Model-Based Domain Generalization
Authors: Alexander Robey
George J. Pappas
Hamed Hassani
Venue: NeurIPS 2021 Poster (2021)
Keywords: Domain generalization
out-of-distribution robustness
constrained optimization

Abstract: Despite remarkable success in a variety of applications, it is well-known that deep learning can fail catastrophically when presented with out-of-distribution data.  Toward addressing this challenge, we consider the \emph{domain generalization} problem, wherein predictors are trained using data drawn from a family of related training domains and then evaluated on a distinct and unseen test domain.  We show that under a natural model of data generation and a concomitant invariance condition, the domain generalization problem is equivalent to an infinite-dimensional constrained statistical learning problem; this problem forms the basis of our approach, which we call Model-Based Domain Generalization.  Due to the inherent challenges in solving constrained optimization problems in deep learning, we exploit nonconvex duality theory to develop unconstrained relaxations of this statistical problem with tight bounds on the duality gap.  Based on this theoretical motivation, we propose a novel domain generalization algorithm with convergence guarantees.   In our experiments, we report improvements of up to 30% over state-of-the-art domain generalization baselines on several benchmarks including ColoredMNIST, Camelyon17-WILDS, FMoW-WILDS, and PACS.

Model-Based Domain Generalization
Alexander Robey
George J. Pappas Hamed Hassani
Department of Electrical and Systems Engineering
University of Pennsylvania {arobey1, pappasg, hassani paw.ppppaa..assaaaaeeeeee.eeee
upenn. edu
Abstract
Despite remarkable success in a variety of applications, it is well-known that deep learning can fail catastrophically when presented with out-of-distribttion data. Toward addressing this challenge, we consider the domain generalization problem, wherein predictors are trained using data drawn from a family of related training domains and then evaluated on distinct and unseen test domain. We show that under a natural model of data generation and a concomitant invariance condition, the domain generalization problem is equivalent to aefinite-dimensioon con- strained statistical learning problem; this problem forms the basis of our approach, which we call Model-Based Domain Generalization. Due the inherent chal- lenges in solving constrained optimization problems in deep learning, we exploit nonconvex duality theory develop unconstrained relaxations of this statistical problem with tight bounds on the duality gap. Based on this theoretical motivation, we propose a novel domain generalization algorithm with convergence guarantees. In our experiments, we report improvements of up to 30% over state-of-the-art domain generalization baselines on several benchmarks including ColoredMNIST, Camelyon17-WW 7-WILDS FMoW-WILDS, and PACS, Our code is publicly available the following link: htttp://ggttae com arobey//bbby/
1 Introduction
Despite well-documented success in numerous applications 1+3 the complex prediction rules learned by modern machine learning methods can fail catastrophically when presented with out- of-distribution (OOD) data [5 Indeed, rapidly growing bodies of work conclusively show that state-of-the-ar methods are vulnerable to distributional shifts arising from spurious correlations
12-12) adversarial attacks [37 sub-populations [8 21. and naturally-occurring variation 24 This failure mode is particularly pernicious in safety-critical applications, wherein the
shifts that arise in fields such medical imaging 255 28) autonomous driving 24 311, and roboties 32 are known to lead to unsafe behavior. And while some progress has been made toward addressing these vulnerabilities, the inability of modern machine learning methods generalize to OOD data is one of the most significant barriers to deployment in safety-criticll applications 335 36
In the last decade, the domain generalization community has emerged in effort improve the OOD performance of machine learning methods [37] 500 In this field, predictors are trained on data drawn from family of related training domains and then evaluated domain. Although a variety of approaches have been proposed for this setting 44 42]. it was recently shown that that existing domain generalization algorithm can significantly outperform empirical risk minimization (ERM) 4 over the training domains when ERM is properly tuned and equipped with ofte--of-the-ar architectures 44] 45 and data augmentation techniques 40 Therefore, due to the prevalence of OOD data in safety critical applications, it is of the utmost importance that new algorithms be proposed which can improve the OOD performance of machine learning methods.
distinct and unseen test
In this paper, we introduce a new framework for domain generalization which we call Model-Bavee Domain Generalization (MBDG). The key idea in framework is to first learn transformations
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

that map data between domains and then to subsequently enforce invariance to these transformations. Under a general model of covariate shift and a novel notion of invariance to learned transformations, we use this framework to rigorously re-formulate the domain generalization problem as a semi-infinite constrained optimization problem. We then use this re-formulation to prove that a tight approximation of the domain generalization problem can be obtained by solving the empirical, parameterized dual for this semi-infinite problem. Finally, motivated by these theoretical insights, we propose a new algorithm for domain generalization; extensive experimental evidence shows that our algorithm advances the state-of-the-art on a range of benchmarks by up to thirty percentage points.
Contributions. Our contributions can be summarized as follows:
We propose a new framework for domain generalization which invariance is enforced to underlying transformations of data which capture inter-domain variation.
Under a general model of covariate shift, we rigorously prove the equivalence of the domain generalization problem to novel semi-infinite constrained statistical learning problem. We derive data vata-dependent duality gap bounds for the empirical parameterized dual of this semi semi-infinite problem. proving that tight approximations of the domain generalization problem can be obtained by solving this dual problem under the covariate shift assumption.
We introduce primal-dual style algorithm for domain generalization in which invariance is enforced over unsupervised generative models trained on data from the training domains.
We empirically show that our algorithm significantly outperforms state-of-the-art baselines on several standard benchmarks, including ColoredMNIST, Camelyon -WILDD and PACS.
2 Related work
Domain generalization. The rapid acceleration of domain generalization research has led to an abundance of principled algorithms, many of which distill knowledge from an array of disparate fields toward resolving OOD failure modes 47 50 Among such works, one prominent thrust has been to
learn predictors which have internal feature representations that are consistent across domains [51-
62]. This approach is also popular in the field of unsupervised domain adaptation (63467), wherein it is assumed that unlabeled data from the test domain is available during training 668770 Also related
are works that seek to learn kernel-based embedding of each domain in an underlying feature space
7 72.. and those that employ Model-Agnostic Meta Learning [3 to adapt to unseen domains
42
14 XI] Recently, another prominent direction has been to design weight-sharing
[82|85] and
instance re-weighting schemes [86888) 88] Unlike any of these approaches, we explicitly enforce hard invariance-based constraints on the underlying statistical domain generalization problem.
Data augmentation. Another approach to improve OOD performance to augment the available training data. Among such methods, perhaps the most common is to leverage various forms of data augmentation (89-96) Recently, several approaches have used style-transfer techniques and image- to-image translation networks [97-1104| to augment the training domains with artificially-genenaa data [105) Alternatively, rather than generating new data, 113 all remove textural features in the data to encourage domain invariance. Unlike the majority of these works, we do not perform data augmentation directly on the training objective; rather, we derive a principled primal-dual style algorithm which enforces invariance constraints on data generated by unsupervised generative models.
3 Domain generalization
The domain generalization setting is characterized by pair of random variables (X, Y) over instances E X C Rd and corresponding labels y E Y, where (X, Y) is jointly distributed according to an unknown probability distribution YY. Ultimately, the objective in this setting is to learn predictor such that f(X) s Y. meaning that f should be able to predict the labels of corresponding instances x for each (x, 3) ~ P(X,Y) However, unlike in standard supervised learning tasks, the domain generalization problem is complicated by the assumption that one cannot sample directly from P(X,Y) Rather, it is assumed that we can only measure (X, Y) under different environmental conditions, each of which corrupts varies the data in different way. For example, in medical imaging tasks, these environmental conditions might correspond to the imaging techniques and stain patterns used at different hospitals; this is illustrated in Figure
To formalize this notion of environmental variation, we assume that data is drawn from a set of environments domains Eal (see Figure lb Concretely, each domain E Eall can be identified with
2

---

Training Arimatss
unobserved
Validarion
Staptta's
X
Hospital
R-
X G
Eall
domain
Hagital 5
Hanpitall
X
X
X
observed
observed with
with ds
(a) In domain generalization, the
(b) Each data point in domain generalization task is observed in particular domain e &au. The set of all domains Eall can be thought of as an abstract space lying in IR. In Camelyon17 1---1IDWDD this space Eall corresponds the of all possible hospitals.
(c) We assume that the variation from domain to domain is charac- terized by an underlying genera- tive model G(z,, e). which trans- forms the unobserved random vari- able X + G(X, e) == X where X" represents X observed in any domain e Emn.
data drawn from family of re- lated domains. For example, the Camelyoni7-W dataset [20]. which contains images of cells, the domains correspond to different hospitals where these images were captured.
Figure 1: An overview of the domain generalization problem setting used in this paper.
pair of random variables (X', y²), which together denote the observation of the random variable pair (X.Y) in environment e. Given samples from a finite subset Euain Ç Eall of domains, the goal of the domain generalization problem is to learn predictor f that generalizes across all possible environments, implying that f(x) R Y. This can be summarized as follows:
Problem 3.1 (Domain generalization). Let Euain Ç Eall be a finite subset of training domains, and assume that for each E &train have access to dataset D ((rf,, )= sampled i.i.d. from F(X Y*). Given function class F and loss function & y x y + R20. our goal is learn a predictor f E F using the data from the datasets DE that minimizes the worst-case risk over the entire family of domains Enn. That is, we want to solve the following optimization problem:
minimize max Ep(x,,y.) (((()))(()))))))))))) Y.). JEF
(DG)
e
In essence, in Problem 3.1 we seek a predictor f E F that generalizes from the finite set of training domains Erain to perform well on the set of all domains Eall However, note that while the inner maximization in (DG) is over the set of all training domains &all by assumption we do not have access to data from any the domains e Ean' Erain making this problem challenging to solve. Indeed, as generalizing to arbitrary test domains is impossible 66.) further structure is often assumed on the topology of Em and on the corresponding distributions P().,Y").
Disentangling the sources of variation across euvironments. generalization task can be characterized by the extent to which the distribution of data in the unseen test domains Ean't Erain resembles the distribution of data in the training domains Emain- For instance, if the domains are assumed to be convex combinations of the training domains, as is often the case in multi-source domain generalization (177)199, Problem 3.1 can be seen as an instance of
The difficulty of a particular domain
distributionally robust optimization [10). More generally, similar spirit to [TC. we identify two forms of variation across domains: covariate shift and concept shift. These shifts characterize the extent to which the marginal distributions over instances IPX*)) and the instance-conditional distributions IP( ((* differ between domains. We capture these shifts in the following definition:
Definition 3.2 (Covariate shift & concept shift). Problem 3.1 is said experience covariate shift if environmental variation due to differences between the set of marginal distributions over instances { On the other hand, Problem 3.1 is said to experience concept shift if environmental variation is due to changes amongst the instance-conditiona distributions (P(Y* (X*) )
The growing domain generalization literature encompasses great deal of past work, wherein both of these shifts have been studied various contexts 12111125 Indeed, as this literature has grown, new benchmarks have been developed which span the gamut between covariate and concept shift [126]. However, a large-scale empirical study recently showed that no existing algorithm can significantly outperform ERM across these standard domain generalization benchmarks when ERM is carefully
3

---

implemented 46 As ERM is known to fail in the presence natural distribution shifts 127.. this result highlights the critical need for new algorithms that can go beyond ERM toward solving Problem G.T
4 Model-based domain generalization
In what follows, we introduce a new framework for domain generalization that call Mlodel-Based Domain Generalization (MBDG). In particular, we prove that when Problem [.1 is characterized solely by covariate shift, then under a natural invariance-base based condition, Problem is equivalent to an diinite-dimensionall constrained statistical learning problem, which forms the basis of MBDG.
Formal assumptions for MBDG. In general, domain generalization tasks can be characterized by both covariate and concept shift. However, in this paper, we restrict the scope of our theoretical analysis to focus on problems in which inter-domain variation is due solely to covariate shift through an underlying model of data generation. Formally, we assume that the data in each domain e E Eall is generated from the underlying random variable pair (X, Y) via an unknown function G.
Assumption 4.1. Let de denote Dirac distribution for E Eall- We assume that there exist: F measurable function G X' x Eall + X, which we refer to as a domain transformation model, that parameterizes the inter-domain covariate shift via P(X*) = G # (IP(X) x de) Ve E Eall, where # denotes the push- forward measure and = denotes equality in distribution.
Informally, this assumption specifies that there should exist a function G that relates the random variables X and X via X + e(X,e) = X In past work, this particular setting in which the instances X measured in an environment e are related the underlying random variable X has been referred to as domain shift [128] $1.8]. In our medical imaging example, the domain shift captured by a domain transformation model would characterize the mapping from the underlying distribution P(X) over different cells to the distribution P(X*) of images of these cells observed at particular hospital; this is illustrated in Figure Ic wherein inter-domain variation is due to varying colors and stain patterns encountered at different hospitals. On the other hand, in this example example, the label y Y describing whether a given cell contains a cancerous tumor should not depend on the lighting and stain patterns used at different hospitals. In this sense, while in other applications, e.g. the datasets introduced [0. the instance-conditional distributions can vary across domains, in this paper we assume that inter-domain variation is solely characterized by the domain shift due to G.
Assumption 4.2 (Domain shift). We assume that inter-domain variation is solely characterized by domain shift in the marginal distributions IPX))), as described in Assumption 4.1 As a consequence, we assume that the destance-conditionl distributions P(Y+I**) are stable across domains, meaning that y and are equivalent in distribution and that for each x E X and E V, it holds that
P(Y- = y| = x) = P( = X| = G( e)) Ve E Ean-
(I)
Pulling back Problem 3.1
The structure imposed on Problem 3.1 by Assumptions and 4.2 pro-
vides a concrete way of parameterizing large families of distributional shifts in domain generalization problems. Indeed, the utility of these assumptions is that when taken together, they provide the basis for pulling-back Problem onto the underlying distribution P.X, Y) via the domain transformation model G. This insight is captured in the following proposition:
Proposition 4.3. Under Assumptions
4.1 and 4.2 Problem 3.] is equivalent to
minimize max Fx(X,) ((((((X,))))) JEF
eEEn
The proof of this fact is straightforward consequence of the decomposition P(X., (Y, P(Y P(X*) in conjunction with Assumptions 4.1 and 4.2 (see Appendix C.2 Note that this result allows us to implicitly absorb each of the domain distributions IPXX", Y into the domain transformation model. Thus, the outer expectation in is defined over the underlying distribution PXX, Y). On the other hand, just as in (GG,, this problem is still challenging statistical min-max problem. To this end, we next introduce a new notion of invariance with respect to domain transformation models, which allows us reformulate the problem in 2 semi-infinite constrained optimization problem.
Crrucillly, although we assume the existence of a domain transformation model G. we emphasize that for many problems, it may be impossible to obtain or derive a simple analytic expression for G. This topic will be discussed at length in Section and in Appendix G

---

A new notion of model-based invariance. Common to much of the domain generalization literature the idea that predictors should be invariant to inter-domain changes. For instance. in [TQ] the authors seek to learn an equipredictive representation Φ X - 2 [19]] i.e. an intermediate representation that satisfies P(Y=: (xx+)) = motivation for this approach, it has been shown that current algorithms which seek equipredictive representations do not significantly improve over ERM
(x(==))) Vei, e2 E Ean. Despite compelling theoretical
330 133] With this in mind and given and 4.2 we introduce new definition of
the additional structure introduced in Assumptions invariance with respect to the variation captured by the underlying domain transformation model G.
Definition 4.4 (G-invariance). Given a domain transformation model G. we say a classifier f is G-invariant if it holds for all E Ean that f()) = f(G(z, e)) almost surely when ~ IP(X).
Concretely, this definition says that predictor f is G-invariant environmental changes under G(I,e) cannot change the prediction returned by f. Intuitively, this notion of invariance couples with the definition of domain shift, in the sense that we expect that a prediction should return the same prediction for any realization of data under G. Thus, whereas equipredictive representations are designed to enforce invariance of an intermediate representation space 2, Definition 4.4 is designed to enforce invariance directly on the predictions made by f. In this way. in the setting of Figure D. G-invariance would imply that the predictor f would return the same label for given cluster of cells regardless of the hospital at which these cells were imaged.
Formulating the MBDG optimization problem. The G-invariance property described in the previous section is the key toward reformulating the min-max problem in 2 Indeed, the following proposition follows from Assumptions
4.1
and
4.2
and from the definition of G-invariance.
Proposition 4.5. Under Assumptions 4.1 and 4.2
if we restrict the domain F of Problem 3.1 the
set of G-invariant predictors, then Problem
3.1 is equivalent to the following constrained problem:
p - minimize JeF
R(f) Ep(x,)) ((f(x),))
(MBDG)
subject to
a.c. x P(X) Ve E Eall
Here a.e. stands for "almost everywhere" and R(f) is the statistical risk of predictor with respect to the underlying random variable pair (X,Y). Note that unlike (2. MBDG is not a composite optimization problem, meaning that the inner maximization has been eliminated. In essence, the proof of Proposition r.oree on the fact that G-invariance implies that predictions should not change across domains (see Appendix Model-Based Domain Generalization framework. To explicitly contrast this problem to Problem 3.1 we introduce the following concrete problem formulation for Model-Based Domain Generalization.
The optimization problem in MBDG forms the basis of our
Problem 4.6 -Model-Based Domain Generalization), As in Problem 3.1 let Eusin S Eall be finite subset of training domains and assume that we have access to datasets De Ve E &Erain- Then under Assumptions and 4.2 the goal of Model-Based Domain Generalization is to use the data from the training datasets to solve the semi-infinite constrained optimization problem in [MDGG.
Problem 4.6 offers principled perspective on Problem when data varies WRT underlying domain transformation model. However, just as solving the min-max problem of Problem 3.1 is known to be difficult, the problem in (MBDG) is also challenging to solve for several reasons:
(CI) The G-invariance constraint in MBDG) is strict and thus challenging to enforce.
Problem is constrained problem over an infiniteddimensional functional space F.
We do have access to the set of all domains Eall or to the underlying distribution P(X., Y). (C4) We also generally do not have access to the underlying domain transformation model G.
In the ensuing sections, we explicitly address each of these challenges toward developing a tractable method for approximately solving Problem 4.6 with guarantees on optimality. In particular, we discuss challenges (CI), (C2), and (C3) in Section 5 We then discuss (C4) in Section F
5 Data-dependent duality gap for MBDG
In this section, we offer principled analysis of Problem 4.6
particular, we first address (C1) by introducing tight relaxation of the G-invariance constraint. Next, to resolve the fundamental
5

---

difficulty involved in solving constrained statistical problems highlighted in (C2), we formulate the parameterized dual problem. which is unconstrained and thus more suitable for learning with deep neural networks. Finally, to address (C3), we introduce an empirical version of the parameterized dual problem and explicitly characterize the data-dependent duality gap between this problem and Problem 4.6 At high level, this analysis results in an unconstrained optimization problem which is guaranteed to produce a solution that is close to the solution of Problem 3.1 (see Theorem 5.3). In this section, we have chosen to present our results somewhat informally by deferring preliminary results, regularity assumptions, and proofs to the appendices.
A Addressng (C1) by relaxing the G-invariance constraint. One of the most fundamental challenges in solving Problem 4.6 is the difficulty of enforcing the G-invariance equality constraint. To alleviate some of this difficulty, we introduce the following relaxation of Problem 4.6
(7) - minimize R(f) s.t. (f) A Ep(x) FEF
Sy Ve E Eill
(3)
where 7 >> 0 is fixed margin the controls the extent to which we enforce G-invariance and d P()) x P()) + Rzo is a distance metric over the space of probability distributions on V. While at first glance this problem may appear to be a significant relaxation of the MBDG optimization problem in (MDDG.. when y = O and under mild conditions on d, the two problems are equivalent in the sense that PT(O) (0) p* (see Proposition B.I Indeed, we note that the conditions we require on d are not restrictive, and include the KL-divergence and more generally the family of f f-divergences. Moreover, when the margin 7 strictly larger than zero, under the assumption that the perturbation
function P (r) is L-Lipschitz continuous, we show in Remark B.2 that [P* - (1)) < Ly, meaning that the gap between the problems is relatively small when y is chosen to be small. In particular, when strong duality holds for (MBDG). this Lipschitz constant L is equal to the L' norm of the optimal dual variable for (MBDG) (see Remark B.4).
Addressing (C2) by formulating the parameterized dual problem. As written, the relaxation in B is an L or the space of continuous functions). Optimization in this infinice-dimensional function space not tractable, and thus we follow the standard convention by leveraging a finite-dimensional parameterization of F, such as the class of deep neural networks [T34| 1335. The approximation power of such a parameterization can be captured in the following definition:
constrained optimization problem over functional space F (e.g-
Definition 5.1 (--parameterization). Let H C RP be a finite-dimensie
parameter space. For
a function y H x X' - y is said to be an e-parameterization of F if it holds that for each f E F, there exists parameter E H. such that Ex(x) [p(, x)) f(x)]l S
The benefit of using such a parameterization is that optimization is generally more tractable in the parameterized space A, == (x((0.) 0 H) C F. However, typical parameterizations often lead to nonconvex problems, wherein methods such as SGD cannot guarantee constraint satisfaction. And while several heuristic algorithms have been designed to enforce constraints over common parametric classes [136 [136 [41] these approaches cannot provide guarantees on the underlying statistical problem of interest 142 Thus, to provide guarantees on the underlying statistical problem in Problem 4.6
given an e-parameterization yp of F, we consider the following saddle-point problem:
/ [C (c)
(1) A maximize min R(0) + XEP(E..) 8EH
(4)
Em
where P(Ea)) is the space of normalized probability distributions over Eall and À E P(Em)) is the beemi-infinite) dual variable. Here we have slightly abused notation to write R(A)= = R(p((f -)) and (0) L -)). One think of 4 as the dual problem to 3 solved over the parametric space Ar- Notice that unlike Problem 4.6 the problem in 4 is wnconstrained, making it much more amenable for optimization over the class of deep neural networks. Moreover, under mild conditions, the optimality gap between and can be explicitly bounded as follows:
Proposition 5.2 (Parameterization gap). Let 7 > O be given. Under mild regularity assumptions (see Assumption C.1 in Appendix C.3 on l and d, there exists small universal constant l such that
P*(7) < D261) < P*))) + ek (1 || 1:))
(5)
where Xion is the optimal dual variable for a perturbed version of in which the constraints are tightened to hold with margin y ke.

---

Algorithm Model-Based Domain Generalization (MBDG)
1: Hyperparameters: Primal step size fhy > O, dual step size T) 2 0, margin >> 0
2: repeat
3: for minibatch {(r,, ,5 )3= in training dataset do
4: ē - GENERATEIMAGE(RA vj [m]]
> Generate model-based images
5: distReg(0) 6 (1/m) دز d(p((, 2z)) (00,
Calculate distance regularizer
6:
(0) t (1/m) Σ j=1 & (1 p(),
> Calculate classification loss
7:
- 0 - Tp Val loss (0) + distReg
>> Primal step for
8:
A [A + TJ (d) -7]1+
Dual step for
9: end for
until convergence
11:
procedure GENERATEIMAGE(S)
13:
Sample e ~ (0, return ((r,e)
é is a latent code for MUNIT
b Return image produced by MUNIT
15:
end procedure
In this way, solving the parameterized dual problem in provides solution that can be used to recover a close approximation
the primal problem in B To see this, observe that Prop. 5.2 implies that (7) (*)) < ck(1 + |e ll.)). This tells that the gap between P* (r) and D (7) is small when we use a tight E-parameterization of F.
Addressing (C3) by bounding the empirical duality gap. The parameterized dual problem in gives us a principled way to address Problem 4.6 the context of deep learning. However, complicat- ing matters is the fact that we do not have access to the full distribution IPXX, Y) to data from any of the domains in Ean't Erain In practice, it is ubiquitous to solve optimization problems such as over finite sample of N data points drawn from P(X, i More specifically, given {(zz, #;)} drawn i. according to (X,Y) we consider the empirical counterpart of
Dr N, Emin (x) - maximize min Â(B, A) 2 R()))
Σ [C) 1100-10 e'Euusn
(0(0) X(e)
(6)
Xe)20, eEEmn BEH
Egmin
where k(() and (0) are the empirical counterparts of R(f) and C(f). Notably, the duality gap between the solution to (6) and (MBDG can be explicitly bounded as follows.
Theorem 5.3 (Data-dependent duality gap). Let >> 0 be given, and let yp be an c-parameterization of F. Under mild regularity assumptions on l and d and assuming that A, has finite VC-dimension, with probability 8 over the N samples from IP(X, Y) we have that
P NE (7) S Toy + ek (1 || (1 + (Viog(N)/V)
where L is the Lipschitz constant of P*(1) and le and Xpon are as defined in Proposition 5.2
The key message to take away from Theorem 5.3 is that given samples from P(X, Y), the duality gap incurred by solving the empirical problem in 6 is small when (a) the G-invariance margin is small, (b) the parametric space A, is a close approximation many samples. Thus, assuming that Assumptions 4.1 and hold, the solution to Problem 3.1 is closely-ape apssel--pproximmate by the solution to the empirical, parameterized dual problem in b
F, and (c) we have access to sufficiently
MBDG: A principled algorithm for domain generalization
Motivated by these theoretical insights, we now introduce a new domain generalization algorithm which is widely applicable to problems with or without covariate shift. Our algorithm consists of two steps. First, we learn an approximation of the underlying domain transformation model G(x, e) using the data from the training domains Eurain Next, leverage G toward solving the unconstrained dual optimization problem
6
via a primal-dual iteration.
Inddee, in practice we do not have access to any samples from P(X, Y). In Section 6 we argue that the N samples from [P Y) can be replaced by the Lecemi ne samples drawn from the training datasets 2)_
7

---

Learning domain transformation models from data. Regarding challenge (C4), critical to our approach is having access to the domain transformation model G. For the vast majority settings, the underlying function G(ar, e) is not known priori and cannot be represented by simple expression. For example, obtaining a closed-form expression for model that captures the variation in coloration, brightness, and contrast in the dataset shown in Figure I it is impossible to learn the true underlying domain transformation model when one only has access to data from the training domains, we argue that realistic approximation of the underlying model can be learned from this data. To this end, to learn a domain transformation model, we train multimodal image-to-ing tomaa translation networks on the training data. These networks are designed to transform samples from one dataset so that they resemble a diverse collection of images from another dataset. In particular, in each of the experiments in Section / use the MUNIT architecture introduced in 1002 to parameterize learned domain transformation models. As shown in Figure 5 and in Appendix models trained using the MUNIT architecture learn accurate and diverse transformations of the training data, which often generalize to generate images from new domains.
would be very challenging. While in general
Primal-dual iteration. Given learned approximation G(x, E) of the underlying domain transfor- mation model, the next step in our procedure is to use a primal-dual iteration [E3] toward solving 6 using the datasets D We note that while our theory calls for data drawn from IPXX, Y). the G-invariance condition implies that when O is feasible, (o x) as (o(0, x²) when x ~ IP(X), I ~ P*(X), and x" = G(z, e). Therefore, the data from LeEE D is a useful proxy for data drawn from P(X, Y). As the outer maximization in C is a linear program in X, the primal-dual iteration can be characterized by alternating between the following steps:
g(t+1) E p- argmin A(B,A(0) PE
(8)
((+) + (x) -n (2:00) -000--))
(9)
Here H++ = ma +}. 17 is the dual step size. and p- argmin denotes solution that p-close to being minimizer, i.e. we should that have A(g(++1), (()) S minoeæ (0,,(()) + p. We call ( the primal step, and we call ( the dual step. Furthermore, it can be shown that if this iteration is run for sufficiently many steps and with small enough step size, the iteration convergences with high probability to a solution which closely approximates the solution to Problem 4.6
Theorem 6.1 (Primal-dual convergence). Assuming that and are [0, B) -boudded H has finite VC-dimension, and under mild regularity conditions on Oo. the primal-dual pair (0(7), ((T) obtained after running the alternating primal-dual iteration in and for T steps with step size 7, where
1 27p
2k
T
+
and
1 Evrin B
(10)
satisfies IP - A(g(T), 1 ()) < K(p,K,c) + O( O(Vlog(N)/N) Here i is constant that captures the regularity of 66) (see Appendix C.6) and ((p, Ki, c) is a small constant depending on p. K, and c.
This means that by solving the empirical dual problem for sufficiently many steps, we can reach solution that is close to solving the Model-Based Domain Generalization problem in Problem 4.6
Implementation of MBDG. In practice, because (a) it may not be tractable to find a p-minimizer over H. at each iteration and (b) there may be a large number of domains in Evain. we propose two modifications of the primal-dual iteration in which we replace { with a stochastic gradient step and we use only one dual variable for all of the domains; we call this algorithm MBDG (see Algorithm M We provide results in Appendix E where one dual variable is used per training domain.
7 Experiments
We now evaluate the performance of MBDG on a range of standard domain generalization benchmarks. In the main text, we present results on ColoredMNIST, Camelyon17-WILDS, FMON-WILDS and PACS; we defer results on VLCS the supplemental. For ColoredMNIST, PACS, and VLCS, we use the DomainBed package Hd. facilitating comparison to a range of baselines. Model selection for each of these datasets was performed using hold-one-out cross-validation. For Camelyon17-WIILD and WILDS, we use the repository provided with the WILDS dataset suite, and we perform model-selection using the dustr--diiti validation set provided in the WILDS repository. Further details concerning hyperparameter tuning and model selection are deferred to Appendix
8

---

Table I: ColOredMNIST. We report accuracies for ColoredM- NIST. Model-selection was performed via cross-validation
MBDG on ColoredMNIST
0.050
distReg(0)
Algorithm
+90%
+80%
-90%
Avg
0.025
ERM
50.0 - 0.2
50.1 = 0.2
10.0 + 0.0
36.7
IRM
46.7 1: 2.4
$1.2
12.1..3 0.3
23.1 1 10.7
40.3
GroupDRO
50.1
0.5
10.2
0.000
Mixup
36.6 10.9
53.4
1..
33.4
MLDG
0.6
5.31
1.10
36.7
Dual Variable
CORAL
1.0
192
10.2
39.7
MMD
50.3
5..0
9.9
36.8
+90%
DANN
49.9
6..1
10.0
40.7
+80%
CDANN
63.2 10.1
4.5
9.2:.22
39.1
90%
MTL
4.9
50.7
1..1
35.0
SagNet
4..9
4..7
1..0
36.5
1000
2000
3000
ARM
5.3
0.3
10.2
36.8
Step
VREx
0.4
50.5
1.0
RSC
49.6
0.47
10.1 + 0.0
36.5
Figure 2:
Tracking dual variables.
MBDA
+ 0.1
50.7 + 0.1
22.5
48.3
We show the values of distReg (0) and the dual variables À for the each MBDG models in Table M The mar-
MBDG-DA
72.7 0.2
71.4 0.1
3.1
MBDG-Reg
73.3
73.7 +
0.1
58.1
MBDG
73.7 + 0.1
68.4 0.0
63.5 0.0
68.5
gin 7 0.025 is shown in red.
7.1
ColoredMNIST
MBDG-Reg on ColOredMNIST
We first consider the ColoredMNIST dataset 10], which is a standard domain generalization benchmark created by colorizing subsets of the MNIST dataset [E4]. This dataset contains three domains, each of which is characterized by different level of correlation between the label and digit color. As shown Table each baseline by nearly
0.06
distReg(())
+90%
0.04
0.02
-90%
I the MBDG algorithm improves 30%. To understand the reasons behind this improvement, we consider three ablation studies on ColoredMNIST.
0.00
1000
2000
3000
Step
Figure 3: Regularized MBDG. We show the regularization value for each domain in ColoredMNIST for fixed dual variable X = 1.0.
Tracking the dual variables. For the three MBDG classifiers in Table I we plot the regularization term distReg(O) and the corresponding dual variable at each training step in Figure 2 Observe that for the +90% and +80% domains, the dual variables decay to zero, as the constraint is satisfied early on in training.
On the other hand, the constraint for the -90% domain is not satisfied early on in training, and in response, the dual variable increases, gradually foreing constraint satisfaction. As we shown in the next subsection, without the dual update step, the constraints may never be satisfied (see Figure 3
common trick for encouraging constraint satisfaction to introduce soft constraints by adding regularizer multiplied by a fixed multiplier to the objective. While this approach yields a related problem to (see Appendix B.4] where the dual variables are fixed, there are few formal guarantees for this approach. Moreover, we show in Table I that when the dual variable is fixed during training (MBDG-Reg in Table D the performance drops significantly vis-a-vis MBDG. Notice that relative Figure 2 the value of distReg(O) is much larger than the margin, meaning that the constraint is not being satisfied.
Regularization vs. dual ascent.
Ablation on data augmentation. To study the efficacy of the MBDG algorithm, we consider two natural alternatives MBDG: (1) ERM with data augmentation through the learned model G(z, e) (MBDA); and (2) MBDG with data augmentation through G(x, e) on the training objective (MBDG- DA). shown at the bottom of Table I while these variants significantly outperform the baselines, they not perform nearly as well as MBDG. Thus, while data augmentation can in some cases improve performance, the primal-dual iteration a much more effective tool for enforcing invariance.
Camelyon 1oanelyon7-WWI.DD W--WID and FMOW-WILDS
We next consider the Hamelyom17-WIIDD and FMoN-WILDS datasets from the WILDS family of domain generalization baselines 20) Table shows that on Comelyon17-WILDD, MBDG improves by more than 20 percentage points the state-of-the-art baselines.

---

100
Models for Camelyon17
Original
Sumples from domun tranafiemation G(sur)
Test Accuracy
Gaselyca17 WILDS
90)
80
PAN--
70
WILDS
B+C
CJ
RA RA RAMUNIT (Geom)
(Color)
PACE
Figure Known vs. learned mod- els G(x,e) e). We compare the perfor- mance of MBDG for known models (first five columns) against model that was trained with the data from the training domains using MUNIT.
Figure 5: Samples from learned models G e). We show samples from domain transformation models learned from the training data via the MUNIT architecture for Camelyyn11-WWILDS, FMOW-WILDS, and PACS.
Distribution of ((
Table WILDS datasets. We report accuracies for Camelyon17 and FMoW. For both datasets, we use the distrf-distribbution validation set provided in the WILDS repository to perform model selection.
Regularizer
Algorithm Camelyon17-WII.DS
FMoW-WILDS
ERM
73.3 1 9.9
51.3 (0.4)
ERM
IRM
MBDG
IRM
60.9 1 15.3
51.1 (0.4)
Training algorithm
ARM
62.1 1 6.4
47.9 (0.3)
Figure 6: Measuing invariance. We mea- sure the invariance of ERM, IRM, and MBDG to images generated by a model G learned for Camelyon17-WIIDD
CORAL
59.2 15.1
49.6 (0.5)
MBDG
94.8 0.4
52.3 = 0.5
Measuring G-invariance of trained classifiers. Section we restricted our attention predictors satisfying the G-invariance condition. To test whether our algorithm successfully enforces G- invariance when G is learned from data, we measure the distribution of distReg (f) over all of the instances from the training domains of Gamelyon17-WILD for ERM, IRM, and MBDG. In Figure g observe that whereas MBDG quite robust to changes under G, ERM and IRM are not nearly as robust. This property is key to the ability of MBDG to learn invariant representations across domains.
Ablation on learning models vs. data augmentation. Rather than learning G from data, a heuristic alternative is to replace the GENERATELMAGE procedure in Algorithm | with standard data aug- mentation transformations. In Figure we investigate this approach with five different forms of data augmentation: B+C (brightness and contrast), CJ (color jitter), and three variants of RandAug- ment 1:5 (RA, RA-Geom, and RA-Color). More details concerning these data augmentation schemes are given in Appendix The bars in Figure X show that although these schemes offer strong performance in our MBDG framework, the learned model trained using MUNIT performs best.
7.3 PACS
In this subsection, we highlight se- lected results for the PACS dataset. Due to spatial limitations, we re- port the top-performing baselines the main text, and defer the full set of results to Appendix D Notably, MBDG beats the current SOTA by nearly 2% when averaged over the four domains. Of note in Table is the result on the "sketch" t (S) subset, wherein MBDG improves by more than 10% over the baselines.
Table PACS. We report classification accuracies for PACS. Model-selection was performed via cross-validation.
Algorithm
A
C
Avg
ERM
83.2
1.3
76.8
1.7
97.2
0.3
74.8
83.0
MTL RSC
85.6 1.5
78.9
0.6
97.1
= 0.3
73.1 2.7
83.7
83.7
1.7
82.9
1.1
95.6
+ 0.7
68.1 + 1.5
82.6
MBDG
80.6 1.1
79.3 0.2
97.0 = 0.4
85.2 =
85.6
10

---

8 Acknowledgements and disclosure of funding
The authors would like to thank Samuel Sokota for his helpful feedback.
The research of Alexander Robey and Hamed Hassani is supported by NSF Grants 1837253, 1943064, 1934876, AFOSR Grant FA9550-20-1-4 10-11111,, and DCIST-CRA. Alexander Robey and George J. Pappas are supported by the NSF-Simons Program on the Mathematics of Deep Learning, ARL CRA DCIST W911NF-17-2-0181 and NSF 2038873.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(C) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you' rou're using/curating? [N/A] All of the external code used for this project is open-sourced and freely-available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
11

---

References
[1] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning, nature, 521(75533:436-444, 2015.
[2] Carlos Esteves, Christine Allen-Blanchen Xiaowei Zhou, and Kostas Daniilidis. Polar transformer networks. arXiv preprint arXiv:1709.01889, 2017.
[3] Carlos Esteves, Christine Allen- Blancheean,,,te,, Ameesh Makadia, and Kostas Daniilidis Learning so (3) equivariant representations with spherical cnns. Proceedings of the European Conference on Computer Vision (ECCV), pages 52-68, 2018.
[4] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. arXiv preprini arXiv: 02025, 2015.
[5] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint a903:: /2261, 2019.
[6] Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexan- der Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' Amour, Dan Moldovan, et al. On robustness and transferability of convolutional neural networks. arXiv preprint wxXiv:2007.0855, 2020.
[7] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33, 2020.
[8] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of dut-of-distribution generalization, arXiv preprint arXiv:2006. 16241, 2020.
[9] Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 2011, pages 1521-1528. IEEE, 2011.
[10] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint UeXiv:1907.0220 2019.
[11] Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In International Conference on Machine Learning, pages 145-155. PMLR, 2020.
[12] Chaochao Lu, Yuhuai Wu, Jose Miguel Hemándezz-Lobtto and Bernhard Schölkopf. Nonlinear invariant risk minimization: A causal approach. arXiv preprint 1.2.::2202.1235:. 2021.
[13] Battista Biggio, Igino Corona, Davide Maiorea, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databuses, pages 387-402. Springer, 2013.
[14] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. arXiv preprini arXiv: 14/2:6572. 2014.
[15] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 2017.
[16] Eric Wong and J Zico Kolter. Provable Defenses Against Adversarial Examples Via the Convex Outer Adversarial Polytope. arXiv preprint arXiv: 2017.
[17] Edgar Dobriban, Hamed Hassani, David Hong, and Alexander Robey. Provable tradeoffs in adversarially robust classification. arXiv preprint arXiv:2006. 05161, 2020.
[18] Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpop- ulation shift. arXiv preprint arXiv:2008. 04859, 2020.
12

---

[19] Nimit S Sohoni, Jared A Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-gratd Fine-grained robustness grasee-granedd classification problems. arXiv preprint arXiv:201 1.12220111122440 12945, 2020.
[20] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Sara Beery, al. Wilds: A benchmark of in-the-wild distribution shifts. arXiv preprint arXiv:2012.0742) 2020.
[21] Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006.0999) 2020.
[22] Alexander Robey, Hamed Hassani, and George Pappas. Model-based robust deep learning. arXiv preprint arXiv:2005. 10247, 2020.
[23] Eric Wong and J Zico Kolter. Learning perturbation sets for robust machine learning. arXiv preprint arXiv:2007.088: 2007.08000 2020.
[24] Sven Gowal, Chongli Qin, Po-Sen Huang, Taylan Cemgil, Krishnamurthy Dvijotham, Timothy Mann, and Pushmeet Kohli. Achieving robustness in the wild via adversarial mixing with disentangled representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1211-1220, 2020.
[25] Andre Esteva, Alexandre Robicquet, Bharath Ramsundar, Volodymyr Kuleshov, Mark De- Pristo, Katherine Chou, Claire Cui, Greg Corrado, Sebastian Thrun, and Jeff Dean. A guide to deep learning in healthcare. Nature medicine,
2019.
[26] Li Yao, Jordan Prosky, Ben Covington, and Kevin Lyman. A strong baseline for domain adaptation and generalization in medical imaging. arXiv preprint 19Xi:v1904.0066 2019.
[27] Haoliang Li, YuFei Wang, Renjie Wan, Shiqi Wang, Tie- Tie-Qiang Li, and Alex Kot. Domain generalization for medical imaging classification with linear-dependency regularization. arXiv preprint arXiv:2009. /2829, 2020.
[28] Vishnu M Bashyam, Jimit Doshi, Guray Erus, Dhivya Srinivasan, Ahmed Abdulkadir, Mo- hamad Habes, Yong Fan, Colin L Masters, Paul Maruff, Chuanjun Zhuo et al. Medical image harmonization using deep learning based canonical mapping: Toward robust and generalizable learning in imaging. arXiv preprint arXiv:2010.05355 2020.
[29] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006. /0742, 2020.
[30] Luona Yang. Xiaodan Liang. Tairui Wang, and Eric Xing. Real-to-virtte domain unification for end-to-end autonomous driving. In Proceedings of the European conference computer vision (ECCV), pages 530-545, 2018
[31] Yang Zhang, Philip David, and Boqing Gong. Curriculum domain adaptation for semantic segmentation of urban scenes. In Proceedings of the IEEE International Conference on Computer Vision, pages 2020-2030, 2017.
[32] Ryan Julian, Benjamin Swanson, Gaurav Sukhatme, Sergey Levine, Chelsea Finn, and Karol Hausman. Never stop learning: The effectiveness of fine-tuning in robotic reinforcement learning. arXiv e-prints, pages arXiv-2004, 2020.
[33] Anoopkumar Sonar, Vincent Pacelli, and Anirudha Majumdar. Invariant policy optimization: Towards stronger generalization in reinforcement learning. arXiv preprint arXiv:2006.0100) 2020.
[34] Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, and Alexandre Bayen. Robust reinforcement learning using adversarial populations. arXiv preprint arXiv:2008.01825 2020.
[35] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?" explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD international conference knowledge discovery and data mining, pages 1135-1144, 1144, 2016.
13

---

[36] Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition, 84:317-311, 2018.
[37] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classifi- cation tasks to a new unlabeled sample. Advances in neural information processing systems, 24:2178-211 2166, 2011.
[38] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pages 10-18, 2013.
[39] Gilles Blanchard, Aniket Anand Deshmukh, Urun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning, arXiv preprint arXiv:1711.07910, 2017.
[40] Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. arXiv preprint arXiv:2007002554. 2, 2020.
[41] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pages 443-450. Springer, 2016.
[42] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta- learning for domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[43] Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions on neural networks, 1065):88-9999 1999.
[44] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of IEEE conference on computer vision and pattern recognition, pages 770-778, 2016
[45] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700-4708, 2017.
[46] Ishaan Guirajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434 2020.
[47] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. arXiv preprint arXiv:2103.02503, 2021.
[48] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, and Tao Qin. Generalizing unseen domains: A survey on domain generalization. arXiv preprint arXIv:2103.03097, 2021.
[49] Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint UrXiv:2104.09937 2021.
[50] Alexis Bellot and Mihaela van der Schaar. Accounting for unobserved confounding in domain generalization. arXiv preprint arXiv:2007.10653 2020.
[51] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversann training of neural networks. The journal of machine learning research, 17(1):2096-200 2016.
[52] Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H. Falk, and Ioannis Mitliagkas. Adversarial target-invariant representation learning for domain generalization. arXiv preprint arXiv: 191.1/1110000 1.00804, 2019.
[53] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5400-5409, 2018
14

---

[54] Saeid Motiaan, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. In Proceedings of the IEEE international conference computer vision, pages 5715-5725, 2017.
[55] Muhammad Ghifary, David Balduzzi, W Bastiaan Kleijn, and Mengjie Zhang. Scatter compo- nent analysis: A unified framework for domain adaptation and domain generalization. IEEE transactions on pattern analysis and machine intelligence, 39(7):1414- 1430, 2016.
[56] Shoubo Hu, Kun Zhang, Zhitang Chen, and Laiwan Chan. Domain generalization via multido- main discriminant analysis. In Uncertainty in Artificial Intelligence, pages 292-302. PMLR, 2020.
[57] Maximilian lsse, Jakub M Tomezak, Christos Louizos, and Max Welling. Diva: Domain invariant variational autoencoders. In Medical Imaging with Deep Learning, pages 322-348.
PMLR, 2020.
[58] Kei Akuzawa, Yusuke Iwasawa, and Yutaka Matsuo. Adversarial invariant feature learning with accuracy constraint for domain generalization. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 315-331. Spr-ner. 2019.
[59] Prithvijit Chattopadhyay, Yogesh Balaji, and Judy Hoffman. Learning to balance specificity and invariance for in and out of domain generalization. In European Conference Computer Vision, pages 301-318. Springer, 2020.
[60] Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common-specificc low-rank decomposition. In International Conference on Machine Learning, pages 7728-7738. 7738. PMLR, 2020.
[61] Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via gross-gradient training. arXiv preprint arXiv:1804. 10745, 2018.
[62] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages
2018.
[63] Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira, et al. Analysis of represen- tations for domain adaptation. Advances in neural information processing systems, 19:137, 2007.
[64] Hal Daumé III. Frustratingly easy domain adaptation. arXiv preprint arXiv:0907. 2009.
[65] Sinno Jialin Pan, Ivor W Tsang, James Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 12(2):199-210) 2010.
[66] Eric Tzeng, Judy Hoffmann, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167-7176, 2017.
[67] Bo Fu, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Learning to detect open classes for universal domain adaptation. In European Conference on Computer Vision, pages 567-583, Springer, 2020.
[68] Vishal M Patel, Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Visual domain adaptation: A survey of recent advances. IEEE signal processing magazine, 32(3):53-69, 2015.
[69] Gabriela Csurka. Domain adaptation for visual applications: A comprehensive survey, arXiv preprint 1xxiv11702.05374, 2017.
[70] Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey, Neurocomputing, 312:135-153: 2018
[71] Abhimanyu Dubey, Vignesh Ramanathan, Alex Pentland, and Dhruv Mahajan. Adaptive methods for rorldworld domain generalization. arXiv preprint 15X7: 2021.
15

---

[72] Aniket Anand Deshmukh, Yunwen Lei, Srinagesh Sharma, Urun Dogan, James W Cutler, and Clayton Scott. A generalization error bound for multi-class domain generalization. arXiv preprint arXiv: /905. 10392, 2019.
[73] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1126-1133 135. PMLR, 2017.
[74] Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain generalization using mettareguularization. Advances in Neural Information Processing Systems, 11:998-1008, 2018.
[75] Qi Dou, Daniel C Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via model-agnostic learning of semantic features. arXiv preprint arXiv:1910.1 2019.
[76] Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M Hospedales. Episodic training for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1446-1455, 2019.
[77] Yang Shu, Zhangjie Cao, Chenyu Wang, Jianmin Wang, and Mingsheng Long. Open domain generalization with domain- meta-learning. arXiv preprint arXiv:2104.0383 2021.
[78] Yiying Li, Yongxin Yang, Wei Zhou, and Timothy Hospedales. Feature-critic networks for heterogeneous domain generalization. In International Conference on Machine Learning, pages 1915-3924. PMLR, 2019.
[79] Yufei Wang, Haoliang Li, and Alex C Kot. Heterogeneous domain generalization via domain mixup. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3622-33 -626.. IEEE, 2020.
[80] Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12556-12565, 2020.
[81] Marvin Zhang, Henrik Marklund, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adap- tive risk minimization: A meta-learning approach for tackling group shift. arXiv preprint arXiv:2007.02931) 2020.
[82] Massimiliano Mancini, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. Robust place categorization with deep domain generalization. IEEE Robotics and Automation Letters, 3(3):2093-2N) 2100, 2018.
[83] Massimiliano Mancini, Samuel Rota Bulò, Barbara Caputo, and Elisa Ricci. Best sources forward: domain generalization through source-specific nets. In 2018 25th IEEE international conference on image processing (ICIP), pages 1353-1357. IEEE, 2018
[84] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pages 5542-5550, 2017.
[85] Zhengming Ding and Yun Fu. Deep domain generalization with structured low-rank constraint. IEEE Transactions on Image Processing, 27(1):304-313, 2017.
[86] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv: 1,07311,,
[87] Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning, pages 2037. PMLR, 2018.
[88] Fredrik D Johansson, Nathan Kallus, Uri Shalit, and David Sontag. Learning weighted representations for generalization across designs. arXiv preprint 180,00880,,0858, 2018.
16

---

[89] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks, Advances in neural information processing systems, 25:1097- 1105, 2012.
[90] Dan Hendrycks. Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: simple data processing method to improve robustness and uncertainty. arXiv preprini arXiv: 1rXiv 2019.
[91] Shuxiao Chen, Edgar Dobriban, and Jane H Lee. Invariance reduces variance: Understanding data augmentation in deep learning and beyond. arXiv preprint arXi::1907.10900 2019.
[92] Ling Zhang, Xiaosong Wang, Dong Yang. Thomas Sanford, Stephanie Harmon, Baris Turkbey, Holger Roth, Andriy Myronenko, Daguang Xu, and Ziyue Xu. When unseen domain gener- alization is unnecessary? rethinking data augmentation. arXiv preprint (88iv:9906.03444
2019.
[93] Riccardo Volpi. Hongseok Namkoong, Ozan Sener, John Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. arXiv preprint arXiv:1805.12011 2018.
[94] Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang. Qi Tian, and Wenjun Zhang Adversarial domain adaptation with domain mixup. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6502-6509, 2020.
[95] Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint Ttt::2000000677, 2020.
[96] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint Xiv::1710009411 2017.
[97] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv:1406.2661 2014.
[98] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
[99] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1809. 1096, 2018.
[100] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolu- tional neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2414 2414-2423, 2016.
[101] Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image trans- lation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages
2017.
[102] Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image- to-image translation. In Proceedings of the European conference on computer vision (ECCV), pages 172-189, 2018
[103] Amjad Almahairi, Sai Rajeswar, Alessandro Sordoni, Philip Bachman, and Aaron Courville. Augmented eyclegan: Learning many-to-many mappings from unpaired data. arXiv preprint arXiv:1802. 10151, 2018
[104] Paolo Russo, Fabio M Carlucci. Tatiana Tommasi, and Barbara Caputo. From source to target and back: symmetric bi-directional adaptive gan. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8099 1199-8108, 8108, 201 8.
[105] Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep adveinaadversariil image generation for domain generalisation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13025-13032 2020.
17

---

[106] Fabio M Carlucci, Antonio D' IIInoceete, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2229-2238, 2019.
[107] Simon Vandenhende, Bert De Brabandere. Davy Neven, and Luc Van Gool. three-player gan: generating hard samples to improve classification networks. In 2019 16th International Conference on Machine Vision Applications (MVA), pages 1-6. IEFE, 2019.
[108] Vinicius F Arruda, Thiago M Paixão, Rodrigo F Berriel, Alberto F De Souza, Claudine Badue, Nicu Sehe, and Thiago Oliveira-San Santos. Cross-domain car detection using unsupervised image-to-image translation: From day to night. In 2019 International Joint Conference on Neural Networks (LJCNN), pages 1-8. IEEE, 2019.
[109] Mohammad Mahfujur Rahman, Clinton Fookes, Mahsa Baktashmotlagh, and Sridha Sridharan. Multi-component image translation for deep domain generalization. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 579-588. IEEE, 2019.
[110] Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni-Vineee Boqing Gong. Domain randomization and pyramid consistency: Simulation-to-reat general- ization without accessing target domain data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2100-2110, 2019.
Kurt Keutzer, and
[111] Zak Murez, Soheil Kolouri, David Kriegman, Ravi Ramamoorthi, and Kyungnam Kim. Image to image translation for domain adaptation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 4500-4509, 2018.
[112] Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization. arXiv preprint arXiv:2104.05833, 2021.
[113] Haohan Wang, Zexue He, Zachary Lipton, and Eric Xing. Learning robust representations 2019.
by projecting superficial statistics out. arXiv preprint
[114] Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap via style-agnostic networks. arXiv preprint arXiv: /645, 2019.
[115] Nader Asadi, Amir M Sarfi, Mehrdad Hosseinzadeh, Zahra Karimpour, and Mahdi Eftekhari. Towards shape biased unsupervised representation learning for domain generalization. arXiv preprint arXiv: (eXXiv::90...m 0929,0 2019.
[116] David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Priol, and Aaron Courville. nut-of-distribution generalization via risk extrapolation (rex), arXiv preprint ArXiv:2003.000 00688, 2020.
[117] Chuang Gan, Tianbao Yang, and Boqing Gong. Learning attributes equals multi-source domain generalization. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 87-97, 2016.
[118] Toshihiko Matsuura and Tatsuya Harada. Domain generalization using mixture of multiple latent domains. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 1 11749-11755 2020.
[119] Niu, Wen Li, and Dong Xu. Visual recognition by learning from web data: weakly supervised domain generalization approach. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 2774-2783 2015.
[120] Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust optimization. Princeton university press, 2009.
[121] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. theory of learning from different domains. Machine learning, 1911):151-175.211
18

---

[122] Shai Ben David, Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility theorems for domain adaptation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 129-136. JMLR Workshop and Conference Proceedings, 2010.
[123] Andrew Bagnell. Robust supervised learning. In AAAI, pages 714-719, 2005.
[124] Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. arXiv preprint arXiv:/206.6471, 2012.
[125] Zachary Lipton, Yu-Xin XuaXiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference machine learning, pages 5122-3130. PMLR, 2018.
[126] Nanyang Ye, Kaican Li, Lanqing Hong, Haoyue Bai, Yiting Chen, Fengwei Zhou, and Zhenguo Li. Ood-bench: Benchmarking and understanding out-of-distribution generalization datasets and algorithms. arXiv preprini arXiv:2106.0372/ 2021.
[127] Amir Rosenfeld, Richard Zemel, and John K Tsotsos. The elephant the room. arXiv preprint arXiv:/808. 03305, 2018.
[128] Joaquin QuiñonerooCandell Masashi Sugiyama, Neil D Lawrence, and Anton Schwaighofer. Dataset shift in machine learming. Mit Press, 2009.
[129] Masanori Koyama and Shoichiro Yamaguchi. Out-of-distribution generalization with maximal invariant predictor. arXiv preprint arXivv2008..0 0888,, 2020.
[130] Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. The risks of invariant risk mini- mization. arXiv preprint arXiv:2010.05761, 2020.
[131] Pritish Kamath, Akilesh Tangella, Danica J Sutherland, and Nathan Srebro. Does invariant risk minimization capture invariance? arXiv preprint arXiv:2101.01 134, 2021.
[132] Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. Understanding the failure modes of out-of-distribution generalization. arXiv preprint arXiv:2010.15775 2020.
[133] Kartik Ahuja, Jun Wang, Amit Dhurandhar, Karthikeyan Shanmugam, and Kush R Varshney. Empirical or invariant risk minimization? sample complexity perspective. arXiv preprint arXiv:2010. /6412, 2020.
[134] Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, 4(2):251-257, 1991.
[135] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-366) 1989.
[136] Deepak Pathak, Philipp Krahenbuhl, and Trevor Darrell. Constrained convolutional neural networks for weakly supervised segmentation. Proceedings of the IEEE international conference computer vision, pages 1796-1880 2015.
[137] Steven Chen, Kelsey Saulnier, Nikolay Atanasov, Daniel D Lee, Vijay Kumar, George J Pappas, and Manfred Morari. Approximating explicit model predictive control using constrained neural networks. In 2018 Annual American control conference (ACC), pages 1520-1527. IEEE, 2018.
[138] Thomas Frerix, Mathias Nießner, and Daniel Cremers. Homogeneous linear inequality constraints for neural network activations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pages 748-749, 2020.
[139] Brandon Amos and J Zico Kolter. Optnet: networks. In International Conference on Machine Learning, pages 135-145. PMLR, 2017.
Differentiable optimization layer in neural
[140] Sathya N Ravi, Tuan Dinh, Vishnu Lokhande, and Vikas Singh. Constrained deep learning us- ing conditional gradient and applications in computer vision. arXiv preprint arXiv: 1803 06453, 2018.
19

---

[141] Priya L Donti, David Rolnick, and J Zico Kolter. De3: A learning method for optimization with hard constraints. arXiv preprint arXiv:2104. 12225, 2021.
[142] Luiz FO Chamon, Santiago Paternain, Miguel Calvo-Fullana, and Alejandro Ribeiro. The empirical duality gap of constrained statistical learning. In ICASSP 2020-2020 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages IEEE, 2020.
[143] Dimitri P Bertsekas and Athena Scientific. Convex optimization algorithmss Athena Scientific Belmont, 2015.
[144] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: hutp://yann. lecun. com/exdb/nnist, 2, 2010.
[145] Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[146] Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
[147] Richard F Bass. Real analysis for graduate students. Createspace Ind Pub, 2013.
[148] Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[149] Miguel Goberna and MA López. Recent contributions linear semi-infinite optimization. 4OR, 15(3):221 -242, 2017.
[150] Luiz Chamon and Alejandro Ribeiro. Probably approximately correct constrained learning. Advances in Neural Information Processing Systems, 33, 2020.
[151] Elias M Stein and Rami Shakarchi. Functional analysis: introduction to further topics in analysis, volume 4. Princeton University Press, 2011.
[152] Luiz FO Chamon, Santiago Paternain, Miguel Calvo-Fullana, and Alejandro Ribeiro. Con- strained learning with non-convex losses. arXiv preprint arXiv:2103.05132 2021.
[153] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 12././/226666. 69800, 2014.
[154] Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loie Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230, 2018.
[155] Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision, pages 2551-2559, 2015.
[156] Asha Anoosheh, Eirikur Agustsson, Radu Timofte, and Luc Van Gool. Combogan: Unre- strained scalability for image domain translation. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 783 183-790,11 2018.
[157] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Wor Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 81 8188-8197, 2020.
20


Reviews:
[{"review_id": "vxIFPliboGZ", "reviewer": "Reviewer_VPwo", "summary": "This paper proposes a framework for domain generalization which involves learning a transformation between domains and optimizing the worst case loss over these transformations. They formulate this as a min max optimization problem and eventually solve the dual of a relaxed version of the problem. Empirically, they find that their algorithm improves a lot for ColoredMNIST and Camelyon-WILDS especially, which are datasets where the domain transformations are somewhat simple (color shift). They get a modest gain on a satellite dataset FMoW-WILDS.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Significance\n\nDomain generalization is an important problem, although it's unclear whether it is feasible to make meaningful progress. Implicitly, this paper shows that for particular domain generalization problems with simple transformation shifts, we can learn these transformations well and generalize OOD. Indeed, we shouldn't expect to be able to learn transformations between domains for general shifts, which can be very complex and hard to generalize across domains. The paper could state this more clearly, that the method needs to take advantage of some structure in the transformations (like color shift). \n\nOriginality\n\nThe overall setup is quite similar to [1], which learns generative perturbations of the data for adversarial learning. The difference is that the learning perturbations here use the different domains. [1] uses PGD adversarial training to optimize the min-max problem; this brings to question whether the primal-dual iterations are needed in the MBDG framework, and whether the problem in Prop 4.3 can be optimized directly through adversarial training.\n\nQuality\n\nThe paper has some good ablations, for example it shows empirically that learned transformations on the training domains (using MUNIT) improve over fixed augmentations such as RandAugment for the MBDG framework. They give a mathematical derivation of the framework through the language of optimization, although it's unclear whether we can get theoretical insights from the general mathematical framework beyond suggesting an optimization algorithm (which perhaps could be replaced with adversarial training?). \n\n- When testing the regularization version of MBDG, the paper shows that with $\\lambda =1.0$ the performance isn't as good as MBDG with primal-dual iterations, but it's not clear whether $\\lambda$ is varied. A proper comparison would have a sweep over $\\lambda$, with some protocol for selecting $\\lambda$ (some kind of validation accuracy, either ID or OOD).\n- While the abstract states improvements \"up to 30\\%\", the range of improvement is pretty wide (30\\% on ColoredMNIST. 20\\% on Camelyon17 but 1% on FMoW), and care should be taken when presenting these results faithfully. \n- According to Figure 4, RandAugment with color is quite effective with MBDG for Camelyon. What is the comparison with ERM + RandAugment? \n- The MBDG model improves on average over the PACS domains, but the improvement is mostly driven by improvement on sketch. This suggests that the MBDG model is best suited for particular shifts (yet to be characterized).\n- For example, the G-invariance is the main special part of the theory, and it's clear from the relaxation formulation that when the G-invariance margin is small, then the gap is smaller between the relaxed problem and the original. The theory has some limitations, like that the hypothesis classes are of finite VC dimension, so it doesn't hold for deep models.\n- G-invariance seems stronger than what is needed to solve the problem in Prop 4.3. In particular, it isn't necessary that $f(x) = f(G(x,e))$, but that for all $e, e' \\in \\varepsilon_{all}$, $f(G(x,e)) = f(G(x,e'))$. Maybe G-invariance works better in practice, but it's unclear from a formulation standpoint why this is formulated this way.\n\nClarity\n\nThe paper is clearly written, although the paper seems a little unfinished, with no conclusion. Secion 7.3 (PACS) lacks an adequate description of the task. \n\n\n[1] Eric Wong and J. Zico Kolter. Learning Perturbation Sets for Robust Machine Learning, 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nWhile invariance and generalization to new domains could improve real world tasks like Camelyon, the authors mention that invariance is not always a desirable property, and models could be overly invariant / enforcing invariance can worsen extrapolation or at least change the inductive bias. They do not believe that there will be negative societal impacts relative to the body of work that already exists. \n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "ISiReC4yxh", "reviewer": "Reviewer_Lmtm", "summary": "This paper considers the domain generalization setting where data in all domains can be transformed into one another by a transformation G. Then, authors then propose to learn a predictor invariant to this transformation. Experimental results show some encouraging results.\nOverall approach presented is interesting, stimulating and the use of different optimization techniques to formulate the problem is pleasant. However, there is a ‘fatal’ problem in the framework, either by a mathematical mistake (i.e., moving from Eq 31 to 32 in the appendix), or authors have failed to declare a very strong assumption that that p(y|x) = p(y | G(x,e)) (see detailed comments below). If this point can be clarified, I would be willing to reconsider my rating.\n\n----- post-rebuttal\nI want to thank the reviewer for their thorough rebuttal. While I still want to see some solid analysis on the effect of the transform G and the assumption, the new discussions made sense and early empirical results on testing far domains are encouraging. The authors, please, make sure you factor them into the paper as they will not only enhance the understanding but also increase the validity of your work.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors implicitly assume that: after transport x ~ P(x) by transformation G( . , e), the labeling mechanism is unchanged, i.e., P(Y | x) = P(Y | G(x,e)) (Proposition 4.3)\n\nThis assumption allows authors to sample from a common distribution, i.e., (x, y) ~ P(x, y) instead of sample directly from distribution of each domain, i.e., (x, y) ~ P^e(x, y) (Proposition 4.3, equation 32). Concretely, we will derive an equivalent sampling process to (x, y) ~ P^e(x, y) as follow:\n(x, y) ~ P^e(x, y) can be cast into: first x ~ P^e(x), then y ~ P^e(Y|x) (Proposition 4.3, equation 28)\nDue to Assumption 4.1, P^e(Y|x) = P(Y|x) for all x, the sampling process is: first x ~ P^e(x), then y ~ P(Y|x) (Proposition 4.3, equation 30)\nDue to Assumption 4.2, there is some G(. , e) that transports P(x) to P^e(x). This means the sampling process x ~ P^e(x) can be replaced by x ~ P(x), then passing G(x, e) to the label generative process. Therefore, the labeling mechanism must be y ~ P(Y| G(x, e)). The whole process is x ~ P(x), y ~ P(Y| G(x, e)), and is equivalent to (x, y) ~ P^e(x, y).\nHowever, the authors instead sample label as y ~ P(Y| x) with x is from P(x), and G(x, e) is only used in the loss function (Proposition 4.3, equation 30, inner-loop expectation). This leads to sampling process x ~ P(x), y ~ P(Y| x) is not equivalent to x ~ P(x), y ~ P(Y| G(x, e)) due to the change in labeling mechanism.\n\nAuthors’ derivation can work under further assumption that P(Y|X) is G-invariant: “P(Y| x) = P(Y| G(x, e)) for all x and e”. However, this is a rather strong assumption, for example it doesn’t hold when there is a large shift in label marginals. Note that this requirement is an additional assumption, and is not coincide with Assumption 4.1, since Assumption 4.1 says all domains e must use the same set of condition distributions, i.e., P^e(Y|x) = P(Y|x) for all x, but it doesn’t require P(Y|x) = P(Y|x’) for 2 different x, x’.\n\nTo illustrate further, let R be the data space and {-1, 1} be the label set. Moreover, all domains P^e(x,y) must share the same conditional P(Y|X), that if X<0, assign Y = -1, and if X>0, assign Y=1. Considering P^e(x, y) with P^e(x) = Uniform[0,1], we have for all x ~ P^e(x), label y=1. Furthermore, let P(x,y) be another distribution with P(x) = Uniform[-1,0], then for all x ~ P(x), label y = -1. Next, we will find a sampling process equivalent to (x, y) ~ P^e(x, y). We can first sample from x ~ P(x), and then shift data by G(x) = x+1 to obtain P^e(x). However, if we label by y ~ P(Y|x), all labels will be -1, because X<0 and P(Y = -1 | X<0) =1, and hence we do not obtain the equivalence of (x, y) ~ P^e(x, y). The correct sampling process must be x ~ P(x), transport to G(x), and then label y ~ P(Y|G(x)), which will now return y=1, since G(x)>0 and P(Y = 1 | X>0) =1. This toy experiment shows that this paper’s approach might fail in situations where domains have different label margins.\n\nUsing rather unrealistic assumption that labeling mechanism is unchanged after transforming x into G(x, e), for all x ~ P(x) and for all domain e, authors define the key optimization problem (Problem 4.6) in Proposition 4.5, which seek to find a function f that is also invariant to transformation G, i.e., f(x) = f(G(x, e)). However, as demonstrated before, assumption “P(Y| x) = P(Y| G(x, e)) for all x and e” is unrealistic, therefore this optimization problem doesn’t guarantee to reflect real datasets.\n\nGeneralization of G is critical to generalization of f, which is expressed in the infinite-dimensional constraint term in the Lagragian. However, in the empirical version, this constraint is approximated by summing over training domains, which is usually of small number in most real datasets. Therefore, G could learn to transform between these training domains, but might not be able to do so for target domains, notably target domains of large dissimilarity to training domains, because G is a map between high-dimensional input space, trained using only a small number of domains. Therefore, for these “distant” target domains, it is unlikely that f generalizes, and f can only work on target domains closely similar to training domains. It is therefore interesting to see (1) the performance of predictor f in the case where the target domain is very different to each training domain, (2) performance of this method on other datasets with more diverse domains, and (3) how the number of training domains affects the generalization of f.\n\nThe form of distance d is not discussed.\nTypos and other errors\nLine 217, P(\\mathcal{E}_{all}) is not explicitly defined.\nLine 926, expectation over P^e(x) is missing\nLine 973, eq 54, drop expectation over risk difference\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author didn’t adequately address the limitation of the transformation G. Moreover, since there is an implicit assumption, this method might not work properly in some situation (shifted label marginals)", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "53b1kldhJyp", "reviewer": "Reviewer_LPjW", "summary": "This paper proposes a new model specifically designed for Out-of-Domain (OOD) detection. The authors start by setting up the domain generalization problem by explaining that (x,y) ~ p_e(x,y) is sampled from different domains ‘e’ in the training and test phase. Making certain assumptions such as (1) the inter-domain variation is only because of p(x), and (2) existing a transformative network g(x,e) that can map observation x to different domains e, the authors propose to slightly modify the problem definition which they call “Model-Based Domain Generalization”. This problem is slightly easier to deal with and under certain conditions, guarantees optimality. The authors finally do an extensive set of experiments to demonstrate the proposed algorithms significantly improves OOD detection results over previous models, as well as the role of regularizing theta with the distance regularizer. ", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Overall Comment]\n\nI think this is a good paper. OOD Detection is definitely both an important and popular topic in the field, especially given that, as the authors point out, it’s impact on ML safety. So I certainly think the OOD community can benefit from the findings of this paper. The transition from problem 3.1 to 4.6 certainly makes sense and I think it’s quite elegant. I applaud the authors given that I think this is a lot of content for a single paper, and overall, I think the authors did a good job condensing their theoretical and experimental contributions while ensuring that things are as clear as possible. Given both the methodology and the improvements of the OOD results, I’m happy to recommend acceptance.\n\n[Minor Comments]\n\n-The problem was set-up quite well. I’m not too familiar with the OOD detection field so I found Section 3 quite useful and easy to read (Figure 1 in particular). \n\n-Could the authors comment on the distinction between OOD detection in discriminative models (which is the setting in the current manuscript) and in purely generative models like Flows and VAEs etc? \n\n-While I understand the space constraints, having no conclusion or discussion that summarizes the work is something I would consider a major flaw given the venue, especially given the fact NeurIPS allowed an additional page this year. I will  leave to the AC and other reviews to determine to what extent this is acceptable. \n\n-The supplementary contains the main manuscript as well. \n\n-I think the related work was discussed very well. \n\n[Clarity]\n\nThe paper reads well overall. Even though this is a lot of content for a single paper, I think overall the authors did a good job at condensing the materials while making sure that the presented ideas are mostly clear.  \n\nTypos:\nPage 5, (C3) “We do ‘NOT’ have access to the set of all domains ...”\n\n[Significance]\n\nGiven both the the extensive theoretical studies and the improvements of the OOD results, I think the paper can make a valuable contribution to NeurIPS.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors sufficiently discussed the potential impact of OOD.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers reached a consensus that the paper has an interesting and novel idea and achieved good improvements over prior works on many (though not all) benchmark datasets. The AC recommends acceptance. 
Decision: Accept (Poster)