MADG: Margin-based Adversarial Learning for Domain Generalization
Authors: Aveen Dayal
Vimal K B
Linga Reddy Cenkeramaddi
C Krishna Mohan
Abhinav Kumar
Vineeth N. Balasubramanian
Venue: NeurIPS 2023 poster (2023)
Keywords: Domain Generalization
Margin Loss
Adversarial Learning
Domain Adaptation

Abstract: Domain Generalization (DG) techniques have emerged as a popular approach to address the challenges of domain shift in Deep Learning (DL), with the goal of generalizing well to the target domain unseen during the training. In recent years, numerous methods have been proposed to address the DG setting, among which one popular approach is the adversarial learning-based methodology. The main idea behind adversarial DG methods is to learn domain-invariant features by minimizing a discrepancy metric. However, most adversarial DG methods use 0-1 loss based $\mathcal{H}\Delta\mathcal{H}$ divergence metric. In contrast, the margin loss-based discrepancy metric has the following advantages: more informative, tighter, practical, and efficiently optimizable. To mitigate this gap, this work proposes a novel adversarial learning DG algorithm, $\textbf{MADG}$, motivated by a margin loss-based discrepancy metric. The proposed $\textbf{MADG}$ model learns domain-invariant features across all source domains and uses adversarial training to generalize well to the unseen target domain. We also provide a theoretical analysis of the proposed $\textbf{MADG}$ model based on the unseen target error bound. Specifically, we construct the link between the source and unseen domains in the real-valued hypothesis space and derive the generalization bound using margin loss and Rademacher complexity. We extensively experiment with the $\textbf{MADG}$ model on popular real-world DG datasets, VLCS, PACS, OfficeHome, DomainNet, and TerraIncognita. We evaluate the proposed algorithm on DomainBed's benchmark and observe consistent performance across all the datasets.

MADG: Margin-based Adversarial Learning for Domain Generalization
Aveen Dayal
Vimal Indian Institute of Technology Hyderabad vimalkb96@gmail com
Indian Institute of Technology Hyderabad ai2iresch1003eiithhhaa in
Linga Reddy Cenkeramaddi
C Krishna Mohan Indian Institute of Technology Hyderabad ckm@cse iith ic
University of Agder
1 Linga.cenkeramen www..eeeaeeedieeeececcaa @uia.no
Abhinav Kumar
Vineeth N Balasubramanian Indian Institute of Technology Hyderabad vineethnbêcse. iith. ac.in
Indian Institute of Technology Hyderabad abhinavkumar@e.. iwrteeeeeittthie with. ac
Abstract
Domain Generalization (DG) techniques have emerged a popular approach address the challenges of domain shift in Deep Learning (DL), with the goal of generalizing well the target domain unseen during the training. In recent years, numerous methods have been proposed to address the DG setting, among which one popular approach is the adversarial learning- based methodology. The main idea behind adversarial DG methods is to learn Nomainn-nvarrion features by minimizing discrepancy metric. However, most adversarial DG methods use 0-1 loss based HAH divergence metric. In contrast, the margin loss-based discrepancy metric has the following advantages: more informative, tighter, practical, and efficiently optimizable. To mitigate this gap, this work proposes novel adversarial learning DG algorithm, MADG, motivated by a margin loss-based discrepancy metric. The proposed MADG model learns domain domain-invariant features across all source domains and uses adversarial training to generalize well to the unseen target domain. We also provide theoretical analysis of the proposed MADG model based the unseen target error bound. Specifically, we construct the link between the source and unseen domains in the real-valued hypothesis space and derive the general- ization bound using margin loss and Rademacher complexity. We extensively experiment with the MADG model on popular real-world DG datasets, VLCS, PACS, OfficeHome, DomainNet, and Terralncognita. We evaluate the proposed algorithm DomainBed's benchmark and observe consistent performance across all the datasets.
1 Introduction
Over the past decade, Deep Neural Networks (DNNs) have demonstrated exceptional performance across various fields, including robotics [1] medical imaging [2], agriculture 3 and more. However, the effectiveness of DNNs in supervised learning environments relies heavily on the independently and identically distributed (f.i.d.) assumption of the training and test (target) data. Unfortunately, in reality, this assumption can be compromised due to domain shifts in target data
[4]. For example, a
model trained on different domains of image data may perform poorly when presented with an image of known label featuring an unseen background or viewpoint, as explained in (5). To address this issue, researchers have developed techniques under the framework of domain adaptation (DA) (6]. The key idea behind DA is to adapt a model trained on source dataset to minimize the generalization
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

error on the target dataset 77] However, the major limitation of DA is that the target data, whether labeled or unlabeled, must be available during training. In contrast, the Domain Generalization (DG) setting aims to leverage knowledge from similar domains to classify previously unseen domains g
Recent years have seen the development of various algorithms addressing the DG setting, among which one popular approach is the adversarial learning- based methodology [9H4]. Other approaches can be broadly classified into meta-learning techniques (15177,, data augmentation methods [18] 20]. self-supervised learning methods 21233, regularization-base
methods (24H29), and so on. Additionally. there have been concerted efforts towards the development of benchmark datasets 33 35] for the DG setting to study methods. The assumption of distribution shift, i.e. smooth variation between the conditional distribution P(Y and the marginal distribution P(X) is also
common in DG literature [6, (7]]
The main objective of DG methods is to model the functional relationship between the input space X' and the label space y for all domains. To achieve this, adversarial DG methods learn a domain- invariant representation by minimizing a discrepancy metric among the source domains. However, despite the myriad efforts, most adversarial learning methods use the 0-1 loss based HAH diver- gence metric [38] for domain alignment. In contrast, divergence metrics based on margin loss are more informative 33), practical, and efficiently optimizable [40]. This work addresses this gap by proposing novel adversarial DG algorithm, MADG, motivated by margin based divergence metric. The proposed MADG leverages margin- based disparity discrepancy [40] to estimate source domain discrepancies in the DG setting and uses adversarial training to ensure that MADG generalizes well to unseen target domains. We also theoretically analyze the proposed MADG algorithm based on bounds for the unseen target error. The proposed generalization bound uses the Rademacher complex- ity framework [41, which provides data-dependent estimates of functional class complexities. The effectiveness of the proposed algorithm is demonstrated through extensive experiments on multiple benchmark DG datasets.
The key contributions of this work can be summarized as follows: (i) We introduce the use of margin loss and a corresponding scoring function to formulate the relationship between domains and develop upper bounds for the unseen target error (the first such margin- based effort in the DG setting, to the best of our knowledge); (ii) We subsequently analyze the generalization bound in terms of functional class complexity using the Rademacher complexity framework; (iii) We propose a novel margin- adversarial DG training algorithm, MADG, motivated by our theoretical results; and (iv) We study the proposed method on five well-known benchmark datasets in the DomainBed setup. providing a higher average accuracy and consistency in model performance across these datasets.
2 Related Work
In this section, we discuss earlier literature proposed specifically for adversarial DG, as well as theoretical analysis for the DG problem in general. We discuss other DG literature across a broader set of categories in the Appendix. The main idea behind existing adversarial DG methods is to minimize the H.H. divergence by employing a minimax optimization between a domain discriminator and a classifier to learn domain-invanne invariant features. One of the early works [9] proposed a method that feature
iteratively divided samples into latent domains via clustering and trained the extractor via adversarial learning. Other efforts extended such an approach of adversarial learning with different divergence metrics and regularization techniques. Lin et al. [10] proposed multi- dataset feature generalization network (MMFA-AAE) based on an adversarial auto-encoder to learn generalized inmain-invariant latent feature representation with the Maximum Mean Discrepancy (MMD) measure to align distributions across multiple domains. Deng et al. 11 examined adversarial censoring techniques learn invariant representations from multiple domains. Zhao et al. [12 proposed an entropy regularization term along with adversarial loss to ensure conditional invariance of learned features. Akuzawa et al. [ proposed the notion of accuracy-constraame and then developed an adversarial feature learning method with accuracy constraint (AFLAC), which explicitly provided invariance on adversarial training. Rahman al. 42 proposed a correlation- aware adversarial DG framework where the features of the source and target data are minimized using correlation alignment along with adversarial learning. All these prior works on adversarial DG methods use a 0-1 loss based HAH discrepancy metric to align source domains. In this work, we instead leverage a margin-base based disparity discrepancy and propose a new adversarial learning strategy founded on our theoretical analysis. The margin loss is advantageous compared to the 0-1 loss as provides informative generalization bounds, tightness, classifier-aware alignment, and efficient optimization. We discuss each of these advantages in detail in Section 4
domain invariance,

---

Early efforts for theoretical unalysis for the DG problem 43 44 the problem setting with corresponding generalization error analysis, and showed empirical results using traditional machine learning methods. A few other efforts 45/47 also followed similar path, and focused on kernel-based approaches in traditional methods. From a deep learning perspective, an early attempt at such theoretically motivated method was proposed in [48] using a convex hull formulation and distribution matching to obtain generalization bounds, which also is an inspiration for parts of our work. In [4)), the authors study theoretical bounds in the DG setting using three new concepts: feature invariance across domains, feature discriminability for the classification task, and a feature expansion function to generalize the first two concepts from the source to the target domain. [50] identified measures relating to the Fisher information, predictive entropy, and maximum mean discrepancy to be good predictors of out-of-distribution generalization of empirical risk minimization (ERM) models in the DG setting.
used kernel based approaches to
[24) proposed generalization bounds in terms of the model's Rademacher complexity and suggested using regularized ERM models for the DG problem. Finally, [14] introduced an online game in which one model (player) reduces the error on test data provided by an adversarial player, but do not provide any empirical studies. In this work, draw inspiration from [48) by considering a convex combination of source domains and leverage this to propose new margin-based approach the theoretical analysis of DG, which helps develop an informative generalization bound. Motivated by this analysis, we propose the MADG algorithm, a novel margin- based adversarial learning approach for DG, which shows consistent performance over other state-of of-the-art methods across benchmarks.
3 Preliminaries
We consider set of source domains Ds, with E (1. and a set of unseen domains Du., with m {1,2...., N. We use D to refer to any of these domains when the index is not relevant. We use the terms 'domain' and 'distribution' interchangeably in this work. Each such domain D C X' V/ where X' is an input space and y is an output space, which is {0, 1) in binary classificatton and {1.... }} in multi-class classification. We use D to denote set of samples drawn independently from D, i.e. D { == where 2; X' and y E V,Vi (1. n}. We use (r. y) to refer to a labeled sample (t), when the index is not relevant.
As in [51. we consider the multi-class setting with a hypothesis space F of scoring functions f: X + R 3 = R where the outputs report the confidence of the prediction on each dimension. Similar to [4]] we use fzz y) denote the component of fxx that corresponds to the label y. In order to obtain the final predicted label, we also consider a labeling function space H containing hy : X + y such that (*) = argmax f"x y). i.e. the predicted label assigned to data sample x is the one resulting in the largest confidence score.
The expected error rate of a classifier h E H with respect to distribution E(z,))= 1 [zx)) * y where is the indicator function. We also define the margin py(+) and the corresponding margin loss of hypothesis for labeled example (x, y) as follows:
defined as erro(h) (h) A
pf v) 2 [xx v) max, tu [xx s/)
err(g' (J) A E [00 pf(,,y)]
(1)
where denotes the composition function and t is:
0
pf
(t) = 1
if
ftstts 5 0
(2)
1
The margin disparity and its empirical version are then defined as:
disp (p) (f',f) f) Ed ( (r)),
(3)
(p) (f f) A ED 2 py hn) d
1 n
2
(4)
where ff and f' are different scoring functions. Margin disparity discrepancy (MDD) is used to between decision boundaries classifiers (using
quantify the degree of discrepancy/disagreeeeee their margins) trained on different domains in our context. Such measure can be used to evaluate the generalization performance of a model across multiple domains. The MDD and its empirical version are defined in Eqn. (5) and Eqn. (6) below, respectively.
3

---

d/r (s,, supperr MEF (dispbs) (2) f) disppsssss (fr
(5)
d dff (Ds., Dss) rEFF disp (p) Dag (f', disp (p) Ds, (s'
(6)
We use the above terminologies from 40) which however focused on domain adaptation (with one source domain and one seen target domain). Handling multiple seen domains and an unseen target domain in the DG setting is non-trivial, which we focus on in this work.
Margin-based Approach to Domain Generalization: Theory
This section presents the theoretical motivation for our margin- based approach to the Domain General- ization (DG) problem. Our approach is based considering the margin disparity discrepancy (MDD), defined above, between the source domains, thereby obtaining a generalization bound. We believe such margin-baseet approach has few advantages: (i) Informative: Generalization bounds based on margin loss for classification provide more information (than 0-1 loss-based bounds) by establishing dependency between any margin function satisfying the Lipschitz condition and the upper bounds, as demonstrated in well-known earlier work [39]. (ii) Tightness: In contrast to using a 0-1 loss to identify samples causing disagreement between classifiers, margin loss computes disagreement between scoring functions using a smooth function parameterized by the threshold (p). Consequently, the number of samples causing agreement between classifiers less, resulting in such MDD-based bound being tighter, as shown in Fig. D (iii) Classifter-aware Discrepancy: MDD considers the classifier function while measuring discrepancy; as shown in Eqns 5 and 6 the supremum is com- puted over f' while holding f constant (f learns the posterior distribution P(y (z) discriminatively for the classification task). This provides a classifier-aware approach to computing discrepancy.
(iv) Efficient Optimization and Practicality: The definition of MDD, as in Eqn. (5). only requires taking the supremum over one hypothesis. Therefore, compared to other divergence measures such as HAH [38]. MDD can be optimized with ease and is practically useful, as also stated in [40].
lg
bym
In this section, we derive a generalization bound for an unseen domain based on the margin-based MDD loss in the DG setting. To this end, we first show an upper bound on the unlabeled source domain eтгоr given other labeled source domains (Lemma and Theorem 1). We then leverage this develop the upper bound for the error on an unseen domain that is not necessarily a source domain (Lemma 2, Theorem 2, and Corollary 1). We subsequently analyze the upper bound from Corollary using the Rademacher complexity framework and develop our final generalization bound for the unseen target domain in the DG setting (Lemma 3 and Theorem 3) using our margin- based loss. See 5, we show the formulation of the proposed adversarial learning algorithm, MADG, motivated by the generalization bound in Sec 4, that employs MDD to address the DG problem.
Margin
2i. Figure 1: Space of intersection (agree- ment) in MDD (yellow) is reduced as compared 0-1 loss (blue + yellow) be- tween f and f' for labels {0. 1}.
We begin by considering a setting where training data consists of N, Ds. i = {I, ; N₂ - }} and a single unlabeled source domain Ds, and show an upper bound on error this setting. We later use this to show an upper bound for the DG setting. We first establish an upper bound on the MDD between weighted sum of the labeled source domains and the unlabeled one in Lemma [below.
labeled source domains
Lemma 1 Consider a weighted sum of N, - 1 labeled source distributions defined as D3 IN=1 a,Ds,, where ass are mixture co-efficients s.t. EN=-1 a; a;=1, 1, and f is scoring function. Then
N-
d (Ds, Dsr S
( (Ds,, Dsr)
Detailed proofs for all our theoretical results are provided in the Appendix. It follows from Lemma that an effective way to minimize the discrepancy between the unlabeled source domain and the mixture of labeled source domains the hypothesis space is by minimizing the convex sum of the pairwise MDD between each labeled and unlabeled source domain. Building upon this insight, we provide bounds on the unlabeled source error below in Theorem

---

Theorem Consider scoring function f. unlabeled source domain Ds, and mixture of N, 1 source distributions denoted as Dg == E a,Ds,, where a& is mixture co-efficients s.1. ΣN= a; =,=1. =1=. Then the error on the unlabeled source Ds, is bounded as:
N.
(0) (f) + (f)+dfff))) (Ds. Ds.) errDs, (ht) E as (errbo
(8)
From Theorem I we observe that the unlabeled source error is upper bounded by the labeled source errors, the pairwise discrepancy between each labeled and unlabeled source domain, and the ideal margin loss described below in Remark We can also interpret Theorem | as an upper bound for the multi-souree domain adaptation (DA) setting. While this is not our primary focus in this work, we report preliminary empirical results for a multi-source DA algorithm that reduces the first two terms in Theorem I
Remark 1:
in the Appendix.
(e) (f*) + err (e) Remark 2: The ideal loss À is defined as Å = miny-eF NNN a, errby, and is constant that independent of function f.
(f*)
Building on Theorem | we now develop bounds for the DG setting, where all source domains (N,) labeled, and the target domain is unseen during training. To this end, consider the relationship between the unseen target and multiple source domains, In particular, we derive our error bound on the unseen target domain using the convex hull of the labeled source domains and MDD. We also provide a DG algorithm motivated by our theoretical analysis, which we describe later in Sec
Before stating main theorem, present Lemma 2 which states that if the maximum MDD between any two sources is bounded above by e, then the MDD between any two domains that belong to the convex hull of all the sources will also be bounded by Let A, be the convex hull of source domains defined as A₂ = (D. D = E T,Ds, where Σ n.= ==1
Let ( Ds. Vi,k E (1,2....,N. and J be scoring function. Then the following inequality holds for the MDD between any pair of domains D', DD Agg df' "D")
Lemma 2
We observe from Lemma 2 that in the hypothesis space, the discrepancy among all the domains (seen or unseen) that belong to the convex hull A, can be reduced by minimizing the maximum MDD between two source domains. With the necessary tools at hand, we state the key theorem, i.e., the unseen target error bound for the DG problem below in Theorem P
Theorem 2 Consider mixture of N, source distributions, scoring function f. unseen domain Dun.. and 7 = =d DU) where Du the projection of Dun onto the convex hull of the sources i.e. Du =
(P)
Then, the unseen target error is f.r (uum. m.Dsss E
TI =
bounded as follows:
N, 5
eTT (h5)
Ti (errio) (e) (f) te +
(10)
i=1
Remark 3:
As defined in Theorem
y= (0) (Dum. Du). Two scenarios therefore arise: =
when the unseen domain falls the convex hull, i.e., Du.,, E T:Ds, or > 0 when the unseen domain cannot be represented by available domains alone. Thus this parameter can be interpreted as the need for diverse source domains. The more diverse the source domains are, the smaller the value of 7.
Remark 4: As seen from Lemma 2 e is defined as the upper bound for the MDD between any two domains that belong to the convex hull A, formed by the souree domains. Thus, we can also interpret
e as the highest MDD value among the source domains as shown below Eqn II
<=d c (Ds,,, )s s.t. f.r (Ds,, )s a.. (Ds,, Ds.) Vi, k, i' K'
No}
(11)
Equipped with this definition for we can re-state Theorem Das Corollary [below. Corollary Consider a mixture of N, source distributions, scoring function f. unseen domain Dim, Q df} (Dum, Du), and (Df Ds,.) where dfp) (Ds,,, Ds
d d (Ds, Ds,) Vi,k,i' k'
Then the unseen target error is bounded as follows:
5

---

N₂
Dr (h1) - ni err Ds, (f) + tof (Ds,, +r
_
(12)
As seen from Corollary 1 the unseen target enror is bounded above by the source er- rors, maximum MDD between two source domains, y, and the ideal loss given by = m E merrye
(0) (f*) + errb)) Dum (f*)).
Before we derive the generalization bound with the Rademacher complexity framework, we define 11F = {or (0
f(x,h(a)) h H.SEF. 40 and Rademacher complexity RA,D in Definition
Definition Let F be a class of functions such that f F X x y [a, bj and D {(11.))) .... (Ii:: yn]] } be a fixed sample of size n drawn from 'D over X x V. Then the Rademacher complexity of F is defined as:
n sup Σ fEF
(F) 2 EDDDA
Eg
ozf vi)
(13)
i=1
where ass are independent Rademacher variables that assume values in {-1,+13. We now leverage Lemma 3.6 in [40] to derive our Lemma 3
Lemma For any 8 > O, with probability 1 1-28, 25, the following holds simultaneously for any scoring function f:
(D,, Ds.) dff Ds.) s Ns..Ds (Ixz))+ k Rm. Der (Iz+F)+
log 2n,
log
2m (14)
where n, and nı correspond to the sample size of Ds and Ds,. respectively. The difference between MDD and its empirical version bounded by the Rademacher complexity as seen in Eqn [4] With these definitions, we derive our generalization bound based on Rademacher complexity and empirical MDD as below.
Theorem 3 Given the same setting Corollary I and Lemma 3, for any 8 > 0, with probability 1 - 35, we obtain the following generalization bound for all f:
N₂
errDum (f) $
Σ Ds,
Tee
(err
Ds, (f)) +d (Dsg: DsH X+
k Rn, Ds; (II+F)
k Rime (II+.F)
log
log
N.
2k
log 2
+V
Ra, De. (1,
2ny
2np
2n,
i=1
(15)
defined in (51) and (in the subscript) represents constant functions mapping all points to the same class (see Appendix for more details). Theorem thus establishes relationship between the margin value p and the generalization error. The first two terms on the right-hand side exhibit minimal variation with an increase in the margin especially when p is small and the hypothesis space is rich, thus reducing the overall right-hand side. However, if p exceeds a certain threshold (resulting in a weak classifier), then the first two terms significantly increase, resulting in increase of the overall bound. Choosing better p value thus allows to obtain a desired error bound, making it more informative than a 0-1 loss-based bound. Inspired by this theoretical framework, we now propose our margin-based adversarial learning algorithm for DG in the next section.
where 12F (I
5 MADG: Methodology
This section details the methodology of the proposed MADG algorithm, novel adversarial DG algorithm motivated by the theory in Section[4] As seen in Theorem 3 the unseen target domain enror upper-bounded by the convex sum of the empirical source domain errors, term defined in Eqn. (16). 7. À and complexity terms. We hence aim to minimize the two important terms, viz., the convex sum of the empirical source domain errors and the term (given below).
8 d Dsv), s.t. d.g (Ds,. Ds..) d/ (Ds,, Ds.) Vi, Vi,k, (1....,NN N.) (16)

---

1208
1200
((Ds))
Concatemnte
(Classification Loss)
fi
Source
GRL
MDD:
Σ Dea Da.
Feature Extractor
(Transfer Loss)
(9)
f
Input
GRI
GRL - Grudient Reversul Layer
MDD,
Source No.
f.fl E F.Vl = {I......}
Figure 2: Architecture of the proposed MADG methodology
Efficient estimation of é. As stated in Eqn. (T6), é is defined as the maximum pairwise empirical MDD among all source domains. Similar to Remark 4 and Lemma 2 upper bound of the empirical MDD between any two domains inside the convex hull formed by the source domains. If we minimize the sum of the empirical MDD between different pairs of the source domains, then the size of the convex hull reduces, which in turn minimizes é Thus, one effective way to estimate é to consider the term below in Eqn.
we can also interpret as the
17
N,
N.
Σ
Σ
5.r (Ds. Ds.)
(17)
k=i+1
Minimax optimization. To find the optimal ff in the hypothesis space F, we formulate our objective function as minimization problem. As mentioned earlier, we minimize the sum of the empirical source errors and the è term estimated using Eqn [7]. Thus, the minimization problem can be written as:
N. Σ
min
(p) Ds.
Σ
5.5 (Ds., Dsw))
(18)
k=++1
Since empirical MDD is defined as the supremum of the hypothesis space F, minimizing it is in turn minimax game with strong max player and weaker min player. In order to strengthen the min player, we use feature extractor, G, which further modifies the optimization problem as follows:
min 1.9
Te (rr(()) G(Ds.) (
disp o(Ds,) ( (Ii -
O(Dss) ())
(19)
fi max (disp f(
((Ds.) (fi. disp G(Ds (fi. .)) where f, fl E F and vI (1, {1......}.
where j (N.)) The relationship between i and is as follows: if I then we pick the and value corresponding to the 3rd element of the set k)) = {1,.... N₂ 1. {i+1,..... N.) To solve the minimization problem in Eqn. (19.. design adversarial learning algorithm whose model architecture is shown in Fig. One classifier performs the classification task on all source domains, while j other classifiers, denoted by f", compute the empirical MDD) between different pairs of source domains, which finally sum together as the transfer loss. To compute the source errors in Eqn. (19), we use the standard cross-entropy loss E {Ds,). For convenience of optimization (MDD can be hard to optimize directly using stochastic gradient descent), in practice, following 44).
we approximate the MDD loss (Dia.) (Ds, Ds.)) shown below.
in terms of two loss functions, L and L', as
E(Ds.) (aaa)
(r(o(t))
Dia,y (Ds,, Ds.)) = B (aa))DD Dsy
r. (((((()))
PR (W L (((Gt)))
(20)
We train the feature extractor, G, to minimize the above MDD loss term by using a Gradient Reversal Layer (GRL) proposed in (52), as D(a.n) (Ds,, Ds.) ) is not differentiable w.r. the parameters of f. L and L' are defined as:
7

---

((F(9())).)) a 10s oo (F( =(((0)))))))))) log
hy (o(o)) (f (f((()))
(21)
where (=)
R& for w 1,...,,, k, and X = ep. The final optimization problem that the MADG method solves is shown below.
اك enj'
min 1.9
Nx (E(Ds.))
Σ (Dea (Ds,, Ds.)).
max Fimndy
Ds.))
(22)
We propose adversarial DG algorithm to solve the minimization problem, as shown in Algorithm [ We train the proposed adversarial model in two steps. In the first step, we update the parameters fand G. In the next step, we first do a forward pass to compute the outputs from and then update the parameters f'and g as shown in the algorithm. We further show through our ablation studies that this way of up- dating parameters at different steps outperforms joint update strategy.
Algorithm 1 Margin Margin-based adversarial learning for Domain Generalization (MADG) INPUT: N. labeled source domains for epoch < 1 to oaal_epoccs do
for batch <- to total_ batches do
Update parameters of f. G. using Eqn. 20
MDs + f(G(x)) Vi E {1,2,.... N.}
Compute MDD, as in Eqn. (2)) using DDs;
Compute Transfer loss
= Ei [Ds., Ds.)))
Update parameters a'dnd Eqn.
6 Experiments
end for
end for
Datasets and Implementation Details. We per-
form an extensive evaluation on five benchmark DG datasets for image classification: VLCS (0), PACS (1I, OfficeHome (OH) (2, Terralncognita (TT) 33 and DomainNet (DN) (34], We follow
53 in using "Test-domain validation' procedure for hyperparameter selection. We use the Resnet50 architecture [54] pre-trained on ImageNet dataset [5S] as the feature extractor (G). We train our
model using stochastic gradient descent with momentum [56]. We use a mini-batch that contains 32 samples from all source domains. All other implementation details including hyperparameters such as learning rate, margin, and weight decay are provided in the Appendix.
Baselines and evaluation metrics. We report the results of all baseline models from 35 which are represented by i in Table I We also present the results from [53), which are represented by L. We run our experiments for three trials and report the average accuracy (with standard deviation). Existing state-of-the-ane art methods (see (35 focus on average accuracy across the benchmarks, with each method doing well on subset of datasets. In order to reward consistent performance across datasets while improving average accuracy, we additionally include a ranking-based metric (from ( and go beyond ranking with two new metrics (AD and GD), which more precisely capture the consistent performance the median rank of model's performance across all datasets and is not skewed to best/worst ranks when compared to the mean rank. (ii) Arithmetic mean of differences (AD): This measures the difference of a model's performance w.r.t the
method across datasets: (i) Median rank (M) (53): This measures
model's 's accuracy on given dataset, and then taking its arithmetic mean across datasets. (iii) Geometric mean of differences (GD): This similarly measures the geometric mean of the difference between the highest accuracy and the achieved accuracy of a model across all datasets.
Results. We report the results on different DG benchmark datasets on the considered evaluation metrics in Table accuracy on at least one of the datasets. This is captured well by the metrics AD and GD, which quantify the mean of the differences from the best accuracy across all the datasets. Thus, we observe higher values in the first two columns of Table I where our MADG model performs the best showing its consistency, The MADG algorithm outperforms all other models on average accuracy, median rank, AD, and GD, thus demonstrating consistent performance.
observed that most previous methods are far from the pest-performing model's
The results also show that MADG outperform DANN and CDANN models (which are based on the 0-1 loss discrepancy theory) across most datasets and improves the average accuracy by x= 2% with significantly better AD and GD values too - corroborating our margin-based approach to the DG problem. MADG is 1 % better on the avg accuracy (and lower AD-GD values) when compared to FRM, which was found to be a strong baseline for DG in (35) Our model also reports an accuracy improvement of se 3% on the OfficeHome dataset compared to all other models.
8

---

Table 1: (ccuracy(%) on benchmark DG datasets. Values in red and orange are best and second best- performing models in column, respectively. We report deviation N/A for models that did not report them. OH OfficeHomee dataset, Terralncognita dataset, DN DomainNet dataset. Median rank, AD Arithmetic mean of differences, GD = Geometric mean differences, Avg. = Average accuracy($)).
Algoritlun
AD(L)
GD(C)
M(J)
VLCS
PACS
DN
Avg-(t)
ERM (1998)1
S7
2.2
1.8
8
77.6.0.3
66.4g05
86.7a0a
53.0gs.
41.3g01
65.0
CORAT. (201))
SN
77.720
68.4g09
87.
52.8
41.8.01
65.6
DANN (2016)
59
3.4
2.1
16
79.7
65,3gna
85.2
50.6
38.3
63.8
CDANN (2018)
60
3.2
0.7
14
79.9
65.3gaG
85.810.
50.8
38.5g0a
64.1
MLDG (2018)*
h1
2.3
1.8
6
77.5
66.6gna
86.820.4
52.0
41.6g
64,9
MMD (2018)*
62
5.9
0.9
IO
77. 27.9
66.2204
87.2.tm
52.0
23.5
61.4
IRM (2019)t
18
76.9
63. 63.0gat
84.5
50.5
28.0
60.6
GrupDRO (2019)1
IO
77 77.4
66.2g.0
$7.1
52.4
33.4g0a
63.3
Mixup (2020)
78.1
68.0gaa
86.810
54.4
39.6201
65.4
ARM (2020)1
09
4.1
3,4
14
77 3
64,8
85 8...
51 51,2as.
36.0aMz
63.1
RSC (2020) 67
2.9
9
77.8
66.5gaa
86.2.0..
52.1
38.9
64.3
SagNett(222)
67.5.aa
86.4.0..
52.5
40.8
V-REx (2021)1
78.1
17.2..o.
51.4
30.1
AND-mask (2021)
00
13
76.420.4
66.1
86.4.0.4
49.8
37.9
63.3
Fish (2021)
13
77.8
66.0gam
85.8104
50.8
43.4
64,8
SAND-mask (202
82
5.2
4,2
16
76 76.220.8
65.920s
85.920.1
50,2
32,2a0e
62.1
Fishr (2022)
53
1.5
1.2
78 28.2
68, 68,2sua
86 26.9
53.6
41.8auz
65.7
G2DM (2019)
48
Theory Hased Methods
75.9
(TT(2221))*
47
2.0
8
77.7
66.5
86.7.0.
52.2
40.8
64.8
2002
73
64.3
85,3
002/2021)
76.3
86.6
MADG (ours)
78.7a0.
71.3
86.5204
53.7
39.9
66.0
To highlight our contribution, we also provide grouping of DG methods that are theory- based in Table [] (also denoted with a *), some of which do not report results on all benchmark datasets. For the only such method that reports results on all datasets MTL., MADG achieves 1.2% more average accuracy, 1.3 units less AD, 1.7 units less GD, and a lower median ranking as compared to MTT.. More results, including "Training-domarl model selection results, are provided in the Appendix.
7 More Empirical Analysis and Ablation Studies
Computational Cost. The computational cost of a model is usually measured in terms of GPU RAM occupied (GB) and time taken per step (s). In terms of computational cost, following Eqn. (17). while we iteratively compute MDD over all domains, our computational cost during training is similar to other benchmark methods as shown in Table As seen the table, even simple methods like ERM and Mixup have running times in similar ranges.
Table 2: Computational cost and Average Ac- curacy (%) (Avg.) for different baselines.
Margin. As shown in Theorem 3 the margin value plays an important role, especially with its dependency on the generalization error. In practice, to get the desired margin while train- ing the MADG algorithm, we approximate as exp.p. We experiment with different margin values, م = 1.5, 2, and 3, and report the average accuracy for PACS and VLCS dataset in Table 3 T that أ = 1.5 outperforms other values
Model
GPU(GB)() (ime(s)( Avg.(1)
MADG
10.5
~1.
66.0
Fishr 53
155.7
~066
65.7
Fish 71
~3.4
~ 1.2
64.8
Mixup 63
~8.22
~0.
65.4
ERM
[
~8.
004
65.0
on both datasets. As seen from the definition of the margin loss function Eqn. ① and MDD Eqn.
(5) although having large margin is better to achieve low loss and higher target accuracy, there is tradcoff between optimal margin and loss value. We observe a similar trend in Table
Experiments on Colored MNIST. As stated in Theorem 2 Y is defined as the projection of the unseen domain onto the convex hull. It is also defined in Theorem as the MDD between the convex hull and the unseen domain, which can be further approximated with a combination of two different cross-entropy functions as in Eq (20). This equation is equivalent to balanced Jensen-Shannon (JS) Divergence as shown in Proposition D. in 40 Thus, we can approximate this projection by computing the IS divergence between domains. We experimented on the Colored MNIST dataset, where we noted the 7 values to be small for all domains, shown
Table 3: Accuracy(5e) of MADG model for different ò values.
PACS
VLCS
86.2
78.4
1.5
86.5
78.7
2
86.2
78.4
85.7
77.8

---

in Table
4 This is because the distributions of different domains in Colored MNIST are relatively close to each other when compared to real-world datasets.
As seen in Table 5 the proposed MADG algorithm achieves 65.6% average accuracy, which is significantly higher than the ERM model (57.8% average accuracy), thus showing that using a margin-based DG algorithm, MADG, learns better domain-invariant features on the Colored MNIST dataset. Besides, Table 4 shows that y is small across domains in this dataset, showcasing the promise of the proposed method when the unseen domain is within the convex hull of the source domains.
Table 4: Pairwise JS divergence for the Col- oredMNIST dataset and the approximated value.
+90
+80
-90
~ "y
+90
0
0.019
0.024
0.043
+80
0.019
0.029
0.048
-90
0.024
0.029
0
0.054
MDD classifiers. The proposed MADG algorithm uses 'j' f' classifiers, and each one calculates the MDD between a pair of source domains. In the algorithm, 'j' is defined as
Table 5: Accuracy (%) on ColoredM- NIST dataset (C).
C
ERM
MADG
j = (Nv). In this section, we analyze the algorithm with j = Ns - where takes corresponding and k values from
+90
71.810.4
72.3.10.3
{(1,)) : (2,...,,N.)) ). As seen in Table 6 for 'Ns 1', the model performs low across all the datasets, significantly (~ 2.5%) for the Terralncognita dataset because the model fails to capture all possible domain discrepancies, leading to poor performance by the feature extractor learning invariant features.
+80
72.910.1
74.0-10.2
-90
28.7.10.5
50.5.10.4
Average
57.8 65.6
Table 6: Accuracy($b) of the MADG model for different MDD classifiers. New Ns- 1, Org. (*²
Additional Results. The proposed MADG algorithm is further com- pared with recent methods, MIRO 29 and SD (74), which report results using different hyperparameter selection procedures. For a fair comparison, we run both these methods following our setting and report the results on OfficeHome data as shown in Table As seen from Table 7 the proposed MADG outperforms the SD model on all domains. The MADG model also outperforms MIRO on two domains and achieves the same accuracy on the 'R' domain.
Data
j=Org. j=New
VLCS
78.7
78.5
PACS
86.5
86.4
OH
71.3
70,8
TI
53.7
51.1
Weighted MDD. The proposed MADG algorithm calculates the Transfer loss as the arithmetic sum of the '' MDD losses. This section analyzes the ef- feet on the model's performance using weighted (w) arithmetic sum. The different weights used in this section are wi - VI 1 1 (Average) and
Table 7: Accuracy(%) on OfficeHome (OH)
dataset
OH
Miro
SD
MADG
67.0 10.7
64.8.10.9
68.6 10.5
56.5 10.9
49.9 112
55.5 10.2
No Wi Eiz, MDD, (Dynamic). The performance of
P
79.4
75.6
79.6 +0.3
MDDr
R
81.5 +0.5
79.1 +0.2
81.5 +0.4
the MADG algorithm using different weights is re- ported in Table 8. The model with wi 1, VI
Avg
71.1
67.4
71.3
1...j performs better than other weight values as reported in Table
8 Conclusion
This study presented novel adversarial algorithm for DG called MADG. Our approach is inspired by a theoretical framework that formulates the relationship between the source and unseen domains in real-valued hypothesis space. We used the margin loss function and Rademacher complexity framework to develop generalization bound, which is informative. The proposed bound also shows a depen- dency between the margin value and the generalization error, which can be easily translated into an algorithm. To this end, we developed the MADG algorithm that leverages the MDD metric with minimax optimization to learn domain- invariant features and generalize well to the unseen domain. The algorithm was studied on five well-known DG benchmark datasets and reported consistent performance across all datasets as compared to other methods. We also report several empirical analyses and ablation studies.
Table 8: Accuracy(%) of the MADG with dif- ferent wi values. One:
My = 1. Avg.=Average. Dyn.=Dynamic
One
Avg.
Dyn.
P
97.7
97.8
97.6
A
87.8
87.3
87.5
C
82.2
82.0
82.4
S
78.3
77.4
77.2
Avg
86.5
86.1
86.2
10

---

Acknowledgments
The research of AD is supported in part by the Prime Minister Research Fellowship. Ministry of Education, Government of India. The research of LRC is supported by the Indo-Norwegian Systems (INCAPS) project: 287918 of the International Partnerships for Excellent Education, Research and Innovation (INTPART) program and the Low- Altitude UAV Communication and Tracking (LUCAT) project: 280835 of the IKTPLUSS program from the Research Council of Norway. The research of AK is supported by TiHAN Faculty Fellowship. VNB would like to acknowledge the support through the Govt of India SERB IMPRINT and DST ICPS funding programs for this work. We are grateful to the anonymous reviewers for the feedback that helped improved the presentation of this paper.
Collaboration in Autonomous
References
[1] K. Aggarwal, Singh. M. Chopra, Kumar, and Colace, "Deep learning robotics for strengthening industry 4.0.: opportunities, challenges and future directions." Robotics and AI for Cybersecurity and Critical Infrastructure in Smart Cities, pp. 1-19, 2022.
[2] M. Tsuneki, "Deep learning models in medical image analysis," Journal of Oral Biosciences, vol. 64, no. 3, pp. 312-33332033 12-3200, 2022.
[3] T. Ayoub Shaikh, T. Rasool, and F. Rasheed Lone, "Towards leveraging the role of machine learning and artificial intelligence in precision agriculture and smart farming," " Computers and Electronics in Agriculture, vol. 198, p. 107119, 2022.
[4] J. Quionero-Candela. M. Sugiyama, A. Schwaighofer, and N. D. Lawrence, Dataset Shift in Machine Learning. The MIT Press, 2009.
[5] Wang, Lan, C. Liu, Ouyang, T. Qin, Lu, Chen, W. Zeng, and Yu, "Generalizing to unseen domains: A survey domain generalization." IEEE Transactions on Knowledge and Data Engincering, 2022.
[6] Y. Zhang, "A survey of unsupervised domain adaptation for visual recognition," ar preprint arXiv:21 1xXiv:211200 12.06745, 2021.
[7] X. Liu, C. Yoo, F. Xing, H. Oh, G. El Fakhri, J.-W. Kang, J. Woo et al., "Deep unsupervised domain adaptation: A review of recent advances and perspectives." APSIPA Transactions on Signal and Information Processing, vol. 11, no. 1, 2022.
[8] G. Blanchard, G. Lee, and C. Scott, "Generalizing from several related classification tasks to new unlabeled sample," in Advances Neural Information Processing Systems, vol. 24, 2011.
[9] T. Matsuura and T. Harada, "Domain generalization using a mixture of multiple latent domains," in Proceedings of the AAAI Conference Artificial Intelligence, vol. 34, no. 07, 2020, pp. 11 749-11 756.
[10] S. Lin, C.-T. Li, and A. C. Kot, "Multi-domain adversarial feature generalization for person re-identification." IEEE Transactions on Image Processing, vol. 30, pp. 1596-1607, 2020.
[11] Z. Deng, F. Ding, C. Dwork, R. Hong, G. Parmigiani, P. Patíl, and P. Sur, "Representation via representations: Domain generalization via adversarially learned invariant representations." arXiv preprint arXiv:2006. 1478, 2020.
[12] Zhao, M. Gong, Liu, H. Fu, and Tao, "Domain generalization via entropy regularization," Advances in Neural Information Processing Systems, vol. 33, pp. 16096-16 16 107, 2020.
[13] K. Akuzawa, Y. Iwasawa, and Y. Matsuo, "Adversarial invariant feature learning with accu- racy constraint for domain generalization," in Machine Learning and Knowledge Discovery in Databases: European Conference. Springer, 2020, pp. 315-331.
[14] E. Rosenfeld, P. Ravikumar, and A. Risteski, "An online learning approach to interpolation and extrapolation in domain generalization," in International Conference on Artificial Intelligence and Statisties. PMLR, 2022, pp. 2641-2657.

---

[15] M.-H. Bui, T. Tran, A. Tran, and D. Phung. "Exploiting domain-specific features to enhance domain generalization." Advances in Neural Information Processing Systems, vol. 34, pp. 21 189-21 201, 2021.
[16] V. Narayanan, A. A. Deshmukh, U. Dogan, and V. N. Balasubramanian, "On challenges in unsupervised domain generalization." in NeurlPS 2021 Workshop on Pre-registration in Machine Learning. PMLR, 2022, pp. 42-58.
[17] Y. Shu, Z. Cao, C. Wang, J. Wang, and M. Long, "Open domain generalization with domain- augmented meta-learning." " in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 9624-9633.
[18] Yuan, X. Ma, D. Chen, K. Kuang, F. Wu, and I.. Lin, "Laael--effcieet domain generalization via collaborative exploration and generalization." in Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 2361-2370.
[19] S. Paul, T. Dutta, and S. Biswas, "Universal cross-domain retrieval: Generalizing across classes and domains," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 12 12055-12 064.
[20] Q. Xu, R. Zhang, Y. Zhang. Wang, and Q. Tian, "A fourier-based framework for domain generalization," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 14 14332..
[21] Z. Li, Z. Cui, S. Wang, Y. Qi, X. Ouyang, Q. Chen, Y. Yang, Z. Xue, D. Shen, and J.-Z. Cheng. "Domain generalization for mammography detection via multi-style and multi-view con- trastive learning." in Medical Image Computing and Computer Assisted Intervention-MICCAI Proceedings, 2021, pp. 98-108
[22] D. Kim, Y. Yoo, S. Park, J. Kim, and J. Lee, "Selfreg: Self-supervised contrastive regulariza- tion for domain generalization," Proceedings the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9619-9628.
[23] Y. Chen, Y. Wang, Y. Pan, Yao, X. Tian, and T. Mei, "A style and semantic memory mech- anism for domain generalization." in Proceedings of the IEEE/CVE International Conference on Computer Vision, 2021, pp. 9164-9173.
[24] D. Li, H. Gouk, and T. Hospedales, "Finding lost dg: Explaining domain generalization via model complexity," arXiv preprint arXiv:2202.00566 2022.
[25] M. Zhang, H. Marklund, N. Dhawan, A. Gupta, S. Levine, and C. Finn, "Adaptive risk minimiza- tion: Learning to adapt to domain shift," Advances in Neural Information Processing Systems, vol. 34, pp. 23 664-23 678, 2021.
[26] A. Nasery, S. Thakur, V. Piratla, A. De, and S. Sarawagi, "Training for the future: A simple gra- dient interpolation loss to generalize along time," Advances in Neural Information Processing Systems, vol. 34, pp. 19 198-19 209, 2021.
[27] J. Cha, S. Chun, K. Lee, H.-C. Cho, S. Park, Y. Lee, and S. Park, "Swad: Domain generalization by seeking flat minima," Advances in Neural Information Processing Systems, vol. 34, pp. 22 405-205-444 418, 2021.
[28] A. Robey, G. J. Pappas, and Hassani, "Model-based domain generalization." Advances in Neural Information Processing Systems, vol. 34, pp. 20 210-20222, 2021.
[29] Cha, K. Lee, S. Park, and S. Chun, "Domain generalization by mutual-information regulariza- tion with pre-trained models," in Proceedings of the European Conference on Computer Vision (ECCV), 2022, pp. 440-457.
[30] C. Fang, Y. Xu, and D. N. Rockmore, "Unbiased metric learning: On the utilization of mul- tiple datasets and web images for softening bias," in 2013 IEBE International Conference on Computer Vision, 2013, pp. 1657-1664.
12

---

[3]] D. Li, Y. Yang, Y.-Z. Song, and T. M. Hospedales, "Deeper, broader and artier domain gener- alization." " in Proceedings of the IEEE international conference on computer vision, 2017, pp. 5542-5550.
[32] H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan, "Deep hashing network for unsupervised domain adaptation." " in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 5018-5027.
[33] S. Beery, G. Van Horn, and P. Perona, "Recognition in terra incognita," in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 456-473.
[34] X. Peng. Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang. "Moment matching for multi- source domain adaptation." in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1406-1415.
[35] I. Gulrajani and D. Lopez-Paz, "In search of lost domain generalization," in 9th International Conference on Learning Representations, ICLR, 2021.
[36] A. Gretton, A. Smola, J. Huang, M. Schmittfull, K. Borgwardt, and B. Schölkopf, "Covariate Shift by Kernel Mean Matching," in Dataset Shift in Machine Learning. The MIT Press, 2008.
[37] J. Lin, Y. Tang, Wang, and W. Zhang, "Mitigating both covariate and conditional shift for domain generalization." " in 2022 IEEE 8th International Conference on Cloud Computing and Intelligent Systems (CCIS). IEEE, 2022, pp. 437-443.
[38] S. Been-David, Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan, "A theory of learning from different domains," Machine learning. vol. 79, no. 1, pp. 151-175, 2010.
[39] V. Koltchinskii and D. Panchenko, "Empirical margin distributions and bounding the gen- eralization error of combined classifiers," The Annals of Statistics, vol. 30, no. 1, pp. 1-50, 2002.
[40] Y. Zhang, T. Liu, M. Long, and M. Jordan, "Bridging theory and algorithm for domain adapta- tion," in Proceedings of the 36th International Conference on Machine Learning, vol. 97, 2019, pp. 7404-7413.
[41] P. L. Bartlett and S. Mendelson. "Rademacher and gaussian complexities: Risk bounds and structural results," " Journal of Machine Learning Research, vol. 3, no. Nov. pp. 463-482, 2002.
[42] M. M. Rahman, C. Fookes, M. Baktashmotlagh, and S. Sridharan, "Correlation-awan ad- versarial domain adaptation and generalization," Pattern Recognition, vol. 100, p. 107124, 2020.
[43] G. Blanchard, G. Lee, and C. Scott, "Generalizing from several related classification tasks to new unlabeled sample," Advances in neural information processing systems, vol. 24, 2011.
[44] K. Muandet, D. Balduzzi, and B. Schölkopf, "Domain generalization via invariant feature representation," in International conference on machine learning. PMLR, 2013, pp. 10-18.
[45] A. A. Deshmukh, Y. Lei, S. Sharma, U. Dogan, J. W. Cutler, and C. Scott, "A generalization error bound for multi-class domain generalization," arXiv preprint arXiv: 1905. 10392, 2019.
[46] S. Hu, K. Zhang, Z. Chen, and L. Chan, "Domain generalization via multidomain discriminant analysis," in Uncertainty in Artificial Intelligence. PMLR, 2020, pp. 292-302.
[47] G. Blanchard, A. A. Deshmukh, Ü. Dogan, G. Lee, and C. Scott, "Domain generalization by marginal transfer learning," The Journal of Machine Learning Research, vol. 22, no. 1, pp. 46-100, 2021.
[48] I. Albuquerque, J. Monteiro, M. Darvishi, T. Falk, and I. Mitliagkas, "Generalizing to unseen domains via distribution matching." arXiv preprint arXiv: 1911 .008044 2019.
[49] H. Ye, C. Xie, T. Cai, R. Li, Z. Li, and L. Wang, "Towards a theoretical framework of out-of- distribution generalization." Advances in Neural Information Processing Systems, vol. 34, pp- 539---23 531, 2021.
13

---

[50] R. Vedantam, D. Lopez-Paz, and D. J. Schwab, "An empirical investigation of domain general- ization with empirical risk minimizers," Advances in Neural Information Processing Systems. vol. 34, pp. 28 131-28 143, 2021.
[51] M. Mohri, A. Rostamizadeh, and A. Talwalkar, Foundations of machine learning. MIT press, 2018.
[52] Y. Ganin and V. Lempitsky, "Unsupervised domain adaptation by backpropagation." in International conference on machine learning. PMLR, 2015, pp. 1180-1189.
[53] A. Rame, C. Dancette, and M. Cord, "Fishr: Invariant gradient variances for out-of-distribution generalization," in International Conference Machine Learning. PMLR, 2022, pp. 18 347- 18 377.
[54] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition." in Proceedings of the IEEE conference computer vision and pattern recognition, 2016, pp.
770-778.
[55] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., "Imagenet large scale visual recognition challenge," International journal of computer vision, vol. 115, pp. 21 211-252, 2015.
[56] N. Qian, "On the momentum term in gradient descent learning algorithms," Neural networks, vol. 12, no. 1, pp. 145-151, 1999.
[57] V. N. Vapnik, "An overview of statistical learning theory." IEEE transactions on neural networks. vol. 10, no. 5, pp. 988-999, 1999.
[58] B. Sun and Saenko, "Deep coral: Correlation alignment for deep domain adaptation," " in Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part 14. Springer, 2016, pp. 443-450.
[59] Y. Ganin, Ustinova, H. Ajakan, Germain, H. Larochelle, Laviolette, M. Marchand. and V. Lempitsky, DDomain-adverseee learning research, vol. 17, no. 1, pp. 2096-2030, 2016.
training of neural networks," The journal of machine
[60] Y. Li, X. Tian, M. Gong, Y. Liu, Liu, Zhang, and D. Tao, "Deep domain generalization via conditional invariant adversarial networks," in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 624-639.
[61] D. Li, Y. Yang, Y.-Z. Song, and T. Hospedales, "Learning generalize: Meta-learning for do- main generalization." in Proceedings of the AAAI conference on artificial intelligence, vol. 32, no. 2018.
[62] H. Li, J. Pan, S. Wang, and A. C. Kot, "Domain generalization with adversarial feature learning," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 5400-5409.
[63] M. Arjovsky, I.. Bottou, I. Gulrajani, and D. Lopez-Paz, "Invariant risk minimization," arXiv preprint 190709990, 2019.
[64] S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang, "Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization," arXiv preprint arXiv: 191 .08731, 2019.
[65] S. Yan, H. Song. N. Li, L. Zou, and L. Ren, "Improve unsupervised domain adaptation with mixup training." arXiv preprint arXiv:2001.0066 2020.
[66] M. Zhang. H. Marklund, A. Gupta, S. Levine, and C. Finn, "Adaptive risk minimization: A meta-learning approach for tackling group shift," arXiv preprint arXiv:2007.02931 vol. 8, p. 9, 2020.
[67] Z. Huang, H. Wang, E. P. Xing, and D. Huang, "Self-challenging improves cross-domain generalization," " in Computer VCCV 2020: 16th European Conference, Glasgow, UK,
August 23-28, 2020, Proceedings, Part II 16. Springer, 2020, pp. 124-140.
14

---

[68] H. Nam, H. Lee. J. Park, W. Yoon, and D. Yoo, "Reducing domain gap by reducing style bias," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021,
pp. 1690-8699.
[69] D. Krueger, E. Caballero, J.-H. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. Le Priol, and A. Courville, "fustoo-distiibuttin generalization via risk extrapolation (rex)," in International Conference Machine Learning. PMLR, 2021, pp. 5815-5826.
[70] G. Parascandolo, A. Neitz, A. Orvieto, L. Gresele, and B. Schölkopf, "Learning explanations that are hard to vary," in 9th International Conference on Learning Representations, ICLR, 2021.
[71] Y. Shi, J. Seely, H. Torr, N. Siddharth, A. Hannun, N. Usunier, and G. Synnaeve, "Gradient matching for domain generalization." arXiv preprint arXiv:2104.09937, 2021.
[72] S. Shahtalebi, J.-C. Gagnon-Auder, T. Laleh, M. Faramarzi, K. Ahuja, and I. Rish, "Sand- mask: An enhanced gradient masking strategy for the discovery of invariances in domain generalization," ArXiv, vol. abs/2 abs/2106.02266 02266, 2021.
[73] G. Zhang, H. Zhao, Y. Yu, and P. Poupart, "Quantifying and improving transferability in domain generalization," Advances in Neural Information Processing Systems, vol. 34, pp. 10957-10970, 2021.
[74] M. Pezeshki, O. Kaba, Y. Bengio, A. C. Courville, D. Precup, and G. Lajoie, "Gradient star- vation: A learning proclivity in neural networks," Advances Neural Information Processing Systems, vol. 34, pp. 1272, 2021.
15


Reviews:
[{"review_id": "F1ULTmZjMA", "reviewer": "Reviewer_u5h1", "summary": "This paper investigates domain generalization (DG) using a margin-based theoretical framework. The authors first formulate the generalization upper bound by leveraging the margin disparity discrepancy (MDD). Then, an adversarial learning strategy (MADG) is devised to minimize the empirical MMD between source domains. The experiments on the DomainBed benchmark further demonstrate that the proposed MADG could outperform the current state-of-the-art methods.", "questions": "1. Table 1 could be simplified, as many baselines are unnecessary.\n2. The common approach to compare DG methods is the training-domain validation. Why do the authors mainly utilize the test-domain selection?\n", "limitations": "See above comments. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written, and the method is well-designed. \n2. The theory of margin-based generalization bound is interesting and could benefit the further designs of DG methods. \n3. The proposed MADG could outperform recent state-of-the-art methods.\n", "weaknesses": "1. The primary concern of the reviewer is in its effectiveness. While MADG outperforms most baselines in the DomainBed benchmark, the improvement is marginal/minor. This made it unclear to me if the method actually works better, or it is just a product of optimizing some hyperparameters. \n2. The motivation, “very little work has been done in developing DG algorithms that are well-motivated by theoretical”, lacks soundness. In fact, there have been notable works in the literature that utilize theoretical frameworks to analyze the generalization problem, such as gradient matching [71, 88] and invariant risk minimization [80]. It could be beneficial if the authors could provide a detailed comparison with such theoretical methods.\n3. In addition to the accuracy comparison, could the authors provide a more in-depth analysis regarding the training dynamics? This would help shed light on the inner workings of the MADG approach.\n4. Many advantages, such as efficient optimization, stated in the abstract are not well verified. Specifically, following Eq. (17), it seems that the estimation process could introduce a significant computational cost, as it iteratively computes for all domains. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments and suggestions. We respond below to each of the concerns/suggestions.\n> 1. The primary concern of the reviewer is in its effectiveness. While MADG outperforms most baselines in the DomainBed benchmark, the improvement is marginal/minor. This made it unclear to me if the method actually works better, or it is just a product of optimizing some hyperparameters.\n> \n**Response:** We understand the concern. The current dataset benchmark in domain generalization, DomainBed [32, ICLR 2021], is a challenging one where one model may not outperform all other models across all the constituent datasets (OfficeHome, PACS, VLCS, TerraIncognita, DomainNet) by a significant margin. Even recent papers, SAND-mask [89, ICML 2021 workshop], Fish [88, ICLR 2022] and Fishr [71, ICML 2022] showcase small improvements across these datasets. To further analyze a model’s effectiveness and consistency across these datasets, we evaluate each model with three other metrics (AD, GD, M) as discussed in Sec 6. The proposed model outperforms other models on all three metrics along with the average accuracy metric.\n> 2. The motivation, “very little work has been done in developing DG algorithms that are well-motivated by theoretical”, lacks soundness. In fact, there have been notable works in the literature that utilize theoretical frameworks to analyze the generalization problem, such as gradient matching [71, 88] and invariant risk minimization [80]. It could be beneficial if the authors could provide a detailed comparison with such theoretical methods.\n> \n**Response:** We apologize for not being clearer on this. The gradient matching papers [71,88] propose the idea of learning invariant features by domain-level gradient matching; in particular, [71] use gradient variance matching, and [88] use gradient mean matching. These papers don't develop generalization bounds for the unseen domain; instead, they theoretically analyze the inconsistency score function to find optimal weights across all domains. Even though [80] develops a generalization bound for the unseen domain, it uses a different causal mechanism-based approach. In this work, we present a new perspective to the problem based on margin-based bounds and adversarial learning, which considers convexity relations between domains and the alignment and discrepancy between these domains. We will revisit the phraseage as suggested.\n\n> 3. In addition to the accuracy comparison, could the authors provide a more in-depth analysis regarding the training dynamics? This would help shed light on the inner workings of the MADG approach.\n> \n**Response:** The key aspects of training dynamics have been analyzed in Sec. 5 (L252-258) and Eq. (20) the optimization problem consists of two loss terms, Classification and Transfer Loss, that are updated in an two-step training methodology. Fig. 1 and 2 in rebuttal.pdf shows the classification and transfer loss plots for the Art and Product domain of OfficeHome.\n\n> 4. Many advantages, such as efficient optimization, stated in the abstract are not well verified. Specifically,.. Eq. (17), it seems that the estimation process could introduce a significant computational cost, as it iteratively computes for all domains.\n> \n**Response:** As seen from Eqns (5) and (6), the MDD between two domains is defined as the supremum over only one variable (f\\`) as opposed to other discrepancies used in related work such as H$\\Delta$H, which involves a supremum over two variables. This makes MDD relatively more efficient to optimize when compared to other discrepancy measures; we will clarify this. In terms of computational cost, following Eq.(17), while we iteratively compute MDD over all domains, our computational cost during training is similar to other benchmark methods as shown below in Table 1 in rebuttal.pdf. (To add, even simple methods like ERM and Mixup have running times in similar ranges.)\n\n> Table 1 could be simplified, as many baselines are unnecessary.\n> \n**Response**: Thank you for the suggestion, we will simplify the table by either dividing it into sections or moving some results to the appendix.\n\n> The common approach to compare DG methods is the training-domain validation. Why do the authors mainly utilize the test-domain selection?\n> \n**Response:** As stated in Appendix A2 (L85-86), we follow recent state-of-the-art work [71] (also supported by recent papers [refs A1-A3]) in using the 'Test-domain' method for hyperparameter selection. These recent efforts also discuss reasons for this choice: (i) [71] argues that learning causal mechanisms is not useful in the Training-domain method, esp when correlations are more predictive in training than causal features. The paper suggests that using 'Test-domain' selection is more realistic since a user will easily label a few target samples to validate a model's generalization performance before deploying it; (ii) [A1] states that In-domain (ID) validation sets (Training-domain) eliminates the advantages of using DG-specific models over ERM models, as ID accuracy is often at odds with generalization accuracy. In our work, restricting the $\\rho$ value selection using ID validation can hurt the model's generalization performance as shown in the relationship between margin and generalization in Thm 3; (iii) [A2] argues that high ID accuracy can be achieved by depending on spurious patterns and is not indicative of generalization performance; (iv) [A3] states that Training-domain method suffers from underspecifications as a reason for the failure of ML systems deployed in the real world. It defines underspecification as a scenario where distinct predictors, primarily validated on Training-domain method, report equivalently strong held-out performance but behave differently in test. We follow these recent efforts in using the test-domain method. (We also reported train-domain results in the Appendix, although that was not our focus in this work.)"}, {"review_id": "J70mCqOdVW", "reviewer": "Reviewer_wf8A", "summary": "This paper is commendable for its innovative use of a margin-based theoretical framework to solve domain generalization problems, which contrasts with the largely heuristic and empirical approaches adopted by existing methods. By grounding their approach in a theoretical foundation, the authors provide more interpretable solutions that can contribute to the field of domain generalization. ", "questions": "Please see weakness", "limitations": "The authors have well described their limitations.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The claim presented in the paper is strongly supported by theoretical analysis, as well as the extensive ablation studies presented in section 7. This speaks to the methodological rigour of the study and strengthens its credibility.\n- The development of a theoretical framework for solving domain generalization problems is indeed a significant strength of this paper. This approach not only enhances the comprehensibility and reproducibility of the method, but also contributes to the broader understanding of the problem.", "weaknesses": "- While the average performance of the proposed method is the highest, the performance gap across different tasks is quite significant. The method shows the best performance on the Office Home dataset, but this alone is not enough to convincingly demonstrate the overall superiority of the method. Therefore, the claimed significance of the proposed method appears to be overstated.\n- The authors' claim that a theoretical approach is necessary is quite a strong statement that seems to undermine the importance of empirically validated methods in the field. This makes it hard to fully agree with their motivation.\n- While the authors make a reasonable argument for the use of margin as a metric in generalization, they do not provide a sufficient justification for its relevance in domain generalization problems, particularly where style shifts are involved. This makes it difficult to understand the authors' motivation for their margin-based approach.\n- Similarly, the reasoning behind the use of adversarial learning (min-max framework) is not clearly explained. The proposed method appears to be similar to the one used in [1], and a clear explanation of the differences between these two methods, apart from the task, would be beneficial.\n- The paper lacks a comparison with recent Domain Generalization (DG) works [2-4]. This makes it difficult to assess how the proposed method stacks up against the state of the art.\n\n[1] Maximum Mean Discrepancy Test is Aware of Adversarial Attacks\n\n[2] Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution\n\n[3] Domain Generalization by Mutual-Information Regularization with Pre-trained Models\n\n[4] SIMPLE: Specialized Model-Sample Matching for Domain Generalization", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments and suggestions. We respond below to each of the concerns/suggestions.\n\n> While the average performance of the proposed method is the highest, the performance gap across different tasks is quite significant.....\n\n**Response:** We agree and in Sec 6 (L288- 292), we discuss this concern and in order to reward model's consistent performance across datasets, we include three other metrics (AD, GD, M), discussed in Sec 6, which reward consistent performance across benchmarks. As seen from Table 1 in main paper, we outperform all earlier methods on these three metrics and the average accuracy metric. Besides, the current dataset benchmark in domain generalization, DomainBed [32, ICLR 2021], is a challenging one where one model may not outperform all other models across all the constituent datasets (OfficeHome, PACS, VLCS, TerraIncognita, DomainNet) by a significant margin. Even recent papers, SAND-mask [89, ICML 2021 workshop], Fish [88, ICLR 2022] and Fishr [71, ICML 2022] showcase similar improvements across these datasets. \n\n> The authors' claim that a theoretical approach is necessary is quite a strong statement that seems to undermine the importance of empirically validated methods in the field....\n\n**Response:** In hindsight, we agree with this observation, and thank you for pointing this out. We will revisit the phrasage and temper our claim. Our overall intention was to suggest that having strong theoretical underpinnings along with improved performance is an added advantage for a method. As discussed in Sec 1 (L39-42), our motivation was to develop an adversarial DG algorithm that uses a margin-based discrepancy because of its advantages, discussed in Sec 4. Such an approach has not been done for DG before. We also develop a generalization bound for unseen domain based on margin loss as they are more informative than 0-1 loss based bounds. \n\n> While the authors make a reasonable argument for the use of margin as a metric in generalization, they do not provide a sufficient justification for its relevance in DG problems, particularly where style shifts are involved....\n>\n**Response:** We apologize if this did not come through clearly. As stated in Sec. 4 (L136- 139), one advantage of using margin-based discrepancy (MDD) is that it considers the task (classifier) in its formulation as shown in Eq (5). MDD learns invariant features across domains based on the idea of a bi-classifier prediction. Let's consider the MDD between the two domains ($D_{s_i}$,$D_{s_k}$) as shown in Eqns (5)&(22) (consider the case where $N_s=2$ and $j=1$). We have two different classifiers (scoring functions) that take input from the feature extractor and try to classify $D_{s_i}$ samples correctly, whereas for $D_{s_k}$ samples only classifier $f$ tries to classify it correctly. Simultaneously the two classifiers are also trained to detect $D_{s_k}$ samples far from $D_{s_i}$'s support. This is possible because $D_{s_k}$ samples far from $D_{s_i}$’s support cant be easily discriminated by $f'$ classifier  The feature extractor then tries to generate $D_{s_k}$ features such that they are close to the support of $D_{s_i}$. In this way, we align the domains and learn domain-invariant features. We will clarify this.\n\n> Similarly, the reasoning behind the use of adversarial learning (min-max framework) is not clearly explained. ....a clear explanation of the differences between this and the one used in [1], apart from the task, would be beneficial.\n> \n**Response:** As discussed in Sec 5 (L243-251), we formulate our objective as a minimization problem as shown in Eq (18). We then expand the second term (Empiricial MDD) in this problem using Eq (6). As empirical MDD is defined as the supremum over $f'$ (maximization), the overall optimization now becomes a minimax game. We hence use adversarial learning to solve our objective function. We will explain this clearly. \n\nWe note that our discrepancy, MDD, is different from the more common MMD. The paper [1] uses Maximum Mean Discrepancy (MMD), whereas, in our paper, we use Margin Disparity Discrepancy (MDD), which is fundamentally different from MMD formulation as shown in Eqns (5) and (6).\n\n>The paper lacks a comparison with recent DG works [2-4]. This makes it difficult to assess how the proposed method stacks up against the state of the art.\n>\n**Response:** As in Sec 6 (L278-280) we evaluate our model on five different datasets from DomainBed benchmark [32]. To the best of our knowledge, this is fairly comprehensive, when compared to most recent efforts. To answer this further, [2] evaluates only on the Domainnet dataset of the DomainBed benchmark, and also uses a smaller version of this dataset (3 instead of 6 domains, and 40 classes instead of 345). Note that DomainBed consists of 5 datasets (OfficeHome, PACS, VLCS, TerraIncognita, DomainNet). To provide a comparison with [2], we train our model on their version of the domainnet dataset. Table 1 in attached rebuttal.pdf shows the results. We see that our model outperforms LP-FT [2] on all three domains. As stated in Appendix A2 (L85-86), we follow recent state-of-the-art work [71] (also supported by recent papers [A1-A3]) in using the `Test-domain` methodology for hyperparameter selection. [3] proposes MIRO model and evaluates only for  a training-domain model selection method. For fair comparision, we re-run the MIRO model for three trials on our setting. From Table 2 in rebuttal.pdf, we see that MADG outperforms MIRO on two domains and achieves higher average accuracy. The paper [4] proposes an ensemble algorithm (collection of multiple methods) for DG. Our method is a non-ensemble one, and can be a part of their method by itself. "}, {"review_id": "AfCtO4zW6G", "reviewer": "Reviewer_FJmm", "summary": "- The paper presents a margin-loss based analysis of domain generalization.\n- First, the paper derives a bound on margin disparity discrepancy (MDD) of any unseen domain within the convex hull of source domains in terms of the margin loss on source domains, ideal margin loss, and max MDD between any two source domains.\n- The paper relates this to any unseen domain via projection onto the convex hull of source domains, and an additional factor $\\gamma$\n- The paper then realizes the bound for the empirical setting using a bound based on the Rademacher complexity of the function class\n- Based on this bound, the paper proposes an adversarial learning method to regularize training by minimizing MDD in addition to classification loss.\n- The paper demonstrates the efficacy of the method on a variety of real world datasets in the DomainBed benchmark.", "questions": "Why is only the feature extractor trained to minimize MDD loss and not the classifier (line 261)?\n\nTheorem 2 shows that the error of any unseen domain can be bounded via a projection onto the convex hull of the source domain, by adding in an additional factor $\\gamma$. It is not clear if this factor is small or very large for domain generalization benchmarks. Is there intuition for “how far” an unseen domain in a benchmark is to the source domains?", "limitations": "It is not completely clear how significant the empirical results are. The method slightly outperforms ERM on average, but ERM outperforms the method on 2 out of 5 considered datasets, and is competitive on 2 others. Additionally, the reliance on labelled OOD data for model selection limits the conclusions one can draw about the efficacy of the method on the DomainBed benchmark.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a principled method for domain generalization, and the empirical method follows neatly from the derived generalization bound.\n2. The proposed bound for error on an unseen domain based on margin disparities between source domains is novel to my knowledge. \n3. Empirical results show modest improvements over ERM.\n4. The paper presents empirical ablations for some design choices used in the algorithm.", "weaknesses": "It is not clear if the single optimization step for the adversarial models f’ is sufficient for tightly approximating MDD. The method effectively regularizes a lower bound on the MDD term, which requires the adversarial models to be effective maximizers — this may require many steps for the adversarial models per main-model step.\n\nTable 1 presents results using the model selection strategy of [71], which uses out-of-distribution data as the validation set for picking hyperparameters. Using labelled OOD data for hyperparameter selection limits the conclusions we can draw about if the proposed method works in practice, where we do not usually have access to labelled test data. While the AD, GD, and M metrics allay some of this concern, presenting results with some methodology that could be used for model-selection in practice, such as leave-one-out domain validation would make the empirical results stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments and suggestions. We respond below to each of the concerns/suggestions.\n\n> It is not clear if the single optimization step for the adversarial models f’ is sufficient for tightly approximating MDD....this may require many steps for the adversarial models per main-model step.\n>\n**Response:** As discussed in Algorithm 1, each classifier f' (Adversarial model) along with the feature extractor is updated once per main-model step. We had found this to be sufficient in our empirical studies. To however answer this further, we report results on the Art domain in OfficeHome dataset with additional adversarial updates per main-model step (t) in Table 1 below. (We had observed similar trends in our earlier studies.) As in the table, adversarial update with t=3 achieves only marginally better performance for the added computational cost.\n\nTable 1: Accuracy (%) for Art domain for different adversarial updates per main-model step\n|\\# Adversarial updates| Art| \n|--|--|\n|t=1| 68.6 |\n|t=3|68.7|\n\n>Table 1 presents results using the model selection strategy of [71], which uses OOD data as validation set for picking hyperparameters. Using labelled OOD data for hyperparameter selection limits the conclusions ..... While D, GD, and M metrics allay some of this concern, presenting results with some methodology that could be used for model-selection in practice would make empirical results stronger.\n\n**Response:** As stated in Appendix A2 (L85-86), we follow the recent state-of-the-art work [71] (also supported by recent papers [refs A1-A3]) in using the 'Test-domain' method for hyperparameter selection - primarily for fair comparison of results. (We also reported train-domain results in the Appendix, although that was not our focus in this work.)\nThese recent efforts also discuss reasons for this choice, such as avoiding over-reliance on spurious correlations [71, A2] and the underspecification issue as a key reason for failure of deployed ML systems [A3]. Also, [71] argues that it is more realistic that a user will label a few target samples to validate their model's generalization performance before deploying it in practice. We followed these recent efforts in using the test-domain method.\n\n> Why is only the feature extractor trained to minimize MDD loss and not classifier (line 261)?\n> \n**Response:** As seen in Eqns (5) and(6), MDD is defined as the supremum over only the $f'$ scoring function  while the other function $f$ is fixed. Thus, following this definition, our final optimization problem, Eq (22), maximizes MDD in terms of the $f'$ classifier and minimizes MDD in terms of the feature extractor ($\\mathcal{G}$). The classifier $f$ is fixed when optimizing the MDD loss. \n\n>Theorem 2 shows that error of any unseen domain can be bounded via a projection onto the convex hull of the source domain, by adding an additional factor $\\gamma$. It is not clear if this factor is small or large for DG benchmarks. Is there intuition for “how far” an unseen domain in a benchmark is to the source domains?\n>\n**Response:** As stated in Theorem 2 and Remark 3, $\\gamma$ is defined as the projection of the unseen domain on to the convex hull. It is also defined in Thm 2 as the MDD between the convex hull and the unseen domain, which can be further approximated with a combination of two different cross entropy functions as in Eq (20). This equation is equivalent to a $\\hat{\\rho}$ balanced Jensen-Shannon (JS) Divergence as shown in Propn D.1. in [37]. Thus we can get an approximation of this projection by computing the JS divergence between the two domains (distributions). To answer this better, we studied this paiwise JS divergence between two domains in the OfficeHome dataset, as shown in Table 1 below. Such an analysis provides us with an intuitive value for $\\gamma$ by summing the values across the rows in Table 1. We can see from Table 1 that the unseen accuracy is low when the $\\gamma$ value is high (expectedly). This is because higher $\\gamma$ values incidcate that the unseen domain is far away from the convex hull and hence harder to generalize.  We can see a similar trend for PACS dataset in Table 2 below.\n\nTable 1: Pairwise JS divergence for the OfficHome dataset and the approximated projection value ($\\gamma$) of the unseen domain on the convex hull.\n|OfficeHome|Art|Clipart|Product|Real World| ~$\\gamma$|Accuracy(%)|\n|-----|-----|-----|-----|-----|-----|-----|\n|Art|0|0.36 |0.35|0.09|0.8|68.6|\n|Clipart|0.36|0|0.31|0.41|1.1|55.5|\n|Product|0.35|0.31|0|0.37|1.0|79.6|\n|Real World|0.09|0.41|0.37|0|0.87|81.5|\n\nTable 2: Pairwise JS divergence for the PACS dataset and the approximated projection value ($\\gamma$) of the unseen domain on the convex hull.\n|PACS|Art_painting|Cartoon|Photo|Sketch| ~$\\gamma$|Accuracy(%)|\n|-----|-----|-----|-----|-----|-----|-----|\n|Art_painting|0|0.34 |0.08|0.72|1.18|87.8|\n|Cartoon|0.34|0|0.4|0.41|1.24|55.5|\n|Photo|0.08|0.4|0|0.74|1.21|79.6|\n|Sketch|0.72|0.46|0.74|0|1.92|81.5|\n\n>It is not completely clear how significant the empirical results are....\n>\n**Response:** We understand the concern. The current dataset benchmark in domain generalization, DomainBed [32, ICLR 2021], is a challenging one where one model may not outperform all other models across all the constituent datasets (OfficeHome, PACS, VLCS, TerraIncognita, DomainNet) by a significant margin. Even recent papers, SAND-mask [89, ICML 2021 workshop], Fish [88, ICLR 2022] and Fishr [71, ICML 2022] showcase small improvements across these datasets. To further analyze a model’s effectiveness and consistency across these datasets, we evaluate each model with three other metrics (AD, GD, M) as discussed in Sec 6. The proposed model outperforms other models on all three metrics along with the average accuracy metric. As discussed in Appendix A2 (L85-86), we follow recent state-of-the-art work [71] (also supported by recent papers [A1-A3]) for our experimental settings, for fair comparison."}, {"review_id": "J7v2dJ4Bzi", "reviewer": "Reviewer_MRUq", "summary": "The paper proposes a new adversarial learning objective using a margin based approach for domain generalization. The goal of domain generalization broadly is to build classifiers that are trained on one or more source domains, and are expected to generalize to an unseen target domain. The key idea is to leverage Margin disparity discrepancy (MDD) which quantifies the level of disagreement between decision boundaries of classifiers using their margins. MDD is used as a proxy to understand generalizability of classifiers in this context across multiple domains. Next, the paper establishes an upper bound on the generalization error on any unseen target domain that is within the convex hull of the source domains and MDD. In other words, the objective is to get the decision boundaries across different source domains to agree as much as possible, while also minimizing the empirical errors on them simultaneously.", "questions": "See above", "limitations": "some what, yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The margin perspective for domain generalization is a fresh perspective to this, and the paper takes an interesting approach at using an adversarial learning strategy to optimizing this problem. \n* Detailed theoretical setup and formulation, which builds on existing work and setup the formulation for MADG\n* Results are impressive — on several benchmarks, the proposed method appears to perform competitively. \n* good ablations and analysis of the proposed method.", "weaknesses": "* I found the paper to be hard to read in general, at its core, the paper is proposing to model error on the unseen target using a convex hull of the source domains — which is a standard idea in many generalization papers. The novelty, in my opinion, is to use the MDD as an objective for determining discrepancy between domains, and the adversarial game. This can be clarified significantly to make sections 4, 5 more readable. There is too much of notation and terminology that obfuscates the reader from understanding the key contributions of the paper. I am taking the proofs and theorems at face value, and have not verified them.   \n* As far as i understand, MADG requires to train significantly more models (1 per domain, a feature extractor and the main classifier) in order to compute MDD and perform training. This is vastly more complex than any existing method. It may be that making progress in a hard problem like domain generalization requires this, but i think this trade-off needs to be made more explicit — compute vs generalization performance. Whereas a simple ERM or Mixup have little to no overhead and perform very closely on the metrics considered in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments and suggestions. We respond below to each of their concerns/suggestions.\n\n> I found the paper to be hard to read in general, at its core, the paper is proposing to model error on the unseen target using a convex hull of the source domains — which is a standard idea in many generalization papers. The novelty, in my opinion, is to use the MDD as an objective for determining discrepancy between domains, and the adversarial game. This can be clarified significantly to make sections 4, 5 more readable. There is too much of notation and terminology that obfuscates the reader from understanding the key contributions of the paper. I am taking the proofs and theorems at face value, and have not verified them\n> \n**Response:** Thank you for the suggestion, we will make necessary changes to make sections 4 and 5 more readable.\n\n> As far as i understand, MADG requires to train significantly more models (1 per domain, a feature extractor and the main classifier) in order to compute MDD and perform training. This is vastly more complex than any existing method. It may be that making progress in a hard problem like domain generalization requires this, but i think this trade-off needs to be made more explicit — compute vs generalization performance. Whereas a simple ERM or Mixup have little to no overhead and perform very closely on the metrics considered in the paper.\n> \n**Response:** We agree that the current dataset benchmark in domain generalization, DomainBed [32, ICLR 2021], is a challenging one where one model may not outperform all other models across all the constituent datasets (OfficeHome, PACS, VLCS, TerraIncognita, DomainNet) by a significant margin. Even recent papers, Fish [88, ICLR 2022] and Fishr [71, ICML 2022] showcase small improvements across these datasets. In terms of computational cost, following Eq.(17), while we iteratively compute MDD over all domains, our computational cost during training is similar to other benchmark methods as shown below in Table 1. (To add, even simple methods like ERM and Mixup have running times in similar ranges.)\n\nTable 1: Computation cost (GPU ram occupied [GB] and time per step [s]  during model training) and the average accuracy achieved across all the datasets.\n|Model| GPU RAM occupied (GB) | Time per step (s) | Avg. Accuracy (%)|\n|-------|-------|-------|-------|\n|MADG (ours) | ~10.5 |~1.3 |66.0|\n|Fishr [71] | ~15.7 |~0.6 | 65.7|\n|Fish[88]| ~3.4|~1.2 | 64.8|\n|Mixup [82]| ~8.2 | ~0.4 |65.4|\n|ERM [75] |~8.2 | ~0.4| 65.0|\n"}, {"review_id": "pvHOxhz3Cq", "reviewer": "Reviewer_zZgJ", "summary": "This paper aims to ease the distribution problem from a theoretical perspective, which uses margin loss and a scoring function to describe the relationship between domains, and the generalization bound in terms of functional class complexity is subsequently analyzed. Based on their theoretical analysis, a margin-based adversarial framework, which is developed upon the classical DANN method, is further proposed. Results conducted in the Domanbed benchmark show their results are competitive among existing arts. ", "questions": "See weakness.", "limitations": "It seems domain labels must be available during training, this should be listed as a limitation as domain labels are not always available.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. It is always good to see some theoretical analysis for DG.\n\n2. Experiments will the five datasets are appreciated.\n", "weaknesses": "1. The motivation in the introduction is problematic. The literature is happy to see inspiring theory, but the theory itself is not the purpose. Rather, the paper should emphasize what part of existing work requires a proper theoretical explanation. After all, how to solve domain generalization is the problem. The authors may consider revise the introduction part.\n\n2. The idea of adversarial training for DG (either DANN [77] or MMD [79]), is shown to be less effective than ERM according to different benchmarks [32, a]. However, this work shows significantly better results than ERM, what is the advantage of the proposed MADG compared with them?\n\n3. According to Line 282, and the implementation details in the supplementary material, I think the comparisons are unfair, as the Domainbed uses randomly selected hyperparameters (batch size, learning rate, etc.), which are fixed in their experiments.\n\n4. More effective approaches should be compared, such as SD [b], and Miro [c], and it is suggested to reevaluate their code (at least some of them) on the same device. \n\n[a] OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization, in CVPR'21.\n\n[b] Gradient Starvation: A Learning Proclivity in Neural Networks, in NeurIPS'21.\n\n[c] MIRO: Mutual Information Regularization with Oracle, in ECCV'22. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments and suggestions. We respond below to each of the concerns/suggestions.\n\n> The motivation in the introduction is problematic. The literature is happy to see inspiring theory, but the theory itself is not the purpose. Rather, the paper should emphasize what part of existing work requires a proper theoretical explanation. After all, how to solve domain generalization is the problem. The authors may consider revise the introduction part.\n> \n**Response:** We apologize for not being clearer on this. Our overall intention was to suggest that having strong theoretical underpinnings along with improved performance is an added advantage for a method. We will  revisit the phraseage and temper/substantiate our claim on motivating our method using theoretical framework as suggested. As discussed in Sec 1 (L39-42), our motivation in this work was to develop an adversarial DG algorithm which uses a disparaity discrepancy that is based on margin loss because of its advantages as discussed in Sec 4 (L129-144. We also develop a generalization bound for an unseen domain based on margin loss and scoring functions, which are more informative than 0-1 loss based bounds. \n\n>The idea of adversarial training for DG (either DANN [77] or MMD [79]), is shown to be less effective than ERM according to different benchmarks [32, a]. However, this work shows significantly better results than ERM, what is the advantage of the proposed MADG compared with them?\n>\n**Response:** As discussed in Sec 4 (L136-139), one of the important advantages of the margin-based discrepancy used in MADG (defined in Eq (5) and (6)) is that it considers the classifier f (that is used to perform the task) in its formulation. The adversarial methods in previous works have used discrepancy measures such as MMD [79] (moment matching metric) and DANN [77] (based on H$\\Delta$H discrepancy) that do not consider the task (the classifier) in their formulations. Also as discused in Sec 4 (L132-144), margin-based discrepancy is tighter and can be efficiently optimized as compared to $H\\Delta H$ discrepancy (ours computes the supremum over just one variable, whereas the latter computes over two variables). We hypothesize that these reasons make the margin-based disparity measure used in MADG perform better domain alignment and learn better domain-invariant features for the DG task.\n\n> According to Line 282, and the implementation details in the supplementary material, I think the comparisons are unfair, as Domainbed uses randomly selected hyperparameters (batch size, learning rate, etc.), which are fixed in their experiments.\n> \n**Response:** In DomainBed, hyperparameter tuning is done by selecting a hyperparameter value from a range of values randomly drawn from a distribution as shown in Table 6 in [32]. We follow the same approach. In Table A1 in our appendix, we report the hyperparameter values of the MADG model for each dataset. Simiarl to DomainBed, we employ hyperparameter tuning from among a range of values for each parameter, as detailed in the below Table 1. We will include these, and clarify this in the Appendix.\n\nTable 1: Ranges of values for hyperparameter tuning in our MADG model\n| Hyperparameter | Search Values |\n| -------- | -------- |\n| Margin     | [1, 1.5, 2, 3]     |\n| Learning rate | [0.01, 0.04, 0.001, 0.004, 0.0001] |\n| Momentum | [0.85, 0.9, 0.95] |\n| Weight Decay | [0.005, 0.0005] |\n| Dropout | [0.3, 0.4, 0.45, 0.5] |\n| Learning rate decay | [0.001, 0.0002] |\n\n>More effective approaches should be compared, such as SD [b], and Miro [c], and it is suggested to reevaluate their code (at least some of them) on the same device.\n>\n**Response:**  As discussed in Sec 6 (Lines 278-280) we evaluate our model on five different benchmark datasets (OfficeHome, PACS, VLCS, TerraIncognita and DomainNet) that is followed by recent papers [71,88,89] and DomainBed benchmark [32]. To the best of our knowledge, this is comprehensive for DG work. The paper [b] did not report results on these datasets, however, as suggested, we run the SD model [b] on OfficeHome dataset for three trials and report the average results below in Table 2. As seen from Table 2, MADG (our model) outperforms SD model on all domains.\n\nAlso, as stated in Appendix A2 (L85-86), we follow recent state-of-the-art work [71] (also supported by recent papers [A1-A3]) in their test-domain methodology for hyperparameter selection. The paper [c] proposes the MIRO model and reports results only for training-domain validation-based model selection setting. For fair comparision, based on your suggestion, we run the MIRO model for three trials on our setting. The results are reported below in Table 2 where we can see that the average accuracy of our model outperforms the MIRO model. The MADG model also outperforms MIRO on two domains and achieves the same accuracy on the \\`Real World\\`domain.\n\nTable 2: Accuracy (%) on OfficeHome dataset\n|Model|Art|Clipart|Product|Real World| Average|\n|-------|--------|-------|-------|------|------|\n|Miro[c]|67($\\pm0.7$)|56.5($\\pm0.9$)|79.4($\\pm1.4$)|81.5 ($\\pm0.5$)|71.1|\n|SD [b]|64.8($\\pm0.9$)|49.9($\\pm1.2$)|75.6($\\pm1.3$)|79.1($\\pm0.2$)|67.4\n|MADG (ours)| 68.6($\\pm0.5$)|55.5($\\pm 0.2$)|79.6($\\pm0.3$)|81.5($\\pm0.4$)|71.3|\n\n> It seems domain labels must be available during training, this should be listed as a limitation as domain labels are not always available.\n> \n**Response:** As with recent state-of-the-art DG methods [71],[88], our method requires the knowledge of domain assignment of each data point. We will be happy to state this assumption/limitation explicitly.\n\n[a] OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization, in CVPR'21.\n[b] Gradient Starvation: A Learning Proclivity in Neural Networks, in NeurIPS'21.\n[c] MIRO: Mutual Information Regularization with Oracle, in ECCV'22."}]

Author rebuttal:
We thank all reviewers for their positive feedback: proposed method is well designed [Ru5h1, RFjmm]; the method is well supported by theoretical analysis as well as extensive ablation studies [Rwf8A, RFjmm, RMRuq]; the development of a theoretical framework for DG problem not only enhaces the comprehensibility and reproducibility of the method, but also contributes to the broader understanding of the problem [Rwf8A, Ru5h1, RzZgJ] and the results of the proposed MADG model are good [Ru5h1, RMRuq]. We respond to each reviewer's comments below.

Meta-review: The theoretical results of the proposed method for DG are sound, while the experimental results only show minor improvement in performance. The paper may need a major revision in editing to make it easy and clear to follow.

Overall, this is a borderline paper. As most of the major concerns have been addressed after rebuttal, this paper is slightly above the borderline.
Decision: Accept (poster)