Title: MADG: Margin-based Adversarial Learning for Domain Generalization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Domain Generalization

Margin Loss

Adversarial Learning

Domain Adaptation
Abstract: Domain Generalization (DG) techniques have emerged as a popular approach to address the challenges of domain shift in Deep Learning (DL), with the goal of generalizing well to the target domain unseen during the training. In recent years, numerous methods have been proposed to address the DG setting, among which one popular approach is the adversarial learning-based methodology. The main idea behind adversarial DG methods is to learn domain-invariant features by minimizing a discrepancy metric. However, most adversarial DG methods use 0-1 loss based $\mathcal{H}\Delta\mathcal{H}$ divergence metric. In contrast, the margin loss-based discrepancy metric has the following advantages: more informative, tighter, practical, and efficiently optimizable. To mitigate this gap, this work proposes a novel adversarial learning DG algorithm, $\textbf{MADG}$, motivated by a margin loss-based discrepancy metric. The proposed $\textbf{MADG}$ model learns domain-invariant features across all source domains and uses adversarial training to generalize well to the unseen target domain. We also provide a theoretical analysis of the proposed $\textbf{MADG}$ model based on the unseen target error bound. Specifically, we construct the link between the source and unseen domains in the real-valued hypothesis space and derive the generalization bound using margin loss and Rademacher complexity. We extensively experiment with the $\textbf{MADG}$ model on popular real-world DG datasets, VLCS, PACS, OfficeHome, DomainNet, and TerraIncognita. We evaluate the proposed algorithm on DomainBed's benchmark and observe consistent performance across all the datasets.

MADG: Margin-based Adversarial Learning for Domain Generalization

Abstract

Domain Generalization (DG) techniques have emerged as a popular approach to address the challenges of domain shift in Deep Learning (DL), with the goal of generalizing well to the target domain unseen during the training. In recent years, numerous methods have been proposed to address the DG setting, among which one popular approach is the adversarial learning-based methodology. The main idea behind adversarial DG methods is to learn domain-invariant features by minimizing a discrepancy metric. However, most adversarial DG methods use 0-1 loss based H ∆ H divergence metric. In contrast, the margin loss-based discrepancy metric has the following advantages: more informative, tighter, practical, and efficiently optimizable. To mitigate this gap, this work proposes a novel adversarial learning DG algorithm, MADG , motivated by a margin loss-based discrepancy metric. The proposed MADG model learns domain-invariant features across all source domains and uses adversarial training to generalize well to the unseen target domain. We also provide a theoretical analysis of the proposed MADG model based on the unseen target error bound. Specifically, we construct the link between the source and unseen domains in the real-valued hypothesis space and derive the generalization bound using margin loss and Rademacher complexity. We extensively experiment with the MADG model on popular real-world DG datasets, VLCS, PACS, OfficeHome, DomainNet, and TerraIncognita. We evaluate the proposed algorithm on DomainBed's benchmark and observe consistent performance across all the datasets.

1 Introduction

Over the past decade, Deep Neural Networks (DNNs) have demonstrated exceptional performance across various fields, including robotics [1], medical imaging [2], agriculture [3], and more. However, the effectiveness of DNNs in supervised learning environments relies heavily on the independently and identically distributed ( i.i.d. ) assumption of the training and test (target) data. Unfortunately, in reality, this assumption can be compromised due to domain shifts in target data [4]. For example, a model trained on different domains of image data may perform poorly when presented with an image of a known label featuring an unseen background or viewpoint, as explained in [5]. To address this issue, researchers have developed techniques under the framework of domain adaptation (DA) [6]. The key idea behind DA is to adapt a model trained on a source dataset to minimize the generalization

error on the target dataset [7]. However, the major limitation of DA is that the target data, whether labeled or unlabeled, must be available during training. In contrast, the Domain Generalization (DG) setting aims to leverage knowledge from similar domains to classify previously unseen domains [8].

Recent years have seen the development of various algorithms addressing the DG setting, among which one popular approach is the adversarial learning-based methodology [9-14]. Other approaches can be broadly classified into meta-learning techniques [15-17], data augmentation methods [1820], self-supervised learning methods [21-23], regularization-based methods [24-29], and so on. Additionally, there have been concerted efforts towards the development of benchmark datasets [30-35] for the DG setting to study methods. The assumption of distribution shift, i.e., smooth variation between the conditional distribution P ( Y | X ) and the marginal distribution P ( X ) is also common in DG literature [36; 37].

The main objective of DG methods is to model the functional relationship between the input space X and the label space Y for all domains. To achieve this, adversarial DG methods learn a domaininvariant representation by minimizing a discrepancy metric among the source domains. However, despite the myriad efforts, most adversarial learning methods use the 0-1 loss based H ∆ H divergence metric [38] for domain alignment. In contrast, divergence metrics based on margin loss are more informative [39], practical, and efficiently optimizable [40]. This work addresses this gap by proposing a novel adversarial DG algorithm, MADG , motivated by a margin-based divergence metric. The proposed MADG leverages margin-based disparity discrepancy [40] to estimate source domain discrepancies in the DG setting and uses adversarial training to ensure that MADG generalizes well to unseen target domains. We also theoretically analyze the proposed MADG algorithm based on bounds for the unseen target error. The proposed generalization bound uses the Rademacher complexity framework [41], which provides data-dependent estimates of functional class complexities. The effectiveness of the proposed algorithm is demonstrated through extensive experiments on multiple benchmark DG datasets.

The key contributions of this work can be summarized as follows: (i) We introduce the use of margin loss and a corresponding scoring function to formulate the relationship between domains and develop upper bounds for the unseen target error (the first such margin-based effort in the DG setting, to the best of our knowledge); (ii) We subsequently analyze the generalization bound in terms of functional class complexity using the Rademacher complexity framework; (iii) We propose a novel margin-based adversarial DG training algorithm, MADG , motivated by our theoretical results; and (iv) We study the proposed method on five well-known benchmark datasets in the DomainBed setup, providing a higher average accuracy and consistency in model performance across these datasets.

2 Related Work

In this section, we discuss earlier literature proposed specifically for adversarial DG, as well as theoretical analysis for the DG problem in general. We discuss other DG literature across a broader set of categories in the Appendix. The main idea behind existing adversarial DG methods is to minimize the H ∆ H divergence by employing a minimax optimization between a domain discriminator and a classifier to learn domain-invariant features. One of the early works [9] proposed a method that iteratively divided samples into latent domains via clustering and trained the domain-invariant feature extractor via adversarial learning. Other efforts extended such an approach of adversarial learning with different divergence metrics and regularization techniques. Lin et al. [10] proposed a multidataset feature generalization network (MMFA-AAE) based on an adversarial auto-encoder to learn a generalized domain-invariant latent feature representation with the Maximum Mean Discrepancy (MMD) measure to align distributions across multiple domains. Deng et al. [11] examined adversarial censoring techniques to learn invariant representations from multiple domains. Zhao et al. [12] proposed an entropy regularization term along with adversarial loss to ensure conditional invariance of learned features. Akuzawa et al. [13] proposed the notion of accuracy-constrained domain invariance, and then developed an adversarial feature learning method with accuracy constraint (AFLAC), which explicitly provided invariance on adversarial training. Rahman et al. [42] proposed a correlationaware adversarial DG framework where the features of the source and target data are minimized using correlation alignment along with adversarial learning. All these prior works on adversarial DG methods use a 0-1 loss based H ∆ H discrepancy metric to align source domains. In this work, we instead leverage a margin-based disparity discrepancy and propose a new adversarial learning strategy founded on our theoretical analysis. The margin loss is advantageous compared to the 0-1 loss as it provides informative generalization bounds, tightness, classifier-aware alignment, and efficient optimization. We discuss each of these advantages in detail in Section 4.

Early efforts for theoretical analysis for the DG problem [43; 44] used kernel-based approaches to the problem setting with corresponding generalization error analysis, and showed empirical results using traditional machine learning methods. A few other efforts [45-47] also followed a similar path, and focused on kernel-based approaches in traditional methods. From a deep learning perspective, an early attempt at such a theoretically motivated method was proposed in [48] using a convex hull formulation and distribution matching to obtain generalization bounds, which also is an inspiration for parts of our work. In [49], the authors study theoretical bounds in the DG setting using three new concepts: feature invariance across domains, feature discriminability for the classification task, and a feature expansion function to generalize the first two concepts from the source to the target domain. [50] identified measures relating to the Fisher information, predictive entropy, and maximum mean discrepancy to be good predictors of out-of-distribution generalization of empirical risk minimization (ERM) models in the DG setting. [24] proposed generalization bounds in terms of the model's Rademacher complexity and suggested using regularized ERM models for the DG problem. Finally, [14] introduced an online game in which one model (player) reduces the error on test data provided by an adversarial player, but do not provide any empirical studies. In this work, we draw inspiration from [48] by considering a convex combination of source domains and leverage this to propose a new margin-based approach to the theoretical analysis of DG, which helps develop an informative generalization bound. Motivated by this analysis, we propose the MADG algorithm, a novel margin-based adversarial learning approach for DG, which shows consistent performance over other state-of-the-art methods across benchmarks.

3 Preliminaries

We consider a set of source domains D S i with i ∈ { 1 , 2 , . . . , N s } and a set of unseen domains D U m with m ∈ { 1 , 2 , . . . , N u } . We use D to refer to any of these domains when the index is not relevant. We use the terms 'domain' and 'distribution' interchangeably in this work. Each such domain D ⊆ X × Y , where X is an input space and Y is an output space, which is { 0 , 1 } in binary classification and { 1 , · · · , k } in multi-class classification. We use ˆ D to denote a set of samples drawn independently from D , i.e. ˆ D = { ( x i , y i ) } n i =1 where x i ∈ X and y i ∈ Y , ∀ i ∈ { 1 , 2 , . . . , n } . We use ( x, y ) to refer to a labeled sample ( x i , y i ) , when the index is not relevant.

As in [51], we consider the multi-class setting with a hypothesis space F of scoring functions f : X → R |Y| = R k , where the outputs report the confidence of the prediction on each dimension. Similar to [40], we use f ( x, y ) to denote the component of f ( x ) that corresponds to the label y . In order to obtain the final predicted label, we also consider a labeling function space H containing h f : X → Y such that h f ( x ) = argmax y ∈Y f ( x, y ) , i.e. the predicted label assigned to data sample x is the one resulting in the largest confidence score.

̸

The expected error rate of a classifier h ∈ H with respect to a distribution D is defined as err D ( h ) = ∆ E ( x,y ) ∼D ✶ [ h ( x ) = y ] where ✶ is the indicator function. We also define the margin ρ f ( · ) and the corresponding margin loss of hypothesis f for a labeled example ( x, y ) as follows:

̸

where ◦ denotes the composition function and Φ ρ is:

The margin disparity and its empirical version are then defined as:

where f and f ′ are different scoring functions. Margin disparity discrepancy (MDD) is used to quantify the degree of discrepancy/disagreement between decision boundaries of classifiers (using their margins) trained on different domains in our context. Such a measure can be used to evaluate the generalization performance of a model across multiple domains. The MDD and its empirical version are defined in Eqn. (5) and Eqn. (6) below, respectively.

We use the above terminologies from [40], which however focused on domain adaptation (with one source domain and one seen target domain). Handling multiple seen domains and an unseen target domain in the DG setting is non-trivial, which we focus on in this work.

4 Margin-based Approach to Domain Generalization: Theory

This section presents the theoretical motivation for our margin-based approach to the Domain Generalization (DG) problem. Our approach is based on considering the margin disparity discrepancy (MDD), defined above, between the source domains, thereby obtaining a generalization bound. We believe such a margin-based approach has a few advantages: (i) Informative: Generalization bounds based on margin loss for classification provide more information (than 0-1 loss-based bounds) by establishing a dependency between any margin function satisfying the Lipschitz condition and the upper bounds, as demonstrated in well-known earlier work [39]. (ii) Tightness: In contrast to using a 0-1 loss to identify samples causing disagreement between classifiers, margin loss computes disagreement between scoring functions using a smooth function parameterized by the threshold ( ρ ). Consequently, the number of samples causing agreement between classifiers is less, resulting in such an MDD-based bound being tighter, as shown in Fig. 1. (iii) Classifier-aware Discrepancy: MDD considers the classifier function while measuring discrepancy; as shown in Eqns 5 and 6, the supremum is computed over f ′ while holding f constant ( f learns the posterior distribution P ( y | x ) discriminatively for the classification task). This provides a classifier-aware approach to computing discrepancy.

(iv) Efficient Optimization and Practicality: The definition of MDD, as in Eqn. (5), only requires taking the supremum over one hypothesis. Therefore, compared to other divergence measures such as H ∆ H [38], MDD can be optimized with ease and is practically useful, as also stated in [40].

In this section, we derive a generalization bound for an unseen domain based on the margin-based MDD loss in the DG setting. To this end, we first show an upper bound on the unlabeled source domain error given other labeled source domains (Lemma 1 and Theorem 1). We then leverage this to develop the upper bound for the error on an unseen domain that is not necessarily a source domain (Lemma 2, Theorem 2, and Corollary 1). We subsequently analyze the upper bound

h f  = 0

h f'  = 0

h f'  = 1

h f  = 1

= Margin

from Corollary 1 using the Rademacher complexity framework and develop our final generalization bound for the unseen target domain in the DG setting (Lemma 3 and Theorem 3) using our marginbased loss. In Sec 5, we show the formulation of the proposed adversarial learning algorithm, MADG, motivated by the generalization bound in Sec 4, that employs MDD to address the DG problem.

We begin by considering a setting where training data consists of N s -1 labeled source domains D S i , i = { 1 , . . . , N s -1 } and a single unlabeled source domain D S T , and show an upper bound on error in this setting. We later use this to show an upper bound for the DG setting. We first establish an upper bound on the MDD between a weighted sum of the labeled source domains and the unlabeled one in Lemma 1 below.

Lemma 1 Consider a weighted sum of N s -1 labeled source distributions defined as D ¯ S := ∑ N s -1 i =1 α i D S i , where α i s are mixture co-efficients s.t. ∑ N s -1 i =1 α i = 1 , and f is a scoring function. Then

Detailed proofs for all our theoretical results are provided in the Appendix. It follows from Lemma 1 that an effective way to minimize the discrepancy between the unlabeled source domain and the mixture of labeled source domains in the hypothesis space is by minimizing the convex sum of the pairwise MDD between each labeled and unlabeled source domain. Building upon this insight, we provide bounds on the unlabeled source error below in Theorem 1.

Theorem 1 Consider a scoring function f , unlabeled source domain D S T and a mixture of N s -1 source distributions denoted as D ¯ S := ∑ N s -1 i =1 α i D S i , where α i s is mixture co-efficients s.t. ∑ N s -1 i =1 α i = 1 . Then the error on the unlabeled source D S T is bounded as:

Remark 1: From Theorem 1, we observe that the unlabeled source error is upper bounded by the labeled source errors, the pairwise discrepancy between each labeled and unlabeled source domain, and the ideal margin loss described below in Remark 2. We can also interpret Theorem 1 as an upper bound for the multi-source domain adaptation (DA) setting. While this is not our primary focus in this work, we report preliminary empirical results for a multi-source DA algorithm that reduces the first two terms in Theorem 1 in the Appendix.

Remark 2: The ideal loss ˆ λ is defined as ˆ λ = min f ∗ ∈F ( ∑ N s -1 i =1 α i err ( ρ ) D S i ( f ∗ ) + err ( ρ ) D S T ( f ∗ ) ) and is a constant that is independent of function f .

Building on Theorem 1, we now develop bounds for the DG setting, where all source domains ( N s ) are labeled, and the target domain is unseen during training. To this end, we consider the relationship between the unseen target and multiple source domains. In particular, we derive our error bound on the unseen target domain using the convex hull of the labeled source domains and MDD. We also provide a DG algorithm motivated by our theoretical analysis, which we describe later in Sec 5.

Before stating our main theorem, we present Lemma 2, which states that if the maximum MDD between any two sources is bounded above by ϵ , then the MDD between any two domains that belong to the convex hull of all the sources will also be bounded by ϵ . Let Λ s be the convex hull of source domains defined as Λ s = { ¯ D : ¯ D = ∑ N s i =1 π i D S i where ∑ N s i =1 π i = 1 } .

Lemma 2 Let d ( ρ ) f, F ( D S i , D S k ) ≤ ϵ ∀ i, k ∈ { 1 , 2 , . . . , N s } and f be a scoring function. Then the following inequality holds for the MDD between any pair of domains D ′ , D ′′ ∈ Λ s :

We observe from Lemma 2 that in the hypothesis space, the discrepancy among all the domains (seen or unseen) that belong to the convex hull Λ s can be reduced by minimizing the maximum MDD between two source domains. With the necessary tools at hand, we state the key theorem, i.e., the unseen target error bound for the DG problem below in Theorem 2.

Theorem 2 Consider a mixture of N s source distributions, scoring function f, unseen domain D U m , and γ = d ( ρ ) f, F ( D U m , ¯ D U ) where ¯ D U is the projection of D U m onto the convex hull of the sources i.e. ¯ ( ρ ) ( N ) N

D U = argmin π 1 ,π 2 ,...,π Ns d f, F D U m , ∑ s i =1 π i D S i , ∑ s i =1 π i = 1 . Then, the unseen target error is bounded as follows:

Remark 3:

As defined in Theorem 2,

γ

=

d

(

ρ

)

F

f,

D

U

m

,

¯

D

U

. Two scenarios therefore arise:

γ

= 0

when the unseen domain falls in the convex hull, i.e., D U m = ∑ N s i =1 π i D S i or γ > 0 when the unseen domain cannot be represented by available domains alone. Thus this parameter can be interpreted as the need for diverse source domains. The more diverse the source domains are, the smaller the value of γ .

Remark 4: As seen from Lemma 2, ϵ is defined as the upper bound for the MDD between any two domains that belong to the convex hull Λ s formed by the source domains. Thus, we can also interpret ϵ as the highest MDD value among the source domains as shown below in Eqn 11.

Equipped with this definition for ϵ , we can re-state Theorem 2 as Corollary 1 below.

Corollary 1 Consider a mixture of N s source distributions, scoring function f, unseen domain D U m , γ = d ( ρ ) f, F ( D U m , ¯ D U ) , and d ( ρ ) f, F ( D S i ′ , D S k ′ ) where d ( ρ ) f, F ( D S i ′ , D S k ′ ) ≥ d ( ρ ) f, F ( D S i , D S k ) ∀ i, k, i ′ , k ′ ∈ { 1 , 2 , . . . , N s } . Then the unseen target error is bounded as follows:

(

)

As seen from Corollary 1, the unseen target error is bounded above by the source errors, maximum MDD between two source domains, γ , and the ideal loss given by ¯ λ = min f ∗ ∈F ( ∑ N s i =1 π i err ( ρ ) D S i ( f ∗ ) + err ( ρ ) D Um ( f ∗ ) ) .

Before we derive the generalization bound with the Rademacher complexity framework, we define Π H F = { x ↦→ f ( x, h ( x ) ) ∣ ∣ ∣ ∣ h ∈ H , f ∈ F } [40] and Rademacher complexity R n, D in Definition 1.

Definition 1 Let F be a class of functions such that f ∈ F : X × Y → [ a, b ] and ˆ D = { ( x 1 , y 1 ) . . . , ( x n , y n ) } be a fixed sample of size n drawn from D over X ×Y . Then the Rademacher complexity of F is defined as:

where σ i s are independent Rademacher variables that assume values in {-1 , +1 } . We now leverage Lemma 3.6 in [40] to derive our Lemma 3.

Lemma 3 For any δ > 0 , with probability 1 -2 δ , the following holds simultaneously for any scoring function f :

where n i and n k correspond to the sample size of ˆ D S i and ˆ D S k , respectively. The difference between MDDand its empirical version is bounded by the Rademacher complexity as seen in Eqn 14. With these definitions, we derive our generalization bound based on Rademacher complexity and empirical MDDas below.

Theorem 3 Given the same setting as Corollary 1 and Lemma 3, for any δ > 0 , with probability 1 -3 δ , we obtain the following generalization bound for all f :

where Π 1 F = ∆ { x ↦→ f ( x, y ) ∣ ∣ y ∈ Y , f ∈ F } as defined in [51] and 1 (in the subscript) represents constant functions mapping all points to the same class (see Appendix for more details). Theorem 3 thus establishes a relationship between the margin value ρ and the generalization error. The first two terms on the right-hand side exhibit minimal variation with an increase in the margin ρ , especially when ρ is small and the hypothesis space is rich, thus reducing the overall right-hand side. However, if ρ exceeds a certain threshold (resulting in a weak classifier), then the first two terms significantly increase, resulting in increase of the overall bound. Choosing a better ρ value thus allows one to obtain a desired error bound, making it more informative than a 0-1 loss-based bound. Inspired by this theoretical framework, we now propose our margin-based adversarial learning algorithm for DG in the next section.

5 MADG: Methodology

This section details the methodology of the proposed MADG algorithm, a novel adversarial DG algorithm motivated by the theory in Section 4. As seen in Theorem 3, the unseen target domain error is upper-bounded by the convex sum of the empirical source domain errors, ˆ ϵ term defined in Eqn. (16), γ , ¯ λ and complexity terms. We hence aim to minimize the two important terms, viz., the convex sum of the empirical source domain errors and the ˆ ϵ term (given below).

Source 1

Source 2

Source N S

GRL

GRL

Concatenate

all sources

Feature Extractor

(Classification Loss)

(Transfer Loss)

Input

Efficient estimation of ˆ ϵ . As stated in Eqn. (16), ˆ ϵ is defined as the maximum pairwise empirical MDDamong all source domains. Similar to Remark 4 and Lemma 2, we can also interpret ˆ ϵ as the upper bound of the empirical MDD between any two domains inside the convex hull formed by the source domains. If we minimize the sum of the empirical MDD between different pairs of the source domains, then the size of the convex hull reduces, which in turn minimizes ˆ ϵ . Thus, one effective way to estimate ˆ ϵ is to consider the term below in Eqn. (17).

Minimax optimization. To find the optimal f in the hypothesis space F , we formulate our objective function as a minimization problem. As mentioned earlier, we minimize the sum of the empirical source errors and the ˆ ϵ term estimated using Eqn (17). Thus, the minimization problem can be written as:

Since empirical MDD is defined as the supremum of the hypothesis space F , minimizing it is in turn a minimax game with a strong max player and weaker min player. In order to strengthen the min player, we use a feature extractor, G , which further modifies the optimization problem as follows:

where j = ( N s 2 ) . The relationship between l, i and k is as follows: if l = 3 then we pick the i and k value corresponding to the 3 rd element of the set { ( i, k ) : i = { 1 , . . . , N s -1 } , k = { i +1 , . . . , N s } } . To solve the minimization problem in Eqn. (19), we design an adversarial learning algorithm whose model architecture is shown in Fig. 2. One classifier f performs the classification task on all source domains, while j other classifiers, denoted by f ′ , compute the empirical MDD between different pairs of source domains, which finally sum together as the transfer loss . To compute the source errors in Eqn. (19), we use the standard cross-entropy loss E ( ˆ D S i ) . For convenience of optimization (MDD can be hard to optimize directly using stochastic gradient descent), in practice, following [40], we approximate the MDD loss as ( D (ˆ ρ,l ) ( ˆ D S i , ˆ D S k ) ) in terms of two loss functions, L and L ′ , as shown below.

We train the feature extractor, G , to minimize the above MDD loss term by using a Gradient Reversal Layer (GRL) proposed in [52], as D (ˆ ρ,l ) ( ˆ D S i , ˆ D S k ) is not differentiable w.r.t the parameters of f . L and L ′ are defined as:

where σ w ( z ) = e zw ∑ k i =1 e z i , z ∈ R k , for w = 1 , . . . , k, and ˆ ρ = e ρ . The final optimization problem that the MADG method solves is shown below.

We propose an adversarial DG algorithm to solve the minimization problem, as shown in Algorithm 1. We train the proposed adversarial model in two steps. In the first step, we update the parameters f and G . In the next step, we first do a forward pass to compute the outputs from f and then update the parameters f ′ and G as shown in the algorithm. We further show through our ablation studies that this way of updating parameters at different steps outperforms a joint update strategy.

6 Experiments

Datasets and Implementation Details. We per-

Algorithm 1 Margin-based adversarial learning for Domain Generalization (MADG)

INPUT: N s labeled source domains for epoch ← 1 to total_epochs do

for batch

1

to total_batches

Update parameters of do

f

,

G

, using Eqn. (20)

ˆ

y

f

(

G

(

x

))

∀

i

i

ˆ

D

S

s

Compute

MDD

l

as in Eqn. (20) using

=

(

D

j

∑

(

l

=1

)

i

(ˆ

ρ,l

)

ˆ

D

S

,

ˆ

D

S

k

)

Update parameters of f ′ and G as in Eqn. (22)

end for end for

form an extensive evaluation on five benchmark DG datasets for image classification: VLCS [30], PACS [31], OfficeHome (OH) [32], TerraIncognita (TI) [33] and DomainNet (DN) [34].We follow [53] in using 'Test-domain validation' procedure for hyperparameter selection. We use the Resnet50 architecture [54] pre-trained on ImageNet dataset [55] as the feature extractor ( G ). We train our model using stochastic gradient descent with momentum [56]. We use a mini-batch that contains 32 samples from all source domains. All other implementation details including hyperparameters such as learning rate, margin, and weight decay are provided in the Appendix.

Baselines and evaluation metrics. We report the results of all baseline models from [35], which are represented by † in Table 1. We also present the results from [53], which are represented by ⊥ . We run our experiments for three trials and report the average accuracy (with standard deviation). Existing state-of-the-art methods (see [35]) focus on average accuracy across the benchmarks, with each method doing well on a subset of datasets. In order to reward consistent performance across datasets while improving average accuracy, we additionally include a ranking-based metric (from [53]) and go beyond ranking with two new metrics (AD and GD), which more precisely capture the consistent performance of a method across datasets: (i) Median rank (M) [53]: This measures the median rank of a model's performance across all datasets and is not skewed to best/worst ranks when compared to the mean rank. (ii) Arithmetic mean of differences (AD): This measures the difference of a model's performance w.r.t the highest-performing model's accuracy on a given dataset, and then taking its arithmetic mean across datasets. (iii) Geometric mean of differences (GD): This similarly measures the geometric mean of the difference between the highest accuracy and the achieved accuracy of a model across all datasets.

Results. We report the results on different DG benchmark datasets on the considered evaluation metrics in Table 1. It is observed that most previous methods are far from the best-performing model's accuracy on at least one of the datasets. This is captured well by the metrics AD and GD, which quantify the mean of the differences from the best accuracy across all the datasets. Thus, we observe higher values in the first two columns of Table 1, where our MADG model performs the best showing its consistency. The MADG algorithm outperforms all other models on average accuracy, median rank, AD, and GD, thus demonstrating consistent performance.

The results also show that MADG outperform DANN and CDANN models (which are based on the 0-1 loss discrepancy theory) across most datasets and improves the average accuracy by ≈ 2% with significantly better AD and GD values too - corroborating our margin-based approach to the DG problem. MADG is 1% better on the avg accuracy (and lower AD-GD values) when compared to ERM, which was found to be a strong baseline for DG in [35]. Our model also reports an accuracy improvement of ≈ 3% on the OfficeHome dataset compared to all other models.

,

2

, . . . , N

}

ˆ

y

ˆ

D

S

∈ {

1

i

To highlight our contribution, we also provide a grouping of DG methods that are theory-based in Table 1 (also denoted with a ∗ ), some of which do not report results on all benchmark datasets. For the only such method that reports results on all datasets - MTL, MADG achieves 1 . 2% more average accuracy, 1.3 units less AD, 1.7 units less GD, and a lower median ranking as compared to MTL. More results, including 'Training-domain' model selection results, are provided in the Appendix.

7 More Empirical Analysis and Ablation Studies

Computational Cost. The computational cost of a model is usually measured in terms of GPU RAM occupied (GB) and time taken per step (s). In terms of computational cost, following Eqn. (17), while we iteratively compute MDD over all domains, our computational cost during training is similar to other benchmark methods as shown in Table 2. As seen in the table, even simple methods like ERM and Mixup have running times in similar ranges.

Margin. As shown in Theorem 3, the margin value ρ plays an important role, especially with its dependency on the generalization error. In practice, to get the desired margin while training the MADG algorithm, we approximate it as ˆ ρ = ∆ exp ρ . We experiment with different margin values, ˆ ρ = 1 , 1 . 5 , 2 , and 3 , and report the average accuracy for PACS and VLCS dataset in Table 3. Table 3 shows that ˆ ρ = 1 . 5 outperforms other values

on both datasets. As seen from the definition of the margin loss function Eqn. (1) and MDD Eqn. (5), although having a large margin is better to achieve low loss and higher target accuracy, there is a tradeoff between optimal margin and loss value. We observe a similar trend in Table 3.

Experiments on Colored MNIST. As stated in Theorem 2, γ is defined as the projection of the unseen domain onto the convex hull. It is also defined in Theorem 2 as the MDD between the convex hull and the unseen domain, which can be further approximated with a combination of two different cross-entropy functions as in Eq (20). This equation is equivalent to a balanced Jensen-Shannon (JS) Divergence as shown in Proposition D.1. in [40]. Thus, we can approximate this projection by computing the JS divergence between domains. We experimented on the Colored MNIST dataset, where we noted the γ values to be small for all domains, as shown

in Table 4. This is because the distributions of different domains in Colored MNIST are relatively close to each other when compared to real-world datasets.

As seen in Table 5, the proposed MADG algorithm achieves 65.6% average accuracy, which is significantly higher than the ERM model (57.8% average accuracy), thus showing that using a margin-based DG algorithm, MADG, learns better domain-invariant features on the Colored MNIST dataset. Besides, Table 4 shows that γ is small across domains in this dataset, showcasing the promise of the proposed method when the unseen domain is within the convex hull of the source domains.

MDD classifiers. The proposed MADG algorithm uses 'j' f ′ classifiers, and each one calculates the MDD between a pair of source domains. In the algorithm, 'j' is defined as j = ( N S 2 ) . In this section, we analyze the algorithm with j = N S -1 where l takes corresponding i and k values from { (1 , i ) : i = { 2 , . . . , N s }} . As seen in Table 6, for ' N S -1 ', the model performs low across all the datasets, significantly ( ∼ 2 . 5% ) for the TerraIncognita dataset because the model fails to capture all possible domain discrepancies, leading to poor performance by the feature extractor in learning invariant features.

Additional Results. The proposed MADG algorithm is further compared with recent methods, MIRO [29] and SD [74], which report results using different hyperparameter selection procedures. For a fair comparison, we run both these methods following our setting and report the results on OfficeHome data as shown in Table 7. As seen from Table 7, the proposed MADG outperforms the SD model on all domains. The MADGmodel also outperforms MIRO on two domains and achieves the same accuracy on the 'R'domain.

Weighted MDD. The proposed MADG algorithm calculates the Transfer loss as the arithmetic sum of the 'j' MDD losses. This section analyzes the effect on the model's performance using a weighted ( w ) arithmetic sum. The different weights used in this section are w l = 1 N S , ∀ l = 1 . . . j (Average) and w l = MDD L ∑ j l =1 MDD l (Dynamic). The performance of the MADG algorithm using different weights is reported in Table 8. The model with w l = 1 , ∀ l =

1 . . . j performs better than other weight values as reported in Table 8.

8 Conclusion

This study presented a novel adversarial algorithm for DG called MADG . Our approach is inspired by a theoretical framework that formulates the relationship between the source and unseen domains in real-valued hypothesis space. We used the margin loss function and Rademacher complexity framework to develop a generalization bound, which is informative. The proposed bound also shows a dependency between the margin value and the generalization error, which can be easily translated into an algorithm. To this end, we developed the MADG algorithm that leverages the MDD metric with minimax optimization to learn domain-invariant features and generalize well to the unseen domain. The algorithm was studied on five well-known DG benchmark datasets and reported consistent performance across

all datasets as compared to other methods. We also report several empirical analyses and ablation studies.

Acknowledgments

The research of AD is supported in part by the Prime Minister Research Fellowship, Ministry of Education, Government of India. The research of LRC is supported by the Indo-Norwegian Collaboration in Autonomous Cyber-Physical Systems (INCAPS) project: 287918 of the International Partnerships for Excellent Education, Research and Innovation (INTPART) program and the LowAltitude UAV Communication and Tracking (LUCAT) project: 280835 of the IKTPLUSS program from the Research Council of Norway. The research of AK is supported by TiHAN Faculty Fellowship. VNB would like to acknowledge the support through the Govt of India SERB IMPRINT and DST ICPS funding programs for this work. We are grateful to the anonymous reviewers for the feedback that helped improved the presentation of this paper.

References

K. Aggarwal, S. K. Singh, M. Chopra, S. Kumar, and F. Colace, 'Deep learning in robotics for strengthening industry 4.0.: opportunities, challenges and future directions,' Robotics and AI for Cybersecurity and Critical Infrastructure in Smart Cities, pp. 1-19, 2022.

M. Tsuneki, 'Deep learning models in medical image analysis,' Journal of Oral Biosciences, vol. 64, no. 3, pp. 312-320, 2022.

T. Ayoub Shaikh, T. Rasool, and F. Rasheed Lone, 'Towards leveraging the role of machine learning and artificial intelligence in precision agriculture and smart farming,' Computers and Electronics in Agriculture, vol. 198, p. 107119, 2022.

J. Quionero-Candela, M. Sugiyama, A. Schwaighofer, and N. D. Lawrence, Dataset Shift in Machine Learning. The MIT Press, 2009.

J. Wang, C. Lan, C. Liu, Y. Ouyang, T. Qin, W. Lu, Y. Chen, W. Zeng, and P. Yu, 'Generalizing to unseen domains: A survey on domain generalization,' IEEE Transactions on Knowledge and Data Engineering, 2022.

Y. Zhang, 'A survey of unsupervised domain adaptation for visual recognition,' arXiv preprint arXiv:2112.06745, 2021.

X. Liu, C. Yoo, F. Xing, H. Oh, G. El Fakhri, J.-W. Kang, J. Woo et al., 'Deep unsupervised domain adaptation: A review of recent advances and perspectives,' APSIPA Transactions on Signal and Information Processing, vol. 11, no. 1, 2022.

G. Blanchard, G. Lee, and C. Scott, 'Generalizing from several related classification tasks to a new unlabeled sample,' in Advances in Neural Information Processing Systems, vol. 24, 2011.

T. Matsuura and T. Harada, 'Domain generalization using a mixture of multiple latent domains,' in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 07, 2020, pp. 11 749-11 756.

S. Lin, C.-T. Li, and A. C. Kot, 'Multi-domain adversarial feature generalization for person re-identification,' IEEE Transactions on Image Processing, vol. 30, pp. 1596-1607, 2020.

Z. Deng, F. Ding, C. Dwork, R. Hong, G. Parmigiani, P. Patil, and P. Sur, 'Representation via representations: Domain generalization via adversarially learned invariant representations,' arXiv preprint arXiv:2006.11478, 2020.

S. Zhao, M. Gong, T. Liu, H. Fu, and D. Tao, 'Domain generalization via entropy regularization,' Advances in Neural Information Processing Systems, vol. 33, pp. 16 096-16 107, 2020.

K. Akuzawa, Y. Iwasawa, and Y. Matsuo, 'Adversarial invariant feature learning with accuracy constraint for domain generalization,' in Machine Learning and Knowledge Discovery in Databases: European Conference. Springer, 2020, pp. 315-331.

E. Rosenfeld, P. Ravikumar, and A. Risteski, 'An online learning approach to interpolation and extrapolation in domain generalization,' in International Conference on Artificial Intelligence and Statistics. PMLR, 2022, pp. 2641-2657.

M.-H. Bui, T. Tran, A. Tran, and D. Phung, 'Exploiting domain-specific features to enhance domain generalization,' Advances in Neural Information Processing Systems, vol. 34, pp. 21 189-21 201, 2021.

V. Narayanan, A. A. Deshmukh, U. Dogan, and V. N. Balasubramanian, 'On challenges in unsupervised domain generalization,' in NeurIPS 2021 Workshop on Pre-registration in Machine Learning. PMLR, 2022, pp. 42-58.

Y. Shu, Z. Cao, C. Wang, J. Wang, and M. Long, 'Open domain generalization with domainaugmented meta-learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 9624-9633.

J. Yuan, X. Ma, D. Chen, K. Kuang, F. Wu, and L. Lin, 'Label-efficient domain generalization via collaborative exploration and generalization,' in Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 2361-2370.

S. Paul, T. Dutta, and S. Biswas, 'Universal cross-domain retrieval: Generalizing across classes and domains,' in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 12 056-12 064.

Q. Xu, R. Zhang, Y. Zhang, Y. Wang, and Q. Tian, 'A fourier-based framework for domain generalization,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 14 383-14 392.

Z. Li, Z. Cui, S. Wang, Y. Qi, X. Ouyang, Q. Chen, Y. Yang, Z. Xue, D. Shen, and J.-Z. Cheng, 'Domain generalization for mammography detection via multi-style and multi-view contrastive learning,' in Medical Image Computing and Computer Assisted Intervention-MICCAI Proceedings, 2021, pp. 98-108.

D. Kim, Y. Yoo, S. Park, J. Kim, and J. Lee, 'Selfreg: Self-supervised contrastive regularization for domain generalization,' in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9619-9628.

Y. Chen, Y. Wang, Y. Pan, T. Yao, X. Tian, and T. Mei, 'A style and semantic memory mechanism for domain generalization,' in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9164-9173.

D. Li, H. Gouk, and T. Hospedales, 'Finding lost dg: Explaining domain generalization via model complexity,' arXiv preprint arXiv:2202.00563, 2022.

M. Zhang, H. Marklund, N. Dhawan, A. Gupta, S. Levine, and C. Finn, 'Adaptive risk minimization: Learning to adapt to domain shift,' Advances in Neural Information Processing Systems, vol. 34, pp. 23 664-23 678, 2021.

A. Nasery, S. Thakur, V. Piratla, A. De, and S. Sarawagi, 'Training for the future: A simple gradient interpolation loss to generalize along time,' Advances in Neural Information Processing Systems, vol. 34, pp. 19 198-19 209, 2021.

J. Cha, S. Chun, K. Lee, H.-C. Cho, S. Park, Y. Lee, and S. Park, 'Swad: Domain generalization by seeking flat minima,' Advances in Neural Information Processing Systems, vol. 34, pp. 22 405-22 418, 2021.

A. Robey, G. J. Pappas, and H. Hassani, 'Model-based domain generalization,' Advances in Neural Information Processing Systems, vol. 34, pp. 20 210-20 229, 2021.

J. Cha, K. Lee, S. Park, and S. Chun, 'Domain generalization by mutual-information regularization with pre-trained models,' in Proceedings of the European Conference on Computer Vision (ECCV), 2022, pp. 440-457.

C. Fang, Y. Xu, and D. N. Rockmore, 'Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias,' in 2013 IEEE International Conference on Computer Vision, 2013, pp. 1657-1664.

D. Li, Y. Yang, Y.-Z. Song, and T. M. Hospedales, 'Deeper, broader and artier domain generalization,' in Proceedings of the IEEE international conference on computer vision, 2017, pp. 5542-5550.

H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan, 'Deep hashing network for unsupervised domain adaptation,' in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 5018-5027.

S. Beery, G. Van Horn, and P. Perona, 'Recognition in terra incognita,' in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 456-473.

X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang, 'Moment matching for multi-source domain adaptation,' in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1406-1415.

I. Gulrajani and D. Lopez-Paz, 'In search of lost domain generalization,' in 9th International Conference on Learning Representations, ICLR, 2021.

A. Gretton, A. Smola, J. Huang, M. Schmittfull, K. Borgwardt, and B. Schölkopf, 'Covariate Shift by Kernel Mean Matching,' in Dataset Shift in Machine Learning. The MIT Press, 2008.

J. Lin, Y. Tang, J. Wang, and W. Zhang, 'Mitigating both covariate and conditional shift for domain generalization,' in 2022 IEEE 8th International Conference on Cloud Computing and Intelligent Systems (CCIS). IEEE, 2022, pp. 437-443.

S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan, 'A theory of learning from different domains,' Machine learning, vol. 79, no. 1, pp. 151-175, 2010.

V. Koltchinskii and D. Panchenko, 'Empirical margin distributions and bounding the generalization error of combined classifiers,' The Annals of Statistics, vol. 30, no. 1, pp. 1-50, 2002.

Y. Zhang, T. Liu, M. Long, and M. Jordan, 'Bridging theory and algorithm for domain adaptation,' in Proceedings of the 36th International Conference on Machine Learning, vol. 97, 2019, pp. 7404-7413.

P. L. Bartlett and S. Mendelson, 'Rademacher and gaussian complexities: Risk bounds and structural results,' Journal of Machine Learning Research, vol. 3, no. Nov, pp. 463-482, 2002.

M. M. Rahman, C. Fookes, M. Baktashmotlagh, and S. Sridharan, 'Correlation-aware adversarial domain adaptation and generalization,' Pattern Recognition, vol. 100, p. 107124, 2020.

G. Blanchard, G. Lee, and C. Scott, 'Generalizing from several related classification tasks to a new unlabeled sample,' Advances in neural information processing systems, vol. 24, 2011.

K. Muandet, D. Balduzzi, and B. Schölkopf, 'Domain generalization via invariant feature representation,' in International conference on machine learning. PMLR, 2013, pp. 10-18.

A. A. Deshmukh, Y. Lei, S. Sharma, U. Dogan, J. W. Cutler, and C. Scott, 'A generalization error bound for multi-class domain generalization,' arXiv preprint arXiv:1905.10392, 2019.

S. Hu, K. Zhang, Z. Chen, and L. Chan, 'Domain generalization via multidomain discriminant analysis,' in Uncertainty in Artificial Intelligence. PMLR, 2020, pp. 292-302.

G. Blanchard, A. A. Deshmukh, Ü. Dogan, G. Lee, and C. Scott, 'Domain generalization by marginal transfer learning,' The Journal of Machine Learning Research, vol. 22, no. 1, pp. 46-100, 2021.

I. Albuquerque, J. Monteiro, M. Darvishi, T. H. Falk, and I. Mitliagkas, 'Generalizing to unseen domains via distribution matching,' arXiv preprint arXiv:1911.00804, 2019.

H. Ye, C. Xie, T. Cai, R. Li, Z. Li, and L. Wang, 'Towards a theoretical framework of out-ofdistribution generalization,' Advances in Neural Information Processing Systems, vol. 34, pp. 23 519-23 531, 2021.

R. Vedantam, D. Lopez-Paz, and D. J. Schwab, 'An empirical investigation of domain generalization with empirical risk minimizers,' Advances in Neural Information Processing Systems, vol. 34, pp. 28 131-28 143, 2021.

M. Mohri, A. Rostamizadeh, and A. Talwalkar, Foundations of machine learning. MIT press, 2018.

Y. Ganin and V. Lempitsky, 'Unsupervised domain adaptation by backpropagation,' in International conference on machine learning. PMLR, 2015, pp. 1180-1189.

A. Rame, C. Dancette, and M. Cord, 'Fishr: Invariant gradient variances for out-of-distribution generalization,' in International Conference on Machine Learning. PMLR, 2022, pp. 18 34718 377.

K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., 'Imagenet large scale visual recognition challenge,' International journal of computer vision, vol. 115, pp. 211-252, 2015.

N. Qian, 'On the momentum term in gradient descent learning algorithms,' Neural networks, vol. 12, no. 1, pp. 145-151, 1999.

V. N. Vapnik, 'An overview of statistical learning theory,' IEEE transactions on neural networks, vol. 10, no. 5, pp. 988-999, 1999.

B. Sun and K. Saenko, 'Deep coral: Correlation alignment for deep domain adaptation,' in Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14. Springer, 2016, pp. 443-450.

Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, 'Domain-adversarial training of neural networks,' The journal of machine learning research, vol. 17, no. 1, pp. 2096-2030, 2016.

Y. Li, X. Tian, M. Gong, Y. Liu, T. Liu, K. Zhang, and D. Tao, 'Deep domain generalization via conditional invariant adversarial networks,' in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 624-639.

D. Li, Y. Yang, Y.-Z. Song, and T. Hospedales, 'Learning to generalize: Meta-learning for domain generalization,' in Proceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1, 2018.

H. Li, S. J. Pan, S. Wang, and A. C. Kot, 'Domain generalization with adversarial feature learning,' in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 5400-5409.

M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz, 'Invariant risk minimization,' arXiv preprint arXiv:1907.02893, 2019.

S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang, 'Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization,' arXiv preprint arXiv:1911.08731, 2019.

S. Yan, H. Song, N. Li, L. Zou, and L. Ren, 'Improve unsupervised domain adaptation with mixup training,' arXiv preprint arXiv:2001.00677, 2020.

M. Zhang, H. Marklund, A. Gupta, S. Levine, and C. Finn, 'Adaptive risk minimization: A meta-learning approach for tackling group shift,' arXiv preprint arXiv:2007.02931, vol. 8, p. 9, 2020.

Z. Huang, H. Wang, E. P. Xing, and D. Huang, 'Self-challenging improves cross-domain generalization,' in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16. Springer, 2020, pp. 124-140.

H. Nam, H. Lee, J. Park, W. Yoon, and D. Yoo, 'Reducing domain gap by reducing style bias,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 8690-8699.

D. Krueger, E. Caballero, J.-H. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. Le Priol, and A. Courville, 'Out-of-distribution generalization via risk extrapolation (rex),' in International Conference on Machine Learning. PMLR, 2021, pp. 5815-5826.

G. Parascandolo, A. Neitz, A. Orvieto, L. Gresele, and B. Schölkopf, 'Learning explanations that are hard to vary,' in 9th International Conference on Learning Representations, ICLR, 2021.

Y. Shi, J. Seely, P. H. Torr, N. Siddharth, A. Hannun, N. Usunier, and G. Synnaeve, 'Gradient matching for domain generalization,' arXiv preprint arXiv:2104.09937, 2021.

S. Shahtalebi, J.-C. Gagnon-Audet, T. Laleh, M. Faramarzi, K. Ahuja, and I. Rish, 'Sandmask: An enhanced gradient masking strategy for the discovery of invariances in domain generalization,' ArXiv, vol. abs/2106.02266, 2021.

G. Zhang, H. Zhao, Y. Yu, and P. Poupart, 'Quantifying and improving transferability in domain generalization,' Advances in Neural Information Processing Systems, vol. 34, pp. 10 957-10 970, 2021.

M. Pezeshki, O. Kaba, Y. Bengio, A. C. Courville, D. Precup, and G. Lajoie, 'Gradient starvation: A learning proclivity in neural networks,' Advances in Neural Information Processing Systems, vol. 34, pp. 1256-1272, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to ease the distribution problem from a theoretical perspective, which uses margin loss and a scoring function to describe the relationship between domains, and the generalization bound in terms of functional class complexity is subsequently analyzed. Based on their theoretical analysis, a margin-based adversarial framework, which is developed upon the classical DANN method, is further proposed. Results conducted in the Domanbed benchmark show their results are competitive among existing arts.\n\nSTRENGTHS:\n1. It is always good to see some theoretical analysis for DG.\n\n2. Experiments will the five datasets are appreciated.\n\nWEAKNESSES:\n1. The motivation in the introduction is problematic. The literature is happy to see inspiring theory, but the theory itself is not the purpose. Rather, the paper should emphasize what part of existing work requires a proper theoretical explanation. After all, how to solve domain generalization is the problem. The authors may consider revise the introduction part.\n\n2. The idea of adversarial training for DG (either DANN [77] or MMD [79]), is shown to be less effective than ERM according to different benchmarks [32, a]. However, this work shows significantly better results than ERM, what is the advantage of the proposed MADG compared with them?\n\n3. According to Line 282, and the implementation details in the supplementary material, I think the comparisons are unfair, as the Domainbed uses randomly selected hyperparameters (batch size, learning rate, etc.), which are fixed in their experiments.\n\n4. More effective approaches should be compared, such as SD [b], and Miro [c], and it is suggested to reevaluate their code (at least some of them) on the same device. \n\n[a] OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization, in CVPR'21.\n\n[b] Gradient Starvation: A Learning Proclivity in Neural Networks, in NeurIPS'21.\n\n[c] MIRO: Mutual Information Regularization with Oracle, in ECCV'22.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nIt seems domain labels must be available during training, this should be listed as a limitation as domain labels are not always available.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new adversarial learning objective using a margin based approach for domain generalization. The goal of domain generalization broadly is to build classifiers that are trained on one or more source domains, and are expected to generalize to an unseen target domain. The key idea is to leverage Margin disparity discrepancy (MDD) which quantifies the level of disagreement between decision boundaries of classifiers using their margins. MDD is used as a proxy to understand generalizability of classifiers in this context across multiple domains. Next, the paper establishes an upper bound on the generalization error on any unseen target domain that is within the convex hull of the source domains and MDD. In other words, the objective is to get the decision boundaries across different source domains to agree as much as possible, while also minimizing the empirical errors on them simultaneously.\n\nSTRENGTHS:\n* The margin perspective for domain generalization is a fresh perspective to this, and the paper takes an interesting approach at using an adversarial learning strategy to optimizing this problem. \n* Detailed theoretical setup and formulation, which builds on existing work and setup the formulation for MADG\n* Results are impressive — on several benchmarks, the proposed method appears to perform competitively. \n* good ablations and analysis of the proposed method.\n\nWEAKNESSES:\n* I found the paper to be hard to read in general, at its core, the paper is proposing to model error on the unseen target using a convex hull of the source domains — which is a standard idea in many generalization papers. The novelty, in my opinion, is to use the MDD as an objective for determining discrepancy between domains, and the adversarial game. This can be clarified significantly to make sections 4, 5 more readable. There is too much of notation and terminology that obfuscates the reader from understanding the key contributions of the paper. I am taking the proofs and theorems at face value, and have not verified them.   \n* As far as i understand, MADG requires to train significantly more models (1 per domain, a feature extractor and the main classifier) in order to compute MDD and perform training. This is vastly more complex than any existing method. It may be that making progress in a hard problem like domain generalization requires this, but i think this trade-off needs to be made more explicit — compute vs generalization performance. Whereas a simple ERM or Mixup have little to no overhead and perform very closely on the metrics considered in the paper.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nsome what, yes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The paper presents a margin-loss based analysis of domain generalization.\n- First, the paper derives a bound on margin disparity discrepancy (MDD) of any unseen domain within the convex hull of source domains in terms of the margin loss on source domains, ideal margin loss, and max MDD between any two source domains.\n- The paper relates this to any unseen domain via projection onto the convex hull of source domains, and an additional factor $\\gamma$\n- The paper then realizes the bound for the empirical setting using a bound based on the Rademacher complexity of the function class\n- Based on this bound, the paper proposes an adversarial learning method to regularize training by minimizing MDD in addition to classification loss.\n- The paper demonstrates the efficacy of the method on a variety of real world datasets in the DomainBed benchmark.\n\nSTRENGTHS:\n1. The paper proposes a principled method for domain generalization, and the empirical method follows neatly from the derived generalization bound.\n2. The proposed bound for error on an unseen domain based on margin disparities between source domains is novel to my knowledge. \n3. Empirical results show modest improvements over ERM.\n4. The paper presents empirical ablations for some design choices used in the algorithm.\n\nWEAKNESSES:\nIt is not clear if the single optimization step for the adversarial models f’ is sufficient for tightly approximating MDD. The method effectively regularizes a lower bound on the MDD term, which requires the adversarial models to be effective maximizers — this may require many steps for the adversarial models per main-model step.\n\nTable 1 presents results using the model selection strategy of [71], which uses out-of-distribution data as the validation set for picking hyperparameters. Using labelled OOD data for hyperparameter selection limits the conclusions we can draw about if the proposed method works in practice, where we do not usually have access to labelled test data. While the AD, GD, and M metrics allay some of this concern, presenting results with some methodology that could be used for model-selection in practice, such as leave-one-out domain validation would make the empirical results stronger.\n\nQUESTIONS:\nWhy is only the feature extractor trained to minimize MDD loss and not the classifier (line 261)?\n\nTheorem 2 shows that the error of any unseen domain can be bounded via a projection onto the convex hull of the source domain, by adding in an additional factor $\\gamma$. It is not clear if this factor is small or very large for domain generalization benchmarks. Is there intuition for “how far” an unseen domain in a benchmark is to the source domains?\n\nLIMITATIONS:\nIt is not completely clear how significant the empirical results are. The method slightly outperforms ERM on average, but ERM outperforms the method on 2 out of 5 considered datasets, and is competitive on 2 others. Additionally, the reliance on labelled OOD data for model selection limits the conclusions one can draw about the efficacy of the method on the DomainBed benchmark.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is commendable for its innovative use of a margin-based theoretical framework to solve domain generalization problems, which contrasts with the largely heuristic and empirical approaches adopted by existing methods. By grounding their approach in a theoretical foundation, the authors provide more interpretable solutions that can contribute to the field of domain generalization.\n\nSTRENGTHS:\n- The claim presented in the paper is strongly supported by theoretical analysis, as well as the extensive ablation studies presented in section 7. This speaks to the methodological rigour of the study and strengthens its credibility.\n- The development of a theoretical framework for solving domain generalization problems is indeed a significant strength of this paper. This approach not only enhances the comprehensibility and reproducibility of the method, but also contributes to the broader understanding of the problem.\n\nWEAKNESSES:\n- While the average performance of the proposed method is the highest, the performance gap across different tasks is quite significant. The method shows the best performance on the Office Home dataset, but this alone is not enough to convincingly demonstrate the overall superiority of the method. Therefore, the claimed significance of the proposed method appears to be overstated.\n- The authors' claim that a theoretical approach is necessary is quite a strong statement that seems to undermine the importance of empirically validated methods in the field. This makes it hard to fully agree with their motivation.\n- While the authors make a reasonable argument for the use of margin as a metric in generalization, they do not provide a sufficient justification for its relevance in domain generalization problems, particularly where style shifts are involved. This makes it difficult to understand the authors' motivation for their margin-based approach.\n- Similarly, the reasoning behind the use of adversarial learning (min-max framework) is not clearly explained. The proposed method appears to be similar to the one used in [1], and a clear explanation of the differences between these two methods, apart from the task, would be beneficial.\n- The paper lacks a comparison with recent Domain Generalization (DG) works [2-4]. This makes it difficult to assess how the proposed method stacks up against the state of the art.\n\n[1] Maximum Mean Discrepancy Test is Aware of Adversarial Attacks\n\n[2] Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution\n\n[3] Domain Generalization by Mutual-Information Regularization with Pre-trained Models\n\n[4] SIMPLE: Specialized Model-Sample Matching for Domain Generalization\n\nQUESTIONS:\nPlease see weakness\n\nLIMITATIONS:\nThe authors have well described their limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates domain generalization (DG) using a margin-based theoretical framework. The authors first formulate the generalization upper bound by leveraging the margin disparity discrepancy (MDD). Then, an adversarial learning strategy (MADG) is devised to minimize the empirical MMD between source domains. The experiments on the DomainBed benchmark further demonstrate that the proposed MADG could outperform the current state-of-the-art methods.\n\nSTRENGTHS:\n1. The paper is clearly written, and the method is well-designed. \n2. The theory of margin-based generalization bound is interesting and could benefit the further designs of DG methods. \n3. The proposed MADG could outperform recent state-of-the-art methods.\n\nWEAKNESSES:\n1. The primary concern of the reviewer is in its effectiveness. While MADG outperforms most baselines in the DomainBed benchmark, the improvement is marginal/minor. This made it unclear to me if the method actually works better, or it is just a product of optimizing some hyperparameters. \n2. The motivation, “very little work has been done in developing DG algorithms that are well-motivated by theoretical”, lacks soundness. In fact, there have been notable works in the literature that utilize theoretical frameworks to analyze the generalization problem, such as gradient matching [71, 88] and invariant risk minimization [80]. It could be beneficial if the authors could provide a detailed comparison with such theoretical methods.\n3. In addition to the accuracy comparison, could the authors provide a more in-depth analysis regarding the training dynamics? This would help shed light on the inner workings of the MADG approach.\n4. Many advantages, such as efficient optimization, stated in the abstract are not well verified. Specifically, following Eq. (17), it seems that the estimation process could introduce a significant computational cost, as it iteratively computes for all domains.\n\nQUESTIONS:\n1. Table 1 could be simplified, as many baselines are unnecessary.\n2. The common approach to compare DG methods is the training-domain validation. Why do the authors mainly utilize the test-domain selection?\n\nLIMITATIONS:\nSee above comments.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}