SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\epsilon = \mathrm {d} _ {f, \mathcal {F}} ^ {(\rho)} \left(\mathcal {D} _ {\mathcal {S} _ {i ^ {\prime}}}, \mathcal {D} _ {\mathcal {S} _ {k ^ {\prime}}}\right), \mathbf {s}. \mathbf {t}. \mathrm {d} _ {f, \mathcal {F}} ^ {(\rho)} \left(\mathcal {D} _ {\mathcal {S} _ {i ^ {\prime}}}, \mathcal {D} _ {\mathcal {S} _ {k ^ {\prime}}}\right) \geq \mathrm {d} _ {f, \mathcal {F}} ^ {(\rho)} \left(\mathcal {D} _ {\mathcal {S} _ {i}}, \mathcal {D} _ {\mathcal {S} _ {k}}\right) \forall i, k, i ^ {\prime}, k ^ {\prime} \in \{1, \dots , N _ {s} \} \tag {11}
$$
Equipped with this definition for  $\epsilon$ , we can re-state Theorem 2 as Corollary 1 below.
Corollary 1 Consider a mixture of  $N_{s}$  source distributions, scoring function  $f$ , unseen domain  $\mathcal{D}_{U_m}$ ,  $\gamma = d_{f,\mathcal{F}}^{(\rho)}(\mathcal{D}_{U_m},\bar{\mathcal{D}}_U)$ , and  $d_{f,\mathcal{F}}^{(\rho)}(\mathcal{D}_{S_{i'}},\mathcal{D}_{S_{k'}})$  where  $d_{f,\mathcal{F}}^{(\rho)}(\mathcal{D}_{S_{i'}},\mathcal{D}_{S_{k'}})\geq d_{f,\mathcal{F}}^{(\rho)}(\mathcal{D}_{S_i},\mathcal{D}_{S_k})$ $\forall i,k,i',k'\in \{1,2,\ldots ,N_s\}$ . Then the unseen target error is bounded as follows:
$$
\operatorname {e r r} _ {\mathcal {D} _ {U _ {m}}} \left(h _ {f}\right) \leq \sum_ {i = 1} ^ {N _ {s}} \pi_ {i} \left(\operatorname {e r r} _ {\mathcal {D} _ {S _ {i}}} ^ {(\rho)} (f)\right) + d _ {f, \mathcal {F}} ^ {(\rho)} \left(\mathcal {D} _ {S _ {i}}, \mathcal {D} _ {S _ {k ^ {\prime}}}\right) + \gamma + \bar {\lambda} \tag {12}
$$
As seen from Corollary 1, the unseen target error is bounded above by the source errors, maximum MDD between two source domains,  $\gamma$ , and the ideal loss given by  $\bar{\lambda} = \min_{f^{*}\in \mathcal{F}}\Bigg(\sum_{i = 1}^{N_{s}}\pi_{i}err_{\mathcal{D}_{S_{i}}}^{(\rho)}(f^{*}) + err_{\mathcal{D}_{U_{m}}}^{(\rho)}(f^{*})\Bigg)$ .
Before we derive the generalization bound with the Rademacher complexity framework, we define  $\Pi_{\mathcal{H}}\mathcal{F} = \left\{x\mapsto f\big(x,h(x)\big)\Bigg{|}h\in \mathcal{H},f\in \mathcal{F}\right\}$  [40] and Rademacher complexity  $\Re_{n,\mathcal{D}}$  in Definition 1.
Definition 1 Let  $\mathcal{F}$  be a class of functions such that  $f\in \mathcal{F}:\mathcal{X}\times \mathcal{Y}\to [a,b]$  and  $\hat{D} = \{(x_{1},y_{1}),\ldots ,(x_{n},y_{n})\}$  be a fixed sample of size  $n$  drawn from  $\mathcal{D}$  over  $\mathcal{X}\times \mathcal{Y}$ . Then the Rademacher complexity of  $\mathcal{F}$  is defined as:
$$
\mathfrak {R} _ {n, \mathcal {D}} (\mathcal {F}) \triangleq \mathbb {E} _ {\hat {D} \sim \mathcal {D} ^ {n}} \left[ \mathbb {E} _ {\sigma} \left[ \sup  _ {f \in \mathcal {F}} \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} f \left(x _ {i}, y _ {i}\right) \right] \right] \tag {13}
$$
where  $\sigma_{i}$ s are independent Rademacher variables that assume values in  $\{-1, + 1\}$ . We now leverage Lemma 3.6 in [40] to derive our Lemma 3.
Lemma 3 For any  $\delta >0$ , with probability  $1 - 2\delta$ , the following holds simultaneously for any scoring function  $f$ :
$$
\left| d _ {f, \mathcal {F}} ^ {(\rho)} \left(\hat {D} _ {\mathcal {S} _ {i}}, \hat {D} _ {\mathcal {S} _ {k}}\right) - d _ {f, \mathcal {F}} ^ {(\rho)} \left(\mathcal {D} _ {\mathcal {S} _ {i}}, \mathcal {D} _ {\mathcal {S} _ {k}}\right) \right| \leq \frac {k}{\rho} \Re_ {n _ {i}, \mathcal {D} _ {\mathcal {S} _ {i}}} \left(\Pi_ {\mathcal {H}} \mathcal {F}\right) + \frac {k}{\rho} \Re_ {n _ {k}, \mathcal {D} _ {\mathcal {S} _ {k}}} \left(\Pi_ {\mathcal {H}} \mathcal {F}\right) + \sqrt {\frac {\log \frac {2}{\delta}}{2 n _ {i}}} + \sqrt {\frac {\log \frac {2}{\delta}}{2 n _ {k}}} \tag {14}
$$
where  $n_i$  and  $n_k$  correspond to the sample size of  $\hat{D}_{S_i}$  and  $\hat{D}_{S_k}$ , respectively. The difference between MDD and its empirical version is bounded by the Rademacher complexity as seen in Eqn 14. With these definitions, we derive our generalization bound based on Rademacher complexity and empirical MDD as below.
Theorem 3 Given the same setting as Corollary 1 and Lemma 3, for any  $\delta >0$ , with probability  $1 - 3\delta$ , we obtain the following generalization bound for all  $f$ :
$$
\begin{array}{l} e r r _ {\mathcal {D} _ {U _ {m}}} (f) \leq \sum_ {i = 1} ^ {N _ {s}} \pi_ {i} \Big (e r r _ {\hat {D} _ {\mathcal {S} _ {i}}} ^ {\rho} (f) \Big) + d _ {f, \mathcal {F}} ^ {(\rho)} \big (\hat {D} _ {\mathcal {S} _ {i ^ {\prime}}}, \hat {D} _ {\mathcal {S} _ {k ^ {\prime}}} \big) + \gamma + \bar {\lambda} + \frac {k}{\rho} \mathfrak {R} _ {n _ {i}, \mathcal {D} _ {\mathcal {S} _ {i}}} \left(\Pi_ {\mathcal {H}} \mathcal {F}\right) \\ + \frac {k}{\rho} \Re_ {n _ {k}, \mathcal {D} _ {\mathcal {S} _ {k}}} (\Pi_ {\mathcal {H}} \mathcal {F}) + \sqrt {\frac {\log \frac {2}{\delta}}{2 n _ {i ^ {\prime}}}} + \sqrt {\frac {\log \frac {2}{\delta}}{2 n _ {k ^ {\prime}}}} + \sum_ {i = 1} ^ {N _ {s}} \pi_ {i} \left(\frac {2 k ^ {2}}{\rho} \Re_ {n _ {i}, \mathcal {D} _ {\mathbf {S} _ {i}}} (\Pi_ {1} \mathcal {F}) + \sqrt {\frac {\log \frac {2}{\delta}}{2 n _ {i}}}\right) \tag {15} \\ \end{array}
$$
where  $\Pi_1\mathcal{F} \triangleq \{x \mapsto f(x,y)|y \in \mathcal{V}, f \in \mathcal{F}\}$  as defined in [51] and 1 (in the subscript) represents constant functions mapping all points to the same class (see Appendix for more details). Theorem 3 thus establishes a relationship between the margin value  $\rho$  and the generalization error. The first two terms on the right-hand side exhibit minimal variation with an increase in the margin  $\rho$ , especially when  $\rho$  is small and the hypothesis space is rich, thus reducing the overall right-hand side. However, if  $\rho$  exceeds a certain threshold (resulting in a weak classifier), then the first two terms significantly increase, resulting in increase of the overall bound. Choosing a better  $\rho$  value thus allows one to obtain a desired error bound, making it more informative than a 0-1 loss-based bound. Inspired by this theoretical framework, we now propose our margin-based adversarial learning algorithm for DG in the next section.
5 MADG: Methodology
This section details the methodology of the proposed MADG algorithm, a novel adversarial DG algorithm motivated by the theory in Section 4. As seen in Theorem 3, the unseen target domain error is upper-bounded by the convex sum of the empirical source domain errors,  $\hat{\epsilon}$  term defined in Eqn. (16),  $\gamma$ ,  $\bar{\lambda}$  and complexity terms. We hence aim to minimize the two important terms, viz., the convex sum of the empirical source domain errors and the  $\hat{\epsilon}$  term (given below).
$$
\hat {\epsilon} = \mathrm {d} _ {f, \mathcal {F}} ^ {(\rho)} \left(\hat {D} _ {\mathcal {S} _ {i ^ {\prime}}}, \hat {D} _ {\mathcal {S} _ {k ^ {\prime}}}\right), \mathbf {s}. \mathbf {t}. \mathrm {d} _ {f, \mathcal {F}} ^ {(\rho)} \left(\hat {D} _ {\mathcal {S} _ {i ^ {\prime}}}, \hat {D} _ {\mathcal {S} _ {k ^ {\prime}}}\right) \geq \mathrm {d} _ {f, \mathcal {F}} ^ {(\rho)} \left(\hat {D} _ {\mathcal {S} _ {i}}, \hat {D} _ {\mathcal {S} _ {k}}\right) \forall i, k, i ^ {\prime}, k ^ {\prime} \in \{1, \dots , N _ {s} \} \tag {16}
$$
Figure 2: Architecture of the proposed MADG methodology
Efficient estimation of  $\hat{\epsilon}$ . As stated in Eqn. (16),  $\hat{\epsilon}$  is defined as the maximum pairwise empirical MDD among all source domains. Similar to Remark 4 and Lemma 2, we can also interpret  $\hat{\epsilon}$  as the upper bound of the empirical MDD between any two domains inside the convex hull formed by the source domains. If we minimize the sum of the empirical MDD between different pairs of the source domains, then the size of the convex hull reduces, which in turn minimizes  $\hat{\epsilon}$ . Thus, one effective way to estimate  $\hat{\epsilon}$  is to consider the term below in Eqn. (17).
$$
\sum_ {i = 1} ^ {N _ {s} - 1} \sum_ {k = i + 1} ^ {N _ {s}} \mathrm {d} _ {f, \mathcal {F}} ^ {(\rho)} \left(\hat {D} _ {S _ {i}}, \hat {D} _ {S _ {k}}\right) \tag {17}
$$
Minimax optimization. To find the optimal  $f$  in the hypothesis space  $\mathcal{F}$ , we formulate our objective function as a minimization problem. As mentioned earlier, we minimize the sum of the empirical source errors and the  $\hat{\epsilon}$  term estimated using Eqn (17). Thus, the minimization problem can be written as:
$$
\min  _ {f \in \mathcal {F}} \sum_ {i = 1} ^ {N _ {s}} \pi_ {i} \left(e r r _ {\hat {D} _ {S _ {i}}} ^ {(\rho)} (f)\right) + \sum_ {i = 1} ^ {N _ {s} - 1} \sum_ {k = i + 1} ^ {N _ {s}} \mathrm {d} _ {f, \mathcal {F}} ^ {(\rho)} \left(\hat {D} _ {S _ {i}}, \hat {D} _ {S _ {k}}\right) \tag {18}
$$
Since empirical MDD is defined as the supremum of the hypothesis space  $\mathcal{F}$ , minimizing it is in turn a minimax game with a strong max player and weaker min player. In order to strengthen the min player, we use a feature extractor,  $\mathcal{G}$ , which further modifies the optimization problem as follows:
$$
\min  _ {f, \mathcal {G}} \sum_ {i = 1} ^ {N _ {s}} \pi_ {i} \left(e r r _ {\mathcal {G} \left(\hat {D} _ {S _ {i}}\right)} ^ {(\rho)} (f)\right) + \sum_ {l = 1} ^ {j} \left(\operatorname {d i s p} _ {\mathcal {G} \left(\hat {D} _ {S _ {i}}\right)} ^ {(\rho)} \left(f _ {l} ^ {*}, f\right) - \operatorname {d i s p} _ {\mathcal {G} \left(\hat {D} _ {S _ {k}}\right)} ^ {(\rho)} \left(f _ {l} ^ {*}, f\right)\right), \tag {19}
$$
$$
f _ {l} ^ {*} = \max  _ {f _ {l} ^ {\prime}} \Big (\mathrm {d i s p} _ {\mathcal {G} (\hat {D} _ {\mathcal {S} _ {i}})} ^ {(\rho)} \big (f _ {l} ^ {\prime}, f \big) - \mathrm {d i s p} _ {\mathcal {G} (\hat {D} _ {\mathcal {S} _ {k}})} ^ {(\rho)} \big (f _ {l} ^ {\prime}, f \big) \Big), \text {w h e r e} f, f _ {l} ^ {\prime} \in \mathcal {F} \text {a n d} \forall l = \{1, \ldots , j \}.
$$
where  $j = \binom{N_s}{2}$ . The relationship between  $l, i$  and  $k$  is as follows: if  $l = 3$  then we pick the  $i$  and  $k$  value corresponding to the  $3^{\mathrm{rd}}$  element of the set  $\left\{(i,k): i = \{1,\dots,N_s - 1\}, k = \{i + 1,\dots,N_s\}\right\}$ . To solve the minimization problem in Eqn. (19), we design an adversarial learning algorithm whose model architecture is shown in Fig. 2. One classifier  $f$  performs the classification task on all source domains, while  $j$  other classifiers, denoted by  $f'$ , compute the empirical MDD between different pairs of source domains, which finally sum together as the transfer loss. To compute the source errors in Eqn. (19), we use the standard cross-entropy loss  $\mathcal{E}(\hat{D}_{S_i})$ . For convenience of optimization (MDD can be hard to optimize directly using stochastic gradient descent), in practice, following [40], we approximate the MDD loss as  $\left(\mathcal{D}_{(\hat{\rho},l)}(\hat{D}_{S_i},\hat{D}_{S_k})\right)$  in terms of two loss functions,  $L$  and  $L'$ , as shown below.
$$
\mathcal {E} \big (\hat {D} _ {\mathcal {S} _ {i}} \big) = \mathbb {E} _ {(x, y) \sim \hat {D} _ {\mathcal {S} _ {i}}} \Big [ L \Big (f \big (\mathcal {G} (x) \big), y \Big) \Big ]
$$
$$
\mathcal {D} _ {(\hat {\rho}, l)} \left(\hat {D} _ {\mathcal {S} _ {i}}, \hat {D} _ {\mathcal {S} _ {k}}\right) = \mathbb {E} _ {(x, y) \sim \hat {D} _ {\mathcal {S} _ {k}}} \left[ L ^ {\prime} \left(f _ {l} ^ {\prime} (\mathcal {G} (x)), f (\mathcal {G} (x))\right) \right] - \hat {\rho} \mathbb {E} _ {(x, y) \sim \hat {D} _ {\mathcal {S} _ {i}}} \left[ L \left(f _ {l} ^ {\prime} (\mathcal {G} (x)), f (\mathcal {G} (x))\right) \right] \tag {20}
$$
We train the feature extractor,  $\mathcal{G}$ , to minimize the above MDD loss term by using a Gradient Reversal Layer (GRL) proposed in [52], as  $\mathcal{D}_{(\hat{\rho},l)}(\hat{D}_{\mathcal{S}_i},\hat{D}_{\mathcal{S}_k})$  is not differentiable w.r.t the parameters of  $f$ .  $L$  and  $L^{\prime}$  are defined as:
$$
L \left(f (\mathcal {G} (x)), y\right) \triangleq - \log \left[ \sigma_ {y} \left(f (\mathcal {G} (x))\right) \right], L ^ {\prime} \left(f ^ {\prime} (\mathcal {G} (x)), f (\mathcal {G} (x))\right) \triangleq \log \left[ 1 - \sigma_ {h _ {f} (\mathcal {G} (x))} \left(f ^ {\prime} (\mathcal {G} (x))\right) \right] \tag {21}
$$
where  $\sigma_w(z) = \frac{e^{z_w}}{\sum_{i=1}^k e^{z_i}}$ ,  $z \in \mathbb{R}^k$ , for  $w = 1, \ldots, k$ , and  $\hat{\rho} = e^\rho$ . The final optimization problem that the MADG method solves is shown below.
$$
\min  _ {f, \mathcal {G}} \sum_ {i = 1} ^ {N _ {s}} \pi_ {i} \left(\mathcal {E} \left(\hat {D} _ {\mathcal {S} _ {i}}\right)\right) + \sum_ {l = 1} ^ {j} \left(\mathcal {D} _ {(\hat {\rho}, l)} \left(\hat {D} _ {\mathcal {S} _ {i}}, \hat {D} _ {\mathcal {S} _ {k}}\right)\right), \max  _ {f _ {1} ^ {\prime}, \dots , f _ {j} ^ {\prime}} \sum_ {l = 1} ^ {j} \left(\mathcal {D} _ {(\hat {\rho}, l)} \left(\hat {D} _ {\mathcal {S} _ {i}}, \hat {D} _ {\mathcal {S} _ {k}}\right)\right) \tag {22}
$$
We propose an adversarial DG algorithm to solve the minimization problem, as shown in Algorithm 1. We train the proposed adversarial model in two steps. In the first step, we update the parameters  $f$  and  $\mathcal{G}$ . In the next step, we first do a forward pass to compute the outputs from  $f$  and then update the parameters  $f'$  and  $\mathcal{G}$  as shown in the algorithm. We further show through our ablation studies that this way of updating parameters at different steps outperforms a joint update strategy.
Algorithm 1 Margin-based adversarial learning for Domain Generalization (MADG)
INPUT:  $N_{s}$  labeled source domains
for epoch  $\leftarrow 1$  to total_epochs do
for batch  $\leftarrow 1$  to total_batches do
Update parameters of  $f, \mathcal{G}$ , using Eqn. (20)
$$
\hat {y} _ {\hat {D} _ {S}}. \leftarrow f (\mathcal {G} (x)) \forall i \in \{1, 2, \dots , N _ {s} \}
$$
Compute  $MDD_{l}$  as in Eqn. (20) using  $\hat{y}_{\hat{D}S_{i}}$
Compute Transfer loss
$$
= \sum_ {l = 1} ^ {j} \left(\mathcal {D} _ {(\hat {\rho}, l)} \left(\hat {D} _ {\mathcal {S} _ {i}}, \hat {D} _ {\mathcal {S} _ {k}}\right)\right)
$$
Update parameters of  $f^{\prime}$  and  $\mathcal{G}$  as in Eqn. (22)
Datasets and Implementation Details. We per
form an extensive evaluation on five benchmark DG datasets for image classification: VLCS [30], PACS [31], OfficeHome (OH) [32], TerraIncognita (TI) [33] and DomainNet (DN) [34]. We follow [53] in using 'Test-domain validation' procedure for hyperparameter selection. We use the Resnet50 architecture [54] pre-trained on ImageNet dataset [55] as the feature extractor  $(\mathcal{G})$ . We train our model using stochastic gradient descent with momentum [56]. We use a mini-batch that contains 32 samples from all source domains. All other implementation details including hyperparameters such as learning rate, margin, and weight decay are provided in the Appendix.
Baselines and evaluation metrics. We report the results of all baseline models from [35], which are represented by  $\dagger$  in Table 1. We also present the results from [53], which are represented by  $\bot$ . We run our experiments for three trials and report the average accuracy (with standard deviation). Existing state-of-the-art methods (see [35]) focus on average accuracy across the benchmarks, with each method doing well on a subset of datasets. In order to reward consistent performance across datasets while improving average accuracy, we additionally include a ranking-based metric (from [53]) and go beyond ranking with two new metrics (AD and GD), which more precisely capture the consistent performance of a method across datasets: (i) Median rank (M) [53]: This measures the median rank of a model's performance across all datasets and is not skewed to best/worst ranks when compared to the mean rank. (ii) Arithmetic mean of differences (AD): This measures the difference of a model's performance w.r.t the highest-performing model's accuracy on a given dataset, and then taking its arithmetic mean across datasets. (iii) Geometric mean of differences (GD): This similarly measures the geometric mean of the difference between the highest accuracy and the achieved accuracy of a model across all datasets.
Results. We report the results on different DG benchmark datasets on the considered evaluation metrics in Table 1. It is observed that most previous methods are far from the best-performing model's accuracy on at least one of the datasets. This is captured well by the metrics AD and GD, which quantify the mean of the differences from the best accuracy across all the datasets. Thus, we observe higher values in the first two columns of Table 1, where our MADG model performs the best showing its consistency. The MADG algorithm outperforms all other models on average accuracy, median rank, AD, and GD, thus demonstrating consistent performance.
The results also show that MADG outperform DANN and CDANN models (which are based on the 0-1 loss discrepancy theory) across most datasets and improves the average accuracy by  $\approx 2\%$  with significantly better AD and GD values too - corroborating our margin-based approach to the DG problem. MADG is  $1\%$  better on the avg accuracy (and lower AD-GD values) when compared to ERM, which was found to be a strong baseline for DG in [35]. Our model also reports an accuracy improvement of  $\approx 3\%$  on the OfficeHome dataset compared to all other models.
Table 1: Accuracy(%) on benchmark DG datasets. Values in red and orange are best and second best-performing models in a column, respectively. We report deviation as  $\pm$ N/A for models that did not report them. OH = OfficeHome dataset, TI = TerraIncognita dataset, DN = DomainNet dataset, M = Median rank, AD = Arithmetic mean of differences, GD = Geometric mean of differences, Avg. = Average accuracy(%)
Algorithm	AD(↓)	GD(↓)	M(↓)	VLCS	OH	PACS	TI	DN	Avg.(↑)
ERM (1998)† [57]	2.2	1.8	8	77.6±0.3	66.4±0.5	86.7±0.3	53.0±0.3	41.3±0.1	65.0
CORAL (2016)† [58]	1.9	1.1	3	77.7±0.2	68.4±0.2	87.1±0.5	52.8±0.2	41.8±0.1	65.6
DANN (2016)† [59]	3.4	2.1	16	79.7±0.5	65.3±0.8	85.2±0.2	50.6±0.4	38.3±0.1	63.8
CDANN (2018)† [60]	3.2	0.7	14	79.9±0.2	65.3±0.5	85.8±0.8	50.8±0.6	38.5±0.2	64.1
MLDG (2018)† [61]	2.3	1.8	6	77.5±0.1	66.6±0.3	86.8±0.4	52.0±0.1	41.6±0.1	64.9
MMD (2018)† [62]	5.9	0.9	10	77.9±0.1	66.2±0.3	87.2±0.1	52.0±0.4	23.5±9.4	61.4
IRM (2019)† [63]	6.7	5.3	18	76.9±0.6	63.0±2.7	84.5±1.1	50.5±0.7	28.0±5.1	60.6
GroupDRO (2019)† [64]	3.9	1.9	10	77.4±0.5	66.2±0.6	87.1±0.1	52.4±0.1	33.4±0.3	63.3
Mixup (2020)† [65]	1.9	0.4	5	78.1±0.3	68.0±0.2	86.8±0.3	54.4±0.3	39.6±0.1	65.4
ARM (2020)† [66]	4.1	3.4	14	77.8±0.3	64.8±0.4	85.8±0.2	51.2±0.5	36.0±0.2	63.1
RSC (2020)† [67]	2.9	2.5	9	77.8±0.6	66.5±0.6	86.2±0.5	52.1±0.2	38.9±0.6	64.3
SagNet (2021)† [68]	2.3	2.0	6	77.6±0.1	67.5±0.2	86.4±0.4	52.5±0.4	40.8±0.2	65.0
V-REx (2021)† [69]	4.7	0.8	12	78.1±0.2	65.7±0.3	87.2±0.6	51.4±0.5	30.1±3.7	62.5
AND-mask (2021)‡ [70]	3.9	3.3	13	76.4±0.4	66.1±0.2	86.4±0.4	49.8±0.4	37.9±0.6	63.3
Fish (2021)‡ [71]	2.5	0.6	13	77.8±0.6	66.0±2.9	85.8±0.6	50.8±0.4	43.4±0.3	64.8
SAND-mask (2021)‡ [72]	5.2	4.2	16	76.2±0.5	65.9±0.5	85.9±0.4	50.2±0.1	32.2±0.6	62.1
Fisher (2022)‡ [53]	1.5	1.2	3	78.2±0.2	68.2±0.2	86.9±0.2	53.6±0.4	41.8±0.2	65.7
G2DM (2019)‡*[48]	Theory Based Methods	-	-	-	75.9±N/A	-	73.6±N/A	-	-	-
MTL(2021)‡*[47]	2.5	2.0	8	77.7±0.5	66.5±0.4	86.7±0.2	52.2±0.4	40.8±0.1	64.8
Transfer(2021)* [73]	-	-	-	-	64.3±N/A	85.3±N/A	-	-	-
Ood(2021)* [49]	-	-	-	76.3±N/A	68.1±N/A	86.6±N/A	-	-	-
MADG (ours)	Theory Based Methods	1.2	0.3	3	78.7±0.2	71.3±0.3	86.5±0.4	53.7±0.5	39.9±0.4	66.0
To highlight our contribution, we also provide a grouping of DG methods that are theory-based in Table 1 (also denoted with a *), some of which do not report results on all benchmark datasets. For the only such method that reports results on all datasets - MTL, MADG achieves  $1.2\%$  more average accuracy, 1.3 units less AD, 1.7 units less GD, and a lower median ranking as compared to MTL. More results, including 'Training-domain' model selection results, are provided in the Appendix.
7 More Empirical Analysis and Ablation Studies
Computational Cost. The computational cost of a model is usually measured in terms of GPU RAM occupied (GB) and time taken per step (s). In terms of computational cost, following Eqn. (17), while we iteratively compute MDD over all domains, our computational cost during training is similar to other benchmark methods as shown in Table 2. As seen in the table, even simple methods like ERM
and Mixup have running times in similar ranges.
Margin. As shown in Theorem 3, the margin value  $\rho$  plays an important role, especially with its dependency on the generalization error. In practice, to get the desired margin while training the MADG algorithm, we approximate it as  $\hat{\rho} \triangleq \exp \rho$ . We experiment with different margin values,  $\hat{\rho} = 1, 1.5, 2,$  and 3, and report the average accuracy for PACS and VLCS dataset in Table 3. Table 3 shows that  $\hat{\rho} = 1.5$  outperforms other values
Table 2: Computational cost and Average Accuracy  $(\%)$  (Avg.) for different baselines.
Model	GPU(GB)(↓)	Time(s)(↓)	Avg.(↑)
MADG	~10.5	~1.3	66.0
Fishr [53]	~15.7	~0.6	65.7
Fish [71]	~3.4	~1.2	64.8
Mixup [65]	~8.2	~0.4	65.4
ERM [57]	~8.2	~0.4	65.0
on both datasets. As seen from the definition of the margin loss function Eqn. (1) and MDD Eqn. (5), although having a large margin is better to achieve low loss and higher target accuracy, there is a tradeoff between optimal margin and loss value. We observe a similar trend in Table 3.
Experiments on Colored MNIST. As stated in Theorem 2,  $\gamma$  is defined as the projection of the unseen domain onto the convex hull. It is also defined in Theorem 2 as the MDD between the convex hull and the unseen domain, which can be further approximated with a combination of two different cross-entropy functions as in Eq (20). This equation is equivalent to a balanced Jensen-Shannon (JS) Divergence as shown in Proposition D.1. in [40]. Thus, we can approximate this projection by computing the JS divergence between domains. We experimented on the Colored MNIST dataset, where we noted the  $\gamma$  values to be small for all domains, as shown
Table 3: Accuracy(%) of MADG model for different  $\widehat{\rho }$  values.
ρ	PACS	VLCS
1	86.2	78.4
1.5	86.5	78.7
2	86.2	78.4
3	85.7	77.8
in Table 4. This is because the distributions of different domains in Colored MNIST are relatively close to each other when compared to real-world datasets.
As seen in Table 5, the proposed MADG algorithm achieves  $65.6\%$  average accuracy, which is significantly higher than the ERM model (57.8% average accuracy), thus showing that using a margin-based DG algorithm, MADG, learns better domain-invariant features on the Colored MNIST dataset. Besides, Table 4 shows that  $\gamma$  is small across domains in this dataset, showcasing the promise of the proposed method when the unseen domain is within the convex hull of the source domains.
Table 4: Pairwise JS divergence for the ColoredMNIST dataset and the approximated  $\gamma$  value.
+90	+80	-90	~γ
+90	0	0.019	0.024	0.043
+80	0.019	0	0.029	0.048
-90	0.024	0.029	0	0.054
MDD classifiers. The proposed MADG algorithm uses 'j'  $f^{\prime}$  classifiers, and each one calculates the MDD between a pair of source domains. In the algorithm, 'j' is defined as  $j = \binom{N_S}{2}$ . In this section, we analyze the algorithm with  $j = N_S - 1$  where  $l$  takes corresponding  $i$  and  $k$  values from  $\{(1,i):i = \{2,\dots ,N_s\} \}$ . As seen in Table 6, for  $N_{S} - 1$ , the model performs low across all the datasets, significantly ( $\sim 2.5\%$ ) for the TerraIncognita dataset because the model fails to capture all possible domain discrepancies, leading to poor performance by the feature extractor in learning invariant featu
Additional Results. The proposed MADG algorithm is further compared with recent methods, MIRO [29] and SD [74], which report results using different hyperparameter selection procedures. For a fair comparison, we run both these methods following our setting and report the results on OfficeHome data as shown in Table 7. As seen from Table 7, the proposed MADG outperforms the SD model on all domains. The MADG model also outperforms MIRO on two domains and achieves the same accuracy on the 'R' domain.