SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Aveen Dayal
Indian Institute of Technology Hyderabad ai21resch11003@iith.ac.in
Vimal K B
Indian Institute of Technology Hyderabad vimalkb96@gmail.com
Linga Reddy Cenkeramaddi
University of Agder linga.cenkeramaddi@uia.no
C Krishna Mohan
Indian Institute of Technology Hyderabad ckm@cse.iith.ac.in
Indian Institute of Technology Hyderabad abhinavkumar@ee.iith.ac.in
Vineeth N Balasubramanian
Indian Institute of Technology Hyderabad vineethnb@cse.iith.ac.in
Domain Generalization (DG) techniques have emerged as a popular approach to address the challenges of domain shift in Deep Learning (DL), with the goal of generalizing well to the target domain unseen during the training. In recent years, numerous methods have been proposed to address the DG setting, among which one popular approach is the adversarial learning-based methodology. The main idea behind adversarial DG methods is to learn domain-invariant features by minimizing a discrepancy metric. However, most adversarial DG methods use 0-1 loss based  $\mathcal{H}\Delta \mathcal{H}$  divergence metric. In contrast, the margin loss-based discrepancy metric has the following advantages: more informative, tighter, practical, and efficiently estimizable. To mitigate this gap, this work proposes a novel adversarial learning DG algorithm, MADG, motivated by a margin loss-based discrepancy metric. The proposed MADG model learns domain-invariant features across all source domains and uses adversarial training to generalize well to the unseen target domain. We also provide a theoretical analysis of the proposed MADG model based on the unseen target error bound. Specifically, we construct the link between the source and unseen domains in the real-valued hypothesis space and derive the generalization bound using margin loss and Rademacher complexity. We extensively experiment with the MADG model on popular real-world DG datasets, VLCS, PACS, OfficeHome, DomainNet, and TerraIncognita. We evaluate the proposed algorithm on DomainBed's benchmark and observe consistent performance across all the datasets.
Over the past decade, Deep Neural Networks (DNNs) have demonstrated exceptional performance across various fields, including robotics [1], medical imaging [2], agriculture [3], and more. However, the effectiveness of DNNs in supervised learning environments relies heavily on the independently and identically distributed (i.i.d.) assumption of the training and test (target) data. Unfortunately, in reality, this assumption can be compromised due to domain shifts in target data [4]. For example, a model trained on different domains of image data may perform poorly when presented with an image of a known label featuring an unseen background or viewpoint, as explained in [5]. To address this issue, researchers have developed techniques under the framework of domain adaptation (DA) [6]. The key idea behind DA is to adapt a model trained on a source dataset to minimize the generalization
error on the target dataset [7]. However, the major limitation of DA is that the target data, whether labeled or unlabeled, must be available during training. In contrast, the Domain Generalization (DG) setting aims to leverage knowledge from similar domains to classify previously unseen domains [8].
Recent years have seen the development of various algorithms addressing the DG setting, among which one popular approach is the adversarial learning-based methodology [9-14]. Other approaches can be broadly classified into meta-learning techniques [15-17], data augmentation methods [18-20], self-supervised learning methods [21-23], regularization-based methods [24-29], and so on. Additionally, there have been concerted efforts towards the development of benchmark datasets [30-35] for the DG setting to study methods. The assumption of distribution shift, i.e., smooth variation between the conditional distribution  $\mathbb{P}(Y|X)$  and the marginal distribution  $\mathbb{P}(X)$  is also common in DG literature [36; 37].
The main objective of DG methods is to model the functional relationship between the input space  $\mathcal{X}$  and the label space  $\mathcal{Y}$  for all domains. To achieve this, adversarial DG methods learn a domain-invariant representation by minimizing a discrepancy metric among the source domains. However, despite the myriad efforts, most adversarial learning methods use the 0-1 loss based  $\mathcal{H}\Delta\mathcal{H}$  divergence metric [38] for domain alignment. In contrast, divergence metrics based on margin loss are more informative [39], practical, and efficiently estimable [40]. This work addresses this gap by proposing a novel adversarial DG algorithm, MADG, motivated by a margin-based divergence metric. The proposed MADG leverages margin-based disparity discrepancy [40] to estimate source domain discrepancies in the DG setting and uses adversarial training to ensure that MADG generalizes well to unseen target domains. We also theoretically analyze the proposed MADG algorithm based on bounds for the unseen target error. The proposed generalization bound uses the Rademacher complexity framework [41], which provides data-dependent estimates of functional class complexities. The effectiveness of the proposed algorithm is demonstrated through extensive experiments on multiple benchmark DG datasets.
The key contributions of this work can be summarized as follows: (i) We introduce the use of margin loss and a corresponding scoring function to formulate the relationship between domains and develop upper bounds for the unseen target error (the first such margin-based effort in the DG setting, to the best of our knowledge); (ii) We subsequently analyze the generalization bound in terms of functional class complexity using the Rademacher complexity framework; (iii) We propose a novel margin-based adversarial DG training algorithm, MADG, motivated by our theoretical results; and (iv) We study the proposed method on five well-known benchmark datasets in the DomainBed setup, providing a higher average accuracy and consistency in model performance across these datasets.
In this section, we discuss earlier literature proposed specifically for adversarial DG, as well as theoretical analysis for the DG problem in general. We discuss other DG literature across a broader set of categories in the Appendix. The main idea behind existing adversarial DG methods is to minimize the  $\mathcal{H}\Delta \mathcal{H}$  divergence by employing a minimax optimization between a domain discriminator and a classifier to learn domain-invariant features. One of the early works [9] proposed a method that iteratively divided samples into latent domains via clustering and trained the domain-invariant feature extractor via adversarial learning. Other efforts extended such an approach of adversarial learning with different divergence metrics and regularization techniques. Lin et al. [10] proposed a multi-dataset feature generalization network (MMFA-AAE) based on an adversarial auto-encoder to learn a generalized domain-invariant latent feature representation with the Maximum Mean Discrepancy (MMD) measure to align distributions across multiple domains. Deng et al. [11] examined adversarial censoring techniques to learn invariant representations from multiple domains. Zhao et al. [12] proposed an entropy regularization term along with adversarial loss to ensure conditional invariance of learned features. Akuzawa et al. [13] proposed the notion of accuracy-constrained domain invariance, and then developed an adversarial feature learning method with accuracy constraint (AFLAC), which explicitly provided invariance on adversarial training. Rahman et al. [42] proposed a correlation-aware adversarial DG framework where the features of the source and target data are minimized using correlation alignment along with adversarial learning. All these prior works on adversarial DG methods use a 0-1 loss based  $\mathcal{H}\Delta \mathcal{H}$  discrepancy metric to align source domains. In this work, we instead leverage a margin-based disparity discrepancy and propose a new adversarial learning strategy founded on our theoretical analysis. The margin loss is advantageous compared to the 0-1 loss as it provides informative generalization bounds, tightness, classifier-aware alignment, and efficient optimization. We discuss each of these advantages in detail in Section 4.
Early efforts for theoretical analysis for the DG problem [43; 44] used kernel-based approaches to the problem setting with corresponding generalization error analysis, and showed empirical results using traditional machine learning methods. A few other efforts [45-47] also followed a similar path, and focused on kernel-based approaches in traditional methods. From a deep learning perspective, an early attempt at such a theoretically motivated method was proposed in [48] using a convex hull formulation and distribution matching to obtain generalization bounds, which also is an inspiration for parts of our work. In [49], the authors study theoretical bounds in the DG setting using three new concepts: feature invariance across domains, feature discriminability for the classification task, and a feature expansion function to generalize the first two concepts from the source to the target domain. [50] identified measures relating to the Fisher information, predictive entropy, and maximum mean discrepancy to be good predictors of out-of-distribution generalization of empirical risk minimization (ERM) models in the DG setting. [24] proposed generalization bounds in terms of the model's Rademacher complexity and suggested using regularized ERM models for the DG problem. Finally, [14] introduced an online game in which one model (player) reduces the error on test data provided by an adversarial player, but do not provide any empirical studies. In this work, we draw inspiration from [48] by considering a convex combination of source domains and leverage this to propose a new margin-based approach to the theoretical analysis of DG, which helps develop an informative generalization bound. Motivated by this analysis, we propose the MADG algorithm, a novel margin-based adversarial learning approach for DG, which shows consistent performance over other state-of-the-art methods across benchmarks.
We consider a set of source domains  $\mathcal{D}_{S_i}$  with  $i\in \{1,2,\ldots ,N_s\}$  and a set of unseen domains  $\mathcal{D}_{U_m}$  with  $m\in \{1,2,\dots ,N_u\}$ . We use  $\mathcal{D}$  to refer to any of these domains when the index is not relevant. We use the terms 'domain' and 'distribution' interchangeably in this work. Each such domain  $\mathcal{D}\subseteq \mathcal{X}\times \mathcal{Y}$ , where  $\mathcal{X}$  is an input space and  $\mathcal{V}$  is an output space, which is  $\{0,1\}$  in binary classification and  $\{1,\dots ,k\}$  in multi-class classification. We use  $\hat{D}$  to denote a set of samples drawn independently from  $\mathcal{D}$ , i.e.  $\hat{D} = \{(x_{i},y_{i})\}_{i = 1}^{n}$  where  $x_{i}\in \mathcal{X}$  and  $y_{i}\in \mathcal{V},\forall i\in \{1,2,\ldots ,n\}$ . We use  $(x,y)$  to refer to a labeled sample  $(x_{i},y_{i})$ , when the index is not relevant.
As in [51], we consider the multi-class setting with a hypothesis space  $\mathcal{F}$  of scoring functions  $f:\mathcal{X}\to \mathbb{R}^{|\mathcal{V}|} = \mathbb{R}^k$ , where the outputs report the confidence of the prediction on each dimension. Similar to [40], we use  $f(x,y)$  to denote the component of  $f(x)$  that corresponds to the label  $y$ . In order to obtain the final predicted label, we also consider a labeling function space  $\mathcal{H}$  containing  $h_f:\mathcal{X}\rightarrow \mathcal{V}$  such that  $h_f(x) = \operatorname*{argmax}_{y\in \mathcal{V}}f(x,y)$ , i.e. the predicted label assigned to data sample  $x$  is the one resulting in the largest confidence score.
The expected error rate of a classifier  $h \in \mathcal{H}$  with respect to a distribution  $\mathcal{D}$  is defined as  $err_{\mathcal{D}}(h) \triangleq \mathbb{E}_{(x,y) \sim \mathcal{D}} \mathbb{1}[h(x) \neq y]$  where  $\mathbb{1}$  is the indicator function. We also define the margin  $\rho_f(\cdot)$  and the corresponding margin loss of hypothesis  $f$  for a labeled example  $(x,y)$  as follows:
$$
\rho_ {f} (x, y) \triangleq \frac {1}{2} \left(f (x, y) - \max  _ {y ^ {\prime} \neq y} f \left(x, y ^ {\prime}\right)\right); \quad e r r _ {\mathcal {D}} ^ {(\rho)} (f) \triangleq \mathbb {E} _ {x \sim \mathcal {D}} \left[ \Phi_ {\rho} \circ \rho_ {f} (x, y) \right] \tag {1}
$$
where  $\circ$  denotes the composition function and  $\Phi_{\rho}$  is:
$$
\Phi_ {\rho} (t) = \left\{ \begin{array}{l l} 0 & \text {i f} \rho \leq t \\ 1 - \frac {t}{\rho} & \text {i f} 0 \leq t \leq \rho \\ 1 & \text {i f} t \leq 0 \end{array} \right. \tag {2}
$$
The margin disparity and its empirical version are then defined as:
$$
\operatorname {d i s p} _ {\mathcal {D}} ^ {(\rho)} \left(f ^ {\prime}, f\right) \triangleq \mathbb {E} _ {\mathcal {D}} \left[ \Phi_ {\rho} \circ \rho_ {f ^ {\prime}} (., h _ {f}) \right] \tag {3}
$$
$$
\operatorname {d i s p} _ {\hat {D}} ^ {(\rho)} \left(f ^ {\prime}, f\right) \triangleq \mathbb {E} _ {\hat {D}} \left[ \Phi_ {\rho} \circ \rho_ {f ^ {\prime}} (., h _ {f}) \right] = \frac {1}{n} \sum_ {i = 1} ^ {n} \Phi_ {\rho} \circ \rho_ {f ^ {\prime}} \left(x _ {i}, h _ {f} \left(x _ {i}\right)\right) \tag {4}
$$
where  $f$  and  $f'$  are different scoring functions. Margin disparity discrepancy (MDD) is used to quantify the degree of discrepancy/disagreement between decision boundaries of classifiers (using their margins) trained on different domains in our context. Such a measure can be used to evaluate the generalization performance of a model across multiple domains. The MDD and its empirical version are defined in Eqn. (5) and Eqn. (6) below, respectively.
$$
\mathrm {d} _ {f, \mathcal {F}} ^ {(\rho)} \left(\mathcal {D} _ {S _ {i}}, \mathcal {D} _ {S _ {k}}\right) \triangleq \sup  _ {f ^ {\prime} \in \mathcal {F}} \left(\operatorname {d i s p} _ {\mathcal {D} _ {S _ {k}}} ^ {(\rho)} \left(f ^ {\prime}, f\right) - \operatorname {d i s p} _ {\mathcal {D} _ {S _ {i}}} ^ {(\rho)} \left(f ^ {\prime}, f\right)\right) \tag {5}
$$
$$
\mathrm {d} _ {f, \mathcal {F}} ^ {(\rho)} \left(\hat {D} _ {S _ {i}}, \hat {D} _ {S _ {k}}\right) \triangleq \sup  _ {f ^ {\prime} \in \mathcal {F}} \left(\operatorname {d i s p} _ {\hat {D} _ {S _ {k}}} ^ {(\rho)} \left(f ^ {\prime}, f\right) - \operatorname {d i s p} _ {\hat {D} _ {S _ {i}}} ^ {(\rho)} \left(f ^ {\prime}, f\right)\right) \tag {6}
$$
We use the above terminologies from [40], which however focused on domain adaptation (with one source domain and one seen target domain). Handling multiple seen domains and an unseen target domain in the DG setting is non-trivial, which we focus on in this work.
4 Margin-based Approach to Domain Generalization: Theory
This section presents the theoretical motivation for our margin-based approach to the Domain Generalization (DG) problem. Our approach is based on considering the margin disparity discrepancy (MDD), defined above, between the source domains, thereby obtaining a generalization bound. We believe such a margin-based approach has a few advantages: (i) Informative: Generalization bounds based on margin loss for classification provide more information (than 0-1 loss-based bounds) by establishing a dependency between any margin function satisfying the Lipschitz condition and the upper bounds, as demonstrated in well-known earlier work [39]. (ii) Tightness: In contrast to using a 0-1 loss to identify samples causing disagreement between classifiers, margin loss computes disagreement between scoring functions using a smooth function parameterized by the threshold  $(\rho)$ . Consequently, the number of samples causing agreement between classifiers is less, resulting in such an MDD-based bound being tighter, as shown in Fig. 1. (iii) Classifier-aware Discrepancy: MDD considers the classifier function while measuring discrepancy; as shown in Eqns 5 and 6, the supremum is computed over  $f'$  while holding  $f$  constant ( $f$  learns the posterior distribution  $\mathbb{P}(y|x)$  discriminatively for the classification task). This provides a classifier-aware approach to computing discrepancy.
(iv) Efficient Optimization and Practicality: The definition of MDD, as in Eqn. (5), only requires taking the supremum over one hypothesis. Therefore, compared to other divergence measures such as  $\mathcal{H}\Delta \mathcal{H}$  [38], MDD can be optimized with ease and is practically useful, as also stated in [40].
In this section, we derive a generalization bound for an unseen domain based on the margin-based MDD loss in the DG setting. To this end, we first show an upper bound on the unlabeled source domain error given other labeled source domains (Lemma 1 and Theorem 1). We then leverage this to develop the upper bound for the error on an unseen domain that is not necessarily a source domain (Lemma 2, Theorem 2, and Corollary 1). We subsequently analyze the upper bound
Figure 1: Space of intersection (agreement) in MDD (yellow) is reduced as compared to 0-1 loss (blue + yellow) between  $f$  and  $f'$  for labels  $\{0,1\}$ .
from Corollary 1 using the Rademacher complexity framework and develop our final generalization bound for the unseen target domain in the DG setting (Lemma 3 and Theorem 3) using our margin-based loss. In Sec 5, we show the formulation of the proposed adversarial learning algorithm, MADG, motivated by the generalization bound in Sec 4, that employs MDD to address the DG problem.
We begin by considering a setting where training data consists of  $N_{s} - 1$  labeled source domains  $\mathcal{D}_{S_i}, i = \{1, \dots, N_s - 1\}$  and a single unlabeled source domain  $\mathcal{D}_{S_T}$ , and show an upper bound on error in this setting. We later use this to show an upper bound for the DG setting. We first establish an upper bound on the MDD between a weighted sum of the labeled source domains and the unlabeled one in Lemma 1 below.
Lemma 1 Consider a weighted sum of  $N_{s} - 1$  labeled source distributions defined as  $\mathcal{D}_{\bar{S}} := \sum_{i=1}^{N_{s}-1} \alpha_{i} \mathcal{D}_{S_{i}}$ , where  $\alpha_{i} s$  are mixture co-efficients s.t.  $\sum_{i=1}^{N_{s}-1} \alpha_{i} = 1$ , and  $f$  is a scoring function. Then
$$
d _ {f, \mathcal {F}} ^ {(\rho)} \left(\mathcal {D} _ {\bar {S}}, \mathcal {D} _ {\mathcal {S} _ {T}}\right) \leq \sum_ {i = 1} ^ {N _ {s} - 1} \alpha_ {i} d _ {f, \mathcal {F}} ^ {(\rho)} \left(\mathcal {D} _ {\mathcal {S} _ {i}}, \mathcal {D} _ {\mathcal {S} _ {T}}\right) \tag {7}
$$
Detailed proofs for all our theoretical results are provided in the Appendix. It follows from Lemma 1 that an effective way to minimize the discrepancy between the unlabeled source domain and the mixture of labeled source domains in the hypothesis space is by minimizing the convex sum of the pairwise MDD between each labeled and unlabeled source domain. Building upon this insight, we provide bounds on the unlabeled source error below in Theorem 1.
Theorem 1 Consider a scoring function  $f$ , unlabeled source domain  $\mathcal{D}_{\mathcal{S}_T}$  and a mixture of  $N_s - 1$  source distributions denoted as  $\mathcal{D}_{\bar{\mathcal{S}}} := \sum_{i=1}^{N_s-1} \alpha_i \mathcal{D}_{\mathcal{S}_i}$ , where  $\alpha_i$  is a mixture co-efficients s.t.  $\sum_{i=1}^{N_s-1} \alpha_i = 1$ . Then the error on the unlabeled source  $\mathcal{D}_{\mathcal{S}_T}$  is bounded as:
$$
\operatorname {e r r} _ {\mathcal {D} _ {S _ {T}}} \left(h _ {f}\right) \leq \sum_ {i = 1} ^ {N _ {s} - 1} \alpha_ {i} \left(\operatorname {e r r} _ {\mathcal {D} _ {S _ {i}}} ^ {(\rho)} (f) + d _ {f, \mathcal {F}} ^ {(\rho)} \left(\mathcal {D} _ {S _ {i}}, \mathcal {D} _ {S _ {T}}\right)\right) + \hat {\lambda} \tag {8}
$$
Remark 1: From Theorem 1, we observe that the unlabeled source error is upper bounded by the labeled source errors, the pairwise discrepancy between each labeled and unlabeled source domain, and the ideal margin loss described below in Remark 2. We can also interpret Theorem 1 as an upper bound for the multi-source domain adaptation (DA) setting. While this is not our primary focus in this work, we report preliminary empirical results for a multi-source DA algorithm that reduces the first two terms in Theorem 1 in the Appendix.
Remark 2: The ideal loss  $\hat{\lambda}$  is defined as  $\hat{\lambda} = \min_{f^{*}\in \mathcal{F}}\Bigl (\sum_{i = 1}^{N_{s} - 1}\alpha_{i}err_{\mathcal{D}_{S_{i}}}^{(\rho)}(f^{*}) + err_{\mathcal{D}_{S_{T}}}^{(\rho)}(f^{*})\Bigr)$  and is a constant that is independent of function  $f$
Building on Theorem 1, we now develop bounds for the DG setting, where all source domains  $(N_{s})$  are labeled, and the target domain is unseen during training. To this end, we consider the relationship between the unseen target and multiple source domains. In particular, we derive our error bound on the unseen target domain using the convex hull of the labeled source domains and MDD. We also provide a DG algorithm motivated by our theoretical analysis, which we describe later in Sec 5.
Before stating our main theorem, we present Lemma 2, which states that if the maximum MDD between any two sources is bounded above by  $\epsilon$ , then the MDD between any two domains that belong to the convex hull of all the sources will also be bounded by  $\epsilon$ . Let  $\Lambda_s$  be the convex hull of source domains defined as  $\Lambda_s = \left\{\bar{D}:\bar{D} = \sum_{i = 1}^{N_s}\pi_i\mathcal{D}_{S_i}\text{ where } \sum_{i = 1}^{N_s}\pi_i = 1\right\}$ .
Lemma 2 Let  $d_{f,\mathcal{F}}^{(\rho)}\left(\mathcal{D}_{\mathcal{S}_i}, \mathcal{D}_{\mathcal{S}_k}\right) \leq \epsilon \forall i, k \in \{1,2,\dots,N_s\}$  and  $f$  be a scoring function. Then the following inequality holds for the MDD between any pair of domains  $\mathcal{D}', \mathcal{D}'' \in \Lambda_s$ :
$$
d _ {f, \mathcal {F}} ^ {(\rho)} \left(\mathcal {D} ^ {\prime}, \mathcal {D} ^ {\prime \prime}\right) \leq \epsilon \tag {9}
$$
We observe from Lemma 2 that in the hypothesis space, the discrepancy among all the domains (seen or unseen) that belong to the convex hull  $\Lambda_s$  can be reduced by minimizing the maximum MDD between two source domains. With the necessary tools at hand, we state the key theorem, i.e., the unseen target error bound for the DG problem below in Theorem 2.
Theorem 2 Consider a mixture of  $N_{s}$  source distributions, scoring function  $f$ , unseen domain  $\mathcal{D}_{U_m}$ , and  $\gamma = d_{f,\mathcal{F}}^{(\rho)}\big(\mathcal{D}_{U_m},\bar{\mathcal{D}}_U\big)$  where  $\bar{\mathcal{D}}_U$  is the projection of  $\mathcal{D}_{U_m}$  onto the convex hull of the sources i.e.  $\bar{\mathcal{D}}_U = \arg \min_{\pi_1,\pi_2,\dots ,\pi_{N_s}}d_{f,\mathcal{F}}^{(\rho)}\Big(\mathcal{D}_{U_m},\sum_{i = 1}^{N_s}\pi_i\mathcal{D}_{S_i}\Big),\sum_{i = 1}^{N_s}\pi_i = 1.$  Then, the unseen target error is bounded as follows:
$$
\operatorname {e r r} _ {\mathcal {D} _ {U _ {m}}} \left(h _ {f}\right) \leq \sum_ {i = 1} ^ {N _ {s}} \pi_ {i} \left(\operatorname {e r r} _ {\mathcal {D} _ {S _ {i}}} ^ {(\rho)} (f)\right) + \epsilon + \gamma + \bar {\lambda} \tag {10}
$$
Remark 3: As defined in Theorem 2,  $\gamma = \mathrm{d}_{f,\mathcal{F}}^{(\rho)}\big(\mathcal{D}_{U_m},\bar{\mathcal{D}}_U\big)$ . Two scenarios therefore arise:  $\gamma = 0$  when the unseen domain falls in the convex hull, i.e.,  $\mathcal{D}_{U_m} = \sum_{i = 1}^{N_s}\pi_i\mathcal{D}_{S_i}$  or  $\gamma >0$  when the unseen domain cannot be represented by available domains alone. Thus this parameter can be interpreted as the need for diverse source domains. The more diverse the source domains are, the smaller the value of  $\gamma$ .
Remark 4: As seen from Lemma 2,  $\epsilon$  is defined as the upper bound for the MDD between any two domains that belong to the convex hull  $\Lambda_s$  formed by the source domains. Thus, we can also interpret  $\epsilon$  as the highest MDD value among the source domains as shown below in Eqn 11.