SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qizhou Wang $^{1*}$  Zhen Fang $^{2*}$  Yonggang Zhang $^{1}$  Feng Liu $^{3}$  Yixuan Li $^{4}$  Bo Han $^{1\dagger}$
$^{1}$ Department of Computer Science, Hong Kong Baptist University
$^{2}$ Australian Artificial Intelligence Institute, University of Technology Sydney
$^{3}$ School of Computing and Information Systems, The University of Melbourne
$^{4}$ Department of Computer Sciences, University of Wisconsin-Madison
{csqzwang, csygzhang, bhanml} @comp.hkbu.edu.hk
zhen.fang@uts.edu.au fengliu.ml@gmail.com sharonli@cs.wisc.edu
Open-world classification systems should discern out-of-distribution (OOD) data whose labels deviate from those of in-distribution (ID) cases, motivating recent studies in OOD detection. Advanced works, despite their promising progress, may still fail in the open world, owing to the lack of knowledge about unseen OOD data in advance. Although one can access auxiliary OOD data (distinct from unseen ones) for model training, it remains to analyze how such auxiliary data will work in the open world. To this end, we delve into such a problem from a learning theory perspective, finding that the distribution discrepancy between the auxiliary and the unseen real OOD data is the key to affecting the open-world detection performance. Accordingly, we propose Distributional-Augmented OOD Learning (DAL), alleviating the OOD distribution discrepancy by crafting an OOD distribution set that contains all distributions in a Wasserstein ball centered on the auxiliary OOD distribution. We justify that the predictor trained over the worst OOD data in the ball can shrink the OOD distribution discrepancy, thus improving the open-world detection performance given only the auxiliary OOD data. We conduct extensive evaluations across representative OOD detection setups, demonstrating the superiority of our DAL over its advanced counterparts. The code is publicly available at: https://github.com/tmlr-group/DAL.
Deep learning in the open world often encounters out-of-distribution (OOD) data of which the label space is disjoint with that of the in-distribution (ID) cases (Hendrycks and Gimpel, 2017; Fang et al., 2022). It leads to the well-known OOD detection problem, where the predictor should make accurate predictions for ID data and detect anomalies from OOD cases (Bulusu et al., 2020; Yang et al., 2021). Nowadays, OOD detection has attracted intensive attention in reliable machine learning due to its integral role in safety-critical applications (Cao et al., 2020; Shen et al., 2021).
OOD detection remains challenging since predictors can make over-confidence predictions for OOD data (Hendrycks et al., 2019), motivating recent studies towards effective OOD detection. Therein, outlier exposure (Hendrycks et al., 2019; Ming et al., 2022) is among the most potent ones, learning from auxiliary OOD data to discern ID and OOD patterns. However, due to the openness of the OOD task objective (Wang et al., 2023), auxiliary OOD data can arbitrarily differ from the (unseen) real OOD data in the open world. So, to formally understand their consequences, we model the difference
$^{\dagger}$  Correspondence to Bo Han (bhanml@comp.hkbu.edu.hk).
between auxiliary and real OOD data by their distribution discrepancy, measured by the Wasserstein distance (Villani, 2021, 2008). Then, we reveal the negative impacts of such OOD distribution discrepancy on the real detection power, with a larger distribution discrepancy indicating a lower performance on real OOD data, cf., Eq. (4).
The OOD distribution discrepancy threatens the open-world detection performance for outlier exposure. Therefore, we raise a natural question in how to alleviate such an OOD distribution discrepancy. Hence, this paper establishes a promising learning framework named Distributional-Augmented OOD Learning (DAL). Therein, we augment the auxiliary OOD distribution by crafting an OOD
distribution set containing all distributions in a Wasserstein ball (Villani, 2021, 2008), centered on the auxiliary OOD distribution. Then, by making the predictor learn from the worst OOD distribution in the set, cf., Eq. (8), one can alleviate the negative impacts of the distribution discrepancy. Moreover, our proposed framework enjoys the learning guarantees towards the expected risk with respect to the real OOD distribution, making OOD detection stay effective when facing unseen data (cf., Theorem 3). Figure 1 provides a conceptual explanation: learning from the worst OOD distribution ensures the uniformly well performance inside the Wasserstein ball, enlarging the influence of the auxiliary OOD distribution. Thus, one can shrink the OOD distribution discrepancy between the auxiliary and the real OOD data and improve OOD detection.
In realization, the primal learning objective in Eq. (8) is generally intractable due to the infinite-dimensional optimization for the worst OOD distribution search. Instead, we adopt the dual form with respect to the original learning problem (cf., Theorem 1), transforming it into a tractable problem of the worst OOD data search in a finite-dimensional space. Furthermore, following Du et al. (2022a); Mehra et al. (2022), the data search procedure is conducted in the embedding space, which can benefit the open-world performance of OOD detection with decent costs of additional computation.
We conduct extensive experiments over representative OOD detection setups, revealing the open-world performance of our method toward effective OOD detection. For example, our DAL reduces the average FPR95 by 1.99 to 13.46 on CIFAR benchmarks compared with the conventional outlier exposure (Hendrycks et al., 2019). Overall, we summarize our contributions into three folds:
- We measure the difference between the auxiliary and the real OOD data by the Wasserstein distance, and establish an effective learning framework, named DAL, to mitigate the OOD distribution discrepancy issue. We further guarantee our performance with respect to unseen real OOD data via Theorem 3, which is new to previous works.
- DAL leads to a practical method in Algorithm 1, learning from the worst cases in the Wasserstein ball to improve the open-world detection performance. Overall, our method solves the dual problem, which performs the worst-case search in the embedding space, which is simple to compute yet effective in OOD detection.
- We conduct extensive experiments in Section 5 to evaluate our effectiveness, ranging from the well-known CIFAR benchmarks to the challenging ImageNet settings. The empirical results comprehensively demonstrate our superiority over advanced counterparts, and the improvement is mainly attributed to our distributional-augmented learning framework.
A detailed overview of existing OOD detection methods and theories can be found in Appendix A, and a summary of the important notations can be found in Appendix B.
Figure 1: A heuristic illustration for our DAL. A large distribution discrepancy between the auxiliary and the unseen OOD data will hurt the real detection effectiveness. However, by ensuring uniformly well performance inside the Wasserstein ball, we can mitigate the distribution discrepancy and thus improve the detection power in the open world.
2 Outlier Exposure
Let  $\mathcal{X}$  denote the feature space and  $\mathcal{Y} = \{1,\dots ,C\}$  denote the label space with respect to the ID distribution. We consider the ID distribution  $D_{X_{\mathrm{I}}Y_{\mathrm{I}}}$ , a joint distribution defined over  $\mathcal{X}\times \mathcal{Y}$ , where  $X_{\mathrm{I}}$  and  $Y_{\mathrm{I}}$  are random variables whose outputs are from spaces  $\mathcal{X}$  and  $\mathcal{Y}$ . We also have an OOD joint distribution  $D_{X_{\mathrm{O}}Y_{\mathrm{O}}}$ , where  $X_{\mathrm{O}}$  is a random variable from  $\mathcal{X}$ , but  $Y_{\mathrm{O}}$  is a random variable whose outputs do not belong to  $\mathcal{Y}$ , i.e.,  $Y_{\mathrm{O}}\notin \mathcal{Y}$  (Fang et al., 2022).
The classical OOD detection (Hendrycks and Gimpel, 2017; Yang et al., 2021) typically considers an open-world setting, where the real OOD data drawn from  $D_{X_0Y_0}$  are unseen during training. Recently, Fang et al. (2022) have provided several strong conditions necessary to ensure the success of the classical OOD setting. Furthermore, to increase the possibility of success for OOD detection and weaken the strong conditions proposed by Fang et al. (2022), advanced works (Hendrycks et al., 2019; Chen et al., 2021) introduce a promising approach named outlier exposure, where a set of auxiliary OOD data is employed as a surrogate of real OOD data. Here, we provide a formal definition.
Problem 1 (OOD Detection with Outlier Exposure). Let  $D_{X_1Y_1}$ ,  $D_{X_0}$ , and  $D_{X_A}$  be the ID joint distribution, the OOD distribution, and the auxiliary OOD distribution, respectively. Given the sets of samples called the ID and the auxiliary OOD data, namely,
$$
S = \{(\mathbf {x} _ {\mathrm {I}} ^ {1}, y _ {\mathrm {I}} ^ {1}), \dots , (\mathbf {x} _ {\mathrm {I}} ^ {n}, y _ {\mathrm {I}} ^ {n}) \} \sim D _ {X _ {\mathrm {I}} Y _ {\mathrm {I}}} ^ {n}, i. i. d., T = \{\mathbf {x} _ {\mathrm {A}} ^ {1}, \dots , \mathbf {x} _ {\mathrm {A}} ^ {m} \} \sim D _ {X _ {\mathrm {A}}} ^ {m}, i. i. d.,
$$
outlier exposure trains a predictor  $\mathbf{f}$  by using the training data  $S$  and  $T$ , such that for any test data  $\mathbf{x}$ : 1) if  $\mathbf{x}$  is an observation from  $D_{X_1}$ , the predictor  $\mathbf{f}$  can classify  $\mathbf{x}$  into its correct ID label; otherwise 2) if  $\mathbf{x}$  is an observation from  $D_{X_0}$ , the predictor  $\mathbf{f}$  can detect  $\mathbf{x}$  as an OOD case.
OD Scoring. Many existing methods detect OOD data by using various score-based strategies (Hendrycks and Gimpel, 2017; Lee et al., 2018a; Liu et al., 2020; Sun et al., 2022). In general, given a model  $\mathbf{f}:\mathcal{X}\to \mathbb{R}^C$  and a scoring function  $s(\cdot ;\mathbf{f}):\mathcal{X}\rightarrow \mathbb{R}$ , the OOD detector  $g_{\lambda}$  is given by:
$$
g _ {\lambda} (\mathbf {x}) = \mathrm {I D}, \text {i f} s (\mathbf {x}; \mathbf {f}) \geq \lambda ; \text {o t h e r w i s e}, g _ {\lambda} (\mathbf {x}) = \mathrm {O O D},
$$
where  $\lambda$  is a given threshold. For example, as a well-known baseline scoring function, the maximum softmax prediction (MSP) (Hendrycks and Gimpel, 2017) is given by:
$$
s _ {\mathrm {M S P}} (\mathbf {x}; \mathbf {f}) = \max  _ {k \in \mathcal {Y}} \operatorname {s o f t m a x} _ {k} \mathbf {f} (\mathbf {x}), \tag {1}
$$
with  $\operatorname{softmax}_k(\cdot)$  denoting the  $k$ -th dimension of the softmax output.
Model and Risks. We denote  $\mathbf{f}_{\mathbf{w}}: \mathcal{X} \to \mathbb{R}^{C}$  the predictor with parameters  $\mathbf{w} \in \mathcal{W}$ , with  $\mathcal{W}$  the parameter space. We consider the loss functions  $\ell$  and  $\ell_{\mathrm{OE}}$  w.r.t. the ID and the OOD cases, respectively. Then, the expected and the empirical  $ID$  risks of the model  $\mathbf{f}_{\mathbf{w}}$  can be written as:
$$
R _ {\mathrm {I}} (\mathbf {w}) = \mathbb {E} _ {(\mathbf {x}, y) \sim D _ {X _ {\mathrm {I}} Y _ {\mathrm {I}}}} \ell (\mathbf {f} _ {\mathbf {w}}; \mathbf {x}, y) \mathrm {a n d} \widehat {R} _ {\mathrm {I}} (\mathbf {w}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \ell (\mathbf {f} _ {\mathbf {w}}; \mathbf {x} _ {\mathrm {I}} ^ {i}, y _ {\mathrm {I}} ^ {i}).
$$
The expected and the empirical auxiliary OOD risks are then given by
$$
R _ {\mathrm {A}} (\mathbf {w}) = \mathbb {E} _ {\mathbf {x} \sim D _ {X _ {\mathrm {A}}}} \ell_ {\mathrm {O E}} \left(\mathbf {f} _ {\mathbf {w}}; \mathbf {x}\right) \text {a n d} \widehat {R} _ {\mathrm {A}} (\mathbf {w}) = \frac {1}{m} \sum_ {i = 1} ^ {m} \ell_ {\mathrm {O E}} \left(\mathbf {f} _ {\mathbf {w}}; \mathbf {x} _ {\mathrm {A}} ^ {i}\right),
$$
and the expected real OOD risk is given by  $R_{\mathrm{O}}(\mathbf{w}) = \mathbb{E}_{\mathbf{x} \sim D_{X_{\mathrm{O}}}} \ell_{\mathrm{OE}}(\mathbf{f}_{\mathbf{w}}; \mathbf{x})$ . Accordingly, we can define the expected detection risk with respect to real OOD data, following
$$
R _ {D} (\mathbf {w}) = R _ {\mathrm {I}} (\mathbf {w}) + \alpha R _ {\mathrm {O}} (\mathbf {w}), \tag {2}
$$
where  $\alpha$  is the trade-off parameter.
Learning Strategy. After the scoring function is selected, one can obtain the OOD detector if the model  $\mathbf{f}_{\mathbf{w}}$  is given. Under the Problem 1 of outlier exposure, a common learning strategy is to optimize the empirical ID and auxiliary OOD risk jointly (Hendrycks et al., 2019), namely,
$$
\min  _ {\mathbf {w} \in \mathcal {W}} \left[ \widehat {R} _ {\mathrm {I}} (\mathbf {w}) + \alpha \widehat {R} _ {\mathrm {A}} (\mathbf {w}) \right]. \tag {3}
$$
Note that the auxiliary OOD data are employed in Eq. (3), which can arbitrarily differ from the real OOD cases. Then, it is generally expected that the predictor  $\mathbf{f}_{\mathbf{w}}$  , trained over the auxiliary OOD data, can perform well even on unseen OOD data, i.e., a small value of  $R_{D}(\mathbf{w})$  is expected.
To the general learning strategy in Eq. (3), intuitively, if the auxiliary data are sampled from a distribution similar to real ones, the predictor will perform well for real OOD data. However, auxiliary and real OOD data differ in practice, posing us to suspect their open-world detection performance. To formally study the problem, we measure the difference between auxiliary and real OOD data in the distribution level, motivating our discussion of OOD distribution discrepancy.
Distribution Discrepancy. In this paper, we adopt a classical measurement for the distribution discrepancy—Optimal Transport Cost (Sinha et al., 2018; Mehra et al., 2022).
Definition 1 (Optimal Transport Cost and Wasserstein-1 Distance (Villani, 2021, 2008)). Given a cost function  $c: \mathcal{X} \times \mathcal{X} \to \mathbb{R}_+$ , the optimal transport cost between two distributions  $D$  and  $D'$  is
$$
\mathrm {W} _ {c} (D, D ^ {\prime}) = \inf _ {\pi \in \Pi (D, D ^ {\prime})} \mathbb {E} _ {(\mathbf {x}, \mathbf {x} ^ {\prime}) \sim \pi} c (\mathbf {x}, \mathbf {x} ^ {\prime}),
$$
where  $\Pi(D, D')$  is the space of all couplings for  $D$  and  $D'$ . Furthermore, if the cost  $c$  is a metric, then the optimal transport cost is also called the Wasserstein-1 distance.
Based on Definition 1, we use the distribution discrepancy to measure the difference between the auxiliary and the real OOD data, namely,  $\mathrm{W}_c(D_{X_0},D_{X_A})$ . Then, we can formally study the impacts of such a discrepancy on the detection performance of the predictor. Under certain assumptions (cf., Corollary 1), we can prove that with high probability, the following generalization bound holds:
$$
R _ {D} (\widehat {\mathbf {w}}) \leq \min  _ {\mathbf {w} \in \mathcal {W}} \left(R _ {\mathrm {I}} (\mathbf {w}) + \alpha R _ {\mathrm {A}} (\mathbf {w})\right) + \alpha L _ {c} \mathrm {W} _ {c} \left(D _ {X _ {\mathrm {O}}}, D _ {X _ {\mathrm {A}}}\right) + \mathcal {O} (1 / \sqrt {n}) + \mathcal {O} (1 / \sqrt {m}), \tag {4}
$$
where  $\widehat{\mathbf{w}}$  is the parameter learned by Eq. (3), i.e.,  $\widehat{\mathbf{w}}\in \arg \min_{\mathbf{w}\in \mathcal{W}}\widehat{R}_{\mathrm{I}}(\mathbf{w}) + \alpha \widehat{R}_{\mathrm{A}}(\mathbf{w}),L_{c}$  is the Lipschitz constant of  $\ell_{\mathrm{OE}}$  w.r.t. the cost function  $c(\cdot ,\cdot)$  (see Theorem 3). In general, the expected detection risk  $R_{D}(\widehat{\mathbf{w}})$  measures the expected performance on unseen OOD data given the predictor trained on the auxiliary OOD data. Then, due to the upper bound, the impacts of the OOD distribution discrepancy are reflected by the Wasserstein-1 distance between the auxiliary and the real OOD data, i.e.,  $\dot{\mathrm{W}}_c(D_{X_O},D_{X_A})$ . Therefore, although classical outlier exposure can improve OOD detection to some extent, it fails to ensure reliable detection of unseen OOD data, in that a larger distribution discrepancy generally indicates a worse guarantee for open-world OOD detection.
The key to improve the detection performance is mitigating the negative impact induced by the OOD distribution discrepancy. To tackle this problem, a simple lemma inspires us:
Lemma 1. Let  $d(\cdot, \cdot)$  be the distance to measure the discrepancy between distributions. Given a space  $\mathfrak{D}$  consisting of some OOD distributions, if  $D_{X_{\Lambda}} \in \mathfrak{D}$ , then
$$
\inf  _ {D _ {X ^ {\prime}} \in \mathfrak {D}} d \left(D _ {X ^ {\prime}}, D _ {X _ {\mathrm {O}}} \right) \leq d \left(D _ {X _ {\mathrm {A}}}, D _ {X _ {\mathrm {O}}}\right). \tag {5}
$$
If  $d(\cdot, \cdot)$  is the Optimal Transport Cost in Definition 1, the cost function  $c$  is a continuous metric, and  $\mathfrak{D}$  is the Wasserstein-1 ball with a radius  $\rho > 0$ , i.e.,  $\mathfrak{D} = \{D_{X'} : W_c(D_{X'}, D_{X_A}) \leq \rho\}$ , then
$$
\inf  _ {D _ {X ^ {\prime}} \in \mathfrak {D}} \mathrm {W} _ {\mathrm {c}} \left(D _ {X ^ {\prime}}, D _ {X _ {\mathrm {O}}}\right) \leq \max  \left\{\mathrm {W} _ {\mathrm {c}} \left(D _ {X _ {\mathrm {A}}}, D _ {X _ {\mathrm {O}}}\right) - \rho , 0 \right\}. \tag {6}
$$
In the light of Lemma 1, we introduce a specific set of distributions  $\mathfrak{D}$ , augmented around the auxiliary OOD distribution. It makes it possible to mitigate the distribution discrepancy, following Eqs. (5) and (6). Therefore, instead of choosing a model  $\mathbf{f}_{\mathbf{w}}$  that directly minimizes the empirical risk in Eq. (3), we target augmenting the auxiliary OOD data within the distribution space  $\mathfrak{D}$ , namely,
$$
\min  _ {\mathbf {w} \in \mathcal {W}} \left[ \widehat {R} _ {\mathrm {I}} (\mathbf {w}) + \alpha \sup  _ {D _ {X ^ {\prime}} \in \mathfrak {D}} \mathbb {E} _ {\mathbf {x} \sim D _ {X ^ {\prime}}} \ell_ {\mathrm {O E}} \left(\mathbf {f} _ {\mathbf {w}}; \mathbf {x}\right) \right], \text {s u b j e c t t o} \widehat {D} _ {X _ {\mathrm {A}}} \in \mathfrak {D}, \tag {7}
$$
where  $\widehat{D}_{X_{\mathrm{A}}}$  is the empirical form of  $D_{X_{\mathrm{A}}}$ , i.e.,  $\widehat{D}_{X_{\mathrm{A}}} = \frac{1}{m}\sum_{i=1}^{m}\delta_{\mathbf{x}_{\mathrm{A}}^i}$  and  $\delta_{\mathbf{x}_{\mathrm{A}}^i}$  is the dirac measure.
4 Learning Framework
This section proposes a general learning framework to mitigate the OOD distribution discrepancy. As aforementioned, we consider an augmented set of OOD distributions to improve OOD detection, thus named Distributional-Augmented OOD Learning (DAL).
To begin with, we need to select a suitable distribution space  $\mathfrak{D}$  for the tractable solutions of Eq. (7). Generally, the choice of  $\mathfrak{D}$  influences both the richness of the auxiliary data as well as the tractability of the resulting optimization problem. Previous works have developed a series of distribution spaces, e.g., the distribution ball based on  $f$ -divergences (Namkoong and Duchi, 2016; Michel et al., 2021) and maximum mean discrepancy (MMD) (Staib and Jegelka, 2019). However, there are several drawbacks for the distribution balls based on  $f$ -divergences and MMD: 1) any  $f$ -divergence-based space  $\mathfrak{D}$  contains only distributions within the same support set as  $\widehat{D}_{X_{\mathrm{A}}}$ ; and 2) the effective solutions in the MMD-based space have not been provided (Staib and Jegelka, 2019).
Instead, motivated by Sinha et al. (2018); Mehra et al. (2022); Dai et al. (2023) and Theorem 1, we consider the Wasserstein ball. For any  $\rho > 0$ , we define the augmented OOD distribution set as
$$
\mathfrak {D} = \left\{D _ {X ^ {\prime}}: \mathrm {W} _ {c} \left(D _ {X ^ {\prime}}, \widehat {D} _ {X _ {\mathrm {A}}} \right) \leq \rho \right\},
$$
and consider the following optimization problem:
$$
\min  _ {\mathbf {w} \in \mathcal {W}} \widehat {R} _ {D} (\mathbf {w}; \rho) = \min  _ {\mathbf {w} \in \mathcal {W}} \left[ \widehat {R} _ {\mathrm {I}} (\mathbf {w}) + \alpha \widehat {R} _ {\mathrm {O}} (\mathbf {w}; \rho) \right], \tag {8}
$$
$$
\widehat {R} _ {\mathrm {O}} (\mathbf {w}; \rho) = \sup  _ {\mathrm {W} _ {c} \left(D _ {X ^ {\prime}}, \widehat {D} _ {X _ {\mathrm {A}}}\right) \leq \rho} \mathbb {E} _ {\mathbf {x} \sim D _ {X ^ {\prime}}} \ell_ {\mathrm {O E}} \left(\mathbf {f} _ {\mathbf {w}}; \mathbf {x}\right). \tag {9}
$$
However, the optimization problem in Eq. (8) is intractable due to the infinite-dimensional search for the distribution  $D_{X^{\prime}}$ . Fortunately, the following dual theorem provides a solution:
Theorem 1 (Blanchet and KarthyekRajhaaA. (2016)). Let  $c(\cdot, \cdot)$  be a continuous metric and  $\phi_{\gamma}(\mathbf{w}; \mathbf{x}) = \sup_{\mathbf{x}' \in \mathcal{X}} \{\ell(\mathbf{f}_{\mathbf{w}}; \mathbf{x}') - \gamma c(\mathbf{x}', \mathbf{x})\}$  be the robust surrogate function. Then, for any  $\rho > 0$
$$
\widehat {R} _ {D} (\mathbf {w}; \rho) = \widehat {R} _ {\mathrm {I}} (\mathbf {w}) + \alpha \inf  _ {\gamma \geq 0} \left\{\gamma \rho + \frac {1}{m} \sum_ {i = 1} ^ {m} \phi_ {\gamma} (\mathbf {w}; \mathbf {x} _ {\mathrm {A}} ^ {i}) \right\}. \tag {10}
$$
Theorem 1 provides a feasible surrogate for the original optimization problem in Eq. (8), transforming the infinite-dimensional problem to its finite counterpart, i.e., the data feature search. We use Eq. (10) to design our algorithm, cf., Section 4.2.
4.1 Theoretical Supports
This section provides the theoretical support for our DAL. Specifically, 1) Theorem 2 shows that the empirical model given by Eq. (8) can achieve consistent learning performance, and 2) Theorem 3 further demonstrates the expected detection risk estimation, i.e.,  $R_{D}(\mathbf{w})$ , with respect to the empirical model given by Eq. (8). All the proofs can be found in Appendix C. To state our theoretical results, we use the notation  $R_{D}(\mathbf{w};\rho)$  to represent the ideal form of  $\widehat{R}_D(\mathbf{w};\rho)$ , which is defined by
$$
R _ {D} (\mathbf {w}; \rho) = R _ {\mathrm {I}} (\mathbf {w}) + \alpha R _ {\mathrm {O}} (\mathbf {w}; \rho),
$$
$$
R _ {\mathrm {O}} (\mathbf {w}; \rho) = \sup  _ {\mathrm {W} _ {c} (D _ {X ^ {\prime}}, D _ {X _ {\mathrm {A}}}) \leq \rho} \mathbb {E} _ {\mathbf {x} \sim D _ {X ^ {\prime}}} \ell_ {\mathrm {O E}} (\mathbf {f} _ {\mathbf {w}}; \mathbf {x}).
$$
Similar to Sinha et al. (2018), our results rely on the covering number (cf., Appendix C.1) for the model classes  $\mathcal{F} = \{\ell (\mathbf{f}_{\mathbf{w}};\cdot):\mathbf{w}\in \mathcal{W}\}$  and  $\mathcal{F}_{\mathrm{OE}} = \{\ell_{\mathrm{OE}}(\mathbf{f}_{\mathbf{w}};\cdot):\mathbf{w}\in \mathcal{W}\}$  to represent their complexity. Intuitively, the covering numbers  $\mathcal{N}(\mathcal{F},\epsilon ,L^{\infty})$  and  $\mathcal{N}(\mathcal{F}_{\mathrm{OE}},\epsilon ,L^{\infty})$  are the minimal numbers of  $L^{\infty}$  balls of radius  $\epsilon >0$  needed to cover the model classes  $\mathcal{F}$  and  $\mathcal{F}_{\mathrm{OE}}$ , respectively. Now, we demonstrate that DAL can achieve consistent performance under mild assumptions.
Theorem 2 (Excess Generalization Bound). Assume that  $0 \leq \ell(\mathbf{f}_{\mathbf{w}}; \mathbf{x}, y) \leq M_{\ell}$ ,  $0 \leq \ell_{\mathrm{OE}}(\mathbf{f}_{\mathbf{w}}; \mathbf{x}) \leq M_{\ell_{\mathrm{OE}}}$ , and  $c(\cdot, \cdot): \mathcal{X} \times \mathcal{X} \to \mathbb{R}_+$  is a continuous metric. Let  $\hat{\mathbf{w}}$  be the optimal solution of Eq. (8), i.e.,  $\hat{\mathbf{w}} \in \arg \min_{\mathbf{w} \in \mathcal{W}} \widehat{R}_D(\mathbf{w}; \rho)$ . Then with the probability at least  $1 - 4e^{-t} > 0$ ,
$$
R _ {D} (\widehat {\mathbf {w}}; \rho) - \min  _ {\mathbf {w} \in \mathcal {W}} R _ {D} (\mathbf {w}; \rho) \leq \epsilon (n, m; t), \tag {11}
$$
for any  $\rho >0$  , where
$$
\begin{array}{l} \epsilon (n, m; t) = \frac {b _ {0} M _ {\ell}}{\sqrt {n}} \int_ {0} ^ {1} \sqrt {\log \mathcal {N} (\mathcal {F} , M _ {\ell} \epsilon , L ^ {\infty})} d \epsilon + 2 M _ {\ell} \sqrt {\frac {2 t}{n}} \\ + \alpha b _ {1} \sqrt {\frac {M _ {\ell_ {\mathrm {O E}}} ^ {3}}{\rho^ {2} m}} \int_ {0} ^ {1} \sqrt {\log \mathcal {N} (\mathcal {F} _ {\mathrm {O E}} , M _ {\ell_ {\mathrm {O E}}} \epsilon , L ^ {\infty})} \mathrm {d} \epsilon + \alpha b _ {2} M _ {\ell_ {\mathrm {O E}}} \sqrt {\frac {2 t}{m}}, \\ \end{array}
$$