SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xin Zou Weiwei Liu*
Out-of-distribution (OOD) generalization has attracted increasing research attention in recent years, due to its promising experimental results in real-world applications. Interestingly, we find that existing OOD generalization methods are vulnerable to adversarial attacks. This motivates us to study OOD adversarial robustness. We first present theoretical analyses of OOD adversarial robustness in two different complementary settings. Motivated by the theoretical results, we design two algorithms to improve the OOD adversarial robustness. Finally, we conduct experiments to validate the effectiveness of our proposed algorithms. Our code is available at https://github.com/ZouXinn/OOD-Adv.
Recent years have witnessed the remarkable success of modern machine learning techniques in many applications. A fundamental assumption of most machine learning algorithms is that the training and test data are drawn from the same underlying distribution. However, this assumption is always violated in many practical applications. The test environment is influenced by a range of factors, such as the distributional shifts across the photos caused by different cameras in image classification tasks, the voices of different persons in voice recognition tasks, and the variations between scenes in self-driving tasks [48]. Therefore, there is now a rapidly growing body of research with a focus on generalizing to unseen distributions, namely out-of-distribution (OOD) generalization [56].
Deep neural networks (DNNs) have achieved state-of-the-art performance in many fields. However, several prior works [59, 25] have demonstrated that DNNs may be vulnerable to imperceptibly changed adversarial examples, which has increased focus on the adversarial robustness of the models. Adversarial robustness refers to the invariance of a model to small perturbations of its input [54], while adversarial accuracy refers to a model's prediction performance on adversarial examples generated by an attacker. However, the adversarial robustness of OOD generalization models (OOD adversarial robustness) is less explored, despite its importance in many systems requiring high security such as self-driving cars. We evaluate the adversarial robustness of the models trained with the current OOD generalization algorithms (the detailed experimental settings can be found in Appendix C.3), and present the results in Table 1. Surprisingly, under the PGD-20 [44] attack, the algorithms achieve nearly  $0\%$  adversarial accuracy on RotatedMNIST [21], VLCS [15], PACS [38], and OfficeHome [62], and no more than  $10\%$  adversarial accuracy on ColoredMNIST [4]. These results show that even if the OOD generalization algorithms generalize well in different scenes, they remain highly vulnerable to adversarial attacks.
Motivated by these limitations of existing algorithms, we provide theoretical analyses for OOD adversarial robustness in two different but complementary OOD settings. We then design two
*Corresponding author: Weiwei Liu (liuweiwei863@gmail.com).
Table 1: The results  $(\%)$  for some of the current OOD generalization algorithms. We present the results in the form of a/b: here, a is the OOD adversarial accuracy under PGD-20 attack [44]; and b is the OOD adversarial accuracy under AutoAttack [12]. We conduct the  $\ell_{\infty}$ -norm attack. We use the perturbation radius  $\epsilon = 0.1$  for RotatedMNIST and ColoredMNIST, and  $\epsilon = \frac{4}{255}$  for VLCS. For the architecture, following [26], we use a small CNN-architecture for RotatedMNIST and ColoredMNIST, and ResNet-50 [27] for VLCS, PACS and OfficeHome. Since [64] do not realize MAT and LDAT for RotatedMNIST, we use X to denote the unrealized results. For more details about the algorithms, please refer to Appendix C.1. We use RM, CM, and OH as the abbreviation of RotatedMNIST, ColoredMNIST, and OfficeHome, respectively.
Algorithm	RM	CM	VLCS	PACS	OH	Avg
ERM [61]	0.6/0.0	5.8/X	0.0/0.0	0.3/0.6	0.4/0.0	1.4/X
MLDG [39]	0.2/0.0	4.8/X	0.0/0.0	0.1/0.3	0.6/0.1	1.2/X
CDANN [42]	0.9/0.0	8.2/X	3.0/0.0	1.5/0.3	0.1/0.0	2.7/X
VREx [33]	0.2/0.0	6.4/X	0.0/0.0	0.0/0.3	0.4/0.1	1.4/X
RSC [28]	1.0/0.0	3.9/X	0.0/0.0	0.1/0.4	0.7/0.0	1.1/X
MAT [64]	X/X	10.7/X	0.0/0.0	0.7/1.4	0.8/0.1	X/X
LDAT [64]	X/X	7.9/X	0.0/0.0	0.1/0.3	0.4/0.1	X/X
baseline algorithms to improve the OOD adversarial robustness, based on the implications of our theory, and validate the effectiveness of our proposed algorithms through experiments.
1. We evaluate the adversarial robustness of current OOD generalization algorithms and experimentally verify that the current OOD generalization algorithms are vulnerable to adversarial attacks.
2. We present theoretical analyses for the adversarial OOD generalization error bounds in the average case and the limited training environments case. Specifically, our bounds in limited training environments involve a "distance" term between the training and the test environments. We further use a toy example to illustrate how the "distance" term affects the OOD adversarial robustness, which is verified by the experimental results in Section 5.
3. Inspired by our theory, we propose two algorithms to improve OOD adversarial robustness. Extensive experiments show that our algorithms are able to achieve more than  $53\%$  average adversarial accuracy over the datasets.
The remainder of this article is structured as follows: §2 introduces related works. §3 presents our main theory. §4 shows our two theory-driven algorithms. §5 provides our experimental results. Finally, the conclusions are presented in the last section.
2.1 Adversarial Robustness
[59] show that DNNs are fragile to imperceptible distortions in the input space. One of the most popular methods used to improve adversarial robustness is adversarial training (AT). The seminal AT work, the fast gradient sign method [25, FGSM], perturbs a sample towards its gradient direction to increase the loss, then uses the generated sample to train the model. Following this line of research, [44, 47, 34, 11] propose iterative variants of the gradient attack with improved AT frameworks. [70, 30, 41] investigates the adversarial robustness from the perspective of ordinary differential equations. Recently, [50, 66] utilize the data from generative models as data augmentation to improve adversarial robustness. Besides, [43] analyze the trade-off between robustness and fairness, [37] study the worst-class adversarial robustness in adversarial training.
For the theoretical perspective, [46] study the PAC learnability of adversarial robust learning, [69] extend the work of [46] to multiclass case, [72, 31, 5, 18, 67] give theoretical analyses to adversarial training by standard uniform convergence argumentation and giving a bound of the Rademacher complexity, and [65, 78] study adversarial robustness under self-supervised learning.
2.2 Out-of-distribution generalization
OOD generalization aims to train a model with data from the training environments so that it is capable of generalizing to an unseen environment. A large number of algorithms have been developed that aim to improve OOD generalization. One series of works focuses on minimizing the discrepancies between the training environments [40, 17, 42, 58, 1]. The most related work among them is [1], which measures the discrepancy between the domains by  $d_{\mathcal{H}}(S,T)$ , while we focus on adversarial robustness and use  $d_{\ell(\mathcal{H})}^B(S,T)$ . Meta-learning domain generalization [39, MLDG] leverages the meta-learning approach and simulates train/test distributional shift during training by synthesizing virtual testing domains within each mini-batch. [53, GroupDRO] studies applying distributionally robust optimization (DRO) [19, 36, 77, 13] to learn models that instead minimize the worst-case training loss over a set of pre-defined groups. Another line of works [68, 64] conducts adversarial training to improve the OOD generalization performance. In this work, we focus on improving the OOD adversarial robustness.
From the theoretical perspective, [10] introduce a formal framework and argue that OOD generalization can be viewed as a kind of supervised learning problem by augmenting the original feature space with the marginal distribution of feature vectors. In [51], OOD generalization is cast into an online game where a player (model) minimizes the risk for a "new" distribution presented by an adversary at each time-step. [14] propose a probabilistic framework for domain generalization called Probable Domain Generalization, wherein the key idea is that distribution shifts seen during training should inform us of probable shifts at test time. Notably, all these works focus on OOD generalization performance, while we present theoretical results for OOD adversarial robustness.
2.3 Works relating domain shifts and adversarial robustness
[8] focuses on improving the adversarial robustness of the models by regarding the adversarial distribution as the target domain and then applying the domain adaptation methods, while we focus on improving the model's adversarial robustness on the OOD distribution. [71] studies the relationship between adversarial robustness and OOD generalization, it shows that good adversarial robustness implies good OOD performance when the target domain lies in a Wasserstein ball. While we study the OOD adversarial robustness and propose algorithms to improve OOD adversarial robustness. [2] empirically analyzes the transferability of models' adversarial/certified robustness under distributional shifts. It shows that adversarily trained models do not generalize better without fine-tuning and that the accuracy-robustness trade-off generalizes to the unseen domain. Its results for adversarial robustness can also be found in our experimental results. [29] investigates how to improve the adversarial robustness of a model against ensemble attacks or unseen attacks. [29] regards the adversarial examples for each type of attack (such as FGSM, PGD, CW, and so on) as a domain, and utilizes the OOD generalization methods to improve the models generalization performance under different (maybe unseen) attacks.
[45, 22] study the relationship between the dependence on spurious correlations and the adversarial robustness of the models and show that adversarial training increases the model's reliance on spurious features. [60] studies the relationship between fairness and adversarial robustness and shows that models that are fairer will be less adversarily robust. However, they do not consider the adversarial robustness of the model on the unseen target domain, which is the topic of this paper.
3 Theoretical Analysis for OOD Adversarial Robustness
In this section, we present two theorems in two different settings, each of which inspires an algorithm designed to improve the OOD adversarial robustness. The proofs of all results in this section can be found in Appendix A. We first introduce some notations and basic setups.
Notations. We define  $[n] \coloneqq \{1, 2, \dots, n\}$ . We denote scalars and vectors with lowercase letters and lowercase bold letters respectively. We use uppercase letters to denote matrices or random variables, and uppercase bold letters to denote random vectors or random matrices. For a vector  $\pmb{x} \in \mathbb{R}^n$ , we define the  $\ell_p$ -norm of  $\pmb{x}$  as  $\| \pmb{x} \|_p \coloneqq \left( \sum_{i=1}^n |x_i|^p \right)^{1/p}$  for  $p \in [1, \infty)$ , where  $x_i$  is the  $i$ -th element of  $\pmb{x}$ ; for  $p = \infty$ , we define  $\| \pmb{x} \|_\infty \coloneqq \max_{1 \leq i \leq n} |x_i|$ . For a matrix  $A \in \mathbb{R}^{m \times n}$ , the Frobenius norm of  $A$
is defined as  $\| A\| _F\coloneqq \left(\sum_{i = 1}^m\sum_{j = 1}^n A_{ij}^2\right)^{\frac{1}{2}}$ , where  $A_{ij}$  is the entry of  $A$  at the  $i$ -th row and  $j$ -th column. We define the determinant of  $A$  as  $\operatorname*{det}(A)$ .  $\mathcal{N}(\pmb {\mu},\Sigma)$  represents the multivariable Gaussian distribution with mean vector  $\pmb{\mu}$  and covariance matrix  $\Sigma$ . Given  $f,g:\mathbb{R}\to \mathbb{R}_+$ , we write  $f = \mathcal{O}(g)$  if there exist  $x_0,\alpha \in \mathbb{R}_+$  such that for all  $x > x_0$ , we have  $f(x)\leq \alpha g(x)$ . We use  $\mathrm{sign}(\cdot)$  to denote the sign function [57].
Setup. Let  $\mathcal{X} \in \mathbb{R}^m$  be the input space and  $\mathcal{Y}$  be the label space. We set  $\mathcal{Y} = \{\pm 1\}$ ,  $\mathcal{Y} = \{1,2,\dots,K\}$  (where  $K$  is the number of classes), and  $\mathcal{Y} = \mathbb{R}$  for the binary classification problem, the multi-class classification problem, and the regression problem, respectively. We use  $\ell: \mathcal{Y} \times \mathcal{Y} \to \mathbb{R}_+$  as the loss function. We consider learning with the hypothesis class  $\mathcal{H} \subseteq \{h: \mathcal{X} \to \mathcal{Y}\}$ . Given a distribution  $\mathcal{D}$  on  $\mathcal{X} \times \mathcal{Y}$ , the error of  $h \in \mathcal{H}$  with respect to the loss function  $\ell$  under the distribution  $\mathcal{D}$  is  $\mathcal{R}_{\mathcal{D}}(\ell,h) = \underset{(\boldsymbol{x},y) \sim \mathcal{D}}{\mathbb{E}}[\ell(h(\boldsymbol{x}),y)]$ , where  $\boldsymbol{x} \in \mathcal{X}$  and  $y \in \mathcal{Y}$ . We further define  $\mathcal{B}(\cdot): \mathcal{X} \to 2^{\mathcal{X}}$
as a perturbation function that maps an input  $\pmb{x}$  to a subset  $\mathcal{B}(\pmb{x})\subseteq \mathcal{X}$ , where  $2^{\mathcal{X}}$  is the power set of  $\mathcal{X}$ . The adversarial error of the predictor  $h$  under the perturbation function  $\mathcal{B}(\cdot)$  is defined as
$$
\mathcal {R} _ {\mathcal {D}} ^ {\mathcal {B}} (\ell , h) = \underset {(\boldsymbol {x}, y) \sim \mathcal {D}} {\mathbb {E}} \left[ \sup  _ {\boldsymbol {x} ^ {\prime} \in \mathcal {B} (\boldsymbol {x})} \ell (h (\boldsymbol {x} ^ {\prime}), y) \right].
$$
3.1 The Average Case
In this section, following [20, 14, 52], we consider the average case, i.e., the case in which the target environment follows a distribution. In this case, we aim to minimize the average target adversarial error of the hypothesis  $h \in \mathcal{H}$ , where the average is taken over the distribution of the target environment.
Suppose that  $\mathcal{P}(\mathcal{X}\times \mathcal{Y})$  is the set of all possible probability measures (environments) on  $\mathcal{X}\times \mathcal{Y}$  for the task of interest. Assume there is a prior  $p$  on  $\mathcal{P}(\mathcal{X}\times \mathcal{Y})$ , which is the distribution of the environments. Moreover, suppose the process of sampling training and test data is as follows:
(1). We generate the training data according to the following two steps: (i) we sample  $t$  training environments from  $p$ , i.e.,  $\mathcal{D}_1, \dots, \mathcal{D}_t \sim p$ ; (ii) the examples  $\widehat{\mathcal{D}}_i = \{(\boldsymbol{x}_{i1}, y_{i1}), \dots, (\boldsymbol{x}_{in_i}, y_{in_i})\} \sim \mathcal{D}_i^{n_i}$  are drawn independent and identically distributed (i.i.d.) from  $\mathcal{D}_i$ , where  $n_i$  is the size of  $\widehat{\mathcal{D}}_i$  and  $i \in [t]$ . To simplify the notations, we also use  $\widehat{\mathcal{D}}_i$  to denote the empirical distribution of the dataset  $\{(x_{i1}, y_{i1}), \dots, (x_{in_i}, y_{in_i})\}$ . Let  $\widehat{\mathcal{D}} = \frac{1}{t} \sum_{i=1}^{t} \widehat{\mathcal{D}}_i$  and  $\hat{p} = \frac{1}{t} \sum_{i=1}^{t} \mathcal{D}_i$ . To simplify the problem, we assume  $n_1 = n_2 = \dots = n_t = n$ , but note that with a more careful analysis, our result in the average case can be extended to the case in which  $n_1, \dots, n_t$  are not necessarily the same.
(2). For each test example, we first sample an environment from  $p$ , i.e.,  $\mathcal{D} \sim p$ , then sample an example  $(\pmb{x}, y) \sim \mathcal{D}$ .
Let  $\mathcal{L}_p(\ell, h) = \underset{\mathcal{D} \sim p}{\mathbb{E}}[\mathcal{R}_{\mathcal{D}}(\ell, h)]$  be the average risk of  $h$  on prior  $p$ . For the perturbation function  $\mathcal{B}(\cdot)$ , we define  $\mathcal{L}_p^{\mathcal{B}}(\ell, h) = \underset{\mathcal{D} \sim p}{\mathbb{E}}[\mathcal{R}_{\mathcal{D}}^{\mathcal{B}}(\ell, h)]$  as the average adversarial risk of  $h$ . The empirical Rademacher complexity [55, Chapter 26] of the hypothesis class  $\mathcal{H}$  is defined as follows:
$$
\mathfrak {R} _ {n} (\mathcal {H}) = \mathbb {E} _ {\boldsymbol {\sigma}} \left[ \sup  _ {h \in \mathcal {H}} \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} h (\boldsymbol {x} _ {i}) \right],
$$
where  $\sigma$  is a Rademacher random vector with i.i.d. entries, and  $\{\pmb{x}_1,\dots ,\pmb{x}_n\}$  is a set of data points. The following theorem presents an upper bound for the average adversarial risk of  $h$ .
Theorem 3.1. Suppose the loss function  $\ell$  is bounded, i.e.,  $\ell \in [0,U]$ . Then with probability of at least  $1 - \delta$  over the sampling of  $\widehat{\mathcal{D}}_1,\dots ,\widehat{\mathcal{D}}_t$ , the following bound holds for all  $h\in \mathcal{H}$ :
$$
\mathcal {L} _ {p} ^ {\mathcal {B}} (\ell , h) \leq \mathcal {L} _ {\widehat {\mathcal {D}}} ^ {\mathcal {B}} (\ell , h) + 2 \Re_ {t} (\widetilde {\mathcal {G}}) + 2 \Re_ {t n} (\widetilde {\mathcal {G}}) + 3 U \sqrt {\frac {\ln 4 / \delta}{2 t}} + 3 U \sqrt {\frac {\ln 4 / \delta}{2 t n}},
$$
$$
\widetilde {\mathcal {G}} = \left\{g _ {h}: \mathcal {X} \times \mathcal {Y} \rightarrow \mathbb {R} _ {+} \Big | g _ {h} (\boldsymbol {x}, y) = \sup  _ {\boldsymbol {x} ^ {\prime} \in \mathcal {B} (\boldsymbol {x})} \ell (h (\boldsymbol {x} ^ {\prime}, y)), h \in \mathcal {H} \right\}.
$$
Theorem 3.1 presents a bound for all hypotheses  $h \in \mathcal{H}$ . We now consider the convergence property of the adversarial empirical risk minimization (AERM) algorithm in the average case. We define the output of the AERM algorithm as  $\hat{h} \in \operatorname*{arginf}_{h \in \mathcal{H}} \mathcal{L}_{\widehat{\mathcal{D}}}^{\mathcal{B}}(\ell, h)$ . We then define the hypothesis with the best adversarial generalization performance as  $h^{\star} \in \operatorname*{arginf}_{h \in \mathcal{H}} \mathcal{L}_p^{\mathcal{B}}(\ell, h)$ .
The next corollary shows an upper bound for the excess risk [63, Chapter 4] of AERM:
Corollary 3.2. Suppose the loss function  $\ell$  is bounded, i.e.,  $\ell \in [0,U]$ . Then with probability of at least  $1 - \delta$  over the sampling of  $\widehat{\mathcal{D}}_1,\dots ,\widehat{\mathcal{D}}_t$ , the following bound holds:
$$
\mathcal {L} _ {p} ^ {\mathcal {B}} (\ell , \hat {h}) \leq \mathcal {L} _ {p} ^ {\mathcal {B}} (\ell , h ^ {\star}) + 4 \Re_ {t} (\widetilde {\mathcal {G}}) + 4 \Re_ {t n} (\widetilde {\mathcal {G}}) + 3 U \sqrt {\frac {\ln 8 / \delta}{2 t}} + 3 U \sqrt {\frac {\ln 8 / \delta}{2 t n}}.
$$
Remark 1. The convergence rate for both Theorem 3.1 and Corollary 3.2 is  $\mathcal{O}\left(t^{-\frac{1}{2}}\right)$ , which prompts us to ask: can we derive a tighter bound that has a faster convergence rate than  $\mathcal{O}\left(t^{-\frac{1}{2}}\right)$ ? The next section provides an affirmative answer to this question.
3.2 Theory with Limited Environments
In this section, we provide the theoretical analysis for the limited training environment case, i.e., in the case in which  $t = \mathcal{O}(1)$ . Suppose that we have  $t$  training environments  $\mathcal{D}_1, \dots, \mathcal{D}_t$  and one unknown target environment  $\mathcal{T}$ . Our goal is to find a predictor that obtains good adversarial robustness on the target environment  $\mathcal{T}$ . Assume we obtain  $n$  examples from each training distribution  $\mathcal{D}_i$ .
First, we introduce a discrepancy between the distributions. Based on the hypothesis class  $\mathcal{H}$ , the seminal work for unsupervised domain adaptation (UDA), [9], defines the discrepancy between two distributions as follows:
$$
d _ {\mathcal {H}} (\mathcal {D}, \mathcal {D} ^ {\prime}) = 2 \sup  _ {h \in \mathcal {H}} \left| \underset {\boldsymbol {x} \sim \mathcal {D}} {\mathbb {P}} [ h (\boldsymbol {x}) = 1 ] - \underset {\boldsymbol {x} \sim \mathcal {D} ^ {\prime}} {\mathbb {P}} [ h (\boldsymbol {x}) = 1 ] \right|.
$$
[9] analyze the generalization error bound of the target domain for UDA by  $d_{\mathcal{H}\Delta \mathcal{H}}(\cdot ,\cdot)$ , where  $g\in \mathcal{H}\Delta \mathcal{H}\Longleftrightarrow g = h\oplus h^{\prime}$  for some  $h,h^{\prime}\in \mathcal{H}$ ; here,  $\oplus$  is the XOR operator. However, the above definition is limited to the binary classification. This paper extends  $d_{\mathcal{H}}$  to the multi-class adversarial case. Given the loss function  $\ell$ , distributions  $P,Q$  on  $\mathcal{X}\times \mathcal{Y}$ , and hypothesis class  $\mathcal{H}$ , we define the adversarial discrepancy as follows:
$$
d_{\ell (\mathcal{H})}^{\mathcal{B}}(P,Q) = \sup_{h\in \mathcal{H}}\bigg|\underset {(\boldsymbol {x},y)\sim P}{\mathbb{E}}\left[ \sup_{\boldsymbol{x}^{\prime}\in \mathcal{B}(\boldsymbol {x})}\ell (h(\boldsymbol{x}^{\prime}),y)\right] - \underset {(\boldsymbol {x},y)\sim Q}{\mathbb{E}}\left[ \sup_{\boldsymbol{x}^{\prime}\in \mathcal{B}(\boldsymbol {x})}\ell (h(\boldsymbol{x}^{\prime}),y)\right]\bigg|.
$$
When  $\mathcal{B}(\pmb{x}) = \{\pmb{x}\}$ ,  $d_{\ell(\mathcal{H})}^{\mathcal{B}}(P, Q)$  becomes the standard case. It can be easily verified that  $d_{\ell(\mathcal{H})}^{\mathcal{B}}(\cdot, \cdot)$  is symmetric and satisfies the triangle inequality (the proof can be found in Appendix A.3); thus,  $d_{\ell(\mathcal{H})}^{\mathcal{B}}(\cdot, \cdot)$  is a pseudometric.
Comparison of  $d_{\mathcal{H}}(\cdot, \cdot)$  and  $d_{\ell(\mathcal{H})}^{\mathcal{B}}(\cdot, \cdot)$ . The theory outlined in [9, Theorem 2] presents an upper bound with a  $d_{\mathcal{H}\Delta \mathcal{H}}(\cdot, \cdot)$  term. To align the feature distributions of the source and target domain, we need to calculate  $d_{\mathcal{H}\Delta \mathcal{H}}(P, Q)$ , which takes the supremum over two hypotheses  $h, h' \in \mathcal{H}$ :
$$
d _ {\mathcal {H} \Delta \mathcal {H}} (P, Q) = 2 \operatorname * {s u p} _ {h, h ^ {\prime} \in \mathcal {H}} \left| \underset {\boldsymbol {x} \sim P} {\mathbb {P}} \left[ h (\boldsymbol {x}) \neq h ^ {\prime} (\boldsymbol {x}) \right] - \underset {\boldsymbol {x} \sim Q} {\mathbb {P}} \left[ h (\boldsymbol {x}) \neq h ^ {\prime} (\boldsymbol {x}) \right] \right|.
$$
However, the definition of  $d_{\ell(\mathcal{H})}^{\mathcal{B}}(\cdot, \cdot)$  shows that:  $d_{\ell(\mathcal{H})}^{\mathcal{B}}(P, Q)$  takes the supremum over one hypothesis  $h \in \mathcal{H}$ , which is easier to optimize [76] and can thus significantly ease the minimax optimization in Section 4.2.
Theorem 3.3. For a given but unknown target distribution  $\mathcal{T}$ , let  $\Delta^{t-1} \coloneqq \{(\lambda_1, \dots, \lambda_t) | \lambda_i \geq 0, \sum_{i=1}^t \lambda_i = 1\}$  be the  $(t-1)$ -dimensional simplex, and  $\operatorname{Conv}(\mathfrak{D}) \coloneqq \left\{\sum_{i=1}^t \lambda_i \mathcal{D}_i \mid \lambda \in \Delta^{t-1}\right\}$  be the convex hull of  $\mathfrak{D} = \{\mathcal{D}_1, \dots, \mathcal{D}_t\}$ . Let  $\mathcal{T}_P \in \underset{\mathcal{D} \in \operatorname{Conv}(\mathfrak{D})}{\arg \inf} d_{\ell(\mathcal{H})}^{\mathcal{B}}(\mathcal{D}, \mathcal{T})$  be the "projection" of  $\mathcal{T}$
onto  $\mathsf{Conv}(\mathfrak{D})$ , and  $\lambda^{\star}$  be the weight vector where  $\mathcal{T}_P = \sum_{i=1}^{t} \lambda_i^{\star} \mathcal{D}_i$ . Assume  $\ell \in [0, U]$ . Then: with probability of at least  $1 - \delta$ , for all  $h \in \mathcal{H}$ ,
$$
\begin{array}{l} \mathcal {R} _ {\mathcal {T}} ^ {\mathcal {B}} (\ell , h) \leq \frac {1}{t} \sum_ {i = 1} ^ {t} \mathcal {R} _ {\widehat {\mathcal {D}} _ {i}} ^ {\mathcal {B}} (\ell , h) + \frac {1}{t} \sum_ {i} \sum_ {j} \lambda_ {j} ^ {\star} d _ {\ell (\mathcal {H})} ^ {\mathcal {B}} (\widehat {\mathcal {D}} _ {i}, \widehat {\mathcal {D}} _ {j}) + d _ {\ell (\mathcal {H})} ^ {\mathcal {B}} (\mathcal {T}, \mathcal {T} _ {P}) \\ + 4 \Re_ {t n} (\widetilde {\mathcal {G}}) + 2 \Re_ {n} (\widetilde {\mathcal {G}}) + 6 U \sqrt {\frac {\ln 8 / \delta}{2 t n}} + 3 U \sqrt {\frac {\ln (1 6 t / \delta)}{2 n}}. \\ \end{array}
$$