SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sloan Nietert
nietert@cs.cornell.edu
Ziv Goldfeld
goldfeld@cornell.edu
Soroosh Shafiee
shafiee@cornell.edu
Distributionally robust optimization (DRO) is an effective approach for data-driven decision-making in the presence of uncertainty. Geometric uncertainty due to sampling or localized perturbations of data points is captured by Wasserstein DRO (WDRO), which seeks to learn a model that performs uniformly well over a Wasserstein ball centered around the observed data distribution. However, WDRO fails to account for non-geometric perturbations such as adversarial outliers, which can greatly distort the Wasserstein distance measurement and impede the learned model. We address this gap by proposing a novel outlier-robust WDRO framework for decision-making under both geometric (Wasserstein) perturbations and non-geometric (total variation (TV)) contamination that allows an  $\varepsilon$ -fraction of data to be arbitrarily corrupted. We design an uncertainty set using a certain robust Wasserstein ball that accounts for both perturbation types and derive minimax optimal excess risk bounds for this procedure that explicitly capture the Wasserstein and TV risks. We prove a strong duality result that enables tractable convex reformulations and efficient computation of our outlier-robust WDRO problem. When the loss function depends only on low-dimensional features of the data, we eliminate certain dimension dependencies from the risk bounds that are unavoidable in the general setting. Finally, we present experiments validating our theory on standard regression and classification tasks.
The safety and effectiveness of various operations rely on making informed, data-driven decisions in uncertain environments. Distributionally robust optimization (DRO) has emerged as a powerful framework for decision-making in the presence of uncertainties. In particular, Wasserstein DRO (WDRO) captures uncertainties of geometric nature, e.g., due to sampling or localized (adversarial) perturbations of the data points. The WdRO problem is a two-player zero-sum game between a learner (decision-maker), who chooses a decision  $\theta \in \Theta$ , and Nature (adversary), who chooses a distribution  $\nu$  from an ambiguity set defined as the  $p$ -Wasserstein ball of a prescribed radius around the observed data distribution  $\tilde{\mu}$ . Namely, WDRO is given by<sup>1</sup>
$$
\inf  _ {\theta \in \Theta} \sup  _ {\nu : \mathsf {W} _ {p} (\nu , \tilde {\mu}) \leq \rho} \mathbb {E} _ {Z \sim \nu} [ \ell (\theta , Z) ], \tag {1}
$$
whose solution  $\hat{\theta} \in \Theta$  is chosen to minimize risk over the Wasserstein ball with respect to (w.r.t.) the loss function  $\ell$ . WDRO has received considerable attention in many fields, including machine learning [6, 22, 45, 48, 59], estimation and filtering [36, 37, 46], and chance constraint programming [12, 55].
In many practical scenarios, the observed data may be contaminated by non-geometric perturbations, such as adversarial outliers. Unfortunately, the WDRO problem from (1) is not suited for handling this
<sup>1</sup>Here,  $\mathsf{W}_p(\mu, \nu) \coloneqq \inf_{\pi \in \Pi(\mu, \nu)} \left( \int \| x - y \|^p d\pi(x, y) \right)^{1/p}$  is the  $p$ -Wasserstein metric between  $\mu$  and  $\nu$ , where  $\Pi(\mu, \nu)$  is the set of all their couplings.
issue, as even a small fraction of outliers can greatly distort the  $W_{p}$  measurement and impede decision-making. In this work, we address this gap by proposing a novel outlier-robust WDRO framework that can learn well-performing decisions even in the presence of outliers. We couple it with a comprehensive theory of excess risk bounds, statistical guarantees, and computationally-tractable reformulations, as well as supporting numerical results.
We consider a scenario where the observed data distribution  $\tilde{\mu}$  is subject to both geometric (Wasserstein) perturbations and non-geometric (total variation (TV)) contamination, which allows an  $\varepsilon$ -fraction of data to be arbitrarily corrupted. Namely, if  $\mu$  is the true (unknown) data distribution, then the Wasserstein perturbation maps it to some  $\mu'$  with  $\mathsf{W}_p(\mu', \mu) \leq \rho$ , and the TV contamination step further produces  $\tilde{\mu}$  with  $\| \tilde{\mu} - \mu' \|_{\mathsf{TV}} \leq \varepsilon$  (e.g., in the special case of the Huber model [28],  $\tilde{\mu} = (1 - \varepsilon)\mu' + \varepsilon \alpha$  where  $\alpha$  is an arbitrary noise distribution). To enable robust decision-making under this model, we replace the Wasserstein ambiguity set in (1) with a ball w.r.t. the recently proposed outlier-robust Wasserstein distance  $\mathsf{W}_p^\varepsilon$  [38, 39]. The  $\mathsf{W}_p^\varepsilon$  distance (see (2) ahead) filters out the  $\varepsilon$ -fraction of mass from the contaminated distribution that contributed most to the transportation cost, and then measures the  $\mathsf{W}_p$  distance
Figure 1: A visual depiction of a clean measure  $\mu \in \mathcal{P}(\mathbb{R})$  and a corrupted observation  $\tilde{\mu}\in \mathcal{P}(\mathbb{R})$  satisfying  $\mathsf{W}_p^\varepsilon (\tilde{\mu},\mu)\leq \rho$
post-filtering. To obtain well-performing solutions for our WDRO problem, the  $\mathsf{W}_p^\varepsilon$  ball is intersected with a set that encodes standard moment assumptions on the uncorrupted data distribution, which are necessary for meaningful outlier-robust estimation guarantees.
We establish the minimax optimal excess risk bounds for the decision  $\hat{\theta}$  that solves the proposed outlier-robust WDRO problem. The bounds control the gap  $\mathbb{E}[\ell (\hat{\theta},Z)] - \mathbb{E}[\ell (\theta_{\star},Z)]$ , where  $Z\sim \mu$  follows the true data distribution and  $\theta_{\star} = \mathrm{argmin}_{\theta}\mathbb{E}[\ell (\theta ,Z)]$  is the optimal decision, subject to regularity properties of  $\ell_{\star} = \ell (\theta_{\star},\cdot)$ . In turn, our bounds imply that the learner can make effective decisions using outlier-robust WDRO based on the contaminated observation  $\tilde{\mu}$ , so long as  $\ell_{\star}$  has low variational complexity. The bounds capture this complexity using the Lipschitz or Sobolev seminorms of  $\ell_{\star}$  and clarify the distinct effect of each perturbation (Wasserstein versus TV) on the quality of the learned  $\hat{\theta}$  solution. We further establish their minimax optimality when  $p = 1$ , by providing a matching lower bound in the setting when an adversary picks a class of Lipschitz functions over which the learner must perform uniformly well. The excess risk bounds become looser as the data dimension  $d$  grows. We show that this degradation is alleviated when the loss function depends on the data only through  $k$ -dimensional affine features, by providing risk bounds that adapt to  $k$  instead of  $d$ .
We then move to study the computational side of the problem, which may initially appear intractable due to non-convexity of the constraint set. We resolve this via a cheap preprocessing step that computes a coarse robust estimate of the mean [34] and replaces the original constraint set (that involves the true mean) with a version centered around the estimate. We adapt our excess risk bounds to this formulation and then prove a strong duality theorem. The dual form is reminiscent of the one for classical WDRO with adaptations reflecting the constraint to the clean distribution family and the partial transportation under  $\mathbb{W}_p^\varepsilon$ . Under additional convexity conditions on the loss, we further derive an efficiently-computable, finite-dimensional, convex reformulation. The optimization results are also adapted to the setting with low-dimensional features. Using the developed machinery, we present experiments that validate our theory on simple regression/classification tasks and demonstrate the superiority of the proposed approach over classical WRDO, when the observed data is contaminated.
Distributionally robust optimization. The Wasserstein distance has emerged as a powerful tool for modeling uncertainty in the data generating distribution. It was first used to construct an ambiguity set around the empirical distribution in [40]. Recent advancements in convex reformulations and
approximations of the WDRO problem, as discussed in [8, 20, 35], have brought notable computational advantages. Additionally, WDRO is linked to various forms of variation [2, 9, 19, 43] and Lipschitz [7, 11, 44] regularization, which contribute to its success in practice. Robust generalization guarantees can also be provided by WDRO via measure concentration argument or transportation inequalities [18, 30, 31, 51, 53, 54]. Several works have raised concerns regarding the sensitivity of standard DRO to outliers [24, 27, 58]. An attempt to address this was proposed in [56] using a refined risk function based on a family of  $f$ -divergences. This formulation aims to prevent DRO from overfitting to potential outliers but is not robust to geometric perturbations. Further, their risk bounds require a moment condition to hold uniformly over  $\Theta$ , in contrast to our bounds that depend only on  $\theta_{\star}$ . We are able to address these limitations by setting a WDRO framework based on partial transportation. While partial OT has been previously used in the context of DRO problems, it was introduced to address stochastic programs with side information in [17] rather than to account for outlier robustness. Another closely related line of work is presented in [4, 5], where the ambiguity set is constructed using an  $f$ -divergence to mitigate statistical errors and the Prokhorov distance to handle outlier data. The proposed model is both computationally efficient and statistically reliable. However, they have not investigated its minimax optimality or robustness against the Huber contamination model, which we aim to do in this paper. Additionally, a best-case favorable analysis approach has been proposed in [29] to address outlier data. This approach is an alternative to the worst-case distributionally robust method. However, it requires solving a non-convex optimization problem, significantly impacting its scalability, and is not accompanied by any proof of minimax optimality.
Robust statistics. The problem of learning from data under TV  $\varepsilon$ -corruptions dates back to [28]. Over the years, various robust and sample-efficient estimators, particularly for mean and scale parameters, have been developed in the robust statistics community; see [41] for a comprehensive survey. The theoretical computer science community, on the other hand, has focused on developing computationally efficient estimators that achieve optimal estimation rates in high dimensions [13, 16]. Relatedly, the probably approximate correct (PAC) learning framework has been well-studied in similar models [1, 10]. Recently, [58] developed a unified robust estimation framework based on minimum distance estimation that gives sharp population-limit and promising finite-sample guarantees for mean and covariance estimation, as well linear regression. Their analysis centers on a generalized resilience quantity, which is essential to our work. We are unaware of any results in the settings above which extend to combined TV and  $W_{p}$  corruptions. Finally, our analysis relies on the outlier-robust Wasserstein distance from [38, 39], which was shown to yield an optimal minimum distance estimate for robust distribution estimation under  $W_{p}$  loss.
Notation. Consider a closed, non-empty set  $\mathcal{Z} \subseteq \mathbb{R}^d$  equipped with the Euclidean norm  $\|\cdot\|$ . A continuously differentiable function  $f: \mathcal{Z} \to \mathbb{R}$  is called  $\alpha$ -smooth if  $\|\nabla f(z) - \nabla f(z')\| \leq \alpha \|z - z'\|$ , for all  $z, z' \in \mathcal{Z}$ . The perspective function of a lower semi-continuous (l.s.c.) and convex function  $f$  is  $P_f(x, \lambda) \coloneqq \lambda f(x / \lambda)$  for  $\lambda > 0$ , with  $P_f(x, \lambda) = \lim_{\lambda \to 0} \lambda f(x / \lambda)$  when  $\lambda = 0$ . The convex conjugate of  $f$  is  $f^*(y) \coloneqq \sup_{x \in \mathbb{R}^d} y^\top x - f(x)$ . We denote by  $\chi_{\mathcal{Z}}$  the indicator function of  $\mathcal{Z}$ , that is,  $\chi_{\mathcal{Z}}(z) = 0$  if  $z \in \mathcal{Z}$  and  $\chi_{\mathcal{Z}}(z) = \infty$  otherwise. The convex conjugate of  $\chi_{\mathcal{Z}}$ , denoted by  $\chi_{\mathcal{Z}}^*$ , is termed as the support function of  $\mathcal{Z}$ .
We use  $\mathcal{M}(\mathcal{Z})$  for the set of signed Radon measures on  $\mathcal{Z}$  equipped with the TV norm  $\| \mu \|_{\mathrm{TV}} \coloneqq \frac{1}{2} |\mu|(\mathcal{Z})$ , and write  $\mu \leq \nu$  for set-wise inequality. The class of Borel probability measures on  $\mathcal{Z}$  is denoted by  $\mathcal{P}(\mathcal{Z})$ . We write  $\mathbb{E}_{\mu}[f(Z)]$  for expectation of  $f(Z)$  with  $Z \sim \mu$ ; when clear from the context, the random variable is dropped and we write  $\mathbb{E}_{\mu}[f]$ . Let  $\Sigma_{\mu}$  denote the covariance matrix of  $\mu \in \mathcal{P}_2(\mathcal{Z})$ . Define  $\mathcal{P}_p(\mathcal{Z}) \coloneqq \{\mu \in \mathcal{P}(\mathcal{Z}) : \inf_{z_0 \in \mathcal{Z}} \mathbb{E}_{\mu}[\| Z - z_0\|^p] < \infty\}$ . The push-forward of  $f$  through  $\mu \in \mathcal{P}(\mathcal{Z})$  is  $f_{\#}\mu(\cdot) \coloneqq \mu(f^{-1}(\cdot))$ , and, for  $\mathcal{A} \subseteq \mathcal{P}(\mathcal{Z})$ , write  $f_{\#}\mathcal{A} \coloneqq \{f_{\#}\mu : \mu \in \mathcal{A}\}$ . The  $p$ th order homogeneous Sobolev (semi)norm of continuously differentiable  $f: \mathcal{Z} \to \mathbb{R}$  w.r.t.  $\mu$  is  $\| f\|_{\dot{H}^{1,p}(\mu)} \coloneqq \mathbb{E}_{\mu}[\|\nabla f\|^p]^{1/p}$ . The set of integers up to  $n \in \mathbb{N}$  is denoted by  $[n]$ ; we also use the shorthand  $[x]_{+} = \max\{x, 0\}$ . We write  $\lesssim, \gtrsim, \asymp$  for inequalities/equality up to absolute constants.
Classical and outlier-robust Wasserstein distances. For  $p \in [1, \infty)$ , the  $p$ -Wasserstein distance between  $\mu, \nu \in \mathcal{P}_p(\mathcal{Z})$  is  $\mathsf{W}_p(\mu, \nu) := \inf_{\pi \in \Pi(\mu, \nu)} \left( \mathbb{E}_{\pi} \left[ \| X - Y \|^p \right] \right)^{1/p}$ , where  $\Pi(\mu, \nu) := \{ \pi \in \mathcal{P}(\mathcal{Z}^2) : \pi(\cdot \times \mathcal{Z}) = \mu, \pi(\mathcal{Z} \times \cdot) = \nu \}$  is the set of all their couplings. Some basic properties of
$\mathsf{W}_p$  are (see, e.g., [42, 52]): (i)  $\mathsf{W}_p$  is a metric on  $\mathcal{P}_p(\mathcal{Z})$ ; (ii) the distance is monotone in the order, i.e.,  $\mathsf{W}_p \leq \mathsf{W}_q$  for  $p \leq q$ ; and (iii)  $\mathsf{W}_p$  metrizes weak convergence plus convergence of  $p$ th moments:  $\mathsf{W}_p(\mu_n, \mu) \to 0$  if and only if  $\mu_n \stackrel{w}{\to} \mu$  and  $\int \|x\|^p d\mu_n(x) \to \int \|x\|^p d\mu(x)$ .
To handle corrupted data, we employ the  $\varepsilon$ -outlier-robust  $p$ -Wasserstein distance<sup>2</sup>, defined by
$$
\mathsf {W} _ {p} ^ {\varepsilon} (\mu , \nu) := \inf  _ { \begin{array}{c} \mu^ {\prime} \in \mathcal {P} \left(\mathbb {R} ^ {d}\right) \\ \| \mu^ {\prime} - \mu \| _ {\mathsf {T V}} \leq \varepsilon \end{array} } \mathsf {W} _ {p} (\mu^ {\prime}, \nu) = \inf  _ { \begin{array}{c} \nu^ {\prime} \in \mathcal {P} \left(\mathbb {R} ^ {d}\right) \\ \| \nu^ {\prime} - \nu \| _ {\mathsf {T V}} \leq \varepsilon \end{array} } \mathsf {W} _ {p} (\mu , \nu^ {\prime}). \tag {2}
$$
The second equality is a useful consequence of Lemma 4 in [39] (see Appendix A for details, along with an interpretation of  $\mathsf{W}_p^\varepsilon$  as a partial OT distance).
Robust statistics. Resilience is a standard sufficient condition for population-limit robust statistics bounds [49, 58]. The  $p$ -Wasserstein resilience of a measure  $\mu \in \mathcal{P}(\mathcal{Z})$  is defined by
$$
\tau_{p}(\mu ,\varepsilon):= \sup_{\mu^{\prime}\leq \frac{1}{1 - \varepsilon}\mu}\sup_{\mu^{\prime}\leq \frac{1}{1 - \varepsilon}\mu}\mathsf{W}_{p}(\mu^{\prime},\mu),
$$
and that of a family  $\mathcal{G} \subseteq \mathcal{P}(\mathbb{R})$  by  $\tau_p(\mathcal{G}, \varepsilon) \coloneqq \sup_{\mu \in \mathcal{G}} \tau_p(\mu, \varepsilon)$ . The relation between  $\mathsf{W}_p$  resilience and robust estimation is formalized in the following proposition.
Proposition 1 (Robust estimation under  $\mathsf{W}_p$  resilience [39]). Fix  $0\leq \varepsilon \leq 0.49$ . For any clean distribution  $\mu \in \mathcal{G}\subseteq \mathcal{P}(\mathcal{Z})$  and corrupted measure  $\tilde{\mu}\in \mathcal{P}(\mathcal{Z})$  such that  $\mathsf{W}_p^\varepsilon (\tilde{\mu},\mu)\leq \rho$ , the minimum distance estimate  $\hat{\mu} = \mathrm{argmin}_{\nu \in \mathcal{G}}\mathsf{W}_p^\varepsilon (\nu ,\tilde{\mu})$  satisfies  $\mathsf{W}_p(\hat{\mu},\mu)\lesssim \rho +\tau_p(\mathcal{G},2\varepsilon)$ .<sup>3</sup>
Throughout, we focus on the bounded covariance class  $\mathcal{G}_{\mathrm{cov}}\coloneqq \{\mu \in \mathcal{P}(\mathcal{Z}):\Sigma_{\mu}\preceq I_d\}$
Proposition 2 ( $W_p$  resilience bound for  $\mathcal{G}_{\mathrm{cov}}$  [39]). Fixing  $0 \leq \varepsilon \leq 0.99$  and  $1 \leq p \leq 2$ , we have  $\tau_p(\mathcal{G}_{\mathrm{cov}}, \varepsilon) \lesssim \sqrt{d} \varepsilon^{1/p-1/2}$ .
3 Outlier-robust WDRO
We perform stochastic optimization with respect to an unknown data distribution  $\mu$ , given access only to a corrupted version  $\tilde{\mu}$ . We allow both localized Wasserstein perturbations, that map  $\mu$  to some  $\mu'$  with  $W_{p}(\mu, \mu') \leq \rho$ , and TV  $\varepsilon$ -contamination that takes  $\mu'$  to  $\tilde{\mu}$  with  $\| \tilde{\mu} - \mu' \|_{\mathrm{TV}} \leq \varepsilon$ . Equivalently, both perturbations are captured by  $W_{p}^{\varepsilon}(\tilde{\mu}, \mu) \leq \rho$ . To simplify notation, we henceforth suppress the dependence of the loss function  $\ell$  on the model parameters  $\theta \in \Theta$ , writing  $\ell$  for  $\ell(\theta, \cdot)$  for a specific function and  $\mathcal{L} = \{\ell(\theta, \cdot)\}_{\theta \in \Theta}$  for the whole class. Our full model is as follows.
Setting A: Fix a  $p$ -Wasserstein radius  $\rho \geq 0$  and TV contamination level  $\varepsilon \in [0,0.49]$ . Let  $\mathcal{L} \subseteq \mathbb{R}^{\mathcal{Z}}$  be a family of real-valued loss functions on  $\mathcal{Z}$ , such that each  $\ell \in \mathcal{L}$  is l.s.c. with  $\sup_{z \in \mathcal{Z}} \frac{\ell(z)}{1 + \|z\|^p} < \infty$ , and fix a class  $\mathcal{G} \subseteq \mathcal{P}_p(\mathcal{Z})$  encoding distributional assumptions. We consider the following model:
(i) Nature selects a distribution  $\mu \in \mathcal{G}$ , unknown to the learner;
(ii) The learner observes a corrupted measure  $\tilde{\mu} \in \mathcal{P}(\mathcal{Z})$  such that  $\mathsf{W}_p^\varepsilon (\tilde{\mu},\mu)\leq \rho$
(iii) The learner selects a decision  $\hat{\ell} \in \mathcal{L}$  and suffers excess risk  $\mathbb{E}_{\mu}[\hat{\ell}] - \inf_{\ell \in \mathcal{L}} \mathbb{E}_{\mu}[\ell]$ .
We seek a decision-making procedure for the learner which provides strong excess risk guarantees when  $\ell_{\star} = \operatorname*{argmin}_{\ell \in \mathcal{L}}\mathbb{E}_{\mu}[\ell ]$  is appropriately "simple." To achieve this, we introduce the  $\varepsilon$ -outlier-robust  $p$ -Wasserstein DRO problem:
$$
\inf  _ {\ell \in \mathcal {L}} \sup  _ {\nu \in \mathcal {G}:   \mathsf {W} _ {p} ^ {\varepsilon} (\tilde {\mu}, \nu) \leq \rho} \mathbb {E} _ {\nu} [ \ell ]. \quad \text {(O R - W D R O)}
$$
3.1 Excess Risk Bounds
We quantify the excess risk of decisions made using OR-WDRO for the two most popular choices of order,  $p = 1,2$ . Proofs are provided in Supplement B.
2While not a metric,  $W_{p}^{\varepsilon}$  is symmetric and satisfies an approximate triangle inequality ([39], Proposition 3).
If a minimizer does not exist for either problem, an infinizing sequence will achieve the same guarantee.
We defer explicit modeling of sampling to Section 3.2 but note that the following results immediately transfer to the  $n$ -sample setting so long as  $\rho$  is taken to be larger than  $\mathsf{W}_p(\hat{\mu}_n,\mu)$  with high probability.
Theorem 1 (OR-WDRO risk bound). Under Setting  $A$ , let  $\hat{\ell}$  minimize (OR-WDRO). Then, writing  $c = 2(1 - \varepsilon)^{-1/p}$ , the excess risk is bounded by
$$
\mathbb{E}_{\mu}[\hat{\ell} ] - \mathbb{E}_{\mu}[\ell_{\star}]\\ \leq \left\{ \begin{array}{ll}\| \ell_{\star}\|_{\text{Lip}}\bigl(c\rho +2\tau_{1}(\mathcal{G},2\varepsilon)\bigr), & p = 1,\ell_{\star}Lipschitz\\ \| \ell_{\star}\|_{\dot{H}^{1,2}(\mu)}\bigl(c\rho +2\tau_{2}(\mathcal{G},2\varepsilon)\bigr) + \frac{1}{2}\alpha \bigl(c\rho +2\tau_{2}(\mathcal{G},2\varepsilon)\bigr)^{2}, & p = 2,\ell_{\star}\text{$\alpha$ - smooth} \end{array} \right..
$$
Note that  $c = O(1)$  since  $\varepsilon \leq 0.49$ . These bounds imply that the learner can make effective decisions when  $\ell_{\star}$  has low variational complexity<sup>5</sup>. In contrast, there are simple regression settings with TV corruption that drive the excess risk of standard WDRO to infinity. Our proof derives both results as a special case of a general bound in terms of the  $W_{p}$  regularizer, defined by  $\mathcal{R}_{\mu,p}(\rho; \ell) := \sup_{\nu' \in \mathcal{P}(\mathcal{Z}): W_{p}(\nu', \nu) \leq \rho} \mathbb{E}_{\nu'}[\ell] - \mathbb{E}_{\nu}[\ell]$ . Introduced in [18], this quantity appears implicitly throughout the WDRO literature. In particular, for each  $\ell \in \mathcal{L}$ , we derive the following bound:
$$
\mathbb {E} _ {\mu} [ \hat {\ell} ] - \mathbb {E} _ {\mu} [ \ell ] \leq \mathcal {R} _ {\mu , p} \left(\underbrace {c \rho} _ {\mathrm {W} _ {p}} + \underbrace {2 \tau_ {p} (\mathcal {G} , 2 \varepsilon)} _ {\mathrm {T V}}; \ell\right), \tag {3}
$$
whose radius reveals the effect of each perturbation (viz. Wasserstein versus TV) on the quality of the decision. The first bound of the theorem follows by plugging in  $p = 1$  and controlling  $\mathcal{R}_{\mu,1}$  via Kantorovich duality. The second bound uses  $p = 2$  and controls  $\mathcal{R}_{\mu,2}$  by replacing  $\ell$  with its Taylor expansion about  $Z \sim \mu$ . We now instantiate Theorem 1 for the bounded covariance class  $\mathcal{G}_{\mathrm{cov}}$ .
Corollary 1 (Risk bounds for  $\mathcal{G}_{\mathrm{cov}}$ ). Under the setting of Theorem 1 with  $\mathcal{G} \subseteq \mathcal{G}_{\mathrm{cov}}$ , we have
$$
\mathbb {E} _ {\mu} [ \hat {\ell} ] - \mathbb {E} _ {\mu} [ \ell_ {\star} ] \lesssim \left\{ \begin{array}{l l} \| \ell_ {\star} \| _ {\operatorname {L i p}} \big (\rho + \sqrt {d \varepsilon} \big), & p = 1, \ell_ {\star}   L i p s c h i t z \\ \| \ell_ {\star} \| _ {\dot {H} ^ {1, 2} (\mu)} (\rho + \sqrt {d}) + \alpha (\rho^ {2} + d), & p = 2, \ell_ {\star}   \alpha \text {- s m o o t h} \end{array} \right..
$$
Since  $\mathcal{G}_{\mathrm{cov}}$  encodes second moment constraints,  $\tau_{2}(\mathcal{G}_{\mathrm{cov}},\varepsilon)\asymp d$  is independent of  $\varepsilon$ . Therefore, the first bound is preferable as  $\varepsilon \to 0$  if  $\| \ell_{\star}\|_{\dot{H}^{1,2}(\mu)}\approx \| \ell_{\star}\|_{\mathrm{Lip}}$ , while the second is better when  $\varepsilon = \Omega (1)$  and  $\| \ell_{\star}\|_{\dot{H}^{1,2}(\mu)}\ll \| \ell_{\star}\|_{\mathrm{Lip}}$ . Distinct trade-offs are observed under stronger tail bounds like sub-Gaussianity, i.e., for  $\mathcal{G}_{\mathrm{subG}}\coloneqq \{\mu \in \mathcal{P}(\mathcal{Z}):\mathbb{E}_{\mu}[e^{(\theta^{\top}(Z - \mathbb{E}[Z])^{2}}]\leq 2,\forall \theta \in \mathbb{S}^{d - 1}\}$ .
Corollary 2 (Risk bounds for  $\mathcal{G}_{\mathrm{subG}}$ ). Under the setting of Theorem 1 with  $\mathcal{G} \subseteq \mathcal{G}_{\mathrm{subG}}$ , the excess risk  $\mathbb{E}_{\mu}[\hat{\ell}] - \mathbb{E}_{\mu}[\ell_{\star}]$  is bounded up to constants by