SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} _ {\mu} [ \hat {\ell} ] - \mathbb {E} _ {\mu} [ \ell_ {\star} ] \lesssim \left\{ \begin{array}{l l} \| \ell_ {\star} \| _ {\mathrm {L i p}} \big (\sqrt {k} \rho_ {0} + \sqrt {k \varepsilon} + \delta \big), & p = 1, \ell_ {\star}   L i p s c h i t z \\ \| \ell_ {\star} \| _ {\dot {H} ^ {1, 2} (\mu)} \big (\sqrt {k} \rho_ {0} + \sqrt {k} + \delta \big) + \alpha \big (\sqrt {k} \rho_ {0} + \sqrt {k} + \delta \big) ^ {2}, & p = 2, \ell_ {\star}   \alpha - s m o o t h \end{array} \right..
$$
Here, the stronger requirement for the robust mean estimate, the restriction to additive contamination, and the need to optimize over the centered  $\mathcal{G}_{\mathrm{cov}}$  class rather than  $\mathcal{G}_2$  all stem from the fact that the resilience term  $\tau_p((\mathcal{G}_{\mathrm{cov}})_k,\varepsilon)$  scales with  $\sqrt{k}$  rather than  $\sqrt{d}$ . Fortunately, efficient computation is still possible. First, we employ iterative filtering [15] for dimension-free robust mean estimation.
Proposition 9 (Refined robust mean estimation). Consider Setting  $B$  or  $B'$  with  $\mathcal{G} = \mathcal{G}_{\mathrm{cov}}$  and  $\varepsilon \leq 1/12$ . For  $n = \tilde{\Omega}(d)$ , there exists an iterative filtering algorithm which takes  $\tilde{\mu}_n$  as input, runs in time  $\tilde{O}(nd^2)$ , and outputs  $z_0 \in \mathbb{R}^d$  such that  $\|z_0 - \mathbb{E}_\mu[Z]\| \lesssim \rho_0 + 1$  with probability at least 0.99.
The analysis requires care when  $p = 1$ , since  $W_{1}$  perturbations can arbitrarily increase the initial covariance bound. Fortunately, this increase can be controlled by trimming out a few samples.
Next, we show that computing the inner worst-case expectation in (7) can be simplified into a minimization problem involving only a scalar and a positive semidefinite matrix provided the following assumption holds (which is indeed the case in the setting of Proposition 8).
Assumption 3 (Slater condition II). Given the distribution  $\tilde{\mu}_n$  and fixed point  $z_0$ , there exists  $\nu_0 \in \mathcal{P}(\mathcal{Z})$  such that  $\mathsf{W}_p^\varepsilon (\tilde{\mu}_n\| \nu_0) < \rho$  and  $\mathbb{E}_{\nu_0}[(Z - z_0)(Z - z_0)^{\dagger}] \prec \sigma^2 I_d$ . Further, we require  $\rho > 0$ .
Proposition 10 (Strong duality). Under Assumption 3, for any  $\ell \in \mathcal{L}$  and  $z_0\in \mathbb{R}^d$  , we have
$$
\sup_{\substack{\nu \in \mathcal{G}_{\mathrm{cov}}(\sigma ,z_{0}):\\ \mathsf{W}_{p}^{\varepsilon}(\tilde{\mu}_{n}\| \nu)\leq \rho}}\mathbb{E}_{\nu}[\ell ] = \inf_{\substack{\Lambda_{1}\in \mathbb{Q}_{+}^{d}\\ \lambda_{2}\in \mathbb{R}_{+},\alpha \in \mathbb{R}}} - z_{0}^{\top}\Lambda_{1}z_{0} + \sigma^{2}\operatorname {Tr}[\Lambda_{1}] + \lambda_{2}\rho^{p} + \alpha +\frac{1}{1 - \varepsilon}\mathbb{E}_{\tilde{\mu}_{n}}\left[\overline{\ell} (\cdot ;\Lambda_{1},\lambda_{2},\alpha)\right],
$$
where  $\bar{\ell}(z; \Lambda_1, \lambda_2, \alpha) \coloneqq \sup_{\xi \in \mathcal{Z}} [\ell(\xi) - \xi^\top \Lambda_1\xi + 2\xi^\top \Lambda_1z_0 - \lambda_2\|\xi - z\|^{p} - \alpha]_{+}$ .
The minimization problem over the variables  $(\Lambda_1, \lambda_2, \alpha)$  belongs to the class of stochastic convex optimization problems. As before, we show that under the convexity condition from Assumption 2 we obtain a tractable reformulation that does not involve an extra optimization problem for evaluating  $\bar{\ell}$ .
Theorem 5 (Convex reformulation). Under Assumption 3, for any  $\ell \in \mathcal{L}$  satisfying Assumption 2 and  $z_0\in \mathbb{R}^d$ , we have
$$
\sup_{\substack{\nu \in \mathcal{G}_{\mathrm{cov}}(\sigma ,z_{0}):\\ \mathsf{W}_{p}^{\varepsilon}(\tilde{\mu}_{n}\| \nu)\leq \rho}}\mathbb{E}_{\nu}[\ell ] = \left\{ \begin{array}{ll}\inf \quad -z_{0}^{\top}\Lambda_{1}z_{0} + \sigma^{2}\operatorname {Tr}[\Lambda_{1}] + \lambda_{2}\rho^{p} + \alpha +\frac{1}{n(1 - \varepsilon)}\sum_{i\in [n]}s_{i}\\ \text{s.t.}\quad \alpha \in \mathbb{R},  \Lambda_{1}\in \mathbb{Q}_{+}^{d},  \lambda_{2}\in \mathbb{R}_{+},  s\in \mathbb{R}_{+}^{n},\\ \tau_{ij}\in \mathbb{R}_{+},  \zeta_{ij}^{\ell},\zeta_{ij}^{\mathcal{G}},\zeta_{ij}^{\mathsf{W}},\zeta_{ij}^{\mathcal{Z}}\in \mathbb{R}^{d}, & \forall i\in [n],\forall j\in [J]\\ s_{i}\geq (-\ell_{j})^{*}(\zeta_{ij}^{\ell}) + \tau_{ij} + \tilde{Z}_{i}^{\top}\zeta_{ij}^{\mathsf{W}}\\ \hskip 142.26378pt + P_{h}(\zeta_{ij}^{\mathsf{W}},\lambda_{2}) + \chi_{\mathcal{Z}}^{*}(\zeta_{ij}^{\mathcal{Z}}) - \alpha , & \forall i\in [n],\forall j\in [J]\\ \zeta_{ij}^{\ell} + \zeta_{ij}^{\mathcal{G}} + \zeta_{ij}^{\mathsf{W}} + \zeta_{ij}^{\mathcal{Z}} = 2\Lambda_{1}z_{0},  (\zeta_{ij}^{\mathcal{G}})^{\top}\Lambda_{1}^{-1}\zeta_{ij}^{\mathcal{G}}\leq 4\tau_{ij}, & \forall i\in [n],\forall j\in [J], \end{array} \right.
$$
where  $P_{h}$  is the perspective function of  $h$  defined in (6).
Lastly, we implement our tractable reformulations and validate their excess risk bounds. Fixing  $\mathcal{Z} = \mathcal{X}\times \mathcal{Y} = \mathbb{R}^{d - 1}\times \mathbb{R}$ , we focus on linear regression with the mean absolute deviation loss, i.e.,  $\mathcal{L} = \{\ell_{\theta}(x,y) = |\theta^{\top}x - y|:\theta \in \mathbb{R}^{d}\}$ . The experiments below were run in 80 minutes on an M1 MacBook Air with 16GB RAM. See Supplement G for full details and additional experiments treating classification and multivariate regression. Code is available at https://github.com/sbnietert/outlier-robust-WDRO.
Fix  $\mathcal{Z} = \mathbb{R}^d$  for  $d\geq 2$ $\rho = \varepsilon = 0.1$  and  $\theta_{\star}\in$ $\mathbb{S}^{d - 2}$  . Taking  $X\sim \mathcal{N}(0,I_{d - 1})$  , we fix clean data  $(X,\theta_{\star}^{\top}X)\sim \mu$  and corrupted data  $(\tilde{X},\tilde{Y})\sim \tilde{\mu}$  such that  $(\tilde{X},\tilde{Y}) = (X,\theta_{\star}^{\top}X + \rho)$  with probability  $1 - \varepsilon$  and  $(\tilde{X},\tilde{Y}) = (CX, - C^{2}\theta_{\star}^{\top}X)$  with probability  $\varepsilon$  , so that  $\mathsf{W}_p^{\varepsilon_0}(\tilde{\mu}\| \mu)\leq \rho$  . In Figure 2 (top), we fix  $d = 10,C = 8$  and compare the excess risk  $\mathbb{E}_{\mu}[\ell_{\hat{\theta}}] - \mathbb{E}_{\mu}[\ell_{\theta_{\star}}]$  of standard WDRO and outlierrobust WDRO with  $\mathcal{G} = \mathcal{G}_2$  , as described by Proposition 4 and implemented via Theorem 2. The results are averaged over  $T = 20$  runs for sample size  $n\in \{10,20,50,75,100\}$  . We run outlier-robust WDRO with corruption fraction  $\hat{\varepsilon}\in \{0,\varepsilon ,2\varepsilon \}$  ,achieving low excess risk when  $\hat{\varepsilon}\geq \varepsilon$  as predicted. In Figure 2 (bottom), to highlight the Section 4 improve
ments due to low-dimensional structure, we fix  $n = 20$ ,  $C = 100$  and compare the excess risk of outlier-robust WDRO with  $\mathcal{G} = \mathcal{G}_2$  to that with  $\mathcal{G} = \mathcal{G}_{\mathrm{cov}}$ , as described by Proposition 8 and implemented via Theorem 5. We average over  $T = 10$  runs and present results for dimension  $d \in \{5, 10, 25, 40\}$ . Confidence bands in both plots depict the top and bottom  $10\%$  quantiles among 100 bootstrapped means from the  $T$  runs. Implementations were performed in MATLAB using the YALMIP toolbox [33] and the Gurobi and SeDuMi solvers [23, 50].
Figure 2: Excess risk of standard WDRO and several forms of outlier-robust WDRO for linear regression under  $W_{p}$  and TV corruptions, with varied sample size and dimension.
In this work, we have introduced a novel framework for outlier-robust WDRO that allows for both geometric and non-geometric perturbations of the data distribution, as captured by  $\mathsf{W}_p$  and TV, respectively. We provided minimax optimal excess risk bounds and strong duality results, with the latter enabling efficient computation via convex reformulations. There are numerous directions for future work, including refined statistical guarantees for  $\rho \ll \rho_0 + n^{-1 / d}$  and convex reformulations for distribution families beyond  $\mathcal{G}_{\mathrm{cov}}$ . Overall, our approach enables principled, data-driven decision-making in realistic scenarios where observations may be subject to adversarial contamination.
[43] S. Shafieezadeh-Abadeh, L. Aolaritei, F. Dörfler, and D. Kuhn. New perspectives on regularization and computation in optimal transport-based distributionally robust optimization. arXiv preprint arXiv:2303.03900, 2023.
[44] S. Shafieezadeh-Abadeh, D. Kuhn, and P. Mohajerin Esfahani. Regularization via mass transportation. Journal of Machine Learning Research, 20(103):1-68, 2019.
[45] S. Shafieezadeh-Abadeh, P. Mohajerin Esfahani, and D. Kuhn. Distributionally robust logistic regression. In Advances in Neural Information Processing Systems, 2015.
[46] S. Shafieezadeh-Abadeh, V. A. Nguyen, D. Kuhn, and P. Mohajerin Esfahani. Wasserstein distributionally robust Kalman filtering. In Advances in Neural Information Processing Systems, 2018.
[47] A. Shapiro. On duality theory of conic linear problems. In M. Á. Goberna and M. A. López, editors, Semi-Infinite Programming, pages 135-165. Kluwer Academic Publishers, 2001.
[48] A. Sinha, H. Namkoong, and J. Duchi. Certifying some distributional robustness with principled adversarial training. In International Conference on Learning Representations, 2018.
[49] J. Steinhardt, M. Charikar, and G. Valiant. Resilience: A criterion for learning in the presence of arbitrary outliers. In Innovations in Theoretical Computer Science Conference, 2018.
[50] J. F. Sturm. Using SeDuMi 1.02, A MATLAB toolbox for optimization over symmetric cones. Optimization Methods and Software, 11(1-4):625-653, 1999.
[51] Z. Tu, J. Zhang, and D. Tao. Theoretical analysis of adversarial learning: A minimax approach. In Advances in Neural Information Processing Systems, 2019.
[52] C. Villani. Topics in Optimal Transportation. American Mathematical Society, 2003.
[53] R. Volpi, H. Namkoong, O. Sener, J. Duchi, V. Murino, and S. Savarese. Generalizing to unseen domains via adversarial data augmentation. In Advances in Neural Information Processing Systems, pages 5339-5349, 2018.
[54] Y. Wang, X. Ma, J. Bailey, J. Yi, B. Zhou, and Q. Gu. On the convergence and robustness of adversarial training. In International Conference on Machine Learning, 2019.
[55] W. Xie. On distributionally robust chance constrained programs with Wasserstein distance. Mathematical Programming, 186(1):115-155, 2021.
[56] R. Zhai, C. Dan, Z. Kolter, and P. Ravikumar. DORO: Distributional and outlier robust optimization. In International Conference on Machine Learning, 2021.
[57] J. Zhen, D. Kuhn, and W. Wiesemann. A unified theory of robust and distributionally robust optimization via the primal-worst-equals-dual-best principle. Operations Research (Forthcoming), 2023.
[58] B. Zhu, J. Jiao, and J. Steinhardt. Generalized resilience and robust statistics. The Annals of Statistics, 50(4):2256 - 2283, 2022.
[59] S. Zhu, L. Xie, M. Zhang, R. Gao, and Y. Xie. Distributionally robust weighted k-nearest neighbors. In Advances in Neural Information Processing Systems, 2022.
A Preliminary Results on Robust OT and Wasserstein DRO
We first recall properties of the robust Wasserstein distance  $\mathsf{W}_p^\varepsilon$  which will be used throughout the supplement. To start, we show that our definition coincides with another based on partial OT, considered in [39]. In what follows, we fix  $p\geq 1$ , write  $c\mathcal{P}(\mathcal{Z})\coloneqq \{c\mu :\mu \in \mathcal{P}(\mathcal{Z})\}$ , and, for  $\mu ,\nu \in c\mathcal{P}(\mathcal{Z})$ , we define  $\Pi (\mu ,\nu)\coloneqq c\Pi (\mu /c,\nu /c)$  and  $\mathsf{W}_p(\mu ,\nu)^p\coloneqq c\mathsf{W}_p(\mu /c,\nu /c)$ .
Lemma 1  $(\mathsf{W}_p^\varepsilon$  as partial OT). Fix  $\varepsilon \in [0,1]$ . For any  $\mu, \nu \in \mathcal{P}(\mathcal{Z})$ , we have
$$
\mathsf{W}_{p}^{\varepsilon}(\mu ,\nu):= \inf_{\substack{\mu^{\prime}\in \mathcal{P}(\mathcal{Z})\\ \| \mu^{\prime} - \mu \|_{\mathsf{TV}}\leq \varepsilon}}\mathsf{W}_{p}(\mu^{\prime},\nu) = \inf_{\substack{\nu^{\prime}\in \mathcal{P}(\mathcal{Z})\\ \| \nu^{\prime} - \nu \|_{\mathsf{TV}}\leq \varepsilon}}\mathsf{W}_{p}(\mu ,\nu^{\prime}) = \inf_{\substack{\mu^{\prime},\nu^{\prime}\in (1 - \varepsilon)\mathcal{P}(\mathcal{Z})\\ \mu^{\prime}\leq \mu ,\nu^{\prime}\leq \nu}}\mathsf{W}_{p}(\mu^{\prime},\nu^{\prime}).
$$
Proof. We write  $\underline{\mathsf{W}}_p^\varepsilon (\mu ,\nu)$  for the rightmost expression; this is definition of  $\mathsf{W}_p^\varepsilon$  considered in [39]. We first show that  $\underline{\mathsf{W}}_p^\varepsilon (\mu ,\nu)\leq \mathsf{W}_p^\varepsilon (\mu ,\nu)$ . Fix any  $\mu^{\prime}$  feasible for the  $\mathsf{W}_p^\varepsilon$  problem. Then, by the approximate triangle inequality for  $\underline{\mathsf{W}}_p^\varepsilon$  (Proposition 3 of [39]), we have
$$
\underline {{\mathsf {W}}} _ {p} ^ {\varepsilon} (\mu , \nu) \leq \underline {{\mathsf {W}}} _ {p} ^ {\varepsilon} (\mu , \mu^ {\prime}) + \mathsf {W} _ {p} (\mu^ {\prime}, \nu) \leq \mathsf {W} _ {p} (\mu^ {\prime}, \nu).
$$
Indeed, writing  $c \coloneqq (\mu \wedge \mu')(\mathcal{Z}) \geq 1 - \varepsilon$ , the last inequality uses that  $\underline{\mathsf{W}}_p^\varepsilon(\mu, \mu') \leq \mathsf{W}_p\left(\frac{1 - \varepsilon}{c} \mu \wedge \mu', \frac{1 - \varepsilon}{c} \mu \wedge \mu'\right) = 0$ . Infimizing over feasible  $\mu'$  gives that  $\underline{\mathsf{W}}_p^\varepsilon(\mu, \nu) \leq \mathsf{W}_p^\varepsilon(\mu, \nu)$ .
For the other direction, take any  $\mu', \nu'$  feasible for the  $\underline{\mathsf{W}}_p^\varepsilon$  problem. Let  $\pi \in \Pi(\mu', \nu')$  be any optimal coupling for the  $\mathsf{W}_p(\mu', \nu')$  problem, and write  $\mu'' = \mu' + (\nu - \nu') \in \mathcal{P}(\mathcal{Z})$ . Defining the coupling  $\pi' = \pi + (\mathrm{Id}, \mathrm{Id})_{\#} (\nu - \nu') \in \Pi(\mu'', \nu)$ , we compute
$$
\mathsf {W} _ {p} (\mu^ {\prime \prime}, \nu) ^ {p} \leq \int_ {\mathcal {Z} \times \mathcal {Z}} \| x - y \| ^ {p} \mathrm {d} \pi^ {\prime} (x, y) = \int_ {\mathcal {Z} \times \mathcal {Z}} \| x - y \| ^ {p} \mathrm {d} \pi (x, y) = \mathsf {W} _ {p} (\mu^ {\prime}, \nu^ {\prime}).
$$
By construction,  $\| \mu^{\prime \prime} - \mu \|_{\mathsf{TV}}\leq \varepsilon$  , and so  $\mathsf{W}_p^\varepsilon (\mu ,\nu)\leq \mathsf{W}_p(\mu ',\nu ')$  . Infimizing over feasible  $\mu^{\prime},\nu^{\prime}$  gives that  $\mathsf{W}_p^\varepsilon (\mu ,\nu)\leq \underline{\mathsf{W}}_p^\varepsilon (\mu ,\nu)$
We thus inherit several results for  $\underline{\mathsf{W}}_p$  given in [39].
Lemma 2 (Approximate triangle inequality [39]). If  $\mu, \nu, \kappa \in \mathcal{P}(\mathcal{Z})$  and  $\varepsilon_1, \varepsilon_2 \in [0,1]$ , then
$$
\mathsf {W} _ {p} ^ {\varepsilon_ {1} + \varepsilon_ {2}} (\mu , \nu) \leq \mathsf {W} _ {p} ^ {\varepsilon_ {1}} (\mu , \kappa) + \mathsf {W} _ {p} ^ {\varepsilon_ {2}} (\kappa , \nu).
$$
Lemma 3  $(\mathsf{W}_p^\varepsilon$  modulus of continuity, [39], Lemma 3). For any  $\mathcal{G} \subseteq \mathcal{P}(\mathcal{Z})$ , we have
$$
\sup_{\substack{\alpha ,\beta \in \mathcal{G}\\ \mathsf{W}_{p}^{\varepsilon}(\alpha ,\beta)\leq \rho}}\mathsf{W}_{p}(\alpha ,\beta)\leq (1 - \varepsilon)^{-1 / p}\rho +2\tau_{p}(\mathcal{G},\varepsilon).
$$
Lemma 4 (One-sided vs. two-sided  $\mathsf{W}_p^\varepsilon$ ). For  $\mu, \nu \in \mathcal{P}(\mathcal{Z})$ , we have
$$
\mathsf {W} _ {p} ^ {\varepsilon} (\mu \| \nu) \leq (1 - \varepsilon) ^ {- 1 / p} \mathsf {W} _ {p} ^ {\varepsilon} (\mu , \nu) + \tau_ {p} (\nu , \varepsilon).
$$
Proof. Fix any  $\mu', \nu' \in (1 - \varepsilon)\mathcal{P}(\mathcal{Z})$  with  $\mu' \leq \mu$  and  $\nu' \leq \nu$ . By design, we have
$$
\begin{array}{l} \mathsf {W} _ {p} ^ {\varepsilon} (\mu \| \nu) \leq \mathsf {W} _ {p} \left(\frac {1}{1 - \varepsilon} \mu^ {\prime}, \nu\right) \\ \leq \mathsf {W} _ {p} \left(\frac {1}{1 - \varepsilon} \mu^ {\prime}, \frac {1}{1 - \varepsilon} \nu^ {\prime}\right) + \mathsf {W} _ {p} \left(\frac {1}{1 - \varepsilon} \nu^ {\prime}, \nu\right) \\ = (1 - \varepsilon) ^ {- 1 / p} \mathsf {W} _ {p} (\mu^ {\prime}, \nu^ {\prime}) + \tau_ {p} (\nu , \varepsilon) \\ \end{array}
$$
Infimizing over  $\mu^{\prime}$  and  $\nu^{\prime}$  and applying Lemma 1 gives the lemma.
Next, we specify explicit constants for the  $W_{p}$  resilience of  $\mathcal{G}_{\mathrm{cov}}$ . Our analysis goes through the related notion of mean resilience [49], defined by  $\tau (\mu ,\varepsilon) = \sup_{\mu^{\prime}\in \mathcal{P}(\mathcal{Z}):\mu^{\prime}\leq \frac{1}{1 - \varepsilon}\mu}\| \mathbb{E}_{\mu^{\prime}}[Z] - \mathbb{E}_{\mu}[Z]\|$ . We say that  $Z\sim \mu \in \mathcal{P}(\mathcal{Z})$  is  $(\tau_0,\varepsilon)$ -resilient in mean or under  $W_{p}$  if  $\mu \in \tau (\tau_0,\varepsilon)$  or  $\mu \in \tau_p(\tau_0,\varepsilon)$ .
Lemma 5 ( $W_p$  resilience for  $\mathcal{G}_2$  and  $\mathcal{G}_{\mathrm{cov}}$ ). Fix  $\varepsilon \in [0,1)$  and  $\sigma \geq 0$ . For  $1 \leq p \leq 2$ , we have  $\tau_p(\mathcal{G}_2(\sigma), \varepsilon) \leq 4\sigma \varepsilon^{1/p-1/2}(1 - \varepsilon)^{-1/p}$ . Moreover, we have  $\tau_p(\mathcal{G}_{\mathrm{cov}}(\sigma), \varepsilon) \leq \tau_p(\mathcal{G}_2(\sqrt{d}\sigma), \varepsilon)$ .
Proof. Fix any  $Z \sim \mu \in \mathcal{G}_2(\sigma, z_0)$ . By definition, we have  $\mathbb{E}[(\| Z - z_0\|^p)^{2/p}] = \mathbb{E}[\| Z - z_0\|^2] \leq \sigma^2 = (\sigma^p)^{2/p}$ . Thus, standard bounds (e.g., Lemma E.2 of [58]) give that  $\| Z - z_0\|^p$  is  $(\sigma^p \varepsilon^{1-p/2}(1-\varepsilon)^{-1}, \varepsilon)$ -resilient in mean. By Lemma 7 of [39], we thus have that  $Z$  is  $(2\sigma \varepsilon^{1/p-1/2}(1-\varepsilon)^{-1/p} + 2\varepsilon^{1/p}\sigma, \varepsilon)$ -resilient under  $\mathsf{W}_p$ . This gives the first result. For the second, we observe that for  $Z \sim \mu \in \mathcal{G}_{\mathrm{cov}}(\sigma)$ , we have  $\mathbb{E}[\| Z - \mathbb{E}[Z]\|^2] = \operatorname{tr}(\Sigma_\mu) \leq \sqrt{d}\sigma$ .
Lastly, we turn to the  $\mathsf{W}_p$  regularizer.
Lemma 6 (Controlling  $W_{p}$  regularizer, [18], Lemmas 1 and 2). For any  $\ell \in \mathcal{L}$ , we have  $\mathcal{R}_{\nu,1}(\rho; \ell) \leq \rho \| \ell \|_{\mathrm{Lip}}$ , with equality if  $\ell$  is convex. For  $\alpha$ -smooth  $\ell$ , we have  $|\mathcal{R}_{\nu,2}(\rho; \ell) - \rho \| \ell \|_{\dot{H}^{1,2}(\nu)}| \leq \frac{1}{2}\alpha\rho^2$ .
The factor of  $1/2$  under smoothness was not present in Lemma 2 of [18], since the proof only used that  $|\ell(z) - \ell(z_0) - \nabla(z_0)^\top(z - z_0)| \leq \alpha \|z - z_0\|^2$ , instead of the tight upper bound of  $\frac{\alpha}{2}\|z - z_0\|^2$ .
This quantity naturally bounds the excess risk of standard WDRO.
Lemma 7 (WDRO excess risk bound). Under Setting  $A$  with  $\varepsilon = 0$ , the standard WDRO estimate  $\hat{\ell} = \operatorname{argmin}_{\ell \in \mathcal{L}} \sup_{\nu \in \mathcal{P}(\mathcal{Z}): \mathsf{W}_p(\tilde{\mu}, \nu) \leq \rho} \mathbb{E}_{\nu}[\ell]$  satisfies  $\mathbb{E}_{\mu}[\hat{\ell}] - \mathbb{E}_{\mu}[\ell_\star] \leq \mathcal{R}_{\mu, p}(2\rho; \ell_\star)$ .
Proof. We bound
$$
\begin{array}{l} \mathbb{E}_{\mu}[\hat{\ell} ] - \mathbb{E}_{\mu}[\ell_{\star}]\leq \sup_{\substack{\nu \in \mathcal{P}(\mathcal{Z})\\ \mathsf{W}_{p}(\nu ,\tilde{\mu})\leq \rho}}\mathbb{E}_{\nu}[\hat{\ell} ] - \mathbb{E}_{\mu}[\ell_{\star}]\\ (\mathsf{W}_{p}(\mu ,\tilde{\mu})\leq \rho) \\ \leq \sup_{\substack{\nu \in \mathcal{P}(\mathcal{Z})\\ \mathsf{W}_{p}(\nu ,\tilde{\mu})\leq \rho}}\mathbb{E}_{\nu}[\ell_{\star}] - \mathbb{E}_{\mu}[\ell_{\star}] \qquad \qquad (\text{optimality of}\hat{\ell}) \\ \leq \sup_{\substack{\nu \in \mathcal{P}(\mathcal{Z})\\ \mathsf{W}_{p}(\nu ,\mu)\leq 2\rho}}\mathbb{E}_{\nu}[\ell_{\star}] - \mathbb{E}_{\mu}[\ell_{\star}] \qquad \qquad (\mathsf{W}_{p}\text{triangle inequality}) \\ = \mathcal {R} _ {\mu , p} (2 \rho ; \ell_ {\star}), \\ \end{array}
$$
Note that this bound does not incorporate the distributional assumptions encoded by  $\mathcal{G}$ .
$$
\begin{array}{l} \mathbb{E}_{\mu}[\hat{\ell} ] - \mathbb{E}_{\mu}[\ell ]\leq \sup_{\substack{\nu \in \mathcal{G}\\ \mathsf{W}_{p}^{\varepsilon}(\tilde{\mu},\nu)\leq \rho}}\mathbb{E}_{\nu}[\hat{\ell} ] - \mathbb{E}_{\mu}[\ell ]\qquad \qquad (\mu \in \mathcal{G},\mathsf{W}_{p}^{\varepsilon}(\tilde{\mu},\mu)\leq \rho) \\ \leq \sup_{\substack{\nu \in \mathcal{G}\\ \mathsf{W}_{p}^{\varepsilon}(\tilde{\mu},\nu)\leq \rho}}\mathbb{E}_{\nu}[\ell ] - \mathbb{E}_{\mu}[\ell ]\qquad \qquad (\hat{\ell}\text{optimal for (OR-WDRO)}) \\ \leq \sup  _ {\substack {\nu \in \mathcal {G} \\ \mathsf {W} _ {p} ^ {2 \varepsilon} (\nu , \mu) \leq 2 \rho}} \mathbb {E} _ {\nu} [ \ell ] - \mathbb {E} _ {\mu} [ \ell ] (Lemma 2) \\ \leq \sup  _ {\substack {\nu \in \mathcal {G} \\ W _ {p} (\nu , \mu) \leq c \rho + 2 \tau_ {p} (\mathcal {G}, 2 \varepsilon)}} \mathbb {E} _ {\nu} [ \ell ] - \mathbb {E} _ {\mu} [ \ell ] (Lemma3) \\ \leq \sup_{\substack{\nu \in \mathcal{P}(\mathcal{Z})\\ \mathsf{W}_{p}(\nu ,\mu)\leq c\rho +2\tau_{p}(\mathcal{G},2\varepsilon)}}\mathbb{E}_{\nu}[\ell ] - \mathbb{E}_{\mu}[\ell ]\qquad \qquad (\mathcal{G}\subseteq \mathcal{P}(\mathcal{Z})) \\ = \mathcal {R} _ {\mu , p} \big (c \rho + 2 \tau_ {p} (\mathcal {G}, 2 \varepsilon); \ell \big). \\ \end{array}
$$
Combining this bound with Lemma 6 gives the theorem.
The corollary follows as an immediate consequence of Theorem 1 and the resilience bounds of Proposition 2.
B.3 Proof of Corollary 2
The corollary follows as an immediate consequence of Theorem 1 and the resilience bound  $\tau_{p}(\mathcal{G}_{\mathrm{subG}},\varepsilon)\lesssim \sqrt{d + p + \log\frac{1}{\varepsilon}}\varepsilon^{1 / p}$  established in [39, Theorem 2].
B.4 Proof of Proposition 3
For ease of presentation, suppose  $d = 2m$  is even. Consider  $\mathbb{R}^d$  as  $\mathbb{R}^m \times \mathbb{R}^m$ , fix  $w \in \mathbb{R}^m$  with  $\| w \| = \rho$ , and let  $\mathcal{L}$  consist of the following loss functions:
$$
\begin{array}{l} \ell_ {+, 0} (x, y) := L \| x + y \| \\ \ell_ {-, 0} (x, y) := L \| x - y \| \\ \ell_ {+, 1} (x, y) := L \| x + y - w \| \\ \ell_ {-, 1} (x, y) := L \| x - y + w \|. \\ \end{array}
$$
Fixing corrupted measure  $\tilde{\mu} = \delta_0$ , we consider the following candidates for the clean measure  $\mu$ :
$$
\begin{array}{l} \mu_ {+, 0} := (1 - \varepsilon) \delta_ {0} + \varepsilon (\operatorname {I d}, - \operatorname {I d}) _ {\#} \mathcal {N} (0, I _ {d} / \varepsilon) \\ \mu_ {-, 0} := (1 - \varepsilon) \delta_ {0} + \varepsilon (\operatorname {I d}, + \operatorname {I d}) _ {\#} \mathcal {N} (0, I _ {d} / \varepsilon) \\ \mu_ {+, 1} := (1 - \varepsilon) \delta_ {(0, w)} + \varepsilon (\operatorname {I d}, - \operatorname {I d} + w) _ {\#} \mathcal {N} (0, I _ {d} / \varepsilon) \\ \mu_ {-, 1} := (1 - \varepsilon) \delta_ {(0, w)} + \varepsilon (\operatorname {I d}, \operatorname {I d} + w)) _ {\#} \mathcal {N} (0, I _ {d} / \varepsilon), \\ \end{array}
$$
where  $\operatorname{Id}: x \mapsto x$  is the identity map. By design,  $\mathsf{W}_1^\varepsilon (\tilde{\mu}\| \mu_{+,0})$ ,  $\mathsf{W}_1^\varepsilon (\tilde{\mu}\| \mu_{-,0})$ ,  $\mathsf{W}_1^\varepsilon (\tilde{\mu}\| \mu_{+,1})$ , and  $\mathsf{W}_1^\varepsilon (\tilde{\mu}\| \mu_{-,1})$  are all at most  $\rho$  and  $\mu_{+,0}, \mu_{-,0}, \mu_{+,1}, \mu_{-,1} \in \mathcal{G}_{\mathrm{cov}}$ . Moreover,
$$
\begin{array}{l} \mathbb {E} _ {\mu_ {+, 0}} [ \ell_ {+, 0} ] = \mathbb {E} _ {\mu_ {-, 0}} [ \ell_ {-, 0} ] = \mathbb {E} _ {\mu_ {+, 1}} [ (\ell_ {+, 1}) = \mathbb {E} _ {\mu_ {-, 1}} [ \ell_ {-, 1} ] = 0 \\ \mathbb {E} _ {\mu_ {+, 0}} [ \ell_ {-, 0} ] = \mathbb {E} _ {\mu_ {-, 0}} [ \ell_ {+, 0} ] = \mathbb {E} _ {\mu_ {+, 1}} [ \ell_ {-, 1} ] = \mathbb {E} _ {\mu_ {-, 1}} [ \ell_ {+, 1} ] = 2 L \varepsilon \mathbb {E} _ {Z \sim \mathcal {N} (0, I _ {d / \varepsilon})} [ \| Z \| ] \gtrsim L \sqrt {d \varepsilon} \\ \mathbb {E} _ {\mu_ {+, 0}} [ \ell_ {+, 1} ] = \mathbb {E} _ {\mu_ {+, 1}} [ \ell_ {+, 0} ] = \mathbb {E} _ {\mu_ {-, 0}} [ \ell_ {-, 1} ] = \mathbb {E} _ {\mu_ {-, 1}} [ \ell_ {-, 0} ] = L \| w \| = L \rho . \\ \end{array}
$$
Thus, for any  $\hat{\ell} = \mathsf{D}(\tilde{\mu})\in \mathcal{L}$ , there exists  $\mu \in \{\mu_{+,0},\mu_{-,0},\mu_{+,1},\mu_{-,1}\}$  such that