SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\{ \begin{array}{l l} \| \ell_ {\star} \| _ {\mathrm {L i p}} \Big (\rho + \sqrt {d + \log \frac {1}{\varepsilon}}   \varepsilon \Big), & p = 1, \ell_ {\star}   L i p s c h i t z \\ \| \ell_ {\star} \| _ {\dot {H} ^ {1, 2} (\mu)} \Big (\rho + \sqrt {(d + \log \frac {1}{\varepsilon}) \varepsilon} \Big) + \alpha \Big (\rho^ {2} + \big (d + \log \frac {1}{\varepsilon} \big)   \varepsilon \Big), & p = 2, \ell_ {\star}   \alpha - s m o o t h \end{array} \right..
$$
Remark 1 (Comparison to MDE under  $\mathsf{W}_p^\varepsilon$ ). We note that the excess risk  $\mathcal{R}_{\mu,p}\big(c\rho + 2\tau_p(\mathcal{G},2\varepsilon); \ell_\star \big)$  from (3) can alternatively be obtained by performing standard  $p$ -WDRO with an expanded radius  $c\rho + 2\tau_1(\mathcal{G},2\varepsilon)$  around the minimum distance estimate  $\hat{\mu} = \operatorname*{argmin}_{\nu \in \mathcal{G}} \mathsf{W}_1^\varepsilon (\hat{\mu},\nu)$ . However, obtaining  $\hat{\mu}$  is an expensive preprocessing step, and we are unaware of any efficient algorithms for such MDE in the finite-sample setting. In Supplement D, we explore recentering WDRO around a tractable estimate obtained from iterative filtering [15], but find the resulting risk to be highly suboptimal. Furthermore, the improvements to our risk bounds under low-dimensional structure, which are derived in Section 4, do not extend to decisions obtained from these alternative procedures.
We now show that Theorem 1 cannot be improved in general. In particular, the first bound is minimax optimal over Lipschitz loss families when  $\mu \in \mathcal{G}_{\mathrm{cov}}$ .
Proposition 3 (Lower bound). Fix  $\mathcal{Z} = \mathbb{R}^d$  and  $\varepsilon \in [0,0.49]$ . For any  $L \geq 0$ , there exists a family  $\mathcal{L} \subseteq \mathrm{Lip}_L(\mathbb{R}^d)$ , independent of  $\varepsilon$ , such that for any decision rule  $\mathsf{D} : \mathcal{P}(\mathcal{Z}) \to \mathcal{L}$  there exists a pair  $(\mu, \tilde{\mu}) \in \mathcal{G}_{\mathrm{cov}} \times \mathcal{P}(\mathcal{Z})$  with  $\mathsf{W}_1^\varepsilon(\tilde{\mu}, \mu) \leq \rho$  satisfying  $\mathbb{E}_{\mu}[\mathsf{D}(\tilde{\mu})] - \inf_{\ell \in \mathcal{L}} \mathbb{E}_{\mu}[\ell] \gtrsim L\big(\rho + \sqrt{d\varepsilon}\big)$ .
Each family  $\mathcal{L}$  encodes a multivariate regression problem. Our proof combines a one-dimensional lower bound of [49] for linear regression with lower bounds of [39] for robust estimation under  $\mathrm{W}_1$ .
The same bounds hold up to  $\varepsilon$  additive slack if  $\ell_{\star}$  is only  $\varepsilon$ -approximately optimal for (OR-WDRO).
Under  $\mathsf{W}_p^\varepsilon$  perturbations, one may perform outlier-robust WDRO using any  $p' \in [1,p]$ , which may be advantageous in terms of the TV component of the excess risk.
3.2 Statistical Guarantees
We next formalize a finite-sample model and adapt our excess risk bounds to it.
Setting B: Fix  $\rho, \varepsilon, \mathcal{L}, \mathcal{G}$  as in Setting A, and let  $Z_{1}, \ldots, Z_{n}$  be identically and independently distributed (i.i.d.) according to  $\mu \in \mathcal{G}$ , with empirical measure  $\hat{\mu}_{n} = \frac{1}{n} \sum_{i=1}^{n} \delta_{Z_{i}}$ . Upon observing these clean samples, Nature applies a  $\mathsf{W}_{p}$  perturbation of size  $\rho_{0}$ , producing  $\{Z_{i}'\}_{i=1}^{n}$  with empirical measure  $\mu_{n}'$  such that  $\mathsf{W}_{p}(\hat{\mu}_{n}, \mu_{n}') \leq \rho_{0}$ . Finally, Nature corrupts up to  $\lfloor \varepsilon n \rfloor$  samples to obtain  $\{\tilde{Z}_{i}\}_{i=1}^{n}$  with empirical measure  $\tilde{\mu}_{n}$  such that  $\| \tilde{\mu}_{n} - \mu' \|_{\mathsf{TV}} = \frac{1}{n} \sum_{i=1}^{n} \mathbb{1} \{\tilde{Z}_{i} \neq Z_{i}\} \leq \varepsilon$ . Equivalently, the final dataset satisfies  $\mathsf{W}_{p}^{\varepsilon}(\tilde{\mu}_{n}, \hat{\mu}_{n}) \leq \rho_{0}$ . The learner is now tasked with selecting  $\hat{\ell} \in \mathcal{L}$  given  $\tilde{\mu}_{n}$ .
The results from Section 3 apply whenever  $\rho \geq \rho_0 + \mathsf{W}_p(\mu ,\hat{\mu}_n)$ . In particular, we obtain the following corollary as an immediate consequence of Theorem 1 and Theorem 3.1 of [32].
Corollary 3 (Finite-sample risk bounds). Under Setting  $B$ , fix  $\hat{\ell} \in \mathcal{L}$  minimizing (OR-WDRO) centered at  $\tilde{\mu} = \tilde{\mu}_n$  with  $\rho \geq \rho_0 + 100\mathbb{E}[\mathsf{W}_p(\mu ,\hat{\mu}_n)]$ . Then the excess risk bounds of Theorem 1 hold with probability at least 0.99. If  $\mathcal{G} \in \{\mathcal{G}_{\mathrm{cov}},\mathcal{G}_{\mathrm{subG}}\}$ ,  $p = 1$ , and  $d \geq 3$ , or if  $\mathcal{G} = \mathcal{G}_{\mathrm{subG}}$ ,  $p = 2$ , and  $d \geq 5$ , then  $\mathbb{E}[\mathsf{W}_p(\mu ,\hat{\mu}_n)] \lesssim \sqrt{dn}^{-1 / d}$ .
Remark 2 (Smaller radius). In the classic WDRO setting with  $\rho_0 = \varepsilon = 0$ , the radius  $\rho$  can be taken significantly smaller than  $n^{-1 / d}$  if  $\mathcal{L}$  and  $\mu$  are sufficiently well-behaved. For example, [18] proves that  $\rho = \widetilde{O} (n^{-1 / 2})$  gives meaningful risk bounds when  $\mu$  satisfies a  $T_{2}$  transportation inequality. While this high-level condition may be hard to verify in practice, Supplement E shows that this improvement can be lifted to an instance of our outlier-robust WDRO problem.
3.3 Tractable Reformulations and Computation
For computation, we restrict to  $\mu \in \mathcal{G}_{\mathrm{cov}}$ . Initially, (OR-WDRO) may appear intractable, since  $\mathcal{G}_{\mathrm{cov}}$  is non-convex when viewed as a subset of the cone  $\mathcal{M}_{+}(\mathcal{Z})$ . Moreover, enforcing membership to this class is non-trivial. To remedy these issues, we use a cheap preprocessing step to obtain a robust estimate  $z_0 \in \mathcal{Z}$  of the mean  $\mathbb{E}_{\mu}[Z]$ , and we optimize over the modified class  $\mathcal{G}_2(\sigma, z_0) := \{\nu \in \mathcal{P}(\mathcal{Z}) : \mathbb{E}_{\nu}[\|Z - z_0\|^2] \leq \sigma^2\}$ , with  $\sigma \gtrsim \|z_0 - \mathbb{E}_{\mu}[Z]\| + \sqrt{d}$  taken so that  $\mu \in \mathcal{G}_2(\sigma, z_0)$ . Finally, for technical reasons, we switch to the one-sided robust distance  $\mathsf{W}_p^\varepsilon(\mu|\nu) := \inf_{\mu' \in \mathcal{P}(\mathbb{R}^d): \mu' \leq \frac{1}{1-\varepsilon}\mu} \mathsf{W}_p(\mu', \nu)$ . Altogether, we arrive at the modified DRO problem
$$
\inf  _ {\ell \in \mathcal {L}} \sup  _ {\nu \in \mathcal {G} _ {2} (\sigma , z _ {0}): \mathbb {W} _ {p} ^ {\varepsilon} (\tilde {\mu} _ {n} \| \nu) \leq \rho} \mathbb {E} _ {\nu} [ \ell ], \tag {4}
$$
which, as stated next, admits risk bounds matching Corollary 1 up to empirical approximation error.
Proposition 4 (Risk bound for modified problem). Consider Setting  $B$  with  $\mathcal{G} \subseteq \mathcal{G}_{\mathrm{cov}}$ . Fix  $z_0 \in \mathcal{Z}$  such that  $\| z_0 - \mathbb{E}_\mu[Z]\| \leq E = O(\rho_0 + \sqrt{d})$ , and suppose that  $\mathsf{W}_p(\hat{\mu}_n, \mu) \leq \delta$ . Take  $\hat{\ell}$  minimizing (4) with  $\rho = (\rho_0 + \delta)(1 - \varepsilon)^{-1/p} + \tau_p(\mathcal{G}_{\mathrm{cov}}, \varepsilon)$  and  $\sigma = \sqrt{d} + E$ . We then have
$$
\mathbb {E} _ {\mu} [ \hat {\ell} ] - \mathbb {E} _ {\mu} [ \ell_ {\star} ] \lesssim \left\{ \begin{array}{l l} \| \ell_ {\star} \| _ {\mathrm {L i p}} \big (\rho_ {0} + \sqrt {d \varepsilon} + \delta \big), & p = 1, \ell_ {\star} L i p s c h i t z \\ \| \ell_ {\star} \| _ {\dot {H} ^ {1, 2} (\mu)} \big (\rho_ {0} + \sqrt {d} + \delta \big) + \alpha \big (\rho_ {0} + \sqrt {d} + \delta \big) ^ {2}, & p = 2, \ell_ {\star} \alpha - s m o o t h \end{array} \right..
$$
Parameters  $\rho, \sigma$  are taken so that  $\mu \in \mathcal{G}_2(\sigma, z_0)$  and  $\mathsf{W}_p^\varepsilon(\tilde{\mu}_n \| \mu) \leq \rho$ . Noting this, the proof mirrors that of Theorem 1, using a  $\mathsf{W}_p$  resilience bound for  $\mathcal{G}_2(\sigma, z_0)$ . To ensure  $\mathsf{W}_p(\hat{\mu}_n, \mu) \leq \delta$  with decent probability, one should take  $\delta$  to be an upper bound on  $\sup_{\nu \in \mathcal{G}} \mathbb{E}[\mathsf{W}_p(\hat{\nu}_n, \nu)]$ . When  $p = 2$ , this quantity is only finite if  $\mathcal{Z}$  is bounded or if  $\mathcal{G}$  encodes stronger tail bounds than  $\mathcal{G}_{\mathrm{cov}}$  (see, e.g., [32]).
For efficient computation, we must specify a robust mean estimation algorithm to obtain  $z_0$  and a procedure for solving (4). The former is achieved by taking a coordinate-wise trimmed mean.
Proposition 5 (Coarse robust mean estimation). Consider Setting  $B$  with  $\mathcal{G} \subseteq \mathcal{G}_{\mathrm{cov}}$  and  $\varepsilon \leq 1/3$ . For  $n = \Omega(\log(d))$ , there is a trimmed mean procedure, which applied coordinate-wise to  $\{\tilde{Z}_i\}_{i=1}^n$ , returns  $z_0 \in \mathbb{R}^d$  with  $\|z_0 - \mathbb{E}_\mu[Z]\| \lesssim \sqrt{d} + \rho_0$  with probability at least 0.99, in time  $\tilde{O}(d)$ .
<sup>7</sup>In general,  $\{\tilde{Z}_i\}_{i=1}^n$  may be any measurable function of  $\{Z_i\}_{i=1}^n$  and independent randomness such that  $\mathsf{W}_p^\varepsilon(\tilde{\mu}_n, \hat{\mu}_n) \leq \rho_0$ .
<sup>8</sup>We say that  $\mu \in T_2(\tau)$  if  $\mathsf{W}_2(\nu, \mu) \leq \sqrt{\tau \mathsf{H}(\nu \| \mu)}$ , for all  $\nu \in \mathcal{P}_2(\mathcal{Z})$ , where  $\mathsf{H}(\nu \| \mu)$  is relative entropy.
More sophisticated methods, e.g., iterative filtering [15], achieve dimension-free estimation guarantees at the cost of additional sample and computational complexity. We will return to these techniques in Section 4, but overlook them for now since they do not impact worst-case excess risk bounds.
We next show that the inner maximization problem of (4) can be simplified to a minimization problem involving only two scalars provided the following assumption holds.
Assumption 1 (Slater condition I). Given the distribution  $\tilde{\mu}_n$  and the fixed point  $z_0$ , there exists  $\nu_0 \in \mathcal{P}(\mathcal{Z})$  such that  $\mathsf{W}_p^\varepsilon (\tilde{\mu}_n\| \nu_0) < \rho$  and  $\mathbb{E}\nu_0[\| Z - z_0\|^2 ] < \sigma^2$ . Additionally, we require  $\rho > 0$ .
Notice that Assumption 1 indeed holds for  $\nu_0 = \mu$  as applied in Proposition 4.
Proposition 6 (Strong duality). Under Assumption 1, for any  $\ell \in \mathcal{L}$  and  $z_0\in \mathbb{R}^d$  , we have
$$
\sup  _ {\substack {\nu \in \mathcal {G} _ {2} (\sigma , z _ {0}): \\ \mathbb {W} _ {p} ^ {\varepsilon} (\tilde {\mu} _ {n} \| \nu) \leq \rho}} \mathbb {E} _ {\nu} [ \ell ] = \inf  _ {\substack {\lambda_ {1}, \lambda_ {2} \in \mathbb {R} _ {+} \\ \alpha \in \mathbb {R}}} \lambda_ {1} \sigma^ {2} + \lambda_ {2} \rho^ {p} + \alpha + \frac {1}{1 - \varepsilon} \mathbb {E} _ {\tilde {\mu} _ {n}} \left[ \bar {\ell} (\cdot ; \lambda_ {1}, \lambda_ {2}, \alpha) \right], \tag{5}
$$
where  $\overline{\ell}(z; \lambda_1, \lambda_2, \alpha) \coloneqq \sup_{\xi \in \mathcal{Z}} \left[ \ell(\xi) - \lambda_1 \| \xi - z_0 \|^2 - \lambda_2 \| \xi - z \|^p - \alpha \right]_+$ .
The minimization problem over  $(\lambda_1, \lambda_2, \alpha)$  is an instance of stochastic convex optimization, where the expectation of the implicit function  $\overline{\ell}$  is taken w.r.t. the contaminated empirical measure  $\tilde{\mu}_n$ . In contrast, the dual reformulation for classical WDRO only involves  $\lambda_2$  and takes the expectation of the implicit function  $\underline{\ell}(z; \lambda_2) := \sup_{\xi \in \mathcal{Z}} \ell(\xi) - \lambda_2 \| \xi - z \|^p$  w.r.t.  $\tilde{\mu}_n$ . The additional  $\lambda_1$  variable above is introduced to account for the clean family  $\mathcal{G}_2(\sigma, z_0)$ , and the use of partial transportation under  $W_p^\varepsilon$  results in the introduction of the operator  $[\cdot]_+$  and the decision variable  $\alpha$ .
Remark 3 (Connection to conditional value at risk (CVaR)). The CVaR of a Borel measurable loss function  $\ell$  acting on a random vector  $Z\sim \mu \in \mathcal{P}(\mathcal{Z})$  with risk level  $\varepsilon \in (0,1)$  is defined as
$$
\operatorname {C V a R} _ {1 - \varepsilon , \mu} [ \ell (Z) ] = \inf  _ {\alpha \in \mathbb {R}} \alpha + \frac {1}{1 - \varepsilon} \mathbb {E} _ {Z \sim \mu} \left[ [ \ell (Z) - \alpha ] _ {+} \right].
$$
CVaR is also known as expected shortfall and is equivalent to the conditional expectation of  $\ell (Z)$  given that it is above an  $\varepsilon$  threshold. This concept is often used in finance to evaluate the market risk of a portfolio. With this definition, the result of Proposition 6 can be written as
$$
\sup_{\substack{\nu \in \mathcal{G}_{2}(\sigma ,z_{0}):\\ \mathsf{W}_{p}^{\varepsilon}(\tilde{\mu}_{n}\| \nu)\leq \rho}}\mathbb{E}_{\nu}[\ell ] = \inf_{\lambda_{1},\lambda_{2}\in \mathbb{R}_{+}}\lambda_{1}\sigma^{2} + \lambda_{2}\rho^{p} + \mathrm{CVaR}_{1 - \varepsilon ,\tilde{\mu}_{n}}\bigg[ \sup_{\xi \in \mathcal{Z}}\ell (\xi) - \lambda_{1}\| \xi -z_{0}\|^{2} - \lambda_{2}\| \xi -Z\|^{p}\bigg].
$$
When  $\varepsilon \to 0$  and  $\sigma \to \infty$ , whence CVaR reduces to expected value and the constrained class  $\mathcal{G}_2(\sigma, z_0)$  expands to  $\mathcal{P}(\mathcal{Z})$ , the dual formulation above reduces to that of classical WDRO [8, 21].
Evaluating  $\overline{\ell}$  requires solving a maximization problem, which could be in itself challenging. To overcome this, we impose additional convexity assumptions, which are standard for WDRO [35, 43].
Assumption 2 (Convexity condition). The loss  $\ell$  is a pointwise maximum of finitely many concave functions, i.e.,  $\ell (\xi) = \max_{j\in [J]}\ell_j(\xi)$ , for some  $J\in \mathbb{N}$ , where  $\ell_{j}$  is real-valued, l.s.c., and concave<sup>9</sup>. The set  $\mathcal{Z}$  is closed and convex. The atoms of  $\tilde{\mu}_n$  are in the relative interior of  $\mathcal{Z}$ .
Theorem 2 (Convex reformulation). Under Assumption 1, for any  $\ell \in \mathcal{L}$  satisfying Assumption 2 and  $z_0\in \mathbb{R}^d$ , we have
$$
\sup_{\substack{\nu \in \mathcal{G}_{q}(\sigma ,z_{0}):\\ \mathsf{W}_{p}^{\varepsilon}(\tilde{\mu}_{n}\| \nu)\leq \rho}}\mathbb{E}_{\nu}[\ell ] = \left\{ \begin{array}{ll}\inf & \lambda_{1}\sigma^{2} + \lambda_{2}\rho^{p} + \alpha +\frac{1}{n(1 - \varepsilon)}\sum_{i\in [n]}s_{i}\\ \text{s.t.} & \alpha \in \mathbb{R},\lambda_{1},\lambda_{2}\in \mathbb{R}_{+},s,\tau_{ij}\in \mathbb{R}^{n}_{+},\zeta_{ij}^{\ell},\zeta_{ij}^{\mathcal{G}},\zeta_{ij}^{\mathsf{W}},\zeta_{ij}^{\mathcal{Z}}\in \mathbb{R}^{d},  \forall i\in [n],\forall j\in [J]\\ & s_{i}\geq (-\ell_{j})^{*}(\zeta_{ij}^{\ell}) + z_{0}^{\top}\zeta_{ij}^{\mathcal{G}} + \tau_{ij}\\ & \qquad +\tilde{Z}_{i}^{\top}\zeta_{ij}^{\mathsf{W}} + P_{h}(\zeta_{ij}^{\mathsf{W}},\lambda_{2}) + \chi^{*}_{\mathcal{Z}}(\zeta_{ij}^{\mathcal{Z}}) - \alpha ,\quad \forall i\in [n],\forall j\in [J]\\ & \zeta_{ij}^{\ell} + \zeta_{ij}^{\mathcal{G}} + \zeta_{ij}^{\mathsf{W}} + \zeta_{ij}^{\mathcal{Z}} = 0,  \| \zeta_{ij}^{\mathcal{G}}\|^{2}\leq \lambda_{1}\tau_{ij},\qquad \forall i\in [n],\forall j\in [J], \end{array} \right.
$$
where  $P_{h}$  is the perspective function (i.e.,  $P_{h}(\zeta, \lambda) = \lambda h(\zeta / \lambda)$ ) of
${}^{9}$  Generally,any continuous function can be approximated arbitrarily well by a maximum of finitely many concave functions. However,the number of functions needed may be arbitrarily large in general. Fortunately, some losses like the  ${\ell }_{\infty }$  -norm  $\parallel z{\parallel }_{\infty } = \mathop{\max }\limits_{{i \in  \left\lbrack  d\right\rbrack  ,a \in  \left\{  {\pm 1}\right\}  }}{\sigma z}_{i}$  require only  $\operatorname{poly}\left( d\right)$  pieces.
$$
h (\zeta) := \left\{ \begin{array}{l l} \chi_ {\{z \in \mathbb {R} ^ {d}: \| z \| \leq 1 \}} (\zeta), & p = 1 \\ \frac {(p - 1) ^ {p - 1}}{p ^ {p}} \| \zeta \| ^ {\frac {p}{p - 1}}, & p > 1. \end{array} \right. \tag {6}
$$
The minimization problem in Theorem 2 is a finite-dimensional convex program. In Section 5, we use this result in conjunction with Proposition 5 to efficiently perform outlier-robust WDRO.
We conclude this section by characterizing the worst-case distribution, i.e., the optimal adversarial strategy, for our outlier-robust WDRO problem. To that end, we need the primal formulation below.
Theorem 3 (Worst-case distribution). Under Assumption 1, for any  $\ell \in \mathcal{L}$  satisfying Assumption 2 and  $z_0\in \mathbb{R}^d$ , we have
$$
\sup_{\substack{\nu \in \mathcal{G}_{q}(\sigma ,z_{0}):\\ \mathsf{W}_{p}^{\varepsilon}(\tilde{\mu}_{n}\| \nu)\leq \rho}}\mathbb{E}_{\nu}[\ell ] = \left\{ \begin{array}{ll}\max \text{} & -\sum_{(i,j)\in [n]\times [J]}P_{-\ell_{j}}(\xi_{ij},q_{ij})\\ \text{s.t.} & q_{ij}\in \mathbb{R}_{+},  \xi_{ij}\in q_{ij}\cdot \mathcal{Z}\\ & \sum_{j\in [J]}q_{ij}\leq \frac{1}{n(1 - \varepsilon)}\\ & \sum_{(i,j)\in [n]\times [J]}q_{ij} = 1\\ & \sum_{(i,j)\in [n]\times [J]}P_{\| \cdot \|^{p}}(\xi_{ij} - q_{ij}\tilde{Z}_{i},q_{ij})\leq \rho \\ & \sum_{(i,j)\in [n]\times [J]}P_{\| \cdot \|^{2}}(\xi_{ij} - q_{ij}z_{0},q_{ij})\leq \sigma^{2} \end{array} \right.
$$
The discrete distribution  $\nu^{\star} = \sum_{(i,j)\in \mathcal{Q}}q_{ij}^{\star}\delta_{\xi_{ij}^{\star} / q_{ij}^{\star}}$  achieves the worst-case expectation on the left-hand side, where  $(q_{ij}^{\star},\xi_{ij}^{\star})_{(i,j)\in [n]\times [J]}$  are optimizers of the maximization problem on the right and  $\mathcal{Q}:= \{(i,j)\in [n]\times [J]:q_{ij}^{\star} > 0\}$ .
The maximization problem from Theorem 3 is the conjugate dual of the minimization in Theorem 2. Subsequently, we propose a systematic approach for constructing a discrete distribution based on a solution derived from the maximization problem that achieves the worst-case expected loss.
Remark 4 (Comparison to WDRO worst-case distribution). Recall that our robust WDRO approach reduces to the classic WDRO approach as  $\varepsilon = 0$  and  $\sigma \to \infty$ . Consequently, this implies that the constraints  $\sum_{j \in [J]} q_{ij} \leq 1 / (n(1 - \varepsilon))$  and  $\sum_{(i,j) \in [n] \times [J]} P_{\| \cdot \|^2} (\xi_{ij} - q_{ij} z_0, q_{ij}) \leq \sigma^2$  can be dropped under this specific choice of  $\varepsilon$  and  $\sigma$ . As a result, our construction simplifies to the approach presented in [35, Theorem 4.4] for WDRO problems.
Remark 5 (Parameter tuning). In practice,  $\varepsilon$ ,  $\rho_0$ , and the relevant tail bound may be unknown. Thus, in Appendix F, we consider learning under Setting B with  $\mathcal{G} = \mathcal{G}_{\mathrm{cov}}(\sigma)$  for potentially unknown  $\varepsilon$ ,  $\sigma$ , and  $\rho_0$ . First, we observe that knowledge of upper bounds on these parameters is sufficient to attain risk bounds scaling in terms of said upper bounds. This approach avoids meticulous parameter tuning but may result in suboptimal risk. To efficiently match our risk bounds with known parameters, we show that it is necessary and sufficient to know  $\rho_0$  and at least one of  $\varepsilon$  or  $\sigma$  (up to constant factors).
4 Low-Dimensional Features
While Proposition 3 shows that the excess risk bounds from Theorem 1 cannot be improved in general, finer guarantees can be derived when the optimal loss function depends only on  $k$ -dimensional affine features of the data. Defining  $\mathcal{G}^{(k)}$  as the union of the projections  $\{U_{\#}\mu : \mu \in \mathcal{G}\}$  over  $U \in \mathbb{R}^{k \times d}$  with  $UU^{\top} = I_k^{10}$ , we improve the excess risk bound of Theorem 1 for this setting.
Theorem 4 (Excess risk bound). Under Setting  $A$ , let  $\hat{\ell}$  minimize (OR-WDRO), and assume that  $\ell_{\star} = \underline{\ell} \circ A$  for an affine map  $A: \mathbb{R}^d \to \mathbb{R}^k$  and some  $\underline{\ell}: \mathbb{R}^k \to \mathbb{R}$ . Writing  $c = 2(1 - \varepsilon)^{-1/p}$ , we have
$$
\mathbb{E}_{\mu}[\hat{\ell} ] - \mathbb{E}_{\mu}[\ell_{\star}]\\ \leq \left\{ \begin{array}{ll}\| \ell_{\star}\|_{\text{Lip}}\big(c\rho +2\tau_{1}(\mathcal{G}^{(k)},2\varepsilon)\big), & p = 1,\ell_{\star}Lipschitz\\ \| \ell_{\star}\|_{\dot{H}^{1,2}(\mu)}\big(c\rho +2\tau_{2}(\mathcal{G}^{(k)},2\varepsilon)\big) + \frac{1}{2}\alpha \big(c\rho +2\tau_{2}(\mathcal{G}^{(k)},2\varepsilon)\big)^{2}, & p = 2,\ell_{\star}\text{$\alpha$ - smooth} \end{array} \right..
$$
This dependence on  $\mathcal{G}^{(k)}$  rather than  $\mathcal{G} = \mathcal{G}^{(d)}$  is a substantial improvement when  $k\ll d$
<sup>10</sup>If  $\mathcal{G}$  is closed under isometries, like  $\mathcal{G}_{\mathrm{cov}}$  , then  $\mathcal{G}^{(k)} = \{U_{\#}\mu :\mu \in \mathcal{G}\}$  for any such  $U$
Corollary 4 (Risk bounds for  $\mathcal{G}_{\mathrm{cov}}$ ). Under the setting of Theorem 4 with  $\mathcal{G} \subseteq \mathcal{G}_{\mathrm{cov}}$ , we have
$$
\mathbb {E} _ {\mu} [ \hat {\ell} ] - \mathbb {E} _ {\mu} [ \ell_ {\star} ] \lesssim \left\{ \begin{array}{l l} \| \ell_ {\star} \| _ {\mathrm {L i p}} \big (\rho + \sqrt {k \varepsilon} \big), & p = 1, \ell_ {\star}   L i p s c h i t z \\ \| \ell_ {\star} \| _ {\dot {H} ^ {1, 2} (\mu)} (\rho + \sqrt {k}) + \alpha (\rho^ {2} + k), & p = 2, \ell_ {\star}   \alpha \text {- s m o o t h}. \end{array} \right.
$$
We again have a matching lower bound for the Lipschitz setting, this time using  $k$ -variate regression.
Proposition 7 (Lower bound). Fix  $\mathcal{Z} = \mathbb{R}^d$  and  $\varepsilon \in [0,0.49]$ . For any  $L \geq 0$ , there exists a family  $\mathcal{L} \subseteq \mathrm{Lip}_L(\mathbb{R}^d)$ , independent of  $\varepsilon$ , such that each  $\ell \in \mathcal{L}$  decomposes as  $\ell = \underline{\ell} \circ A$  for  $A \in \mathbb{R}^{k \times d}$  and  $\underline{\ell} : \mathbb{R}^k \to \mathbb{R}$ , and such that for any decision rule  $\mathsf{D} : \mathcal{P}(\mathcal{Z}) \to \mathcal{L}$  there exists a pair  $(\mu, \tilde{\mu}) \in \mathcal{G}_{\mathrm{cov}} \times \mathcal{P}(\mathcal{Z})$  with  $\mathsf{W}_1^\varepsilon(\tilde{\mu}, \mu) \leq \rho$  satisfying  $\mathbb{E}_{\mu}[\mathsf{D}(\tilde{\mu})] - \inf_{\ell \in \mathcal{L}} \mathbb{E}_{\mu}[\ell] \gtrsim L\big(\rho + \sqrt{k\varepsilon}\big)$ .
For computation, we turn to a slightly modified  $n$ -sample contamination model. Our analysis for the low-dimensional case only supports additive TV corruptions (sometimes called Huber contamination).
Setting  $\mathbf{B}'$ : Fix  $\rho, \varepsilon, \mathcal{L}, \mathcal{G}$  as in Setting A, and fix  $m = \lceil (1 - \varepsilon)n \rceil$  for some  $n \in \mathbb{N}$ . Let  $Z_{1}, \ldots, Z_{m}$  be drawn i.i.d. from  $\mu \in \mathcal{G}$ , with empirical measure  $\hat{\mu}_{m} = \frac{1}{m}\sum_{i=1}^{m}\delta_{Z_{i}}$ . Upon observing these clean samples, Nature applies a  $\mathsf{W}_{p}$  perturbation of size  $\rho_{0}$ , producing  $\{Z_{i}'\}_{i=1}^{m}$  with empirical measure  $\mu_{m}'$  such that  $\mathsf{W}_{p}(\hat{\mu}_{m},\mu_{m}') \leq \rho_{0}$ . Finally, Nature adds  $\lfloor \varepsilon n \rfloor$  samples to obtain  $\{\tilde{Z}_i\}_{i=1}^n$  with empirical measure  $\tilde{\mu}_{n}$  such that  $\mu_{m}' \leq \frac{1}{1 - \varepsilon}\tilde{\mu}_{n}$ . Equivalently, the final dataset satisfies  $\mathsf{W}_{p}^{\varepsilon}(\tilde{\mu}_{n}\|\hat{\mu}_{m}) \leq \rho_{0}$ .
As before, we modify (OR-WDRO) using a centered alternative to  $\mathcal{G}_{\mathrm{cov}}$ . Defining  $\mathcal{G}_{\mathrm{cov}}(\sigma, z_0) := \{\mu \in \mathcal{P}(\mathcal{Z}) : \mathbb{E}_\mu[(Z - z_0)(Z - z_0)^\top] \preceq \sigma^2 I_d\}$ , we consider the outlier-robust WDRO problem
$$
\inf  _ {\ell \in \mathcal {L}} \sup  _ {\nu \in \mathcal {G} _ {\operatorname {c o v}} (\sigma , z _ {0}): \mathbb {W} _ {p} ^ {\varepsilon} (\tilde {\mu} _ {n} \| \nu) \leq \rho} \mathbb {E} _ {\nu} [ \ell ]. \tag {7}
$$
To start, we provide a corresponding risk bound which matches Corollary 4 when  $k = O(1)$ .
Proposition 8 (Risk bound for modified problem). Consider Setting  $B'$  with  $\mathcal{G} \subseteq \mathcal{G}_{\mathrm{cov}}$ , and assume  $\ell_{\star} = \underline{\ell} \circ A$  for affine  $A: \mathbb{R}^d \to \mathbb{R}^k$  and  $\underline{\ell}: \mathbb{R}^k \to \mathbb{R}$ . Fix  $z_0 \in \mathcal{Z}$  such that  $\| z_0 - \mathbb{E}_\mu[Z] \| \leq E = O(\rho_0 + 1)$ , and assume  $W_p(\hat{\mu}_m, \mu) \leq \delta$ . If  $\hat{\ell}$  minimizes (7) with  $\rho = \rho_0 + \delta$  and  $\sigma = 1 + E$ , then