SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We simply instantiate the lower bound construction from Proposition 3 in  $\mathbb{R}^k$ , viewed as a subspace of  $\mathbb{R}^d$ . Extending each  $\ell \in \mathcal{L}$  to  $\mathbb{R}^d$  by  $\ell(z) = \ell(z_{1:k})$ , the same lower bound applies with  $d \gets k$ .
C.4 Proof of Proposition 8
For  $Z\sim \mu \in \mathcal{G}_{\mathrm{cov}}$  and  $z_0,w\in \mathbb{R}^d$  , we bound
$$
\begin{array}{l} \mathbb {E} \left[ w ^ {\top} (Z - z _ {0}) (Z - z _ {0}) ^ {\top} w \right] ^ {\frac {1}{2}} = \mathbb {E} \left[ w ^ {\top} (Z - z _ {0}) ^ {2} \right] ^ {\frac {1}{2}} \\ \leq \mathbb {E} \left[ w ^ {\top} (Z - \mathbb {E} [ Z ]) ^ {2} \right] ^ {\frac {1}{2}} + \left| w ^ {\top} (\mathbb {E} [ Z ] - z _ {0}) \right| \\ \leq \| w \| \left(1 + \| z _ {0} - \mathbb {E} [ Z ] \|\right). \\ \end{array}
$$
Consequently, we have  $\mu \in \mathcal{G}_{\mathrm{cov}}(1 + \| z_0 - \mathbb{E}[Z]\|, z_0) \subseteq \mathcal{G}_{\mathrm{cov}}(\sigma, z_0)$ . Moreover, we have  $\mathsf{W}_p^\varepsilon (\tilde{\mu}_n,\mu) \leq \mathsf{W}_p(\mu_m',\mu) \leq \rho_0 + \delta = \rho$ . Decomposing  $\ell_{\star} = \bar{\ell} \circ Q$  as in Lemma 12 and writing  $\mu_k = Q_\# \mu$ , the same approach applied in the proof of Theorem 4 gives
$$
\mathbb {E} _ {\mu} [ \hat {\ell} ] - \mathbb {E} _ {\mu} [ \ell_ {\star} ] \leq \mathcal {R} _ {\mu_ {k}, p} \left(c \rho + 2 \tau_ {p} \left(Q _ {\#} \mathcal {G} _ {\operatorname {c o v}} (\sigma , z _ {0}), 2 \varepsilon\right); \tilde {\ell}\right)
$$
$$
\leq \mathcal {R} _ {\mu_ {k}, p} \left(c \rho + O \left(\sigma \sqrt {k} \varepsilon^ {\frac {1}{p} - \frac {1}{2}}\right); \tilde {\ell}\right).
$$
$$
c \rho + O (\sigma \sqrt {k \varepsilon}) \lesssim \rho_ {0} + \delta + (1 + \rho_ {0}) \sqrt {k \varepsilon} \lesssim \sqrt {k} \rho_ {0} + \sqrt {k \varepsilon} + \delta ,
$$
$$
c \rho + O (\sigma \sqrt {k}) \lesssim \rho_ {0} + \delta + (1 + \rho_ {0}) \sqrt {k} \lesssim \sqrt {k} \rho_ {0} + \sqrt {k} + \delta
$$
We then conclude as in Theorem 4.
C.5 Proof of Proposition 9
Since iterative filtering works by identifying a subset of samples with bounded covariance and  $\mathsf{W}_p$  perturbations can arbitrarily increase second moments when  $p < 2$ , it is not immediately clear how to apply this method. Fortunately,  $\mathsf{W}_2$  perturbations have a bounded affect on second moments, and, by trimming out a small fraction of samples, we can ensure that a  $\mathsf{W}_1$  step is bounded under  $\mathsf{W}_2$ .
Lemma 13. For any  $\mu, \nu \in \mathcal{P}(\mathbb{R}^d)$  and  $0 < \gamma \leq 1$ , there exists  $\nu' \in \mathcal{P}(\mathbb{R}^d)$  with  $\| \nu' - \nu \|_{\mathrm{TV}} \leq \tau$  such that  $\mathsf{W}_1(\mu, \nu') \leq \mathsf{W}_1(\mu, \nu)$  and  $\mathsf{W}_{\infty}(\mu, \nu') \leq \mathsf{W}_1(\mu, \nu) / \gamma$ .
Proof. Let  $(X,Y)$  be a coupling of  $\mu$  and  $\nu$  with  $\mathbb{E}[\| X - Y\| ] = \mathsf{W}_1(\mu ,\nu)$ . Writing  $\Delta = \| X - Y\|$ , the event  $E$  that  $\Delta \leq \mathsf{W}_1(\mu ,\nu) / \gamma$  has probability at least  $1 - \gamma$  by Markov's inequality. We shall take  $\nu^{\prime}$  be the law of  $Y^{\prime} = \mathbb{1}_{E}Y + (1 - \mathbb{1}_{E})X$ . By design,  $\| \nu^{\prime} - \nu \|_{\mathsf{TV}}\leq \gamma$ , and
$$
\mathsf {W} _ {1} (\mu , \nu^ {\prime}) \leq \mathbb {E} [ \| X - Y ^ {\prime} \| ^ {p} ] = \mathbb {E} [ \mathbb {1} _ {E} \| X - Y \| ^ {p} ] \leq \mathsf {W} _ {1} (\mu , \nu).
$$
Finally, we bound  $\mathsf{W}_{\infty}(\mu ,\nu^{\prime})\leq \| \mathbb{1}_E\Delta \|_{\infty}\leq \mathsf{W}_1(\mu ,\nu) / \gamma$
For consistency between Settings B and  $\mathbf{B}^{\prime}$ , we let  $m = n$  if we are the former. Thus, in both cases, we have  $\mathsf{W}_p(\hat{\mu}_m,\mu_m')\leq \rho_0$  and  $\| \mu_m' - \tilde{\mu}_n\|_{\mathsf{TV}}\leq \varepsilon \leq \varepsilon_0\coloneqq 1 / 12$ . It is well known that the empirical measure  $\hat{\mu}_m$  will inherit the bounded covariance of  $\mu$  for  $m$  sufficiently large, so long as a small fraction of samples are trimmed out. In particular, by Lemma A.18 of [15] and our sample complexity requirement, there exists a uniform discrete measure  $\alpha_{k}$  over a subset of  $k = (1 - \varepsilon_0 / 120)m$  points, such that  $\| \mathbb{E}_{\alpha_k}[Z] - \mathbb{E}_{\mu}[Z]\| \lesssim 1$  and  $\Sigma_{\alpha_k}\preceq O(1)I_d$  with probability at least 0.99.
Moreover, applying Lemma 13 with  $\gamma = \varepsilon_0 / 120$ , there exists  $\beta \in \mathcal{P}(\mathbb{R}^d)$  with  $\| \beta - \mu_m' \|_{\mathrm{TV}} \leq \varepsilon_0 / 120$  and  $\mathsf{W}_2(\beta, \hat{\mu}_n) \leq 240\rho_0 / \varepsilon_0$ . Combining, we have that  $\mathsf{W}_2^{\varepsilon_0 / 120 + \varepsilon_0 / 120 + \varepsilon_0}(\alpha_m, \tilde{\mu}_n) = \mathsf{W}_2^{61\varepsilon_0 / 60}(\alpha_k, \tilde{\mu}_n) \leq 240\rho_0 / \varepsilon_0$ , and so there exists  $\kappa \in \mathcal{P}(\mathbb{R}^d)$  such that  $\mathsf{W}_{\infty}(\alpha_k, \kappa) \leq 240\rho_0 / \varepsilon_0$  and  $\| \kappa - \tilde{\mu}_n \|_{\mathrm{TV}} \leq 61\varepsilon_0 / 60$ . The  $W_{\infty}$  bound implies that  $\Sigma_{\kappa} \preceq O(1 + \rho_0^2\varepsilon_0^{-2})I_d$ .
Thus, by the proof of Theorem 4.1 in [26] and our sample complexity requirement, the iterative filtering algorithm (Algorithm 1 therein, based on that of [15]) applied with an outlier fraction of  $61 / 60\varepsilon_0\leq 1 / 10$  returns a reweighting of  $\tilde{\mu}_n$  whose mean  $z_{0}\in \mathbb{R}^{d}$  is within  $O(\sqrt{\varepsilon_0} +\rho_0 / \sqrt{\varepsilon_0}) = O(1 + \rho_0)$  of that of  $\kappa$ . Thus, we obtain
$$
\begin{array}{l} \left\| z _ {0} - \mathbb {E} _ {\mu} [ Z ] \right\| \leq \left\| \mathbb {E} _ {\kappa} [ Z ] - \mathbb {E} _ {\mu} [ Z ] \right\| + O \left(1 + \rho_ {0}\right) \\ \leq \left\| \mathbb {E} _ {\alpha_ {k}} [ Z ] - \mathbb {E} _ {\mu} [ Z ] \right\| + O \left(1 + \rho_ {0}\right) \\ \leq O (1 + \rho_ {0}), \\ \end{array}
$$
C.6 Proof of Proposition 10
$$
\sup_{\substack{\nu \in \mathcal{G}_{\mathrm{cov}}(\sigma ,z_{0}):\\ \mathsf{W}^{\varepsilon}_{p}(\tilde{\mu}_{n}\| \nu)\leq \rho}}\mathbb{E}_{\nu}[\ell ] = \sup_{\substack{\mu^{\prime},\nu \in \mathcal{P}(\mathcal{Z})\\ \pi \in \Pi (\mu^{\prime},\nu)}}\left\{ \begin{array}{c}\mathbb{E}_{\nu}[ (Z - z_{0})(Z - z_{0})^{\top}] \preceq \sigma I_{d},\\ \mathbb{E}_{\nu}[\ell ]:\mathbb{E}_{\pi}[\| Z^{\prime} - Z\|^{p}]\leq \rho^{p},\\ \mu^{\prime}\leq \frac{1}{1 - \varepsilon}\tilde{\mu}_{n} \end{array} \right\}
$$
$$
\begin{array}{l} = \sup_{\substack{m\in \mathbb{R}^{n}\\ \nu_{1},\ldots ,\nu_{n}\in \mathcal{P}(\mathcal{Z})}}\left\{\sum_{i\in [n]}m_{i}\mathbb{E}_{\nu_{i}}[(Z_{i} - z_{0})(Z_{i} - z_{0})^{\top}]\preceq \sigma I_{d},\right.\\ \sum_{i\in [n]}m_{i}\mathbb{E}_{\nu_{i}}[\| \tilde{Z}_{i} - Z_{i}\|^{p}]\leq \rho^{p},\\ 0\leq m_{i}\leq \frac{1}{n(1 - \varepsilon)},  \forall i\in [n],\\ \sum_{i\in [n]}m_{i} = 1\Bigg\} \\ = \sup  _ { \begin{array}{c} m \in \mathbb {R} ^ {n} \\ \nu_ {1} ^ {\prime}, \ldots , \nu_ {n} ^ {\prime} \geq 0 \end{array} } \left\{ \begin{array}{l l} & \sum_ {i \in [ n ]} \int_ {\mathcal {Z}} (z _ {i} - z _ {0}) (z _ {i} - z _ {0}) ^ {\top} d \nu_ {i} ^ {\prime} (z _ {i}) \preceq \sigma I _ {d}, \\ & \sum_ {i \in [ n ]} \int_ {\mathcal {Z}} \| z _ {i} - \tilde {Z} _ {i} \| ^ {p} d \nu_ {i} ^ {\prime} (z _ {i}) \leq \rho^ {p}, \\ \sum_ {i \in [ n ]} \mathbb {E} _ {\nu_ {i} ^ {\prime}} [ \ell ]: & 0 \leq m _ {i} \leq \frac {1}{n (1 - \varepsilon)},   \forall i \in [ n ], \\ & \sum_ {i \in [ n ]} m _ {i} = 1 \\ & \int_ {\mathcal {Z}} d \nu_ {i} ^ {\prime} (z _ {i}) = m _ {i},   \forall i \in [ n ] \end{array} \right\}, \\ \end{array}
$$
where the first equality follows from the definitions of  $\underline{G}_{\mathrm{cov}}(\sigma, z_0)$  and  $\mathsf{W}_p^\varepsilon (\tilde{\mu}_n\| \nu)$ . The second and the third equalities follow from the same variable substitution as in the proof of Proposition 6. The last optimization problem admits the dual form
$$
\inf_{\substack{\Lambda_{1}\in \mathbb{Q}_{+}^{d},\lambda_{2}\in \mathbb{R}_{+}\\ r,s\in \mathbb{R}^{n},\alpha \in \mathbb{R}}}\left\{ \begin{array}{ll} - z_{0}^{\top}\Lambda_{1}z_{0} + \sigma   \mathrm{Tr}[\Lambda_{1}] + \lambda_{2}\rho^{p} + \alpha +\frac{\sum_{i\in[n]}s_{i}}{n(1 - \varepsilon)}: & \\ s_{i}\geq \max \{0,r_{i} - \alpha \} ,\quad \forall i\in [n],\\ r_{i}\geq \ell (\xi) - \xi^{\top}\Lambda_{1}\xi +2\xi^{\top}\Lambda_{1}z_{0} - \lambda_{2}\| \xi -\tilde{Z}_{i}\|^{p}, & \forall \xi \in \mathcal{Z},\forall i\in [n] \end{array} \right\} .
$$
Strong duality holds thanks to Assumption 3 and [47, Proposition 3.4]. The proof of the second claim concludes by removing the decision variables  $r$  and  $s$  and using the definition of  $\tilde{\mu}_n$ .
C.7 Proof of Theorem 5
By Proposition 10 and exploiting the definition of  $\tilde{\mu}_n$ , we have
$$
\begin{array}{l} \sup  \quad \mathbb {E} _ {\nu} [ \ell ] \\ \nu \in \mathcal {G} _ {\text {c o v}} (\sigma , z _ {0}): \\ \mathsf {W} _ {p} ^ {\varepsilon} (\tilde {\mu} _ {n} \| \nu) \leq \rho \\ = \left\{ \begin{array}{l l} \inf  & - z _ {0} ^ {\top} \Lambda_ {1} z _ {0} + \sigma \operatorname {T r} [ \Lambda_ {1} ] + \lambda_ {2} \rho^ {p} + \alpha + \frac {1}{n (1 - \varepsilon)} \sum_ {i \in [ n ]} s _ {i} \\ \text {s . t .} & \Lambda_ {1} \in \mathbb {Q} _ {+} ^ {d},   \lambda_ {2} \in \mathbb {R} _ {+},   s \in \mathbb {R} _ {+} ^ {n} \\ & s _ {i} \geq \sup  _ {\xi \in \mathcal {Z}} \ell (\xi) - \xi^ {\top} \Lambda_ {1} \xi + 2 \xi^ {\top} \Lambda_ {1} z _ {0} - \lambda_ {2} \| \xi - \tilde {Z} _ {i} \| ^ {p} - \alpha \quad \forall i \in [ n ] \end{array} \right. \\ = \left\{ \begin{array}{l l} \inf  & - z _ {0} ^ {\top} \Lambda_ {1} z _ {0} + \sigma \operatorname {T r} \left[ \Lambda_ {1} \right] + \lambda_ {2} \rho^ {p} + \frac {1}{n (1 - \varepsilon)} \sum_ {i \in [ n ]} s _ {i} \\ \text {s . t .} & \Lambda_ {1} \in \mathbb {Q} _ {+} ^ {d}, \lambda_ {2} \in \mathbb {R} _ {+}, s \in \mathbb {R} _ {+} ^ {n} \\ & s _ {i} \geq \sup  _ {\xi \in \mathcal {Z}} \ell_ {j} (\xi) - \xi^ {\top} \Lambda_ {1} \xi + 2 \xi^ {\top} \Lambda_ {1} z _ {0} - \lambda_ {2} \| \xi - \tilde {Z} _ {i} \| ^ {p} - \alpha \quad \forall i \in [ n ], \forall j \in [ J ] \end{array} \right. \tag {10} \\ \end{array}
$$
$$
\begin{array}{l} \sup  _ {\xi \in \mathcal {Z}} \ell_ {j} (\xi) - \xi^ {\top} \Lambda_ {1} \xi + 2 \xi^ {\top} \Lambda_ {1} z _ {0} - \lambda_ {2} \| \xi - \tilde {Z} _ {i} \| ^ {p} - \alpha \\ = \left\{ \begin{array}{l l} \inf  & (- \ell_ {j}) ^ {*} (\zeta_ {i j} ^ {\ell}) + \frac {1}{4} (\zeta_ {i j} ^ {\mathcal {G}}) ^ {\top} \Lambda_ {1} ^ {- 1} \zeta_ {i j} ^ {\mathcal {G}} + \tilde {Z} _ {i} ^ {\top} \zeta_ {i j} ^ {\mathsf {W}} + \lambda_ {2} h (\zeta_ {i j} ^ {\mathsf {W}} / \lambda_ {2}, p) + \chi_ {\mathcal {Z}} ^ {*} (\zeta_ {i j} ^ {\mathcal {Z}}) - \alpha \\ \text {s . t .} & \zeta_ {i j} ^ {\ell}, \zeta_ {i j} ^ {\mathcal {G}}, \zeta_ {i j} ^ {\mathsf {W}}, \zeta_ {i j} ^ {\mathcal {Z}} \in \mathbb {R} ^ {d},   \zeta_ {i j} ^ {\ell} + \zeta_ {i j} ^ {\mathcal {G}} + \zeta_ {i j} ^ {\mathsf {W}} + \zeta_ {i j} ^ {\mathcal {Z}} = 2 \Lambda_ {1} z _ {0} \end{array} \right. \\ \end{array}
$$
where the equality is a result of strong duality due to [57, Theorem 2] and Lemma 11. The claim follows by introducing the epigraph variable  $\tau_{ij}$  for the term  $\frac{1}{4} (\zeta_{ij}^{\mathcal{G}})^{\top}\Lambda_1^{-1}\zeta_{ij}^{\mathcal{G}}$ , and substituting all resulting dual minimization problems into (10) and eliminating the corresponding minimization operators. Note that by the Schur complement argument, we have
$$
\frac {1}{4} (\zeta_ {i j} ^ {\mathcal {G}}) ^ {\top} \Lambda_ {1} ^ {- 1} \zeta_ {i j} ^ {\mathcal {G}} \leq \tau_ {i j} \iff \left[ \begin{array}{c c} \Lambda_ {1} & \zeta_ {i j} ^ {\mathcal {G}} \\ (\zeta_ {i j} ^ {\mathcal {G}}) ^ {\top} & 4 \tau_ {i j} \end{array} \right] \succeq 0,
$$
which implies that the resulting reformulation is indeed convex.
D Comparison to WDRO with Expanded Radius around Minimum Distance Estimate (Remark 1)
First, we prove the claimed excess risk bound for WDRO with an expanded radius around the minimum distance estimate  $\hat{\mu} = \hat{\mu} (\tilde{\mu},\mathcal{G},\varepsilon)\coloneqq \mathrm{argmin}_{\nu \in \mathcal{G}}\mathsf{W}_p^\varepsilon (\nu ,\tilde{\mu})$ . We write  $c = 2(1 - \varepsilon)^{-1 / p}$  as in Theorem 1.
Lemma 14. Under Setting  $A$ , let  $\hat{\ell} = \operatorname{argmin}_{\ell \in \mathcal{L}} \sup_{\nu \in \mathcal{P}(\mathcal{Z}): \mathsf{W}_p(\nu, \hat{\mu}) \leq \rho'} \mathbb{E}_{\nu}[\ell]$ , for the expanded radius  $\rho' := c\rho + 2\tau_p(\mathcal{G}, 2\varepsilon)$ . We then have  $\mathbb{E}_{\mu}[\hat{\ell}] - \mathbb{E}_{\mu}[\ell_\star] \leq \mathcal{R}_{\mu,p}(c\rho + 2\tau_p(\mathcal{G}, 2\varepsilon); \ell_\star)$ .
Proof. Since  $\mathsf{W}_p^\varepsilon (\mu ,\tilde{\mu})\leq \rho$  and  $\mu \in \mathcal{G}$ , we have  $\mathsf{W}_p^\varepsilon (\hat{\mu},\tilde{\mu})\leq \rho$ . Thus, Lemma 2 gives that  $\mathsf{W}_p^{2\varepsilon}(\hat{\mu},\mu)\leq 2\rho$ . By Lemma 3, we then have  $\mathsf{W}_p(\hat{\mu},\mu)\leq c\rho +2\tau_p(\mathcal{G},2\varepsilon)$ , and so Lemma 7 gives the desired result.
In practice, we are unaware of efficient finite-sample algorithms to compute  $\hat{\mu}$ . For the class  $\mathcal{G}_{\mathrm{cov}}$ , we instead propose the spectral reweighing estimate  $\check{\mu} = \check{\mu} (\tilde{\mu},\varepsilon) \coloneqq \operatorname*{argmin}_{\nu \in \mathcal{P}_2(\mathcal{Z}), \nu \leq \frac{1}{1 - 2\varepsilon}\tilde{\mu}}\|\Sigma_{\nu}\|_{\mathrm{op}}$ , where  $\|\cdot\|_{\mathrm{op}}$  is the matrix operator norm (see [26] for varied applications of spectral reweighing). In practice, when  $\tilde{\mu} = \tilde{\mu}_n$  is an  $n$ -sample empirical measure, one can efficiently obtain a feasible measure  $\nu$  whose objective value is optimal up to constant factors for the problem with  $\varepsilon \gets 3\varepsilon$ , using the iterative filtering algorithm [15]. We work with the exact minimizer  $\tilde{\mu}$  for convenience, but our results are robust to such approximate solutions.
Lemma 15. Under Setting  $A$  with  $\mathcal{G} = \mathcal{G}_{\mathrm{cov}}$ ,  $p = 1$ , and  $0 < \varepsilon \leq 0.2$ , we have  $\mathsf{W}_1(\check{\mu},\mu) \lesssim \sqrt{d}\rho + \sqrt{d\varepsilon}$ , and this bound is tight; that is, there exists an instance  $(\mu ,\tilde{\mu}) \in \mathcal{G}_{\mathrm{cov}} \times \mathcal{P}(\mathbb{R}^d)$  with  $\mathsf{W}_1^\varepsilon (\mu ,\tilde{\mu}) \leq \rho$  such that  $\mathsf{W}_1(\check{\mu},\mu) \gtrsim \sqrt{d}\rho + \sqrt{d\varepsilon}$ . Consequently, the WDRO estimate  $\tilde{\ell} = \operatorname*{argmin}_{\ell \in \mathcal{L}}\sup_{\nu \in \mathcal{P}(\mathcal{Z}): \mathsf{W}_1(\nu ,\check{\mu}) \leq \check{\rho}}\mathbb{E}_{\nu}[\ell ]$  satisfies  $\mathbb{E}_{\mu}[\check{\ell}] - \mathbb{E}_{\mu}[\ell_{\star}] \leq \mathcal{R}_{\mu ,1}(O(\sqrt{d}\rho +\sqrt{d\varepsilon});\ell_{\star})$ .
Proof. Upper bound: For the upper bound on  $\mathsf{W}_1$  estimation error, fix any  $\tilde{\mu} \in \mathcal{P}(\mathbb{R}^d)$  with  $\mathsf{W}_1^\varepsilon (\tilde{\mu},\mu) \leq \rho$ . Take any  $\mu' \in \mathcal{P}(\mathbb{R}^d)$  such that  $\mathsf{W}_1(\mu',\mu) \leq \rho$  and  $\| \mu' - \tilde{\mu}\|_{\mathsf{TV}} \leq \varepsilon$ . By Lemma 13, there exists  $\alpha \in \mathcal{P}(\mathbb{R}^d)$  with  $\mathsf{W}_1(\alpha,\mu) \leq \rho$ ,  $\mathsf{W}_2(\alpha,\mu) \leq \rho \sqrt{2 / \varepsilon}$ , and  $\| \alpha - \tilde{\mu}\|_{\mathsf{TV}} \leq 2\varepsilon$ . Fixing an optimal coupling  $\pi \in \Pi (\alpha ,\mu)$  for the  $\mathsf{W}_2(\alpha ,\mu)$  problem and letting  $(Z,W) \sim \pi$ , we bound
$$
\| \Sigma_ {\alpha} \| _ {\mathrm {o p}} ^ {\frac {1}{2}} = \sup  _ {\theta \in \mathbb {S} ^ {d - 1}} \mathbb {E} \left[ \theta^ {\top} (Z - \mathbb {E} [ Z ]) ^ {2} \right] ^ {\frac {1}{2}} \leq \sup  _ {\theta \in \mathbb {S} ^ {d - 1}} \mathbb {E} \left[ \theta^ {\top} (Z - \mathbb {E} [ W ]) ^ {2} \right] ^ {\frac {1}{2}} \leq \| \Sigma_ {\mu} \| _ {\mathrm {o p}} ^ {\frac {1}{2}} + \mathsf {W} _ {2} (\alpha , \mu)
$$
Thus,  $\alpha \in \mathcal{G}_{\mathrm{cov}}(1 + \rho \sqrt{2 / \varepsilon})$ . Write  $\beta \coloneqq \frac{1}{(\alpha \wedge \tilde{\mu})(\mathbb{R}^d)}\alpha \wedge \tilde{\mu}$ , and note that this midpoint measure is feasible for the problem defining  $\breve{\mu}$ . Hence, we have
$$
\| \Sigma_ {\tilde {\mu}} \| _ {\mathrm {o p}} \leq \| \Sigma_ {\beta} \| _ {\mathrm {o p}} \leq \sup  _ {\theta \in \mathbb {S} ^ {d - 1}} \mathbb {E} _ {\beta} \big [ (\theta^ {\top} (Z - \mathbb {E} _ {\alpha} [ Z ])) ^ {2} \big ] \leq \frac {1}{1 - 2 \varepsilon} \| \Sigma_ {\alpha} \| _ {\mathrm {o p}} \leq \frac {1}{1 - 2 \varepsilon} (1 + \rho \sqrt {2 / \varepsilon}) ^ {2},
$$
and so  $\tilde{\mu} \in \mathcal{G}_{\mathrm{cov}}\big((1 - 2\varepsilon)^{-1/2}(1 + \rho \sqrt{2/\varepsilon})\big)$ . Moreover, we have  $\| \breve{\mu} - \alpha \|_{\mathsf{TV}} \leq 4\varepsilon$ . Thus, using Lemma 5 and the fact that  $4\varepsilon$  is bounded away from 1, we bound
$$
\mathsf {W} _ {1} (\breve {\mu}, \alpha) \leq \mathsf {W} _ {1} \left(\breve {\mu}, \frac {1}{(\breve {\mu} \wedge \alpha) (\mathcal {R} ^ {d})} \breve {\mu} \wedge \alpha\right) + \mathsf {W} _ {1} \left(\frac {1}{(\breve {\mu} \wedge \alpha) (\mathcal {R} ^ {d})} \breve {\mu} \wedge \alpha\right) \lesssim \sqrt {d} \rho + \sqrt {d \varepsilon}.
$$
By the triangle inequality, we have  $W_{1}(\check{\mu},\mu)\lesssim \sqrt{d}\rho +\sqrt{d\varepsilon}$ . The risk bound follows by Lemma 7. Taking the final error measurement using distance between means instead of  $W_{1}$ , we observe that  $\| \mathbb{E}_{\tilde{\mu}}[Z] - \mathbb{E}_{\mu}[Z]\| _2\lesssim \rho +\sqrt{\varepsilon}$
Lower bound: To see that this guarantee cannot be improved, fix clean measure  $\mu = \delta_0 \in \mathcal{G}_{\mathrm{cov}}$  and consider the corrupted measure  $\tilde{\mu} = (1 - 3\varepsilon)\delta_0 + 2\varepsilon \left(\frac{1}{2}\delta_{\frac{\rho}{2\varepsilon}}\mathbf{e}_1 + \frac{1}{2}\delta_{-\frac{\rho}{2\varepsilon}}\mathbf{e}_1\right) + \varepsilon \mathcal{N}\left(0,\frac{\rho^2}{100\varepsilon^2} I_d\right)$ , constructed so that  $\mathsf{W}_1^\varepsilon (\tilde{\mu},\mu) \leq \rho$ . Intuitively, iterative filtering seeks to drive down the operator norm of covariance matrix and will thus focus on removing mass from the second mixture component.
To formalize this, we decompose the output of filtering as  $(1 - 2\varepsilon)\breve{\mu} = (1 - 3\varepsilon -\tau)\delta_0 + \alpha +\beta$  , where  $0\leq \tau \leq 2\varepsilon$  and  $\alpha ,\beta \in \mathcal{M}_{+}(\mathbb{R}^{d})$  such that  $\alpha \leq 2\varepsilon \bigl (\frac{1}{2}\delta_{\frac{\rho}{2\varepsilon}\mathbf{e}_1} + \frac{1}{2}\delta_{-\frac{\rho}{2\varepsilon}\mathbf{e}_1}\bigr)$  and  $\beta \leq \varepsilon \mathcal{N}\big(0,\frac{\rho^2}{100\varepsilon^2} I_d\big)$  We have  $\tau +(2\varepsilon -\alpha (\mathbb{R}^d)) + (\varepsilon -\beta (\mathbb{R}^d)) = 2\varepsilon$  by the definition of  $\tilde{\mu}$  . Further note that  $\| \mathbb{E}_{\tilde{\mu}}[Z]\| \lesssim$
$\sqrt{\varepsilon} + \rho \lesssim \rho$ , by the bounds above. Now suppose for sake of contradiction that  $\beta(\mathbb{R}^d) \leq \varepsilon / 2$ . Then  $\alpha(\mathbb{R}^d) \geq \varepsilon / 2$ , and so we bound
$$
\begin{array}{l} \| \Sigma_ {\tilde {\mu}} \| _ {\mathrm {o p}} ^ {\frac {1}{2}} \geq \sqrt {\mathbb {E} _ {\tilde {\mu}} [ Z _ {1} ^ {2} ]} - \| \mathbb {E} _ {\tilde {\mu}} [ Z ] \| \\ \geq \sqrt {\frac {\varepsilon}{2 (1 - 2 \varepsilon)} \frac {\rho^ {2}}{4 \varepsilon^ {2}}} - \| \mathbb {E} _ {\tilde {\mu}} [ Z ] \| \\ \geq \frac {\rho}{\sqrt {8 \varepsilon}} - O (\rho). \\ \end{array}
$$
On the other hand, another feasible outcome for spectral reweighing is  $\mu' = \frac{1}{1 - \varepsilon} (1 - 3\varepsilon) \delta_0 + \varepsilon \mathcal{N}\left(0, \frac{\rho^2}{100\varepsilon^2} I_d\right)$ , for which we have  $\| \Sigma_{\mu'}\|_{\mathrm{op}}^{1/2} = \frac{\rho}{10\sqrt{\varepsilon}}$ . Since  $10 > \sqrt{8}$ , this contradicts optimality of  $\check{\mu}$  if  $\varepsilon \leq c\rho^2$  for a sufficiently small constant  $c$ . However, if  $\varepsilon > c\rho^2$ , then a lower bound of  $\Omega(\sqrt{d\varepsilon})$  suffices. This bound holds even without Wasserstein perturbations; see the  $\mathcal{G}_{\mathrm{cov}}$  lower risk bound of  $\Omega(\sqrt{d\varepsilon})$  in Theorem 2 of [38].
We now suppose that  $\beta (\mathbb{R}^d)\geq \varepsilon /2$ . Let  $Z\sim \mathcal{N}\big(0,\frac{\rho^2}{100\varepsilon^2}\big)$  and write  $F$  for the CDF of  $\| Z\| ^2$  (which has a scaled  $\chi_d^2$  distribution). We then have  $\int \| z\| \mathrm{d}\beta (z)\geq \varepsilon \mathbb{E}\big[\| Z\| \mid \| Z\| ^2\leq F^{-1}(1 / 2)\big]\gtrsim \sqrt{d}\rho$  using concentration of  $\chi_d^2$  about its mean. Thus,  $\mathsf{W}_1(\tilde{\mu},\mu)\geq \mathbb{E}_{\tilde{\mu}}[\| Z\| ] - \mathbb{E}_{\mu}[\| Z\| ]\gtrsim \sqrt{d}\rho .$
E Smaller Robustness Radius for Outlier-Robust WDRO (Remark 2)
In the classical WDRO setting with  $\rho_0 = \varepsilon = 0$ , the radius  $\rho$  can often be taken significantly smaller than  $n^{-1 / d}$  if  $\mathcal{L}$  and  $\mu$  are sufficiently well-behaved. In particular, when  $\mu$  satisfies a  $T_{2}$  transportation inequality, [18] proves that  $\rho = \widetilde{O} (n^{-1 / 2})$  gives meaningful risk bounds. Recall that  $\mu \in T_2(\tau)$  if
$$
\mathsf {W} _ {2} (\nu , \mu) \leq \sqrt {\tau \mathsf {H} (\nu \| \mu)}, \quad \forall \nu \in \mathcal {P} _ {2} (\mathcal {Z}),
$$
where  $\mathsf{H}(\nu \| \mu) \coloneqq \int_{\mathcal{Z}} \log (d\nu / d\mu)d\nu$  denotes relative entropy when  $\nu \ll \mu$  (and is  $+\infty$  otherwise). We note that  $T_2$  is implied by the log-Sobolev inequality, which holds for example when  $\mu$  has strongly log-concave density. Under  $T_2$ , [18] shows the following.
Proposition 11 (Example 3 in [18]). Fix  $\mathcal{Z} = \mathbb{R}^d \times \mathbb{R}$ ,  $\tau, B > 0$ , and an  $\alpha$ -smooth and  $L$ -Lipschitz function  $f: \mathbb{R} \to \mathbb{R}$ . Consider the parameterized family of loss functions  $\mathcal{L} = \{(x, y) \mapsto \ell_{\theta}(x, y) = f(\theta^\top x - y) : \theta \in \Theta\}$ , where  $\Theta \subset \{\theta \in \mathbb{R}^d : \| \theta \| \leq B\}$ . Fix  $\mu \in \mathcal{P}(\mathcal{Z})$  whose first marginal  $\mu_X = \mu(\cdot \times \mathbb{R})$  satisfies  $\mu_X \in T_2(\tau)$  and such that  $\inf_{\theta \in \Theta} \mathbb{E}_{\mu}[f'(\theta^\top X - Y)^2] > 0$ . Write
$$
\sigma = \sup  _ {\theta \in \Theta} \frac {\mathbb {E} _ {\mu} \left[ f ^ {\prime} \left(\theta^ {\top} X , Y\right) ^ {4} \right] ^ {\frac {1}{2}}}{\mathbb {E} _ {\mu} \left[ f ^ {\prime} \left(\theta^ {\top} X , Y\right) ^ {2} \right]} \leq \frac {L ^ {2}}{\inf  _ {\theta \in \Theta} \mathbb {E} _ {\mu} \left[ f ^ {\prime} \left(\theta^ {\top} X , Y\right) ^ {2} \right]} <   \infty .
$$
For  $t > 0$ , define
$$
\begin{array}{l} \rho_ {n} = \sqrt {\frac {\tau t \left(1 + d \log (2 + 2 B n)\right)}{n}} \left(1 + \sigma \sqrt {\frac {2 t (1 + d \log (2 + 2 B n))}{n}}\right), \\ \delta_ {n} = \frac {2 L + 2 B \alpha \mathbb {E} _ {\mu} [ \| X \| ] + B ^ {2} \alpha^ {2} \operatorname {V a r} _ {\mu} (\| X \|) + \rho_ {n} \sqrt {\mathbb {E} _ {\mu} \left[ (L + B \alpha \| X \|) ^ {2} \right] + \operatorname {V a r} _ {\mu} \left((L + B \alpha \| X \|) ^ {2}\right)}}{n}, \\ \end{array}
$$
$$
\eta_ {n} = \frac {2 \alpha B ^ {2} \tau t \left(1 + d \log (2 + 2 B n)\right)}{n}.
$$
Then, with probability at least  $1 - 2 / n - 2e^{-t}$ , we have
$$
\left| \mathbb {E} _ {\mu} \left[ \ell_ {\theta} \right] - \mathbb {E} _ {\hat {\mu} _ {n}} \left[ \ell_ {\theta} \right] \right| \leq \mathcal {R} _ {\hat {\mu} _ {n}, 2} \left(\rho_ {n}; \ell_ {\theta}\right) + \delta_ {n} + \eta_ {n} \quad \forall \theta \in \Theta . \tag {11}
$$
We note that (11) is stated without the absolute value on the right hand side, but that this strengthened result holds due to the discussion after [18, Theorem 1]. This generalization bound immediately gives the following excess risk bound.