SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Corollary 5. Assume  $n \geq 800$ . Fix  $\rho_n, \delta_n, \eta_n$ , and  $\mathcal{L}$  as in Proposition 11 with  $t = 7$ , and take  $\ell_{\hat{\theta}} \in \mathcal{L}$  minimizing (1) with  $p = 2$  and radius  $\rho = \rho_n$ . Then, with probability at least 0.99, we have
$$
\mathbb {E} _ {\mu} [ \ell_ {\hat {\theta}} ] - \mathbb {E} _ {\mu} [ \ell_ {\theta} ] \lesssim \rho_ {n} \| \ell_ {\theta} \| _ {\dot {H} ^ {1, 2} (\mu)} + \alpha \rho_ {n} ^ {2} + \delta_ {n} + \eta_ {n} \quad \forall \theta \in \Theta .
$$
Proof. By Proposition 11 and [18, Remark 1], we have
$$
\mathbb {E} _ {\mu} [ \ell_ {\hat {\theta}} ] - \mathbb {E} _ {\mu} [ \ell_ {\theta} ] \leq 2 \mathcal {R} _ {\hat {\mu} _ {n}, 2} (\rho_ {n}; \ell_ {\theta}) + 2 \delta_ {n} + 2 \eta_ {n} \quad \forall \theta \in \Theta ,
$$
with probability at least  $1 - 2 / n - 2e^{-t} \geq 0.995$ . Since  $\ell_{\theta}$  is  $\alpha$ -smooth, Lemma 6 gives that
$$
\mathbb {E} _ {\mu} [ \ell_ {\hat {\theta}} ] - \mathbb {E} _ {\mu} [ \ell_ {\theta} ] \leq 2 \rho_ {n} \| \ell_ {\theta} \| _ {\dot {H} ^ {1, 2} (\hat {\mu} _ {n})} + 2 \alpha \rho_ {n} ^ {2} + 2 \delta_ {n} + 2 \eta_ {n} \quad \forall \theta \in \Theta ,
$$
with probability at least 0.995. By Markov's inequality, we can substitute  $\hat{\mu}_n$  with  $\mu$  at the cost of a constant factor blow-up in excess risk and a decrease in the confidence probability to, say, 0.99.
In the example above, excess risk is controlled by the 2-Wasserstein regularizer with radius  $\rho_{n} = O(n^{-1 / 2})$ , up to  $O(n^{-1})$  correction terms, which is significantly smaller than the typical radius size of  $O(n^{-1 / d})$ . We shall now lift this improvement to the outlier-robust setting. Similar to Proposition 4, we perform outlier-robust DRO with a modified choice of  $\mathcal{A}$ . This time, writing  $\mathcal{G}_2(\sigma) = \cup_{z_0\in \mathcal{Z}}\mathcal{G}_2(\sigma ,z_0)$ , we have the following.
Proposition 12 (Outlier-robust WDRO under  $T_{2}$ ). Assume  $n \geq 800$  and  $\mu \in \mathcal{G}_{\mathrm{cov}}$ . Fix  $\rho_{n}, \delta_{n}, \eta_{n}$  and  $\mathcal{L}$  as in Proposition 11 with  $t = 8$ , and take  $\ell_{\hat{\theta}}$  minimizing (OR-WDRO) with center  $\tilde{\mu}_n$ , radius  $\rho = \rho_0 + 15\rho_n + 200\sqrt{d}$ , and  $\mathcal{A} = \mathcal{G}_2(15\sqrt{d} + \rho_n)$ . Then, with probability at least 0.99, we have
$$
\mathbb {E} _ {\mu} [ \ell_ {\hat {\theta}} ] - \mathbb {E} _ {\mu} [ \ell_ {\theta} ] \lesssim \| \ell_ {\theta} \| _ {\dot {H} ^ {1, 2} (\mu)} \big (\rho_ {0} + \rho_ {n} + \sqrt {d} \big) + \alpha \big (\rho_ {0} + \rho_ {n} + \sqrt {d} \big) ^ {2} + \delta_ {n} + \eta_ {n} \quad \forall \theta \in \Theta .
$$
Proof. Noting that  $\mathcal{G}_{\mathrm{cov}} \subseteq \mathcal{G}_2(\sqrt{d})$ , we have by Markov's inequality that  $\hat{\mu}_n \in \mathcal{G}_2(15\sqrt{d})$  with probability at least 0.995. In other words, there exists  $z_0 \in \mathcal{Z}$  such that  $\mathsf{W}_2(\hat{\mu}_n, \delta_{z_0}) \leq 15\sqrt{d}$ . Thus, for any  $\nu \in \mathcal{P}(\mathcal{Z})$  with  $\mathsf{W}_2(\hat{\mu}_n, \nu) \leq \rho_n$ , we have  $\mathsf{W}_2(\nu, \delta_{z_0}) \leq 15\sqrt{d} + \rho_n$ , and so  $\nu \in \mathcal{G}_2(15\sqrt{d} + \rho_n)$ . By Lemmas 4 and 5, this implies that
$$
\begin{array}{l} \mathsf {W} _ {2} ^ {\varepsilon} (\tilde {\mu} _ {n} \| \nu) \leq \mathsf {W} _ {2} ^ {\varepsilon} (\tilde {\mu} _ {n}, \nu) + \tau_ {2} (\nu , \varepsilon) \\ \leq \rho_ {0} + \rho_ {n} + 8 (1 5 \sqrt {d} + \rho_ {n}) (1 - \varepsilon) ^ {- 1 / 2} \\ <   \rho . \\ \end{array}
$$
Next, by Proposition 11, with probability at least  $1 - 1/400 + 2e^{-8} \geq 0.9985$ , we have for each  $\theta \in \Theta$  that
$$
\begin{array}{l} \mathbb {E} _ {\mu} [ \ell_ {\hat {\theta}} ] - E _ {\mu} [ \ell_ {\theta} ] \leq \mathbb {E} _ {\hat {\mu} _ {n}} [ \ell_ {\hat {\theta}} ] + \mathcal {R} _ {\hat {\mu} _ {n}, 2} (\rho_ {n}; \ell_ {\hat {\theta}}) - \mathbb {E} _ {\mu} [ \ell_ {\theta} ] + \delta_ {n} + \eta_ {n} \\ = \mathbb {E} _ {\hat {\mu} _ {n}} [ \ell_ {\hat {\theta}} ] + \left(\sup _ {\substack {\nu \in \mathcal {P} (\mathcal {Z}) \\ \mathsf {W} _ {2} (\hat {\mu} _ {n}, \nu) \leq \rho_ {n}}} \mathbb {E} _ {\nu} [ \ell_ {\hat {\theta}} ] - \mathbb {E} _ {\hat {\mu} _ {n}} [ \ell_ {\hat {\theta}} ]\right) - \mathbb {E} _ {\mu} [ \ell_ {\theta} ] + \delta_ {n} + \eta_ {n} \\ = \mathbb{E}_{\hat{\mu}_{n}}[\ell_{\hat{\theta}}] + \left(\sup_{\substack{\nu \in \mathcal{G}_{2}(15\sqrt{d} +\rho_{n})\\ W_{2}(\hat{\mu}_{n},\nu)\leq \rho_{n}}}\mathbb{E}_{\nu}[\ell_{\hat{\theta}}] - \mathbb{E}_{\hat{\mu}_{n}}[\ell_{\hat{\theta}}]\right) - \mathbb{E}_{\mu}[\ell_{\theta}] + \delta_{n} + \eta_{n} \\ \leq \mathbb{E}_{\hat{\mu}_{n}}[\ell_{\hat{\theta}}] + \left(\sup_{\substack{\nu \in \mathcal{G}_{2}(15\sqrt{d} +\rho_{n})\\ \mathsf{W}_{2}^{\varepsilon}(\tilde{\mu}_{n}\| \nu)\leq \rho}}\mathbb{E}_{\nu}[\ell_{\hat{\theta}}] - \mathbb{E}_{\hat{\mu}_{n}}[\ell_{\hat{\theta}}]\right) - \mathbb{E}_{\mu}[\ell_{\theta}] + \delta_{n} + \eta_{n}. \\ \end{array}
$$
Let  $c = 2\left(\frac{1 - \varepsilon}{1 - 2\varepsilon}\right)^{1 / p}$ . Using optimality of  $\hat{\ell}$  and Lemma 2, we further bound
$$
\begin{array}{l} \mathbb{E}_{\mu}[\ell_{\hat{\theta}}] - E_{\mu}[\ell_{\theta}]\leq \left(\sup_{\substack{\nu \in \mathcal{G}_{2}(15\sqrt{d} +\rho_{n})\\ \mathsf{W}_{2}^{\varepsilon}(\tilde{\mu}_{n}\| \nu)\leq \rho}}\mathbb{E}_{\nu}[\ell_{\theta}] - \mathbb{E}_{\hat{\mu}_{n}}[\ell_{\theta}]\right) + \mathbb{E}_{\hat{\mu}_{n}}[\ell_{\theta}] - \mathbb{E}_{\mu}[\ell_{\theta}] + \delta_{n} + \eta_{n} \\ \leq \left(\sup_{\substack{\nu \in \mathcal{G}_{2}(15\sqrt{d} +\rho_{n})\\ \mathsf{W}_{2}^{\varepsilon}(\tilde{\mu}_{n}\| \nu)\leq \rho}}\mathbb{E}_{\nu}[\ell_{\theta}] - \mathbb{E}_{\hat{\mu}_{n}}[\ell_{\theta}]\right) + \mathcal{R}_{\hat{\mu}_{n},2}(\rho_{n};\ell_{\theta}) + 2\delta_{n} + 2\eta_{n} \\ \leq \left(\sup_{\substack{\nu \in \mathcal{G}_{2}(15\sqrt{d} +\rho_{n})\\ W_{2}^{2\varepsilon}(\hat{\mu}_{n},\nu)\leq c\rho}}\mathbb{E}_{\nu}[\ell_{\theta}] - \mathbb{E}_{\hat{\mu}_{n}}[\ell_{\theta}]\right) + \mathcal{R}_{\hat{\mu}_{n},2}(\rho_{n};\ell_{\theta}) + 2\delta_{n} + 2\eta_{n} \\ \leq \left(\sup_{\substack{\nu \in \mathcal{G}_{2}(15\sqrt{d} +\rho_{n})\\ W_{2}(\nu ,\hat{\mu}_{n})\leq c\rho +\tau_{2}(\hat{\mu}_{n},2\varepsilon) + \tau_{2}(\mathcal{A},2\varepsilon)}}\mathbb{E}_{\nu}[\ell_{\theta}] - \mathbb{E}_{\hat{\mu}_{n}}[\ell_{\theta}]\right) + \mathcal{R}_{\hat{\mu}_{n},2}(\rho_{n};\ell_{\theta}) + 2\delta_{n} + 2\eta_{n} \\ \leq \mathcal {R} _ {\hat {\mu} _ {n} 2} \left(c \rho + \tau_ {2} (\hat {\mu} _ {n}, 2 \varepsilon) + \tau_ {2} (\mathcal {A}, 2 \varepsilon); \ell_ {\theta}\right) + \mathcal {R} _ {\hat {\mu} _ {n}, 2} \left(\rho_ {n}; \ell_ {\theta}\right) + 2 \delta_ {n} + 2 \eta_ {n} \\ \lesssim \left\| \ell_ {\theta} \right\| _ {\dot {H} ^ {1, 2} (\hat {\mu} _ {n})} \left(\rho_ {0} + \rho_ {n} + \sqrt {d}\right) + \alpha \left(\rho_ {0} + \rho_ {n} + \sqrt {d}\right) ^ {2} + \delta_ {n} + \gamma_ {n}. \\ \end{array}
$$
As in Corollary 5, we can substitute  $\hat{\mu}_n$  with  $\mu$  at the cost of a constant factor increase in excess risk along with a small decrease in the confidence probability (in this case, sufficiently small such that the total failure probability is at most 0.01).
The main goal of Proposition 12 was to demonstrate that one can expect improved excess risk bounds for outlier-robust WDRO in situations where such improvements hold for standard WDRO. We conjecture that similar guarantees hold for additional settings and under milder assumptions like the  $T_{1}$  inequality, but leave such refinements for future work. In particular, for the class  $\mathcal{G}_{\mathrm{cov}}$ , it would be desirable to prove such bounds when  $p = 1$ , so that the TV contribution to the risk vanishes as  $\varepsilon \to 0$ .
F Parameter Tuning (Remark 5)
To clarify the parameter selection process, we consider Setting B with the class  $\mathcal{G} = \mathcal{G}_{\mathrm{cov}}(\sigma)$ ,  $p = 1$ , and  $\varepsilon \leq 1/3$ . We aim to efficiently achieve excess risk
$$
\mathbb {E} _ {\mu} [ \hat {\ell} ] - \mathbb {E} _ {\mu} [ \ell_ {\star} ] \lesssim \| \ell_ {\star} \| _ {\operatorname {L i p}} \left(\rho_ {0} + \sigma \sqrt {d \varepsilon} + \sigma \sqrt {d} n ^ {- 1 / d}\right), \tag {12}
$$
matching Proposition 4, when
$$
\hat {\ell} = \operatorname {a r g m i n} _ {\ell \in \mathcal {L}} \sup  _ {\nu \in \mathcal {G} _ {2} (\hat {\sigma}, z _ {0}): \mathbb {W} _ {1} ^ {\hat {\rho}} (\tilde {\mu} _ {n}, \nu) \leq \hat {\rho}} \mathbb {E} _ {\nu} [ \ell ] \tag {13}
$$
for some parameter guesses  $\hat{\sigma}$ ,  $\hat{\varepsilon}$ , and  $\hat{\rho}$ , and a robust mean estimate  $z_0$ . First, we observe that the coordinate-wise trimmed mean estimate from Proposition 5 is computed without knowledge of the parameters, so it is safe to assume that  $\| z_0 - \mathbb{E}_\mu [Z]\| \lesssim \sqrt{d} +\rho_0$ . If the parameter guesses are conservative, i.e.,  $\hat{\sigma}\geq \sigma ,\hat{\varepsilon}\geq \varepsilon$ , and  $\hat{\rho}\geq \rho_0 + \mathsf{W}_1(\mu ,\hat{\mu}_n)$ , then we may still employ Proposition 4. If they are not too large, i.e.,  $\hat{\sigma}\lesssim \sigma ,\hat{\varepsilon}\lesssim \varepsilon$ , and  $\hat{\rho}\leq \rho_0 + \sigma \sqrt{dn^{-1 / d}}$ , this gives the desired excess risk.
We now explore what prior knowledge of  $\sigma$ ,  $\varepsilon$ , and  $\rho_0$  is needed to obtain such guesses. First, we show that effective learning is impossible without knowledge of  $\rho_0$ , even for standard WDRO (i.e.,  $\varepsilon = 0$ ,  $\sigma = \infty$ ). For ease of presentation, we present the following lower bound without sampling.
Lemma 16. There exists a family of loss functions  $\mathcal{L}$  over  $\mathbb{R}$  such that, for any  $C > 0$  and decision rule  $\mathrm{D}:\mathcal{P}(\mathbb{R})\to \mathcal{L}$ , there are  $\mu ,\tilde{\mu}\in \mathcal{P}(\mathbb{R})$  such that  $\mathbb{E}_{\mu}[\mathsf{D}(\tilde{\mu})] > C(\inf_{\ell \in \mathcal{L}}\mathbb{E}_{\mu}[\ell ] + \mathsf{W}_1(\mu ,\tilde{\mu})\| \ell \|_{\mathrm{Lip}})$ .
Proof. Let  $\mathcal{L} = \{\ell_{\theta} : \theta > 0\}$ , where  $\ell_{\theta}(z) \coloneqq z / \theta + \theta$ . By design, we have  $\|\ell_{\theta}\|_{\mathrm{Lip}} = 1 / \theta$ . Let  $\tilde{\mu} = \delta_0$  and write  $\mathsf{D}(\tilde{\mu}) = \ell_{\hat{\theta}}$  for some  $\hat{\theta} > 0$ . We then set  $\mu = \delta_{\rho}$  for  $\rho = 10\hat{\theta}^2 C^2$ . This gives
$$
\mathbb {E} _ {\mu} [ \mathsf {D} (\tilde {\mu}) ] = \ell_ {\hat {\theta}} (\rho) = \frac {\rho}{\hat {\theta}} + \hat {\theta} > \frac {\rho}{\hat {\theta}} = 1 0 C ^ {2} \hat {\theta},
$$
$$
\inf _ {\ell \in \mathcal {L}} \mathbb {E} _ {\mu} [ \ell ] + \mathsf {W} _ {1} (\mu , \tilde {\mu}) \| \ell \| _ {\mathrm {L i p}} = \inf _ {\theta > 0} \ell_ {\theta} (\rho) + \frac {\rho}{\theta} = \inf _ {\theta > 0} \frac {2 \rho}{\theta} + \theta = 2 \sqrt {2 \rho} = \sqrt {8 0} C \hat {\theta}.
$$
Thus, we have  $\mathbb{E}_{\mu}[\mathsf{D}(\tilde{\mu})] > \inf_{\ell \in \mathcal{L}}\mathbb{E}_{\mu}[\ell ] + \mathsf{W}_1(\mu ,\tilde{\mu})\| \ell \|_{\mathrm{Lip}}$  , as desired.
Thus, we assume in what follows that  $\hat{\rho} = \rho_0$  is known. Moreover, we require knowledge of at least one of  $\varepsilon$  and  $\sigma$ . If both are unknown, then is information theoretically impossible to meaningfully distinguish inliers from outliers (see Exercise 1.7b of [16] for a discussion of this issue in the setting of robust mean estimation). If  $\varepsilon$  is known, then we can choose  $\hat{\sigma}$  as  $2^i$  for the smallest  $i$  such that the supremum of Eq. (13) is feasible (or, equivalently, such that the associated dual is bounded for some fixed  $\ell \in \mathcal{L}$ ). We can overshoot by at most a factor of two and thus achieve the desired risk bound. Using binary search, this adds a multiplicative overhead logarithmic in the ratio of the initial guess for  $\sigma$  and its true value. The same approach can be employed if  $\sigma$  is known but not  $\varepsilon$ .
We now provide several experiments in addition to those in the main body. Code is again provided at https://github.com/sbnietert/outlier-robust-WDRO.
First, in Fig. 3, we extend the experiment summarized Fig. 2 (top) to include runs with  $\hat{\varepsilon} = \varepsilon$  and varied Wasserstein radius  $\hat{\rho}\in \{\rho /2,\rho ,2\rho \}$ . For this simple learning problem and perturbation model, we find that the choice of  $\hat{\rho}$  plays a minor role in the resulting excess risk. We emphasize that, in the worst case, selection of  $\hat{\rho}$  can be critical, as demonstrated by Lemma 16.
Next, we consider linear classification with the hinge loss, i.e.  $\mathcal{L} = \{\ell_{\theta}(x,y) = \max \{0,1 - y(\theta^{\top}x)\} : \theta \in \mathbb{R}^{d - 1}\}$ . This time (to ensure that the resulting optimization problem is convex), our approach supports Euclidean Wasserstein perturbations in the feature space, but no Wasserstein perturbations in the label space (this corresponds to using  $\mathcal{Z} = \mathbb{R}^{d - 1} \times \mathbb{R}$  equipped with the (extended) norm  $\| (x,y)\| = \| x\|_2 + \infty \cdot \mathbb{1}\{y \neq 0\}$ . We consider clean data  $(X,Y = \mathrm{sign}(\theta_{\star}^{\top}X)) \sim \mu$ , where  $X \sim \mathcal{N}(0,I_{d - 1})$ . The corrupted data  $(\tilde{X},\tilde{Y}) \sim \tilde{\mu}$  satisfies  $(\tilde{X},\tilde{Y}) = (X + \rho e_1,Y)$  with probability  $1 - \varepsilon$  and  $(\tilde{X},\tilde{Y}) = (100X, -Y)$  with probability  $\varepsilon$ , so that  $\mathsf{W}_p^\varepsilon (\tilde{\mu}\| \mu) \leq \rho$ . In Figure 4 (left), we fix  $d = 10$  and compare the excess risk  $\mathbb{E}_{\mu}[\ell_{\hat{\theta}}] - \mathbb{E}_{\mu}[\ell_{\theta_{\star}}]$  of standard
Figure 3: Excess risk of standard WDRO and outlier-robust WDRO for linear regression under  $W_{p}$  and TV corruptions, with varied sample size and dimension.
WDRO and outlier-robust WDRO with  $\mathcal{G} = \mathcal{G}_2$ , as described by Proposition 4 and implemented via Theorem 2. The results are averaged over  $T = 20$  runs for sample size  $n \in \{10, 20, 50, 75, 100\}$ . We note that this example cannot drive the excess risk of standard WDRO to infinity, so the separation between standard and outlier-robust WDRO is less striking than regression, though still present.
We further present results for multivariate regression. This time, we consider  $\mathcal{Z} = \mathbb{R}^{d\times k}$  equipped with the  $\ell_2$  norm, and use the loss family  $\mathcal{L} = \{\ell_M(x,y) = \| Mx - y\| _1:M\in \mathbb{R}^{k\times d}\}$ . We consider clean data  $(X,Y = M_{\star}^{\top}X)\sim \mu$ , where  $M_{\star}\in \mathbb{R}^{k\times d}$  and  $X$  have standard normal entries. The corrupted data  $(\tilde{X},\tilde{Y})\sim \tilde{\mu}$  satisfies  $(\tilde{X},\tilde{Y}) = (X + \rho e_1,Y)$  with probability  $1 - \varepsilon$  and  $(\tilde{X},\tilde{Y}) = (10X, - 100M_{\star}X)$  with probability  $\varepsilon$ , so that  $\mathsf{W}_p^\varepsilon (\tilde{\mu}\| \mu)\leq \rho$ . In Figure 4 (right), we fix  $d = 10$  and  $k = 3$ , and compare the excess risk  $\mathbb{E}_{\mu}[\ell_{\hat{M}}] - \mathbb{E}_{\mu}[\ell_{M_{\star}}]$  of standard WDRO and outlier-robust WDRO with  $\mathcal{G} = \mathcal{G}_2$ , as described by Proposition 4 and implemented via Theorem 2. The results are averaged over  $T = 10$  runs for sample size  $n\in \{10,20,50,75,100\}$ . We are restricted to low  $k$  since the  $\ell_1$  norm in the losses is expressed as the maximum of  $2^{k}$  concave functions (specifically, we use that  $\ell_M(x,y) = \max_{\alpha \in \{-1,1\}^k}\alpha^\top (Mx - y)$ ).
Finally, we turn to a classification task with image data. We train a robust linear classifier to distinguish
Figure 4: Excess risk of standard WDRO and outlier-robust WDRO for classification and multivariate linear regression under  $W_{p}$  and TV corruptions, with varied sample size.
between the MNIST [14] digits "3" and "8" when  $10\%$  of training labels are flipped uniformly at random, using outlier-robust WDRO with  $\mathcal{G} = \mathcal{G}_2$  and the hinge loss, as above, applied with  $\varepsilon = 0.1$  and  $\rho = 0.01$ . To ensure tractability, we first pass the images through principal component analysis to reduce the input to 50 dimensions, and we again use Theorem 2 for implementation. In Fig. 5, we plot the classification accuracy of our robust classifier compared to one learned via standard WDRO for training set size  $n \in \{10, 20, 50, 100, 150\}$ , averaged over  $T = 10$  runs. In this case, we do not witness a noticeable improvement over standard WDRO. We suspect that the relevant  $\mathcal{G}_2$  class is too large to be of significant use in this high-dimensional setting.
For all experiments, confidence bands are plotted representing the top and bottom  $10\%$  quantiles
Figure 5: Classification accuracy of standard WDRO and outlier-robust WDRO predictors for MNIST digit classification under random label flips, with varied number of training samples.
among 100 bootstrapped means from the  $T$  runs. The additional experiments were performed on an M1 Macbook Air with 16GB RAM in roughly 30 minutes each.