SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} _ {\mu} [ \hat {\ell} ] - \inf  _ {\ell \in \mathcal {L}} \mathbb {E} _ {\mu} [ \ell ] = \mathbb {E} _ {\mu} [ \hat {\ell} ] \gtrsim L \max  \{\rho , \sqrt {d \varepsilon} \} \asymp L \big (\rho + \sqrt {d \varepsilon} \big).
$$
Since  $\mu \in \mathcal{G}_{\mathrm{cov}}$  , we have
$$
\begin{array}{l} \mathbb {E} _ {\mu} \left[ \| Z - z _ {0} \| ^ {2} \right] ^ {\frac {1}{2}} \leq \mathbb {E} _ {\mu} [ \| Z - \mathbb {E} _ {\mu} [ Z ] \| ^ {2} ] ^ {\frac {1}{2}} + \| \mathbb {E} _ {\mu} [ Z ] - z _ {0} \| \\ = \operatorname {t r} \left(\Sigma_ {\mu}\right) ^ {\frac {1}{2}} + \| \mathbb {E} _ {\mu} [ Z ] - z _ {0} \| \\ \leq \sqrt {d} + \| \mathbb {E} _ {\mu} [ Z ] - z _ {0} \| \leq \sigma . \\ \end{array}
$$
Consequently, we have  $\mu \in \mathcal{G}_2(\sigma, z_0)$ . Next, we bound
$$
\begin{array}{l} \mathsf{W}_{p}^{\varepsilon}(\tilde{\mu}_{n}\| \mu) := \inf_{\substack{\nu \in \mathcal{P}(\mathcal{Z})\\ \nu \leq \frac{1}{1 - \varepsilon}\tilde{\mu}_{n}}}\mathsf{W}_{p}(\nu ,\mu) \\ \leq \inf_{\substack{\nu ,\mu^{\prime}\in \mathcal{P}(\mathcal{Z})\\ \nu \leq \frac{1}{1 - \varepsilon}\tilde{\mu}_{n}\\ \mu^{\prime}\leq \frac{1}{1 - \varepsilon}\mu}}\mathsf{W}_{p}(\nu ,\mu^{\prime}) + \mathsf{W}_{p}(\mu^{\prime},\mu) \\ \leq \inf  _ { \begin{array}{c} \nu , \mu^ {\prime} \in \mathcal {P} (\mathcal {Z}) \\ \nu \leq \frac {1}{1 - \varepsilon} \tilde {\mu} _ {n} \\ \mu^ {\prime} \leq \frac {1}{1 - \varepsilon} \mu \end{array} } \mathsf {W} _ {p} (\nu , \mu^ {\prime}) + \tau_ {p} (\mu , \varepsilon) \qquad \qquad \qquad \text {(d e f i n i t i o n o f} \tau_ {p}) \\ \end{array}
$$
$$
\begin{array}{l} = (1 - \varepsilon) ^ {- \frac {1}{p}} \mathsf {W} _ {p} ^ {\varepsilon} (\tilde {\mu} _ {n}, \mu) + \tau_ {p} (\mu , \varepsilon) \tag {Lemma 1} \\ \leq (1 - \varepsilon) ^ {- \frac {1}{p}} (\rho_ {0} + \delta) + \tau_ {p} (\mathcal {G} _ {\mathrm {c o v}}, \varepsilon) = \rho . \\ \end{array}
$$
Writing  $\mathcal{G}' = \mathcal{G}_2(\sigma, z_0)$  and mirroring the proof of Theorem 1, we have for each  $\ell \in \mathcal{L}$  that
$$
\begin{array}{l} \mathbb{E}_{\mu}[\hat{\ell} ] - \mathbb{E}_{\mu}[\ell ]\leq \sup_{\substack{\nu \in \mathcal{G}^{\prime}\\ \mathsf{W}_{p}^{\varepsilon}(\tilde{\mu}_{n}\| \nu)\leq \rho}}\mathbb{E}_{\nu}[\hat{\ell} ] - \mathbb{E}_{\mu}[\ell ] \\ \leq \sup_{\substack{\nu \in \mathcal{G}^{\prime}\\ \mathsf{W}_{p}^{\varepsilon}(\tilde{\mu}_{n}\| \nu)\leq \rho}}\mathbb{E}_{\nu}[\ell ] - \mathbb{E}_{\mu}[\ell ]\qquad (\hat{\ell}\text{optimal for (4)}) \\ \leq \sup  _ {\substack {\nu \in \mathcal {G} ^ {\prime} \\ \mathsf {W} _ {p} ^ {2 \varepsilon} (\nu , \mu) \leq 2 \rho}} \mathbb {E} _ {\nu} [ \ell ] - \mathbb {E} _ {\mu} [ \ell ] (Lemma 2) \\ \leq \sup  _ {\substack {\nu \in \mathcal {G} ^ {\prime} \\ \mathsf {W} _ {p} (\nu , \mu) \leq c \rho + 2 \tau_ {p} (\mathcal {G} ^ {\prime}, 2 \varepsilon)}} \mathbb {E} _ {\nu} [ \ell ] - \mathbb {E} _ {\mu} [ \ell ] (Lemma 3) \\ \leq \sup_{\substack{\nu \in \mathcal{P}(\mathcal{Z})\\ \mathsf{W}_{p}(\nu ,\mu)\leq c\rho +2\tau_{p}(\mathcal{G}^{\prime},2\varepsilon)}}\mathbb{E}_{\nu}[\ell ] - \mathbb{E}_{\mu}[\ell ] \\ = \mathcal {R} _ {\mu , p} (c \rho + 2 \tau_ {p} (\mathcal {G} ^ {\prime}, 2 \varepsilon); \ell) \\ \leq \mathcal {R} _ {\mu , p} \left(c \rho + 8 \sigma (2 \varepsilon) ^ {\frac {1}{p} - \frac {1}{2}} (1 - 2 \varepsilon) ^ {- \frac {1}{p}}; \ell\right). (Lemma 5) \\ \end{array}
$$
When  $p = 1$ , we bound the regularizer radius by
$$
c \rho + 8 \sigma \sqrt {2 \varepsilon} (1 - 2 \varepsilon) ^ {- 1} \lesssim \rho_ {0} + \delta + \tau_ {1} (\mathcal {G} _ {\mathrm {c o v}}, \varepsilon) + (\sqrt {d} + \rho_ {0}) \sqrt {\varepsilon} \lesssim \rho_ {0} + \delta + \sqrt {d \varepsilon},
$$
using Lemma 5. Similarly, when  $p = 2$ , we bound the radius by
$$
c \rho + 8 \sigma (1 - 2 \varepsilon) ^ {- \frac {1}{2}} \lesssim \rho_ {0} + \delta + \tau_ {2} (\mathcal {G} _ {\mathrm {c o v}}, \varepsilon) + (\sqrt {d} + \rho_ {0}) \lesssim \rho_ {0} + \delta + \sqrt {d}.
$$
Taking  $\ell = \ell_{\star}$  and applying Lemma 7 gives the proposition.
B.6 Proof of Proposition 5
To start, we fix  $d = 1$ . Given  $0 \leq \gamma < 1/2$  and  $\nu \in \mathcal{P}(\mathbb{R})$  with cumulative distribution function (CDF)  $F_{\nu}$ , define the  $\gamma$ -trimming  $\mathsf{T}_{\gamma}(\nu) \in \mathcal{P}(\mathbb{R})$  as the law of  $F_{\nu}^{-1}(U)$ , where  $U \sim \mathrm{Unif}([\gamma, 1 - \gamma])$ , and let  $m_{\gamma}(\nu) \coloneqq \mathbb{E}_{T_{\gamma}(\nu)}[Z]$  denote the  $\gamma$ -trimmedmed mean. If  $\nu = \frac{1}{|A|} \sum_{a \in A} \delta_{a}$  is uniform over a finite set  $A = \{a_1 < a_2 < \dots < a_n\}$  and  $\gamma n$  is an integer, we have  $m_{\gamma}(\nu) = \frac{1}{(1 - 2\gamma)n} \sum_{i = \gamma n + 1}^{(1 - \gamma)n} a_i$ .
Our robust mean estimate when  $d = 1$  is  $z_{0} = m_{\gamma}(\tilde{\mu}_{n})$  with  $\gamma = 1/3$ . The smaller choice of  $\gamma = \varepsilon$  gives tighter guarantees at the cost of increased sample complexity; we keep the larger choice since we only require a coarse estimate.
Lemma 8. Consider Setting  $B$  with  $d = 1$ ,  $\mathcal{G} = \mathcal{G}_{\mathrm{cov}}$ ,  $\rho_0 = 0$ ,  $\varepsilon \leq 1/3$ . Fix sample size  $n = \Omega (\log (1 / \delta))$ , for  $0 < \delta < 1/2$ . Then,  $\| m_{\gamma}(\tilde{\mu}_n) - \mathbb{E}_\mu [Z]\| \lesssim 1$  with probability at least  $1 - \delta$ .
Proof. This follows by Proposition 1.18 of [16] applied to the distribution  $\mu$  with corruption fraction  $\gamma, \varepsilon' = 4\gamma/3 < 1/2$ , and resilience bound  $\tau(\mathcal{G}_{\mathrm{cov}}, 2\varepsilon') \lesssim \sqrt{\gamma} \lesssim 1$ .
Now, since we are free to permute the order of the TV and  $\mathsf{W}_p$  corruptions (see Lemma 1), there exist  $\{W_i\}_{i=1}^n \subseteq \mathbb{R}$  with empirical measure  $\nu_n \in \mathcal{P}(\mathbb{R})$  such that  $\| \nu_n - \hat{\mu}_n \|_{\mathsf{TV}} \leq \varepsilon$  and  $\mathsf{W}_p(\nu_n, \tilde{\mu}_n) \leq \rho_0$ . By Lemma 8, we have  $|m_\gamma(\nu_n) - \mathbb{E}_\mu[Z]| \lesssim 1$ . Of course, we do not observe  $\nu_n$ , so this result is not immediately useful. To apply this fact, we use that  $\mathsf{T}_\gamma$  is an approximate Wasserstein contraction.
Lemma 9. If  $\alpha, \beta \in \mathcal{P}(\mathbb{R})$  and  $0 \leq \gamma < 1/2$ , then  $\mathsf{W}_p(\mathsf{T}_{\gamma}(\alpha), \mathsf{T}_{\gamma}(\beta)) \leq (1 - 2\gamma)^{-1/p}\mathsf{W}_p(\alpha, \beta)$ .
Proof. Writing  $F_{\alpha}$  and  $F_{\beta}$  for the CDFs of  $\alpha$  and  $\beta$ , respectively, we compute
$$
\mathrm {W} _ {p} \left(\mathrm {T} _ {\gamma} (\alpha), \mathrm {T} _ {\gamma} (\beta)\right) ^ {p} = \frac {1}{1 - 2 \gamma} \int_ {\gamma} ^ {1 - \gamma} \left| F _ {\alpha} ^ {- 1} (t) - F _ {\beta} ^ {- 1} (t) \right| ^ {p} \mathrm {d} t
$$
$$
\leq \frac {1}{1 - 2 \gamma} \int_ {0} ^ {1} | F _ {\alpha} ^ {- 1} (t) - F _ {\beta} ^ {- 1} (t) | ^ {p} \mathrm {d} t = \frac {1}{1 - 2 \gamma} \mathsf {W} _ {p} (\alpha , \beta) ^ {p},
$$
Applying Lemma 9, we bound
$$
\begin{array}{l} \left| m _ {\gamma} \left(\tilde {\mu} _ {n}\right) - \mathbb {E} _ {\mu} [ Z ] \right| \leq \left| m _ {\gamma} \left(\nu_ {n}\right) - \mathbb {E} _ {\mu} [ Z ] \right| + \left| m _ {\gamma} \left(\nu_ {n}\right) - m _ {\gamma} \left(\tilde {\mu} _ {n}\right) \right| \\ \leq \left| m _ {\gamma} \left(\nu_ {n}\right) - \mathbb {E} _ {\mu} [ Z ] \right| + \mathrm {W} _ {1} \left(\mathrm {T} _ {\gamma} \left(\nu_ {n}\right), \mathrm {T} _ {\gamma} \left(\tilde {\mu} _ {n}\right)\right) \\ \lesssim 1 + W _ {1} \left(\nu_ {n}, \tilde {\mu} _ {n}\right) \\ \lesssim 1 + \rho_ {0}. \\ \end{array}
$$
This matches the proposition statement when  $d = 1$ .
For general  $d > 1$ , we propose the coordinate-wise trimmed estimate  $z_0 \in \mathbb{R}^d$  given by  $(z_0)_i = m_\gamma(\mathbf{e}_i^\top \tilde{\mu}_n)$ . Plugging in  $\delta \gets 1/(100d)$  into Lemma 8 and taking a union bound over coordinates, we condition on the 0.99 probability event that the one-dimensional bound holds for all coordinates. We can then bound
$$
\begin{array}{l} \left\| z _ {0} - \mathbb {E} _ {\mu} [ Z ] \right\| ^ {2} = \sum_ {i = 1} ^ {d} \left(m _ {\gamma} \left(\mathbf {e} _ {i} ^ {\top} \# \tilde {\mu} _ {n}\right) - \mathbb {E} _ {\mu} \left[ \mathbf {e} _ {i} ^ {\top} Z \right]\right) ^ {2} \\ \lesssim d + \sum_ {i = 1} ^ {d} \mathsf {W} _ {1} (\mathbf {e} _ {i \#} ^ {\top} \nu_ {n}, \mathbf {e} _ {i \#} ^ {\top} \tilde {\mu} _ {n}) ^ {2} \\ \leq d + \mathsf {W} _ {1} (\nu_ {n}, \tilde {\mu} _ {n}) ^ {2} \\ \leq d + \rho_ {0} ^ {2}, \\ \end{array}
$$
as desired. The penultimate inequality is a consequence of the reverse Minkowski inequality.
Lemma 10. Fix  $\alpha, \beta \in \mathcal{P}(\mathbb{R}^d)$ . Write  $\alpha_i = \mathbf{e}_i^\top_{\#} \alpha$  and  $\beta_i = \mathbf{e}_i^\top_{\#} \beta$ ,  $i \in [d]$ , for their coordinate-wise marginals. We then have
$$
\sum_ {i = 1} ^ {d} \mathrm {W} _ {1} \left(\alpha_ {i}, \beta_ {i}\right) ^ {2} \leq \mathrm {W} _ {1} (\alpha , \beta) ^ {2}. \tag {8}
$$
Proof. Take  $(X,Y)$  to be an optimal coupling for the  $W_{1}(\alpha, \beta)$  problem. Writing  $\Delta_{i} = \| Y_{i} - X_{i}\|^{2}$  for  $i \in [d]$ , the right hand side of (8) can be written as the  $L^{1/2}$  norm  $\| \sum_{i=1}^{d} \Delta_{i}\|_{1/2}$ . We then bound
$$
\sum_ {i = 1} ^ {d} \mathsf {W} _ {1} \left(\alpha_ {i}, \beta_ {i}\right) ^ {2} \leq \sum_ {i = 1} ^ {d} \| \Delta_ {i} \| _ {1 / 2} \leq \left\| \sum_ {i = 1} ^ {d} \Delta_ {i} \right\| _ {1 / 2},
$$
where the final inequality follows by the reverse Minkowski inequality for  $L^{1/2}$ .
B.7 Proof of Proposition 6
$$
\begin{array}{l} \sup_{\substack{\nu \in \mathcal{G}_{2}(\sigma ,z_{0}):\\ \mathsf{W}_{p}^{\varepsilon}(\tilde{\mu}_{n}\| \nu)\leq \rho}}\mathbb{E}_{\nu}[\ell ] = \sup_{\substack{\mu^{\prime},\nu \in \mathcal{P}(\mathcal{Z})\\ \pi \in \Pi (\mu^{\prime},\nu)}}\left\{ \begin{array}{c}\mathbb{E}_{\nu}[\| Z - z_{0}\|^{2}]\leq \sigma^{2},\\ \mathbb{E}_{\nu}[\ell ]:\mathbb{E}_{\pi}[\| Z^{\prime} - Z\|^{p}]\leq \rho^{p},\\ \mu^{\prime}\leq \frac{1}{1 - \varepsilon}\tilde{\mu}_{n} \end{array} \right\} \\ = \sup_{\substack{m\in \mathbb{R}^{n}\\\nu_{1},\ldots ,\nu_{n}\in \mathcal{P}(\mathcal{Z})}}\left\{\sum_{i\in [n]}m_{i}\mathbb{E}_{\nu_{i}}[\| Z_{i} - z_{0}\|^{2}]\leq \sigma^{2},\right.\\\left.\sum_{i\in [n]}m_{i}\mathbb{E}_{\nu_{i}}[\| \tilde{Z}_{i} - Z_{i}\|^{p}]\leq \rho^{p},\right.\\0\leq m_{i}\leq \frac{1}{n(1 - \varepsilon)},  \forall i\in [n]\\\sum_{i\in [n]}m_{i} = 1\right\} , \\ \end{array}
$$
where the first equality follows from the definitions of  $\mathcal{G}_2(\sigma, z_0)$  and  $\mathsf{W}_p^\varepsilon(\tilde{\mu}_n \| \nu)$ . The second equality holds because  $\tilde{\mu}_n = \frac{1}{n} \sum_{i \in [n]} \delta_{\tilde{Z}_i}$ , which implies that the distributions  $\mu', \nu$  and  $\pi$  take the form
$\mu' = \sum_{i \in [n]} m_i \delta_{\tilde{Z}_i}$ ,  $\nu = \sum_{i \in [n]} m_i \nu_i$ , and  $\pi = \sum_{i \in [n]} m_i \delta_{\tilde{Z}_i} \otimes \nu_i$ , respectively. Note that the distribution  $\nu_i$  models the probability distribution of the random variable  $Z$  condition on the event that  $Z' = \tilde{z}_i$ . Using the definition of the expectation operator and introducing the positive measure  $\nu_i' = m_i \nu_i$  for every  $i \in [n]$ , we arrive at
$$
\begin{array}{l} \sup_{\substack{\nu \in \mathcal{G}_{2}(\sigma ,z_{0}):\\ \mathsf{W}_{p}^{\varepsilon}(\tilde{\mu}_{n}\| \nu)\leq \rho}}\mathbb{E}_{\nu}[\ell ] = \sup_{\substack{m\in \mathbb{R}^{n}\\ \nu_{1}^{\prime},\ldots ,\nu_{n}^{\prime}\geq 0}}\left\{ \begin{array}{ll}\sum_{i\in [n]}\int_{\mathcal{Z}}\| z_{i} - z_{0}\|^{2}d\nu_{i}^{\prime}(z_{i})\leq \sigma^{2},\\ \sum_{i\in [n]}\int_{\mathcal{Z}}\| z_{i} - \tilde{Z}_{i}\|^{p}d\nu_{i}^{\prime}(z_{i})\leq \rho^{p},\\ \sum_{i\in [n]} \mathbb{E}_{\nu_{i}^{\prime}}[\ell ]: & 0\leq m_{i}\leq \frac{1}{n(1 - \varepsilon)},  \forall i\in [n],\\ \sum_{i\in [n]}m_{i} = 1\\ \int_{\mathcal{Z}}d\nu_{i}^{\prime}(z_{i}) = m_{i},  \forall i\in [n] \end{array} \right\} \\ = \inf_{\substack{\lambda_{1},\lambda_{2}\in \mathbb{R}_{+}\\ r,s\in \mathbb{R}^{n},\alpha \in \mathbb{R}}}\left\{\lambda_{1}\sigma^{q} + \lambda_{2}\rho^{p} + \alpha +\frac{\sum_{i\in[n]}s_{i}}{n(1 - \varepsilon)}: \begin{array}{c}s_{i}\geq \max \{0,r_{i} - \alpha \} ,  \forall i\in [n],\\ r_{i}\geq \ell (\xi) - \lambda_{1}\| \xi -z_{0}\|^{2} - \lambda_{2}\| \xi -\tilde{Z}_{i}\|^{p},\\ \forall \xi \in \mathcal{Z},\forall i\in [n] \end{array} \right\} , \\ \end{array}
$$
where the second equality follows from strong duality, which holds because the Slater condition outlined in [47, Proposition 3.4] is satisfied thanks to Assumption 1. The proof concludes by removing the decision variables  $r$  and  $s$  and using the definition of  $\tilde{\mu}_n$ .
B.8 Proof of Theorem 2
The proof requires the following preparatory lemma. We say that the function  $f$  is proper if  $f(x) > -\infty$  and  $\operatorname{dom}(f) \neq \emptyset$ .
Lemma 11. The followings hold.
(i) Let  $f(x) = \lambda g(x - x_0)$ , where  $\lambda \geq 0$  and  $g: \mathbb{R}^d \to \mathbb{R}$  is l.s.c. and convex. Then,  $f^{*}(y) = x_{0}^{\top} y + \lambda g^{*}(y / \lambda)$ .
(ii) Let  $f(x) = \| x\| ^p$  for some  $p\geq 1$ . Then,  $f^{*}(y) = h(y)$ , where the function  $h$  is defined as in (6).
(iii) Let  $f(x) = x^{\top} \Sigma x$  for some  $\Sigma \succ 0$ . Then,  $f^{*}(y) = \frac{1}{4} y^{\top} \Sigma^{-1} y$ .
Proof. The claims follow from [25, §E, Proposition 1.3.1], [57, Lemma B.8 (ii)] and [25, §E, Example 1.1.3], respectively.  $\square$
Proof of Theorem 2. By Proposition 6 and exploiting the definition of  $\tilde{\mu}_n$ , we have
$$
\begin{array}{l} \sup_{\substack{\nu \in \mathcal{G}_{2}(\sigma ,z_{0}):\\ \mathsf{W}_{p}^{\varepsilon}(\tilde{\mu}_{n}\| \nu)\leq \rho}}\mathbb{E}_{\nu}[\ell ] = \left\{ \begin{array}{ll}\inf & \lambda_{1}\sigma^{2} + \lambda_{2}\rho^{p} + \alpha +\frac{1}{n(1 - \varepsilon)}\sum_{i\in [n]}s_{i}\\ \text{s.t.} & \alpha \in \mathbb{R},  \lambda_{1},\lambda_{2}\in \mathbb{R}_{+},  s\in \mathbb{R}_{+}^{n}\\ & s_{i}\geq \sup_{\xi \in \mathcal{Z}}\ell (\xi) - \lambda_{1}\| \xi -z_{0}\|^{2} - \lambda_{2}\| \xi -\tilde{Z}_{i}\|^{p} - \alpha \quad \forall i\in [n] \end{array} \right. \\ = \left\{ \begin{array}{l} \inf  _ {\text {s . t .}} \lambda_ {1} \sigma^ {2} + \lambda_ {2} \rho^ {p} + \alpha + \frac {1}{n (1 - \varepsilon)} \sum_ {i \in [ n ]} s _ {i} \\ \sup  _ {\xi \in \mathcal {Z}} \lambda_ {1}, \lambda_ {2} \in \mathbb {R} _ {+}, s \in \mathbb {R} _ {+} ^ {n} \\ s _ {i} \geq \sup  _ {\xi \in \mathcal {Z}} \ell_ {j} (\xi) - \lambda_ {1} \| \xi - z _ {0} \| ^ {2} \\ - \lambda_ {2} \| \xi - \tilde {Z} _ {i} \| ^ {p} - \alpha \quad \forall i \in [ n ], \forall j \in [ J ] \end{array} \right. \tag {9} \\ \end{array}
$$
where the second equality follows from Assumption 2. For any fixed  $i \in [n]$  and  $j \in [J]$ , we have
$$
\begin{array}{l} \sup _ {\xi \in \mathcal {Z}} \ell_ {j} (\xi) - \lambda_ {1} \| \xi - z _ {0} \| ^ {2} - \lambda_ {2} \| \xi - \tilde {Z} _ {i} \| ^ {p} - \alpha \\ = \left\{ \begin{array}{l l} \inf  & (- \ell_ {j}) ^ {*} (\zeta_ {i j} ^ {\ell}) + z _ {0} ^ {\top} \zeta_ {i j} ^ {\mathcal {G}} + \tau_ {i j} + \tilde {Z} _ {i} ^ {\top} \zeta_ {i j} ^ {\mathsf {W}} + P _ {h} (\zeta_ {i j} ^ {\mathsf {W}}, \lambda_ {2}) + \chi_ {\mathcal {Z}} ^ {*} (\zeta_ {i j} ^ {\mathcal {Z}}) - \alpha \\ \text {s . t .} & \tau_ {i j} \in \mathbb {R} _ {+} ^ {n},   \zeta_ {i j} ^ {\ell}, \zeta_ {i j} ^ {\mathcal {G}}, \zeta_ {i j} ^ {\mathsf {W}}, \zeta_ {i j} ^ {\mathcal {Z}} \in \mathbb {R} ^ {d},   \zeta_ {i j} ^ {\ell} + \zeta_ {i j} ^ {\mathcal {G}} + \zeta_ {i j} ^ {\mathsf {W}} + \zeta_ {i j} ^ {\mathcal {Z}} = 0,   \| \zeta_ {i j} ^ {\mathcal {G}} \| ^ {2} \leq \lambda_ {1} \tau_ {i j} \end{array} \right. \\ \end{array}
$$
where the equality is a result of strong duality due to [57, Theorem 2] and Lemma 11. The claim follows by substituting all resulting dual minimization problems into (9) and eliminating the corresponding minimization operators.
B.9 Proof of Theorem 3
Thanks to Remark 3, we have
$$
\begin{array}{l} \sup  \quad \mathbb {E} _ {\nu} [ \ell ] \\ \nu \in \mathcal {G} _ {2} (\sigma , z _ {0}): \\ \mathbb {W} _ {p} ^ {\varepsilon} (\tilde {\mu} _ {n} \| \nu) \leq \rho \\ = \inf  _ {\lambda_ {1}, \lambda_ {2} \in \mathbb {R} _ {+}} \lambda_ {1} \sigma^ {2} + \lambda_ {2} \rho^ {p} + \mathrm {C V a R} _ {1 - \varepsilon , \tilde {\mu} _ {n}} \left[ \sup  _ {\xi \in \mathcal {Z}} \ell (\xi) - \lambda_ {1} \| \xi - z _ {0} \| ^ {2} - \lambda_ {2} \| \xi - Z \| ^ {p} \right] \\ = \inf  _ {\lambda_ {1}, \lambda_ {2} \in \mathbb {R} _ {+}} \sup  _ {m \in \mathcal {M} _ {\varepsilon}} \lambda_ {1} \sigma^ {2} + \lambda_ {2} \rho^ {p} + \sum_ {i \in [ n ]} m _ {i} \left[ \sup  _ {\xi \in \mathcal {Z}} \ell (\xi) - \lambda_ {1} \| \xi - z _ {0} \| ^ {2} - \lambda_ {2} \| \xi - \tilde {Z} _ {i} \| ^ {p} \right], \\ \end{array}
$$
where  $\mathcal{M}_{\varepsilon} := \{m \in \mathbb{R}_+^n : m_i \leq 1 / (n(1 - \varepsilon)), \forall i \in [n], \sum_{i \in [n]} m_i = 1\}$ , and the equality follows from the primal representation of the CVaR as a coherent risk measure [3, Example 4.1] and the fact that  $\tilde{\mu}_n$  is discrete. By Assumption 2, we have
$$
\begin{array}{l} \sup _ {m \in \mathcal {M} _ {\varepsilon}} \sum_ {i \in [ n ]} m _ {i} \left[ \sup _ {\xi \in \mathcal {Z}} \ell (\xi) - \lambda_ {1} \| \xi - z _ {0} \| ^ {2} - \lambda_ {2} \| \xi - \tilde {Z} _ {i} \| ^ {p} \right] \\ = \sup  _ {m \in \mathcal {M} _ {\varepsilon}} \sup  _ {\xi_ {i j} \in \mathcal {Z}} \sum_ {i \in [ n ]} m _ {i} \left[ \max  _ {j \in [ J ]} \ell \left(\xi_ {i j}\right) - \lambda_ {1} \| \xi_ {i j} - z _ {0} \| ^ {2} - \lambda_ {2} \| \xi_ {i j} - \tilde {Z} _ {i} \| ^ {p} \right] \\ = \sup _ {q \in \mathcal {Q} _ {\varepsilon}} \sup _ {\xi_ {i j} \in \mathcal {Z}} \sum_ {(i, j) \in [ n ] \times [ J ]} q _ {i j} \left[ \ell (\xi_ {i j}) - \lambda_ {1} \| \xi_ {i j} - z _ {0} \| ^ {2} - \lambda_ {2} \| \xi_ {i j} - \tilde {Z} _ {i} \| ^ {p} \right], \\ \end{array}
$$
where  $\mathcal{Q}_{\varepsilon} := \{q \in \mathbb{R}_+^{n \times J} : \sum_{j \in [J]} q_{ij} \leq \frac{1}{n(1 - \varepsilon)}, \forall i \in [n], \sum_{(i,j) \in [n] \times [J]} q_{ij} = 1\}$ , and the last equality easily follows by introducing the variables  $q_{ij}$  as a means to merge the variables  $m_i$  and the maximum operator. Note that the final supremum problem is nonconvex as we have bilinearity between  $q_{ij}$  and  $\xi_{ij}$ . Using the definition of the perspective function and the simple variable substitution  $\xi_{ij} \gets \xi_{ij} / q_{ij}$ , however, one can convexify this problem and arrive at
$$
\begin{array}{l} \sup_{\substack{\nu \in \mathcal{G}_{2}(\sigma ,z_{0}):\\ \mathbb{W}_{p}^{\varepsilon}(\tilde{\mu}_{n}\| \nu)\leq \rho}}\mathbb{E}_{\nu}[\ell ] = \inf_{\lambda_{1},\lambda_{2}\in \mathbb{R}_{+}}\sup_{\substack{q\in \mathcal{Q}_{\varepsilon}\\ \xi_{ij}\in q_{ij}\cdot \mathcal{Z}}}\left\{\lambda_{1}\sigma^{2} + \lambda_{2}\rho^{p} - \sum_{(i,j)\in [n]\times [J]}P_{-\ell_{j}}(\xi_{ij},q_{ij})\right. \\ \left. - \lambda_ {1} P _ {\| \cdot \| ^ {2}} \left(\xi_ {i j} - q _ {i j} z _ {0}, q _ {i j}\right) - \lambda_ {2} P _ {\| \cdot \| ^ {p}} \left(\xi_ {i j} - q _ {i j} \tilde {Z} _ {i}, q _ {i j}\right) \right\}. \\ \end{array}
$$
Note that strong duality holds similar to the proof of [57, Section 6]. This allows us to interchange the infimum and supremum without changing the optimal value of the problem. Then, infinizing over  $\lambda_{1}$  and  $\lambda_{2}$ , and noticing that the resulting supremum problem is solvable, since the feasible set is compact, conclude the first part of the proof. Following the discussion in [57, § 6], it is easy to show that the proposed discrete distribution  $\nu^{\star}$  solves the worst-case expectation problem. The details are omitted for brevity. This concludes the proof.
Lemma 12. Under the setting of Theorem 4, we may decompose  $\ell_{\star} = \tilde{\ell}\circ Q$  for  $Q\in \mathbb{R}^{k\times d}$  with  $QQ^{\top} = I_k$  and some  $\tilde{\ell}:\mathbb{R}^k\to \mathbb{R}^d$ . For any such decomposition, we have
$$
\sup_{\substack{\nu \in \mathcal{G}\\ \mathsf{W}_{p}^{\varepsilon}(\nu ,\tilde{\mu})\leq \rho}}\mathbb{E}_{\nu}[\ell_{\star}] = \sup_{\substack{\nu \in \mathcal{G}_{k}\\ \mathsf{W}_{p}^{\varepsilon}(\nu ,Q_{\#}\tilde{\mu})\leq \rho}}\mathbb{E}_{\nu}[\tilde{\ell} ].
$$
Proof. To start, we decompose  $A(z) = RQz + z_0$ , where  $Q \in \mathbb{R}^{k \times d}$  with  $QQ^\top = I_k$ ,  $R \in \mathbb{R}^{k \times k}$  and  $z_0 \in \mathbb{R}^k$ . Note that the orthogonality condition ensures that  $Q^\top$  isometrically embeds  $\mathbb{R}^k$  into  $\mathbb{R}^d$ . We then choose  $\tilde{\ell}(w) = \underline{\ell}(Rw + z_0)$ .
Next, given  $\nu \in \mathcal{G}$ , we have  $Q_{\#} \nu \in \mathcal{G}^{(k)}$  with  $\mathsf{W}_p^\varepsilon(Q_{\#} \nu, Q_{\#} \tilde{\mu}) \leq \mathsf{W}_p^\varepsilon(\nu, \tilde{\mu})$ , and  $\mathbb{E}_{\nu}[\ell] = \mathbb{E}_{Q_{\#} \nu}[\tilde{\ell}]$ . Thus, the RHS supremum is always at least as large as the LHS. It remains to show the reverse.
Fix  $\nu \in \mathcal{G}_k$  with  $\mathsf{W}_p^\varepsilon (\nu ,Q_\# \tilde{\mu})$ . Take any  $\nu^{\prime}\in \mathcal{P}(\mathbb{R}^{k})$  with  $\mathsf{W}_p(\nu ,\nu ')\leq \rho$  and  $\| \nu^{\prime} - Q_{\#}\tilde{\mu}\|_{\mathrm{TV}}\leq \varepsilon$ . Write  $\kappa = Q_{\#}^{\top}\nu \in \mathcal{G}$  and  $\kappa^{\prime} = Q_{\#}^{\top}\nu^{\prime}$ . Since  $Q^{\top}$  is an isometric embedding, we have  $\kappa \in \mathcal{G}$ ,  $\mathsf{W}_p(\kappa ,\kappa ') = \mathsf{W}_p(\nu ,\nu ')\leq \rho$ , and  $\| \kappa^{\prime} - \tilde{\mu}\|_{\mathsf{TV}} = \| \nu^{\prime} - Q_{\#}\tilde{\mu}\|_{\mathsf{TV}}\leq \varepsilon$ . Finally, we have  $\mathbb{E}_{\nu}[\ell ] = \mathbb{E}_{\kappa}[\tilde{\ell} ]$ . Thus, the RHS supremum is no greater than the LHS, and we have the desired equality.
Writing  $\mu_{k} = Q_{\#}\mu$  , we mirror the proof of Theorem 1 and bound
$$
\begin{array}{l} \mathbb{E}_{\mu}[\hat{\ell} ] - \mathbb{E}_{\mu}[\ell_{\star}] \leq \sup_{\substack{\nu \in \mathcal{G}\\ \mathsf{W}_{p}^{\varepsilon}(\tilde{\mu},\nu)\leq \rho}}\mathbb{E}_{\nu}[\hat{\ell} ] - \mathbb{E}_{\mu}[\ell_{\star}]\\ (\mu \in \mathcal{G},\mathsf{W}_{p}^{\varepsilon}(\tilde{\mu},\mu)\leq \rho) \\ \leq \sup_{\substack{\nu \in \mathcal{G}\\ \mathsf{W}_{p}^{\varepsilon}(\tilde{\mu},\nu)\leq \rho}}\mathbb{E}_{\nu}[\ell_{\star}] - \mathbb{E}_{\mu}[\ell_{\star}] \qquad \qquad (\hat{\ell}\text{optimal for (OR-WDRO)}) \\ = \sup  _ {\substack {\nu \in \mathcal {G} ^ {(k)} \\ \mathsf {W} _ {p} ^ {\varepsilon} (\nu , \tilde {Q} _ {\#} \tilde {\mu}) \leq \rho}} \mathbb {E} _ {\nu} [ \tilde {\ell} ] - \mathbb {E} _ {\mu_ {k}} [ \tilde {\ell} ] (Lemma 12) \\ \leq \sup  _ {\substack {\nu \in \mathcal {G} ^ {(k)} \\ \mathsf {W} _ {p} ^ {2 \varepsilon} (\nu , \mu_ {k}) \leq 2 \rho}} \mathbb {E} _ {\nu} [ \tilde {\ell} ] - \mathbb {E} _ {\mu_ {k}} [ \tilde {\ell} ] (Lemma 2) \\ \leq \sup  _ {\substack {\nu \in \mathcal {G} ^ {(k)} \\ \mathsf {W} _ {p} (\nu , \mu_ {k}) \leq c \rho + 2 \tau_ {p} (\mathcal {G} ^ {(k)}, 2 \varepsilon)}} \mathbb {E} _ {\nu} [ \tilde {\ell} ] - \mathbb {E} _ {\mu_ {k}} [ \tilde {\ell} ] (Lemma3) \\ \leq \sup_{\substack{\nu \in \mathcal{P}(\mathcal{Z})\\ \mathsf{W}_{p}(\nu ,\mu_{k})\leq c\rho +2\tau_{p}(\mathcal{G}^{(k)},2\varepsilon)}}\mathbb{E}_{\nu}[\tilde{\ell} ] - \mathbb{E}_{\mu_{k}}[\tilde{\ell} ]\qquad \qquad (\mathcal{G}\subseteq \mathcal{P}(\mathcal{Z})) \\ = \mathcal {R} _ {\mu_ {k}, p} \left(c \rho + 2 \tau_ {p} \left(\mathcal {G} ^ {(k)}, 2 \varepsilon\right); \tilde {\ell}\right). \\ \end{array}
$$
To obtain the theorem, we apply Lemma 6 and observe that  $\| \tilde{\ell}\|_{\mathrm{Lip}} = \| \ell_{\star}\|_{\mathrm{Lip}}$  and  $\| \tilde{\ell}\|_{\dot{H}^{1,2}(\mu_k)} = \|\ell_\star\|_{\dot{H}^{1,2}(\mu)}$  (since  $Q^{\top}$  is an isometric embedding from  $\mathbb{R}^k$  into  $\mathbb{R}^d$ ).
C.2 Proof of Corollary 4
This follows as an immediate consequence of Theorem 4 and Corollary 1.
C.3 Proof of Proposition 7