SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Waiss Azizian, Franck Iutzeler, Jérôme Malick
Univ. Grenoble Alpes, CNRS, Grenoble INP
Grenoble, 38000, France
firstname.lastname@univ-grenoble-alpes.fr
Wasserstein distributionally robust estimators have emerged as powerful models for prediction and decision-making under uncertainty. These estimators provide attractive generalization guarantees: the robust objective obtained from the training distribution is an exact upper bound on the true risk with high probability. However, existing guarantees either suffer from the curse of dimensionality, are restricted to specific settings, or lead to spurious error terms. In this paper, we show that these generalization guarantees actually hold on general classes of models, do not suffer from the curse of dimensionality, and can even cover distribution shifts at testing. We also prove that these results carry over to the newly-introduced regularized versions of Wasserstein distributionally robust problems.
1.1 Generalization and (Wasserstein) Distributionally Robust Models
We consider the fundamental question of generalization of machine learning models. Let us denote by  $f_{\theta}$  the loss induced by a model parametrized by  $\theta$  for some uncertain variable  $\xi$  (typically a data point). When  $\xi$  follows some distribution  $\mathrm{P}$ , seeking the best parameter  $\theta$  writes as minimizing the expected loss
$$
\min  _ {\theta \in \Theta} \mathbb {E} _ {\xi \sim \mathrm {P}} \left[ f _ {\theta} (\xi) \right].
$$
We usually do not have a direct knowledge of  $\mathrm{P}$  but rather we have access to samples  $(\xi_{i})_{i = 1}^{n}$  independently drawn from  $\mathrm{P}$ . The empirical risk minimization approach then consists in minimizing the expected loss over the associated empirical distribution  $\widehat{\mathrm{P}}_{\mathrm{n}} = \frac{1}{n}\sum_{i = 1}^{n}\delta_{\xi_{i}}$  (as a proxy for the expected loss over  $\mathrm{P}$ ), i.e.,
$$
\min _ {\theta \in \Theta} \mathbb {E} _ {\xi \sim \widehat {\mathrm {P}} _ {\mathrm {n}}} [ f _ {\theta} (\xi) ] \left(= \frac {1}{n} \sum_ {i = 1} ^ {n} f _ {\theta} (\xi_ {i})\right).
$$
Classical statistical learning theory ensures that, with high probability,  $\mathbb{E}_{\mathrm{P}}[f_{\theta}]$  is close to  $\mathbb{E}_{\widehat{\mathrm{P}}_{\mathrm{n}}} [f_{\theta}]$  up to  $\mathcal{O}(1 / \sqrt{n})$  error terms, see e.g., the monographs Boucheron et al. (2013); Wainwright (2019).
A practical drawback of empirical risk minimization is that it can lead to over-confident decisions (when  $\mathbb{E}_{\widehat{\mathrm{P}}_{\mathrm{n}}} [f_{\theta}] < \mathbb{E}_{\mathrm{P}} [f_{\theta}]$ , the real loss can be higher that the empirical one (Esfahani and Kuhn, 2018)). In addition, this approach is also sensitive to distribution shifts between training and application. To overcome these drawbacks, an approach gaining momentum in machine learning is distributionally robust optimization, which consists in minimizing the worst expectation of the loss when the distribution lives in a neighborhood of  $\widehat{\mathrm{P}}_{\mathrm{n}}$ :
$$
\min  _ {\theta \in \Theta} \sup  _ {\mathrm {Q} \in \mathcal {U} \left(\widehat {\mathrm {P}} _ {\mathrm {n}}\right)} \mathbb {E} _ {\xi \sim \mathrm {Q}} \left[ f _ {\theta} (\xi) \right], \tag {1}
$$
where the inner sup is thus taken over  $\mathrm{Q}$  in the neighborhood  $\mathcal{U}(\widehat{\mathrm{P}}_{\mathrm{n}})$  of  $\widehat{\mathrm{P}}_{\mathrm{n}}$  in the space of probability distributions. Popular choices of distribution neighborhoods are based on the Kullback-Leibler (KL) divergence (Laguel et al., 2020; Levy et al., 2020), kernel tools (Zhu et al., 2021a; Staib and Jegelka, 2019; Zhu et al., 2021b), moments (Delage and Ye, 2010; Goh and Sim, 2010), or Wasserstein distance (Shafieezadeh Abadeh et al., 2015; Esfahani and Kuhn, 2018). If  $\mathrm{P} \in \mathcal{U}(\widehat{\mathrm{P}}_{\mathrm{n}})$ , distributionally robust models can benefit from direct generalization guarantees as,
$$
\sup  _ {\mathrm {Q} \in \mathcal {U} \left(\widehat {\mathrm {P}} _ {\mathrm {n}}\right)} \mathbb {E} _ {\xi \sim \mathrm {Q}} [ f _ {\theta} (\xi) ] \geq \mathbb {E} _ {\xi \sim \mathrm {P}} [ f _ {\theta} (\xi) ]. \tag {2}
$$
Thus, for well-chosen neighborhoods  $\mathcal{U}(\widehat{\mathrm{P}}_{\mathrm{n}})$ , distributionally robust objectives are able to provide exact upper-bounds on the expected loss over distribution P, i.e., the true risk.
Wasserstein distributionally robust optimization (WDRO) problems correspond to (1) with
$$
\mathcal {U} (\widehat {\mathrm {P}} _ {\mathrm {n}}) = \left\{\mathrm {Q} \in \mathcal {P} (\Xi): W (\widehat {\mathrm {P}} _ {\mathrm {n}}, \mathrm {Q}) \leq \rho \right\},
$$
where  $W(\widehat{\mathrm{P}}_{\mathrm{n}}, \mathrm{Q})$  denotes the Wasserstein distance between  $\widehat{\mathrm{P}}_{\mathrm{n}}$  and  $\mathrm{Q}$  and  $\rho > 0$  controls the required level of robustness around  $\widehat{\mathrm{P}}_{\mathrm{n}}$ . As a natural metric to compare discrete and absolutely continuous probability distributions, the Wasserstein distance has attracted a lot of interest in both machine learning (Shafieezadeh Abadeh et al., 2015; Sinha et al., 2018; Shafieezadeh-Abadeh et al., 2019; Li et al., 2020; Kwon et al., 2020) and operation research (Zhao and Guan, 2018; Arrigo et al., 2022) communities; see e.g., the review articles Blanchet et al. (2021); Kuhn et al. (2019).
WDRO benefits from out-of-the-box generalization guarantees in the form of (2) since it inherits the concentration properties of the Wasserstein distance (Esfahani and Kuhn, 2018). More precisely, under mild assumptions on  $\mathrm{P}$ , (Fournier and Guillin, 2015) establishes that  $W(\widehat{\mathrm{P}}_{\mathrm{n}},\mathrm{P})\leq \rho$  with high probability as soon as  $\rho \sim 1 / n^{1 / d}$  where  $d$  denotes the dimension of the samples space. Thus, a major issue is the prescribed radius  $\rho$  suffers from the curse of the dimensionality: when  $d$  is large,  $\rho$  decreases slowly as the number of samples  $n$  increases. This constrains with other distributionally robust optimization ambiguity sets, such as Maximum Mean Discrepancy (MMD) (Staib and Jegelka, 2019; Zeng and Lam, 2022), where the radius scales as  $1 / \sqrt{n}$ . Moreover, the existing scaling for WDRO is overly conservative for WDRO objectives since recent works (Blanchet et al., 2022a; Blanchet and Shapiro, 2023) prove that a radius behaving as  $1 / \sqrt{n}$  is asymptotically optimal. The main difference with (Esfahani and Kuhn, 2018) is that they — and us — consider the WDRO objective as a whole, instead of proceeding in two steps: first considering the Wasserstein distance independently and invoking concentration results on the Wasserstein distance and then plugging this result in the WDRO problem.
1.2 Contributions and related works
In this paper, we show that WDRO provides exact upper-bounds on the true risk with high probability. More precisely, we prove non-asymptotic generalization bounds of the form of (2), that hold for general classes of functions, and that only require  $\rho$  to scale as  $1 / \sqrt{n}$  and not  $1 / n^{1 / d}$ . To do so, we construct an interval for the radius  $\rho$  for which it is both sufficiently large so that we can go from the empirical to the true estimator (i.e., at least of the order of  $1 / \sqrt{n}$ ) and sufficiently small so that the robust problem does not become degenerate (i.e., smaller than some critical radius, that we introduce as an explicit constant). Our results imply proving concentration results on Wasserstein Distributionally Robust objectives that are of independent interest.
This work is part of a rich and recent line of research about theoretical guarantees on WDRO for machine learning. One of this first results, Lee and Raginsky (2018), provides generalization guarantees, for a general class of models and a fixed  $\rho$ , that, however, become degenerate as the radius goes to zero. In the particular case of linear models, WDRO models admit an explicit form that allows Shafieezadeh-Abadeh et al. (2019); Chen and Paschalidis (2018) to provide generalization guarantees (2) with the radius scaling as  $1 / \sqrt{n}$ . The case of general classes of models, possibly non-linear, is more intricate. Sinha et al. (2018) showed that a modified version of (2) holds at the price of non-negligible error terms. Gao (2022); An and Gao (2021) made another step towards broad generalization guarantees for WDRO but with error terms that vanish only when  $\rho$  goes to zero.
In contrast, our analysis provides exact generalization guarantees in the form (2) without additional error terms, that hold for general classes of functions and allow for a non-vanishing uncertainty radius to cover for distribution shifts at testing. Moreover, our guarantees also carry over to the recently introduced regularized versions of WDRO (Wang et al., 2023; Azizian et al., 2023), whose statistical properties have not been studied yet.
This paper is organized as follows. In Section 2, we introduce notations and our blanket assumptions. In Section 3, we present our main results, an idea of proof, and discussions. The complete proofs are deferred to the appendix.
2 Setup and Assumptions
In this section, we formalize our setting and introduce Wasserstein Distributionally Robust risks.
2.1 Wasserstein Distributionally Robust risk functions
In this paper, we consider as a samples space  $\Xi$  a subset of  $\mathbb{R}^d$  equipped with the Euclidean norm  $\|\cdot\|$ . We rely on Wasserstein distances of order 2, in line with the seminal work Blanchet et al. (2022a) on generalization of WDRO. This distance is defined for two distributions Q,  $Q'$  in the set of probability distributions on  $\Xi$ , denoted by  $\mathcal{P}(\Xi)$ , as
$$
W _ {2} (\mathrm {Q}, \mathrm {Q} ^ {\prime}) := \left(\inf  _ {\pi \in \mathcal {P} (\Xi \times \Xi), \pi_ {1} = \mathrm {Q}, \pi_ {2} = \mathrm {Q} ^ {\prime}} \mathbb {E} _ {(\xi , \zeta) \sim \pi} \left[ \frac {1}{2} \| \xi - \zeta \| ^ {2} \right]\right) ^ {1 / 2},
$$
where  $\mathcal{P}(\Xi \times \Xi)$  is the set of probability distributions in the product space  $\Xi \times \Xi$ , and  $\pi_1$  (resp.  $\pi_2$ ) denotes the first (resp. second) marginal of  $\pi$ .
We denote by  $f: \Xi \to \mathbb{R}$  the loss function of some model over the sample space. The model may depend on some parameter  $\theta$ , that we drop for now to lighten the notations; instead, we consider a class of functions  $\mathcal{F}$  encompassing our various models and losses of interest (we come back to classes of parametric models of the form  $\mathcal{F} = \{f_{\theta}: \theta \in \Theta\}$  in Section 4).
We define the empirical Wasserstein Distributionally Robust risk  $\widehat{\mathcal{R}}_{\rho^2}(f)$  centered on  $\widehat{\mathrm{P}}_{\mathrm{n}}$  and similarly the true robust risk  $\mathcal{R}_{\rho^2}(f)$  centered on  $\mathrm{P}$  as
$$
\widehat {\mathcal {R}} _ {\rho^ {2}} (f) := \sup  _ { \begin{array}{c} Q \in \mathcal {P} (\Xi) \\ W _ {2} ^ {2} (\widehat {P} _ {n}, Q) \leq \rho^ {2} \end{array} } \mathbb {E} _ {\xi \sim Q} [ f (\xi) ] \quad \text {a n d} \quad \mathcal {R} _ {\rho^ {2}} (f) := \sup  _ { \begin{array}{c} Q \in \mathcal {P} (\Xi) \\ W _ {2} ^ {2} (P, Q) \leq \rho^ {2} \end{array} } \mathbb {E} _ {\xi \sim Q} [ f (\xi) ]. \tag {3}
$$
Note that  $\widehat{\mathcal{R}}_{\rho^2}(f)$ , which is based on the empirical distribution  $\widehat{\mathrm{P}}_{\mathrm{n}}$ , is a computable proxy for the true robust risk  $\mathcal{R}_{\rho^2}(f)$ . Note also that the true robust risk  $\mathcal{R}_{\rho^2}(f)$  immediately upper-bounds the true (non-robust) risk  $\mathbb{E}_{\xi \sim \mathrm{P}}[f(\xi)]$  and also upper-bounds  $\mathbb{E}_{\xi \sim \mathrm{Q}}[f(\xi)]$  for neighboring distributions  $\mathrm{Q}$  that correspond to distributions shifts of magnitude smaller than  $\rho$  in Wasserstein distance.
2.2 Regularized versions
Entropic regularization of WDRO problems was recently studied in Wang et al. (2023); Blanchet and Kang (2020); Piat et al. (2022); Azizian et al. (2023) and used in Dapogny et al. (2023); Song et al. (2023); Wang and Xie (2022); Wang et al. (2022). Inspired by the entropic regularization in optimal transport (OT) (Peyre and Cuturi, 2019, Chap. 4), the idea is to regularize the objective by adding a KL divergence, that is defined, for any transport plan  $\pi \in \mathcal{P}(\Xi \times \Xi)$  and a fixed reference  $\overline{\pi} \in \mathcal{P}(\Xi \times \Xi)$ , by
$$
\operatorname {K L} (\pi   | \overline {{\pi}}) = \left\{ \begin{array}{l l} \int \log \frac {\mathrm {d} \pi}{\mathrm {d} \overline {{\pi}}}   \mathrm {d} \pi & \text {w h e n} \pi \ll \overline {{\pi}} \\ + \infty & \text {o t h e r w i s e .} \end{array} \right.
$$
Unlike in OT though, the choice of the reference measure in WDRO is not neutral and introduces a bias in the robust objective (Azizian et al., 2023). For their theoretical convenience, we take reference measures that have Gaussian conditional distributions
$$
\pi_ {\sigma} (\mathrm {d} \zeta | \xi) \propto 1 _ {\zeta \in \Xi} e ^ {- \frac {\| \xi - \zeta \| ^ {2}}{2 \sigma^ {2}}} \mathrm {d} \zeta , \quad \text {f o r a l l} \xi \in \Xi , \tag {4}
$$
where  $\sigma > 0$  controls the spread of the second marginals, following Wang et al. (2023); Azizian et al. (2023). Then, the regularized version of  $\widehat{\mathcal{R}}_{\rho^2}(f)$  (WDRO empirical risk) is given by
$$
\begin{array}{l} \widehat {\mathcal {R}} _ {\rho^ {2}} ^ {\varepsilon} (f) := \quad \sup  _ {\widehat {\mathbf {\Pi}}} \mathbb {E} _ {\xi \sim \pi_ {2}} [ f (\xi) ] - \varepsilon \operatorname {K L} (\pi | \pi_ {\sigma} ^ {n}) \quad \text {w i t h} \pi_ {\sigma} ^ {n} = \widehat {\mathrm {P}} _ {\mathrm {n}} (\mathrm {d} \xi) \pi_ {\sigma} (\mathrm {d} \zeta | \xi) \tag {5} \\ \pi \in \mathcal {P} (\Xi \times \Xi), \pi_ {1} = \widehat {\mathrm {P}} _ {\mathrm {n}} \\ \mathbb {E} _ {(\xi , \zeta) \sim \pi} \left[ \frac {1}{2} \| \xi - \zeta \| ^ {2} \right] \leq \rho^ {2} \\ \end{array}
$$
and similarly, the regularized version of  $\mathcal{R}_{\rho^2}(f)$  is given by
$$
\begin{array}{l} \mathcal {R} _ {\rho^ {2}} ^ {\varepsilon} (f) := \sup  _ {\xi \in (\overline {{\Gamma}}, \overline {{\Gamma}})} \mathbb {E} _ {\xi \sim \pi_ {2}} [ f (\xi) ] - \varepsilon \operatorname {K L} (\pi | \pi_ {\sigma}) \quad \text {w i t h} \pi_ {\sigma} = \mathrm {P} (\mathrm {d} \xi) \pi_ {\sigma} (\mathrm {d} \zeta | \xi). \tag {6} \\ \pi \in \mathcal {P} (\Xi \times \Xi), \pi_ {1} = \mathrm {P} \\ \mathbb {E} _ {(\xi , \zeta) \sim \pi} \left[ \frac {1}{2} \| \xi - \zeta \| ^ {2} \right] \leq \rho^ {2} \\ \end{array}
$$
These regularized risks have been studied in terms of computational or approximation properties, but their statistical properties have not been investigated yet. The analysis we develop for WDRO estimators is general enough to carry over to these settings.
In (5) and (6), note finally that the regularization is added as a penalization in the supremum, rather than in the constraint. As in Wang et al. (2023), penalizing in the constraint leads to an ambiguity set defined by the regularized Wasserstein distance, that we introduce in Section 3.2. We refer to Azizian et al. (2023) for a unified presentation of the two penalizations.
2.3 Blanket assumptions
Our analysis is carried under the following set of assumptions that will be in place throughout the paper. First, we assume that the sample space  $\Xi \subset \mathbb{R}^d$  is convex and compact, which is in line with previous work, e.g., (Lee and Raginsky, 2018; An and Gao, 2021).
Assumption 1 (On the set  $\Xi$ ). The sample space  $\Xi$  is a compact convex subset of  $\mathbb{R}^d$ .
Second, we require the class of loss functions  $\mathcal{F}$  to be sufficiently regular. In particular, we assume that they have Lipschitz continuous gradients.
Assumption 2 (On the function class). The functions of  $\mathcal{F}$  are twice differentiable, uniformly bounded, and their derivatives are uniformly bounded and uniformly Lipschitz.
Finally, we assume that  $\widehat{\mathrm{P}}_{\mathrm{n}}$  is made of independent and identically distributed (i.i.d.) samples of  $\mathrm{P}$  and that  $\mathrm{P}$  is supported on the interior of  $\Xi$  (which can be done without loss of generality by slightly enlarging  $\Xi$  if needed).
Assumption 3 (On the distributions).  $\widehat{\mathrm{P}}_{\mathrm{n}} = \frac{1}{n}\sum_{i = 1}^{n}\delta_{\xi_i}$  where  $\xi_1,\ldots ,\xi_n$  are i.i.d. samples of P. We further assume that there is some  $R > 0$  such that  $\mathrm{P}$  satisfies  $\mathrm{supp~P} + \overline{\mathbb{B}} (0,R)\subset \Xi$
3 Main results and discussions
The main results of our paper establish that the empirical robust risk provides high probability bounds, of the form of (2), on the true risk. Since the results and assumptions slightly differ between the WDRO models and their regularized counterparts, we present them separately in Section 3.1 and Section 3.2. In Section 3.3, we provide the common outline for the proofs of these results, the proofs themselves being provided in the appendix. Finally, in Section 4, we detail some examples.
3.1 Exact generalization guarantees for WDRO models
In this section, we require the two following additional assumptions on the function class. The first assumption is common in the WDRO literature, see e.g., Blanchet et al. (2022a); Blanchet and Shapiro (2023); Gao (2022); An and Gao (2021).
Assumption 4. The quantity  $\inf_{f\in \mathcal{F}}\mathbb{E}_{\mathrm{P}}\left[\| \nabla f\| ^2\right]$  is positive.
The second assumption we consider in this section makes use of the notation  $d(\xi, A)$ , for a set  $A \subset \Xi$  and a point  $\xi \in \Xi$ , to denote the distance between  $\xi$  and  $A$ , i.e.,  $d(\xi, A) = \inf_{\zeta \in A} \| \xi - \zeta \|$ .
1. For any  $R > 0$ , there exists  $\Delta > 0$  such that,
$$
\forall f \in \mathcal {F}, \forall \zeta \in \Xi , d (\zeta , \arg \max  f) \geq R \Longrightarrow f (\zeta) - \max  f \leq - \Delta .
$$
2. The following growth condition holds: there exist  $\mu > 0$  and  $L > 0$  such that, for all  $f \in \mathcal{F}, \xi \in \Xi$  and  $\xi^*$  a projection of  $\xi$  on  $\arg \max f$ , i.e.,  $\xi^* \in \arg \min_{\arg \max f} \| \xi - \cdot \|$ ,
$$
f (\xi^ {*}) \geq f (\xi) + \frac {\mu}{2} \| \xi - \xi^ {*} \| ^ {2} - \frac {L}{6} \| \xi - \xi^ {*} \| ^ {3}.
$$
The first item of this assumption has a natural interpretation: we show in Lemma A.7, that it is equivalent to the relative compactness of the function space  $\mathcal{F}$  w.r.t. to the distance
$$
D (f, g) := \| f - g \| _ {\infty} + D _ {H} (\arg \max  f, \arg \max  g),
$$
where  $D_H$  denotes the (Hausdorff) distance between sets and  $\|f\|_{\infty} \coloneqq \sup_{\xi \in \Xi} |f(\xi)|$  is the infinity norm. The last one is a structural assumption on the functions  $\mathcal{F}$  that is new in our context but is actually very close to the so-called parametric Morse-Bott condition, introduced in the bilevel optimization (Arbel and Mairal, 2022), see Section A.5.
We now state our main generalization result for WDRO risks.
Theorem 3.1. Under Assumptions 4 and 5, there is an explicit constant  $\rho_{c}$  depending only on  $\mathcal{F}$  and  $\mathrm{P}$  such that for any  $\delta \in (0,1)$  and  $n\geq 1$  , if
$$
\mathcal {O} \left(\sqrt {\frac {1 + \log 1 / \delta}{n}}\right) \leq \rho \leq \frac {\rho_ {c}}{2} - \mathcal {O} \left(\sqrt {\frac {1 + \log 1 / \delta}{n}}\right) \tag {7}
$$
then, there is  $\rho_{n} = \mathcal{O}\left(\sqrt{\frac{1 + \log 1 / \delta}{n}}\right)$  such that, with probability  $1 - \delta$
$$
\forall f \in \mathcal {F}, \quad \widehat {\mathcal {R}} _ {\rho^ {2}} (f) \geq \mathbb {E} _ {\xi \sim \mathrm {Q}} [ f (\xi) ] \quad \text {f o r a l l Q s u c h t h a t} W _ {2} ^ {2} (\mathrm {P}, \mathrm {Q}) \leq \rho (\rho - \rho_ {n}). \tag {8}
$$
In particular, with probability  $1 - \delta$ , we have
$$
\forall f \in \mathcal {F}, \quad \widehat {\mathcal {R}} _ {\rho^ {2}} (f) \geq \mathbb {E} _ {\xi \sim \mathrm {P}} [ f (\xi) ]. \tag {9}
$$
The second part of the result, (9), is an exact generalization bound: it is an actual upper-bound on the true risk  $\mathbb{E}_{\xi \sim \mathrm{P}}[f(\xi)]$ , that we cannot access in general, through a quantity that we can actually compute with  $\widehat{\mathrm{P}}_{\mathrm{n}}$ . The first part of the result, (8) gives us insight into the robustness guarantees offered by the WDRO risk. Indeed, it tells us that, when  $\rho$  is greater than the minimal radius  $\rho_{n} \propto 1 / \sqrt{n}$  by some margin, the empirical robust risk  $\widehat{\mathcal{R}}_{\rho^2}(f)$  is an upper-bound on the loss even with some perturbations of the true distribution. Hence, as long as  $\rho$  is large enough, the WDRO objective enables us to guarantee the performance of our model even in the event of a distribution shift at testing time. In other words, the empirical robust risk is an exact upper-bound on the true robust risk  $\mathcal{R}_{\rho (\rho -\rho_n)}(f)$  with a reduced radius.
The range of admissible radiuses is described by (7). The lower-bound, roughly proportional to  $1 / \sqrt{n}$ , is optimal, following the results of Blanchet et al. (2022a). The upper-bound, almost independent of  $n$ , depends on a constant  $\rho_{c}$ , that we call critical radius and that has an interesting interpretation, that we formalize in the following remark. Note, finally, that, the big-O notation in this theorem has a slightly stronger meaning<sup>1</sup> than the usual one, being non-asymptotic in  $n$  and  $\delta$ .
Remark 3.2 (Interpretation of critical radius). The critical radius  $\rho_{c}$ , appearing in (7), is defined by
$$
\rho_ {c} ^ {2} := \inf _ {f \in \mathcal {F}} \mathbb {E} _ {\xi \sim \mathrm {P}} \left[ \frac {1}{2} d ^ {2} (\xi , \arg \max f) \right].
$$
$^{1}\mathrm{Eg.}, \rho_{n} = \mathcal{O}\left(\sqrt{\frac{1 + \log 1 / \delta}{n}}\right)$  means that  $\exists C > 0$  such that  $\rho_{n} \leq C \sqrt{\frac{1 + \log 1 / \delta}{n}}$  for all  $\delta \in (0,1)$  and  $n \geq 1$ .
It can be interpreted as the threshold at which the WDRO problem w.r.t.  $\mathrm{P}$  starts becoming degenerate. Indeed, when  $\rho^2\geq \mathbb{E}_{\xi \sim \mathrm{P}}\big[\frac{1}{2} d^2 (\xi ,\arg \max f)\big]$  for some  $f\in \mathcal{F}$  that we fix, the distribution Q given by the second marginal of the transport plan  $\pi$  defined by,
$$
\pi (\mathrm {d} \xi , \mathrm {d} \zeta) := \mathrm {P} (\mathrm {d} \xi) \delta_ {\zeta^ {\star} (\xi)} (\mathrm {d} \zeta) \quad \text {w h e r e} \quad \zeta^ {\star} (\xi) \in \operatorname * {a r g   m i n} _ {\zeta \in \arg \max  f} d ^ {2} (\xi , \zeta),
$$
$$
W _ {2} ^ {2} (\mathrm {P}, \mathrm {Q}) \leq \mathbb {E} _ {(\xi , \zeta) \sim \pi} \left[ \frac {1}{2} \| \xi - \zeta \| ^ {2} \right] = \mathbb {E} _ {\xi \sim \mathrm {P}} \left[ \frac {1}{2} d ^ {2} (\xi , \arg \max f) \right] \leq \rho^ {2}.
$$
As a consequence, the robust problem is equal to
$$
\mathcal{R}_{\rho^{2}}(f) = \sup_{\substack{\mathrm{Q}\in \mathcal{P}(\Xi)\\ W_{2}^{2}(\mathrm{P},\mathrm{Q})\leq \rho^{2}}}\mathbb{E}_{\xi \sim \mathrm{Q}}[f(\xi)] = \max_{\xi \in \Xi}f(\xi).
$$
Thus, when the radius exceeds  $\rho_{c}$ , there is some  $f$  such that the robust problem becomes degenerate as it does not depend on  $\mathrm{P}$  nor  $\rho$  anymore.
Finally, note that we can obtain the same generalization guarantee as Theorem 3.1 without Assumption 5 at the expense of losing the above interpretation on the condition on the radius. More precisely, we have the following result.
Theorem 3.3. Let Assumption 4 hold. For any  $\delta \in (0,1)$  and  $n\geq 1$ , if  $\rho$  satisfies (7), and if in addition, it is smaller than a positive constant which depends only on  $\mathrm{P}$ ,  $\mathcal{F}$  and  $\Xi$ , then both conclusions of Theorem 3.1 hold.