SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This theorem can be compared to existing results, and in particular with Gao (2022); An and Gao (2021). These two papers provide generalization bounds for WDRO under a similar assumption on  $\mathcal{F}$  and a weakened version of Assumption 3. However, these generalization bounds involve extra error terms, that require  $\rho$  to be vanishing. In comparison, with a similar set of assumptions, Theorem 3.3 improves on these two issues, by allowing  $\rho$  not to vanish as  $n\to \infty$  and by providing the exact upper-bound (9). Allowing non-vanishing radiuses is an attractive feature of our results that enables us to cover distribution shifts.
3.2 Regularized WDRO models
The analysis that we develop for the standard WDRO estimators is general enough to also cover the regularized versions presented in Section 2.2. We thus obtain the following Theorem 3.4 which is the first generalization guarantee for regularized WDRO. This theorem is very similar to Theorem 3.1 with still a couple of differences. First, the regularization leads to ambiguity sets defined in terms of  $W_{2,\tau}(\mathrm{P},\cdot)$ , the regularized Wasserstein distance to the true distribution  $\mathrm{P}$ , defined, for some regularization parameter  $\tau >0$ , as
$$
W _ {2, \tau} ^ {2} (\mathrm {P}, \mathrm {Q}) := \inf \left\{\mathbb {E} _ {\pi} \left[ \frac {1}{2} \| \xi - \zeta \| ^ {2} \right] + \tau \operatorname {K L} (\pi | \pi_ {\sigma}): \pi \in \mathcal {P} (\Xi \times \Xi), \pi_ {1} = \mathrm {P}, \pi_ {2} = \mathrm {Q} \right\},
$$
where  $\pi_{\sigma}$  appears in the definition of the regularized robust risk (6). Besides, the regularization allows us to avoid Assumptions 4 and 5 to show our generalization result.
Theorem 3.4. For  $\sigma = \sigma_0\rho$  with  $\sigma_0 > 0$ ,  $\varepsilon = \varepsilon_0\rho$  with  $\varepsilon_0 > 0$  such that  $\varepsilon_0 / \sigma_0^2$  is small enough depending on  $\mathcal{F}$ ,  $\mathrm{P}$ ,  $\Xi$ , there is an explicit constant  $\rho_c$  depending only on  $\mathcal{F}$ ,  $\mathrm{P}$  and  $\Xi$  such that for all  $\delta \in (0,1)$  and  $n\geq 1$ , if
$$
\mathcal {O} \left(\sqrt {\frac {1 + \log 1 / \delta}{n}}\right) \leq \rho \leq \frac {\rho_ {c}}{2} - \mathcal {O} \left(\frac {1}{\sqrt {n}}\right), \quad a n d \quad \rho_ {c} \geq \mathcal {O} \left(\frac {1}{n ^ {1 / 6}} + \left(\frac {1 + \log 1 / \delta}{n}\right) ^ {1 / 4}\right),
$$
then, there are  $\tau = \mathcal{O}(\varepsilon \rho)$  and  $\rho_{n} = \mathcal{O}\left(\sqrt{\frac{1 + \log 1 / \delta}{n}}\right)$  such that, with probability at least  $1 - \delta$
$$
\forall f \in \mathcal {F}, \quad \widehat {\mathcal {R}} _ {\rho^ {2}} ^ {\varepsilon} (f) \geq \mathbb {E} _ {\xi \sim \mathrm {Q}} [ f (\xi) ] \quad \text {f o r a l l Q s u c h t h a t} W _ {2, \tau} ^ {2} (\mathrm {P}, \mathrm {Q}) \leq \rho (\rho - \rho_ {n}). \tag {10}
$$
Furthermore, when  $\sigma_0$  and  $\sigma$  are small enough depending on  $\mathrm{P}$  and  $\Xi$ , with probability  $1 - \delta$ ,
$$
\forall f \in \mathcal {F}, \quad \widehat {\mathcal {R}} _ {\rho^ {2}} ^ {\varepsilon} (f) \geq \mathbb {E} _ {\xi \sim \mathrm {P}} \mathbb {E} _ {\zeta \sim \pi_ {\sigma} (\cdot | \xi)} [ f (\zeta) ].
$$
The first part of the theorem, (10), guarantees that the empirical robust risk is an upper-bound on the loss even with some perturbations of the true distribution. As in OT, the regularization added to the Wasserstein metric induces a bias that may prevent  $W_{2,\tau}^2(\mathrm{P},\mathrm{P})$  from being null. As a result, the second part of the theorem involves a smoothed version of the true risk: the empirical robust risk provides an exact upper-bound the true expectation of a convolution of the loss with  $\pi_{\sigma}$ .
A few additional comments are in order:
- Our result prescribes the scaling of the regularization parameters:  $\varepsilon$  and  $\sigma$  should be taken proportional to  $\rho$ .
- The critical radius  $\rho_{c}$  has a slightly more intricate definition, yet the same interpretation as in the standard WDRO case inRemark 3.2; see Section D.2.
- The regularized OT distances do not suffer from the curse of dimensionality (Genevay et al., 2019). However this property does not directly carry over to regularized WDRO. Indeed, we cannot choose the same reference measure as in OT and we have to fix the measure  $\pi_{\sigma}$ , introducing a bias. As a consequence, we have to extend the analysis of the previous section to obtain the claimed guarantees that avoid the curse of dimensionality.
3.3 Idea of the proofs
In this section, we present the main ideas of the proofs of Theorem 3.1, Theorem 3.3, and Theorem 3.4. The full proofs are detailed in appendix; we point to relevant sections along the discussion. First, we recall the duality results for WDRO that play a crucial role in our analysis. Second, we present a rough sketch of proofs that is common to both the standard and the regularized cases. Finally, we provide a refinement of our results that is a by-product of our analysis.
Duality in WDRO. Duality has been a central tool in both the theoretical analyses and computational schemes of WDRO from the onset (Shafieezadeh Abadeh et al., 2015; Esfahani and Kuhn, 2018). The expressions of the dual of WDRO problems for both the standard case (Gao and Kleywegt, 2016; Blanchet and Murthy, 2019) and the regularized case (Wang et al., 2023; Azizian et al., 2023) can be written with the following dual generator function  $\phi$  defined as
$$
\phi (f, \xi , \lambda , \varepsilon , \sigma) := \left\{ \begin{array}{l l} \sup  _ {\zeta \in \Xi} \left\{f (\zeta) - \frac {\lambda}{2} \| \xi - \zeta \| ^ {2} \right\} & \text {i f} \varepsilon = 0 \\ \varepsilon \log \left(\mathbb {E} _ {\zeta \sim \pi_ {\sigma} (\cdot | \xi)} \exp \left(\frac {f (\zeta) - \lambda \| \xi - \zeta \| ^ {2} / 2}{\varepsilon}\right)\right) & \text {i f} \varepsilon > 0, \end{array} \right. \tag {11}
$$
where  $\lambda$  is the dual variable associated to the Wasserstein constraint in (3), (5) and (6). The effect of regularization appears here clearly as a smoothing of the supremum. Note also that this function depends on the conditional reference measures  $\pi_{\sigma}(\cdot |\xi)$  but not on other probability distributions. Then, under some general assumptions (specified in Section 2.3 in appendix), the existing strong duality results yield that the (regularized) empirical robust risk writes
$$
\widehat {\mathcal {R}} _ {\rho^ {2}} ^ {\varepsilon} (f) = \inf  _ {\lambda \geq 0} \lambda \rho^ {2} + \mathbb {E} _ {\xi \sim \widehat {\mathrm {P}} _ {\mathrm {n}}} [ \phi (f, \xi , \lambda , \varepsilon , \sigma) ], \tag {12}
$$
and, similarly, the (regularized) true robust risk writes
$$
\mathcal {R} _ {\rho^ {2}} ^ {\varepsilon} (f) = \inf  _ {\lambda \geq 0} \lambda \rho^ {2} + \mathbb {E} _ {\xi \sim \mathrm {P}} [ \phi (f, \xi , \lambda , \varepsilon , \sigma) ]. \tag {13}
$$
These expressions for the risks are the bedrock of our analysis.
Sketch of proof. In both the standard case and the regularized case, our proof is built on two main parts: the first part is to obtain a concentration bound on the dual problems that crucially relies on a lower bound of the dual multiplier; the second part then consists in establishing such a lower bound. All the bounds are valid with high probability, and we drop the dependency on the confidence level  $\delta$  of the theorems for simplicity.
For the first part of the proof (Section B), we assume that there is a deterministic lower-bound  $\underline{\lambda} > 0$  on the optimal dual multiplier in (12) that holds with high-probability. As a consequence, we can
restrict the range of  $\lambda$  in (12) to obtain:
$$
\widehat {\mathcal {R}} _ {\rho^ {2}} ^ {\varepsilon} (f) = \inf  _ {\lambda \geq \underline {{\lambda}}} \left\{\lambda \rho^ {2} + \mathbb {E} _ {\xi \sim \widehat {\mathrm {P}} _ {\mathrm {n}}} [ \phi (f, \xi , \lambda , \varepsilon , \sigma) ] \right\}
$$
$$
= \inf  _ {\lambda \geq \underline {{\lambda}}} \left\{\lambda \rho^ {2} + \mathbb {E} _ {\xi \sim \mathrm {P}} [ \phi (f, \xi , \lambda , \varepsilon , \sigma) ] - \lambda \frac {\mathbb {E} _ {\xi \sim \mathrm {P}} [ \phi (f , \xi , \lambda , \varepsilon , \sigma) ] - \mathbb {E} _ {\xi \sim \widehat {\mathrm {P}} _ {\mathrm {n}}} [ \phi (f , \xi , \lambda , \varepsilon , \sigma) ]}{\lambda} \right\}
$$
$$
\geq \inf  _ {\lambda \geq \underline {{\lambda}}} \left\{\lambda \rho^ {2} + \mathbb {E} _ {\xi \sim \mathrm {P}} [ \phi (f, \xi , \lambda , \varepsilon , \sigma) ] - \lambda \sup  _ {\lambda^ {\prime} \geq \underline {{\lambda}}} \frac {\mathbb {E} _ {\xi \sim \mathrm {P}} [ \phi (f , \xi , \lambda^ {\prime} , \varepsilon , \sigma) ] - \mathbb {E} _ {\xi \sim \widehat {\mathrm {P}} _ {\mathrm {n}}} [ \phi (f , \xi , \lambda^ {\prime} , \varepsilon , \sigma) ]}{\lambda^ {\prime}} \right\}
$$
$$
\geq \inf  _ {\lambda \geq \underline {{\lambda}}} \left\{\lambda \rho^ {2} + \mathbb {E} _ {\xi \sim \mathrm {P}} \left[ \phi (f, \xi , \lambda , \varepsilon , \sigma) \right] - \lambda \bar {\rho} _ {n} ^ {2} \right\}
$$
$$
\geq \mathcal {R} _ {\rho^ {2} - \bar {\rho} _ {n} ^ {2}} ^ {\varepsilon} (f). \tag {14}
$$
In the above, we used that the inner supremum, which is random, can be bounded by a deterministic and explicit quantity that we call  $\overline{\rho}_n^2$ , i.e.,
$$
\overline {{\rho}} _ {n} ^ {2} \geq \sup  _ {\lambda^ {\prime} \geq \underline {{\lambda}}} \frac {\mathbb {E} _ {\xi \sim \mathrm {P}} [ \phi (f , \xi , \lambda^ {\prime} , \varepsilon , \sigma) ] - \mathbb {E} _ {\xi \sim \widehat {\mathrm {P}} _ {\mathrm {n}}} [ \phi (f , \xi , \lambda^ {\prime} , \varepsilon , \sigma) ]}{\lambda^ {\prime}} \quad \text {w i t h h i g h p r o b a b i l i t y}.
$$
Hence, we obtain an upper-bound on the robust risk w.r.t. the true distribution with radius  $\rho^2 -\overline{\rho}_n^2$  Moreover, we show that  $\overline{\rho}_n^2 = \mathcal{O}(1 / (\underline{\lambda}\sqrt{n}))$  which highlights the need for a precise lower bound  $\underline{\lambda}$  to control the decrease in radius.
The second part of the proof thus consists in showing that the dual variable is indeed bounded away from 0, which means that the Wasserstein constraint is sufficiently active. We have to handle two cases differently:
- when  $\rho$  is small, i.e., close to  $\rho_{n}$  (Section C),
- when  $\rho$  is large, i.e., close to the critical radius  $\rho_c$  (Section D). Note that the additional Assumption 5 is required here: where we need to control the behaviors of  $f \in \mathcal{F}$  close to their maxima (see (11) for  $\varepsilon = 0$  and small  $\lambda$ ).
In both cases we obtain that  $\underline{\lambda}$  scales as  $1 / \rho$  for the respective ranges of admissible radiuses. As a consequence  $\overline{\rho}_n^2$  is bounded by  $\rho \rho_{n}$  with  $\rho_{n} = \mathcal{O}(1 / \sqrt{n})$  and (14) becomes
$$
\widehat {\mathcal {R}} _ {\rho^ {2}} ^ {\varepsilon} (f) \geq \mathcal {R} _ {\rho (\rho - \rho_ {n})} ^ {\varepsilon} (f), \tag {15}
$$
which leads to our main results.
Extension: upper and lower bounds on the empirical robust risk. The proof that we sketched above actually shows that  $\mathcal{R}_{\rho (\rho -\rho_n)}^{\varepsilon}(f)$  is a lower bound of  $\widehat{\mathcal{R}}_{\rho^2}^{\varepsilon}(f)$ . This proof technique also yields an upper bound by exchanging the roles of  $\mathrm{P}$  and  $\widehat{\mathrm{P}}_{\mathrm{n}}$ .
Theorem 3.5. In the setting of either Theorem 3.1, Theorem 3.3 or Theorem 3.4 (with  $\varepsilon = 0$  or  $\varepsilon > 0$ ), with probability at least  $1 - \delta$ , it holds that
$$
\forall f \in \mathcal {F}, \quad \mathcal {R} _ {\rho (\rho - \rho_ {n})} ^ {\varepsilon} (f) \leq \widehat {\mathcal {R}} _ {\rho^ {2}} ^ {\varepsilon} (f) \leq \mathcal {R} _ {\rho (\rho + \rho_ {n})} ^ {\varepsilon} (f),
$$
$$
w i t h \rho_ {n} = \mathcal {O} \left(\sqrt {\frac {1 + \log 1 / \delta}{n}}\right).
$$
This result shows how two robust objectives w.r.t. P provide upper and lower bounds on the empirical robust risk, with only slight variations in the radius. Furthermore, when the number of data points  $n$  grows, both sides of the bound converge to the same quantity  $\mathcal{R}_{\rho^2}^\varepsilon(f)$ . Hence our generalization bounds of the form (15) are asymptotically tight.
As a final remark, we underline that the proofs of this theorem and of the previous ones rely on the cost being the squared Euclidean norm and the extension to more general cost functions is left as future work. In particular, the Laplace approximation of Section A.3 in the regularized case and the analysis of Section D.1 in the standard WDRO case would need further work to accommodate general cost functions.
4 Examples: parametric models
Our main theorems Theorems 3.1, 3.3 and 3.4 involve a general class  $\mathcal{F}$  of loss functions. We explain in this section how to instantiate our results in the important class of parametric models. We then illustrate this setting with logistic regression and linear regression in Examples 4.1 and 4.2.
Let us consider the class of functions of the form
$$
\mathcal {F} = \{\xi \mapsto f (\theta , \xi): \theta \in \Theta \} \quad \text {w i t h} f: \Theta \times \Xi \longrightarrow \mathbb {R} \tag {16}
$$
where  $\Theta$ , the parameter space, is a subset of  $\mathbb{R}^p$  and  $\Xi$ , the sample space, is a subset of  $\mathbb{R}^d$ .
For instance, this covers the case of linear models of the form  $f(\theta, \xi) = \ell(\langle \xi, \theta \rangle)$  with  $\ell$  a convex loss. This class of models is studied by Shafieezadeh-Abadeh et al. (2019); Chen and Paschalidis (2018) in a slightly different setting, where they obtain a closed form for the robust objective and then establish a generalization bound similar to (9).
Let us show how to instantiate our theorems in the case of (16).
- If  $f$  is twice continuously differentiable on a neighborhood of  $\Theta \times \Xi$  with  $\Theta$  and  $\Xi$  both compact, then Assumption 2 is immediately satisfied. Therefore, Theorem 3.4 can be readily applied and its generalization guarantee hold.
- As for Assumption 4, it is equivalent to, for all  $\theta \in \Theta$ ,  $\mathrm{P}(\nabla_{\xi}f(\theta ,\xi)\neq 0) > 0$ . Thus disregarding the degenerate case of  $\nabla_{\xi}f(\theta ,\xi)$  being null for P-almost every  $\xi$  (e.g., when the loss does not depend on  $\xi$ ), we are in the setting of Theorem 3.3.
- Satisfying Assumption 5, needed for Theorem 3.1, requires some problem-dependent developments, see the examples below. Note though that the second item of Assumption 5 is implied by the parametric Morse-Bott property (Arbel and Mairal, 2022); see Section A.5.
We discuss linear and non-linear examples of this framework. In light of the above, we focus our discussion on Assumption 5. We first present the examples of linear models, Examples 4.1 and 4.2, where the latter assumption is satisfied. We then consider several examples of nonlinear models: kernel regression (Example 4.3), smooth neural networks (Example 4.4) and families of invertible mappings (Example 4.5). In Section H, we also provide numerical illustrations for linear models.
Example 4.1 (Logistic Regression). For a training sample  $(x,y)\in \mathbb{R}^p\times \{-1, + 1\}$ , the logistic loss for a parameter  $\theta \in \mathbb{R}^p$  is given by  $\log \bigl (1 + e^{-y\langle x,\theta\rangle}\bigr)$ . It fits into our framework by defining  $f(\theta ,\xi) = \log \bigl (1 + e^{\langle \xi ,\theta \rangle}\bigr)$  with  $\xi$  playing the role of  $-y\times x$ . We assume that  $\Theta$  is a compact set that does not include the origin, and, for the sake of simplicity, we take  $\Xi$  as a closed Euclidean ball, i.e.,  $\Xi = \overline{\mathbb{B}} (0,r)$ . We are going to show that Assumption 5 is satisfied, and, for this, we need the following elements. For any  $\theta$ , the maximizer of  $f(\theta ,\cdot)$  over  $\Xi = \overline{\mathbb{B}} (0,r)$  is reached at  $\xi^{*}\coloneqq \frac{r\theta}{\|\theta\|}$ . Besides, for any  $\xi \in \Xi$ , it holds that
$$
r ^ {2} \geq \| \xi \| ^ {2} = \| \xi^ {*} \| ^ {2} + 2 \langle \xi^ {*}, \xi - \xi^ {*} \rangle + \| \xi - \xi^ {*} \| ^ {2},
$$
so that, since  $\| \xi^{*}\| = r$ , we have
$$
\langle \xi^ {*}, \xi^ {*} - \xi \rangle \geq \frac {1}{2} \| \xi - \xi^ {*} \| ^ {2}. \tag {17}
$$
We can now turn to the verification of Assumption 5.
1. Take some  $R > 0$  and some  $\xi \in \Xi$  such that  $\| \xi - \xi^* \| \geq R$ . Then, (17) yields
$$
\langle \theta , \xi \rangle - \langle \theta , \xi^ {*} \rangle = \frac {\| \theta \|}{r} \langle \xi^ {*}, \xi - \xi^ {*} \rangle \leq - \frac {\| \theta \|}{2 r} \| \xi - \xi^ {*} \| ^ {2} \leq - \frac {d (0 , \Theta) R ^ {2}}{2 r}. \tag {18}
$$
Since  $u \mapsto \log(1 + e^u)$  is increasing, this yields that  $f(\theta, \xi) - f(\theta, \xi^*)$  is bounded away from 0 by a negative constant uniformly in  $\theta$ . The first item of Assumption 5 is thus satisfied.
2. Fix  $\theta \in \Theta$ ; by Taylor expanding  $u \mapsto \log(1 + e^u)$  around  $\langle \theta, \xi^* \rangle$  we get
$$
f (\theta , \xi) = f (\theta , \xi^ {*}) + \frac {1}{1 + e ^ {- \langle \theta , \xi^ {*} \rangle}} \langle \theta , \xi - \xi^ {*} \rangle + \mathcal {O} (\langle \theta , \xi - \xi^ {*} \rangle) ^ {2},
$$
where the big- $O$  remainder is uniform over  $\theta \in \Theta$ . Using the first inequality in (18), we get for  $\xi$  close enough to  $\xi^{*}$
$$
f (\theta , \xi) \leq f (\theta , \xi^ {*}) - \frac {1}{2 \left(1 + e ^ {- \langle \theta , \xi^ {*} \rangle}\right)} \langle \theta , \xi - \xi^ {*} \rangle \leq f (\theta , \xi^ {*}) - \frac {\| \theta \|}{4 r} \| \xi - \xi^ {*} \| ^ {2}.
$$
This shows that the second item of Assumption 5 is satisfied locally around  $\xi^{*}$ . It can be made global by using the uniform Lipschitz-continuity of  $f$ , which introduces a term of the form  $\frac{L}{6} \| \xi - \xi^{*} \|^{3}$ .
Example 4.2 (Linear Regression). With samples of the form  $\xi = (x,y)\in \mathbb{R}^p\times \mathbb{R}$  and parameters  $\theta \in \mathbb{R}^p$ , the loss is given by  $f(\theta ,\xi) = \frac{1}{2} (\langle \theta ,x\rangle -y)^2$ . Similarly to the previous example, we take  $\Theta$  as a compact set of  $\mathbb{R}^d$  that does not include the origin and  $\Xi$  of the form  $\overline{\mathbb{B}} (0,r)\times [-r',r']$ . The maximizers of  $f(\theta ,\cdot)$  on  $\Xi$  are  $\xi_1^* = (r\theta /\| \theta \| , - r')$  and  $\xi_2^* = (-r\theta /\| \theta \| ,r')$ . By symmetry, one can restrict to the case of  $\xi_1^*$  and  $\langle \theta ,x\rangle -y\geq 0$ ; the same rationale as above can then be applied.
Example 4.3 (Kernel Ridge Regression). Using a kernel  $k: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$  with  $\mathcal{X}$  compact and  $k$  smooth, for instance Gaussian or polynomial, we consider the following class of loss functions:
$$
f (\theta , \xi) = \frac {1}{2} \left(\sum_ {i = 1} ^ {m} \alpha_ {i} k (x, x _ {i}) - y\right) ^ {2} + \frac {\mu}{2} \| \alpha \| _ {2} ^ {2}.
$$
where  $\xi = (x,y)$ ,  $\Xi$  is some compact subset of  $\mathcal{X} \times \mathbb{R}$ ,  $\theta = (\alpha_1, \ldots, \alpha_m, x_1, \ldots, x_m)$ ,  $\Theta = A_m \times \mathcal{X}_m$ ,  $m$  is a fixed integer,  $A$  is a compact subset of  $\mathbb{R}^m$ ,  $\mathcal{X}_m$  can be any closed subset of  $\mathcal{X}^m$  and  $\mu \geq 0$  is the regularization parameter. A typical choice for  $\mathcal{X}_m$  would be the datapoints of the training set. This class then fits into our framework of parametric models above. Finally, further information on the kernel would be needed to ensure that Assumption 5 is satisfied.
Example 4.4 (Smooth Neural Networks). Denote by  $\mathcal{NN}(x,\theta,\sigma)$  a multi-linear perceptron that takes  $x$  as input, has weights  $\theta$  and a smooth activation function  $\sigma$ , for instance the hyperbolic tangent or the Gaussian Error Linear Units (GELU). We choose  $\ell(\hat{y},y)$  a smooth loss function and we consider the loss  $f(\theta,(x,y)) = \ell(\mathcal{NN}(x,\theta,\sigma),y)$  with  $\theta \in \Theta$  some compact set. Provided that the inputs  $(x,y)$  lie in a compact set  $\Xi$ , this class fits the parametric framework above. Note that we require  $\sigma$  to be smooth, further work would be required for non-smooth activation functions.
Example 4.5 (Family of diffeomorphisms). Consider maps  $h \colon \Xi \to \Xi$  and  $(\theta, \xi) \in \Theta \times \Xi \mapsto g_{\theta}(\xi) \in \Xi$  and define the parametric loss  $f(\theta, \xi) = h(g_{\theta}(\xi))$ . Assume that these functions are twice differentiable, that  $h$  satisfies the second item of Assumption 5 and that, for every  $\theta \in \Theta$ ,  $g_{\theta}$  has a inverse  $g_{\theta}^{-1}$  which is also continuously differentiable in a neighborhood of  $\Theta \times \Xi$ .
As before, this setting fits into the framework above. We now show that Assumption 5 is satisfied.
1. Since  $h$  is continuous,  $h$  satisfies the first item of Assumption 5. It is satisfied by  $\mathcal{F}$  as well thanks to  $g_{\theta}^{-1}$  being Lipschitz-continuous in  $\xi$  uniformly in  $\theta$  by compactness of  $\Theta \times \Xi$ .
2. Take  $C$  such that both  $g_{\theta}$  and  $g_{\theta}^{-1}$  are  $C$ -Lipschitz in  $\xi$  uniformly in  $\theta$ . Since  $\arg \max f(\theta, \cdot) = g_{\theta}^{-1}(\arg \max h)$ , it holds that  $\min_{\xi^{\star} \in \arg \max h} \| g_{\theta}(\xi) - \xi^{\star} \| = \min_{\xi^{\star} \in \arg \max f(\theta, \cdot)} \| g_{\theta}(\xi) - g_{\theta}(\xi^{\star}) \|$  which lies between  $C^{-1} \min_{\xi^{\star} \in \arg \max f(\theta, \cdot)} \| \xi - \xi^{\star} \|$  and  $C \min_{\xi^{\star} \in \arg \max f(\theta, \cdot)} \| \xi - \xi^{\star} \|$ . Combined with  $h$  satisfying the second item of Assumption 5, this shows that  $f$  satisfies this condition as well.
5 Conclusion and perspectives
In this work, we provide generalization guarantees for WDRO models that improve over existing literature in the following aspects: our results avoid the curse of dimensionality, provide exact upper bounds without spurious error terms, and allow for distribution shifts during testing. We obtained these bounds through the development of an original concentration result on the dual of WDRO. Our framework is general enough to cover regularized versions of the WDRO problem: they enjoy similar generalization guarantees as standard WDRO, with less restrictive assumptions.
Our work could be naturally extended in several ways. For instance, it might be possible to relax any of the assumptions (on the sample space, the sampling process, the Wasserstein metric, and the class of functions) at the expense of additional technical work. Moreover, the crucial role played by the radius of the Wasserstein ball calls for a principled and efficient procedure to select it.
This work has been supported by MIAI Grenoble Alpes (ANR-19-P3IA-0003).
J. Wang, R. Moore, Y. Xie, and R. Kamaleswaran. Improving sepsis prediction model generalization with optimal transport. In Machine Learning for Health, pages 474-488. PMLR, 2022.
J. Wang, R. Gao, and Y. Xie. Sinkhorn distributionally robust optimization. arXiv preprint arXiv:2109.11926, 2023.
Y. Zeng and H. Lam. Generalization bounds with minimal dependency on hypothesis class via distributionally robust optimization. Advances in Neural Information Processing Systems, 35: 27576-27590, 2022.
C. Zhao and Y. Guan. Data-driven risk-averse stochastic optimization with Wasserstein metric. Operations Research Letters, 46(2):262-267, 2018.
J.-J. Zhu, W. Jitkrittum, M. Diehl, and B. Scholkopf. Kernel distributionally robust optimization: Generalized duality theorem and stochastic approximation. In International Conference on Artificial Intelligence and Statistics, pages 280-288. PMLR, 2021a.
J.-J. Zhu, C. Kouridi, Y. Nemmour, and B. Schölkopf. Adversarially robust kernel smoothing. arXiv preprint arXiv:2102.08474, 2021b.