SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Natalie S. Frank
nf1066@nyu.edu
Jonathan Niles-Weed
jnw@cims.nyu.edu
We study the consistency of surrogate risks for robust binary classification. It is common to learn robust classifiers by adversarial training, which seeks to minimize the expected 0-1 loss when each example can be maliciously corrupted within a small ball. We give a simple and complete characterization of the set of surrogate loss functions that are consistent, i.e., that can replace the 0-1 loss without affecting the minimizing sequences of the original adversarial risk, for any data distribution. We also prove a quantitative version of adversarial consistency for the  $\rho$ -margin loss. Our results reveal that the class of adversially consistent surrogates is substantially smaller than in the standard setting, where many common surrogates are known to be consistent.
A central issue in the study of neural nets is their susceptibility to adversarial perturbations—perturbations imperceptible to the human eye can cause a neural net to misclassify an image [Szegedy et al., 2013, Biggio et al., 2013]. The same phenomenon appears in other types of data such as speech and text. As deep nets are used in applications such as self-driving cars and medical imaging [Paschali et al., 2018, Li et al., 2021], training classifiers robust to adversarial perturbations is a central question in machine learning.
The foundational theory of surrogates for classification in well understood. In the standard classification setting, one seeks to minimize the classification risk—the proportion of incorrectly classified data. Since minimizing the classification risk is typically computationally intractable [Ben-David et al., 2003], a common approach is to instead minimize a better-behaved alternative called the surrogate risk. However, one must verify that classifiers with low surrogate risk also achieve low classification risk. If for every data distribution, a sequence of functions minimizing the surrogate also minimizes the classification risk, the surrogate risk is called consistent. Many classic papers study the consistency of surrogate risks in the standard classification setting [Bartlett et al., 2006, Lin, 2004, Steinwart, 2007, Philip M. Long, 2013, Mingyuan Zhang, 2020].
Unlike the standard case, however, little is known about the consistency of surrogate risks in the context of adversarial training, which involves risks that compute the supremum of a surrogate loss function over an  $\epsilon$ -ball. Though this question has been partially studied in the literature [Awasthi et al., 2021a,c, Meunier et al., 2022], a general theory is lacking. Existing results reveal, however, that the situation is substantially different from the standard case: for instance, [Meunier et al., 2022] show that no convex surrogate can be adversarially consistent. To our knowledge, no adversarily consistent risks are known.
In this work, we give a complete characterization of adversarial consistency for surrogate losses.
- In Section 4 we give a surprisingly simple necessary and sufficient condition for adversarial consistency:
Informal Theorem. Under reasonable assumptions on the surrogate loss  $\phi$ , the supremum-based  $\phi$ -risk is adversely consistent if and only if  $\inf_{\alpha} \phi(\alpha) / 2 + \phi(-\alpha) / 2 < \phi(0)$ .
In particular, this result proves consistency for any loss function that is not midpoint convex at the origin.
- In Section 5, we specialize to the case of the  $\rho$ -margin loss, where we obtain a quantitative proof of adversarial consistency by explicitly bounding the excess adversarial risk.
To the best of the authors' knowledge, this paper is the first to prove that a loss-based learning procedure is consistent for a wide range of distributions in the adversarial setting. As mentioned above, the  $\rho$ -margin loss  $\phi_{\rho}(\alpha) = \min(1, \max(1 - \alpha / \rho, 0))$  satisfies the conditions of Informal Theorem above, as does the shifted sigmoid loss  $\phi_{\tau}(\alpha) = 1 / (1 + \exp(\alpha - \tau))$  with  $\tau > 0$ , which confirms a conjecture of Meunier et al. [2022]. By contrast, all convex losses satisfy  $\inf_{\alpha} \phi(\alpha) / 2 + \phi(-\alpha) / 2 = \phi(0)$ , and are therefore not adversarily consistent.
In addition to consistency, one would hope to obtain a quantitative comparison between the adversarial surrogate risk and the adversarial classification risk. Our bound in Section 5 shows that the excess error of the adversarial  $\rho$ -margin loss is a linear upper bound on the adversarial classification error, which implies that minimizing the adversarial  $\rho$ -margin loss is an effective procedure for minimizing the adversarial classification error. Extending the bound in Section 5 to further losses remains an open question.
Many previous works have studied the consistency of surrogate risks [Bartlett et al., 2006, Lin, 2004, Steinwart, 2007, Philip M. Long, 2013, Mingyuan Zhang, 2020]. The classic papers by [Bartlett et al., 2006, Lin, 2004, Zhang, 2004] explore the consistency of surrogate risks over all measurable functions. The works [Philip M. Long, 2013, Mingyuan Zhang, 2020, Awasthi et al., 2022] study  $\mathcal{H}$ -consistency, which is consistency restricted to a smaller set of functions. Steinwart [2007] generalizes some of these results into a framework referred to as calibration. Awasthi et al. [2021a], Bao et al. [2021], Awasthi et al. [2021c], Meunier et al. [2022] then use this framework to analyze the calibration of adversarial surrogate losses. Furthermore Meunier et al. [2022] relate calibration to consistency for adversarial losses in certain cases — they show that no convex loss is adversarily consistent. They also conjecture that a class of surrogate losses called the odd shifted losses are adversially consistent. Meunier et al. [2022] also show that in a restricted setting, surrogates are consistent for 'optimal attacks'. The proof of our result formalizes this intuition. Simultaneous work [Mao et al., 2023] shows that the  $\rho$ -margin loss is adversarily  $\mathcal{H}$ -consistent for typical function classes. Lastly, Bhattacharjee and Chaudhuri [2020, 2021] use a different set of techniques to study the consistency of non-parametric methods in adversarial scenarios.
Our results rely on recent works establishing the properties of minimizers to surrogate adversarial risks. [Awasthi et al., 2021b, Pydi and Jog, 2021, Bungert et al., 2021] all proved the existence of minimizers to the adversarial risk and [Pydi and Jog, 2021] proved a minimax theorem for the zero-one loss. Building on the work of [Pydi and Jog, 2021], [Frank and Niles-Weed, 2023] later proved similar existence and minimax statements for arbitrary surrogate losses. Trillos et al. [2022, 2023] extend some of these results to the multiclass case. Lastly, [Trillos and Murray, 2020] study further properties of the minimizers to the adversarial classification loss.
This section contains the necessary background for our results. Section 3.1 gives precise definitions for the main concepts, and Section 3.2 describes the minimax theorems that are at the heart of our proof.
3.1 Surrogate Risks
This paper studies binary classification on  $\mathbb{R}^d$ . Explicitly, labels are  $\{-1, + 1\}$  and the data is distributed according to a distribution  $\mathcal{D}$  on the set  $\mathbb{R}^d\times \{-1, + 1\}$ . The measures  $\mathbb{P}_1,\mathbb{P}_0$  define the relative probabilities of finding points with a given label in a region of  $\mathbb{R}^d$ . Formally, define measures on  $\mathbb{R}^d$  by
$$
\mathbb {P} _ {1} (A) = \mathcal {D} (A \times \{+ 1 \}), \mathbb {P} _ {0} (A) = \mathcal {D} (A \times \{- 1 \}).
$$
The classification risk  $R(f)$  is then the probability of misclassifying a point under  $\mathcal{D}$ :
$$
R (f) = \int \mathbf {1} _ {f (\mathbf {x}) \leq 0} d \mathbb {P} _ {1} + \int \mathbf {1} _ {f (\mathbf {x}) > 0} d \mathbb {P} _ {0}. \tag {1}
$$
The surrogate to  $R$  is
$$
R _ {\phi} (f) = \int \phi (f) d \mathbb {P} _ {1} + \int \phi (- f) d \mathbb {P} _ {0}. \tag {2}
$$
A classifier can be obtained by minimizing either  $R$  or  $R_{\phi}$  over the set of all measurable functions. A point  $\mathbf{x}$  is then classified according to sign  $f$ . There are many possible choices for  $\phi$  — typically one chooses a loss that is easy to optimize. In this paper, we assume that
Assumption 1.  $\phi$  is non-increasing, non-negative, continuous, and  $\lim_{\alpha \to \infty}\phi (\alpha) = 0$
Most surrogate losses in machine learning satisfy this assumption. Learning algorithms typically optimize the risk in (2) using an iterative procedure, which produces a sequence of functions that minimizes  $R_{\phi}$ . We call  $R_{\phi}$  a consistent risk and  $\phi$  a consistent loss if for all distributions, every minimizing sequence of  $R_{\phi}$  is also a minimizing sequence of  $R$ . Alternatively, the risks  $R$ ,  $R_{\phi}$  can be expressed in terms of the quantities  $\mathbb{P} = \mathbb{P}_0 + \mathbb{P}_1$  and  $\eta = d\mathbb{P}_1 / d\mathbb{P}$ . For all  $\eta \in [0,1]$ , define
$$
C (\eta , \alpha) = \eta \mathbf {1} _ {\alpha \leq 0} + (1 - \eta) \mathbf {1} _ {\alpha > 0}, \quad C ^ {*} (\eta) = \inf  _ {\alpha} C (\eta , \alpha), \tag {3}
$$
$$
C _ {\phi} (\eta , \alpha) = \eta \phi (\alpha) + (1 - \eta) \phi (- \alpha), \quad C _ {\phi} ^ {*} (\eta) = \inf  _ {\alpha} C _ {\phi} (\eta , \alpha) \tag {4}
$$
For more on the definitions of  $R, R_{\phi}, C, C_{\phi}$ , see [Bartlett et al., 2006] or Sections 3.1 and 3.2 of [Frank and Niles-Weed, 2023]. Using these definitions,  $R(f) = \int C(\eta(\mathbf{x}), f(\mathbf{x})) d\mathbb{P}$  and
$$
R _ {\phi} (f) = \int C _ {\phi} (\eta (\mathbf {x}), f (\mathbf {x})) d \mathbb {P} \tag {5}
$$
This alternative view of the risks  $R$  and  $R_{\phi}$  provides a 'pointwise' criterion for consistency—if the function  $f(\mathbf{x})$  minimizes  $C_{\phi}(\eta(\mathbf{x}), \cdot)$  at each point, then it also minimizes  $R_{\phi}$ . However, minimizers to  $C_{\phi}(\eta, \cdot)$  over  $\mathbb{R}$  do not always exist—consider for instance  $\eta = 1$  for the exponential loss  $\phi(\alpha) = e^{-\alpha}$ . In general, for minimizers of  $C_{\phi}(\eta, \cdot)$  to exist, one must work over the extended real numbers  $\overline{\mathbb{R}} = \mathbb{R} \cup \{-\infty, +\infty\}$ . The following proposition proved in Appendix A implies that 'pointwise' considerations also extend to minimizing sequences of functions.
Proposition 1. The following are equivalent:
1)  $\phi$  is consistent
2) Every minimizing sequence of  $C_{\phi}(\eta, \cdot)$  is also a minimizing sequence of  $C(\eta, \cdot)$ .
3) Every  $\overline{\mathbb{R}}$ -valued minimizer of  $R_{\phi}$  is a minimizer of  $R$
This result is well-known in prior literature; in particular the equivalence between 2) and 3) is closely related to the equivalence between calibration and consistency in the non-adversarial setting [Steinwart, 2007]. Most importantly, the equivalence between 1) and 3) reduces studying minimizing sequences of functionals to studying minimizers of functions. We will show that the equivalence
In the context of standard (non-adversarial) learning, the concept we defined as consistency is often referred to as calibration, see for instance [Bartlett et al., 2006, Steinwart, 2007]. We opt for the term 'consistency' as the prior works [Awasthi et al., 2021a,c, Meunier et al., 2022] use calibration to refer to a different but related concept in the adversarial setting.
between 1) and 2) has an analog in the adversarial scenario, but the equivalence between 1) and 3) does not.
In the adversarial classification setting, every  $x$ -value is perturbed by a malicious adversary before undergoing classification by  $f$ . We assume that these perturbations are bounded by  $\epsilon$  in some norm  $\| \cdot \|$  and furthermore, the adversary knows both our classifier  $f$  and the true label of the point  $\mathbf{x}$ . In other words,  $f$  misclassifies  $(\mathbf{x}, y)$  when there is a point  $\mathbf{x}' \in \overline{B_{\epsilon}(\mathbf{x})}$  for which  $\mathbf{1}_{f(\mathbf{x}') \leq 0} = 1$  for  $y = +1$  and  $\mathbf{1}_{f(\mathbf{x}') > 0} = 1$  for  $y = -1$ . Conveniently, this criterion can be expressed in terms of suprema. For any function  $g$ , we define
$$
S_{\epsilon}(g)(\mathbf{x}) = \sup_{\| \mathbf{h}\| \leq \epsilon}g(\mathbf{x} + \mathbf{h})
$$
A point  $\mathbf{x}$  with label  $+1$  is misclassified when  $S_{\epsilon}(\mathbf{1}_{f\leq 0})(\mathbf{x}) = 1$  and a point  $\mathbf{x}$  with label  $-1$  is misclassified when  $S_{\epsilon}(\mathbf{1}_{f > 0})(\mathbf{x}) = 1$ . Hence the expected fraction of errors under the adversarial attack is
$$
R ^ {\epsilon} (f) = \int S _ {\epsilon} \left(\mathbf {1} _ {f \leq 0}\right) d \mathbb {P} _ {1} + \int S _ {\epsilon} \left(\mathbf {1} _ {f > 0}\right) d \mathbb {P} _ {0}, \tag {6}
$$
which is called the adversarial classification risk  $\mathbf{r}^{2}$ . Again, optimizing the empirical version of (6) is computationally intractable so instead one minimizes a surrogate of the form
$$
R _ {\phi} ^ {\epsilon} (f) = \int S _ {\epsilon} (\phi \circ f) d \mathbb {P} _ {1} + \int S _ {\epsilon} (\phi \circ - f) d \mathbb {P} _ {0} \tag {7}
$$
Due to the supremum in this expression, we refer to such a risk as a supremum-based surrogate. We define adversarial consistency as
Definition 1. The risk  $R_{\phi}^{\epsilon}$  is adversarially consistent if for every data distribution, every sequence  $f_{n}$  which minimizes  $R_{\phi}^{\epsilon}$  over all Borel measurable functions also minimizes  $R^{\epsilon}$ . We say that the loss  $\phi$  is adversarially consistent if the risk  $R_{\phi}^{\epsilon}$  is adversarially consistent.
Many convex and non-convex losses are consistent in standard classification [Bartlett et al., 2006, Zhang, 2004, Steinwart, 2007, Reid and Williamson, 2009, Lin, 2004]. By contrast, adversarial consistency often fails. For instance, Meunier et al. [2022] show that convex losses are not adversarily consistent. Furthermore, their example shows that the equivalence between 1) and 3) in Proposition 1 does not hold in the adversarial context. Thus, to understand adversarial consistency, it does not suffice to compare minimizers of  $R_{\phi}^{\epsilon}$  and  $R^{\epsilon}$ . To illustrate this distinction, we show the following result, adapted from [Meunier et al., 2022].
Proposition 2. Assume that  $\inf_{\alpha}\phi (\alpha) / 2 + \phi (-\alpha) / 2 = \phi (0)$ . Then  $\phi$  is not adversely consistent.
Proof. Let  $\mathbb{P}_0 = \mathbb{P}_1$  be the uniform distribution on the ball  $\overline{B_R(0)}$  and let  $\epsilon = 2R$ . Let  $\phi$  be a loss function for which  $\inf_{\alpha} \phi(\alpha) / 2 + \phi(-\alpha) / 2 = C_\phi^*(1/2) = \phi(0)$ . Notice that  $\inf_f R^\epsilon(f) \geq \inf_f R(f)$  and  $\inf_f R_\phi^\epsilon(f) \geq \inf_f R_\phi(f)$ . Since  $\mathbb{P}_0 = \mathbb{P}_1$ , the optimal non-adversarial risk is  $\inf_f R(f) = 1/2$ . Moreover, as  $C_\phi^*(1/2) = \phi(0)$ , the optimal non-adversarial surrogate risk is  $\inf_f R_\phi(f) = C_\phi^*(1/2) = \phi(0)$ . Thus, for the function  $f^* \equiv 0$ ,  $R^\epsilon(f^*) = \inf_f R(f) = 1/2$  and  $R_\phi^\epsilon(f^*) = \inf_f R_\phi(f) = \phi(0)$ . Therefore  $f^*$  minimizes both  $R_\phi^\epsilon$  and  $R^\epsilon$ . Now consider the sequence of functions
$$
f _ {n} (\mathbf {x}) = \left\{ \begin{array}{l l} \frac {1}{n} & \mathbf {x} = 0 \\ - \frac {1}{n} & \mathbf {x} \neq 0 \end{array} \right.
$$
Because  $\epsilon = 2R$ , every point in the support of the distribution can be perturbed to every other point. Thus  $S_{\epsilon}(\phi \circ f_n)(\mathbf{x}) = \phi(-1/n)$  and  $S_{\epsilon}(\phi \circ -f_n)(\mathbf{x}) = \phi(-1/n)$ . However,  $S_{\epsilon}(\mathbf{1}_{f \leq 0}) = 1$  and  $S_{\epsilon}(\mathbf{1}_{f > 0}) = 1$ . Therefore,  $R_{\phi}^{\epsilon}(f_n) = \phi(-1/n)$  while  $R^{\epsilon}(f_n) = 1$  for all  $n$ . As  $\phi$  is continuous,  $\lim_{n \to \infty} R_{\phi}^{\epsilon}(f_n) = \phi(0)$ . Thus  $f_n$  is a minimizing sequence of  $R_{\phi}^{\epsilon}$  but not of  $R^{\epsilon}$ , so  $\phi$  is not adversely consistent.
$^2$ Defining this integral requires some care because for a Borel function  $g$ ,  $S_{\epsilon}(g)$  may not be measurable; see Section 3.3 and Appendix A of [Frank and Niles-Weed, 2023] for details.
This example shows that if  $C_{\phi}^{*}(1 / 2) = \phi (0)$ , then  $\phi$  is not adversarially consistent. The main result of this paper is that this is the only obstruction to adversarial consistency:  $\phi$  is adversarially consistent if and only if  $C_{\phi}^{*}(1 / 2) < \phi (0)$ .
We begin by showing that this condition suffices for consistency in the non-adversarial setting. Surprisingly, despite the wealth of work on this topic, this condition does not appear to be known.
Proposition 3. If  $C_{\phi}^{*}(1 / 2) < \phi (0)$ , then  $\phi$  is consistent.
See Appendix C for a proof.
Again, some losses that satisfy this property are the  $\rho$ -margin loss  $\phi_{\rho}(\alpha) = \min(1, \max(1 - \alpha / \rho, 0))$  and the shifted sigmoid loss proposed by Meunier et al. [2022],  $\phi(\alpha) = 1 / (1 + \exp(\alpha - \tau))$ ,  $\tau > 0$ . (In fact, one can show that the class of shifted odd losses proposed by Meunier et al. [2022] satisfy  $C_{\phi}^{*}(1/2) < \phi(0)$ .
Notice that all convex losses satisfy  $C_{\phi}^{*}(1 / 2) = \phi (0)$ :
$$
C _ {\phi} ^ {*} (1 / 2) = \inf _ {\alpha} \frac {1}{2} \phi (\alpha) + \frac {1}{2} \phi (- \alpha) \geq \phi (0)
$$
The opposite inequality follows from the observation that  $C_{\phi}^{*}(1 / 2) \leq C_{\phi}(1 / 2,0) = \phi (0)$ . In contrast, recall that a convex loss  $\phi$  with  $\phi^{\prime}(0) < 0$  is consistent [Bartlett et al., 2006].
As conjectured by prior work Bao et al. [2021], Meunier et al. [2022], the fundamental reason losses with  $C_{\phi}^{*}(1/2) < \phi(0)$  are adversarially consistent is that minimizers of  $C_{\phi}(\eta, \cdot)$  are uniformly bounded away from 0 for all  $\eta$ :
Lemma 1. The loss  $\phi$  satisfies  $C_{\phi}^{*}(1 / 2) < \phi(0)$  iff there is an  $a > 0$  for which any minimizer  $\alpha^{*}$  of  $C_{\phi}(\eta, \cdot)$  satisfies  $|\alpha| \geq a$ .
See C for a proof. Concretely, one can show that for the  $\rho$ -margin loss  $\phi_{\rho}$ , a minimizer  $\alpha^{*}$  of  $C_{\phi_{\rho}}(\eta, \cdot)$  must satisfy  $|\alpha^{*}| \geq \rho$ . Similarly, a minimizer  $\alpha^{*}$  of  $C_{\phi_{\tau}}(\eta, \cdot)$  of the shifted sigmoid loss  $\phi_{\tau} = 1 / (1 + \exp(\alpha - \tau))$ ,  $\tau > 0$  is always either  $-\infty$  or  $+\infty$ . In 4, we use this property to show that minimizing sequences of  $R_{\phi}^{\epsilon}$  must be uniformly bounded away from zero, thus ruling out the counterexample presented in Proposition 2.
3.2 Minimax Theorems for Adversarial Risks
We study the consistency of  $\phi$  by comparing minimizing sequences of  $R_{\phi}^{\epsilon}$  with those of  $R^{\epsilon}$ . In the next section, in order to compare these minimizing sequences, we will attempt to re-write the adversarial loss in a 'pointwise' manner similar to Proposition 1. In order to achieve this representation of the adversarial loss, we apply minimax and complimentary slackness theorems from [Pydi and Jog, 2021, Frank and Niles-Weed, 2023].
Before presenting these results, we introduce the  $\infty$ -Wasserstein metric from optimal transport. For two finite probability measures  $\mathbb{Q}, \mathbb{Q}'$  satisfying  $\mathbb{Q}(\mathbb{R}^d) = \mathbb{Q}'(\mathbb{R}^d)$ , let  $\Pi(\mathbb{Q}, \mathbb{Q}')$  be the set of couplings between  $\mathbb{Q}$  and  $\mathbb{Q}'$ :
$$
\Pi (\mathbb {Q}, \mathbb {Q} ^ {\prime}) = \{\gamma : \text {m e a s u r e o n} \mathbb {R} ^ {d} \times \mathbb {R} ^ {d} \text {w i t h} \gamma (A \times \mathbb {R} ^ {d}) = \mathbb {Q} (A), \gamma (\mathbb {R} ^ {d} \times A) = \mathbb {Q} ^ {\prime} (A) \}
$$
The distance between  $\mathbb{Q}'$  and  $\mathbb{Q}$  in the Wasserstein  $\infty$ -metric  $W_{\infty}$  is defined as
$$
W_{\infty}(\mathbb{Q},\mathbb{Q}^{\prime}) = \inf_{\gamma \in \Pi (\mathbb{Q},\mathbb{Q}^{\prime})}\operatorname *{ess  sup}_{(\mathbf{x},\mathbf{y})\sim \gamma}\| \mathbf{x} - \mathbf{y}\| .
$$
The  $W_{\infty}$  distance is in fact a metric on the space of measures. We denote the  $\infty$ -Wasserstein ball around a measure  $\mathbb{Q}$  by
$$
\mathcal {B} _ {\epsilon} ^ {\infty} (\mathbb {Q}) = \left\{\mathbb {Q} ^ {\prime}: \mathbb {Q} ^ {\prime} \text {B o r e l}, \quad W _ {\infty} (\mathbb {Q}, \mathbb {Q} ^ {\prime}) \leq \epsilon \right\}
$$
Informally, the measure  $\mathbb{Q}'$  is in  $\mathcal{B}_{\epsilon}^{\infty}(\mathbb{Q})$  if perturbing points by at most  $\epsilon$  under the measure  $\mathbb{Q}$  can produce  $\mathbb{Q}'$ . As a result, Wasserstein  $\infty$ -balls are fairly useful for modeling adversarial attacks. Specifically, one can show:
Lemma 2. For any function  $g$  and measures  $\mathbb{Q}'$ ,  $\mathbb{Q}$  with  $W_{\infty}(\mathbb{Q}', \mathbb{Q}) \leq \epsilon$ , the inequality  $\int S_{\epsilon}(g)d\mathbb{Q} \geq \int gd\mathbb{Q}'$  holds.
See Appendix D for a proof.
Minimax theorems from prior work use this framework to introduce dual problems to the adversarial classification risks (6) and (7). Let  $\mathbb{P}_0^{\prime},\mathbb{P}_1^{\prime}$  be finite Borel measures and define
$$
\bar {R} \left(\mathbb {P} _ {0} ^ {\prime}, \mathbb {P} _ {1} ^ {\prime}\right) = \int C ^ {*} \left(\frac {d \mathbb {P} _ {1} ^ {\prime}}{d \left(\mathbb {P} _ {0} ^ {\prime} + \mathbb {P} _ {1} ^ {\prime}\right)}\right) d \left(\mathbb {P} _ {0} ^ {\prime} + \mathbb {P} _ {1} ^ {\prime}\right) \tag {8}
$$
where  $C^*$  is defined by (3). The next theorem states that maximizing  $\bar{R}$  over  $W_{\infty}$  balls is in fact a dual problem to minimizing  $R^{\epsilon}$ .
Theorem 1. Let  $\bar{R}$  be defined by (8).
$$
\inf  _ {\substack {f \text {Borel} \\ \mathbb {R} \text {-valued}}} R ^ {\epsilon} (f) = \sup  _ {\substack {\mathbb {P} _ {0} ^ {\prime} \in \mathcal {B} _ {\epsilon} ^ {\infty} (\mathbb {P} _ {0}) \\ \mathbb {P} _ {1} ^ {\prime} \in \mathcal {B} _ {\epsilon} ^ {\infty} (\mathbb {P} _ {1})}} \bar {R} \left(\mathbb {P} _ {0} ^ {\prime}, \mathbb {P} _ {1} ^ {\prime}\right) \tag{9}
$$
and furthermore equality is attained for some Borel measurable  $\hat{f}$  and  $\hat{\mathbb{P}}_1, \hat{\mathbb{P}}_0$  with  $W_{\infty}(\hat{\mathbb{P}}_0, \mathbb{P}_0) \leq \epsilon$  and  $W_{\infty}(\hat{\mathbb{P}}_1, \mathbb{P}_1) \leq \epsilon$ .
The first to show such a theorem was Pydi and Jog [2021]. In comparison to their Theorem 8, Theorem 1 removes the assumption that  $\mathbb{P}_0, \mathbb{P}_1$  are absolutely continuous with respect to Lebesgue measure and shows that the minimizer  $\hat{f}$  is in fact Borel. We prove this theorem in Appendix E. Frank and Niles-Weed [2023] prove a similar statement for the surrogate risk  $R_{\phi}^{\epsilon}$ . This time, the dual objective is
$$
\bar {R} _ {\phi} \left(\mathbb {P} _ {0} ^ {\prime}, \mathbb {P} _ {1} ^ {\prime}\right) = \int C _ {\phi} ^ {*} \left(\frac {d \mathbb {P} _ {1} ^ {\prime}}{d \left(\mathbb {P} _ {0} ^ {\prime} + \mathbb {P} _ {1} ^ {\prime}\right)}\right) d \left(\mathbb {P} _ {0} ^ {\prime} + \mathbb {P} _ {1} ^ {\prime}\right) \tag {10}
$$
with  $C_{\phi}^{*}$  defined by (4).
Theorem 2. Assume that Assumption 1 holds, and define  $\bar{R}_{\phi}$  by (10). Then
$$
\inf  _ { \begin{array}{c} f B o r e l, \\ f \mathbb {R} - v a l u e d \end{array} } R _ {\phi} ^ {\epsilon} (f) = \sup  _ { \begin{array}{c} \mathbb {P} _ {0} ^ {\prime} \in \mathcal {B} _ {\epsilon} ^ {\infty} (\mathbb {P} _ {0}) \\ \mathbb {P} _ {1} ^ {\prime} \in \mathcal {B} _ {\epsilon} ^ {\infty} (\mathbb {P} _ {1}) \end{array} } \bar {R} _ {\phi} \left(\mathbb {P} _ {0} ^ {\prime}, \mathbb {P} _ {1} ^ {\prime}\right) \tag {11}
$$
and furthermore equality in the dual problem is attained for some  $\mathbb{P}_1^*,\mathbb{P}_0^*$  with  $W_{\infty}(\mathbb{P}_0^*,\mathbb{P}_0)\leq \epsilon$  and  $W_{\infty}(\mathbb{P}_1^*,\mathbb{P}_1)\leq \epsilon$