SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Frank and Niles-Weed [2023] proved this statement in Theorem 6 but with the infimum taken over  $\overline{\mathbb{R}}$ -valued functions. To extend the result to  $\mathbb{R}$ -valued functions as in Theorem 2, we show that  $\inf_{f \text{ Borel}, f \text{ Borel-valued}} R_{\phi}^{\epsilon}(f) = \inf_{f \text{ Borel}, f \text{ Borel-valued}} R_{\phi}^{\epsilon}(f)$  in Appendix B.
4 Adversarily Consistent Losses
This section contains our main results on adversarial consistency. In light of Proposition 2, our main task is to show that a loss satisfying  $C_{\phi}^{*}(1 / 2) < \phi(0)$  is adversarially consistent.
At a high level, we will show that every minimizing sequence of  $R_{\phi}^{\epsilon}$  must also minimize  $R^{\epsilon}$ . However, directly analyzing minimizing sequences  $\{f_n\}$  of  $R_{\phi}^{\epsilon}$  and  $R^{\epsilon}$  is challenging due to the supremums in the definitions of the adversarial risks. We therefore develop alternate characterizations of minimizing sequences to both functionals, based on complimentary slackness conditions derived from the convex duality results of Section 3.2. However, unlike standard complementary slackness conditions well known from convex optimization, these theorems allow us to characterize minimizing sequences as well as minimizers.
4.1 Approximate Complimentary Slackness
We first state this slackness result for the surrogate case, due to Frank and Niles-Weed [2023, Lemmas 16 and 26] and Theorem 2.
Proposition 4. Let  $(\mathbb{P}_0^*,\mathbb{P}_1^*)$  be any maximizers of  $\bar{R}_{\phi}$  over  $\mathcal{B}_{\epsilon}^{\infty}(\mathbb{P}_i)$ . Define  $\mathbb{P}^{*} = \mathbb{P}_{0}^{*} + \mathbb{P}_{1}^{*}$ ,  $\eta^{*} = d\mathbb{P}_{1}^{*} / d\mathbb{P}^{*}$ . If  $f_{n}$  is a minimizing sequence for  $R_{\phi}^{\epsilon}$ , then the following hold:
$$
\lim  _ {n \rightarrow \infty} \int C _ {\phi} \left(\eta^ {*}, f _ {n}\right) d \mathbb {P} ^ {*} = \int C _ {\phi} ^ {*} \left(\eta^ {*}\right) d \mathbb {P} ^ {*}. \tag {12}
$$
$$
\lim  _ {n \rightarrow \infty} \int S _ {\epsilon} \left(\phi \circ f _ {n}\right) d \mathbb {P} _ {1} - \int \phi \circ f _ {n} d \mathbb {P} _ {1} ^ {*} = 0, \quad \lim  _ {n \rightarrow \infty} \int S _ {\epsilon} \left(\phi \circ - f _ {n}\right) d \mathbb {P} _ {0} - \int \phi \circ - f _ {n} d \mathbb {P} _ {0} ^ {*} = 0 \tag {13}
$$
Proof. Let  $R_{\phi,*}^{\epsilon}$  be the minimal value of  $R_{\phi}^{\epsilon}$  and choose a  $\delta > 0$ . Then for sufficiently large  $N$ ,  $n \geq N$  implies that  $R_{\phi}^{\epsilon}(f_n) \leq R_{\phi,*}^{\epsilon} + \delta$ . Lemma 2 and the definition of  $C_{\phi}^{*}$  in (4) further imply that
$$
R _ {\phi , *} ^ {\epsilon} + \delta \geq \int S _ {\epsilon} (\phi \circ f _ {n}) d \mathbb {P} _ {1} + \int S _ {\epsilon} (\phi \circ - f _ {n}) d \mathbb {P} _ {0} \geq \int \phi \circ f _ {n} d \mathbb {P} _ {1} ^ {*} + \int \phi \circ - f _ {n} d \mathbb {P} _ {0} ^ {*} \geq R _ {\phi , *} ^ {\epsilon} \tag {14}
$$
As  $R_{\phi ,*}^{\epsilon} = \int C_{\phi}^{*}(\eta^{*})d\mathbb{P}^{*}$ , this relation immediately implies (12).
Next, Lemma 2 again implies that
$$
\int S _ {\epsilon} \left(\phi \circ f _ {n}\right) d \mathbb {P} _ {1} \geq \int \phi \circ f _ {n} d \mathbb {P} _ {1} ^ {*} \quad \text {a n d} \quad \int S _ {\epsilon} \left(\phi \circ - f _ {n}\right) d \mathbb {P} _ {0} \geq \int \phi \circ - f _ {n} d \mathbb {P} _ {0} ^ {*} \tag {15}
$$
while (14) implies that
$$
R _ {\phi , *} ^ {\epsilon} - \int \phi \circ f _ {n} d \mathbb {P} _ {1} ^ {*} + \int \phi \circ - f _ {n} d \mathbb {P} _ {0} ^ {*} \leq 0.
$$
Therefore, subtracting  $\int \phi \circ f_n d\mathbb{P}_1^* + \int \phi \circ -f_n d\mathbb{P}_0^*$  from (14) results in
$$
\delta \geq \left(\int S _ {\epsilon} \left(\phi \circ f _ {n}\right) d \mathbb {P} _ {1} - \int \phi \circ f _ {n} d \mathbb {P} _ {1} ^ {*}\right) + \left(\int S _ {\epsilon} \left(\phi \circ - f _ {n}\right) d \mathbb {P} _ {0} - \int \phi \circ - f _ {n} d \mathbb {P} _ {0} ^ {*}\right) \geq 0. \tag {16}
$$
Again, (15) implies that the quantities on parentheses are both positive which implies (13).
Proposition 4 shows that minimizing sequences of  $R_{\phi}^{\epsilon}$  satisfy two properties: 1) The sequence  $\{f_n\}$  must minimize the standard  $\phi$ -risk  $R_{\phi}$  with measures  $\mathbb{P}_0^*, \mathbb{P}_1^*$  in place of  $\mathbb{P}_0, \mathbb{P}_1, 2$ . At the limit, the measures  $\mathbb{P}_0^*, \mathbb{P}_1^*$  are best adversarial attacks on  $\phi \circ f_n, \phi \circ -f_n$ . In fact, one can show that  $\{f_n\}$  is a minimizing sequence of  $R_{\phi}^{\epsilon}$  if and only if it satisfies these properties. Crucially, a very similar characterization holds for minimizers of the adversarial classification loss. We state and prove the 'only if' direction of this characterization in Proposition 5.
Proposition 5. Let  $f_{n}$  be a sequence and let  $\mathbb{P}_0^*, \mathbb{P}_1^*$  be measures in  $\mathcal{B}_{\epsilon}^{\infty}(\mathbb{P}_i)$ . Define  $\mathbb{P}^{*} = \mathbb{P}_{0}^{*} + \mathbb{P}_{1}^{*}$ ,  $\eta^{*} = d\mathbb{P}_{1}^{*} / d\mathbb{P}^{*}$ . If the following two conditions hold:
$$
\lim  _ {n \rightarrow \infty} \int C \left(\eta^ {*}, f _ {n}\right) d \mathbb {P} ^ {*} = \int C ^ {*} \left(\eta^ {*}\right) d \mathbb {P} ^ {*} \tag {17}
$$
$$
\lim  _ {n \rightarrow \infty} \int S _ {\epsilon} \left(\mathbf {1} _ {f _ {n} \leq 0}\right) d \mathbb {P} _ {1} - \int \mathbf {1} _ {f _ {n} \leq 0} d \mathbb {P} _ {1} ^ {*} = 0, \quad \lim  _ {n \rightarrow \infty} \int S _ {\epsilon} \left(\mathbf {1} _ {f _ {n} > 0}\right) d \mathbb {P} _ {0} - \int \mathbf {1} _ {f _ {n} > 0} d \mathbb {P} _ {0} ^ {*} = 0, \tag {18}
$$
then  $f_{n}$  is a minimizing sequence of  $R^{\epsilon}$ .
Proof. Equation 17 implies that the limit  $\lim_{n\to \infty}C(\eta^{*},f_{n})d\mathbb{P}^{*}$  exists. Thus (17) and (18) imply that
$$
\begin{array}{l} \lim  _ {n \rightarrow \infty} R ^ {\epsilon} (f _ {n}) = \lim  _ {n \rightarrow \infty} \int S _ {\epsilon} (\mathbf {1} _ {f _ {n} \leq 0}) d \mathbb {P} _ {1} + \int S _ {\epsilon} (\mathbf {1} _ {f _ {n} > 0}) d \mathbb {P} _ {0} = \lim  _ {n \rightarrow \infty} \int \mathbf {1} _ {f _ {n} \leq 0} d \mathbb {P} _ {1} ^ {*} + \int \mathbf {1} _ {f _ {n} > 0} d \mathbb {P} _ {0} ^ {*} \\ = \lim  _ {n \rightarrow \infty} \int C \left(\eta^ {*}, f _ {n}\right) d \mathbb {P} ^ {*} = \int C ^ {*} \left(\eta^ {*}\right) d \mathbb {P} ^ {*} = \bar {R} \left(\mathbb {P} _ {0} ^ {*}, \mathbb {P} _ {1} ^ {*}\right). \\ \end{array}
$$
Therefore, Strong duality (Theorem 1) then implies that
$$
\lim_{n\to \infty}R^{\epsilon}(f_{n})\leq \sup_{\substack{\mathbb{P}_{0}^{\prime}\in \mathcal{B}_{\epsilon}^{\infty}(\mathbb{P}_{0})\\ \mathbb{P}_{1}^{\prime}\in \mathcal{B}_{\epsilon}^{\infty}(\mathbb{P}_{1})}}\bar{R} (\mathbb{P}_{0}^{\prime},\mathbb{P}_{1}^{\prime}) = \inf_{\substack{f  \text{Borel}\\ \mathbb{R}\text{-valued}}}R^{\epsilon}(f)
$$
and therefore,  $f_{n}$  is a minimizing sequence.
We end this section by comparing the different criteria for consistency presented in Proposition 1 with Propositions 4 and 5. Together, Propositions 4 and 5 will allow us to compare minimizing sequences of  $R_{\phi}^{\epsilon}$  to those of  $R^{\epsilon}$  by showing that any sequence satisfying (12)-(13) must also satisfy (17)-(18). This statement is the analog to 2) of Proposition 1. Indeed, because  $C_{\phi}(\eta^{*},f_{n})\geq C_{\phi}^{*}(\eta^{*})$ , (12) is actually equivalent to to  $C_{\phi}(\eta^{*},f_n)\to C_{\phi}^{*}(\eta^{*})$  in  $L^1 (\mathbb{P}^*)$ . However, the extra criterion (18) implies an additional constraint on the structure of the minimizing sequence. This additional constraint is the reason 3) of Proposition 1 is false in the adversarial setting. In the restricted situation where  $\bar{R}_{\phi} = \bar{R}$ , Meunier et al. [2022] show that (12) implies (17) (Proposition 4.2). However, this observation does not suffice to conclude consistency.
4.2 Adversarial Consistency
We are now in a position to prove consistency. Before presenting the full proof, we pause to discuss the overall strategy. Consistency will follow from three considerations. First, every minimizing sequence of  $R_{\phi}^{\epsilon}$  satisfies conditions (12) and (13). Second, conditions (12) and (13) imply the very similar conditions (17) and (18). Finally, any function sequence satisfying (17) and (18) must be a minimizing sequence to  $R^{\epsilon}$ . The first and last steps are the content of Propositions 4 and 5, so it remains to justify the middle step.
Verifying that (12) implies (17) is straightforward. The relation (12) actually states that  $f_{n}$  minimizes the standard surrogate risk with respect to the distribution given by  $\mathbb{P}_0^*, \mathbb{P}_1^*$ . Therefore (12) implies (17) so long as  $\phi$  is consistent.
The main difficulty is verifying (18), due to the discontinuity of  $\mathbf{1}_{\alpha < 0}$ ,  $\mathbf{1}_{\alpha \geq 0}$  at 0. Due to this discontinuity, one cannot directly argue that (13) implies (18): to simplify the discussion, assume that  $\phi$  is strictly decreasing on a neighborhood of the origin, in which case  $\mathbf{1}_{\alpha < 0} = \mathbf{1}_{\phi (\alpha) > \phi (0)}$  and  $\mathbf{1}_{\alpha \geq 0} = \mathbf{1}_{\phi (-\alpha) \geq \phi (0)}$ . Recall that according to (13), in the limit  $n \to \infty$ ,  $\mathbb{P}_0^*, \mathbb{P}_1^*$  are the strongest attack in  $\mathcal{B}_{\epsilon}^{\infty}(\mathbb{P}_0) \times \mathcal{B}_{\epsilon}^{\infty}(\mathbb{P}_1)$ , or informally,  $S_{\epsilon}(\phi \circ f_n)(\mathbf{x})$  approaches  $\phi(f_n(\mathbf{x}'))$  for an optimal perturbation  $\mathbf{x}'$  w.h.p., with a similar condition for  $\phi \circ -f_n$ . However, due to the discontinuity of  $\mathbf{1}_{\phi(-\alpha) \geq \phi(0)}$  at  $\phi(0)$ , if  $f_n(\mathbf{x}') \to 0$  as  $n \to \infty$ , this relation does not imply that  $\mathbf{1}_{S_{\epsilon}(\phi \circ -f_n)(\mathbf{x}) \geq \phi(0)}$  approaches  $\mathbf{1}_{\phi \circ -f_n(\mathbf{x}') \geq 0}$ .
Lemma 1 says that if  $C_{\phi}^{*}(1/2) < \phi(0)$ , minimizers of  $C_{\phi}(\eta, \cdot)$  are uniformly bounded away from 0. This fact suggests that minimizing sequences will also be bounded away from the origin, which will allow us to avoid the discontinuity there. Concretely, we show:
Lemma 3. Let  $C_{\phi}^{*}(1/2) < \phi(0)$ . Then there is a  $\delta > 0$  and a  $c > 0$  with  $\phi(c) < \phi(0)$  for which  $\alpha \in [-c, c]$  implies  $C_{\phi}(\eta, \alpha) \geq C_{\phi}^{*}(\eta) + \delta$ , uniformly in  $\eta$ . Furthermore, for this value of  $c$ , if  $\alpha > c$  then  $\phi(\alpha) < \phi(c)$ .
We prove this lemma in Appendix C. Because  $C_{\phi}(\eta^{*},f_{n})\to C_{\phi}^{*}(\eta^{*})$  in  $L^1 (\mathbb{P}^*)$ , Lemma 3 implies that
$$
\lim  _ {n \rightarrow \infty} \mathbb {P} ^ {*} (f _ {n} \in [ - c, c ]) = 0. \tag {19}
$$
This relation is the key fact that allows us to show that (13) implies (18). The condition  $C_{\phi}^{*}(1/2) < \phi(0)$  is essential for this step of the argument.
Lastly, Lemma 2 implies that  $\int S_{\epsilon}(\mathbf{1}_{f_n\geq 0})d\mathbb{P}_1\geq \int \mathbf{1}_{f_n\geq 0}d\mathbb{P}_1^*$  and thus to validate (18), it suffices to verify the opposite inequality in the limit  $n\to \infty$
Lemma 4. Let  $f_n$  be a sequence of functions and let  $\mathbb{P}_0^* \in \mathcal{B}_{\epsilon}^\infty(\mathbb{P}_0)$ ,  $\mathbb{P}_1^* \in \mathcal{B}_{\epsilon}^\infty(\mathbb{P}_1)$ . The equation
$$
\lim  _ {n \rightarrow \infty} \sup  _ {n \rightarrow \infty} \int S _ {\epsilon} \left(\mathbf {1} _ {f _ {n} \leq 0}\right) d \mathbb {P} _ {1} \leq \lim  _ {n \rightarrow \infty} \inf  _ {n \rightarrow \infty} \int \mathbf {1} _ {f _ {n} \leq 0} d \mathbb {P} _ {1} ^ {*} \tag {20}
$$
implies the first relation of (18) and
$$
\lim  _ {n \rightarrow \infty} \sup  _ {n \rightarrow \infty} \int S _ {\epsilon} \left(\mathbf {1} _ {f _ {n} > 0}\right) d \mathbb {P} _ {0} \leq \lim  _ {n \rightarrow \infty} \inf  _ {n \rightarrow \infty} \int \mathbf {1} _ {f _ {n} > 0} d \mathbb {P} _ {0} ^ {*} \tag {21}
$$
implies the second relation of (18).
See Appendix F for a proof. These considerations suffice to prove the main result of this paper:
Theorem 3. The loss  $\phi$  is adversely consistent if and only if  $C_{\phi}^{*}(1 / 2) < \phi (0)$ .
Proof. The 'only if' portion of the statement is Proposition 2.
To show the 'if' statement, recall the standard analysis fact:  $\lim_{n\to \infty}a_n = a$  iff for all subsequences  $\{a_{n_j}\}$  of  $\{a_{n}\}$ , there is a further subsequence  $a_{n_{j_k}}$  for which  $\lim_{k\to \infty}a_{n_{j_k}} = a$ . This result implies that to prove  $R_{\phi}^{\epsilon}$  is consistent, it suffices to show that every minimizing sequence  $f_{n}$  of  $R_{\phi}^{\epsilon}$  has a subsequence  $f_{n_j}$  that minimizes  $R^{\epsilon}$ .
Let  $f_{n}$  be a minimizing sequence of  $R_{\phi}^{\epsilon}$ . For convenience, pick a subsequence  $f_{n_j}$  for which the limits  $\lim_{j\to \infty}\int S_{\epsilon}(\mathbf{1}_{f_{n_j}} < 0)d\mathbb{P}_0$ ,  $\lim_{j\to \infty}\int S_{\epsilon}(\mathbf{1}_{f_{n_j}}\geq 0)d\mathbb{P}_1$  both exist. For notational clarity, we drop the  $j$  subscript and denote this sequence as  $f_{n}$ .
By Proposition 4, the equations (12) and (13) hold. We will argue that  $f_{n}$  is in fact a minimizing sequence of  $R^{\epsilon}$  by verifying the conditions of Proposition 5.
First, the relation (12) states that the sequence  $f_{n}$  minimizes the standard  $\phi$ -risk for the distribution given by  $\mathbb{P}_0^*$  and  $\mathbb{P}_1^*$ . As the loss  $\phi$  is consistent by Proposition 3, the sequence  $f_{n}$  must minimize the standard classification risk for the distribution  $\mathbb{P}_0^*, \mathbb{P}_1^*$ . This statement implies (17). Next we will argue that (18) holds.
Let  $c, \delta$  be as in Lemma 3. Because  $C_{\phi}(\eta^{*}, f_{n}) \geq C_{\phi}^{*}(\eta^{*})$ , (12) implies that  $C_{\phi}(\eta^{*}, f_{n})$  converges to  $C_{\phi}^{*}(\eta^{*})$  in  $L^1$ . However,  $L^1$  convergence implies convergence in measure (see for instance Proposition 2.29 of [Folland, 1999]), and therefore  $\lim_{n \to \infty} \mathbb{P}^{*}\left(C_{\phi}(\eta^{*}, f_{n}) > C_{\phi}^{*}(\eta^{*}) + \delta\right) = 0$ . Lemma 3 then implies that for  $i = 0, 1$
$$
\lim  _ {n \rightarrow \infty} \mathbb {P} _ {i} ^ {*} (f _ {n} \in [ - c, c ]) = 0. \tag {22}
$$
Next, because  $\phi$  is non-increasing,  $f\leq 0$  implies  $\phi (f)\geq \phi (0)$  and thus  $\mathbf{1}_{f\leq 0}\leq \mathbf{1}_{\phi \circ f\geq \phi (0)}$ . Furthermore, as the function  $\alpha \mapsto \mathbf{1}_{\alpha \geq 0}$  is monotone and upper semi-continuous,
$$
\int S _ {\epsilon} \left(\mathbf {1} _ {f _ {n} \leq 0}\right) d \mathbb {P} _ {1} \leq \int S _ {\epsilon} \left(\mathbf {1} _ {\phi \circ f _ {n} \geq \phi (0)}\right) d \mathbb {P} _ {1} \leq \int \mathbf {1} _ {S _ {\epsilon} (\phi \circ f _ {n}) \geq \phi (0)} d \mathbb {P} _ {1}. \tag {23}
$$
Let  $\gamma_{i}$  be a coupling between  $\mathbb{P}_i$  and  $\mathbb{P}_i^*$  for which  $\operatorname{ess} \sup_{(\mathbf{x},\mathbf{y}) \sim \gamma_i} \| \mathbf{x} - \mathbf{y} \| \leq \epsilon$ . Then the measure  $\gamma_{i}$  is supported on  $\Delta_{\epsilon} = \{(\mathbf{x},\mathbf{y}) : \| \mathbf{x} - \mathbf{y} \| \leq \epsilon\}$ . Furthermore, as  $S_{\epsilon}(\phi \circ f_n)(\mathbf{x}) \geq \phi \circ f_n(\mathbf{x}')$  everywhere on  $\Delta_{\epsilon}$ , the relation  $S_{\epsilon}(\phi \circ f_n)(\mathbf{x}) \geq \phi \circ f_n(\mathbf{x}')$  actually holds by (13) a.e. Therefore, (13) actually implies that  $S_{\epsilon}(\phi \circ f_n)(\mathbf{x}) - \phi \circ f_n(\mathbf{x}')$  converges in  $\gamma_{1}$ -measure to 0. In particular, since  $\phi(c) < \phi(0)$ ,  $\lim_{n \to \infty} \gamma_1(S_\epsilon(\phi \circ f_n)(\mathbf{x}) - \phi(f_n(\mathbf{x}')) \geq \phi(0) - \phi(c)) = 0$  and thus  $\lim_{n \to \infty} \gamma_1(S_\epsilon(\phi \circ f_n)(\mathbf{x}) \geq \phi(0) \cap \phi \circ f_n(\mathbf{x}') < \phi(c)) = 0$ . Therefore,
$$
\begin{array}{l} \liminf_{n\to \infty}\mathbb{P}_{1}(S_{\epsilon}(\phi \circ f_{n})(\mathbf{x})\geq \phi (0)) = \liminf_{n\to \infty}\gamma_{1}(S_{\epsilon}(\phi \circ f_{n})(\mathbf{x})\geq \phi (0)\cap \phi \circ f_{n}(\mathbf{x}^{\prime})\geq \phi (c)) \\ \leq \operatorname * {l i m i n f} _ {n \rightarrow \infty} \gamma_ {1} (\phi \circ f _ {n} (\mathbf {x} ^ {\prime}) \geq \phi (c)) = \operatorname * {l i m i n f} _ {n \rightarrow \infty} \mathbb {P} _ {1} ^ {*} (\phi \circ f _ {n} (\mathbf {x} ^ {\prime}) \geq \phi (c)) \\ \end{array}
$$
This calculation implies
$$
\lim  _ {n \rightarrow \infty} \inf  _ {n \rightarrow \infty} \int \mathbf {1} _ {S _ {\epsilon} (\phi \circ f _ {n}) (\mathbf {x}) \geq \phi (0)} d \mathbb {P} _ {1} \leq \lim  _ {n \rightarrow \infty} \inf  _ {n \rightarrow \infty} \int \mathbf {1} _ {\phi \circ f _ {n} (\mathbf {x} ^ {\prime}) \geq \phi (c)} d \mathbb {P} _ {1} ^ {*} \leq \lim  _ {n \rightarrow \infty} \inf  _ {n \rightarrow \infty} \int \mathbf {1} _ {f _ {n} \leq c} d \mathbb {P} _ {1} ^ {*} \tag {24}
$$
The last inequality follows because Lemma 3 states that  $\alpha > c$  implies  $\phi(\alpha) < \phi(c)$  and therefore  $\mathbf{1}_{\phi \circ f_n \geq \phi(c)} \leq \mathbf{1}_{f_n \leq c}$ . Equation 22 then implies
$$
\lim  _ {n \rightarrow \infty} \inf  _ {\phi \circ f _ {n} \geq \phi (0)} d \mathbb {P} _ {1} ^ {*} \leq \lim  _ {n \rightarrow \infty} \inf  _ {\phi \circ f _ {n} \leq c} d \mathbb {P} _ {1} ^ {*} = \lim  _ {n \rightarrow \infty} \inf  _ {\phi \circ f _ {n} \leq - c} d \mathbb {P} _ {1} ^ {*}. \tag {25}
$$
Recall that the sequence  $f_{n}$  was chosen so that the limit  $\lim_{n\to \infty}\int S_{\epsilon}(\mathbf{1}_{f_n\leq 0})d\mathbb{P}_1$  exists. Combining this fact with (23), (24), and (25) results in
$$
\lim  _ {n \rightarrow \infty} \int S _ {\epsilon} \left(\mathbf {1} _ {f _ {n} \leq 0}\right) d \mathbb {P} _ {1} \leq \lim  _ {n \rightarrow \infty} \inf  _ {n \rightarrow \infty} \int \mathbf {1} _ {f _ {n} \leq - c} d \mathbb {P} _ {1} ^ {*} \leq \lim  _ {n \rightarrow \infty} \inf  _ {n \rightarrow \infty} \int \mathbf {1} _ {f _ {n} \leq 0} d \mathbb {P} _ {1} ^ {*} \tag {26}
$$
The first relation of (18) then follows from (26) together with Lemma 4.
A similar argument implies the second relation of (18). Because  $\mathbf{1}_{f > 0} = \mathbf{1}_{-f < 0} \leq \mathbf{1}_{-f \leq 0}$ , the same chain of inequalities as (23), (24), and (25) implies that
$$
\lim _ {n \to \infty} \int S _ {\epsilon} (\mathbf {1} _ {f _ {n} > 0}) d \mathbb {P} _ {0} \leq \lim _ {n \to \infty} \int S _ {\epsilon} (\mathbf {1} _ {- f _ {n} \leq 0}) d \mathbb {P} _ {0} \leq \lim _ {n \to \infty} \inf _ {n \to \infty} \int \mathbf {1} _ {- f _ {n} \leq - c} d \mathbb {P} _ {0} ^ {*} = \lim _ {n \to \infty} \inf _ {n \to \infty} \int \mathbf {1} _ {f _ {n} \geq c} d \mathbb {P} _ {0} ^ {*}
$$
As  $c > 0$ , it follows that  $\lim \sup_{n\to \infty}\int S_{\epsilon}(\mathbf{1}_{f_n > 0})d\mathbb{P}_0\leq \lim \inf_{n\to \infty}\int \mathbf{1}_{f_n > 0}d\mathbb{P}_0^*$ . Once again, the second expression of (18) follows from this relation and Lemma 4.
5 Quantitative Bounds for the  $\rho$ -Margin Loss
As discussed in the introduction, statistical consistency is not the only property one would want from a surrogate. Hopefully, minimizing a surrogate will also efficiently minimize the classification loss. Bartlett et al. [2006], Steinwart [2007], Reid and Williamson [2009] prove bounds of the form  $R(f) - R_{*} \leq G_{\phi}(R_{\phi}^{*}(f) - R_{\phi,*})$  for a function  $G_{\phi}$  and  $R_{*} = \inf_{f} R(f), R_{\phi,*} = \inf_{f} R_{\phi}(f)$ . The function  $G_{\phi}$  is an upper bound on the rate of convergence of the classification risk in terms of the rate of convergence of the surrogate risk. One would hope that  $G_{\phi}$  is not logarithmic, as such a bound could imply that reducing  $R(f) - R_{*}$  by a quantity  $\Delta$  could require an exponential change of  $e^{\Delta}$  in  $R_{\phi}(f) - R_{\phi,*}$ . Bartlett et al. [2006] compute such  $G_{\phi}$  for several popular losses in the standard classification setting. For example, they show the bounds  $G_{\phi}(\theta) = \theta$  for the hinge loss  $\phi(\alpha) = (1 - \alpha)_+$  and  $G_{\phi}(\theta) = \sqrt{\theta}$  for the squared hinge loss  $\phi(\alpha) = (1 - \alpha)_+^2$ . On can prove an analogous bound for the  $\rho$ -margin loss in the adversarial setting:
Theorem 4. Let  $\phi_{\rho} = \min(1, \max(1 - \alpha / \rho, 0))$  be the  $\rho$ -margin loss,  $R_{*}^{\epsilon} = \inf_{f} R^{\epsilon}(f)$ , and  $R_{\phi_{\rho}, *}^{\epsilon}(f) = \inf_{f} R_{\phi_{\rho}}^{\epsilon}(f)$ . Then
$$
R ^ {\epsilon} (f) - R _ {*} ^ {\epsilon} \leq R _ {\phi_ {\rho}} ^ {\epsilon} (f) - R _ {\phi_ {\rho}, *} ^ {\epsilon}.
$$
Notice that this theorem immediately implies that the  $\rho$ -margin loss is in fact adversarially consistent. The proof below is completely independent of the argument in Section 4.
Proof. Notice that for the  $\rho$ -margin loss,  $C_{\phi_{\rho}}^{*} = C^{*}$  and therefore, the optimal  $\phi_{\rho}$ -risk  $R_{\phi_{\rho},*}^{\epsilon}$  equals the optimal adversarial classification risk  $R_{*}^{\epsilon}$ . However, since  $\phi_{\rho}(\alpha) \geq 1_{\alpha \leq 0}$  and  $\phi_{\rho}(-\alpha) \geq 1_{\alpha > 0}$  for any  $\alpha$ , one can conclude that  $R^{\epsilon}(f) \leq R_{\phi_{\rho}}^{\epsilon}(f)$ . Therefore,
$$
R ^ {\epsilon} (f) - R _ {*} ^ {\epsilon} = R ^ {\epsilon} (f) - R _ {\phi_ {\rho}, *} ^ {\epsilon} \leq R _ {\phi_ {\rho}} ^ {\epsilon} (f) - R _ {\phi_ {\rho}, *} ^ {\epsilon}
$$
This bound implies that reducing the excess adversarial  $\rho$ -margin loss by  $\Delta$  also reduces an upper bound on the excess adversarial classification loss by  $\Delta$ . Thus, one would expect that minimizing the adversarial  $\rho$ -margin risk would be an effective procedure for minimizing the adversarial classification risk.
Extending Theorem 4 to other losses remains an open problem. In the non-adversarial scenario, many prior works develop techniques for computing such bounds. These include the  $\Psi$ -transform of [Bartlett et al., 2006], calibration analysis in [Steinwart, 2007], and special techniques for proper losses in [Reid and Williamson, 2009].
Contemporary work [Mao et al., 2023] derives an  $\mathcal{H}$ -consistency surrogate risk bound for a variant of the adversarial  $\rho$ -margin loss.
In conclusion, we proved that the adversarial training procedure is consistent for perturbations in an  $\epsilon$ -ball if an only if  $C_{\phi}^{*}(1/2) < \phi(0)$ . The technique that proved consistency extends to perturbation sets which satisfy existence and minimax theorems analogous to Theorems 1 and 2. Furthermore, we showed a quantitative excess risk bound for the adversarial  $\rho$ -margin loss. Finding such bounds for other losses remains an open problem. We hope that insights to consistency and the structure of adversarial learning will lead to the design of better adversarial learning algorithms.
Natalie Frank was supported in part by the Research Training Group in Modeling and Simulation funded by the National Science Foundation via grant RTG/DMS - 1646339. Jonathan Niles-Weed was supported in part by a Sloan Research Fellowship.