SlowGuess/NeurIPS_2023_OCR huggingface dataset:
which implies that  $s \leq \frac{1}{l}$  in this case. Finally, we consider the third case. We assume WLOG that  $\mathbf{x}_1 \in A_f$  and  $\mathbf{x}_2 \in (A_f \cup B_f)^c$ . Substitute into Equation (B.42), we have
$$
\begin{array}{l} s = \frac {1}{d _ {p} \left(\mathbf {x} _ {1} , \mathbf {x} _ {2}\right)} \cdot \left| \frac {d _ {p} \left(\mathbf {x} _ {1} , A _ {f}\right)}{d _ {p} \left(\mathbf {x} _ {1} , A _ {f}\right) + d _ {p} \left(\mathbf {x} _ {1} , B _ {f}\right)} - \frac {d _ {p} \left(\mathbf {x} _ {2} , A _ {f}\right)}{d _ {p} \left(\mathbf {x} _ {2} , A _ {f}\right) + d _ {p} \left(\mathbf {x} _ {2} , B _ {f}\right)} \right| \tag {B.45} \\ = \frac {1}{d _ {p} (\mathbf {x} _ {1} , \mathbf {x} _ {2})} \cdot \frac {d _ {p} (\mathbf {x} _ {2} , A _ {f})}{d _ {p} (\mathbf {x} _ {2} , A _ {f}) + d _ {p} (\mathbf {x} _ {2} , B _ {f})} \leq \frac {1}{2 \lambda} \\ \end{array}
$$
To sum up above, we have  $\sup_{\mathbf{x}_1 \neq \mathbf{x}_2} s = \frac{1}{\lambda}$ , which implies that  $f_b$  is  $\frac{1}{\lambda}$ -Lipschitz continuous, as is required.
Corollary 5.3. Given  $\epsilon >0$ ,  $r > 0$  and  $S\subset [0,1]^d$ , there is a ReLU network  $\tilde{\phi}$  with  $O((1 / r\epsilon)^{d})\cdot O(d^{2} + d\log (1 / \epsilon))$  parameters that can approximate  $\phi (\cdot ;r,S)$  to precision  $\epsilon$ .
Proof of Corollary 5.3. We prove this corollary in a similar manner as Corollary 5.3, i.e., we upper bound the supremum of
$$
\begin{array}{l} s := \frac {\left| \phi \left(\mathbf {x} _ {1} ; r , S\right) - \phi \left(\mathbf {x} _ {2} ; r , S\right) \right|}{d _ {p} \left(\mathbf {x} _ {1} , \mathbf {x} _ {2}\right)} = \frac {1}{d _ {p} \left(\mathbf {x} _ {1} , \mathbf {x} _ {2}\right)} \cdot \left| \frac {d _ {p} \left(\mathbf {x} _ {1} , S\right)}{r + d _ {p} \left(\mathbf {x} _ {1} , S\right)} - \frac {d _ {p} \left(\mathbf {x} _ {2} , S\right)}{r + d _ {p} \left(\mathbf {x} _ {2} , S\right)} \right| \tag {B.46} \\ = \frac {r}{d _ {p} (\mathbf {x} _ {1} , \mathbf {x} _ {2})} \cdot \left| \frac {1}{r + d _ {p} (\mathbf {x} _ {1} , S)} - \frac {1}{r + d _ {p} (\mathbf {x} _ {2} , S)} \right|. \\ \end{array}
$$
We also consider three cases in this proof:
1. both of  $\mathbf{x}_1, \mathbf{x}_2 \in S$ , or
2. both of  $\mathbf{x}_1, \mathbf{x}_2 \in S^c$ , and
3. either  $\mathbf{x}_1$  or  $\mathbf{x}_2$  is in  $S$ .
In case 1, we see at once that  $s = 0$ . When both of  $\mathbf{x}_1, \mathbf{x}_2 \in S^c$ , we assume WLOG that  $d_p(\mathbf{x}_1, S) > d_p(\mathbf{x}_2, S)$ . By Equation (B.46), we have
$$
s = \frac {d _ {p} \left(\mathbf {x} _ {1} , S\right) - d _ {p} \left(\mathbf {x} _ {2} , S\right)}{d _ {p} \left(\mathbf {x} _ {1} , \mathbf {x} _ {2}\right)} \cdot \frac {r}{\left(r + d _ {p} \left(\mathbf {x} _ {1} , S\right)\right) \left(r + d _ {p} \left(\mathbf {x} _ {2} , S\right)\right)} \leq \frac {1}{r}. \tag {B.47}
$$
Analysis similar to Equation (B.47) shows that
$$
s = \frac {r}{d _ {p} \left(\mathbf {x} _ {1} , \mathbf {x} _ {2}\right)} \cdot \left(\frac {1}{r} - \frac {1}{r + d _ {p} \left(\mathbf {x} _ {2} , S\right)}\right) \leq \frac {1}{r}. \tag {B.48}
$$
To sum up above, we have  $\sup_{\mathbf{x}_1 \neq \mathbf{x}_2} s = \frac{1}{\lambda}$ , which implies that  $\phi$  is  $\frac{1}{r}$ -Lipschitz continuous, as is required.
$$
\mathcal {O} \left(\max  \left\{\frac {1}{\lambda \epsilon}, \frac {2}{r \epsilon} \right\} ^ {d}\right) \cdot \mathcal {O} \left(d ^ {2} + d \log \left(\frac {1}{\epsilon}\right)\right) + \mathcal {O} \left(\log^ {2} \left(\frac {1}{\epsilon}\right)\right) \tag {B.49}
$$
Proof of Proposition 5.6. This proposition can be derived directly from Lemma 5.5, Corollary 5.3, and Corollary 5.4.
$$
\left\| \tilde {f} _ {t} - f _ {t} \right\| _ {\infty} \leq \epsilon , \left\| \tilde {f} _ {s} - f _ {s} \right\| _ {\infty} \leq \epsilon \tag {B.50}
$$
$$
\tilde {f} _ {t} (\mathbf {x}) \cdot \tilde {f} _ {t} \left(\mathbf {x} _ {a}\right) \leq 2 \epsilon (1 + \epsilon) ^ {2} + 2 \epsilon^ {2}. \tag {B.51}
$$
Proof of Theorem 5.7. Consider  $\mathbf{x} \in A_f$  WLOG. If  $f_s$  is robust against adversarial examples at  $\mathbf{x} \in B(\mathbf{x}, \delta) \cap \mathcal{M} \subset A_f$ , then there is nothing to prove. If not, denote the adversarial example of  $f_s$  at  $\mathbf{x}$  by  $\mathbf{x}_a$ . Since  $\mathbf{x} \in A_f \subset \mathcal{M}$ , there is
$$
\tilde {f} _ {s} (\mathbf {x}) \geq \tilde {f} _ {b} (\mathbf {x}) \cdot \tilde {\phi} _ {\text {o f f}} (\mathbf {x}) - \epsilon \geq (1 - \epsilon) ^ {2} - \epsilon > 0 \tag {B.52}
$$
and we thus have  $\tilde{f}_s(\mathbf{x}_a) < 0$ . By  $\mathbf{x}_a \in B(\mathbf{x}; \delta)$  and the assumption  $\delta < \frac{\lambda}{2}$ , we have
$$
\tilde {f} _ {b} \left(\mathbf {x} _ {a}\right) = 1 - \frac {2 d _ {p} \left(\mathbf {x} _ {a} , A _ {f}\right)}{d _ {p} \left(\mathbf {x} _ {a} , B _ {f}\right) + d _ {p} \left(\mathbf {x} _ {a} , A _ {f}\right)} \geq 1 - \frac {2 \delta}{2 \lambda} \geq \frac {1}{2}. \tag {B.53}
$$
To obtain  $\tilde{\times} (\tilde{f}_b,\tilde{\phi}_{\mathrm{off}})(\mathbf{x}_a) < 0$ , there must be  $\tilde{f}_b(\mathbf{x}_a)\cdot \tilde{\phi}_{\mathrm{off}}(\mathbf{x}_a) < \epsilon$ , which implies that
$$
\tilde {\phi} _ {\text {o f f}} \left(\mathbf {x} _ {a}\right) <   2 \epsilon . \tag {B.54}
$$
Now consider  $\tilde{f}_t(\mathbf{x})$  and  $\tilde{f}_t(\mathbf{x}_a)$ . By definition, we have  $\tilde{f}_b^* (\mathbf{x})\in [1 - \epsilon ,1 + \epsilon ]$ ,  $\tilde{\phi}_{\mathrm{off}}(\mathbf{x})\in [1 - \epsilon ,1 + \epsilon ]$ , and thus
$$
\tilde {f} _ {t} (\mathbf {x}) = \tilde {\times} \left(\tilde {f} _ {b} ^ {*}, \tilde {\phi} _ {\text {o f f}}\right) (\mathbf {x}) \leq (1 + \epsilon) ^ {2} + \epsilon . \tag {B.55}
$$
Similar to Equation (B.53), there is
$$
\tilde {f} _ {b} ^ {*} (\mathbf {x} _ {a}) = 1 - \frac {2 d _ {p} (\mathbf {x} _ {a} , A)}{d _ {p} (\mathbf {x} _ {a} , B) + d _ {p} (\mathbf {x} _ {a} , A)} \leq 1 \tag {B.56}
$$
Combining Equations (B.54) to (B.56) together, we have
$$
\tilde {f} _ {t} (\mathbf {x}) \cdot \tilde {f} _ {t} \left(\mathbf {x} _ {a}\right) \leq 2 \epsilon (1 + \epsilon) ^ {2} + 2 \epsilon^ {2}. \tag {B.57}
$$
as is required.
1. if  $B(\mathbf{x}, \delta) \cap \mathcal{M} \subset S_{\mathrm{crit}} \cup S_{\mathrm{wrg}}$ , then  $\tilde{f}_t$  and  $\tilde{f}_s$  are both robust against adversarial examples;
2. if  $B(\mathbf{x},\delta)\cap \mathcal{M}\subset A\cup B$  and  $\mathbf{x}\in A_f\cup B_f$  , then the adversarial examples of  $\tilde{f}_s$  at  $\mathbf{x}$  (if exists) cannot transfer to  $\tilde{f}_t$
Proof of Theorem 5.8. The proof of the first result is also straightforward, which is omitted here. It remains to prove the second result. By Equation (7), denote
$$
f _ {s} (\mathbf {x}) = f _ {b} (\mathbf {x}) \cdot \phi_ {\text {o n}} (\mathbf {x}; f _ {b}) = \frac {d _ {2} \left(\mathbf {x} , B _ {f}\right) - d _ {2} \left(\mathbf {x} , A _ {f}\right)}{d _ {2} \left(\mathbf {x} , B _ {f}\right) + d _ {2} \left(\mathbf {x} , A _ {f}\right)} \cdot \frac {\alpha \delta - d _ {2} \left(\mathbf {x} , \mathcal {N} _ {\delta} \left(A _ {f} \cup B _ {f}\right)\right)}{\alpha \delta + d _ {2} \left(\mathbf {x} , \mathcal {N} _ {\delta} \left(A _ {f} \cup B _ {f}\right)\right)}. \tag {B.58}
$$
For  $\forall \mathbf{x} \in A \cup B$  such that  $B(\mathbf{x}, \delta) \cap \mathcal{M} \subset A \cup B$  and  $\mathbf{x} \in A_f \cup B_f$ , denote the unspecific adversarial example (if exist) of  $f_s$  at  $\mathbf{x}$  by  $\mathbf{x}_a$ . Assume that  $\mathbf{x} \in A \cap A_f$  WLOG. By definition, we have
$$
\tilde {f} _ {b} (\mathbf {x}) \in [ 1 - \epsilon , 1 + \epsilon ]. \tag {B.59}
$$
By the  $2\lambda$ -separated assumption of  $A_{f}$  and  $B_{f}$ , we have and
$$
\tilde {f} _ {b} (\mathbf {x} _ {a}) \in [ 1 - \epsilon , 1 + \epsilon ]. \tag {B.60}
$$
Since  $\mathbf{x} \in A_f \cup B_f$ , we have  $\mathbf{x} \in \mathcal{N}_{\delta}(A_f \cup B_f)$  and
$$
\tilde {\phi} _ {\text {o n}} (\mathbf {x}; f _ {b}) \in [ 1 - \epsilon , 1 + \epsilon ], \tag {B.61}
$$
$$
\tilde {f} _ {s} (\mathbf {x}) = \tilde {\times} \left(\tilde {f} _ {b}, \tilde {\phi} _ {\mathrm {o n}} (\cdot ; f _ {b})\right) (\mathbf {x}) \geq (1 - \epsilon) ^ {2} - \epsilon > 0. \tag {B.62}
$$
From  $\tilde{f}_s(\mathbf{x})\tilde{f}_s(\mathbf{x}_a) < 0$ , we can obtain that  $\tilde{f}_s(\mathbf{x}_a) < 0$ , which implies that
$$
\tilde {\phi} _ {\text {o n}} \left(\mathbf {x} _ {a}; f _ {b}\right) \cdot \tilde {f} _ {b} (\mathbf {x}) <   \epsilon , \tag {B.63}
$$
$$
\tilde {\phi} _ {\mathrm {o n}} \left(\mathbf {x} _ {a}; f _ {b}\right) <   \frac {\epsilon}{1 - \epsilon} <   2 \epsilon . \tag {B.64}
$$
By definition, we have
$$
\tilde {f _ {t}} (\mathbf {x}) = \tilde {\times} (\tilde {f _ {b}} ^ {*}, \tilde {\phi} _ {\mathrm {o n}} (\cdot ; f _ {b} ^ {*})) (\mathbf {x}) \geq (1 - \epsilon) ^ {2} - \epsilon > 0. \tag {B.65}
$$
By  $\mathbf{x} \in A$  and the  $2\lambda$ -separated assumption of  $A$  and  $B$ , we have
$$
\tilde {f} _ {b} ^ {*} (\mathbf {x} _ {a}) = 1 - \frac {2 d _ {p} (\mathbf {x} _ {a} , A)}{d _ {p} (\mathbf {x} _ {a} , B) + d _ {p} (\mathbf {x} _ {a} , A)} \geq 1 - \frac {2 \delta}{2 \lambda} \geq \frac {1}{2}. \tag {B.66}
$$
By  $B(\mathbf{x},\delta)\cap \mathcal{M}\subset A$  , we have  $\mathbf{x}_a\in \mathcal{N}_{\delta}(A\cup B))$  , i.e.,
$$
\phi_ {\text {o n}} \left(\mathbf {x} _ {a}; f _ {b} ^ {*}\right) \in [ 1 - \epsilon , 1 + \epsilon ] \tag {B.67}
$$
Together, we have
$$
\tilde {f} _ {t} \left(\mathbf {x} _ {a}\right) = \tilde {\times} \left(\tilde {f} _ {b} ^ {*}, \tilde {\phi} _ {\text {o n}} (\cdot ; f _ {b} ^ {*})\right) \left(\mathbf {x} _ {a}\right) \geq \frac {1 - \epsilon}{2} > 0, \tag {B.68}
$$
Proposition 5.9. For any classifier  $f^{*}$  with  $R_{\mathrm{std}}(f^{*}) = 0$  and perturbation radius  $\delta \in (r_{\delta}(f^{*}), \lambda)$ , there is  $f \in \mathcal{F}_{\mathcal{M}}$  such that
1.  $R_{\mathrm{std}}(f) = R_{\mathrm{std}}(f^{*})$  , and
2. for  $\forall \mathbf{x} \in A \cup B$ , if  $\mathbf{x}_a$  is an adversarial example of  $f^*$  at  $\mathbf{x}$ , then exists  $\mathbf{x}_a' \in B(\mathbf{x}_a, r_\delta(f^*)) / 4)$  such that  $\mathbf{x}_a'$  is an adversarial example of  $f$ .
Proof of Proposition 5.9. Define the following set
$$
S _ {a} := \left\{\mathbf {x} \in [ 0, 1 ] ^ {d} \cap (A \cup B) ^ {c}: \exists \mathbf {x} ^ {\prime} \in A \cup B \text {s . t .} \mathbf {x} ^ {\prime} \in B (\mathbf {x}; \delta), f ^ {*} (\mathbf {x}) f ^ {*} \left(\mathbf {x} ^ {\prime}\right) <   0 \right\}, \tag {B.69}
$$
and let
$$
G = \left(\bigcup_ {\mathbf {x} \in S _ {a}} B (\mathbf {x}, r _ {\delta} \left(f ^ {*}\right) / 2)\right) ^ {c}. \tag {B.70}
$$
By definition,  $A\cup B\in G$  .Consider
$$
f (\mathbf {x}) = f _ {b} ^ {*} (\mathbf {x}) \cdot \phi (\mathbf {x}; r _ {\delta} (f ^ {*}) / 4, G). \tag {B.71}
$$
Since  $A \cup B \in G$ , we have  $R_{\mathrm{std}}(f) = R_{\mathrm{std}}(f_b^*) = 0 = R_{\mathrm{std}}(f)^*$ . For  $\forall \mathbf{x} \in A \cup B$ , if  $\mathbf{x}_a$  is an adversarial example of  $f^*$  at  $\mathbf{x}$ , then
$$
\frac {r _ {\delta} \left(f ^ {*}\right)}{4} \leq d _ {p} \left(\mathbf {x} _ {a}, G\right) \leq \frac {r _ {\delta} \left(f ^ {*}\right)}{2}, \tag {B.72}
$$
which implies that exists  $\mathbf{x}_a' \in B(\mathbf{x}_a, r_\delta(f^*) / 4)$  such that  $\mathbf{x}_a'$  is an adversarial example of  $f$ .