SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\mathbb{E}_{g \sim \mu}[h(g(x))]^2 = (1 - y\mathbb{E}_{g \sim \mu}[h(g(x))])^2$ , and exponential loss  $e^{-y\mathbb{E}_{g \sim \mu}[h(g(x))]}$ . Furthermore, the class of Lipschitz functions  $\ell: \mathbb{R} \to \mathbb{R}$  on the margin  $y\mathbb{E}_{g \sim \mu}[h(g(x))]$  enables us to capture levels of robustness between the average- and worst-case. For example, taking  $\ell(t) = \frac{1 - t}{2}$  results in the loss  $\ell_{\mathcal{G},\mu}(h,(x,y)) = \ell(y\mathbb{E}_{g \sim \mu}[h(g(x)]) = \mathbb{P}_{g \sim \mu}[h(g(x)) \neq y]$ , corresponding to average-case robustness, or data augmentation. On the other hand, taking  $\ell(t) = \min(\frac{1 - t}{2\rho}, 1)$  for some  $\rho \in (0,1)$ , results in the loss
$$
\ell_ {\mathcal {G}, \mu} (h, (x, y)) = \ell (y \mathbb {E} _ {g \sim \mu} [ h (g (x)) ]) = \min  \left(\frac {\mathbb {P} _ {g \sim \mu} [ h (g (x)) \neq y ]}{\rho}, 1\right)
$$
which corresponds to a notion of robustness that becomes stricter as  $\rho$  approaches 0. We note that some of the losses in our family were studied by Rice et al. [2021]. However, their focus was on evaluating robustness, while ours is about (proper) learnability.
Lemma 4.1 shows that for hypothesis classes  $\mathcal{H}$  with finite VC dimension, for any  $(\mathcal{G},\mu)$ , all  $L$ -Lipschitz loss functions  $\ell_{\mathcal{G},\mu}(h,(x,y))$  enjoy the uniform convergence property.
Lemma 4.1 (Uniform Convergence of Lipschitz Loss). Let  $\mathcal{H}$  be a hypothesis class with finite VC dimension,  $(\mathcal{G},\mu)$  be a perturbation set and measure, and  $\ell_{\mathcal{G},\mu}(h,(x,y)) = \ell(y\mathbb{E}_{g\sim \mu}[h(g(x)])$  such that  $\ell: \mathbb{R} \to \mathbb{R}$  is a  $L$ -Lipschitz function. With probability at least  $1 - \delta$  over a sample  $S \sim \mathcal{D}^n$  of size  $n = O\left(\frac{\mathrm{VC}(\mathcal{H})L^2\ln\left(\frac{L}{\epsilon}\right) + \ln\left(\frac{1}{\delta}\right)}{\epsilon^2}\right)$ , for all  $h \in \mathcal{H}$  simultaneously,
$$
\left| \mathbb {E} _ {\mathcal {D}} \left[ \ell_ {\mathcal {G}, \mu} (h, (x, y)) \right] - \hat {\mathbb {E}} _ {S} \left[ \ell_ {\mathcal {G}, \mu} (h, (x, y)) \right] \right| \leq \epsilon .
$$
Proof. Let  $\mathrm{VC}(\mathcal{H}) = d$  and  $S = \{(x_1, y_1), \ldots, (x_m, y_m)\}$  be a set of examples drawn i.i.d from  $\mathcal{D}$ . Define the loss class  $\mathcal{L}_{\mathcal{G},\mu}^{\mathcal{H}} = \{(x,y) \mapsto \ell_{\mathcal{G},\mu}(h,(x,y)) : h \in \mathcal{H}\}$ . Observe that we can reparameterize  $\mathcal{L}_{\mathcal{G},\mu}^{\mathcal{H}}$  as the composition of a  $L$ -Lipschitz function  $\ell(x)$  and the function class  $\mathcal{F}_{\mathcal{G},\mu}^{\mathcal{H}} = \{(x,y) \mapsto y\mathbb{E}_{g \sim \mu}[h(g(x))] : h \in \mathcal{H}\}$ . By Proposition 2.1 to show the uniform convergence property of  $\ell_{\mathcal{G},\mu}(h,(x,y))$ , it suffices to upper bound  $\hat{\Re}_m(\mathcal{L}_{\mathcal{G},\mu}^{\mathcal{H}}) = \hat{\Re}_m(\ell \circ \mathcal{F}_{\mathcal{G},\mu}^{\mathcal{H}})$ , the empirical Rademacher complexity of the loss class. Since  $\ell$  is  $L$ -Lipschitz, by Ledoux-Talagrand's contraction principle [Ledoux and Talagrand, 1991], it follows that  $\hat{\Re}_m(\mathcal{L}_{\mathcal{G},\mu}^{\mathcal{H}}) = \hat{\Re}_m(\ell \circ \mathcal{F}_{\mathcal{G},\mu}^{\mathcal{H}}) \leq L \cdot \hat{\Re}_m(\mathcal{F}_{\mathcal{G},\mu}^{\mathcal{H}})$ . Thus, it actually suffices to upperbound  $\hat{\Re}_m(\mathcal{F}_{\mathcal{G},\mu}^{\mathcal{H}})$  instead. Starting with the definition of the empirical Rademacher complexity:
$$
\begin{array}{l} \hat {\mathfrak {R}} _ {m} \left(\mathcal {F} _ {\mathcal {G}, \mu} ^ {\mathcal {H}}\right) = \frac {1}{m} \mathbb {E} _ {\sigma \sim \{\pm 1 \} ^ {m}} \left[ \sup  _ {h \in \mathcal {H}} \left(\sum_ {i = 1} ^ {m} \sigma_ {i} y _ {i} \mathbb {E} _ {g \sim \mu} [ h (g (x _ {i})) ]\right) \right] \\ = \frac {1}{m} \mathbb {E} _ {\sigma \sim \{\pm 1 \} ^ {m}} \left[ \operatorname * {s u p} _ {h \in \mathcal {H}} \left(\mathbb {E} _ {g \sim \mu} \left[ \sum_ {i = 1} ^ {m} \sigma_ {i} h (g (x _ {i}))\right)\right) \right] \\ \leq \mathbb {E} _ {g \sim \mu} \left[ \frac {1}{m} \mathbb {E} _ {\sigma \sim \{\pm 1 \} ^ {m}} \left[ \sup _ {h \in \mathcal {H}} \sum_ {i = 1} ^ {m} \sigma_ {i} h (g (x _ {i})) \right] \right], \\ \end{array}
$$
where the last inequality follows from Jensen's inequality and Fubini's Theorem. Note that the quantity  $\frac{1}{m}\mathbb{E}_{\sigma \sim \{\pm 1\} ^m}\left[\sup_{h\in \mathcal{H}}\sum_{i = 1}^m\sigma_ih(g(x_i))\right]$  is the empirical Rademacher complexity of the hypothesis class  $\mathcal{H}$  over the sample  $\{g(x_1),\ldots ,g(x_m)\}$  drawn i.i.d from the distribution defined by first sampling from the marginal data distribution,  $x\sim \mathcal{D}_{\mathcal{X}}$  , and then applying the transformation  $g(x)$  . By standard VC arguments,  $\hat{\Re}_m(\mathcal{H})\leq O\left(\sqrt{\frac{d\ln(\frac{m}{d})}{m}}\right)$  , which implies that  $\hat{\Re}_m(\mathcal{F}_{\mathcal{G},\mu}^{\mathcal{H}})\leq$ $\mathbb{E}_{g\sim \mu}\left[\hat{\Re}_m(\mathcal{H})\right]\leq O\left(\sqrt{\frac{d\ln(\frac{m}{d})}{m}}\right)$  . Putting things together, we get  $\hat{\Re}_m(\mathcal{L}_{\mathcal{G},\mu}^{\mathcal{H}}) = \hat{\Re}_m(\ell \circ \mathcal{F}_{\mathcal{G},\mu}^{\mathcal{H}})\leq$ $O\left(\sqrt{\frac{dL^{2}\ln(\frac{m}{d})}{m}}\right)$  . Proposition 2.1 then implies that with probability  $1 - \delta$  over a sample  $S\sim \mathcal{D}^m$  of size  $m = O\left(\frac{dL^2\ln(\frac{L}{\epsilon}) + \ln(\frac{1}{\delta})}{\epsilon^2}\right)$  , we have
$$
\left| \mathbb {E} _ {\mathcal {D}} \left[ \ell_ {\mathcal {G}, \mu} (h, (x, y)) \right] - \hat {\mathbb {E}} _ {S} \left[ \ell_ {\mathcal {G}, \mu} (h, (x, y)) \right] \right| \leq \epsilon
$$
for all  $h\in \mathcal{H}$  simultaneously.
Uniform convergence of Lipschitz-losses immediately implies proper learning via ERM.
Theorem 4.2. Let  $\ell_{\mathcal{G},\mu}(h,(x,y)) = \ell (y\mathbb{E}_{g\sim \mu}[h(g(x)])$  such that  $\ell :\mathbb{R}\to \mathbb{R}$  is a L-Lipschitz function. For every hypothesis class  $\mathcal{H}$ , perturbation set and measure  $(\mathcal{G},\mu)$ , and  $(\epsilon ,\delta)\in (0,1)^2$ , the proper learning rule  $\mathcal{A}(S) = \arg \min_{h\in \mathcal{H}}\hat{\mathbb{E}}_S[\ell_{\mathcal{G},\mu}(h,(x,y))]$ , for any distribution  $\mathcal{D}$  over  $\mathcal{X}\times \mathcal{Y}$ , achieves, with probability at least  $1 - \delta$  over a sample  $S\sim \mathcal{D}^n$  of size  $n\geq O\left(\frac{\mathrm{VC}(\mathcal{H})L^2\ln(\frac{L}{\epsilon}) + \ln(\frac{1}{\delta})}{\epsilon^2}\right)$ , the guarantee
$$
\mathbb {E} _ {\mathcal {D}} \left[ \ell_ {\mathcal {G}, \mu} (\mathcal {A} (S), (x, y)) \right] \leq \inf _ {h \in \mathcal {H}} \mathbb {E} _ {\mathcal {D}} \left[ \ell_ {\mathcal {G}, \mu} (h, (x, y)) \right] + \epsilon .
$$
At a high-level, Theorem 4.2 shows finite VC dimension is sufficient for achieving robustness between the average- and worst-case using ERM. In fact, the next theorem, whose proof can be found in Appendix C.2, shows that finite VC dimension may not even be necessary for this to be true. The proof of Theorem 4.3 involves considering the well-known infinite VC class  $\mathcal{H} = \{x \mapsto \operatorname{sign}(\sin(wx)): w \in \mathbb{R}\}$  and picking  $(\mathcal{G}, \mu)$  such that  $\mathbb{E}_{g \sim \mu}[h(g(x))]'$  is essentially constant in  $x$  for all hypothesis  $h \in \mathcal{H}$ .
Theorem 4.3. Let  $\ell_{\mathcal{G},\mu}(h,(x,y)) = \ell (y\mathbb{E}_{g\sim \mu}[h(g(x)]))$  such that  $\ell :\mathbb{R}\to \mathbb{R}$  is a L-Lipschitz function. There exists  $\mathcal{H}$  and  $(\mathcal{G},\mu)$  such that  $\mathrm{VC}(\mathcal{H}) = \infty$  but  $\mathcal{H}$  is still properly learnable with respect to  $\ell_{\mathcal{G},\mu}(h,(x,y))$
Together, Theorems 4.2 and 4.3 showcase an interesting trade-off. Theorem 4.3 indicates that by carefully choosing  $(\mathcal{G},\mu)$ , the complexity of  $\mathcal{H}$  can be essentially smoothed out. On the other hand, Theorem 4.2 shows that any complexity in  $(\mathcal{G},\mu)$  can be smoothed out if  $\mathcal{H}$  has finite VC dimension. This interplay between the complexities of  $\mathcal{H}$  and  $(\mathcal{G},\mu)$  closely matches the intuition of Chapelle et al. [2000] in their work on Vicinal Risk Minimization. Note that the results in this section do not contradict that of Section 3 because  $\ell_{\mathcal{G},\mu}^{\rho}(h,(x,y))$  is a non-Lipschitz function of  $y\mathbb{E}_{g\sim \mu}[h(g(x))]$ .
We end this section by noting that Lipschitzness of  $\ell_{\mathcal{G},\mu}$  is sufficient but, in full generality, not necessary for proper learnability. For example, the loss function that completely ignores  $(\mathcal{G},\mu)$  and just computes the 0-1 loss is not Lipschitz, however, it is learnable via ERM when the VC dimension of  $\mathcal{H}$  is finite.
5 Proper Learnability Under Relaxed Competition
The results of Section 3 show that relaxing the adversarially robust loss may not always enable proper learning, even for very natural robust loss relaxations. In this section, we show that this bottleneck can be alleviated if we also allow the learner to compete against a slightly stronger notion of robustness. Furthermore, we expand on this idea by exploring other robust learning settings where allowing the learner to compete against a stronger notion of robustness enables proper learnability. We denote this type of modification to the standard adversarially and probabilistically robust leaning settings as robust learning under relaxed competition. Prior works on Tolerantly Robust PAC Learning [Ashtiani et al., 2022; Bhattacharjee et al., 2022] mentioned in the introduction fit under this umbrella.
Our main tool in this section is Lemma 5.1, which we term as Sandwich Uniform Convergence (SUC). Roughly speaking, SUC provides a sufficient condition under which ERM outputs a predictor that generalizes well with respect to a stricter notion of loss. A special case of SUC has implicitly appeared in margin theory (e.g., see Mohri et al. [2018, Section 5.4]), where one evaluates the 0-1 risk of the output hypothesis against the optimal margin 0-1 risk.
Lemma 5.1 (Sandwich Uniform Convergence). Let  $\ell_1(h,(x,y))$  and  $\ell_2(h,(x,y))$  be bounded, nonnegative loss functions such that for all  $h\in \mathcal{H}$  and  $(x,y)\in \mathcal{X}\times \mathcal{Y}$ , we have  $\ell_1(h,(x,y))\leq \ell_2(h,(x,y))\leq 1$ . If there exists a loss function  $\tilde{\ell} (h,(x,y))$  such that  $\ell_1(h,(x,y))\leq \tilde{\ell} (h,(x,y))\leq \ell_2(h,(x,y))$  and  $\tilde{\ell} (h,(x,y))$  enjoys the uniform convergence property with sample complexity  $n(\epsilon ,\delta)$ , then the learning rule  $\mathcal{A}(S) = \inf_{h\in \mathcal{H}}\hat{\mathbb{E}}_S[\ell_2(h,(x,y))]$  achieves, with probability  $1 - \delta$  over a sample  $S\sim \mathcal{D}^m$  of size  $m\geq n(\epsilon /2,\delta /2) + O\left(\frac{\ln(\frac{1}{\delta})}{\epsilon^2}\right)$ , the guarantee
$$
\mathbb {E} _ {\mathcal {D}} \left[ \ell_ {1} (\mathcal {A} (S), (x, y)) \right] \leq \inf  _ {h \in \mathcal {H}} \mathbb {E} _ {\mathcal {D}} \left[ \ell_ {2} (h, (x, y)) \right] + \epsilon .
$$
Proof. Let  $\mathcal{A}(S) = \inf_{h\in \mathcal{H}}\mathbb{E}_S[\ell_2(h,(x,y))]$ . By uniform convergence of  $\tilde{\ell} (h,(x,y))$ , we have that for sample size  $m = n(\frac{\epsilon}{2},\frac{\delta}{2})$ , with probability at least  $1 - \frac{\delta}{2}$ , over a sample  $S\sim \mathcal{D}^m$ , for every hypothesis  $h\in \mathcal{H}$  simultaneously,
$$
\mathbb {E} _ {\mathcal {D}} \left[ \tilde {\ell} (h, (x, y)) \right] \leq \hat {\mathbb {E}} _ {S} \left[ \tilde {\ell} (h, (x, y)) \right] + \frac {\epsilon}{2}.
$$
In particular, this implies that for  $\hat{h} = \mathcal{A}(S)$ , we have
$$
\mathbb {E} _ {\mathcal {D}} \left[ \tilde {\ell} (\hat {h}, (x, y)) \right] \leq \hat {\mathbb {E}} _ {S} \left[ \tilde {\ell} (\hat {h}, (x, y)) \right] + \frac {\epsilon}{2}.
$$
Since,  $\ell_1(h,(x,y))\leq \tilde{\ell} (h,(x,y))\leq \ell_2(h,(x,y))$  , we have that
$$
\mathbb {E} _ {\mathcal {D}} \left[ \ell_ {1} (\hat {h}, (x, y)) \right] \leq \hat {\mathbb {E}} _ {S} \left[ \ell_ {2} (h ^ {*}, (x, y)) \right] + \frac {\epsilon}{2}
$$
where  $h^* = \inf_{h \in \mathcal{H}} \mathbb{E}_{\mathcal{D}}[\ell_2(h, (x, y))]$ . It now remains to upper bound  $\hat{\mathbb{E}}_S[\ell_2(h^*, (x, y))]$  with high probability. However, a standard Hoeffding bound tells us that with probability  $1 - \frac{\delta}{2}$  over a sample  $S$  of size  $O\left(\frac{\ln\left(\frac{1}{\delta}\right)}{\epsilon^2}\right), \hat{\mathbb{E}}_S[\ell_2(h^*, (x, y))] \leq \mathbb{E}_{\mathcal{D}}[\ell_2(h^*, (x, y)] + \frac{\epsilon}{2}$ . Thus, by union bound, we get that with probability at least  $1 - \delta$ ,  $\mathbb{E}_{\mathcal{D}}\left[\ell_1(\hat{h}, (x, y))\right] \leq \mathbb{E}_{\mathcal{D}}\left[\ell_2(h^*, (x, y))\right] + \epsilon$ , using a sample of size  $n(\epsilon / 2, \delta / 2) + O\left(\frac{\ln\left(\frac{1}{\delta}\right)}{\epsilon^2}\right)$ .
Lemma 5.1 only requires the existence of such a sandwiched loss function that enjoys uniform convergence—we do not actually require it to be computable. In the next two sections, we exploit this fact to give three new generalization guarantees for the empirical risk minimizer over the adversarially robust loss  $\ell_{\mathcal{G}}(h,(x,y))$  and  $\rho$ -probabilistically robust loss  $\ell_{\mathcal{G},\mu}^{\rho}(h,(x,y))$ , hereafter denoted by  $\mathrm{RERM}(S;\mathcal{G}) \coloneqq \arg \min_{h\in \mathcal{H}}\hat{\mathbb{E}}_S[\ell_{\mathcal{G}}(h,(x,y))]$  and  $\mathrm{PRERM}(S;(\mathcal{G},\mu),\rho) \coloneqq \arg \min_{h\in \mathcal{H}}\hat{\mathbb{E}}_S[\ell_{\mathcal{G},\mu}^{\rho}(h,(x,y))]$  respectively.
5.1  $(\rho, \rho^{*})$ -Probabilistically Robust PAC Learning
In light of the hardness result of Section 3, we slightly tweak the learning setup in Definition 1 by allowing  $\mathcal{A}$  to compete against the hypothesis minimizing the probabilistic robust risk at a level  $\rho^{*} < \rho$ . Under this further relaxation, we show that proper learning becomes possible, and that too, via PRERM. In particular, Theorem 5.2 shows that while VC classes are not properly  $\rho$ -probabilistically robust PAC learnable, they are properly  $(\rho, \rho^{*})$ -probabilistically robust PAC learnable.
Theorem 5.2 (Proper  $(\rho, \rho^{*})$ -Probabilistically Robust PAC Learner). Let  $0 \leq \rho^{*} < \rho$ . Then, for every hypothesis class  $\mathcal{H}$ , perturbation set and measure  $(\mathcal{G}, \mu)$ , and  $(\epsilon, \delta) \in (0,1)^{2}$ , the proper learning rule  $\mathcal{A}(S) = \mathrm{PREM}(S; (\mathcal{G}, \mu), \rho^{*})$ , for any distribution  $\mathcal{D}$  over  $\mathcal{X} \times \mathcal{Y}$ , achieves, with probability at least  $1 - \delta$  over a sample  $S \sim \mathcal{D}^n$  of size  $n \geq O\left(\frac{\frac{\mathrm{VC}(\mathcal{H})}{(\rho - \rho^{*})^{2}} \ln \left(\frac{1}{(\rho - \rho^{*})\epsilon}\right) + \ln \left(\frac{1}{\delta}\right)}{\epsilon^{2}}\right)$ , the guarantee
$$
R _ {\mathcal {G}, \mu} ^ {\rho} (\mathcal {A} (S); \mathcal {D}) \leq \inf  _ {h \in \mathcal {H}} R _ {\mathcal {G}, \mu} ^ {\rho^ {*}} (h; \mathcal {D}) + \epsilon .
$$
In contrast to Section 3, where proper learning is not always possible, Theorem 5.2 shows that if we compare our learner to the best hypothesis for a slightly stronger level of probabilistic robustness, then not only is proper learning possible for VC classes, but it is possible via ERM. Our main technique to prove Theorem 5.2 is to consider a different probabilistically robust loss function that is (1) a Lipschitz function of  $y \mathbb{E}_{g \sim \mu} [h(g(x)) \neq y]$  and (2) can be sandwiched in between  $\ell_{\mathcal{G}, \mu}^{\rho^*}$  and  $\ell_{\mathcal{G}, \mu}^{\rho}$ . Then, Theorem 5.2 follows from Lemma 5.1. The full proof is in Appendix D.1.
5.2  $(\rho, \mathcal{G})$ -Probabilistically Robust PAC Learning
Can measure-independent learning guarantees be achieved if we instead compare the learner's probabilistically robust risk  $R_{\mathcal{G},\mu}^{\rho}$  to the best adversarially robust risk  $R_{\mathcal{G}}$  over  $\mathcal{H}$ ? We answer this in the affirmative by using SUC. We show that if one wants to compete against the best hypothesis for the worst-case adversarially robust risk, it is sufficient to run RERM.
Theorem 5.3 (Proper  $(\rho, \mathcal{G})$ -Probabilistically Robust PAC Learner). For every hypothesis class  $\mathcal{H}$ , perturbation set  $\mathcal{G}$ , and  $(\epsilon, \delta) \in (0,1)^2$ , the proper learning rule  $\mathcal{A}(S) = \mathrm{RERM}(S; \mathcal{G})$ , for any measure  $\mu$  over  $\mathcal{G}$  and any distribution  $\mathcal{D}$  over  $\mathcal{X} \times \mathcal{Y}$ , achieves, with probability at least  $1 - \delta$  over a sample  $S \sim \mathcal{D}^n$  of size  $n \geq O\left(\frac{\frac{\mathrm{VC}(\mathcal{H})}{\rho^2} \ln\left(\frac{1}{\rho\epsilon}\right) + \ln\left(\frac{1}{\delta}\right)}{\epsilon^2}\right)$ , the guarantee
$$
R _ {\mathcal {G}, \mu} ^ {\rho} (\mathcal {A} (S); \mathcal {D}) \leq \inf  _ {h \in \mathcal {H}} R _ {\mathcal {G}} (h; \mathcal {D}) + \epsilon .
$$
The proof of Theorem 5.3 can be found in Appendix D.2, which follows directly from Lemma 5.1 by a suitable choice of the sandwiched loss  $\ell$ . We make a few remarks about the practical importance of Theorem 5.3. Theorem 5.3 implies that for any pre-specified perturbation function class  $\mathcal{G}$  (for example  $\ell_p$  balls), running RERM is sufficient to obtain a hypothesis that is probabilistically robust with respect to any fixed measure  $\mu$  over  $\mathcal{G}$ . Moreover, the level of robustness of the predictor output by RERM, as measured by  $1 - \rho$ , scales directly with the sample size - the more samples one has, the smaller  $\rho$  can be made. Alternatively, for a fixed sample size  $m$ , desired error  $\epsilon$  and confidence  $\delta$ , one can use the sample complexity guarantee in Theorem 5.3 to back-solve the robustness guarantee  $\rho$ .
We highlight that the generalization bound in Theorem 5.3, is a measure-independent guarantee. This means that  $\rho$  does not quantify the level of robustness of the output hypothesis with respect to any one particular measure, but for any measure. This is desirable, as in contrast to the previous section, the  $\rho$  here more succinctly quantifies the level of robustness achieved by the output classifier. Lastly, we highlight that while the sample complexity of adversarially robust PAC learning can be exponential in the VC dimension of  $\mathcal{H}$  [Montasser et al., 2019], this is not the case for  $(\rho, \mathcal{G})$ -probabilistically robust PAC learning, where we only get a linear dependence on VC dimension.
5.3 Tolerantly Robust PAC Learning
In Tolerantly Robust PAC Learning [Bhattacharjee et al., 2022, Ashtiani et al., 2022], the learner's adversarily robust risk under a perturbation set  $\mathcal{G}$  is compared with the best achievable adversially robust risk for a larger perturbation set  $\mathcal{G}' \supset \mathcal{G}$ . Ashtiani et al. [2022] study the setting where both  $\mathcal{G}$  and  $\mathcal{G}'$  induce  $\ell_p$  balls with radius  $r$  and  $(1 + \gamma)r$  respectively. In the work of Bhattacharjee et al. [2022],  $\mathcal{G}$  is arbitrary, but  $\mathcal{G}'$  is constructed such that it induces perturbation sets that are the union of balls with radius  $\gamma$  that cover  $\mathcal{G}$ . Critically, Bhattacharjee et al. [2022] show that, under certain assumptions, running RERM over a larger perturbation set  $\mathcal{G}'$  is sufficient for Tolerantly Robust PAC learning. In this section, we take a slightly different approach to Tolerantly Robust PAC learning. Instead of having the learner compete against the best possible risk for a larger perturbation set, we have the learner still compete against the best possible adversially robust risk over  $\mathcal{G}$ , but evaluate the learner's adversially robust risk using a smaller perturbation set  $\mathcal{G}' \subset \mathcal{G}$ .
For what  $\mathcal{G}' \subset \mathcal{G}$  is Tolerantly Robust PAC learning via RERM possible? As an immediate result of Lemma 5.1 and Vapnik's "General Learning", finite VC dimension of the loss class  $\mathcal{L}_{\mathcal{G}'}^{\mathcal{H}} = \{(x,y) \mapsto \ell_{\mathcal{G}'}(h,(x,y)): h \in \mathcal{H}\}$  is sufficient. Note that finite VC dimension of  $\mathcal{L}_{\mathcal{G}'}^{\mathcal{H}}$  implies that the loss function  $\ell_{\mathcal{G}'}(h,(x,y))$  enjoys the uniform convergence property with sample complexity  $O\left(\frac{\mathrm{VC}(\mathcal{L}_{\mathcal{H}}^{\mathcal{G}'}) + \ln(\frac{1}{\delta})}{\epsilon^2}\right)$ . Thus, taking  $\ell_1(h,(x,y)) = \tilde{\ell}(h,(x,y)) = \ell_{\mathcal{G}'}(h,(x,y))$  and  $\ell_2(h,(x,y)) = \ell_{\mathcal{G}}(h,(x,y))$  in Lemma 5.1, we have that if there exists a  $\mathcal{G}' \subset \mathcal{G}$  such that  $\mathrm{VC}(\mathcal{L}_{\mathcal{H}}^{\mathcal{G}'}) < \infty$ , then with probability  $1 - \delta$  over a sample  $S \sim \mathcal{D}^n$  of size  $n = O\left(\frac{\mathrm{VC}(\mathcal{L}_{\mathcal{G}'}^{\mathcal{H}}) + \ln(\frac{1}{\delta})}{\epsilon^2}\right)$ ,
$$
R _ {\mathcal {G} ^ {\prime}} (\mathcal {A} (S); \mathcal {D}) \leq \inf  _ {h \in \mathcal {H}} R _ {\mathcal {G}} (h; \mathcal {D}) + \epsilon ,
$$
where  $\mathcal{A}(S) = \mathrm{RERM}(S;\mathcal{G})$
Alternatively, if  $\mathcal{G}' \subset \mathcal{G}$  such that there exists a finite subset  $\tilde{\mathcal{G}} \subset \mathcal{G}$  where  $\ell_{\mathcal{G}'}(h, (x,y)) \leq \ell_{\tilde{\mathcal{G}}} (h, (x,y))$ , then Tolerantly Robust PAC learning via RERM is possible with sample complexity that scales according to  $O\left(\frac{\mathrm{VC}(\mathcal{H})\log(|\tilde{\mathcal{G}}|) + \ln(\frac{1}{\delta})}{\epsilon^2}\right)$ . This result essentially comes from the fact that the VC dimension of the loss class for any finite perturbation set  $\tilde{\mathcal{G}}$  incurs only a log  $(|\tilde{\mathcal{G}}|)$  blow-up from the VC dimension of  $\mathcal{H}$  (see Lemma 1.1 in Attias et al. [202]). Thus, finite VC dimension of
$\mathcal{H}$  implies finite VC dimension of the loss class  $\mathcal{L}_{\mathcal{H}}^{\tilde{\mathcal{G}}}$  which implies uniform convergence of the loss  $\ell_{\tilde{\mathcal{G}}}(\bar{h},(x,y))$ , as needed for Lemma 5.1 to hold.
We now give an example where such a finite approximation of  $\mathcal{G}'$  is possible. In order to do so, we will need to consider a metric space of perturbation functions  $(\mathcal{G}, d)$  and define a notion of "nice" perturbation sets, similar to "regular" hypothesis classes from Bhattacharjee et al. [2022].
Definition 4 (r-Nice Perturbation Set). Let  $\mathcal{H}$  be a hypothesis class and  $(\mathcal{G},d)$  a metric space of perturbation functions. Let  $B_{r}(g)\coloneqq \{g^{\prime}\in \mathcal{G}:d(g,g^{\prime})\leq r\}$  denote a closed ball of radius  $r$  centered around  $g\in \mathcal{G}$ . We say that  $\mathcal{G}^{\prime}\subset \mathcal{G}$  is  $r$ -Nice with respect to  $\mathcal{H}$ , if for all  $x\in \mathcal{X}$ ,  $h\in \mathcal{H}$ , and  $g\in \mathcal{G}^{\prime}$ , there exists a  $g^{*}\in \mathcal{G}$ , such that  $g\in B_r(g^*)$  and  $h(g(x)) = h(g'(x))$  for all  $g^{\prime}\in B_r(g^{*})$ .
Definition 4 prevents a situation where a hypothesis  $h \in \mathcal{H}$  is non-robust to an isolated perturbation function  $g \in \mathcal{G}'$  for any given labelled example  $(x, y) \in \mathcal{X} \times \mathcal{Y}$ . If a hypothesis  $h$  is non-robust to a perturbation  $g \in \mathcal{G}'$ , then Definition 4 asserts that there must exist a small ball of perturbation functions in  $\mathcal{G}$  over which  $h$  is also non-robust. Next, we define the covering number.