SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Definition 5 (Covering Number). Let  $(\mathcal{M},d)$  be a metric space, let  $\kappa \subset \mathcal{M}$  be a subset, and  $r > 0$ . Let  $B_{r}(x) = \{x^{\prime}\in \mathcal{M}:d(x,x^{\prime})\leq r\}$  denote the ball of radius  $r$  centered around  $x\in \mathcal{M}$ . A subset  $\mathcal{C}\subset \mathcal{M}$  is an  $r$ -covering of  $\kappa$  if  $\kappa \subset \bigcup_{c\in \mathcal{C}}B_r(c)$ . The covering number of  $\kappa$ , denoted  $\mathcal{N}_r(\mathcal{K},d)$ , is the smallest cardinality of any  $r$ -covering of  $\kappa$ .
Finally, let  $\mathcal{G}_{2r}^{\prime} = \bigcup_{g\in \mathcal{G}^{\prime}}B_{2r}(g)$  denote the union over all balls of radius  $2r$  with centers in  $\mathcal{G}^\prime$ . Theorem [5.4] then states that if there exists a set  $\mathcal{G}^{\prime}\subset \mathcal{G}$  that is  $r$ -Nice with respect to  $\mathcal{H}$ , then Tolerantly Robust PAC learning is possible via RERM with sample complexity that scales logarithmically with  $\mathcal{N}_r(\mathcal{G}_{2r}^{\prime},d)$ .
Theorem 5.4 (Tolerantly Robust PAC learning under Nice Perturbations). Let  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  be a hypothesis class and  $(\mathcal{G}, d)$  be a metric space of perturbation functions. Given a subset  $\overline{\mathcal{G}'} \subset \mathcal{G}$  such that  $\mathcal{G}'$  is  $r$ -Nice with respect to  $\mathcal{H}$ , then the proper learning rule  $\mathcal{A}(S) = \mathrm{RERM}(S; \mathcal{G})$ , for any distribution  $\mathcal{D}$  over  $\mathcal{X} \times \mathcal{Y}$ , achieves, with probability at least  $1 - \delta$  over a sample  $S \sim \mathcal{D}^n$  of size  $n \geq O\left(\frac{\mathrm{VC}(\mathcal{H}) \log(\mathcal{N}_r(\mathcal{G}_{2r}', d)) + \ln(\frac{1}{\delta})}{\epsilon^2}\right)$ , the guarantee
$$
R _ {\mathcal {G} ^ {\prime}} (\mathcal {A} (S); \mathcal {D}) \leq \inf  _ {h \in \mathcal {H}} R _ {\mathcal {G}} (h; \mathcal {D}) + \epsilon .
$$
In Appendix D we give a full proof and show that  $\ell_p$  balls are  $r$ -Nice perturbation sets for robustly learning halfspaces. Note that Theorem 5.4 does not require apriori knowledge of the  $r$ -Nice perturbation set  $\mathcal{G}'$ , but just its existence. Therefore, Theorem 5.4 applies to the largest possible  $r$ -Nice perturbation subset of  $\mathcal{G}$ . This is important from a practical standpoint as computing an  $r$ -Nice perturbation set might not be computationally tractable. Accordingly, while RERM may not be a proper adversarially robust PAC learner [Montasser et al., 2019], Theorem 5.4 shows that RERM is a proper tolerantly robust PAC learner.
In this work, we show that there exists natural relaxations of the adversarially robust loss for which finite VC dimension is still not sufficient for proper learning. On the other hand, we identify a large set of Lipschitz robust loss relaxations for which finite VC dimension is sufficient for proper learning. In addition, we give new generalization guarantees for RERM. As future work, we are interested in understanding whether our robust loss relaxations can be used to mitigate the tradeoff between achieving adversarial robustness and maintaining high nominal performance. In addition, it is also interesting to find a combinatorial characterization of proper probabilistically robust PAC learning with respect to  $\ell_{\mathcal{G},\mu}^{\rho}$ .
We acknowledge the support of NSF via grant IIS-2007055. We also acknowledge Idan Attias for introducing us to Tolerantly Robust PAC Learning. VR acknowledges the support of the NSF Graduate Research Fellowship.
A Mao, M Mohri, and Y Zhong. Cross-entropy loss functions: Theoretical analysis and applications. arxiv. arXiv preprint arXiv:2304.07288, 2023.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
Omar Montasser, Steve Hanneke, and Nathan Srebro. Vc classes are adversarially robustly learnable, but only improperly. In Conference on Learning Theory, pages 2512-2530. PMLR, 2019.
Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial examples. arXiv preprint arXiv:1801.09344, 2018.
Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John C Duchi, and Percy Liang. Adversarial training can hurt generalization. arXiv preprint arXiv:1906.06032, 2019.
Leslie Rice, Anna Bair, Huan Zhang, and J Zico Kolter. Robustness between the worst and average case. Advances in Neural Information Processing Systems, 34:27840-27851, 2021.
Alexander Robey, Luiz FO Chamon, George J Pappas, and Hamed Hassani. Probabilistically robust learning: Balancing average-and worst-case performance. arXiv preprint arXiv:2202.01136, 2022.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adversarily robust generalization requires more data. Advances in neural information processing systems, 31, 2018.
Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin-Yu Chen, and Yupeng Gao. Is robustness the cost of accuracy? a comprehensive study on the robustness of 18 deep image classification models. In Proceedings of the European Conference on Computer Vision (ECCV), pages 631-648, 2018.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint arXiv:1805.12152, 2018.
Vladimir Vapnik. Estimation of dependences based on empirical data. Springer Science & Business Media, 2006.
Vladimir Naumovich Vapnik and Aleksei Yakovlevich Chervonenkis. On uniform convergence of the frequencies of events to their probabilities. Teoriya Veroyatnosti i ee Primeneniya, 16(2):264-279, 1971.
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International conference on machine learning, pages 5286-5295. PMLR, 2018.
Yao-Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Russ R Salakhutdinov, and Kamalika Chaudhuri. A closer look at accuracy vs. robustness. Advances in neural information processing systems, 33:8588-8601, 2020.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pages 7472-7482. PMLR, 2019.