SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this work, we developed optimal learners for realizable regression in PAC learning and online learning. Moreover, we identified combinatorial dimensions that characterize learnability in these settings. We hope that our work can lead to simplified characterizations for these problems. We believe that the main limitation of our work is that the OIG-based dimension we propose is more complicated than the dimensions that have been proposed in the past, like the fat-shattering dimension (which, as we explain, does not characterize learnability in the realizable regression setting). Nevertheless, despite its complexity, this is the first dimension that characterizes learnability in the realizable regression setting. More to that, our work leaves as an important next step to prove (or disprove) the conjecture that the (combinatorial and simpler)  $\gamma$ -DS dimension is qualitatively equivalent to the  $\gamma$ -OIG dimension. Another future direction, that is not directly related to this conjecture, is to better understand the gap between the fat-shattering dimension and the OIG-based dimension.
Amin Karbasi acknowledges funding in direct support of this work from NSF (IIS-1845032), ONR (N00014- 19-1-2406), and the AI Institute for Learning-Enabled Optimization at Scale (TILOS). Grigoris Velegkas is supported by TILOS, the Onassis Foundation, and the Bodossaki Foundation. This work was done in part while some of the authors were visiting Archimedes AI Research Center.
[AACSZ22] Ishaq Aden-Ali, Yeshwanth Cherapanamjeri, Abhishek Shetty, and Nikita Zhivotovskiy. The one-inclusion graph algorithm is not always optimal. arXiv preprint arXiv:2212.09270, 2022. 4
[AACSZ23] Ishaq Aden-Ali, Yeshwanth Cherapanamjeri, Abhishek Shetty, and Nikita Zhivotovsky. Optimal pac bounds without uniform convergence. arXiv preprint arXiv:2304.09167, 2023. 4
[AB99] Martin Anthony and Peter L Bartlett. Neural network learning: Theoretical foundations, volume 9. Cambridge university press Cambridge, 1999. 18, 21
[ABDCBH97] Noga Alon, Shai Ben-David, Nicolo Cesa-Bianchi, and David Haussler. Scale-sensitive dimensions, uniform convergence, and learnability. Journal of the ACM (JACM), 44(4):615-631, 1997. 2, 4
$\left[\mathrm{ABDH}^{+}20\right]$  Hassan Ashtiani, Shai Ben-David, Nicholas JA Harvey, Christopher Liaw, Abbas Mehrabian, and Yaniv Plan. Near-optimal sample complexity bounds for robust learning of gaussian mixtures via compression schemes. Journal of the ACM (JACM), 67(6):1-42, 2020. 25
[ABED+21] Noga Alon, Omri Ben-Eliezer, Yuval Dagan, Shay Moran, Moni Naor, and Eylon Yogev. Adversarial laws of large numbers and optimal regret in online classification. In Proceedings of the 53rd annual ACM SIGACT symposium on theory of computing, pages 447–455, 2021. 3, 4
[AH22] Idan Attias and Steve Hanneke. Adversarily robust learning of real-valued functions. arXiv preprint arXiv:2206.12977, 2022. 4, 25
[AHHM22] Noga Alon, Steve Hanneke, Ron Holzman, and Shay Moran. A theory of pac learnability of partial concept classes. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS), pages 658-671. IEEE, 2022. 4, 7, 17, 19, 25
[AHM22] Idan Attias, Steve Hanneke, and Yishay Mansour. A characterization of semi-supervised adversarially-robust pac learnability. arXiv preprint arXiv:2202.05420, 2022. 25
[Bac21] Francis Bach. Learning theory from first principles. Online version, 2021. 1
[BBM05] Peter Bartlett, Olivier Bousquet, and Shahar Mendelson. Local rademacher complexities. Annals of Statistics, 33(4):1497-1537, 2005. 4
$\left[\mathrm{BCD}^{+}22\right]$  Nataly Brukhim, Daniel Carmon, Irit Dinur, Shay Moran, and Amir Yehudayoff. A characterization of multiclass learnability. In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS), pages 943-955. IEEE, 2022. 1, 4, 5, 7, 9, 25, 28
[BDGR22] Adam Block, Yuval Dagan, Noah Golowich, and Alexander Rakhlin. Smoothed online learning is as easy as statistical learning. In Conference on Learning Theory, pages 1716-1786. PMLR, 2022. 4
[BDHM+19] Shai Ben-David, Pavel Hrubes, Shay Moran, Amir Shpilka, and Amir Yehudayoff. Learnability can be undecidable. Nature Machine Intelligence, 1(1):44-48, 2019. 8, 30
[FHMM23] Yuval Filmus, Steve Hanneke, Idan Mehalel, and Shay Moran. Optimal prediction using expert advice and randomized littlestone dimension. arXiv preprint arXiv:2302.13849, 2023. 30
[FR20] Dylan Foster and Alexander Rakhlin. Beyond ucb: Optimal and efficient contextual bandits with regression oracles. In International Conference on Machine Learning, pages 3199-3210. PMLR, 2020. 4
[FW95] Sally Floyd and Manfred Warmuth. Sample compression, learnability, and the vapnik-chervonenkis dimension. Machine learning, 21(3):269-304, 1995. 25
[GBC16] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016. 1
[GHST05] Thore Graepel, Ralf Herbrich, and John Shawe-Taylor. Pac-bayesian compression bounds on the prediction error of learning algorithms for classification. Machine Learning, 59:55-76, 2005. 25
[GKN14] Lee-Ad Gottlieb, Aryeh Kontorovich, and Pinhas Nisnevitch. Near-optimal sample compression for nearest neighbors. Advances in Neural Information Processing Systems, 27, 2014. 25
[Gol21] Noah Golowich. Differentially private nonparametric regression under a growth condition. In Conference on Learning Theory, pages 2149-2192. PMLR, 2021. 4
[Han16] Steve Hanneke. The optimal sample complexity of pac learning. The Journal of Machine Learning Research, 17(1):1319-1333, 2016. 1, 4
[HKLM22] Max Hopkins, Daniel M Kane, Shachar Lovett, and Gaurav Mahajan. Realizable learning is all you need. In Conference on Learning Theory, pages 3015-3069. PMLR, 2022. 4, 31
[HKS18] Steve Hanneke, Aryeh Kontorovich, and Menachem Sadigurschi. Agnostic sample compression for linear regression. arXiv preprint arXiv:1810.01864, 2018. 25
[HKS19] Steve Hanneke, Aryeh Kontorovich, and Menachem Sadigurschi. Sample compression for real-valued learners. In Algorithmic Learning Theory, pages 466-488. PMLR, 2019. 2, 4, 25, 26
[HLW94] David Haussler, Nick Littlestone, and Manfred K Warmuth. Predicting  $\{0,1\}$ -functions on randomly drawn points. Information and Computation, 115(2):248-292, 1994. 1, 4, 7
[JS03] Tadeusz Januszkiiewicz and Jacek Światkowski. Hyperbolic coxeter groups of large dimension. Commentarii Mathematici Helvetici, 78(3):555-583, 2003. 28
[Kég03] Balázs Kégl. Robust regression by boosting the median. In Learning Theory and Kernel Machines, pages 258-272. Springer, 2003. 8, 25
[KKMV23] Alkis Kalavasis, Amin Karbasi, Shay Moran, and Grigoris Velegkas. Statistical indistinguishability of learning algorithms. arXiv preprint arXiv:2305.14311, 2023. 4
[KS94] Michael J Kearns and Robert E Schapire. Efficient distribution-free learning of probabilistic concepts. Journal of Computer and System Sciences, 48(3):464-497, 1994. 5
[KS23] Pieter Kleer and Hans Simon. Primal and dual combinatorial dimensions. Discrete Applied Mathematics, 327:185-196, 2023. 4
[KSW17] Aryeh Kontorovich, Sivan Sabato, and Roi Weiss. Nearest-neighbor sample compression: Efficiency, consistency, infinite dimensions. Advances in Neural Information Processing Systems, 30, 2017. 25
[KVK22] Alkis Kalavasis, Grigoris Velegkas, and Amin Karbasi. Multiclass learnability beyond the pac framework: Universal rates and partial concept classes. arXiv preprint arXiv:2210.02297, 2022. 4
[Lit88] Nick Littlestone. Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. Machine learning, 2:285-318, 1988. 1, 2, 4, 9, 10
[NLW86] Nick Littlestone and Manfred Warmuth. Relating data compression and learnability. 1986. 8, 25, 26
[Men02] Shahar Mendelson. Improving the sample complexity using global data. IEEE transactions on Information Theory, 48(7):1977-1991, 2002. 4, 8, 9
[MHS19] Omar Montasser, Steve Hanneke, and Nathan Srebro. Vc classes are adversarially robustly learnable, but only improperly. In Conference on Learning Theory, pages 2512-2530. PMLR, 2019. 25
[MHS20] Omar Montasser, Steve Hanneke, and Nati Srebro. Reducing adversarially robust learning to non-robust pac learning. Advances in Neural Information Processing Systems, 33:14626–14637, 2020. 25
[MHS21] Omar Montasser, Steve Hanneke, and Nathan Srebro. Adversarily robust learning with unknown perturbation sets. In Conference on Learning Theory, pages 3452-3482. PMLR, 2021. 25
[MHS22] Omar Montasser, Steve Hanneke, and Nati Srebro. Adversarily robust learning: A generic minimax optimal learner and characterization. Advances in Neural Information Processing Systems, 35:37458-37470, 2022. 4, 8, 25
[MKFI22] Jason Millionis, Alkis Kalavasis, Dimitris Fotakis, and Stratis Ioannidis. Differentially private regression with unbounded covariates. In International Conference on Artificial Intelligence and Statistics, pages 3242-3273. PMLR, 2022. 4
[MY16] Shay Moran and Amir Yehudayoff. Sample compression schemes for vc classes. Journal of the ACM (JACM), 63(3):1-10, 2016. 25, 27
[Nat89] Balas K Natarajan. On learning sets and functions. Machine Learning, 4(1):67-97, 1989. 1, 4, 5, 6
[Osai3] Damian L Osajda. A construction of hyperbolic coxeter groups. Commentarii Mathematici Helvetici, 88(2):353-367, 2013. 28
[Pol90] David Pollard. Empirical processes: theory and applications. Ims, 1990. 5
[Pol12] David Pollard. Convergence of stochastic processes. Springer Science & Business Media, 2012. 2
[RBR09] Benjamin IP Rubinstein, Peter L Bartlett, and J Hyam Rubinstein. Shifting: One-inclusion mistake bounds and sample compression. Journal of Computer and System Sciences, 75(1):37-59, 2009. 4, 7
[RS14] Alexander Rakhlin and Karthik Sridharan. Online non-parametric regression. In Conference on Learning Theory, pages 1232-1264. PMLR, 2014. 3, 4
[RST10] Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Online learning: Random averages, combinatorial parameters, and learnability. Advances in Neural Information Processing Systems, 23, 2010. 3
[RST15a] Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Online learning via sequential complexities. J. Mach. Learn. Res., 16(1):155-186, 2015. 3, 4, 30
[RST15b] Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Sequential complexities and uniform martingale laws of large numbers. *Probability theory and related fields*, 161:111-153, 2015. 3, 4, 30
[RST17] Alexander Rakhlin, Karthik Sridharan, and Alexandre B Tsybakov. Empirical entropy, minimax regret and minimax risk. 2017. 4
[RST23] Vinod Raman, Unique Subedi, and Ambuj Tewari. A characterization of online multiclass learnability. arXiv preprint arXiv:2303.17716, 2023. 1, 3, 4
[She17] Or Sheffet. Differentially private ordinary least squares. In International Conference on Machine Learning, pages 3105-3114. PMLR, 2017. 4
[Sim97] Hans Ulrich Simon. Bounds on the number of examples needed for learning functions. SIAM Journal on Computing, 26(3):751-763, 1997. 2, 3, 4, 5, 6, 9
[SLX22] David Simchi-Levi and Yunzong Xu. Bypassing the monster: A faster and simpler optimal algorithm for contextual bandits under realizability. Mathematics of Operations Research, 47(3):1904-1931, 2022. 4
[SMB22] Han Shao, Omar Montasser, and Avrim Blum. A theory of pac learnability under transformation invariances. arXiv preprint arXiv:2202.07552, 2022. 4
[SSSSS10] Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Learnability, stability and uniform convergence. The Journal of Machine Learning Research, 11:2635-2670, 2010. 3
[Val84] Leslie G Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134-1142, 1984. 1
[Vap99] Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions on neural networks, 10(5):988-999, 1999. 1
[VC71] V. Vapnik and A. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications, 16(2):264-280, 1971. 6
[WHEY15] Yair Wiener, Steve Hanneke, and Ran El-Yaniv. A compression technique for analyzing disagreement-based active learning. J. Mach. Learn. Res., 16:713-745, 2015. 25
A Notation and Definitions
We first overview standard definitions about sample complexity in PAC learning.
PAC Sample Complexity. The realizable PAC sample complexity  $\mathcal{M}(\mathcal{H};\varepsilon ,\delta)$  of  $\mathcal{H}$  is defined as
$$
\mathcal {M} (\mathcal {H}; \varepsilon , \delta) = \inf  _ {A \in \mathcal {A}} \mathcal {M} _ {A} (\mathcal {H}; \varepsilon , \delta), \tag {1}
$$
where the infimum is over all possible learning algorithms and  $\mathcal{M}_A(\mathcal{H};\varepsilon ,\delta)$  is the minimal integer such that for any  $m\geq \mathcal{M}_A(\mathcal{H};\varepsilon ,\delta)$ , every distribution  $\mathcal{D}_{\mathcal{X}}$  on  $\mathcal{X}$ , and, true target  $h^\star \in \mathcal{H}$ , the expected loss  $\mathbf{E}_{x\sim \mathcal{D}_{\mathcal{X}}}[\ell (A(T)(x),h^{\star}(x))]$  of  $A$  is at most  $\varepsilon$  with probability  $1 - \delta$  over the training set  $T = \{(x,h^{\star}(x)):x\in S\}$ ,  $S\sim \mathcal{D}_{\mathcal{X}}^{m}$ .
PAC Cut-Off Sample Complexity. We slightly overload the notation of the sample complexity and we define
$$
\mathcal {M} (\mathcal {H}; \varepsilon , \delta , \gamma) = \inf  _ {A \in \mathcal {A}} \mathcal {M} _ {A} (\mathcal {H}; \varepsilon , \delta , \gamma), \tag {2}
$$
where the infimum is over all possible learning algorithms and  $\mathcal{M}_A(\mathcal{H};\varepsilon ,\delta ,\gamma)$  is the minimal integer such that for any  $m\geq \mathcal{M}_A(\mathcal{H};\varepsilon ,\delta ,\gamma)$ , every distribution  $\mathcal{D}_{\mathcal{X}}$  on  $\mathcal{X}$ , and, true target  $h^\star \in \mathcal{H}$ , the expected cut-off loss  $\mathbf{Pr}_{x\sim \mathcal{D}_{\mathcal{X}}}[\ell (A(T)(x),h^{\star}(x)) > \gamma ]$  of  $A$  is at most  $\varepsilon$  with probability  $1 - \delta$  over the training set  $T = \{(x,h^{\star}(x)):x\in S\}$ ,  $S\sim \mathcal{D}_{\mathcal{X}}^{m}$ .
Lemma 1 (Equivalence Between Sample Complexities). For every  $\varepsilon, \delta \in (0,1)^2$  and every  $\mathcal{H} \subseteq [0,1]^{\mathcal{X}}$ , where  $\mathcal{X}$  is the input domain, it holds that
$$
\mathcal {M} (\mathcal {H}; \sqrt {\varepsilon}, \delta , \sqrt {\varepsilon}) \leq \mathcal {M} (\mathcal {H}; \varepsilon , \delta) \leq \mathcal {M} (\mathcal {H}; \varepsilon / 2, \delta , \varepsilon / 2)
$$
Proof. Let  $A$  be a learning algorithm. We will prove the statement for each fixed  $A$  and for each data-generating distribution  $\mathcal{D}$ , so the result follows by taking the infimum over the learning algorithms.
Assume that the cut-off sample complexity of  $A$  is  $\mathcal{M}_A(\mathcal{H};\varepsilon /2,\delta ,\varepsilon /2)$ . Then, with probability  $1 - \delta$  over the training sample  $S\sim \mathcal{D}$ , for its expected loss it holds that
$$
\underset {x \sim \mathcal {D} _ {\mathcal {X}}} {\mathbf {E}} \left[ \ell (A (S; x), h ^ {\star} (x)) \right] \leq \frac {\varepsilon}{2} + \left(1 - \frac {\varepsilon}{2}\right) \cdot \frac {\varepsilon}{2} \leq \varepsilon ,
$$
thus,  $\mathcal{M}_A(\mathcal{H};\varepsilon ,\delta)\leq \mathcal{M}_A(\mathcal{H};\varepsilon /2,\delta ,\varepsilon /2)$
The other direction follows by using Markov's inequality. In particular, if we have that with probability at least  $1 - \delta$  over  $S\sim \mathcal{D}$  it holds that
$$
\underset {x \sim \mathcal {D} _ {\mathcal {X}}} {\mathbf {E}} \left[ \ell (A (S; x), h ^ {\star} (x)) \right] \leq \varepsilon ,
$$
then Markov's inequality gives us that
$$
\operatorname * {P r} _ {x \sim \mathcal {D} _ {\mathcal {X}}} \left[ \ell (A (S; x), h ^ {\star} (x)) \geq \sqrt {\varepsilon} \right] \leq \sqrt {\varepsilon},
$$
which shows that  $\mathcal{M}_A(\mathcal{H};\varepsilon ,\delta)\geq \mathcal{M}_A(\mathcal{H};\sqrt{\varepsilon},\delta ,\sqrt{\varepsilon})$
ERM Sample Complexity. In the special case where  $\mathcal{A}$  is the class ERM of all possible ERM algorithms, i.e., algorithms that return a hypothesis whose sample error is exactly 0, we define the ERM sample complexity as the number of samples required by the worst-case ERM algorithm, i.e.,
$$
\mathcal {M} _ {\mathrm {E R M}} (\mathcal {H}; \varepsilon , \delta) = \sup  _ {A \in \mathrm {E R M}} \mathcal {M} _ {A} (\mathcal {H}; \varepsilon , \delta), \tag {3}
$$
and its cut-off analogue as
$$
\mathcal {M} _ {\mathrm {E R M}} (\mathcal {H}; \varepsilon , \delta , \gamma) = \sup  _ {A \in \mathrm {E R M}} \mathcal {M} _ {A} (\mathcal {H}; \varepsilon , \delta , \gamma), \tag {4}
$$
B  $\gamma$ -Graph Dimension and ERM Learnability
In this section, we show that  $\gamma$ -graph dimension determines the learnability of  $\mathcal{H} \subseteq [0,1]^{\mathcal{X}}$  using any ERM learner. We first revisit the notion of partial concept classes which will be useful for deriving our algorithms.
B.1 Partial Concept Classes and A Naive Approach that Fails
[AHHM22] proposed an extension of the binary PAC model to handle partial concept classes, where  $\mathcal{H} \subseteq \{0,1,\star\}^{\mathcal{X}}$ , for some input domain  $\mathcal{X}$ , where  $h(x) = \star$  should be thought of as  $h$  not knowing the label of  $x \in \mathcal{X}$ . The main motivation behind their work is that partial classes allow one to conveniently express data-dependent assumptions. As an intuitive example, a halfspace with margin is a partial function that is undefined inside the forbidden margin and is a well-defined halfspace outside the margin boundaries. Instead of dealing with concept classes  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  where each concept  $h \in \mathcal{H}$  is a total function  $h: \mathcal{X} \to \mathcal{Y}$ , we study partial concept classes  $\mathcal{H} \subseteq (\mathcal{Y} \cup \{\star\})^{\mathcal{X}}$ , where each concept  $h$  is now a partial function and  $h(x) = \star$  means that the function  $h$  is undefined at  $x$ . We define the support of  $h$  as the set  $\operatorname{supp}(h) = \{x \in \mathcal{X}: h(x) \neq \star\}$ . Similarly as in the case of total classes, we say that a finite sequence  $S = (x_{1},y_{1},\ldots,x_{n},y_{n})$  is realizable with respect to  $\mathcal{H}$  if there exists some  $h^{*} \in \mathcal{H}$  such that  $h^{*}(x_{i}) = y_{i}, \forall i \in [n]$ .
An important notion related to partial concept classes is that of disambiguation.
Definition 14 (Disambiguation of Partial Concept Class [AHHM22]). Let  $\mathcal{X}$  be an input domain. A total concept class  $\overline{\mathcal{H}} \subseteq \{0,1\}^{\mathcal{X}}$  is a special type of a partial concept. Given some partial concept class  $\mathcal{H} \subseteq \{0,1,\star\}^{\mathcal{X}}$  we say that  $\overline{H}$  is a disambiguation of  $\mathcal{H}$  if for any finite sequence  $S \in (\mathcal{X} \times \{0,1\})^{*}$  if  $S$  is realizable with respect to  $\mathcal{H}$ , then  $S$  is realizable with respect to  $\overline{\mathcal{H}}$ .
Intuitively, by disambiguating a partial concept class we convert it to a total concept class without reducing its "expressivity".
Let us first describe an approach to prove the upper bound, i.e., that if the scaled-graph dimension is finite for all scales then the class is ERM learnable, that does not work. We could perform the following transformation, inspired by the multiclass setting [DSS14]: for any  $h \in \mathcal{H} \subseteq [0,1]^{\mathcal{X}}$ , let us consider the function  $\widetilde{h}: \mathcal{X} \times [0,1] \to \{0,1\}$  with  $\widetilde{h}(x,y) = 1$  if and only if  $h(x) = y$ ,  $\widetilde{h}(x,y) = 0$  if and only if  $\ell(h(x),y) > \varepsilon$  and  $\widetilde{h}(x,y) = \star$  otherwise. This induces a new binary partial hypothesis class  $\widetilde{\mathcal{H}} = \{\widetilde{h}: h \in \mathcal{H}\} \subseteq \{0,1,\star\}^{\mathcal{X}}$ . We note that  $\mathbb{D}_{\varepsilon}^{\mathrm{G}}(\mathcal{H}) = \mathrm{VC}(\widetilde{\mathcal{H}})$ . However, we cannot use ERM for the partial concept class since in general this approach fails. In particular, a sufficient condition for applying ERM is that VC( $\{\mathrm{supp}(\widetilde{h}): \widetilde{h} \in \widetilde{\mathcal{H}}\}$  <  $\infty$ .
Remark 1. Predicting  $\star$  in [AHHM22] implies a mistake for the setting of partial concept classes. However, in our regression setting,  $\star$  is interpreted differently and corresponds to loss at most  $\gamma$  which is desirable. In particular, the hard instance for proper learners in the partial concepts paper (see Proposition 4 in [AHHM22]) is good in settings where predicting  $\star$  does not count as a mistake, as in our regression case.
B.2 Main Result
We are now ready to state the main result of this section. We will prove the next statement.
Theorem 1. Let  $\ell$  be the absolute loss function. For every class  $\mathcal{H} \subseteq [0,1]^{\mathcal{X}}$  and for any  $\varepsilon, \delta, \gamma \in (0,1)^3$ , the sample complexity bound for realizable PAC regression by any ERM satisfies
$$
\Omega \left(\frac {\mathbb {D} _ {\gamma} ^ {\mathrm {G}} (\mathcal {H}) + \log (1 / \delta)}{\varepsilon}\right) \leq \mathcal {M} _ {\mathrm {E R M}} (\mathcal {H}; \varepsilon , \delta , \gamma) \leq O \left(\frac {\mathbb {D} _ {\gamma} ^ {\mathrm {G}} (\mathcal {H}) \log (1 / \varepsilon) + \log (1 / \delta)}{\varepsilon}\right).
$$
In particular, any ERM algorithm  $\mathbb{A}$  achieves
$$
\underset {x \sim \mathcal {D} _ {\mathcal {X}}} {\mathbf {E}} [ \ell (A (S; x), h ^ {\star} (x) ] \leq \inf  _ {\gamma \in [ 0, 1 ]} \gamma + \widetilde {\Theta} \left(\frac {\mathbb {D} _ {\gamma} ^ {\mathrm {G}} (\mathcal {H}) + \log (1 / \delta)}{n}\right),
$$
with probability at least  $1 - \delta$  over  $S$  of size  $n$ .
Proof. We prove the upper bound and the lower bound of the statement separately.
Upper Bound for the ERM learner. We deal with the cut-off loss problem with parameters  $(\varepsilon, \delta, \gamma) \in (0,1)^3$ . Our proof is based on a technique that uses a "ghost" sample to establish generalization guarantees of the algorithm. Let us denote
$$
\operatorname {e r} _ {\mathcal {D}, \gamma} (h) \triangleq \underset {x \sim \mathcal {D} _ {\mathcal {X}}} {\mathbf {P r}} [ \ell (h (x), h ^ {\star} (x)) > \gamma ], \tag {1}
$$
and for a dataset  $z\in (\mathcal{X}\times [0,1])^n$
$$
\widehat {\mathrm {e r}} _ {z, \gamma} (h) \triangleq \frac {1}{| z |} \sum_ {(x, y) \in z} \mathbb {I} \left\{\ell (h (x), y) > \gamma \right\}. \tag {2}
$$
We will start by showing the next symmetrization lemma in our setting. Essentially, it bounds the probability that there exists a bad ERM learner by the probability that there exists an ERM learner on a sample  $r$  whose performance on a hidden sample  $s$  is bad. For a similar result, see Lemma 4.4 in [AB99].
Lemma 2 (Symmetrization). Let  $\varepsilon, \gamma \in (0,1)^2$ ,  $n > 0$ . Fix  $Z = \mathcal{X} \times [0,1]$ . Let
$$
Q _ {\varepsilon , \gamma} = \{z \in Z ^ {n}: \exists h \in \mathcal {H}: \widehat {\operatorname {e r}} _ {z, \gamma} (h) = 0, \operatorname {e r} _ {\mathcal {D}, \gamma} (h) > \varepsilon \} \tag {3}
$$
$$
R _ {\varepsilon , \gamma} = \left\{(r, s) \in Z ^ {n} \times Z ^ {n}: \exists h \in \mathcal {H}: \operatorname {e r} _ {\mathcal {D}, \gamma} (h) > \varepsilon , \widehat {\operatorname {e r}} _ {r, \gamma} (h) = 0, \widehat {\operatorname {e r}} _ {s, \gamma} (h) \geq \varepsilon / 2 \right\}. \tag {4}
$$
Then, for  $n \geq c / \varepsilon$ , where  $c$  is some absolute constant, we have that
$$
\mathcal {D} ^ {n} \left(Q _ {\varepsilon , \gamma}\right) \leq 2 \mathcal {D} ^ {2 n} \left(R _ {\varepsilon , \gamma}\right).
$$
Proof. We will show that  $\mathcal{D}^{2n}(R_{\varepsilon, \gamma}) \geq \frac{\mathcal{D}^n(Q_{\varepsilon, \gamma})}{2}$ . By the definition of  $R_{\varepsilon, \gamma}$  we can write
$$
\mathcal {D} ^ {2 n} (R _ {\varepsilon , \gamma}) = \int_ {Q _ {\varepsilon , \gamma}} \mathcal {D} ^ {n} (s: \exists h \in \mathcal {H}, \operatorname {e r} _ {\mathcal {D}, \gamma} (h) > \varepsilon , \widehat {\operatorname {e r}} _ {r, \gamma} (h) = 0, \widehat {\operatorname {e r}} _ {s, \gamma} (h) \geq \varepsilon / 2) d \mathcal {D} ^ {n} (r).
$$