SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Steve Hanneke  
Department of Computer Science  
Purdue University  
steve.hanneke@gmail.com
Aryeh Kontorovich  
Department of Computer Science  
Ben-Gurion University of the Negev  
karyeh@cs.bgu.ac.il
Guy Kornowski  
Department of Computer Science and Applied Mathematics  
Weizmann Institute of Science  
guy.kornowski@weizmann.ac.il
We generalize the notion of average Lipschitz smoothness proposed by Ashlagi et al. [2021] by extending it to Hölder smoothness. This measure of the "effective smoothness" of a function is sensitive to the underlying distribution and can be dramatically smaller than its classic "worst-case" Hölder constant. We consider both the realizable and the agnostic (noisy) regression settings, proving upper and lower risk bounds in terms of the average Hölder smoothness; these rates improve upon both previously known rates even in the special case of average Lipschitz smoothness. Moreover, our lower bound is tight in the realizable setting up to log factors, thus we establish the minimax rate. From an algorithmic perspective, since our notion of average smoothness is defined with respect to the unknown underlying distribution, the learner does not have an explicit representation of the function class, hence is unable to execute ERM. Nevertheless, we provide distinct learning algorithms that achieve both (nearly) optimal learning rates. Our results hold in any totally bounded metric space, and are stated in terms of its intrinsic geometry. Overall, our results show that the classic worst-case notion of Hölder smoothness can be essentially replaced by its average, yielding considerably sharper guarantees.
A fundamental theme throughout learning theory and statistics is that "smooth" functions ought to be easier to learn than "rough" ones — an intuition that has been formalized and rigorously established in various frameworks [Györfi et al., 2002, Tsybakov, 2008, Giné and Nickl, 2021]. Hölder continuity is a natural and well-studied notion of smoothness that measures the extent to which nearby points can differ in function value and includes Lipschitz continuity as an important special case.
These global moduli of smoothness, while convenient for theoretical analysis, suffer from the shortcoming of being overly pessimistic. Indeed, being distribution-independent, they fail to distinguish a function that is highly oscillatory everywhere from one that is smooth over most of the probability mass; see Figure 1 for a simple illustration. Moreover, classically studied classes of average smoothness (e.g. Besov space) typically fix some distribution in advance (predominantly uniform), and then turn to consider smooth functions with respect to that single distribution. Thus, from a distribution-free statistical learning perspective — where the underlying distribution is assumed to be unknown — such classes fall short.
Figure 1: Illustration of a function and a measure  $\mu$  exhibiting a large gap between "worst-case" smoothness (occurring in low density regions) and average-smoothness with respect to  $\mu$ .
Seeking to address these drawbacks, Ashlagi et al. [2021] proposed a natural notion of average Lipschitz smoothness with respect to a distribution. Their average Lipschitz modulus can be considerably (even infinitely) smaller than the standard Lipschitz constant, while still being able to control the excess risk. However, the risk bounds obtained by Ashlagi et al. are far from optimal, while the optimal rates for distribution-free learning of average smoothness classes remained unknown. In particular, the cost of adapting to the smoothness with respect to the underlying distribution (in contrast to using classic worst-case smoothness) remained unclear so far.
Our contributions. In this work, we generalize the aforementioned notion of average Lipschitz smoothness by extending it to Hölder smoothness of any exponent  $\beta \in (0,1]$ . After formally defining the average Hölder smoothness of a function with respect to a distribution in Section 2.1, our contributions can be summarized as follows:
- Bracketing numbers upper bound (Theorem 3.1). We establish a nearly-optimal distribution-free bound on the bracketing entropy of our proposed average-smooth function class, serving as the main crux on which we base our analyses throughout the paper. In particular, although it is known that asymptotically empirical covering numbers yield sharper bounds than bracketing numbers, $^{1}$  in the case of average smoothness we reveal that the latter are tight up to a logarithmic factor.
- Realizable sample complexity (Theorem 3.4). We derive a nearly-optimal sample complexity required for uniform convergence of average-Hölder functions in the realizable case, which was not previously known even in the special case of average Lipschitz functions.
- Optimal realizable learning algorithm (Theorem 4.1). Since our notion of average smoothness is defined with respect to the unknown sampling distribution, the learner does not have an explicit representation of the function class, and hence is unable to execute ERM. We are able to overcome this obstacle by constructing a realizable nonparametric regression algorithm with a nearly-optimal learning rate. Such a rate was not previously known even in the special case of average Lipschitz smoothness.
- Agnostic learning algorithm (Theorem 5.1). We provide yet another learning algorithm for the fully agnostic (i.e. noisy) regression setting. Once again, our derived rate was not previously known even in the special case of average Lipschitz smoothness.
- Matching lower bound (Theorem 6.1). We prove a lower bound, showing that all the results mentioned above are tight up to logarithmic factors in the realizable case, establishing the (nearly) minimax risk rate for average-smooth classes.
- Illustrative comparisons (Section 7). Finally, we illustrate the extent to which the proposed smoothness notion is sharper than previously studied notions. We provide examples in which
Uniform convergence of the  $L_{1}(\mathcal{D})$  distance between the upper and lower bracket functions implies that, in the limit of sample size, the  $\varepsilon$ -bracket functions are almost surely an empirical  $(1 + o(1))\varepsilon$ -cover; see Section 2 for a reminder of relevant definitions.
$^{2}$ Indeed, the learner cannot know for sure whether any given non-classically Hölder function belongs to the average-Hölder class.
the "optimistic" average-Hölder constant is infinitely apart from both its "pessimistic" worst-case counterpart, or even the average-Lipschitz  $(\beta = 1)$  constant, exemplifying the substantial (possibly infinite) speed-ups in terms of learning rates.
1.1 Related work.
The sample complexities associated to distribution-free learning of (classic) Hölder classes is well covered in the literature, see for example the books by Györfi et al. [2002], Tsybakov [2008].
Previous notions of average smoothness include Bounded Variation (BV) [Appell et al., 2014] in dimensions one and higher [Kuipers and Niederreiter, 1974, Niederreiter and Talay, 2006]. One-dimensional BV has found learning-theoretic applications [Bartlett et al., 1997, Long, 2004, Anthony and Bartlett, 1999], but to our knowledge the higher-dimensional variants have not. Moreover, the positive results require  $\mu$  to be uniformly distributed on a segment, and the aforementioned results break down for more general measures — especially if  $\mu$  is not known to the learner.
Sobolev spaces, and the Sobolev-Slobodetskii norm in particular [Agranovich, 2015], bear some resemblance to our average Hölder smoothness. However, Ashlagi et al. [2021, Appendix I] demonstrate that from a learning-theoretic perspective this notion is inadequate for general (i.e., non-uniform or Lebesgue) measures, as it cannot be used to control sample complexity. Results for controlling bracketing in terms of various measures of average smoothness include Nickl and Potscher [2007], who bound the bracketing numbers of Besov- and Sobolev-type classes and Malykhin [2010], who used the averaged modulus of continuity developed by Sendov and Popov [1988]; again, these are all defined under the Lebesgue measure. While it is easy to define these smoothness notions with respect to arbitrary distributions, we are not aware of any existing work to bound their corresponding sample complexity (or even their covering or bracketing numbers) in a distribution-independent manner. Moreover, the smoothness notion studied in this paper is defined over arbitrary metric spaces, whereas previous notions are typically restricted to Euclidean structures (or variants thereof). Despite of the considerable generality of our setting, we are able to provide tight bounds for all metric spaces alike, without requiring specialized analyses.
A seminal work on recovering functions with spatially inhomogeneous smoothness from noisy samples is Donoho and Johnstone [1998]. Arguably in the spirit of  $\mu$ -dependent Hölder smoothness, some of the classic results on  $k$ -NN risk decay rates were refined by Chaudhuri and Dasgupta [2014] via an analysis that captures the interplay between the metric and the sampling distribution. Another related notion is that of Probabilistic Lipschitzness (PL) [Urner and Ben-David, 2013, Urner et al., 2013, Kpotufe et al., 2015], which seeks to relax a hard Lipschitz condition on the regression function. While PL is in the same spirit as our notion, one critical distinction from our work is that, while existing analyses of learning under PL have focused specifically on binary classification, our interest in the present work is learning real-valued functions.
As previously mentioned, the main feature setting this work apart from others studying regression under average smoothness is that our notion is defined with respect to a general, unknown measure  $\mu$ . The notable exception is, of course, Ashlagi et al. [2021] — who introduced the framework of efficiently learning smooth-on-average functions with respect to an unknown distribution. Although extending their definition from Lipschitz to Hölder average smoothness was straightforward, optimal minimax rates are likely inaccessible via their techniques, which relied on empirical covering numbers. Estimating the magnitude of these random objects was a formidable challenge, and Ashlagi et al. were only able to do so to within an additive error decaying with sample size; this sampling noise appears to present an inherent obstruction to optimal rates. Thus, our results required a novel technique to overcome this obstruction, which we did by tightly controlling the bracketing entropy. Our Hölder-type extension is a direct adaptation of the Pointwise Minimum Slope Extension (PMSE) developed for the Lipschitz special case by Ashlagi et al., which in turn is closely related to the one introduced by Oberman [2008].
2 Preliminaries.
Setting. Throughout the paper we consider functions  $f: \Omega \to [0,1]$  where  $(\Omega, \rho)$  is a metric space. We will consider a distribution  $\mathcal{D}$  over  $\Omega \times [0,1]$  with marginal  $\mu$  over  $\Omega$ , such that  $(\Omega, \rho, \mu)$  forms a metric probability space (namely,  $\mu$  is supported on the Borel  $\sigma$ -algebra induced by  $\rho$ ). We associate
to any measurable function  $f: \Omega \rightarrow [0,1]$  its  $L_{1}$  risk  $L_{\mathcal{D}}(f) \coloneqq \mathbb{E}_{(X,Y) \sim \mathcal{D}} |f(X) - Y|$ , and its empirical risk  $L_{S}(f) \coloneqq \frac{1}{n} \sum_{i=1}^{n} |f(X_{i}) - Y_{i}|$  with respect to a sample  $S = (X_{i}, Y_{i})_{i=1}^{n} \sim \mathcal{D}^{n}$ . More generally, we associate to any measurable function its  $L_{1}$  norm  $\| f \|_{L_{1}(\mu)} \coloneqq \mathbb{E}_{X \sim \mu} |f(X)|$  and given a sample  $(X_{1}, \ldots, X_{n}) \sim \mu^{n}$  we denote its  $L_{1}$  norm with respect to the empirical measure  $\| f \|_{L_{1}(\mu_{n})} \coloneqq \frac{1}{n} \sum_{i=1}^{n} |f(X_{i})|$ .
We say that a distribution  $\mathcal{D}$  over  $\Omega \times [0,1]$  is realizable by a function class  $\mathcal{F} \subset [0,1]^{\Omega}$  if there exists an  $f^{*} \in \mathcal{F}$  such that  $L_{\mathcal{D}}(f^{*}) = 0$ . Thus,  $f^{*}(X) = Y$  almost surely, where  $(X,Y) \sim \mathcal{D}$ .
Metric notions. The diameter of  $A \subset \Omega$  is  $\mathrm{diam}(A) := \sup_{x, x' \in A} \rho(x, x')$ , and we denote by  $B(x, r) := \{x' \in \Omega : \rho(x, x') \leq r\}$  the closed ball around  $x \in \Omega$  of radius  $r > 0$ . For  $t > 0$ ,  $A, B \subset \Omega$ , we say that  $A$  is a  $t$ -cover of  $B$  if  $B \subset \bigcup_{a \in A} B(a, t)$ , and define the  $t$ -covering number  $\mathcal{N}_B(t)$  to be the minimal cardinality of any  $t$ -cover of  $B$ . We say that  $A \subset B \subset \Omega$  is a  $t$ -packing of  $B$  if  $\rho(a, a') \geq t$  for all  $a \neq a' \in A$ . We call  $V$  a  $t$ -net of  $B$  if it is a  $t$ -cover and a  $t$ -packing. The induced Voronoi partition of  $B$  with respect to a net  $V$  is its partitioning into subsets sharing the same nearest neighbor in  $V$  (with ties broken in some consistent arbitrary manner). A metric space  $(\Omega, \rho)$  is said to be **doubling** if there exists  $d \in \mathbb{N}$  such that every  $r$ -ball in  $\Omega$  is contained in the union of some  $d$ $r/2$ -balls. The **doubling dimension** is defined as  $\min_{d \geq 1} \log_2 d$  where the minimum is taken over  $d$  satisfying the doubling property.
**Bracketing.** Given any two functions  $l, u: \Omega \to [0,1]$ , we say that  $f: \Omega \to [0,1]$  belongs to the bracket  $[l,u]$  if  $l \leq f \leq u$ . A set of brackets  $\mathcal{B}$  is said to cover a function class  $\mathcal{F}$  if any function in  $\mathcal{F}$  belongs to some bracket in  $\mathcal{B}$ . We say that  $[l,u]$  is a  $t$ -bracket with respect to a norm  $\| \cdot \|$  if  $\| u - l \| \leq t$ . The  $t$ -bracketing number  $\mathcal{N}_{[l]}(\mathcal{F}, \| \cdot \|, t)$  is defined as the minimal cardinality of any set of  $t$ -brackets that covers  $\mathcal{F}$ . The logarithm of this quantity is called the bracketing entropy.
Remark 2.1 (Covering vs. bracketing). Having recalled two notions that quantify the "size" of a normed function space  $(\mathcal{F},\| \cdot \|)$  — namely, its covering and bracketing numbers — it is useful to note they are related through
$$
\mathcal {N} _ {\mathcal {F}} (\varepsilon) \leq \mathcal {N} _ {[ ]} (\mathcal {F}, \| \cdot \|, 2 \varepsilon), \tag {1}
$$
though no converse inequality of this sort holds in general. On the other hand, the main advantage of using bracketing numbers for generalization bounds is that it suffices to bound the ambient bracketing numbers with respect to the distribution-specific metric, as opposed to the empirical covering numbers which are necessary to guarantee generalization [van der Vaart and Wellner, 1996, Section 2.1.1].
Strong and weak mean. For any non-negative random variable  $Z$  we define its weak mean by  $\mathbb{W}[Z] \coloneqq \sup_{t > 0} t \operatorname*{Pr}[Z \geq t]$ , and note that  $\mathbb{W}[Z] \leq \mathbb{E}[Z]$  by Markov's inequality. In the special case where  $Z$  has finite support of size  $N \geq 3$  where each atom has mass  $1/N$  we have the reverse inequality  $\mathbb{E}[Z] \leq 2 \log(N) \mathbb{W}[Z]$  [Ashlagi et al., 2021, Lemma 22].
2.1 Average smoothness.
For  $\beta \in (0,1]$  and  $f:\Omega \to \mathbb{R}$ , we define its  $\beta$ -slope at  $x\in \Omega$  to be  $\Lambda_f^\beta (x)\coloneqq \sup_{y\in \Omega \setminus \{x\}}\frac{|f(x) - f(y)|}{\rho(x,y)^\beta}$ . Recall that  $f$  is called  $\beta$ -Hölder continuous if  $\| f\|_{\mathrm{H}\tilde{\mathrm{o}} l^{\beta}}\coloneqq \sup_{x\in \Omega}\Lambda_f^\beta (x) < \infty$ , with this quantity serving as its Hölder seminorm. In particular, when  $\beta = 1$  these are exactly the Lipschitz functions equipped with the Lipschitz seminorm. For a metric probability space  $(\Omega ,\rho ,\mu)$ , we consider the average  $\beta$ -slope to be the mean of  $\Lambda_f^\beta (X)$  where  $X\sim \mu$ . Namely, we define
$$
\overline {{\Lambda}} _ {f} ^ {\beta} (\mu) := \mathbb {E} _ {X \sim \mu} [ \Lambda_ {f} ^ {\beta} (X) ],
$$
$$
\widetilde {\Lambda} _ {f} ^ {\beta} (\mu) := \mathbb {W} _ {X \sim \mu} \left[ \Lambda_ {f} ^ {\beta} (X) \right] = \sup  _ {t > 0} t \cdot \mu \left(x: \Lambda_ {f} ^ {\beta} (x) \geq t\right).
$$
Notably,
$$
\widetilde {\Lambda} _ {f} ^ {\beta} (\mu) \leq \overline {{\Lambda}} _ {f} ^ {\beta} (\mu) \leq \| f \| _ {\mathrm {H o l} ^ {\beta}}, \tag {2}
$$
where each subsequent pair can be infinitely apart — as we demonstrate in Section 7. Having defined notions of averaged smoothness, we can further define their corresponding function spaces
$$
\operatorname {H o l} _ {L} ^ {\beta} (\Omega) := \left\{f: \Omega \to [ 0, 1 ]: \| f \| _ {\operatorname {H o l} ^ {\beta}} \leq L \right\},
$$
$$
\overline {{\mathrm {H o l}}} _ {L} ^ {\beta} (\Omega , \mu) := \left\{f: \Omega \to [ 0, 1 ]: \overline {{\Lambda}} _ {f} ^ {\beta} (\mu) \leq L \right\},
$$
$$
\widetilde {\mathrm {H o l}} _ {L} ^ {\beta} (\Omega , \mu) := \left\{f: \Omega \to [ 0, 1 ]: \widetilde {\Lambda} _ {f} ^ {\beta} (\mu) \leq L \right\}.
$$
We occasionally omit  $\mu$  when it is clear from context. Note that  $\mathrm{H\bar{o}l}_L^\beta (\Omega)\subset \overline{\mathrm{H}\bar{o}\mathrm{l}}_L^\beta (\Omega ,\mu)\subset$ $\widetilde{\mathrm{H}\bar{o}\mathrm{l}}_L^\beta (\Omega ,\mu)$  due to Eq. (2), where both containments are strict in general. The special case of  $\beta = 1$  recovers the average-Lipschitz spaces  $\mathrm{Lip}_L(\Omega)\subset \overline{\mathrm{Lip}}_L(\Omega ,\mu)\subset \widetilde{\mathrm{Lip}}_L(\Omega ,\mu)$  studied by Ashlagi et al. [2021].
3 Generalization bounds
Our first goal is to bound the bracketing entropy (namely, the logarithm of the bracketing number) of average-Hölder classes. We present this bound in full generality in terms of the underlying metric space, as captured by its covering number (see Corollary 3.5 for the typical scaling of covering numbers). As we will soon establish, this bound implies nearly-tight generalization guarantees in terms of the average smoothness constant.
Theorem 3.1. For any metric probability space  $(\Omega, \rho, \mu)$ , any  $\beta \in (0,1]$  and any  $0 < \epsilon < L$ , it holds
$$
\begin{array}{l} \log \mathcal {N} _ {[ ]} (\overline {{\mathrm {H o l}}} _ {L} ^ {\beta} (\Omega , \mu), L _ {1} (\mu), \varepsilon) \leq \log \mathcal {N} _ {[ ]} (\widetilde {\mathrm {H o l}} _ {L} ^ {\beta} (\Omega , \mu), L _ {1} (\mu), \varepsilon) \\ \leq \mathcal {N} _ {\Omega} \left(\left(\frac {\varepsilon}{1 2 8 L \log (1 / \varepsilon)}\right) ^ {1 / \beta}\right) \cdot \log \left(\frac {1 6 \log_ {2} (1 / \varepsilon)}{\varepsilon}\right). \\ \end{array}
$$
Crucially, the bound above does not depend on  $\mu$ , allowing us to obtain distribution-free generalization guarantees. We defer the proof of Theorem 3.1 to Appendix B.1. We start by showing that bounding the bracketing entropy implies a generalization bound in the realizable case:
Proposition 3.2. Suppose  $(\Omega, \rho)$  is a metric space,  $\mathcal{F} \subseteq [0,1]^{\Omega}$  is a function class, and let  $\mathcal{D}$  be a distribution over  $\Omega \times [0,1]$  which is realizable by  $\mathcal{F}$ , with marginal  $\mu$  over  $\Omega$ . Then with probability at least  $1 - \delta$  over drawing a sample  $S \sim \mathcal{D}^n$  it holds that for all  $f \in \mathcal{F}$ :
$$
L _ {\mathcal {D}} (f) \leq 1. 0 1 L _ {S} (f) + \inf  _ {\alpha \geq 0} \left(\alpha + \frac {2 0 5 \log \mathcal {N} _ {[ ]} (\mathcal {F} , L _ {1} (\mu) , \alpha)}{n}\right) + \frac {2 0 5 \log (1 / \delta)}{n}.
$$
Remark 3.3 (Constant is arbitrary). In Proposition 3.2 and in what follows, the constant multiplying  $L_{S}(f)$  is arbitrary, and can be replaced by  $(1 + \gamma)$  for any  $\gamma > 0$  at the expense of multiplying the remaining summands by  $\gamma^{-1}$ . In the next section we will provide a realizable regression algorithm that returns an approximate empirical risk minimizer  $f$  for which  $L_{S}(f) \approx 0$ , thus this constant will not matter for our purposes.
We prove Proposition 3.2 in Appendix B.2. By combining Theorem 3.1 with Proposition 3.2 and setting  $\alpha = \varepsilon /2$ , we obtain the following realizable sample complexity result.
Theorem 3.4. For any metric space  $(\Omega, \rho)$ , any  $\beta \in (0,1]$  and any  $0 < \epsilon < L$ , let  $\mathcal{D}$  be a distribution over  $\Omega \times [0,1]$  realizable by  $\widetilde{\mathrm{Hol}}_L^\beta(\Omega, \mu)$ . Then there exists  $N = N(\beta, \varepsilon, \delta) \in \mathbb{N}$  satisfying
$$
N = \widetilde {O} \left(\frac {\mathcal {N} _ {\Omega} \left(\left(\frac {\varepsilon}{2 5 6 L \log (1 / \varepsilon)}\right) ^ {1 / \beta}\right) + \log (1 / \delta)}{\varepsilon}\right)
$$
such that as long as  $n \geq N$ , with probability at least  $1 - \delta$  over drawing a sample  $S \sim \mathcal{D}^n$  it holds that for all  $f \in \widetilde{\mathrm{H}\tilde{\mathrm{o}}\mathrm{l}}_L^\beta(\Omega, \mu)$ :
$$
L _ {\mathcal {D}} (f) \leq 1. 0 1 L _ {S} (f) + \varepsilon .
$$
The same claim holds for the smaller class  $\overline{\mathrm{H}\bar{\mathrm{o}}\mathrm{l}}_L^\beta (\Omega ,\mu)$
Corollary 3.5 (Doubling metrics). In most cases of interest,  $(\Omega, \rho)$  is a doubling metric space of some dimension  $d$ , e.g. when  $\Omega$  is a subset of  $\mathbb{R}^d$  (or more generally a  $d$ -dimensional Banach space). For  $d$ -dimensional doubling spaces of finite diameter we have  $\mathcal{N}_{\Omega}(\varepsilon) \lesssim \left(\frac{1}{\varepsilon}\right)^d$  [Gottlieb et al., 2016, Lemma 2.1], which, plugged into Theorem 3.4, yields the simplified sample complexity bound
$$
N = \widetilde {O} \left(\frac {L ^ {d / \beta}}{\varepsilon^ {(d + \beta) / \beta}}\right),
$$
$$
L _ {\mathcal {D}} (f) \leq 1. 0 1 L _ {S} (f) + \widetilde {O} \left(\frac {L ^ {d / (d + \beta)}}{n ^ {\beta / (d + \beta)}}\right),
$$
up to a constant which depends (exponentially) on  $d$ , but is independent of  $L, n$ .
Remark 3.6 (Tightness). The bounds in Theorem 3.1 and Theorem 3.4 are both tight up to logarithmic factors, as we will prove in Section 6.
4 Realizable learning algorithm
Recall that without knowing  $\mu$ , the underlying distribution over  $\Omega$ , we cannot know for sure whether a function  $f$  belongs to  $\overline{\mathrm{H}\circ\mathrm{l}}_L^\beta (\Omega ,\mu)$  (except for the trivial case  $f\in \mathrm{H}\circ \mathrm{l}_L^\beta (\Omega)$ ). This gives rise to the challenge of designing a fully empirical algorithm — since standard empirical risk minimization is not possible. To that end, we provide the following algorithmic result with optimal guarantees (up to logarithmic factors).
Theorem 4.1. For any metric space  $(\Omega, \rho)$ , any  $\beta \in (0,1]$  and any  $0 < \epsilon < L$ , let  $\mathcal{D}$  be a distribution over  $\Omega \times [0,1]$  realizable by  $\overline{\mathrm{H}\bar{\mathrm{o}}\mathrm{l}}_L^\beta (\Omega ,\mu)$ . Then there exists a polynomial time learning algorithm  $\mathcal{A}$  which, given a sample  $S \sim \mathcal{D}^n$  of size  $n \geq N$  for some  $N = N(\beta ,\varepsilon ,\delta) \in \mathbb{N}$  satisfying
$$
N = \widetilde {O} \left(\frac {\mathcal {N} _ {\Omega} \left(\left(\frac {\varepsilon}{2 5 6 L \log (1 / \varepsilon)}\right) ^ {1 / \beta}\right) + \log (1 / \delta)}{\varepsilon}\right),
$$
constructs a hypothesis  $f = \mathcal{A}(S)$  such that  $L_{\mathcal{D}}(f) \leq \varepsilon$  with probability at least  $1 - \delta$ .
Remark 4.2 (Doubling metrics). As mentioned in Corollary 3.5, in most cases of interest we have  $\mathcal{N}_{\Omega}(\varepsilon) \lesssim \left(\frac{1}{\varepsilon}\right)^d$ . In that case, the algorithm above has sample complexity
$$
L _ {\mathcal {D}} (f) = \widetilde {O} \left(\frac {L ^ {d / (d + \beta)}}{n ^ {\beta / (d + \beta)}}\right),
$$