Title: Near-optimal learning with average Hölder smoothness
Venue: NeurIPS 2023
Accepted: yes
Keywords: Hölder smoothness

average smoothness

bracketing numbers

generalization

risk bounds

metric space
Abstract: We generalize the notion of average Lipschitz smoothness proposed by Ashlagi et al. (COLT 2021) by extending it to Hölder smoothness. This measure of the "effective smoothness" of a function is sensitive to the underlying distribution and can be dramatically smaller than its classic "worst-case" Hölder constant.
We consider both the realizable and the agnostic (noisy) regression settings, proving upper and lower risk bounds in terms of the average Hölder smoothness; these rates improve upon both previously known rates even in the special case of average Lipschitz smoothness.
Moreover, our lower bound is tight in the realizable setting up to log factors, thus we establish the minimax rate.
From an algorithmic perspective, since our notion of average smoothness is defined with respect to the unknown underlying distribution, the learner does not have an explicit representation of the function class, hence is unable to execute ERM. Nevertheless, we provide distinct learning algorithms that achieve both (nearly) optimal learning rates.
Our results hold in any totally bounded metric space, and are stated in terms of its intrinsic geometry.
Overall, our results show that the classic worst-case notion of Hölder smoothness can be essentially replaced by its average, yielding considerably sharper guarantees.

Near-optimal learning with average Hölder smoothness

Abstract

We generalize the notion of average Lipschitz smoothness proposed by Ashlagi et al. [2021] by extending it to Hölder smoothness. This measure of the 'effective smoothness' of a function is sensitive to the underlying distribution and can be dramatically smaller than its classic 'worst-case' Hölder constant. We consider both the realizable and the agnostic (noisy) regression settings, proving upper and lower risk bounds in terms of the average Hölder smoothness; these rates improve upon both previously known rates even in the special case of average Lipschitz smoothness. Moreover, our lower bound is tight in the realizable setting up to log factors, thus we establish the minimax rate. From an algorithmic perspective, since our notion of average smoothness is defined with respect to the unknown underlying distribution, the learner does not have an explicit representation of the function class, hence is unable to execute ERM. Nevertheless, we provide distinct learning algorithms that achieve both (nearly) optimal learning rates. Our results hold in any totally bounded metric space, and are stated in terms of its intrinsic geometry. Overall, our results show that the classic worst-case notion of Hölder smoothness can be essentially replaced by its average, yielding considerably sharper guarantees.

1 Introduction

A fundamental theme throughout learning theory and statistics is that 'smooth' functions ought to be easier to learn than 'rough' ones - an intuition that has been formalized and rigorously established in various frameworks [Györfi et al., 2002, Tsybakov, 2008, Giné and Nickl, 2021]. Hölder continuity is a natural and well-studied notion of smoothness that measures the extent to which nearby points can differ in function value and includes Lipschitz continuity as an important special case.

These global moduli of smoothness, while convenient for theoretical analysis, suffer from the shortcoming of being overly pessimistic. Indeed, being distribution-independent, they fail to distinguish a function that is highly oscillatory everywhere from one that is smooth over most of the probability mass; see Figure 1 for a simple illustration. Moreover, classically studied classes of average smoothness (e.g. Besov space) typically fix some distribution in advance (predominantly uniform), and then turn to consider smooth functions with respect to that single distribution. Thus, from a distribution-free statistical learning perspective - where the underlying distribution is assumed to be unknown - such classes fall short.

Seeking to address these drawbacks, Ashlagi et al. [2021] proposed a natural notion of average Lipschitz smoothness with respect to a distribution. Their average Lipschitz modulus can be considerably (even infinitely) smaller than the standard Lipschitz constant, while still being able to control the excess risk. However, the risk bounds obtained by Ashlagi et al. are far from optimal, while the optimal rates for distribution-free learning of average smoothness classes remained unknown. In particular, the cost of adapting to the smoothness with respect to the underlying distribution (in contrast to using classic worst-case smoothness) remained unclear so far.

Our contributions. In this work, we generalize the aforementioned notion of average Lipschitz smoothness by extending it to Hölder smoothness of any exponent β ∈ (0 , 1] . After formally defining the average Hölder smoothness of a function with respect to a distribution in Section 2.1, our contributions can be summarized as follows:

Bracketing numbers upper bound (Theorem 3.1). We establish a nearly-optimal distribution-free bound on the bracketing entropy of our proposed average-smooth function class, serving as the main crux on which we base our analyses throughout the paper. In particular, although it is known that asymptotically empirical covering numbers yield sharper bounds than bracketing numbers, 1 in the case of average smoothness we reveal that the latter are tight up to a logarithmic factor.

Realizable sample complexity (Theorem 3.4). We derive a nearly-optimal sample complexity required for uniform convergence of average-Hölder functions in the realizable case, which was not previously known even in the special case of average Lipschitz functions.

Optimal realizable learning algorithm (Theorem 4.1). Since our notion of average smoothness is defined with respect to the unknown sampling distribution, the learner does not have an explicit representation of the function class, and hence is unable to execute ERM. 2 We are able to overcome this obstacle by constructing a realizable nonparametric regression algorithm with a nearly-optimal learning rate. Such a rate was not previously known even in the special case of average Lipschitz smoothness.

Agnostic learning algorithm (Theorem 5.1). We provide yet another learning algorithm for the fully agnostic (i.e. noisy) regression setting. Once again, our derived rate was not previously known even in the special case of average Lipschitz smoothness.

Matching lower bound (Theorem 6.1). We prove a lower bound, showing that all the results mentioned above are tight up to logarithmic factors in the realizable case, establishing the (nearly) minimax risk rate for average-smooth classes.

Illustrative comparisons (Section 7). Finally, we illustrate the extent to which the proposed smoothness notion is sharper than previously studied notions. We provide examples in which

the 'optimistic' average-Hölder constant is infinitely apart from both its 'pessimistic' worst-case counterpart, or even the average-Lipschitz ( β = 1 ) constant, exemplifying the substantial (possibly infinite) speed-ups in terms of learning rates.

1.1 Related work.

The sample complexities associated to distribution-free learning of (classic) Hölder classes is well covered in the literature, see for example the books by Györfi et al. [2002], Tsybakov [2008].

Previous notions of average smoothness include Bounded Variation (BV) [Appell et al., 2014] in dimensions one and higher [Kuipers and Niederreiter, 1974, Niederreiter and Talay, 2006]. Onedimensional BV has found learning-theoretic applications [Bartlett et al., 1997, Long, 2004, Anthony and Bartlett, 1999], but to our knowledge the higher-dimensional variants have not. Moreover, the positive results require µ to be uniformly distributed on a segment, and the aforementioned results break down for more general measures - especially if µ is not known to the learner.

Sobolev spaces, and the Sobolev-Slobodetskii norm in particular [Agranovich, 2015], bear some resemblance to our average Hölder smoothness. However, Ashlagi et al. [2021, Appendix I] demonstrate that from a learning-theoretic perspective this notion is inadequate for general (i.e., non-uniform or Lebesgue) measures, as it cannot be used to control sample complexity. Results for controlling bracketing in terms of various measures of average smoothness include Nickl and Pötscher [2007], who bound the bracketing numbers of Besov- and Sobolev-type classes and Malykhin [2010], who used the averaged modulus of continuity developed by Sendov and Popov [1988]; again, these are all defined under the Lebesgue measure. While it is easy to define these smoothness notions with respect to arbitrary distributions, we are not aware of any existing work to bound their corresponding sample complexity (or even their covering or bracketing numbers) in a distribution-independent manner. Moreover, the smoothness notion studied in this paper is defined over arbitrary metric spaces, whereas previous notions are typically restricted to Euclidean structures (or variants thereof). Despite of the considerable generality of our setting, we are able to provide tight bounds for all metric spaces alike, without requiring specialized analyses.

A seminal work on recovering functions with spatially inhomogeneous smoothness from noisy samples is Donoho and Johnstone [1998]. Arguably in the spirit of µ -dependent Hölder smoothness, some of the classic results on k -NN risk decay rates were refined by Chaudhuri and Dasgupta [2014] via an analysis that captures the interplay between the metric and the sampling distribution. Another related notion is that of Probabilistic Lipschitzness (PL) [Urner and Ben-David, 2013, Urner et al., 2013, Kpotufe et al., 2015], which seeks to relax a hard Lipschitz condition on the regression function. While PL is in the same spirit as our notion, one critical distinction from our work is that, while existing analyses of learning under PL have focused specifically on binary classification, our interest in the present work is learning real-valued functions.

As previously mentioned, the main feature setting this work apart from others studying regression under average smoothness is that our notion is defined with respect to a general, unknown measure µ . The notable exception is, of course, Ashlagi et al. [2021] - who introduced the framework of efficiently learning smooth-on-average functions with respect to an unknown distribution. Although extending their definition from Lipschitz to Hölder average smoothness was straightforward, optimal minimax rates are likely inaccessible via their techniques, which relied on empirical covering numbers. Estimating the magnitude of these random objects was a formidable challenge, and Ashlagi et al. were only able to do so to within an additive error decaying with sample size; this sampling noise appears to present an inherent obstruction to optimal rates. Thus, our results required a novel technique to overcome this obstruction, which we did by tightly controlling the bracketing entropy. Our Höldertype extension is a direct adaptation of the Pointwise Minimum Slope Extension (PMSE) developed for the Lipschitz special case by Ashlagi et al., which in turn is closely related to the one introduced by Oberman [2008].

2 Preliminaries.

Setting. Throughout the paper we consider functions f : Ω → [0 , 1] where (Ω , ρ ) is a metric space. We will consider a distribution D over Ω × [0 , 1] with marginal µ over Ω , such that (Ω , ρ, µ ) forms a metric probability space (namely, µ is supported on the Borel σ -algebra induced by ρ ). We associate

to any measurable function f : Ω → [0 , 1] its L 1 risk L D ( f ) := E ( X,Y ) ∼D | f ( X ) -Y | , and its empirical risk L S ( f ) := 1 n ∑ n i =1 | f ( X i ) -Y i | with respect to a sample S = ( X i , Y i ) n i =1 ∼ D n . More generally, we associate to any measurable function its L 1 norm ∥ f ∥ L 1 ( µ ) := E X ∼ µ | f ( X ) | , and given a sample ( X 1 , . . . , X n ) ∼ µ n we denote its L 1 norm with respect to the empirical measure ∥ f ∥ L 1 ( µ n ) := 1 n ∑ n i =1 | f ( X i ) | .

We say that a distribution D over Ω × [0 , 1] is realizable by a function class F ⊂ [0 , 1] Ω if there exists an f ∗ ∈ F such that L D ( f ∗ ) = 0 . Thus, f ∗ ( X ) = Y almost surely, where ( X,Y ) ∼ D .

̸

Metric notions. The diameter of A ⊂ Ω is diam( A ) := sup x,x ′ ∈ A ρ ( x, x ′ ) , and we denote by B ( x, r ) := { x ′ ∈ Ω : ρ ( x, x ′ ) ≤ r } the closed ball around x ∈ Ω of radius r > 0 . For t > 0 , A,B ⊂ Ω , we say that A is a t -cover of B if B ⊂ ⋃ a ∈ A B ( a, t ) , and define the t -covering number N B ( t ) to be the minimal cardinality of any t -cover of B . We say that A ⊂ B ⊂ Ω is a t -packing of B if ρ ( a, a ′ ) ≥ t for all a = a ′ ∈ A . We call V a t -net of B if it is a t -cover and a t -packing. The induced Voronoi partition of B with respect to a net V is its partitioning into subsets sharing the same nearest neighbor in V (with ties broken in some consistent arbitrary manner). A metric space (Ω , ρ ) is said to be doubling if there exists d ∈ N such that every r -ball in Ω is contained in the union of some d r/ 2 -balls. The doubling dimension is defined as min d ≥ 1 log 2 d where the minimum is taken over d satisfying the doubling property.

Bracketing. Given any two functions l, u : Ω → [0 , 1] , we say that f : Ω → [0 , 1] belongs to the bracket [ l, u ] if l ≤ f ≤ u . A set of brackets B is said to cover a function class F if any function in F belongs to some bracket in B . We say that [ l, u ] is a t -bracket with respect to a norm ∥ · ∥ if ∥ u -l ∥ ≤ t . The t -bracketing number N [ ] ( F , ∥ · ∥ , t ) is defined as the minimal cardinality of any set of t -brackets that covers F . The logarithm of this quantity is called the bracketing entropy .

Remark 2.1 (Covering vs. bracketing) . Having recalled two notions that quantify the 'size' of a normed function space ( F , ∥ · ∥ ) -namely, its covering and bracketing numbers - it is useful to note they are related through

though no converse inequality of this sort holds in general. On the other hand, the main advantage of using bracketing numbers for generalization bounds is that it suffices to bound the ambient bracketing numbers with respect to the distribution-specific metric, as opposed to the empirical covering numbers which are necessary to guarantee generalization [van der Vaart and Wellner, 1996, Section 2.1.1].

Strong and weak mean. For any non-negative random variable Z we define its weak mean by W [ Z ] := sup t> 0 t Pr[ Z ≥ t ] , and note that W [ Z ] ≤ E [ Z ] by Markov's inequality. In the special case where Z has finite support of size N ≥ 3 where each atom has mass 1 /N we have the reverse inequality E [ Z ] ≤ 2 log( N ) W [ Z ] [Ashlagi et al., 2021, Lemma 22].

2.1 Average smoothness.

Recall that f is called β -Hölder continuous if ∥ f ∥ H¨ ol β := sup x ∈ Ω Λ β f ( x ) < ∞ , with this quantity serving as its Hölder seminorm. In particular, when β = 1 these are exactly the Lipschitz functions equipped with the Lipschitz seminorm. For a metric probability space (Ω , ρ, µ ) , we consider the average β -slope to be the mean of Λ β f ( X ) where X ∼ µ . Namely, we define

Notably,

where each subsequent pair can be infinitely apart - as we demonstrate in Section 7. Having defined notions of averaged smoothness, we can further define their corresponding function spaces

We occasionally omit µ when it is clear from context. Note that H¨ ol β L (Ω) ⊂ H¨ ol β L (Ω , µ ) ⊂ ˜ H¨ ol β L (Ω , µ ) due to Eq. (2), where both containments are strict in general. The special case of β = 1 recovers the average-Lipschitz spaces Lip L (Ω) ⊂ Lip L (Ω , µ ) ⊂ ˜ Lip L (Ω , µ ) studied by Ashlagi et al. [2021].

3 Generalization bounds

Our first goal is to bound the bracketing entropy (namely, the logarithm of the bracketing number) of average-Hölder classes. We present this bound in full generality in terms of the underlying metric space, as captured by its covering number (see Corollary 3.5 for the typical scaling of covering numbers). As we will soon establish, this bound implies nearly-tight generalization guarantees in terms of the average smoothness constant.

Theorem 3.1. For any metric probability space (Ω , ρ, µ ) , any β ∈ (0 , 1] and any 0 < ϵ < L , it holds

Crucially, the bound above does not depend on µ , allowing us to obtain distribution-free generalization guarantees. We defer the proof of Theorem 3.1 to Appendix B.1. We start by showing that bounding the bracketing entropy implies a generalization bound in the realizable case:

Proposition 3.2. Suppose (Ω , ρ ) is a metric space, F ⊆ [0 , 1] Ω is a function class, and let D be a distribution over Ω × [0 , 1] which is realizable by F , with marginal µ over Ω . Then with probability at least 1 -δ over drawing a sample S ∼ D n it holds that for all f ∈ F :

Remark 3.3 (Constant is arbitrary) . In Proposition 3.2 and in what follows, the constant multiplying L S ( f ) is arbitrary, and can be replaced by (1 + γ ) for any γ > 0 at the expense of multiplying the remaining summands by γ -1 . In the next section we will provide a realizable regression algorithm that returns an approximate empirical risk minimizer f for which L S ( f ) ≈ 0 , thus this constant will not matter for our purposes.

We prove Proposition 3.2 in Appendix B.2. By combining Theorem 3.1 with Proposition 3.2 and setting α = ε/ 2 , we obtain the following realizable sample complexity result.

Theorem 3.4. For any metric space (Ω , ρ ) , any β ∈ (0 , 1] and any 0 < ϵ < L , let D be a distribution over Ω × [0 , 1] realizable by ˜ H¨ ol β L (Ω , µ ) . Then there exists N = N ( β, ε, δ ) ∈ N satisfying

such that as long as n ≥ N , with probability at least 1 -δ over drawing a sample S ∼ D n it holds that for all f ∈ ˜ H¨ ol β L (Ω , µ ) :

The same claim holds for the smaller class H¨ ol β L (Ω , µ ) .

Corollary 3.5 (Doubling metrics) . In most cases of interest, (Ω , ρ ) is a doubling metric space of some dimension d , 3 e.g. when Ω is a subset of R d (or more generally a d -dimensional Banach space). For d -dimensional doubling spaces of finite diameter we have N Ω ( ε ) ≲ ( 1 ε ) d [Gottlieb et al., 2016, Lemma 2.1], which, plugged into Theorem 3.4, yields the simplified sample complexity bound

up to a constant which depends (exponentially) on d , but is independent of L, n .

Remark 3.6 (Tightness) . The bounds in Theorem 3.1 and Theorem 3.4 are both tight up to logarithmic factors, as we will prove in Section 6.

4 Realizable learning algorithm

Recall that without knowing µ , the underlying distribution over Ω , we cannot know for sure whether a function f belongs to H¨ ol β L (Ω , µ ) (except for the trivial case f ∈ H¨ ol β L (Ω) ). This gives rise to the challenge of designing a fully empirical algorithm - since standard empirical risk minimization is not possible. To that end, we provide the following algorithmic result with optimal guarantees (up to logarithmic factors).

Theorem 4.1. For any metric space (Ω , ρ ) , any β ∈ (0 , 1] and any 0 < ϵ < L , let D be a distribution over Ω × [0 , 1] realizable by H¨ ol β L (Ω , µ ) . Then there exists a polynomial time learning algorithm A , which, given a sample S ∼ D n of size n ≥ N for some N = N ( β, ε, δ ) ∈ N satisfying

constructs a hypothesis f = A ( S ) such that L D ( f ) ≤ ε with probability at least 1 -δ .

Remark 4.2 (Doubling metrics) . As mentioned in Corollary 3.5, in most cases of interest we have N Ω ( ε ) ≲ ( 1 ε ) d . In that case, the algorithm above has sample complexity

up to a constant which depends (exponentially) on d , but is independent of L, n .

Remark 4.3 (Computational complexity) . The algorithm constructed in Theorem 4.1 involves a one-time preprocessing step after which f ( x ) can be evaluated at any given x ∈ Ω in O ( n 2 ) time. We note that the computation at inference time matches that of (classic) Lipschitz/Hölder regression (e.g. Gottlieb et al., 2017). Furthermore, the computational complexity of the preprocessing step is similar to that in Ashlagi et al. [2021, Theorem 7] for the average Lipschitz case, where it is shown to run in time ˜ O ( n 2 ) . The complexity analysis of our prepossessing step is entirely analogous to theirs, and we forgo repeating it here.

We will now outline the proof of Theorem 4.1, which appears in Appendix B.3. The key idea is to analyze a natural fully-empirical quantity that will serve as an estimator of the true unknown average

or equivalently or equivalently

smoothness. To that end, given a sample S = ( X i , Y i ) n i =1 ∼ D n and a function f : Ω → [0 , 1] , consider the following quantity which can be established directly from the data:

̸

Namely, this is the empirical average smoothness with respect to the sampled points. It would suit us well if the empirical average smoothness of a function did not greatly exceed its true average smoothness, with high probability. The fact something like this turns out to be true is somewhat surprising and may be of independent interest:

Proposition B.1. (Informal) Let f ∗ : Ω → [0 , 1] . Then with high probability ̂ Λ β f ∗ ≲ Λ β f ∗ .

The proposition above implies that restricting to the sample, and letting ̂ f ( X i ) := Y i yields a function over { X 1 , . . . , X n } which is empirically average-smooth over the sample (with high probability). We then turn to show that any such function can be approximately extended to the whole space, in a way that guarantees its average smoothness with respect to the underlying distribution .

Proposition B.3. (Informal) Let ̂ f : { X 1 , . . . , X n } → [0 , 1] where ( X i ) n i =1 ∼ µ n . Then it is possible to construct f : Ω → [0 , 1] such that with high probability f ( X i ) ≈ ̂ f ( X i ) for all i ∈ [ N ] , and Λ β f ( µ ) ≲ ̂ Λ β ̂ f .

We will now sketch the procedure described in Proposition B.3, which serves as the main challenge in proving Theorem 4.1. Roughly speaking, the algorithm sorts the sampled points with respect to their relative slope to one another. Then, it discards a fraction of the sampled points with largest relative slope, which can be thought of as 'outliers'. Then, the algorithm proceeds to extend the function in a smooth fashion among the remaining 'well-behaved' samples. A careful probabilistic analysis shows that disregarding just the right amount of samples induces small error, while being average-smooth with high probability.

Overall this procedure yields a function f : Ω → [0 , 1] which is an approximate empirical-minimizer (since f ( X i ) ≈ ̂ f ( X i ) = Y i ), while guaranteed to be averagely-smooth with respect to the unknown distribution. Thus we can apply the uniform convergence of Theorem 3.4, proving Theorem 4.1.

5 Agnostic learning algorithm

Noticeably, up to this point, both the uniform convergence result we derived (Theorem 3.4) as well as the algorithmic result (Theorem 4.1) are tailored for the realizable regression setting. Inspired by a recent result of Hopkins et al. [2022] that showed a reduction from agnostic learning to realizable learning, we provide an algorithm for agnostic (i.e. noisy) regression of average-smooth functions. It is worth noting that the following algorithm does not require any prior assumption on the noise model, unlike many nonparametric regression methods, due to our distribution free analysis.

Theorem 5.1. There exists a learning algorithm A such that for any metric space (Ω , ρ ) , any β ∈ (0 , 1] , 0 < ϵ < L , and any distribution D over Ω × [0 , 1] , given a sample S ∼ D n of size n ≥ N for some N = N ( β, ε, δ ) satisfying

the algorithm constructs a hypothesis f = A ( S ) such that L D ( f ) ≤ inf f ∗ ∈ H¨ ol β L (Ω ,µ ) L D ( f ∗ ) + ϵ with probability at least 1 -δ .

Remark 5.2 (Doubling metrics) . As mentioned in Corollary 3.5, in most cases of interest we have N Ω ( ε ) ≲ ( 1 ε ) d . In that case, the algorithm above has sample complexity

or equivalently

up to a constant which depends (exponentially) on d , but is independent of L, n .

Though our agnostic algorithm is similar in spirit to that obtained by the reduction of Hopkins et al. [2022], our analysis is self-contained and crucially relies on the bracketing bound given by Theorem 3.1, as well as analyzing the empirical smoothness estimator as provided by Proposition B.1. We also note that unlike our algorithm for realizable learning, the agnostic algorithm is not computationally efficient. This seems to be inherent for such reductions, and we do not know whether this blow-up in running time can be avoided or not.

We will now describe the proof of Theorem 5.1 which appears in Appendix B.4. Given a sample S of size n , consider dividing it into two sub-samples S 1 ∪ S 2 = S of size n/ 2 each. We first use S 1 in order to construct an empirical ϵ -net h 1 , . . . , h N : S 1 → [0 , 1] , namely a set of functions which are sufficiently empirically smooth over the sample, yet far away enough from one another when averaged over the sample. Recalling that bracketing numbers upper bound covering numbers (Eq. (1)), and since Theorem 3.1 holds true for every measure (in particular for the empirical measure), we can bound log N ≲ N Ω (( ϵ/L ) 1 /β ) . Moreover, using Proposition B.1 we know that f ∗ := arg min f ∈ H¨ ol β L (Ω ,µ ) L D ( f ) is likely to be ˜ O ( L ) average-smooth over S 1 , so there must exist some h j with ϵ excess empirical loss (since f ∗ is in the class we are ϵ -covering). Thus running the realizable algorithm of Theorem 4.1 over all { h 1 , . . . , h N } , producing f 1 , . . . , f N : Ω → [0 , 1] , yields at least one function which has both small excess empirical error, while being smooth with respect to the underlying distribution. Finally, running ERM over { f 1 , . . . , f N } with respect to the fresh sample S 2 reveals such a good candidate function within log( N )+log(1 /δ ) ϵ 2 samples by applying standard uniform convergence for finite classes (i.e. Hoeffding's inequality with the union bound).

6 Lower bound

We now turn to show that the bounds proved in Theorem 3.1, Theorem 3.4 and Theorem 4.1 are all tight up to logarithmic factors. In fact, since the bracketing entropy bound of Theorem 3.1 implies the generalization bound of Theorem 3.4 and the latter implies the sample complexity in Theorem 4.1, it is enough to show that the latter is nearly optimal.

Theorem 6.1. For any β ∈ (0 , 1] , ε ∈ (0 , 1) any metric space (Ω , ρ ) and L ≥ 8 diam(Ω) , there exists a distribution D over Ω × [0 , 1] which is realizable by H¨ ol β L (Ω) such that any learning algorithm that produces f = A ( S ) with L D ( f ) ≤ ε with constant probability, must have sample complexity

Remark 6.2 (Typical case) . In most cases of interest it holds that N Ω ( ε ) ≳ ( 1 ε ) d for some constant d , e.g. when Ω is a subset of non-empty interior in R d (or more generally in any d -dimensional Banach space). 4 That being the case, Theorem 6.1 yields the simplified sample complexity lower bound of

Equivalently, we obtain an excess risk lower bound of

We will now provide a proof sketch of Theorem 6.1, while the full proof appears in Appendix B.5. Suppose K ⊂ Ω is a ( ε/L ) 1 /β -net of most of Ω , yet x 0 ∈ Ω is some 'isolated' point at constant

distance away from K (we show that such x 0 , K always exist). Let µ be the measure that assigns 1 -ε probability mass to x 0 , while the rest of the probability mass is distributed uniformly over K . Now consider a (random) function that independently assigns either 0 or 1 to each point in K uniformly, and is constant over x 0 . Since points in K are ( ε/L ) 1 /β away from one another, the local β -slope at each point in K is roughly 1 / (( ε/L ) 1 /β ) β = L/ε , while the slope at x 0 is small since it is far enough from other points. Averaging over the space with respect to µ , we see that the function is µ ( K ) · L/ε = L average-Hölder. Now, we imitate the standard lower bound proof for VC classes over K : Since any point in K is sampled with probability ε/ | K | , any learning algorithm with much fewer than | K | /ε ≈ N Ω (( ε/L ) 1 /β ) /ε examples will guess wrong a large portion of K , suffering L 1 -loss of at least order of µ ( K ) = ε .

7 Illustrative examples

Having established the control that average-Hölder smoothness has on generalization, we illustrate the vast possible gap between the average smoothness and it's 'worst-case' classic counterpart. Indeed, in the examples we provide, the gap is infinite. Moreover, we also show that classes of average-Hölder smoothness are significantly richer than the previously studied average-Lipschitz, motivating the more general Hölder framework considered in this work. Finally, it is illuminating to notice that both claims to follow actually consist of the same simple function f ( x ) = 1 [ x > 1 2 ] though with respect to different distributions, emphasizing the crucial role of the underlying distribution in terms of establishing the function classes.

Claim 7.1. For any L > 0 , β ∈ (0 , 1) , there exist f : Ω → [0 , 1] and a probability measure µ such that

f is average-Hölder: f ∈ H¨ ol β L (Ω , µ ) .

f is not Hölder with any finite Hölder constant: For all M > 0 : f / ∈ H¨ ol β M (Ω) .

f is not (even) weakly-average-Lipschitz with any finite modulus: For all M > 0 : f / ∈ ˜ Lip M (Ω , µ ) .

Claim 7.2. For any L > 0 , β ∈ (0 , 1) , there exist f : Ω → [0 , 1] and a probability measure µ such that

f is weakly-average-Hölder: f ∈ ˜ H¨ ol β L (Ω , µ ) .

f is not strongly-average-Hölder with any finite modulus: For all M > 0 : f / ∈ H¨ ol β M (Ω) .

f is not (even) weakly-average-Lipschitz with any finite modulus: For all M > 0 : f / ∈ ˜ Lip M (Ω , µ ) .

We prove both of the claims above in Appendix B.6.

8 Discussion

In this work, we have defined a notion of an average-Hölder smoothness, extending the averageLipschitz one introduced by Ashlagi et al. [2021]. Using proof techniques based on bracketing numbers, we have established the minimax rate for average-smoothness classes in the realizable setting with respect to the L 1 risk up to logarithmic factors, and have provided a nontrivial learning algorithm that attains this nearly-optimal learning rate. Moreover, we have also provided yet another learning algorithm for the agnostic setting. All of these results improve upon previously known rates even in the special case of average-Lipschitz classes.

A few notes are in order. First, the choice of focusing on L 1 risk as opposed to general L p losses is merely a matter of conciseness, as to avoid introducing additional parameters. Indeed, the only place throughout the proofs which we use the L 1 loss is in the proof of Proposition 3.2, where we show that the loss-class L F := { x ↦→| f ( x ) -f ∗ ( x ) | : f ∈ F} satisfies

It is easy to show via essentially the same proof that for any p ∈ [1 , ∞ ) , the L p -composed lossclass satisfies N [ ] ( L F , L 1 ( µ ) , α ) ≤ N [ ] ( F , L 1 ( µ ) , α 1 /p ) , and the remaining proofs can be invoked verbatim. This yields a realizable sample complexity (in the typical, d -dimensional case) of order N = ˜ O ( L d/pβ ε ( d + pβ ) /pβ ) , or equivalently L p -risk decay rate of L D ( f ) = ˜ O ( L d/ ( d + pβ ) n pβ/ ( d + pβ ) ) which are also easily translatable to their corresponding agnostic rates.

Focusing again on L 1 minimax rates of average-Hölder classes, it is interesting to compare them to the minimax rates of 'classic' (i.e., worst-case) Hölder classes. Schreuder [2020] has shown the minimax risk to be of order n -β/d , whereas we showed the average-smooth case has the slightly worse rate of n -β/ ( d + β ) (which cannot be improved, due to our matching lower bound). However, comparing the rates alone is rather misleading, since both risks are multiplied by a factor depending on their corresponding Hölder constant, which can be considerably smaller in the average-case result. Still, it is interesting to note that in the asymptotic regime there is a marginal advantage in case the learned function is worst-case Hölder, as opposed to Hölder on average.

Our work leaves open several questions. A relatively straightforward one is to compute the minimax rates and construct an optimal algorithm for the classification setting, which is not addressed by this paper. Moreover, there is a slight mismatch between our established upper and lower bounds in the agnostic setting, ranging between ˜ O ( n -β/ ( d +2 β ) ) and Ω( n -β/ ( d + β ) ) . Closing this gap is an interesting problem which we leave for future work.

References

Mikhail S. Agranovich. Sobolev spaces, their generalizations and elliptic problems in smooth and Lipschitz domains . Springer Monographs in Mathematics. Springer, Cham, 2015. ISBN 978-3319-14647-8; 978-3-319-14648-5. doi: 10.1007/978-3-319-14648-5. URL https://doi.org/ 10.1007/978-3-319-14648-5 . Revised translation of the 2013 Russian original.

Martin Anthony and Peter L. Bartlett. Neural Network Learning: Theoretical Foundations . Cambridge University Press, Cambridge, 1999. ISBN 0-521-57353-X. doi: 10.1017/CBO9780511624216. URL http://dx.doi.org/10.1017/CBO9780511624216 .

Jürgen Appell, Józef Bana´ s, and Nelson Merentes. Bounded variation and around , volume 17 of De Gruyter Series in Nonlinear Analysis and Applications . De Gruyter, Berlin, 2014. ISBN 978-3-11-026507-1; 978-3-11-026511-8.

Yair Ashlagi, Lee-Ad Gottlieb, and Aryeh Kontorovich. Functions with average smoothness: structure, algorithms, and learning. In Conference on Learning Theory , pages 186-236. PMLR, 2021.

Peter L. Bartlett, Sanjeev R. Kulkarni, and S. E. Posner. Covering numbers for real-valued function classes. IEEE Trans. Information Theory , 43(5):1721-1724, 1997. doi: 10.1109/18.623181. URL https://doi.org/10.1109/18.623181 .

Kamalika Chaudhuri and Sanjoy Dasgupta. Rates of convergence for nearest neighbor classification. In NIPS , 2014.

David L. Donoho and Iain M. Johnstone. Minimax estimation via wavelet shrinkage. Ann. Statist. , 26 (3):879-921, 1998. ISSN 0090-5364. doi: 10.1214/aos/1024691081. URL https://doi.org/ 10.1214/aos/1024691081 .

Evarist Giné and Richard Nickl. Mathematical foundations of infinite-dimensional statistical models . Cambridge university press, 2021.

Lee-Ad Gottlieb, Aryeh Kontorovich, and Robert Krauthgamer. Adaptive metric dimensionality reduction. Theoretical Computer Science , 620:105-118, 2016.

Lee-Ad Gottlieb, Aryeh Kontorovich, and Robert Krauthgamer. Efficient regression in metric spaces via approximate lipschitz extension. IEEE Transactions on Information Theory , 63(8):4838-4849, 2017. doi: 10.1109/TIT.2017.2713820.

László Györfi, Michael Kohler, Adam Krzy˙ zak, and Harro Walk. A distribution-free theory of nonparametric regression . Springer Series in Statistics. Springer-Verlag, New York, 2002. ISBN 0-387-95441-4. doi: 10.1007/b97848. URL http://dx.doi.org/10.1007/b97848 .

Max Hopkins, Daniel M Kane, Shachar Lovett, and Gaurav Mahajan. Realizable learning is all you need. In Conference on Learning Theory , pages 3015-3069. PMLR, 2022.

Michael J Kearns and Umesh Vazirani. An introduction to computational learning theory . MIT press, 1994.

Samory Kpotufe, Ruth Urner, and Shai Ben-David. Hierarchical label queries with data-dependent partitions. In Peter Grünwald, Elad Hazan, and Satyen Kale, editors, Proceedings of The 28th Conference on Learning Theory, COLT 2015, Paris, France, July 3-6, 2015 , volume 40 of JMLR Workshop and Conference Proceedings , pages 1176-1189. JMLR.org, 2015. URL http:// proceedings.mlr.press/v40/Kpotufe15.html .

Kuipers and H. Niederreiter. Uniform distribution of sequences . Wiley-Interscience [John Wiley &Sons], New York-London-Sydney, 1974. Pure and Applied Mathematics.

Philip M. Long. Efficient algorithms for learning functions with bounded variation. Inf. Comput. , 188 (1):99-115, 2004. doi: 10.1016/S0890-5401(03)00164-0. URL https://doi.org/10.1016/ S0890-5401(03)00164-0 .

Yu. V. Malykhin. Averaged modulus of continuity and bracket compactness. Mat. Zametki , 87(3): 468-471, 2010. ISSN 0025-567X. doi: 10.1134/S0001434610030181. URL https://doi.org/ 10.1134/S0001434610030181 .

Richard Nickl and Benedikt M. Pötscher. Bracketing metric entropy rates and empirical central limit theorems for function classes of Besov- and Sobolev-type. J. Theoret. Probab. , 20(2):177-199, 2007. ISSN 0894-9840. doi: 10.1007/s10959-007-0058-1. URL https://doi.org/10.1007/ s10959-007-0058-1 .

Harald Niederreiter and Denis Talay, editors. Monte Carlo and quasi-Monte Carlo methods 2004 , 2006. Springer-Verlag, Berlin. ISBN 978-3-540-25541-3; 3-540-25541-9. doi: 10.1007/3-540-31186-6. URL https://doi.org/10.1007/3-540-31186-6 .

Adam M. Oberman. An explicit solution of the Lipschitz extension problem. Proc. Amer. Math. Soc. , 136(12):4329-4338, 2008. ISSN 0002-9939. doi: 10.1090/S0002-9939-08-09457-4. URL https://doi.org/10.1090/S0002-9939-08-09457-4 .

Nicolas Schreuder. Bounding the expectation of the supremum of empirical processes indexed by hölder classes. Mathematical Methods of Statistics , 29(1):76-86, 2020.

Blagovest Sendov and Vasil A. Popov. The averaged moduli of smoothness . Pure and Applied Mathematics (New York). John Wiley & Sons, Ltd., Chichester, 1988. ISBN 0-471-91952-7. Applications in numerical methods and approximation, A Wiley-Interscience Publication.

Alexandre B. Tsybakov. Introduction to Nonparametric Estimation . Springer Publishing Company, Incorporated, 1st edition, 2008. ISBN 0387790519.

Ruth Urner and Shai Ben-David. Probabilistic Lipschitzness: A niceness assumption for deterministic labels. In Learning Faster from Easy Data WorkshopNIPS , 2013.

Ruth Urner, Sharon Wullf, and Shai Ben-David. PLAL: Cluster-based active learning. In Conference on Learning Theory , 2013.

Aad W. van der Vaart and Jon A. Wellner. Weak Convergence and Empirical Processes . Springer, 1996.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

A Minimal β -slope Hölder extension

In this section we describe a procedure that extends Hölder functions in an optimally smoothest manner at every point, as it will serve as a crucial ingredient in our proofs. That is, given a subset of a metric space A ⊂ Ω and a function f : Ω → [0 , 1] , it produces F A : Ω → [0 , 1] such that

It extends f | A : F A | A = f | A .

For any ˜ F : Ω → [0 , 1] that extends f | A , it holds that Λ β F A ( x ) ≤ Λ β ˜ F ( x ) for all x ∈ Ω .

Such a procedure was described for Lipschitz extensions (namely when β = 1 ) in Ashlagi et al. [2021]. The purpose of this section is to generalize this procedure to any Hölder exponent.

Throughout this section we fix β ∈ (0 , 1] , ∅ ̸ = A ⊂ Ω and f : Ω → [0 , 1] , and will always assume the following.

Assumption A.1. ∥ f | A ∥ H¨ ol β < ∞ and diam( A ) < ∞ .

Keeping in mind that the case we are really interested in is when A is finite (i.e. a sample), the conditions above are trivially satisfied. Nonetheless, everything we will present continues to hold in this more general setting. For u, v ∈ A we introduce the following notation:

Definition A.2. We define the β -pointwise minimal slope extension ( β -PMSE) to be the function F A : Ω → R satisfying

In the degenerate case in which f ( u ) = f ( v ) for all u, v ∈ A , define F A ( x ) := f ( u ) for some (and hence any) u ∈ A .

Theorem A.3. Let ∅ ̸ = A ⊂ Ω , f : Ω → [0 , 1] , such that Assumption A.1 holds. Then F A : Ω → [0 , 1] is well defined, and satisfies for any x ∈ Ω : Λ β F A ( x ) ≤ Λ β f ( x ) . Furthermore, if A is finite, then F A ( x ) can be computed for any x ∈ Ω within O ( | A | 2 ) arithmetic operations.

Remark A.4. When R x ( · , · ) has a unique maximizer ( u ∗ x , v ∗ x ) ∈ A × A , the definition of F A simplifies to

We conclude that under Assumption A.1, we can assume without loss of generality that for each x ∈ Ω there is such a unique maximizer (since the function is well defined, thus does not depend on the choice of the maximizer). Furthermore, this readily shows that when A is finite, we can compute F A ( x ) for any x ∈ Ω within O ( | A | 2 ) arithmetic operations - simply by finding this maximizer.

Proof. (of Theorem A.3)

We will assume that there exist u, v ∈ A such that f ( u ) = f ( v ) , since the degenerate (constant extension) case is trivial to verify. This assumption implies that R ∗ x > 0 . It is also easy to verify that sup x ∈ Ω R ∗ x < ∞ ⇐⇒ ∥ f ∥ H¨ ol β < ∞ .

̸

Lemma A.5. F A is well defined. Namely, under Assumption A.1 the limit lim ε → 0 + Φ x ( ε ) ∈ [0 , 1] exists.

Proof. Fix x ∈ Ω (we will omit the x subscripts from now on). Let ε < R ∗ / 2 , ( u, v ) , ( u ′ , v ′ ) ∈ W ( ε ) . Note that R ( u, v ) > 0 and that F ( u, v ) = f ( v ) -R ( u, v ) ρ ( x, v ) β . Hence

and the same clearly holds if we replace ( u, v ) by ( u ′ , v ′ ) . Assume without loss of generality that F ( u, v ) ≤ F ( u ′ , v ′ ) , hence f ( u ) ≤ F ( u, v ) ≤ F ( u ′ , v ′ ) ≤ f ( v ′ ) . We get

We conclude that if diam( A ) < ∞ then lim ε → 0 + Φ x ( ε ) indeed exists.

̸

It remains to prove the optimality of the β -slope. Throughout the proof we will denote for any u = v ∈ Ω :

and for any point x ∈ Ω , subset B ⊂ Ω and function g : Ω → [0 , 1] we let

The proof is split into three claims.

Let x ∈ Ω \ A , and let ( u ∗ , v ∗ ) ∈ A × A be its associated maximizer of R x . Recall Eq. (4) from which we can deduce that F A ( u ∗ ) ≤ F A ( x ) ≤ F A ( v ∗ ) . Also note that a simple rearrangement based on Eq. (3) (and the fact that f and F A agree on A ) shows that S ( u ∗ , x ) = R x ( u ∗ , v ∗ ) = S ( x, v ∗ ) . Furthermore, we claim that Λ β F A ( x, A ) := sup y ∈ A \{ x } S ( x, y ) = S ( x, u ∗ ) . If this were not true then we would have S ( x, y ) > S ( x, u ∗ ) = S ( x, v ∗ ) for some y ∈ A \ { x, u ∗ , v ∗ } . Using the mediant inequality, if f ( y ) ≥ f ( x ) this implies

while if f ( y ) < f ( x ) then

both contradicting the maximizing property of ( u ∗ , v ∗ ) - so indeed Λ β F A ( x, A ) = S ( x, u ∗ ) = S ( x, v ∗ ) . In particular, we see that if F A ( x ) ≥ f ( x ) then

while if F A ( x ) < f ( x ) then

proving Claim I in either case.

Claim II. ∀ x ∈ Ω \ A : Λ β F A ( x, Ω \ A ) ≤ Λ β F A ( x, A ) , in particular Λ β F A ( x, Ω) = Λ β F A ( x, A ) . It suffices to show that for any x, y ∈ Ω \ A :

since taking the supremum of the left hand side over y ∈ Ω \ A shows the claim. Let ( u ∗ x , v ∗ x ) , ( u ∗ y , v ∗ y ) the associated maximizers of R x , R y respectively, and note that by definition we have

We assume without loss of generality that Λ β F A ( x, A ) ≤ Λ β F A ( y, A ) , and recall that by Eq. (4) we can deduce that F A ( u ∗ x ) ≤ F A ( x ) ≤ F A ( v ∗ x ) and F A ( u ∗ y ) ≤ F A ( y ) ≤ F A ( v ∗ y ) . Now suppose by contradiction that S ( x, y ) > Λ β F A ( x, A ) . If F A ( x ) ≤ F A ( y ) then

thus S ( x, v ∗ y ) = | F A ( x ) -F A ( v ∗ y ) | ρ ( x,v ∗ y ) β > Λ β F A ( x, A ) which contradicts Eq. (5). On the other hand, if F A ( x ) > F A ( y ) then

thus S ( x, u ∗ y ) = | F A ( x ) -F A ( u ∗ y ) | ρ ( x,u ∗ y ) β > Λ β F A ( x, A ) which contradicts Eq. (5), and proves claim Claim II.

Let x ∈ A . Assume towards contradiction that there exists y / ∈ A such that

We denote by ( u ∗ y , v ∗ y ) ∈ A × A the maximizer of R y ( · , · ) . Recall that since x ∈ A , in the proof of Claim I we showed that S ( x, y ) ≤ S ( y, u ∗ y ) = S ( y, v ∗ y ) . If F A ( x ) ≤ F A ( y ) ≤ F A ( v ∗ y ) then

while on the other hand if F A ( x ) > F A ( y ) ≥ F A ( u ∗ y ) then

where in both calculations we used the mediant inequality. Both inequalities above contradict the definition of Λ β F A ( x, A ) , thus proving Claim III.

Combining the ingredients. We are now ready to finish the proof. For x ∈ Ω , if x ∈ A then Claim III provides the desired inequality. Otherwise, if x ∈ Ω \ A then

B Proofs

B.1 Proof of Theorem 3.1

We start by stating a strengthened version of the triangle inequality (also known as the 'snowflake' triangle inequality) which we will use later on. For any β ∈ (0 , 1] , x = y, z ∈ Ω :

̸

Indeed, this follows from

Let 0 < ε < 1 4 , denote K := ⌈ log 2 (1 /ε ) ⌉ , ε ′ := 1 ( K +1)2 K and note that

Let N = { x 1 , . . . , x | N | } be a ( ε ′ 32 L ) 1 /β -net of Ω of size | N | = N Ω ( ( ε ′ 32 L ) 1 /β ) , and let Π = { C 1 , . . . , C | N | } be its induced Voronoi partition. We define B = { [ l j , u j ] } j ∈ J ⊂ [0 , 1] Ω × [0 , 1] Ω to be the pairs of functions constructed as follows:

l, u are both constant over every cell C i ∈ Π , and map each cell to a value in { 0 , ε ′ 2 , ε ′ , 3 ε ′ 2 , . . . , 1 } .

Choose some cells S 1 ⊂ Π such that µ ( ⋃ C i ∈ S 1 C i ) ≤ ε ′ and set for any C i ∈ S 1 : l | C i = 0 , u | C i = 1 .

For m = 2 , . . . , K choose some 'unchosen' cells S m ⊂ Π \ ⋃ j<m S j such that µ ( ⋃ C i ∈ S m C i ) ≤ 2 m -1 ε ′ and set for any C i ∈ S m : l | C i ∈ { 0 , 1 2 m , 2 2 m , . . . , 2 m -2 2 m } , , u | C i = l + 1 2 m -1 .

In the 'remaining' cells S K +1 := Π \ ⋃ j ≤ K S j set for any C i ∈ S K +1 :

Notice that for any [ l, u ] ∈ B we have

Furthermore, we can bound |B| by noticing that any such l is defined by its values over | N | cells who all belong to { 0 , ε ′ 2 , ε ′ , . . . , 1 } , and that once l is fixed then any associated u has at most K +1 possible values over each cell since it equals l + 1 2 m -1 for some m ∈ [ K +1] . Thus

where the last inequality uses Eq. (7) and definition of K . In order to finish the proof, in remains to show that B indeed cover ˜ H¨ ol β L (Ω , µ ) as brackets. Namely, that for any f ∈ ˜ H¨ ol β L (Ω , µ ) there exist [ l, u ] ∈ B such that l ≤ f ≤ u . To that end, let f ∈ ˜ H¨ ol β L (Ω , µ ) . Denote

and notice that ⋃ { C i ∈ S f 1 } ⊆ { x : Λ β f ( x ) ≥ L/ε ′ } = ⇒ µ ( ⋃ { C i ∈ S f 1 } ) ≤ ε ′ . Hence we can pick [ l, u ] ∈ B such that ( l | C i , u | C i ) ≡ (0 , 1) for any C i ∈ S f 1 (serving as S 1 in their construction). Clearly any such l, u bound f over these cells. Furthermore, for m = 2 , . . . , K we denote

and notice that ⋃ { C i ∈ S f m } ⊆ { x : Λ β f ( x ) ≥ L/ (2 m -1 ε ′ ) } = ⇒ µ ( ⋃ { C i ∈ S f m } ) ≤ 2 m -1 ε ′ . Consequently we can let S f m serve as S m in the construction of [ l, u ] ∈ B , assuming we will show such a choice can serve as a bracket of f over such cells. Indeed, for any x ∈ C i we have

which by the triangle inequality shows in particular that for any x, y ∈ C i :

So clearly there exists α i ∈ { 0 , 1 2 m , 2 2 m , . . . , 2 m -2 2 m } such that α i ≤ f | C i ≤ α i + 1 2 m -1 , and by setting l | C i , u | C i = ( α i , α i + 1 2 m -1 ) for any C i ∈ S f m we ensure the bracketing property. Finally, for any of the remaining cells S f K +1 := Π \ ⋃ j ≤ K S f j we get by construction that ∃ z i ∈ C i : Λ β f ( z i ) < L 2 K ε ′ (otherwise they would satisfy the condition for some previously constructed S f m ). Hence for any x ∈ C i we have

which by the triangle inequality shows that for any x, y ∈ C i :

So as before, there clearly exists α i ∈ { 0 , 1 2 K +1 , 2 2 K +1 , . . . , 2 K +1 -2 2 K +1 } such that α i ≤ f | C i ≤ α i + 1 2 K , and by setting l | C i , u | C i = ( α i , α i + 1 2 K ) for any C i ∈ S f m we ensure the bracketing property over all of Ω , which finishes the proof.

B.2 Proof of Proposition 3.2

Recalling that the realizability assumption ensures a 'perfect' predictor f ∗ ∈ F , we start by introducing the loss class L F ⊂ [0 , 1] Ω :

Fix α > 0 . We observe that L F is no larger than F in terms of bracketing entropy, namely

Indeed, suppose we are given an α -bracketing of F denoted by B α , and denote for any f ∈ F by [ l f , u f ] ∈ B α its associated bracket. Then any ℓ f ∈ L F is inside the bracket [ l ℓ f , u ℓ f ] where

It is straightforward to verify that ∥ u ℓ f -l ℓ f ∥ L 1 ( µ ) ≤ ∥ u f -l f ∥ L 1 ( µ ) ≤ α , and clearly the set of all such brackets is of size at most |B α | , yielding Eq. (8).

Now notice that for any f ∈ F :

In order to bound the right hand side, fix some l ℓ f , and note that Var( l ℓ f ) ≤ ∥ l 2 ℓ f ∥ L 1 ( µ ) ≤ ∥ l ℓ f ∥ L 1 ( µ ) , since l ℓ f ( x ) ∈ [0 , 1] . Thus by Bernstein's inequality and the AM-GM inequality we get that with probability at least 1 -γ :

Setting γ = δ/ N [ ] ( F , L 1 ( µ ) , α ) and taking a union bound over l ℓ f whose number is bounded due to Eq. (8), we see that with probability 1 -δ :

Plugging this back into Eq. (9), and minimizing over α > 0 finishes the proof.

B.3 Proof of Theorem 4.1

Proposition B.1. Let f : Ω → [0 , 1] . Then with probability at least 1 -δ/ 2 over drawing a sample it holds that

Corollary B.2. If D is realizable by H¨ ol β L (Ω , µ ) , then for f ∗ : Ω → [0 , 1] such that L D ( f ∗ ) = 0 it holds with probability at least 1 -δ/ 2 : ̂ Λ β f ∗ ≤ 5 log 2 (4 n/δ ) L . Hence, ̂ f ( X i ) := f ∗ ( X i ) = Y i satisfies L S ( ̂ f ) = 0 and ̂ Λ β ̂ f ≤ 5 log 2 (4 n/δ ) L .

Proof. (of Proposition B.1) Fix f : Ω → [0 , 1] . Given a sample ( X i ) n i =1 ∼ µ n which induces an empirical measure µ n , we get

̸

where the last inequality follows from the reversed strong-weak mean inequality for uniform measures. We will now show that with high probability W X ∼ µ n [Λ β f ( X )] ≲ W X ∼ µ [Λ β f ( X )] = ˜ Λ β f . To that end, we denote for any t > 0 : M f ( t ) := { x : Λ β f ( x ) ≥ t } ⊂ Ω , let K := ˜ Λ β f ( µ ) , N := ⌈ 2 log(4 n/δ ) log log(4 n/δ ) ⌉ and note that

We will bound all three summands above. We easily bound the first term by

For the second term, denote for any t > 0 by M + f ( t ) ⊃ M f ( t ) a containing set for which 1 n ≤ µ ( M + f ( t )) ≤ µ ( M f ( t ))+ 1 n . We can always assume without loss of generality that such a set exists. 5 By the multiplicative Chernoff bound we have for any t, α > 0 :

hence by the union bound we get with probability at least 1 -Ne α (1+ α ) 1+ α :

Letting α = log(4 n/δ ) -1 , by our choice of N = ⌈ 2 log(4 n/δ ) log log(4 n/δ ) ⌉ we get that with probability at least 1 -δ/ 4 :

In order to bound the last term in Eq. (11), we observe that the empirical measure satisfies for any A ⊂ Ω : µ n ( A ) < 1 n ⇐⇒ µ n ( A ) = 0 , and that M f ( s ) ⊂ M f ( t ) for s > t . Furthermore, by definition of K = ˜ Λ β f ( µ ) we have µ ( M f ( t )) ≤ K t , hence by Markov's inequality

̸

̸

̸

For t := 2 N K yields Pr S [ sup s ≥ 2 n K µ n ( M f ( s )) = 0 ] ≤ n 2 N ≤ δ 4 . Combining this with Eq. (12), Eq. (13) and plugging back into Eq. (11), we get that with probability at least 1 -δ/ 2 :

Recalling Eq. (10), we get overall that

Simplifying the expression above finishes the proof.

Proposition B.3. Under the same setting, for any γ > 0 there exists an algorithm that given a sample S ∼ D n and any function ̂ f : S → [0 , 1] , provided that n ≥ N for N = ˜ O ( N Ω ( γ )+log(1 /δ ) γ ) , constructs a function f : Ω → [0 , 1] such that with probability at least 1 -δ/ 2 :

Λ β f ( µ ) ≤ 5 ̂ Λ β ̂ f .

Proof. Throughout the proof, we denote for any point x ∈ Ω , subset B ⊂ Ω and function g : B → [0 , 1] :

Give the sample S = ( X i , Y i ) n i =1 , we denote S x = ( X i ) n i =1 . Let γ > 0 . The algorithm constructs f : Ω → [0 , 1] as follows:

Let S x ( γ ) ⊂ S x consist of the ⌊ γn ⌋ points whose Λ ̂ f ( · , S x ) values are the largest (with ties broken arbitrarily), and S ′ x ( γ ) := S x \ S x ( γ ) be the rest.

Let A ⊂ S ′ x ( γ ) be a γ 1 /β -net of S ′ x ( γ ) .

Define f : Ω → [0 , 1] to be the β -PMSE extension of ̂ f from A to Ω as defined in Definition A.2 (and analyzed throughout Appendix A).

We will prove that f satisfies both requirements. For the first requirement, since f | A = ̂ f | A and S x = S ′ x ( γ ) ⊎ S x ( γ ) we have

The first summand above is bounded by γ since 0 ≤ f, ̂ f ≤ 1 = ⇒ | f ( x ) -̂ f ( x ) | ≤ 1 and | S x ( γ ) | ≤ γn . In order to bound the second term, we denote by N A : S ′ x ( γ ) → A to be the mapping of each element to its nearest neighbor in the net, and note that ρ ( x, N A ( x )) ≤ γ 1 /β . Then

So overall we get ∥ f -̂ f ∥ L 1 ( µ n ) ≤ γ +2 γL = γ (1 + 2 L ) as claimed in the first bullet.

We move on to prove the second bullet. Let U ⊂ Ω be a γ 1 /β 4 -net of Ω , Π be its induced Voronoi partition and let m := | Π | ≤ N Ω ( γ 1 /β / 4) . Let Consider the following partition of Π into 'light' and 'heavy' cells:

We will now state three lemmas required for the proof, two of which are due to [Ashlagi et al., 2021].

̸

Lemma B.4. Suppose A ⊂ Ω and that f : Ω → [0 , 1] is the β -PMSE extension of some function from A to Ω . Let E ⊂ Ω such that diam( E ) β ≤ 1 2 min x = x ′ ∈ A ρ ( x, x ′ ) β . Then sup x,x ′ ∈ E Λ β f ( x ) Λ β f ( x ′ ) ≤ 2 .

Proof. Let u ∗ x , v ∗ x ∈ A be the pair of points which satisfy Λ β f ( x ) = f ( v ∗ x ) -f ( u ∗ x ) ρ ( v ∗ x ,x ) β + ρ ( u ∗ x ,x ) β . By assumption on E , we know that 2diam( E ) β ≤ ρ ( v ∗ x , u ∗ x ) β ≤ ρ ( v ∗ x , x ) β + ρ ( u ∗ x , x ) β , hence

Lemma B.5 (Ashlagi et al., 2021, Lemma 16) . If nγ 2 ≥ m , then

Lemma B.6 (Ashlagi et al., 2021, Lemma 17) . ∥ f ∥ H¨ ol β ≤ 2 L γ .

Equipped with the three lemmas, we calculate

The first summand above is bounded with high probability using Lemma B.5 and Lemma B.6, since under the event described in Lemma B.5 we have:

In order to bound the second term in Eq. (14), let C ∈ Π , x ′ ∈ C and note that by applying Lemma B.4 to E := S x ∩ C we get that Λ β f ( x ′ ) ≤ 2 min x ∈ S x ∩ C Λ β f ( x ) . Thus, under the high probability event described in Lemma B.5 we have

where the last inequality is due to the extension property of Theorem A.3. Overall, plugging these bounds into Eq. (14) and using the union bound to ensure all required events to hold simultaneously, we see that the desired second bullet holds holds with probability at least 1 -m exp( -nγ/ 4 m ) -exp ( -n ( γ -√ m/n ) 2 / 2 ) . A straightforward computation shows that by our assumption on n being large enough, this probability exceeds 1 -δ/ 2 as required.

We are now ready to finish the proof of Theorem 4.1. Let γ > 0 . By Corollary B.2, we can construct ̂ f : S → [0 , 1] such that with probability at least 1 -δ/ 2 : L S ( ̂ f ) = 0 and ̂ Λ β ̂ f ≤

5 log 2 (4 n/δ ) L . Assuming n is appropriately large, we further apply Proposition B.3 in order to obtain f : Ω → [0 , 1] such that with probability at least 1 -δ/ 2 : f ∈ H¨ ol β 25 log 2 (4 n/δ ) L (Ω) and also L S ( f ) ≤ L S ( ̂ f ) + γ (1 + 2 L ) = γ (1 + 2 L ) . By the union bound, we get that with probability at least 1 -δ :

where ( ∗ ) is justified by setting γ = Θ( ε/L ) and applying Theorem 3.4 for appropriately large n .

B.4 Proof of Theorem 5.1

Given a sample S = ( X i , Y i ) n i =1 ∼ D n , denote the empirically smooth class

Consider the following procedure:

( Empirical cover ) Construct h 1 , . . . , h N ∈ ̂ H¨ ol for maximal N such that ∀ i = j ∈ [ N ] : ∥ h i -h j ∥ L 1 ( µ n ) ≥ ϵ 4 .

̸

( Run realizable algorithm on cover ) For any j ∈ [ N ] , execute the realizable algorithm A realizable of Theorem 4.1 on the 'relabeled' dataset ( X i , h j ( X i )) ⌊ n/ 2 ⌋ i =1 , and obtain f j : Ω → [0 , 1] .

( ERM ) Return arg min f 1 ,...,f N ∑ n i = ⌊ n/ 2 ⌋ +1 | f j ( X i ) -Y i | .

We will now prove that the algorithm above satisfies the theorem. Let f ∗ ∈ arg min f ∈ H¨ ol β L (Ω ,µ ) L D ( f ) , 6 and note that by Proposition B.1 (as explained in Corollary B.1) we have f ∗ ∈ ̂ H¨ ol with probability at least 1 -δ/ 2 . By construction, h 1 , . . . , h N is a maximal ϵ 4 -packing of ̂ H¨ ol , which is known to imply that it is also a ϵ 4 -net [Vershynin, 2018, Lemma 4.2.8] with respect to the metric L 1 ( µ n ) . In particular, this implies that there exists j ∗ ∈ [ N ] such that

Further note for any j ∈ [ N ] : h j ∈ ̂ H¨ ol , so our realizable algorithm (as manifested in Proposition B.3 for γ = Θ( ϵ/L ) ) when fed the 'smoothed' labels ( X i , h j ( X i )) ⌊ n/ 2 ⌋ i =1 will produce f j such that L S ( f j ) ≤ L S ( h j ) ≤ ϵ 4 and Λ β f j ( µ ) ≤ 5 ̂ Λ β h j ≤ 25 log 2 (4 n/δ ) L . In particular

Finally, by Eq. (1) and Theorem 3.1 (which holds for any measure, in particular for the empirical measure µ n )

Hence, by a standard Chernoff-Hoeffding bound over the finite class { f 1 , . . . , f N } , step (3) of the algorithm yields ϵ 2 excess risk as long as n 2 = Ω ( log( N )+log(1 /δ ) ϵ 2 ) .

B.5 Proof of Theorem 6.1

We start by providing a simple structural result which we will use for our lower bound construction, showing that in any metric space there exists a sufficiently isolated point from a large enough subset.

Lemma B.7. There exists a point x 0 ∈ Ω and a subset K ⊂ Ω such that

∀ x ∈ K : ρ ( x 0 , x ) ≥ diam(Ω) 4 .

̸

∀ x = y ∈ K : ρ ( x, y ) ≥ ( ε/L ) 1 /β .

Proof. Denote D := diam(Ω) , let x 0 , x 1 be two points such that ρ ( x 0 , x 1 ) > D/ 2 , and let Π = { C 0 , C 1 } be a Voronoi partition of Ω induced by { x 0 , x 1 } . For γ > 0 , let N γ be a maximal γ -packing of Ω . By the pigeonhole principle there must exist a cell C i ∈ Π such that | C i ∩ N γ | ≥ | N γ | / 2 , which we assume without loss of generality to be C 1 . Now note that any x ∈ C 1 satisfies ρ ( x, x 0 ) ≥ 1 2 ρ ( x, x 0 ) + 1 2 ρ ( x, x 1 ) ≥ 1 2 ρ ( x 0 , x 1 ) > D/ 4 . Finally, set γ := ε 1 /β and let K ⊂ C 1 ∩ N γ be any subset of size ⌊ N Ω (( ε/L ) 1 /β ) 2 ⌋ .

Given x 0 , K from the lemma above, we denote K = { x 0 } ∪ K and define the distribution µ over Ω supported on K such that µ ( x 0 ) = 1 -ε 2 and µ ( x ) = ε 2 | K | for all x ∈ K . From now on, the proof is similar to a classic lower bound strategy for VC classes in the realizable case (e.g. Kearns and Vazirani, 1994, Proof of Theorem 3.5). To that end, it is enough to provide a distribution over functions in H¨ ol β L (Ω , µ ) such that with constant probability any algorithm must suffer significant loss for some function supported by the distribution.

We define such a distribution over functions f : K →{ 0 , 1 } as follows: Pr[ f ( x 0 ) = 0] = 1 , while for any x ∈ K : Pr[ f ( x ) = 0] = Pr[ f ( x ) = 1] = 1 2 independently of other points. We will now show that any such f : K →{ 0 , 1 } is average Hölder smooth with respect to µ . Indeed, for every x ∈ K :

while hence

Finally, we define the (random) function f ∗ : Ω → [0 , 1] to be the β -PMSE extension of f from K to Ω as defined in Definition A.2, and note that f ∗ satisfies the required smoothness assumption. Setting D over Ω × [0 , 1] to have marginal µ and Y = f ∗ ( X ) , we ensure that D is indeed realizable by H¨ ol β L (Ω) .

Now assume A is a learning algorithm which is given a sample S of size | S | ≤ | K | 4 ε and produces A ( S ) : Ω → [0 , 1] . Wecall a point x ∈ K "misclassified" by the algorithm if | A ( S )( x ) -f ∗ ( x ) | ≥ 1 2 , and denote the set of misclassified points by M ⊂ K . Recalling that ∀ x ∈ K : Pr[ f ( x ) = 0] = Pr[ f ( x ) = 1] = 1 2 independently, and that µ ( x ) = ε 2 | K | , we observe that with probability at least 1 2 the algorithm will misclassify more than | K | / 2 points. 7 Thus, we get that with probability at least 1 2 :

By rescaling ε , we see that in order to obtain L D ( A ( S )) ≤ ε the sample size must be of size

B.6 Proofs from Section 7

Proof of Claim 7.1. Let β ∈ (0 , 1) . Consider the unit segment Ω = [0 , 1] with the standard metric, equipped with the probability measure µ with density dµ dx = 1 Z | x -1 2 | β -1 2 (where Z = ∫ 1 0 | x -1 2 | β -1 2 < ∞ is a normalizing constant). We examine the function f ( x ) = 1 [ x > 1 2 ] which is clearly not Hölder continuous since it is discontinuous. Furthermore,

hence f / ∈ ˜ Lip M (Ω , µ ) for all M > 0 . On the other hand, Λ β f ( x ) = 1 | x -1 2 | β so

thus f ∈ H¨ ol β L (Ω) for some L < ∞ . Note that by normalizing the function, the claim holds even for L = 1 .

̸

Proof of Claim 7.2. Let β ∈ (0 , 1) . Consider the unit segment Ω = [0 , 1] with the standard metric, equipped with the probability measure µ with density dµ dx = 1 Z | x -1 2 | β -1 (where Z = ∫ 1 0 | x -1 2 | β -1 dx < ∞ is a normalizing constant). We examine the function f ( x ) = 1 [ x > 1 2 ] . Note that for any x = 1 2 : Λ 1 f ( x ) = 1 | x -1 2 | , hence

This shows that

̸

hence f / ∈ ˜ Lip M (Ω , µ ) for all M > 0 . Furthermore, for x = 1 2 : Λ β f ( x ) = 1 | x -1 2 | β so

hence f / ∈ ˜ H¨ ol β M (Ω , µ ) for all M > 0 . On the other hand

thus f ∈ ˜ H¨ ol β L (Ω) for some L < ∞ . Note that by normalizing the function, the claim holds even for L = 1 .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a notion named average Hölder smoothness, which is a generalization of average Lipschitz smoothness. They gave sample complexity bounds of learning the averge Hölder smooth function class in both the realizable and agnostic setting with corresponding algorithms. They also gave lower bounds which matches the upper bound up to logarithmic factors.\n\nSTRENGTHS:\n1. The writing is clear. The paper is easy to follow and the authors explain their contributions well. The technical part in the main article, although concise, highlights the major part of the proof.\n2. The contributions are valuable. The sample complexity of learning an average smooth Hölder function class is previous unknown even in the special case of average Lipschitz smoothness and so is the lower bound. The authors also showed the class of average smooth Hölder functions are strictly richer than previous notions like average smooth Lipschitz functions.\n3. The algorithm is novel and has great potential. Indeed the classical ERM fails on learning the average smooth Hölder function class in the distribution free setting so it is conceivable that the algorithm could be used to solve some real problems arising from practice that couldn't be solved before.\n4. The results are quite complete. The upper bound is tight up to only logarithmic factors and the bounds is analogous to classical PAC learning sample complexity bounds.\n\nWEAKNESSES:\n1. I'm not quite convinced that there's no efficient algorithm for the agnostic case unless more evidence is provided. The reduction from agnostic setting to realizable setting is quite generic. But for this learning problem, it feels quite possible that we can find an efficient algorithm. \n2. There are some typos. In line 182 it should be \"There exits a positive number $N$\" instead of a \"function\".\n\nQUESTIONS:\n1. Is it possible to derive tighter bounds for subclasses of average Hölder smooth function classes? In the paper only learning from the entire class is considered but it could be much easier to learn from a subclass in some cases, right? Is it possible to derive a bound dependent on the specific function class? If so, will the average Hölder smooth parameter still show up somewhere in the bound?\n\n2. I think it's better to give some concrete examples in applications, like giving a setting where ERM fails and we can only use your algorithm. Learning finite sums with a distribution concentrating on positions where the smooth parameter is small may be a good example.\n\n3. I think maybe you should compare your bound more with the classical PAC learning bound to help reader better understand. \n\n4. Are there any other works giving sample complexity bounds on distribution free learning for generic or specific function classes? Maybe adding these to the related work section.\n\nLIMITATIONS:\nThe authors addressed the limitation of their theoretical work properly.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the notion of \"average Hölder smoothness\", generalizing the notion of Holder smoothness.\nAuthors motivate the introduction of this class, explaining that the Holder constant is a supremum over many slopes that can eventually not be representative of the general behavior of this slope.\nAfter a long recall on needed notions, authors listed their theorems.\n\nThe first notable one (Thm 3.1) controls the backeting entropy of the newly defined class of functions with its covering number.\nThis theorem is crucial as\n    1. the bracketing entropy is used to encode the number of \"packs\" (brackets) one needs to enclose all the average holder smooths functions, and then naturally bounds the generalization error (See Thm 3.2),\n    2. the covering number is more intuitive and known in classical case.\nThen, Thm 3.4 follows, providing the generalization error gap on this class of function depending on the number of sampled data.\n\nThe next 2 sections discuss learning algorithms to minimize the empirical function keeping the generalization gap low. Note that an approximation of holder constants is used since its exact value cannot be computed.\n\nFinally, lower bounds are computed in the section 6.\n\nOverall, this paper seems self contained and complete, generalizing results obtained on average Lipschitz continuous function class.\n\nSTRENGTHS:\nThe paper is self-contained, well written, notions are explained clearly and the reasoning is explained step by step to obtain a generalization bound on a newly proposed class of functions that is much larger than the classical ones and describes functions better.\n\nWEAKNESSES:\nMinor:\n\n- l.147: What about N=1? What is the log basis? I checked in [Ashlagi et al., 2021, Lemma 22]: the log basis is e and there is a small mistake at the very end of their proof. The actual bound should be $E[Z] \\leq (1 + log(N)) W[Z]$. Finally, the provided bound only holds for $N\\geq 3$.\n\nQUESTIONS:\n- What motivates these definitions of « semi-local » holder constants? Would we be able to say something with $\\Lambda_f^{\\beta}(x) = inf_{\\varepsilon>0} sup_{y\\in B(x, \\varepsilon) -\\lbrace x \\rbrace} \\frac{|f(x)-f(y)|}{\\rho(x, y)^{\\beta}}$ ?\n- If I understand correctly, Thm 3.1 and 3.4 are claimed as contribution, but not Proposition 3.2. Where can we find it?\n- Thm 3.4: Why did we lose the $log(1/\\varepsilon)$ term from Th 3.1?\n\nLIMITATIONS:\n- Cor 3.5: Number of needed data for good generalization is exponential in d.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper generalizes the notion of average Lipschitz smoothness proposed by Ashlagi et al. by extending it to Hölder smoothness. Eventually, the class of the average holder smooth function is much larger than the class of the worst-case holder smooth function. Further, the paper provides upper and lower risk bounds in terms of the average Hölder smoothness for realizable regression and noisy regression.\n\nSTRENGTHS:\nThough I am not an expert in this area, the results provided in the paper are clearly better than the previous result for the average Lipschitz case. The bounds provided in the paper for bracketing number is distribution-free. Using this distribution free bound on the bracketing number, the authors prove sample complexity for obtaining uniform convergence of average-Hölder functions in the realizable case. the learner does not have an explicit representation of the function class, running an ERM is not possible in this case. Hence, the authors construct a non-parametric regression algorithm with nearly an optimal rate. \nThe authors also provide results for agnostic settings using the reduction from agnostic to realizable settings proposed in the previous work of  Hopkins et al. [2022]. A matching lower bound is also provided in the paper.\n\nWEAKNESSES:\nOverall, I find the paper quite well written. However, it would be great if more details about the algorithm could be provided in the paper. Again, this is coming from non an expert in this area. To obtain the result in theorem 4.1, what is the modification needed in the algorithm as running ERM is not possible? It would be great if that could be explained in detail. The same goes for the agnostic result. \nAlso, the worst-case rate in remark 4.2 is exponential in dimension, can the authors please comment that under what conditions we might get rid of exponential dimension independence? For example, approximation results in sobolev space get rid of exponential dependence if the function is differentiable O(d) times.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}