Title: Online Consistency of the Nearest Neighbor Rule
Venue: NeurIPS 2024
Accepted: yes
Keywords: Nearest Neighbor Classification

Online Learning

Smoothed Analysis
Abstract: In the realizable online setting, a learner is tasked with making predictions for a stream of instances, where the correct answer is revealed after each prediction. A learning rule is online consistent if its mistake rate eventually vanishes. The nearest neighbor rule is fundamental prediction strategy, but it is only known to be consistent under strong statistical or geometric assumptions: the instances come i.i.d. or the label classes are well-separated. We prove online consistency for all measurable functions in doubling metric spaces under the mild assumption that instances are generated by a process that is uniformly absolutely continuous with respect to an underlying finite, upper doubling measure.

Online Consistency of the Nearest Neighbor Rule

Abstract

In the realizable online setting, a learner is tasked with making predictions for a stream of instances, where the correct answer is revealed after each prediction. A learning rule is online consistent if its mistake rate eventually vanishes. The nearest neighbor rule (Fix and Hodges, 1951) is a fundamental prediction strategy, but it is only known to be consistent under strong statistical or geometric assumptions-the instances come i.i.d. or the label classes are well-separated. We prove online consistency for all measurable functions in doubling metric spaces under the mild assumption that the instances are generated by a process that is uniformly absolutely continuous with respect to a finite, upper doubling measure.

1 Introduction

In online classification, a learner faces a never-ending stream of prediction tasks. For all times n :

-the learner is presented with an instance X n ,

-the learner makes a prediction ˆ Y n ,

-the ground-truth label Y n is revealed.

When the instances come from some underlying metric space, one of the simplest prediction rules the learner can employ is the nearest neighbor rule (Fix and Hodges, 1951). This learner memorizes everything it sees, and when it comes time to make a prediction for a new instance X n , it looks for the most similar data point in memory, predicting with that nearest neighbor's label. In this work, we are interested in simple conditions under which the nearest neighbor rule is in fact a reasonable strategy, where the rate at which the learner makes mistakes eventually vanishes.

Let ( X , ρ, ν ) be a metric measure space where ρ is a separable metric and ν is a finite Borel measure. We study the realizable setting, in which the ground-truth labels Y n = η ( X n ) are given by some measurable label function η : X → Y . Let X = ( X n ) n ≥ 0 be any stochastic process. It induces a nearest neighbor process ˜ X = ( ˜ X ) n> 0 , which is any process satisfying:

The nearest neighbor rule predicts using the label ˆ Y n = η ( ˜ X n ) , and it is online consistent when the asymptotic mistake rate on the problem instance ( X , η ) goes to zero:

glyph[negationslash]

There are two representative results for the online consistency of the 1-nearest neighbor rule in this setting-both impose strong constraints on either X or η . Cover and Hart (1967) assume the

glyph[negationslash]

statistical constraint that X is an i.i.d. process. In contrast, Kulkarni and Posner (1995) allow for arbitrary processes. However, they assume the geometric constraint that points of different classes η ( x ) = η ( x ′ ) are uniformly separated ρ ( x, x ′ ) > c > 0 in a totally bounded space.

The result of Kulkarni and Posner (1995) turns out to be tight in the absence of further assumptions. As long as there are points belonging to different classes that are arbitrarily close together, then there are adversarial sequences on which the nearest neighbor rule is not online consistent (Proposition 2). Still, this negative result does not necessarily spell doom for the nearest neighbor rule in all non-i.i.d. settings; one of our aims is to understand just how pathological these worst-case sequences are.

The upshot of this work is that worst-case sequences on which the nearest neighbor rule fails to learn are in fact extremely rare-under quite mild constraints on X and η , they almost never occur. The nearest neighbor rule is online consistent under much broader conditions than previously known.

1.1 Main results

Consistency for functions with negligible boundary We first consider learning label functions with negligible boundary , where almost every point has a positive separation from other classes. As the separation may be instance-dependent, this relaxes the uniform separation condition of Kulkarni and Posner (1995), and it is equivalent to the assumption used by Cover and Hart (1967).

It turns out that if the label function has negligible boundary, we can cover essentially all of X with mutually-labeling balls (Definition 8). On such a ball, the nearest neighbor rule makes at most one mistake (Lemma 9). So, progress is monotonic: eventually, all mistakes must come from a remainder region with arbitrarily small ν -mass (roughly, points arbitrarily close to the decision boundary).

It follows that if we can limit the rate at which X comes from regions with arbitrarily small mass, we can also limit the mistake rate of the nearest neighbor rule. To this end, we formalize the notion of an ergodically dominated process (Definition 3), which is a process where the asymptotic rate of landing in a region A is bounded as a function of ν ( A ) . In particular, these processes do not hit regions with arbitrarily small mass at a constant rate. With little ado, we can show that the nearest neighbor rule is consistent when X is ergodically dominated and η has negligible boundary (Theorem 7). Stronger, quantitative assumptions also yield rates of convergence (Theorem F.2).

Universal consistency This first consistency result for functions with negligible boundaries is quite general and captures many settings of interest (e.g. classification on R d with smooth decision boundaries). But as not all functions have ν -negligible boundaries, we also study when the nearest neighbor rule is universally consistent (i.e. consistent for any measurable η ). This question is trickier since boundary points, which are hard for our learner, need not be localized to a set of measure zero.

We proceed by giving more structure to ( X , ρ, ν ) and X . First, we let ρ be a d -doubling metric (Definition 11). This is helpful because every measurable function η can then be approximated arbitrarily well by a function η ′ with negligible boundary (Proposition 13). And so, it seems that we might be able to deduce universal consistency of the nearest neighbor rule almost directly from the previous result-learning η is perhaps not so different from learning η ′ when their disagreement region is made to be vanishingly small. However, this turns out not to be the case.

For example, Blanchard (2022) constructs a classification problem where the nearest neighbor rule is not consistent, but X is a 1-doubling space (the unit interval [0 , 1] with the usual metric), η is measurable, and X is ergodically dominated. The problem is that, even if the disagreement region is made to be extremely small, its influence on nearest neighbor predictions is not limited to the times when instances land in it. These instances exert influence when they themselves are nearest neighbors of downstream instances. In other words, 'bad points' can accumulate in the memory of

the nearest neighbor learner, and their influence grow and shrink with their Voronoi cells (regions where they are nearest neighbors). As the region on which the nearest neighbor learner is prone to make mistakes waxes and wanes throughout time, we cannot argue that progress is monotonic in the same way as before. In short, a tail constraint on X is no longer sufficient for consistency.

To show universal consistency, we constrain X at every moment in time. Ergodic domination only ensured that the time-averaged rate at which X hits small regions is small. We now require the timeuniform rate to also be small, a strictly stronger condition. Formally, a uniformly dominated process (Definition 4) is one where the probability that the instance X n lands in a region A is bounded as a function of ν ( A ) at each point in time. Intuitively, ergodic domination is retrospective: looking back, how often do points land in A ? In contrast, uniform domination is a generative constraint: at any point in time, how easily can the underlying mechanism generating X select a point in A ?

The basic argument then is that even though 'bad points' can accumulate in space over time, in a doubling space, their Voronoi cells also tend to shrink quickly when they are hit by further instances. If the mass of these Voronoi cells were to shrink as well, then it would become increasingly unlikely that these instances are nearest neighbors of downstream points whenever X is uniformly dominated. So that having small metric entropy implies having small mass, we let the measure be upper doubling (Definition 11), by which we mean that the mass of a ball of radius r is bounded by O ( r d ) .

We first prove the following result, which may be of interest in its own right, about the behavior of nearest neighbor processes: when X is a uniformly dominated process in an upper doubling space, any nearest neighbor process ˜ X is ergodically dominated (Theorem 14). Simply put, this means that if two functions η and η ′ rarely disagree, then the average rate at which nearest neighbor processes land in their disagreement region is also bounded. Universal consistency now follows fairly easily.

Let η be closely approximated by some η ′ with negligible boundary. The asymptotic mistake rate of the nearest neighbor rule on η ′ is zero when X is uniformly dominated. On the other hand, if the mistake rate on η is large, this discrepancy must be due to the influence of their (very small) disagreement region. But, a large discrepancy is not possible as the nearest neighbor process is unable to significantly amplify the influence of very small regions. Thus, the nearest neighbor rule is universally consistent on upper doubling spaces for uniformly dominated processes (Theorem 12).

Notation Given a sequence X , we let X <n denote the set { X 1 , . . . , X n -1 } . The symbol ✶ denotes the indicator function, which is equal to 1 when the event that follows it occurs and 0 otherwise. We let B ( x, r ) ⊂ X denote the open ball of radius r centered at x . For any x ∈ X and Z ⊂ X , let:

2 Non-convergence for worst-case sequences

To motivate the study of non-worst case sequences, let's first consider a worst-case example showing that the nearest neighbor rule can make a mistake in each round learning a threshold function:

Example 1 (Failing to learn a threshold) . Let X = [ -1 , 1] and let η ( x ) = ✶ { x ≥ 0 } be a threshold function. Let X be defined by X n = ( -1 / 3) n . The nearest neighbor rule makes a mistake every round n +1 : the nearest neighbor of X n +1 is X n , which has the opposite sign (see Figure 1).

More generally, the hardness of a point for the nearest neighbor rule depends on its separation from points of different classes-hard sequences exist precisely whenever the classes are not separated:

Proposition 2 (Non-convergence in the worst-case) . Let ( X , ρ ) be a totally bounded metric space. Given η : X → Y , there is a sequence of instances ( X n ) n on which the nearest neighbor rule is not online consistent on η if and only if there is no positive separation between classes:

glyph[negationslash]

While the nearest neighbor rule can fail to learn many functions of interest, in both the example and the proof of the proposition, the mode of failure depended on the ability of a worst-case adversary to select instances with arbitrary precision. This can be seen in the threshold example, where the interval on which the learner can make a mistake shrinks exponentially quickly.

0

X

1

X

2

X

3

· · ·

However, this mode of failure may not always be present, especially when the ability of the adversary to select instances from small regions of space diminishes with the sizes of those regions. This may be the case if the adversary is limited in computation, information, or ill-will toward the learner. The question remains: what is the behavior of the nearest neighbor rule in such non-worst-case settings?

3 Two general classes of non-worst case sequences

To study this, let's formalize the generating process. At each time step n , conditioned on the past outcomes X <n , an adaptive adversary constructs a distribution over X from which X n is drawn. In this way, any particular choice of adaptive adversary corresponds to a stochastic process, which defines a probability measure over the space of all sequences of instances. We are interested in the almost-sure online consistency of the nearest neighbor rule under these measures.

In the standard i.i.d. setting, each X n is drawn from the same underlying distribution. In the worstcase setting, the conditional distribution of X n | X <n may be point masses and so X may be an arbitrary sequence. We introduce two mildly-constrained classes of non-worst-case processes. Both are given with respect to an underlying reference measure ν (which we assumed to be finite).

Ergodically dominated processes The first can be considered a class of budgeted adversaries . For any region A ⊂ X , the asymptotic rate at which the adversary can select points in A is bounded by a function ε ( · ) of ν ( A ) . In particular, we require ε ( δ ) - → 0 as δ goes to zero. Instances from an ergodically dominated process do not concentrate in regions with small mass in retrospect.

Definition 3 (Ergodic continuity) . A stochastic process X is ergodically dominated by ν if for any ε > 0 , there exists δ > 0 such that when a measurable set A ⊂ X satisfies ν ( A ) < δ , then:

We say that X is ergodically continuous with respect to ν at rate ε ( δ ) .

As pointed out by Hanneke (2021), the set function A ↦→ lim sup N →∞ 1 N ∑ N n =1 ✶ { X n ∈ A } is a submeasure. Then, ergodic continuity requires it to be absolutely continuous with respect to ν . These processes are closely related to the C 1 -processes introduced by Hanneke (2021). There, the submeasures must be exhaustive , which Talagrand (2008) shows to be a strictly weaker condition than absolute continuity. Hanneke (2021) also demonstrates that there are C 1 -processes on the unit interval for which the nearest neighbor rule is not universally online consistent.

Uniformly dominated processes The following can be thought of as a class of bounded precision adversaries. Each time step, the probability that the adversary selects an instance from a region A is bounded by a function ε ( · ) of ν ( A ) . Thus, it provides a time-uniform condition:

Definition 4 (Uniform absolute continuity) . A stochastic process X is uniformly dominated by ν if for any ε > 0 , there exists δ > 0 such that when a measurable set A ⊂ X satisfies ν ( A ) < δ , then:

We say that X is uniformly absolutely continuous with respect to ν at rate ε ( δ ) .

This class can be seen as a strict generalization of the σ -smoothed processes introduced by Haghtalab et al. (2020), which satisfy the Lipschitz rate ε ( δ ) = δ/σ (Definition F.3). This stronger, quantitative condition allows us to give rates of convergence for smoothed processes in Appendix F.

Time-averaged behavior of uniformly dominated processes Wenowshow that ergodic continuity is a weaker condition than uniform absolute continuity. And intuitively, the former lets us prove consistency when the hard or atypical instances come from a small, fixed region of space. The latter will be needed when the hard regions of space evolve over time.

In the following, we can think of ( A n ) n as a sequence of hard regions (e.g. the region on which the learner can make mistakes). By the martingale law of large numbers, if the mass of these regions eventually remain small, then the average rate at which X lands in hard regions also becomes small:

Lemma 5. Let X be uniformly dominated by ν at rate ε ( δ ) , and let ( F n ) n be its natural filtration. Let A n be an F n -predictable sequence where lim sup n →∞ ν ( A n ) < δ almost surely. Then:

Setting A n to A shows that ergodic continuity is weaker than uniform absolute continuity. In fact, as it only constrains the tail of X , it is strictly weaker (the ergodically-dominated adversary is stronger).

4 Consistency for functions with negligible boundaries

The inductive bias built into the nearest neighbor rule is that most points are surrounded by other points of the same class (though one might have to zoom in very close to the point). We first consider label functions for which this inductive bias is correct ν -almost everywhere. Ergodic continuity with respect to ν shall be enough for online consistency. To formalize these functions, we define:

Definition 6 (Boundary point) . Let η : X → Y be measurable. Let margin η ( x ) be the distance from x to points of other classes, and let ∂ η X denote the boundary of η , points with no margin:

glyph[negationslash]

Let F 0 = { η measurable : ν ( ∂ η X ) = 0 } denote the set of functions with negligible boundaries .

The class F 0 captures many label functions of interest. For example, when X is a Euclidean space equipped with the Lebesgue measure, then label functions with smooth decision boundaries have negligible boundaries-the boundary forms a lower-dimensional manifold with zero measure.

Theorem 7 (Online consistency for F 0 ) . Let ( X , ρ, ν ) be a metric measure space, where ρ is a separable metric and ν is a finite Borel measure. Let X be ergodically dominated by ν and let η have ν -negligible boundary. The nearest neighbor rule is online consistent with respect to ( X , η ) .

To prove this, we introduce the notion of a mutually-labeling set , which are subsets U of X that share a single label under η , and whose diameter is less than the distance to reach a different class:

Definition 8 (Mutually-labeling set) . A set U ⊂ X is mutually-labeling for η if for all x ∈ U ,

See Figure 2 for a picture. This construct is useful because the nearest neighbor rule makes at most one mistake per mutually-labeling set-see Lemma 9. Moreover, it is easy to construct such sets; Lemma 10 shows that sufficiently small balls centered at non-boundary points are mutually-labeling.

Lemma 9. Let U be a mutually-labeling set for η . Let X be an arbitrary process. Then:

glyph[negationslash]

Lemma 10. For any 0 < r < margin η ( x ) / 3 , the ball B ( x, r ) is mutually-labeling for η .

Proof of Theorem 7. Fix ε > 0 . We first prove that the asymptotic mistake rate is bounded by ε . Choose δ > 0 such that the ergodic domination condition, Equation (2), holds. We claim that we can cover all of X by a finite number K δ < ∞ of mutually-labeling sets, except for a region A δ of small mass ν ( A δ ) < δ . By Lemma 9, at most one mistake can be made on each of the mutuallylabeling set, so that all but finitely many mistakes come from A δ . Thus, the asymptotic mistake rate is bounded by the rate at which X can hit A δ . By definition of ergodic continuity, this is less than ε , glyph[negationslash]

To finish the proof, we construct the above collection of mutually-labeling sets. By Lemma 10, we can cover almost all of X by the collection of mutually-labeling balls, since we only miss out on the boundary points, which is ν -negligible. This collection has a countable subcover C = { B 1 , B 2 , . . . } because ρ is separable. By the finiteness of ν and the continuity of measures, when K δ is sufficiently large, the first K δ balls cover everything but a region A δ of mass ν ( A δ ) < δ ,

A picture is also given in Figure 2.

5 Universal consistency on upper doubling spaces

We now show that the nearest neighbor rule is universally online consistent (that is, consistent for any measurable function) whenever X is uniformly dominated and X is an upper doubling space:

Definition 11 (Upper doubling) . A metric space ( X , ρ ) is doubling with doubling dimension d if every ball B ( x, r ) can be covered by 2 d balls of radii r/ 2 . A d -doubling space with measure ν is upper doubling if there exists c > 0 such that for all B ( x, r ) , we have ν ( B ( x, r ) ) ≤ cr d .

This notion was introduced under a somewhat more general form by Hyt¨ onen (2010), and it relaxes the condition of a doubling measure space. For example, R d with the glyph[lscript] ∞ -distance and Lebesgue measure is readily seen to be upper doubling with doubling dimension d .

Theorem 12 (Universal consistency) . Let ( X , ρ, ν ) be an upper doubling metric measure space, where ρ is a separable metric and ν is a finite Borel measure. Let X be uniformly dominated by ν . For any measurable η , the nearest neighbor rule is online consistent with respect to ( X , η ) .

The doubling metric condition is helpful because the set F 0 of functions with negligible boundaries is then dense in L 1 ( X ; ν ) , as shown in Proposition 13. That is, any measurable η can be arbitrarily well-approximated by F 0 , for which we know the nearest neighbor rule is consistent.

Proposition 13 ( F 0 is dense in L 1 ) . Let ( X , ρ, ν ) be a metric measure spaces where ρ is doubling and ν is a finite Borel measure. Then, the set F 0 is dense in L 1 ( X , ν ) .

glyph[negationslash]

To show universal consistency, it is not enough that η can be well-approximated by a function η 0 with negligible boundary. We also need to know that their disagreement region { η = η 0 } cannot have excessive influence on the behavior of nearest neighbor predictions. The upper doubling condition allows us to show that when X is uniformly dominated, then ˜ X is ergodically dominated. We will use this to limit the rate that nearest neighbors come from regions where η is poorly approximated.

Theorem 14 (Ergodic continuity of nearest neighbor processes) . Let ( X , ρ, ν ) be a upper doubling space with bounded diameter. Suppose that a process X is uniformly dominated by ν at a rate ε ( δ ) . There exists constants c 1 , c 2 > 0 such that for any measurable set A ⊂ X with ν ( A ) < δ 0 ,

If we do not optimize the bound and just let δ = δ 0 , this shows that when X is uniformly dominated at rate ε ( δ ) , then ˜ X is ergodically dominated at a slower rate O ( ε ( δ ) log 1 δ ) . We can now show:

Proof of Theorem 12. Fix ε > 0 . Let X be uniformly dominated and let η be measurable. We prove that the asymptotic mistake rate of the nearest neighbor rule for ( X , η ) is bounded by 3 ε almost surely. The result follows by simultaneously applying this to a sequence ε i ↓ 0 .

glyph[negationslash]

Let η 0 ∈ F 0 be a δ 0 -accurate approximation of η so that ν ( { η = η 0 } ) < δ 0 . We will set δ 0 later. If at time n , the nearest neighbor rule makes a mistake, then at least one of three events must occur:

(a) The functions η and η 0 disagree on X n ,

(b) The functions η and η disagree on ˜ X

0 n , (a) (b)

(c) The nearest neighbor rule errs at time n on ( X , η )

0 . η 0 ( X n ) η 0 ( ˜ X n )

Lemma 5 implies that (a) contributes at most ε ( δ 0 ) to the asymptotic mistake rate, while Theorem 7 implies that (c) contributes nothing. We just need to bound the contribution of (b) for when the nearest neighbor process lands in the disagreement region of η and η 0 . For this, we can almost directly apply Theorem 14, except that it assumes that the process takes place in a bounded space.

It turns out that because ρ is doubling and ν is finite, there is a bounded region X ε ⊂ X that captures the vast majority of X and ˜ X ; Lemma D.4 bounds the rate at which either process escapes X ε ,

glyph[negationslash]

Having accounted for mistakes outside of X ε , we can consider the amended event (b ′ ) that η and η 0 disagree on A = X ε ∩ { η = η 0 } . Since ν ( A ) < δ 0 , we now apply Theorem 14; when c 1 and c 2 are the corresponding constants given by for the space ( X ε , ρ, ν ) , it suffices to set δ 0 > 0 so that:

Then, to the asymptotic mistake rate, the events in (a) contribute at most ε ( δ 0 ) < ε , (b ′ ) contribute another ε , (c) contribute nothing, and (d) contribute ε . Together, they yield the target 3 ε bound.

6 Ergodic continuity of nearest neighbor processes

The nearest neighbor rule does not forget; and so, a data point X n can be the nearest neighbor of an unbounded number of downstream instances X >n . In this section, we ask a trickier question: at what rate can a set of instances in X contain nearest neighbors of downstream points? More intuitively, how much influence can a set of instances exert through the nearest neighbor process?

Theorem 14 gives one result of this form: informally, if a process X can generate points from small regions only very rarely, then these points cannot make up a significantly larger fraction of ˜ X . More generally, we consider the long-term influence of any asymptotically rate-limited subsequence of X . Formally, to indicate the instances whose long-term influence we wish to bound, we define:

Definition 15 (Indicator process) . An indicator process I = ( I n ) n is a sequence of { 0 , 1 } -random variables. It induces a counter k ( n ) and the sequence of stopping times ( τ k ) k where:

That is, k ( n ) is the number of indications given by time n , while τ k is the time of the k th indication. We say that I is asymptotically rate-limited by γ > 0 if almost surely, lim sup n →∞ k ( n ) /n < γ .

Notation 16 (Indicated instances) . Let X be a process and I be an indicator process. Let X [ I <n ] denote the subset of instances in X indicated by time n (not inclusive), so that:

For uniformly dominated processes in upper doubling spaces, if this set of instances doesn't grow too fast, then its time-averaged influence is limited when filtered through the nearest neighbor process. For simplicity, in this section, we will also assume that the space is bounded.

Theorem 17 (Long-term influence bound) . Let ( X , ρ, ν ) be a bounded, upper doubling space. There are constants c 1 , c 2 > 0 so that the following holds. Let X be uniformly dominated at rate ε ( δ ) and let I be an indicator process adapted to X asymptotically rate-limited by γ > 0 . For any δ > 0 , the rate that the indicated instances X [ I <n ] contain a nearest neighbor ˜ X n is at most:

The ergodic continuity of ˜ X follows when we take I n to be ✶ { X n ∈ A } and optimize the bound. 6.1 A metric bound for nearest neighbor events

To prove Theorem 17, we need to balance two opposing dynamics. On one hand, more and more indicated instances fill the space as time goes on. On the other, the V oronoi cells of these instancesregions in which they are nearest neighbors-tend to shrink as they are hit. Mathematically, this can be seen by decomposing the event that an indicated instance is a nearest neighbor like so:

The natural proof strategy following this would be to bound the probability that the left-hand side event occurs by arguing that the indicated Voronoi cells have small ν -mass, so that these cells would be rarely hit if X is uniformly dominated. However, this decomposition does not seem to be very fruitful as it is difficult to directly control how the mass of V oronoi cells evolve over time.

Instead, our proof strategy makes use of both notions of size available to us in metric measure spaces. Besides the measure of a set, recall the packing number of a set, a notion of metric entropy:

Definition 18 (Packing and packing number) . Let r > 0 . A set Z ⊂ X is an r -packing if all of its points are bounded away from each other by a distance r ,

The r -packing number P r ( U ) of U is the maximum possible size of an r -packing Z contained in U .

The packing number bounds the number of times that the nearest neighbor distance ρ ( X n , ˜ X n ) is large. In particular, for any r > 0 , the nearest neighbor distance can exceed r at most P r ( X ) times. This is because such a set of instances must then form an r -packing. As a slight generalization:

Definition 19 ( r -separated event) . Let X be a process and r > 0 . The r -separated event at time n is the event E r n that X n is r -separated from all past instances X <n ,

Given a subset U , the ( U, r ) -separated events are the events E U,r n := E r n ∩ { X n ∈ U } .

Lemma 20 (Packing bound) . Let ( X , ρ ) be a metric space, U ⊂ X be a subset, and r > 0 . For any process X , the number of ( U, r ) -separated events is bounded by the r -packing number of U ,

Thus, we now have two ways of bounding how often an instance can appear in the nearest neighbor process: the number of times it can be a distant nearest neighbor is bounded by the packing number, while the rate it can be a close nearest neighbor can be limited by the measure of small balls. The basic proof idea for Theorem 17 will be to slowly trade off the measure bound for the metric bound via an alternative decomposition of the event ˜ X n ∈ X [ I <n ] based on the cover tree data structure.

6.2 The cover-tree decomposition

Definitions In this section, let X be a bounded metric space, which we may rescale to have unit diameter. Let us recall the cover tree data structure, introduced by (Beygelzimer et al., 2006), which is an efficient multi-scale ball cover for a given dataset: each data point is covered by a ball of every radius 2 -glyph[lscript] for all glyph[lscript] ∈ N 0 ≡ N ∪ { 0 } . First, we'll introduce the dyadic cone as a multi-scale cover of a single data point. Then, we inductively construct the cover tree as a union of dyadic cones.

Definition 21 (Dyadic cone) . Let ( X , ρ ) have unit diameter and let x ∈ X . A dyadic cone centered at x of rank L ∈ N 0 is the discrete collection of balls:

When L ′ ≥ L , we also refer to cone( x ; L ′ ) within cone( x ; L ) as its rankL ′ tail .

One possible way of constructing a multi-scale covering of a dataset is to take the union of all dyadic cones of rank zero centered all data points. However, this cover is not particularly efficient, with many redundant large-scale balls. Instead, the cover tree is defined as a union of dyadic cones with adaptive ranks. Each rank is chosen to be minimal while still covering each point at all scales:

Definition 22 (Sequentially-constructed cover trees) . Let ( X , ρ ) have unit diameter and A = ( a k ) k be a dataset in X without duplicates. The cover trees ( C k ) k with insertion ranks ( L k ) k are defined:

We say that a k was inserted into the cover tree at the L k th rank.

Points in A ≤ k are covered by the cover tree C k at all scales. All other points are also covered by balls in the doubled cover tree, where the radii are comparable to their distances to A ≤ k . In particular, we define a cover-tree neighbor map c k : X \ A ≤ k →C k as any one satisfying:

The following lemma shows that such a map always exist.

Lemma 23. The cover tree C k for A ≤ k has a cover-tree neighbor map c k .

Proof. Let x ∈ X \ A ≤ k and let a ∈ A ≤ k be a nearest neighbor, so that ρ ( x, a ) = ρ ( x, A ≤ k ) . There exists glyph[lscript] ∈ N such that 2 -( glyph[lscript] +1) ≤ ρ ( x, a ) < 2 -glyph[lscript] since x is not in A ≤ k . As a is covered at all scales, there is a ball B ∈ C k of radius 2 -glyph[lscript] that contains a . By triangle inequality, x is contained in 2 B .

Later, we shall also make use of the rooted tree structure on A induced by the cover trees:

Definition 24 (Tree structure) . For the above sequence of cover trees and insertion ranks, we let a 1 be the root of A . For all k > 1 , there is a ball B ( a j , 2 -L k +1 ) ∈ C k -1 containing a k . We assign such an a j to be the parent of a k , and we say that a k is its child . A set of instances inserted at the same rank to the same parent is called a generation of children. The number of generations of children a k has at time n defines the upper triangular array ( G k,n ) k ≤ n :

Application to nearest neighbor analysis Let ( C k ) k be a sequence of cover trees for the sequence of indicated instances A = ( X τ k ) k . Using cover-tree neighbors, we obtain a decomposition of the event that an indicated instance is a nearest neighbor (Lemma 25). Whereas the earlier Voronoi decomposition proposed in Equation (6) corresponds to the partition of X induced by the nearest neighbor map, this one is induced by the cover-tree neighbor map given by Equation (7).

Lemma 25 (Cover tree decomposition) . Let ( X , ρ ) have unit diameter, let X be a process in X , and let I be an indicator process. For any n , let C be a cover tree for X [ I <n ] with a cover-tree neighbor map c . Assume that X n / ∈ X [ I <n ] is not equal to one of the indicated instances. Then:

In particular, the event within the union indexed by B = B ( a, r ) ∈ C is contained in E 2 B,r/ 2 n .

Proof sketch of Theorem 17. Fix N . Let C be a cover tree for X [ I <N ] . A purely metric bound is:

Here, (i) follows from Lemma 25, where B r denotes a ball of radius r in C , (ii) is a larger summation, and (iii) applies the metric bound Lemma 20 and the fact that X is a d -doubling space.

Of course, this bound is vacuous since the cover tree has infinitely many balls. Instead of paying for every ball in C via the metric bound, we can decompose C into two pieces: a relatively slow-growing collection of large balls for which we apply the metric bound, and a collection of dyadic tails with small combined measure. In particular, we adaptively adjust the tail for each indicated instance so that the combined tail events always have ν -mass at most δ . As X is uniformly dominated, the asymptotic rate at which the tail events occur is no more than ε ( δ ) .

The basic idea for choosing the tails is this. First, when an indicated instance is inserted into the cover tree, we immediately shrink the tail of this new instance by removing O ( 1 d log c δ ) of its largest balls, where ( c, d ) are the parameters associated to the upper doubling condition. Secondly, if this new instance is the first of a new generation of children, we also remove the largest ball from the tail of its parent's dyadic cone. We account for the events corresponding to these removed balls via the packing bound, contributing 2 2 d to the finite bound per removed ball. The indicated instances appear at a rate asymptotically bounded by γ , so the cumulative packing bound grows at a rate of O ( γ · 2 2 d log c δ ) . Treating c and d as constants, we obtain an overall bound of:

See Figure 3 for a picture. Mathematically, we select the rank T k,n of the tail of the k th indicated instance at time n as follows. Let L k be the insertion rank of the k th indicated instance and let k ( n ) be the number of indicated instances that have arrived by time n . Set the rank to:

Lemma E.1 shows that this choice ensures that the combined tail event has ν -mass at most δ .

7 Related work

Nearest neighbor rule The nearest neighbor rule and its generalizations form a fundamental and well-studied part of non-parametric estimation, where the vast majority of work considers the i.i.d. setting (Fix and Hodges, 1951; Cover and Hart, 1967; Stone, 1977; Devroye et al., 1994; C´ erou and Guyader, 2006; Chaudhuri and Dasgupta, 2014). Also see the surveys Devroye et al. (2013); Dasgupta and Kpotufe (2021). Far less is known in the non-i.i.d. setting. For positive results, Holst and Irle (2001); Ryabko (2006) relax to conditional independence where there may be a distinct distribution over instances per class; Kulkarni and Posner (1995) study arbitrary processes. Blanchard (2022) shows a negative result: in some non-worst-case online settings where other learners succeed, the nearest neighbor learner may fail. In the batch learning setting, Dasgupta (2012) and Ben-David and Urner (2014), considered consistency when training and test data distributions differ.

Non-worst-case online learning Our results echo a motif of smoothed analysis (Spielman and Teng, 2004): worst-case analyses of algorithms yield safeguards by refraining from making difficultto-test assumptions, but they can be overly-pessimistic and fail to explain the observed behavior of algorithms (Roughgarden, 2021). Recently, two independent lines of work have emerged, introducing new classes of constrained stochastic processes for non-worst-case online learning: smoothed online learning (Rakhlin et al., 2011; Haghtalab et al., 2020, 2022; Block et al., 2022), and optimistic universal learning (Hanneke et al., 2021; Blanchard and Cosson, 2022; Blanchard, 2022). We connect the classes in these work through the strictly increasing chain of stochastic processes:

i.i.d. ⊂ smoothed ⊂ uniformly dominated ⊂ ergodically dominated ⊂ C 1 ⊂ arbitrary .

To summarize the behavior of the nearest neighbor rule in these settings, (a) we characterize when the nearest neighbor rule is consistent for arbitrary sequences (Proposition 2); (b) Blanchard and Cosson (2022) shows that the nearest neighbor process is not universally consistent for the C 1 -processes defined by Hanneke (2021); (c) we show, via a new and simple proof, that the original Cover and Hart (1967) F 0 -consistency result for i.i.d. processes actually holds for a much larger class of ergodically dominated processes (Theorem 7); (d) we show universal consistency in upper doubling spaces for uniformly dominated processes (Theorem 12); and (e) we prove rates of convergence for smoothed processes in doubling length spaces (Theorem F.2).

Acknowledgments and Disclosure of Funding

We thank the National Science Foundation for support under grant IIS-2211386.

References

Shai Ben-David and Ruth Urner. Domain adaptation-can quantity compensate for quality? Annals of Mathematics and Artificial Intelligence , 70(3):185-202, 2014.

Alina Beygelzimer, Sham Kakade, and John Langford. Cover trees for nearest neighbor. In Proceedings of the 23rd international conference on Machine learning , pages 97-104, 2006.

Moise Blanchard. Universal online learning: An optimistically universal learning rule. In Conference on Learning Theory , pages 1077-1125. PMLR, 2022.

Moise Blanchard and Romain Cosson. Universal online learning with bounded loss: Reduction to binary classification. In Conference on Learning Theory , pages 479-495. PMLR, 2022.

Adam Block, Yuval Dagan, Noah Golowich, and Alexander Rakhlin. Smoothed online learning is as easy as statistical learning. In Conference on Learning Theory , pages 1716-1786. PMLR, 2022.

Fr´ ed´ eric C´ erou and Arnaud Guyader. Nearest neighbor classification in infinite dimension. ESAIM: Probability and Statistics , 10:340-355, 2006.

Kamalika Chaudhuri and Sanjoy Dasgupta. Rates of convergence for nearest neighbor classification. Advances in Neural Information Processing Systems , 27, 2014.

Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions on information theory , 13(1):21-27, 1967.

Sanjoy Dasgupta. Consistency of nearest neighbor classification under selective sampling. In Conference on Learning Theory , pages 18-1. JMLR Workshop and Conference Proceedings, 2012.

Sanjoy Dasgupta and Samory Kpotufe. Nearest Neighbor Classification and Search , page 403-423. Cambridge University Press, 2021. doi: 10.1017/9781108637435.024.

Luc Devroye, Laszlo Gyorfi, Adam Krzyzak, and G´ abor Lugosi. On the strong universal consistency of nearest neighbor regression function estimates. The Annals of Statistics , 22(3):1371-1385, 1994.

Luc Devroye, L´ aszl´ o Gy¨ orfi, and G´ abor Lugosi. A probabilistic theory of pattern recognition , volume 31. Springer Science & Business Media, 2013.

Rick Durrett. Probability: theory and examples , volume 49. Cambridge university press, 2019.

Evelyn Fix and Joseph Lawson Hodges. Discriminatory analysis, nonparametric discrimination. USAF School of Aviation Medicine, Randolph Field, Texas, Project 21-49-004, Report 4, Contract AD41(128)-31 , 1951.

Mikhael Gromov, Misha Katz, Pierre Pansu, and Stephen Semmes. Metric structures for Riemannian and non-Riemannian spaces , volume 152. Springer, 1999.

Nika Haghtalab, Tim Roughgarden, and Abhishek Shetty. Smoothed analysis of online and differentially private learning. Advances in Neural Information Processing Systems , 33:9203-9215, 2020.

Nika Haghtalab, Tim Roughgarden, and Abhishek Shetty. Smoothed analysis with adaptive adversaries. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS) , pages 942-953. IEEE, 2022.

Steve Hanneke. Learning whenever learning is possible: Universal learning under general stochastic processes. Journal of Machine Learning Research , 22(130):1-116, 2021.

Steve Hanneke, Roi Livni, and Shay Moran. Online learning with simple predictors and a combinatorial characterization of minimax in 0/1 games. In Conference on Learning Theory , pages 2289-2314. PMLR, 2021.

Juha Heinonen. Lectures on analysis on metric spaces . Springer Science & Business Media, 2001.

MHolst and A Irle. Nearest neighbor classification with dependent training sequences. Annals of statistics , pages 1424-1442, 2001.

Tuomas Hyt¨ onen. A framework for non-homogeneous analysis on metric spaces, and the rbmo space of tolsa. Publicacions Matematiques , pages 485-504, 2010.

Sanjeev R Kulkarni and Steven E Posner. Rates of convergence of nearest neighbor estimation under arbitrary sampling. IEEE Transactions on Information Theory , 41(4):1028-1039, 1995.

Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Online learning: Stochastic and constrained adversaries. arXiv preprint arXiv:1104.5070 , 2011.

Tim Roughgarden. Beyond the worst-case analysis of algorithms . Cambridge University Press, 2021.

Daniil Ryabko. Pattern recognition for conditionally independent data. Journal of Machine Learning Research , 7(4), 2006.

Daniel A Spielman and Shang-Hua Teng. Smoothed analysis of algorithms: Why the simplex algorithm usually takes polynomial time. Journal of the ACM (JACM) , 51(3):385-463, 2004.

Charles J Stone. Consistent nonparametric regression. The annals of statistics , pages 595-620, 1977.

Michel Talagrand. Maharam's problem. Annals of mathematics , pages 981-1009, 2008.

A Proofs for Section 2

Proposition 2 (Non-convergence in the worst-case) . Let ( X , ρ ) be a totally bounded metric space. Given η : X → Y , there is a sequence of instances ( X n ) n on which the nearest neighbor rule is not online consistent on η if and only if there is no positive separation between classes:

glyph[negationslash]

Proof. Suppose that there is a positive separation between classes, so that margin η ( x ) > c > 0 for all x ∈ X . By Lemma 10, the collection of open balls B ( x, c/ 3) for all x ∈ X forms a cover of X by mutually labeling sets. Because X is totally bounded, there is a finite subcover of X by these mutually labeling balls. By Lemma 9, each of these sets admits at most one mistake, so the nearest neighbor rule makes at most finitely many mistakes, achieving online consistency.

On the other hand, suppose the class are not positively separated. We claim that there is a sequence of pairs ( X 2 n +1 , X 2 n +2 ) where a nearest neighbor of X 2 n +2 has a different label:

glyph[negationslash]

On this sequence X , the nearest neighbor rule makes a mistake at every even time step, so that:

glyph[negationslash]

It fails to be online consistent on ( X , η ) .

To prove the claim for a fixed n . First, define the minimal non-zero interpoint distance in X ≤ 2 n :

glyph[negationslash]

Let x, x ′ ∈ X have different labels and ρ ( x, x ′ ) < r/ 3 . There are two cases:

The points x and x ′ have distinct nearest neighbors ˜ x and ˜ x ′ in X ≤ 2 n . Then, the triangle inequality shows that:

Since ρ ( x, x ′ ) < r/ 3 , we obtain that:

This implies one of ρ ( x, ˜ x ) or ρ ( x ′ , ˜ x ′ ) is strictly larger than r/ 3 . Without loss of generality, let ρ ( x ′ , ˜ x ′ ) > r/ 3 . Then, set X 2 n +1 = x and X 2 n +2 = x ′ . In this case, X 2 n +2 has the unique nearest neighbor X 2 n +1 , which has a different label.

glyph[negationslash]

The points x and x ′ have the same nearest neighbor z in Z . Then, by exchanging x and x ′ if necessary, we have that η ( z ) = η ( x ′ ) . Set X 2 n +1 = x and X 2 n +2 = x ′ . The nearest neighbor of X 2 n +2 is either X 2 n +1 or z . Both have different label than X 2 n +2 .

B Proofs for Section 3

Lemma 5. Let X be uniformly dominated by ν at rate ε ( δ ) , and let ( F n ) n be its natural filtration. Let A n be an F n -predictable sequence where lim sup n →∞ ν ( A n ) < δ almost surely. Then:

Proof. The following sequence ( Z n ) n is a martingale-difference sequence:

We obtain:

where (a) converges to zero by the martingale law of large numbers, and (b) is bounded by ε ( δ ) because the uniform domination implies that:

For completeness, we include a version of the strong law of large numbers (SLLN).

Theorem B.1 (Strong law of large numbers for martingales, (Durrett, 2019, Exercise 4.4.11)) . Let ( M n ) n ≥ 0 be a martingale and let Z n = M n -M n -1 for n > 0 . If E [ Z 2 n ] < K < ∞ , then:

C Proofs for Section 4

Lemma 9. Let U be a mutually-labeling set for η . Let X be an arbitrary process. Then:

glyph[negationslash]

Proof. For the sake of contradiction, suppose that there are two times n < m for which the indicated event occurs. In particular, X n , X m ∈ U and a mistake was made on the latter point:

glyph[negationslash]

As both are contained in U , we have:

On the other hand, the margin of X m is upper bounded:

Together, they contradict the mutually-labeling property of U .

Lemma 10. For any 0 < r < margin η ( x ) / 3 , the ball B ( x, r ) is mutually-labeling for η .

glyph[negationslash]

Proof. Let x ′ ∈ B ( x, r ) . Since x ′ is within the margin of x , they must share the same label. For any point z from a different class η ( z ) = η ( x ) , we obtain by the definition of the margin of x :

Subtract ρ ( x, x ′ ) from both sides and apply the triangle inequality ρ ( x, z ) -ρ ( x, x ′ ) ≤ ρ ( x ′ , z ) :

Finally, taking infimum over all points z in other classes implies:

On the other hand, the diameter of the ball B ( x, r ) is at most 2 r < 2 · margin η ( x ) / 3 . Combining this with the above proves that B ( x, r ) is mutually labeling.

D Proofs for Section 5

Proposition 13 ( F 0 is dense in L 1 ) . Let ( X , ρ, ν ) be a metric measure spaces where ρ is doubling and ν is a finite Borel measure. Then, the set F 0 is dense in L 1 ( X , ν ) .

The proof of this follows the standard approach: for any η ∈ L 1 ( X , ν ) , we can construct a countable partition of almost all of X such that η overwhelmingly assigns each partition the same label, by which we mean up to a δ -fraction of that partition. The existence of such a partition requires a version of the Lebesgue differentiation theorem. The theorem in basic settings applies to doubling metric measure spaces (e.g. Heinonen (2001)), in which all balls satisfy the doubling property:

As our setting only assumes a doubling metric space with a finite Borel measure (not necessarily a doubling measure), this property is not guaranteed to hold. However, Hyt¨ onen (2010) shows that there are still a plethora of balls that satisfy such a property, which is defined as:

Definition D.1 (Doubling ball, Hyt¨ onen (2010)) . Let ( X , ρ, ν ) be a metric measure space. We say that a ball B ( x, r ) in X is ( α, β ) -doubling for some α, β > 1 whenever:

To prove that F 0 is dense in L 1 , we apply a version of the Lebesgue differentiation theorem restricted to (5 , β ) -doubling balls in a space (Theorem D.2). This is followed by a corresponding Vitali covering theorem for such balls (Theorem D.3). With these ingredients, the proof is routine:

Proof. Let η ∈ L 1 ( X , ν ) and fix δ > 0 . We show that we can construct a function η ′ ∈ F 0 with negligible boundary such that ‖ η -η ′ ‖ L 1 ( X ,ν ) < δ .

To do so, we make use of a version of the Lebesgue differentiation theorem for doubling spaces with finite Borel measures, Theorem D.2. It states almost every x ∈ X has a positive sequence r n ↓ 0 such that the following Lebesgue differentiation condition is satisfied:

glyph[negationslash]

and where all balls B ( x, r n ) are (5 , β ) -doubling. Let's denote by Leb( η ) the set of points satisfying this Lebesgue condition. In this case, for each x ∈ Leb( η ) , we can choose a ball B ( x, r x ) that is (5 , β ) -doubling and whose labels overwhelming agree with η ( x ) :

glyph[negationslash]

By Theorem D.3, which is a version of the Vitali covering lemma for collections of (5 , β ) -doubling balls, there is a countable disjoint subfamily of balls C ⊂ { B ( x, r x ) : x ∈ Leb( η ) } that covers almost all of X . Enumerate the balls in C by B ( x i , r i ) , and define the function:

By the dominated convergence theorem, we have that ‖ η -η ′ ‖ L 1 ( X ,ν ) < δ . Furthermore, points in the balls B ( x i , r i ) are not boundary points; η ′ has negligible boundary.

Theorem D.2 (Lebesgue differentiation theorem, Corollary 3.6, Hyt¨ onen (2010)) . Suppose that ( X , ρ, ν ) is a metric measure space where ρ is a doubling metric and ν is a finite Borel measure. There exists β > 1 such that the following holds. Let η ∈ L 1 ( X , ν ) be bounded. For ν -almost every x ∈ X , there exists a positive sequence r n ↓ 0 such that:

and the balls B ( x, r n ) are (5 , β ) -doubling.

Theorem D.3 (Vitali covering theorem, Theorem 1.6, Heinonen (2001)) . Let ( X , ρ, ν ) be a doubling metric space with a finite measure. Let β > 1 . Let A ⊂ X be any set and let F be any family of balls such that for each x ∈ A , there exists some sequence r n ↓ 0 where:

Then, there is a countable disjoint subfamily of F that covers ν -almost all of A .

This is a slightly more general version of the Vitali covering theorem than given by Heinonen (2001), although it is implied by the proof given there. In that proof, the sequence of (5 , β ) -doubling balls was deduced from a stronger condition; here, we directly assume its existence.

Lemma D.4. Let ( X , ρ, ν ) be a metric measure space where ν is a finite Borel measure. Suppose that X is a process that is uniformly dominated by ν at rate ε ( δ ) . For any ε > 0 , there exists a region X ′ ⊂ X with bounded diameter such that:

Proof. Fix 0 < ε < 1 . Let δ > 0 be sufficiently small so that ε ( δ ) < ε . Let x ∈ X . For any positive sequence r n ↑ ∞ , the balls converge B ( x, r n ) ↑ X . As ν is finite, by the continuity of measure, there is sufficiently large r n such that ν ( B ) > 1 -δ , where B = B ( x, r n ) . Let X ′ = 3 B .

Suppose that τ = min { n : X n ∈ B } is the first time that X n hits the ball B . By Lemma 10, the ball B has the mutually-labeling property for the function ✶ 3 B . Thus, for n > τ , whenever the nearest neighbor falls outside of 3 B , this implies that X n falls outside of B . It follows that the chain holds:

And as we also have { X n / ∈ 3 B } ⊂ { X n / ∈ B } , the bound follows:

The stopping time τ is almost surely finite. The Borel-Cantelli lemma shows that τ is almost surely eventually bounded above by some n ∈ N , since the following sum converges:

Thus, the asymptotic rate of X or ˜ X escaping X ′ is bounded by the rate at which X avoids B . By Lemma 5, this is at most ε .

Theorem 14 (Ergodic continuity of nearest neighbor processes) . Let ( X , ρ, ν ) be a upper doubling space with bounded diameter. Suppose that a process X is uniformly dominated by ν at a rate ε ( δ ) . There exists constants c 1 , c 2 > 0 such that for any measurable set A ⊂ X with ν ( A ) < δ 0 ,

Proof of Theorem 14. Fix any measurable set A ⊂ X where ν ( A ) < δ 0 . Define the indicator process I by I n = ✶ { X n ∈ A } . Lemma 5 implies that I is asymptotically ε ( δ 0 ) -rate-limited. Then, Theorem 17 immediately implies that there are constants c 1 , c 2 > 0 such that:

Optimizing the bound implies the result.

E Proofs for Section 6

Theorem 17 (Long-term influence bound) . Let ( X , ρ, ν ) be a bounded, upper doubling space. There are constants c 1 , c 2 > 0 so that the following holds. Let X be uniformly dominated at rate ε ( δ ) and let I be an indicator process adapted to X asymptotically rate-limited by γ > 0 . For any δ > 0 , the rate that the indicated instances X [ I <n ] contain a nearest neighbor ˜ X n is at most:

Proof of Theorem 17. By rescaling, we assume without loss of generality that X has unit diameter. Recall from Definition 15 that given an indicator process I , we defined k ( n ) to be the counter for the number of indicated instances up through time n , and τ k to be the time of the k th indicated instance. In particular, we have by assumption that lim sup k ( n ) /n < γ almost surely.

n

→∞

Let ( C k ) k be a chain of sequentially-constructed cover trees associated to the sequence of indicated instances A = ( X τ k ) k . Let L k be the insertion rank of the k th indicated instance X τ k . Recall that this means that C k is constructed by taking a union of C k -1 with the dyadic cone centered at X τ k of rank L k . For the k th indicated instance and for all time n ≥ τ k , define the tail rank T k,n to:

where ( c, d ) are parameters associated to the upper doubling condition, and G k,k ( n ) is the number of generations of children it has by time n . Let's define the δ -tail of the cover tree at time n to be the union of the rank-( T k,n +1) tails of X τ k and let A n be the union of the doubled balls:

By Lemma E.1, the mass of the tail is ν ( A n ) < δ . We now apply Lemma 25 to bound the number of times an indicated instance is a nearest neighbor. To do so, we need that X n is almost never contained in X [ I <n ] . This follows from the upper doubling condition, which implies that singleton sets have zero mass. By uniform domination, X n is almost surely never equal to a previous instance. And so, Lemma 25 implies that:

and that almost surely:

We bound (a) and (b) separately:

(a) The first term is bounded by Lemma 20:

The number of balls in C k ( N ) \T N can be computed by counting for each indicated instance the number of balls not in its ( T k,N +1) -tail:

where the last inequality holds because the total number of generations of children is no more than the number of nodes in the tree. As the indicator process is asymptotically rate-limited by γ , we obtain:

(b) Since the mass of the tail is bounded ν ( A n ) < δ , the second term can be asymptotically controlled on average by Lemma 5,

Summing these bounds and choosing c 1 = 2 2 d ( 3 + 1 d lg c ) and c 2 = 2 2 d d completes the proof.

Lemma E.1 (Cover tree δ -tail) . Let ( X , ρ, ν ) be a upper doubling metric space with unit diameter. Let ( C k ) k be a chain of cover trees for the sequence A = ( a k ) k and let ( L k ) k be its sequence of insertion ranks. Define the array of tail ranks ( T k,n ) k ≤ n and tail sets ( A n ) n ,

where d is the doubling dimension and c is the upper doubling constant in Definition 11. For all n , the mass of the tail is bounded ν ( A n ) < δ .

Proof. Fix n and let S ⊂ A ≤ n a generation of children with rank L . This set of instances along with their parent forms a 2 -L packing of a 2 -L +1 ball. Therefore, | S | ≤ 2 d -1 since X is doubling.

We extend S to include all its descendants: let S be the subset in A ≤ n that contains S and has the property that if x ′ is a child of x ∈ S , then x ′ ∈ S . Let the tail centered at S be the following union:

In particular, when S = { a 1 } , then L = 0 and S = { a 1 , . . . , a n } . It suffices to show:

We proceed by induction on the rank L in decreasing order.

Base case: let L be the maximal rank of any instance in A ≤ n and let S be any generation of children with maximal rank. These instances have no further descendants, so S = S . For each a k ∈ S , we also have G k,n = 0 . By a union bound:

where the first inequality uses the measure condition of upper doubling spaces, and the second inequality follows from our choice of T k,n and that | S | < 2 d .

Inductive step: suppose that the claim holds for all generations with rank glyph[lscript] > L . Let S be any generation with rank L . For each a k ∈ S , let G k,n be the collection of its generations of children, which all have rank strictly greater than L . When S ′ ∈ G k,n is a generation, let S ′ extend S ′ with its descendants and L ( S ′ ) be the rank of S ′ . Then:

By the inductive hypothesis, we have:

where (i) follows by union bound, (ii) by the inductive hypothesis, (iii) by the geometric series formula, and (iv) by the upper bound that | S | < 2 d -1 .

Lemma 20 (Packing bound) . Let ( X , ρ ) be a metric space, U ⊂ X be a subset, and r > 0 . For any process X , the number of ( U, r ) -separated events is bounded by the r -packing number of U ,

Proof. Let Z ⊂ X be the collection of instances X n ∈ U with nearest neighbor distance at least r :

Fix n such that n ∈ Z . For all X m ∈ Z where m<n , we have:

For all X m ∈ Z where m>n , we also have:

Thus, Z is an r -packing of the set U , and so | Z | ≤ P r ( U ) .

Lemma 25 (Cover tree decomposition) . Let ( X , ρ ) have unit diameter, let X be a process in X , and let I be an indicator process. For any n , let C be a cover tree for X [ I <n ] with a cover-tree neighbor map c . Assume that X n / ∈ X [ I <n ] is not equal to one of the indicated instances. Then:

In particular, the event within the union indexed by B = B ( a, r ) ∈ C is contained in E 2 B,r/ 2 n .

Proof. Let c k ( X n ) = B r be an r -ball. Then, by Equation (7), the definition of a cover-tree neighbor, we have that X n is r/ 2 -separated from X [ I <n ] . When the indicated instances contain a nearest neighbor, then X n is also r/ 2 -separated from X <n . By the other condition of a cover-tree neighbor, we have that X n ∈ 2 B r . Together, this implies the (2 B r , r/ 2) -separated event.

F Rates of convergence for smoothed processes

In this section, we show that our proof techniques can be extended to obtain rates of convergence for uniformly dominated processes. Recall that our basic strategy in Section 4 was to decompose X into two pieces: (i) a region that is a finite union of mutually-labeling sets and (ii) a remainder A δ with small mass ν ( A δ ) < δ . When η ∈ F 0 has negligible boundary, then δ can be taken to be arbitrarily small: this is possible because points that cannot be covered by mutually-labeling sets are boundary points, which have measure zero. We can adapt this idea to yield rates by quantifying the number of mutually-labeling sets required to cover all but a region of δ -mass. To this end, we define:

Definition F.1 (Mutually-labeling covering number) . Let V ⊂ X . The mutually-labeling covering number N ML ,η ( V ) for η is the size of a minimal covering of V by mutually-labeling sets of η .

Then, an expected mistake bound on a process X that is uniformly dominated at a rate ε ( δ ) is obtained by separately counting mistakes on V and V c , where we bound mistakes (i) on V by its mutually-labeling covering number and (ii) on V c by its ν -mass:

glyph[negationslash]

Theorem F.2 (Convergence rate) . Let ( X , ρ, ν ) be a metric measure space with separable metric ρ and finite Borel measure ν . Let η be measurable and let X be uniformly dominated by ν at rate ε ( δ ) . Fix p > 0 . With probability at least 1 -p , the following holds simultaneously for all N ∈ N :

glyph[negationslash]

Of course, this bound could be vacuous, for example if every point of η is a boundary point. In the following, we restrict ourselves to a setting with stronger stochastic and geometric assumptions. This allows us to provide quantitative bounds on the two terms N ML ,η ( V ) and ε ( ν ( V )) in Theorem F.2. For this stronger setting, we obtain a convergence rate Theorem F.5 by balancing these two terms.

F.1 Convergence rate for smoothed processes in length metric spaces

Here, we consider a class of uniformly dominated processes with a stronger condition that is often studied in smoothed online learning . They satisfy Lipschitz continuity with a rate ε ( δ ) = σ -1 · δ :

Definition F.3 (Smoothed process) . Let σ > 0 . A process X is σ -smoothed with respect to ν if:

We also work in length spaces (Definition G.8 or Gromov et al. (1999)), in which distances between points are given by the infimum of lengths over continuous paths between those points. The appealing property of length spaces is that the margins of points are equal to the distance to the boundary:

Lemma F.4 (Margin in length spaces) . Let ( X , ρ ) be a length space. Let η be a function. Then,

In this case, it is natural to restrict V ⊂ X in Theorem F.2 to the sets of the form:

These are the set of points whose margin is at least r . Then, we need to control the mutual-labeling covering number of V r and the ν -masses of V c r . When X is a length space, these can be bounded in terms of the geometry of the boundary ∂ X . The reason is that in length spaces, points with small margins are also close to boundary points: here, V c r precisely coincides with the r -expansion ∂ X r of the boundary. And when X is a doubling space, we can quantify the bounds in terms of the box-counting dimension b ( ∂ η X ) and the Minkowski content m ( ∂ η X ) of the boundary.

In particular, Proposition G.9 shows that for small r ,

where the hand-waving inequality can be made rigorous by replacing b = b + o (1) and m = m + o (1) . For example, this yields convergence rates on smoothed processes, by plugging Equation (9) into Theorem F.2. For simplicity, assume that b > 1 so that the O ((2 N log N ) 1 / 2 ) term is lower-order. After optimizing r , we obtain the following result:

Theorem F.5 (Convergence rate for smoothed processes) . Let ( X , ρ, ν ) be a bounded length space with a doubling metric and finite Borel measure. Let η ∈ F 0 . Let X be a σ -smoothed process. Suppose that the boundary ∂ η X has box-counting dimension b > 1 and Minkowski content m . For any choice of c 1 , c 2 , p > 0 , there exists constants C 0 , C 1 > 0 such that with probability at least 1 -p , the following holds simultaneously for all N ∈ N :

glyph[negationslash]

G Proofs for Appendix F

Proof of Lemma F.4 To show that m c ( x ) = ρ ( x, ∂ X ) , we prove left and right inequalities.

First, the margin is upper bounded by m c ( x ) ≤ ρ ( x, ∂ X ) . To see this, fix δ > 0 . By the definition of the distance between x and the set ∂ X , there is a boundary point z ∈ ∂ X such that:

And as boundary points are arbitrarily close to at least two classes, there exists x ′ ∈ X close to z :

while also belonging to a different class than x . By the definition of m c ( x ) and by triangle inequality, we obtain that for all δ > 0 , there exists some x ′ satisfying:

Letting δ go to zero yields the first inequality.

For the other, we claim that if γ : [0 , 1] →X is a continuous path from x to x ′ with c ( x ) = c ( x ′ ) , then there exists a point γ ( t ) contained in ∂ X . If the claim is true, then the other inequality holds:

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

where (i) the infimum above is taken over all continuous paths γ from x to x ′ , (ii) applies the definition of a length space, and (iii) applies the definition of the margin.

To prove the claim, let t be the first time a point on the path has a different label than x . Formally, glyph[negationslash]

To show that γ ( t ) ∈ ∂ X , we need to exhibit a point γ ( s ) that is δ -close to γ ( t ) with a different label, given any δ > 0 . Indeed, such a s exists by the definition of t and the continuity of γ .

To obtain bounds on the mutually-labeling covering number N ML ( V r ) , we need to introduce the notion of the box-counting dimension a set A ⊂ X and the doubling dimension of a metric space X . Let us first recall the following definitions and results from analysis and measure theory.

Definition G.1 (Covering number) . Given r > 0 and A ⊂ X , the r -covering number N r ( A ) of A is size of a minimal covering of A by balls with radius r .

Definition G.2 (Box-counting dimension) . The (upper) box-counting dimension of A ⊂ X is:

The box-counting dimension implies a bound on the covering number N r ( A ) of r -b ( A )+ o (1) . The following lemma is a straightforward conversion of the asymptotic limit into a quantitative bound.

Lemma G.3 (Box-counting upper bound on N r ) . Let X be bounded with diameter R . Let A ⊂ X have box-counting dimension b ( A ) . Then, for all c > 0 , there exists a constant C > 0 such that:

Proof. Fix c > 0 . By the definition of b ( A ) , there exists r 0 > 0 such that whenever 0 < r < r 0 ,

Because N r ( A ) is non-increasing in r , we can extend the bound to all 0 < r < R ,

where r ∧ r 0 := min { r, r 0 } . In fact, we have min { r, r 0 } > r · r 0 /R , and so:

To finish the proof, it suffices to let C = ( r 0 /R ) -( b ( A )+ c ) .

Lemma G.4 (Lemma 2.3, Hyt¨ onen (2010)) . Let ( X , ρ ) be doubling with doubling dimension d . There exists a constant C > 0 such that for all balls B ( x, r ) , the covering number is bounded:

To obtain bounds on the mass ν ( V c r ) , we need to introduce the Minkowski content of a set A ⊂ X . First, recall that the r -expansion of a set A fattens the set to all points of distance within r of A :

Definition G.5 ( r -expansion) . Let A ⊂ X be a set and r > 0 . The r -expansion A r of A is:

The Minkowski content of A is the rate at which an infinitesimal fattening of A increases its mass: Definition G.6 (Minkowski content) . Let (upper) Minkowski content of A ⊂ X is:

The following lemma bounding the covering number of the r -expansion of a set in terms of the doubling dimension will also be helpful:

LemmaG.7 (Covering the r -expansion of a set) . Let ( X , ρ ) have finite doubling dimension d . There exists a constant C > 0 such that for all A ⊂ X , we have:

Proof. Let A be covered by the balls B ( x 1 , r ) , . . . , B ( x n , r ) where n = N r ( A ) . Then, by the triangle inequality, the r -expansion A r is covered by the r -expanded balls, B ( x 1 , 2 r ) , . . . , B ( x n , 2 r ) . Now, by Lemma G.4, each expanded ball B ( x i , 2 r ) can be covered by C 2 d balls with radius r . It follows that covering A r needs at most C 2 d n balls with radius r .

G.1 Bounding geometric quantities of ∂ η X

Definition G.8 (Length space) . A metric space ( X , ρ ) is a length space if for all x, x ′ ∈ X ,

where γ : [0 , 1] →X include all continuous paths from x to x ′ and glyph[lscript] ( γ ) is the length of the path γ .

Proposition G.9 (Geometric quantities of ∂ X ) . Let ( X , ρ, ν ) be a bounded length space with finite doubling dimension and Borel measure. Let η ∈ F 0 . Then, for any c 1 , c 2 > 0 , there is a constant C > 0 and r 0 > 0 so that for all 0 < r < r 0 ,

Proof of Proposition G.9 Recall that V r and ∂ X are defined in terms of the margin:

While the complement V c r always contains the expansion ∂ η X r , generally V c r can be much larger. But when X is a length space, equality holds:

Lemma G.10. Let ( X , ρ ) be a length space. Then, for all r > 0 :

Proof. Lemma F.4 shows that when X is a length space, margin η ( x ) = ρ ( x, ∂ η X ) . Thus:

The question of bounding N ML ,η ( V r ) and ν ( V c r ) becomes that of N ML ,η ( X \ ∂ η X r ) and ν ( ∂ η X r ) . Proposition G.11 (Upper bound on N ML ) . Let X be a bounded length space with finite doubling dimension Γ and diameter R . For η ∈ F 0 , let b be the box-counting dimension of ∂ η X . Then, for any c > 0 , there exists a constant C > 0 such that for all r > 0 :

Proof. We can write X \ ∂ η X r as a union of layers of the form L k := ∂ X 2 k +1 r \ ∂ X 2 k r ,

Then, we can upper bound the mutually-labeling covering number by the sum:

To upper bound N ML ( L k ) , first note that by Lemma G.10,

Thus, the margin of any point x ∈ L k is at least 2 k r . By Lemma 10, the ball B ( x, 2 k r/ 3) is a mutually-labeling set, so that N ML ( L k ) ≤ N 2 k r/ 3 ( L k ) . In fact, we obtain the following:

where (i) holds because the radius 2 k -2 r is less than 2 k r/ 3 , (ii) follows because ∂ X 2 k +1 r contains L k and so has larger covering number, (iii) makes use of the definition doubling dimension three times to convert the 2 k -2 -covering number to a 2 k +1 -covering number, (iv) applies Lemma G.7 to convert the covering number of the expansion to that of the boundary set, and (v) upper bounds the covering number in terms of the box-dimension of ∂ X by Lemma G.3.

By combining Equations (10) and (11), we obtain:

where the geometric series converges to 2 -( b + c ) . We finish by relabeling the constants.

This shows that N ( V r ) = r -( d ( ∂ X )+ o (1)) . Next we show that ν ( V c r ) = ( m ( ∂ X ) + o (1) ) · r . This is immediate from the definition of the Minkowski content m ( ∂ X ) .

Proposition G.12 (Upper bound on ν ) . Let ( X , ρ, ν ) be a metric Borel space. Let η ∈ F 0 whose boundary ∂ η X has Minkowski content m . Then, for any c > 0 , there exists some r 0 such that for all 0 < r < r 0 :

Proof. Since the boundary has measure zero, the definition of Minkowski content states that there exists r 0 > 0 so that for all 0 < r < r 0 ,

The result follows by multiplying through by r .

Together, Propositions G.11 and G.12 prove Proposition G.9.

G.2 Proofs of convergence rates

Proof of Theorem F.2 Fix V ⊂ X . Let X be uniformly dominated. Denote by A n ⊂ X region on which the nearest neighbor rule makes a mistake at time n . We can count the total number of mistakes separately on V and V c :

glyph[negationslash]

At most one mistake can be made per mutually-labeling set on V , so the first summation can be bounded by N ML ( V ) . The second term can be bounded by the number of times X n hits V c :

By Azuma-Hoeffding's, we have that with probability at least 1 -p/ 2 N 2 :

Because the process is uniformly dominated, we also have E [ ✶ { X n ∈ V c }|F n -1 ] < ε ( ν ( V c ) ) . By taking a union bound over all N ∈ N , we obtain that with probability at least 1 -p , glyph[negationslash]

The result follows from optimizing V , and by noting at most N mistakes can be made in N time. glyph[squaresolid]

glyph[squaresolid]

Proof of Theorem F.5 Given c 1 , c 2 > 0 , Proposition G.9 yields C, r 0 > 0 so that when 0 < r < r 0 ,

From Theorem F.2, it follows that with probability at least 1 -p , we have for all T :

glyph[negationslash]

where r is optimized at:

provided that r ∗ N < r 0 . This will eventually hold for sufficiently large N > N 0 . For N ≤ N 0 , we can use the coarser mistake bound N 0 . Thus, for all N ∈ N :

glyph[negationslash]

Because we assumed b > 1 , the √ N log N term is eventually dominated by the N ( b + o (1)) / ( b +1) term when N > N ′ 0 is sufficiently large. We obtain the result by setting C 0 as below, and noting that we can simplify the exponent because ( b + c 1 ) / ( b + c 1 +1) < ( b + c 1 ) / ( b +1) .

glyph[squaresolid]

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: There are two prior work on the consistency of the nearest neighbor rule in the realizable setting. Both require much stronger assumptions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The assumptions used throughout are clearly described in the theorem statements. Additional limitations are addressed in the related work section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: To our knowledge, the paper is complete and correct.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This is a theoretical work.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This is a theoretical work.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This is a theoretical work.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This is a theoretical work.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This is a theoretical work.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The Code of Ethics do not seem to apply to this work.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work focuses on extending our theoretical understanding a long-existing algorithm.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This is a theoretical work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This is a paper did not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No crowdsourcing or experimentation with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of non-uniform consistency of the nearest neighbor prediction rule when the instances are generated by a random process that has conditional marginals dominated by some underlying reference measure. In particular, it generalizes the well-known results of Cover and Hart (1967) for iid processes and the result of Kulkarni and Posner (1995) for arbitrary processes with strong assumptions on labeling functions. The paper shows that under the uniform dominance condition of the process, the NN rule is consistent for any measurable function, which is further extended to finite sample rates with additional assumptions on both the instance space and processes.\n\nSTRENGTHS:\nI believe studying the consistency of the nearest neighbor prediction rule under a general random process is an important problem for understanding its limitations. This paper makes a significant step toward such a goal. From a philosophical perspective, this work is analogous to the recent line of work in beyond worst-case online learning in the infinite consistency paradigm. The paper also develops several original techniques, such as the control of the nearest neighbor process in Theorem 15, which may be of independent interest.\n\nOverall, I think this is an interesting paper and is suitable for publication at NeurIPS.\n\nWEAKNESSES:\nMy main complaint about this paper is its writing style. The paper leans too much on geometric concepts (such as doubling metric spaces, length spaces, box-counting dimension, and Minkowski content), which, in my opinion, are not necessary to demonstrate the learning-theoretical insights and place an undue burden on the reviewers. Why not just focus on the Euclidean space in the main text and leave the generalizations for the appendix or a journal version?\n\nI have also the following specific comments:\n\n1. Can the authors comment on why all the main theorems assume uniform dominance rather than the more relaxed ergodic dominance? My understanding is that the only place where uniform dominance is used is in the Borel-Cantelli argument in the proof of Lemma D.4. Am I right?\n\n2. To be honest, I don't quite understand how the \"sequentially-constructed cover tree\" works. The sketch from lines 237-253 doesn't really help too much. Are the whole arguments merely to improve the $1/\\delta$ to $\\log(1/\\delta)$? Why can't one simply extend the argument in Example 17 using multiple indicators, if the goal is only consistency?\n\n3. I would recommend the authors move part of Section 7 to the appendix and put more effort in the main text on explaining the proof of Theorem 15.\n\n4. Can the authors comment on how the results from Blanchard (2022) compare to your results? Does the kC1NN rule proposed therein achieve consistency in your setting?\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the consistency (or mistake-bound) of the nearest neighbour rule in the realizable online setting when the instances are not necessarily i.i.d. but drawn from a well-behaved stochastic process. The authors prove that when the underlying stochastic process is uniformly dominated, the nearest neighbour rule is consistent for labeling functions that have negligible boundaries. To prove this they introduce the notion of labeling functions on mutually labeling sets, which they then generalize to labeling functions on upper doubling metric measure spaces. Finally they also give convergence rates for smooth stochastic processes, which is a special case of the uniformly dominated setting considered in the paper.\n\nSTRENGTHS:\nThe central question of this paper is very well-motivated and an important question in the online learning literature. The paper manages to address the gap of online consistency of the nearest neighbour rule, and outline various interesting conditions for settings in which nearest neighbours rule is consistent. The paper is also quite well written.\n\nWEAKNESSES:\nOne comment I have regarding the technical novelty is that the proof techniques in this paper are fairly standard across the literature on online learning and consistency of nearest neighbours. I mention this in case the authors would like to highlight something that is not actually a standard technique. This comment does not affect the score.\n\n1. It seems that the paper only proves results for the $1$-NN setting. See point 2 of the questions section.\n2. Section 6 seems to serve expository purposes. See point 4 of the questions section.\n\nQUESTIONS:\n1. The authors mention on line 307 that \"Dasgupta (2012) ....... considered consistency for non-online settings....\" This seems to be a gross oversimplification. I would like a clarification on this point. \nDasgupta (2012): Consistency of nearest neighbor classification under selective sampling, COLT 2012.\n\n2. Do the proof techniques extend naturally to the $k$-NN setup or the $k_n$-NN setup? This is not immediately clear to me from the paper.\n\n3. Do the proof techniques extend naturally to the agnostic case where there is no underlying labeling function but instead $Z_n=(X_n,Y_n)$ are jointly drawn according to some underlying stochastic process $Z=(Z_n)_{n>0}$? The proof techniques would then boil down to computing mistake bound against the Bayes optimal classifier in this case.\n\n4. I am not sure about the significance of Theorem 14 or Section 6, and would like a clarification on the following point. The notion of consistency for uniformly dominated stochastic processes should straightforwardly imply a notion of consistency for stochastic processes which are dominated by much weaker notions of convergence. I suspect that Theorem 15 can be generalized to stochastic processes which are mixing, and the parameter $\\gamma$ can be replaced by some function of the mixing coefficient.\n\nLIMITATIONS:\nThe paper does a good job at clearly specifying the assumptions used in the setup, but sometimes does not address settings which are not explicitly considered in the paper. Please refer to the Questions section for clarifications.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

comments: SUMMARY:
This paper studies the nearest neighbor rule in the realizable online setting and closely checks under what assumptions it can achieve online consistency, i.e. the mistake rate eventually vanishes as the number of samples increases. It proves that for all measurable functions in doubling metric spaces when the samples are generated by a uniformly absolutely continuous process with respect to an underlying finite, upper doubling measure, the online consistency is achieved.

STRENGTHS:
While the prior work showed the online consistency for nearest neighbor rule under much stronger assumptions (instances are i.i.d. or the label classes are well-separated), this paper improves the understanding of this problem by showing it in the uniformly dominated regime. This result potentially gives us an understanding of other algorithms when studying online learning beyond i.i.d. and smoothed adversary settings.

WEAKNESSES:
While at this point, the review does not notice any significant weakness that needs to be addressed, the writings of the paper could be a bit too technical.

QUESTIONS:
Why ergodic continuity is related to uniformly absolute continuity such that it is included in the proof?

LIMITATIONS:
Yes, the limitations are adequately addressed. There is no potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13