Title: Consistency of the $k_n$-nearest neighbor rule under adaptive sampling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Smoothed analysis

online learning

non-parametric estimation

consistency
Abstract: In the adaptive sampling model of online learning, future prediction tasks can be arbitrarily dependent on the past. Every round, an adversary selects an instance to test the learner. After the learner makes a prediction, a noisy label is drawn from an underlying conditional label distribution and is revealed to both learner and adversary. A learner is consistent if it eventually performs no worse than the Bayes predictor. We study the $k_n$-nearest neighbor learner within this setting. In the worst-case, the learner will fail because an adaptive process can generate spurious patterns out of noise. However, under the mild smoothing assumption that the process generating the instances is uniformly absolutely continuous and that choice of $(k_n)_n$ is reasonable, the $k_n$-nearest neighbor rule is online consistent.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors consider online binary classification under an adaptive sampling model (i.e., the sequence of prediction tasks can adapt to the sequence of observed labels). A KNN based procedure is proposed and is shown to be consistent (approach Bayes optimal performance) under the assumption of \"uniform absolute continuity\" of the data process (a type of smoothness condition) and appropriate choice of $k$. Under a stronger assumption of ``upper doubling\" on the metric space,  KNN is further shown to satisfy universal consistency for any measurable label function $\\eta$. It is also established that in the worst-case setting, the $k_n$-nearest neighbor rule can fail to be consistent.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, the results presented in the paper constitute solid theoretical contributions, that is generalized from previous work on i.i.d to the sequential and noisy label setting. The claims are sound as far as I can tell, and the paper is well-written with clear presentation. The problem itself is also of great interest to the ML community at large.\n\n\nMy main concern about this paper is its practicality. The distance calculation upon seeing every new observation, and the memory requirement for such a KNN-based algorithm is not scalable to high dimension and/or large data. In another direction: how robust is the choice of $k$ here, since the smoothness parameter $\\epsilon$ is almost impossible to know exactly. I also don't think conditions like upper doubling is easy to check in practice.\n\nThe paper can use some reorganization: the related work section should come earlier with a conclusion in the end. I find some parts a bit technical and better pushed to the appendix with sketches of the argument in the main text sufficient.\n\nQUESTIONS:\n- I'm a bit confused about Line 106. Are the labels not supposed to be random?\n\n- The results seem to me very geometry-tied, which may be an artifact of the particular algorithm under consideration. Is there a sense in which kNN is the best thing to do here? The setup is very close to a bandit / online optimization setting,  where the type of guarantees / assumptions / optimal algorithms are quite different from the proposal here.\n\nLIMITATIONS:\nYes it's adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the consistency of the k_n-nearest neighbor rule under an adaptive adversary and label noise. They show that the k_n-nearest neighbor rule is not conistent under such an adaptive adverasary. The authors complement this negative result by showing that one can recover consistency by placing additional mild assumptions on the adversary.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n - I think the results of the paper are interesting and provide a clear contribution to the existing literature on the consistency of k_n-nearest neighbors. \n- Moreover, I think this paper contributes nicely to the growing literature on beyond-worse-case analysis \n- Finally, I thought the Theorem statements were surprisingly clean and easy to parse.\n\nWeaknesses: My main gripe with this paper is its organization. While the authors do a nice job of summarizing the main results in Section 1.2, it was not immediately clear how the remaining sections map to these main results. Moreover, I often got lost reading through some of the (often verbose) prose wondering what the main point is and how it connects to the main contributions stated in Section 1.2. For example, instead of providing a proof sketch of Proposition 7, the authors provide a \"false proof of convergence.\" What is the point of lines 133-137?  What is the point of Section 3 how is it used in the proof of the main claims in Section 1.2?  Definition 12, Lemma 13, and Lemma 14 are not about k_n-nearest neighbor. How do these connect to the main claims and what is the point of Section 3.1? In my opinion, a paper should be focused on substantiating its main claims, with sections clearly labeled indicating which claim is being addressed. This is not the case with this paper, which to me, reads as if it were written from a stream of consciousness (e.g. lines 138-142).\n\nQUESTIONS:\n- See Weakness above for questions regarding organization\n- What are some examples of upper doubling spaces?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of asymptotically proper prediction of binary labels for online (adversarial) sampling from a given space. The authors provide a complex mathematical analysis of the algorithm based on $k_n$-Nearest Neighbor, for certain parameters $k_n$ depending on the sampling round n and for data generation process that, although allowing adversarial actions, it guarantees that intractable worst-cases generation patterns occur with negligible probability. Additionally, the authors claim that their analysis includes the presence of noise.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n-\tThe problem itself is among fundamental theoretical ML problems\n\n-\tMathematical analysis is quite complex\n\nCons:\n\n-\tThe novelty of the analyzed setting is unclear, comparing to the previous work. It looks like the main difference is noise, but I couldn’t find its formal definition and use in technical parts. I also couldn’t find a satisfactory comparison with the previous work(s)\n\n-\tThe result holds under certain assumptions, including Borel measurable spaces, continuous label distribution function (or measurable function in upper doubling space), etc. \n\n-\tThe authors assume that the reader is fluent in various aspects of mathematical statistics and topology\n\nDetail comments:\n\nThe title does not say anything about the actual angles of the paper – the problem, adversarial setting, noise, etc.\n\nThe concept of noise is not formally defined in the main paper, only by the means of examples. It is not clear what are real difference in the setting and techniques between this work and Dasgupta and So, NeurIPS 2024. The authors are not clear in referencing results to previous work, especially results stated without any proof. \n\nThe proof of the main Theorem 5 is somehow stopped and hand-waved closer to the end. I could follow the analysis (up to some presented level of details) up to Lemma 17, but Section 5 is far too sketchy.\n\nQUESTIONS:\n1.\tCompare results with previous work, focusing on differences.\n2.\tDefine noise formally.\n3.\tEmphasize novel techniques and their impact in the analysis.\n4.\tHow to obtain proofs of Theorems 2 and 5 from technical lemmas?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the consistency of $k_n$-nn classifiers under noise & adaptive (non-iid) data generation processes in general metric spaces. This continues a line of work on the consistency of the $k_n$-classifier under more challenging settings and belongs to the broader scope of online learning in between worst-case adversarial and iid . In particular this paper extends the results of Dasgupta and So (NeurIPS 2024) from 1-NN to general $k_n$-nn and from noiseless to benign noise. They have two main results:  $k_n$ is consistent (for specific choices of $(k_n)_n$) under a distributional assumption related to absolutely continuity if additionally\n\n1.  and the Bayes optimal classifier is continuous, or\n2. the metric space satisfies a certain doubling dimension based assumption.\n\nThey also discuss that under weaker assumptions consistency typically fails.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper studies an interesting and challenging problem. It greatly extends the work by Dasgupta and So (2024) on consistency results for 1-nn to $k_n$-nn. It fits nicely into the recent line of work on online learnability between adversarial/worst-case and iid (i.e., learning under general adaptive stochastic processes). \n\nThe structure of the paper is not ideal. On first sight the purpose of sections 2-5 is not very clear. Section 2 and 3 provide some more intuition on the challenges in this setting, Section 4 and 5 give a proof overview on the two main results. Perhaps try to clarify this.\n\nAlso the discussion of related work is rather limited. Please discuss more exactly how you relate to similar settings (e.g., Hanneke et al., 2021). In particular please discuss how your assumption relate to previous assumptions in other works (as you did for Haghtalab et al., saying you generalize it).\n\nQUESTIONS:\nConsider adding some corollaries for common settings (Euclidean etc.).\n\nYour assumption on $k_n$ in Equation (2) requires knowledge of $\\varepsilon$. It is of course not really realistic to know $\\varepsilon$. Is there some way to make it work without knowing $\\varepsilon$? E.g., guess it by some doubling trick etc.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}