Title: Minimax Adaptive Online Nonparametric Regression over Besov spaces
Venue: NeurIPS 2025
Accepted: yes
Keywords: Online non-parametric regression

Adaptive learning

Wavelet-based methods

Besov spaces
Abstract: We study online adversarial regression with convex losses against a rich class of continuous yet highly irregular competitor functions,% prediction rules, 
modeled by Besov spaces $B_{pq}^s$ with general parameters $1 \leq p,q \leq \infty$ and smoothness $s > \tfrac{d}{p}$. 
We introduce an adaptive
wavelet-based algorithm that performs sequential prediction without prior knowledge of $(s,p,q)$, and establish minimax-optimal regret bounds against any comparator in $B_{pq}^s$.
We further design a locally adaptive extension capable of sequentially adapting to spatially inhomogeneous smoothness. This adaptive mechanism adjusts the resolution of the predictions over both time and space, yielding refined regret bounds in terms of local regularity. Consequently, in heterogeneous environments, our adaptive guarantees can significantly surpass those obtained by standard global methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles online learning in Besov space using a wavelet decomposition approach. It presents two online learning algorithms. The first can be thought of as a regression problem over a (truncated) wavelet basis containing 1 \"coarse\" subbasis and J-1 \"detail\" subbases. Each wavelet function corresponds to a cubic volume of the bounded domain built based on a doubling approach. The second algorithm builds upon this first to tackle heterogeneously smooth functions. The aforementioned cubic volumes are arranged into a tree so that predictors can be localized. They are then combined into a final prediction using specific weights, learnt jointly with the local predictors. Regret bounds and convergence results are derived for both algorithms, which are shown to adapt to the underlying target function.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper puts forward and analyzes minimax optimal algorithms for online approximation of Besov functions, including for functions with heterogeneous degrees of smoothness. While mostly clearly written, the density of the material in some parts warrants some improvements in clarity. In particular,\n\n- While the conclusion mentions the exponential dependence on the smoothness parameter, it would be good to also have a discussion on the space complexity of the algorithms on the dimension. This is admittedly an issue in many wavelet methods, but I wonder if some form of adaptive tesselation could help (i.e., using an adapted wavelet basis)?\n\n- It should be pointed out that a large portion of the complexity of the proposed algorithms is encapsulated in Assumption 1. In particular, the complexity of mirror descent updates is not taken into account in Table 2 (Appendix F). While they are polynomial in nature, the constants could be detrimental depending on the conditioning of the problem (potentially bringing another issue aside from the exponential dependence in smoothness).\n\n- Assumption 2 is also not easy to interpret and could use more contextualization to make the paper more self-contained. As it is, the algorithms presented in the paper are really more \"meta-algorithms.\"\n\n- This is particularly the case for Algorithm 2, which requires substantial work from the reader to parse. In particular with respect to the description of the local predictor and its relationship with the global predictor at scale $j_0$. For instance, line 284 states that the local expert predictor (at scale $j_0$) is the restriction of the global predictor (at scale $j_0$) to the chosen subcube with the additional modification that the scaling coefficients $\\alpha_{j0}$ are set to node specific coefficients $a_n$, which are not particularly clear (except for the fact that they \"are supported on a grid $\\mathcal{A}$ of precision $T^{-1/2}$\"). It is unclear at this point how (or whether) the basis coefficients are shared among predictors (or what the role of $a_n$ is in this context). It would also be helpful to more explicitly define other quantities fundamental to the procedure, such as the active index set $\\Gamma_{e,t}$.\n\nQUESTIONS:\nSee \"Strengths And Weaknesses\"\n\nLIMITATIONS:\nYes, the authors have discussed limitations of the work in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of online nonparametric regression where the hypothesis class is a general Besov space. An adaptive algorithm based on wavelet decomposition and parameter free online learning is first proposed to achieve the minimax optimal regret bound without knowing the parameters of the hypothesis class beforehand. Then, inspired by (Liautaud et al., 2025), an improved algorithm based on expert aggregation further improves the result to local adaptivity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The considered problem is a challenging one that meaningfully extend the setting from an earlier work (Liautaud et al., 2025) with a similar theme. \n\n- The technical development in this paper is solid as far as I can tell. \n\n- The results are quantitatively strong, achieving optimality across a number of settings and parameter regimes. \n\nWeaknesses\n\n- It is claimed in the paper that the proposed algorithm is the first adaptive algorithm to bridge wavelet theory with online nonparametric learning, which I think is not accurate. Online nonparametric regression is known to be related to the so called dynamic online learning problem as discussed in the related work section of Baby and Wang (2019), where there is no context but the comparing hypothesis can be time varying. This setting is a special case of online nonparametric regression where the feature space is just the sequence of time indices 1,...,T. Baby and Wang's works (2019,2020) combined wavelet theory with averaging and the VAW forecaster to achieve the optimal dynamic regret, extending earlier works on trend filtering to the sequential setting. More recently, Zhang et al (2023) combined wavelet with parameter free online learning to tackle the case with general convex loss functions, and their algorithm appears to be identical to Algorithm 1 proposed by the submission despite using discrete wavelets. The relation to these works needs to be thoroughly discussed. \n\n- The technical development of the paper closely parallels the earlier work of Liautaud et al (2025), and the novelty compared to that is not sufficiently highlighted. Quantitatively, the proposed algorithm can handle more general hypothesis classes than Liautaud et al's, but given the similarity of the two algorithms, it is currently less clear what is the key technical contribution and how significant is that. \n\nReferences\n\nBaby, Dheeraj, and Yu-Xiang Wang. \"Online forecasting of total-variation-bounded sequences.\" Advances in Neural Information Processing Systems 32 (2019).\n\nBaby, Dheeraj, and Yu-Xiang Wang. \"Adaptive online estimation of piecewise polynomial trends.\" Advances in Neural Information Processing Systems 33 (2020): 20462-20472.\n\nZhang, Zhiyu, Ashok Cutkosky, and Yannis Paschalidis. \"Unconstrained dynamic regret via sparse coding.\" Advances in Neural Information Processing Systems 36 (2023): 74636-74670.\n\nQUESTIONS:\n1. I did not understand the expert construction in Algorithm 2. Could the authors elaborate on it? The description of the algorithm is also dense, therefore if possible, I would suggest revising the writing to highlight the key idea. \n\n2. I am also wondering if Algorithm 1 can directly achieve local adaptivity without expert aggregation, or put in another way, whether the analysis of Theorem 1 is loose. Any comment or insight would be helpful.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

comments: SUMMARY:
This paper develops online learning algorithms that adapt to local irregularities of the function class played by the adversary. The algorithms are based on parameter free online linear optimization algorithms, applied to each component of a wavelet decomposition separately, then aggregated. The resulting bounds are adaptive to the besov norm of a comparator function. Theorem 1 gives a trichotemy showing that different rates are possible with the stated algorithms depending on the relationship between dimension and smoothness. Theorem 2 gives further bounds in the case of spatially inhomogeneous smoothness. The boudns match the minimax rates in a wide range of cases.

STRENGTHS AND WEAKNESSES:
The paper is well written and easy to follow. The work combines recent results in parameter free optimization with sophisticated techniques from applied maths in order to develop something broadly applicable and useful, as well as of theoretical interest.

QUESTIONS:
-

LIMITATIONS:
-
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies online nonparametric regression using a competitive framework, i.e., bounding regret compared to the best predictor in a given function class (here, given by a Besov space). Specifically, the paper proposes a wavelet-based Algorithm 1 that is simultaneously competitive with predictors from any Besov space, up to a predetermined smoothness level $S$. The paper then proposes a \"spatially adaptive\" Algorithm 2, which uses a collection of experts  (each trained using Algorithm 1) assigned to different dyadic partitions of the space, and shows that Algorithm 2 obtains correspondingly stronger regret guarantees.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. Though dense, the paper is generally well-written and easy to follow. In particular, the paper does a good job both discussing demonstrative special cases/examples (e.g., Figure 3) and summarizing the results at a high level (e.g., with Figure 2).\n2. The proposed method does not require any strong knowledge of the data-generating process; in particular, it is impressive that the same algorithm is competitive with predictors from any Besov space with $s < S$.\n\n### Weaknesses\n1. As someone used to nonparametric regression in an offline, statistical/stochastic context and unfamiliar with the online learning setting, it took some adjustment to follow the paper's ideas, such as the \"competitive\" evaluation framework. The competitive framework seems designed to optimize prediction performance (under a particular loss function) while minimizing the assumptions made the data-generating process. However, it's unclear to me how much one can infer about the underlying data-generating process from this approach. For example, if we assume that the data is generated from a Besov-smooth function (i.e., $f(x) = \\mathbb{E}[Y|X = x]$ for some $f \\in B_{p,q}^s$) and train a predictor $\\hat f$, what do the given bounds tell us about $f$ (perhaps with some additional assumptions on the noise)?\n2. The main paper doesn't really discuss the proofs of the main results at all. I understand that space is limited, but a few sentences would be nice to give an idea of the key ideas/overall approach.\n\nQUESTIONS:\n1. What exactly is meant by the term \"parameter-free\" used throughout the manuscript? Does this refer to the fact that the parameters $s$, $p$, etc. need not be available to the algorithm? How does this differ from the term \"adaptive\" used throughout the paper and elsewhere in the statistical literature?\n\n2. A key result in the statistical literature on Besov function estimation (due to [13, 14, etc.]) is that linear estimators (e.g., simple projection or kernel estimators) necessarily fail to achieve minimax optimal convergence rates (under certain loss functions) over many Besov spaces, and that some nonlinear mechanism, such as thresholding, is needed. If I understand correctly, the construction of the active wavelet set plays a similar role here. Is there an analogous lower bound for this setting, e.g., where a simpler class of estimators that are competitive with Holder-smooth functions provably fail to compete with Besov-smooth functions? This might also be related to the spatial adaptivity advantages of Algorithm 2 over Algorithm 1, but I'm not sure if this is the same, since usually spatial adaptivity is within a single Besov space rather than across different (locally restricted) Besov spaces.\n\n3. The loss function is assumed to be both convex and Lipschitz; does this need to be true on the whole real line, or can this assumption be restricted to a bounded domain? On an unbounded domain, this is a lot more restrictive (i.e., the loss function is essentially linear outside some bounded region).\n\n4. Could the authors provide a bit more intuition for Assumption 1? It's not really clear to me what it means. Maybe an example (of the type of behavior it is restricting) would be helpful?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}