Minimax Adaptive Online Nonparametric Regression over Besov spaces
Authors: Paul Liautaud
Pierre Gaillard
Olivier Wintenberger
Venue: NeurIPS 2025 spotlight ()
Keywords: Online non-parametric regression
Adaptive learning
Wavelet-based methods
Besov spaces
Primary area: general_machine_learning

Abstract: We study online adversarial regression with convex losses against a rich class of continuous yet highly irregular competitor functions,% prediction rules, 
modeled by Besov spaces $B_{pq}^s$ with general parameters $1 \leq p,q \leq \infty$ and smoothness $s > \tfrac{d}{p}$. 
We introduce an adaptive
wavelet-based algorithm that performs sequential prediction without prior knowledge of $(s,p,q)$, and establish minimax-optimal regret bounds against any comparator in $B_{pq}^s$.
We further design a locally adaptive extension capable of sequentially adapting to spatially inhomogeneous smoothness. This adaptive mechanism adjusts the resolution of the predictions over both time and space, yielding refined regret bounds in terms of local regularity. Consequently, in heterogeneous environments, our adaptive guarantees can significantly surpass those obtained by standard global methods.

Minimax Adaptive Online Nonparametric Regression over Besov spaces
Paul Liautaud Sorbonne Université, CNRS, LPSM F-75005 Paris, France
Pierre Gaillard
Université Grenoble Alpes, Inria CNRS, Grenoble INP, LJK 38000 Grenoble, France pierre. saiirreggiaaaaaiiiiaaa
paul Llaaattuu@rrrboee--ueeeeessteeee
Olivier Wintenberger
Sorbonne Université, CNRS, LPSM F-75005 Paris, France Institut Pauli, CNRS and University of Vienna
Oskar Makar-Morgenstem-PPlaze 1, 1090 Wien, Austria
olivier wttp::/wtteeeeeerrooooooeeeeeeaa
Abstract
We study online adversarial regression with convex losses against rich class of continuous yet highly irregular competitor functions.modeled by Besov spaces By with general parameters 1 & p. & oo and smoothness > -. We introduce adaptive wavelet-based algorithm that performs sequential prediction without prior knowledge of (s, (s, 9), and establish minimax-optimal regret bounds against any comparator in Byg. We further design locally adaptive extension capable of sequentially adapting to spatially inhomogencous smoothness. This adaptive mech- anism adjusts the resolution of the predictions over both time and space, yielding refined regret bounds in terms of local regularity. Consequently, in heterogencous environments, our adaptive guarantees can significantly surpass those obtained by standard global methods.
Introduction
We consider the online regression framework [5, 6]. where inputs æ1,..., It.... X arrive in a stream, and the task is to sequentially predict response fe(xx.) using an online predictive algorithm fe : X' -> IR based on past observations s = =,...., 1 and the current input æ., The goal is to design sequence of predictors (f:) in the competitive approach, i.e., with guarantees that hold uniformly over all individual (and potentially adversarial) data sequences. Prediction accuracy is assessed time using sequence of convex loss functions (f))131 for instance (x,,)x))) lf(( vil or (ff( 2 where Ye the observed response associated with æt After T > rounds, the performance of the algorithm is measured through its regret with respect to competitive continuous functions f.
T
Rr(f) = Σ (.(f((z))) - Ea(((r)))
t=1
Much of the early literature [7, 17, 23, 25] focuses on competitors f belonging smooth benchmark classes, such as Lipschitz or kernel-based functions. In this work, we extend this setting by designing constructive algorithms that are competitive with a much richer class of prediction rules, namely functions in general Besov spaces [20, 37, 40J. Building on wavelet-based representations, we design an adaptive algorithm that achieves optimal regret performance (1), against broad classes of prediction rules, modeled by Besov spaces. Wavelets [8, 11] are indeed a powerful and widely used tool for
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

capturing local features and regularities in signals. Their applications range from image segmentation and change point detection EEG analysis and financial time series. Moreover, in many practical scenarios, the environment may exhibit spatial heterogeneity, with varying degrees of regularity across the domain. This motivates the need for methods that can adapt locally to different smoothness levels. To tackle this challenge, we develop a locally adaptive algorithm [28, 29] that sequentially adjusts the resolution of its predictions over space, effectively adapting to inhomogeneous regularity. Our analysis provides minimax optimal regret guarantees that depend on the local smoothness of the target function, improving upon globally-tuned algorithms.
Wavelets and multiscale approaches. Classical wavelet-based methods for statistical function estimation have been primarily developed and analyzed in the batch (i.i.d.) setting, where the entire dataset is available upfront. Notable examples include the wavelet shrinkage procedure of [15], which achieves near-minn mar-mmiimmx estimation rates over Besov spaces. More generally, wavelets play a central role in the signal processing and compressed sensing framework developed by [30], where they are well understood and widely applied. In the context of adaptive and nonparametric estimation, [1, 2] introduced universal algorithms based on tree-structured approximations, closely related in spirit to wavelet thresholding. While these methods are computationally efficient and amenable to online implementation, their theoretical analysis is performed in the batch statistical learning setting and focuses on specific classes of approximation spaces. Multiscale and chaining ideas have also emerged in the online learning literature, beginning with the early work of [5] and continuing more recently in [34] (in a non-constructive fashion) and [17], although typically without relying on explicit wavelet constructions. Recently, [43] studied an online algorithm that combines discrete wavelets with parameter- parameter-free learning to minimize dynamic regret under general convex losses. Beyond this, the combination of wavelet-based representations with principled online nonparametric learning guarantees remains largely unexplored. Our work contributes to this direction by developing an online algorithm that leverages multiscale wavelet structures with theoretical regret guarantees over large nonparametric Besov function classes.
Online nonparametric regression. A classical line of work in online regression focuses on competing with smooth benchmark functions with a given degree of smoothness > 0. For instance, [7, 17, 29] design constructive online algorithms that achieve optimal regret against Lipschitz functions (s & 1) by using chaining-basse techniques and exploiting regularity properties such as uniform continuity to build refined predictors. Much of the early literature also focused on reproducing kernel Hilbert spaces (RKHS) [3, 4, 19, 25, 38], which correspond to the case where the smoothness index satisfies
and 2. This setting offers convenient geometric properties, such as inner products and representer theorems, but it excludes many natural function classes of interest, e.g. general (P'x') spaces with p 1, Sobolev spaces with low smoothness 8, or more generally Besov spaces. A key milestone in the direction of generalizing beyond RKHS is the work [40], which introduces the method of defensive forecasting to compete with wild prediction rules, i... rules drawn from general Banach spaces (e.g.. I''('), P > 2). Their framework shows that online learning is possible in highly irregular settings and provides regret bounds that depend on the geometry of the underlying Banach space. However, their analysis yields bounds that depend solely on the integrability parameter p > and does not account for any additional smoothness structure that the benchmark functions may possess. This motivates the need for online learning strategies that adapt not only integrability, but also to spatial regularity or smoothness. Another paper in this line is [42], where they study the performance of Sobolev kernels on restricted classes of Sobolev spaces W (X') with integrability p > and smoothness s > S. Going one step further, our paper proposes algorithm with regret guarantees against any competitor in general Besov spaces B for any integrability parameters & p.g & oc and smoothness s > g. This generalizes and improves upon previous methods by addressing a broad range of function spaces. More importantly, none of the constructive methods mentioned above provide the minimax optimal rate for generic Besov spaces, as established by [34]. To the best of our knowledge, we present the first constructive and adaptive algorithm that bridges wavelet theory with online nonparametric learning, while providing minimax optimal regret guarantees against functions in general Besov spaces. Table 1 summarizes our contributions and the corresponding regret rates in the literature.
Local adaptivity in inhomogeneous smoothness regimes. Many real-world functions exhibit spatially varying regularity, motivating the development of locally adaptive methods. In the batch setting, [14] pioneered spatially adaptive wavelet estimators that adjust to unknown smoothness. Bayesian approaches such as [36] further model locally Hölder functions with hierarchical priors. In

---

Table 1: Comparison of regret rates and parameter requirements for online regression.
Paper
Setting ((4) square losses, a > !
Input Parameters
Regret Rate
Vovk [39]
/ E P.g.
s.p, |1
T Lx
f C Bie p 2.9 (f
r
Vovk (40]
s.p, 11
/ W2 =66 p
T 3
/ = W,, p32,,
T Z
Gaillard and Gerchinovitz (17)
s.p. 11/
f c W2. p> 2.
/ E Wy, 232,33
T Pfatt
Zadorozhnyi et al. [42]
/ Wg. 2>2,
T
/ & Rip 1, 1. gorpe
This work- Alg.
f e p2..
T
distribution- free framework, [21, 27] introduced the notion of average smoothness, based on averaging local Hölder semi-norms at fixed degree of regularity. In the online setting, [28, 29] developed algorithms that sequentially adapt to local smoothness across time. However, these approaches typically focus on adapting to local norms while assuming a fixed degree of regularity. In contrast, our method jointly adapts to both the local regularity norm and the local smoothness exponent, enabling a data-driven compromise that is well suited to highly inhomogeneous environments.
Context and notation. Throughout the paper, domain of Rd d > 1. For any subset X" C X, we set its diameter as | = sup,, || vll Without loss of generality, we assume that X' is a regular hypercube of volume | We denote the horizon of time by T > 1. The sequence losses (4x) are assumed to be convex and G-Lipschitz for some G > O. For any natural integer E N, we denote [k] == (O...., k}.
assume the following. X' denotes compact
Background and function representation
X'' + IR that lie in (22) equipped with the standard inner product (f.g) fx f=)g((x) de. To design a sequential algorithm we rely on a multiscale representation of f based on an orthonormal wavelet basis. For a chosen starting scale ju E N, we write:
We consider compactly supported functions :
(2) where the families () and (dE,A))EAA,)33 form orthonormal basis of L2) We now highlight the key properties of the expansion (2), and refer the interested reader to Appendix E for further details.
EnER, &oo. ES= EXEN, 33
Scaling (coarse-scale) component. The functions ( = 2504/2 2504/2 k) are the scaling functions at resolution level jo. constructed from fixed father wavelet 6. They span
Vi3 == span s: Aja1:
where the index set Aoo satisfies AAo < d for some constant The corresponding coefficients & == (Jo))) are known as the scaling coefficients.
Wavelet (detail-scale) components. The functions ( 22'/" k) are the wavelet functions at scale j, obtained from a fixed mother wavelet Φ. Here, the multi-index & encodes both spatial position and directional information in d dimensions see Appendix E for a brief summary of the tensor-proet product construction used to define such wavelets in dimension > 1. The detail space at level j is defined as
Wi span E Az),
where A, indexes the active wavelet functions whose supports intersect X, and A₃ & A2dd j jo. for the same constant are called detail coefficients at scale j. Within the multiresolution analysis framework, the sequence of spaces (V))sez. forms a nested hierarchy with V, C V/+1 and dense union in (²RR)), while the wavelet spaces W, orthogonal complements such that Vj+1 V, ① W,. Figure 1 illustrates the hierarchical, stage-wise approximation process over levels that enables multiresolution analysis.
as above with no loss of generality. The coefficients B := (f. vy.a)
3

---

The decomposition (2) is classical result from multiresolu- tion analysis (see [22, Chap. 3] for deeper introduction), and holds for all functions in (2)) when the basis functions are derived from appropriately constructed wavelets. More gener- ally, similar dyadic expansions can also be built from splines or piecewise polynomial systems. this work, we focus on the wavelet setting as described above; we refer to [11, Chap. 6] and [8] for further details.
One property of this representation (2) is that can begin at any arbitrary scale Jo E N, offering flexibility to adapt the starting resolution level. In particular, we will later allow jo to be selected in data-driven and local fashion.
Throughout the paper, we do not rely on a specific wavelet basis, but require that it satisfies the standard S-regularity property for some S E N", as recalled in Definition 2 in Appendix E. This condition ximplies compact support, smoothness, vanishing moments, and bounded overlap. Notable examples include the compactly supported orthonormal wavelets of Daubechies [11, Chap. 7], and the biorthogonal, symmetric, and highly regular wavelet bases of Cohen [9] see Figure for illustration of approximation with S-regular Daubechies wavelets. When working with S-regular wavelet bases, the expansion (2) converges not only in L2 but also in other function spaces, such (x) for p e (or the space of uniformly continuous functions) depending on whether E (X).) > 1. This broader convergence behavior is a key reason for adopting such bases.
Approximation with Daubechies wavelets of regularity S (top) and S 5 (bottom) at levels J = 1, ,,,..,5 5.
Figure
Approximation results with wavelets. Approximation properties of wavelet expansions are by now classical and well understood; see, e.g., [8, 13, 20] for reviews. In particular, for functions f belonging smoothness spaces such Besov spaces Big with > p (so that f L (2)), one can construct an approximation f using nonlinear methods - see [12]. For instance, the so-called best N-term approximant f in S-regular wavelet basis (S s, see Definition 2) achieves the bound |l - al X N- where the hidden factor depends on the wavelet basis and the norm of the target function f. The precise construction of f and justification of this rate are provided in the Appendix (see the proof of Theorem 1). These rates will serve as a benchmark in our analysis of the regret (1).
Parameter-free online wavelet decomposition
In this section, we develop a sequential algorithm that performs an online, parameter-free decomposi- tion of incoming data using wavelets. The key idea is to learn wavelet coefficients incrementally without prior knowledge of the function's regularity or the optimal resolution depth while obtaining strong regret guarantees over broad function classes.
3.1 Algorithm: Online Wavelet Decomposition
Let ( } denote an S-regular wavelet basis as introduced in the previous section. We consider an online predictor based on a wavelet expansion (2) that begins at scale jo E and is truncated at level J > Jo. where the predictor at time > takes the form:
f((x)= = Σ a30.
Σ Σ (x))
REASo
3-000 REA,
where the scaling and detail coefficients fa5o. 3g.n.t} are updated sequentially over time.
Online optimization of wavelet coefficients. Our algorithm maintains and updates the collection of scaling and detail coefficients {aju.k,er (3.k.e) } in (3) across scales and positions k. At each round t > 1, after observing a new input v, the prediction computed using only the coefficients and basis functions whose support intersects x.. This defines the active index set at time t
T = {(v, k) 2% a (x)) + #OJu U Ü (G,A) (x)) #0} & Ago Ü Ag. j-10
50

---

Only the coefficients indexed by r, are updated at time 1, based on gradient feedback from the loss. The full procedure is summarized in Algorithm
Algorithm 1: Online Wavelet Decomposition at time
Input :Current coefficients {u5o, 83,t.);; active index set I, defined in (4).
1 Predict with coefficients in I',
f((x)) EGiMen Syku (2.((2))
where & stands for either 65 or d5j,; similarly, Cj = k or 93,
2 Receive gradients (9sa.e) associated with the active coefficients, defined in Eq. (5);
3 for (j.k) Te do
Update Cya.a Cjk.tt1 by approximately minimizing
c (+ (s. (z))((z)))
using corresponding gradient 03...t and parameter-free update rule satisfying Assumption 1;
Output: Updated coefficients (..t. B.d.+1)..
Computation of gradients. We assume that after making its prediction, the algorithm receives first-order feedback in the form of gradients {gga..) with respect to each active coefficient Cj...s. where cj denotes either a scaling coefficient kija.k, or wavelet coefficient B3 These gradients are efficiently computed using the chain rule:
Aj = (ooa(f(( (p) + (1)))(()))) a-ey,u..
((M((.)) (a))
(5)
where f9,. is either Φjo. 13.8 depending the scale, and E, is the derivative of the loss function with respect to its prediction argument. Note that the gradient expression (5) vanishes whenever the corresponding basis function satisfies (2x) O. This justifies restricting the optimization step at round to the active set I' defined in (4).
Assumption on the gradient step. To analyze the regret (1) of our method, we assume that the update rule used in Algorithm satisfies a parameter- free regret guarantee of the following form.
Assumption (Parameter-free regret bound). Let T > and suppose gi.... gr E (-G, G] are the gradients observed over time. We assume that the coefficient update rule satisfies, for any e TR,
T Σ
(i(c c) < (lcc))lla &lc-ail (i-aal(CVV 19. c20)
for some C1, C2
This regret bound holds with an additional term GG, E > as small as possible, for a broad class of first-order online learning algorithms, such online mirror descent with self-tuned learning rates or coin-betting style updates [33]. For simplicity we consider that Assumption 1 holds omitting this 0 in the sequel. These algorithms
additional term and the dependence in the hyperparameter and they provide optimal adaptivity to Jel, at the expense of
are referred to as "parameter-free logarithmic factors absorbed in C1 and C2 (see, e.g., [10, 32]). Note also that this type of algorithms are explicit and maintain their iterates through a closed-form update, resulting in low computational complexity see, for instance, the update rule in [32, Eq. (9)].
3.2 Regret analysis of Online Wavelet Decomposition (Alg. 1)
In this section, we analyze the regret performance of Algorithm under general convex losses (Lt), and against broad class of potentially irregular prediction rules, specifically those lying in Besov spaces B% (2'), which we introduce later. We focus in particular on the case s x. ensuring that the competing functions are continuous and bounded; see standard embedding results [20, 37]. As corollary, we show that Algorithm I achieves minimax-optimal rates when competing against functions in Hölder spaces, while automatically adapting to the unknown regularity of the target function.
á
5

---

Besov spaces, Besov spaces Big (X) constitute a classical family of function spaces indexed by three parameters: smoothness parameter s O, an integrability parameter p E [1, and a summability parameter q co]. For those un- familiar with Besov spaces, this space can be intuitively viewed as the space of functions with > 0 derivatives
Smmoth functious
in L"))), with p > 1, and parameter q > 1 allows for additional finer control of the regularity of the underlying functions, These spaces interpolate between Sobolev and Hölder spaces and are designed capture both smooth and non-smooth behaviors in functions. There exist several equivalent definitions of Besov spaces (e.g. using differ- ences, or interpolation theory): we refer to [20, 22, 37] for detailed and general background work, we adopt the wavelet characterization, which is par- ticularly well suited for the analysis of our wavelet-based algorithm.
optinaal regred
concave: 2 20-
optimal regne
Besov spaces. In this
continuuus functions See Appendia D
Let & > and let (djo.e: v..+.} be an orthonormal S S-regular wavelet basis with S (see Definition 2). We say that function f belongs to the Besov space Big the following wavelet-based norm is finite, with s' = d.
Figure 2: Diagram (4/r 8) of regret
regimes against Besov spaces.
1/9 13,113
[| == ==Iaasulla +
Σ 2
if & <
13300
(6)
I15 = 11 + sup 2 11 3330
if oo,
where ag (5.,)) denotes the vector of scaling coefficients at level jo. and B3 (U, Va,,))EA, the wavelet coefficients scale j > jo-
We now present our first result, establishing regret guarantees for Algorithm when competing against irregular but bounded prediction rules.
Theorem (Regret against Besov predictors). Let 1, 8 p,q oo with s 5
Let k x be an S-regular wavelet basis (Definition for some $ 8. Suppose Algorithm I is run with updates satisfying Assumption 1, starting at ei = 0, 3, = 0. 3 jo and using a wavelet expansion from scale jo to J = = 1og, T. Then, for any f Big (X), the regret satisfies:
SVT
ifs d or p <<<
Rr(f) CGILflle, (T--
else,
where C = C(A, s, C.) > depend only on 8, p. the wavelet basis, and the Ci, C2 in Assumption 1.
Theorem is proved in Appendix A, where we provide the full statement including explicit constants. Our bounds are minimax-aplimal in the regimes s >> $ when facing convex losses; see [35]. Note that logarithmic factors in I' may be absorbed into the constant in the regret bound of Theorem 1, typically in the case p The details are provided in Appendix A.
Adaptivity and tradeoff. Importantly, our procedure is adaptive to the Besov norm [| and the parameters (s, p. 4) whenever s < S. Notably, via the usual embeddings, f can belong simultaneously to several Besov spaces Brc with different norms || fll depending on and p. Remarkably, our algorithm effectively competes against any oracle associated to the best (which are not necessarily the largest) values of (s, p). yielding regret bound of type:
RT( & inf CG ('(.)) s.P
(
where the infimum is taken over all admissible pairs (s.p) such that f E (ipso))) the exponent r(s.p) reflects the rate in each regime according to (s. (see Theorem 1), and the constant C C(A, s.s.p Ci, C2) detailed in Appendix A.
Complexity and choice of wavelet basis. While one can use wavelets with infinite regularity (i.c.. S co), such as Meyer wavelets, these not compactly supported in space and thus lack

---

good localization properties. In practice, it is more common to use compactly supported wavelets that offer good trade-off between smoothness and spatial localization. Their compact support implies that most basis functions 35,6 vanish at any given input xy: only the indices & E A, such that xi E s))p( contribute to the prediction. For example, Daubechies wavelets of regularity S are supported on the hypercube [0, .S3 so at most O($4) coefficients per scale j are nonzero at any point 2s. Among all wavelet families, the Haar basis (corresponding to S = 1) yields the most efficient updates, its basis functions are non-overlapping. but it is limited capturing piecewise constant (i.e., Lipschitz-1) regularity; see [29] for an algorithm exploiting this structure and Figure I for an illustration in the cases S 1 and S closed-form function of its scalar gradient, satisfying Assumption and leading to ((1) cost per coefficient and keeping the overall update as light standard gradient descent (see [10, 32]). As a result, the per-round computational cost of our algorithm scales as O(JS"), where J is the total number of levels.
Each wavelet coefficient is updated through
The case of Hölder function spaces (X) (X). We previously showed that Algorithm 1 effectively competes against any comparator in the broad class of Besov spaces B In particular, by classical embedding results, when p q
has the identification 6 (X)) - (X') where
"E" (2¹) is the set of Hölder continuous functions. A function f E (x(x) with E (0, 1] if it satisfies
the Hölder condition:
(8) where > 0 is the smallest such constant, denoted Ifls- For > 1, we extend the definition by requiring that all derivatives D" exist and satisfy (8) with exponent [s] for any multi-indices m E N' such that [m] = [s].
(f() - f(y) & L||= - ylا for all E X,
We now state corollary of Theorem for Hölder continuous functions, expressed in terms of the Hölder semi-norm Is). and sup norm ||
Corollary 1 (Regret against Hölder predictors). Let T 3 1 and > 0. Let (6jn,:: vy,a} be an S-regular wavelet basis with S > Under the same assumptions in Theorem 1, Algorithm I has regret bounded for any I (X) as
VT
if d is
2
Rr(l) &
loga(T)vT T
< <
where = C(C1, A. s) is constant independent of T and f, and depend only on P. the wavelet basis, and Assumption 1.
We prove Corollary in Appendix B. Our results are minimax-optimal for general convex losses, as established in [34, 35J, and improve over the guarantees of [29], which are restricted to functions with at most Lipschitz regularity (s & 1). In contrast, our method adapts to any smoothness level O. Moreover, Corollary 1 shows that Algorithm 1 adapts simultaneously to both the smoothness and the Hölder semi-norm [fl, of any competitor f e '6*. As the Besov case, our algorithm automatically trades off between leveraging higher smoothness and benefiting from smaller If see (7). This tradeoff will be discussed and exemplified in the next section.
4 Adaptive learning in inhomogeneous regularity regimes
In this section, we extend Algorithm to enable local adaptivity, with a particular focus on settings where the target function exhibits spatially inhomogeneous regularity; see Figure 3 for an illustration. Our method is inspired by the localized chaining approach of [29], and we show that it can adapt to local variations in regularity across broad class of functions in Besov spaces Bg. This adaptive procedure also yields improved global regret rates over those of Theorem for exp-concave loss functions with optimal guarantees formally established in Theorem 2.
4.1 Adaptive Online Wavelet Regression
We begin by describing the partitionning process we use in our strategy, and we further describe the aggregation procedure leading to our adaptive Algorithm 2.
Partitioning tree. A common strategy to construct partitions of X is via hierarchical refinement, with dyadic partitions being a canonical example. Fix Jo E N For each jo [Jo], let Dj,, = (2') denote the collection of dyadic subcubes of X' at resolution level jo. where each subcube has side
7

---

length | jo. We define the full multiscale dyadic collection as D = -1p= Dio spanning scales jo O...., Jo. This collection is naturally aligned with tree structure T T= (D) with node set ((T). Each node E (T) is associated with unique cube X, D at some level 1(m) [Jo], such that Rr E Din.. For any fixed scale jo E [0. Jol, the cubes in D3 form uniform partition of X, and each cube Xr Dja, with 1(m) = jo has side length Xx, | Furthermore, each node n at level [(x) = jo has 2d children corresponding to the dyadic subcubes X, C Nr, at level I(n') jo + 1. Finally, we refer to any subtree T' C T that shares the same root and whose leaves terminal nodes C(T') form (potentially non-uniform) partition of X' as a pruning of T; see [29, Def. 2]. The goal is to design algorithm that effectively tracks the best partition induced by such prunings T'.
Local adaptation via multi-scale expert aggregation. Our objective is to identify the optimal starting scale jo locally over X, in order to adapt to the spatial variability in function regularity. Intuitively, allowing finer-scale precision in regions with lower regularity can significantly improve prediction accuracy. To this end, we launch a family of global predictors fio of the form (3), each initialized at different starting scale ju E [Ju] and sharing common maximum scale J = [ log, T). Following the tree structure T of depth Jo we associate each node n E (V(T) to starting scale ju = 1(m) and local expert predictor fn... subregion Nr and whose scaling coefficients are set to a an in (3). The scaling coefficients are supported on grid A of precision 7 / The local predictor then associated with a restricted scaling index set jo.... C Ajo and wavelet index set A,," C Ay for j > ju. both supported on N.. For simplicity, we define the tuples = (n.a.. belonging to some expert set & C CT(T) x A whose cardinal is bounded as |E] & [|
fs
the restriction of the global predictor fo to the
At each time t 3 1, we define the set of active experts at round f as : fe= (n, an) E E x4 E ) The prediction is then formed by aggregating the outputs of active local multi-scale experts in Es, yielding:
f((z)) = = We,t (e:)() CEE,
where
Wele E [0, 1J.
Σ eEE,
We,: 1,
99)
and [| max(- B,, B.-(BB.)i)))) denotes the clipping operator in [-B, Each localized expert fe, _fnan.. is trained independently using Algorithm 1, with scaling coefficients initialized at some a., and contributes only within its local region X... This framework mirrors an instance of the sleeping expert problem, as described for example [18], and requires standard sleeping reduction, such as the one in line 4, and then used in lines 5-7 of Algorithm 2. The weights (wee))ee are updated over time in line 7 using weight procedure based on gradients V. E (-C, Gglel that satisfies Assumption 2. The overall procedure is summarized in Algorithm 2.
Algorithm 2: Adaptive Online Wavelet Regression
Input :Bounds G, B 0; Set of experts E; Initial uniform weights W1 (We,))ees; Initial prediction functions (fe.))es;
1 for t = 1 to T do
2 Receive Zr; Reveal active expert set & and local active index set
De. T= Aggin n Nu for every = (n, Ee, jo 1(n)
with T. as in (4);
Reduce weights We,t + ment/ Excs, we., if e &, and wr.e = else
5 Predict f(())) = Enes, (e.)fe.e((e) using active weights (we,))ege, Reveal gradient V, Va, (Eees, (f)))f..( Eefe, w.. f(xx)) Update we41 weight(wr, V.) with weight satisfying Assumption 2; for e E E, do
8
Reveal gradient g.,t = (93. in (5), on active index set Fe,6
10
Update fe, using Algorithm with input l',, and g.,
Output: :f+i = Eces (o.Tt1
Assumption on the aggregation algorithm. Our method relies on any expert-aggregattio algo-
rithm satisfying second-order regret bound, stated in Assumption 2. State-of-the-art aggregation

---

algorithms, such as those proposed in [18, 26, 41], satisfy this second-order regret bound and are compatible with the sleeping expert setting.
Assumption 2. Let V....., V E [-G, 6 for T and Ō > Assume that the weight vectors W: = (EE, initialized with a uniform distribution WI, are updated via the weight function in Algorithm 2 and satisfy the following second-order regrel bound:
T V..w. - & Cayy log((E)) E(VT VV.)
for all e E E, where Cs, C4 > 0 are constants.
Note that for Assumption 2 hold, the loss gradients V. must be uniformly bounded in the sup-norm by G. Indeed, this is ensured by two factors: first, we consider oracle prediction rules in (2t), ensuring that their outputs are also uniformly bounded, and second the predictions produced in (9) are clipped to bounded range.
4.2 Regret guarantees under spatially inhomogeneous smoothness (Alg. 2)
Local Besov regularity. Let f Big for some fixed 1 p.9 oo and s :. Let T' be pruning of T and (Ax)nEc(T) be the associated partition of X. To model spatially varying smoothness, we define the local Besov regularity function over each region N, as
(10) where the restriction fix. belongs to the Besov space Bi over the domain X,, for fixed global parameters 1 & P.q & oc. For each region, we denote by || the corresponding local Besov norm (6).
8n == sup (a fix, E (p.)) >
More generally, one could define 'fully" local Besov spaces via triplets (sn.Pa. 9n), allowing both smoothness and the integrability parameters to vary across regions. We leave the analysis of such fully tion (d = schiemes to future work, and we focus on local adaptation in terms of smoothness only. adaptive
0.5
0.8
0.9
25
X
Xs
Figure 3: Example of inhomogeneous func-
We prove regret bound for Algorithm 2, expressed in terms of the local regularity of any competitor Besov space and that achieves minimax-optimal rates with convex or exp-concave loss functions. Theorem 2. Let T 3 1, & P.q & 00,8 >>>>> e. Let f E (x) and > ll Let T' be any pruning of T, together with a collection of local smoothness indices (sn)neccr) and of local norms (IIslla) me defined in (10). Then, under the same assumptions Theorem I and Assumption 2, Algorithm 2 satisfies
Sf) REA(T')
Σ
(B--Zhn (2 1(0)an [fllem)) 25 v Tu 23
(2-1(m)am || Tul Tull )
Bv
and moreover we also have, if (Rt) are exp-concave:
SC E
B''=mm 24
2om 2m (2-1(m)4m
28
2om
NEE(T))
+ 1())
[[]| [Tal 1- (Tal-=
2 + B
where & hides logarithmic factors in T, and constants independent of or T.
The proof of Theorem 2 is deferred to Appendix C. Taking T' the pruning corresponding to the root of T, Theorem 2 yields minimax-optimal rates in the case min, 8n $. both for convex and exp-concave losses simultaneously. Importantly, the local adaptivity of Algorithm 2 is reflected in the regret bounds in Theorem 2, which now depend on the local Besov regularity of the target function. This is especially advantageous in inhomogeneous settings where the function alternates between smooth and highly irregular regions; see Figure 3 for an illustrative example. such cases, our approach can substantially improve the overall regret compared to classical global adaptive methods that aim to recover the largest (but worst case) smoothness exponent such that f E B (X'), assuming the semi-norm al,, is uniformly bounded.

---

Adaptive trade-off between smoothness and norm. To illustrate the benefit of our strategy, consider the function f æ E [0. 1 x", with & E (1. We have f (x'),, with global semi- norm Ifls (as defined in (8)). However, the semi-norm (f], becomes large near x 0 due to the unbounded second derivative, which equals a(s - 1)=² with < and explodes when æ - As result, this directly affects the regret bound of non-local algorithms. Now consider partition Xi - [0, 6] and X/ = (ò. 1J induced by some pruning, with 5 = 2 for some jo > 1. The function f belongs to & (((2)) near
0, and to (X2) with bounded semi-norm Iflz over X2. Estimating under this higher regularity on X₂ yields improved guarantees. Our adaptive algorithm automatically exploits this spatial inhomogeneity by focusing on the relevant local smoothness and local semi-norm, leading to improved overall performance. Indeed, applying Theorem 2 to functions in 8 (2²), with z and exp-concave losses, yields:
Rr( & (111)) [Ti 2 + (1
(11)
where |fl2 = (1 1f s(s 1)0-~ and |f]. < oo by definition. Equation (11) illustrates trade-off:: [Ta] negligible- i.e., only small fraction of the data falls near 0 then the second term dominates, and we obtain a regret rate of O(T Conversely, we incur the worst-case rate ((T)/(+))) but is diluted by the small measure of [ = 8. Finally, consider the case where the data are uniformly distributed, i.., [Til R2 öT and [Ta] R (1 8yT'. We obtain:
Rr(l) $ 8 (8T) श
20==2)
(1 Syr
8T
Optimizing over 6, which is automatically handled by our procedure that selects the best pruning. yields regret rate of O with (4/s, 1/2,+1)), improving upon the worst-case rate ((T achieved by any non-local algorithm.
Conclusion and perspectives
We proposed adaptive havelet-based algorithms and analyzed them in the competitive online learning framework against comparator functions in general Besov spaces. Our algorithms achieve minimax- optimal regret guarantees while adapting simultancously to the regularity of the target function, the convexity properties of the loss functions, and spatially inhomogeneous smoothness resulting in significant improvements over globally-tuned methods. A limitation of our algorithm, which is common with wavelet-based methods, is the exponential increase in dimensional complexity with the regularity parameter S. An interesting parallel can be drawn with traditional wavelet thresholding methods [15] used in the batch setting: in an online manner, our purameter-free algorithm implicitly mimics their behavior by selectively updating coefficients across scales, without requiring explicit thresholds prior knowledge of the function's 's regularity. Finally, like most prior work, we focused 9). A natural and compelling direction for future research is to extend this analysis to competitors in general L" spaces with < oc, which would likely require new analytical tools beyond those used here.
on functions embedded in 1 (i.e., with
Acknowledgments
The authors would like to thank Stéphane Jaffard and Albert Cohen for insightful discussions and suggestions on wavelet and approximation theory. P.L.. is supported by PhD scholarship from the Sorbonne Center for Artificial Intelligence (SCAI), Paris.
References
[1] Peter Binev, Albert Cohen, Wolfgang Dahmen, Ronald Devere, Vladimir Temlyakov, and Peter Bartlett. Universal algorithms for learning theory part i: Piccewise constant functions. Journal of Machine Learning Research, 6(9), 2005.
[2] Peter Binev, Albert Cohen, Wolfgang Dahmen, and Ronald DeVore. Universal algorithms for learning theory. part ii: Piecewise polynomial functions. Constructive approximation, 26: 127-152, 2007.
[3] Daniele Calandriello, Alessandro Lazaric and Michal Valko. Efficient second-order online kernel learning with adaptive embedding. Advances Neural Information Processing Systems, 30, 2017.
10

---

[4] Daniele Calandriello, Alessandro Lazaric, and Michal Valko. Second-order kernel online convex optimization with adaptive sketching. In International Conference on Machine Learning, pages 645-653. PMLR, 2017.
[5] Nicolo Desa-Bianchi and Gábor Lugosi. On prediction of individual sequences. The Annals of Statistics,
1999.
[6] Nicolo Resa-Biancch and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
[7] Nicolò Cesa-Bianchi, Pierre Gaillard, Claudio Gentile, and Séhastien Gerchinovitz. Algorithmic chaining and the role of partial feedback in online nonparametric learning. In Conference on Learning Theory, pages 465-481. PMLR, 2017.
[8] Albert Cohen. Numerical analysis of wavelet methods, volume 32. Elsevier, 2003.
[9] Albert Cohen, Ingrid Daubechies, and J-C Feauveau. Biorthogonal bases of compactly supported wavelets. Communicutions pure and applied mathematics, 45(5):485-50) 1992.
[10] Ashok Cutkosky and Francesco Orabona. Black-box reductions for parameter-free online learning in banach spaces. In Conference On Learning Theory, pages 1493-1529. PMLR, 2018.
[11] Ingrid Daubechics. Ten lectures on wavelets. SIAM, 1992.
[12] Ronald A DeVore. Nonlinear approximation. Acta numerica, 7:51-150, 1998.
[13] Ronald A De and George G Lorentz. Constructive approximation, volume 303. Springer Science Business Media, 1993.
[14] David L Donoho and Iain M Johnstone. Ideal spatial adaptation by wavelet shrinkage. biometrika, (3)))2225-44 455, 1994.
[15] David L. Donoho and lain M Johnstone. Minimax estimation via wavelet shrinkage. The annals of Statistics, 26(3):879-92 1998.
[16] David L Donoho, lain M Johnstone, Gérard Kerkyacharian, and Dominique Picard. Density estimation by wavelet thresholding. The Annals of statistics, pages 508 (08-539, 1996.
[17] Pierre Gaillard and Sébastien Gerchinovitz. A chaining algorithm for online nonparametric regression. In Conference on Learning Theory, pages 764-796. PMLR, 2015.
[18] Pierre Gaillard, Gilles Stoltz, and Tim Van Erven. A second-order bound with excess losses. In Conference on Learning Theory, pages 176-196. PMLR, 2014.
[19] Alex Gammerman, Yuri Kalnishkan, and Vladimir Vovk. On-line prediction with kernels and the complexity approximation principle. In Proceedings of the 20th conference on Uncertainty artificial intelligence, pages 170-176, 2004.
[20] Evarist Giné and Richard Nickl. Mathematical foundations of infinite-dimensional statistical models. Cambridge university press, 2021.
[21] Steve Hanneke, Aryeh Kontorovich, and Guy Kornowski. Efficient agnostic learning with average smoothness. In International Conference on Algorithmic Learning Theory, pages 719-731. PMLR, 2024.
[22] Wolfgang Härdle, Gerard Kerkyacharian, Dominique Picard, and Alexander Tsybakov. Wavelets, approximation, and statistical applications, volume 129. Springer Science & Business Media, 2012.
[23] Elad Hazan and Nimrod Megiddo. Online learning with prior knowledge. In Learning Theory: 20th Annual Conference on Learning Theory, COLT 2007, San Diego, CA, USA; June 13-15, 2007. Proceedings 20, pages 499-513. Springer, 2007.
[24] Elad Hazan et Introduction to online convex optimization. Foundations and Trendso in Optimization, 157-4): 2016.

---

[25] Rémi Jézéquel, Pierre Gaillard, and Alessandro Rudi. Efficient online learning with kernels for adversarial large scale problems. Advances in Neural Information Processing Systems, 32. 2019.
[26] Wouter M Koolen and Tim Van Erven. Second-order quantile methods for experts and combina- torial games. Conference on Learning Theory, pages 1155-1175, PMLR, 2015.
[27] Guy Kornowski, Steve Hanneke, and Aryeh Kontorovich, Near-optimal learning with average hölder smoothness. Advances in Neural Information Processing Systems, 16621135-211555 2023.
[28] Iía Kuzborskij and Nicolo Cesa-Bianchi. Locally-daaptiv nonparametric online learning. Advances Neural Information Processing Systems, 33:1679-1689, 2020.
[29] Paul Liautaud, Pierre Gaillard, and Olivier Wintenherger. Minimax-oppimal and locally-adaptive online nonparametric regression. In Algorithmic Learning Theory, pages 702-735, PMLR, 2025.
[30] Stéphane Mallat. A Wavelet Tour of Signal Processing. Academic Press, 1999.
[31] Yves Meyer. Ondelettes et opérateurs. I: Ondelettes, 1990.
[32] Zakaria Mhammedi and Wouter M Koolen. Lipschitz and comparator-norm adaptivity in online learning. In Conference on Learning Theory, pages 2858-2887. PMLR, 2020.
[33] Francesco Orabona and Dávid Pál. Coin betting and parameter-free online learning. Advances in Neural Information Processing Systems, 29, 2016.
[34] Alexander Rakhlin and Karthik Sridharan. Online non-parametric regression. In Conference Learning Theory, pages 1232-1264. PMLR, 2014.
[35] Alexander Rakhlin and Karthik Sridharan. Online nonparametric regression with general loss functions. arXiv preprint arXiv:1501.0659% 2015.
[36] Veronika Ročková and Judith Rousseau. Ideal bayesian spatial adaptation. Journal of the American Statistical Association, 1196547) 2024.
[37] Hans Triebel. Theory of Function Spaces. Birkhäuser, 2006.
[38] Vladimir Vovk. On-line regression competitive with reproducing kernel hilbert spaces. In International Conference on Theory and Applications of Models of Computation, pages 452-463. Springer, 2006.
[39] Vladimir Vovk. Metric entropy in competitive on-line prediction. arXiv preprint cs/0609045, 2006.
[40] Vladimir Vovk. Competing with wild prediction rules. Machine Learning, 69:193-212, 2007.
[41] Olivier Wintenberger. Optimal learning with bernstein online aggregation. Machine Learning. 106:119-141, 2017.
[42] Oleksandr Zadorozhnyi, Pierre Gaillard, Sebastien Gerschinovitz, and Alessandro Rudi. Online nonparametric regression with sobolev kernels. arXiv preprint arXiv:2 2021.
[43] Zhiyu Zhang, Ashok Cutkosky, and Yannis Paschalidis. Unconstrained dynamic regret via sparse coding. Advances in Neural Information Processing Systems,
2023.
12

---

APPENDIX
Contents
1
Introduction
2 Background and function representation
3
Parameter-free online wavelet decomposition
3.1 Algorithm: Online Wavelet Decomposition
3.2
Regret analysis of Online Wavelet Decomposition (Alg. 1)
4
Adaptive learning in inhomogeneous regularity regimes
4.1
Adaptive Online Wavelet Regression.
4.2
Regret guarantees under spatially inhomogeneous smoothness (Alg. 2)
A Proof of Theorem 1
14
B Proof of Corollary 1
20
C Proof of Theorem 2
22
D Discussion on the unbounded case:
29
Review multi-resolution analysis
30
Summary of the results and comparison to the literature
32
Besov embeddings usual functional spaces
33
Summary of optimal regret in Online Nonparametric Regression
33
13

---

A Proof of Theorem
Let & P.q & 00,0 < < < S and f E arg men 12, ( Ei ((f(x))) the best function to fit the T data over X' B, BJ. We start the proof with a decomposition regret, with any oracle function f' R
Rr(s) = E
T (((
Σ - +
estimation regret
approximation regret.
Nonlinear oracle.
Let jo E N and J > jo to be optimized. We first recall that we use wavelet development defined for any J 2 jo
1x(x)= = E (o.
Σ I REAj
E
(12)
kEAJo
with O'jo.k (f, (o.k)) k) and B3 U.,)),) 2 Jo. is truncated wavelet expansion up level J 3 jo- Using a truncated approximation fa with large value of J in (12) can lead to suboptimal regret performance, as it requires estimating large number of wavelet coefficients, thereby incurring high estimation error. To address this, we introduce nonlinear oracle f" that depends only on a selected subset of coefficients across the J levels. This approach, known best-term nonlinear approximation, surveyed in the textbook [12], while constructions close to us in spirit can be found in [15, 16]. We now make this oracle explicit and show that balances approximation and estimation errors, in particular achieving minimax optimality in our setting. We define the oracle as
= fs far,
(13)
where f.. is a truncated wavelet expansion up to level J < Js in (12) (i.e. we keep all the detail coefficients up level J"), and the nonlinear part
fa- Σ Baadssa
A' A(C(U,)) J <{E))
(AA)A
only wavelet coefficients indexed by oracle set A' drawn from the finer scales j (J", J]. The cardinality of A', i.e., the number of retained coefficients, will be optimized in the analysis. Intuition: The component fa- of the oracle consists of the |A* largest coefficients chosen adaptively from the fine scales greater than J The procedure is termed nonlinear because the choice of coefficients varies with the function f. rather than being a fixed linear rule, contrary to the first J' levels which keep all coefficients independently of the function.
For any E Aj.j > jo, define 2 == f with s' = + _ 1 as in (6). Observe that the definition of the Besov norm (6) allows control the set {0,, since
x
A,,
J < j J} in terms of no-m
& [(-J)) 1
Σ Σ RE
&
& (J-J))--5) 51+
Σ JURJJJ
Σ KEAj
los.ap)
+ 11
C₁,
(14)
by Hölder's inequality if > p, else by convexity with g p. Let A" denote the set of indices corresponding to the A* largest wavelet coefficients (in absolute value) among all (v,.)) with j [J" + 1, and Aj. The cardinality |A'| that is, the number of wavelet coefficients retained in the nonlinear component of the oracle estimator (13)-will be selected later in the analysis tuning parameter. Let > J*. We have that
|
min (AAEA
l &
Σ 12 & Of < <c, GADEA-
and in particular since k) & A", |u mingy,,gea A")-- lny one has
kū & A', [A" || Cf
( |B,r -,2-5*1A77
(15)
We are now ready to analyze the regret in two steps-that estimation error and an approximation error.
14

---

Step 1: Bounding the estimation regret. We set
T Σ & (E (xAn)pga(())) - & (EG SARPsa
Ri = Σ
-
where the sum is over all scaling and detail coefficients with indices in {(Jo, k) & E R): j 3 Ju: & E A, J. where CG stands for either the scaling coefficient aso.k or the detail coefficient 3,.k (and their sequential counterparts C,,,, depend on t), and Pa, denotes either the scaling function $3o.k or the wavelet function 2.,k.
Since á ++ E<()) convex and both fi, l are linear the (g,a), then o f and a I" are convex in } and we have by convexity:
R1
Ii cj.b):
5.k
where 0s.k. = (33 by Equation (5). Observe that max; S3.k. & 2 G||ppl> =: G, for any j. k. Then, first by Assumption 1, and second by the structure of the oracle (13) - namely, that jj > J" such that (j,k) & A", we have C, = - we get:
R1 E Σ
Bp..el o)u))
jik
Σ
loyall
CIVE ghAll + CaC.s)
Ciyy
3,4
legal
(CVV 19s...1 lessaal + C2G5s
+
lesal
Ci (IVVE leand + + (16)
GARJAA 3>0'
:Ri(ff)))
Ri (fa))
where Ct, C2 > 0 are factors (possibly including T;; see Assumption 1). The estimation regret Ri is thus controlled in (16) by sum of individual regrets over the nonzero coefficients Cz that define f = for + fx-, The sum naturally splits into two parts: the linear part Rí (fr=) over ((jo,) : &s E ((.A)) : jo E & J" & E A,), and the nonlinear part Rı (fxx) over the indices in A".
Linear part: bounding R1 (f))).. The wavelet basis (dia. } is assumed to be S-regular with S > 8, so we can invoke the characterization of Besov spaces with |lf < do (see Eq. (6)). Let p.p' '> be such that + ト 1. Applying Hölder's inequality to the detail coefficients at levels
j E J'I, we obtain, with G, G2 || jo:
Σ
Σ 18g
Ci Civ Eim |0 2 C2G C2G1111l022
JO REAS
13) C1 Σ (VEi ( x GG (,,1
JLEA,
keA,
Σ
113
CiV EEREs (ETH 19) C2Gllull ||)
Σ J
| lp
Ci -,1(( -) V ExEss Eier 19 C202 |A,
(17)
where the last inequality uses | < 3,,1 [|/l]] for a vector x of dimension |A,| and ||a||
(++ := max(:;0) 0}.
Repeating for the scaling coefficients for & E Ajos summing in j = jo...., J and bounding [Ayl x and | 1o we get:
(f(+)) & ||a5 llp (G12060 - +))+ V ExEN, Ei k.o. C=G de (2 47)
+ E/ |13 (GA220(1--)) VInee, Et le... E 2 x2 (18)
15

---

where we recall the scaling coefficients are ai (ag).:) and the detail coefficients at scale j are 3 (3
On the other hand, over each level j x ja. one has
95. 2
REIN
KEA
Σ
(
kEA, t=1
T
G2 2
Σ 1=1
Σ KEA
12
(19)
where we used the fact that (xx)) & G (since 1 + (0) is G-Liischitz), the definition of $5.. and we applied Jensen's inequality. Equation (19) also holds for the scaling level, replacing $5,. by $50.k over the index set Ajo-
By D.2, one has
sup2 Σn 16(x - k)l² & Mollol and sup, Σx |6(x - k:)2 & My 1
With - 1 + + +)+ we get from (18) and (19)
Ri(fs-) & Aa [CI(M)llo))))) Callolo )l 11 002 -
(Cr (C((M)I))=)) Callvil
||, M/ 2 -
(20)
3=30
Then since |l oc in (6), we apply Hölder's inequality with q. o' that entails
1 11
Σ
2 2-1 1-pl0+1-900 32200000 - 5) 111 2/2 2 (4-3)44
30
==j
2 14'4 (a # - #1+)) き 200 # 1113,,133) 1
30
< |ffl
2 54 $- +1+),
since || 11 & 1| 11,1,, a'>1
1=0
Finally, we get from (20) with ||a ll & |
Ri(fx) & AG M +CITT+C2tti
J Σ (21)
+ +CIVT+C22*""
==j0
with 3 := d(} - - and M = (max(Meeloll>>> M 1oo 11 1w)12 1 < 0
Nonlinear part: bounding (fa+)) Let A ,/_ where |A;1 & |A, is now the oracle sparse set made of positions at level j. One has by (17) on the levels = J + l.....J,
Ri (fa)a) < Σ 113, (CIA) # E En lnnll + CcG2 4 | 1171
J J'C
< GM JJK
Σ
- +
16

---

where second inequality follows from (19). Then, using Hölder's inequality with y > one has
Σ 2 13,11//
Σ
2 - 11113 Ilp 2- $) |3} - -
=
JTCjEJ
JECJEJ
< E
Σ
2-
JeijEy
Finally, since Exeegs |Aj1 |A7), one has
E 2- 5 & A |1 11- 314 21442 2-
Σ 2- 3
| | - & a^ (2 3
2
JCCJEJ
JJMjEd
by Hölder's inequality in the case 2 2 and since & >>> d. Similarly, for the second term we have
2 2- 5
Σ _ |331م|131
& llfll
21 11 3.
J JJ
All in one, with [A 1- < 1 +)+
one has
Ri(fa-) & GMIIfll + -2- 11000911770 -1 #1
(22)
2-
Bound on Rí: We use (21) and (22) and we reach
Ri < N. (fr-) + (fa-)
, M (22 +)+
+ +XCCVT++ C22
Σ 2-38 58
3=30
2 (1-
+ -
+)+
(23)
2
where we recall / = 1 d(} +)) and s' + and |Ad coefficients we keep below level J
| is the number of 'non-linear'
Step 2: Bounding the approximation regret. We now bound the term incurred by approximating
by its nonlinear wavelet approximation ʃ Using the C-Lipschitz property of each loss &, and the uniform bound on the approximation error, we obtain:
Ra Σ ( & &G Eir(e)) & GT fll
(24)
With /" = f.- SA- and fo the truncated wavelet expansion (12) at level J > J > ju we have with the triangle inequality
Ra (fs + - + f
First, since fs are both wavelet expansion truncated respectively at level and J, one has
[[(fs- + fx-) fallx0 =
Σ (A
(1.))s
2
sup ki.
13 ERES,, 10/2'- =l) + by definition of dy..
j=d*11
Σ 2/42-30
by Definition D.2 and (15)
j=
(--" d
replacing s' and with > d
< MOCHAAAT
(25)
2
17

---

Second, with y. using the characterizations of Besov spaces and classical results on Sobolev embeddings (see, c.g., [20, Prop. 4.3.8] or [8, 13]), Big (2) C B (x) and one has
II5s - foloo & Mollflla,, Σ 2 & Mol1s1lo, 2ー 2--
(26)
523
where g. Finally, with (25) and (26) one has
GMyll 1y (2 (2- 10--3l--110 D (J - 2- 317
Ra
(27)
2" 2-
Step 3: Optimization J JJ. | and conclusion. Let ju following regret bound
From (23) and (27) we reach the
(f) = Ri + Ra & CGIIfIIRA [ + (1 +1+22-1 2-18
1
+
3 (J - 2 1A - (28)
with C' some constant that can change from line to another (depending on A, lo M, M......
s d( 1 1 We keep the explicit dependence on J, J", and |A*], as we now aim to optimize the upper bound with respect these parameters. We have three different regimes depending on the sign of / in (28). Observe that
if 2. p 2,
and we also have s p'
Case 1: B > 0. This regime corresponds to sufficiently regular functions: since d this corresponds to the case
p
p< 2,
or
In this case, the geometric sum is bounded by
2
1 2-5
1=
Choosing
2
J" 1 loga (T)
2 2- +) |1 T
|A 2dd
(29)
2- DT=
J log, (T)
[ = 2 =T
with $ >>> 0 and S > s, and this entails a total regret
(f) & CGI (a+C)))T 2+ 1-2-8 T
+ (1+
(
1 9 - l082(1))
13 (5-621)
)
(30)
With s > d/2, we have 71-4/4 < VT and = = VT).
Remark. The notation O(-) here hides (Tg factors that appear when p << q. This originates from the nonlinear oracle construction in the analysis (see Inequality (14)). In addition, log terms may also be absorbed into the constants C1, C2 coming from the parameter-free subroutine (see Assumption 1). This remark also holds for the remaining cases.
18

---

Case 2: = 0. This critical regime occurs when 2 2 and
The sum becomes:
2'
2- J
==
Choosing [A") and J as in (29) yields the bound:
Rr(f) < f (C++Ca)VV 2 log. T +
13 ) ( l083(T)
} (31)
+VT 1+
That is Rf( = OGl log, (T)T).
Case 3: 3 <<0 0. This corresponds to the low regularity case: 3 $ and
<s<< d 2
pd2 2
and
p
Here, the geometric sum is bounded as:
2 2-J'
18 & 2-8-1
With J", and in (29), the regret bound becomes:
Rr(f) & CGIIfl
[G Ca)+C 1
T
+
+
2- -
16 (3-0)4)) ) (32)
+ 1+
[
-1) 108:
d
With V VTT T one has Rr OG T" T2.
19

---

B Proof of Corollary 1
The proof is based on that of Theorem 1, in Appendix A, case q oc.
Let 0 E arg min (e))(x) Ei (x)())) the best function fit the T data over X and =- defined as (12). One key point is that in the case Hölder-smooth function (p q == oo), the nonlinear set A of wavelet coefficients will not be needed achieve optimal rates. We start with decomposition of regret with the oracle f" fJ as in the proof of Theorem 1 in Appendix A and we have:
T
Rr(f) = ( Euth(t)) +
Σ
=:
=:Ra
Step 1: Bounding estimation regret Ra.
From (16), one has
Rí
Σ
|
Civy CiVE
10 C2 20
AEAjo
Civy CIV ET 19 + 12 2 (33)
18
1=30 kEAj
where C2, C2 > are relative to Assumption 1, aj, refers the scaling coefficients and 83 the detail coefficients.
Since the wavelet basis foro.k v. is assumed to be S-regular with S > (Definition 2) and E 66 (X)), by Proposition 1, the detail coefficients every level j > ju are bounded as:
|05x1 KV,, vi,+)| C(p, s)].). 2)
where C(vi, s) is a positive constant that only depends on the S-regular wavelet basis and [/]s refers to the semi-norm of f defined in (8). For the scaling level jo. then for every k. one has:
10.0.al = =Kff & ll 11 & 2 44 11lll11flle.
where we used
2/04/2 6(2"0 k) da "=2 20022-J0 [ [d(ui) du 2 2 ||+|
[| 1 &
3d
and ||+l| < since the scaling function ø is assumed to be localized (e.g. compactly supported).
Then, plugging the above upper bound, we get with Bio.t, & 2
& G| aig -Asal 11 (C12 2 VT C22 "
T
Ci
Σ
1 + 12
REA,
Using Cauchy-Sch -shwar's inequality as long as the form of the gradients in (5) and the bound (19). we have over each level j E
[Jo J].
Σ kEAj
Σ
gikeel & GAÈ2d
Σ
12/2,,, 1))²
REA,
< (()) 2
where || EKEA, |v - 1 00
1p < oo (see D.2) and |Ayl X Finally, with M
max(M, Il, Il )
R1 & GMA ||
200 (CIVT C2) 8)) (ol.llll.C.VT... (CIVT C22f1) Σ 2 (oo (34)
Setting jo = 0, the sum can be upper-bounded with different cases as
(1 - 2-- 2)
if < 28
Σ 2 1(
<
J +1
if d = 28
1=
2- (J41)
(1) 1)
if >
20

---

Step 2: Bounding the approximation regret. Following (27), one has:
Ra = Σ ( & G Σ (m)) & ff & C.GTTf).2"* (35)
where C4 Ca( see [20, Prop. 4.1.5] for instance with assumption D.3.
Step 3: upper-bounding Rr( We need to balance (34) and (35), and finding the optimal J > 0. i.e. jo - and
Taking jo d < 2s, d = 2s and d 2s.
logg (T) entails the desired bound in the cases
Remark. In the preceding proof we showed that single (linear) global resolution level J competitors, in contrast to general
T log, 7] suffices to attain the minimax regret for competitors in Big (X'), which require a nonlinear mechanism (see Appendix A). Nevertheless, even the Hölder-smooth case one may take a larger level In the analysis, set the oracle coefficients $,.6 combined with Assumption 1, reduces to sum over the remaining nonzero coefficients-naaneee
as log, T > 1 log, T as in Theorem I. for levels j à loga T ; the estimation regret,
those in the linear part- and leads to the same rates.
21

---

C Proof of Theorem 2
The proof uses a first key result that we state and prove right after.
Theorem 3 (Local regret over Besov spaces). Let T 3 1, P.q & 00,8> d and f E Bip. Under the same assumptions of Theorem and Assumption 2, Algorithm with ll & B has regret
Rr(s) & Ginff { (EnECTTY) B B
IIf 20M (2lp(en))))
and if (4x) are exp-concave:
Rf( & Ginff + Enecery 1f 2- |Tuu (
where & hides logarithmic factors in T, and constants independent of f or T, L(T') denotes the set of leaves in pruning T' C., l1fll are local Besov norms, 1(n) is the level of node a E L(T'), and the local rate exponent is given by
2
sn 3 or d otherwise.
r(8n,)) =
Remark. Theorem 3 holds for any pruning T' of T. particular, our procedure effectively competes against the best pruning with respect to the profile of the competitor f. Intuitively, Algorithm achieves a spatial trade-off over the input space: it can refine locally by going deeper with high I(n) at the cost of increasing the number of leaves |C(T")), while remaining coarser and less accurate in other regions, with fewer leaves to compete against. In particular, when applying the result specific pruning, show in Theorem 4 that Algorithm achieves minimax-optimal (local) regret when facing exp-concave losses.
Let & p.g & oc, f. Bi such that B |l this is possible since f is continuous over X' with the condition d/p and embedding of B in L
Proof of Theorem 3.
Grid for scaling coefficients at starting scale jo- Observe that
|aso.t| . KS (3..)) ||/ Il
|| < 4n
BB
Let Ejo O. We define the regular grid As. of coo ppeciiiion used to learn the scaling coefficients level jo. denoted (agn.), with
|A 2 2B |011 cJu
points, regularly spaced in the interval -2 2 |éll, B2- [@]]] In the following, we will use a local grid Aun, to learn scaling coefficient On at a scale jo 1(m) locally over the space Y. In particular, we will carefully set the local precision Ei(N)) to handle regret terms.
Definition of the oracle associated to pruning. Let be (XT))EZ(T") be the associated partition of X. Let Aqn) denote the grid of precision Ei(n) as described above. We define the prediction function of pruning T', at any time >
pruning of T and P(T')
(T()()== = Σ (m)am
x E X,
BEC(T)
where each fn.m,.. is sequential predictor type (3), with starting scale jo 1(x), restricted to X,,, and initialized at the oracle scaling coefficients
(on,k) = =arg ||a Osoonlloer aEA((n)
that is, the best approximating vector in the grid Anx)) for the subset of scaling coefficients &joom, E Ay..n, whose basis functions dj.. are supported on X.. For simplicity, we slightly abuse notation by writing a E Ayn, treating the grid as a tensor grid of the same dimension as In particular, the number of coefficients in Ajo whose supports intersect X,, satisfies
Ajo
[Ago 12 [()) & A,
since 1(ni jo-
22

---

Decomposition of regret.
We have a decomposition of regret as:
Regr (f) = 4 - -
(36)
R1
R2
Hi is the regret related to the estimation error of the expert-aggregatioo algorithm compared to some oracle partition P(T') associated to T i.e. the error the algorithm commits while aiming the oracle partition P(T'). On the other hand, R₂ related to the error of the model predicting over subregions in P(T',, against some function f c B3 and corresponds to the (localised) regret discussed in Theorem 1.
Step 1: Upper-bounding R2 as local regrets. Recall that P(T') form a partition X. Hence, for any 2; X, the prediction at time t is fr (xx)) [ a with ((T')) the unique node such that 2, E N, at time t. Then, R₂ can be written as follows:
T
Ra n== = Σ
E (G(TT.(E.)) -((()))))
E RE
E
Σ
(((i)....((t))))) (z(f(x))
NEA(T))
Σ
E
(elfnannstee)) (x.))
(37)
REC(TT)
where se, = (1 (I E C. and (37) because [fnan,) & fommat and & is convex and has minimum in [-B, BJ with B B IIfll
The decomposition in (37) represents sum of local error approximations of the function f over the partition P(T'), using predictors fn,a,, n E L(T"). Recall that for every n E L(T'), fn... is prediction function associated to a wavelet decomposition (3), where the scaling coefficients start at a. over X, and with ju I(n). In proof of Theorem (Appendix A) showed that any wavelet decomposition adapts to any regularity via Rg. of f. Thus, the approximation eтror of fo. with respect to f remains similar to that in (27), but now with regard to Besov function with local smoothness and norm ||], == || Il al (xx) over X, see (10). Specifically, from (37), (23), (27), we get (without applying Hölder's 's inequality on the scaling coefficients):
&
Σ REC(T))
G
Σ kE
laso. an, [T + C22
|||B 11, 201 +)+ (Ci E Et l0s.. C2G1 2'2 200 2200+) #1
estimation wavelet coefficients (23)
(1(x)) How |1 | 4
ostimation noulisear unnoler coefficients
+ CaGlIfll (2 (60) P Tal Jo) - (1
(38)
approsimation entor X., stale
with C1, C2 as in Assumption and Cs a constant that can be deduced from (27) and jo = I(n) for
each L(T'). In particular, by definition of arg min aina ||a aaloo and given that Aijy) is grid with precision Ei(N) 0, one has
(i(n)
j0 an.al
for every k E Aga.
From (38), one can bound the absolute values of the scaling terms by Eugn and using A & For every E L(T"), let
El(w) = B(2 Unjd a
VT)
23

---

that gives for every [(T))
el(m)
(CIv C22
Uold
X B(C + C2T
Kald
kE Join
where we used v TT.l/VT & 1. Then, one can factorize the sum in j and the approximation term by 2 2-1(m)== over each n E L(T"). Finally, applying Hölder's i inequality over the sum in j (sce (21)) and following the same optimization steps in Jú, Jny AAr in Proof of Theorem I we get, with M defined as (21):
Ra & (CM + +)
+GG
Σ
Gollffle,
2-Mn)s
[TT. [T 17.2--
if
>
or <2
(39)
else,
nEC(T)
where C. CC, C2, C3, Sn, t. can be deduced from similar calculation in (30), (31) and (32) and can include log dependencies.
Step 2: Upper-bounding the estimation error Ri. Hi is due to the error incurred by sequentially learning the prediction rule fr. associated with an oracle pruning T of T, along with the best scaling coefficients (a.)ee((T)) selected from the grid (A(()))e Note that each time i, only a subset of nodes for any time > 1, we define in Algorithm 2 the set of active experts at round as
are active and output predictions. Specifically,
= ((n, an) me X..).
Moreover, assume bounded gradients: for any time t and expert e E,,
|v.=l= ((.(f((x)))) (fee < GB,
which satisfies Assumption 2 with = BG.
Using standard sleeping reduction, one can prove that, for any expert (n, an), n E t > 1 see Proof of Theorem 2 in [29] Eq. (31)-(35);
(z.)) (x.))....())))) ZEE, < ((((())))((()))) - fron + convexity of
= =(V, Wr - V(naa)).)
(40)
= -VTw V (mann))
(41)
Then, with T, = (1 X.).n E(TT):
R Σ Σ (l((f(=))) - (()) + (Nn, E L(T')) partition of
REC(TT)
Σ nEC(T')
Σ
(VT.w.
w W. W by (41)
Σ
+ by Assumption 2 log (IE) ET (V. wi (inan)e) C.ci
"EE(T")
=CBBG)((T))+ + log (1E1)
Σ
(V.w, (iaa)))))
(42)
nEc(T') V
where the last equality holds because for any n E L(T'), V. V(naa)) = 0 if æs X. and
= BG.
The proof goes on with two different cases depending on the losses' convex properties:
Case 1: (4 convex.
Observe that at any time e VV & & and ||wr]] = 1, which gives V.T.ll wel < BG.
Then, from (42)
Rí & CB + 2C's log (Iε) BG Σ
Ta
(43)
nEC(T')
24

---

In case of convex losses, we finally have by (36), (39) and (43):
(fgr(f) & 2C 222BBBG log (1E1)
Σ
VLT. + NGBMMIITIICCCCCCC
nEC(T)
if > 1 or
+ AGM
Σ
Callfle,, 2 gln(n)
(44)
[Talt=3
else,
NEE(T')
where |8] & (2 since for every n one has |Ajo.ll and
(A(m) =
[2]
by the choice of the precision ((()) = -B2 Km)a T 3 In particular, the grids have number of points that does not grow exponentially with 7', making the construction computationally feasible. Finally, since (44) holds for all pruning T' of our main tree T, one can take the infimum over all pruning to get the desired upper-bound.
Case 2: (f)) n-exp-concave.
If the sequence of loss functions (L+) is n-exp-coneave for some n then thanks to [24, Lemma 4.3] we have for any 0 < & & min g. and all t > 1, E L(T'), using (41):
(Le(f((x)))
(2) ( E.. & V. w. ((n.an).
(V. w, ((nan).)) (45) 2
Summing (45) over t E (T) and L(T'), we get:
R1
V. we (nr,an). Σ Σ WEE(T')
Σ Σ
( w.
V V(naa)))) )
REA(T)) CET
& & nEL(T")
Σ
Σ LET
(V 1eee-VVnnnnn ((nan)a))
P Σ
(V. w: V(n.)))))
2
REA(TT)
(46)
where last inequality is by (42) and we set Õ2 C' / log ((E)) and & BG. Young's inequality gives, for any v > 0, the following upper-bound:
V w. V (n.a)))))
1
+
(V.
((maa)).))
(47)
2y
2
TTT.
TET
Finally, plugging (47) with = na/s > 0 in (46), we get
C2 Σ
R1 & CBGGL(T') C2
Σ
(v. w. - Vimaniiee
2μ
20' te
nEC(T')
Σ
V. w.
((nm).))))
REC(T)) T
C3 log (1E))
C.BG L(T')),
(48)
24h
again with |8| & (21+) Then, one can deduce the final bound from equations (36), (39) and (48) and taking the infimum over the prunings T'.
Worst case regret bound. Note that since we assume that || B, and that all local predictors fe,e E Algorithm are clipped in I-B, B), we first have for any x E X,
[f((a]]= = E we.c|(e.(EE)ll & Σ We,t = B.
25

---

Thus,
T Regr (1) Σ -((((=)))
G)f((z)) - s(xi))
+- G-Lipschitz
G
[f((z))) + [f(xa)
2BGT
(49)
We now restate complete version of Theorem from the main text and provide its proof below.
Theorem 4. Let'Tss 3 1,1 p.q x0, $, Let f E Big and B 1. Let T' be any pruning of together with collection of local smoothness indices (En)ae defined as in (10) and local norms I. Then, under the same assumptions of Theorem and Assumption Algorithm satisfies
& E NEC(T")
B (B'-/"
25 (2 12-1mx) [..1l=.)) 2 v Th 2,,44
(2-1im)an |1 Tnl -- s + [T.]
and moreover we also have, if (ft) are exp-concave:
(f) E [² 22 (2 -(1))) [[51ba)) sho
NEE(T')
+ 22-
[|[|,-,, [T. (T.1-
No 2 + B).
where S hides logarithmic factors in T, and constants independent of or T.
Proof of Theorem 4.
Let T' C T be some pruming of T. We define Tia the extension of T' such that all terminal nodes n E C(T') is extended with a tree T, of depth h, N. In particular, for any n'' L(Te)),((n)) I(n) + h.. with n E L(T'). See Figure 4 for an illustrative example.
Figure 4: Example of an extended tree Tú = TUUT U......... T/ formed by a subtree T' (black nodes) and its extensions (colored nodes). Each dotted corresponds to a subtree T. rooted at leaf n E L(T') and extended to depth h. The depths vary: h2 ha 0 (black boxes), hı = ha = 1 (orange), ha he (blue), and hy (green). The leaves of T appear different levels depending on the values of (hw) and the level 1(n.) of the leaves E L(T") = (1, 4,
Observe that the total number of leaves in the extended pruning Tou is
[((Ta))
Σ
[C(T.)).
(50)
REC(T))
26

---

Define
n NEN').
(51)
Remark also that by definition of the Besov norm in (6) (and via the usual embedding (p.g) fixed see, c.g., [20]), one has for any n E L(T"), Is,, > II E ((Ta)). In particular, every tree
extension T, at node n E L(T)) has |C(T,)] = 2nd leaves.
Case (4r) convex. Applying Theorem in the convex case on the extended pruning Tess gives
Rr(f) & CG
Σ (BVITad WEDITOL
lIflls,, 22 2-1(n")s - ("")
(52)
with C' some constant that hides log I' terms that can change from inequality to another and 1 *s' is the local rate described Theorem 3. Note that by (51) one has r', & Thy n' E ((T)).
Recall that for every nt' E L(Tea),, I[n']]n==)))))))))) I(n) h, for L(T') and since every leaves in L(T,) is partitioning each terminal node E T", one has by Jensen's incquality:
Σ e'E2(T2)
Thel
Σ Σ
VI Σ VIC(TR)ITT
(53)
REL(T')n'EC(T)
nEC(T)
Then, by (51),(52) and (53) one gets (with Tnd In, ((TA))
RT( CG E
E
(BVITa/ IIfIla., (10))hh)) (Tail"M
nEC(T))
n'EC(T)
CG E
(BVIC(TATTTT)
EnEECITY) (T).
(54)
REC(T)
Further, applying Hölder's inequality over the sum over n' E L(T.)) in (54) with (1 (1- + (rr is constant over the sum in n'):
Rr( & CG
Σ
(BVIC(TRNIITA II (0)))) hn | (EWEE(TT) (F)l)
nEZ(T')
(55)
= CG
Σ
(BVITall2and + lfflla,, (p-u) (Tal"))
NEL(T")
where we used
Σ
TTdl= [Ta]
and 2 2-hnen [c =2 =-nhnn (2dthn) (eghhnn--1 11-t 2dhn (1-2 28 -ma).
R(EC(T
Define the local regrets under the sum over (TT)) by
Rn(()) = B/TT. [l. 1(n)an (d))an -m-) ITal"
optimize in h. N. This leads to two different cases depending the values of the local exponent Tn. defined in Theorem 3.
that
Case 8n > $ or < 2: Fn The local regret grows
B Th 2hnd + Il. 10m))mn
VT.
Therefore, setting An max (0, log, (2- (m)on || B-1) this entails
Rn) & C ma B² gi-xxe the (2 um))n |
T.
Case =1 1 9x
The local regret grows as:
(n( = B B + |lfll..? K())a [Tn)"
and the best choice hn in this case that entails
Rm. (f) B B + 1.l -m0
27

---

Finally, in the case (&)) are convex losses, we deduce that the regret is upper bounded as
Rr(f) CG
E Rn)
nEC(T)
& CG Σ
(max(B, B, B¹ zi (2 (2-1ln)am [[flla.) sin WTTml) m3
nEC(T))
+ (BVTa 2 [lf [T,l TTol
(56)
Case (4) exp-concave. gives
Applying Theorem the exp-concave case on the extended pruning Te
Rr(f) & CG BIC(T2))+
E
Il -
2 Unn)s, |Tar 1"
(57)
R'EC(TA))
with C' some constant that hides log I' terms and that can change from an incquality to another and ra [2 "a' } is the local rate described in Theorem 3. Note that [C((Ta)) -nnECTTy ((T))) and again for every n' C(Teu),
1(1)) h. for n L(T') and rn Tnn E ((Tr). We get
RT(f) & CG
E NEE(T)
(IC((T)) II (10)44))) EnEETTTy (T
(58)
Using [C(T))) 2nd and applying Hölder's 's inequality over the sum over n' e L(T.) in (58) with To) tra 1 as in (55) entails
RT( & CG
Σ nEC(T')
(B2nnd |1,,,,, 2- (M) -- rn [T, "").
Again, we define the local regrets under the sum over (TT))) as
Rr RT( == B2ddd 2 (1 - |T,I""
that we optimize in A., E N. The cases are the same as for the convex case, according to the values of the local exponent r., defined in Theorem
Case > 1 or 2: Tn
The local regret grows as
(f( add + |lf 2-1(m]an - Tal
Afterwards, optimizing in lú, such that
B2nnd 2 2- || f|,,, ,,,, 2 halan 4) TT.
leads to hr max {0, [ 2. logz ((B (B-12-Kkn)s) || pmm. ? DJ that entails
Rm (f) C max B, B²= 2ul a (2 (g-1(m)an 1(m)an flll 2a
TT 2om
Case $: 1 à
The local regret grows as
(f) B2rnd 21m))n [|, [T,]' -7
and the best choice is hn 0 which entails
R. (f) 2 1(m)) [| [T.] 1-2
Finally, with (f.) exp-concave losses, the regret is bounded
Rr(f) & CG
Σ Rn(
nEe(T")
& CG Σ NEC(T')
max (B, 13- zm (2 -( [15llsm) zon [Ta 2mn }1, 1.
(59)
+ (B+22 + 1BB+2-1m 1(x)a || [Ta] --
(60)
Remark. Taking T' as the pruning associated to the root, this entails (7T = O - which is minimax-optimal for this case see [35].
28

---

D Discussion on the unbounded case: < d
As in most previous works in statistical learning, this paper primarily considers competitive functions e (2) with > 5. which ensures that E (X)) with || < ce.
natural question whether our Algorithm remains competitive that is, achieves sublinear regret in the more challenging regime where _ Indeed, in the case
f. prediction rules may no
longer be bounded in sup-norm. For example, the function f(z) = x 1= belongs to [,o 1]) for p < 2 but not to ([o, 1]), illustrating the type of singularity permitted when g. In such cases, the boundedness condition Il oo < oc required in (24) may fail, where fa denotes the truncated J-level wavelet expansion defined in (2). Nevertheless, we discuss how Algorithm can still offer performance guarantees in certain settings, particularly when the input data {œt} are well distributed over X.
Indeed, by Hölder's inequality, (24) be upper bounded as:
Σ (ff(z))) - f(xx)) < T T (f)())) (((e))
(61)
where the sum on right-hand side defines an empirical e semi-metric over the input data (#t H denoted:
(fo,,)) = - Σ (f((..)) f(xe)))
The upper bound (61) suggests that tighter control may be obtained by focusing the empirical norm (Js, f) rather than on the sup-norm, which may not be finite.
First case: the empirical semi-norm di approximates the L" norm. Assume that the semi-norm df, f) is close to the true L norm lifs fller. Such equivalence is expected when the data {ze} are well distributed over X, for example when ~ ((X)) i..dd., or when æ, are equally spaced, such 2, & for t 1.... T. By the law of large numbers standard concentration arguments, one typically has ff,, = Ilfa fllze in expectation or with high probability.
Classical approximation results (c.g., [20, Prop. 4.3.8)) then yield [[s A|1 X 2- for f E (X) (2)) Optimizing over J to balance estimation and approximation regrets leads regret bound of O(T'-3 ă)- see the proof of Theorem A, last case 3 < 0. This regret is sublinear as soon as > 0 and becomes linear when = 0, as is typical for f E LP. Nevertheless, minimax analysis from [34, 35] shows that a regret of O(T is possible, which improves upon our bound whenever < Whether constructive algorithm achieving this minimax regret exists in the regime question.
remains, the best of our knowledge, open and interesting
Second case: the semi-norm d.. fails to approximate the L norm. If the data points {æx) are concentrated near singularities (e.g., g. near 0 the example above), the empirical norm de (fa, f) can differ significantly from the true norm Ilf- fllee, making the latter less informative in practice. In such adversarial or non-uniform settings, it seems preferable to control the empirical norm de (fs, directly, as more accurately reflects the distribution of the observed data. Addressing this challenge may require adaptive sampling strategies, localization techniques, or alternative norms that account for the geometry or density of the input distribution.
29

---

E Review of multi-resolution analysis
In this section we present some of the basic ingredients of wavelet theory. Let's assume we have multivariate function f Rd R.
Definition 1 (Scaling function). We say that function ó E L²(RR) is the scaling function of multiresolution analysis (MRA) it satisfies the following conditions:
1. the family
{x δ(x = -1 0(x. - -n.) :.):ke kekkezg
is an ortho-normal basis, that is (o(. - 1:) - n)) = Ōz.ni
2. the linear spaces
Vo {s = EnEza cu - (cx): Exezo of 6o))...... (h= f(2.)) if f Vo).... are nested, i.e. V,-1 C VV, for all j > 0.
We note that under these two conditions, it is immediate that the functions
f 24//22(2" -k) k e 2d}
form an ortho-normal basis of the space Vj.j E N. One can define the projection kernel of / over Vj (from here also say kernel projection scale or level j)
K,f(x)=
Σ (f:GsN))sAN))) KEZ
/ (,(z,))f(o)d
(62)
with (2 = Enezd , (z)))),,())) = Exers 2 602'022 - n))((2n) (which is not of convolution type) but has comparable approximation properties that we detail after.
Incremental construction via wavelets. Since the spaces (Vz) are nested, one can define nontrivial subspaces the orthogonal complements Wj Vi+1 VG. We can then telescope these orthogonal complements to see that each space V, 2े jo can be written as
Vn = Vi +
+ wi 1=j0
for any jo E N.
Let s be mother wavelet corresponding to the scaling function The associated wavelets are defined as follows: for E = {0, 133d {0}, we set
3 (w)) = 4f (x1) ... (xa); ii = 2512 (2'x n), j 0,0, n E 2
where 20 Φ, r vo. For each j. these functions form orthonormal basis of W,. Analogously, one can now observe that for every 2 ju.
K,f = Kjof
Σ
(Kurf - -Kf)
(63)
1=
where each increment the sum can be written as
Kj+1f - K2S Σ E
(1. (1.V
where for each j 1, the set
1/ = 22/2 (2'2 (2' F kE
forms a basis of W2 for some wavelet vh with E 10 (0). For simplicity, we include the index e in the multi-index Finally, the set (o),.k, } constitutes wavelet basis. For our results we will not be needing particular wavelet basis, but any that satisfies the following key properties.
Definition (S-regular wavelet basis). Let S N" and jo The multiresolution wavelet basis
{6x = o(. k:,, Vi.e 2///2 "02"""""" 10(2"--0) -k))
of (Rd) with associated projection kernel K(x, v) E. (x()))d(()) is said be S-regular if the following conditions satisfied:
30

---

(D.1) Vanishing moments and normalization:
Jua (xx) x" du for all multi-indices. with |a| < Moreover, for all R and with & |a] <. Jg K(v,u+ u) 11, Jaa K(u u u =
(aa =1.
(D.2) Bounded basis sums:
Ma = supuead Ex 16(z k)] <0;
M3 == supaeR Σx 115x k:] < co.
(D.3) Kernel decay: For k(r. equal to K(x.y) or Ee 1/ k)pb( k), there exist constants
C1, 02 > O and a bounded integrable function & : [0, 00) - R such that
supuead [k(u, u)] @1
Cs Sa [|ll/
Case of bounded compact X CR. The above definition applies wavelet systems on R but can be extended to compact domains X C Ied using standard coundary-corrected or periodized con- structions. Notable examples include the compactly supported orthonormal wavelets of Daubechies [11, Chapter 7] and the biorthogonal, symmetric, and highly regular wavelet bases of Cohen et al. [9]. Just as in the case of R. we can build a tensor-product wavelet basis ( $5,,)), for example using periodic or boundary-corrected Daubechies wavelets. At the j-th level, there are now O(2id) wavelets y.e. which we index by E A,, the set of indices corresponding to wavelets at level j. This coincides with the expansion used in Equation (2).
Control of wavelet coefficients and characterization of Hölder spaces. Remarkably, the norm of the space 8 (x)) ) has useful characterisation by wavelet bases see [31] or [20] for review on the characterisation of smoothness according to wavelet basis.
Proposition 1. Let O thus have the following:
f E " =
sup V5Aa & C1
(64)
where C = C(b,, S) is some constant that depends only on the (S-regular) wavelet basis.
Proof. Let ub be a compactly supported wavelet in Rd with S vanishing moments, i.e.,
J(e x² (x) dx = 0 for all multi-indices /3 with | S.
the wavelet vanishing moments match the regularity of : Let (2)) = 204/2 (2'x k) be the wavelet at scale and location & e z The wavelet coefficient is given by
Assume that fee (R"))) for some > 0, with
C (..bj,)) [ (()))),(()) de.
We define the center the wavelet support as FG,b == 2 and write a Taylor expansion of f at zy,t
f(x) Pay.a (x) Rey, (x),
where Pay, the Taylor polynomial degree [s] and RRa,, (x)) < lfbll xy.ollãc for near ZsA and where [f]. suppm|=[s] [| flla-im|- Using the vanishing moments of v., we have
C, [ Ray, (x)))5,())
Now perform the change of variables u = 2xx k, 5o 2 + uga and de = 2- 3-ddus du:
C = 5ddf2 5d/2
2 u))
By the Hölder remainder estimate, we have RRaz (1.,A + 2-) & 1fla|12-' 'ul 1fl.2 |u
Therefore,
leyull & 1f1.2 (=+d//))))//)
(v( || du,
and since v is compactly supported and smooth, the integral is finite. Hence, defining C(b,s) f [[]]]|||||] du < oc we get the result.
Remark. The smoothness o translates into faster decay of the coefficients given sufficiently (S > regular wavelets.
31

---

F Summary of the results and comparison to the literature
Table 2: Regret rates, parameter requirements and time complexity for online regression algorithm with (f,)
square losses 8 > d/7-
Paper
Setting
Input Parameters
Regret Rate
Complexity
Vork [39]
B l1f
exp(7) Td
Vovk[40]
f He p2 Sew" p=00,
Not fessible
wew2. 232,
exp(s7)
Guillard and Gerchinovitz [17]
1 Wg. p22,8
ya-g
exp(a7)
p=.. d=
(oly
Linutaud al. (29)
fe p= (PA.))
B
(ob(()
Zadorowhnyi al. [42]
We. 2
T'
poly(T)d
Alg.
J Bia p.9 porps2
VT
poly(Tys
This work
J Kigo 932
I Rio p.
T
Alg.2
T2se<sssp.03 MMll
poly(7)s*
S Rie p22.g31,5
Minimax rater
1 B p.
Non constractive
T
Non constructive
Rakhlia and Sridharan [34, 35]
f Bi p22.93
Comparison to Vovk [40]. Vovk [40] provide a general analysis for prediction in Banach spaces, focusing on the regime > 4/y. They achieve regret rates of (/1) for certain Besov spaces Bi with p > 2 and q E [p,(p-1), p]. These rates are independent of the smoothness parameter 8, except in the case p oc, where they obtain - However, this remains suboptimal in their setting with square loss. In contrast, our analysis yields the minimax-optimal rate O(T 2444) over broader class of Besov spaces Byg with arbitrary p. E [1, oo] and > 4/p.
Comparison to Vovk [39]. Vovk [39] investigates prediction under general metric entropy con- ditions, proposing algorithms that compete with reference class of functions in terms of covering numbers. While their approach is highly general and applies to a broad range of normed spaces, the regret bounds they derive, of order O(T) -f2)) still do not match the minimax-optiman rates known for functions in B
Comparison Zadorozhnyi et al. [42]. Their approach focuses on Sobolev spaces Wx)) with p 3 2 and s > L. and they obtain suboptimal rates, in the regime & < < of O(T' - +1,, for arbitrarily small e. In comparison, our rates O(T*= -2ta) ) are minimax-optimal over a broader class of Besov spaces Bbg with arbitrary p, q satisfying d. which include the Sobolev balls considered in their work.
Computational complexity. Most existing work in online nonparametric regression over Besov spaces (including Sobolev spaces), such Rakhlin and Sridharan [34, 35], Vovk [39, 40], does not provide efficient (i.e., polynomial-time algorithms. The work by Rakhlin and Sridharan [34, 35] offers minimax-optimal analysis, but does not yield constructive procedures- computing the offset Rademacher complexity, as required by their method, is numerically infeasible in practice. The approach of using the Exponentiated Weighted Average (EWA) algorithm in nonparametric settings, as proposed by Vovk [39], suffers from both suboptimal regret rates and prohibitive computational complexity. since it requires updating the weights of each expert in covering net, leading total cost of (eexp(T)). Vovk [40] introduce the defensive forecasting approach, which also avoids efficient implementation as relies on the so-called Banach feature map a representation that is typically inaccessible or intractable in practice. The Chaining EWA forecaster of Gaillard and Gerchinovitz [17] achieves optimal regret bounds in the online nonparametric setting. However, its algorithm is provably polynomial- polynomial-time only in the case and p, its direct implementation requires O(exp(dT)) operations. Zadorozhnyi et al. [42] propose an efficient algorithm with total computational complexity of order O(T -(T*+40 dT²). We note that their algorithm has a linear cost d, making particularly suitable for high-dimensional settings with smooth competitors in (x()') (with )
and d = 1; in general dimensions
32

---

Finally, our algorithms are both optimal and efficient, with computational costs (after T rounds) of
O(TxJxS") 0 S de (og,(T)))) and X[xx|XX - o 0
S
log, (T)*
de
for Algorithm and Algorithm 2 respectively (taking partitioning tree of maximum depth Ja 8 de log, T.).
G Besov embeddings in usual functional spaces
We refer [8. 13, 20] for precise statements of the classical embedding theorems. For convenience, we recall some of the most useful embeddings in Table 3.
Table 3: Classical emheddings of Besov spaces Biy
Condition on (s. p.
Target Space
Embedding Type
8>
L
Continuous embedding
p¹ 9=
L
Critical embedding
8> d -
P&T
L
Continuous embedding
>
Bpg
Continuous embedding
8 & p2, 6
Byav2
Continuous embedding
B%
W
Equivalence (for N)
8
Norm equivalence with Hölder
H Summary of optimal regret in Online Nonparametric Regression
This section summarizes the results in [34, 35] for mimimax-optimal rate of regret in the adversarial online nonparametric regression setting.
Proposition 2 ([35)). Assume the sequential entropy at scale 0 a) for the target class function. Optimal regret is then summarized the table:
Table 4: Optimal regret for different loss functions
Loss Function
Range on a
Optimal Regret
E (0, 2]
T2
bsooute loss
a > 2
T
E (0, 2]
7 24a
Square loss
T
In particular, for Hölder functions (**)),, > 0, and (2) s > one has a = d
33

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We give an overview of the related literature and of our main results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: After providing each result we discuss the limitations with respect to the literature. We discuss the computational efficiency of our algorithm.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
34

---

All the theorems, formulas, and proofs in the paper are numbered, cross- referenced and well detailed.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [NA]
Justification:
We only provide graph to illustrate and explain our theoretical results.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
35

---

Answer:
[NA]
Justification:
We do not have experimental results on data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
Sonedeee/CodSuummiieie wwwsssssssoiooiooiooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips wttpccccpppccccceeececccccccccct guides/ /CodeSubmic / seniisssionn iesionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification:
No experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: No experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
36

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: No experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https / /ttp///eerpp///eeee cc htcpppulcc/Ehhccuuieeeeeeoooeeee
Answer: [NA]
Justification: Yes and have reviewed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact in the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
37

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks as it is general theoretical work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
38

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowd-sourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
39

---

The core methods development in this research involves pencil and paper. No LLM was used this end.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /atass///nuurttee wweurips.cc/Conferreeeee.. cw/Coc///Ccc/ccc/cc/c / for what should or should be described.
40


Reviews:
[{"review_id": "yavtRqUlOy", "reviewer": "Reviewer_GTjW", "summary": "This paper studies online nonparametric regression using a competitive framework, i.e., bounding regret compared to the best predictor in a given function class (here, given by a Besov space). Specifically, the paper proposes a wavelet-based Algorithm 1 that is simultaneously competitive with predictors from any Besov space, up to a predetermined smoothness level $S$. The paper then proposes a \"spatially adaptive\" Algorithm 2, which uses a collection of experts  (each trained using Algorithm 1) assigned to different dyadic partitions of the space, and shows that Algorithm 2 obtains correspondingly stronger regret guarantees.", "questions": "1. What exactly is meant by the term \"parameter-free\" used throughout the manuscript? Does this refer to the fact that the parameters $s$, $p$, etc. need not be available to the algorithm? How does this differ from the term \"adaptive\" used throughout the paper and elsewhere in the statistical literature?\n\n2. A key result in the statistical literature on Besov function estimation (due to [13, 14, etc.]) is that linear estimators (e.g., simple projection or kernel estimators) necessarily fail to achieve minimax optimal convergence rates (under certain loss functions) over many Besov spaces, and that some nonlinear mechanism, such as thresholding, is needed. If I understand correctly, the construction of the active wavelet set plays a similar role here. Is there an analogous lower bound for this setting, e.g., where a simpler class of estimators that are competitive with Holder-smooth functions provably fail to compete with Besov-smooth functions? This might also be related to the spatial adaptivity advantages of Algorithm 2 over Algorithm 1, but I'm not sure if this is the same, since usually spatial adaptivity is within a single Besov space rather than across different (locally restricted) Besov spaces.\n\n3. The loss function is assumed to be both convex and Lipschitz; does this need to be true on the whole real line, or can this assumption be restricted to a bounded domain? On an unbounded domain, this is a lot more restrictive (i.e., the loss function is essentially linear outside some bounded region).\n\n4. Could the authors provide a bit more intuition for Assumption 1? It's not really clear to me what it means. Maybe an example (of the type of behavior it is restricting) would be helpful?", "limitations": "N/A", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. Though dense, the paper is generally well-written and easy to follow. In particular, the paper does a good job both discussing demonstrative special cases/examples (e.g., Figure 3) and summarizing the results at a high level (e.g., with Figure 2).\n2. The proposed method does not require any strong knowledge of the data-generating process; in particular, it is impressive that the same algorithm is competitive with predictors from any Besov space with $s < S$.\n\n### Weaknesses\n1. As someone used to nonparametric regression in an offline, statistical/stochastic context and unfamiliar with the online learning setting, it took some adjustment to follow the paper's ideas, such as the \"competitive\" evaluation framework. The competitive framework seems designed to optimize prediction performance (under a particular loss function) while minimizing the assumptions made the data-generating process. However, it's unclear to me how much one can infer about the underlying data-generating process from this approach. For example, if we assume that the data is generated from a Besov-smooth function (i.e., $f(x) = \\mathbb{E}[Y|X = x]$ for some $f \\in B_{p,q}^s$) and train a predictor $\\hat f$, what do the given bounds tell us about $f$ (perhaps with some additional assumptions on the noise)?\n2. The main paper doesn't really discuss the proofs of the main results at all. I understand that space is limited, but a few sentences would be nice to give an idea of the key ideas/overall approach.", "was_revised": true, "final_justification": "Thanks to the authors for their response. I found the comparisions to the batch setting helpful, and suggest these discussions be added to the paper. I still find Assumption 1 rather mysterious and think concrete examples (violating & satisfying the assumption) would help here. Also, I am not familiar enough with the online learning literature to follow the concerns about novelty (compared to prior work by Baby & Wang and Zhang et al.) brought up by Reviewer LkUW. Given these, I have reduced by confidence from 3 to 2, but am retaining my overall rating of 5.", "rebuttal": "Dear Reviewer,\n\nThank you very much for your constructive and insightful comments.\n\n**I) Parameter-freeness and adaptivity:** Your question about the distinction between \"parameter-free\" and \"adaptive\" indeed points to an important subtlety in terminology, and we appreciate the opportunity to clarify this in our paper.\n\n*First about the terminology:* in the online optimization literature, the term **'parameter-free'** refers to algorithms that do not require tuning any user-specified hyperparameter (e.g., a learning rate or regularization coefficient), in order to achieve optimal regret bounds. More precisely, it typically denotes algorithms satisfying bounds of the form\n\n$$\n\\sum_{t=1}^T g_t(c_t - c) \\leq O\\left(|c| \\sqrt{\\sum_{t=1}^T g_t^2}\\right),\n$$\n\nfor any comparator $c \\in \\mathbb R$. This implies *automatic adaptivity to the norm* $|c|$, without prior knowledge on it. It is a central notion in the monograph by F. Orabona (*A Modern Introduction to Online Learning*, Chapter 10), as well as in [10, 31]. This is sometimes also called 'comparator(-norm) adaptivity' (see [30]). \n\n*Regarding our setting:* this parameter-freeness property is encapsulated in a general Assumption 1 (with starting at $c_1 = 0$), which allows us to plug in any such online algorithm as a gradient step in our wavelet-based Algorithm 1. This assumption is then crucial in our analysis: we bound part of the regret using this norm-adaptive behavior with respect to each wavelet coefficient. Intuitively, it allows us to control the regret through both the structure and magnitude of the scale/detail coefficients $|c_{j,k}| \\lesssim 2^{-j s'}$.\n\n*Relation to statistical adaptivity:* in contrast, **'adaptive'** in the statistical literature usually refers to algorithms that utomatically adapt to unknown regularity parameters of the target function (such as smoothness $s$, integrability $p$, etc.), often achieving minimax-optimal rates without prior knowledge of these quantities. In our case, we use *parameter-free subroutines* to build a *statistically adaptive* method: our regret bounds hold uniformly over Besov functions indexed by any $s, p, q$, without these parameters being given to the algorithm.\n\n**II) Convexity and Lipschitz assumptions on the losses:**\n\nIn our analysis, the loss function only needs to be convex and Lipschitz on the domain where the algorithm operates - i.e., over the range of predictions $\\hat f_t(x_t)$. Typically, if we assume that the comparator $f$ is such that $|f|_\\infty \\leq B$, we may clip $\\hat f_t(x_t)$ (as done for Alg. 2) and require convexity and Lipschitz assumptions on $[-B,B]$ only. \n\n\n**III) Nonlinear estimation and spatial adaptivity:** Thank you for this insightful and technically deep question.\n\n- You are absolutely right that, in the offline statistical setting (Donoho et al.), linear estimators fail to achieve minimax rates over many Besov spaces. In our online setting, the same phenomenon arises, hence the need for a nonlinear mechanism. The algorithm dynamically updates the wavelet coefficients in the active set over time, thereby mimicking a nonlinear adaptation oracle.This is performed via the use of parameter-free subroutines that - roughly speaking - directly track the best (i.e. thresholded) wavelet coefficients. Indeed this mechanism is essential to achieving minimax-optimal regret bounds while adapting to the unknown Besov regularity of the target function. We will add comments in the main text to better highlight this, and introduce the nonlinear oracle that our algorithm implicitly mimics.\n\n- *Regarding your observation 'this might also be related to the spatial adaptivity advantages of Algorithm 2 over Algorithm 1':* Our Algorithm 1 already mimicks the behavior of the ideal selection of $(j,k)$ indices - that is, the optimal set of active wavelet coefficients, which serves as the indices constituting the (thresholded) nonlinear oracle. This is achieved through the use of parameter-free subroutines that effectively target the 'thresholded coefficients'.    \nIn contrast, Algorithm 2 also supports *local* adaptivity across different regularity classes, not just within a fixed Besov space. Donoho et al’s notion of spatial adaptivity in [13] refers to adaptation to inhomogeneities within a single (global) Besov space - e.g., detecting jumps or peaks via wavelet thresholding. Our method goes beyond the (global) spatial adaptivity considered in [13] and enables the algorithm to handle heterogeneous smoothness $s$. In particular, we show that this local strategy can adapt to different levels of smoothness across regions and improves over global guarantees. We will clarify this distinction more explicitly in the final version and add comments to emphasize the parallel with the litterature of Donoho's and al.\n\n- *About the lower bound:* While our current focus is on establishing that our algorithm achieves optimal performance across a range of Besov spaces, we do not provide an explicit lower bound showing the failure of linear estimators in the online setting. Nevertheless, such lower bounds can be derived: since the *offline/batch i.i.d.* smoothing problem is strictly 'easier' than the *online* one, any lower bound on the minimax risk (e.g., MSE) of linear smoothers in the offline setting (as in [14]) implies a lower bound on the regret of linear predictors in our online setting. We will consider adding such a result in the supplementary material of the final version to further motivate the need for a nonlinear mechanism.\n\n**IV) Guarantees in the batch setting:**\n\nOur guarantees in the paper are stated in terms of *regret* and  we make no assumptions on the data-generating process - only boundedness of the observations. However, one may naturally be interested in guarantees in a *batch* or *statistical* setting. Indeed, our *regret* upper-bounds can be used to derive *excess risk* bounds via standard *online-to-batch conversion* (e.g., see N. Cesa-Bianchi, A. Conconi, and C. Gentile. *On the generalization ability of on-line learning algorithms*). Typically, if we now consider iid data $(X,Y)$, and if we define a batch estimator $\\bar f_T = \\frac{1}{T} \\sum_{t=1}^T \\hat f_t$ as the average of the online predictors, then the expected excess risk of $\\bar f_T$ with respect to any Besov-smooth function $f$ would typically scale as our regret bound divided by $T$. Note that $\\mathbb E[Y|X]$ may be any bounded measurable function and non necessary Besov.   \nFinally, if the question is to obtain guarantees on the mean integrated squared error (MISE), i.e., $\\mathbb{E}[\\|\\bar f_T - f\\|_2^2]$, with $f \\in B^s\\_{pq}$ then in the specific i.i.d. case where $Y = f(X) + \\varepsilon, E[\\varepsilon | X] = 0$ with i.i.d. uniform design, our regret bounds provide also optimal guarantees whenever $s \\ge d/2$. Interestingly, we believe that Algorithm 2 could also be extended to derive guarantees for general $L^{p'}$-risks with $p' > p$ in our setting - a promising and challenging direction for future work, given the rich literature by Donoho, Johnstone, Kerkyacharian, and Picard."}, {"review_id": "clJ3rcaPLi", "reviewer": "Reviewer_ZsGV", "summary": "This paper develops online learning algorithms that adapt to local irregularities of the function class played by the adversary. The algorithms are based on parameter free online linear optimization algorithms, applied to each component of a wavelet decomposition separately, then aggregated. The resulting bounds are adaptive to the besov norm of a comparator function. Theorem 1 gives a trichotemy showing that different rates are possible with the stated algorithms depending on the relationship between dimension and smoothness. Theorem 2 gives further bounds in the case of spatially inhomogeneous smoothness. The boudns match the minimax rates in a wide range of cases.", "questions": "-", "limitations": "-", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well written and easy to follow. The work combines recent results in parameter free optimization with sophisticated techniques from applied maths in order to develop something broadly applicable and useful, as well as of theoretical interest.", "was_revised": true, "final_justification": "I am still in favour of accepting. I maintain my score.", "rebuttal": "Dear Reviewer,\n\nThank you very much for encouraging feedback. We sincerely appreciate your positive assessment of our work. We will make sure to replace citations of arXiv versions with the corresponding published versions in the final submission."}, {"review_id": "2XeuO9gLJB", "reviewer": "Reviewer_LkUW", "summary": "This paper studies the problem of online nonparametric regression where the hypothesis class is a general Besov space. An adaptive algorithm based on wavelet decomposition and parameter free online learning is first proposed to achieve the minimax optimal regret bound without knowing the parameters of the hypothesis class beforehand. Then, inspired by (Liautaud et al., 2025), an improved algorithm based on expert aggregation further improves the result to local adaptivity.", "questions": "1. I did not understand the expert construction in Algorithm 2. Could the authors elaborate on it? The description of the algorithm is also dense, therefore if possible, I would suggest revising the writing to highlight the key idea. \n\n2. I am also wondering if Algorithm 1 can directly achieve local adaptivity without expert aggregation, or put in another way, whether the analysis of Theorem 1 is loose. Any comment or insight would be helpful.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- The considered problem is a challenging one that meaningfully extend the setting from an earlier work (Liautaud et al., 2025) with a similar theme. \n\n- The technical development in this paper is solid as far as I can tell. \n\n- The results are quantitatively strong, achieving optimality across a number of settings and parameter regimes. \n\nWeaknesses\n\n- It is claimed in the paper that the proposed algorithm is the first adaptive algorithm to bridge wavelet theory with online nonparametric learning, which I think is not accurate. Online nonparametric regression is known to be related to the so called dynamic online learning problem as discussed in the related work section of Baby and Wang (2019), where there is no context but the comparing hypothesis can be time varying. This setting is a special case of online nonparametric regression where the feature space is just the sequence of time indices 1,...,T. Baby and Wang's works (2019,2020) combined wavelet theory with averaging and the VAW forecaster to achieve the optimal dynamic regret, extending earlier works on trend filtering to the sequential setting. More recently, Zhang et al (2023) combined wavelet with parameter free online learning to tackle the case with general convex loss functions, and their algorithm appears to be identical to Algorithm 1 proposed by the submission despite using discrete wavelets. The relation to these works needs to be thoroughly discussed. \n\n- The technical development of the paper closely parallels the earlier work of Liautaud et al (2025), and the novelty compared to that is not sufficiently highlighted. Quantitatively, the proposed algorithm can handle more general hypothesis classes than Liautaud et al's, but given the similarity of the two algorithms, it is currently less clear what is the key technical contribution and how significant is that. \n\nReferences\n\nBaby, Dheeraj, and Yu-Xiang Wang. \"Online forecasting of total-variation-bounded sequences.\" Advances in Neural Information Processing Systems 32 (2019).\n\nBaby, Dheeraj, and Yu-Xiang Wang. \"Adaptive online estimation of piecewise polynomial trends.\" Advances in Neural Information Processing Systems 33 (2020): 20462-20472.\n\nZhang, Zhiyu, Ashok Cutkosky, and Yannis Paschalidis. \"Unconstrained dynamic regret via sparse coding.\" Advances in Neural Information Processing Systems 36 (2023): 74636-74670.", "was_revised": true, "final_justification": "Overall, my evaluation of the paper is that it makes some good contributions in terms of the final results, but the significance, novelty and impact of these results are not particularly strong. It probably started as an attempt to generalize the work of Liautaud et al, with the use of wavelet machinery being the main technical novelty. However, the works of Baby & Wang and Zhang et al, which the authors were not aware of, have shown similar ideas before, therefore the significance and novelty of this paper are unfortunately weakened to a certain extent.\n\nI have no major problem seeing this paper accepted, but if there exists an acceptance threshold, then I feel this paper is not significantly above it. As the result, I've increased my final score to 4.", "rebuttal": "Dear Reviewer,\n\nThank you for your feedback and for pointing out these references. Below, we address your comments point by point and hope these clarifications help demonstrate the broader scope and novelty of our approach.\n\n**I) On prior work of Baby and Wang:**\nWe acknowledge that wavelet-based techniques have been explored in the line of work by Baby and Wang. Nevertheless, their analysis focuses on a specific application: they use an online algorithm (VAW) for batch denoising with wavelets, where the data is assumed to be i.i.d. and lies on a uniform grid. As such, their setting is stochastic and non-adversarial, and aligns closely with problems studied in the signal processing community. In contrast, our work develops a wavelet-based algorithm specifically for the *online adversarial learning* setting. We will make this distinction more precise in the related work section of the final version.\n\nThat said, our setting and contributions significantly generalize this line of works in several key aspects, here are some:\n- while their method apply to specific discrete Sobolev- or Hölder-type classes, they remain confined to structured sequences over 1D grids. The input domain is implicitly the time axis $\\{1, \\dots, T\\}$. In contrast, our algorithms operate in a general contextual setting $x_t \\in \\mathbb{R}^d$, without relying on grid structure or discretization. This allows approximation over general domains $\\mathcal X$ via multiscale wavelet decompositions and in an adversarial setting. They use the discrete wavelet transform (DWT) over sequences, while we construct and analyze algorithms using continuous wavelet expansions in general Besov spaces $B^s_{p,q}$.\n- they consider discrete sequences $\\theta\\_{1:T} = [f(x_t)]\\_{1:T} \\in \\mathbb{R}^T$ over the 1D time index $\\{1, \\dots, T\\}$, with regularity enforced via finite differences (e.g., total variation of order $k, \\mathrm{TV}^k(C_T)$). Typically, in our setting the data context $(x_t)$ does not live on a grid and is unknown in advance. It is sequentially revealed in an arbitrary ordering. In our setting, their total variation constant $C_T$ could behave as $C_T = O(T)$, making their regret upper bounds vacuous. \n- finally, their function class $\\mathrm{TV}^k(C_T)$ is a particular class of functions sandwiched between Besov-type spaces\n  $\n  B^{k+1}\\_{11} \\subset \\mathrm{TV}^k \\subsetneq B^{k+1}\\_{1\\infty}\n  $\n  all of which are naturally handled by our method, since we are competitive with any $B_{pq}^s$. In particular our algorithm also achieves their minimax optimal dynamic regret in their setting. Letting $\\ell\\_t(\\hat y) = (y_t - \\hat y)^2$, suppose $f(x\\_t) = \\theta\\_{1:T}[t]$ for some $\\theta\\_{1:T} \\in \\mathrm{TV}^k(C\\_T) \\subset B^{k+1}\\_{1,\\infty}$, and $y_t = \\theta_{1:T}[t] + \\varepsilon_t$ with independent Gaussian noise. \nWhile our predictions $(\\hat f_t)$ make *no stochastic assumptions* beyond the boundedness of $(y_t)$, our regret (for $s = k+1$) directly upper bounds their dynamic regret when taking the expectation (see their discussion in Appendix B, paragraph competitive online nonparametric regression). Thereby, our method also achieves their optimal dynamic regret of $O(T^{1/(2k+3)})$.\n\nWe will add these distinctions in the related of the final version.\n\n**II) About Zhang et al. (2023):**  \nThe recent work by Zhang et al. (2023) introduces an online algorithm that combines discrete wavelets with parameter-free learning to minimize dynamic regret under general convex losses. While their method does share structural similarities with our Algorithm 1 - both maintain and update coefficients using a parameter-free subroutine - the wavelet basis they use, their goals, and their analysis differ fundamentally from ours.  \nZhang et al. focus on obtaining sparse dynamic regret bounds in a parametric framework (finite dictionary). Although they draw connections to nonparametric regression, by considering large dictionaries thanks to their sparsity results, they do not derive explicit rates with respect to function classes such as Besov or Hölder spaces. In contrast, selecting an appropriate wavelet basis and analyzing rates with respect to these function classes is central to our approach - and not addressed in their analysis.  \nMoreover, their algorithm does not consider the exp-concave setting, where, similar to our Algorithm 1, it would fail to achieve minimax rates. In contrast, our Algorithm 2 is specifically designed for this setting and introduces both conceptual and technical innovations to achieve optimal performance.  \nThank you for pointing out this relevant reference; we will add it to the related work section and clarify this distinction in the final version.\n\n**III) Contributions and clarification regarding Algorithm 2:** \n\n- *About contributions in regard to the work of Liautaud et al.:*  \nOur current work significantly generalizes the previous setting by competing against a much broader class of functions - namely, Besov-smooth functions - whereas the earlier work of Liautaud et al. focused on Lipschitz-smooth functions. This extension requires a substantially more intricate analysis of the algorithm due to the richer structure and potential irregularity of Besov spaces. For instance, the need for a nonlinear oracle analysis and sparse wavelet representations across scales, or the challenge of handling spatially inhomogeneous smoothness. In particular, the analysis must adapt to both the regularity and integrability properties encoded by the Besov parameters $(s, p, q)$, which introduces significant technical complexity compared to standard Hölder classes with $p=q=\\infty$. \nTo our knowledge, this is the first algorithm designed and analyzed with optimal guarantees against adversarial Besov-smooth environments, covering the full range of smoothness regimes embedded in $L^\\infty$, in a fully *online* and *adversarial* setting. In addition, we provide concrete examples to show that our method achieves faster rates via local adaptation, particularly when the target function exhibits high regularity locally. In contrast, the approach in Liautaud et al. is inherently limited to $s \\leq 1$.\nWe will consider adding further intuition and key steps from our analysis to the main text in order to better highlight these contributions.  \n\n- *About Algorithm 2:* It introduces a spatially adaptive, tree-structured framework that enables local adaptation to heterogeneous Besov regularity, improving over the global guarantees of Algorithm 1. Briefly, it leverages the localized support of wavelets to build region-specific predictors with tailored coefficients, which are then aggregated using a second-order expert procedure to achieve local adaptivity and optimal regret. In particular, it supports recovery in settings with spatially varying smoothness $s$, adapting *locally* instead of paying for the lowest regularity *globally*.\nRegarding your question: Algorithm 1 does not achieve such local adaptivity, as it applies a uniform strategy across the entire domain $\\mathcal X$. This limitation is intrinsic to the global regret bound in Theorem 1. Algorithm 2 overcomes this by aggregating local predictors specialized to subregions and tuned to the corresponding local regularity levels - thereby recovering different local rates within a unified framework. We also highlight that Algorithm 2 supports acceleration for exp-concave losses, which is not captured by Algorithm 1.\nFinally, we agree that the expert construction in Algorithm 2 may appear dense, and we will work to clarify its core idea in the main of the final version."}, {"review_id": "vU0gJhFbmK", "reviewer": "Reviewer_eiQR", "summary": "This paper tackles online learning in Besov space using a wavelet decomposition approach. It presents two online learning algorithms. The first can be thought of as a regression problem over a (truncated) wavelet basis containing 1 \"coarse\" subbasis and J-1 \"detail\" subbases. Each wavelet function corresponds to a cubic volume of the bounded domain built based on a doubling approach. The second algorithm builds upon this first to tackle heterogeneously smooth functions. The aforementioned cubic volumes are arranged into a tree so that predictors can be localized. They are then combined into a final prediction using specific weights, learnt jointly with the local predictors. Regret bounds and convergence results are derived for both algorithms, which are shown to adapt to the underlying target function.", "questions": "See \"Strengths And Weaknesses\"", "limitations": "Yes, the authors have discussed limitations of the work in the conclusion.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper puts forward and analyzes minimax optimal algorithms for online approximation of Besov functions, including for functions with heterogeneous degrees of smoothness. While mostly clearly written, the density of the material in some parts warrants some improvements in clarity. In particular,\n\n- While the conclusion mentions the exponential dependence on the smoothness parameter, it would be good to also have a discussion on the space complexity of the algorithms on the dimension. This is admittedly an issue in many wavelet methods, but I wonder if some form of adaptive tesselation could help (i.e., using an adapted wavelet basis)?\n\n- It should be pointed out that a large portion of the complexity of the proposed algorithms is encapsulated in Assumption 1. In particular, the complexity of mirror descent updates is not taken into account in Table 2 (Appendix F). While they are polynomial in nature, the constants could be detrimental depending on the conditioning of the problem (potentially bringing another issue aside from the exponential dependence in smoothness).\n\n- Assumption 2 is also not easy to interpret and could use more contextualization to make the paper more self-contained. As it is, the algorithms presented in the paper are really more \"meta-algorithms.\"\n\n- This is particularly the case for Algorithm 2, which requires substantial work from the reader to parse. In particular with respect to the description of the local predictor and its relationship with the global predictor at scale $j_0$. For instance, line 284 states that the local expert predictor (at scale $j_0$) is the restriction of the global predictor (at scale $j_0$) to the chosen subcube with the additional modification that the scaling coefficients $\\alpha_{j0}$ are set to node specific coefficients $a_n$, which are not particularly clear (except for the fact that they \"are supported on a grid $\\mathcal{A}$ of precision $T^{-1/2}$\"). It is unclear at this point how (or whether) the basis coefficients are shared among predictors (or what the role of $a_n$ is in this context). It would also be helpful to more explicitly define other quantities fundamental to the procedure, such as the active index set $\\Gamma_{e,t}$.", "was_revised": true, "final_justification": "I am satisfied with the authors' arguments. I, however, strongly encourage them to include these relevant discussions in the manuscript as I think it will only improve its clarity.", "rebuttal": "Dear Reviewer,\n\nThank you very much for your constructive feedback. We appreciate your thoughtful reading and your suggestions, which helped us identify several areas for improvement. Below, we respond to your main points about complexity and assumptions.\n\n- **On exponential dependence in dimension and adaptive tessellation:**\nYou are absolutely right that wavelet-based methods typically incur exponential complexity in the dimension - a known limitation of multiscale constructions. While our theoretical guarantees apply in arbitrary dimension, the practical space complexity grows as $O(S^d)$, where $S$ is the smoothness parameter and related to the size of the support of the wavelets.\nThis dependence arises even with compactly supported wavelets such as Daubechies, which are known to have minimal support $O(S)$ and overlap while achieving a target regularity $S$. As such, to the best of our knowledge, this overhead is intrinsic to wavelet representations in high dimensions.\nWe agree that exploring adaptive tessellation or basis/regularity adaptation is a promising direction. We would be happy to integrate any references or ideas you may have along those lines.\n\n- **Complexity of the gradient step:**  \nThank you for raising this point. While Table 2 does not explicitly break down the complexity of the update step, we clarify it now. Indeed, the 'parameter-free' descent subroutines used in Algorithm 1 are lighter then general mirror descent algorithms. In our setting, the updates are simple, closed-form computations that resemble standard gradient descent steps - see, e.g., [30, 31]. In particular, each wavelet coefficient $c_{j,k,t} \\in \\mathbb{R}$ is updated with a closed-form function of the scalar gradient $g_{j,k,t}$. This results in a computational cost of $O(1)$ per coefficient (simple algebraic computations), making the overall update efficient even though Assumption 1 is stated in a general form. We will clarify this point explicitly in the final version.\n\n\n- **On Assumption 2:**\nAs with Assumption 1, we chose to state Assumption 2 in a general form to allow flexibility in the choice of expert subroutines. Any second-order regret algorithm satisfying this assumption - such as those in [16, 40] - can be plugged in. This abstraction allows us to present and analyze our algorithms as meta-algorithms, emphasizing their generality and modular design. That said, we agree that more context could help, and so we will consider clarifying this point in the final version.\n\n- **About Algorithm 2:**\nThe basic idea behind Algorithm 2 is to best track the *local profile* of the comparator function. This is achieved by running multiple predictors $\\hat f_{j_0,n}$ in parallel, each associated with a different starting scale $j_0$ and a different spatial localization $\\mathcal X_n\\subset \\mathcal X$ - leveraging the localized support of wavelets. The predictors are aggregated using a standard second-order expert aggregation procedure (satisfying Assumption 2). More precisely, each predictor is also associated to scaling coefficients and these latter are learnt on a discretized grid $\\mathcal A$ and are directly optimized through the expert aggregation process (while the detail coefficients are still learnt with Algorithm 1).\nWe agree that the current description may be too dense, and we will revise the final version to better explain this mechanism."}]

Author rebuttal:

Meta-review: The paper proposes an adaptive wavelet based  algorithm for online  non-parametric regression in the adversarial setting that adapts to the parameters including smoothness parameters of the Besov space. The algorithm is the first constructive algorithm to obtain the minimax optimal regret bound for this setting. The reviewers all seem to be in agreement that the paper is worth an accept. I agree with them.
Decision: Accept (spotlight)