SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In a nutshell, our strategy is to apply an unconstrained static OLO algorithm on the transform domain, and in a coordinate-wise fashion. This is remarkably simple, but also contains a few twists. To make it concrete, let us start with a single feature vector.
Size 1 dictionary Consider an index  $n \in [1:N]$ , which is associated to the feature  $h_{1:T,n} := [h_{1,n},\ldots,h_{T,n}] \in \mathbb{R}^{dT}$ . We suppress the index  $n$  and write it as  $h_{1:T} = [h_1,\dots,h_T]$ . For any comparator  $u_{1:T} \in \operatorname{span}(h_{1:T})$ , there exists  $\hat{u} \in \mathbb{R}$  such that  $u_{1:T} = h_{1:T}\hat{u}$ . The cumulative loss of  $u_{1:T}$  can be rewritten as
$$
\langle g _ {1: T}, u _ {1: T} \rangle = \langle g _ {1: T}, h _ {1: T} \rangle \hat {u} = \sum_ {t = 1} ^ {T} \langle g _ {t}, h _ {t} \rangle \hat {u},
$$
which is the loss of the coefficient  $\hat{u}$  in a 1D OLO problem with surrogate loss gradients  $\langle g_t, h_t \rangle$ . Essentially, to compete with a 1D comparator subspace  $\operatorname{span}(h_{1:T})$ , it suffices to run a 1D static regret algorithm  $\mathcal{A}$  that competes with  $\hat{u} \in \mathbb{R}$ . Such a procedure is presented as Algorithm 1.
It still remains to choose the static algorithm  $\mathcal{A}$ . Technically, all known static comparator adaptive algorithms can be applied. As an illustrative example, we adopt the FREEGRAD algorithm [MK20], which simultaneously achieves static comparator adaptivity and second order gradient adaptivity [DHS11].<sup>13</sup> Its pseudocode and static regret bound are presented in Appendix C.1 for completeness.
In summary, our single feature learner (Algorithm 1) has the following simplified guarantee, with the full gradient adaptive version deferred to Appendix C.
<sup>11</sup>Despite also using features, the considered setting slightly differs from the standard notion of regression, as the loss function here does not necessarily have a minimizer. We use the term OLR for cleaner exposition.
<sup>12</sup>The assumptions on the features are mild: an important special case is  $\max_t \| h_{t,n} \|_2 = 1$ , as in the Haar wavelet dictionary. We impose these assumptions to apply unconstrained static algorithms verbatim.
<sup>13</sup>A gradient adaptive regret bound refines our definition Eq.(1) by depending on the actually encountered environment  $\mathcal{E}$  as well. FREEGRAD enjoys another favorable property called scale-freeness: the predictions are invariant to any positive scaling of the loss gradients and the Lipschitz constant  $G$ .
Algorithm 1 Sparse coding with size 1 dictionary.
Require: An algorithm  $\mathcal{A}$  for static 1D unconstrained OLO with  $G$ -Lipschitz losses; and a nonzero feature vector  $h_{1:T} \subset \mathbb{R}^{dT}$ .
1: for  $t = 1,2,\ldots ,\mathbf{d}\mathbf{o}$
2: Receive  $h_t \in \mathbb{R}^d$
3: If  $h_t$  is nonzero, query  $\mathcal{A}$  for its output, and assign it to  $\hat{x}_t \in \mathbb{R}$ ; otherwise,  $\hat{x}_t$  is arbitrary.
4: Predict  $x_{t} = \hat{x}_{t}h_{t}\in \mathbb{R}^{d}$ , and receive the loss gradient  $g_{t}\in \mathbb{R}^{d}$ .
5: If  $h_t$  is nonzero, compute  $\hat{g}_t = \langle g_t, h_t \rangle$  and send it to  $\mathcal{A}$  as its surrogate loss gradient.
6: end for
Lemma 2.1. Let  $\varepsilon >0$  be an arbitrary hyperparameter for FREEGRAD (Algorithm 3 in Appendix C.1). Applying its 1D version as the static subroutine, for all  $T\in \mathbb{N}_{+}$  and  $u_{1:T}\in \mathrm{span}(h_{1:T})$ , Algorithm 1 guarantees
$$
\operatorname {R e g r e t} _ {T} \left(u _ {1: T}\right) \leq \varepsilon G + \| u _ {1: T} \| _ {2} G \cdot \operatorname {p o l y l o g} \left(\max  _ {t} \| u _ {t} \| _ {2}, T, \varepsilon^ {- 1}\right).
$$
Note that the hyperparameter  $\varepsilon$  can be arbitrarily small. Further neglecting poly-logarithmic factors, the bound is essentially  $\tilde{O}\left(G\left\| u_{1:T}\right\| _2\right)$ .
General dictionary Given the above single feature learner, let us turn to the general setting with  $N$  features. We run  $N$  copies of Algorithm 1 in parallel, aggregate their predictions, and the regret bound sums Lemma 2.1, similar to [Cut19] in the static setting. The pseudocode is presented as Algorithm 2, and the regret bound is Theorem 1.
Algorithm 2 Sparse coding with general dictionary.
Require: A dictionary  $\mathcal{H} = [h_{t,n}]$ , where  $h_{t,n} \in \mathbb{R}^d$ ; and a hyperparameter  $\varepsilon > 0$ .
1: For all  $n \in [1:N]$ , initialize a copy of Algorithm 1 as  $\mathcal{A}_n$ . It runs the 1D version of Algorithm 3 as a subroutine, with hyperparameter  $\varepsilon / N$ .
2: for  $t = 1,2,\ldots ,\mathbf{d}\mathbf{o}$
3:Receive  $\mathcal{H}_t = [h_{t,n}]_{1\leq n\leq N}$  . For all  $n$  , send  $h_{t,n}$  to  $\mathcal{A}_n$  , and query its prediction  $w_{t,n}$
4: Predict  $x_{t} = \sum_{n=1}^{N} w_{t,n}$ .
5: Receive loss gradient  $g_{t}$ , and send it to  $\mathcal{A}_1, \ldots, \mathcal{A}_N$  as loss gradients.
6: end for
Theorem 1. Consider any collection of signals  $z^{(n)} \in \operatorname{span}(h_{1:T,n}), \forall n$ . We define its reconstruction error (for the comparator  $u_{1:T}$ ) as  $z^{(0)} = u_{1:T} - \sum_{n=1}^{N} z^{(n)} \in \mathbb{R}^{dT}$ . Then, for all  $T \in \mathbb{N}_+$  and  $u_{1:T} \in \mathbb{R}^{dT}$ , Algorithm 2 guarantees
$$
\operatorname {R e g r e t} _ {T} (u _ {1: T}) \leq \varepsilon G + G \left(\sum_ {n = 1} ^ {N} \left\| z ^ {(n)} \right\| _ {2}\right) \cdot \operatorname {p o l y l o g} \left(\max  _ {t, n} \left\| z _ {t} ^ {(n)} \right\| _ {2}, T, N, \varepsilon^ {- 1}\right) + G \sum_ {t = 1} ^ {T} \left\| z _ {t} ^ {(0)} \right\| _ {2},
$$
where  $z_{t}^{(n)}\in \mathbb{R}^{d}$  is the t-th round component of the sequence  $z^{(n)}\in \mathbb{R}^{dT}$ .
To interpret this very general result, let us consider a few concrete settings.
- Static regret. If the size  $N = d$  and the dictionary  $\mathcal{H}_t = I_d$ , then for any static comparator  $(u_t = u \in \mathbb{R}^d)$ , we can let  $z^{(n)}$  be the projection of the sequence  $u_{1:T}$  onto  $\operatorname{span}(h_{1:T,n})$ . This leaves zero reconstruction error, i.e.,  $u_{1:T} = \sum_{n=1}^{N} z^{(n)}$ . Theorem 1 reduces to
$$
\operatorname {R e g r e t} _ {T} \left(u _ {1: T}\right) \leq \varepsilon G + \| u \| _ {1} G \sqrt {T} \cdot \operatorname {p o l y l o g} \left(\| u \| _ {\infty}, T, d, \varepsilon^ {- 1}\right), \tag {6}
$$
which recovers a standard  $\tilde{O} (\| u\| _1\sqrt{T})$  bound in coordinate-wise unconstrained static OLO [Ora19, Section 9.3]. The gradient adaptive version yields a better  $\tilde{O} (\| u\| _2\sqrt{T})$  bound, c.f., Appendix C.2.
- Orthogonal dictionary. Entering the dynamic realm, we now consider the situation where feature vectors are orthogonal (standard in signal processing), and the comparator  $u_{1:T} \in \operatorname{span}(\mathcal{H})$ . Same as the static setting, we are free to define  $z^{(n)}$  as the projection
$$
z ^ {(n)} = \langle h _ {1: T, n}, u _ {1: T} \rangle \frac {h _ {1 : T , n}}{\| h _ {1 : T , n} \| _ {2} ^ {2}}.
$$
Due to orthogonality, the projection preserves the energy of the time domain signal, i.e.,  $E = \| u_{1:T} \|_2^2 = \sum_{n=1}^{N} \left\| z^{(n)} \right\|_2^2$ . By further defining Sparsity $_{\mathcal{H}} := (\sum_{n=1}^{N} \left\| z^{(n)} \right\|_2)^2 / \sum_{n=1}^{N} \left\| z^{(n)} \right\|_2^2$  (arbitrary when the denominator is zero), Theorem 1 reduces to
$$
\operatorname {R e g r e t} _ {T} \left(u _ {1: T}\right) \leq \tilde {O} \left(\sqrt {E \cdot \operatorname {S p a r s i t y} _ {\mathcal {H}}}\right). \tag {7}
$$
Note that as the squared  $L_{1} / L_{2}$  ratio, Sparsity $_{\mathcal{H}}$  is a classical sparsity measure [HR09] of the decomposed signals  $\{z^{(n)}\}_{1 \leq n \leq N}$ : if there are only  $N_{0} \leq N$  nonzero vectors within this collection, then Sparsity $_{\mathcal{H}} \leq N_{0}$  due to the Cauchy-Schwarz inequality. Therefore, the generic sparsity adaptive bound Eq.(7) depends on (i) the energy of the comparator  $u_{1:T}$ ; and (ii) the sparsity of its representation, without knowing either condition beforehand. The easier the comparator is (low energy, and sparse on  $\mathcal{H}$ ), the lower the bound becomes.
- Overparameterization. So far we have only considered  $N \leq dT$ , where feature vectors can be orthogonal. However, a key idea in signal processing is to use redundant features ( $N \gg dT$ ) to obtain sparser representations. Theorem 1 implies a feature selection property in this context: since it applies to any decomposition of  $u_{1:T}$ , as long as  $u_{1:T}$  can be represented by a subset  $\tilde{\mathcal{H}}$  of orthogonal features within  $\mathcal{H}$ , the regret bound adapts to Sparsity $_{\tilde{\mathcal{H}}}$ , the sparsity of  $u_{1:T}$  measured on  $\tilde{\mathcal{H}}$ . That is, we are theoretically justified to assemble smaller dictionaries into a larger one – the regret bound adapts to the quality of the optimal (comparator-dependent) sub-dictionary  $\tilde{\mathcal{H}}$ .
How to choose the dictionary  $\mathcal{H}$ ? In practice, we may use prior knowledge on the dynamics of the environment. For example, if the environment is periodic, such as the weather or the traffic, then a good choice could be the Fourier dictionary. Similarly, wavelet dictionaries are useful for piecewise regular environments. Another possibility is to learn the dictionary from offline datasets, which is also called representation learning. Overall, such prior knowledge is not required to be correct – our algorithm can take any dictionary as input, and the regret bound naturally adapts to its quality. The established connection between adaptivity and signal structures is a key benefit of our framework.
Figure 1: The power law.
Power law For a more specific discussion, let us consider an empirically justified setup. In signal processing, the study of sparsity has been partially motivated by the power law [Pri21]: under the standard Fourier or wavelet transforms, the  $n$ -th largest transform domain coefficient of many real signals can have magnitude roughly proportional to  $n^{-\alpha}$ , where  $\alpha \in (0.5, 1)$ . We also observe this phenomenon from a weather dataset, with details presented in Appendix E.1. Figure 1 plots the sorted Fourier coefficients of an actual temperature sequence, on a log-log scale. A fitted dashed line is shown in orange, with (negative) slope  $\alpha = 0.68$ .
When the power law holds, our bound Eq.(7) has a more interpretable form. Assuming  $d = 1$  and  $N = T$ ,
$$
\mathrm {S p a r s i t y} _ {\mathcal {H}} = \frac {\left(\sum_ {n = 1} ^ {T} n ^ {- \alpha}\right) ^ {2}}{\sum_ {n = 1} ^ {T} n ^ {- 2 \alpha}} = O \left(T ^ {2 - 2 \alpha}\right).
$$
In a typical setting of  $E = \Theta (T)$ , we obtain a sublinear  $\tilde{O} (T^{1.5 - \alpha})$  dynamic regret bound.
3 The Haar OLR algorithm
This section presents the quantitative contributions of this paper: despite its generality, our sparse coding framework can improve existing results [JC22]. Our workhorse is the ability of wavelet bases to sparsely represent smooth signals.
3.1 Haar wavelet
Wavelet is a fundamental topic in signal processing, with long lasting impact throughout modern data science. Roughly, the motivation is that a signal can simultaneously exhibit nonstationarity at different time scales, such as slow drifts and fast jumps, therefore to faithfully represent it, we should apply feature vectors with different resolutions. We will only use the simplest Haar wavelets, which is already sufficient. Readers are referred to [Mal08, Joh19] for a thorough introduction to this topic.
Specifically, we start from the 1D setting  $(d = 1)$  with a dyadic horizon  $(T = 2^{m}$ , for some  $m\in \mathbb{N}_{+})$ . The Haar wavelet dictionary consists of  $T$  (unnormized) orthogonal feature vectors, indexed by a scale parameter  $j\in [1:\log_2T]$  and a location parameter  $l\in [1:2^{-j}T]$ . Given a  $(j,l)$  pair, define a feature  $h^{(j,l)} = [h_1^{(j,l)},\dots,h_T^{(j,l)}]\in \mathbb{R}^T$  entry-wise as
$$
h _ {t} ^ {(j, l)} = \left\{ \begin{array}{l l} 1, & t \in [ 2 ^ {j} (l - 1) + 1: 2 ^ {j} (l - 1) + 2 ^ {j - 1} ]; \\ - 1, & t \in [ 2 ^ {j} (l - 1) + 2 ^ {j - 1} + 1: 2 ^ {j} l ]; \\ 0, & \text {e l s e .} \end{array} \right.
$$
It means that  $h^{(j,l)}$  is only nonzero on a length-  $2^j$  interval, while changing its sign once in the middle of this interval. Collecting all the  $(j,l)$  pairs yield  $T - 1$  features; then, we incorporate an extra all-one feature  $h^* = [1,\dots ,1]$  to complete this size  $T$  dictionary.
The defined features can be assembled into the columns of a matrix  $\mathrm{Haar}_m$ . To help with the intuition,  $\mathrm{Haar}_2$  with  $T = 4$  is presented in Eq.(8). The columns from the left to the right are  $h^*$ ,  $h^{(2,1)}$ ,  $h^{(1,1)}$  and  $h^{(1,2)}$ . Observe that they are orthogonal, and the norm assumption from Section 2.1 is satisfied. Therefore, our sparsity adaptive regret bound Eq.(7) is applicable.
$$
\mathrm {H a a r} _ {2} = \left[ \begin{array}{c c c c} 1 & 1 & 1 & 0 \\ 1 & 1 & - 1 & 0 \\ 1 & - 1 & 0 & 1 \\ 1 & - 1 & 0 & - 1 \end{array} \right]. \tag {8}
$$
Given this 1D Haar wavelet dictionary, we apply a minor variant of Algorithm 2 to prevent the dimension  $d$  from appearing in the regret bound. When  $d = 1$ , the algorithm is exactly Algorithm 2, where intuitions are most clearly demonstrated. Then, the doubling trick [SS11, Section 2.3.1] is adopted to relax the knowledge of  $T$ . The pseudocode is presented as Algorithm 5 in Appendix D.
**Computation** An appealing property is that most Haar wavelet features are supported on short local intervals. Despite  $N = T$ , there are only  $\log_2 T$  active features in each round. Therefore, the runtime of our algorithm is  $O(d\log T)$  per round, matching that of all the baselines we compare to. This local property holds for compactly supported wavelets, most notably the Daubechies family [Dau88, CDV93]. The latter can represent more general, piecewise polynomial signals.
3.2 Main result
For almost static environments, our Haar OLR algorithm guarantees the following bounds, by relating comparator smoothness to the sparsity of its Haar wavelet representation. Different from [JC22] which only contains  $P$ -dependent bounds, we also provide a  $K$ -switching regret bound, in order to avoid using  $P = O(KM)$ .<sup>14</sup> Interestingly, the proofs of the following two bounds are quite different: the first uses exact sparsity, while the second uses approximate sparsity.
Theorem 2 (Switching regret). For all  $T \in \mathbb{N}_{+}$  and  $u_{1:T} \in \mathbb{R}^{dT}$ , Algorithm 5 guarantees
$$
\operatorname {R e g r e t} _ {T} \left(u _ {1: T}\right) \leq \tilde {O} \left(\| \bar {u} \| _ {2} \sqrt {T} + \sqrt {K \bar {E}}\right). \tag {9}
$$
Theorem 3 (Path length bound). For all  $T \in \mathbb{N}_+$  and  $u_{1:T} \in \mathbb{R}^{dT}$ , Algorithm 5 guarantees
$$
\operatorname {R e g r e t} _ {T} \left(u _ {1: T}\right) \leq \tilde {O} \left(\| \bar {u} \| _ {2} \sqrt {T} + \sqrt {P \bar {S}}\right). \tag {10}
$$
It can be verified (Appendix A) that for all comparators  $u_{1:T}$ , our bounds are at least as good as prior works (Table 1). The optimality is a more subtle issue, as one should compare upper bound functions (of  $u_{1:T}$ ) to lower bound functions in a global manner, rather than comparing the exponents of  $T$  in minimax online learning.
Nonetheless, we present two examples of  $u_{1:T}$ , where the improvement can be clearly seen through better exponents of  $T$ . To give it a concrete background, suppose we want to sequentially predict a 1D time series  $z_1, \ldots, z_T \in \mathbb{R}$ . This could be formulated as a OCO problem where the decision  $x_t$  there is our prediction of  $z_t$ , and the loss function is the absolute loss  $l_t(x) = |x - z_t|$ . A natural
<sup>14</sup>Recall that one of our motivations is to remove  $M$  from the existing bounds.
choice of the comparator is the ground truth sequence  $z_{1:T}$ , and due to Eq.(2), any upper bound on  $\mathrm{Regret}_T(z_{1:T})$  also upper-bounds the total forecasting loss of our algorithm. Below we present specific 1D comparator sequences  $u_{1:T}$  to demonstrate the strength of our results, which could be intuitively thought as the true time series  $z_{1:T}$  in this more restricted discussion.
Example 1 (Tracking outliers). Consider the situation where  $u_{1:T}$  has a locally outlying scale: we set all the instantaneous comparators  $u_t$  to 1, except  $k \leq \sqrt{T}$  consecutive members which are set to  $\sqrt{T}$ . Crucially,  $|\bar{u}| = O(1)$  and  $\bar{S} = O(k\sqrt{T})$ , while  $M = \sqrt{T}$  and  $S = \Theta(T)$ . With details deferred to Appendix D.7, both our bounds, i.e., Eq.(9) and (10), are  $\tilde{O}(\sqrt{kT})$ , while the fine baseline Eq.(4) is  $\tilde{O}(T^{3/4})$ , and the coarse baseline Eq.(3) is  $\tilde{O}(T)$ . The largest gain is observed when  $k$  is a constant, i.e., the comparator is subject to a short but large perturbation.
Example 2 (Persistent oscillation). Consider the situation where  $\bar{u} = 1$ , and all the instantaneous comparators oscillate around  $\bar{u}$ :  $u_{t} = \bar{u} + \alpha_{t} / \sqrt{T}$ .  $\alpha_{t} = 1$  or  $-1$ , and it only switches sign for  $k$  times. Notice that  $\bar{S} = \sqrt{T}$ , while  $S = \Theta(T)$ . All the baselines are  $\tilde{O}\left(\sqrt{T} + k^{1/2}T^{1/4}\right)$ , while both our bounds are  $\tilde{O}(\sqrt{T})$ . The largest gain is observed when  $k = T - 1$ , i.e., the comparator switches in every round.
In summary, we show that existing bounds are suboptimal, while the optimality of our results remains to be studied. It highlights the importance of comparator energy and variability in the pursuit of better algorithms, which have not received enough attention in the literature. Next, we briefly sketch the proofs of these bounds.
Proof sketch The switching regret bound mostly follows from a very simple observation: if a sequence is constant throughout the support of a Haar wavelet feature, then its transform domain coefficient for this feature is zero. As features on the same scale  $j$  do not overlap, a  $K$ -switching comparator can only induce  $K$  nonzero coefficients on the  $j$ -th scale. There are at most  $K\log_2T$  nonzero coefficients in total, therefore  $\mathrm{Sparsity}_{\mathcal{H}} = \tilde{O}(K)$ . The bound Eq.(9) is obtained by applying this argument after taking out the average of  $u_{1:T}$ .
As for the path length bound, the idea is to consider the reconstructed sequences, using transform domain coefficients on a single scale  $j$ . These are usually called detail sequences in the wavelet literature [Mal08]. Each detail sequence has a relatively simple structure, whose path length and variability can be associated to the magnitude of its transform domain coefficients. Moreover, as these detail sequences are certain "locally averaged" and "globally centered" versions of the actual comparator  $u_{1:T}$ , their regularities are dominated by the regularity of  $u_{1:T}$  itself. In combination, this yields a relation between  $P\bar{S}$  and the coefficients'  $L_1$  norm, i.e.,  $\sum_{n=1}^{N}\left\|z^{(n)}\right\|_2$  in Theorem 1, from which the bound is established.
Compared to the analysis of [JC22], the key advantage of our analysis is the decoupling of function approximation from the generic sparsity-based regret bound. The former is algorithm-independent, while the latter can be conveniently combined with advances in static online learning. With the help of approximation theory (e.g., Fourier features, wavelets, and possibly deep learning further down the line), intuitions are arguably clearer in this way, and solutions could be more precise (compared to analyses that "mix" function approximation with regret minimization).
Additional discussion Finally, due to limited space, we defer additional discussion of our technical results to Appendix F, including
- The related use of Multi-Resolution Analysis (MRA) in the existing online learning literature.
- The comparison between Lipschitz and strongly convex losses in unconstrained dynamic OCO.
This paper presents a unified study of unconstrained and dynamic online learning, where the two problem structures are naturally connected via comparator adaptivity. Building on the synergy between static parameter-free algorithms and temporal representations, we develop an algorithmic framework achieving a generic sparsity-adaptive regret bound. Equipped with the wavelet dictionary, our framework improves the quantitative results from [JC22], by adapting to finer characterizations of the comparator sequence.
We thank Vivek Goyal for helpful pointers to the signal processing literature, and the NeurIPS reviewers for their constructive feedback. This research was partially supported by the NSF under grants CCF-2200052, DMS-1664644, and IIS-1914792, by the ONR under grant N00014-19-1-2571, by the DOE under grant DE-AC02-05CH11231, by the NIH under grant UL54 TR004130, and by Boston University.