Title: Unconstrained Dynamic Regret via Sparse Coding
Venue: NeurIPS 2023
Accepted: yes
Keywords: Dynamic online learning

parameter-free online learning

time series forecasting

wavelet
Abstract: Motivated by the challenge of nonstationarity in sequential decision making, we study Online Convex Optimization (OCO) under the coupling of two problem structures: the domain is unbounded, and the comparator sequence $u_1,\ldots,u_T$ is arbitrarily time-varying. As no algorithm can guarantee low regret simultaneously against all comparator sequences, handling this setting requires moving from minimax optimality to comparator adaptivity. That is, sensible regret bounds should depend on certain complexity measures of the comparator relative to one's prior knowledge. This paper achieves a new type of such adaptive regret bounds leveraging a sparse coding framework. The complexity of the comparator is measured by its energy and its sparsity on a user-specified dictionary, which offers considerable versatility. For example, equipped with a wavelet dictionary, our framework improves the state-of-the-art bound (Jacobsen & Cutkosky, 2022) by adapting to both ($i$) the magnitude of the comparator average $||\bar u||=||\sum_{t=1}^Tu_t/T||$, rather than the maximum $\max_t||u_t||$; and ($ii$) the comparator variability $\sum_{t=1}^T||u_t-\bar u||$, rather than the uncentered sum $\sum_{t=1}^T||u_t||$. Furthermore, our proof is simpler due to decoupling function approximation from regret minimization.

Unconstrained Dynamic Regret via Sparse Coding

Abstract

Motivated by the challenge of nonstationarity in sequential decision making, we study Online Convex Optimization (OCO) under the coupling of two problem structures: the domain is unbounded, and the comparator sequence u 1 , . . . , u T is arbitrarily time-varying. As no algorithm can guarantee low regret simultaneously against all comparator sequences, handling this setting requires moving from minimax optimality to comparator adaptivity. That is, sensible regret bounds should depend on certain complexity measures of the comparator relative to one's prior knowledge. This paper achieves a new type of such adaptive regret bounds leveraging a sparse coding framework. The complexity of the comparator is measured by its energy and its sparsity on a user-specified dictionary, which offers considerable versatility. For example, equipped with a wavelet dictionary, our framework improves the state-of-the-art bound [JC22] by adapting to both ( i ) the magnitude of the comparator average ∥ ¯ u ∥ = ∥ ∑ T t =1 u t /T ∥ , rather than the maximum max t ∥ u t ∥ ; and ( ii ) the comparator variability ∑ T t =1 ∥ u t -¯ u ∥ , rather than the uncentered sum ∑ T t =1 ∥ u t ∥ . Furthermore, our analysis is simpler due to decoupling function approximation from regret minimization.

1 Introduction

Nonstationarity is prevalent in sequential decision making, which poses a critical challenge to the vast majority of existing approaches developed offline. Consider weather forecasting for example [SBG + 21]. A meteorologist typically starts from the governing physical equations and simulates them online using high performance computing; the imperfection of this physical model can lead to time-varying patterns in its forecasting error. Alternatively, a machine learning scientist may build a data-driven model from historical weather datasets, but its online deployment is subject to distribution shifts. If the structure of such nonstationarity can be exploited in our algorithm, then we may expect better forecasting performance. This paper investigates the problem from a theoretical angle - we aim to improve nonstationary online decision making by incorporating temporal representations .

Concretely, we study Online Convex Optimization (OCO), which is a repeated game between us (the player) and an adversarial environment E . In each (the t -th) round, with a mutually known Lipschitz constant G :

We make a prediction x t ∈ R d based on the observations before the t -th round.

The environment E reveals a convex loss function l t : R d → R dependent on our prediction history x 1 , . . . , x t ; l t is G -Lipschitz with respect to ∥·∥ 2 .

We suffer the loss l t ( x t ) .

The game ends after T rounds, and then, our total loss is compared to that of an alternative sequence of predictions u 1 , . . . , u T ∈ R d . Without knowing the time horizon T , the environment E and the

comparator sequence { u t } t ∈ Z , our goal is to achieve low unconstrained dynamic regret

Fixing any comparator { u t } t ∈ Z : if such an expression can be upper-bounded by a sublinear function of T , then asymptotically, in any environment, we perform at least as well as the { u t } t ∈ Z sequence.

The above setting deviates from the most standard setting of OCO [Haz16, Ora19] in two ways.

Structure 1. The domain R d is unbounded.

Structure 2. The comparator is allowed to be time-varying.

While the latter has been studied extensively in the literature (since [Zin03]) to account for nonstationarity, most existing approaches require a time-invariant bounded domain to set the hyperparameters properly, which, to some extent, limits the amount of nonstationarity they can handle. One might argue that most practical problems have a finite range, which could be heuristically estimated from offline datasets. However, such a heuristic is not robust in nature, as underestimates will invalidate the theoretical analysis, and overestimates will make the regret bound excessively conservative. It is thus important to study the more challenging unconstrained dynamic setting 2 combining the two problem structures, where algorithms cannot rely on pre-selected range estimates at all.

Taking a closer look at their analysis, it is perhaps a little surprising that these two problem structures share a common theme, despite being studied mostly separately. In either the unconstrained static setting [MO14, OP16, CO18] or the bounded dynamic setting [Zin03, HW15, ZLZ18], the standard form of minimax optimality [ABRT08, AABR09, RS14b] becomes vacuous, as it is impossible to guarantee that sup u 1: T Regret T ( u 1: T ) is sublinear in T . Circumventing this issue relies on comparator adaptivity 3 - instead of only depending on T , any appropriate regret upper bound, denoted by Bound T ( u 1: T ) , should also depend on the comparator u 1: T through a certain complexity measure . Intuitively, despite the intractability of hard comparators, nonvacuous bounds can be established against 'easy ones'. A total loss bound then follows from the oracle inequality

A crucial observation is that the complexity of u 1: T is not uniquely defined: one could imagine bounding Regret T ( u 1: T ) by many different non-comparable functions of u 1: T . Essentially, this complexity measure serves as a Bayesian prior : 4 choosing it amounts to assigning different priorities to different comparators u 1: T ∈ R d × T . The associated algorithm guarantees lower Bound T ( u 1: T ) against comparators with higher priority, and due to Eq.(2), the total loss of our algorithm is low if some of these high priority comparators actually achieve low loss ∑ T t =1 l t ( u t ) . Such a Bayesian reasoning highlights the importance of versatility in this workflow: in order to place an arbitrary application-dependent prior, we need a versatile algorithmic framework that adapts to a wide range of complexity measures. This leads to the limitations of existing results, discussed next.

To our knowledge, [JC22] is the only existing work that considers our setting. Two unconstrained dynamic regret bounds are presented based on three statistics of the comparator sequence, the maximum range M := max t ∥ u t ∥ 2 , the norm sum S := ∑ T t =1 ∥ u t ∥ 2 and the path length P := ∑ T -1 t =1 ∥ u t +1 -u t ∥ 2 . First, using a 1D unconstrained static algorithm as a simple range scaler, the paper achieves [JC22, Lemma 10]

Then, by developing a customized mirror descent approach, most of the effort is devoted to improving MT to S [JC22, Theorem 4], i.e., adapting to the magnitude of individual u t .

Despite the strengths of these results and their nontrivial analysis, a shared limitation is that both bounds depend explicitly on the path length P . Intuitively, it means that good performance is only guaranteed in almost static environments: in the typical situation of S = Θ( T ) , these bounds are only sublinear when P = o ( T ) , which rules out important persistent dynamics such as periodicity. Moreover, even the second bound still depends on MS instead of a finer characterization of each individual u t 's magnitude. That is, the mission of removing M is not fully accomplished yet. 5

The goal of this paper is to extend comparator adaptivity to a wider range of complexity measures. For almost static environments in particular, quantitative benefits will be obtained from specific instances of this general approach.

1.1 Contribution

The contributions of this paper are twofold.

First, we present an algorithmic framework achieving a new type of unconstrained dynamic regret bounds. It is based on a conversion to vector-output Online Linear Regression (OLR): given a dictionary H of orthogonal feature vectors spanning the sequence space R dT , we use an unconstrained static OCO algorithm to linearly aggregate these feature vectors, which are themselves time-varying prediction sequences. Such a procedure guarantees

where E = ∑ T t =1 ∥ u t ∥ 2 2 is the energy of the comparator u 1: T , and Sparsity H measures the sparsity of u 1: T on the dictionary H . 6 Both E and Sparsity H are unknown beforehand.

Compared to [JC22], the main advantage of this framework is its versatility. Prior knowledge on the transform domain can be incorporated by picking H , and favorable algorithmic properties can be conveniently inherited from static online learning.

Our second contribution is quantitative: although [JC22] is specifically crafted to handle almost static environments, we show that equipped with a Haar wavelet dictionary, our framework actually guarantees better bounds (Table 1) in this setting, which is a surprising finding to us.

With the comparator average ¯ u := ∑ T t =1 u t /T and the first order variability ¯ S := ∑ T t =1 ∥ u t -¯ u ∥ 2 , our Haar wavelet algorithm guarantees

(

)

regret bound, the hallmark of gradient adaptivity is the so-called 'second-order bound'

O

D

√ ∑

T

t

=1

∥

g

t

∥

2

2

,

It improves Eq.(4) by ( i ) a better dependence on the comparator magnitude ( √ MS → ∥ ¯ u ∥ 2 √ T ); and ( ii ) decoupling the bias ¯ u from the characterization of variability ( √ PS → √ P ¯ S ).

̸

With the number of switches K := ∑ T -1 t =1 1 [ u t +1 = u t ] and the second order variability ¯ E := ∑ T t =1 ∥ u t -¯ u ∥ 2 2 , the same Haar wavelet algorithm guarantees an unconstrained switching regret bound

which improves the existing ˜ O ( √ (1 + K ) MS ) bound resulting from Eq.(4) and P = O ( KM ) .

Due to the local property of wavelets, our algorithm runs in O ( d log T ) time per round, matching that of the baselines. As for the regret, our bounds are never worse than the baselines, and in two examples corresponding to ∥ ¯ u ∥ 2 ≪ M and ¯ S ≪ S , they reduce to clearly improved rates in T . Furthermore, our analysis follows from the generic regret bound Eq.(5) and the wavelet approximation theory , providing an intriguing connection between disparate fields.

The paper concludes with an application in fine-tuning time series forecasters, where unconstrained dynamic OCO is naturally motivated. Due to limited space, this is deferred to Appendix E, with experiments that support our theoretical results.

1.2 Related work

Our paper addresses the connection between unconstrained OCO and dynamic OCO. Although they both embody the idea of comparator adaptivity, unified studies have been scarce.

Unconstrained OCO To obtain static regret bounds in OCO, Online Gradient Descent (OGD) [Zin03] is often the default approach. With learning rate η , it guarantees O ( η -1 ∥ u ∥ 2 2 + ηT ) regret with respect to any unconstrained static comparator u ∈ R d , and the optimal choice in hindsight is η = O ( ∥ u ∥ 2 / √ T ) . Without the oracle knowledge of ∥ u ∥ 2 , it is impossible to tune η optimally. To address this issue, a series of works (also called parameter-free online learning ) [SM12, MA13, GY14, MO14, OP16, FKMS17, CO18, FRS18, MK20, CLW21, ZCP22] developed vastly different strategies to achieve the oracle optimal rate O ( ∥ u ∥ √ T ) up to logarithmic factors. That is, the algorithm performs as if the complexity measure ∥ u ∥ is known beforehand.

There is certain flexibility in the choice of the norm ∥·∥ : L 1 and L 2 norm bounds were presented in [SM12], while Banach norm bounds were developed by [FRS18, CO18]. Historically, the connection between the L 1 norm and sparsity has powered breakthroughs in batch data science, including LASSO [Tib96] and compressed sensing [CRT06]. However, the parallel path in online learning remains less studied: while the sparsity implication of the L 1 norm adaptive bounds has been discussed in the literature [SM12, Ger13, vdH19], there is in general a lack of downstream investigations with concrete benefits. In this paper, we show that the sparsity of the comparator can be naturally associated to the structural simplicity of a nonstationary environment.

Dynamic OCO Comparing against dynamic sequences is a classical research topic. It is clear that one cannot go beyond linear regret in the worst case, therefore various notions of complexity should be introduced.

The closest topic to ours is the universal dynamic regret , where the regret bound adapts to the complexity of an arbitrary u 1: T on a bounded domain with L p -diameter D . In the most common framework, the complexity measure is an L p,q norm of the difference sequence { u t +1 -u t } , such as the L p, 1 norm, i.e., the path length P = ∑ T -1 t =1 ∥ u t +1 -u t ∥ p [HW01]. 7 Omitting the dependence on the dimension d (thus also the choice of p ), the optimal bound under convex Lipschitz losses is ˜ O ( √ ( D + P ) DT ) [Zin03, HW15, JRSS15, GS16, ZLZ18], while the accelerated rate 8 ˜ O ( P 2 / 3 T 1 / 3 ∨ 1) can be achieved with strong convexity [BW21, BW22]. Improvements have been

studied under the additional smoothness assumption [ZZZZ20, ZZZZ21]. These bounds subsume results in switching (a.k.a., shifting) regret , where the complexity of u 1: T is measured by its number of switches K , as P is dominated by DK .

A notable exception is the dynamic model framework from [HW15, ZLZ18]. Still considering a bounded domain, it takes a collection of dynamic models as input, which are mappings from the domain to itself. Then, the complexity of a comparator u 1: T is measured by how well it can be reconstructed by the best dynamic model in hindsight. Essentially, the use of temporal representations is somewhat similar to the dictionary in our framework. The important difference is that instead of using the best feature (or the best convex combination of the features) to measure the comparator, we use linear combinations of the features - this allows handling unconstrained domains through subspace modeling.

Besides the universal dynamic regret, there are other notions of dynamic regret that do not induce oracle inequalities like Eq.(2), including ( i ) the restricted dynamic regret [YZJY16, ZYY + 17, BW19, BW20, BZW21], which depends on the complexity of certain offline optimal comparators; 9 and ( ii ) regret bounds that depend on the functional variation ∑ T -1 t =1 max x | l t ( x ) -l t +1 ( x ) | [BGZ15, CWW19]. They are not as relevant to our purpose, due to being vacuous on unbounded domains under the linear losses - this is an important setting in our investigation.

Unconstrained (universal) dynamic regret To our knowledge, [JC22] is the only work studying the universal dynamic regret without a bounded domain, whose contributions have been summarized in our Introduction. Here we survey some negative results in the literature.

The restricted dynamic regret is a special case of the universal dynamic regret, therefore lower bounds for the former apply to the latter as well. For convex Lipschitz losses [YZJY16] and strongly convex losses [BW19], any algorithm should suffer the dynamic regret of Ω( P ) and Ω( P 2 ) , respectively.

For dynamic OCO on bounded domains, a recurring analysis goes through the notion of strong adaptivity [DGSS15]: one first achieves low static regret bounds on every subinterval of the time horizon [1 : T ] , and then assembles these local bounds appropriately to bound the global dynamic regret [ZYZ + 18, Cut20, BW21, BW22]. Following this route in the unconstrained setting appears to be challenging, as [JC22, Section 4] showed that (a natural form of) strong adaptivity cannot be achieved there.

Additional discussions of related works are deferred to Appendix B, including the more general problem of online nonparametric regression , the more specific problem of parametric time series forecasting , and other orthogonal uses of sparsity in online learning.

1.3 Notation

For two integers a ≤ b , [ a : b ] is the set of all integers c such that a ≤ c ≤ b . The brackets are removed when on the subscript, denoting a tuple with indices in [ a : b ] . Treating all vectors as column vectors, span( A ) represents the column space of a matrix A . log is natural logarithm when the base is omitted, and log + ( · ) := 0 ∨ log( · ) . polylog denotes a poly-logarithmic function of its input. 0 represents a zero vector whose dimension depends on the context.

2 The general sparse coding framework

This section presents our sparse coding framework, achieving the generic sparsity adaptive regret bound Eq.(5). The key idea is to view online learning on the sequence space R dT , rather than the default domain R d . Despite its central role in signal processing (e.g., the Fourier transform ), such a view is (in our opinion) under-explored by the online learning community. 10

2.1 Setting

To begin with, we follow the conventions in online learning [Haz16, Ora19] to linearize convex losses. Consider that instead of the full loss function l t , we only observe its subgradient g t ∈ ∂l t ( x t ) at our prediction x t . By using the linear loss ⟨ g t , ·⟩ as a surrogate, we can still upper bound the regret Eq.(1) due to l t ( x t ) -l t ( u ) ≤ ⟨ g t , x t -u ⟩ . The linear loss problem is also called Online Linear Optimization (OLO), where each observation g t is a d dimensional vector satisfying ∥ g t ∥ 2 ≤ G .

Now, consider the length T sequences of predictions x 1: T , gradients g 1: T and comparators u 1: T . Let us flatten everything and treat them as dT dimensional vectors, concatenating per-round quantities in R d . These are called signals . The comparator statistics could be more succinctly represented using vector notations, e.g., the energy E = ∑ T t =1 ∥ u t ∥ 2 2 = ∥ u 1: T ∥ 2 2 .

Our framework requires a dictionary matrix H ∈ R dT × N , possibly revealed online, whose columns are N nonzero feature vectors . We write H in an equivalent block form as [ h t,n ] 1 ≤ t ≤ T, 1 ≤ n ≤ N , where each block h t,n ∈ R d × 1 . The accompanied linear transform u = H ˆ u relates a signal u ∈ R dT to a coefficient vector ˆ u ∈ R N (if it exists). Adopting the convention in signal processing, we will call R dT the time domain , and R N the transform domain . In general, symbols without hat refer to time domain quantities, while their transform domain counterparts are denoted with hat.

Summarizing the above, we consider the following concise interaction protocol. 11 Despite its parametric appearance, our main focus is on the nonparametric regime, where the dictionary size N scales with the amount of data T .

Vector-output OLR with linear losses In the t -th round, our algorithm observes a d -by-N feature matrix H t := [ h t,n ] 1 ≤ n ≤ N , linearly combines its columns into a prediction x t ∈ R d , receives a loss gradient g t ∈ R d , and then suffers the linear loss ⟨ g t , x t ⟩ . We assume that 12 ∥ h t,n ∥ 2 ≤ 1 , ∑ T t =1 ∥ h t,n ∥ 2 2 ≥ 1 and ∥ g t ∥ 2 ≤ G . The performance metric is the unconstrained dynamic regret defined in Eq.(1).

2.2 Main result

In a nutshell, our strategy is to apply an unconstrained static OLO algorithm on the transform domain, and in a coordinate-wise fashion. This is remarkably simple, but also contains a few twists. To make it concrete, let us start with a single feature vector.

Size 1 dictionary Consider an index n ∈ [1 : N ] , which is associated to the feature h 1: T,n := [ h 1 ,n , . . . , h T,n ] ∈ R dT . We suppress the index n and write it as h 1: T = [ h 1 , . . . , h T ] . For any comparator u 1: T ∈ span( h 1: T ) , there exists ˆ u ∈ R such that u 1: T = h 1: T ˆ u . The cumulative loss of u 1: T can be rewritten as

which is the loss of the coefficient ˆ u in a 1D OLO problem with surrogate loss gradients ⟨ g t , h t ⟩ . Essentially, to compete with a 1D comparator subspace span( h 1: T ) , it suffices to run a 1D static regret algorithm A that competes with ˆ u ∈ R . Such a procedure is presented as Algorithm 1.

It still remains to choose the static algorithm A . Technically, all known static comparator adaptive algorithms can be applied. As an illustrative example, we adopt the FREEGRAD algorithm [MK20], which simultaneously achieves static comparator adaptivity and second order gradient adaptivity [DHS11]. 13 Its pseudocode and static regret bound are presented in Appendix C.1 for completeness.

In summary, our single feature learner (Algorithm 1) has the following simplified guarantee, with the full gradient adaptive version deferred to Appendix C.

Algorithm 1 Sparse coding with size 1 dictionary.

Require: An algorithm A for static 1D unconstrained OLO with G -Lipschitz losses; and a nonzero feature vector h 1: T ⊂ R dT .

1: for t = 1 , 2 , . . . , do

2: Receive h t ∈ R d .

3: If h t is nonzero, query A for its output, and assign it to ˆ x t ∈ R ; otherwise, ˆ x t is arbitrary.

4: Predict x t = ˆ x t h t ∈ R d , and receive the loss gradient g t ∈ R d .

5: If h t is nonzero, compute ˆ g t = ⟨ g t , h t ⟩ and send it to A as its surrogate loss gradient.

6: end for

Lemma 2.1. Let ε > 0 be an arbitrary hyperparameter for FREEGRAD (Algorithm 3 in Appendix C.1). Applying its 1D version as the static subroutine, for all T ∈ N + and u 1: T ∈ span( h 1: T ) , Algorithm 1 guarantees

Note that the hyperparameter ε can be arbitrarily small. Further neglecting poly-logarithmic factors, the bound is essentially ˜ O ( G ∥ u 1: T ∥ 2 ) .

General dictionary Given the above single feature learner, let us turn to the general setting with N features. We run N copies of Algorithm 1 in parallel, aggregate their predictions, and the regret bound sums Lemma 2.1, similar to [Cut19] in the static setting. The pseudocode is presented as Algorithm 2, and the regret bound is Theorem 1.

Algorithm 2 Sparse coding with general dictionary.

Require: A dictionary H = [ h t,n ] , where h t,n ∈ R d ; and a hyperparameter ε > 0 .

1: For all n ∈ [1 : N ] , initialize a copy of Algorithm 1 as A n . It runs the 1D version of Algorithm 3 as a subroutine, with hyperparameter ε/N .

2: for t = 1 , 2 , . . . , do

3: Receive H t = [ h t,n ] 1 ≤ n ≤ N . For all n , send h t,n to A n , and query its prediction w t,n .

4: Predict x t = ∑ N n =1 w t,n .

5: Receive loss gradient g t , and send it to A 1 , . . . , A N as loss gradients.

6: end for

Theorem 1. Consider any collection of signals z ( n ) ∈ span( h 1: T,n ) , ∀ n . We define its reconstruction error (for the comparator u 1: T ) as z (0) = u 1: T -∑ N n =1 z ( n ) ∈ R dT . Then, for all T ∈ N + and u 1: T ∈ R dT , Algorithm 2 guarantees

where z ( n ) t ∈ R d is the t -th round component of the sequence z ( n ) ∈ R dT .

To interpret this very general result, let us consider a few concrete settings.

Static regret. If the size N = d and the dictionary H t = I d , then for any static comparator ( u t = u ∈ R d ), we can let z ( n ) be the projection of the sequence u 1: T onto span( h 1: T,n ) . This leaves zero reconstruction error, i.e., u 1: T = ∑ N n =1 z ( n ) . Theorem 1 reduces to

which recovers a standard ˜ O ( ∥ u ∥ 1 T ) bound in coordinate-wise unconstrained static OLO [Ora19, Section 9.3]. The gradient adaptive version yields a better ˜ O ( ∥ u ∥ 2 √ T ) bound, c.f., Appendix C.2.

Orthogonal dictionary. Entering the dynamic realm, we now consider the situation where feature vectors are orthogonal (standard in signal processing), and the comparator u 1: T ∈ span( H ) . Same as the static setting, we are free to define z ( n ) as the projection

Due to orthogonality, the projection preserves the energy of the time domain signal, i.e, E = ∥ u 1: T ∥ 2 2 = ∑ N n =1 ∥ ∥ z ( n ) ∥ ∥ 2 2 . By further defining Sparsity H := ( ∑ N n =1 ∥ ∥ z ( n ) ∥ ∥ 2 ) 2 / ∑ N n =1 ∥ ∥ z ( n ) ∥ ∥ 2 2 (arbitrary when the denominator is zero), Theorem 1 reduces to

Note that as the squared L 1 / L 2 ratio, Sparsity H is a classical sparsity measure [HR09] of the decomposed signals { z ( n ) } 1 ≤ n ≤ N : if there are only N 0 ≤ N nonzero vectors within this collection, then Sparsity H ≤ N 0 due to the Cauchy-Schwarz inequality. Therefore, the generic sparsity adaptive bound Eq.(7) depends on ( i ) the energy of the comparator u 1: T ; and ( ii ) the sparsity of its representation, without knowing either condition beforehand. The easier the comparator is (low energy, and sparse on H ), the lower the bound becomes.

Overparameterization. So far we have only considered N ≤ dT , where feature vectors can be orthogonal. However, a key idea in signal processing is to use redundant features ( N ≫ dT ) to obtain sparser representations. Theorem 1 implies a feature selection property in this context: since it applies to any decomposition of u 1: T , as long as u 1: T can be represented by a subset ˜ H of orthogonal features within H , the regret bound adapts to Sparsity ˜ H , the sparsity of u 1: T measured on ˜ H . That is, we are theoretically justified to assemble smaller dictionaries into a larger one - the regret bound adapts to the quality of the optimal (comparator-dependent) sub-dictionary ˜ H .

How to choose the dictionary H ? In practice, we may use prior knowledge on the dynamics of the environment. For example, if the environment is periodic, such as the weather or the traffic, then a good choice could be the Fourier dictionary. Similarly, wavelet dictionaries are useful for piecewise regular environments. Another possibility is to learn the dictionary from offline datasets, which is also called representation learning . Overall, such prior knowledge is not required to be correct - our algorithm can take any dictionary as input, and the regret bound naturally adapts to its quality. The established connection between adaptivity and signal structures is a key benefit of our framework.

10 0

10 1

10 2

10 3

10 4

10 5

Indices (log)

10 0

10 2

10 4

10 6

Fourier coefficients (abs, log)

Data

Fitted,

=0.68

Power law For a more specific discussion, let us consider an empirically justified setup. In signal processing, the study of sparsity has been partially motivated by the power law [Pri21]: under the standard Fourier or wavelet transforms, the n -th largest transform domain coefficient of many real signals can have magnitude roughly proportional to n -α , where α ∈ (0 . 5 , 1) . We also observe this phenomenon from a weather dataset, with details presented in Appendix E.1. Figure 1 plots the sorted Fourier coefficients of an actual temperature sequence, on a log-log scale. A fitted dashed line is shown in orange, with (negative) slope α = 0 . 68 .

When the power law holds, our bound Eq.(7) has a more interpretable form. Assuming d = 1 and N = T ,

In a typical setting of E = Θ( T ) , we obtain a sublinear ˜ O ( T 1 . 5 -α ) dynamic regret bound.

3 The Haar OLR algorithm

This section presents the quantitative contributions of this paper: despite its generality, our sparse coding framework can improve existing results [JC22]. Our workhorse is the ability of wavelet bases to sparsely represent smooth signals.

3.1 Haar wavelet

Wavelet is a fundamental topic in signal processing, with long lasting impact throughout modern data science. Roughly, the motivation is that a signal can simultaneously exhibit nonstationarity at different time scales, such as slow drifts and fast jumps, therefore to faithfully represent it, we should apply feature vectors with different resolutions. We will only use the simplest Haar wavelets, which is already sufficient. Readers are referred to [Mal08, Joh19] for a thorough introduction to this topic.

Specifically, we start from the 1D setting ( d = 1 ) with a dyadic horizon ( T = 2 m , for some m ∈ N + ). The Haar wavelet dictionary consists of T (unnormalized) orthogonal feature vectors, indexed by a scale parameter j ∈ [1 : log 2 T ] and a location parameter l ∈ [1 : 2 -j T ] . Given a ( j, l ) pair, define a feature h ( j,l ) = [ h ( j,l ) 1 , . . . , h ( j,l ) T ] ∈ R T entry-wise as

It means that h ( j,l ) is only nonzero on a length2 j interval, while changing its sign once in the middle of this interval. Collecting all the ( j, l ) pairs yield T -1 features; then, we incorporate an extra all-one feature h ∗ = [1 , . . . , 1] to complete this size T dictionary.

The defined features can be assembled into the columns of a matrix Haar m . To help with the intuition, Haar 2 with T = 4 is presented in Eq.(8). The columns from the left to the right are h ∗ , h (2 , 1) , h (1 , 1) and h (1 , 2) . Observe that they are orthogonal, and the norm assumption from Section 2.1 is satisfied. Therefore, our sparsity adaptive regret bound Eq.(7) is applicable.

Given this 1D Haar wavelet dictionary, we apply a minor variant of Algorithm 2 to prevent the dimension d from appearing in the regret bound. When d = 1 , the algorithm is exactly Algorithm 2, where intuitions are most clearly demonstrated. Then, the doubling trick [SS11, Section 2.3.1] is adopted to relax the knowledge of T . The pseudocode is presented as Algorithm 5 in Appendix D.

Computation An appealing property is that most Haar wavelet features are supported on short local intervals. Despite N = T , there are only log 2 T active features in each round. Therefore, the runtime of our algorithm is O ( d log T ) per round, matching that of all the baselines we compare to. This local property holds for compactly supported wavelets, most notably the Daubechies family [Dau88, CDV93]. The latter can represent more general, piecewise polynomial signals.

3.2 Main result

For almost static environments, our Haar OLR algorithm guarantees the following bounds, by relating comparator smoothness to the sparsity of its Haar wavelet representation. Different from [JC22] which only contains P -dependent bounds, we also provide a K -switching regret bound, in order to avoid using P = O ( KM ) . 14 Interestingly, the proofs of the following two bounds are quite different: the first uses exact sparsity , while the second uses approximate sparsity .

Theorem 2 (Switching regret) . For all T ∈ N + and u 1: T ∈ R dT , Algorithm 5 guarantees

Theorem 3 (Path length bound) . For all T ∈ N + and u 1: T ∈ R dT , Algorithm 5 guarantees

It can be verified (Appendix A) that for all comparators u 1: T , our bounds are at least as good as prior works (Table 1). The optimality is a more subtle issue, as one should compare upper bound functions (of u 1: T ) to lower bound functions in a global manner, rather than comparing the exponents of T in minimax online learning.

Nonetheless, we present two examples of u 1: T , where the improvement can be clearly seen through better exponents of T . To give it a concrete background, suppose we want to sequentially predict a 1D time series z 1 , . . . , z T ∈ R . This could be formulated as a OCO problem where the decision x t there is our prediction of z t , and the loss function is the absolute loss l t ( x ) = | x -z t | . A natural

choice of the comparator is the ground truth sequence z 1: T , and due to Eq.(2), any upper bound on Regret T ( z 1: T ) also upper-bounds the total forecasting loss of our algorithm. Below we present specific 1D comparator sequences u 1: T to demonstrate the strength of our results, which could be intuitively thought as the true time series z 1: T in this more restricted discussion.

Example 1 (Tracking outliers) . Consider the situation where u 1: T has a locally outlying scale: we set all the instantaneous comparators u t to 1 , except k ≤ √ T consecutive members which are set to √ T . Crucially, | ¯ u | = O (1) and ¯ S = O ( k √ T ) , while M = √ T and S = Θ( T ) . With details deferred to Appendix D.7, both our bounds, i.e., Eq.(9) and (10), are ˜ O ( √ kT ) , while the fine baseline Eq.(4) is ˜ O ( T 3 / 4 ) , and the coarse baseline Eq.(3) is ˜ O ( T ) . The largest gain is observed when k is a constant, i.e., the comparator is subject to a short but large perturbation.

Example 2 (Persistent oscillation) . Consider the situation where ¯ u = 1 , and all the instantaneous comparators oscillate around ¯ u : u t = ¯ u + α t / √ T . α t = 1 or -1 , and it only switches sign for k times. Notice that ¯ S = √ T , while S = Θ( T ) . All the baselines are ˜ O ( √ T + k 1 / 2 T 1 / 4 ) , while both our bounds are ˜ O ( √ T ) . The largest gain is observed when k = T -1 , i.e., the comparator switches in every round.

In summary, we show that existing bounds are suboptimal, while the optimality of our results remains to be studied. It highlights the importance of comparator energy and variability in the pursuit of better algorithms, which have not received enough attention in the literature. Next, we briefly sketch the proofs of these bounds.

Proof sketch The switching regret bound mostly follows from a very simple observation: if a sequence is constant throughout the support of a Haar wavelet feature, then its transform domain coefficient for this feature is zero. As features on the same scale j do not overlap, a K -switching comparator can only induce K nonzero coefficients on the j -th scale. There are at most K log 2 T nonzero coefficients in total, therefore Sparsity H = ˜ O ( K ) . The bound Eq.(9) is obtained by applying this argument after taking out the average of u 1: T .

As for the path length bound, the idea is to consider the reconstructed sequences, using transform domain coefficients on a single scale j . These are usually called detail sequences in the wavelet literature [Mal08]. Each detail sequence has a relatively simple structure, whose path length and variability can be associated to the magnitude of its transform domain coefficients. Moreover, as these detail sequences are certain 'locally averaged' and 'globally centered' versions of the actual comparator u 1: T , their regularities are dominated by the regularity of u 1: T itself. In combination, this yields a relation between P ¯ S and the coefficients' L 1 norm, i.e., ∑ N n =1 ∥ ∥ z ( n ) ∥ ∥ 2 in Theorem 1, from which the bound is established.

Compared to the analysis of [JC22], the key advantage of our analysis is the decoupling of function approximation from the generic sparsity-based regret bound. The former is algorithm-independent, while the latter can be conveniently combined with advances in static online learning. With the help of approximation theory (e.g., Fourier features, wavelets, and possibly deep learning further down the line), intuitions are arguably clearer in this way, and solutions could be more precise (compared to analyses that 'mix' function approximation with regret minimization).

Additional discussion Finally, due to limited space, we defer additional discussion of our technical results to Appendix F, including

The related use of Multi-Resolution Analysis (MRA) in the existing online learning literature.

The comparison between Lipschitz and strongly convex losses in unconstrained dynamic OCO.

4 Conclusion

This paper presents a unified study of unconstrained and dynamic online learning, where the two problem structures are naturally connected via comparator adaptivity. Building on the synergy between static parameter-free algorithms and temporal representations, we develop an algorithmic framework achieving a generic sparsity-adaptive regret bound. Equipped with the wavelet dictionary, our framework improves the quantitative results from [JC22], by adapting to finer characterizations of the comparator sequence.

Acknowledgments and Disclosure of Funding

We thank Vivek Goyal for helpful pointers to the signal processing literature, and the NeurIPS reviewers for their constructive feedback. This research was partially supported by the NSF under grants CCF-2200052, DMS-1664644, and IIS-1914792, by the ONR under grant N00014-19-1-2571, by the DOE under grant DE-AC02-05CH11231, by the NIH under grant UL54 TR004130, and by Boston University.

References

[AABR09] Jacob Abernethy, Alekh Agarwal, Peter L Bartlett, and Alexander Rakhlin. A stochastic view of optimal regret through minimax duality. In Conference on Learning Theory , 2009.

[ABRT08] Jacob Abernethy, Peter L Bartlett, Alexander Rakhlin, and Ambuj Tewari. Optimal strategies and minimax lower bounds for online convex games. In Conference on Learning Theory , pages 415-423, 2008.

[AHMS13] Oren Anava, Elad Hazan, Shie Mannor, and Ohad Shamir. Online learning for time series prediction. In Conference on learning theory , pages 172-184. PMLR, 2013.

[AHZ15] Oren Anava, Elad Hazan, and Assaf Zeevi. Online time series prediction with missing data. In International conference on machine learning , pages 2191-2199. PMLR, 2015.

[AM16] Oren Anava and Shie Mannor. Heteroscedastic sequences: beyond gaussianity. In International Conference on Machine Learning , pages 755-763. PMLR, 2016.

[AW01] Katy S Azoury and Manfred K Warmuth. Relative loss bounds for on-line density estimation with the exponential family of distributions. Machine Learning , 43(3):211-246, 2001.

[BGZ15] Omar Besbes, Yonatan Gur, and Assaf Zeevi. Non-stationary stochastic optimization. Operations research , 63(5):1227-1244, 2015.

[BW19] Dheeraj Baby and Yu-Xiang Wang. Online forecasting of total-variation-bounded sequences. Advances in Neural Information Processing Systems , 32, 2019.

[BW20] Dheeraj Baby and Yu-Xiang Wang. Adaptive online estimation of piecewise polynomial trends. Advances in Neural Information Processing Systems , 33:20462-20472, 2020.

[BW21] Dheeraj Baby and Yu-Xiang Wang. Optimal dynamic regret in exp-concave online learning. In Conference on Learning Theory , pages 359-409. PMLR, 2021.

[BW22] Dheeraj Baby and Yu-Xiang Wang. Optimal dynamic regret in proper online learning with strongly convex losses and beyond. In International Conference on Artificial Intelligence and Statistics , pages 1805-1845. PMLR, 2022.

[BZW21] Dheeraj Baby, Xuandong Zhao, and Yu-Xiang Wang. An optimal reduction of TV-denoising to adaptive online learning. In International Conference on Artificial Intelligence and Statistics , pages 2899-2907. PMLR, 2021.

[CDV93] Albert Cohen, Ingrid Daubechies, and Pierre Vial. Wavelets on the interval and fast wavelet transforms. Applied and computational harmonic analysis , 1993.

[CLW21] Liyu Chen, Haipeng Luo, and Chen-Yu Wei. Impossible tuning made possible: A new expert algorithm and its applications. In Conference on Learning Theory , pages 1216-1259. PMLR, 2021.

[CO18] Ashok Cutkosky and Francesco Orabona. Black-box reductions for parameter-free online learning in banach spaces. In Conference On Learning Theory , pages 1493-1529. PMLR, 2018.

[CRT06] Emmanuel J Cand` es, Justin Romberg, and Terence Tao. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on information theory , 52(2):489-509, 2006.

[Cut19] Ashok Cutkosky. Combining online learning guarantees. In Conference on Learning Theory , pages 895-913. PMLR, 2019.

[Cut20] Ashok Cutkosky. Parameter-free, dynamic, and strongly-adaptive online learning. In International Conference on Machine Learning , pages 2250-2259. PMLR, 2020.

[CWW19] Xi Chen, Yining Wang, and Yu-Xiang Wang. Nonstationary stochastic optimization under L p,q -variation measures. Operations Research , 67(6):1752-1765, 2019.

[Dau88] Ingrid Daubechies. Orthonormal bases of compactly supported wavelets. Communications on pure and applied mathematics , 41(7):909-996, 1988.

[DGSS15] Amit Daniely, Alon Gonen, and Shai Shalev-Shwartz. Strongly adaptive online learning. In International Conference on Machine Learning , pages 1405-1411. PMLR, 2015.

[DHS11] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research , 12(7), 2011.

[DSSST10] John C Duchi, Shai Shalev-Shwartz, Yoram Singer, and Ambuj Tewari. Composite objective mirror descent. In COLT , volume 10, pages 14-26. Citeseer, 2010.

[FKK16] Dean Foster, Satyen Kale, and Howard Karloff. Online sparse linear regression. In Conference on Learning Theory , pages 960-970. PMLR, 2016.

[FKMS17] Dylan J Foster, Satyen Kale, Mehryar Mohri, and Karthik Sridharan. Parameter-free online learning via model selection. Advances in Neural Information Processing Systems , 30, 2017.

[FRS18] Dylan J Foster, Alexander Rakhlin, and Karthik Sridharan. Online learning: Sufficient statistics and the Burkholder method. In Conference On Learning Theory , pages 3028-3064. PMLR, 2018.

[Ger13] S´ ebastien Gerchinovitz. Sparsity regret bounds for individual sequences in online linear regression. The Journal of Machine Learning Research , 14(1):729-769, 2013.

[GG15] Pierre Gaillard and S´ ebastien Gerchinovitz. A chaining algorithm for online nonparametric regression. In Conference on Learning Theory , pages 764-796. PMLR, 2015.

[GS16] Andras Gyorgy and Csaba Szepesv´ ari. Shifting regret, mirror descent, and matrices. In International Conference on Machine Learning , pages 2943-2951. PMLR, 2016.

[GW18] Pierre Gaillard and Olivier Wintenberger. Efficient online algorithms for fast-rate regret bounds under sparsity. Advances in Neural Information Processing Systems , 31, 2018.

[GY14] S´ ebastien Gerchinovitz and Jia Yuan Yu. Adaptive and optimal online linear regression on L 1 -balls. Theoretical Computer Science , 519:4-28, 2014.

[Haz16] Elad Hazan. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2(3-4):157-325, 2016.

[HLS + 18] Elad Hazan, Holden Lee, Karan Singh, Cyril Zhang, and Yi Zhang. Spectral filtering for general linear dynamical systems. Advances in Neural Information Processing Systems , 31, 2018.

[HR09] Niall Hurley and Scott Rickard. Comparing measures of sparsity. IEEE Transactions on Information Theory , 55(10):4723-4741, 2009.

[HW01] Mark Herbster and Manfred K Warmuth. Tracking the best linear predictor. The Journal of Machine Learning Research , 1:281-309, 2001.

[HW15] Eric C Hall and Rebecca M Willett. Online convex optimization in dynamic environments. IEEE Journal of Selected Topics in Signal Processing , 9(4):647-662, 2015.

[JC22] Andrew Jacobsen and Ashok Cutkosky. Parameter-free mirror descent. In Conference on Learning Theory , pages 4160-4211. PMLR, 2022.

[Joh19] Iain M Johnstone. Gaussian estimation: Sequence and wavelet models. Unpublished lecture notes , 2019. https://imjohnstone.su.domains/GE_09_16_19.pdf .

[JOWW17] Kwang-Sung Jun, Francesco Orabona, Stephen Wright, and Rebecca Willett. Improved strongly adaptive online learning using coin betting. In Artificial Intelligence and Statistics , pages 943-951. PMLR, 2017.

[JRSS15] Ali Jadbabaie, Alexander Rakhlin, Shahin Shahrampour, and Karthik Sridharan. Online optimization: Competing with dynamic comparators. In Artificial Intelligence and Statistics , pages 398-406. PMLR, 2015.

[Kal14] Satyen Kale. Open problem: Efficient online sparse regression. In Conference on Learning Theory , pages 1299-1301. PMLR, 2014.

[KKLP17] Satyen Kale, Zohar Karnin, Tengyuan Liang, and D´ avid P´ al. Adaptive feature selection: Computationally efficient online sparse linear regression under rip. In International Conference on Machine Learning , pages 1780-1788. PMLR, 2017.

[KM16] Vitaly Kuznetsov and Mehryar Mohri. Time series prediction and online learning. In Conference on Learning Theory , pages 1190-1213. PMLR, 2016.

[LLZ09] John Langford, Lihong Li, and Tong Zhang. Sparse online learning via truncated gradient. Journal of Machine Learning Research , 10(3), 2009.

[LS15] Haipeng Luo and Robert E Schapire. Achieving all with no parameters: Adanormalhedge. In Conference on Learning Theory , pages 1286-1304. PMLR, 2015.

[MA13] Brendan McMahan and Jacob Abernethy. Minimax optimal algorithms for unconstrained linear optimization. Advances in Neural Information Processing Systems , 26:2724-2732, 2013.

[Mal08] Stephane Mallat. A Wavelet Tour of Signal Processing: The Sparse Way . Academic Press, 2008.

[MK20] Zakaria Mhammedi and Wouter M Koolen. Lipschitz and comparator-norm adaptivity in online learning. In Conference on Learning Theory , pages 2858-2887. PMLR, 2020.

[MO14] H Brendan McMahan and Francesco Orabona. Unconstrained online linear learning in hilbert spaces: Minimax algorithms and normal approximations. In Conference on Learning Theory , pages 1020-1039. PMLR, 2014.

[OP16] Francesco Orabona and D´ avid P´ al. Coin betting and parameter-free online learning. Advances in Neural Information Processing Systems , 29, 2016.

[Ora19] Francesco Orabona. A modern introduction to online learning. arXiv preprint arXiv:1912.13213 , 2019.

[Pri21] Eric Price. Sparse recovery. In Tim Roughgarden, editor, Beyond the Worst-Case Analysis of Algorithms , page 140-164. Cambridge University Press, 2021.

[RS14a] Alexander Rakhlin and Karthik Sridharan. Online non-parametric regression. In Conference on Learning Theory , pages 1232-1264. PMLR, 2014.

[RS14b] Alexander Rakhlin and Karthik Sridharan. Statistical learning and sequential prediction. Unpublished lecture notes , 2014. http://www.mit.edu/ ~ rakhlin/courses/stat928/stat928_ notes.pdf .

[SBG + 21] Martin G Schultz, Clara Betancourt, Bing Gong, Felix Kleinert, Michael Langguth, Lukas Hubert Leufen, Amirpasha Mozaffari, and Scarlet Stadtler. Can deep learning beat numerical weather prediction? Philosophical Transactions of the Royal Society A , 379(2194):20200097, 2021.

[SM12] Matthew Streeter and Brendan Mcmahan. No-regret algorithms for unconstrained online convex optimization. Advances in Neural Information Processing Systems , 25, 2012.

[SS11] Shai Shalev-Shwartz. Online learning and online convex optimization. Foundations and trends in Machine Learning , 4(2):107-194, 2011.

[SST11] Shai Shalev-Shwartz and Ambuj Tewari. Stochastic methods for L 1 -regularized loss minimization. Journal of Machine Learning Research , 12:1865-1892, 2011.

[Tib96] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological) , 58(1):267-288, 1996.

[vdH19] Dirk van der Hoeven. User-specified local differential privacy in unconstrained adaptive online learning. Advances in Neural Information Processing Systems , 32, 2019.

[Vov01] Volodya Vovk. Competitive on-line statistics. International Statistical Review , 69(2):213-248, 2001.

[Xia09] Lin Xiao. Dual averaging method for regularized stochastic learning and online optimization. Advances in Neural Information Processing Systems , 22, 2009.

[YZJY16] Tianbao Yang, Lijun Zhang, Rong Jin, and Jinfeng Yi. Tracking slowly moving clairvoyant: Optimal dynamic regret of online learning with true and noisy gradient. In International Conference on Machine Learning , pages 449-457. PMLR, 2016.

[ZCP22] Zhiyu Zhang, Ashok Cutkosky, and Ioannis Paschalidis. PDE-based optimal strategy for unconstrained online learning. In International Conference on Machine Learning , pages 26085-26115. PMLR, 2022.

[Zin03] Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning , pages 928-936, 2003.

[ZLZ18] Lijun Zhang, Shiyin Lu, and Zhi-Hua Zhou. Adaptive online learning in dynamic environments. Advances in neural information processing systems , 31, 2018.

[ZYY + 17] Lijun Zhang, Tianbao Yang, Jinfeng Yi, Rong Jin, and Zhi-Hua Zhou. Improved dynamic regret for non-degenerate functions. Advances in Neural Information Processing Systems , 30, 2017.

[ZYZ + 18] Lijun Zhang, Tianbao Yang, Zhi-Hua Zhou, et al. Dynamic regret of strongly adaptive methods. In International conference on machine learning , pages 5882-5891. PMLR, 2018.

[ZZZZ20] Peng Zhao, Yu-Jie Zhang, Lijun Zhang, and Zhi-Hua Zhou. Dynamic regret of convex and smooth functions. Advances in Neural Information Processing Systems , 33:12510-12520, 2020.

[ZZZZ21] Peng Zhao, Yu-Jie Zhang, Lijun Zhang, and Zhi-Hua Zhou. Adaptivity and nonstationarity: Problem-dependent dynamic regret for online convex optimization. arXiv preprint arXiv:2112.14368 , 2021.

Appendix

Organization Appendix A summarizes a list of comparator statistics involved in our theoretical analysis. Appendix B surveys additional related works. Appendix C, D and E respectively present details of our general sparse coding framework, its special version with wavelet dictionaries, and applications in time series forecasting. Finally, Appendix F presents additional discussion of our technical results.

A List of comparator statistics

A major task in comparator adaptive online learning is finding suitable statistics to quantify the regularity of a comparator sequence. Several such statistics are defined throughout this paper, which are summarized in Table 2. Note that for the definition of sparsity on the last line, we assume the dictionary H is orthogonal, and the sequence u 1: T is contained in its span. Then, z ( n ) is defined as the projection of u 1: T onto a feature vector h 1: T,n , i.e.,

This is well defined due to our assumption ∥ h 1: T,n ∥ 2 ≥ 1 from Section 2.1.

̸

Next, let us discuss their relations, in order to interpret our quantitative contribution more clearly (Table 1). It is clear that S ≤ MT , therefore the fine baseline Eq.(4) from [JC22] improves the coarse one Eq.(3); comparing their associated switching regret bounds follows the same reasoning.

To compare our results to the baselines, observe that

Therefore,

That is, both our path-length-dependent bound and the switching regret bound are at least as good as the results from [JC22]. Concrete benefits are demonstrated in Example 1 and 2.

Finally, as a sanity check, our path-length-dependent bound does not violate the lower bound Ω( P ) : even when ¯ u = 0 ,

therefore our bound is never better than ˜ O ( P ) .

B More on related work

Online regression Our sparse coding framework converts unconstrained dynamic OCO to a special form of online regression. The standard setting of the latter [RS14a] considers a repeated game as well: in each round, we observe a covariate x t ∈ R d , make a prediction ˆ y t ∈ R (which depends on x t ), and then observe a label y t ∈ R . The performance metric is the minimax regret under the square loss

Roughly, the problem is of a nonparametric type if the complexity of the function class F is not fixed a priori, but grows with T (i.e., the amount of data).

Overall, such an online regression problem is highly general, as static OCO is recovered if x t is time-invariant. The setting we utilize is a variant with ( i ) vector output; ( ii ) general convex losses; ( iii ) x t specified by the dictionary, possibly being sparse itself (e.g., wavelets); and ( iv ) the function class F being linear, but unbounded. As discussed in Footnote 11, our setting deviates from the conventional definition of regression, as a general convex loss function does not necessarily have minimizers. We adopt the terminology of 'regression' for streamlined exposition.

Existing works on online nonparametric regression [RS14a, GG15] have established the relation of this problem to certain path length characterizations of dynamic regret. However, the generality of this setting makes the analysis challenging, and especially, algorithms can be computationally expensive. With a bounded domain assumption (on predictions ˆ y t ), a recent breakthrough [BW21] simultaneously achieved several notions of optimality for path-length-dependent bounds, with efficient computation. Readers are referred to [BW21, Appendix A] for a thorough discussion of this line of works.

For the special case of Online Linear Regression (OLR) with square losses, the celebrated VAW forecaster [AW01, Vov01] guarantees O ( N log T ) regret against any unbounded coefficient vector ˆ u ∈ R N , where N is the dimension of the feature space. Such a fast rate becomes vacuous in the nonparametric regime (when N > T ) [GY14], therefore [Ger13] proposed a sparsity regret bound ˜ O ( ∥ ˆ u ∥ 0 ) and an accompanying inefficient algorithm as its high dimensional generalization. Efficient computation was addressed by [GW18], but the obtained result only applies to bounded ˆ u . In a rough sense, such sparsity regret bounds are the square loss and feature-based analogue of the L 1 -norm parameter-free bounds in OLO [Ora19, Chapter 9]. They are also closely related to sparsity oracle inequalities in statistics, as reviewed by [Ger13].

Parametric time series models For time series forecasting, most prior works are devoted to parametric strategies with strong inductive bias, such as the ARMA model, state space models, and more recent deep learning models. Online learning has been applied to such models as well [AHMS13, AHZ15, AM16, KM16, HLS + 18], leading to forecasting guarantees under mild statistical assumptions. When convexity is present, some of these problems could be reframed as special cases of our OLR problem, with a constant-size dictionary that does not grow with T ; for example, learning the autoregressive model corresponds to defining the features as the fixed-length observation history. Also, Appendix E shows that given a parametric time series forecaster (possibly without performance guarantees), our algorithm can be applied on top of it, in order to provably correct its nonstationary bias.

Other sparsity topics in OL Finally, we review other sparsity-related topics in online learning, which do not fit into the scope of this paper. [LLZ09, Xia09, DSSST10, SST11] considered using

online learning to solve batch L 1 regularized problems. The goal is to achieve sparse predictions instead of sparsity adaptive regret bounds. [Kal14, FKK16, KKLP17] studied online sparse regression , where only a subset of features are available in each round. The challenge is to handle bandit feedback in OLR.

C Detail on the general framework

This section presents details on our general sparse coding framework. Appendix C.1 introduces the static subroutine we adopt from [MK20]. Appendix C.2 proves our main results, but with additional gradient adaptivity compared to the main paper.

C.1 Unconstrained static subroutine

The following static OCO algorithm and its guarantee are due to [MK20, Section 3.1]. We assume that ∥ ˆ g t ∥ 2 ≤ ˆ G , and ˆ G > 0 .

Algorithm 3 FREEGRAD [MK20, Definition 4]: scale-free and gradient adaptive unconstrained static OLO.

Require: A hyperparameter ε > 0 ; dimension d ; Lipschitz constant ˆ G .

1: Initialize a gradient sum counter s = 0 ∈ R d and a variance counter v = ˆ G 2 .

2: for t = 1 , 2 , . . . do

3: Predict

4: Observe the loss gradient ˆ g t .

5: Update s ← s +ˆ g t , and v ← v +ˆ g 2 t .

6: end for

Lemma C.1 (Theorem 20 of [MK20]) . With any hyperparameter ε > 0 , for all T ∈ N + and ˆ u ∈ R , Algorithm 3 guarantees

where

C.2 Proof of the main result

We now present the analysis of our general sparse coding framework. The following lemma is a slightly more general version of Lemma 2.1 in the main paper, which characterizes the performance of our single direction learner (Algorithm 1). Recall that g t ∈ ∂l t ( x t ) from the OCO-OLO reduction.

Lemma C.2 (Lemma 2.1, full) . Let ε > 0 be an arbitrary hyperparameter for Algorithm 3. Applying its 1D version as the static subroutine, for all T ∈ N + and u 1: T ∈ span( h 1: T ) , against any adversary E , Algorithm 1 guarantees

The simplified form (Lemma 2.1) is recovered by using ∥ h 1: T ∥ 2 ≥ 1 and ∥ g t ∥ 2 ≤ G .

Proof of Lemma 2.1. Subsuming poly-logarithmic factors, the static regret bound of our static subroutine (Algorithm 3) can be written as

where ˆ u is any 1D static comparator that the subroutine handles.

Now, for any single-directional comparator u 1: T ∈ span( h 1: T ) considered in this lemma, there exists ˆ u ∈ R such that u 1: T = ˆ uh 1: T . The dynamic regret can be rewritten as

and the RHS can be bounded using the static regret bound above. Note that | ˆ g t | = |⟨ g t , h t ⟩| ≤ G , therefore the surrogate Lipschitz constant ˆ G from the static regret bound can be assigned to G . In summary,

where the last line is due to our assumption that ∥ h 1: T ∥ 2 ≥ 1 .

Next, we prove the unconstrained dynamic regret bound with general dictionaries (Theorem 1).

Theorem 4 (Theorem 1, full) . Consider any collection of signals z ( n ) ∈ span( h 1: T,n ) , ∀ n . We define its reconstruction error (for the comparator u 1: T ) as z (0) = u 1: T -∑ N n =1 z ( n ) ∈ R dT . Then, for all T ∈ N + and u 1: T ∈ R dT , against any adversary E , Algorithm 2 guarantees

where z ( n ) t ∈ R d is the t -th round component of the sequence z ( n ) ∈ R dT .

Proof of Theorem 4. The idea of this theorem is a dynamic analogue of [Cut19] to aggregate the regret bound of single direction learners. For all decomposition u 1: T = ∑ N n =0 z ( n ) such that z ( n ) ∈ span( h 1: T,n ) for all n ∈ [1 : T ] , we have

For the first term on the RHS, 〈 -g 1: T , z (0) 〉 = -∑ T t =1 〈 g t , z ( n ) t 〉 . As for the rest, we plug in Lemma C.2, with hyperparameter ε/N .

Next, we show how this dynamic regret bound recovers the static regret bound in R d . As discussed in Section 2.2, the static setting amounts to picking N = d and H t = I d , and the decomposed signals z ( n ) are determined by orthogonal projection of the static comparator sequence u 1: T = [ u, . . . , u ] .

Specifically, z ( n ) t is a d -dimensional vector which is zero except the n -th entry; its n -th entry equals the n -th entry of the static comparator u . If we index the gradient as g t = [ g t, 1 , . . . , g t,d ] ∈ R d and the static comparator as u = [ u 1 , . . . , u d ] ∈ R d , then 〈 g t , z ( n ) t 〉 = g t,n u n . Applying Theorem 4, against static u 1: T ,

In the asymptotic regime with large T , Regret T ( u 1: T ) = ˜ O ( ∥ u ∥ 2 √ T ) .

D Detail on the wavelet algorithm

This section presents details of our wavelet algorithm. The pseudocode is presented in Appendix D.1. Appendix D.2 introduces the wavelet-specific notations for our analysis. Appendix D.3 presents a generic sparsity based bound for our algorithm. Appendix D.4 and D.5 prove our main results. Auxiliary lemmas are contained in Appendix D.6. Finally, Appendix D.7 works out the details of the two examples from the main paper.

D.1 Pseudocode

For all m ∈ N + , let T = 2 m , and let Haar m be the T × T Haar dictionary matrix defined in Section 3.1, for d = 1 . We apply the following variant (Algorithm 4) of our sparse coding framework, in order to remove all d dependence from the final regret bound. It adopts the d dimensional version of the static subroutine (FREEGRAD), instead of the 1D version in Section 2. The pseudocode mirrors the combination of Algorithm 1 and 2.

It is equivalent to view Algorithm 4 as operating on a dT × dT 'master' dictionary matrix H , defined block-wise as the following: for all ( i, j ) ∈ [1 : T ] 2 , the ( i, j ) -th block of H is the product of the

Algorithm 4 Haar OLR with known time horizon.

Require: A time horizon T = 2 m ; the T × T Haar dictionary matrix Haar m ; and a hyperparameter ε > 0 (default is 1 ).

1: Let N = T . For all n ∈ [1 : N ] , initialize a copy of the d dimensional version of Algorithm 3 (FREEGRAD) as A n , with hyperparameter ε/N .

2: for t = 1 , 2 , . . . , do

3: Receive the t -th row of Haar m , and index it as [ h t, 1 , . . . , h t,N ] ; note that h t,n ∈ R .

4: for n = 1 , 2 , . . . , N do

̸

5: If h t,n = 0 , query A n for its output, and assign it to ˆ x t,n ∈ R d ; otherwise, ˆ x t,n is arbitrary.

6: Define w t,n = h t,n ˆ x t,n ∈ R d .

7: end for

9: for n = 1 , 2 , . . . , N do

8: Predict x t = ∑ N n =1 w t,n ∈ R d , receive loss gradient g t ∈ R d .

̸

10: If h t,n = 0 , compute ˆ g t,n = h t,n g t and send it to A n as its surrogate loss gradient.

11: end for

12: end for

Algorithm 5 Anytime Haar OLR (Algorithm 4 with doubling trick).

1: for m = 1 , 2 , . . . , do

2: Run Algorithm 4 for 2 m rounds, which uses the matrix Haar m . The hyperparameter is set to 1 .

3: end for

( i, j ) -th entry of Haar m (which is a scalar) and the d -dimensional identity matrix I d . That is, H is a block matrix; each block is a diagonal matrix with equal diagonal entries determined by Haar m . Roughly, the algorithm measures distances in R d by the L 2 norm, while measuring R T by the L 1 norm.

Algorithm 4 alone is not sufficient for our purpose: it must take an integer m and run for a fixed T = 2 m rounds. We apply a meta algorithm (Algorithm 5), which simply restarts the known T algorithm using the classical doubling trick, c.f., [SS11, Section 2.3.1].

D.2 More background

Although the analysis of our framework is simpler than [JC22], a challenge is carefully indexing all the quantities to account for the vectorized setting. It is thus important to introduce a few notations to streamline the presentation. Haar m is the T × T Haar dictionary matrix defined in Section 3.1, with T = 2 m . Recall the statistics of the comparator sequence, summarized in Appendix A.

Local interval Given any scale-location pair ( j, l ) , let the support I ( j,l ) be the time interval where the feature h ( j,l ) is nonzero. That is,

Moreover, let I ( j,l ) + denote the first half of this interval, and I ( j,l ) -for the second half. h ( j,l ) is 1 on I ( j,l ) + , and -1 on I ( j,l ) -.

Normalization Let ˜ Haar m be the orthonormal matrix obtained by scaling the columns of Haar m . The normalized feature vectors are also denoted by tilde, i.e., instead of h ∗ and h ( j,l ) , the normalized features are ˜ h ∗ and ˜ h ( j,l ) . They are vectors in R T , with the t -th component denoted by ˜ h ∗ t and ˜ h ( j,l ) t , in R .

Coordinate sequence Consider any comparator sequence u 1: T ∈ R dT . For all coordinate i ∈ [1 : d ] , we define its i -th coordinate sequence as u ( i ) 1: T ∈ R T : the t -th entry of this coordinate sequence u ( i ) 1: T , denoted by u ( i ) t , is the i -th coordinate of u t .

Transform domain coefficient We will also use the transform domain coefficients of u 1: T , under the Haar wavelet transform. Recall that in the single-feature, generic setting (Section 2.2), we denoted a single transform domain coefficient by ˆ u ∈ R . With wavelets, the transform domain encodes dT -dimensional vectors. According to our convention so far, we will denote them by scale-location pairs ( j, l ) : given a ( j, l ) pair, the 'coefficient' ˆ u ( j,l ) is a d -dimensional vector. There are T -1 pairs of ( j, l ) in total; complementing the representation, we use another ˆ u ∗ ∈ R d to represent the 'coefficient' for the all-one feature.

Given any scale parameter j ∈ [1 : log 2 T ] and location parameter l ∈ [1 : 2 -j T ] , let

and for the all-one feature,

That is, each entry is the inner product between the normalized feature and a coordinate sequence from u 1: T .

Due to the orthonormality of the applied transform (specified by the normalized features ˜ h ∗ and ˜ h ( j,l ) ), the energy is preserved between the time domain and the transform domain, i.e.,

and also the second order variability (the energy of the centered dynamic component within u 1: T ),

Moreover, since ˜ h ∗ equals 1 / √ T times the all-one vector,

Detail reconstruction Given the transform domain coefficients, we can reconstruct details of the comparator u 1: T on the time domain. Similar to our notation in the generic framework (Section 2.2), we keep the letter z , but replace the index n by ( j, l ) , which is more suitable for indexing wavelets.

Let z ( j,l ) ∈ R dT be the detail of u 1: T along the ( j, l ) -th feature. It is the concatenation of T vectors in R d , and for all t , the t -th of these vectors is defined by

Similarly, we can define the detail z ∗ along the feature ˜ h ∗ . Its t -th component is

and clearly, the RHS does not depend on t since ˜ h ∗ is the normalization of the all-one feature h ∗ .

Let us also sum the details across different locations. Given a scale j , let

Note that the summands are sequences that do not overlap: at each entry, only one of the summand sequence is nonzero. The full reconstruction is obtained by summing all the details,

Statistics of the detail sequence We can define statistics of the detail sequences just like the statistics of the comparator u 1: T . Specifically, define the first order variability of the ( j, l ) -th detail as

Note that since the z ( j,l ) t sequence is centered (with average being equal to 0 ), its first order variability equals its norm sum, c.f., Appendix A. Summing over the locations, the first order variability at the j -th scale is

which equals ∑ l ¯ S ( j,l ) .

Similarly, we can define the path length of the detail sequences. A caveat is that we only count the path length within the support I ( j,l ) of the feature h ( j,l ) ,

The comparator's movement when the support changes does not count. Summing over the locations,

D.3 Generic sparsity adaptive bound

With the notation from the previous subsection, we now present a generic sparsity adaptive regret bound for Algorithm 4 (fixed T Haar OLR). Since the latter is a variant of our main sparse coding framework (Section 2), the result can be analogously derived, although the notations need to be treated carefully.

Lemma D.1. For any m , T = 2 m and u 1: T ∈ R dT , with any hyperparameter ε > 0 , Algorithm 4 guarantees

The proof sums the regret bound of the d -dimensional version of the static subroutine (Lemma C.1), across T different copies. It is very similar to Theorem 1, therefore omitted.

It might be more convenient to use the transform domain coefficients ˆ u ( j,l ) in the bound, rather than the reconstructed details z ( j,l ) . In this case, we have

Similarly,

Therefore,

D.4 Unconstrained switching regret

In the K -switching regret, the complexity of the comparator is characterized by its amount of switches. The idea is that, if the comparator u 1: T is static on a support I ( j,l ) for some ( j, l ) , then the corresponding transform domain coefficient ˆ u ( j,l ) = 0 ∈ R d . We have the following bound for the fixed T algorithm (Algorithm 4).

Lemma D.2. For any m , T = 2 m and u 1: T ∈ R dT , Algorithm 4 with the hyperparameter ε = 1 guarantees

Proof of Lemma D.2. Consider any scale j . Since the supports { I ( j,l ) } l do not overlap, if u 1: T shifts K times, then there are at most K choices of location l such that the transform domain coefficient ˆ u ( j,l ) is nonzero. Furthermore, since there are log 2 T scales in total, there are at most K log 2 T pairs of ( i, l ) such that ˆ u ( j,l ) is nonzero. Therefore, using Cauchy-Schwarz and Eq.(11),

Plugging this into Eq.(13), and further using Eq.(12) for ∥ ˆ u ∗ ∥ 2 complete the proof.

The anytime bound in general follows from the classical doubling trick. A twist is that the analysis is slightly more involved than the standard one, e.g., [SS11, Section 2.3.1], as we also need to relate the comparator statistics on each block to those for the entire signal u 1: T .

Theorem 2 (Switching regret) . For all T ∈ N + and u 1: T ∈ R dT , Algorithm 5 guarantees

Proof of Theorem 2. First, assume T can be exactly decomposed into m ∗ segments with dyadic lengths 2 1 , . . . , 2 m ∗ . We use ¯ u m , K m and ¯ E m to represent the statistics of the comparator sequence on the length 2 m block, and let I m denote the time interval that this block operates on. ¯ u , K and S denote the statistics of the entire signal u 1: T , c.f., Appendix A. From Lemma D.2,

The first term follows from the standard doubling trick analysis [SS11, Section 2.3.1],

As for the second term in Eq.(14), using Cauchy-Schwarz,

m ∗ = O (log T ) , and also observe that the sum (in the parenthesis) on the RHS equals the second order variability of the following signal: for any time t in the m -th block, the signal's component is ¯ u m ∈ R d . This signal is a locally averaged version of the original comparator u 1: T , and the key idea is that local averaging decreases the variability. Formally, due to Lemma D.7, we have

For the third term in Eq.(14), using Cauchy-Schwarz again,

The sum of K m is straightforward. The inequality for the sum of ¯ E m follows from the observation that on the m -th block, ¯ u m minimizes ∑ t ∈ I m ∥ u t -x ∥ 2 2 with respect to x ∈ R d .

Also, notice that the second term in Eq.(14) is dominated by the third term. If K = 0 , then both √ ¯ E and √ K ¯ E equal 0. If K ≥ 1 , then √ ¯ E ≤ √ K ¯ E . Therefore, Eq.(14) can be written as

As for the general setting where T cannot be exactly decomposed into dyadic blocks: consider the smallest T ∗ > T such that T ∗ can be decomposed. Due to doubling intervals, T ∗ ≤ 2 T . Let us consider a hypothetical length T ∗ game with the rounds t > T constructed as follows: the loss gradient g t = 0 ∈ R d , and u t = ¯ u . In this case, with K and ¯ E still representing the statistics of the length T sequence u 1: T , the number of switches on the entire time interval [1 : T ∗ ] is at most K +1 , and the second order variability on [1 : T ∗ ] is ¯ E ; furthermore, it is clear that ( K +1) ¯ E ≤ 2 K ¯ E . The regret of any algorithm on this hypothetical length T ∗ game is the same as the length T game, therefore bounding the latter follows from bounding the former.

D.5 Path-length-based bound

Next, we turn to bounds that depend on the path length P of the comparator u 1: T . Similar to the switching regret analysis, we will first consider the setting with fixed dyadic T (Algorithm 4), and then extend its guarantee through a doubling trick.

D.5.1 Fixed dyadic horizon

In the following, we consider Algorithm 4; assume T = 2 m for some m . The static component (i.e., z ∗ ) and the dynamic component (i.e., u 1: T -z ∗ ) of u 1: T are analyzed separately; the former is fairly standard, while the latter is more challenging. We will first consider the dynamic component, and proceed in three steps.

Step 1 Considering any scale j , we aim to show ∑ l ∥ ∥ ˆ u ( j,l ) ∥ ∥ 2 ≤ √ P ( j ) ¯ S ( j ) , which relates the transform domain coefficients to the regularity of the reconstructed signals.

Lemma D.3. For all ( j, l ) pair, and

Rewriting P ( j,l ) and ¯ S ( j,l ) ,

which yields the equality in the lemma. The second part follows from Cauchy-Schwarz.

Proof of Lemma D.3. Let us start from the first part of this lemma, and express the detail sequence z ( j,l ) , and equivalently z ( j ) , more explicitly on its support I ( j,l ) .

Step 2 Showing that P ( j ) ≤ P and ¯ S ( j ) ≤ ¯ S . That is, the reconstructed signals are easier than the original comparator u 1: T . Here, P and ¯ S should be considered separately.

Lemma D.4. For any u 1: T and any scale parameter j ∗ , P ( j ∗ ) ≤ P .

Proof of Lemma D.4. From the definition of P and the reconstruction of u 1: T from detail sequences,

where the last equality is due to z ∗ being a constant sequence.

Consider removing 'shorter' scales with 1 ≤ j < j ∗ , which is equivalent to local averaging, c.f., Appendix D.6. Due to Lemma D.7, the path length does not increase, i.e,

Then, we can further remove the rounds where the path length is not counted in P ( j ∗ ) , i.e., when a time t ∈ I ( j ∗ ,l ) but t +1 ∈ I ( j ∗ ,l +1) .

Now, consider any location l , which determines the time interval I ( j ∗ ,l ) = [2 j ∗ ( l -1) + 1 : 2 j ∗ l ] . Any detail sequence z ( j ) with scale j > j ∗ is constant on this time interval, thus removing it does not change the path length at all. Therefore,

As for the first order variability,

Lemma D.5. For any u 1: T and any scale parameter j ∗ , ¯ S ( j ∗ ) ≤ ¯ S .

Proof of Lemma D.5. From the definition, noticing that ¯ u is entirely captured by the all-one feature,

Due to Lemma D.7, removing short scales amounts to local averaging, which decreases the variability.

For any l , consider the support of the ( j ∗ , l ) -th feature, I ( j ∗ ,l ) . Observe that ∑ j>j ∗ z ( j ) t is time invariant throughout I ( j ∗ ,l ) , let us denote it as v ∈ R d . Meanwhile, for some w ∈ R d , z ( j ∗ ) t equals w on I ( j ∗ ,l ) + , the first half of this interval, while being -w on the second half I ( j ∗ ,l ) -of this interval. Therefore,

Combining the above,

Step 3 Summarizing the above relations, and using the property that there are only log 2 T scales. Lemma D.6. For any m , T = 2 m and u 1: T ∈ R dT , Algorithm 4 with the hyperparameter ε = 1 guarantees

Proof of Lemma D.6. Starting from the generic regret bound, Eq.(13) for Algorithm 4.

Due to Eq.(12), ∥ ˆ u ∗ ∥ 2 = ∥ ¯ u ∥ 2 √ T . Then, combining Lemma D.3, D.4 and D.5,

Plugging it into the generic bound completes the proof.

D.5.2 Anytime bound

Now we are ready to prove an anytime unconstrained dynamic regret bound that depends on the path length.

Theorem 3 (Path length bound) . For all T ∈ N + and u 1: T ∈ R dT , Algorithm 5 guarantees

Proof of Theorem 3. Similar to the analysis of the switching regret (Theorem 2), we first consider the situation where the time horizon T can be exactly decomposed into m ∗ segments with dyadic lengths 2 1 , . . . , 2 m ∗ . In this situation, we have

where the second line follows from the proof of Theorem 2, specifically Eq.(15) and Eq.(16). Now let us consider the remaining sum on the RHS. Using Cauchy-Schwarz,

where

The last sum on the RHS is the first order variability of a locally averaged version of u 1: T . Due to Lemma D.7,

Combining everything above,

It remains to show that √ ¯ E ≤ √ P ¯ S , thus the former can be absorbed into the latter. Plugging in the definitions, this is equivalent to showing

and it suffices to prove ∥ u t -¯ u ∥ ≤ P for all t ∈ [1 : T ] . This is completed in Lemma D.8. Till this point, we have shown the desirable result in the situation of 'exact dyadic partitioning'.

To complete the proof, we turn to the general situation where T cannot be partitioned into dyadic blocks. This follows from a similar 'padding' construction from the proof of Theorem 2. Let T ∗ = 2 ⌈ log 2 T ⌉ , and by definition, T ∗ ≤ 2 T . Let us consider a hypothetical length T ∗ game with the rounds t > T constructed as follows: the loss gradient g t = 0 ∈ R d , and u t = ¯ u . Then, the regret of any algorithm on the length T ∗ hypothetical game equals its regret on the actual length T game, and the regret bound for the former applies to the latter as well: if we write P ∗ and ¯ S ∗ as the statistics of the extended length T ∗ comparator, then

Clearly, ¯ S ∗ = ¯ S and T ∗ ≤ 2 T . As for the path length, P ∗ = P + ∥ u T -¯ u ∥ 2 , and due to Lemma D.8, ∥ u T -¯ u ∥ 2 ≤ P . Plugging it back completes the proof.

D.6 Useful lemma

Our analysis uses two auxiliary lemmas. First, we show that local averaging makes a signal 'more regular'. Consider any signal u 1: T ∈ R dT , with the t -th round component u t ∈ R d . Local averaging refers to replacing any k consecutive components of u 1: T by their average, i.e., setting

for some τ ∈ [0 : T -k ] .

Lemma D.7. Let a signal w 1: T ∈ R dT be the result of u 1: T after local averaging, and ¯ w = T -1 ∑ T t =1 w t ∈ R d . Then, the path length, the norm sum and the energy of w 1: T , including their centered versions, are all dominated by those of u 1: T . That is,

∑ T -1 t =1 ∥ w t +1 -w t ∥ 2 ≤ ∑ T -1 t =1 ∥ u t +1 -u t ∥ 2 ;

∑ T t =1 ∥ w t -¯ w ∥ 2 ≤ ∑ T t =1 ∥ u t -¯ u ∥ 2 ;

Proof of Lemma D.7. Starting from the first part of the lemma, we prove for the general case of 0 < τ < T -k . The boundary cases ( τ = 0 and τ = T -k ) are analogous.

Local averaging only affects the path length caused by the averaged entries u τ +1 , . . . , u τ + k , and the two entries u τ and u τ + k +1 right besides averaging boundary; this original path length quantity in

u 1: T is ∑ k i =0 ∥ u τ + i +1 -u τ + i ∥ 2 . After averaging, the path length among these entries becomes

Now consider the second part of the lemma. After local averaging, ¯ w = ¯ u . The affected part of the signal contributes to the following first order variability

As for the third part of the lemma,

where the last inequality is due to AM-QM inequality.

The final part of the proof is the uncentered version of Part 2 and 3, which follows the same steps. In fact, any fixed reference point (for the variability) works, i.e., for all v ∈ R d ,

We also use another simple lemma.

Lemma D.8. Consider any comparator sequence u 1: T . For all t , we have ∥ u t -¯ u ∥ 2 ≤ P .

Proof of Lemma D.8. Starting from the definition,

and for all i, t ∈ [1 : T ] , ∥ u t -u i ∥ 2 ≤ P due to triangle inequality.

D.7 Quantitative example

This subsection presents details of our two quantitative examples, Example 1 and 2.

Tracking outliers We first calculate the relevant statistics of the comparator u 1: T . Note that we assume k ≤ √ T , and in this way, there is only a small amount of u t with large magnitude, which can then be called outliers.

| ¯ u | = Θ(1) , M = √ T , K = 1 or 2 , P = Θ( T ) , ¯ E ≤ E = kT + T -k = Θ( kT ) . As for S and ¯ S , √

Intuitively, we have | ¯ u | = Θ(1) while M = √ T ; ¯ S = O ( k √ T ) while S = Θ( T ) . This explains the improvements detailed next. For each algorithm considered in Table 1, we evaluate both its switching regret bound and its path-length-dependent bound.

The minimax algorithm ADER [ZLZ18] is not applicable, as M grows with T and can be larger than any fixed diameter D .

The P -dependent bound of the coarse baseline [JC22, Algorithm 6], c.f., Eq.(3), is

With P = O ( KM ) , the resulting K -dependent bound is

The P -dependent bound of the fine baseline [JC22, Algorithm 2], c.f., Eq.(4), is

With P = O ( KM ) , the resulting K -dependent bound is

Our path length bound is

Same for our switching regret bound,

Persistent oscillation Again, we calculate the statistics of the comparator u 1: T . ¯ u = 1 , M ≤ 2 , K = k , P = Θ( k/ √ T ) . Crucially, ¯ S = √ T and ¯ E = Θ(1) , while S = Θ( T ) and E = Θ( T ) . Here the K -dependent bounds of the baselines are loose compared to their corresponding P -dependent bounds, due to using the relation P = O ( KM ) . Therefore we will only evaluate their P -dependent bounds.

Suppose one knows that M ≤ 2 beforehand, then ADER can be applied with D = 2 . The regret bound is

One could check that the P -dependent bounds of the coarse and the fine baselines are also

For our algorithm, the P -dependent bound is

The K -dependent bound is

E Application: Time series forecasting

This section presents an application of our framework in time series forecasting. 15 Roughly speaking, we aim to address the following question:

Given a black box forecaster, can we make it provably robust against (structured) nonstationarity?

Along the way, our objective is to show that

Simultaneously handling unconstrained domains and dynamic comparators in online learning brings downstream benefits in time series forecasting.

Our sparse coding framework can enhance empirically developed forecasting strategies.

Setting Let us consider the following forecasting problem, which resembles the online learning game introduced at the beginning of this paper. The difference is that, here, we further assume access to a black box forecaster A . In each (the t -th) round,

The black box forecaster A produces a prediction a t ∈ R d based on the observed history ( z 1: t -1 and l 1: t -1 ).

After observing a t , we make a prediction x t ∈ R d .

The environment reveals a true value z t ∈ R d and a convex loss function l t : R d → R . l t is G -Lipschitz with respect to ∥·∥ 2 , and z t is one of its minimizer satisfying l t ( z t ) = 0 .

Our goal is to achieve low total loss ∑ T t =1 l t ( x t ) . Since trivially picking x t = a t already achieves a total loss of ∑ T t =1 l t ( a t ) , our goal is to improve it in certain situations, by designing a more sophisticated prediction rule based on a t .

Intuition In the above setting, A can be any algorithm that predicts z 1: T in a reasonable, but non-robust manner. Taking the weather forecasting for example, there are a few notable cases.

A is a simulator of the governing meteorological equations, which uses the online observations z 1: t -1 as boundary conditions.

A is an autoregressive model, which predicts a linear combination of the past observations. The coefficients are determined by statistical modeling.

A is a large deep learning model trained on offline datasets (e.g., the weather history at geographically similar locations).

Even though such forecasters typically lack performance guarantees, their predictions can be used to construct time-varying Bayesian priors (see our discussion in the Introduction): given a t , we will apply a fine-tuning adjustment δ t to predict x t = a t + δ t . Intuitively, the total loss is low if a t is close to the true value z t , i.e., when the prior is good.

Reduction to unconstrained dynamic regret Concretely, if x t = a t + δ t , then due to convexity, for all subgradients g t ∈ ∂l t ( x t ) we have l t ( x t ) -l t ( z t ) ≤ ⟨ g t , δ t ⟩ - ⟨ g t , z t -a t ⟩ . The RHS is the instantaneous regret of δ t in an OLO problem with loss gradient g t and comparator z t -a t . Applying our unconstrained dynamic OLO algorithm, the total loss in forecasting can be bounded as

That is, the total loss bound adapts to the complexity of the error sequence z 1: T -a 1: T (of the given black box forecaster). This contains a 1: T = 0 as a special case, where no side information is assumed.

Let us compare this bound to the baseline ∑ T t =1 l t ( a t ) , which corresponds to trivially picking x t = a t .

If z 1: T = a 1: T , i.e., the black box A is perfect, then the baseline loss is ∑ T t =1 l t ( a t ) = 0 . In this case, due to Theorem 4, our general sparse coding framework guarantees ∑ T t =1 l t ( x t ) ≤ εG , where ε > 0 is an arbitrary hyperparameter. That is, our algorithm is worse than the baseline by at most a constant.

̸

If z 1: T = a 1: T , then in general, the baseline loss ∑ T t =1 l t ( a t ) is linear in T . In contrast, our algorithm could guarantee a sublinear Regret T ( z 1: T -a 1: T ) , thus also a sublinear total loss, when the error sequence z 1: T -a 1: T is structurally simple (e.g., sparse under a transform, or low path length) with respect to our prior knowledge.

In summary, the idea is that by sacrificing at most a constant loss when A is perfect ( z 1: T = a 1: T ), we could robustify A against certain structured unseen environments, improving the linear total loss to a sublinear rate.

Importance of unconstrained domain The above application critically relies on the ability of our algorithm to handle unconstrained domains. To demonstrate this, suppose we instead use the bounded domain algorithm from [ZLZ18] to pick the fine-tuning adjustment δ t . Then, the above analysis only holds if an upper bound D of the maximum error max t ∥ z t -a t ∥ 2 is known a priori - this is a stringent requirement in practice. Furthermore, when z 1: T = a 1: T , such an alternative approach only guarantees ∑ T t =1 l t ( x t ) ≤ ˜ O ( D √ T ) , which is considerably worse than the baseline 0 . In other words, the alternative fine-tuning strategy could ruin the black box forecaster A , when the latter performs well.

In the rest of this section, we present experiments for this time series application. Appendix E.1 demonstrates the power law phenomenon, which shows that both the time series z 1: T and the error sequence z 1: T -a 1: T could exhibit exploitable structures. This implies good performance guarantees using our theoretical framework. Appendix E.2 goes one step further by actually testing the fine-tuning performance of our algorithm.

E.1 Power law phenomenon

This subsection further verifies the power law phenomenon discussed in Section 2.2, with both wavelet and Fourier dictionaries. The goal is to present concrete examples where signal structures can be exploited by our framework, generating more interpretable, sublinear regret bounds.

Wavelet dictionary We first verify the power law on the Haar wavelet dictionary. Intuitively it is suitable when the dynamics of the environment exhibits switching behavior. To this end, consider the following stochastic time series model

where { β t } and { ζ t } are iid random variables satisfying ζ t ∼ Uniform( -q, q ) and

Picking T = 2 15 = 32768 , p = 0 . 0005 and q = 0 . 005 , we generate four sample paths of z 1: T using four arbitrary random seeds (2020, 2021, 2022 and 2023), and the obtained time domain signals are plotted in the first row of Figure 2. As the switching probability p is chosen to be low enough, all the sample paths exhibit a small amount of sharp switches, corrupted by the noise term ζ t . According to our intuition from signal processing, the Haar wavelet transform of these signals is sparse.

Now let us verify this intuition. We take the Haar wavelet transform of these signals, sort the transform domain coefficients and plot the results on log-log scales - these are shown as the solid blue lines in the second row of Figure 2. Using the largest 100 transform domain coefficients on each plot, we fit a liner model using least square, which is shown as the dashed orange line. The slope of each line is -α , where α is displayed in the legend. It can be seen that for all four sample paths, the fitted α is within (0 . 5 , 1) , thus justifying the power law phenomenon [Pri21]. Given α , the regret of our Haar wavelet algorithm is ˜ O ( T 1 . 5 -α ) , as shown in Section 2.2.

As for the implication in time series forecasting, let us consider forecasting z 1: T with a 1: T = 0 , i.e., without the external forecaster A . Given the power law, the total forecasting loss of our fine-tuning approach is ∑ T t =1 l t ( x t ) ≤ ˜ O ( T 1 . 5 -α ) .

We also remark that although only four sample paths are demonstrated, we observe the power law phenomenon on all random seeds we tried in the experiment.

0

5000

10000 15000 20000 25000 30000

Time

1.0

0.5

0.0

0.5

1.0

Signal

0

5000

10000 15000 20000 25000 30000

Time

1.0

0.5

0.0

0.5

1.0

Signal

0

5000

10000 15000 20000 25000 30000

Time

1.0

0.5

0.0

0.5

1.0

Signal

0

5000

10000 15000 20000 25000 30000

Time

1.5

1.0

0.5

0.0

0.5

1.0

1.5

Signal

10 0

10 1

10 2

10 3

10 4

Indices (log)

10

6

10

4

10

2

10 0

10 2

Wavelet coefficients (abs, log)

Data

Fitted,

=0.90

(a) Seed

= 2020

.

10 0

10 1

10 2

10 3

10 4

Indices (log)

10

5

10

3

10

1

10 1

Wavelet coefficients (abs, log)

Data

Fitted,

=0.83

(b) Seed

= 2021

.

10 0

10 1

10 2

10 3

10 4

Indices (log)

10

6

10

4

10

2

10 0

10 2

Wavelet coefficients (abs, log)

Data

Fitted,

=0.89

(c) Seed

= 2022

.

10 0

10 1

10 2

10 3

10 4

Indices (log)

10

6

10

4

10

2

10 0

10 2

Wavelet coefficients (abs, log)

Data

Fitted,

=0.66

(d) Seed

= 2023

.

Fourier dictionary Next, we verify the power law on the Fourier dictionary. Here we use the Jena weather forecasting dataset, 16 which records the weather data at a German city, Jena, every 10 minutes. We take the data from Jan 1st, 2010 till July 1st, 2022, consisting of T = 656956 time steps. Two different modalities, namely the temperature and the humidity, are considered. The actual temperature and humidity sequences are plotted in Figure 3.

2010

2012

2014

2016

2018

2020

2022

Time

20

10

0

10

20

30

40

Temperature (°C)

2010

2012

2014

2016

2018

2020

2022

Time

20

40

60

80

100

Relative humidity (%)

For the sequence of temperature z 1: T , we perform its Discrete Fourier Transform (DFT), which returns T complex number as the frequency domain coefficients. We discard the second half of the coefficients due to symmetry, since the input of the transform is real. For the remaining coefficients, we take their absolute values, sort them and plot the result on a log-log plot. Similar to the wavelet experiment, we also fit a linear model using the largest 100 transform domain coefficients. These are shown as Figure 4 (Left), which exhibit the power law phenomenon.

Furthermore, we perform the same procedure on the temperature difference sequence { z t +1 -z t } , where the t -th entry is the change of temperature from the t -th time step to the t +1 -th time step. The result is shown as Figure 4 (Right). Although the tail is heavier, we can still observe similar power-law phenomenon for large transform domain coefficients.

Now, let us discuss again the implication of the observed power law in time series forecasting. First, consider forecasting z 1: T without A . Given the power law of z 1: T itself, the Fourier version of our forecaster guarantees sublinear total loss. Next, consider forecasting z 1: T with A being the zeroth-order hold forecaster, i.e, a t = z t -1 . The power law of the difference sequence { z t +1 -z t } implies good forecasting performance of our framework in this context.

10 0

10 1

10 2

10 3

10 4

10 5

Indices (log)

10 0

10 2

10 4

10 6

Fourier coefficients (abs, log)

Data

Fitted,

=0.68

10 0

10 1

10 2

10 3

10 4

10 5

Indices (log)

10 0

10 1

10 2

10 3

10 4

Fourier coefficients of diff (abs, log)

Data

Fitted,

=0.69

Parallel results on the humidity sequence are reported in Figure 5, with a similar qualitative behavior. It illustrates the prevalence of the power law across different types of the data.

10 0

10 1

10 2

10 3

10 4

10 5

Indices (log)

10 1

10 3

10 5

10 7

Fourier coefficients (abs, log)

Data

Fitted,

=0.73

10 0

10 1

10 2

10 3

10 4

10 5

Indices (log)

10 0

10 1

10 2

10 3

10 4

10 5

Fourier coefficients of diff (abs, log)

Data

Fitted,

=0.70

E.2 Fine-tuning forecaster

Finally, we test the performance of our forecasting framework on the synthetic switching data and the actual temperature sequence. For the first case, our framework is equipped with the Haar wavelet dictionary. The Fourier dictionary is adopted in the second case. In both cases, we compare our algorithm against the baseline from [JC22]. More specifically, we take our online learning algorithm (Algorithm 2) and the algorithm from [JC22], plug them both into the time series forecasting workflow introduced at the beginning of this section, and then compare their total forecasting loss.

Concretely, let us start from the wavelet dictionary.

Wavelet dictionary In this case, consider the setting without the external forecaster A . We run both online learning algorithms (our Algorithm 2 and the baseline [JC22, Algorithm 2]), and use their outputs directly as the time series predictions. Our Algorithm 2 is equipped with the Haar wavelet dictionary defined in Section 3.1. The configurations of the time series model are the same as the previous subsection, with T = 2 15 = 32768 , p = 0 . 0005 and q = 0 . 005 . The loss functions l t are the absolute loss.

Both algorithms require a confidence hyperparameter ε , and we set it to 1 . Since the time series data Eq.(17) is random, we run both algorithms on 10 random seeds, and calculate their total loss. Our algorithm achieves a total loss of 44048, which is considerably lower than the baseline's total loss 62465. This is consistent with the theoretical results developed so far.

Fourier dictionary Next, we turn to the task of temperature forecasting. The data is reported in the previous subsection. We take its first T = 50000 entries, and assign it to the true time series z 1: T ; the loss functions l t are the absolute loss. The black box forecaster A is assigned to the zeroth-order hold forecaster, i.e., a t = z t -1 .

For our framework, we need to specify the dictionary. Although using the entire DFT matrix could lead to low regret guarantees (as demonstrated by the power law), this is computationally challenging. Instead, we exploit the fact that the weather is naturally periodic, with the period of one day. Picking the base frequency ω accordingly, we define features indexed by k (the harmonic order) as

By specifying the maximum order K , we obtain 2 K features { h t, 2 k -1 , h t, 2 k } k ∈ [1: K ] from this construction. An all-one feature is further added, making the dictionary size N = 2 K +1 .

Again, we set the confidence hyperparameter ε = 1 for our algorithm. The total loss as a function of the dictionary size N is plotted in Figure 6. Notably, the case of N = 0 is equivalent to trivially following the advice of the given forecaster A : x t = a t = z t -1 . It can be seen that our fine-tuning framework ( N > 0 ) actually results in better performance, due to exploiting the structures in the error sequence z 1: T -a 1: T .

0 1

3

5

7

9

11

13

15

17

19

Number of features

7.2

7.4

7.6

7.8

8.0

8.2

Total loss

1e3

We also test the fine-tuning performance of the algorithm from [JC22]. Same as the above, we set ε = 1 . The total loss achieved by this alternative algorithm is 8238, which is around the same as A itself, and significantly higher than the total loss of our algorithm with moderate amount of features ( N > 5 ). This fits the intuition from this paper: the environment contains persistent dynamics, which the algorithm from [JC22] cannot handle.

F Additional discussion

MRA in online learning On a broader scope, wavelets embody the idea of Multi-Resolution Analysis (MRA), which is reminiscent of the classical geometric covering (GC) construction in adaptive online learning [DGSS15]. Such a construction starts from a class of GC time intervals , which are equivalent to the support of Haar wavelet features. On each GC interval, a static online learning algorithm is defined (corresponding to using an all-one feature, c.f., Section 2.2); and then, the outputs of these 'local' algorithms are aggregated by a sleeping expert algorithm on top [LS15, JOWW17]. Algorithmically, our innovation is introducing sign changes in the features, accompanied by a different, additive way to aggregate base algorithms. For tackling nonstationarity, both approaches have their own strengths: the GC construction can produce strongly adaptive guarantees on subintervals of the time horizon, while our algorithm does not need a bounded domain. Their possible connections are intriguing.

Lipschitz vs strongly convex losses We also comment on the choice of loss functions in unconstrained dynamic OCO. Besides the Lipschitz assumption we impose, a fruitful line of works by Baby and Wang [BW19, BW20, BW21, BZW21, BW22] considered an alternative setting with strong convexity, motivated by the prevalence of the square loss in statistics. Their focus is primarily on bounded domains, as [BW19] showed that evaluated under the square loss, a lower bound for the unconstrained dynamic regret is Ω( P 2 ) . A sublinear regret bound here requires P = o ( √ T ) , rather than P = o ( T ) with Lipschitz losses - that is, the environment is required to be 'more static' than the typical requirement in the Lipschitz setting.

Essentially, such a behavior is due to the large penalty that the square loss imposes on outliers. An adversary in online learning can deliberately pick the loss functions such that some of the player's predictions are large outliers with 'huge' (square) losses, while the offline optimal comparator sequence suffers zero losses. Using the Lipschitz losses instead may offer an advantage on unbounded domains, due to being more tolerant to these outliers. Furthermore, Lipschitz losses do not necessarily have minimizers - this is useful for decision problems (as opposed to estimation ), where a ground truth may not exist. 17

Future work For future works, several interesting questions could stem from this paper. For example,

Our regret bound is stated against individual comparator sequences. One could investigate the implication of this result in stochastic environments, where the comparator statistics may take more concrete forms.

Besides the sparsity and the energy studied in this paper, an interesting open problem is investigating alternative complexity measures of the comparator, possibly drawing connections to statistical learning theory.

Our framework builds on pre-defined dictionary inputs. The quantitative benefit of using a datadependent dictionary is unclear.

Beyond wavelets, one may investigate the combination of the sparse coding framework with other function approximators, such as neural networks.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies adaptive online convex optimization, with focus on adapting to unbounded domain and arbitrary time-varying comparator sequence. Different from previous studies which mostly consider the path-length as the prior which appears in the dynamic regret bound, this paper aims to enlarge the range of priors. For a given dictionary of orthogonal feature vectors, it is shown that an $\\tilde{O}(\\sqrt{E\\cdot \\text{Sparsity}_H})$ regret bound can be achieved, which not only recovers previous results on path-length, but also allows more flexibility. This paper also strictly improves the result of JC22 by replacing comparator-dependent terms in the regret bound by strictly smaller ones.\n\nSTRENGTHS:\nThis paper makes solid contributions to adaptive online convex optimization. It extends the scope of comparator adaptivity from the traditional path-length, to other priors on the comparator via a novel sparse coding framework. The achieved bounds involve more refined dependence on the comparator, and are never worse than previous results. In particular, the special case of the wavelet dictionary improves the SotA result in JC22.\n\nWEAKNESSES:\nThe main weakness is the unclear presentation of technical results, which are hard to fully understand and interpret. Though the claimed results seem promising, I have to vote for rejection for now because I can't verify the correctness without a better understanding of the results. Here are some confusions I met.\n\nFor the size 1 dictionary case, isn't the assumption too strong which seems to trivialize Lemma 2.1? In line 211, the comparator is assumed to lie in the span of a single vector $h_{1:T}$, which is just a scaling of the vector $h_{1:T}$. Since the dictionary is fixed and revealed to the player, the range of potential comparators seems very restricted (the degree of freedom is just one through $\\hat{u}$).\n\nThe same issue carries over to the main result Theorem 1, which assumes the signal $z^{(n)}$ lies in the span of a single vector. Even if the reconstruction error creates certain flexibility, to achieve vanishing regret one still needs to guarantee the comparator is very close to $\\sum_{n=1}^N z^{(n)}$, which seems under-expressive: for a given dictionary, the overall degree of freedom is just $n$, while for arbitrary time-varying comparators the degree of freedom is $dT$.\n\nTwo examples are presented right after the main theorem. The case of static regret is well understood. However, I feel the case of orthogonal dictionary requires more technical details, at least to readers unfamiliar with signal processing. For example, what exactly does the \"orthogonal\" means here? (which vector is orthogonal to which?) It would be great if you can provide an intuitive example here to show the power of your method over previous results.\n\nI'm also concerned with how to choose the dictionary. It seems a good choice of dictionary requires prior knowledge. It's not clear to me if one can adapt to a class of dictionaries.\n\nQUESTIONS:\nCan you provide more technical details to address my questions discussed above? In particular, can you provide 1: an example of dictionary that recovers the traditional path-length case, and 2: an example of dictionary that corresponds to some other prior different from path-length?\n\nCan you provide details on using the Fourier dictionary to tackle periodic environments? It looks interesting to me because the naive idea of setting $H_t=(-1)^t I_d$ won't work.\n\nLIMITATIONS:\nThe presentation of technical results is not very clear.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe problem that this paper tries to tackle is the unconstrained online convex optimization with dynamic regret. Previous works usually assume that the comparator sequence is arbitrary and maybe time-varying with some fixed form of comparator measurement in the final dynamic regret bound. For this paper, it proposes a new way of the comparator measurement by first using a pre-defined subspace (by the users) to characterize the comparator sequence and then upper bound the dynamic regret in terms of the pre-defined subspace complexity as well as the reconstruction error between the subspace and the comparator sequence. It shows that for the almost static environment, the proposed algorithm + wavelet constructed subspace could have better regret than the existing works' in (maybe) the constant part.\n\nSTRENGTHS:\n1) It provides the researchers a new perspective of measuring the comparator sequence when upper bounding the dynamic regret, which is interesting and useful in real applications.\n2) The specific wavelet based subspace version could result in a regret better than the existing works' with (maybe) smaller constant part.\n\nWEAKNESSES:\n1) The proposed algorithm framework only moves the dynamic regret from previous works in the constant value part. And in order to compare the actual improvement in terms of the constant part, the author/s need to provide the complete regret result comparison instead of an order O() based one.\n2) The paper result depends heavily on the existing work [MK20], and the subspace based comparator sequence measurement is the only part that makes it interesting, although previous works like [HW15, ZLZ18] has already shown such an idea as pointed out by the author/s.\n3) The author/s explained the motivation of tackling unconstrained dynamic setting rather than the constrained one. But for the finite range argument in the paper, more often than not, the finite range usually comes from the requirement of the model output and not just some heuristic estimation. I think it's better to also provide some examples to demonstrate the motivation of this paper.\n\nQUESTIONS:\n1) The proposed algorithm requires the user to provide the subspace, which is very critical for the final dynamic regret result. If the user provided subspace is not good enough, does that mean the resulted algorithm will have bad performance? If so, how to guarantee the performance?\n2) Since the proposed algorithm depends on the Algorithm 3 a lot, do you think it may make more sense to move Algorithm 3 from the Appendix to the main paper?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors examine the dynamic regret of Online Convex Optimization (OCO) within the context of unbounded comparator sequences. To address this issue, they introduce a novel framework of sparse dictionary coding for online optimization. Following this, the authors provide theoretical proof for the regret bounds applicable to different types of dictionary matrices - the general dictionary matrix, orthogonal dictionary matrix, and the Haar wavelet dictionary matrix. Of notable mention is the result pertaining to the Haar wavelet dictionary matrix, where the authors establish a regret bound that surpasses the current state-of-the-art.\n\nSTRENGTHS:\n- Originality & Significance: The introduction of the sparse coding framework for Online Convex Optimization (OCO), complemented with the corresponding proof, is highly innovative. Furthermore, considering the dynamic regret of OCO in the unbounded domain is very crucial. The result that the authors obtained is related to the comparator average, first-order variability, and second-order variability, rather than the path length. This indicates that the authors have achieved smaller dynamic regret under more adaptive conditions, thereby enriching optimization theory within the community.  \n- Quality & Clarity: The work is clear, well written, and technically sound.\n\nWEAKNESSES:\n- Comparing the proposed method with the meta-expert optimistic online gradient descent method as described in [1] could be beneficial. Particularly in Examples 1 and 2, it appears that the meta-expert optimistic online gradient descent method can also achieve a regret of $\\mathcal O(\\sqrt T)$, considering the path-length is actually $\\mathcal O(\\sqrt T)$. This comparison might offer a broader perspective.\n\n- I'm concerned about the computational complexity of $\\mathcal O(d\\log T)$ for high-dimensional problems, especially when compared to the $\\mathcal O(\\log T)$ complexity of both the meta-expert OGD and the meta-expert optimistic OGD methods. If possible, an in-depth discussion on this issue would be helpful for readers.\n\n- Additionally, the main text seems to miss Algorithms 3-5. This omission may cause a minor confusion for readers during their first read. Rectifying this could help in enhancing the flow and clarity of the paper.\n\nRef: [1] P Zhao et.al., Adaptivity and Non-stationarity: Problem-dependent Dynamic Regret for Online Convex Optimization.\n\nQUESTIONS:\nSee above discussions.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the universal dynamic regret minimization problem with the unconstraint decision domain. The authors proposed the sparsing coding framework, which converts the dynamic regret minimization problem in the time domain into a static regret minimization problem in the transfer domain. The comparator-adaptive static regret bound in the transfer domain implies a dynamic regret bound in the time domain. Specifically, by choosing the dictionary as the Haar wavelet base, this paper achieves improved dynamic regret bound with better dependence on the range of the comparator sequence. Several concrete examples are provided to illustrate the superiority of the proposed bound.\n\nSTRENGTHS:\n+ This paper provides a novel framework to obtain universal dynamic regret bound via sparsing coding. The conversion from comparator-adaptive static regret bound to dynamic regret bound in the time domain is interesting to me.\n+ This paper achieves improved universal dynamic regret bound with the new framework, which is strictly better than the existing results. Several examples are provided to illustrate the advantages of the proposed bound.\n+ This paper is well-written and provides a sufficient discussion of the related literature.\n\nWEAKNESSES:\nI do not find a major weakness in the paper, but there are still some minor comments:\n\n- about the general formulation (Eq. (5)): it would be nice to mention that the dynamic regret bound (Eq.(5)) is not universal dynamic regret bound in general as it only holds for the comparator $u_{1:T}\\in\\mbox{span}(h_{1:T})$. The universal dynamic regret bound can only be achieved with an appropriate choice of the dictionary.\n\n- about the examples: this paper has listed several examples with the specific choice of the comparator sequence to show the superiority of the proposed bound. However, since the main focus of this paper is the universal dynamic regret, it is unclear in which situation the listed comparator sequence is an appropriate benchmark that can minimize the right-hand side of Eq (2). I suggest the authors provide more concrete examples to illustrate the advantage of the proposed bound with certain specific loss functions.\n\nQUESTIONS:\n- could you provide more concrete examples to illustrate the superiority of the proposed bound? (please refer to the second point of the weakness for more details)\n- is it possible to achieve the $\\tilde{O}\\left(\\Vert \\bar{u}\\Vert\\sqrt{T}+\\sqrt{P\\bar{S}}\\right)$ bound without the knowledge of $T$. Could you highlight what is the main difference to obtain such a bound?\n\nLIMITATIONS:\nOne of the main limitations of the paper is that the tightness of the proposed bound is still unclear. The authors discuss this issue at the end of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}