Title: Online Convex Optimisation: The Optimal Switching Regret for all Segmentations Simultaneously
Venue: NeurIPS 2024
Accepted: yes
Keywords: Online Convex Optimisation

Non-stationary Learning
Abstract: We consider the classic problem of online convex optimisation. Whereas the notion of static regret is relevant for stationary problems, the notion of switching regret is more appropriate for non-stationary problems. A switching regret is defined relative to any segmentation of the trial sequence, and is equal to the sum of the static regrets of each segment. In this paper we show that, perhaps surprisingly, we can achieve the asymptotically optimal switching regret on every possible segmentation simultaneously. Our algorithm for doing so is very efficient: having a space and per-trial time complexity that is logarithmic in the time-horizon. Our algorithm also obtains novel bounds on its dynamic regret: being adaptive to variations in the rate of change of the comparator sequence.

The Optimal Switching Regret for all Segmentations

Abstract

We consider the classic problem of online convex optimisation. Whereas the notion of static regret is relevant for stationary problems, the notion of switching regret is more appropriate for non-stationary problems. A switching regret is defined relative to any segmentation of the trial sequence, and is equal to the sum of the static regrets of each segment. In this paper we show that, perhaps surprisingly, we can achieve the asymptotically optimal switching regret on every possible segmentation simultaneously. Our algorithm for doing so is very efficient: having a space and per-trial time complexity that is logarithmic in the time-horizon. Our algorithm also obtains novel bounds on its dynamic regret: being adaptive to variations in the rate of change of the comparator sequence.

1 Introduction

We consider the classic problem of online convex optimisation: a problem with numerous real-world applications. In this problem we have an action set which is a bounded convex subset of some euclidean space. On each trial we select an action from this set and then receive a convex function of bounded gradient, which associates the action with a loss . The aim is to minimise the cumulative loss. The static regret is defined as the cumulative loss of the algorithm minus that of the best constant action in retrospect. It has been shown that the minimax static regret is Θ( √ T ) where T is the time horizon, and that it is achieved by the classic mirror descent family of algorithms [2]. However, in dynamic environments a more sensible notion of regret is the switching regret , which is defined relative to any segmentation of the trial sequence and is equal to the sum of the static regrets over all segments. Clearly, if the segmentation is known a-priori then the minimax switching regret is Θ( ∑ k √ Λ k ) where Λ k is the length of the k -th segment, and it is obtained by running mirror descent independently on each segment. Tracking algorithms , instead, attempt to bound the switching regret on every possible segmentation of the trial sequence simultaneously. However, as far as we are aware, the best such bound until now was O ( ∑ k √ Λ k ln( T )) which is a factor of O ( √ ln( T )) higher than the optimal if we knew the segmentation a-priori. In this paper we (quite remarkably) get rid of this factor: hence obtaining the asymptotically optimal switching regret of O ( ∑ k √ Λ k ) for every possible segmentation simultaneously. Not only is our algorithm optimal, but it is also parameter-free and efficient: having both a space and per-trial time complexity of O (ln( T )) .

Vasilios Mavroudis

In fact, our algorithm RESET is a meta-algorithm which utilises any base algorithm for the online convex optimisation problem at hand. Using online gradient descent [11] as our base algorithm gives us the above O ( ∑ k √ Λ k ) switching regret bound. However, the constant under the O is dependent on the action set and the possible gradients. By choosing a more appropriate base algorithm that is tailored to the specific problem we can achieve lower constant factors. In particular, when faced with the classic problem of prediction with expert advice with N experts, using Hedge [5] as our base algorithm yields the asymptotically optimal O ( ∑ k min(Λ k , √ ln( N )Λ k )) switching regret (a novel result in itself).

We note that although, like strongly adaptive algorithms [4], we are adaptive to heterogeneous segment lengths, we are not necessarily strongly adaptive: in that we do not bound the static regret on any particular segment.

Whilst switching regret models discrete changes in the environment, a continuously changing environment is better modeled by the notion of dynamic regret , which is the difference between the loss of the algorithm and that of any comparator sequence of actions. It is known that algorithms exist which bound the dynamic regret by O ( √ (1 + P ) T ) where P is the path length of the comparator sequence. However, this bound is not adaptive to variations in the rate that the comparator sequence is changing. RESET, with online gradient descent as the base algorithm, rectifies this: improving the dynamic regret to O ( ∑ k √ (1 + P k )Λ k ) for any segmentation in which the path length in the k -th segment is P k . We note that this implies the O ( ∑ k √ Λ k ) bound on switching regret. However, since we are forced to use online gradient descent as our base algorithm here, this result is not strictly more general than our switching regret result.

Related works: Mirror descent was introduced in [2] to find minimisers of convex functions in convex sets. The same algorithm, however, can also be applied to online convex optimisation: the Hedge algorithm of [5] implementing a special case when the convex set is a simplex and the convex functions are linear (the so-called experts problem). The O ( √ T ) static regret of Mirror descent was shown to be optimal in [1]. The work [6] studied the non-stationary case in the experts setting: modifying Hedge to give an algorithm Fixed share which takes a parameter Φ and has a switching regret of O ( √ Φ T ln( T/ Φ)) for any segmentation with Φ segments. One issue with Fixed share, however, is that it does not adapt to heterogeneous segment lengths. In order to remedy this, [4] gave a strongly adaptive algorithm which achieved a static regret of O (ln( T ) √ Λ) on any segment of length Λ . This was improved to O ( √ ln( T )Λ) in [8]. The work [9] took parameters a, b ∈ N and achieved a static regret of O ( √ (1 + ln( b/a ))Λ) for any segment of length Λ ∈ [ a, b ] . However, this still leads to a switching regret of O ( ∑ k √ Λ k ln( T )) for general segmentations. Our work finally achieves the optimal O ( ∑ k √ Λ k ) . The work [11] showed that gradient descent achieves a dynamic regret of O ((1 + P ) √ T ) , which was improved to O ( √ (1 + P ) T ) in [10]. However, prior to both these works, the work [7] obtained the O ( √ (1 + P ) T ) bound subject to an optimal parameter tuning. Our work dramatically improves on this bound by being adaptive to variations in the rate of change of the comparator sequence.

Notation: Let N be the set of natural numbers excluding 0 . Given A ∈ N we define [ A ] := { a ∈ N | a ≤ A } , we define ∆ A := { a ∈ [0 , 1] A | ∑ i ∈ [ A ] a i = 1 } , and we define A N to be the set of natural numbers that are multiples of A . Given a predicate p we define J p K := 0 if p is false and define J p K := 1 if p is true. Given q, s ∈ N with s ≥ q let ⟨ q, s ⟩ := { a ∈ N | q ≤ a ≤ s } .

2 Problem and Results

In this section we introduce the online convex optimisation problem and state the results of this paper. In particular we define and compare the notions of switching and dynamic regret, giving the bounds obtained by our meta-algorithm RESET. Another common notion of regret, not necessarily bounded by RESET, is strongly adaptive regret which we discuss in Section 3.3.

2.1 Online Convex Optimisation

Here we describe the classic problem of online convex optimisation , which our meta-algorithm RESET solves. In this problem we have known bounded convex subsets X , G of some euclidean

space. We define L to be the set of all convex functions that map X into R and whose sub-gradients lie in G . The problem proceeds in T trials. On each trial t ∈ [ T ] the following happens:

We choose some action x t ∈ X .

We receive some loss function ℓ t ∈ L .

Our aim is to minimise the cumulative loss:

Without loss of generality we shall assume that for all t ∈ [ T ] and all x ∈ X we have ℓ t ( x ) ∈ [0 , 1] . This is without loss of generality as both X and the sub-gradients of ℓ t are bounded and our algorithm RESET, when using mirror descent as the base algorithm, is invariant to any constant addition to any loss function. Also, without loss of generality, assume that T is an integer power of two.

An example of online convex optimisation is prediction with expert advice . Here we have some N ∈ N and a set of N experts : where on each trial each expert is associated with a loss in [0 , 1] . On each trial we must select an expert (incurring the loss associated with that expert) and then observe the vector of losses for that trial. For this problem we choose X := ∆ N and G := [0 , 1] N . On each trial t we draw our expert from the probability vector x t and define the loss function ℓ t to be the linear function such that for all i ∈ [ N ] we have that ℓ t ( e i ) (that is, the loss of the i -th basis element of R N ) is the loss associated with expert i on trial t . Note that ℓ t ( x t ) is our expected loss on trial t .

2.2 Switching Regret

We now define the notion of switching regret . A segment is any set of the form ⟨ q, s ⟩ for q, s ∈ [ T ] with s ≥ q . The static regret with respect to such a segment I is defined as:

which is the total loss incurred on the segment minus that which would have been obtained by always choosing the best constant action in retrospect. A segmentation S is defined as any partition of [ T ] into segments. The switching regret with respect to such a segmentation S is defined as:

which is the sum of the static regrets on each segment of S . Note that R † ( S ) is the total loss of the algorithm minus that which would have been obtained by the best sequence of actions which is constant over each segment of S . The following theorem establishes a lower bound on the switching regret with respect to any fixed segmentation, even in the special case in which all the loss functions are linear:

Theorem 2.1. For any segmentation S and any algorithm for the online convex optimisation problem, there exists (except in trivial cases) a sequence:

where the constant under the Ω is dependent only on X and G .

Proof. Apply the static regret lower bound of [1] to each segment independently.

In this paper we develop an algorithm RESET which has an upper-bound that matches this lower bound for every possible segmentation S simultaneously. RESET utilises any algorithm (called the base algorithm ) for the online convex optimisation problem at hand. The base algorithm must take a parameter Λ ∈ [ T ] and guarantee that R ([Λ]) ∈ O ( √ Λ) if it were used directly. We note that online gradient descent [11] is always one such possibility. Computationally, to use online gradient descent, we must be able to compute subgradients of the loss functions and euclidean projections into the set X . The following theorem bounds the switching regret of RESET.

of linear functions, in which:

Theorem 2.2. Suppose γ ∈ R is such that for all Λ ∈ [ T ] , when the base algorithm is run with parameter Λ , it is guaranteed that: √

Then, for any segmentation S , RESET achieves a switching regret of:

where:

Proof. See Section 4.

Clearly, theorems 2.1 and 2.2 show that, for any fixed pair ( X , G ) , RESET has the asymptotically optimal switching regret for every segmentation simultaneously. However, our result is stronger. For example, take the problem of prediction with expert advice defined above. Here the HEDGE algorithm attains:

which is shown to be optimal via (the proofs of) theorems 3.6 and 3.7 of [3] and by noting that we can always force O (Λ) regret if Λ ≤ ln( N ) . Although there is a slight technicality here when segments have length less than ln( N ) , the proof of Theorem 2.2 still works in exactly the same way to show that (by applying RESET to HEDGE) we have the asymptotically optimal switching regret for every value of N and every segmentation simultaneously.

RESET is also very efficient, as shown in the following theorem.

Theorem 2.3. Given that the base algorithm runs in a time of ξ per trial and requires a space of ξ ′ , RESET has a per-trial time complexity of O ( ξ ln( T )) and space complexity of O ( ξ ′ ln( T )) .

Proof. Immediate from the RESET algorithm.

2.3 Dynamic Regret

Switching regret measures the performance of the algorithm against the best comparator sequence of actions that is constant in each segment. Dynamic regret, on the other hand, measures the performance of the algorithm against any comparator sequence. Specifically, given any sequence of actions:

then the dynamic regret with respect to E is defined as:

To bound the dynamic regret of RESET we introduce the following notion of path length . Specifically, given the above sequence E and a segment I , the path length of E in the segment I is defined as:

The current state of the art for dynamic regret is the algorithm ADER [10] which achieves a dynamic regret of:

In this paper we significantly improve on this result, as shown in the following theorem.

Theorem 2.4. When using online gradient descent [11] as the base algorithm, RESET achieves, for any comparator sequence E and any segmentation S , a dynamic regret of:

where the constant under the O is dependent only on X and G .

Note that, for any segmentation, the dynamic regret bound of RESET is asymptotically equal to that of running ADER on each segment independently. To achieve this with ADER one would need to know the specific segmentation a-priori. As for the switching regret, RESET achieves this for every segmentation simultaneously. We note that our bound is a significant improvement on that of ADER since it is adaptive to variation in the rate of change of the comparator sequence.

We note that, given a segmentation S , the switching regret R † ( S ) is equal to the maximum dynamic regret R ∗ ( E ) across all sequences E that are constant in each segment of S . For all I ∈ S , the fact that such an E is constant on I implies that the path length P ( E , I ) is in O (1) . This means that Theorem 2.4 implies the switching regret bound of Theorem 2.2 up to a constant factor (dependent on γ, X and G ). However, to obtain Theorem 2.4 we must use online gradient descent as our base algorithm. For prediction with expert advice, for example, online gradient descent is not asymptotically optimal for every value of N simultaneously. Hence, when considering switching regret only, Theorem 2.2 is a stronger result.

3 The Algorithm

In this section we describe our meta-algorithm RESET ( Re cursion over Se gment T ree). We first introduce the notation that we will use to describe the base algorithm.

3.1 The Base Algorithm

We now define the notation that we use to describe our base algorithm. The base algorithm utilises a data-structure D (which contains the parameter) and is composed of the following three subrountines:

Given Λ ∈ [ T ] , the subroutine INITIALISE (Λ) returns the initial data-structure with parameter Λ .

At the start of a trial, given the current data-structure D , the subroutine QUERY ( D ) returns the output action of the base algorithm for that trial.

At the end of a trial t , given the current data-structure D and the loss function ℓ t , the subroutine UPDATE ( D , ℓ t ) returns the updated data-structure (ready for the next trial).

The assumption in Theorem 2.2 implies the following. Suppose we have trials q, s ∈ [ T ] with s ≥ q , and on each trial t ∈ ⟨ s, q ⟩ we do the following:

If t = s then D t ← INITIALISE ( q -s +1) .

w t ← QUERY ( D t ) .

D t +1 ← UPDATE ( D t , ℓ t ) .

Then we have:

3.2 RESET

We now introduce our meta-algorithm RESET. First let τ := log 2 ( T ) and define the function ψ : R × R × R × N → R by:

The pseudocode of RESET is given in Algorithm 1.

We now describe RESET. We have a set of τ +1 levels , where each level i ∈ [ τ ] ∪ { 0 } hosts an instance of the base algorithm with parameter 2 i and which will reset every 2 i trials. On each trial t ,

Algorithm 1 RESET

we denote the data-structure of the base algorithm associated with level i by D i t . Each level also has an associated number in [0 , 1] called the mixing weight . On each trial t , we denote the mixing weight associated with level i by µ i t . We note that the mixing weight µ 0 t is not necessary.

We now describe the creation of the action x t on trial t . Note first that each level i ∈ [ τ ] ∪{ 0 } has an associated action w i t which is defined as the output of QUERY ( D i t ) , so is the action selected by the base algorithm for level i on trial t . We call these actions the base actions . The action x t is created by the following recursive process. For each level i in order we construct an action z i t called the propagating action . This action is constructed by a convex combination of the preceding propagating action z i -1 t and the base action w i t . Specifically, we start by setting:

and then, for all levels i ∈ [ τ ] , once z i -1 t has been constructed we set:

Finally, we output:

We now turn to the update at the end of trial t . For all levels i ∈ [ τ ] ∪ { 0 } we have the following two cases.

If t ∈ 2 i N then we set:

so that the mixing weight and instance of the base algorithm hosted by level i are reset. Note that the parameter of the base algorithm is 2 i . This is since it is reset every 2 i trials.

On the other hand, if t / ∈ 2 i N then we set:

Note that the update of the mixing weight is based on the losses of w i t and z i -1 t . If z i -1 t has a higher loss than w i t , in that the base action performs better than the lower-level propagating action, then the mixing weight increases. This means that the weight of the base action, in the convex combination forming the propagating action of level i , increases. If w i t has a higher loss than z i -1 t then the opposite happens.

Figure 1 illustrates RESET.

0

1

2

3

0

1

2

3

1

2

3

4

5

6

7

8

Left: The generation of the base actions and mixing weights. Purple numbers denote levels and black numbers denote trials. Each segment (rectangle) in the figure runs an instance of the base algorithm (generating the base actions). The mixing weights for each level reset at the start of each segment. Mixing weight updates are dependent on the segment length.

Right: The computation of the action x t on trial t . Purple numbers denote levels. Blue balls denote base actions, red balls denote mixing weights, and black balls denote propagating actions. The final black arrow is the output x t .

3.3 Comparison to Strongly Adaptive Online Learner

RESET has some similarities to the SAOL algorithm of [4]. Unlike RESET, SAOL is strongly adaptive , in that it bounds the static regret on any segment. Specifically, for any segment I , SAOL achieves:

This, however, leads to a switching regret bound that is a factor O (ln( T )) off the optimal. This additional factor was improved to O ( √ ln( T )) by [8].

Like RESET, SAOL has τ +1 levels and utilises a base algorithm which constructs, for every trial t and level i , a base action w i t in exactly the same way as RESET. It then generates, for each trial t , the final action x t as a convex combination of the base actions. We note that in RESET, the action x t is also a convex combination of the base actions, where for each level i ∈ [ τ ] , the coefficient of w i t is equal to:

The crucial difference between SAOL and RESET is that, whilst SAOL updates each coefficient in the convex combination directly, the coefficients in RESET are updated by updating each mixing weight directly. It is due to this, and the particular way that the mixing weights are updated, that RESET attains, unlike SAOL, the optimal switching regret.

4 Analysis

Here we prove Theorem 2.2. We will show how to modify this proof in order to prove Theorem 2.4 at the end of this section. All lemmas stated in this section are proved in Appendix A.

Choose any segmentation S . Let Φ := |S| and for all k ∈ [Φ + 1] define σ k such that the k -th segment in S is ⟨ σ k , σ k +1 -1 ⟩ . We define a comparator sequence ⟨ ϵ t | t ∈ [ T ] ⟩ as follows. For all k ∈ [Φ] define the action:

and then, for all t ∈ ⟨ σ k , σ k +1 -1 ⟩ , define ϵ t := ˜ ϵ k . Note that:

In addition let α := 2 √ ln(2) / ( √ 2 -1) . With these definitions in hand we now begin the analysis.

4.1 Hedge

The updates for our mixing weights follow the classic algorithm HEDGE [5]. In particular, for each level i ∈ [ τ ] we maintain an instance of HEDGE (which restarts every 2 i trials) with two experts . On each trial t , the weight of the first expert is µ i t and the weight of the second is 1 -µ i t . The loss of the first expert is ℓ t ( w i t ) and the loss of the second is ℓ t ( z i -1 t ) . The purpose of the function ψ is then to update the weights according to the HEDGE algorithm. The following lemma is a classic result about HEDGE.

Lemma 4.1. Given trials q, s ∈ [ T ] with q ≤ s , and a sequence ⟨ ( a t , b t ) | t ∈ ⟨ q, s ⟩⟩ such that for all t ∈ ⟨ q, s ⟩ we have a t , b t ∈ [0 , 1] , and a sequence ⟨ ρ t | t ∈ ⟨ q, s ⟩⟩ defined recursively such that for all t ∈ ⟨ q, s -1 ⟩ we have:

then we have:

4.2 The Segment Tree

In this subsection we define the segment tree , which is the geometrical structure that our analysis is based on. The segment tree is a full, balanced, binary tree B whose leaves are the elements of [ T ] in order from left to right. Given any internal vertex v ∈ B , let ◁ ( v ) and ▷ ( v ) be its left and right child respectively. Given any vertex v ∈ B , let ◀ ( v ) and ▶ ( v ) be its left-most and right-most descendent respectively (noting that these are both elements of [ T ] ). Let r be the root of B . Given a vertex v ∈ B \ { r } , let ↑ ( v ) be the parent of v . Given a vertex v ∈ B , let h ( v ) be equal to the height of v (that is, the height of the tree B minus the depth of v , so that leaves have height 0 ).

Each vertex v ∈ B represents the segment ⟨ ◀ ( v ) , ▶ ( v ) ⟩ . i.e. Each vertex represents a segment in the left hand side of Figure 1 (when t = 8 ). We call a vertex v ∈ B stationary iff there exists k ∈ [Φ] with σ k ≤ ◀ ( v ) and ▶ ( v ) < σ k +1 . Let H be the set of all stationary vertices. We call a vertex v ∈ B fundamental iff both:

v ∈ H .

v = r or ↑ ( v ) / ∈ H .

Let F be the set of all fundamental vertices. We call a vertex v ∈ B relevant iff it is an ancestor of a fundamental vertex. Let A be the set of all relevant vertices. For all relevant vertices v ∈ A we define Q ( v ) to be the set of descendants of v that are contained in F .

We have, from the algorithm, the following lemma about the vertices of the segment tree:

Lemma 4.2. Given any vertex v ∈ B we have:

and:

and for all t ∈ [ ◀ ( v ) , ▶ ( v ) -1] we have:

Note that this lemma shows that for all v ∈ B we run a single instance of both the base algorithm and HEDGE over the segment ⟨ ◀ ( v ) , ▶ ( v ) ⟩ , as illustrated in Figure 1.

4.3 The Recursive Equations

We now derive the recursive equations that our analysis is based on. First note that for all v ∈ F there exists u ∈ X such that ϵ t = u for all t ∈ ⟨ ◀ ( v ) , ▶ ( v ) ⟩ . Hence, Lemma 4.2 and Equation (1), and the fact that w h ( v ) t is the output of QUERY ( D h ( v ) t ) , lead to the following lemma.

Lemma 4.3. For all v ∈ F we have:

Note that, from the algorithm and the convexity of the loss functions, we have, for all t ∈ [ T ] and all v ∈ B with h ( v ) = 0 , that:

̸

So, by lemmas 4.2 and 4.1, we have the following lemma.

̸

Lemma 4.4. For all vertices v ∈ B with h ( v ) = 0 we have:

Noting that z 0 t = w 0 t for all t ∈ [ T ] , lemmas 4.3 and 4.4 immediately imply the following recursive equations. For all v ∈ F we have:

and for all v ∈ A \ F we have:

4.4 Performing the Recursion

We now utilise equations (3) and (4) to perform the recursion. Specifically, we have the following inductive hypothesis for vertices in A , which is proved by induction up the tree B from the vertices in F to the root. The reason it holds for vertices in F comes direct from Equation (3). For a vertex in A\F , once the inductive hypothesis has been shown to hold for both its children, it is then shown to hold for the vertex itself by Equation (4). The inductive hypothesis is given in the following lemma.

Lemma 4.5. For all v ∈ A we have:

In particular, this inductive hypothesis holds for v = r . By Equation (2) this gives us the following.

Lemma 4.6. We have:

We now have a bound on the switching regret. It is, however, not yet written in terms of the segment lengths. To write it in terms of the segment lengths we partition F into a sequence of sets ⟨F k | k ∈ [Φ] ⟩ such that, for all k ∈ [Φ] , we define F k to be equal to the set of all v ∈ F such that σ k ≤ ◀ ( v ) and ▶ ( v ) < σ k +1 . We now have the following lemma.

Lemma 4.7. For all k ∈ [Φ] we have:

Combining lemmas 4.6 and 4.7 gives us:

as required. This completes the proof of Theorem 2.2.

4.5 Dynamic Regret Analysis

We now prove Theorem 2.4. Let the base algorithm be online gradient descent as in [11]. Take any segmentation S ∗ and any comparator sequence E . Let Ψ = |S ∗ | and for all j ∈ [Ψ] let I j be the j -th segment in S ∗ . For all j ∈ [Ψ] , note that we can choose a natural number:

and a partition S ′ j of I j into N j segments such that for all I ∈ S ′ j we have:

S

:=

⋃

∈

j

[Ψ]

Note that Equation (5) implies that for all I ∈ S we have:

We now modify the analysis of the switching regret as follows. In the analysis of the switching regret we defined a comparator sequence ⟨ ϵ t | t ∈ [ T ] ⟩ . In this analysis we instead define this comparator sequence as equal to E . Using the segmentation S , construct the segment tree and the sets A and F as in the analysis of the switching regret. Since our base algorithm is gradient descent we have, direct from [11], the following lemma.

Lemma 4.8. Suppose we have trials q, s ∈ [ T ] with s ≥ q , and on each trial t ∈ ⟨ s, q ⟩ we do the following:

If t = s then D t ← INITIALISE ( q -s +1) .

w t ← QUERY ( D t ) .

D t +1 ← UPDATE ( D t , ℓ t ) .

Then we have:

This lemma, along with Lemma 4.2 and Equation (6), gives us the following.

Lemma 4.9. For all v ∈ F we have:

This lemma is essentially identical to, and will be used instead of, Lemma 4.3. Following the rest of the analysis of the switching regret gives us:

Note that for all j ∈ [Ψ] we have |S ′ j | = N j and hence:

so we have proved Theorem 2.4.

References

Jacob D. Abernethy, Alekh Agarwal, Peter L. Bartlett, and Alexander Rakhlin. A stochastic view of optimal regret through minimax duality. Conference on Learning Theory , 2009.

Charles E. Blair. Problem complexity and method efficiency in optimization (a. s. nemirovsky and d. b. yudin). Siam Review , 27:264-265, 1985.

Nicolò Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. 2006.

Amit Daniely, Alon Gonen, and Shai Shalev-Shwartz. Strongly adaptive online learning. International Conference on Machine Learning , 2015.

Yoav Freund and Robert E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. In European Conference on Computational Learning Theory , 1997.

Mark Herbster and Manfred K. Warmuth. Tracking the best expert. Machine Learning , 32:151178, 1995.

Mark Herbster and Manfred K. Warmuth. Tracking the best linear predictor. J. Mach. Learn. Res. , 1:281-309, 2001.

Kwang-Sung Jun, Francesco Orabona, Stephen J. Wright, and Rebecca M. Willett. Improved strongly adaptive online learning using coin betting. In International Conference on Artificial Intelligence and Statistics , 2016.

Yuanyu Wan, TU Wei-Wei, and Lijun Zhang. Strongly adaptive online learning over partial intervals. Science China Information Sciences , 65, 2022.

Lijun Zhang, Shiyin Lu, and Zhi-Hua Zhou. Adaptive online learning in dynamic environments. Neural Information Processing Systems , 2018.

Martin A. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In International Conference on Machine Learning , 2003.

A Proofs

Here we prove, in order, all the lemmas in the analysis.

A.1 Lemma 4.1

Direct from [5] using only two experts, were, on each trial t , we have that:

The loss of the first expert is a t and the loss of the second is b t .

The weight of the first expert is ρ t and the weight of the second is 1 -ρ t .

A.2 Lemma 4.2

The equality:

comes directly from the definition of h ( v ) . We also have that:

and for all t ∈ [ ◀ ( v ) , ▶ ( v ) -1] we have:

Hence, by the algorithm, the lemma holds.

A.3 Lemma 4.3

Note first that, since w h ( v ) t is the output of QUERY ( D h ( v ) t ) , Lemma 4.2 and Equation (1) immediately give us:

Since v ∈ F we have that v ∈ H so there exists k ∈ [Φ] with σ k ≤ ◀ ( v ) and ▶ ( v ) < σ k +1 . This implies that for all t ∈ ⟨ ◀ ( v ) , ▶ ( v ) ⟩ we have t ∈ ⟨ σ k , σ k +1 -1 ⟩ so that ϵ t := ˜ ϵ k . Hence, we have:

Substituting into Equation (7) gives us the result.

A.4 Lemma 4.4

Consider Lemma 4.1. In this lemma choose q := ◀ ( v ) and s := ▶ ( v ) . For all t ∈ ⟨ q, s ⟩ choose:

Then by Lemma 4.2 we have, for all t ∈ [ q, s ] , that:

so that, by the algorithm and the convexity of the loss functions we have, for all t ∈ ⟨ q, s ⟩ , that:

and hence, by Lemma 4.1, we have:

The result then follows from the fact that, by Lemma 4.2, we have s -q +1 = 2 h ( v ) .

A.5 Lemma 4.5

For k ∈ N ∪ { 0 } we define:

so our inductive hypothesis is that for all v ∈ A we have:

If v ∈ F then, by the definition of F , we have Q ( v ) = { v } so the inductive hypothesis holds by Equation (3). Now suppose we have some ˜ v ∈ A \ F . Note that, by the definition of F , we have ◁ (˜ v ) , ▷ (˜ v ) ∈ A so all that is left to prove the inductive hypothesis is to prove that, if the inductive hypothesis holds for both v = ◁ (˜ v ) and v = ▷ (˜ v ) , then it also holds for v = ˜ v . So suppose that the inductive hypothesis holds for v = ◁ (˜ v ) and v = ▷ (˜ v ) . First note that:

and, by the definition of F (and a simple induction up the tree B from the vertices in F ), we have:

so that:

Equations (4), (8) and (9), imply:

Applying the inductive hypothesis to the terms:

and noting that Q (˜ v ) = Q ( ◁ (˜ v )) ∪ Q ( ▷ (˜ v )) and for all q ∈ Q (˜ v ) we have:

shows the inductive hypothesis holds for v = ˜ v . We have hence proved that the inductive hypothesis holds for all v ∈ A .

A.6 Lemma 4.6

For all q ∈ Q ( r ) we have:

so, since r ∈ A and Q ( r ) = F , Lemma 4.5 gives us:

Since ◀ ( r ) = 1 , ▶ ( r ) = T and, by the algorithm, z h ( r ) t = z τ t = x t for all t ∈ [ T ] , we then have, by Equation (2), that:

as required.

A.7 Lemma 4.7

First let:

We take the inductive hypothesis that for all non-empty finite sets Z ⊆ N ∪ { 0 } we have:

and we prove by induction on |Z| . In the case that |Z| = 1 we have Z = { i } for some i ∈ N ∪ { 0 } and hence:

so the inductive hypothesis holds for |Z| = 1 . Now suppose we have some j ∈ N and that the inductive hypothesis holds for |Z| = j . We now show that it holds for |Z| = j +1 which will prove that the inductive hypothesis holds always. Specifically, let i := max Z , let Z ′ := Z \ { i } and let i ′ := max Z ′ . Define:

Note that:

so that y < 1 and hence:

since the term on the left is monotonic decreasing with y . This implies that:

so that, by the inductive hypothesis (applied to the set Z ′ ), we have:

so the inductive hypothesis holds for |Z| = j +1 . We have hence proved that the inductive hypothesis holds always.

Now note that, by the definition of F , we have that the set {⟨ ◀ ( v ) , ▶ ( v ) ⟩ | v ∈ F k } is a partition of ⟨ σ k , σ k +1 -1 ⟩ so that, by Lemma 4.2, we have:

Assume, for contradiction, that there exists three distinct vertices v, v ′ , v ′′ ∈ F k with h ( v ) = h ( v ′ ) = h ( v ′′ ) . Without loss of generality assume that v and v ′′ are the leftmost and rightmost of the three vertices respectively. Also without loss of generality assume that v ′ is the right child of its parent ↑ ( v ′ ) . Then v must either be equal to or lie to the left of the left child of ↑ ( v ′ ) and hence:

and so that ↑ ( v ′ ) ∈ H . But this contradicts the fact that v ′ ∈ F .

We have hence shown that for any i ∈ [ τ ] ∪ { 0 } there are at most two distinct vertices v, v ′ ∈ F k with h ( v ) = h ( v ′ ) = i . This means that we can partition F k into two disjoint sets U k and V k such that for all i ∈ [ τ ] ∪ { 0 } there exists at most one element v of U k with h ( v ) = i and at most one element v ′ of V k with h ( v ′ ) = i . By Equation (10) we then have:

so that:

Noting that for all y, y ′ > 0 we have √ y + √ y ′ ≤ √ 2 √ y + y ′ and c = ξ √ 2 , the above inequality, along with Equation (11), gives us:

as required.

A.8 Lemma 4.8

Direct from [11].

A.9 Lemma 4.9

By lemmas 4.2 and 4.8 we immediately have that:

Since v ∈ F we have that there exists k ∈ [Φ] such that ◀ ( v ) ≥ σ k and ▶ ( v ) < σ k +1 . Hence, by Equation (6) , we have that:

Substituting into Equation (12) gives us the result.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [NA]

Justification: Our algorithm always achieves the stated bounds on regret and time-complexity and (when using the doubling trick) is parameter free.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theorems are either referenced or proved.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper conforms to the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work is theoretical in nature and we cannot foresee any societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This work is theoretical in nature and we cannot foresee any risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of tracking regret (aka. switching cost). The previous start-of-the-art result has a $\\ln T$ dependence on the final regret bound. This paper claims to remove the extra $\\ln T$ (more specifically, an $\\sqrt{\\ln T}$ factor) and achieve the optimal switching regret.\n\nSTRENGTHS:\nThe solution is clear and intuitive. I checked the whole proof and it seems to be correct. The combination way of $z_t^i = \\mu_t^i w_t^i + (1 - \\mu_t^i) z_t^{i-1}$ is impressive as it is different from the traditional weighted combination of $z_t^i = \\sum_i \\mu_t^i w_t^i$. This novel combination method has proven to play an important role in achieving the optimal switching regret. However, Neither the reason why this weighted combination method is able to achieve the desired optimal switching regret nor the comparison with the traditional combination method are explained in the paper. I suggest that the authors could revise the paper to add corresponding discussions.\n\nWEAKNESSES:\nI found the solution impressive. However, the current presentation of the paper is not satisfying enough. As stated in the above part, the reason why this weighted combination method is able to achieve the desired optimal switching regret and the comparison with the traditional combination method are explained in the paper, which are thought to be necessary from my point of view. I believe that the proofs are correct and the method is of importance. However, I am inclined that the current presentation does not meet the acceptance bar of NeurIPS and thus another round of polishment and review would be better.\n\nQUESTIONS:\nThis paper can be further improved by considering the following minor issues:\n1. The investigation and citations of the literature are insufficient. For example, in Line 18, when mirror descent is mentioned for the first time, there should be a citation, but not and only listing the related citations in the related work part.\n2. Some writings are not consistent with standard practices. After the equation in Line 85, there should be a ',' followed by 'which' (not the capital 'Which' in the current version).\n3. I believe that Theorem 2.1 exists in the literature as it is pretty straightforward and simple (by using the lower bound for static regret to attain a lower bound for the switching regret). However, there is no citation on this theorem. I suggest that the authors conduct a more careful check of the literature to refine this part.\n4. The idea of 'multiple levels' in the proposed method is highly similar to the geometric covering proposed in [1], but without necessary discussions about the similarities or differences between them (I believe that the idea is the same). If they are similar, there should be a corresponding citation. The current presentation claims this idea to be one of the contributions of this work, which is misleading. Otherwise, if they are indeed different, I suggest that the authors could add a corresponding discussion in the next version.\n5. The Section 'Analysis' is highly mathematical with little intuition. This section is more like a proof which should be put in the appendix. In the main paper, there should be more discussions about the proof sketch of the analysis, the reason why this paper uses such an algorithm, etc. I suggest that the authors could carefully polish the paper to enhance its readability. \n6. The citations are somewhat inaccurate. For example, [3] (the index in the paper) is actually published at ICML 2015, but the current citation leads to arXiv. And [9] is actually published in NIPS 2018, but the current citation leads to arXiv.\n7. The last part of dynamic regret is a little confusing. Do the authors mean that under the condition that the path length on each segment is bounded by a constant, a switching-regret-type dynamic regret can be achieved? If so, it does not make much sense because under such a strong condition, each segment is nearly stationary, and the dynamic regret degenerates to switching regret in this case.\n\nReferences:\n1. Strongly adaptive online learning, ICML 2015\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and, if applicable, potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides an algorithm which achieves a strongly-adaptive*\nswitching regret guarantee, removing the logarithmic penalty\ntypically associated with efficient algorithms achieving strongly-adaptive guarantees.\n\n--- \n*Update after rebuttal: The results are only for switching regret, *not* strongly-adaptive switching regret guarantees\n\nSTRENGTHS:\nIf the result is correct, it seems like it could be a breakthrough result,\nremoving a logarithmic penalty from strongly-adaptive guarantees that\nseemed to be necessary.\n\nWEAKNESSES:\nIt seems like the segment tree would still require $O(T)$ memory, would it not?\nWhereas existing approaches achieve $O(\\log(T))$ per-round memory and computation.\nSo it seems like the most interestin\nSo if the result is correct, perhaps the takeaway is that the log(T) penalty\nis the cost of reducing the memory consumption of the algorithm.\n\nThe dynamic regret result doesn't seem particularly convincing since it only holds for\ncomparator sequences which do not move much, ie, for sequences that are essentially\njust switching sequences. In fact it seems strange that any interesting\ndynamic regret result could follow from a switching regret one, since the\noptimal switching regret follows from the optimal $\\sqrt{P_T T}$ dynamic regret\nby throwing away information; if the comparator only changes at times\n$\\sigma_1,\\ldots,\\sigma_{S+1}$ then the path-length bound is $\\sum_t \\|u_t-u_{t-1}\\|=\\sum_i \\|u_{\\sigma_i}-u_{\\sigma_{i+1}}\\|\\le S$ for $u\\in\\Delta_{d-1}$\n\n\n\nThe presentation is hard to follow. This work would also probably benefit from having diagrams\ndemonstrating the segment tree in the notation given here.\n\nQUESTIONS:\n- The approach seems very similar in spirit to the classic geometric covering intervals approach,\n  which can also be seen as breaking up [1,T] according to a particular\n  partition tree (see e.g. \"Partition Tree Weighting\" Veness et al. 2013).\n  Could you explain what exactly the segment tree approach is doing to\n  improve over the geometric covering intervals approach to avoid the log(T)\n  penalty?\n- Section 3.3 and Theorem 18 of Chen et al. (2021) (\"Impossible Tuning Made Possible: A New Expert Algorithm and its Applications\")\n  seems to imply that a strongly-adaptive switching regret guarantee is not even possible in the first place.\n  Could you comment on how this does not contradict the results presented here? Is this work leveraging some\n  additional assumption?\n- The result seems to forego all other types of adaptivity, and just accepts\n  penalties of $\\sqrt{b-a}$ on each interval $[b-a]$ instead of adapting to the\n  losses of the experts. Do you think it is possible to get some form of\n  adaptivity to the loss sequence as well, or is this a fundamental limitation of the approach?\n\nLIMITATIONS:\nThe limitations are largely unaddressed. It's clear that some compromises are made to achieve this result,\nbut it's still very unclear to me what exactly they are, and these should have been laid out more clearly in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies online convex optimisation under a non-stationary environment. The authors focus on the switching regret, which evaluates the regret on every possible segment of the trials. This paper demonstrates that the additional logarithmic factor $O(\\log T)$ shown in previous results can be improved, and the authors obtain an algorithm with an optimal guarantee.\n\nSTRENGTHS:\nThis paper studies online convex optimisation under more challenging scenarios, and focuses on an important theoretical question: whether the additional logarithmic factor can be improved.\n\nWEAKNESSES:\n- As a theoretical paper that claims to make fundamental improvements, it would be beneficial for the descriptions of theorems to be more precise, rigorous, and comprehensive. For example, I would expect the authors to clarify under which assumptions ($\\mathcal{X}$, loss functions) the theorems hold.\n- The terminologies used in this paper are ambiguous. The term \"switching regret\" or \"tracking regret\" might often be used in a prediction with experts' advice setting, which could be treated as a degenerated version of dynamic regret, a measure considered widely for general convex optimization. The authors do not introduce the term \"adaptive regret\" [1, 2], which seems to be the same optimization objective in this paper, to my understanding. In [2], it is demonstrated that adaptive regret is stronger than tracking regret (or switching regret).\n- There lacks a clear statement of the algorithm. It might improve the readability if the authors could explain the main idea and the key technique straightforwardly about how to achieve the optimal results.\n- It would be better if the authors could provide more discussions about related works. See details in 'Questions'.\n\n1. E. Hazan and C. Seshadhri. Efﬁcient learning algorithms for changing environments.\n2. A. Daniely, A. Gonen, and S. Shalev-Shwartz. Strongly adaptive online learning.\n\nQUESTIONS:\n- Does Theorem 4.1 hold for any bounded convex set and Lipschitz convex loss functions? What is the meaning of $\\gamma$ in Theorem 4.1?\n- In Theorem 4 of [3], it is shown that the adaptive regret guarantee can recover the dynamic regret guarantee. Could the author explain the difference between the results achieved in Section 5 and those?\n- What is the relationship between the proposed segment tree and the geometric covering [2]?\n- At Line 144, it seems that new algorithms are initialized at some specific trials. The analysis of such an algorithm can be reduced to analyzing the sleeping expert problem [3], which usually requires the aggregating algorithm to have specific properties for minimizing general convex functions. Does the aggregating algorithm used in this paper have such properties?\n- At Line 189, this paper analyzes the aggregating step in a black-box manner by using Theorem 3.2. The aggregating algorithm might be sensitive to the decision point at time $q$ in order to provide guarantees on any interval starting at time $q$. How does the proof avoid the analysis of this issue?\n\n3. H. Luo and R. E. Schapire. Achieving all with no parameters: Adanormalhedge.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of switching and dynamic regret in online convex optimization (OCO). Given any segmentation of $[T]$, the switching regret is equal to the sum of static regrets on each segment. For the switching regret, the best-known bound obtained by prior works was $\\mathcal{O}(\\sum_{k}\\sqrt{\\Lambda_{k} \\ln T})$, where $\\Lambda_{k}$ is the length of the $k$-th segment. However, the lower bound is $\\Omega(\\sum_{k} \\sqrt{\\Lambda_{k}})$, which follows by independently applying the lower bound on the static regret of OCO with convex functions, on each segment. The current paper removes the extra $\\sqrt{\\ln T}$ factor by proposing an efficient algorithm RESET (Recursion over Segment Tree), which has a running time $\\mathcal{O}(\\ln T)$ per iteration (disregarding the cost associated with treating the feasible set). \n\nFor the dynamic regret, it is known from Zhang et al. that a dynamic regret bound of $\\mathcal{O}(\\sqrt{\\mathcal{P}T})$, where $\\mathcal{P}$ denotes the path-length of the comparator sequence, is possible. However, the bound is not sensitive to variation in the path-length. The current work improves upon this by showing that RESET achieves $\\mathcal{O}(\\sum_{k} \\sqrt{\\Lambda_k})$ dynamic regret for any segmentation that has $\\mathcal{O}(1)$ path length on each segment. When the segments have varying lengths, this bound is significantly better.\n\nSTRENGTHS:\nThe considered problem is theoretically interesting and the presented algorithm is quite simple with a clean analysis. The writing is to the point. The idea of having a segment tree is natural since segment trees are known to handle interval updates, e.g. sum-query updates over intervals in an array efficiently. The subroutines used by RESET are Mirror Descent and Hedge. The algorithm can be viewed as updates over the (level, time) space, i.e. at each time $t$ we update the parameters $w_{t} ^ {i}, \\mu_{t} ^ {i}, z_{t} ^ {i}$ by Mirror Descent, Hedge, a convex combination (with parameter $\\mu_{t} ^ i$) of $z_{t}^{i - 1}$ and $w_{t} ^ {i}$. However, the analysis proceeds by translating these updates to a recursion defined over the nodes of a segment tree; the recursion is obtained by the updates and the regret guarantees of Mirror Descent and Hedge. The dynamic regret guarantee follows straightforwardly from the analysis of switching regret.\n\nWEAKNESSES:\nI do not see any immediate weakness regarding the contribution of the work. Although the presentation of the paper is good, there is a scope for improvement. I suggest the authors use appropriate punctuation marks to end/continue equations and sentences.\n\nMinor Typo: Definition of Bregman Divergence in Section 3.1.\n\nQUESTIONS:\nLines 183--184 are not very clear to me. If I imagine a full, balanced tree with 15 nodes ($T = 8$) with all levels except the leaf nodes being filled, I do not see how 183 holds when $v$ is the parent of the node with the number $7$. Is that a typo?\n\nLIMITATIONS:\nYes, the authors have adequately addressed this.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}