Title: Online Non-convex Learning in Dynamic Environments
Venue: NeurIPS 2024
Accepted: yes
Keywords: Online Learning

Non-convexity

Dynamic Environments
Abstract: This paper considers the problem of online learning with non-convex loss functions in dynamic environments. Recently, Suggala and Netrapalli [2020] demonstrated that follow the perturbed leader (FTPL) can achieve optimal regret for non-convex losses, but their results are limited to static environments. In this research, we examine dynamic environments and choose \emph{dynamic regret} and \emph{adaptive regret} to measure the performance. First, we propose an algorithm named FTPL-D by restarting FTPL periodically and establish $O(T^\frac{2}{3}(V_T+1)^\frac{1}{3})$ dynamic regret with the prior knowledge of $V_T$, which is the variation of loss functions. In the case that $V_T$ is unknown, we run multiple FTPL-D with different restarting parameters as experts and use a meta-algorithm to track the best one on the fly. To address the challenge of non-convexity, we utilize randomized sampling in the process of tracking experts. Next, we present a novel algorithm called FTPL-A that dynamically maintains a group of FTPL experts and combines them with an advanced meta-algorithm to obtain $O(\sqrt{\tau\log{T}})$ adaptive regret for any interval of length $\tau$. Moreover, we demonstrate that FTPL-A also attains an $\tilde{O}(T^\frac{2}{3}(V_T+1)^\frac{1}{3})$ dynamic regret bound. Finally, we discuss the application to online constrained meta-learning and conduct experiments to verify the effectiveness of our methods.

Online Non-convex Learning in Dynamic Environments

Abstract

This paper considers the problem of online learning with non-convex loss functions in dynamic environments. Recently, Suggala and Netrapalli [2020] demonstrated that follow the perturbed leader (FTPL) can achieve optimal regret for non-convex losses, but their results are limited to static environments. In this research, we examine dynamic environments and choose dynamic regret and adaptive regret to measure the performance. First, we propose an algorithm named FTPL-D by restarting FTPL periodically and establish O ( T 2 3 ( V T +1) 1 3 ) dynamic regret with the prior knowledge of V T , which is the variation of loss functions. In the case that V T is unknown, we run multiple FTPL-D with different restarting parameters as experts and use a meta-algorithm to track the best one on the fly. To address the challenge of non-convexity, we utilize randomized sampling in the process of tracking experts. Next, we present a novel algorithm called FTPL-A that dynamically maintains a group of FTPL experts and combines them with an advanced meta-algorithm to obtain O ( √ τ log T ) adaptive regret for any interval of length τ . Moreover, we demonstrate that FTPL-A also attains an ˜ O ( T 2 3 ( V T +1) 1 3 ) dynamic regret bound. Finally, we discuss the application to online constrained meta-learning and conduct experiments to verify the effectiveness of our methods.

1 Introduction

Online learning is a powerful model for sequential decision-making tasks, supported by wellestablished theoretical guarantees [Cesa-Bianchi and Lugosi, 2006, Orabona, 2019]. It can be regarded as a repeated game between a learner and an adversary. In each round t , the learner first selects a decision x t ∈ K , where K is a candidate set. Then a loss function f t ( · ): K → R is revealed and the learner suffers a loss f t ( x t ) . The goal of the learner is to minimize the cumulative loss ∑ T t =1 f t ( x t ) over all rounds. Traditionally, the performance measure is static regret or simply regret

defined as the difference between the cumulative loss of the online learner and that of the optimal decision chosen in hindsight. During the past decades, numerous algorithms and theories have been developed to minimize static regret. An extensively researched setting is online convex optimization (OCO) [Shalev-Shwartz et al., 2012], in which the losses are assumed to be convex and Lipschitzcontinuous. For OCO, many efficient methods such as online gradient descent (OGD) attain O ( √ T ) regret [Zinkevich, 2003], which is known to be minimax optimal [Abernethy et al., 2008].

On the other hand, when the losses are non-convex, minimizing regret becomes computationally challenging. This is because minimizing regret implies optimization, and general non-convex optimization is known to be NP-hard. In light of this computational barrier, some recent works examine the notion of local regret instead [Hazan et al., 2017, Aydore et al., 2019, Hallak et al., 2021, Guan et al., 2023]. However, these researches focus on finding local optima and hence do not guarantee vanishing regret that grows sublinear in T . To ensure vanishing regret, another class of studies assumes access to a sampling oracle [Krichene et al., 2015, Yang et al., 2018, Héliou et al., 2020] or an offline optimization oracle [Agarwal et al., 2019, Suggala and Netrapalli, 2020]. Particularly, Suggala and Netrapalli [2020] have demonstrated that for general non-convex and Lipschitz-continuous losses, FTPL can achieve O ( √ T ) regret, which matches the optimal result in the convex setting. However, their study solely focuses on static environments, as it uses regret as the only metric to measure the performance. In dynamic environments, the distribution of loss functions may change over time, causing a shift in the optimal decision. In this scenario, the static regret in (1) is not a suitable measure since the comparator is fixed [Zhang, 2020, Cesa-Bianchi and Orabona, 2021]. To overcome this limitation, studies in OCO have introduced new performance metrics: dynamic regret [Zinkevich, 2003, Zhang et al., 2018a] and adaptive regret [Hazan and Seshadhri, 2007, Daniely et al., 2015].

In dynamic regret [Zinkevich, 2003], the learner is compared against a sequence of local minimizers:

where x ∗ t ∈ arg min x ∈K f t ( x ) is a minimizer of f t ( x ) over domain K . It is recognized that in the worst-case scenario, achieving sublinear dynamic regret is impossible unless we introduce certain constraints on the comparator sequence or the function sequence [Jadbabaie et al., 2015]. One such example is the functional variation defined below

If the value of V T is known in advance, Besbes et al. [2015] showed that a restarted OGD achieves O ( T 2 3 ( V T +1) 1 3 ) dynamic regret for convex functions.

Strongly adaptive regret [Daniely et al., 2015] is another widely used performance metric in dynamic environments. It is defined as the maximum static regret over any interval of length τ :

Given the dynamic nature of environments, where the optimal decisions can vary across intervals, minimizing static regret over any interval of length τ is essentially competing against changing comparator. For convex functions, the best known result for strongly adaptive regret is O ( √ τ log T ) [Jun et al., 2017].

In this paper, we consider minimizing dynamic regret and adaptive regret in the non-convex setting. For dynamic regret minimization, we first propose an algorithm named FTPL-D by restarting FTPL periodically and establish an O ( T 2 3 ( V T + 1) 1 3 ) dynamic regret bound, which requires the prior knowledge of V T to set the optimal restarting frequency. To get rid of this limitation, we then propose the second algorithm named FTPL-D+ by running multiple instances of FTPL-D with different restarting frequencies and combining them with a meta-algorithm to track the best one. Given that the loss functions are non-convex, deterministic algorithms cannot achieve a vanishing regret [Suggala and Netrapalli, 2020]. Hence, we choose Hedge [Cesa-Bianchi and Lugosi, 2006] with randomized sampling as our meta-algorithm. Correspondingly, in each round, we sample one expert based on their weights instead of computing the weighted average of all experts. We prove that FTPL-D+ enjoys the same order of dynamic regret bound as that of FTPL-D without the prior knowledge of V T .

For adaptive regret minimization, we propose an algorithm named FTPL-A, where we construct a set of intervals dynamically and instantiate an expert of FTPL to minimize the static regret for every interval. To combine these experts whose numbers vary at different time steps, we use AdaNormalHedge [Luo and Schapire, 2015] that can deal with sleeping experts as our meta-algorithm

and again employ randomized sampling in it to select experts. We prove that FTPL-A achieves an O ( √ τ log T ) adaptive regret bound for any interval of length τ . Besides, we demonstrate that our FTPL-A also obtains an ˜ O ( T 2 3 ( V T +1) 1 3 ) dynamic regret bound, indicating its effectiveness in minimizing dynamic regret and adaptive regret simultaneously. In brief, our dynamic regret and strongly adaptive regret for non-convex loss functions are on the same order as those for convex functions. We compare our results and previous ones in Table 1.

Moreover, we discuss the application of our methods to online constrained meta-learning [Xu and Zhu, 2023] and conduct experiments. The empirical results demonstrate the effectiveness of our methods in dynamic regret and adaptive regret minimization. We highlight the main contributions of this paper below.

For dynamic regret minimization, we first propose a novel algorithm named FTPL-D and establish O ( T 2 3 ( V T +1) 1 3 ) dynamic regret. To eliminate the dependence on prior knowledge of V T , we then propose FTPL-D+ and provide a dynamic regret bound of the same order.

For adaptive regret minimization, we develop a novel algorithm named FTPL-A and establish an O ( √ τ log T ) strongly adaptive regret bound. Moreover, we prove that FTPL-A also ensures an ˜ O ( T 2 3 ( V T +1) 1 3 ) dynamic regret bound.

We discuss the application to online constrained meta-learning and conduct experiments to verify the effectiveness of our methods.

2 Related Work

In this section, we give a brief introduction to previous works in OCO and online non-convex learning. More related works are reviewed in Appendix B.

Online Convex Optimization The existing works in OCO mostly focus on static regret. For instance, OGD [Zinkevich, 2003] and follow the regularized leader [Hazan et al., 2016] both achieve an O ( √ T ) regret bound. The O ( √ T ) regret bound for Lipschitz-convex functions is known to be minimax optimal [Abernethy et al., 2008]. To cope with dynamic environments, Zinkevich [2003] introduced the dynamic regret in (2), and there are numerous studies dedicated to the worst-case scenario because of its mathematical tractability [Besbes et al., 2015, Jadbabaie et al., 2015, Mokhtari et al., 2016, Yang et al., 2016, Zhang et al., 2017, Baby and Wang, 2019, Zhao and Zhang, 2021, Wan et al., 2023]. As mentioned earlier, dynamic regret is often bounded in terms of certain regularities of the comparator sequence or the function sequence. In particular, Besbes et al. [2015] proposed the functional variation in (3) to evaluate the movement of loss functions. They equipped restarted OGD with an O ( T 2 3 ( V T +1) 1 3 ) dynamic regret bound, but require to know V T beforehand.

Besides dynamic regret, another metric in dynamic environments is adaptive regret. Adaptive regret has been examined under the setting of prediction with expert advice (PEA) [Littlestone and Warmuth, 1994, Freund et al., 1997, Gyorgy et al., 2012, Adamskiy et al., 2012, Luo and Schapire, 2015] and OCO [Hazan and Seshadhri, 2007, Daniely et al., 2015, Jun et al., 2017, Zhang et al., 2019, 2021, Yang et al., 2024]. In the following, we focus on the latter one. To minimize (4), Daniely et al. [2015] developed a meta-algorithm called strongly adaptive online learner and used it to design

two-layer structured online algorithms, which enjoy an O ( √ τ log T ) strongly adaptive regret bound for convex functions. Later, Jun et al. [2017] proposed a novel meta-algorithm and improved the strongly adaptive regret bound to O ( √ τ log T ) .

Online Non-convex Learning To avoid the NP-hardness of non-convex optimization, Hazan et al. [2017] proposed a computational tractable notion of local regret and developed algorithms that attain the optimal local regret bound efficiently. Hallak et al. [2021] extended the local regret minimization framework to non-smooth settings. Guan et al. [2023] further examined the cases where a limited number of gradient oracles or value oracles are available. Although these techniques can efficiently minimize local regret, they do not guarantee to find the global optima and achieve vanishing regret.

Another line of work still focuses on the notion of static regret, but assumes access to a sampling oracle or an offline optimization oracle. Assuming access to a sampling oracle, Krichene et al. [2015] proved that the Hedge algorithm is capable of achieving O ( √ T log T ) regret over a specific feasible set. Yang et al. [2018] later improved the regret bound to O ( √ T ) by partitioning the feasible set with a layered structure and using a novel weighting method. Besides, Héliou et al. [2020] examined the dual averaging (DA) algorithm with an imperfect value-feedback model of the loss function, which also attained an O ( √ T ) regret bound. However, these algorithms rely on a sampling oracle on a continuum that is computationally intractable .

Under the hypothesis that the algorithm has access to an offline optimization oracle, Agarwal et al. [2019] showed that FTPL achieves O ( T 2 3 ) regret for general non-convex loss functions with Lipschitz continuousity. In the same setting, Suggala and Netrapalli [2020] improved the regret bound to O ( √ T ) . It should be noted that assuming access to an offline optimization oracle is reasonable since some simple algorithms such as stochastic gradient descent, are able to find approximate global optima quickly, even for non-convex objective functions.

By contrast, the studies on dynamic environments are limited. Aydore et al. [2019] introduced a variant of local regret for dynamic environments and proposed a novel algorithm to minimize it. Héliou et al. [2020] obtained O ( T 2 3 ( V T +1) 1 3 ) dynamic regret with their imperfect feedback method. However, it needs the prior knowledge of V T to choose the optimal stepsize in DA and still relies on a computationally intractable sampling oracle.

3 Preliminaries

In this section, we introduce the problem setting and FTPL [Suggala and Netrapalli, 2020].

Assumption 1. The feasible set K is bounded and has glyph[lscript] ∞ -diameter at most D , i.e., for all x , y ∈ K , ‖ x -y ‖ ∞ ≤ D .

Assumption 2. The sequence of loss functions f t ( · ) are L -Lipschitz with respect to glyph[lscript] 1 -norm, i.e., for all x , y ∈ K , | f t ( x ) -f t ( y ) | ≤ L ‖ x -y ‖ 1 .

We proceed to introduce FTPL in Algorithm 1, which is the subroutine of our algorithms. FTPL relies on the offline optimization oracle below [Suggala and Netrapalli, 2020].

Definition 1. An offline optimization oracle takes input as a function f ( · ) : K → R and a d -dimensional vector σ , and returns an approximate minimizer of x ↦→ f ( x ) -〈 σ, x 〉 . An optimization oracle is called ' ( α, β ) -approximate optimization oracle' if it returns x ∗ ∈ K such that

We denote such an optimization oracle with O α,β ( f i -σ ) .

Given the access to an ( α, β ) -approximate optimization oracle, the main idea of FTPL is to add a small perturbation to the cumulative loss and follow the 'perturbed' leader:

where σ t ∈ R d is a random perturbation such that σ t,j , the j -th coordinate of σ t , is sampled from the exponential distribution with parameter η , that is

Algorithm 1 Follow the Perturbed Leader (FTPL)

1: Input: feasible set K , approximation optimization oracle O α,β , parameter of exponential distribution η

2: for t = 1 to T do

3: Generate random vector σ t by (6)

4: Predict x t according to (5)

5: Observe loss function f t ( · )

6: end for

Algorithm 2 FTPL-D

1: Input: feasible set K , length of interval γ , approximation optimization oracle O α,β , parameter of exponential distribution η

2: for t = 1 to T do

3: Compute s γ = glyph[floorleft] ( t -1) /γ glyph[floorright] ∗ γ +1

4: Generate random vector σ t by (6)

5: Predict x t according to (7)

6: Observe loss function f t ( · )

7: end for

Remark . We choose FTPL as our subroutine due to its simplicity and reasonable assumption of the optimization oracle. However, it is worth noting that our methods are not limited to FTPL alone. We can choose any algorithm that has a static regret guarantee in online non-convex learning and extend it to dynamic environments.

4 Online Non-convex Learning with Dynamic Regret

In this section, we propose our first algorithm named FTPL-D based on FTPL and establish a dynamic regret bound with the prior knowledge of V T . After that, we propose the second algorithm named FTPL-D+ which is equipped with the same bound without the prior knowledge of V T .

4.1 Follow the Perturbed Leader with Dynamic Regret

Let dynamic regret defined in (2) be the performance measure. Our goal is to bound the dynamic regret by the functional variation V T .

Following the work of Besbes et al. [2015], we apply the restarting strategy to FTPL. The key idea is to partition the time horizon T into consecutive intervals, where the length of each interval is controlled by a parameter γ , and then restart FTPL at the beginning of each interval. Our algorithm, which we call follow the perturbed leader with dynamic regret (FTPL-D), is described in Algorithm 2. At each time step t , we mark the beginning of the current interval as s γ in Step 3. Then FTPL-D follows the perturbed leader within the current interval, which spans from s γ to t . Specifically,

The following theorem presents the dynamic regret of Algorithm 2 for general non-convex functions. √

Theorem 1. Under Assumptions 1 and 2, and setting η = 1 / dγ , Algorithm 2 ensures

where

If the value of V T is known, by choosing γ = min {⌊ ( T V T ) 2 3 ⌋ , T } , we have

Algorithm 3 FTPL-D+

1: Input: feasible set K , H = { γ 1 , ..., γ N } , step size ρ

2: Activate a set of experts { E i | γ i ∈ H} by invoking Algorithm 2 for each γ i ∈ H

3: For each expert E i , set w i 1 = 1 /N for i ∈ [ N ]

4: for t = 1 to T do

5: Receive x i t from each expert E i

6: Draw x t according to P ( x t = x i t ) = w i t

7: Output x t and observe loss function f t ( · )

8: Update the weight of experts according to (9)

9: Send loss function f t ( · ) to each expert E i

10: end for

Remark . Theorem 1 demonstrates that for general non-convex functions, when α = O (1 / √ T ) and β = O (1 /T ) , which mirrors the settings used by Suggala and Netrapalli [2020, Page 4], our FTPL-D achieves an O ( T 2 3 ( V T + 1) 1 3 ) dynamic regret bound that matches the existing bound for convex functions [Besbes et al., 2015].

4.2 FTPL-D+

The dynamic regret in Theorem 1 requires the prior knowledge of V T to choose the optimal parameter γ . However, in practice, V T is usually unknown, posing a challenge in selecting the best γ . Studies on OCO also encounter the problem of searching for the optimal parameter for their algorithms [van Erven and Koolen, 2016, Zhang et al., 2018a, Wan et al., 2021, 2024]. The main idea in their solutions is activating multiple instances of their algorithms as experts, and tracking the best one with a meta-algorithm such as Hedge [Cesa-Bianchi and Lugosi, 2006], which assigns a weight to each expert and computes the weighted average of their advice.

Inspired by the above idea, we choose FTPL-D as the expert algorithm and propose our second algorithm called improved FTPL-D (FTPL-D+). Note that both the loss functions and the feasible set are non-convex, so we cannot combine different experts' predictions by the weighted average. According to the discussion of Cesa-Bianchi and Lugosi [2006, Chapter 4.1], we can tackle this issue by randomized sampling, where we sample one expert according to the weights, and then output the prediction of that expert.

The details of FTPL-D+ are shown in Algorithm 3, and we describe the main steps below. We first define a set H = { γ 1 , ..., γ N } of N values for parameter γ . Then we activate a set of experts { E i | γ i ∈ H} by invoking Algorithm 2 as E i = FTPL-D ( K , γ i , O α,β , η ) . For each expert E i , we denote its weight at round t as w i t , which is initiated as w i 1 = 1 /N . At each round t , we receive a prediction x i t from the expert E i . To utilize these predictions { x i t | i ∈ [ N ] } , we select x i t with the probability w i t and submit it as the output x t , i.e., P ( x t = x i t ) = w i t . After the loss function is revealed, we update the weights according to the following rule [Cesa-Bianchi and Lugosi, 2006]:

where ρ > 0 is the step size. We present the dynamic regret of Algorithm 3 in the following theorem.

Theorem 2. Let H = { γ i = 2 i | i = 1 , · · · N } where N = glyph[floorleft] log 2 T glyph[floorright] , and ρ = 1 dDL √ 8 ln N T . Under Assumptions 1 and 2, Algorithm 3 ensures

Remark . Theorem 2 shows that our FTPL-D+ achieves the same order of dynamic regret bound as Algorithm 2 without the prior knowledge of V T .

t

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

...

I

0

[

]

[

]

[

]

[

]

[

]

[

]

[

]

[

]

[

]

[

]

[

]

[

]

[

]

[

]

[

]

[

]

[

]

[

]

[

]

[

]

...

I

1

[

]

[

]

[

]

[

]

[

]

[

]

[

]

[

]

[

]

[

...

I

2

[

]

[

]

[

]

[

]

[

...

I

3

[

]

[

...

I

4

[

...

Algorithm 4 FTPL-A

1: for t = 1 to T do 2: for I ∈ C t do 3: Create an expert E I which runs FTPL with η = 1 / √ d | I | 4: For the expert E I , set R t -1 ,I = C t -1 ,I = 0 5: Add expert E S I to the set of active experts A t 6: end for 7: Remove all experts who end at the round t from A t 8: Receive the predictions x t,I of each expert E I ∈ A t and calculate its weight w t,I by (10) 9: Draw x t according to P ( x t = x t,I ) = w t,I 10: Output x t and observe loss function f t ( · ) 11: Compute the instantaneous expected loss defined in (11) 12: For each expert E I ∈ A t , update R t,I and C t,I by (12) 13: Send loss function f t ( · ) to each expert E I ∈ A t 14: end for

5 Online Non-convex Learning with Adaptive Regret

In this section, we further consider strongly adaptive regret defined in (4) as the performance metric in dynamic environments. We develop our third algorithm, named follow the perturbed leader with adaptive regret (FTPL-A), and establish its adaptive and dynamic regret bounds.

Previous research on adaptive regret [Hazan and Seshadhri, 2007, Daniely et al., 2015, Wang et al., 2024] has adopted a two-layered framework to design adaptive online algorithms. The basic idea is to dynamically construct a set of intervals and run an expert algorithm to minimize the static regret within each interval. These experts are then combined by a meta-algorithm. Building upon this idea, our FTPL-A includes three components: an expert algorithm, a set of intervals, and a meta-algorithm. In the following, we illustrate them separately.

First, we use FTPL as the expert algorithm, which enjoys O ( √ | I | ) static regret for a given interval I [Suggala and Netrapalli, 2020]. Then, following the work of Daniely et al. [2015], we build geometric covering (GC) intervals shown in Fig. 2:

That is, each I k is a partition of N \{ 1 , . . . , 2 k -1 } into consecutive intervals of length 2 k . For each interval I ∈ I , we activate an instance of FTPL as the expert E I to minimize the regret over I .

Next, we choose AdaNormalHedge [Luo and Schapire, 2015] to track the best expert. Similar to the Hedge forecaster we used in FTPL-D+, we maintain a weight for each expert, but in a specific form of a potential function:

where [ x ] + = max(0 , x ) and Φ(0 , 0) is defined to be 1, and the weight function is determined by

Due to the non-convexity of the loss functions, as explained in Section 4.2, we use randomized sampling to generate the output.

The procedure of FTPL-A is summarized in Algorithm 4. We describe the main steps below. For brevity, we denote the set of intervals starting from round t as C t , and the set of active experts at round t as A t . In Step 3, for each interval I ∈ C t , we run an instance of FTPL as E I with the optimal parameters. In Step 4, the two variables R t -1 ,I and C t -1 ,I are initialized as 0 for E I , where R t -1 ,I denotes the expected regret of E I up to round t -1 , and C t -1 ,I denotes the expected absolute regret. In Steps 5 and 7, the active expert set A t is maintained by adding new experts and removing expired ones. In Step 8, the expert's weight is calculated by [Luo and Schapire, 2015]:

Similarly, we sample one expert (prediction) according to the weights in Step 9, and submit it as the output in Step 10. In Step 11, the instantaneous expected loss is computed as the weighted average of different experts' losses:

where x t,I denotes the prediction of expert E I at round t . In Step 12, we update R t,I and C t,I by

The following theorem demonstrates the strongly adaptive regret bound of Algorithm 4 for general non-convex loss functions.

Theorem 3. Under Assumptions 1 and 2, Algorithm 4 ensures

Remark . Theorem 3 demonstrates that when α = O (1 / τ ) and β = O (1 /τ ) , FTPL-A is strongly adaptive with the same order of O ( √ τ log T ) regret as that for convex functions [Jun et al., 2017].

Furthermore, we notice that Zhang et al. [2018b] investigated the relationship between adaptive regret and dynamic regret. They showed that dynamic regret can be bounded by the strongly adaptive regret and the functional variation. According to their findings, we have the following theorem that presents the dynamic regret bound of Algorithm 4.

Theorem 4. Under Assumptions 1 and 2, Algorithm 4 ensures

Remark . Theorem 4 shows that FTPL-A achieves nearly the same dynamic regret as FTPL-D+, indicating its ability to minimize adaptive and dynamic regret simultaneously. However, compared to FTPL-A, FTPL-D+ does not require the construction of GC intervals and uses a simpler metaalgorithm, making it easier to implement.

6 Application to Online Constrained Meta-Learning

Online non-convex learning offers a wide range of applications [Neel et al., 2020, Vietri et al., 2020, Ghai et al., 2021, Castiglioni et al., 2022]. By applying our methods, we can extend these applications to dynamic environments. In this section, we discuss the application to online constrained meta-learning [Xu and Zhu, 2023] and conduct experiments to support our theoretical results.

Online constrained meta-learning. Meta-learning, also known as learning-to-learn, focuses on acquiring a prior meta-parameter that enables fast adaptation to new tasks. Recently, Xu and Zhu [2023] proposed the setting of online constrained meta-learning, which aims to learn the metaparameter from a sequence of constrained learning tasks {T 1 , . . . , T T } . Each task T t is characterized by its data distributions D t and constraint limits c t . At each round t , a training dataset D tr t , which contains data sampled i.i.d. from D t , is available to the learner. Then the learner adapts the metaparameter φ t to the task-specific parameter θ t by a within-task algorithm A lg with D tr t and c t . After

0

50

100

150

200

Round (task index)

0

50

100

Instantaneous loss

FTPL

FTPL-A

FTPL-D+

0

50

100

150

200

Round (task index)

0

2

4

Cumulative loss

×10 3

FTPL

FTPL-A

FTPL-D+

0

50

100

150

200

Round (task index)

0

20

40

60

Instantaneous loss

FTPL

FTPL-A

FTPL-D+

0

50

100

150

200

Round (task index)

0

1

2

Cumulative loss

×10 3

FTPL

FTPL-A

FTPL-D+

the learner deploys θ t , it obtains a validation dataset D val t by sampling data from D t and defines the meta-objective function at current round as

where L ( θ, D val t ) denotes the loss of θ on D val t . Xu and Zhu [2023] updated the meta-parameter by using FTPL to the non-convex meta-objective function on all revealed tasks. With this approach, they effectively controlled the static regret of the meta-objective function between φ t and the fixed optimal meta-parameter φ ∗ . However, when the distribution of tasks is shifting, the fixed φ ∗ may not be adapted to each task well. In such a scenario, it is better to use dynamic regret or adaptive regret as the performance metric, since they measures the learner's performance against the changing optimal meta-parameter. By applying Algorithms 3 and 4, we can minimize the two metrics efficiently. To evaluate our methods, we conduct experiments on a sequence of tasks in imitation learning and another in few-shot learning, respectively.

6.1 Experiments on meta-imitation learning

Setup. We use the demonstration data given by Huang et al. [2019] and set the total number of tasks T = 200 . At each round t ∈ [ T ] , a human expert writes a different letter in a free space without obstacle. The learner can observe a demonstration of the letter and is asked to write the same letter in a cluttered environment. We provide the details of problem formulation and implement setting in Appendix C.1. By controlling the letter to be imitated for each task, we simulate two types of dynamic environments: (i) abruptly changing environments; (ii) gradually evolving environments. In (i), we split the time horizon evenly into 4 stages, and set the imitation target as capital letters "M", "E", "T" and "A" for the 4 stages. In this way, the optimal meta-parameter φ ∗ drifts notably every 50 round. In (ii), we choose the capital letter "A" as the imitation target. Additionally, we rotate the letter "A" at a small random angle δ t ∈ (0 , 0 . 05] in each round. This ensures the optimal meta-parameter φ ∗ undergoes slow and smooth shifts. We compare our FTPL-D+ and FTPL-A with FTPL [Suggala and Netrapalli, 2020] in the above two scenarios.

Results. We repeat the experiments five times with different random seeds and plot the loss (mean and standard deviation) in Fig. 3. More results are provided in Appendix C.1. For abruptly changing environments, Fig. 3(a) shows that, in comparison to FTPL, our methods adapt more quickly to the new task distribution after the demonstration shifts, which occurs at T = 50 , 100 , and 150 . Fig. 3(b) demonstrates that our methods perform significantly better than FTPL in terms of cumulative loss. For gradually evolving environments, it can be observed from Fig. 3(c) that as the angle of rotation increases, both our methods maintain low loss, while FTPL exhibits a significant increase in instantaneous loss after T = 100 . Fig. 3(d) also shows a notable advantage in cumulative loss. Moreover, our FTPL-D+ and FTPL-A achieve comparable performance in the two types of dynamic environments.

6.2 Experiments on few-shot image classification with robustness

Setup. We conduct the experiments on CUB-200-2011 (referred to as CUB) dataset [Wah et al., 2011], which includes 200 fine-grained categories of birds. There are T = 150 of robust 5-way 5-shot image classification tasks, each task containing images from 5 classes with only 5 training samples per class. At each round t ∈ [ T ] , the model is updated using a few data samples and is required to have high accuracy on both clean and perturbed test data. We provide the details of problem

0

50

100

150

Round (task index)

0.40

0.45

0.50

0.55

0.60

Clean accuracy

FTPL

FTPL-A

FTPL-D+

(a) Clean accuracy

0

50

100

150

Round (task index)

0.30

0.35

0.40

0.45

0.50

0.55

PGD accuracy

FTPL

FTPL-A

FTPL-D+

(b) PGD accuracy

0

50

100

150

Round (task index)

0.30

0.35

0.40

0.45

0.50

0.55

B-score

FTPL

FTPL-A

FTPL-D+

(c) B-score

formulation and implement setting in Appendix C.2. We resize the input images to 84 × 84, and apply the same data augmentation as in Ye et al. [2021] and Xu and Zhu [2023]. A four-layer convolutional neural network (Conv-4) is employed as the backbone, comprising four blocks. Each block consists of a convolutional layer with 64 kernels of size 3 × 3, stride 1, and zero padding, followed by a batch normalization layer, a ReLU activation function, and lastly a 2 × 2 max-pooling layer. Following the convolutional layers, the network uses a fully connected linear layer with 5 neurons as a classifier to output the prediction for the input image. To simulate real-world dynamic environments, we select three groups of bird categories from the CUB dataset based on their habitats: water birds, forest birds, and grassland birds. We then split the time horizon T evenly into 3 stages, within each stage the tasks are sampled from one group of bird categories. This simulates abruptly changing environments where the optimal meta-parameter φ ∗ drifts every 50 round. We compare our FTPL-D+ and FTPL-A with FTPL [Suggala and Netrapalli, 2020].

Metrics and results. The performance of the task-specific model is evaluated by: (i) clean accuracy; (ii) PGD accuracy; (iii) B-score. The clean accuracy is the accuracy on the clean test dataset, and PGD accuracy is the accuracy on the corrupted test dataset, which is obtained by adding perturbation on the clean test dataset by the Projected Gradient Descent (PGD) method [Kurakin et al., 2018]. Balance Score (B-score) [Ye et al., 2021] measures both clean accuracy and PGD accuracy. It is defined as B-score = 2 × ( CA × PA ) / ( CA + PA ) , where CA and PA denote clean accuracy and PGD accuracy respectively. We report the three metrics against the number of tasks in Fig. 3. As evidenced by the results, FTPL-D+ and FTPL-A attain comparable performance, and both significantly outperform FTPL in terms of all three metrics after the environmental changes at T = 50 and 100 . This suggests that our methods are effective in rapidly adapting to the new task distribution.

7 Conclusion and Future Work

This paper investigates online non-convex learning in dynamic environments, using dynamic regret and adaptive regret as performance metrics. For dynamic regret minimization, we propose FTPL-D with an O ( T 2 3 ( V T +1) 1 3 ) regret bound. To eliminate the dependence on prior knowledge of V T , we propose FTPL-D+, which runs multiple instances of FTPL-D and uses a meta-algorithm to track the best one. For adaptive regret minimization, we propose FTPL-A with a regret bound of O ( √ τ log T ) . Finally, we discuss the application to online constrained meta-learning, and the conducted experiments verify the effectiveness of our methods.

Currently, we bound the dynamic regret by the functional variation. A natural problem is whether we can derive regret bounds based on other regularities, such as the path length of comparators that is widely used in prior works for non-stationary online convex optimization [Zhang et al., 2018a, Zhao et al., 2020, 2024, Baby and Wang, 2021, Cutkosky, 2020]. This is left as a future work to explore.

References

Jacob Abernethy, Peter L Bartlett, Alexander Rakhlin, and Ambuj Tewari. Optimal strategies and minimax lower bounds for online convex games. In Proceedings of the 21st Annual Conference on Learning Theory , pages 414-424, 2008.

Dmitry Adamskiy, Wouter M Koolen, Alexey Chernov, and Vladimir Vovk. A closer look at adaptive regret. In Proceedings of the 23rd International Conference on Algorithmic Learning Theory , pages 290-304, 2012.

Naman Agarwal, Alon Gonen, and Elad Hazan. Learning in non-convex games with an optimization oracle. In Proceedings of the 32nd Annual Conference on Learning Theory , pages 18-29, 2019.

Sergul Aydore, Tianhao Zhu, and Dean P Foster. Dynamic local regret for non-convex online forecasting. In Advances in Neural Information Processing Systems 32 , pages 7982-7991, 2019.

Dheeraj Baby and Yu-Xiang Wang. Online forecasting of total-variation-bounded sequences. In Advances in Neural Information Processing Systems 32 , page 11071-11081, 2019.

Dheeraj Baby and Yu-Xiang Wang. Optimal dynamic regret in exp-concave online learning. In Proceedings of the 34th Annual Conference on Learning Theory , pages 359-409, 2021.

Dheeraj Baby and Yu-Xiang Wang. Optimal dynamic regret in proper online learning with strongly convex losses and beyond. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics , pages 1805-1845, 2022.

Omar Besbes, Yonatan Gur, and Assaf Zeevi. Non-stationary stochastic optimization. Operations research , 63(5):1227-1244, 2015.

Aude Billard, Yann Epars, Sylvain Calinon, Stefan Schaal, and Gordon Cheng. Discovering optimal imitation strategies. Robotics and autonomous systems , 47(2-3):69-77, 2004.

Matteo Castiglioni, Andrea Celli, Alberto Marchesi, Giulia Romano, and Nicola Gatti. A unifying framework for online optimization with long-term constraints. In Advances in Neural Information Processing Systems 35 , pages 33589-33602, 2022.

Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games . Cambridge University Press, 2006.

Nicolò Cesa-Bianchi and Francesco Orabona. Online learning algorithms. Annual review of statistics and its application , 8:165-190, 2021.

Luiz Chamon and Alejandro Ribeiro. Probably approximately correct constrained learning. In Advances in Neural Information Processing Systems 33 , pages 16722-16735, 2020.

Ashok Cutkosky. Parameter-free, dynamic, and strongly-adaptive online learning. In Proceedings of the 37th International Conference on Machine Learning , pages 2250-2259, 2020.

Amit Daniely, Alon Gonen, and Shai Shalev-Shwartz. Strongly adaptive online learning. In Proceedings of the 32nd International Conference on Machine Learning , pages 1405-1411, 2015.

Yoav Freund, Robert E Schapire, Yoram Singer, and Manfred K Warmuth. Using and combining predictors that specialize. In Proceedings of the 29th Annual ACM Symposium on Theory of Computing , pages 334-343, 1997.

Xiand Gao, Xiaobo Li, and Shuzhong Zhang. Online learning with non-convex losses and nonstationary regret. In Proceedings of the 21st International Conference on Artificial Intelligence and Statistics , pages 235-243, 2018.

Udaya Ghai, David Snyder, Anirudha Majumdar, and Elad Hazan. Generating adversarial disturbances for controller verification. In Proceedings of the 3rd Learning for Dynamics and Control , pages 1192-1204, 2021.

Ziwei Guan, Yi Zhou, and Yingbin Liang. Online nonconvex optimization with limited instantaneous oracle feedback. In Proceedings of the 36th Annual Conference on Learning Theory , pages 3328-3355, 2023.

András Gyorgy, Tamás Linder, and Gábor Lugosi. Efficient tracking of large classes of experts. IEEE Transactions on Information Theory , 58(11):6709-6725, 2012.

Nadav Hallak, Panayotis Mertikopoulos, and Volkan Cevher. Regret minimization in stochastic non-convex learning via a proximal-gradient approach. In Proceedings of the 38th International Conference on Machine Learning , pages 4008-4017, 2021.

Elad Hazan and Edgar Minasyan. Faster projection-free online learning. In Proceedings of 33rd Annual Conference on Learning Theory , pages 1877-1893, 2020.

Elad Hazan and Comandur Seshadhri. Adaptive algorithms for online decision problems. Electronic Colloquium on Computational Complexity , 14(88), 2007.

Elad Hazan, Karan Singh, and Cyril Zhang. Efficient regret minimization in non-convex games. In Proceedings of the 34th International Conference on Machine Learning , pages 1433-1441, 2017.

Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends® in Optimiza-

tion , 2(3-4):157-325, 2016.

Amélie Héliou, Matthieu Martin, Panayotis Mertikopoulos, and Thibaud Rahier. Online non-convex optimization with imperfect feedback. In Advances in Neural Information Processing Systems 33 , pages 17224-17235, 2020.

Amélie Héliou, Matthieu Martin, Panayotis Mertikopoulos, and Thibaud Rahier. Zeroth-order non-convex learning via hierarchical dual averaging. In Proceedings of the 38th International Conference on Machine Learning , pages 4192-4202, 2021.

Yanlong Huang, Leonel Rozo, Joao Silvério, and Darwin G Caldwell. Kernelized movement primitives. The International Journal of Robotics Research , 38(7):833-852, 2019.

Ali Jadbabaie, Alexander Rakhlin, Shahin Shahrampour, and Karthik Sridharan. Online optimization: Competing with dynamic comparators. In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics , pages 398-406, 2015.

Kwang-Sung Jun, Francesco Orabona, Stephen Wright, and Rebecca Willett. Improved strongly adaptive online learning using coin betting. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , pages 943-951, 2017.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ArXiv preprint , arXiv:1412.6980, 2014.

Walid Krichene, Maximilian Balandat, Claire Tomlin, and Alexandre Bayen. The hedge algorithm on a continuum. In Proceedings of the 32nd International Conference on Machine Learning , pages 824-832, 2015.

Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security , pages 99-112. Chapman and Hall/CRC, 2018.

Antoine Lesage-Landry, Joshua A Taylor, and Iman Shames. Second-order online nonconvex optimization. IEEE Transactions on Automatic Control , 66(10):4866-4872, 2020.

Nick Littlestone and Manfred K Warmuth. The weighted majority algorithm. Information and Computation , 108(2):212-261, 1994.

Haipeng Luo and Robert E Schapire. Achieving all with no parameters: Adanormalhedge. In Proceedings of the 28th Annual Conference on Learning Theory , pages 1286-1304, 2015.

Aryan Mokhtari, Shahin Shahrampour, Ali Jadbabaie, and Alejandro Ribeiro. Online optimization in dynamic environments: Improved regret rates for strongly convex problems. In Proceedings of the 55th IEEE Conference on Decision and Control , pages 7195-7201, 2016.

Seth Neel, Aaron Roth, Giuseppe Vietri, and Steven Wu. Oracle efficient private non-convex optimization. In Proceedings of the 37th International Conference on Machine Learning , pages 7243-7252, 2020.

Francesco Orabona. A modern introduction to online learning. ArXiv preprint , arXiv:1912.13213, 2019.

Abhishek Roy, Krishnakumar Balasubramanian, Saeed Ghadimi, and Prasant Mohapatra. Multipoint bandit algorithms for nonstationary online nonconvex optimization. ArXiv preprint , arXiv:1907.13616, 2019.

Shai Shalev-Shwartz et al. Online learning and online convex optimization. Foundations and Trends® in Machine Learning , 4(2):107-194, 2012.

Arun Sai Suggala and Praneeth Netrapalli. Online non-convex learning: Following the perturbed leader is optimal. In Proceedings of the 31st International Conference on Algorithmic Learning Theory , pages 845-861, 2020.

Tim van Erven and Wouter M Koolen. Metagrad: Multiple learning rates in online learning. In Advances in Neural Information Processing Systems 29 , page 3666-3674, 2016.

Giuseppe Vietri, Grace Tian, Mark Bun, Thomas Steinke, and Steven Wu. New oracle-efficient algorithms for private synthetic data release. In Proceedings of the 37th International Conference on Machine Learning , pages 9765-9774, 2020.

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.

Yuanyu Wan, Bo Xue, and Lijun Zhang. Projection-free online learning in dynamic environments. In Proceedings of the 35th AAAI Conference on Artificial Intelligence , pages 10067-10075, 2021.

Yuanyu Wan, Lijun Zhang, and Mingli Song. Improved dynamic regret for online frank-wolfe. In Proceedings of the 36th Annual Conference on Learning Theory , pages 3304-3327, 2023.

Yuanyu Wan, Chang Yao, Mingli Song, and Lijun Zhang. Non-stationary online convex optimization with arbitrary delays. In Proceedings of the 41st International Conference on Machine Learning , pages 49991-50011, 2024.

Yibo Wang, Yuanyu Wan, Shimao Zhang, and Lijun Zhang. Distributed projection-free online learning for smooth and convex losses. In Proceedings of the 37th AAAI Conference on Artificial Intelligence , pages 10226-10234, 2023.

Yibo Wang, Wenhao Yang, Wei Jiang, Shiyin Lu, Bin Wang, Haihong Tang, Yuanyu Wan, and Lijun Zhang. Non-stationary projection-free online learning with dynamic and adaptive regret guarantees. In Proceedings of the 38th AAAI Conference on Artificial Intelligence , pages 15671-15679, 2024.

Siyuan Xu and Minghui Zhu. Online constrained meta-learning: Provable guarantees for generalization. In Advance in Neural Information Processing Systems 37 , pages 15531-15544, 2023.

Lin Yang, Lei Deng, Mohammad H Hajiesmaili, Cheng Tan, and Wing Shing Wong. An optimal algorithm for online non-convex learning. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 2(2):1-25, 2018.

Tianbao Yang, Lijun Zhang, Rong Jin, and Jinfeng Yi. Tracking slowly moving clairvoyant: Optimal dynamic regret of online learning with true and noisy gradient. In Proceedings of the 33rd International Conference on Machine Learning , pages 449-457, 2016.

Wenhao Yang, Wei Jiang, Yibo Wang, Ping Yang, Yao Hu, and Lijun Zhang. Small-loss adaptive regret for online convex optimization. In Proceedings of the 41st International Conference on Machine Learning , pages 56156-56195, 2024.

Feiyang Ye, Baijiong Lin, Zhixiong Yue, Pengxin Guo, Qiao Xiao, and Yu Zhang. Multi-objective meta learning. In Advances in Neural Information Processing Systems 34 , pages 21338-21351, 2021.

Lijun Zhang. Online learning in changing environments. In Proceedings of the 29th International Joint Conference on Artificial Intelligence , pages 5178-5182, 2020.

Lijun Zhang, Tianbao Yang, Jinfeng Yi, Rong Jin, and Zhi-Hua Zhou. Improved dynamic regret for non-degenerate functions. In Advance in Neural Information Processing Systems 30 , pages 732-741, 2017.

Lijun Zhang, Shiyin Lu, and Zhi-Hua Zhou. Adaptive online learning in dynamic environments. In Advances in Neural Information Processing Systems 31 , pages 1323-1333, 2018a.

Lijun Zhang, Tianbao Yang, Rong Jin, and Zhi-Hua Zhou. Dynamic regret of strongly adaptive methods. In Proceedings of the 35th International Conference on Machine Learning , pages 5882-5891, 2018b.

Lijun Zhang, Tie-Yan Liu, and Zhi-Hua Zhou. Adaptive regret of convex and smooth functions. In Proceedings of the 36th International Conference on Machine Learning , pages 7414-7423, 2019.

Lijun Zhang, Shiyin Lu, and Tianbao Yang. Minimizing dynamic regret and adaptive regret simultaneously. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics , pages 309-319, 2020.

Lijun Zhang, Guanghui Wang, Wei-Wei Tu, Wei Jiang, and Zhi-Hua Zhou. Dual adaptivity: A universal algorithm for minimizing the adaptive regret of convex functions. In Advances in Neural Information Processing Systems 34 , pages 24968-24980, 2021.

Lijun Zhang, Wei Jiang, Jinfeng Yi, and Tianbao Yang. Smoothed online convex optimization based on discounted-normal-predictor. In Advances in Neural Information Processing Systems 35 , pages 4928-4942, 2022.

Peng Zhao and Lijun Zhang. Improved analysis for dynamic regret of strongly convex and smooth functions. In Proceedings of the 3rd Conference on Learning for Dynamics and Control , pages 48-59, 2021.

Peng Zhao, Yu-Jie Zhang, Lijun Zhang, and Zhi-Hua Zhou. Dynamic regret of convex and smooth functions. In Advances in Neural Information Processing Systems 33 , pages 12510-12520, 2020.

Peng Zhao, Yan-Feng Xie, Lijun Zhang, and Zhi-Hua Zhou. Efficient methods for non-stationary online learning. In Advances in Neural Information Processing Systems 35 , pages 11573-11585, 2022.

Peng Zhao, Yu-Jie Zhang, Lijun Zhang, and Zhi-Hua Zhou. Adaptivity and non-stationarity: Problemdependent dynamic regret for online convex optimization. Journal of Machine Learning Research , 25(98):1 - 52, 2024.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning , pages 928-936, 2003.

A Theoretical Analysis

A.1 Proof of Theorem 1

Let r = T γ , q i = ( i -1) γ +1 for i = 1 , . . . , r , and q r +1 = T +1 . First, we decompose the dynamic regret as

Taking expectation with respect to x t on both sides of (13), we have

To bound a i , we introduce the following lemma, which presents the static regret of Algorithm 1.

Lemma 1. (Theorem 1 in Suggala and Netrapalli [2020]) Suppose Algorithm 1 is using an ( α, β ) -approximate optimization oracle. Under Assumptions 1 and 2, for any fixed η , Algorithm 1 ensures

By applying Lemma 1 to each interval i ∈ [ r ] , we get

Let η = 1 √ dγ , then we have

where c ( α, β, γ ) = 125 DL 2 d 3 2 + (21 βγ + D ) d 3 2 20 + 21 α √ γ 20 +2 dLβγ.

To bound b i , we follow the proof of Theorem 3 in Zhang et al. [2018b]:

Define the local functional variation of the i -th interval as

and it is obvious that ∑ r i =1 V T ( i ) ≤ V T .

For any t ∈ [ q i , q i +1 -1] , we have

Combining (16) and (17), we obtain

Substituting (15) and (18) into (14), we have

where the last inequality uses ∑ r i =1 V T ( i ) ≤ V T and r = ⌈ T γ ⌉ ≤ T γ +1 ≤ 2 T γ .

We proceed to prove the second part of Theorem 1. For brevity, let ˆ c = c ( α, β, T ) . By the definition in (8), we have c ( α, β, γ ) ≤ ˆ c for any γ ≤ T . If V T ≥ 1 √ T , Algorithm 2 with γ = ⌊ ( T V T ) 2 3 ⌋ achieves

If V T ≤ 1 √ T , Algorithm 2 with γ = T ensures

Combining the results above, with the prior knowledge of V T , we have

A.2 Proof of Theorem 2

The dynamic regret of the Algorithm 3 can be devided into two parts: expert regret and meta-regret, as shown below

where x i t denotes the prediction of the i -th expert.

To bound the meta-regret, we follow the proof of Theorem 2.2 in Cesa-Bianchi and Lugosi [2006] and introduce a novel quantity ln W T W 0 . We define

where L i t = ∑ t τ =1 f τ ( x i τ ) denotes the cumulative loss of the i -th expert and N denotes the number of experts. Besides, the weight of expert at round t can be writen as

First, we give a lower bound of ln W T W 0 :

To upper bound ln W T W 0 , we introduce the following lemma of Hoeffding's inequality.

Lemma 2. (Lemma A.1 in Cesa-Bianchi and Lugosi [2006]) Let X be a random variable with a ≤ X ≤ b . Then for any s ∈ R ,

Note that at each round t , we sample an expert (prediction) by P ( x t = x i t ) = w i t , and incur the loss of that expert. Hence, the loss incurred by the meta-algorithm is a random variable f t ( x t ) with probability P [ f t ( x t ) = f t ( x i t ) ] = w i t , and the expected loss is

Under Assumptions 1 and 2, the difference between the maximum and minimum of the loss function at each round can be bounded as

Observe that for each t = 1 , . . . , T ,

Now using Lemma 2, we have

Summing over t = 1 , . . . , T , we have

By combining (22) and (27), we get the meta-regret-bound as follows:

We proceed to present the dynamic regret of the best expert. Let H = { γ i = 2 i | i = 1 , · · · N } where N = glyph[floorleft] log 2 T glyph[floorright] . Assuming the optimal restarting parameter in Theorem 1 is γ ∗ , then there exists some k such that γ k ≤ γ ∗ ≤ γ k +1 . Moreover, we have

According to Theorem 1, for any expert i , we have

where ˆ c = c ( α, β, T ) is given in (8).

Then, for expert k and k +1 ,

and

From Theorem 1, it can be inferred that

Hence, the dynamic regret of the best expert can be bounded as

Substituting (28) and (34) into (20), we get the dynamic regret of Algorithm 3:

A.3 Proof of Theorem 3

The analysis consists of two parts. We first upper bound the strongly adaptive regret over any interval J = [ i, j ] ∈ I . Then, we extend the regret bound to any interval I = [ s, s + τ -1] ⊆ [ T ] .

For any interval J = [ i, j ] ∈ I , we can decompose the regret as

According to Lemma 1 and (15), we have the following expert regret

Next, we analyze the meta-regret. Let N t denote the total number of experts that have been seen up to round t . Recall the definition of GC intervals:

For every k that satisfies 2 k < t , we have that a single interval in I k contains t . Since we activate an expert for each interval, the number of active experts at round t is 1 + glyph[floorleft] log 2 t glyph[floorright] . Taking each round into account, it is easy to verify that

For interval J , we denote the regret at round t as r t,J . By (25), we scale r t,J to ensure

Then, according to Theorems 1 and 3 in Luo and Schapire [2015], we have the following lemma.

Lemma 3. Under Assumptions 1 and 2, for any interval J = [ i, j ] ∈ I , Algorithm 4 ensures

where

Combining (38) and (39), we obtain that

Note that at each round t , we sample x t by P ( x t = x t,J ) = w t,J . Thus, the expected loss incurred by the meta-algorithm is

Also note that ˜ f t ( x t ) is defined as the weighted average loss of all experts, which is of the same form as (41), i.e.

Replacing the weighted average loss in (40) by expected loss, we have

Combining the expert regret in (36) and the meta-regret in (43), we obtain that

where g ( j ) ≤ 1 + ln j +ln(1 + log 2 j ) + ln 5+3ln(1+ j ) 2 .

In the following, we extend the regret bound over J to any interval I = [ s, s + τ -1] ⊆ [ T ] . We first introduce a property of GC intervals as below.

Lemma 4. (Lemma 1.2 in Daniely et al. [2015]) For any interval I = [ s, s + τ -1] ⊆ [ T ] , it can be partitioned into two sequences of disjoint and consecutive intervals, denoted by I -p , . . . , I 0 ∈ D and I 1 , . . . , I q ∈ D , such that

According to Lemma 4, we have that

Hence, the strongly adaptive regret of Algorithm 4 is

where g ( T ) ≤ 1 + ln T +ln(1 + log 2 T ) + ln 5+3ln(1+ T ) 2 .

A.4 Proof of Theorem 4

The analysis is similar to Corollary 5 in Zhang et al. [2018b], which exploits the following lemma to bound the dynamic regret by the strongly adaptive regret and the functional variation.

Lemma 5. (Theorem 3 in Zhang et al. [2018b]) Let u ∗ t ∈ arg min u ∈K f t ( u ) . For all integer k ∈ [ T ] , we have

where the minimization is taken over any sequence of intervals.

For our randomized algorithm, we take the expectation over both sides

Next, we restric to intervals of length τ , and in this case k = T τ . Then we get

Combining (46) and (47), we have

By the fact that

we obtain

In the following, we consider two cases. If V T ≥ √ log T T , we choose

and have

where ˆ c = c ( α, β, T ) .

Otherwise, we choose τ = T , and have

Combining (49) and (50), we have

B More Related Works

In OCO, the computational effective variants of FTPL are investigated in centralized [Hazan and Minasyan, 2020] and decentrailized [Wang et al., 2023] setting. In addition to the worst-case dynamic regret defined in (2), Zinkevich [2003] also introduce the general dynamic regret, which compares the learner against any sequence of comparators

where u 1 , . . . , u T ∈ K . In this pioneer work, Zinkevich [2003] proposed a regularity of comparator sequence called path-length

and showed that OGD attains an O ( √ T (1+ P T )) general dynamic regret bound. Zhang et al. [2018a] strengthened the general dynamic regret bound to O ( √ T (1 + P T )) , which has been shown to be optimal. Improved results can be obtained when the online functions are smooth [Zhao et al., 2020, 2024], strongly convex [Baby and Wang, 2022] or exp-concave [Baby and Wang, 2021]. Moreover, the computational efficiency of general dynamic regret optimization is recently considered [Zhao et al., 2022, Wang et al., 2024].

Notice that by assigning u t ∈ arg min x ∈K f t ( x ) as the local minimizer, we get the worst-case dynamic regret in (2), and there exist numerous studies on the worst-case scenario. When the loss functions are strong convex and smooth [Mokhtari et al., 2016], or when they are smooth and the minimizers lie in the interior of K [Yang et al., 2016], OGD can attain O ( P ∗ T + 1) dynamic regret bound, where P ∗ T := P T ( x ∗ 1 , . . . , x ∗ T ) and x ∗ t ∈ arg min x ∈K f t ( x ) . Another regularity of comparator sequence is the squared path-length [Zhang et al., 2017]:

which could be much smaller than the path-length P ∗ T when local minimizers move slowly. Zhang et al. [2017] demonstrated that the dynamic regret bound can be reduced to O (min { P ∗ T , S ∗ T } +1) for (semi-)strongly convex and smooth functions. Moreover, Besbes et al. [2015] demonstrated that if the value of V T is known, a restarted OGD can achieve O ( T 2 3 ( V T +1) 1 3 ) and O ( √ T ( V T +1)) dynamic regret for convex and strongly convex functions, respectively. Later, Baby and Wang [2019] improved the dynamic regret to O ( T 1 3 ( V T +1) 2 3 ) for 1-dim square loss. More recently, to exploit both comparator sequence and function sequence, Zhao and Zhang [2021] refined the analysis for online multiple gradient descent and attained an O (min { P ∗ T , S ∗ T , V T } +1) dynamic regret bound under the same assumption as Zhang et al. [2017].

Besides dynamic regret, another metric in dynamic environments is adaptive regret. Adaptive regret is introduced by Hazan and Seshadhri [2007] to OCO, but in a weak form:

which is defined as the maximum static regret over any contiguous interval. To minimize (54), Hazan and Seshadhri [2007] proposed follow the leading history (FLH) with O ( √ T log 3 T ) weakly adaptive regret for convex functions. However, this metric does not respect short intervals well since the O ( √ T log 3 T ) bound is meaningless for those intervals of length O ( √ T ) . To eliminate the dominance of long intervals, Daniely et al. [2015] introduced strongly adaptive regret that takes the interval length τ as a parameter, as we indicate in (4). They also developed two-layer algorithms and attain O ( √ τ log T ) strongly adaptive regret for general convex functions. Later, Jun et al. [2017] proposed a novel meta-algorithm named sleeping coin betting (SCB) and improved the strongly adaptive regret bound to O ( √ τ log T ) .

Although both dynamic regret and adaptive regret are designed for dynamic environments, our understanding of their relationship remains limited. Zhang et al. [2018b] first demonstrated that

dynamic regret can be bounded by the strongly adaptive regret and the functional variation. Recent studies shows that it is possible to minimize dynamic regret and adaptive regret simultaneously [Zhang et al., 2020, Cutkosky, 2020, Zhang et al., 2022, Wang et al., 2024].

The metric of dynamic regret has also been investigated in online non-convex learning. LesageLandry et al. [2020] proposed online newton method (ONM), a second-order method, achieving an O ( P ∗ T + 1) dynamic regret bound under the conditions that the starting point is located near the global optimal solution and the loss function is strongly convex around the optimal solution. Gao et al. [2018] examined a class of non-convex functions that satisfy weak pseudo-convex conditions and developed online algorithms that attain O ( √ T ( V T +1)) dynamic regret. They also extended their result to the bandit feedback setting. Roy et al. [2019] studied functions with weak-quasi-convexity and obtained similar results as Gao et al. [2018]. Furthermore, Héliou et al. [2020] applied their methods to bandit setting using a kernel-based estimator. In a subsequent work, Héliou et al. [2021] proposed hierarchical dual averaging (HDA) and improved the dynamic regret bounds under the same setting.

C Omitted Details for Experiments

All experiments are executed on a computer with a 2.50 GHz Intel Xeon Platinum 8255C CPU and an RTX 2080Ti GPU. We follow the experiment settings of Xu and Zhu [2023] and implement our methods based on their code.

C.1 Experiments on meta-imitation learning.

Problem formulation. Imitation learning [Billard et al., 2004] has gained significant attention as a means of transferring human skills to robots. Huang et al. [2019] presents a novel formulation of imitation learning using kernelized movement primitives, which considers nonlinear hard constraints and obstacle avoidance. In their study, the states of the robot are modeled as a linear combination of basis functions, and the demonstrations provided by humans are modeled by a Gaussian mixture model (GMM), where the parameters follow a Gaussian distribution. The objective of their approach is to minimize the divergence between the distributions of the robot state model and the demonstrations, while ensuring that the hard constraints are satisfied. Building upon this idea, Xu and Zhu [2023] further model the states of the robot by a neural network instead of the linear combination of basis functions.

Following the work of Xu and Zhu [2023], in this experiment, we utilize a neural network to model the states of the robot and GMM to model the demonstrations. Specifically, the robot's state ξ ( w,t ) , which includes the joint position q ( w,t ) ∈ R O and velocity ˙ q ( w,t ) , is parameterized by w and is modeled as

where q ( w,t ) is a neural network and takes t as the input and the location q ( w,t ) as the output. The dataset of demonstrations consists of H trajectories, where each trajectory contains N time-state pairs. These pairs, denoted as {{ t n,h , ˆ ξ n,h } N n =1 } H h =1 , are modeled using a GMM. Each demonstration state ˆ ξ n associated with t n is described by a conditional probability distribution with mean ˆ µ n and covariance ˆ Σ n , i.e., ξ n | t n ∼ N (ˆ µ n , ˆ Σ n ) , where ˆ µ n and ˆ Σ n can be computed by the GMM. As formulated in previous works [Huang et al., 2019, Xu and Zhu, 2023], the imitation learning task can be cast as the following constrained optimization problem:

where g i is the i-th state constraint, m is the total constraint number, ˆ µ t and ˆ Σ t are the mean and variance of the demonstration state ˆ ξ ( t ) , i.e., ˆ ξ | t ∼ N (ˆ µ t , ˆ Σ t ) . Here, the training dataset { t n , ˆ µ n , ˆ Σ n } N n =1 is provided by the GMM. It is notable that the demonstrations are collected in a no-collision environment. Thus, it is possible that these demonstrations may not be able to avoid collisions effectively.

0

50

100

150

200

Round (task index)

0

1

2

3

4

5

6

7

Constraint violation

FTPL

FTPL-A

FTPL-D+

(a) Constrained violation metric

0

50

100

150

200

Round (task index)

0

20

40

60

80

100

Cumulative constraint violation

FTPL

FTPL-A

FTPL-D+

(b) Cumulative constrained violation metric

In our scenario, we have a series of imitation learning tasks that are revealed sequentially. In each round t , a new task of Problem (55) is revealed, and its data D tr t = { t n , ˆ µ n , ˆ Σ n } N n =1 and the collision area denoted by { g i } i m =1 and { c i } i m =1 are given. Then we apply the framework of online constrained meta-learning [Xu and Zhu, 2023] and construct the meta objective at round t as

where L ( w, D tr ) is the loss function of the model parameter w on a dataset D tr = { t n , ˆ µ n , ˆ Σ n } N n =1 , and

Next, we update φ t by our algorithms and compute w t = A lg ( λ, φ t , D tr t ) as the task-specific model.

General setup. We use the demonstration data given by Huang et al. [2019]. In each round, the robot needs to imitate the given demonstration and write a capital letter. The demonstration dataset, denoted as D tr t = { t n , ˆ µ n , ˆ Σ n } N n =1 , consists of the sizes, angles, and locations of the letter. These sizes, angles, and locations are randomly sampled from a Gaussian distribution. The robot also needs to avoid a circle collision area, which defines m , g 1 , and c 1 in (55) as m = 1 , c 1 = 0 . 5 and g 1 ( x ) = σ ( -√ ( x 1 -d 1 ) 2 +( x 2 -d 2 ) 2 + r ) , where σ ( x ) = β log(1 + exp( βx )) is a barrier function and, r is the radius of the collision area and r = 2 . The center of the collision area, denoted by ( d 1 , d 2 ) , is sampled from the Gaussian distribution N (0 , 1) for each task. Furthermore, in each task, the robot can access the full shot of demonstrations including 400 data points.

We model the position of the robot by a four-layer neural network with 128 which consists of an input layer of size 8 , followed by 3 hidden layers of size 128 with the ReLU nonlinearities and an output layer of size 2 . The neural network takes { t, t 2 , t 3 , t 4 , sin ( t ) , cos (2 t ) , sin (2 t ) , cos (2 t ) } as the inputs and q ( w,t ) as the outputs. We use the Adam optimizer [Kingma and Ba, 2014] with a learning rate of 0.001 for the optimization.

We simulate two types of dynamic environments: abruptly changing environments and gradually evolving environments. For abruptly changing environments, we use the demonstration data of capital letters "M", "E", "T" and "A" for the 4 stages. For gradually evolving environments, we rotate the letter "A" at a small random angle δ t ∈ (0 , 0 . 05] in each round. Specifically, at round t , we compute with

0

50

100

150

200

Round (task index)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Constraint violation

FTPL

FTPL-A

FTPL-D+

(a) Constrained violation metric

0

50

100

150

200

Round (task index)

0

5

10

15

20

25

30

35

Cumulative constraint violation

FTPL

FTPL-A

FTPL-D+

(b) Cumulative constrained violation metric

the rotation angle δ = ∑ t τ =1 δ τ , define a rotation matrix

and then transform ˆ µ n and ˆ Σ n of the dataset by

Supplementary results. Apart from the loss shown in Fig 3, the constrained violation is another metric considered in Xu and Zhu [2023]. We plot the constrained violation in Fig 5 and Fig 6. The results show that our methods sustain the same and even less constrained violation than FTPL. This is reasonable and can be explained by Proposition 5 in Xu and Zhu [2023] that the constrained violation bound holds for any meta-parameter sequence φ 1: T = { φ 1 , · · · , φ T } .

C.2 Experiments on few-shot image classification with robustness

Problem formulation. Following the problem formulation in Chamon and Ribeiro [2020] and Xu and Zhu [2023], the problem of robust learning for a single image classification task T t can be written as

where 0 < α < 1 is the robustness tolerance parameter, glyph[lscript] is the loss function, D t is the distribution of the original data, and D t, [ P ] is the distribution of the perturbed data, which is generated by the PGD method on D t .

In our setting, a series of robust 5-way 5-shot image classification tasks are revealed sequentially. In each round t , a new task of Problem (59) is revealed, which consists of a 5-shot meta-training dataset D tr t (the support dataset), and a meta-validation dataset D val t (the query dataset). Additionally, the dataset D tr t, [ P ] is also 5-shot and generated by the PGD method [Kurakin et al., 2018] on D tr t . Then the meta objective at round t can be written as

with where L ( w, D tr t ) is the loss function of the model parameter θ on a dataset D tr t . Next, we update φ t by our algorithms and compute θ t = A lg ( λ, φ t , D tr t ) as the task-specific model. In the experiment, we select α = 0 . 3 and λ = 1 . 0 .

Black Footed Albatross

(a) Water birds

Cardinal

Gray Catbird

Spotted Catbird

(b) Forest birds

Eastern Towhee

Scissor T ailed Flycatcher

Horned Lark

Bobolink

(c) Grassland birds

General setup. We conduct the experiments on CUB-200-2011 (referred to as CUB) dataset [Wah et al., 2011]. The input image processing and network architecture are fully provided in Section 6.2. To simulate real-world dynamic environments, we select three groups of bird categories from the CUB dataset based on their habitats: water birds, forest birds, and grassland birds. We show example images in Fig. 7 and provide a full list of classes for each group in Table 2. For each group, we randomly allocate 50% of the classes for training data, 25% for validation data, and 25% for test data. The time horizon T is then evenly divided into three stages, with tasks sampled from one group of bird categories in each stage. Specifically, in each round, we sample a 5-way 5-shot task from the training data classes, involving the selection of 5 classes with 5 images per class. During the meta-testing phase, we sampled 600 5-way 5-shot tasks from the test data classes.

We use the Adam optimizer [Kingma and Ba, 2014] with a learning rate of 0.001 for the optimization and use cross-entropy as the loss function. The adversarial attack on the query set is performed using the PGD method with a perturbation size of glyph[epsilon1] = 2 / 255 . This process involves 7 iterative steps, each with a step size of 2 . 5 glyph[epsilon1] .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Section 4 proposes algorithms for dynamic regret minimization. Section 5 provides the algorithm for adaptive regret minimization. Section 6 reports the application and the experiments.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See the future work in Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Section 3 presents the assumptions. Appendix A provides the proofs of the theorems.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe our algorithms in Sections 4 and 5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code and data are included in supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 6 and Appendix C.1 for experimental details. Full details are available in the code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See the results in Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics and conform to it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work does not present any foreseeable societal consequence.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This work does not pose such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite Xu and Zhu [2023] properly.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This work does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides $O(V_T^{1/3}T^{2/3})$ dynamic regret\nand $O(\\sqrt{\\tau\\log(T)})$ strongly-adaptive regret guarantees for\nonline learning with Lipschitz losses in bounded domains.\n\nSTRENGTHS:\nThe paper fills a gap in the literature by providing the\ndynamic and strongly-adaptive extensions of the FTPL\nalgorithm. The paper reads well and is easy to follow.\n\nWEAKNESSES:\nThe results are straight-forward applications of existing results. Prior analyses provide the\nregret bound of FTPL, and by assuming Lipschitz+bounded domain one immediately has access\nto an experts algorithm which will combine instances of the base algorithm (multiplicative weights update).\nSimilarly, the strongly adaptive result seems to follow easily from the standard\ngeometric covering intervals + AdaNormalHedge.\n\nThe results are adaptive to the temporal variability, $\\sum_{t=2}^{T}\\sup_x|f_t(x)-f_{t-1}(x)|$, which is more\nof a variance measurement than it is a measure of comparator variability. For instance, even\nthe temporal variability can be high even if the comparator is fixed.\nYet studying dynamic\nregret in the first place implies we are concerned with the variability of the comparator sequence; the ideal measure\nis something like path-length, where the \"difficulty\" of the comparator sequence is directly\nreflected in the bound. Here, the bound reflects only seems to\nreflect the difficulty of the loss sequence.\n\nQUESTIONS:\n- what are some examples of interesting losses which are non-convex, but bounded and lipschitz?\n- Can a similar strategy be developed for FTRL? Or is randomization an essential ingredient for the base algorithm?\n- Most online learning guarantees are anytime and wp1, while the results here are only in expectation.\n  Would it be difficult to get a stronger high-probability result, at least?\n\nLIMITATIONS:\nThe weakness in terms of adapting to temporal variability instead of path-length is pointed out in the conclusion", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-28"}

comments: SUMMARY:
This paper presents novel algorithms to tackle the challenges of online learning with non-convex loss functions in dynamic settings. The authors extend the Follow the Perturbed Leader (FTPL) algorithm to dynamic environments by proposing two new algorithms: FTPL-D and FTPL-D+. They demonstrate the effectiveness of their methods through theoretical regret analysis.

STRENGTHS:
The paper introduces innovative extensions to the FTPL algorithm, specifically designed for non-convex and dynamic environments. The paper is well-written and structured, making it easy to follow the theoretical developments and experimental setups. The algorithms are clearly described, and the results are presented in a manner that highlights their significance.

WEAKNESSES:
I think the problem addressed in this paper is not sufficiently central. The main contribution is extending the loss function to a non-convex case under this specific setup, and the proof methods are quite similar to previous works. Additionally, the experimental setup is relatively simple and lacks more practical experiments.

QUESTIONS:
Could the authors include some more challenging experimental tasks? The current tasks are relatively simple.

LIMITATIONS:
The authors did not discuss the limitations in sufficient detail. The last paragraph of the paper only contains some discussion on future directions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes two variant algorithms of the FTPL (follow-the-perturbed-leader) algorithm for online learning with non-convex losses in time-changing environments. The authors analyze these algorithms under the settings where the variability of the environment ($V_T$) is both known and unknown. FTPL-D is their proposed algorithm to minimize dynamic regret, and FTPL-A is their proposed algorithm to minimize adaptive regret, where their regret bounds match the best known bounds in the convex equivalent cases. The authors also provide an imitation-learning-based example to test the validity of their algorithms.\n\nSTRENGTHS:\nThe paper introduces (to the best of my knowledge) novel _variants_ of the FTPL algorithm. The paper is well-organized and easy to follow, with clear explanations of the theoretical concepts and practical implementation details. The authors provide detailed proofs of their theoretical results in the appendices, as well as a numerical experiment to empirically document the performance of the algorithm in a concrete example setting to show that their regret bounds are competitive. The chief technical contribution (perhaps, surprisingly, to me) is that the non-convex adaptive-regret/dynamic-regret bounds match their convex equivalents, where the distinction follows mainly from the oracle requirements.\n\nWEAKNESSES:\n1) Going through the proofs: in equation 17 (to bound $b_i$, it seems that the last inequality is very loose. You are bounding two differences by two summations of differences. I wonder if this can be strengthened?\n2) The analysis for these algorithms seem to be quite straightforward - most of the proofs follow very standard analytic methods in the literature. I wonder why this result has not been discovered earlier?\n3) Is it realistic to have an $O(1/\\sqrt{\\gamma}, 1/\\gamma)$-approximate oracle? In particular, I was under the impression that the oracle defined in equation 5 was for a constant value of the parameter $\\gamma$, but it seems that the choice of $\\gamma$ might depend on $L, d, D$, and $T$ (which can be asymptotically large). Doesn't this also make the offline optimization oracle intractable? \n4) If the dynamic environment changes very quickly, how well can the system practically handle this? i.e. if the variational intensity $V_T$ is very high, the experiments do not seem to model this situation at all...\n5) Further, the computational complexity of this algorithm does not seem well-analyzed. What is the memory requirement of each algorithm? How does it scale with T and with the number of experts?\n6) One minor suggestion is that the authors should compare these proposed algorithms with SOTA algorithms (like online gradient descent (OGD) and its oracle). Further, there appears to be a line of related recent works from the control theory + online optimization literature such as [a,b,c,d] which has also made progress in this problem. For instance, [c] appears to have a more tractable oracle, in particular, than OGD. I wonder if you can also compare the analytic methods and results to these works as well, if they are in fact related?\n\nReferences:\n\n[a] Online Policy Optimization in Unknown Nonlinear Systems [Lin et. al. 2024]\n\n[b] Adaptive Regret for Control of Time-Varying Dynamics [Gradu et. al. 2020]\n\n[c] Online Adaptive Policy Selection in Time-Varying Systems: No-Regret via Contractive Perturbations [Lin et. al. 2024]\n\n[d] Online Control of Unknown Time-Varying Dynamical Systems [Minasyan et. al. 2022]\n\nQUESTIONS:\nPlease address the questions from the above section.\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

comments: SUMMARY:
This paper considers the problem of non-convex online learning in dynamic environments. The authors go on to provide some algorıthmic variants and their respective regret bounds depending on the functional variation. They show that the non-convex dynamic regrets matches the convex ones in the literature.

STRENGTHS:
S1) Regret analysis for non-convex online learning is highly significant.

S2) Deriving dynamic regret bounds for non-convex optimization with existing tools for convex optimization is nice.

S3) Achieving dynamic regret and adaptive regret simultanously is powerful.

WEAKNESSES:
W1) While the derivations are nice, algorithmic contribution is limited.

W2) Existance of an approximate optimization oracle with selectable parameters seems dubious.

W3) Since the $(\alpha,\beta)$-approximate oracle is an integral part of the algorithm, providing regret bounds independent of $(\alpha,\beta)$ seems counter-intuitive.

QUESTIONS:
Major Question: 

My biggest issue is with the missing feasibility discussion. The existence of an $(\alpha,\beta)$-approximate oracle is assumed, which is not a problem by itself. However, in the derivations, $(\alpha,\beta)$ can be set freely depending on $\gamma$, which is an input. This seems like a big issue to me. For static regret, the effect of $\gamma$ is non-existing and regret results may naturally depend on given $\alpha, \beta$, which is not the case for your analyses. Given that all of the results depend on this, I would like for authors to properly address it. Am I missing something here?

\
Minor Questions:

- Line 152: what is $x^*$?

- Line 179: how do you set $\gamma$?

LIMITATIONS:
Most limitations are adequately addressed except for the existence of a parameter-dependent offline optimization oracle.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-11