Title: MetaCURL: Non-stationary Concave Utility Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Concave utility reinforcement learning

non-stationary MDPs

learning with expert advice

online learning
Abstract: We explore online learning in episodic loop-free Markov decision processes on non-stationary environments (changing losses and probability transitions). Our focus is on the Concave Utility Reinforcement Learning problem (CURL), an extension of classical RL for handling convex performance criteria in state-action distributions induced by agent policies. While various machine learning problems can be written as CURL, its non-linearity invalidates traditional Bellman equations. Despite recent solutions to classical CURL, none address non-stationary MDPs. This paper introduces MetaCURL, the first CURL algorithm for non-stationary MDPs. It employs a meta-algorithm running multiple black-box algorithms instances over different intervals, aggregating outputs via a sleeping expert framework. The key hurdle is partial information due to MDP uncertainty. Under partial information on the probability transitions (uncertainty and non-stationarity coming only from external noise, independent of agent state-action pairs), we achieve optimal dynamic regret without prior knowledge of MDP changes. Unlike approaches for RL, MetaCURL handles full adversarial losses, not just stochastic ones. We believe our approach for managing non-stationarity with experts can be of interest to the RL community.

MetaCURL: Non-stationary Concave Utility Reinforcement Learning

Abstract

Weexplore online learning in episodic Markov decision processes on non-stationary environments (changing losses and probability transitions). Our focus is on the Concave Utility Reinforcement Learning problem (CURL), an extension of classical RL for handling convex performance criteria in state-action distributions induced by agent policies. While various machine learning problems can be written as CURL, its non-linearity invalidates traditional Bellman equations. Despite recent solutions to classical CURL, none address non-stationary MDPs. This paper introduces MetaCURL, the first CURL algorithm for non-stationary MDPs. It employs a meta-algorithm running multiple black-box algorithms instances over different intervals, aggregating outputs via a sleeping expert framework. The key hurdle is partial information due to MDP uncertainty. Under partial information on the probability transitions (uncertainty and non-stationarity coming only from external noise, independent of agent state-action pairs), the algorithm achieves optimal dynamic regret without prior knowledge of MDP changes. Unlike approaches for RL, MetaCURL handles adversarial losses. We believe our approach for managing non-stationarity with experts can be of interest to the RL community.

1 Introduction

We consider the task of learning in an episodic Markov decision process (MDP) with a finite state space X , a finite action space A , episodes of length N , and a probability transition kernel p := ( p n ) n ∈ [ N ] such that for all ( x, a ) ∈ X ×A , p n ( ·| x, a ) ∈ S X . For any finite set B , we denote by S B the simplex induced by this set, and by |B| its cardinality. For all d ∈ N we let [ d ] := { 1 , . . . , d } . At each time step n , an agent in state x n chooses an action a n ∼ π n ( ·| x n ) by means of a policy, and moves to the next state x n +1 ∼ p n +1 ( ·| x n , a n ) , inducing a state-action distribution sequence µ π,p := ( µ π,p n ) n ∈ [ N ] , where µ π,p n ∈ S X×A for all n ∈ [ N ] .

In many applications of learning in episodic MDPs, an agent aims at finding an optimal policy π maximizing/minimizing a concave/convex function F of its state-action distribution, known as the Concave Utility Reinforcement Learning (CURL) problem:

CURL extends reinforcement learning (RL) from linear to convex losses. Many machine learning problems can be written in the CURL setting, including: RL, where for a loss function ℓ , F ( µ π,p ) = ⟨ ℓ, µ π,p ⟩ ; pure RL exploration [28], where F ( µ π,p ) = ⟨ µ π,p , log( µ π,p ) ⟩ ; imitation learning [26, 35] and apprenticeship learning [55, 1], where F ( µ π,p ) = D g ( µ π,p , µ ∗ ) , with D g representing a Bregman

38th Conference on Neural Information Processing Systems (NeurIPS 2024).

Nadia Oudjane EDF R & D †

divergence induced by a function g and µ ∗ being a behavior to be imitated; certain instances of meanfield control [7], where F ( µ π,p ) = ⟨ ℓ ( µ π,p ) , µ π,p ⟩ ; mean-field games with potential rewards [34]; among others. The CURL problem alters the additive structure inherent in standard RL, invalidating the classical Bellman equations, requiring the development of new algorithms.

Most of existing works on CURL focus on stationary environments [28, 57, 58, 5, 56, 25, 12, 11], where both the objective function F and the probability transition kernel p remain the same across episodes. However, in practical scenarios, environments are rarely stationary. The work of [39] is the first to address online CURL with objective functions that can change arbitrarily between episodes, also known as adversarial losses [19]. However, their work assumes stationary probability kernels and presents results in terms of static regret (performance comparable to an optimal policy). In non-stationary scenarios, it is more relevant to minimize dynamic regret-the gap between the learner's total loss and that of any policy sequence (see Eq. (5) for formal definition). In this work we address this problem by introducing the first algorithm for CURL handling adversarial objective functions and non-stationary probability transitions, achieving near-optimal dynamic regret.

High-level idea. Our approach, called MetaCURL, draws inspiration from the online learning literature. In online learning [9], non-stationarity is often managed by running multiple black-box algorithm instances from various starting points and dynamically selecting the best performer using an "expert" algorithm. This strategy has demonstrated effectiveness in settings with complete information [29, 59, 47, 33]. With MetaCURL, we extend this concept to decision-making in MDPs. Unlike classical online learning, the main challenge faced is uncertainty. We assume that the probability transition kernel in each episode has a known deterministic structure but is affected by an external noise with unknown distribution, placing us in a setting with only partial information (see Section 2 for more details). The learner is then unable to observe the agent's loss under policies other than the one played.

MetaCURL is a general algorithm that can be applied with any black-box algorithm with low dynamic regret in near-stationary environments. CURL approaches suitable as black-boxes rely on parametric algorithms that would require prior knowledge of the MDP changes to tune their learning rate. MetaCURL also addresses this challenge by simultaneously running multiple learning rates and weighting them in direct proportion to their empirical performance. MetaCURL achieves optimal regret of order ˜ O ( √ ∆ π ∗ T +min { √ ∆ p ∞ T, T 2 / 3 (∆ p ) 1 / 3 } ) , where ∆ p ∞ and ∆ p represent the frequency and magnitude of changes of the probability transition kernel respectively, and ∆ π ∗ is the magnitude of changes of the policy sequence we compare ourselves with in dynamic regret (see Eqs. (6) and (7) for formal definitions). MetaCURL does not require previous knowledge of the degree of non-stationarity of the environment, and can handle adversarial losses. To ensure completeness, we show that Greedy MD-CURL from [39] fulfills the requirements to serve as a black-box algorithm. This is the first dynamic regret analysis for a CURL approach.

Comparisons. Without literature on non-stationary CURL, we review non-stationary RL approaches. Most methods [24, 13, 45, 17, 20, 40, 21] typically rely on prior knowledge of the MDP's nonstationarity degree, while MetaCURL does not. Let ∆ l ∞ and ∆ l represent the frequency and magnitude of change in the RL loss function, respectively 1 . Recently, [54] achieved a regret of ˜ O ( min { √ (∆ p ∞ +∆ l ∞ ) T, T 2 / 3 (∆ p +∆ l ) 1 / 3 } ) , a near-optimal result as demonstrated by [40], without requiring prior knowledge of the environment's variation. However, this regret bound is tied to changes in loss functions, making it ineffective against adversarial losses. In contrast, rather than

̸

1 ∆ l := 1 + ∑ T -1 t =1 ∆ l t and ∆ l ∞ := 1 + ∑ T -1 t =1 1 { ∆ l t = ∆ l t +1 } , where ∆ l t := ∑ N n =1 max x,a | ℓ t n ( x, a ) -ℓ t +1 n ( x, a ) | and ℓ t n ( x, a ) is the expected loss suffered by choosing action a in state x at step n of round t .

depending on the magnitude of variation of the loss function, MetaCURL's bound depends on the magnitude of variation of the policy sequence we use for comparison in dynamic regret. This allows it to handle adversarial losses, and to compare against policies with a more favorable bias-variance trade-off, which may not align with the optimal policies for each loss. In addition, we improve this dependency by paying it as √ ∆ π ∗ T instead of (∆ π ∗ ) 1 / 3 T 2 / 3 . We summarize comparisons in Table 1.

Other related works. The studies by [43, 42] examine the difference between optimizing the objective over infinite trials and the expectation of the objective over a single trial, challenging the traditional CURL formulation in Eq. (1). Here, we retain the classic formulation to align with existing CURL research. Other works on RL with nonlinear objective functions are [46, 16] focusing on rewards over trajectories rather than individual states. In addition to non-stationarity, there is a series of works on RL with adversarial losses but stationary probability transitions, with results only on static regret [48, 30, 18, 50, 32, 14]. Another line of research is known as corruption-robust RL. It differs from non-stationary MDPs in that it assumes a ground-truth MDP and measures adversary malice by the degree of ground-truth corruption [31, 38, 10, 60, 53].

Contributions. We resume our main contributions below:

We introduce MetaCURL, the first algorithm for non-stationary CURL. Under the framework described in Section 2, MetaCURL achieves the optimal dynamic regret bound of order ˜ O ( √ ∆ π ∗ T +min { √ ∆ p ∞ T, T 2 / 3 (∆ p ) 1 / 3 } ) , without requiring previous knowledge of the degree of non-stationarity of the MDP. MetaCURL handles full adversarial losses and improves the dependency of the regret on the total variation of policies. MetaCURL is the first adaptation of Learning with Expert Advice (LEA) to deal with uncertainty in non-stationary MDPs.

We also establish the first dynamic regret upper bound for an online CURL algorithm in a nearly stationary environment, which can serve as a black-box routine for MetaCURL.

Notations. Let ∥ · ∥ 1 be the L 1 norm, and for all v := ( v n ) n ∈ [ N ] , such that v n ∈ R X×A we define ∥ v ∥ ∞ , 1 := sup 1 ≤ n ≤ N ∥ v n ∥ 1 .

2 General framework: non-stationary CURL

When an agent plays a policy π := ( π n ) n ∈ [ N ] in an episodic MDP with probability transition p , it induces a state-action distribution sequence (also called the occupancy-measure [61]), which we denote by µ π,p := ( µ π,p n ) n ∈ [ N ] , with µ π,p n ∈ S X×A . It can be calculated recursively for all ( x, a ) ∈ X and n ∈ [ N ] by taking µ π,p 0 ( x, a ) = µ 0 ( x, a ) fixed, and

Offline CURL. The classic CURL optimization problem in Eq. (1) considers minimizing a function F : ( S X×A ) N → R , here defined as F ( µ ) := ∑ N n =1 f n ( µ n ) with f n a convex function over µ n with values in [0 , 1] , across all policies that induce µ π,p . Note that F is not convex on the policy π . To convexify the problem, we define the set of state-action distributions satisfying the Bellman flow of a MDP with transition kernel p as

For any µ ∈ M p µ 0 , there exists a strategy π such that µ π,p = µ . It suffices to take π n ( a | x ) ∝ µ n ( x, a ) when the normalization factor is non-zero, and arbitrarily defined otherwise. There is thus an equivalence between the CURL problem (optimization on policies) and a convex optimization problem on state-action distributions satisfying the Bellman flow:

Online CURL. In this paper we consider the online CURL problem in a non-stationary setting. We assume a finite-horizon scenario with T episodes. An oblivious adversary generates a sequence of changing objective functions ( F t ) t ∈ [ T ] , with F t being fully communicated to the learner only at the end of episode t . We assume F t is L F -Lipschitz with respect to the ∥ · ∥ ∞ , 1 norm for all t . The

probability transition kernel is also allowed to evolve over time and is denoted by p t at episode t . The learner's objective is then to calculate a sequence of strategies ( π t ) t ∈ [ T ] minimizing a total loss L T := ∑ T t =1 F t ( µ π t ,p t ) , while dealing with adversarial objective functions F t and changing probability transition kernels p t . To measure the learner's performance, we use the notion of dynamic regret (the difference between the learner's total loss and that of any policy sequence ( π t, ∗ ) t ∈ [ T ] ):

Non-stationarity measures. We consider the following two non-stationary measures ∆ p ∞ and ∆ p on the probability transition kernels that respectively measure abrupt and smooth variations:

̸

Regarding dynamic regret, we define for any sequence of policies ( π t, ∗ ) t ∈ [ T ] , its non-stationarity measure as

Moreover, for any interval I ⊆ [ T ] , we write ∆ p I := ∑ t ∈ I ∆ p t and ∆ π ∗ I := ∑ t ∈ I ∆ π ∗ t .

Dynamic's hypothesis. For each episode t , let ( x t 0 , a t 0 ) ∼ µ 0 ( · ) , and for all time steps n ∈ [ N ] ,

where g n represents the deterministic part of the dynamics, and ( ϵ t n ) n ∈ [ N ] is a sequence of independent external noises such that ϵ t n ∼ h t n ( · ) , where h t n is any centered distribution. Note that these dynamics imply that the probability transition kernel can be written as p t n +1 ( x ′ | x, a ) = P ( g n ( x, a, ϵ t n ) = x ′ ) . Different variants of this problem can be considered, depending on the prior information available about the dynamics in Eq. (8). In this article we consider the case where g n is fixed and known by the learner, but h t n is unknown and can change (hence the source of uncertainty and non-stationarity of the transitions). To the best of our knowledge, there are no black-box algorithms in the literature that achieve sublinear regret for online CURL with adversarial losses without relying on model assumptions. In using RL methods to CURL, we believe model-optimistic approaches like UCRL (Upper Confidence RL [4]) could be adapted. However, these methods are computationally expensive, as they require solving an additional optimization problem in every episode. The black-box algorithm for CURL we consider from [39] provides closed-form solutions, which is more computationally efficient, but requires the same dynamic assumption as in Eq. (8). Another class of RL methods is policy optimization (PO), which directly optimizes the policy and often yields closed-form solutions, leading to faster performance. Recent theoretical work [37] has shown that PO methods can achieve near-optimal regret without model assumptions. However, these methods rely on RL's Bellman equations, which do not apply to CURL due to its non-linear nature. We believe that the MetaCURL analysis could potentially be extended to the case where g n is unknown but belongs to a parametric family. We leave this extension for future work.

This particular dynamic is also motivated by many real-world applications:

Controlling a fleet of drones in a known environment, subject to external influences due to weather conditions or human interventions.

Addressing data center power management aiming to cut energy expenses while maintaining service quality. Workload fluctuations cause dynamic job queue transitions, and volatile electricity prices lead to varying operational costs. The probabilities of task processing by each server are predetermined, but the probabilities of task arrival are uncertain [6].

As renewable energy use increases and energy demand grows, balancing production and consumption becomes harder. Certain devices, like electric vehicle batteries and water heaters, can serve as flexible energy storage options. However, this requires electric grids to establish policies regulating when these devices turn on or off to match a desired consumption profile. These profiles can fluctuate daily due to changes in energy production levels. Despite knowing the devices' physical dynamics, household consumption habits remain unpredictable and constantly changing [51, 41].

Outline. In this paper, we propose a novel approach to handle non-stationarity in MDPs, being the first to propose a solution to CURL within this context. We begin in Section 3 by discussing the idea behind our algorithm's construction and the key challenges within our framework. Section 4 introduces MetaCURL, while Section 5 presents the main results of our regret analysis. The proofs' specifics are provided in the appendix.

3 Main idea

A hypothetical learner who achieves optimal regret. Let m> 1 . Assume a hypothetical learner that could compute a sequence of restart times 1 = t 1 < . . . < t m +1 = T + 1 , where for each i ∈ [ m ] we let I i := [ t i , t i +1 -1] , such that p p

Consider any algorithm that, when computing ( π t ) t ∈ I with learning rate λ for any interval I ⊆ [ T ] , attains a dynamic regret relative to any sequence of policies ( π t, ∗ ) t ∈ I upper bounded by

where ( c j ) j ∈ [3] are constants that may depend on the MDP parameters, and on the interval size only in logarithmic terms. This kind of regret bound holds for Greedy MD-CURL from [39] as we show in Appendix G. Suppose the hypothetical learner could also access ∆ π ∗ to calculate the optimal learning rate. Hence, playing such an algorithm for all horizon T with the optimal learning rate, the learner would have a dynamic regret upper bounded by

Optimizing over m , the learner would obtain the optimal regret of order ˜ O ( ∆ π ∗ T +(∆ p ) 1 / 3 T 2 / 3 ) . In the case where the MDP is piece-wise stationary, if the learner takes I i such that ∆ p I i = 0 , it obtains a regret of order O ( √ ∆ π ∗ T + √ ∆ p ∞ T ) , where ∆ p ∞ is the number of times the probability transitions of the MDP change over [ T ] .

A meta algorithm to learn restart times. In reality, the restart times of Eq. (9), and the optimal learning rate, are unknown to the learner. Hence, we propose to build a meta aggregation algorithm to learn both. Let E represent a parametric black-box algorithm with dynamic regret as in Eq. (10). We introduce a meta algorithm M that, takes as input a finite set of learning rates Λ , and at each episode t , initializes | Λ | instances of E , denoted as E t,λ for each λ ∈ Λ . Each E t,λ operates independently within the interval [ t, T ] . At time t , M combines the decisions from the active runs {E s,λ } s ≤ t,λ ∈ Λ by weighted average. The idea is that at time t , some of the outputs of {E s,λ } s ≤ t,λ ∈ Λ are not based on data prior to t ′ < t , so if the environment changes at time t ′ , these outputs can be given a greater weight by the meta algorithm, enabling it to adapt more quickly to the change. At the same time, we expect a larger weight will be given to the empirically best learning rate. Let M ( E , Λ) be the complete algorithm.

Remark 3.1. The meta-algorithm increases the computational complexity of the parametric blackbox algorithm by a factor of T ×| Λ | , as it requires updating t ×| Λ | instances at each episode t . By strategically designing intervals to run the black-box algorithms, previous works on online learning have reduced computational complexity to O (log( T )) [15, 29, 27]. Extending our analysis to these intervals is straightforward, but it would complicate the presentation of the paper. Thus, we decided to present our results using the naive choice of intervals. Also, in Section 5, we show that a learning rate grid with | Λ | = log( T ) is sufficient to obtain the optimal regret.

Regret decomposition. Denote by π t,s,λ the policy output from E s,λ at episode t , for learning rate λ , for all s ≤ t , and by π t the policy output by the meta algorithm M ( E , Λ) to be played by the learner. The regret of M ( E , Λ) can be decomposed as the sum of the regret suffered by the meta algorithm aggregation scheme, M , and the regret from the black-box algorithm, E , played with any learning rate λ ∈ Λ . The dynamic regret, defined in Eq. (5), can be decomposed, for any set of intervals I i = [ t i , t i +1 -1] , with 1 = t 1 < . . . < t m +1 = T +1 , and for any learning rate λ ∈ Λ , as

The black-box regret on I i is exactly the standard regret for T = |I i | with a learning rate of λ . Hence, in order to prove low dynamic regret for M ( E , Λ) we have to: show that M incurs a low dynamic regret in each interval I i ; find a black-box algorithm E for CURL that has dynamic regret as in Eq. (10), and build a learning rate grid Λ allowing us to perform nearly as well as the optimal learning rate.

4 MetaCURL Algorithm

We call our meta-algorithm M MetaCURL. It is based on sleeping experts, is parameter-free, and achieves optimal regret. Its construction is described below.

4.1 Learning with expert advice

General setting. In Learning with Expert Advice (LEA), a learner makes sequential online predictions u 1 , . . . , u T in a decision space U , over a series of T episodes, with the help of K experts [22, 36, 9]. For each round t , each expert k makes a prediction u t,k , and the learner combines the experts' predictions by computing a vector v t := ( v t, 1 , . . . , v t,K ) ∈ S K , and predicting the convex combination of experts' prediction u t := ∑ K k =1 v t,k u t,k . The environment then reveals a convex loss function ℓ t : U → R . Each expert suffers a loss ℓ t,k := ℓ t ( u t,k ) , and the learner suffers a loss ˆ ℓ t := ℓ t ( u t ) . The learner's objective is to keep the cumulative regret with respect to each expert as low as possible. For each expert k , this quantity is defined as Reg [ T ] ( k ) := ∑ T t =1 ˆ ℓ t -ℓ t,k .

Sleeping experts. In our case, each black-box algorithm is an expert that does not produce solutions outside its active interval. This problem can be reduced to the sleeping expert problem [8, 23], where experts are not required to provide solutions at every time step. Let I t,k ∈ { 0 , 1 } define a signal equal to 1 if expert k is active at episode t and 0 otherwise. The algorithm knows ( I t,k ) k ∈ [ K ] and assigns a zero weight to sleeping experts ( I t,k = 0 implies v t,k = 0 ). We would like to have a guarantee with respect to expert k ∈ [ K ] but only when it is active. Hence, we now aim to bound a cumulative regret that depends on the signal I t,k : Reg sleep [ T ] ( k ) := ∑ T t =1 I t,k ( ˆ ℓ t -ℓ t,k ) . There is a generic reduction from the sleeping expert framework to the general LEA setting [3, 2] (see Appendix A.1).

4.2 Meta-aggregation scheme

In every episode t , for every learning rate λ ∈ Λ and s ≤ t , an instance E s,λ of the black-box algorithm acts as an expert computing a policy π t,s,λ . The meta algorithm aims to aggregate these predictions using a sleeping expert approach based on the expert's losses. However, within CURL's framework, the meta algorithm faces two challenges:

Uncertainty. At the episode's end, the learner has full information about the objective function F t . If the learner also knew p t , they could recursively calculate the corresponding state-action distribution µ π t,s,λ ,p t using Eq. (2) and observe the actual loss of each expert, denoted as F t ( µ π t,s,λ ,p t ) . However, given that p t is unknown to the learner, the true loss remains unobservable. Consequently, the metaalgorithm needs to create an estimator ˆ p t for p t and utilize it to estimate the losses. We propose a method to compute an estimator ˆ p t in Subsection 4.3.

Convexity. As discussed in Section 2, the objective functions F t are not convex over the space of polices. However, CURL is equivalent to a convex problem over the state-action distributions satisfying the Bellman's flow as shown in Eq. (4). Therefore, instead of aggregating policies, the meta algorithm aggregates the associated state-action distributions using the probability estimator ˆ p t and the recursive scheme at Eq. (2). We detail MetaCURL in Alg. 1 when employed with the Exponentially Weighted Average forecaster (EWA) as the sleeping expert subroutine (we detail EWA in Appendix A.2).

4.3 Building an estimator of p t

As discussed earlier, applying the learning with experts framework requires estimating the loss of non-played expert policies, which depends on estimating the non-stationary transition probabilities ˆ p t . Standard RL techniques for bounding the L 1 norm between the empirical estimator ˆ p t and the true

Algorithm 1 MetaCURL with EWA

1: Input: number of episodes T , finite set of learning rates Λ , black-box algo. E , EWA learning rate η = √ 8 log( T ) T

2: Initialization: ˆ p 1 n ( ·| x, a ) := 1 |X| for all n ∈ [ N ] , ( x, a ) ∈ X × A

3: for t = 1 , . . . , T do

4: Start | Λ | new instances of E denoted by E t,λ for all λ ∈ Λ , assign each of them a new weight v t,t,λ = 1 | Λ | t , and normalize weight vectors v t,s,λ for s ∈ [ t -1] such that v t := ( v t,s,λ ) s ≤ t,λ ∈ Λ is a probability vector in R t × Λ

5: For s ≤ t and λ ∈ Λ , E s,λ outputs π t,s,λ

6: Compute recursively µ π t,s,λ , ˆ p t using Eq. (2) for all s ≤ t and λ ∈ Λ

7: Aggregate the state-action distributions: µ t := ∑ t s =1 ∑ λ ∈ Λ µ π t,s,λ , ˆ p t v t,s,λ

̸

8: Retrieve π t from µ t : for all n , ( x, a ) ,

9: Learner plays π t : Agent starts at ( x t 0 , a t 0 ) ∼ µ 0 ( · )

10: for n = 1 , . . . , N do

11: Environment draws new state x t n ∼ p t n ( ·| x t n -1 , a t n -1 )

12: Learner observes agent's external noise ε t n

13: Agent chooses an action a t n ∼ π t n ( ·| x t n )

14: end for

15: Objective function F t is exposed

16: Compute experts' losses ℓ t,s,λ := F t ( µ π t,s,λ , ˆ p t ) , for all s ≤ t and λ ∈ Λ

17: Compute the new weight vector v t +1 : for all s ≤ t and λ ∈ Λ ,

18: Use agent's external noise trajectory ( ε t n ) n ∈ [ N ] to compute ˆ p t +1 as in Subsection 4.3

19: end for

dynamics p t [44, 49] are not applicable here due to non-stationarity. To address this, we introduce a second layer of sleeping experts for each ( n, x, a ) ∈ [ N ] ×X ×A , where each expert provides an empirical estimate of p t based on different intervals. We then propose a new loss function in Eq. (12) and conduct a novel regret analysis in Prop. 5.2 to achieve the optimal regret rate.

In each episode t , the learner calculates independent samples x t n,x,a ∼ p t n ( ·| x, a ) utilizing the external noise sequence ( ε t n ) n ∈ [ N ] observed (just let x t n,x,a = g n -1 ( x, a, ε t n -1 ) , see Eq. (8)). Each expert outputs an empirical estimator of p t n ( ·| x, a ) using samples across different intervals. We assume T experts, with expert s active in interval [ s, T ] . Expert s at episode t > s outputs:

We let ˆ p t n ( ·| x, a ) be the result of employing sleeping EWA with experts ˆ p t,s n ( ·| x, a ) , for s < t . Typically, in density estimation with EWA, a logarithmic loss -log( · ) is used. However, in this case -log( · ) can be unbounded, so we opt here for a smoothed logarithmic loss, given by, for all q ∈ S X ,

The definition of this non-standard loss is further clarified during the regret analysis in Section 5. This loss function is 1 -exp concave (see Lemma 4 of [52]), hence the cumulative regret of EWA with respect to each expert s ∈ [ T ] , for all episodes τ ∈ [ s, T ] , satisfies Reg sleep [ s,τ ] ( s ) = ∑ τ t = s ℓ t (ˆ p t n ( ·| x, a )) -ℓ t (ˆ p t,s n ( ·| , x, a )) ≤ log( T ) (for more information on the regret

bounds of EWA with exp-concave losses, see Appendix A.2). We describe the complete online scheme to compute ˆ p t in Alg. 3 at Appendix B.

5 Regret analysis

This section presents the main result concerning MetaCURL's regret analysis. Subsection 5.1 shows an upper bound for R meta when MetaCURL is played with EWA and ˆ p t is computed as in Subsection. 4.3. Subsection 5.2 introduces a learning rate grid for MetaCURL when the black-box algorithm meets the dynamic regret criteria in Eq. (10), providing an upper bound for R black-box . Given the dynamic regret decomposition of Eq. (11), we see that the combination of these results leads to our main result, the full proof of which can be found in appendix (F) :

Theorem 5.1 (Main result) . Let δ ∈ (0 , 1) . Playing MetaCURL, with a parametric black-box algorithm E with dynamic regret as in Eq. (10) , with a learning rate grid Λ := { 2 -j | j = 0 , 1 , 2 , . . . , ⌈ log 2 ( T ) / 2 ⌉ } , and with EWA as the sleeping expert subroutine, we obtain, with probability at least 1 -2 δ , for any sequence of policies ( π t, ∗ ) t ∈ [ T ] ,

5.1 Meta-algorithm analysis

Given the uncertainty in the probability transition, the meta regret term can be decomposed as follows:

Sleeping LEA regret. Referring to Thm. A.1 in Appendix A, using sleeping EWA as the sleeping expert subroutine of MetaCURL, with signals I t,s = 1 for active experts ( s ≤ t ), experts' convex losses ℓ t,s,λ := F t ( µ π t,s,λ , ˆ p t ) , and learner loss ˆ ℓ t := F t ( µ π t , ˆ p t ) , yields, for any m ∈ [ T ] and for any set of intervals I i = [ t i , t i +1 -1] , with 1 = t 1 < . . . < t m +1 = T +1 ,

ˆ p t Estimation regret. In a scenario without uncertainty in the MDP's probability transitions, the meta-algorithm's regret would simply be bounded by Eq. (14), the sleeping expert regret used as a subroutine. However, given the presence of uncertainty, the main challenge in analyzing the meta-regret comes from the regret terms associated with the estimator ˆ p t . We outline this analysis in Prop. 5.2.

Proposition 5.2. Let δ ∈ (0 , 1) , C := √ 1 2 log ( N |X||A| 2 |X| T δ ) , and L F be the Lipschitz constant of F t , with respect to the norm ∥ · ∥ ∞ , 1 , for all t ∈ [ T ] . With a probability of at least 1 -δ , MetaCURL obtains

For any m ∈ [ T ] and for any set of intervals I i = [ t i , t i +1 -1] , with 1 = t 1 < . . . < t m +1 = T +1 , the same bound is valid for ∑ m i =1 ∑ t ∈I i R ˆ p I i ( π t,t i ,λ ) .

Proof. The proof idea is based mainly on the formulation of ˆ p t described in Subsection 4.3. We start by using the convexity of F t to linearize the expression, then we apply Holder's inequality and exploit the L F -Lipschitz property of F t to establish an upper bound based on the L 1 norm difference of the state-action distributions induced by the true probability transition and the estimator. Using Lemma C.5 in Appendix C, we then obtain that

To use the results from Subsection 4.3, we first regularize p t and ˆ p t , for each ( n, x, a ) , by averaging each with the uniform distribution over X , that we denote by p 0 := 1 / |X| . As both probabilities are now lower bounded, we can employ Pinsker's inequality to convert the L 1 norm into a KL divergence. The sum over t ∈ [ T ] of the KL divergence can then be decomposed as follows:

where ˆ p t,t i j ( ·| x, a ) is the empirical estimate of p t j ( ·| x, a ) calculated with the observed data from t i to t -1 , and the expectation is over ˜ x t j,x,a ∼ ( p t j ( ·| x, a ) + p 0 ) / 2 . The second term is the cumulative regret of computing ˆ p t using EWA with loss as in Eq. (12), and is bounded by m log( T ) . We finish and give more details of the proof in Appendix D.

Prop. 5.2 together with Eq. (14) yields the main result of this subsection:

Proposition 5.3 (Meta regret bound) . With the same assumptions as Prop. 5.2, for any m ∈ [ T ] , with probability at least 1 -2 δ ,

5.2 Black-box algorithm analysis

Assuming E is a parametric black-box algorithm with dynamic regret satisfying Eq. (10) for any learning rate λ > 0 , we only need to address the selection of the λ s grid and optimize across λ to achieve our final bound on R black-box [ T ] .

Learning rate grid. The dynamic regret of Eq. (10) implies that any two λ that are a constant factor of each other will guarantee the same upper-bound up to essentially the same constant factor. We therefore choose an exponentially spaced grid

The meta-algorithm aggregation scheme guarantees that the learner performs as well as the best empirical learning rate. We thus obtain a bound on R black-box [ T ] , with its proof in Appendix E:

Proposition 5.4 (Black-box regret bound) . Assume MetaCURL is played with a black-box algorithm satisfying dynamic regret as in Eq. (10) , with learning rate grid as in Eq. (15) . Hence, for any sequence of policies ( π t, ∗ ) t ∈ [ T ] ,

Greedy MD-CURL. Greedy MD-CURL, developed by [39], is a computationally efficient policyoptimization algorithm known for achieving sublinear static regret in online CURL with adversarial objective functions within a stationary MDP. In Thm. G.3 of Appendix G, we extend this analysis showing that Greedy MD-CURL also achieves dynamic regret as in Eq. (10). To our knowledge, this is the first dynamic regret result for a CURL algorithm. Hence, Greedy MD-CURL can be used as a black-box for MetaCURL. We detail Greedy MD-CURL in Alg. 4 in Appendix G.

6 Conclusion, discussion, and future work

In this paper, we present MetaCURL, the first algorithm for dealing with non-stationarity in CURL, a setting covering many problems in the literature that modifies the standard linear RL configuration, making typical RL techniques difficult to use. We also employ a novel approach to deal with nonstationarity in MDPs using the learning with expert advice framework from the online learning literature. The main difficulty in analyzing this method arises from uncertainty about probability transitions. We overcome this problem by employing a second expert scheme, and show that MetaCURL achieves near-optimal regret.

Compared to the RL literature, our approach is more efficient, deals with adversarial losses, and has a better regret dependency concerning the varying losses, but to do so, we need to simplify the assumptions about the dynamics (all uncertainty comes only from the external noise, that is independent of the agent's state-action). There seems to be a trade-off in RL: all algorithms dealing with both non-stationarity and full exploration use UCRL-type approaches, and are thus computationally expensive. We thus leave a question for future work: How can we effectively manage non-stationarity and adversarial losses using efficient algorithms, all while addressing full exploration?

References

P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning (ICML) , 2004.

D. Adamskiy, W. M. Koolen, A. Chernov, and V. Vovk. A closer look at adaptive regret. Journal of Machine Learning Research , 17(23):1-21, 2016.

D. Adamskiy, M. K. K. Warmuth, and W. M. Koolen. Putting bayes to sleep. In Advances in Neural Information Processing Systems (NeurIPS) , volume 25, 2012.

P. Auer, T. Jaksch, and R. Ortner. Near-optimal regret bounds for reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS) , volume 21, 2008.

A. Barakat, I. Fatkhullin, and N. He. Reinforcement learning with general utilities: simpler variance reduction and large state-action space. In International Conference on Machine Learning (ICML) , pages 1753-1800, 2023.

M. Bayati. Power management policy for heterogeneous data center based on histogram and discrete-time mdp. Electronic Notes in Theoretical Computer Science , 337:5-22, 2018. Proceedings of the Ninth International Workshop on the Practical Application of Stochastic Modelling (PASM).

A. Bensoussan, P. Yam, and J. Frehse. Mean Field Games and Mean Field Type Control Theory . SpringerBriefs in Mathematics. Springer, 2013.

A. Blum. Empirical support for winnow and weighted-majority based algorithms: results on a calendar scheduling domain. In A. Prieditis and S. Russell, editors, Machine Learning Proceedings 1995 , pages 64-72, 1995.

N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games . Cambridge University Press, 2006.

Y. Chen, S. S. Du, and K. Jamieson. Improved corruption robust algorithms for episodic reinforcement learning. In M. Meila and T. Zhang, editors, International Conference on Machine Learning (ICML) , volume 139, pages 1561-1570, 2021.

W. C. Cheung. Exploration-exploitation trade-off in reinforcement learning on online markov decision processes with global concave rewards. ArXiv , abs/1905.06466, 2019.

W. C. Cheung. Regret minimization for reinforcement learning with vectorial feedback and complex objectives. In Advances in Neural Information Processing Systems (NeurIPS) , volume 32, 2019.

W. C. Cheung, D. Simchi-Levi, and R. Zhu. Reinforcement learning for non-stationary Markov decision processes: The blessing of (More) optimism. In International Conference on Machine Learning (ICML) , volume 119, pages 1843-1854, 2020.

A. Cohen, Y. Efroni, Y. Mansour, and A. Rosenberg. Minimax regret for stochastic shortest path. In Advances in Neural Information Processing Systems (NeurIPS) , volume 34, pages 28350-28361, 2021.

A. Daniely, A. Gonen, and S. Shalev-Shwartz. Strongly adaptive online learning. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning (ICML) , volume 37, pages 1405-1411, 2015.

R. De Santi, M. Prajapat, and A. Krause. Global reinforcement learning : Beyond linear and convex rewards via submodular semi-gradient methods. In Proceedings of the 41st International Conference on Machine Learning , volume 235, pages 10235-10266, 2024.

O. D. Domingues, P. M'enard, M. Pirotta, E. Kaufmann, and M. Valko. A kernel-based approach to non-stationary reinforcement learning in metric spaces. In International Conference on Artificial Intelligence and Statistics (AISTATS) , volume 130, pages 3538-3546, 2021.

Y. Efroni, L. Shani, A. Rosenberg, and S. Mannor. Optimistic policy optimization with bandit feedback. In International Conference on Machine Learning (ICML) , volume 119, pages 8604-8613, 2020.

E. Even-Dar, S. M. Kakade, and Y. Mansour. Online markov decision processes. Mathematics of Operations Research , 34(3):726-736, 2009.

Y. Fei, Z. Yang, Z. Wang, and Q. Xie. Dynamic regret of policy optimization in non-stationary environments. In Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pages 6743-6754, 2020.

S. Feng, M. Yin, R. Huang, Y.-X. Wang, J. Yang, and Y. Liang. Non-stationary reinforcement learning under general function approximation. In International Conference on Machine Learning (ICML) , volume 202, pages 9976-10007, 2023.

Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences , 55(1):119-139, 1997.

Y. Freund, R. E. Schapire, Y. Singer, and M. K. Warmuth. Using and combining predictors that specialize. In Annual ACM Symposium on Theory of Computing (STOC) , page 334-343, 1997.

P. Gajane, R. Ortner, and P. Auer. A sliding-window algorithm for markov decision processes with arbitrarily changing rewards and transitions. ArXiv , abs/1805.10066, 2018.

M. Geist, J. Pérolat, M. Laurière, R. Elie, S. Perrin, O. Bachem, R. Munos, and O. Pietquin. Concave utility reinforcement learning: The mean-field game viewpoint. In International Conference on Autonomous Agents and Multiagent Systems , page 489-497, 2022.

S. K. S. Ghasemipour, R. Zemel, and S. Gu. A divergence minimization perspective on imitation learning methods. In Proceedings of the Conference on Robot Learning , volume 100, pages 1259-1277, 2020.

A. György, T. Linder, and G. Lugosi. Efficient tracking of large classes of experts. In 2012 IEEE International Symposium on Information Theory Proceedings , pages 885-889, 2012.

E. Hazan, S. Kakade, K. Singh, and A. Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning (ICML) , volume 97, pages 2681-2691, 2019.

E. Hazan and C. Seshadhri. Adaptive algorithms for online decision problems. Electronic Colloquium on Computational Complexity (ECCC) , 14, 01 2007.

C. Jin, T. Jin, H. Luo, S. Sra, and T. Yu. Learning adversarial Markov decision processes with bandit feedback and unknown transition. In International Conference on Machine Learning (ICML) , volume 119, pages 4860-4869, 2020.

T. Jin, J. Liu, C. Rouyer, W. Chang, C.-Y. Wei, and H. Luo. No-regret online reinforcement learning with adversarial losses and transitions. In Advances in Neural Information Processing Systems (NeurIPS) , volume 36, pages 38520-38585, 2023.

T. Jin and H. Luo. Simultaneously learning stochastic and adversarial episodic mdps with known transition. In Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pages 16557-16566, 2020.

K.-S. Jun, F. Orabona, S. Wright, and R. Willett. Improved Strongly Adaptive Online Learning using Coin Betting. In International Conference on Artificial Intelligence and Statistics (AISTATS) , volume 54, pages 943-951, 2017.

P. Lavigne and L. Pfeiffer. Generalized conditional gradient and learning in potential mean field games. Applied Mathematics & Optimization , 88(3):89, Oct 2023.

J. W. Lavington, S. Vaswani, and M. Schmidt. Improved policy optimization for online imitation learning. In Proceedings of The 1st Conference on Lifelong Learning Agents , volume 199, pages 1146-1173, 2022.

N. Littlestone and M. Warmuth. The weighted majority algorithm. Information and Computation , 108(2):212-261, 1994.

H. Luo, C.-Y. Wei, and C.-W. Lee. Policy optimization in adversarial mdps: Improved exploration via dilated bonuses. In Advances in Neural Information Processing Systems (NeurIPS) , volume 34, pages 22931-22942, 2021.

T. Lykouris, M. Simchowitz, A. Slivkins, and W. Sun. Corruption-robust exploration in episodic reinforcement learning. In Conference on Learning Theory (COLT) , volume 134, pages 32423245, 2021.

B. M Moreno, M. Bregere, P. Gaillard, and N. Oudjane. Efficient model-based concave utility reinforcement learning through greedy mirror descent. In International Conference on Artificial Intelligence and Statistics (AISTATS) , volume 238, pages 2206-2214, 2024.

W. Mao, K. Zhang, R. Zhu, D. Simchi-Levi, and T. Basar. Near-optimal model-free reinforcement learning in non-stationary episodic mdps. In International Conference on Machine Learning (ICML) , volume 139, pages 7447-7458, 2021.

B. Marin Moreno, M. Brégère, P. Gaillard, and N. Oudjane. (Online) Convex Optimization for Demand-Side Management: Application to Thermostatically Controlled Loads, Jan. 2023.

M. Mutti, R. De Santi, P. De Bartolomeis, and M. Restelli. Challenging common assumptions in convex reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS) , volume 35, pages 4489-4502, 2022.

M. Mutti, R. D. Santi, P. D. Bartolomeis, and M. Restelli. Convex reinforcement learning in finite trials. Journal of Machine Learning Research , 24(250):1-42, 2023.

G. Neu, A. Gyorgy, and C. Szepesvari. The adversarial stochastic shortest path problem with unknown transition probabilities. In International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 805-813. PMLR, 2012.

R. Ortner, P. Gajane, and P. Auer. Variational regret bounds for reinforcement learning. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference , volume 115, pages 81-90, 2020.

M. Prajapat, M. Mutny, M. Zeilinger, and A. Krause. Submodular reinforcement learning. In International Conference on Learning Representations (ICLR) , 2024.

A. Raj, P. Gaillard, and C. Saad. Non-stationary online regression, 2020.

A. Rosenberg and Y. Mansour. Online convex optimization in adversarial Markov decision processes. In International Conference on Machine Learning (ICML) , volume 97, pages 5478-5486, 2019.

A. Rosenberg and Y. Mansour. Online convex optimization in adversarial Markov decision processes. In International Conference on Machine Learning (ICML) , pages 5478-5486, 2019.

A. Rosenberg and Y. Mansour. Stochastic shortest path with adversarially changing costs. In International Joint Conference on Artificial Intelligence (IJCAI) , pages 2936-2942, 2021.

A. Séguret, C. Wan, and C. Alasseur. A mean field control approach for smart charging with aggregate power demand constraints. In 2021 IEEE PES Innovative Smart Grid Technologies Europe (ISGT Europe) , pages 01-05, 2021.

D. van der Hoeven, N. Zhivotovskiy, and N. Cesa-Bianchi. High-probability risk bounds via sequential predictors, 2023.

C.-Y. Wei, C. Dann, and J. Zimmert. A model selection approach for corruption robust reinforcement learning. In International Conference on Algorithmic Learning Theory (ALT) , volume 167, pages 1043-1096, 2022.

C.-Y. Wei and H. Luo. Non-stationary reinforcement learning without prior knowledge: an optimal black-box approach. In Conference on Learning Theory (COLT) , volume 134, pages 4300-4354, 2021.

T. Zahavy, A. Cohen, H. Kaplan, and Y. Mansour. Apprenticeship learning via frank-wolfe. In AAAI Conference on Artificial Intelligence , 2019.

T. Zahavy, B. O' Donoghue, G. Desjardins, and S. Singh. Reward is enough for convex mdps. In Advances in Neural Information Processing Systems (NeurIPS) , volume 34, pages 25746-25759, 2021.

J. Zhang, A. Koppel, A. S. Bedi, C. Szepesvari, and M. Wang. Variational policy gradient method for reinforcement learning with general utilities. In Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pages 4572-4583, 2020.

J. Zhang, C. Ni, z. Yu, C. Szepesvari, and M. Wang. On the convergence and sample efficiency of variance-reduced policy gradient method. In Advances in Neural Information Processing Systems (NeurIPS) , volume 34, pages 2228-2240, 2021.

L. Zhang, T. Yang, rong jin, and Z.-H. Zhou. Dynamic regret of strongly adaptive methods. In International Conference on Machine Learning (ICML) , volume 80, pages 5882-5891, 2018.

X. Zhang, Y. Chen, X. Zhu, and W. Sun. Robust policy gradient against strong data corruption. In International Conference on Machine Learning (ICML) , volume 139, pages 12391-12401, 2021.

A. Zimin and G. Neu. Online learning in episodic markovian decision processes by relative entropy policy search. In Advances in Neural Information Processing Systems (NeurIPS) , volume 26, pages 1583-1591, 2013.

A Learning with expert advice

In this section, we take a closer look at the Learning with Expert Advice (LEA) framework. We start by presenting, in Subsection A.1, a general reduction of the sleeping experts framework to the standard framework. Thus, any LEA algorithm can be used as a sub-routine for MetaCURL. In Section 5 of the main paper, we show a regret bound for MetaCURL using the Exponentially Weighted Average Forecaster (EWA) algorithm [9], also known as Hedge. In Subsection A.2 we present the main results of playing EWA with convex and exp-concave losses.

Setting. We recall the general setting of learning with expert advice (LEA) as presented in the main paper: a learner makes sequential online predictions u 1 , . . . , u T in a decision space U , over a series of T episodes, with the help of K experts. For each round t , each expert k makes a prediction u t,k , and the learner combines the experts' predictions by computing a vector v t := ( v t, 1 , . . . , v t,K ) ∈ S K , and predicting u t := ∑ K k =1 v t,k u t,k . The environment then reveals a convex loss function ℓ t : U → R . Each expert suffers a loss ℓ t,k := ℓ t ( u t,k ) , and the learner suffers a loss ˆ ℓ t := ℓ t ( u t ) . The learner's objective is to keep the cumulative regret with respect to each expert as low as possible. For each expert k , this quantity is defined as Reg [ T ] ( k ) := ∑ T t =1 ˆ ℓ t -ℓ t,k .

A.1 Sleeping experts

The sleeping expert problem [8, 23] is a LEA framework where experts are not required to provide solutions at every time step. Let I t,k ∈ { 0 , 1 } define a binary signal that equals 1 if expert k is active at episode t and 0 otherwise. The algorithm knows ( I t,k ) k ∈ [ K ] and assigns a zero weight to sleeping experts. We would like to have a guarantee with respect to expert k ∈ [ K ] but only when it is active. Hence, we now aim to bound a cumulative regret that depends on the signal I t,k : Reg sleep [ T ] ( k ) := ∑ T t =1 I t,k ( ˆ ℓ t -ℓ t,k ) . We present a generic reduction from the sleeping expert framework to the standard LEA framework [3, 2]:

Let, for all episodes t ∈ [ T ] ,

We play a standard LEA algorithm with modified outputs where, at episode t , expert k outputs

A standard LEA algorithm gives an upper bound on the regret Reg T ( k ) with respect to each expert k . Using that ∑ K k =1 ˜ u t,k v t,k = ˆ u t , we obtain that

Consequently, the cumulative regret with respect to each expert during the times it is active is upper bounded by the standard regret of playing a LEA algorithm with the modified outputs.

A.2 Exponentially Weighted Average forecaster

The exponentially weighted average forecaster (EWA), also called Hedge, is a LEA algorithm that chooses a weight that decreases exponentially fast with past errors. We present EWA in Alg. 2.

Algorithm 2 EWA (Exponentially Weighted Average)

Input: [ K ] := { 1 , . . . , K } a finite set of experts, v 0 a prior over [ K ] , a learning rate η > 0 for t ∈ { 1 , . . . , T } do

Observe loss function ℓ t , compute the loss suffered by each expert ℓ t,k := ℓ t ( u t,k ) and suffer loss ˆ ℓ t := ℓ ( ∑ K k =1 v t,k u t,k )

Update for all k ∈ [ K ] :

end for

We recall two results of playing EWA with convex losses, and with exp-concave losses, used in the main paper:

Theorem A.1 (EWA with convex losses: Corollary 2 . 2 from [9]) . If the ℓ t losses are convex and take value in [0 , 1] , then the regret of the learner playing EWA with any η > 0 satisfies, for any k ∈ [ K ] ,

In particular, with η = √ 8 log( K ) /T , the upper bound becomes √ ( T/ 2) log( K ) .

Theorem A.2 (EWA with exp-concave losses: Thm. 3 . 2 from [9]) . If the ℓ t losses are η -exp concave, then the regret of the learner playing EWA (with the same value of η ) satisfies, for any k ∈ [ K ] ,

B Algorithm for the online estimation of the probability kernel ( ˆ p t estimator)

Algorithm 3 Online estimation of the probability kernel ( ˆ p t estimator)

1: for t ∈ { 1 , . . . , T } do

2: Get agent's external noise trajectory ( ε t n ) n ∈ [ N ] from MetaCURL

3: for ( n, x, a ) ∈ [ N ] ×X ×A do

4: Compute x t n,x,a = g n -1 ( x, a, ε t n -1 )

5: Update the empirical estimations for s < t and x ′ ∈ X :

6: Initialize a new estimator, ˆ p t,t n ( x ′ | x, a ) = 1 { x t n,x,a = x ′ } for all x ′ ∈ X , assign it a new weight vector v t,t n,x,a = 1 t , and normalize weight vectors v t,s n,x,a for s ∈ [ t -1] such that v t n,x,a := ( v t,s n,x,a ) s ≤ t,λ ∈ Λ is a probability vector in R t

7: Simulate a sample ˜ x t n,x,a from distribution ( p t n ( ·| x, a ) + 1 |X| ) / 2 :

and use it to build the loss function

8: Update weights using EWA with loss ℓ t : for all s ≤ t ,

9: Compute ˆ p t +1 n ( ·| x, a ) = ∑ t s =1 v t +1 ,s n,x,a ˆ p t,s n ( ·| x, a )

10: end for

11: Issue ˆ p t +1 to MetaCURL (line 18 of Alg. 1)

12: end for

C Auxiliary lemmas

We start with some auxiliary results. For t ∈ I := [ t s +1 , t e ] ⊆ [ T ] , we define the average probability distribution for all n and ( x, a ) as

Lemma C.1. Let ˆ p t,t s be the empirical probability transition kernel computed with data from episodes [ t s , t -1] . For any δ ∈ (0 , 1) , with probability 1 -δ ,

simultaneously for all n ∈ [ N ] , ( x, a ) ∈ X × A , t s ∈ [ T -1] , and t ∈ [ t s +1 , T ] .

Proof. For a fixed n ∈ [ N ] , ( x, a ) ∈ X × A , and θ ∈ {-1 , 1 } |X| , we define for all s ∈ I ,

a Bernoulli random variable with mean value given by ∑ x ′ ∈X θ ( x ′ ) p s n ( x ′ | x, a ) .

The sequence of random variables given by ( Y s n,x,a,θ ) s ∈ I is independent, as we assume that the external noises observed at each episode are all independent. Hence, by Hoeffding's inequality we get that, for all ξ > 0 ,

Therefore, we have that

By applying an union bound on all n ∈ [ N ] , ( x, a ) ∈ X × A , and θ ∈ {-1 , 1 } |X| and noting that, for any two probability distributions p, q ∈ ∆ X , we have that

we arrive at the final result.

Lemma C.2. Let t ∈ I := [ t s +1 , t e ] ⊆ [ T ] . For all n ∈ [ N ] , and ( x, a ) ∈ X × A ,

Proof. For t ∈ I , and for all n and ( x, a ) we have that

where recall that we define ∆ p j := max n,s,a ∥ p j +1 n ( ·| x, a ) -p j n ( ·| x, a ) ∥ 1 .

Lemma C.3. Let ˆ p t,t s be the empirical probability transition kernel computed with data from episodes [ t s , t -1] . For any δ ∈ (0 , 1) , with probability 1 -δ ,

simultaneously for all n ∈ [ N ] , ( x, a ) ∈ X × A , t s ∈ [ T -1] , and t ∈ [ t s +1 , T ] .

Proof. The result follows immediately from the triangular inequality and Lemmas C.1 and C.2.

Lemma C.4 (A version of the inverse of Pinsker's inequality) . Let p ′ , q ′ be any distributions over S X . Define

Proof. First, note that q is lower bounded by 1 2 |X| , hence KL ( p | q ) is well defined. Also, by convexity of the simplex, p, q ∈ S X , therefore

Lemma C.5. For any strategy π ∈ ( S A ) X× N , for any two probability kernels p = ( p n ) n ∈ [ N ] and q = ( q n ) n ∈ [ N ] such that p n , q n : X × A × X → [0 , 1] , and for all n ∈ [ N ] ,

Proof. From the definition of a state-action distribution sequence induced by a policy π in a MDP with probability kernel p in Eq. (2), we have that for all ( x, a ) ∈ X × A and n ∈ [ N ] ,

Therefore,

Thus,

Since for n = 0 , ∥ µ π,p 0 -µ π,q 0 ∥ 1 = 0 , by induction we get that

Lemma C.6. For any pair of strategies π, π ′ ∈ (∆ A ) X× N , for any probability kernel p = ( p n ) n ∈ [ N ] such that p n : X × A × X → [0 , 1] , and for all n ∈ [ N ] ,

where ρ π,p i ( x ) := ∑ a ∈A µ π,p i ( x, a ) for all x ∈ X and i ∈ [ N ] .

Proof. Using the recursive relation from Eq. (2) of a state-action distribution induced by a policy π in a MDP with probability transition p we have that

Since µ 0 is fixed for each state-action distribution sequence, by induction we obtain that

completing the proof.

D Proof of Prop. 5.2: R ˆ p [ T ] regret analysis

Proof. Here, we set an upper bound on the term R ˆ p [ T ] where we pay for errors in estimating p t by ˆ p t .

To obtain the first inequality, we use the convexity of F t for all t ∈ [ T ] , then we use Holder's inequality and the fact that F t is L F -Lipschitz, and for the last inequality we use Lemma C.5.

The difficulty in analyzing the L 1 difference between p t and ˆ p t arises from the non-stationarity of p t . To overcome this we want to use the scheme presented in Subsection 4.3. By Cauchy-Schwartz, we get that

Analysis of the L 1 norm. We start by analysing the sum over t ∈ [ T ] of the L 1 norm in term ( ∗ ) . For each n ∈ [ N ] , j ∈ [ n ] and ( x, a ) ∈ X × A ,

where we apply Pinsker's inequality.

Consider a sequence of episodes 1 = t 1 < t 2 < . . . < t m +1 = T +1 , with I i := [ t i , t i +1 -1] , such that ∆ p I i ≤ ∆ p /m for all i ∈ [ m ] . Decomposing the KL sum over t ∈ [ T ] as a sum on the intervals I i , we obtain that

where the expectation of the second term is with respect to ˜ x t j,x,a ∼ ( p t j ( ·| x, a ) + 1 |X| ) / 2 .

We analyse each term separately:

First, note that ( ii ) is the expectation over ˜ x t j,x,a of the cumulative regret of sleeping EWA on interval I i with respect to the expert t i using the loss function ℓ t defined in Eq. (12). This term is upper bounded by log( T ) (see Subection 4.3 of the main paper). From it we deduce that

Regarding term ( i ) , we start by using the inverse of Pinsker's inequality presented in Lemma C.4,

To simplify notations, from now on we let C := √ 1 2 log ( N |X||A| 2 |X| T δ ) . Applying Lemma C.3, we obtain that

Joining the upper bounds of ( i ) and ( ii ) we conclude that

Back to the analysis of R ˆ p [ T ] . Using the inequality in Eq. (17) to bound the L 1 norm of ( ∗ ) on Eq. (16), we obtain that

concluding the proof.

Note that for ∑ m i =1 ∑ t ∈I i R ˆ p I i ( π t,t i ,λ ) (the third term of the meta-regret decomposition of Eq. (13)), following the same procedure as above, we obtain that

Since the second term is independent of π t,t i , the analysis is the same as before and we obtain the same upper bound as for R ˆ p [ T ] ( π t ) .

E Proof of Prop. 5.4: R black-box [ T ] regret analysis

Proof. Assume a Black-box algorithm satisfying the dynamic regret bound of Eq. (10), i.e., for any interval I ⊆ [ T ] , with respect to any sequence of policies ( π t, ∗ ) t ∈ I , and for any learning rate λ ,

Consider any sequence of episodes 1 = t 1 < t 2 < .. . < t m +1 = T + 1 , forming intervals I i := [ t i , t i +1 -1] for all i ∈ [ m ] . We can decompose the black-box regret over [ T ] as

In principle, we would like to select the optimal λ that optimizes this dynamic regret. However, as ∆ π ∗ may be unknown in advance, this is not possible. We show here that running MetaCURL with the learning rate set Λ := { 2 -j | j = 0 , 1 , 2 , . . . , ⌈ [log 2 ( T ) / 2 ⌉} ensures that the optimal empirical learning rate is close to the true optimal one by a factor of 2 and that the learner always plays as well as the optimal empirical learning rate.

Denote by λ ∗ the optimal learning rate and ˆ λ ∗ the empirical optimal learning rate in Λ . Note that

We consider three different cases:

If λ ∗ ≥ 1 : this implies that c 2 ∆ π ∗ + c 3 c 1 T ≥ 1 . Therefore, we have that T ≤ c 2 ∆ π ∗ + c 3 c 1 . As we assume f t n ∈ [0 , 1] for all time steps n ∈ [ N ] and episodes t ∈ [ T ] , then

If λ ∗ ≤ 1 / √ T : this implies that c 2 ∆ π ∗ + c 3 c 1 ≤ 1 . Therefore, taking ˆ λ ∗ = 1 / √ T ∈ Λ , we have that

If λ ∗ ∈ [1 / √ T, 1] : in this case, given the construction of Λ , there is a ˆ λ ∗ ∈ Λ such that λ ∗ ≤ ˆ λ ∗ ≤ 2 λ ∗ . Hence,

Therefore, by taking λ = ˆ λ ∗ in the analysis, we can ensure that

F Proof of Thm. 5.1: Main result

Joining the results from the meta-regret bound and the black-box regret bound, we get the main result of the paper:

Theorem (Main result) . Let δ ∈ (0 , 1) . Playing MetaCURL, with black-box algorithm with dynamic regret as in Eq. (10) , with a learning rate grid Λ := { 2 -j | j = 0 , 1 , 2 , . . . , ⌈ 1 / 2 log 2 ( T ) ⌉ } , and with EWA as the sleeping expert subroutine, we obtain, with probability at least 1 -2 δ , for any sequence of policies ( π t, ∗ ) t ∈ [ T ] ,

Proof. Define a sequence of episodes 1 = t 1 < t 2 < . . . < t m +1 = T +1 , with I i := [ t i , t i +1 -1] , such that ∆ p I i ≤ ∆ p /m for all i ∈ [ m ] .

The dynamic regret of M ( E , Λ) with respect to any sequence of policies ( π t, ∗ ) t ∈ [ T ] , and any λ ∈ Λ , can be decomposed as

From Prop. 5.3, we have that with probability at least 1 -2 δ , and C := √ 1 2 log ( N |X||A| 2 |X| T δ ) ,

In addition, for Λ := { 2 -j | j = 0 , 1 , 2 , . . . , ⌈ 1 / 2 log 2 ( T ) ⌉ } , and λ equal the best empirical learning rate in Λ , Prop. 5.4 yields that, if the black-box algorithm has dynamic regret as in Eq. (10) for any interval in T , then

Therefore, joining both results, we get that,

G Greedy MD-CURL dynamic regret analysis

Here we introduce Greedy MD-CURL developed by [39], a computationally efficient policyoptimization algorithm known for achieving sublinear static regret in online CURL with adversarial objective functions within a stationary MDP. We begin by detailing Greedy MD-CURL as presented in [39] in Alg. 4. We then provide a new analysis upper bounding the dynamic regret of Greedy MD-CURL in a quasi-stationary interval valid for any learning rate λ . Hence, Greedy MD-CURL can be used as a black-box for MetaCURL. This is the first dynamic regret analysis for a CURL approach.

̸

Let M p, ∗ µ 0 denote the subset of M p µ 0 where the corresponding policies π are such that π n ( a | x ) = 0 for all ( x, a ) ∈ X × A , n ∈ [ N ] . For any probability transition p , Γ : M p µ 0 ×M p, ∗ µ 0 → R such that, for all µ ∈ M p µ 0 with its associated policy π , and µ ′ ∈ M p, ∗ µ 0 with its associated policy π ′ , we have

This divergence Γ is a Bregman divergence (see Proposition 4.3 of [39]). Problem (20) implemented with this Bregman divergence Γ has a closed form solution, as showed in [39].

Algorithm 4 Greedy MD-CURL

1: Input: number of episodes T , initial sequence of policies π 1 ∈ ( S A ) X× N , initial state-action distribution µ 0 , learning rate λ > 0 , sequence of parameters ( α t ) t ∈ [ T ] .

2: Initialization: ∀ ( x, a ) , ˆ p 1 ( ·| x, a ) = 1 |X| and µ 1 = ˜ µ 1 := µ π 1 , ˆ p 1

3: for t = 1 , . . . , T do

4: Agent starts at ( x t 0 , a t 0 ) ∼ µ 0 ( · )

5: for n = 1 , . . . , N do

6: Environment draws new state x t n ∼ p n ( ·| x t n -1 , a t n -1 )

7: Learner observes agent's external noise ε t n

8: Agent chooses an action a t n ∼ π t n ( ·| x t n )

9: end for

10: Update probability kernel estimate for all ( x, a ) :

11:

12: Compute policy for the next episode:

13:

14: for all n ∈ [ N ] , ( x, a ) ∈ X × A ,

15: Compute ˜ π t +1 := (1 -α t +1 ) π t +1 + α t +1 / |A|

16: Compute ˜ µ t +1 := µ ˜ π t +1 , ˆ p t +1 as in Eq. (2)

17: end for

18: return ( π t ) t ∈ [ T ]

G.1 Dynamic regret analysis of Greedy MD-CURL

Let us assume we analyze our regret in an interval I := [ t s , t e ] ⊆ [ T ] . We denote by R I the dynamic regret of an instance of Greedy MD-CURL started at episode t s until the end of interval I at episode t e . We denote by π t the policy produced by this instance of Greedy MD-CURL at episode t ∈ I , p t the true probability transition kernel, and

the empirical estimate of the probability kernel at episode t , with data from the beginning of the interval I .

We define and decompose the dynamic regret R I with respect to any sequence of policies ( π t, ∗ ) t ∈ I into three terms as follows:

The terms R MDP I ( π t ) and R MDP I ( π t, ∗ ) pay for our lack of knowledge of the true MDP, forcing us to use its empirical estimate. The term R policy I corresponds to the loss incurred in calculating the policy

by solving the optimization problem given in Eq. (20). Below, we present the first analysis of the dynamic regret for a CURL algorithm. We consider each term separately.

G.1.1 R MDP I analysis

In Section 2 we assume that the deterministic part of the dynamics, given by g n in equation (8) for each time step n , is known in advance. The source of uncertainty and non-stationarity in the MDP comes only from the external noise dynamics, that is independent of the agent's state-action pair. Therefore, we do not need to explore in this setting, so the analysis of the two terms R MDP I ( π t ) and R MDP I ( π t, ∗ ) are the same.

Proposition G.1. With probability at least 1 -δ ,

for all intervals I ∈ [ T ] . The same result is valid for R MDP I ( π t, ∗ ) .

Proof. We start by using the convexity of F t , Holder's inequality, that F t is L F -Lipschitz, and Lemma C.5 to obtain that

Applying Lemmas C.1 and C.2, we have that for any δ ∈ (0 , 1) , with probability 1 -δ ,

Using this to continue the upper bound of Eq. (22), we conclude our proof:

G.1.2 R policy I analysis

Proposition G.2. Let b be a constant defined as

Then, Greedy MD-CURL obtains, for any sequence of policies ( π t, ∗ ) t ∈ I , and for any learning rate λ > 0 ,

Proof. We adapt the proof of Prop. 5 . 7 of [39] that upper bounds the static regret incurred when solving the optimization Problem (20), for a proof that upper bounds the dynamic regret. The main difference is that, in the case of static regret, we compare ourselves to the same policy throughout the interval, whereas in the case of dynamic regret, at each episode we compare ourselves to a different policy given by π t, ∗ . Consequently, the analysis remains the same as in [39] for all terms that do not depend on π t, ∗ , but requires a new analysis in terms that do depend on it.

To simplify notation, we take ℓ t := ∇ F t ( µ π t , ˆ p t ) and µ t := µ π t , ˆ p t . We can use the same reasoning as in appendix D. 5 of [39] to show that

Since the A term does not depend on π t, ∗ , its analysis follows directly from [39], and is given by

where L F is the Lipschitz constant of F t and α t is an input parameter of Greedy MD-CURL.

We then proceed to analyze term B . Again, following the procedure of appendix D. 5 of [39], we obtain that

Let ψ : ( S X×A ) N → R denote the function inducing the Bregman divergence Γ of Eq. (19). [39] further shows that:

( ii ) ≤ 2 N ∑ t ∈ I α t , and this upper bound is found independently of π t, ∗

Only term ( i ) , which involves ∥ µ π ∗ ,t -1 , ˆ p t -µ π ∗ ,t , ˆ p t +1 ∥ ∞ , 1 , depends on the sequence ( π t, ∗ ) t ∈ [ T ] , requiring then a new analysis. For this purpose, we rely on the following two results:

From Lemma 5 . 6 of [39], we have that, for all strategies π ,

From auxiliary Lemma C.6 proved in Appendix C we have that

Therefore, using the triangular inequality and the two results above, we obtain that

Therefore, the bound on term B is given by

Final step: joining all results

Joining the upper bounds on term A from Eq. (23) and on term B from Eq. (24), we have that

If we take the learning rate as α t = 1 /t , then, for all λ > 0 ,

where b := 8 N 2 +2 N 2 log( |A| ) log( | I | ) ( 1 + log( | I | ) ) +2 N log( | I | ) + N log( |A| ) .

G.2 Final Greedy MD-CURL regret analysis

Replacing the bounds of Prop. G.1 and G.2 in Eq. (21) yields the final upper bound of Greedy MDCURL's dynamic regret for any interval I ⊆ T with respect to any sequence of policies ( π t, ∗ ) :

t ∈ I Theorem G.3 (Dynamic regret of Greedy MD-CURL) . Let b be a constant defined as

Let δ ∈ (0 , 1) . With probability at least 1 -2 δ , for any interval I ⊆ [ T ] , for any sequence of policies ( π t, ∗ ) t ∈ I , for any learning rate λ > 0 , and for α t := 1 /t , Greedy MD-CURL obtains

Hence, Greedy MD-CURL meets the requisite dynamic regret bound from Eq. (10) to serve as a black-box algorithm for MetaCURL achieving optimal dynamic regret.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly provide the contributions of our work and our settings in the Introduction. We formally detail our hypothesis in Section 2. We present the new algorithm in Section 4. All regret results related to the new algorithm claimed in the abstract and introduction are stated in Section 5 and proved in the Appendix.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Section 2, we formally detail our assumptions and discuss the strong assumption made about probability transitions, justifying it as a means of providing lowcomplexity methods. We also discuss real-world applications that satisfy these assumptions. In Section 6, we discuss the aspirational goal of addressing this limitation as a future work. In Remark 3.1, we discuss the complexity of the algorithm.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the main results are in the paper, and all proofs are carefully stated in the Appendix. All auxiliary results used to obtain the main results are also included in the Appendix. A proof sketch is provided for the main results in the main paper. All results are properly referenced.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This is a theoretical paper, with no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This is a theoretical paper, with no experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This is a theoretical paper, with no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This is a theoretical paper, with no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This is a theoretical paper, with no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This work conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The results presented in this paper are largely theoretical. The framework provided in this paper is very general and could be applied to any reinforcement learning or concave utility reinforcement learning problem in a tabular MDP. Therefore, as with any reinforcement learning algorithm, it is possible that the algorithms developed from the ideas presented in this paper could be applied in contexts that have negative societal impacts, or in contexts where the reward function has a negative negative societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses concave utility RL in a non-stationary episodic setting, where the transition probabilities as well as the utility function may change from one episode to the other. The paper proposes an algorithm, dubbed MetaCURL, which dynamically select the best performing \"expert\" within a set of baseline algorithms with different starting points and hyper-parameters. The paper analyses the dynamic regret of MetaCURL as a function of some characteristics of the instance, such as the number of abrupt changes and the magnitude of changes, under a deterministic dynamics plus action-independent noise assumption.\n\nSTRENGTHS:\n- Novelty of the setting. Concave utility RL has been studied extensively recently, but I am not aware of any work combining concave utilities and non-stationarity;\n- Simplicity of the approach. The expert-based solution is neat and easy to follow.\n\nWEAKNESSES:\n- Motivation of the setting (1). The paper does not do much to support why studying non-stationarity in CURL is useful/interesting;\n- Motivation of the setting (2). The paper only evaluates the performance of the expected realization rather than the expected performance;\n- Restrictive assumptions. The setting presents several challenges but does not require exploration due to an assumption that makes the stochasticity of the transitions independent from the actions;\n- Analysis. It is unclear whether the analysis brings some interesting techniques beyond what has been done in prior works.\n\nThis paper tackles a setting comprising an eye-popping list of challenges, which include: non-stationarity of transitions, adversarial learning, concave cost functions. However, aside from restrictive assumptions that makes strategic exploration unnecessary, I believe the paper somewhat fails in two aspects: First, to motivate the setting properly, not just as a \"patchwork\" of other settings considered in the literature but as a problem raising from important application; Second, to highlight what is the value of their results in terms of insights and employed techniques.\nTo these reasons, I am currently providing a slightly negative evaluation. I would like to hear from the authors on these concerns. More details below.\n\nQUESTIONS:\nMOTIVATION: NON-STATIONARY CURL.\n\nBoth CURL and non-stationary MDPs are interesting and worth studying, but why the combination of CURL and non-stationarity is interesting? Does the combination brings more challenges than just the sum of its components? Does the algorithm bring novel ideas? Does the analysis bring novel techniques or valuable insights? I feel that the paper shall do more to support why the reported results matter.\n\nMOTIVATION: OCCUPANCY VS SINGLE-TRIAL.\n\nAs it is typical in the original formulation, the paper defines the CURL objective as a concave function of the occupancy measure, i.e., the expected state visitations. More recently (see Mutti et al., Challenging common assumptions in convex reinforcement learning, 2022), it has been shown that there is a crucial difference between optimizing the concave objective over the expected visitation and the expected concave objective. Can the authors comment on why the occupancy formulation is relevant in this setting?\n\nMOTIVATION: NO EXPLORATION.\n\nStudying the regret in a setting in which strategic exploration is unnecessary looks a bit odd. Also, the provided motivation is that applying optimism would induce computational issues, which I do not find fully convincing. There exist tractable no-regret algorithms in RL and it shall be shown why it is not the case for CURL. Moreover, plenty of works in RL studies the regret by relying on planning oracles that are intractable to implement. This is a common choice to isolate statistical complexity from computational considerations. I think that considering $g_n$ unknown but belonging to a known function class would make for a significantly more interesting analysis.\n\nNOVELTY OF THE ANALYSIS.\n\nCan the authors highlight what is novel in the analysis? From reading the paper, the main effort seems to reduce the analysis to a combination of EWA and a known rate for the baseline algorithm.\n\nADDITIONAL REFERENCES.\n\nWhile the work does a good job of placing the contribution with respect to previous literature in non-stationary RL, there are some missing works that seem relevant. As mentioned above, the papers *(Mutti et al., Challenging common assumptions in convex reinforcement learning, 2022; Mutti et al., Convex reinforcement learning in finite-trials, 2023)* study a so-called \"single-trial\" variation of the CURL formulation. The works *(Cheung, Regret minimization for reinforcement learning with vectorial feedback and complex objectives, 2019; Cheung, Exploration-exploitation trade-off in reinforcement learning on online markov decision processes with global concave rewards, 2019)* analyse regret-minimization in a problem formulation akin to an infinite-horizon form of CURL. The papers *(Prajapat et al., Submodular reinforcement learning, 2023; De Santi et al., Global reinforcement learning, 2024)* look also closely related.\n\nOTHER COMMENTS.\n- Examples of applications: the paper makes an effort of describing a few applications that fulfil the dynamics assumption, not why they are  relevant for CURL.\n- Does the analysis need $F_t$ to be fully revealed instead of bandit feedback? If that is the case, can the authors comment on how this assumption may be overcome?\n- The considered policies seem to depend only on the current state of the environment (aka Markovian). Can the author discuss why history-based policies are unnecessary in this setting?\n\nLIMITATIONS:\nThe paper is mostly upfront in mentioning the limitations introduced by restrictive assumptions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on CURL, i.e., the Concave Utility Reinforcement Learning problem, which can be treated an extension of traditional RL  to deal with convex performance criteria. The authors introduce MetaCURL for non-stationary MDPs. This is a theory heavy paper.\n\nSTRENGTHS:\nProofs are provided. It seems that the proposed theorems and propositions are novel.\n\nWEAKNESSES:\nActually, for readers (like me) are not an expert in CURL, some introductory examples are helpful. Additionall, it seems that this paper does not have any experiment.\n\nQUESTIONS:\nFirst of all, I strongly recommend the authors to include at least one or two introductory examples or figures to show the high-level intuition of their settings and motivation. E.g., difference between RL and concave utility RL; a figure showing offline CURL and online CURL; why concave utility RL is important; why do we need to focus on non-stationarity; and so on. Some examples can be within the domains of robotics, medical treatment, and so on.\n\nA good figure example can be found in Fig.1 of [1]. \n\n[1] Levine, Sergey, et al. \"Offline reinforcement learning: Tutorial, review, and perspectives on open problems.\" arXiv preprint arXiv:2005.01643 (2020).\n\nA good introductory example can be done by setting some parameters equal to low dimensions.\n\n# Specific Questions about the Settings\n1. Can your framework handle cases when the state or the action is continuous?\n2. Do we need to assume the occupancy measure is non-zero?\n\n# Inconsistent Notations\n1. Authors mention that L247-248, \"We assume T experts, with expert s active in interval [s, T]\". Actually, such notations are very confusion. $t$ refers to the episode, but here $T$ represents the number of experts. Especially in L209, the authors mention \"over a series of T episodes\". Again, in L127, the authors mention that \"for any interval $I \\subseteq [T]$\", what interval does $[T]$ represent, $[1:T]$ or $[t:T]$? It is extremely hard for readers to understand their paper with inconsistent notations.\n\n2. In L248, the authors mention that \"Expert s at episode t > s outputs\". $s$ refers to the expert, how it can be compared with $t$? \n\n3. In L19, authors claim \"episodes of length N\". Does it mean that each episode has N steps? Do the authors need to assume each episode has the exactly same length?\n\n4. In L248, Why N is treated as a function, $N(x')$? But in L19, authors claim \"episodes of length N\".\n\n5. In L210, \"For each round t\". In L225, \"In every episode t\". Actually, from my perspective, round t is different from episode t.\n\n6. Confusing exponent symbols and notations. In L268, $T^{2/3}$ represents T exponent $2/3$. Is that correct? In your Algorithm 1, symbol $F^{t}$ and $\\hat{p}^{t+1}$ represent exponent $t$?\n\nI strongly recommend the authors to introduce a notation table.\n\n# Specific Questions about MetaCURL\nAs mentioned previously, because the notations are inconsistent, it is really hard for me to fully understand the proposed framework.\n\n1. Learning with Expert Advice (LEA)\n\n    1.1 Do you assume these K experts are optimal? What if there is limited expert knowledge?\n    \n    1.2 What if two experts contradict to each other? Will your algorithm take the average of these two values? E.g., for the state $s_1$, expert 1 takes $a_1$, expert_2 takes $-a_1$, What should be the expert loss in that case? Could you show some numerical intro examples?\n\n2. The authors mention that \"This problem can be reduced to the sleeping expert problem\". However, it is still unclear how and why this problem can be reduced?\n    2.1 The authors mention that \"experts are not required to provide solutions at every time step\". What if at certain step there is not any expert providing solutions? Is there a lower bound of the number of experts that have to be awake?\n\n3. Does the agent have access to the \"external noise sequence $\\left(\\varepsilon_{n}^{t}\\right)$\"? Because $\\epsilon_{n}^{t} \\sim h_{n}^{t}(\\cdot)$, is there any information leakage?\n\n4. Does the algorithm know which expert is active? Is that signal too strong? Can such information be learned?\n\n5. What is the time complexity of algo. 1? \n\n6. Theorem 5.1\n    \n    6.1 In Theorem 5.1, why the complexity is determined by $\\Delta^{\\pi^{*}}$, $\\Delta_{\\infty}^{p}$ and $\\Delta^{p}$ simultanously? What is the intuition of taking the minimum of abrupt and smooth variation? Under the worst case, should we take the maximum of such variations?\n\n    6.2 In [2], their regret bound contains the dimenions for the state space, action space, and the length of each episode. Why in Theorem 5.1, they are not included? Does it mean that your algorithm will not change, when their dimensions are increasing? Still, because of the inconsistent $T$ notation, it is unclear such $T$ represents? (L247-248, \"We assume T experts, with expert s active in interval [s, T]\". In L209, the authors mention \"over a series of T episodes\".)\n\n[2] Rosenberg, Aviv, and Yishay Mansour. \"Online convex optimization in adversarial markov decision processes.\" International Conference on Machine Learning. PMLR, 2019.\n\n7. Because there is not any experiment in the current version, I have general concerns over the practicality of the proposed method, e.g., infinite horizon or continuous spaces.\n\nLIMITATIONS:\nThis paper does not have any introductory examples, or figures, or experiments. However, as such a notation dense paper, it seems that the notations are not consistent, making readers extremely hard to understand their major contributions. Additionally, the agents are assumed to have special strong information, e.g., experts signal or external noise sequence, which might be missing in real-world.\n\n-----------------Updates\n\nThanks for the thoughtful discussions. After careful consideration, I have decided to maintain my current rating of 4.\n\ntl&dr: Recommendation for Rejection: **The majority of reviewers (Reviewer unHu, Reviewer FWrb, Reviewer La9P, Reviewer ZTXg) expressed some concerns regarding the assumptions underlying this paper.** Suggesting inclusion of motivating examples and realistic experiments to enhance community-wide benefits from your insights.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper studied the Concave Utility Reinforcement Learning (CURL) problem in non-stationary environments, which extends classical reinforcement learning (RL) to handle convex performance criteria in state-action distributions induced by agent policies. The paper proposed MetaCURL to address the challenges posed by non-stationary Markov Decision Processes (MDPs) with changing losses and probability transitions by using a meta-algorithm that aggregates multiple black-box algorithm instances over different intervals. The algorithm achieves optimal dynamic regret without requiring prior knowledge of the MDP changes, making it suitable for adversarial losses and providing a significant advancement in the RL community.

STRENGTHS:
1. The designed algorithm is parameter-free, and does not require the knowledge of variation budget of transitions.
2. The results are theoretically solid.

WEAKNESSES:
1. The model falls into the tabular MDPs, where the state-action pairs are finite. I am wondering if similar technique can be applied to the case of function approximation. In addition, there is a missing related work on the non-stationary RL with function approximation [1].
2. The transition dynamics, while there are many suitable applications, is also restrictive for a theoretical work.


Feng et al., Non-stationary Reinforcement Learning under General Function Approximation, ICML, 2023.

QUESTIONS:
Please see the weaknesses part.

LIMITATIONS:
No further limitations to be addressed
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

comments: SUMMARY:
The authors present a policy learning algorithm for non-stationary (+ uncertain) environments and convex utilities. 
The proposed algorithm is a meta-algorithm which runs multiple black-box algorithms and aggregates outputs with something they call a sleeping expert framework. The algorithm achieves optimal dynamic regret.

STRENGTHS:
- Proposed an algorithm for non-stationary and uncertain convex utility MDP
- Achieves optimal dynamic regret

WEAKNESSES:
- The paper considers uncertain/non-stationary noise distribution. However, until 1/3rd of the paper, I got an impression of uncertain dynamics (without any structure assumption).  
- The paper is difficult to follow particularly section 3 from line 175 and section 4

QUESTIONS:
- Are baselines the same as the black-box algorithms? 
- Line 164, what does parametric algorithm mean in this context
- Line 141 why do policy optimization algorithms have lower complexity?
- Any thoughts on how the algorithm changes if g_n is uncertain?

Comments for improving paper:
- line 13: we achieve ---> the algorithm achieves?
- line 15: "full adversarial losses, not just stochastic ones." is ill-posed
- I am familiar with both non-stationarity and uncertainty. However, since the paper builds on these two challenges, I would suggest explaining them (at least in a high-level) early in the introduction. Similarly, for the terms adversarial objectives and learner (line 50).
- Line 133-135: Please include it in the introduction (could be a bit vague) but it's important to note that the structure of MDP is already known and only the noise is uncertain. 
- Maybe write contributions as a paragraph, since points 2,3 are not contributions but explanations of point 1.

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents theoretical results on the CURL algorithm for non-stationary MDPs. The proposed meta CURL models non-stationarity factors using external noise and achieves low dynamic regret in near-stationary environments, with regret only related to the frequency and magnitude of changes. Overall, the work provides solid theoretical results for CURL and non-stationary RL. Although I am not an expert on RL theory (my focus is more on algorithms and applications), I would give an acceptance rating for this initial review and will be engaged in the discussion.\n\nSTRENGTHS:\n- [**Motivation and Significance**]: The problem of non-stationary MDPs is critical in CURL, and this paper provides valuable theoretical insights and guarantees for the CURL algorithm. While I am not an expert in RL theory, I think in general this work is critical for both RL theory and empirical algorithms design. \n- [**Technical Soundness**]: While I am not an expert in RL theory, the theoretical results and proofs presented in the paper appear to be solid and well-constructed, especially on the regret analysis part (only related to changing frequency and magnitude). \n- [**Presentation**]: The paper is generally well-written and the theoretical analysis is clearly presented (accessible even to readers outside the theory domain)\n\nWEAKNESSES:\nSince I am not an expert on RL theory, I have listed most of my questions in this section. \n\nQ1: In equation (8), the noise captures the non-stationary factors. Will these factors from different parts of the MDP (such as the reward function or state dynamics) separately influence or impact the major theory and regret bound?\n\nQ2: In some non-stationary or meta RL works, the non-stationary or distribution change factors are not only from external noise but also from the agent’s policy itself. How does this affect the major theoretical results? Any comments on this point would be appreciated.\n\nQUESTIONS:\nI put the questions in the above section.\n\nLIMITATIONS:\nI do not think this theoretical work could pose any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses online learning in non-stationary episodic loop-free Markov decision processes (MDPs) with changing losses and probability transitions. It extends the Concave Utility Reinforcement Learning (CURL) problem to handle convex performance criteria in state-action distributions, overcoming the non-linearity that invalidates traditional Bellman equations. The introduced MetaCURL algorithm, the first for non-stationary MDPs, runs multiple black-box algorithm instances over different intervals, using a sleeping expert framework to aggregate outputs. MetaCURL achieves optimal dynamic regret under partial information without prior knowledge of MDP changes, handling fully adversarial losses. This approach is expected to be of significant interest to the RL community.\n\nSTRENGTHS:\nThe novelty is clear. That is expanding CURL in non-stationary MDPs.\n\nWEAKNESSES:\nMy comments on weaknesses are listed below.\n\nQUESTIONS:\nThanks for the opportunity to review this paper. I have some questions as follows. \n[Major issues]\n\n1. Novelty \n\n- The statement \"non-requiring prior knowledge of the environment's variations\" (lines 68-69) needs clarification. In Table 1, the dynamic regret of the proposed algorithms still depends on $\\Delta^p_\\infty$ and $\\Delta^p$, which seem to contain information about environmental changes (lines 123-126). How does this align with the claim of not requiring prior knowledge?\n\n2. Dynamic regret representation \n\n- The use of $\\Delta^{\\pi^\\star}_t$ to represent the upper bound of dynamic regret requires further justification. While it may seem reasonable since the optimal policy is defined given the environment, what happens if the optimal policy is not unique? In such cases, $\\Delta^{\\pi^\\star}_t$ could take multiple values, as there can be multiple optimal policies at times $t$ and $t+1$.\n\n3. Dynamic hypothesis\n \n- The dynamic hypothesis presented in line 128 appears to be a strong assumption. Reinforcement learning (RL) is inherently about learning from data, and assuming prior knowledge of how the environment operates seems misaligned with RL's principles. This assumption aligns more with traditional control theory, where the plant dynamics are known. In RL, even in model-based RL, the model is estimated, and the policy is learned simultaneously.\n\n[Minnor issues]\n1. Terminology\n- The term \"loop-free\" used in the abstract needs clarification. This term is not commonly used in the context of learning but is more prevalent in control theory. Is \"loop-free learning\" akin to continual learning with infinite episodes? More precise terminology would help readers understand your contributions better.\n\n2. Regarding with the related works. \n- It seems the paper lacks some recent related works (around 2023,2204)  in theoretical non-stationary RL works. Please include the following to keep the track. \n\n [1] Lee, H., Ding, Y., Lee, J., Jin, M., Lavaei, J., & Sojoudi, S. (2024). Tempo adaptation in non-stationary reinforcement learning. Advances in Neural Information Processing Systems, 36.\n[2] Feng, S., Yin, M., Huang, R., Wang, Y. X., Yang, J., & Liang, Y. (2023, July). Non-stationary reinforcement learning under general function approximation. In International Conference on Machine Learning (pp. 9976-10007). PMLR.\n\nLIMITATIONS:\nThere is no limitations on potential negative societal impact .", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-30"}