Title: NeoRL: Efficient Exploration for Nonepisodic RL
Venue: NeurIPS 2024
Accepted: yes
Keywords: reinforcement learning

single trajectory

nonepisodic setting

average cost MDP
Abstract: We study the problem of nonepisodic reinforcement learning (RL) for nonlinear dynamical systems, where the system dynamics are unknown and the RL agent has to learn from a single trajectory, i.e., without resets. We propose **N**on**e**pisodic **O**ptistmic **RL** (NeoRL), an approach based on the principle of optimism in the face of uncertainty. NeoRL uses well-calibrated probabilistic models and plans optimistically w.r.t. the epistemic uncertainty about the unknown dynamics. Under continuity and bounded energy assumptions on the system, we
provide a first-of-its-kind regret bound of  $\mathcal{O}(\beta_T \sqrt{T \Gamma_T})$ for general nonlinear systems with Gaussian process dynamics. We compare NeoRL to other baselines on several deep RL environments and empirically demonstrate that NeoRL achieves the optimal average cost while incurring the least regret.

NEORL: Efficient Exploration for Nonepisodic RL

Abstract

We study the problem of nonepisodic reinforcement learning (RL) for nonlinear dynamical systems, where the system dynamics are unknown and the RL agent has to learn from a single trajectory, i.e., adapt online and without resets. This setting is ubiquitous in the real world, where resetting is impossible or requires human intervention. We propose N on e pisodic O ptimistic RL ( NEORL ) , an approach based on the principle of optimism in the face of uncertainty. NEORL uses well-calibrated probabilistic models and plans optimistically w.r.t. the epistemic uncertainty about the unknown dynamics. Under continuity and bounded energy assumptions on the system, we provide a first-of-its-kind regret bound of O ( β T √ T Γ T ) for general nonlinear systems with Gaussian process dynamics. We compare NEORL to other baselines on several deep RL environments and empirically demonstrate that NEORL achieves the optimal average cost while incurring the least regret.

1 Introduction

In recent years, data-driven control approaches, such as reinforcement learning (RL), have demonstrated remarkable achievements. However, most RL algorithms are devised for an episodic setting, where during each episode, the agent interacts in the environment for a predetermined episode length or until a termination condition is met. After the episode, the agent is reset back to an initial state from where the next episode commences. Episodes prevent the system from blowing up, i.e., maintain stability, while also restricting exploration to states that are relevant to the task at hand. Moreover, resets ensure that the agent explores close to the initial states and does not end up at undesirable parts of the state space that exhibit low reward. In simulation, resetting is typically straightforward. However, if we wish to enable agents to learn and adapt by interacting online with the real world, resets are often prohibitive since they typically involve manual intervention. Instead, agents should be able to learn autonomously (Sharma et al., 2021b) i.e., from a single trajectory. This problem is extensively studied in adaptive control (Åström & Wittenmark, 2013), where classical works focus on controller design (Lai & Wei, 1982, 1987; Krsti´ c et al., 1992, 1995; Annaswamy, 2023) and not on the exploration/learning aspect of the problem. Only a few works consider these two aspects jointly (Abbasi-Yadkori & Szepesvári, 2011; Cohen et al., 2019; Dean et al., 2020; Simchowitz & Foster, 2020; Zhao et al., 2024). However, these works study linear systems with quadratic costs, i.e., the LQR setting. While several works in the Deep RL community have also studied this problem, (c.f., Section 5), the theoretical results for this setting are fairly limited. In particular, theoretical results mostly exist for the finite state and action spaces (Kearns & Singh, 2002; Brafman & Tennenholtz, 2002; Jaksch et al., 2010) and the extension to nonlinear systems with continuous spaces is much less understood. In our work, we address this gap and propose a practical RL algorithm that is grounded in theory. In particular, we make the following contributions.

Contributions

We propose, NEORL, a novel model-based RL algorithm based on the principle of optimism in the face of uncertainty. NEORL operates in a nonepisodic setting and picks average cost optimal policies optimistically w.r.t. to the model's epistemic uncertainty.

We show that when the dynamics lies in a reproducing kernel Hilbert space (RKHS) of kernel k , NEORL exhibits a regret of O ( β T √ T Γ T ) , where the regret, akin to prior work, is measured w.r.t to the optimal average cost under known dynamics, T is the number of environment steps, β T the calibration coefficient (Chowdhury & Gopalan, 2017; Srinivas et al., 2012) and Γ T the maximum information gain of kernel k (Srinivas et al., 2012). Our regret bound is similar to the ones obtained in the episodic setting (Kakade et al., 2020; Curi et al., 2020; Sukhija et al., 2024; Treven et al., 2024) and Gaussian process (GP) bandit optimization (Srinivas et al., 2012; Chowdhury & Gopalan, 2017; Scarlett et al., 2017) and is sublinear for common kernel such as the exponential kernel. To the best of our knowledge, we are the first to obtain regret bounds for the setting.

We evaluate NEORL on several RL benchmarks against common model-based RL baselines. Our experimental results demonstrate that NEORL consistently achieves sublinear regret, also when neural networks are employed instead of GPs for modeling dynamics. Moreover, in all our experiments, NEORL converges to the optimal average cost.

2 Problem Setting

We consider a discrete-time dynamical system with running costs c .

Here x t ∈ X ⊆ R d x is the state, u t ∈ U ⊆ R d u the control input, and w t ∈ W ⊆ R w the process noise. The dynamics f ∗ are unknown and the cost c is assumed to be known.

Task In this work, we study the average cost RL problem (Puterman, 2014), i.e., we want to learn the solution to the following minimization problem

Moreover, we consider the nonepisodic RL setting where the system starts at an initial state x 0 ∈ X but never resets back during learning, that is, we seek to learn online from a single trajectory. After each step t in the environment, the RL system receives a transition tuple ( x t , u t , x t +1 ) and updates its policy based on the data D t collected thus far during learning. The average cost formulation is common for the nonepisodic setting (Jaksch et al., 2010; Abbasi-Yadkori & Szepesvári, 2011; Cohen et al., 2019; Dean et al., 2020; Simchowitz & Foster, 2020), and the cumulative regret for the learning algorithm in this case is defined as

Studying the average cost criterion for general continuous state-action spaces is challenging even when the dynamics are known, since the average cost exists only for special classes of nonlinear systems (Arapostathis et al., 1993). In the following, we impose assumptions on the dynamics and policy class Π that enable our theoretical analysis.

2.1 Assumptions

Imposing continuity on f ∗ is quite common in the control theory (Khalil, 2015) and reinforcement learning literature (Curi et al., 2020; Sussex et al., 2023; Sukhija et al., 2024). To this end, for our analysis, we make the following assumption.

Assumption 2.1 (Continuity of f ∗ and π ) . The dynamics model f ∗ and all π ∈ Π are continuous.

Next, we make an assumption on the system's stochastic disturbances.

Assumption 2.2 (Process noise distribution) . The process noise is i.i.d. Gaussian with variance 2 i.i.d 2 .

Our analysis can be extended for the more general heteroscedastic case, where σ depends on x . However, for simplicity, we focus on the homoscedastic setting. In the following, we make assumptions on our policy class. To this end, we first introduce the class of K ∞ functions.

Definition 2.3 ( K ∞ -functions) . The function ξ : R ≥ 0 → R ≥ 0 is of class K ∞ , if it is continuous, strictly increasing, ξ (0) = 0 and ξ ( s ) →∞ for s →∞ .

Assumption 2.4 (Policies with bounded energy) . We assume there exists κ, ξ ∈ K ∞ , positive constants K,C u , C l with C u > C l , and γ ∈ (0 , 1) such that for each π ∈ Π we have,

Bounded energy: There exists a Lyapunov function V π : X → [0 , ∞ ) for which ∀ x , x ′ ∈ X ,

where x + = f ∗ ( x , π ( x )) + w .

Bounded norm of cost:

Boundedness of the noise with respect to κ :

The drift condition states that the energy between two timesteps can increase at most by K . In particular, the Lyapunov function V π can be viewed as an energy function for the dynamical system, and the bounded energy condition above ensures that the system is not 'blowing up'. We do not perceive this as restrictive for real-world engineered systems. Other works that study learning nonlinear dynamics (Foster et al., 2020; Sattar & Oymak, 2022; Lale et al., 2021) in the nonepisodic setting also make stability assumptions such as global exponential stability for their analysis. In similar spirit, we make the bounded energy assumption for our policy class. The drift condition on the Lyapunov function is also used to study the ergodicity of Markov chains for continuous state spaces (Meyn & Tweedie, 2012; Hairer & Mattingly, 2011), which is crucial for our analysis of the infinite horizon behavior of the system. Moreover, for a very rich class of problems, the drift condition is satisfied. We highlight this in the corollary below.

Lemma 2.5. Assume f ∗ is uniformly continuous and for all π ∈ Π , x ∈ X , ∥ π ( x ) ∥ ≤ u max . Further assume, there exists π s ∈ Π such that we have constants K,C u , C l with C u > C l , γ ∈ (0 , 1) , κ, α ∈ K ∞ and a Lyapunov function V : X → [0 , ∞ ) for which ∀ x , x ′ ∈ X ,

where x + = f ∗ ( x , π ( x )) + w . Then, V also satisfies the drift condition for all π ∈ Π , i.e., is a Lyapunov function for all policies.

We prove this lemma in Appendix A. Intuitively, if the inputs are bounded, the energy inserted into the system by another policy is also bounded. Nearly all real-world systems have bounded inputs due to the physical limitations of actuators. For these systems, it suffices if only one policy in Π satisfies the drift condition.

The boundedness assumptions for the cost and the noise in Assumption 2.4 are satisfied for a rich class of cost and K ∞ functions.

Under these assumptions, we can show the existence of the average cost solution.

Theorem 2.6 (Existence of Average Cost Solution) . Let Assumption 2.1 - 2.4 hold. Consider any π ∈ Π and let P π denote its transition kernel, i.e., P π ( x , A ) = P ( x + ∈ A| x , π ( x )) for A ⊆ X . Then P π admits a unique invariant measure ¯ P π , and there exists C 2 , C 3 ∈ (0 , ∞ ) , λ ∈ (0 , 1) such that

Average Cost ;

for all x 0 ∈ X .

Theorem 2.6 is a crucial result for our analysis since it implies that the average cost is bounded and independent of the initial state x 0 . Furthermore, it also shows that the bias is bounded. The average cost criterion satisfies the following Bellman equation (Puterman, 2014) below

Accordingly, the bias term plays an important role in the regret analysis (also notice its similarity to our regret term in Equation (3)).

Thus far, we have only made assumptions that make the average cost problem tractable. In the following, we make an assumption on the dynamics that allow us to learn it from data. Moreover, we assume that at each step n we learn a mean estimate µ n of f ∗ and can quantify our uncertainty σ n over the estimate. More formally, we learn a well-calibrated statistical model of f ∗ as defined below.

Definition 2.7 (Well-calibrated statistical model of f ∗ , Rothfuss et al. (2023)) . Let Z def = X × U . An all-time well-calibrated statistical model of the function f ∗ is a sequence {M n ( δ ) } n ≥ 0 , where

if, with probability at least 1 -δ , we have f ∗ ∈ ⋂ n ≥ 0 M n ( δ ) . Here, f j , µ n,j and σ n,j denote the j -th element in the vector-valued functions f , µ n and σ n respectively, and β n ( δ ) ∈ R ≥ 0 is a scalar function that depends on the confidence level δ ∈ (0 , 1] and which is monotonically increasing in n .

Next, we assume that f ∗ resides in a Reproducing Kernel Hilbert Space (RKHS) of vector-valued functions and show that this is sufficient for us to obtain a well-calibrated model.

Assumption 2.8. We assume that the functions f ∗ j , j ∈ 1 , . . . , d x lie in a RKHS with kernel k and have a bounded norm B , that is f ∗ ∈ H d x k,B , with H d x k,B = { f | ∥ f j ∥ k ≤ B,j = 1 , . . . , d x } . Moreover, we assume that k ( x , x ) ≤ σ max for all x ∈ X .

Assumption 2.8 allows us to model f ∗ with GPs for which the mean and epistemic uncertainty ( µ n ( z ) = [ µ n,j ( z )] j ≤ d x , and σ n ( z ) = [ σ n,j ( z )] j ≤ d x ) have an analytical formula

Here, y j 1: n corresponds to the noisy measurements of f ∗ j , i.e., the observed next state from the transitions dataset D 1: n , k n = [ k ( z , z i )] i ≤ nT , z i ∈ D 1: n , and K n = [ k ( z i , z l )] i,l ≤ nT , z i , z l ∈ D 1: n is the data kernel matrix. The restriction on the kernel k ( x , x ) ≤ σ max implies boundedness of f ∗ and has also appeared in works studying the episodic setting for nonlinear systems (Mania et al., 2020; Kakade et al., 2020; Curi et al., 2020; Sukhija et al., 2024; Wagenmaker et al., 2023). We can also define f ∗ such that x k = x k -1 + f ∗ ( x k -1 , u k -1 ) + w k -1 in which case the boundedness of f ∗ captures many real-world systems.

Lemma 2.9 (Well calibrated confidence intervals for RKHS, Rothfuss et al. (2023)) . Let f ∗ ∈ H d x k,B . Suppose µ n and σ n are the posterior mean and variance of a GP with kernel k , c.f., Equation (5) . There exists β n ( δ ) ∝ √ Γ n , for which the tuple ( µ n , σ n , β n ( δ )) is a well-calibrated statistical model of f ∗ .

In summary, in the RKHS setting, a GP is a well-calibrated model. For more general models like Bayesian neural networks (BNNs), methods such as Kuleshov et al. (2018) can be used for calibration. Our results can also be extended beyond the RKHS setting to other classes of well-calibrated models similar to Curi et al. (2020).

Algorithm 1 NEORL: NONEPISODIC OPTIMISTIC RL

Init: Aleatoric uncertainty σ , Probability δ , Statistical model ( µ 0 , σ 0 , β 0 ( δ )) , H 0

end for

3 NEORL

In the following, we present our algorithm: N on e pisodic O ptimistic RL (NEORL) for efficient nonepisodic exploration in continuous state-action spaces. NEORL builds on recent advances in episodic RL (Kakade et al., 2020; Curi et al., 2020; Sukhija et al., 2024; Treven et al., 2024) and leverages the optimism in the face of uncertainty paradigm to pick policies that are optimistic w.r.t. the dynamics within our calibrated statistical model as follows

Here, f n is a dynamical system such that the cost by controlling f n with its optimal policy π n is the lowest among all the plausible systems from M n -1 ∩ M 0 . Note, from Lemma 2.9 we have that f ∗ ∈ M n -1 ∩ M 0 (with high probability) and therefore the solution to Equation (6) gives an optimistic estimate for the average cost. We take the intersection of M n -1 with M 0 to ensure that we maintain at least the same confidence about our model as at the beginning, i.e., n = 0 , during learning.

NEORL proceeds in the following manner. Similar to Jaksch et al. (2010), we bin the total time T the agent spends interacting in the environment into N 'artificial' episodes. At each episode, we pick a policy according to Equation (6) and roll it out for H n steps on the system. Next, we use the data collected during the rollout to update our statistical model. Finally, we double the horizon H n +1 = 2 H n , akin to Simchowitz & Foster (2020), and continue to the next episode without resetting the system back to the initial state x 0 . Intuitively, in the beginning, when our model estimate is not accurate, we update our model more frequently, and with more episodes as our model gets better we reduce the frequency of updates. The algorithm is summarized in Algorithm 1.

3.1 Theoretical Results

In the following, we study the theoretical properties for NEORL and provide a first-of-its-kind bound on the cumulative regret for the average cost criterion for general nonlinear dynamical systems. Our bound depends on the maximum information gain of kernel k (Srinivas et al., 2012), defined as

Γ T represents the complexity of learning f ∗ from T data points and is sublinear for a very rich class of kernels (e.g., O (log d x + d u +1 ( T )) for the exponential (RBF) kernel, O (( d x + d u ) log( T )) for the linear kernel). In Appendix A, we report the dependence of Γ T on T in Table 1.

Theorem 3.1 (Cumulative Regret of NEORL) . Let Assumption 2.1 - 2.8 hold, and define H 0 as the smallest integer such that

Then with probability at least 1 -δ , we have the following regret for NEORL

with D 4 ( x 0 , K, γ ) , D 5 ( x 0 , K, γ ) being bounded constants for bounded ∥ x 0 ∥ , K , and γ < 1 .

Prepare policy

Set horizon

Collect measurements for horizon H n

Update statistical model M n

From Lemma 2.9 we have that β T ∝ √ Γ T and therefore Theorem 3.1 gives sublinear regret for a rich class of RKHS functions. Moreover, it also gives a minimal horizon H 0 that we need to maintain before switching to the next policy. Even for the linear case, fast switching between stable controllers can destabilize the closed-loop system. We ensure this does not happen in our case by having a minimal horizon of H 0 . Theorem 3.1 can also be derived beyond the RKHS setting for a more general class of well-calibrated models. In this case, the maximum information gain is replaced by the model complexity from Curi et al. (2020) (c.f., Curi et al. (2020); Sukhija et al. (2024) for further detail).

In the following, we give an intuitive proof sketch for Theorem 3.1. The detailed proof is provided in Appendix A.

Proof sketch The proof can be split into three main steps. First, we show the ergodicity of the closed-loop system, a sufficient condition for showing the existence of the average cost and bias term, i.e., Theorem 2.6, for every policy π ∈ Π under Assumption 2.1 - 2.4. For this, we use elementary results on Markov chains in measurable spaces from Meyn & Tweedie (2012); Hairer & Mattingly (2011). Second, we show that under Assumption 2.8, the optimistic system selected in Equation (6), retains the same properties as the true system f ∗ , e.g., stability, and therefore also is ergodic. Crucial to show this is that the true system f ∗ and the optimistic system f n are at most β n σ n apart. Finally, in the third step, we show that as we update our model and policy every H n steps, the doubling of the horizon retains the system properties from above, and our accumulated model uncertainties across T environment steps grow with the rate Γ T . For the latter, we use the analysis from Kakade et al. (2020) for the episodic case, to bound the deviation between the optimistic average cost and the true average cost.

3.2 Practical Modifications

For testing NEORL, we make three modifications that simplify its deployment in practice in terms of implementation and computation time. First, instead of doubling the horizon H n we pick a fixed horizon H during the experiment. This makes the planning and training of the agent easier. Next, we use a receding horizon controller, i.e., model predictive control (MPC) (García et al., 1989), instead of directly optimizing for the average cost in Equation (6). MPC is widely used to obtain a feedback controller for the infinite horizon setting. Moreover, while for linear systems, the Riccati equations (Anderson & Moore, 2007) provide an analytical solution to Equation (2), no such solution exists for the nonlinear case and MPC is commonly used as an approximation. Further, under additional assumptions on the cost and dynamics, MPC also obtains a policy with bounded average cost, which is crucial for the nonepisodic case (c.f., Assumption 2.4). We use the iCEM optimizer for planning (Pinneri et al., 2021). Finally, instead of optimizing over M n ∩ M 0 , we optimize directly over M n . This allows us to use the reparameterization trick from Curi et al. (2020) and obtain a simple and tractable optimization problem. In summary, for each step t in the environment, we solve the following optimization problem

Here H MPC is the MPC horizon. We take the first input from the solution of the problem above, i.e., u ∗ 0 , and execute this in the system. We then repeat this procedure for H steps and then update our statistical model M n . The resulting optimization above considers a larger action space as it includes the hallucinated controls η as additional input variables. The hallucinated controls are introduced through the reparameterization trick from (Curi et al., 2020) and are used to directly optimize over models in f ∈ M n -1 . Moreover, the final algorithm can be seen as a natural extension to HUCRL (Curi et al., 2020) for the nonepisodic setting. We summarize the algorithm in Appendix B Algorithm 2. Note while these modifications deviate from our theoretical analysis, empirically they work well for GP and BNN models, c.f., Section 4.

4 Experiments

We evaluate NEORL on the Pendulum-v1 and MountainCar environment from the OpenAI gym benchmark suite (Brockman et al., 2016), Cartpole, Reacher, and Swimmer from the DeepMind control suite (Tassa et al., 2018), the racecar simulator from Kabzan et al. (2020), and a soft robotic arm from Tekinalp et al. (2024). The swimmer and the soft robotic arm are fairly high-dimensional systems - the swimmer has a 28-dimensional state and 5-dimensional action space, and the soft arm

NeTS

NeMean

NePETS

NeoRL

oRL

NePETS

0

200

400

0

5

10

A

(

π

)

Pendulum-GP

0

200

400

Env Steps

T

0

2

4

RT

×

10

3

0

1000

2000

0

20

40

A

(

π

)

CartPole

0

1000

2000

Env Steps

T

0

5

RT

×

10

3

0

500

1000

0

5

10

Pendulum

0

500

1000

Env Steps

T

0

1

2

×

10

3

0

10000

20000

0

.

0

0

.

5

Swimmer

0

10000

20000

Env Steps

T

0

.

0

0

.

5

1

.

0

×

10

4

0

500

1000

0

50

100

MountainCar

0

500

1000

Env Steps

T

0

.

0

0

.

5

1

.

0

×

10

5

0

1000

2000

0

.

0

0

.

5

1

.

0

SoftArm

0

1000

2000

Env Steps

T

0

.

0

0

.

5

1

.

0

×

10

3

0

1000

2000

0

.

0

0

.

2

Reacher

0

1000

2000

Env Steps

T

1

2

3

×

10

1

0

1000

0

.

0

2

.

5

5

.

0

RaceCar

0

1000

Env Steps

T

0

1

2

×

10

3

is represented by a 58-dimensional state and has a 12-dimensional action space. All environments are never reset during learning. Moreover, the Pendulum-v1, MountainCar, CartPole, and Reacher environments operate within a bounded domain and thus inherently satisfy Assumption 2.4. The swimmer, racecar, and soft arm can operate in an unbounded domain but have a cost function that penalizes the distance between the system's state x t and a target state x ∗ . Therefore, the cost encourages the system to move towards the target and remain within a bounded domain.

Baselines In the episodic setting, resets can be used to control the exploration space for the agent. However, in the absence of resets, the agent can explore arbitrarily and end up in states that are irrelevant to the task at hand. Moreover, the agent has to follow an uninterrupted chain of experience, which makes the nonepisodic setting the most challenging one in RL (Kakade, 2003). Accordingly, there are only a few algorithms that consider this setting (c.f., Section 5). In this work, we focus on model-based RL (MBRL) algorithms due to their sample efficiency. In particular, we adopt common MBRL methods for our setting. MBRL algorithms typically differentiate in three ways; ( i ) propagating dynamics for planning (Chua et al., 2018; Osband & Van Roy, 2017; Kakade et al., 2020; Curi et al., 2020), ( ii ) representation of the dynamics model (Ha & Schmidhuber, 2018; Hafner et al., 2019; Kipf et al., 2019), and ( iii ) types of planners (Williams et al., 2017; Hafner et al., 2020; Pinneri et al., 2021). NEORL is independent to the choice of representation or planners. Therefore, we focus on ( i ) and use probabilistic ensembles (Lakshminarayanan et al., 2017) and GPs for modeling our dynamics and MPC with iCEM (Pinneri et al., 2021) as the planner. Common techniques to propagate the dynamics for planning are using the mean, trajectory sampling (Chua et al., 2018), and Thompson

sampling (Osband & Van Roy, 2017). We adapt these three for our setting similar to as discussed in Section 3.2. For all experiments with probabilistic ensembles, we consider TS1 from Chua et al. (2018) for trajectory sampling, and for the GP experiment, we use distribution sampling from Chua et al. (2018). We call the three baselines NEMEAN (nonepisodic mean), NEPETS (nonepisodic PETS), and NETS (nonepisodic Thompson sampling). NEMEAN and NEPETS are greedy w.r.t. the current estimate of the dynamics, i.e., do not explicitly encourage exploration. In our experiments, we show that being greedy does not suffice to converge to the optimal average cost, that is, obtain sublinear regret. The code for our experiments is available online. 2

Convergence to the optimal average cost In Figure 1 we report the normalized average cost and cumulative regret of NEORL, NEMEAN, NEPETS, and NETS. The normalized average cost is defined such that A ( π ∗ ) = 0 for all environments. We observe that NEMEAN fails to converge to the optimal average cost for the Pendulum-v1 environment for both probabilistic ensembles and a GP model. It also fails to solve the MountainCar environment and is unstable for the Reacher and CartPole. In general, NEMEAN performs the worst among all methods. This is similar to the episodic case, where using the mean model often leads to the policy 'overfitting' to the model inaccuracies (Chua et al., 2018). NEPETS performs better than the mean, however still significantly worse than NEORL. Even in the episodic setting, PETS tends to underexplore (Curi et al., 2020). We observe the same for the nonepisodic case, especially for the MountainCar task, which is a challenging RL environment with a sparse cost. Here NEPETS is also not able to achieve the optimal average cost and thus does not have sublinear cumulative regret. NETS performs similarly to NEPETS and is also not able to solve the MountainCar task.

NEORL performs the best among the baselines for all experiments and converges to the optimal average cost achieving sublinear cumulative regret using only ∼ 10 3 environment interactions. Moreover, this observation is consistent between different dynamics models (GPs and probabilistic ensembles) and environments. Even in environments that are unbounded, i.e., Swimmer, SoftArm, and RaceCar, we observe that NEORL converges to the optimal average cost the fastest. We believe this is due to the feedback control from MPC, which has a stabilizing effect.

Calling reset when needed All the experiments in Figure 1 considered the nonepisodic setting where the system was never reset during learning. A special case of our theoretical analysis is the class of policies Π that may call for a reset / 'ask for help' whenever they end up in an undesirable part of the state space. In this setting, the system is typically restricted to a compact subset of the state space X , and the policy class satisfies Assumption 2.4. For many real-world applications, such a policy class can be derived. To simulate this experiment, we consider the CartPoleBalance task in Figure 2, where the goal is to balance the pole in the upright position. A reset is triggered whenever the pole drops. We again observe that NEORL achieves the best performance, i.e., lowest cumulative regret and thus learns to solve the task the fastest. Moreover, it also requires fewer resets than NEMEAN, NEPETS, and NETS.

5 Related Work

Average cost RL for finite state-action spaces A significant amount of work studies the average cost/reward RL setting for finite-state action spaces. Moreover, seminal algorithms such as E 3 (Kearns & Singh, 2002) and Rmax (Brafman & Tennenholtz, 2002) have established PAC bounds for the nonepisodic setting. These bounds are further improved for communicating MDPs by the UCRL2 (Jaksch et al., 2010) algorithm, which, similar to NEORL, is based on the optimism in the face of uncertainty paradigm and picks policies that are optimistic w.r.t. to the estimated dynamics. Their result is extended for weakly-communicating MDPs by REGAL (Bartlett & Tewari, 2012), similar results are derived for Thompson sampling based exploration (Ouyang et al., 2017), and for factored-MDP (Xu & Tewari, 2020). Albeit the significant amount of work for the finite case, progress for continuous state-action spaces has mostly been limited to linear dynamical systems.

Nonepisodic RL for linear systems There is a large body of work for nonepisodic learning with linear systems (Abbasi-Yadkori & Szepesvári, 2011; Cohen et al., 2019; Simchowitz & Foster, 2020; Dean et al., 2020; Lale et al., 2020; Faradonbeh et al., 2020; Abeille & Lazaric, 2020; Treven et al., 2021). For linear systems with quadratic costs, the average reward problem, also known as the linear quadratic-Gaussian (LQG), has a closed-form solution which is obtained via the Riccati equations (Anderson & Moore, 2007). Moreover, for LQG, stability and optimality are intertwined,

0

1000

2000

Env Steps

T

0

1

2

3

4

Total Resets

0

1000

2000

Env Steps

T

0

.

0

0

.

5

1

.

0

RT

×

10

2

CartPoleBalance

NeTS

NeMean

NePETS

NeoRL

making studying linear systems much easier than their nonlinear counterpart. For studying nonlinear systems, additional assumptions on their stability are usually made.

Episodic RL for nonlinear systems In the case of nonlinear systems, guarantees have mostly been established for the episodic setting (Mania et al., 2020; Kakade et al., 2020; Curi et al., 2020; Wagenmaker et al., 2023; Sukhija et al., 2024; Treven et al., 2024). In this setting, the agent begins each episode from an initial state s 0 (or initial state distribution) and interacts with the environment for a fixed horizon H . It uses the data collected from the interactions to update its model. After each episode, the agent is reset back to s 0 . The works mentioned above theoretically study this setting for finite-horizon MDPs and establish regret bounds for general nonlinear systems. Particularly Kakade et al. (2020); Curi et al. (2020); Sukhija et al. (2024); Treven et al. (2024) also use an optimism-based approach similar to ours. Compared to the nonepisodic case, the analysis of episodic RL methods is simpler as resets restrict the agent's exploration around the initial state s 0 and prevent the system from blowing up or visiting states from which the agent cannot recover. However, as discussed in Section 1, resets are often prohibitive and RL agents that learn non-episodically are preferred for many real-world applications.

Nonepisodic RL beyond linear systems Only a few works consider the nonepisodic/singletrajectory case. For instance, a line of work studies data-driven MPC approaches focusing mostly on establishing system-theoretic guarantees such as closed-loop stability and robustness (Berberich & Allgöwer, 2024). From the learning side, Foster et al. (2020); Sattar & Oymak (2022) study the problem of system identification of a closed-loop globally exponentially stable dynamical system from a single trajectory. Lale et al. (2021) study the nonepisodic setting for nonlinear systems with MPC. Moreover, they consider finite-order or exponentially fading NARX systems that lie in the RKHS of infinitely smooth functions, which they further approximate with random Fourier features (Rahimi & Recht, 2007) ϕ with feature size D . Further, they assume access to bounded persistently exciting inputs w.r.t. the feature matrix Φ t Φ ⊺ t . This assumption is generally tough to verify and common excitation strategies such as random exploration often don't perform well for nonlinear systems (Sukhija et al., 2024). The algorithm also operates in two stages, where in the first stage it performs pure exploration for system identification and in the second stage exploitation, i.e., acting greedily w.r.t. the estimated dynamics, akin to NEMEAN. Additionally, the algorithm requires the feature size D to increase with the horizon T . They give a regret bound of O ( T 2 / 3 ) where the regret is measured w.r.t. to the oracle MPC with access to the true dynamics. Lale et al. (2021) also assume exponential input-to-output stability of the system to avoid blow-up during exploration. Our work considers more general RKHS, naturally trades-off exploration and exploitation, does not require apriori knowledge of persistently exciting inputs and gives a regret bound of O ( β T √ T Γ T ) w.r.t. the optimal average cost criterion. Moreover, our regret bound is similar to the ones obtained for nonlinear systems in the episodic case and Gaussian process bandits (Srinivas et al., 2012; Chowdhury & Gopalan, 2017; Scarlett et al., 2017). To the best of our knowledge, we are the first to give such a regret bound for nonlinear systems.

Nonepisodic Deep RL Standard deep RL approaches often fail in the nonepisodic setting (Sharma et al., 2021b). To this end, deep RL algorithms have also been developed for the nonepisodic case. Mostly, these works focus on learning to reset and formulate it from the perspective of safety (Eysenbach et al., 2018) (avoiding undesirable states), chaining multiple controllers (Han et al., 2015), skill discovery/intrinsic exploration (Zhu et al., 2020; Xu et al., 2020), curriculum learning (Sharma et al., 2021a), and learning initial state distributions from demonstrations (Sharma et al., 2022). However, in contrast to us, none of the works above provide any theoretical guarantees.

There are several extensions of model-free deep RL algorithms to the average reward setting (TRPO (Zhang & Ross, 2021), PPO (Ma et al., 2021), and DDPG (Saxena et al., 2023)). However, they mostly focus on maximizing the long-term behavior of the RL agent and allow for resets during learning. Overall, extending RL algorithms for the discounted case to the average one is still an open problem (Dewanto et al., 2020). However, future work in this direction will benefit NEORL. Since average-reward optimizers can be used in combination with NEORL to directly minimize the average cost in a model-based policy optimization (Janner et al., 2019) manner.

6 Conclusion

We propose, NEORL, a novel model-based RL algorithm for the nonepisodic setting with nonlinear dynamics and continuous state and action spaces. NEORL seeks for average-cost optimal policies and leverages the model's epistemic uncertainty to perform optimistic exploration. Similar to the episodic case (Kakade et al., 2020; Curi et al., 2020), we provide a regret bound for NEORL of O ( β T √ T Γ T ) for Gaussian process dynamics. To our knowledge, we are the first to obtain this result in the nonepisodic setting. We compare NEORL to other model-based RL methods on standard deep RL benchmarks. Our experiments demonstrate that NEORL, converges to the optimal average cost of A ( π ∗ ) = 0 across all environments, suffering sublinear regret even when Bayesian neural networks are used to model the dynamics. Moreover, NEORL outperforms all our baselines across all environments requiring only ∼ 10 3 samples for learning.

Future work may consider deriving lower bounds on the regret of NEORL, studying different assumptions on f ∗ and Π , and investigating different notions of optimality such as bias optimality in the nonepisodic setting (Mahadevan, 1996).

Acknowledgments and Disclosure of Funding

We would like to thank Mohammad Reza Karimi, Scott Sussex, and Armin Lederer for the insightful discussions and feedback on this work. This project has received funding from the Swiss National Science Foundation under NCCR Automation, grant agreement 51NF40 180545, and the Microsoft Swiss Joint Research Center.

References

Abbasi-Yadkori, Y. and Szepesvári, C. Regret bounds for the adaptive control of linear quadratic systems. In Conference on Learning Theory , 2011.

Abeille, M. and Lazaric, A. Efficient optimistic exploration in linear-quadratic regulators via lagrangian relaxation. In International Conference on Machine Learning , 2020.

Anderson, B. D. and Moore, J. B. Optimal control: linear quadratic methods . Courier Corporation, 2007.

Annaswamy, A. M. Adaptive control and intersections with reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems , 2023.

Arapostathis, A., Borkar, V. S., Fernández-Gaucherand, E., Ghosh, M. K., and Marcus, S. I. Discrete-time controlled markov processes with average cost criterion: A survey. SIAM Journal on Control and Optimization , 1993.

Åström, K. J. and Wittenmark, B. Adaptive Control . Courier Corporation, 2013.

Bartlett, P. L. and Tewari, A. Regal: A regularization based algorithm for reinforcement learning in weakly communicating mdps. arXiv preprint arXiv:1205.2661 , 2012.

Berberich, J. and Allgöwer, F. An overview of systems-theoretic guarantees in data-driven model predictive control, 2024. URL https://arxiv.org/abs/2406.04130 .

Brafman, R. I. and Tennenholtz, M. R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research , 2002.

Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

Chowdhury, S. R. and Gopalan, A. On kernelized multi-armed bandits. In ICML , 2017.

Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In NeurIPS , 2018.

√

Cohen, A., Koren, T., and Mansour, Y. Learning linear-quadratic regulators efficiently with only T regret. In International Conference on Machine Learning , 2019.

Curi, S., Berkenkamp, F., and Krause, A. Efficient model-based reinforcement learning through optimistic policy search and planning. NeurIPS , 33:14156-14170, 2020.

Dean, S., Mania, H., Matni, N., Recht, B., and Tu, S. On the sample complexity of the linear quadratic regulator. Foundations of Computational Mathematics , 20(4):633-679, 2020.

Dewanto, V., Dunn, G., Eshragh, A., Gallagher, M., and Roosta, F. Average-reward model-free reinforcement learning: a systematic review and literature mapping. arXiv preprint arXiv:2010.08920 , 2020.

Eysenbach, B., Gu, S., Ibarz, J., and Levine, S. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. International Conference on Learning Representations , 2018.

Faradonbeh, M. K. S., Tewari, A., and Michailidis, G. Optimism-based adaptive regulation of linear-quadratic systems. IEEE Transactions on Automatic Control , 2020.

Foster, D., Sarkar, T., and Rakhlin, A. Learning nonlinear dynamical systems from a single trajectory. In Learning for Dynamics and Control , 2020.

García, C. E., Prett, D. M., and Morari, M. Model predictive control: Theory and practice - a survey. Automatica , pp. 335-348, 1989.

Ha, D. and Schmidhuber, J. Recurrent world models facilitate policy evolution. Advances in neural information processing systems , 31, 2018.

Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. Learning latent dynamics for planning from pixels. In International conference on machine learning , 2019.

Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. Dream to control: Learning behaviors by latent imagination. ICLR , 2020.

Hairer, M. and Mattingly, J. C. Yet another look at harris' ergodic theorem for markov chains. In Seminar on Stochastic Analysis, Random Fields and Applications VI: Centro Stefano Franscini, Ascona, May 2008 , pp. 109-117. Springer, 2011.

Han, W., Levine, S., and Abbeel, P. Learning compound multi-step controllers under unknown dynamics. In Intelligent Robots and Systems (IROS) , 2015.

Jaksch, T., Ortner, R., and Auer, P. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research , 2010.

Janner, M., Fu, J., Zhang, M., and Levine, S. When to trust your model: Model-based policy optimization. Advances in neural information processing systems , 2019.

Kabzan, J., Valls, M. I., Reijgwart, V . J., Hendrikx, H. F., Ehmke, C., Prajapat, M., Bühler, A., Gosala, N., Gupta, M., Sivanesan, R., et al. Amz driverless: The full autonomous racing system. Journal of Field Robotics , 2020.

Kakade, S., Krishnamurthy, A., Lowrey, K., Ohnishi, M., and Sun, W. Information theoretic regret bounds for online nonlinear control. NeurIPS , 33:15312-15325, 2020.

Kakade, S. M. On the sample complexity of reinforcement learning . University of London, University College London (United Kingdom), 2003.

Kearns, M. and Singh, S. Near-optimal reinforcement learning in polynomial time. Machine learning , 2002.

Khalil, H. K. Nonlinear control , volume 406. Pearson New York, 2015.

Kipf, T., Van der Pol, E., and Welling, M. Contrastive learning of structured world models. arXiv preprint arXiv:1911.12247 , 2019.

Krsti´ c, M., Kanellakopoulos, I., and Kokotovi´ c, P. Adaptive nonlinear control without overparametrization. Systems & Control Letters , 1992.

Krsti´ c, M., Kokotovic, P. V ., and Kanellakopoulos, I. Nonlinear and adaptive control design . John Wiley & Sons, Inc., 1995.

Kuleshov, V., Fenner, N., and Ermon, S. Accurate uncertainties for deep learning using calibrated regression. In ICML , pp. 2796-2804. PMLR, 2018.

Lai, T. L. and Wei, C. Z. Least squares estimates in stochastic regression models with applications to identification and control of dynamic systems. The Annals of Statistics , 1982.

Lai, T. L. and Wei, C.-Z. Asymptotically efficient self-tuning regulators. SIAM Journal on Control and Optimization , 1987.

Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles, 2017.

Lale, S., Azizzadenesheli, K., Hassibi, B., and Anandkumar, A. Logarithmic regret bound in partially observable linear dynamical systems. Advances in Neural Information Processing Systems , 2020.

Lale, S., Azizzadenesheli, K., Hassibi, B., and Anandkumar, A. Model learning predictive control in nonlinear dynamical systems. In Conference on Decision and Control (CDC) . IEEE, 2021.

Ma, X., Tang, X., Xia, L., Yang, J., and Zhao, Q. Average-reward reinforcement learning with trust region methods. International Joint Conference on Artificial Intelligence , 2021.

Mahadevan, S. Average reward reinforcement learning: Foundations, algorithms, and empirical results. Machine learning , 1996.

Mania, H., Jordan, M. I., and Recht, B. Active learning for nonlinear system identification with guarantees. arXiv preprint arXiv:2006.10277 , 2020.

Meyn, S. P. and Tweedie, R. L. Markov chains and stochastic stability . Springer Science & Business Media, 2012.

Osband, I. and Van Roy, B. Why is posterior sampling better than optimism for reinforcement learning? In International conference on machine learning , 2017.

Ouyang, Y., Gagrani, M., Nayyar, A., and Jain, R. Learning unknown markov decision processes: A thompson sampling approach. Advances in neural information processing systems , 30, 2017.

Pinneri, C., Sawant, S., Blaes, S., Achterhold, J., Stueckler, J., Rolinek, M., and Martius, G. Sample-efficient cross-entropy method for real-time planning. In CORL , Proceedings of Machine Learning Research, pp. 1049-1065, 2021.

Puterman, M. L. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

Rahimi, A. and Recht, B. Random features for large-scale kernel machines. Advances in neural information processing systems , 20, 2007.

Rothfuss, J., Sukhija, B., Birchler, T., Kassraie, P., and Krause, A. Hallucinated adversarial control for conservative offline policy evaluation. UAI , 2023.

Sattar, Y. and Oymak, S. Non-asymptotic and accurate learning of nonlinear dynamical systems. Journal of Machine Learning Research , 2022.

Saxena, N., Khastagir, S., Kolathaya, S., and Bhatnagar, S. Off-policy average reward actor-critic with deterministic policy search. In International Conference on Machine Learning , 2023.

Scarlett, J., Bogunovic, I., and Cevher, V . Lower bounds on regret for noisy Gaussian process bandit optimization. In Conference on Learning Theory , 2017.

Sharma, A., Gupta, A., Levine, S., Hausman, K., and Finn, C. Autonomous reinforcement learning via subgoal curricula. Advances in Neural Information Processing Systems , 2021a.

Sharma, A., Xu, K., Sardana, N., Gupta, A., Hausman, K., Levine, S., and Finn, C. Autonomous reinforcement learning: Formalism and benchmarking. arXiv preprint arXiv:2112.09605 , 2021b.

Sharma, A., Ahmad, R., and Finn, C. A state-distribution matching approach to non-episodic reinforcement learning. International Conference on Machine Learning , 2022.

Simchowitz, M. and Foster, D. Naive exploration is optimal for online lqr. In International Conference on Machine Learning . PMLR, 2020.

Srinivas, N., Krause, A., Kakade, S. M., and Seeger, M. W. Information-theoretic regret bounds for gaussian process optimization in the bandit setting. IEEE Transactions on Information Theory , 2012.

Sukhija, B., Treven, L., Sancaktar, C., Blaes, S., Coros, S., and Krause, A. Optimistic active exploration of dynamical systems. NeurIPS , 2024.

Sussex, S., Makarova, A., and Krause, A. Model-based causal bayesian optimization. In ICLR , May 2023.

Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 , 2018.

Tekinalp, A., Kim, S. H., Bhosale, Y., Parthasarathy, T., Naughton, N., Albazroun, A., Joon, R., Cui, S., Nasiriziba, I., Stölzle, M., Shih, C.-H. C., and Gazzola, M. Gazzolalab/pyelastica: v0.3.2, 2024.

Treven, L., Curi, S., Mutn` y, M., and Krause, A. Learning stabilizing controllers for unstable linear quadratic regulators from a single trajectory. In Learning for Dynamics and Control , 2021.

Treven, L., Hübotter, J., Sukhija, B., Dörfler, F., and Krause, A. Efficient exploration in continuous-time model-based reinforcement learning. NeurIPS , 2024.

Vakili, S., Khezeli, K., and Picheny, V. On information gain and regret bounds in gaussian process bandits. In AISTATS , 2021.

Wagenmaker, A., Shi, G., and Jamieson, K. Optimal exploration for model-based rl in nonlinear systems. arXiv preprint arXiv:2306.09210 , 2023.

Williams, G., Wagener, N., Goldfain, B., Drews, P., Rehg, J. M., Boots, B., and Theodorou, E. A. Information theoretic mpc for model-based reinforcement learning. In ICRA , 2017.

Xu, K., Verma, S., Finn, C., and Levine, S. Continual learning of control primitives: Skill discovery via reset-games. Advances in Neural Information Processing Systems , 2020.

Xu, Z. and Tewari, A. Reinforcement learning in factored mdps: Oracle-efficient algorithms and tighter regret bounds for the non-episodic setting. Advances in Neural Information Processing Systems , 2020.

Zhang, Y. and Ross, K. W. On-policy deep reinforcement learning for the average-reward criterion. In International Conference on Machine Learning , 2021.

Zhao, F., Dörfler, F., Chiuso, A., and You, K. Data-enabled policy optimization for direct adaptive learning of the lqr. arXiv preprint arXiv:2401.14871 , 2024.

Zhu, H., Yu, J., Gupta, A., Shah, D., Hartikainen, K., Singh, A., Kumar, V., and Levine, S. The ingredients of real-world robotic reinforcement learning. arXiv preprint arXiv:2004.12570 , 2020.

Appendices

A Proofs

In this section, we prove Theorem 2.6 and Theorem 3.1. First, we start with the proof of Lemma 2.5.

Proof of Lemma 2.5. We first analyze the following term E w [ V ( f ∗ ( x , π ( x )) + w ) -V ( f ∗ ( x , π s ( x )) + w )] for any π ∈ Π .

Therefore,

Hence, V satisfies the drift condition for π . Furthermore, since V also satisfies positive definiteness by assumption, the bounded energy condition holds for all π ∈ Π .

A.1 Proof of Theorem 2.6

For proving Theorem 2.6, we invoke the results from (Hairer & Mattingly, 2011, Theorem 1.2 - 1.3). For this we require that the Markov chain induced by a policy π satisfies the drift condition. In our setting, this corresponds to Assumption 2.4. Next, we show that the chain satisfies the following minorisation condition.

Lemma A.1 (Minorisation condition) . Consider the system in Equation (1) and let Assumption 2.1 - 2.4 hold. Let P π denote the transition kernel for the policy π ∈ Π , i.e., P π ( x , A ) = P ( x + ∈ A| x , π ( x )) . Then, for all π ∈ Π , exists a constant α ∈ (0 , 1) and a probability measure ζ ( · ) s.t.,

Proof. We prove it in 3 steps. First, we show that C is contained in a compact domain. From the Assumption 2.4 we pick the function ξ ∈ K ∞ . Since C l ξ (0) = 0 , lim s →∞ ξ ( s ) = + ∞ and C l ξ is continuous, there exists M such that C l ξ ( M ) = R . Then for ∥ x ∥ > M we have:

Therefore we have: C ⊆ B ( 0 , M ) def = { x | ∥ x -0 ∥ ≤ M } . In the second step we show that f ( C , π ( C )) is bounded, in particular we show that there exists B > 0 such that: f ( C , π ( C )) ⊆ B ( 0 , B ) . This is true since continuous image of compact set is compact and the observation:

Lemma A.1. It is enough to show that ∀ µ ∈ B ( 0 , B ) , ∀ x ∈ R x we have:

Since f ( B ( 0 , M ) , π ( B ( 0 , M ))) is compact there exists B such that f ( C , π ( C )) ⊆ B ( 0 , B ) . In the last step we prove that α def = 2 -d x e -B 2 /σ 2 and ζ with law of N ( 0 , σ 2 2 ) satisfy condition of d

which can be proven with simple algebraic manipulations.

Through the minorisation condition and Assumption 2.4, we can prove the ergodicity of the closedloop system for a given policy π ∈ Π .

Theorem A.2 (Ergodicity of closed-loop system) . Let Assumption 2.1 - 2.4, consider any probability measures ζ 1 , ζ 2 , and θ > 0 , define P π ζ , ∥ φ ∥ 1+ θV π , ρ π θ as

We have for all π ∈ Π , that P π admits a unique invariant measure ¯ P π . Furthermore, there exist constants C 1 > 0 , θ > 0 , λ ∈ (0 , 1) such that

holds for every measurable function φ : X → R with ∥ φ ∥ 1+ V π < ∞ . Here ( P π ) t denotes the t -step transition kernel under the policy π . Moreover, θ = α 0 / K , and

for any α 0 ∈ (0 , α ) and γ 0 ∈ ( γ +2 K / R , 1) .

Proof. From Assumption 2.4, we have a value function for each policy that satisfies the drift condition. Furthermore, in Lemma A.1 we show that our system also satisfies the minorisation condition for all policies. Under these conditions, we can use the results from Hairer & Mattingly (2011, Theorem 1.2. - 1.3.).

Note that ∥·∥ 1+ θV π represents a family of equivalent norms for any θ > 0 . Now we prove Theorem 2.6.

Proof of Theorem 2.6. From Theorem A.2, we have

where δ x 0 is the dirac measure. Therefore, ( P π ) t is a Cauchy sequence. Furthermore, ρ π θ is complete for the set of probability measures integrating V , thus ρ π θ (( P π ) t , ¯ P π ) → 0 for t →∞ (c.f., Hairer &Mattingly (2011) for more details). In particular, we have for φ such that ∥ φ ∥ 1+ θV π ≤ 1 ,

Note that since all ∥·∥ 1+ θV π norms are equivalent for θ > 0 , if ∥ c ∥ 1+ V π ≤ C (Assumption 2.4), then ∥ c ∥ 1+ θV π ≤ C ′ for some C ′ ∈ (0 , ∞ ) . Furthermore, note that c ( · ) ≥ 0 . Therefore,

In summary,

Consider any T ≥ 0 , and define with ¯ c = E x ∼ ¯ P π [ c ( x , π ( x ))] .

Hence, we have

and for any x 0 in a compact subset of X

Moreover,

Another interesting, inequality that follows from the proof above is the difference in bias inequality.

In summary, we have E x ∼ ¯ P π [ c ( x )] ≤ C ( 1 + K 1 1 -γ )

Consider any t > 0 , and note that from Theorem A.2 we have

where C 2 = C 1 ( ∥ c ∥ 1+ V π + CK 1 1 -γ ) .

Moreover, since the inequality holds for all x 0 , we have

for all probability measures ζ 1 , ζ 2 . To show this holds, define C ′ = max π ∈ Π ∥ c ( x , π ( x )) ∥ 1+ θV π . Furthermore, note that C ′ < ∞ from Assumption 2.4 and ∥ c ( x , π ( x )) / C ′ ∥ 1+ θV π ≤ 1 .

Also, note that there exists C θ ∈ (0 , ∞ ) such that C θ ∥ φ ∥ 1+ θV π ≥ ∥ φ ∥ 1+ V π due to the equivalence of the two norms.

Therefore, for the bias we have

Set C 3 = C ′ C θ .

A.2 Proof of bounded average cost for the optimistic system

In this section, we show that the results from Theorem 2.6 also transfer over to the optimistic dynamics.

Theorem A.3 (Existence of Average Cost Solution for the Optimistic System) . Let Assumption 2.1 2.8 hold. Consider any n > 0 and let π n , f n denote the solution to Equation (6) , P π , f n its transition kernel. Then P π , f n admits a unique invariant measure ¯ P π n , f n and there exists C 2 , C 3 ∈ (0 , ∞ ) , ˆ λ ∈ (0 , 1) such that

Average Cost ;

Bias Cost ;

for all x 0 ∈ X .

Difference in Bias ;

for all probability measures ζ 1 , ζ 2 .

Theorem A.3 shows that the optimistic dynamics f n retain the boundedness property from the true dynamics f ∗ and give a well-defined solution w.r.t. average cost and the bias cost. To prove Theorem A.3 we show that the optimistic system also satisfies the drift and minorisation condition. Then we can invoke the result from Hairer & Mattingly (2011) similar to the proof of Theorem 2.6.

Lemma A.4 (Stability of optimistic system) . Let Assumption 2.1 - 2.8 hold, then we have with probability at least 1 -δ for all n ≥ 0 , π ∈ Π , f ∈ M n ∩ M 0 , that there exists a constant ̂ K > 0 such that where x + = f ( x , π ( x ) + w .

Proof. Note, that V π is uniformly continuous w.r.t. κ

Furthermore, since f ∈ M n ∩ M 0 and therefore f ∈ M 0 , we have that there exists some η ∈ [ -1 , 1] dx such that

Therefore,

where we denoted x ∗ + = f ∗ ( x , π ( x ) + w . Define ̂ K = K + κ (( 1 + √ d x ) β 0 √ d x σ max ) .

Lemma A.5 (Minorisation condition optimistic ystem) . Consider the system

for any n ≥ 0 , π ∈ Π and f ∈ M n ∩ M 0 . Let Assumption 2.1 - 2.8 hold. Let P π , f denote the transition kernel for the policy π ∈ Π i.e., P π , f ( x , A ) = P ( x + ∈ A| x , π ( x ) , f ) . Then, there exists a constant ˆ α ∈ (0 , 1) and a probability measure ˆ ζ ( · ) independent of n s.t.,

with C def = { x ∈ X ; V π ( x ) < ˆ R } for some ˆ R > 2 ̂ K / 1 -γ

Proof. First, we show that C is contained in a compact domain. From the Assumption 2.4 we pick the function ξ ∈ K ∞ . Since C l ξ (0) = 0 , lim s →∞ ξ ( s ) = + ∞ and C l ξ is continuous, there exists M such that C l ξ ( M ) = ˆ R . Then for ∥ x ∥ > M we have:

Therefore we have: C ⊆ B ( 0 , M ) def = { x | ∥ x -0 ∥ ≤ M } . Since for any x ∈ C we have ∥ f ( x , π ( x )) ∥ ≤ ∥ f ∗ ( x , π ( x )) ∥ + β 0 σ max . Since f ∗ is continuous, there exists a B such that f ∗ ( C , π ( C )) ⊂ B ( 0 , B ) . Therefore we have: f ( C , π ( C )) ⊂ B ( 0 , B 1 ) , where B 1 = B + β 0 σ max . In the last step we prove that α def = 2 -d x e -B 2 1 /σ 2 and ζ with law of N ( 0 , σ 2 2 ) satisfy condition of R d x

Lemma A.1. It is enough to show that ∀ µ ∈ B ( 0 , B 1 ) , ∀ x ∈ we have:

which can be proven with simple algebraic manipulations.

Proof of Theorem A.3. As for the true system, the drift condition from Lemma A.4 and the minorisation condition from Lemma A.5 are sufficient to show ergodicity of the optimistic system (c.f., Theorem A.2 or Hairer & Mattingly (2011)). The rest of the proof is similar to Theorem 2.6.

A.3 Proof of Theorem 3.1

Since NEORL works in artificial episodes n ∈ { 0 , N -1 } of varying horizons H n . We denote with x n k the state visited during episode n at time step k ≤ H n . Crucial, to our regret analysis is bounding the first and second moment of V π n ( x n k ) for all n, k . Given the nature of Assumption 2.4, this requires analyzing geometric series. Thus, we start with the following elementary result of geometric series.

Corollary A.6. Consider the sequence { S n } n ≥ 0 with S n ≥ 0 for all n . Let the following hold

for ρ ∈ (0 , 1) and C > 0 . Then we have

Proof.

Lemma A.7. Let Assumption 2.1 - 2.8 hold and let H 0 be the smallest integer such that

Moreover, define ν = C u C l γ H 0 . Note, by definition of H 0 , ν < 1 . Then we have for all k ∈ { 0 , . . . , H n } and n > 0

Bounded expectation over horizon

Bounded expectation over episodes

Moreover, we have

Proof. We start with proving the first claim

We can apply Corollary A.6 to prove the claim. For the second claim, we note that for any π , π ′ and x ∈ X we have from Assumption 2.4

Therefore,

For our choice of H 0 , we have for all n ≥ 0 that C u C l γ H n ≤ C u C l γ H 0 ≤ ν < 1 . From Corollary A.6, we get

Lemma A.8. Let Assumption 2.1 - 2.8 hold and let H 0 be the smallest integer such that

Moreover, define ν = C u C l γ H 0 . Note, by definition of H 0 , ν < 1 . Then we have for all k 0 , . . . , H n and n > 0

∈ { }

Bounded second moment over horizon

Bounded second moment over episodes

Proof. Note that,

We first bound the second term. Let ¯ x n k = f ∗ ( x n k -1 , π n ( x n k -1 )) , i.e., the next state in the absence of transition noise.

Therefore we have

Let D 2 ( x 0 , K, γ, ν ) = 2 KγD ( x 0 , K, γ, ν ) + K 2 + C w . Applying Corollary A.6 we get

Similar to the first moment, we leverage that V π n ( x ) ≤ C u C l V π n -1 ( x ) for all x ∈ X , C u C l γ H n -1 ≤ ν , and get,

Moreover,

Finally, we prove the regret bound of NEORL.

Proof of Theorem 3.1. In the following, let ˆ x n k +1 = f n ( x n k , π n ( x n k ))+ w n k denote the state predicted under the optimistic dynamics and x n k +1 = f ∗ n ( x n k , π n ( x n k )) + w n k the true state.

First, we study the term (A).

Proof for (A) : Note that because f n ∈ M n , there exists a η ∈ [ -1 , 1] d x such that ˆ x n k +1 = µ n ( x n k , π n ( x n k )) + β n σ n ( x n k , π n ( x n k )) η ( x n k ) + w n k . Furthermore, x n k +1 = f ∗ ( x n k , π n ( x n k )) + w n k and the transition noise is Gaussian. Let ζ n 2 ,k and ζ n 1 ,k denote the respective distributions of the

two random variables, i.e., ζ n 1 ,k ∼ N ( f ∗ ( x n k , π n ( x n k )) , σ 2 I ) and ζ n 2 ,k ∼ N ( f n ( x n k , π n ( x n k )) , σ 2 I ) . Next, define ¯ B = E x ∼ ζ n 2 ,k [ B ( π n , f n , x )] , and consider the function h ( x ) = B ( π n , f n , x ) -¯ B . Then we have

Note that E x ∼ ζ n 2 ,k [ h ( x )] = 0 by the definition of h and thus,

In the following, we bound the term above w.r.t. the Chi-squared distance

With d χ ( ζ n 2 ,k , ζ n 1 ,k ) being the Chi-squared distance.

Since both bounds from Equation (17) and bound we got by applying (Kakade et al., 2020, Lemma C.2.,), we can apply minimum and have:

Therefore, following Kakade et al. (2020, Lemma C.2.,) we get

Therefore, we have

Here, for the second and third inequality, we use Cauchy-Schwarz. Now we bound the two terms above individually.

First we bound E x ∼ ζ n [ h 2 ( x ) ] .

Furthermore, we have from Lemma A.8.

In the end, we get

Next, we use the bound from Curi et al. (2020, Lemma 17.) for the second term.

Here Γ T is the maximum information gain.

If we set D 4 ( x 0 , K, γ ) = C ′ (1+ √ d x ) σ ( √ 2 C 2 1 -ˆ λ ) √ (1 + 2 γ 2 ) D 3 ( x 0 , K, γ, ν ) + 2(2 + ˆ K ) 2 , we have

Proof for (B) :

Here D 5 ( x 0 , K, γ ) = 2 C 2 1 -ˆ λ (1 + D ( x 0 , K, γ )) . Finally, for our choice, H n = H 0 2 n , we get

Therefore, N = log 2 ( T H 0 +1 ) . To this end, we get for our regret

This regret is sublinear for a very rich class of functions. We summarize bounds on Γ T from Vakili et al. (2021) in Table 1. Furthermore, note that D 4 ( x 0 , K, γ ) ∈ (0 , ∞ ) for all x 0 ∈ X with ∥ x 0 ∥ < ∞ , K < ∞ , γ ∈ (0 , 1) . The same holds for D 5 ( x 0 , K, γ ) . Moreover, since V π ( x ) is Θ( ζ ( ∥ x ∥ )) , both D 4 and D 5 are Θ( ζ ( ∥ x 0 ∥ )) .

Algorithm 2 Practical NEORL:

Init: Aleatoric uncertainty σ , Probability δ , Statistical model ( µ 0 , σ 0 , β 0 ( δ )) for n = 1 , . . . , N do for h = 1 , . . . , H do min u 0: H MPC -1 , η 0; H MPC -1 E [ H MPC -1 ∑ h =0 c (ˆ x h , u h ) ] ; x 0 = x n h ➤ Solve MPC problem ( x h n , u ∗ 0 , x h +1 n ) ← ROLLOUT ( u ∗ 0 ) ➤ Collect transition end for Update ( µ n , σ n , β n ) ←D n end for

B Practical algorithm and Experimental Details

In this section, we provide the practical algorithm Algorithm 2, provide all hyperparameters used in our experiments in Table 2, and the cost function for the environments. All our experiments within 1-8 hours 3 on a GPU (NVIDIA GeForce RTX 2080 Ti). For NEORL, we use β n = 2 for all the experiments, except for the Swimmer and the SoftArm environment where we use β n = 1 .

×

∥

-

∥

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We highlight the problem setting, algorithm, and theoretical and empirical results in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Section 2 we highlight the assumptions of our work, which also correspond to the limitations of our theoretical analysis and also the setting for which our algorithm yields theoretical guarantees. Further, in ?? we discuss an alternative set of assumptions to the one made in the main paper. Limitations to the theoretical algorithm are discussed in Section 3.2, where also practical modifications are proposed. In our experiments (Section 4) we evaluate our algorithm on settings where the assumptions are not necessarily satisfied.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theoretical results are accompanied by the relevant assumptions that are listed in Section 2 and we provide all proofs in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We use open-source benchmarks, disclose all hyperparameters in Appendix B, and the practical algorithm is explained in Section 3.2. Furthermore, we provide the code as supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the code as supplementary material and the hyperparameters used in our experiments in Appendix B.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All experiment details are explained thoroughly in Section 3.2 and Section 4. Furthermore, all hyperparameters are listed in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All experiments are run with 10 seeds and mean performance with standard error is reported in all our plots.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We give details on computation time in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms, in every respect, to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper proposes a method to improve exploration in reinforcement learning in the nonepisodic setting, and is not tied to specific applications. As such, it shares the many potential societal consequences that are associated with reinforcement learning and automation as a whole, spanning from environmental impact to concerns on ethics and alignment.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release high-risk data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite all creators whose code we used in our experiments in Section 4.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide the code as supplementary material including a readme that explains how to install and run the code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a model-based RL algorithm NeoRL  for continuous state-action spaces in the nonepisodic setting, where the agent learns from a single trajectory without resets.\n\nSTRENGTHS:\n1. The paper provides the first regret bound for nonepisodic RL in general nonlinear systems, addressing a gap in the literature.\n2. NeoRL is grounded in the optimism principle and leverages well-calibrated probabilistic models, providing a theoretically justified exploration strategy.\n3. The experiments demonstrate the practical effectiveness of NEORL, achieving sublinear regret and converging to the optimal average cost in various environments.\n\nWEAKNESSES:\n1. The optimization problem in Equation (6) for policy selection may be computationally expensive, especially for high-dimensional systems. \n2. While the experiments demonstrate the efficacy of NEORL, a more comprehensive evaluation across a diverse set of environments and comparisons with state-of-the-art methods would strengthen the empirical results.\n3. The authors can discuss some data-driven MPC framework, like [1,2].\n\n[1] Berberich J, Köhler J, Müller M A, et al. Data-driven model predictive control with stability and robustness guarantees[J]. IEEE Transactions on Automatic Control, 2020, 66(4): 1702-1717.\n\n[2] Berberich J, Allgöwer F. An overview of systems-theoretic guarantees in data-driven model predictive control[J]. arXiv preprint arXiv:2406.04130, 2024.\n\nQUESTIONS:\n1. How sensitive is the performance of NeoRL to the choice of kernel and the maximum information gain? What guidance can be provided for selecting appropriate kernels in practice?\n2. Can the bounded energy assumption be relaxed or replaced with alternative assumptions to broaden the applicability of NEORL?\n\nLIMITATIONS:\nThe authors have discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a novel model-based RL algorithm, named NeoRL, for nonepisodic RL problems with unknown system dynamics and known costs. A cumulative regret bound is provided for NeoRL with well-calibrated dynamic models. The proposed method achieved lower accumulative regret and average cost compared with baselines in several continuous control tasks. In the end, NeoRL shows it needs less reset before convergence in a reverted pendulum task with automatic reset.\n\nSTRENGTHS:\n1. the research question this paper tries to address is important and interesting, as the difficulty of resetting the environment is a notorious blocker for deploying RL agents in the real world.\n\n2. A bound for cumulative regret is provided for the proposed algorithm when certain conditions are satisfied.\n\n3. The proposed algorithms show lower average cost and cumulative regret than baselines in the empirical evaluation.\n\nWEAKNESSES:\n1. The paper is difficult to follow for readers without much control theory background (such as me), and it is difficult to distinguish the algorithmic contributions of this work. Could the author provide a more intuitive explanation about how NeoRL is connected and different from [1,2] and which part of the algorithm actually improves performance in a non-episodic setting?\n\n2. Another major concern of mine is the selection of baseline algorithms, which are not compared to other recent model-based RL algorithms, such as [3-5] and RL algorithms designed to deal with non-episodic problems [6, 7].\n\n3. No analyses were presented to understand how critical design choices influence the proposed methods' performance, such as the choice of planning horizon H_0 and whether to double the planning horizon.\n\n[1] Treven, L., Hübotter, J., Dorfler, F. and Krause, A., 2024. Efficient \nexploration in continuous-time model-based reinforcement learning. *Advances in Neural Information Processing Systems*, *36*.\n\n[2]Curi S, Berkenkamp F, Krause A. Efficient model-based reinforcement learning through optimistic policy search and planning. Advances in Neural Information Processing Systems. 2020;33:14156-70.\n\n[3]Hafner D, Pasukonis J, Ba J, Lillicrap T. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104. 2023 Jan 10.\n\n[4] Hansen N, Wang X, Su H. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.04955. 2022 Mar 9.\n\n[5] Hansen N, Su H, Wang X. Td-mpc2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16828. 2023 Oct 25.\n\n[6] Sharma A, Ahmad R, Finn C. A state-distribution matching approach to non-episodic reinforcement learning. arXiv preprint arXiv:2205.05212. 2022 May 11.\n\n[7] Chen A, Sharma A, Levine S, Finn C. You only live once: Single-life reinforcement learning. Advances in Neural Information Processing Systems. 2022 Dec 6;35:14784-97.\n\nQUESTIONS:\n1. Could the author provide a more intuitive explanation about which part of the algorithm improves performance in non-episodic settings?\n\n2. Could the author explain why the planning horizon needs to be doubled after each “artificial episode”?\n\n3. Is NeoRL extendable to problems with unknown rewards (cost functions)?\n\nLIMITATIONS:\nThe limitations and assumptions for the theory results are discussed in section 2", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper proposes NeoRL for non-episodic RL with nonlinear dynamical systems. NeoRL has a first-of-its-kind regret bound for general nonlinear systems with Gaussian process dynamics. The paper also proposes a practical implementation of NeoRL with MPC, which significantly outperforms baseline algorithms.

STRENGTHS:
The paper paper proposes NeoRL, which has a first-of-its-kind regret bound for general nonlinear systems with Gaussian process dynamics.

The paper also proposes a practical implementation of NeoRL with MPC, which significantly outperforms baseline algorithms.

The paper is very well-written and easy to follow.

While the basic idea of the algorithm is not novel, the paper considers a very important topic of average RL and has a large impact on the theory of average RL, I think.

WEAKNESSES:
I do not see any particular weakness in this paper. Maybe one weakness is that the tightness of the derived bound is unclear because there is no lower bound for the considered setting, as the authors also mentioned in Conclusion.

QUESTIONS:
- Is there any comment on the tightness of the regret?
- How well does NeoRL scale to high-dimensional environments such as Humanoid?

LIMITATIONS:
The paper discuss the limitation of the theoretical results. I do not see any potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12