Title: Policy Optimization for Continuous Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: exploratory stochastic control

occupation time

performance difference

policy optimization
Abstract: We study reinforcement learning (RL) in the setting of continuous time and space, for an infinite horizon with a discounted objective and the underlying dynamics driven by a stochastic differential equation. Built upon recent advances in the continuous approach to RL, we develop a notion of occupation time (specifically for a discounted objective),  and show how it can be effectively used to derive performance difference and local approximation formulas. We further extend these results to illustrate their applications in the PG (policy gradient) and TRPO/PPO (trust region policy optimization/ proximal policy optimization) methods,  which have been familiar and powerful tools in the discrete RL setting but under-developed in continuous RL. Through numerical experiments, we demonstrate the effectiveness and advantages of our approach.

Policy Optimization for Continuous Reinforcement Learning

Abstract

We study reinforcement learning (RL) in the setting of continuous time and space, for an infinite horizon with a discounted objective and the underlying dynamics driven by a stochastic differential equation. Built upon recent advances in the continuous approach to RL, we develop a notion of occupation time (specifically for a discounted objective), and show how it can be effectively used to derive performance-difference and local-approximation formulas. We further extend these results to illustrate their applications in the PG (policy gradient) and TRPO/PPO (trust region policy optimization/ proximal policy optimization) methods, which have been familiar and powerful tools in the discrete RL setting but under-developed in continuous RL. Through numerical experiments, we demonstrate the effectiveness and advantages of our approach.

1 Introduction

Reinforcement Learning (RL, [52]) has been successfully applied to wide-ranging domains in the past decade, including achieving superhuman performance in games like Atari and Go [35, 48, 49], enhancing Large Language Models using human feedback [8, 10], and showing potentials in improving traditional model-based decisions in healthcare, inventory management, and finance [9, 30, 33]. Most existing works, including all references cited above, are formulated and solved as discrete-time sequential optimization problems such as Markov decision processes (MDPs, [42]). Yet in many applications, agents may need to monitor and interact with the random environment at an ultra-high frequency (e.g., autonomous driving, robot navigation, and high-frequency stock trading), which calls for a continuous-time/space approach.

Recent years have witnessed a fast growing body of research that has extended the frontiers of continuous RL in several important directions including, for instance, modeling the noise or randomness in the environment dynamics as following a stochastic differential equation (SDE), and incorporating an entropy-based regularizer into the objective function [58] to facilitate the exploration-exploitation tradeoff; designing model-free methods and algorithms, along with applications to portfolio optimization [18, 20, 21, 22]; studying regret bounds [5, 54], and so forth.

In this paper, we continue the above trend in continuous RL, focusing on an infinite horizon formulation with a discounted objective and the underlying dynamics driven by an SDE [24, 39]. We are specifically motivated by the following two questions.

( Q1 ) The visitation frequency in MDP (with a discounted objective) is defined as: ρ ( s ) = ∑ ∞ t =0 γ t · P ( Y t = s ) , where { Y t } is a Markov chain with state space S := { s } , and γ ∈ (0 , 1) is a discount factor. It plays an important role in many RL algorithms for MDP. So, a natural question is, what is the continuous counterpart of ρ ( s ) ?

( Q2 ) For continuous RL, how can we characterize the difference in performance between two policies? In particular, can we derive performance-difference formulas similar to those in the MDP

case [23, 45]? Can we adapt and apply the ideas and tools of the efficient policy optimization methods (e.g., [45, 47]) to the continuous RL setting?

Main contributions . We provide a unified theory/framework for policy optimization in continuous time and space. Specifically, we have addressed the above two questions (Q1) and (Q2) by developing the notion of occupation time/measure , specifically for a discounted objective, and focusing on its associated q -value . Based on these two quantities, we derive the performance-difference formula for continuous RL by means of perturbation analysis. Leveraging the performance-difference formula, we develop the continuous counterparts of the policy gradient (PG, [53]) and also propose the local approximation for the performance metric, for which we derive a bound on it and allow the development of a minorization-majorization (MM) algorithm. We further develop the continuous counterparts of trust region policy optimization/ proximal policy optimization (TRPO/PPO) methods in [45, 47], which have been familiar and powerful tools in the discrete RL setting but under-developed in continuous RL, as approximations to the previous algorithms. (What is worth mentioning is that these policy optimization algorithms do not require any a priori discretization of time and space.) Through numerical examples we show the convergence of these algorithms when applied to certain stochastic control tasks in continuous time and space.

Organization of the paper . In Section 2 we present the continuous RL formulation and develop necessary tools. The main results, the performance-difference formula (Theorem 2) and the bound (Theorem 5) are provided in Section 3. In Section 4 we propose two algorithms, policy gradient with random rollout and PPO with adaptive penalty , based on our analyses and theoretical results; and illustrate their performance via numerical experiments. Concluding remarks are summarized in Section 5.

Related works . One line of research on continuous RL focuses on modeling the underlying dynamics as a deterministic system, typically following a deterministic ordinary differential equation. Several papers [9, 37, 44] solve the problems via a priori discretization in either time or space; [13] develops a framework to apply the temporal difference to the continuous setting, and proposes algorithms that combine value iteration or advantage update as in [3, 4, 6] to avoid explicit discretization; [37] further investigates policy gradient methods, followed by more recent studies on model-free continuous RL methods [25, 29, 57] or model-based ones [14]; [55] studies the sensitivity of existing off-policy algorithms along with advantage updating to propose continuous RL algorithms that are robust to time discretization.

The formulation of continuous RL in a stochastic setting (i.e., with the state process driven by an SDE), can be traced back to [38], which however provides no data-driven solution. Recently, [58] develops an exploratory control model for the continuous RL. Built upon this approach and for a finite-horizon objective, [20] studies policy evaluation, and [21] policy gradient. Furthermore, [22] brings forth the notion of q -value, which leads to a continuous analogue of Q -learning. Also worth noting is [2, 28], which studies RL in the mean-field regime where continuous-time processes occur in the limit, and [16] extends the study to jump-diffusion processes.

In discrete-time MDPs, the body of research on bounding the performance difference between two policies also relates to our work: [1, 45] develop a policy improvement bound for the discounted total reward; [12, 64] studies the long-run average reward, and [11] proposes a bound that is continuous with respect to the discount factor.

Notation . For a measurable set A , denote P ( A ) for the set of probability distributions over A . For a vector x , denote by || x || 2 the Euclidean norm of x . For a matrix A , denote by || A || F the Frobenius norm of A , and A 2 := AA ⊤ where A ⊤ is the transpose of A . For a positive-definite matrix A , denote by A 1 2 the square root matrix of A . For A,B two matrices of the same size, denote by A ◦ B the inner product of A and B . For a function f on an Euclidean space, ∇ f (resp. ∇ 2 f ) denotes the gradient (resp. the Hessian) of f . For two distributions P, Q ∈ P ( A ) , denote by W 2 ( P, Q ) the Wasserstein-2 distance (or Quadratic Wasserstein distance) between P and Q :

where Γ( P, Q ) is the set of all couplings of P and Q ; and denote by D KL ( P || Q ) the KL-divergence between P and Q : D KL ( P ∥ Q ) = ∫ p ( x ) log ( p ( x ) q ( x ) ) dx , in which p and q denote the probability densities of P and Q .

2 Formulation and Preliminaries

Continuous RL . We start with a quick formulation of the continuous RL, based on the same modeling framework as in [58]. Assume that the state space is R n , and denote by A the action space. Let π ( · | x ) ∈ P ( A ) be a (state) feedback policy given the state x ∈ R n . A continuous RL problem is formulated by a distributional (or relaxed) control approach [62], which is motivated by the trial and error process in RL. The state dynamics ( X a s , s ≥ 0) is governed by the Itô process:

where ( B t , t ≥ 0) is the m -dimensional Brownian motion, b : R n ×A↦→ R n , σ : R n ×A↦→ R n × m , and the action a s is generated from the distribution π ( · | X a s ) by external randomization . To avoid technical difficulties, we assume that the stochastic processes (1) (and (3), (9) below) are well-defined, see [24, Section 5.3] or [50, Chapter 6] for background.

From now on, write ( X π s , a π s ) for the state and action at time s given by the process (1) under the policy π = { π ( · | x ) ∈ P ( A ) : x ∈ R n } . The goal here is to find the optimal feedback policy π ∗ that maximizes the expected discounted reward over an infinite time horizon:

where r : R n × A ↦→ R + is the running reward of the current state and action ( X π s , a π s ) ; p : R n × A × P ( A ) ↦→ R is a regularizer which facilitates exploration (e.g., in [58], p is taken as the differential entropy defined by p ( x, a, π ( · )) = -log π ( a ) ); γ ≥ 0 is a weight parameter on exploration (also known as the 'temperature' parameter); and β > 0 is a discount factor that measures the time-depreciation of the objective value (or the impatience level of the agent).

Performance metric . A standard approach to solving the problem in (2) is to find a sequence of policies π k = { π k ( · | x ) : x ∈ R n } , k = 1 , 2 , . . . such that the value functions following the policies π k will converge to V ∗ , or be at least increasing in k , i.e., demonstrating policy improvement.

Given a policy π ( · ) , let ˜ b ( x, π ( · )) := ∫ A b ( x, a ) π ( a )d a and ˜ σ ( x, π ( · )) := (∫ A σ 2 ( x, a ) π ( a )d a ) 1 2 . Assume (for technical purpose) that ˜ σ ( x, π ( · )) is positive definite for every x ∈ R n . It is sometimes more convenient to consider the following equivalent SDE representation of (1):

in the sense that there exists a probability measure ˜ P which supports the m -dimensional Brownian motion ( ˜ B s , s ≥ 0) , and for each s ≥ 0 , the distribution of ˜ X s under ˜ P agrees with that of X s under P defined by (1), see Appendix A. Note that the dynamics in (3) does not require external randomization. Also set ˜ r ( x, π ) := ∫ A r ( x, a ) π ( a )d a and ˜ p ( x, π ) := ∫ A p ( x, a, π ) π ( a )d a . We formally define the (state) value function given the feedback policy { π ( · | x ) : x ∈ R n } by

which, under suitable conditions on model parameters ( b, σ, r, p ) and the policy π , is characterized by the Hamilton-Jacobi equation (see [21, 56]):

More technical details regarding the above formulation are spelled out in the Appendix.

We can now define the performance metric as follows:

so V ∗ = max π η ( π ) . The main task of the continuous RL is to approximate max π η ( π ) by constructing a sequence of policies π k , k = 1 , 2 , . . . recursively such that η ( π k ) is non-decreasing.

Policy evaluation . Let's first recall a general approach in [20], which can be used to learn the state value function in (4) or the performance metric in (6) for a given policy π . The idea is that for any T > 0 and a suitable test process ( ξ t , t ≥ 0) ,

If we parameterize V ( x ; π ) = V ϕ ( x ) and choose the special test function ξ t = ∂V ϕ ( X π t ) ∂ϕ , stochastic approximation leads to the online update:

where α > 0 is the learning rate. This recovers the mean-squared TD error (MSTDE) method for policy evaluation in the discrete RL [51].

q -value . The Q-value function [61] and the advantage function [3, 34] in discrete-time MDPs play a critical role in reinforcement learning theory and algorithms. However, as pointed out in [55], these concepts will not apply when the time interval shrinks to 0 (as in the continuous setting). To derive algorithms that fit the need of a continuous stochastic environment, [55, 22] proposed the advantage rate function. Namely, given a policy π and ( t, x, a ) ∈ [0 , ∞ ) × R n ×A , consider a 'perturbed" policy ˆ π as follows. It takes the action a ∈ A on [ t, t +∆ t ) where ∆ t > 0 , and then follows π on [ t + ∆ t, ∞ ) . The corresponding state process X ˆ π given X ˆ π t = x is broken into two pieces. On [ t, t +∆ t ) , it is X a which is the solution to

while on [ t +∆ t, ∞ ) , it is X π following (3) with the initial time-state pair ( t +∆ t, X a t +∆ t ) . With ∆ t > 0 as time discretization, the generalization of the conventional Q-function can be expressed as

where H a ( x, y, A ) := b ( x, a ) · y + 1 2 σ 2 ( x, a ) ◦ A + r ( x, a ) is the (generalized) Hamilton function in stochastic control theory [62]. This motivates the following definition.

Definition 1. [22] For a given policy π ∈ Π and ( x, a ) ∈ R n ×A , define the q -value as

which represents the instantaneous advantage rate of an action in a given state under a given policy.

3 Main Results

This section is concerned with theoretical developments. In Section 3.1, we define the (discounted) occupation time/measure which is the continuous analog of visitation frequency in MDPs. It is crucial in deriving the performance-difference formula in Section 3.2, which spins off two different algorithms - policy gradient and TRPO/PPO. In Section 3.3, we propose a local approximation for the performance metric, and derive a bound from which an MM algorithm is constructed.

3.1 Discounted Occupation Time

Here we first provide an answer to ( Q1 ) by defining the notion of discounted occupation time for the continuous RL.

Definition 2. Let X = ( X t , t ≥ 0) be governed by the SDE (3) , and assume that it has a probability density function p π ( · , t ) at each time t . For each x ∈ R n and t ≥ 0 , define the β -discounted occupation time of X at the state x by

So d π µ ( · ) induces a finite measure on R n with a total mass of β -1 , which we call the discounted occupation measure.

In probability theory, the definition in (12) is referred to as the β -potential of X , which gives discounted visitation frequencies of the state process. We record the following result, which will be useful in the derivation of the performance-difference formula. It is a consequence of the occupation time formula [41, 43].

Lemma 1. Under the conditions in 2, we have E ∫ ∞ 0 e -βs φ ( X s ) d s = ∫ R n d π µ ( x ) φ ( x )d x , for any measurable function φ : R n ↦→ R + for which the expectation exists.

3.2 Performance-Difference Formula

We are now ready to answer ( Q2 ) by deriving the performance-difference formula between two policies in terms of the discounted occupation time in (12) and the q -values in (11).

Theorem 2. Given two feedback policies ˆ π = { ˆ π ( · | x ) : x ∈ R n } and π = { π ( · | x ) : x ∈ R n } , we have:

Proof sketch . The full proof is detailed in Appendix B.1. The essence of the proof is to use the perturbation theory and properties of the discounted occupation time. Define an operator L π : C 2 ( R n ) ↦→ C ( R n ) associated with the diffusion process as:

Then the Hamilton-Jacobi equation that characterizes the state value function can be expressed as:

Note that for any φ ∈ C 2 ( R n ) , we have ∫ R n d π µ ( y )( -L π φ )( y )d y = ∫ R n φ ( y ) µ (d y ) . This allows us to express the performance difference in model-dynamics related terms:

What remains is to reduce the above to the desired result in (13).

□

As discussed in Section 2, our main task is to construct (algorithmically) a sequence of policies π k along which the performance improves. Here we illustrate how some well known approaches of policy improvement (from π to ˆ π ) are instances of the performance difference formula (13).

q -learning and soft q -learning . Since d ˆ π µ ≥ 0 , we only need to ensure that for all x ∈ R n , ∫ A ˆ π ( a | x ) ( q ( x, a ; π ) + γp ( x, a, ˆ π )) d a ≥ 0 . This boils down to the problem that for any x , find v ≡ ˆ π ( ·| x ) to maximize

There are two special cases:

(i) If p ( x, a, v ) ≡ 0 , then v = δ ( a ∗ ) where a ∗ = arg max a q ( x, a, π ) . This is essentially the counterpart of Q -Learning [61] in the discrete time, which we call q -learning.

(ii) If p ( x, a, v ) = -log( v ( a )) , this is known as the entropy regularizer [17, 59]. Concretely, we need to solve

which has a closed form solution with v ∗ ( a ) ∝ exp( q ( x,a,π ) γ ) , i.e. v ∗ is the Boltzmann policy for q -functions. This is a 'soft' ( à la [17]) version of the q -learning mentioned above.

Policy gradient . We may use function approximations to π by a parametric family π θ , with θ ∈ Θ ⊆ R L . For simplicity, we write d θ 0 µ (resp. η ( θ ) ) for d π θ 0 µ (resp. η ( π θ ) ). Setting ˆ π = π θ and π = π θ 0 in (13) and taking derivative with respect to θ on both sides, we get the following result.

Theorem 3 (Policy Gradient) . The policy gradient at π θ 0 is:

where the expectation is w.r.t. ( x, a ) ∼ ( βd θ 0 µ , π θ 0 ) , meaning x ∼ βd θ 0 µ ( · ) and then a ∼ π θ 0 ( · | x ) .

The above formula is indeed the continuous analogue to the well-known PG formula (without regularization) in the MDP setting, where ∇ θ η ( θ ) | θ = θ 0 = 1 β E ( x,a ) [ ∇ θ log ( π θ ( a | x ) ) A ( x, a ; π θ 0 ) ] ([53]), with A denoting the advantage function. Specifically, as a comparison, the formula in (19) replaces the visitation frequency by the occupation time, and the advantage function by the q -function, while keeping the same score function ∇ θ log ( π θ ( a | x ) ) .

3.3 Continuous TRPO/PPO

Leveraging the performance-difference formula derived above, we can now move on to spell out the continuous counterpart of TRPO and PPO originally developed in [45, 47] for the discrete RL.

Local approximation function : Given a feedback policy ˆ π , we define the local approximation function to η (ˆ π ) by

Comparing (20) to the formula (13), we see that the difference is to replace d ˆ π µ ( s ) with d π µ ( s ) . Observe that

i.e. the local approximation function and the true performance objective share the same value and the same gradient with respect to the policy parameters. Thus, the local approximation function can be regarded as the first order approximation to the performance metric. Furthermore, similar to [23, 45], we can apply simulation methods to evaluate the local approximation function only using the data generated from the current policy π :

Next, we provide analysis and bounds on the gap η (ˆ π ) -L π (ˆ π ) , which can then be used to ensure policy improvement (similar to approaches in [45, 64] for discounted/average reward MDP). First, we need some technical conditions on the model dynamics.

Assumption 1. Assume the following conditions for the state dynamics hold true:

(i) Global boundedness: There exists 0 < σ 0 ≤ ¯ σ 0 such that σ 2 0 · I ≤ ˜ σ 2 ( x, a ) ≤ ¯ σ 2 0 · I for all x, a ; (ii) Uniformly Lipschitz: There exists C ˜ σ > 0 such that ∥ ˜ σ ( x, π ) -˜ σ ( x ′ , π ) ∥ F ≤ C ˜ σ ∥ x -x ′ ∥ 2 for all π and x, x ′ ;

(iii) Monotonicity (for drift) or growth condition:

The following lemma provides a Wasserstein-2 bound between the discounted occupation measures d π µ ( · ) and d ˆ π µ ( · ) for two policies π and π ′ .

Lemma 4. Let π, π ′ be two feedback policies, and suppose the conditions in Assumption 1 hold. Define C ˜ b, ˜ σ := 2 C ˜ b +1+2 C 2 ˜ σ and C = sup x,a | b ( x, a ) | 2 + n ¯ σ 2 0 2 σ 2 0 . (Recall n is the dimension of the state.) Assume further that β > C ˜ b, ˜ σ and C < ∞ . Then there is the bound

(The proof is deferred to Appendix B.3.) To derive a performance difference bound, define the Sobolev semi-norm as K := ∥ f ∥ ˙ H 1 := (∫ R n |∇ f ( x ) | 2 d x ) 1 2 , and its dual norm ∥ · ∥ ˙ H -1 as ∥ µ ∥ ˙ H -1 =

sup {|⟨ g, µ ⟩| | ∥ g ∥ ˙ H 1 ≤ 1 } . [32, 40] show the equivalence of this dual norm ∥ µ -v ∥ ˙ H -1 to the Wasserstein-2 distance W 2 ( µ, v ) for any probability measure µ and v . Combining this fact with Lemma 4 yields the following result.

Theorem 5. Suppose the conditions in Lemma 4 hold, and further assume that d ˆ π µ ( x ) , d π µ ( x ) are upper bounded by M for all x ∈ R n . Define K := ∥ f ∥ ˙ H 1 with f ( x ; π, ˆ π ) := ∫ A ˆ π ( a | x ) ( q ( x, a ; π ) + p ( x, a, ˆ π )) d a and C ( µ, π, ˆ π ) := √ MK 2 β 2 C ˜ b, ˜ σ ( β -C ˜ b, ˜ σ ) ( sup x,a ∥ b ( x, a ) ∥ 2 + n ¯ σ 2 0 2 σ 2 0 ) . Assuming C ( µ, π, ˆ π ) < ∞ , we have η (ˆ π ) ≥ L π (ˆ π ) , where

Proof is given in Appendix B. By Theorem 5, we can use the minorization-maximization (MM) algorithm in [19, 23, 26], where L π (ˆ π ) is taken as the surrogate function for η ( π ) . Specifically, given the policy π k , if we can indeed solve the optimization problem max ˆ π L π k (ˆ π ) , and designate its solution as π k +1 . Then, we have

i.e., a guaranteed performance improvement. See also [26, Chapter 7] and [27] for the (global) convergence analysis of the MM algorithm (which exceeds the scope of this work). However, in general this optimization problem is not easy to solve directly since C ( µ, π, ˆ π ) is unknown because of the unknown underlying dynamics, and we may also have to work with sample based estimates of the approximation functions. In the spirit of [45, 47], we provide algorithms in the next section that can be practically implemented by incorporating an adaptive penalty constant C penalty as an alternative to C ( µ, π, ˆ π ) . Consequently, the resulting algorithms may no longer strictly preserve the increasing performance of η at each iteration, but overall increasing trend will be clear (as demonstrated in Figure 3).

4 Algorithms and Experiments

4.1 Sample-based Algorithms

Based on the analysis and results developed above, we provide sample-based estimates of the objective functions that lead to practical algorithms. Here we highlight several hyper-parameters: the learning rate α ; the trajectory truncation parameter (time horizon) T (needs to be sufficiently large); the total sample size N or the sampling interval δ t , with N · δ t = T . Also denote t i := i · δ t , i = 0 , . . . , N -1 , for the time points that we observe data from the environment.

Algorithm 1 CPG: Policy Gradient with exp( β ) random rollout

Input : Policy parameters θ 0 , critic net parameters ϕ 0 , batch/sample size J

1: for k = 0 , 1 , 2 , · · · until θ k converges do

2: Collect a truncated trajectory { X t i , a t i , r t i , p t i } , i = 1 , . . . , N from the environment using π θ k .

3: for i = 0 , . . . , N -1 do : Update the critic parameters as in (8)

4: for j = 1 , , . . . , J do : Draw i.i.d. τ j from exp( β ) , round τ j to the largest multiple of δ t no larger than it, and compute the GAE estimator of q ( X τ j , a τ j )

5: Get an estimator of ∇ j η ( θ k ) as

6: Let ˜ ∇ η ( θ k ) = 1 J ∑ J j =1 ∇ j η ( θ k ) and perform PG update: θ k +1 = θ k + α ˜ ∇ η ( θ k )

Continuous Policy Gradient (CPG) . To estimate the policy gradient (19) from data, we first sample an independent exponential variable τ ∼ exp( β ) to get ( X π τ , a π τ ) ∼ ( d θ 0 µ , π θ 0 ( ·| x )) . If there is a q -function oracle, then we can obtain an unbiased estimate of the policy gradient (of which the convergence analysis follows [63]). Lack of such an oracle, we employ the generalized advantage estimation (GAE) technique [46] to get q ( X t , a t ) ≈ ( Q ∆ t ( X t , a t ; π ) -V ( X t ; π )) /δ t ≈ ( r t δ t + e -βδ t V ( X t + δ t ) -V ( X t ) ) /δ t . This yields the policy gradient Algorithm 1.

Continuous PPO (CPPO) . We now present Algorithm 2, a continuous version of the PPO, also as an approximation to the MM algorithm in Section 3.3. To do so, we need more hyperparameters: the tolerance level ϵ , and the KL-divergence radius δ . Moreover, we set ¯ D KL ( θ ∥ θ k ) := E x ∼ d θ k µ √ D KL ( π θ k ( ·| x ) ∥ π θ ( ·| x )) . (Empirically we find that taking average, instead of supremum, over x does not affect the algorithm performance while reducing computational burden, similar to what's observed in the discrete-time TRPO in [45].)

Algorithm 2 CPPO: PPO with adaptive penalty constant

Input : Policy parameters θ 0 , critic net parameters ϕ 0

1: for k = 0 , 1 , 2 , · · · until θ k converge do

2: Follow the same as Steps 2-6 in Algorithm 1.

3: Compute policy update (by taking a fixed s steps of gradient descent)

4: if ¯ D KL ( θ k +1 ∥ θ k ) ≥ (1 + ϵ ) δ then C k +1 penalty = 2 C k penalty

5: else if ¯ D KL ( θ k +1 ∥ θ k ) ≤ δ/ (1 + ϵ ) then C k +1 penalty = C k penalty / 2

Algorithm 2 is essentially a continuous analogue of the TRPO/PPO methods. Note that in the penalty term we use the mean square-root of the KL-divergence, since we choose the radius δ < 1 ; hence, the square-root distance will dominate in the bound in (23). Moreover, interestingly, throughout our primary experiments, using the square-root KL-divergence outperforms (using the KL-divergence itself). Refer to Appendix C,D for more details.

4.2 Experiments

LQ stochastic control . Consider an environment driven by an SDE with linear state dynamics and quadratic rewards, with b ( x, a ) = Ax + Ba , σ ( x, a ) = Cx + Da , where A,B,C,D ∈ R , p ( x, a, π ) = -log( π ( a | x )) , and r ( x, a ) = -( M 2 x 2 + Rxa + N 2 a 2 + Px + Qa ) , where M ≥ 0 , N > 0 , R, P, Q ∈ R . Linear-quadratic (LQ) control problems play an important role in the control literature, not only because it has elegant and simple solutions but also because more complex, nonlinear problems can be approximated by LQ problems. In general, we do not know the model parameters (e.g., A,B,... ), and the idea is to use continuous RL methods to find the optimal policy.

CPG

1.0

CPPO

：to theoptimal

0.8

0.6

0.4

0.2

0

200

400

600

800

1000

iteration

20.0

CPG

17.5

CPPO

15.0

the

12.5

distance

8

10.0

7.5

5.0

2.5

0.0

0

200

400

600

800

1000

iteration

Here we adopt a Gaussian exploration parameterized by θ as: π θ ( · | x ) = N ( θ 1 x + θ 2 , exp( θ 3 )) , and we also parameterize the value function by ϕ as V ϕ ( x ) = 1 2 ϕ 2 x 2 + ϕ 1 x + ϕ 0 . (In fact, as shown in [58, Theorem 4], the optimal exploration and value functions are of this form, and constants such as θ and ϕ can be computed explicitly given the model dynamics.) We randomly choose a set of initial constants, and compute the optimal θ ∗ and ϕ ∗ with respect to these parameters; refer to Appendix D.1 for more details. Figure 1,2 show the convergence of algorithms for one certain realized trajectory.

In Figure 1, we compute the l 2 distance between the current policy parameters and the optimal ones, i.e. ∥ θ k -θ ∗ ∥ 2 , which tracks the convergence of the policy parameters. In Figure 2, we plot the sample estimated KL divergence between the current policy π k (specified by θ k ) and π ∗ (specified by θ ∗ ), i.e. E x ∼ d θ k µ D KL ( π θ k ( ·| x ) ∥ π θ ( ·| x )) . The reason to consider the KL-divergence between π k and π ∗ is that minimizing the KL-divergence to the optimal solution is equivalent to minimizing the distance between the current policy objective and the optimal objective (see Appendix D.1)). The experiments illustrate that our proposed algorithms do converge to the (local) optimum.

We also compare the performance of CPO and CPPO to the approaches that directly discretize the time, and then apply the classical discrete-time PG and PPO algorithms. See the details in Appendix D.4. The experiments show that our proposed CPO and CPPO are comparable in terms of sample efficiency, and in many cases they outperform the discrete-time algorithms under a range of time discretization.

2-dimensional optimal pair trading . We also consider the 2-dimensional optimal pair trading problem formulated in [36]. The state space is X = ( S, W ) ∈ R 2 with X (0) = ( s 0 , w 0 ) , where S represents the spread between two stocks, and W denotes the corresponding wealth process. The trader intends to maximize the total discounted reward, with the reward function r ( X,a ) = log(1 + W ) . The state dynamics are:

We set p ( x, a, π ) ≡ 0 , and add a constraint on the action: a t ∈ [ -ℓ, ℓ ] . (The action a t is the position taken on the first stock, which can be long/positive or short/negative.) Since the action space is bounded and continuous, we consider a beta distribution for policy parameterization: π θ ( a | X ) := f ( a + ℓ 2 ℓ , α θ ( X ) , β θ ( X ) ) with f ( x, α, β ) := Γ( α + β ) Γ( α )Γ( β ) x α -1 ( 1 -x β -1 ) . For α θ and β θ , we use a 3-layer neural network (NN) parameterized by θ for function approximation; and use another 3-layer NN for value function approximation. (More details are provided in Appendix D.2.)

Figure 3 shows that both algorithms, CPG and CPPO, converge to a local optimum (different between the two), and with an overall increasing trend over iterations. (Averaging is taken over 100 Monte Carlo estimates for each policy evaluation.)

1.350

CPG

CPPO

1.325

1.300

mWW

return (averaged)

1.275

1.250

1.225

1.200

1.175

1.150

0

20

40

60

80

100

120

iteration

5 Conclusion and Further Works

We have developed in this paper the basic theoretical framework for policy optimization in continuous RL, and illustrated its potential applications using numerical experiments.

For further research, two topics are high on our agenda. First, we plan to study the convergence (rate) of the continuous policy gradient and TRPO/PPO, vis-a-vis the error due to the time increment δ t . Our conjecture is that it is likely to be polynomial-bounded under mild assumptions, similar to the analysis in [20, 16]), thus extending beyond the condition required by [63] and [60, 31]. Second, for the bounds on the statistical distance and the performance difference, we want to further develop a consistent bound like the one in [11] (for the discrete setting), i.e., one that remains meaningful when the discount factor β → 0 .

Acknowledgments

Wenpin Tang gratefully acknowledges financial support through NSF grants DMS-2113779 and DMS-2206038, and through a start-up grant at Columbia University. The works of Hanyang Zhao and David Yao are part of a Columbia-CityU/HK collaborative project that is supported by InnotHK Initiative, The Government of the HKSAR and the AIFT Lab. We also thank Yanwei Jia and Xun Yu Zhou for helpful discussions.

References

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning , pages 22-31. PMLR, 2017.

Andrea Agazzi and Jianfeng Lu. Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regime. arXiv:2010.11858 , 2020.

Leemon C Baird. Reinforcement learning in continuous time: Advantage updating. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94) , volume 4, pages 2448-2453. IEEE, 1994.

Leemon C Baird III. Advantage updating. Technical report, WRIGHT LAB WRIGHTPATTERSON AFB OH, 1993.

Matteo Basei, Xin Guo, Anran Hu, and Yufei Zhang. Logarithmic regret for episodic continuoustime linear-quadratic reinforcement learning over a finite-time horizon. Journal of Machine Learning Research , 23(178):1-34, 2022.

Steven Bradtke. Reinforcement learning applied to linear quadratic regulation. Advances in neural information processing systems , 5, 1992.

Gerard Brunick and Steven Shreve. Mimicking an Itô process by a solution of a stochastic differential equation. Ann. Appl. Probab. , 23(4), 2013.

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712 , 2023.

Hans Buehler, Lukas Gonon, Josef Teichmann, and Ben Wood. Deep hedging. Quantitative Finance , 19(8):1271-1291, 2019.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

Jim G Dai and Mark Gluzman. Refined policy improvement bounds for mdps. arXiv preprint arXiv:2107.08068 , 2021.

Jim G Dai and Mark Gluzman. Queueing network controls via deep reinforcement learning. Stochastic Systems , 12(1):30-67, 2022.

Kenji Doya. Reinforcement learning in continuous time and space. Neural computation , 12(1):219-245, 2000.

Jianzhun Du, Joseph Futoma, and Finale Doshi-Velez. Model-based reinforcement learning for semi-markov decision processes with neural odes. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 19805-19816. Curran Associates, Inc., 2020.

Wendell H Fleming and Halil Mete Soner. Controlled Markov processes and viscosity solutions , volume 25. Springer Science & Business Media, 2006.

Xin Guo, Anran Hu, and Yufei Zhang. Reinforcement learning for linear-convex models with jumps via stability analysis of feedback controls. arXiv preprint arXiv:2104.09311 , 2021.

Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning , pages 1352-1361. PMLR, 2017.

Yilie Huang, Yanwei Jia, and Xunyu Zhou. Achieving mean-variance efficiency by continuoustime reinforcement learning. In Proceedings of the Third ACM International Conference on AI in Finance , pages 377-385, 2022.

David R Hunter and Kenneth Lange. A tutorial on MM algorithms. The American Statistician , 58(1):30-37, 2004.

Yanwei Jia and Xun Yu Zhou. Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach. Journal of Machine Learning Research , 23(154):1-55, 2022.

Yanwei Jia and Xun Yu Zhou. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms. Journal of Machine Learning Research , 23(154):1-55, 2022.

Yanwei Jia and Xun Yu Zhou. q-learning in continuous time. arXiv preprint arXiv:2207.00713 , 2022.

Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning , pages 267274, 2002.

Ioannis Karatzas and Steven E. Shreve. Brownian motion and stochastic calculus , volume 113 of Graduate Texts in Mathematics . Springer-Verlag, New York, second edition, 1991.

Jeongho Kim, Jaeuk Shin, and Insoon Yang. Hamilton-jacobi deep q-learning for deterministic continuous-time systems with lipschitz continuous controls. The Journal of Machine Learning Research , 22(1):9363-9396, 2021.

Kenneth Lange. MMoptimization algorithms . SIAM, Philadelphia, PA, 2016.

Kenneth Lange, Joong-Ho Won, Alfonso Landeros, and Hua Zhou. Nonconvex optimization via MM algorithms: Convergence theory. arXiv:2106.02805 , 2021.

James-Michael Leahy, Bekzhan Kerimkulov, David Siska, and Lukasz Szpruch. Convergence of policy gradient for entropy regularized MDPs with neural network approximation in the mean-field regime. In International Conference on Machine Learning , pages 12222-12252, 2022.

Jaeyoung Lee and Richard S. Sutton. Policy iterations for reinforcement learning problems in continuous time and space - fundamental theory and methods. Automatica , 126:109421, 2021.

Yuan Ling, Sadid A Hasan, Vivek Datla, Ashequl Qadir, Kathy Lee, Joey Liu, and Oladimeji Farri. Diagnostic inferencing via improving clinical concept extraction with deep reinforcement learning: A preliminary study. In Machine Learning for Healthcare Conference , pages 271-285. PMLR, 2017.

Boyi Liu, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural proximal/trust region policy optimization attains globally optimal policy, 2023.

Grégoire Loeper. Uniqueness of the solution to the Vlasov-Poisson system with bounded density. Journal de mathématiques pures et appliquées , 86(1):68-79, 2006.

Dhruv Madeka, Kari Torkkola, Carson Eisenach, Dean Foster, and Anna Luo. Deep inventory management. arXiv preprint arXiv:2210.03137 , 2022.

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928-1937. PMLR, 2016.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Supakorn Mudchanatongsuk, James A. Primbs, and Wilfred Wong. Optimal pairs trading: A stochastic control approach. In 2008 American Control Conference , pages 1035-1039, 2008.

Rémi Munos. Policy gradient in continuous time. Journal of Machine Learning Research , 7:771-791, 2006.

Rémi Munos and Paul Bourgine. Reinforcement learning for continuous stochastic control problems. In M. Jordan, M. Kearns, and S. Solla, editors, Advances in Neural Information Processing Systems , volume 10. MIT Press, 1997.

Bernt Øksendal. Stochastic differential equations . Springer, 2003.

Rémi Peyre. Comparison between W2 distance and H-1 norm, and localization of Wasserstein distance. ESAIM: Control, Optimisation and Calculus of Variations , 24(4):1489-1501, 2018.

Jim Pitman and Marc Yor. Hitting, occupation and inverse local times of one-dimensional diffusions: martingale and excursion approaches. Bernoulli , 9(1):1-24, 2003.

Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

Daniel Revuz and Marc Yor. Continuous martingales and Brownian motion , volume 293. Springer Science & Business Media, 2013.

Ahmad EL Sallab, Mohammed Abdou, Etienne Perot, and Senthil Yogamani. Deep reinforcement learning framework for autonomous driving. arXiv preprint arXiv:1704.02532 , 2017.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015.

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 , 2015.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

Daniel W. Stroock and S. R. Srinivasa Varadhan. Multidimensional diffusion processes , volume 233 of Grundlehren der Mathematischen Wissenschaften . Springer-Verlag, Berlin-New York, 1979.

Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning , 3:9-44, 1988.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems , 12, 1999.

Lukasz Szpruch, Tanut Treetanthiploet, and Yufei Zhang. Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models. arXiv:2112.10264 , 2021.

Corentin Tallec, Léonard Blier, and Yann Ollivier. Making deep q-learning methods robust to time discretization. In International Conference on Machine Learning , pages 6096-6104. PMLR, 2019.

Wenpin Tang, Yuming Paul Zhang, and Xun Yu Zhou. Exploratory HJB equations and their convergence. SIAM Journal on Control and Optimization , 60(6):3191-3216, 2022.

Kyriakos G Vamvoudakis. Q-learning for continuous-time linear systems: A model-free infinite horizon optimal control approach. Systems & Control Letters , 100:14-20, 2017.

Haoran Wang, Thaleia Zariphopoulou, and Xun Yu Zhou. Reinforcement learning in continuous time and space: A stochastic control approach. J. Mach. Learn. Res. , 21(198):1-34, 2020.

Haoran Wang and Xun Yu Zhou. Continuous-time mean-variance portfolio selection: A reinforcement learning framework. Mathematical Finance , 30(4):1273-1308, 2020.

Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence, 2019.

Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning , 8:279-292, 1992.

Jiongmin Yong and Xun Yu Zhou. Stochastic controls: Hamiltonian systems and HJB equations , volume 43. Springer Science & Business Media, 1999.

Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Basar. Global convergence of policy gradient methods to (almost) locally optimal policies. SIAM Journal on Control and Optimization , 58(6):3586-3612, 2020.

Yiming Zhang and Keith W Ross. On-policy deep reinforcement learning for the average-reward criterion. In International Conference on Machine Learning , pages 12535-12545. PMLR, 2021.

Appendix A Continuous RL: Formulation and Well-Posedness

A.1 Exploratory Stochastic-Control

For n, m positive integers, let b : R n × A ↦→ R n and σ : R n × A ↦→ R n × m be given functions, where A is a compact action space. A classical stochastic control problem [15, 62] is to control the state (or feature) dynamics governed by an Itô process, defined on a filtered probability space ( Ω , F , P ; { F B s } s ≥ 0 ) , along with an {F B s } -Brownian motion B = { B s , s ≥ 0 } :

where a s is the agent's action (control) at time s . The goal of the stochastic control (discounted objective over an infinite time horizon) is for any time-state pair ( t, x ) in (29), to find the optimal { F B s } s ≥ 0 -progressively measurable sequence of actions a = { a s , s ≥ t } (called the optimal policy) that maximizes the expected total β -discounted reward:

where r : R n ×A↦→ R is the running reward of the current state and action ( X a s , a s ) , and β > 0 is a discount factor that measures the time-depreciation of the objective value (or the impatience level of the agent). Note that the state process X a = { X a s , s ≥ t } depends on the starting (initial) time-state pair ( t, x ) . For ease of notation, we denote by X a instead of X t,x,a = { X t,x,a s , s ≥ t } the solution to the SDE in (29) when there is no ambiguity.

Listed below are the standard assumptions to ensure the well-posedness of the stochastic control problem in (29)-(30).

Assumption 2. The following conditions are assumed throughout: (i) b, σ, r are all continuous functions in their respective arguments;

(ii) b, σ are uniformly Lipschitz continuous in x , i.e., there exists a constant C > 0 such that for φ ∈ { b, σ } ,

(iii) b, σ have linear growth in x and a , i.e., there exists a constant C > 0 such that for φ ∈ { b, σ } ,

(iv) r has polynomial growth in x and a , i.e., there exists a constant C > 0 and µ ≥ 1 such that

The key idea underlying exploratory stochastic control is to use a randomized policy (or relaxed control), i.e., apply a probability distribution to the admissible action space. To do so, let's assume the probability space is rich enough to support a uniform random variable Z that is independent of the Brownian motion B = { B t } . We then expand the original filtered probability space to ( Ω , F , P ; {F s } s ≥ 0 ) , where F s = F B s ∨ σ ( Z ) (i.e., augment F B s with the sigma field generated by

Let π : R n ∋ x ↦→ π ( · | x ) ∈ P ( A ) be a stationary feedback policy given the state at x , where P ( A ) is a suitable collection of probability distributions (with density functions). At each time s , an action a s is generated from the distribution π ( · | X a s ) , i.e. the policy only depends on the current state. In other words, we only consider stationary, or time-independent feedback control policies for the stochastic control problem (29)-(30).

Given a stationary policy π ∈ P ( A ) , an initial state x , and an {F s } -progressively measurable action process a π = { a π s , s ≥ 0 } generated from π , the state process X π = { X π s , s ≥ 0 } follows:

defined on ( Ω , F , P ; {F s } s ≥ 0 ) . It is easy to see that the dynamics in (34) define a time-homogeneous Markov process, such that for each t ≥ 0 and x :

Consequently, the objective in (30) is independent of time t , and is equal to:

Furthermore, following [58], we can add a regularizer to the objective function to encourage exploration (represented by the randomized policy), leading to

where p : R n ×A×P ( A ) ↦→ R is the regularizer, and γ ≥ 0 is a weight parameter on exploration (also known as the 'temperature" parameter). For instance, in [58], p is taken as the differential entropy,

and hence, the 'entropy' regularizer. The same argument as before justifies that V ( t, x ; π ) is independent of time t . That is, for all t ≥ 0 ,

which is the state-value function under the policy π , V ( x ; π ) , in (4), and which, in turn, leads to the performance function η ( π ) in (6). Moreover, recall the main task of the continuous RL is to find (or approximate) η ∗ = max π η ( π ) , where max is over all admissible policies.

A.2 Controlled SDE and the HJ Equation

Note that the exploratory state dynamics in (34) is governed by a general Itô process. It is sometimes more convenient to consider an equivalent SDE representation- in the sense that its (weak) solution has the same distribution as the Itô process in (34) at each fixed time t . It is known ([58]) that when n = m = 1 , the marginal distribution of { X π s , s ≥ 0 } agrees with that of the solution to the SDE, denoted by { ˜ X s , s ≥ 0 } :

where ˜ b ( x, π ( · )) = ∫ A b ( x, a ) π ( a )d a and ˜ σ ( x, π ( · )) = √∫ A σ 2 ( x, a ) π ( a )d a . This result is easily extended to arbitrary n, m , thanks to [7, Corollary 3.7], with the precise statement presented below (assuming n = m for ease of exposition).

Theorem 6. Assume that for a policy π and for every x ,

is positive definite. Then there exists a filtered probability space ( ˜ Ω , ˜ F , { ˜ F t } t ≥ 0 , ˜ P ) that supports a continuous R n -valued adapted process ˜ X and an n -dimensional Brownian motion ˜ B satisfying

where

For each s ≥ 0 , the distribution of ˜ X s under ˜ P agrees with that of X π s under P defined in (34).

As a consequence, the state value function in (37) is identical to

Also define

so we can simplify the value function to

Following the principle of optimality, V then satisfies the HJ equation:

To guarantee that the HJ equation in (40) characterizes the state-value function in (39), we need

Assumption 3. Assume the following conditions hold:

(i) b, σ, r, p are all continuous functions in their respective arguments.

(ii) b, r, p are uniformly Lipschitz continuous in x , i.e., there exists a constant C > 0 such that for φ ∈ { b, r } ,

and

(iii) ˜ σ is globally bounded, i.e., there exist 0 < σ 0 < ¯ σ 0 such that

(iv) the SDE (38) has a weak solution which is unique in distribution. (v) π ( a | x ) is measurable in ( x, a ) and is uniformly Lipschitz continuous in x , i.e., there exists a constant C > 0 such that

Theorem 7. Under Assumption 3, the state-value function in (39) is the unique (subquadratic) viscosity solution to the HJ equation in (40) .

Proof. By [56, Section 3.1], the HJ equation in (40) has a unique (subquadratic) viscosity solution under the conditions (i)-(iii). Further by [21, Lemma 2], the viscosity solution is the state-value function.

Appendix B Proofs of Main Results (in §3)

B.1 Proof of Theorem 2

Recall in the proof sketch of the Theorem in §3, we have defined the operator L π : C 2 ( R n ) ↦→ C ( R n ) as

which leads to the following characterization of the HJ equation:

We need the following two lemmas concerning the operator L π .

Lemma 8. For any φ ∈ C 2 ( R n ) , we have

Proof. The left hand side of the above equation is

where the first equality follows from the definition of the occupation time and the third equality from Itô's formula. □

Lemma 9. Let π, ˆ π be two feedback policies. We have

Proof. By definition of q ( x, a ; π ) in (11), we have

□

Proof of Theorem 2 . Note that in (13), the equation to be proven, the right hand side can be written as ∫ R d ˆ π µ ( y ) f ( x ; π, ˆ π )d y , with

From Lemma 9, we have

On the other hand, for the left hand side of (13), we have

with the second equality following from Lemma 8; and

following the definition of the discounted expected occupation time; moreover, from (41), we have

Hence, combining the last three equations (44,45,46), we have

Thus, we have shown LHS=RHS in (13).

B.2 Proof of Theorem 3

Proof . It suffices to show the integral version of the theorem:

As before, we simplify notation by denoting η ( π θ ) as η ( θ ) and d π θ as d θ . Then, by Theorem 2), we have

Denote

Note that f (0) = 0 , which follows from

Thus,

Dividing both sides by δθ completes the proof, as the first term on the last line above is of higher order than δθ . □

B.3 Proofs of Lemma 4 and Theorem 5

We need a lemma for the perturbation bounds.

Lemma 10. Assume that both ˜ σ 2 ( x, ˆ π ( · )) and ˜ σ 2 ( x, π ( · )) are positive definite and

where σ 0 > 0 , then we have that the difference between the square root matrix is bounded by

If we also assume that the upper bounds, i.e.

by some ¯ σ 0 > σ 0 > 0 , then we have

Proof . Consider a normalized vector x with ∥ x ∥ 2 = 1 is an eigenvector of A 1 2 -B 1 2 with eigenvalue µ then

thus, if A,B ≥ σ 2 0 I , this implies

Furthermore, note that

so

Proof (of Lemma 4). Consider the Wasserstein-2 distance W 2 ( µ, v ) between distribution µ and v as

where Γ( µ, ν ) is the set all probability measures on the product space R n × R n with the marginal distributions being µ and v , and ∥ · ∥ 2 is the standard Euclidean distance. Denote

We want to get an upper bound on W 2 ( ¯ d π µ , ¯ d ˆ π µ ) in terms of the distance between two policies π and ˆ π . Consider a specific coupling ( X t , Y t ) below:

with X 0 = Y 0 , which leads to a joint distribution over R n × R n :

Hence,

It then boils down to estimating E ∥ X s -Y s ∥ 2 2 . By Itô's formula,

Taking expectation on both sides yields

with

and

Combining the above, we get

By Grönwall's inequality, we have

Substituting back into (51), we obtain

Thus, if β > C ˜ b, ˜ σ , we have

Concerning the term C ( π, ˆ π ) , we have

and

Thus we have:

which proves our upper bound.

Proof (of Theorem 5). We have that

where K := sup ˆ π ∥ f ∥ ˙ H 1 < ∞ (more about K in the remarks below). Combining (54) with the estimate in (22) (of Lemma 4) yields the desired result in (23). □

Remarks (on K ). In the performance-difference bound developed above, we assume K is finite:

where f ( x ; π, ˆ π ) := ∫ A ˆ π ( a | x ) ( q ( x, a ; π ) + p ( x, a, ˆ π )) d a . The famous Poincaré inequality can provide a lower bound on this quantity; but we need an upper bound as well, i.e.,

This above is essentially a reverse Poincaré Inequality, which is not likely to hold (in particular, the existence of the constant C ).

Should we indeed have a reverse Poincaré Inequality, then we can further bound f by

and

where the second inequality is from Pinsker's inequality. This way, we would have recovered a similar bound as in the discrete RL. Since we do not have the reverse Poincaré inequality, however, we have to assume that K is finite.

Appendix C Algorithms

C.1 Performance of CPPO with Square-root KL and Linear KL

Here we present a detailed version of the CPPO algorithm. For two probability distributions P and Q over the action space with density functions p and q correspondingly, recall that the KL-divergence between these two is defined as:

Note that bounding the performance difference by the linear KL-divergence D KL ( θ, θ k ) , instead of its square-root counterpart ¯ D KL ( θ ∥ θ k ) , will generally require stronger conditions (which may be difficult to satisfy). For completeness, we present the following algorithm as an important benchmark, the CPPO with linear KL-divergence:

Algorithm 3 CPPO: PPO with adaptive penalty constant (linear KL-divergence)

Input : Policy parameters θ 0 , critic net parameters ϕ 0

1: for k = 0 , 1 , 2 , · · · until θ k converge do

2: Collect a truncated trajectory { X t i , a t i , r t i , p t i } , i = 1 , . . . , N from the environment using π θ k .

3: for i = 0 , . . . , N -1 do : Update the critic parameters as in (8)

4: for j = 1 , , . . . , J do : Draw i.i.d. τ j from exp( β ) , round τ j to the largest multiple of δ t no larger than it, and compute the GAE estimator of q ( X τ j , a τ j )

5: Compute policy update (by taking a fixed s steps of gradient descent)

6: if D KL ( θ k +1 ∥ θ k ) ≥ (1 + ϵ ) δ , then C k +1 penalty = 2 C k penalty .

7: else if D KL ( θ k +1 ∥ θ k ) ≤ δ/ (1 + ϵ ) , then C k +1 penalty = C k penalty / 2 .

A comparison between the above and Algorithm 2 (using square-root KL divergence) is presented in §D.3 below, which clearly illustrates the advantage of square-root KL divergence.

C.2 KL-divergence

We elaborate here on the KL-divergence between the current policy and the optimal policy, along with the entropy regularizer. By the performance difference formula, we have

Notice that by the definition of KL-divergence we defined before, we have

Similar as the previous discussion of soft q -learning, π ∗ is optimal implies that

and the normalization constant is 1 can be proved through considering the exploratory HJB equation, see [22, 56]. Thus

which leads to where

This justifies our claim that the KL-divergence is essentially equivalent to the distance to the optimal performance.

Appendix D Experiments

D.1 Example 1

Recall, in the LQ control problem, the reward function is

with M ≥ 0 , N > 0 , R, Q, P ∈ R and R 2 < MN , and we adopt the entropy regularizor as

Furthermore, suppose that the discount rate satisfies β > 2 A + C 2 +max ( D 2 R 2 -2 NR ( B + CD ) N , 0 ) .

The following results are readily derived from Theorem 4 of [58]. The value function of the optimal policy π ∗ is

and

respectively. Moreover, the optimal feedback control is Gaussian, with density function

For a set of model parameters: A = -1 , B = C = 0 , D = 1 , M = N = Q = 2 , R = P = 1 , β = 1 , γ = 0 . 1 , following the formulas and the parameterized policy π θ ( · | x ) = N ( θ 1 x + θ 2 , exp( θ 3 )) , and the corresponding value function V ϕ ( x ) = 1 2 ϕ 2 x 2 + ϕ 1 x + ϕ 0 , we can derive the optimal parameters:

and

D.2 Example 2

The model parameters are k = 0 . 01 , θ = 7 , η = 0 . 1 , ρ = 0 . 3 , σ = 1 , r f = 0 . 01 , ℓ = 5 . For both the value function and the policy parameterization, we use a 3-layer neural network, and with the initial parameters sampled form the uniform distribution over [-0.5,0.5]. We use the tanh activation function for the hidden layer.

D.3 Performance of CPPO with Square-root KL and Linear KL

We compare the performance of CPPO with square-root KL-divergence (denote as CPPO), and linear KL-divergence (denoted as CPPO (nst) - non square-root) applied to the experiments in Example 1 and Example 2. Figure 4 compares the distance between the current policy parameters and the optimal parameters, with x -axis denoting the iteration times and y -axis denoting the L 2 distance. Figure 5 compares the current expected return, with x -axis denoting the iteration times and y -axis denoting the current performance by taking the average of 100 times of Monte Carlo evaluation. In

CPPO

1.0

CPPO_nst

to the optimal

0.8

0.6

distance t

0.4

2

0.2

0

200

400

600

800

1000

iteration

1.35

CPPO

CPPO_nst

1.30

1.25

1.20

1.15

1.10

1.05

0

20

40

60

80

100

120

both figures, the blue curve represents the algorithm with square-root KL-divergence as opposed to the orange one corresponding to the linear version. Both figures clearly demonstrate the advantage of the former. In particular, the linear version can suffer from getting stuck at the local optimum as demonstrated in Example 1.

D.4 Performance of CPG and CPPO compared to the classical discrete-time algorithms

We conduct experiments to compare the CPG and CPPO to their discrete counterparts. Specifically, we discretize the MDP in Example 1, and implement the classical PG and PPO algorithms. Our results show that in time discretization with step size δt = 0 . 1 and δt = 0 . 05 , the performance of CPG and CPPO is (at least) comparable to their discrete counterparts; in particular, for δt = 0 . 1 , CPG outperforms PG. We have repeated the experiments for 25 random seeds, and plotted both the average performance line and the error bar. These experimental results indicate that the continuous approach has the potential to outperform their discrete counterparts, which is worth further exploring in the future.

CPG-L2

1.0

5t=0.05

0.8

0.6

0.4

0.2

0.0

0

100

200

300

400

500

DPG-L2

5t=0.05

1.0

0.8

0.6

0.4

0.2

0

100

200

300

400

500

CPG-L2

1.0

0.1

0.8

0.6

0.4

0.2

0.0

0

100

200

300

400

500

CPG-KL

20.0

5t=0.05

17.5

15.0

12.5

10.0

7.5

5.0

2.5

0.0

0

100

200

300

400

500

DPG-KL

20.0

5t=0.05

17.5

15.0

12.5

10.0

7.5

5.0

2.5

0.0

0

#

100

200

300

400

500

CPG-KL

20.0

5t=0.1

17.5

15.0

12.5

10.0

7.5

5.0

2.5

0.0

0

100

200

300

400

500

DPG-L2

1.0

5t=0.1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0

100

200

OOE

400

500

CPPO-L2

1.0

5t=0.05

0.8

0.6

0.4

0.2

0.0

0

50

100

150

200

250

DPPO-L2

1.2

0=0.05

1.0

0.8

0.6

0.4

0.2

0.0

0

50

100

150

200

250

DPG-L2

20.0

ot=0.1

17.5

15.0

12.5

10.0

7.5

5.0

2.5

0.0

0

100

200

300

400

500

CPPO-KL

20.0

5t=0.05

17.5

15.0

12.5

10.0

7.5

5.0

2.5

0.0

0

50

100

150

200

250

DPPO-KL

20.0

17.5

15.0

12.5

10.0

7.5

5.0

2.5

0.0

0

50

100

150

200

250

CPPO-L2

5t=0.1

1.0

0.8

0.6

0.4

0.2

0.0

0

50

100

150

200

250

DPPO-L2

1.0

0.8

0.6

0.4

0.2

0.0

0

50

100

150

200

250

CPPO-KL

20.0

5t=0.1

17.5

15.0

12.5

10.0

7.5

5.0

2.5

MW

0.0

0

50

100

150

200

250

DPPO-KL

20.0

5t=0.1

17.5

15.0

12.5

10.0

7.5

5.0

2.5

0.0

0

50

100

150

200

250

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nReinforcement learning (RL) is a powerful tool for solving sequential decision making problems but has primarily been formulated for discrete-time Markov decision processes. However, many real-world systems are more naturally expressed in continuous time, and the proper choice of discretization time step may be challenging. Additionally, a controller operating in continuous time may be more suitable for high frequency applications. Prior work has formulated a continuous-time analog to the advantage function, called the q-value. Building on the concept of a q-value, the paper sets out to derive a continuous-time analog to the discounted occupation probability and performance difference lemma, two core concepts in RL. The authors then show how to compute a gradient of this performance difference lemma to give a continuous analog to the policy gradient. Next, the authors form a local approximation of the performance difference which takes an expectation over the current policy rather than the updated one. They provide bounds on the gap between the true performance metric and this local approximation. Then they use this local approximation to formulate a continuous analog to TRPO\\PPO. The paper evaluates the continuous-time policy gradient and PPO algorithms on synthetic linear-quadratic stochastic control problems. They also consider a two-dimensional optimal pair trading problem and show their CPPO algorithm performs best.\n\nSTRENGTHS:\n- Continuous-time RL is an under-explored area that has application in controlling systems at high frequency and using an adaptive discretization scheme with non-uniform time steps.\n- This appears to be the first work to propose a continuous-time analog in a stochastic setting to the performance difference lemma and derive a continuous PPO algorithm.  \n- The paper is well organized and clearly written. It does a good job explaining the results and provides enough information to support its claims.\n\nWEAKNESSES:\n- There is no discussion or experimental results which illustrate how the continuous-time formulation is advantageous over discrete-time RL. The proposed algorithms still ultimately require us to discretize. As such, there should be some comparison to existing discrete-time RL methods. The hope would be that formulating the problem in continuous time allows us to achieve better performance under certain choices of step size. Another potential benefit of continuous-time RL is the case of unevenly-spaced observations. Experiments which highlight these benefits would significantly strengthen the paper.\n- The problems considered are toy examples which do not tell us much about the scalability of the proposed methods. More complex scenarios would make the paper much more convincing. Especially ones in which the continuous-time formulation provides a clear advantage.\n- The discussion of related work could be better. It is still a little unclear to me how this work is positioned in the continuous-time RL literature.\n\nQUESTIONS:\n- How do these continuous-time methods compare to their discrete-time counterparts when the time step discretization is uniform? How does the performance gap change with choice of discretization time step? What are the main advantages of this formulation?\n- How does this approach scale to harder, more realistic problems?\n- How exactly does the continuous-time policy gradient algorithm in this paper compare to previously proposed approaches?\n\nLIMITATIONS:\nThere is no discussion of limitations of the continuous-time formulation or their specific algorithms. The paper would be stronger if it discussed these. This could include assumptions made in the proofs or the fact that the work assumes the dynamics follow an Ito SDE driven by Brownian motion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the continuous reinforcement learning problem. The proposed method is based on the notion of occupation time for policy gradient, which is analogous to the visitation frequency in discrete Markov decision processes. Empirical evaluations are conducted on two example scenarios for two versions of the algorithm (CPG and CPPO).\n\nSTRENGTHS:\n- Analyze an Important formulation of continuous time and space in reinforcement learning.\n- Theoretical analysis is present to develop a policy gradient counterpart of TRPO/PPO.\n- The empirical evaluation is presented along with theoretical analysis.\n\nWEAKNESSES:\n- Lack of comparison with discrete counterpart.\n- The lack of implementation details (missing hyperparameter, random seed) makes reproducibility challenging.\n- The evaluation of the proposed method is limited to only two hand-crafted examples, which hinders a comprehensive understanding of its implications.\n\nQUESTIONS:\nIn Figures 1 and 2, what does it mean to be the distance to the optimal to be some non-zero value? How does this translate to the reward performance? Reward performance is critical to know whether or not the achieved difference in distance is useful.\n\nThe performance is compared with the proposed two methods, CPG and CPPO (Figure 3), and no baseline is considered. An essential choice would be to use the discrete counterpart (Policy Gradient - PG and PPO). A simple form can be to make the MDP discrete and run PG and PPO. This comparison is even important to understand what kind of problem the continuous RL is useful for.\n\nIs the achieved return by CPPO optimal (empirical best) in Figure 3? \n\nImplementation details need to be included; what are the PPO-specific hyperparameters (e.g., clipping), for Example 2?\n\nFor reproducibility, a common practice is using several seed runs to account for variation in the results. How many random seeds are used for the experiments in Figures 1 and 2? These missing details make the presented results hard to reproduce.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper seeks to answer three research questions: 1) Is there a continuous time analog of the state occupancy measure, 2) Is there a convenient expression for the performance difference between two policies in the continuous time setting, and 3) Can PPO be adapted to fit the continuous time setting? In answering these questions, the paper presents a continuous time occupancy measure, a policy performance difference similar to that of conservative policy iteration and related algorithms, and continuous versions of REINFORCE and PPO are provided. Theoretical results for several properties are given, and experiments demonstrating that the algorithms were able to solve a couple of problems are provided.\n\nSTRENGTHS:\nThe paper provides a thorough theoretical treatment of defining policy gradients for the continuous time RL setting. The paper clearly defines and adequately answers its stated objectives.\n\nWEAKNESSES:\nThe biggest area for improvement in this paper is in the empirical results. While the main results of this paper are, theoretical new practical algorithms are presented. Thus, they deserve proper evaluation and experimentation to educate the reader on the challenges of using them. \n\nFor example, there are no experiments illustrating that there were any special difficulties in applying these algorithms to the continuous setting. There should be experiments illustrating how the hyperparameters, particularly those specific to the continuous time setting, impact the optimization process. Currently, the results only tell us that the algorithms were made to work. Great, but we do not learn anything beyond this trivial result. It is important to develop the reader’s understanding of how they can make the algorithms work or at least what can cause failure. \n\n\nAdditionally, the adaptive penalty term should also be explored since it deviates from the standard PPO implementation. Does it effectively constrain the distribution throughout learning? How does the penalty change over time? These are important questions to answer when introducing a new method. \n\n\nMinor quibble: the opening sentence of the introduction is boring. Many papers have used this form, and my eyes glaze over as soon as I read them. Try keeping this motivation brief and tie it immediately to the scope of the work discussed in this paper.\n\nQUESTIONS:\n- How does the sampling at discretization impact the gradient estimate? \n\n- How does the step size need to be adapted to work in the continuous-time setting? Is it sensitive to the scaling of the time horizon or sampling rate?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}