Policy Optimization for Continuous Reinforcement Learning
Authors: Hanyang Zhao
Wenpin Tang
David Yao
Venue: NeurIPS 2023 poster (2023)
Keywords: exploratory stochastic control
occupation time
performance difference
policy optimization

Abstract: We study reinforcement learning (RL) in the setting of continuous time and space, for an infinite horizon with a discounted objective and the underlying dynamics driven by a stochastic differential equation. Built upon recent advances in the continuous approach to RL, we develop a notion of occupation time (specifically for a discounted objective),  and show how it can be effectively used to derive performance difference and local approximation formulas. We further extend these results to illustrate their applications in the PG (policy gradient) and TRPO/PPO (trust region policy optimization/ proximal policy optimization) methods,  which have been familiar and powerful tools in the discrete RL setting but under-developed in continuous RL. Through numerical experiments, we demonstrate the effectiveness and advantages of our approach.

Policy Optimization for Continuous Reinforcement Learning
Hanyang Zhao Columbia University h226840columbe umbiiaaaoolbba....mo edu
Wenpin Tang Columbia University wt2319@columbia. edu
David D. Yao Columbia University yao coo@columiia.euu
Abstract
We study reinforcement learning (RI.) in the setting of continuous time and space, for an infinite horizon with a discounted objective and the underlying dynamics driven by stochastic differential equation. Built upon recent advances the continuous approach to RL, we develop a notion of occupation time (specifically for discounted objective), and show how it be effectively used derive iifoormnnne-ifffreeme and approximatinn formulas. We further extend these results illustrate their applications the PG (policy gradient) and TRPO/PPO (trust region policy optimization/ proximal policy optimization) methods, which have been familiar and powerful tools in the discrete RL setting but under-developed in continuous RL. Through numerical experiments, we demonstrate the effective- ness and advantages of our approach.
1 Introduction
Reinforcement Learning (RI., (52) has been successfully applied to wide-ranging domains in the past decade, including achieving superhuman performance in games like Atari and Go 35 48
49 enhancing Large Language Models using human feedback g 10], and showing potentials in improving traditional model-based decisions in healthcare, inventory management, and finance 30 (33. Most existing works, including all references cited above, are formulated and solved as discrete-time sequential optimization problems such Markov decision processes (MDPs, 42).. Yet in many applications, agents may need to monitor and interact with the random environment at an ultra-high frequency (e.g., autonomous driving, robot navigation, and high-frequency stock trading), which calls for a continuous sectinuouu-iin//spaaee time/space approach.
Recent years have witnessed a fast growing body of research that has extended the frontiers of contin- uous RI. several important directions including, for instance, modeling the noise or randomness in the environment dynamics as following stochastic differential equation (SDE), and incorporating an entropy-basee regularizer into the objective function (58) facilitate the exploration cetttiiooeeelliitttonnnn tradeoff:: designing model-free methods and algorithms, along with applications to portfolio optimiza- tion 18
22]; studying regret bounds [5] 54], and so forth.
In this paper, we continue the above trend in continuous RL, focusing on an infinite horizon formula- tion with a discounted objective and the underlying dynamics driven by an SDE 24 39 We are specifically motivated by the following two questions.
(Q1) The visitation frequency in MDP (with discounted objective) is defined as: p(s) E -e
P(Y: = *), where (Y:) is Markov chain with state space S = {s}. and E (0, 1) is a discount factor. It plays important role in many RL algorithms for MDP. So, natural question is, what is the continuous counterpart of p(s)?
(Q2) For continuous RL, how can we characterize the difference in performance between two policies? In particular, can we derive performance-differeee formulas similar to those the MDP
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

case 23 45 ? Can we adapt and apply the ideas and tools of the efficient policy optimization methods to the continuous RL setting?
(e.g., 45
Main contributions. We provide a unified theory/framework for policy optimization in continuous time and space. Specifically, we have addressed the above two questions (Q1) and (Q2) by developing the notion of occupation time/measure, specifically for discounted objective, and focusing on its associated g-value. Based on these two quantities, we derive the performance-ciifeeeee
formula for formula, we develop the continuous counterparts of the policy gradient (PG, (531) and also propose the local approximation for the performance metric. for which we derive a bound on it and allow the development of minorization majorizatiin (MM) algorithm. We further develop the continuous minorization-majorrzaatooo
continuous RL by means of perturbation analysis. Leveraging the performance-differe
counterparts of trust region policy optimization/ proximal policy optimization (TRPO/PPO) methods in 477, which have been familiar and powerful tools in the discrete RI. setting but under-developed in continuous RI, as approximations to the previous algorithms. (What is worth mentioning is that these policy optimization algorithms do not require any a priori discretization of time and space.) Through numerical examples we show the convergence of these algorithms when applied to certain stochastic control tasks in continuous time and space.
Organization of the paper. In Section 2 we present the continuous RL formulation and develop difference
necessary tools. The main results, the oerformance-dffferencee formula (Theorem (2 and the bound (Theorem 5] are provided in Section 3 In Section 4 we propose two algorithms, policy gradient with random rollout and PPO with adaptive penalty, based on our analyses and theoretical results; and illustrate their performance via numerical experiments. Concluding remarks are summarized in
Section 5
Related works, One line of research on continuous RL focuses on modeling the underlying dynamics as deterministic system, typically following deterministic ordinary differential equation. Several papers 99 37 444 solve the problems via a priori discretization in either time or space; [13] develops framework apply the temporal difference to the continuous setting, and proposes algorithms that combine value iteration or advantage update as in 4 6 to avoid explicit discretization; (37 further investigates policy gradient methods, followed by more recent studies on model-free continuous RL or mrodel-based ones [4]; (55) studies the sensitivity of existing off-policy algorithms along with advantage updating to propose continuous RI. algorithms that are robust to time discretization.
methods 25 29 57]
The formulation of continuous RL in a stochastic setting (i.e., with the state process driven by an SDE), can be traced back to (38). which however provides no data-driven solution. Recently, (58)
develops an exploratory control model for the continuous RL. Built upon this approach and for
finite-horizon objective, (20) studies policy evaluation, and 21 policy gradient. Furthermore, [22]
brings forth the notion of v-vuue,, which leads
continuous analogue of Q-learning. Also worth
noting is 28), which studies RL in the mean mean-field regime where continuous-tinn time processes occur
in the limit, and [16] extends the study to jump- diffussonn processes.
In discrete-time time MDPs, the body of research on bounding the performance difference between two policies also relates to work: [1 45 develop a policy improvement bound for the discounted total reward; 12 64] studies the long-run average reward, and 11 proposes a bound that is continuous with respect to the discount factor.
Notation.
For a measurable set A. denote P(A) for the set of probability distributions over A. For a vector z, denote by ||ا|| the Euclidean norm of x. For matrix A, denote by ||A|[[ the Frobenius norm of A, and 2 : AAT where A is the transpose of A. For positive-definite matrix A, denote by A the square root matrix of A. For A, B two matrices of the same size, denote by A B the inner product of A and B. For a function f on an Euclidean space, V (resp. V²f) denotes the gradient (resp. the Hessian) of f. For two distributions P. Q E P( denote by W2(P, Q) the Wasserstein-2 distance (or Quadratic Wasserstein distance) between P and Q:
E( (,eint inf 2/13
1/2
Q)) =
(ET(P..)
where I'(P, Q) is the set of all couplings of and Q: and denote by DKL (P(l() the KL-divergence (p(x) dr, in which p and g denote the probability between P and Q: (Pl()))Q) p(x) log ((x)) densities of P and Q.
2

---

2 Formulation and Preliminaries
Continuous RL. We start with quick formulation of the continuous RL, based on the same modeling framework as in [58]. Assume that the state space is R", and denote by A the action space. Let T(- z) P(A) be (state) feedback policy given the state x E R" A continuous RL problem is formulated by a distributional (or relaxed) control approach [62], which is motivated by the trial and error process in RL. The state dynamics (Xş, $ 2 0) is governed by the Itō process:
dx 5 b (X", ag) ds + (X", ag) dB., X5 ~ P E P(RR),
(I)
where (B,, 12 0) is the m-dimensional Brownian motion, b R x x A R. X R_xm and the action 0₂ generated from the distribution x (- X²) by external randomization. To avoid technical difficulties, we assume that the stochastic processes (and (3) (9) below) are well-defined,
see [24, Section 5.3] or [50] Chapter 6] for background.
From now on, write (X a.) for the state and action at time s given by the process I under the policy - (-(- æ)) E P(A) x ]." The goal here is find the optimal feedback policy ". that maximizes the expected discounted reward over an infinite time horizon:
[
e BB (x a + pp (X. a Xx)) X6 ~ 11
V = max E ①
(2)
where r : R x A ET the running reward of the current state and action (X a.. p R" x A x P(A) + R is a regularizer which facilitates exploration (e.g., in [58), is taken as the differential entropy defined by p(z, a, (+)) - log (a));; 7 2 0 is a weight parameter on exploration (also known as the "temperature" parameter); and B > 0 is a discount factor that measures the den-eeppeciation of the objective value (or the impatience level of the agent).
Performance metric. A standard approach to solving the problem in ( is to find a sequence of policies म = {- ar) : ar R") ). k 2,2..... such that the value functions following the policies Th will converge to V*, or be at least increasing in k; i.e., demonstrating policy improvement.
Given a policy (-)), let b(æ, x()) == ʃA b(æ, (()dd and ozz, x()) := (S o² a) $
Assume (for technical purpose) that ā (ar, (-)) is positive definite for every E RD. It is sometimes more convenient to consider the following equivalent SDE representation of (I:
dX, b [x.: (- x.)) ds + ā (Xe. (xa) dB., Xo~
(3)
in the sense that there exists probability measure Po which supports the m-dimennional Brownian
motion (B.. 2 0), and for each 2 O. the distribution of X, under P agrees with that of X, under P defined by (I). see Appendix A Note that the dynamics in (3) does not require external randomization. Also set F(zr, TT == ja r(ar, (a)dd and f(z, v) : SA ((x, a, x ())(a)da. We
formally define the (state) value function given the feedback policy {-( I)) x R" by
V(a;; m)
E
98 (r (x a.) + 7P (X 0g ; क (. (x) ds
Xx x
E
(X,, ((: x A { म[- (
ds
Ao. x
which, under suitable conditions on model parameters o, p) and the policy ग, is characterized
by the Hamilton-Jaobb equation (see [2] 56)):
VV T) b(x, T) (V - 1 (x, V2V(2; F(z, T) >5(x, = 0. (5)
More technical details regarding the above formulation are spelled out in the Appendix.
We can now define the performance metric as follows:
πイ()) / ((zT))u(dx),
(6)
so V max, 1/()). The main task of the continuous RL. to approximate max, 7)()) by construct- ing sequence of policies Ti, k- 1,2,... .. recursively such that 77(T))) is necedccra...
3

---

Policy evaluation. Let's first recall general approach in (00,, which can be used to learn the state value function in 44) or the performance metric in (6) for a given policy и. The idea that for any T > O and suitable test process (G,, I. 12200 0),
E ! & jav (XF:x) + (X2 of dt + Tp (XF, (-1x7)) dt 3V (xT dt] 0. (7)
If we parameterize (z; hT (x) and choose the special test function & approximation leads the online update:
(v*( de
stochastic
8V (XF) jav (xT) + (X7, of) d (XT. xT)) 3V (x7) dt), ao
Φ - +
(8)
where 0 is the learning rate. This recovers the mean-squared TD error (MSTDE) method for policy evaluation in the discrete RI [51).
g-value. The Q-value function 66 and the advantage function 3 34 in discrete-time MDPs play critical role in reinforcement learning theory and algorithms. However, as pointed out (55), these concepts will not apply when the time interval shrinks to (as in the continuous setting). To derive algorithms that fit the need of a continuous stochastic environment, [55] 22 proposed the advantage rate function. Namely, given policy π and (t, x. a) [0, o0) x R A, consider "perturbed" policy á as follows. takes the action a E A on [t, t + At) where At > O, and then follows и on + At, o0). The corresponding state process X given X x is broken into two pieces. On [t,t + At), it is X which is the solution to
(9) while on [t + At, 20), it is X following (3) with the initial time-state pair (t+ At, xfxx( With At > 0 as time discretization, the generalization of the conventional Q-function be expressed as
dx. = b (Xg, a) ds + a (X,,a) dB,, , 8 [t, At), X² x.
Qsi a; T) V(x;)) + Ha
OV
(x;;
Q2v
(o:
BV (V; (1; At o(Ar). (10)
dr
dx
where (z, y, A) == b(x a) y 2000 (z, a) A + r(x, is the (generalized) Hamilton function in stochastic control theory [62]. This motivates the following definition.
Definition 1. (22) For given policy II and (x. a) E R. x A, define the q-value as
Qarla, a; V(;;)) = 7H
OV
(x; Ox (1::))
82V
q T) == lim At-0
BV (z; x),
At
dr
(11)
which represents the instantaneous advantage rate of an action a given state under a given policy,
Main Results
This section is concerned with theoretical developments. In Section 3.1 we define the (discounted) occupation time/measure which is the continuous analog of visitation frequency in MDPs. It is crucial in deriving the performance-differeeee formula in Section 3.2 which spins off two different algorithms- - policy gradient and TRPO/PPO. Section 3.3] we propose local approximation for the performance metric, and derive a bound from which an MM algorithm is constructed.
3.1 Discounted Occupation Time
Here first provide answer to (Q1) by defining the notion of discounted occupation time for the continuous RL.
Definition 2. Let X = (X1, t 2 0) be governed by the SDE 33, and assume that it has a probability density function Dd (.,L) at each time I. For each x E Rat and I. O, define the 3-discounted occupation time of X at the state by
d=(x) = 100 e e s)ss
(12)
So ())())) induces a finite measure on Th with total mass of 3-1, which we call the discounted occupation measure.

---

Concretely, we In probability theory, the definition in (22 is referred to as the 5-potential of X, which gives discounted visitation frequencies of the state process. We record the following result, which will be useful in the derivation of the performance-differeiin formula. It is consequence of the occupation time formula 41 43
Lemma 1. Under the conditions in we have Jo e-Pa (X,) ds = Jan (1) aor any measurable function yp Rh + R2 for which the expectation exists.
3.2 Performance-Differeeee Formula
We are now ready to answer (Q2) by deriving the performance-diffrencc formula between two policies in terms of the discounted occupation time in (12) and the g-values (TI..
Theorem 2. Given two feedback policies क = { x)) x E [Rn and {a(( x E.).
we have:
((+) - --(=)= = J (7((0) U/ a(a x) (q(2, + yp(x, a, +)) da dx.
(13)
Proof sketch. The full proof is detailed in Appendix B.T The essence of the proof to use perturbation theory and properties of the discounted occupation time. Define operator L (²(R")) + C(RR)) associated with the diffusion process as:
(C ()) == -3((x) + b(x, T) Vi((>) + j0( T) v2p(x).
(14)
Then the Hamilton-Jacobi equation that characterizes the state value function can be expressed as:
- --"V(x;)) F(x, ))),,
(15)
Note that for any to E (RR),, we have J2- (- to express the performance difference in model-dynamics related terms:
fgm p(v)u(dy). This allows us
7 (x)) = / (1) [(C - Cy) Fly. #) + xpy,, #) Fly. x) -py.. x)] dy. (16)
What remains is to reduce the above to the desired result in 13
As discussed in Section our main task is to construct (algorithmically) sequence of policies Ta along which the performance improves. Here we illustrate how some well known approaches of policy improvement (from æ to f) are instances of the performance difference formula (3)).
q-learning and soft q-learning. Since di 2 0, we only need ensure that for all E F".
La ((a æ)) (1(x, a; mT + vp(x, a, +)) da 2 0. This boils down to the problem that for any x, find (+2z) maximize
JA v(a) (4(2, a; + yp(x, a, u)) da.
(17)
There are two special cases:
(i) If p(x, v) = 0, then = 6(a") where a" arg max. g(z, a, T). counterpart of Q-Learning 66 in the discrete time, which we call q-learning.
This is essentially the
(ii) If p(æ, a, v) = lo)(u(a)), this is known as the entropy regularizer need solve
17
59
/ v(a) (q(a, a; T) log v(a))
max PEP((A)
(18)
which has closed form solution with v" (a) exp("(e,...) i.e. v" is the Boltzmann policy for functions. This is a "soft" (à la (17) version of the q-learning mentioned above.
Policy gradient. We may use function approximations to π by a parametric family T² with 0 E θ C R24 For simplicity, we write d% (resp. r7()) for dz (resp. n7(T")). Setting म = T and n in 13 and taking derivative with respect to 0 on both sides, we get the following result.
5

---

Theorem 3 (Policy Gradient). The policy gradient at T is:
Von(O) 10=00= 3 (z,,)
(Vo log (= æ)) (g(z, a: v + a, a, + vooop(z, x
(19)
where the expectation is w.r.t (z, a) ~ , ) meaning (-) and then a a x (. x).
The above formula is indeed the continuous analogue to the well-known PG formula (without
regularization) in the setting, where Von(O) 10=6, = (. VV log (= x)) A(x, a; o " (153), with denoting the advantage function. Specifically, as a comparison, the formula in (19) replaces the visitation frequency by the occupation time, and the advantage function by the q-function. while keeping the same score function Ve log (TT (a x)).
3.3 Continuous TRPO/PPO
Leveraging the performance-ditferen formula derived above, now move on to spell out the continuous counterpart of TRPO and PPO originally developed in 45 for the discrete RL.
Local approximation function:
Given a feedback policy म, we define the local approximation
function to (f) by
L (+) ((T) + / (2) [ #(a xx) (a(2, + p(x, a, +)) da dr.
(20)
Comparing (20) the formula that
we see that the difference to replace d (8) with (s). Observe
i (x) ( (ii) Vo (n(*²)) 10=0, Vo (T²"(( l0=0,-
(TT)
ie. the local approximation function and the true performance objective share the same value and the same gradient with respect to the policy parameters. Thus, the local approximation function regarded as the first order approximation to the performance metric. Furthermore, similar to 23] 45), can apply simulation methods evaluate the local approximation function only using the data generated from the current policy m:
be
(a x) (a x)
E L (**+)= = (TT) (B. (m)))
(9(2, (;; rp(1, a,
(21)
Next, we provide analysis and bounds on the gap n ft) (**),, which can then be used to ensure policy improvement (similar to approaches 45 64 for discounted/averagee reward MDP). First, we need some technical conditions the model dynamics.
Assumption 1. Assume the following conditions for the state dynamics hold true:
(i) Global boundedness: There exists 0 < do S such that of I < 82 (z. I oā for all x, o;
(ii) Uniformly Lipschitz: There exists C. > 0 such that (āō σ) (x', x x)]lF I C. I| x' d'12 for all and x, x'; (ii) Monotonicity (for drift) or growth condition:
There exists Cj > 0 such that (x ve-xx' I²)T (b(2, (((z,)))((2' (() S Cj ||= x' 22122 for all a and x, x'.
The following lemma provides Wasserstein-2 bound between the discounted occupation measures (-)-) and (-)()) for two policies π and ,'.
Lemma Let πI It' be two feedback policies, and suppose the conditions in Assumption hold. Define Cg,5 := 20₁ + 1 2C and C' = supa... |b(x,a))² stute.) Assume further that Cb and < cc. Then there is the bound
nā
(Recall is the dimension of the 20
We (3d 3d S
C
max sup [ - (-11=)1 sup ||a(+1)) x x-1x))1
3Cr CC. (,,))
(22)
(The proof deferred Appendix B.3]
B.3 To derive performance difference bound, define the Sobolev
semi-norm K = |lfll (Ja-
ff))) and its dual norm ||
l| 1
|||||| i

---

sup (pp /i) Wasserstein-2 distance W2(,, v) for any probability measure p and v. Combining this fact with Lemma yields the following result.
I191l 11).
B2
40
show the equivalence of this dual norm ||I u|| to the
Theorem 5. Suppose the conditions in Lemma 4 hold, and further assume that df di (x)
upper bounded by M for all x R". Define K Il.fl with f(x; n,, 8) = 14 ((a
VMK r) (g(2, a; x) + +((x, a, it) da and C(u, nI, == 25 (B C2,a) (sup,,a [lb(z, a)ll 20 As-
no
suming C(u, TT, R) oo, we have (f+tt 2 (#),, where
(f (#) C,,, ft) max sup DKI (+())))))) sup v (<)(( (z))
(23)
Proof is given in Appendix B By Theorem we can use the -ttirattonnmmiimizaatinn (MM)
algorithm in 19 23 26). where L L=(#) ) is taken as the surrogate function for ((T)). Specifically,
given the policy Ty. we can indeed solve the optimization problem max; L = (ft,, and designate its
solution as T1+11 Then, we have
(T(T))) 2 L (TE+1) 2 L (TT) (T()
(24)
i.c., a guaranteed performance improvement. See also 126 Chapter 7] and 127 for the (global) convergence analysis of the MM algorithm (which exceeds the scope of this work). However, in general this optimization problem is not easy to solve directly since C(₁,, म, f) is unknown because of the unknown underlying dynamics, and we may also have work with sample based estimates of the approximation functions. In the spirit of 45 47 we provide algorithms in the next section that can be practically implemented by incorporating an adaptive penalty constant Openaity as an alternative to C(u, ग,, ft. Consequently, the resulting algorithms may no longer strictly preserve the increasing performance ' at each iteration, but overall increasing trend will clear (as demonstrated in Figure 3
4
Algorithms and Experiments
4.1
Sample-based Algorithms
Based on the analysis and results developed above, we provide sample-based estimates the objective functions that lead to practical algorithms. Here we highlight several rate or; the trajectory truncation parameter (time horizon) T (needs to be sufficiently large); the total sample size N or the sampling interval 6G, with N -g T. Also denote t; i 1-8e. 0, ....... N 1, for the time points that we observe data from the environment.
the learning
Algorithm 1 CPG: Policy Gradient with exp(3) random rollout
Input: Policy parameters o,. critic net parameters @o. batch/sample size J
1: for k k=00 0, 2, ... until 0g converges do
2: Collect a truncated trajectory (Xt,, az Tt, Pt. }. }.i 1,...,N N from the environment using
TBC
3: for i = 0,.... N
do: Update the critic parameters as in 8
4: for j = 1, JJ do: Draw i.i.d. T, from (xp(3), round t, to the largest multiple of 6, no
larger than it, and compute the GAE estimator of ((,,,,,0))) ar,)
q(x,, ar,) == (r-, 5, te
- -8,
(X, (Xry+6.) - V((,,)) 18c
(25)
5:
Get an estimator of V,n(0,) as
Vo log (ar, Xx,)) (ī(X,,, , ar,) + (r,, + + VVp( Xr,, arys n xx))
3
(26)
6: Let (n(0k)= = Ej=1 V (nn(E)) and perform PG update: ax+1 02 (v((())
7

---

Continuous Policy Gradient (CPG). To estimate the policy gradient (99) from data, we first
sample an independent exponential variable 7 exp(3) get (X7, a=) (dh (-1x)) If
there is q-function oracle, then we can obtain an unbiased estimate of the policy gradient (of
which the convergence analysis follows (63). Lack of such an oracle, we employ the generalized
advantage estimation (GAE) technique [46] to get ,(X,, 42) ~ (Qar(Xe::)) Oz; (z; (Xx: m)) /8, s
(ride + e 88, (((+)))) - V(X))) /6,. This yields the policy gradient Algorithm
Continuous PPO (CPPO). We now present Algorithm 2
continuous version of the PPO, also as an approximation to the MM algorithm in Section 3.3 To do so, we need more hyper- parameters: the tolerance level e, and the Kl.-divergence radius Moreover, we DKL. (0 2n v DKL. (Te, (- (z) [+)-)))) (Empirically we find that taking average, instead of supremum, over I does not affect the algorithm performance while reducing computational burden, similar to what's observed in the discrete-time TRPO in [55].
Algorithm 2 CPPO: PPO with adaptive penalty constant
Input: Policy parameters 0g critic parameters do
for == 0, 2... until 0% converge do
2: Follow the same as Steps 2-6 in Algorithm
3: Compute policy update (by taking a fixed steps of gradient descent)
0 = arg max {1.4 (0) penalty DKL (8110))
(27)
4: if DKL (0x+1 11xb) 2 (1 + c)6 then
ck+1 "pnnaiyy = 20
penalty
5: else if DKL (0&+1 Ok)) < (1 +e) then Ch
Clk.
Ppenally/
pecsllty
Algorithm is essentially continuous analogue of the TRPO/PPO methods. Note that in the penalty term we use the mean square-root of the KL-divergence, since we choose the radius d < 1; hence, the square-root distance will dominate in the bound in (23). Moreover, interestingly, throughout our primary experiments, using the square-root KL-divergence outperforms (using the KL-divergencc itself). Refer to Appendix Cor for more details.
4.2 Experiments
LQ stochastic control. Consider an environment driven by an SDE with linear state dynamics and quadratic rewards, with b(I, a) = Az + Ba, oI, a) Ca + Da, where A, C, E R. p(z, a. T) =- (log(((((z)) O, > 0, R, P, E R. Linear-quadratic (LQ) control problems play an important role in the control literature, not only because it has elegant and simple solutions but also because more complex, nonlinear problems can be approximated by LQ problems. In general, we do not know the model parameters (c.g., A, B...), and the idea is to continuous RL methods to find the optimal policy.
and r(z, = ( 12 + Rra + 7 a² + Px + Qa). where M >
20.0
CPG
CPC
CPPO
17.5
25.0
0.8
12.5
ddbaancee the
0.6
10.0
5.0
255
0.2
200
600
800
1000
200
400
620
800
1000
iteration
Figure 1: Convergence of 9 in la distance
Figure Convergence of mo in Ki-divegence

---

Here we adopt a Gaussian exploration parameterized by 0 as: To(: x) = -N 02, exp(ts)), and we also parameterize the value function by & as V²(x) = 1022 + Φ1 + Φp- (In fact, as shown in (88 Theorem 4], the optimal exploration and value functions are of this form, and constants such as and @ can be computed explicitly given the model dynamics.) We randomly choose set of initial constants, and compute the optimal 0* and o² with respect to these parameters; refer to Appendix [D.] for more details. Figure show the convergence of algorithms for one certain realized trajectory.
In Figure D we compute the l2 distance between the current policy parameters and the optimal ones, Le. ||0g 12, which tracks the convergence of the policy parameters. In Figure 2 we plot the sample estimated KL divergence between the current policy тT (specified by 0%) and " (specified by 8"), i.e. E ==-a DKL((TT (+|z² (+(z))) The reason to consider the KL-divergence between Th
and nT" is that minimizing the KL-divergence to the optimal solution is equivalent to minimizing the distance between the current policy objective and the optimal objective (see Appendix [DII). The experiments illustrate that our proposed algorithms do converge to the (local) optimum.
We also compare the performance of CPO and CPPO the approaches that directly discretize the time, and then apply the classical discrete-time PG and PPO algorithms. See the details in Appendix D.4 The experiments show that our proposed CPO and CPPO are comparable in terms of sample elficiency, and in many cases they outperform the discrete-time algorithms under a range of time discretization.
2-dimensional optimal pair trading. We also consider the 2-dimensional optimal pair trading problem formulated in [36]. The state space is X (S, W) Rº with ( (so: Wp), where S represents the spread between two stocks, and W denotes the corresponding wealth process. The trader intends to maximize the total discounted reward, with the reward function r( log(1 + W). The state dynamics are:
1 n² pon + +ryddt + w,W, (28) ds,- k-0 St + rdd,, dW, = a. W.((B S.) +
We set p(x, a, TT) = O, and add a constraint on the action: a E [-e, 2]. (The action a is the position taken on the first stock, which can be long/positive or short/negative.) Since the action space is bounded and continuous, we consider a beta distribution for policy paramoterization: To X)) :=
(a+,, oxe (x with f(x ar 3) 3-layer neural network (NN) parameterized by 0 for function approximation; and use another 3-layer
[(a+8) x xa=1 (1 1-- For oe and A8s we use F(a)r(3)
NN for value function approximation. (More details are provided in Appendix D.2)
Figure 3 shows that both algorithms, CPG and CPPO, converge to local optimum (different between the two), and with an overall increasing trend over iterations. (Averaging is taken over 100 Monte Carlo estimates for each policy evaluation.)
1999
mm
1323
1350
10 mmm
4.72
Myna M
4.30
4.25
0.20
1150
Figure 3: Performance of both algorithms to the task

---

Conclusion and Further Works
We have developed this paper the basic theoretical framework for policy optimization in continuous RI, and illustrated its potential applications using numerical experiments.
For further research, topics are high of the continuous policy gradient and TRPO/PPO, vis-a-vis the error due to the time increment o,- Our conjecture is that is likely be polynomial-bounde under mild assumptions, similar the analysis in (20 [6]), ), thus extending beyond the condition required by [63 and [60 31 Second, for
agenda. First, plan to study the convergence (rate)
the bounds on the statistical distance and the performance difference, we want to further develop
consistent bound like the one in [TI (for the discrete setting), i.c., one that remains meaningful when
the discount factor 3 O.
Acknowledgments
Wenpin Tang gratefully acknowledges financial support through NSF grants DMS-21 111772 and DMS-2206038, and through a start-up grant at Columbia University. The works of Hanyang Zhao and David Yao are part of a Columbia-CityU/Ne Initiative, The Government of the HKSAR and the AIFT Lab. We also thank Yanwei Jia and Xun Yu Zhou for helpful discussions.
collaborative project that is supported by InnotHK
References
[1] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22-31. PMLR, 2017.
[2] Andrea Agazzi and Jianfeng Lu. Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regime. arXiv:2010. 1858, 2020.
[3] Leemon C Baird. Reinforcement learning in continuous time: Advantage updating. In Proceed- ings of 1994 IEEE International Conference on Neural Networks (ICNN'94) volume 4, pages 2448 2453. IEEE, 1994.
[4] Leemon C Baird III. Advantage updating. Technical report, WRIGHT LAB WRIGHT. PATTERSON AFB OH, 1993.
[5] Matteo Basei, Xin Guo, Anran Hu, and Yufei Zhang. Logarithmic regret for episodic continuous- time linear-quadratic reinforcement learning over a finite-time horizon. Journal of Machine Learning Research, 23(178):1-34 2022.
[6] Steven Bradtke. Reinforcement learning applied to linear quadratic regulation. Advances in neural information processing systems, 5, 1992.
[7] Gerard Brunick and Steven Shreve. Mimicking an process by a solution of a stochastic differential equation. Ann. Appl. Probab., 23(4), 2013.
[8] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.1271 2023.
[9] Hans Buehler, Lukas Gonon, Josef Teichmann, and Ben Wood. Deep hedging. Quantitative Finance, 19(8): 2711-191, 2019.
[10] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[11] Jim G Dai and Mark Gluzman. Refined policy improvement bounds for mdps. arXiv preprint A0X 08068, 2021.
[12] Jim G Dai and Mark Gluzman. Queueing network controls via deep reinforcement learning. Stochastic Systems, 12(1):30-67, 2022.
10

---

[13] Kenji Doya. Reinforcement learning in continuous time and space. Neural computation. 12(1):219-244 245, 2000.
[14] Jianzhun Du, Joseph Futoma, and Finale Doshi-Velez, Model-based reinforcement learning for semi-markov decision processes with neural odes. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33,
pages 19805-19816. Curran Associates, Inc., 2020.
[15] Wendell H Fleming and Halil Mete Soner. Controlled Markov processes and viscosity solutions, volume 25. Springer Science & Business Media, 2006.
[16] Xin Guo, Anran Hu, and Yufei Zhang. Reinforcement learning for linear-convex models with jumps via stability analysis of feedback controls. arXiv preprini arXiv:2 2021.
[17] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pages 1352- 1361. PMLR, 2017.
[18] Yilie Huang, Yanwei Jia, and Xunyu Zhou. Achieving mean-varianee efficiency by continuous- time reinforcement learning. In Proceedings of the Third ACM International Conference on Al in Finance, pages 377-385, 2022.
[19] David R Hunter and Kenneth Lange. A tutorial on MM algorithms. The American Statistician, 58(1):30-33) 2004.
[20] Yanwei Jia and Xun Yu Zhou. Policy evaluation and temporal-different learning in continuous time and space: martingale approach. Journal of Machine Learning Research, 23(154):1-55, 2022.
[21] Yanwei Jia and Xun Yu Zhou. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms. Journal of Machine Learning Research, 23(154): 2022.
[22] Yanwei Jia and Xun Yu Zhou. q-learning in continuous time. arXiv preprint arXiv:2207.00713, 2022.
[23] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pages 267-
274, 2002.
[24] Ioannis Karatzas and Steven E. Shreve. Brownian motion and stochastic calculus, volume 113 of Graduate Texts in Mathematics. Springer- Verlag, New York, second edition, 1991.
[25] Jeongho Kim, Jaeuk Shin, and Insoon Yang. Hamilton-jacobi deep q-learning for deterministic continuous-time systems with lipschitz continuous controls. The Journal of Machine Learning Research, (2(1):9363-99) 2021.
[26] Kenneth Lange. MM optimization algorithms. SIAM, Philadelphia, PA, 2016.
[27] Kenneth Lange, Joong-Ho Won, Alfonso Landeros, and Hua Zhou. Nonconvex optimization via MM algorithms: Convergence theory. arXiv:2106.02805 2021.
[28] James-Michael Leahy, Bekzhan Kerimkulov, David Siska, and Lukasz Szpruch. Convergence of policy gradient for entropy regularized MDPs with neural network approximation in the mean-field regime. In International Conference on Machine Learning, pages 12222-12252, 2022.
[29] Jaeyoung Lee and Richard S. Sutton. Policy iterations for reinforcement learning problems in continuous time and space fundamental theory and methods. Automatica, 126: 109421, 2021.
[30] Yuan Ling, Sadid A Hasan, Vivek Datla, Ashequl Qadir, Kathy Lee, Joey Liu, and Oladimeji Farri. Diagnostic inferencing via improving clinical concept extraction with deep reinforcement learning: preliminary study. In Machine Learning for Healthcare Conference, pages 271-285. PMLR, 2017.

---

[31] Boyi Liu, Qi Cai. Zhuoran Yang, and Zhaoran Wang. Neural proximal/ruust region policy optimization attains globally optimal policy, 2023.
[32] Grégoire Loeper. Uniqueness of the solution to the Vlasov-Poisses system with bounded density. Journal de mathématiques pures et appliquées, 86(1):68-77) 79, 2006.
[33] Dhruv Madeka, Kari Torkkola, Carson Eisenach, Dean Foster, and Anna Luo. Deep inventory management. arXiv preprint wrXiv:2200.03137, 2022.
[34] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference on machine learning, pages 1928-1937. PMI., 2016.
[35] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 131.5562, 2013.
[36] Supakor Mudchanatongsuk, James A. Primbs, and Wilfred Wong. Optimal pairs trading: A stochastic control approach. In 2008 American Control Conference, pages 1035-1039, 2008
[37] Rémi Munos. Policy gradient in continuous time. Journal of Machine Learning Research, 7:771-791, 2006.
[38] Rémi Munos and Paul Bourgine. Reinforcement learning for continuous stochastic control problems. M. Jordan, M. Kearns, and S. Solla, editors, Advances Neural Information Processing Systems, volume 10. MIT Press, 1997.
[39] Bernt Øksendal. Stochastic differential equations. Springer, 2003.
[40] Rémi Peyre. Comparison between W2 distance and H-1 norm, and localization of Wasserstein distance. ESAIM: Control, Optimisation and Calculus of Variations, 24(4):1489-1501, 2018.
[41] Jim Pitman and Mare Yor. Hitting, occupation and inverse local times of one-dimensional diffusions: martingale and excursion approaches. Bernoulli, 91)::122, 2003.
[42] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[43] Daniel Revuz and Mare Yor. Continuous martingales and Brownian motion, volume 293.
Springer Science & Business Media, 2013.
[44] Ahmad EL Sallab, Mohammed Abdou, Etienne Perot, and Senthil Yogamani. Deep rein- forcement learning framework for autonomous driving. arXiv preprint arXiv: (tt.::/704.22533 2017.
[45] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[46] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02433 2015.
[47] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 77070064777777 2017.
[48] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
[49] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. nature, 550(7666))33)4-33999 2017.
12

---

[50] Daniel W. Stroock and S. R. Srinivasa Varadhan. Multidimensional diffussin processes, volume 233 of Grundlehren der Mathematischen Wissenschaften. Springer- Verlag, Berlin-New York, 1979.
[51] Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3:9-44 1988.
[52] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[53] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[54] Lukasz Szpruch, Tanut Treetanthiploet, and Yufei Zhang. Exploration-exploitttii trade-off for continuous-time episodic reinforcement learning with linear-convex models. arXiv:21 10264, 2021.
[55] Corentin Tallec, Léonard Blier, and Yann Ollivier. Making deep q-learning methods robust to time discretization. In International Conference Machine Learning, pages 0196-6104. PMLR, 2019.
[56] Wenpin Tang, Yuming Paul Zhang, and Xun Yu Zhou. Exploratory HJB equations and their convergence. SIAM Journal on Control and Optimization, 60(6):3191-3216 2022.
[57] Kyriakos G Vamvoudakis. Q-learning for continuous time linear systems: A model-free infinite horizon optimal control approach. Systems & Control Letters,
2017.
(58) Haoran Wang, Thaleia Zariphopoulou, and Xun Yu Zhou. Reinforcement learning in continuous time and space: stochastic control approach. J. Mach. Learn. Res., 21((988):-34,, 2020.
[59] Haoran Wang and Xun Yu Zhou. Continuous time mcan-variance portfolio selection: A reinforcement learning framework. Mathematical Finance, 30(4):1273-1308 2020.
[60] Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence, 2019.
[61] Christopher JCH Watkins and Peter Dayan. Q-learning Machine learning, 8:279-292, 1992.
[62] Jiongmin Yong and Xun Yu Zhou. Stochastic controls: Hamiltonian systems and HJB equations, volume 43. Springer Science & Business Media, 1999.
[63] Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Basar, Global convergence policy gradient methods to (almost) locally optimal policies. SIAM Journal on Control and Optimization, 35663335863612, 2020.
[64] Yiming Zhang and Keith W Ross. On-policy deep reinforcement learning for the average-reward criterion. In International Conference Machine Learning, pages 12535-12545. PMLR, 2021.
13

---

Appendix A
Continuous RL: Formulation and Well-Posedness
A.1
Exploratory Stochastic-Control
For n, m positive integers, let b RR x A - IR" and a I x A + Raxm be given functions, where A is compact action space. classical stochastic control problem [5 62 is to control the state (or feature) dynamics governed by an Itō process, defined on a filtered probability space [DFF, (FP) .20) along with an {F.) J-Browinn motion = (Bx, 2 03:
dx; = bb (2) ds + a (X", (a) dB,, 8 821, X, =
(29)
where 0, is the agent's action (control) time The goal of the stochastic control (discounted objective over an infinite time horizon) is for any time-state pair (t,, x) in (29). to find the optimal {F+)) $.20 progressively measurable sequence of actions a = {a,, 8 2 t (called the optimal policy) that maximizes the expected total B-discounted reward:
[. e B(5-() (Xg, ds) X; =
(30)
where r: 1R x A + R is the running reward of the current state and action (X ag), and / >>> 0 is discount factor that measures the time-depreciation of the objective value (or the impatience level of the agent). Note that the state process X {X 2 f depends on the starting (initial) time-state pair (f, x). For ease of notation, we denote by X" instead Xt..raa = (x... 2 t the solution
to the SDE in 29) when there is ambiguity.
Listed below are the standard assumptions to ensure the well-posesness of the stochastic control
problem in (29 h (3
Assumption 2. The following conditions are assumed throughout:
(i) b, o,r are all continuous functions in their respective arguments;
(ii) b, a are uniformly Lipschitz continuous in n, iee, there exists a constant C 0 such that for vp ob
1| a) - s0 (x', ,))l2 S I| 2'11,, for all E A, N, E R";
(31)
(ii) b, have linear growth in x and a, i.e., there exists a constant > 0 Osuch thatfor & E (b,
[1<p(r, a)lla S C(1 + ||=|| + [|all)), for all (x, a) E IR" x A;
(32)
(iv) r has polynomial growth in and a, ie., there exists constant C 0 and 2 1 such that
[r(x,)) a)) s C1 + ||=|1² + [[al))) for all (x, a) R A.
(33)
The key idea underlying exploratory stochastic control is to use randomized policy (or relaxed control), i.., apply a probability distribution to the admissible action space. To do so. let's assume the probability space is rich enough to support uniform random variable Z that is independent of the Brownian motion B {B.}. We then expand the original filtered probability space to
(FF where F. = FB v (Z) (i.e., augment JB with the sigma field generated by Z).
Let aT: R [2) E P(A) be stationary feedback policy given the state at x, where P(AA) is a suitable collection of probability distributions (with density functions). At each time s, an action ag generated from the distribution
Xg),, i.e. the policy only depends on the current state. In other words, we only consider stationary, or time-independent feedback control policies for the stochastic control problem (29 (30).
Given a stationary policy т E P(A), an initial state x, and an proj-prroressiielly measurable action process a {a_ > 0} generated from n, the state process X= {X 2 0} follows:
dx = b (X a.)) ds + a (X),a))dB. 8 2 4, XJ = x,
(34)
defined on FF It is easy see that the dynamics in 34 define time-homogeneous Markov process, such that for each t 2 and a::
(X, X x) d (X5++ XX = =x), 0.
14

---

Consequently, the objective in (3) is independent of time t, and is equal to:
E
(X. a.)) ds X
(35)
Furthermore, following (58], we can add regularizer to the objective function to encourage explo- ration (represented by the randomized policy), leading
4)) ==E J," 3(x-t) Ir (X, 0.) yp (X, 07 ,
.)] x
(36)
where p: _ A P(A) + R is the regularizer, and 7 2 is weight parameter on exploration (also known as the "temperature" parameter). For instance, in [58]. is taken as the differential entropy,
a (-)) log ((),,
and hence, the "entropy" regularizer. The same argument as before justifies that (t, is independent of time t. That is, for 2
((t, (1) = V(z; m) = IE? 1 e (X a a.) + (x ag, (=1])) X
(37)
which is the state-value function under the policy æ. V(x; R), in ( and which, in turn, leads the performance function ()T) in (6) Moreover, recall the main task of the continuous RI. is to find (or approximate) 1y" max, n((),, where max over admissible policies.
A.2 Controlled SDE and the HJ Equation
Note that the exploratory state dynamics in (34) is governed by a general Itô process. It is sometimes the sense that its (weak) solution has the same distribution as the Itô process in 34 at each fixed time t. It is known (158)) that when n m 1, the marginal distribution of [ 2 0} agrees with that of the solution to the SDE, denoted by {X,, 2 0}:
more convenient consider an equivalent SDE representation-
ax, = b 6xxx,, (.1x.) ds + (Xs, (.1x.) dB,,
X. = x.
where b(0, (T)) = - bir, (a)da and (ar, T(-)) VIa J2 (a)da. This result is easily
extended to arbitrary m, thanks to 7 Corollary 3.7J, with the precise statement presented below (assuming m for of exposition).
Theorem 6. Assume that for policy and for every
[ (x R"x*
is positive definite. Then there exists a filtered probability space ②2, F
.P) that supports
120"
a continuous 1V"vuue adapted process X and an n-dimensional Brownian motion B satisfying
dx, = =6xx,, (.1x.) ds + (X.; (-1 (.1 aB.,
Xo x.
(38)
where
b(zz x(-)(()))))))))) / ()) o(x,
For each 2 0, the distribution of X. under agrees with that of X3 under P defined in (34)
As consequence, the state value function in 37 is identical to
((: = E [/ e- / |r(xx. + xp (8. (x,))] x(a X.)da ds Xo = x
15

---

Also define
F(a, m) / r(z, a)r(als)da, p(a,z) / p(a, (t)
can simplify the value function to
V(z; m) =
Ba ((),; + vD ( ds Xo - =
(39)
Following the principle of optimality, V then satisfies the HJ equation:
(V m) b(ar,)) (V;;:) 1 82 (2, () V²V(x; f(x, T)) (8(x, 0.
(40)
To guarantee that the HJ equation in 40) characterizes the state-value function in 39 need
Assumption 3. Assume the following conditions hold:
(i) b, a. r. are all continuous functions in their respective arguments.
(it) b. r. p uniformly Lipschitz continuous in i.e,, there exists constant C 0 such that for E rb
11o(x, a) - (z',,a)) I CC -2'l2, for all a E A, x, E ER"
and
1p(x,,, a, plx', a.a, T) S C | x'1l2, for all a E A, T TEP(A), x, E ER.
(iii) ā is globally bounded, i.e., there exist 0 < < oo such that
o I 0² (x, a) s o 1. for all E A, TER.
(iv) the SDE (38) has weak solution which is unique in distribution. (v) (())x) is measurable (z, a) and uniformly Lipschitz continuous in a, i.e., there exists
constant C >> O such that
JA (<(al)) (alpx)) da & Cl|2 2'112, for all x. E TR.
Theorem 7. Under Assumption 3 the state-value function in is the unique (subquadratic) viscosity solution to the HJ equation 40).
39
Proof. By (56) Section 3.1], the HJ equation in (00) has unique (subquadratic) viscosity solution under the conditions (i)-(i Further by 21] Lemma 2], the viscosity solution is the state-valuc function.
Appendix B Proofs of Main Results (in $3)
B.1
Proof of Theorem 2
Recall in the proof sketch of the Theorem in 3 we have defined the operator L CRR(R)) + C(R")
(C) (x) -3p(x) + b(z, m) Vv(x) + 5o, m)² (2p(x),
which leads to the following characterization the HJ equation:
- F(x, m) + )7(a,))
(41)
We need the following two lemmas concerning the operator L Lemma 8, For any E C² (RR", we have
R"
16

---

Proof. The left hand side of the above equation is
Og
1
= e
BP(())) 8(x.
4)
(xx)
oxxx p8 17,77)
(X.))
ds
dx
2
dx
00
[x((x)) - 8(8
02
(xx)
2
1
ox
84
(xx)
do
6(x,,
(x:
dd.
Or
Ox
(-(8))(((7))
lim (-e (- + (X())
=
8-000
=
((r),
where the first equality follows from the definition of the occupation time and the third equality from Itö's formula.
Lemma 9. Let nT t be two feedback policies. We have
(C - L2) + f(x; ft) F(z, x) T)(2,T)= - #(a a)
(42)
Proof. By definition ax
a; in we have
RHS Sai #(a x H1th I, Ox
oV
(x;
OV
dz (α; 3V (z: da
Jaca #(a (2) (2z Or
OV
1 0² (z)
PV
(x; +
(2; + r(z, - 3V (x; da
2
Oz
= f(x, + (*V²(()
f(z, - f(2, T) - 1)(,,)) -VV))
LHS.
Proof of Theorem 2 Note that in 113 the equation to be proven, the right hand side can be written as Ja d (v)f(x; (2 n, dyy with
( = / t(a (x) (a(a,,;;) + mp(a, a, +)) da.
From Lemma
we have
f(z; = (L LV) F(z,, #) + + F(x, - ~v((z, x).
(43)
On the other hand, for the left hand side of [13 we have
π(T) = S ((::)))(d)) = / [W(-<=))
(44)
with the second equality following from Lemma 8 and
(7 / (1) [F #) vp(U, dy
(45)
following the definition of the discounted expected occupation time; moreover, from 41 we have
0 f [l T) Fly, mT) VMk, T) dy-
(46)
Hence, combining the last three equations 44 46), we have
7(*) (())) - dv(i)) [[C* (T) + Fly. #) + vp(y, #) F(y. T)) 7p(U. dy. (47)
Thus, we have shown LHS=RHH in 13
17

---

B.2
Proof of Theorem 3
Proof. It suffices to show the integral version of the theorem:
Ve (π((r) l==== / of (x) U Ver"(a (a x) (q(a,a;"") mp(a, a,
(48)
7 x 2) Vopla, n )aa dx.
As before, we simplify notation by denoting ny(*)) as (7)) and d' as d Then, by Theorem we have
17 + 60) n0()= = / (x)) V x 00000 (a x) (g(2, 0) + rp(x, 0 + 50)) da dæ. (49)
Denote
[(60) = - 9 (a (x) (a(x, 1p(x, 0 60)) da.
Note that f(0)- O, which follows from
S(0) = / π (a 5) (g(0,, 0) + yp(x, a, 0)) da
av T (a x²) (z, dx
@² (x; - 3V (x; + >p(2, a, 0) da
(z; R)
Oz
3V (1; T) + b(x,))) VV(x;)
2 0² (I, (VV( x))
Thus,
17(0 + 80) (0)) (d +8950 f(00)
660 500 [(00) - (d ,{())
(d f(60) f(0))
(d du. f(00) - f(0)) + (dh, f(0) - f(0)).
Dividing both sides by 50 completes the proof, as the first term on the last line above of higher order than 80.
B.3 Proofs of Lemma 4and Theorem 5
We need a lemma for the perturbation bounds.
Lemma 10. Assume that both x² (x. f()) and (2 x-)) are positive definite and
0² x())) a()) oă
where dp > 0, then we have that the difference between the square root matrix is bounded by
[, 0(x π 122 S 200
(2, #) (2(2, 12
If we also assume that the upper bounds, i.e.
(x), (1,,),
a()) < 0/. I.
by some 5g > oo 0. then have
50
[1ō #) - ö(z T <
A .T ||
20
Proof. Consider a normalized vector & with || 1 an eigenvector of A B with eigenvalue then
x. JT(AA A B)r = _² _T Bb B.)AAr+ TT Bà (A's Bà) )x
= μxx "A= + B+}
18

---

thus, if A, 11.2000 I. I, this implies
1x (A B)æl
S 1|A Bll2- Amia + BB) I|A (202/
# x² < + B2
Furthermore, note that
0² - (2, ((t, xaaa)))da.
8o
(z, 222 08 [((az) [ oह. [| (a/=)]]1
Proof (of Lemma 4 Consider the Wasserstein-2 distance W2( 1) between distribution j and as
( inf ((x,y)~//1 v11 1/2
W2(u,
(TET(u,))
where r',,, v)) is the set all probability measures on the product space IR" x R" with the marginal distributions being μ and and || |l2 is the standard Euclidean distance. Denote
1" = 3d
We want to get an upper bound on (2(d jo
( in terms of the distance between two policies n and Consider a specific coupling (X,, Y,) below:
[dX, = 6X,,, क (- X.)) ds + + (X,, (: (x)) dB.,
(50)
dY, = (Y,, # (. Y,)) ds + ā (Y,, A (: Y,)) dB,.
with Xo = Yo. which leads to joint distribution over IR" x IR:
3 : plo.v) = L
00
31 f(x,)) y)dt B
Hence,
00
W d E (( y12
9 IX, Yolldds.
(51)
then boils down to estimating E X,, - 12 By Itô's formula,
d|xx, Y.1. 2(X, - -Y.) (6 (X(,,, x) - -YY,, + (X,,,) - (Y,,
+ +Tr Tr (6 (X,,T) m) ő (Ys, #)) ds.
Taking expectation both sides yields
E|X, Y.13 2E (X.- - Y.) (x., - #))ds
+Tr [B (X,, - ō (Y., +))
ds
(A)
(B)
(52)
with
(A) = R (X, TT)T(b())))) (b(((bb(,,)) m) - (Y,,, r))ds
+ +B -
( [E((X,-Y))T()
(6 (Y.. #))ds
1
Ci E Y.112 + E| - Y.11 + (Y., bY.,, 7) 1
2
$ <G + ] X, Y. + [6(. b(. #)
and
(B) E (X,, - (Y,, #) 1
< 2B| (X,, T) - ā (Y,, T) 1| + 2 - -Y,, 11?
< 20 E||,, -. 2sup |ā (अ, à (x. 1
= 2Cg E|X, -Y11 + 2 (TT)) 5( (. () 112.00
19

---

Combining the above, we get
B| B - - (20g + 1
- - b(. 2) + 21 (..)) - 5 (,, R) 10
ds
Ch.
((x,8)
By Grönwall's inequality, we have
C( f)
E Xr
(eCoss -
(53)
b,a
Substituting back into 51. we obtain
((T,
000
W W di 2
3 - (E - 1) ds.
Ci jo
Thus, B Cj.a have
C( #)
W2 (x) 5
Cg (8 Ci,
Concerning the term C(T, ft), have
[i6(s - - b(. 12 sup ||b (z, b (z,, #) ||2 S sup [+((1=)) - π (-|xx)] sup a),,a)}b
2,0
and
= sup - (2, || S Vn 50
sup | x-11>)1
20
Thus we have:
C(T, ft = |1((, 7) - b(., ((1,00) + +2llā (,, - (+, #) 11
d. 3.
sup b(z,bbzz a)12 + 2,a
sup || (z)) ( (-|xx)] 1, sup ||*(||z) T(112)1|
200
which proves our upper bound.
Proof (of Theorem
5. We have that
|I An
( (tt
1r - (*)( 1(t dh. 5>
|
B
(54)
K KVM
M
|d Il < 8
W2 (d% d.))
B
where K == sup, || iy1
00 (more about K in the remarks below). Combining estimate in (22) (of Lemma yields the desired result in (3).
54 with the
Remarks (on K). In the performance-difffeennce bound developed above, we assume K is finite:
K = 1ffll ( [V(())
< <0,
where f(x; n, R) := SA (a x)) (g(2, α) + p(x, a, #)) da. The famous Poincaré inequality can provide lower bound this quantity; but need upper bound well, i.e.,
Y K (L_ (V())a))))) sc {
This above essentially reverse Poincaré Inequality, which not likely hold (in particular, the existence of the constant C).
20

---

Should we indeed have a reverse Poincaré Inequality, then we can further bound by
|f(x)) / (#(a x) - .)) (a(a,a;7) p(x, a, R)) dal
$ / |a(a 2) <a |z) ( + p(z, a, R) da
A
< ssu a + +ll,, a, R) Div(T(: x) 4( a)),
and
(L (((o)))e)) Rm
< (J Asup (x + p(z,a, )² (Tv( xx) x))dd)
(fe suu ( + p(z, ) dz)
sup (TL( [)), #(: x)),
R=
where the second inequality is from Pinsker's inequality. This way, we would have recovered similar bound as in the discrete RL. Since we do not have the reverse Poincaré inequality, however, we have to assume that K is finite.
Appendix C Algorithms
C.1 Performance of CPPO with Square-root KL and Linear KL
Here present detailed version of the CPPO algorithm. For two probability distributions P and Q over the action space with density functions and correspondingly. recall that the KL-divergence between these two is defined as:
/ log( )a(a)da, p(a)
((a)
DKL (RI(()
Denote (0110) = E andff DKL(TO, (To, (+(=)][ to distinguish it from DkI.(01(0x) =
E endat v DKL (Tex
which was used in CPPO Algorithm in 2
Note that bounding the performance difference by the linear KL-divergence DKL (0, 0x), instead of its square-root counterpart DKL(#) (0)10)), will generally require stronger conditions (which may be difficult to satisfy). For completeness, we present the following algorithm as an important benchmark, the CPPO with linear Kl--divegeecc:
Algorithm 3 CPPO: PPO with adaptive penalty constant (linear KL-divergence)
Input: Policy parameters Oo. critic net parameters Φn
for k 0, 2. until 0g converge do 2: Collect truncated trajectory (Xt,, ot,, F,, Pt; ;;;=1,11 1,..., N from the environment using
3: TT.
for i = ,,..,,N: do: Update the critic parameters as in (
4: for 1, , ... J do: Draw i.i.d. Ty from exp( round Tj to the largest multiple of 5, no
larger than it, and compute the GAE estimator of ((>,,, (r,)
ar,) = (rr, 38, V (X+,+&)) V(X,,)) /8c.
5: Compute policy update (by taking fixed steps of gradient descent)
Bx+1 = arg max 1 (+(()) Cpemnn DKL (8 (6+)).
6: if DKL (0++1 (0x,) 2 (1 + e)6, then crk+1 20
penalty penalty
7: else if DILL (0x+1110) < 51 + c), then C Ctk
Openallly /
comparison between the above and Algorithm (using square-root KI. divergence) is presented in D.3 below, which clearly illustrates the advantage of square-root KL divergence.
21

---

C.2 KL-divergence
We elaborate here on the KL-divergence between the current policy and the optimal policy, along with the entropy regularizer. By the performance difference formula, we have
/ (x)))) RRm
UA x(a æ)) (g(2, a; x²) - log(T(a))) da dz.
Notice that by the definition of KL-divergence we defined before, we have
DKL(T (++))=))))))] = / log( (alar)
T T (a/r)da.
Similar as the previous discussion soft q-learning, .t" is optimal implies that
x" (a. x)) oc exp( (z,
and the normalization constant is 1 can be proved through considering the exploratory HJB equation, sec 22 56].
Thus
(H / log(T(a/=))<((d))dd fx a(x, x(aaz)da,
which leads
(( .π(=((/>))))))))>">)) = -y Ex-dy ( ( (-1x).
This justifies our claim that the KL-divergence is essentially equivalent to the distance to the optimal performance.
Appendix D Experiments
D.1 Example 1
Recall, in the LQ control problem, the reward function is
N 70² + Px P Qa)
r(x, a) = 2 x² + Roa +
with M 0, 0, R Q. and R² < MN, and adopt the entropy regularizor
p(z, a, n) - -log(T(a)).
DD'R"22RRBBCCH 0).
Furthermore, suppose that the discount rate satisfies 3 > 2A C2
The following results are readily derived from Theorem of 58 policy x² is
The value function of the optimal
1
V(x):
2 k22² k, ko, I E2
where
- 100-224-CC9 D
h2 2
(B+C
1 V ((p (2A+ 2B CD)R (BB
(R? -MM-
2
(B (D=)))((p-(2A-C))) D
P(N-kDD - QR
k = BB CD) + (A p) (N -2 BR
and
(bg Q)
7
N D2 -1)
2тey
to =
+
2p (N D2)
2p
22

---

respectively. Moreover, the optimal feedback control is Gaussian, with density function
((a;a)=<(( (B2 CD) N k2
R) k B
Q
7
N - k2 D²
For set of model parameters: -=-.-==----------=--------
1 0.1, following the formulas and the parameterized policy To(. =((= 02. exp(0s)).
and the corresponding value function (²( = 0222 + 01 + Qo. we can derive the optimal parameters:
Φ- = 10.11914874, - 0.10555128, 0..3518876)
and
0" [- 0.39444872, -0.78889748 -1.40400944]
Table 1: Hyper-parameter values for Example
Alphabet
Description
Value
T
Trajectory Truncation Length discount factor time interval batch size for sampling exp(3)
25
0.005
100
01
learning rate for policy iteration k:
0.02 when S 50 and 0.02 x log( 50 when > 50
02
learning rate for value iteration iteration threshold
0.01 when < 50 and 0.01 x log( when 50
K
2000
8
steps of gradient descent radius
10
8
0.0002
c
tolerance level
0.5
D.2
Example 2
The model parameters are 0.01, -. o 1.000==-,,,0 5. For both the value function and the policy parameterization, we use a 3-laayer neural network, and with the initial parameters sampled form the uniform distribution over [-0.5,0.5]. We use the tanh activation function for the hidden layer.
Table 2: Hyperparameter values for Example 2
Alphabet
Description
Value
Trajectory Truncation Length
25
discount factor time interval batch size for sampling exp(
0.005
100
01
learning rate for policy iteration k
0.005 when k: <. 50 and 0.005 x log 50
when k > 50
Q2
learning rate for value iteration &o
0.01 when & < 50 and 0.01 x log(
50 k
when > 50
K
iteration threshold steps of gradient descent radius tolerance level
200
10
8
0.025
0.5
D.3 Performance of CPPO with Square-root KL and Linear KL
We compare the performance of CPPO with square-root KL-divergence (denote as CPPO), and linear KL-divergence (denoted as CPPO (nst) non square-root) applied to the experiments in Example and Example 2. Figure compares the distance between the current policy parameters and the optimal parameters, with æ-axis denoting the iteration times and y-axis denoting the L2 distance. Figure compares the current expected return, with z-axis denoting the iteration times and y-axis denoting the current performance by taking the average of 100 times of Monte Carlo evaluation. In
23

---

CPPO
10
CPPO_mt _rP__nst
distance optimal
0.8
0.6
0.4
0.2
200
400
1000
Iteration
Figure 4: Performance of CPPO and CPPO (nst) the Example
1.35
CPPO
CPPO_nst ___st
1.30
1.25
1.20
1.15
1.10
1.05
20
40
60
80
100
120
Figure 5: Performance of CPPO and CPPO (nst) the Example 2
both figures, the blue curve represents the algorithm with square-root KL-divergence as opposed to the orange one corresponding to the linear version. Both figures clearly demonstrate the advantage of the former. particular, linear version can suffer from getting stuck at the local optimum as demonstrated in Example] I
D.4 Performance of CPG and CPPO compared to the classical discrete-time algorithms
We conduct experiments to compare the CPG and CPPO their discrete counterparts. Specifically, we discretize the MDP in Example 1, and implement the classical PG and PPO algorithms. Our results show that time discretization with step size 8t 0.1 and 51. 0.05, the performance of CPG and CPPO is (at least) comparable to their discrete counterparts; in particular, for 51 0.1, CPG outperforms PG. We have repeated the experiments for 25 random seeds, and plotted both the average performance line and the error bar. These experimental results indicate that the continuous approach has the potential to outperform their discrete counterparts, which is worth further exploring in the future.
24

---

CPG-4;
CPG-KL
1.0
61-0.05
05-0.05
0.6
12.5
10.0
0.4
7.5
5.0
300
200
302
400
300
Figure CPG in 12 distance (6. 0.05)
Figure 7: CPG in KL distance (6: = 0.05)
DPG-1z
6t-0
20.0
05
17.5
0.8
0.
00
100
200
300
420
500
300
400
Figure 9: DPG KI. distance (6, = 0.05)
Figure 8: DPG in la distance (6. 0.05)
CPG-f:
CPG-KL
0.8
15.0
12.5
0.6
10.0
7.5
0.0
100
200
J03
230
502
420
30L
Figure 10: CPG 12 distance (6, = 0.1)
Figure 11: CPG in KL distance (6, 0.1)
25

---

DPG-4
DPG-12
0t-0.1
15.0
0.
12.5
10.0
7.5
5.0
0
100
200
300
400
500
200
Figure 12: DPG in 12 distance (6g = 0.1)
Figure 13: DPG in KL distance (6; = 0.1)
CPPO-4,
CPPO-KL
6t=0 05
0t0005
15.0
12.5
0.6
10.0
0.4
75
0.
0.0
0.0
100
250
Figure 15: CPPO in KL distance (6, = 0.05)
Figure 14: CPPO in la distance (8, 0.05)
DPPO-4;
20.0
12
6-0
37.5
10
06
20.0
0.
75
5.0
02
25
0.
150
200
250
100
150
200
250
Figure 16: DPPO in lg distance (6, 0.05)
Figure 17: DPPO in KL distance (6, 0.05)
26

---

CPPO-12
CPPO-KL
20.0
6t-0.1
0t-0.1
1.0
17.5
0.8
12.5
10.0
0.4
5.0
0.2
0.0
0.0
0
50
100
150
200
250
150
200
250
Figure 18: CPPO in 12 distance (6, 0.1)
Figure 19: CPPO in KL distance (6, 0.1)
DPPO-4;
DPPO-KL
0r-01
37.5
25.0
12.5
06
30.0
0.4
02
0.0
50
100
150
200
250
0
30
100
150
200
250
Figure 20: DPPO in la distance (de = 0.1)
Figure 21: DPPO in KL distance (6. 0.1)
27


Reviews:
[{"review_id": "xGliZPKzRh", "reviewer": "Reviewer_Mbd6", "summary": "This paper seeks to answer three research questions: 1) Is there a continuous time analog of the state occupancy measure, 2) Is there a convenient expression for the performance difference between two policies in the continuous time setting, and 3) Can PPO be adapted to fit the continuous time setting? In answering these questions, the paper presents a continuous time occupancy measure, a policy performance difference similar to that of conservative policy iteration and related algorithms, and continuous versions of REINFORCE and PPO are provided. Theoretical results for several properties are given, and experiments demonstrating that the algorithms were able to solve a couple of problems are provided. \n", "questions": "- How does the sampling at discretization impact the gradient estimate? \n\n- How does the step size need to be adapted to work in the continuous-time setting? Is it sensitive to the scaling of the time horizon or sampling rate? \n\n", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides a thorough theoretical treatment of defining policy gradients for the continuous time RL setting. The paper clearly defines and adequately answers its stated objectives. ", "weaknesses": "The biggest area for improvement in this paper is in the empirical results. While the main results of this paper are, theoretical new practical algorithms are presented. Thus, they deserve proper evaluation and experimentation to educate the reader on the challenges of using them. \n\nFor example, there are no experiments illustrating that there were any special difficulties in applying these algorithms to the continuous setting. There should be experiments illustrating how the hyperparameters, particularly those specific to the continuous time setting, impact the optimization process. Currently, the results only tell us that the algorithms were made to work. Great, but we do not learn anything beyond this trivial result. It is important to develop the reader’s understanding of how they can make the algorithms work or at least what can cause failure. \n\n\nAdditionally, the adaptive penalty term should also be explored since it deviates from the standard PPO implementation. Does it effectively constrain the distribution throughout learning? How does the penalty change over time? These are important questions to answer when introducing a new method. \n\n\nMinor quibble: the opening sentence of the introduction is boring. Many papers have used this form, and my eyes glaze over as soon as I read them. Try keeping this motivation brief and tie it immediately to the scope of the work discussed in this paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the suggestions and comments.\n\n1. How does the sampling at discretization impact the gradient estimate?\n\nA: We have implemented our proposed algorithms with different (time) step sizes: $\\delta t=0.02$, $\\delta t=0.05$ and $\\delta t=0.1$, and found that the performances are quite similar, suggesting that our CPG algorithm is insensitive to step size; please refer to the link https://www.dropbox.com/scl/fo/mkuqf1nux1ysiwhys9oqn/h?rlkey=8s4ui8i2wknd4za1o11alf1u5&dl=0 for details. \n\n2. Comparison of our proposed algorithm in continuous setting with the discrete MDP algorithms.\n\nA:  We have conducted extra experiments to compare the CPG and CPPO to their discrete counterparts. (The results will be included in the revision). Specifically, we discretize the MDP in Example 1, and implement the classical PG and PPO algorithms. Our results show that in time discretization with step size $\\delta t=0.1$ and $\\delta t= 0.05$, the performance of CPG and CPPO is (at least) comparable to their discrete counterparts; in particular, for $\\delta t=0.1$, CPG outperforms PG. We have repeated the experiments for 25 random seeds, and plotted both the average performance line and the error bar. Please refer to the link https://www.dropbox.com/scl/fo/03g1ub7mvis64yucclqzm/h?rlkey=boyq188kpop55hj5ahyiyom16&dl=0. These experimental results indicate that the continuous approach has the potential to outperform their discrete counterparts. While this is quite preliminary, we do plan to do more thorough investigations along this line in the near future.\n\n3. How does the step size need to be adapted to work in the continuous-time setting?\n\nA: As mentioned above, the performance of our algorithms appears to be quite robust to step size. In the future, we plan to also investigate the sensitivity of the hyperparameters to the environment dynamics (e.g., the time horizon)."}, {"review_id": "GUex6L0BXc", "reviewer": "Reviewer_9Un8", "summary": "This paper investigates the continuous reinforcement learning problem. The proposed method is based on the notion of occupation time for policy gradient, which is analogous to the visitation frequency in discrete Markov decision processes. Empirical evaluations are conducted on two example scenarios for two versions of the algorithm (CPG and CPPO).", "questions": "In Figures 1 and 2, what does it mean to be the distance to the optimal to be some non-zero value? How does this translate to the reward performance? Reward performance is critical to know whether or not the achieved difference in distance is useful.\n\nThe performance is compared with the proposed two methods, CPG and CPPO (Figure 3), and no baseline is considered. An essential choice would be to use the discrete counterpart (Policy Gradient - PG and PPO). A simple form can be to make the MDP discrete and run PG and PPO. This comparison is even important to understand what kind of problem the continuous RL is useful for.\n\nIs the achieved return by CPPO optimal (empirical best) in Figure 3? \n\nImplementation details need to be included; what are the PPO-specific hyperparameters (e.g., clipping), for Example 2?\n\nFor reproducibility, a common practice is using several seed runs to account for variation in the results. How many random seeds are used for the experiments in Figures 1 and 2? These missing details make the presented results hard to reproduce.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Analyze an Important formulation of continuous time and space in reinforcement learning.\n- Theoretical analysis is present to develop a policy gradient counterpart of TRPO/PPO.\n- The empirical evaluation is presented along with theoretical analysis.", "weaknesses": "- Lack of comparison with discrete counterpart.\n- The lack of implementation details (missing hyperparameter, random seed) makes reproducibility challenging.\n- The evaluation of the proposed method is limited to only two hand-crafted examples, which hinders a comprehensive understanding of its implications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments, especially on the experiments.\n\n1. Lack of comparison to the discrete counterpart.\n\nA: We have conducted extra experiments to compare the CPG and CPPO to their discrete counterparts. (The results will be included in the revision). Specifically, we discretize the MDP in Example 1, and implement the classical PG and PPO algorithms. Our results show that in time discretization with step size $\\delta t=0.1$ and $\\delta t= 0.05$, the performance of CPG and CPPO is (at least) comparable to their discrete counterparts; in particular, for $\\delta t=0.1$, CPG outperforms PG. We have repeated the experiments for 25 random seeds, and plotted both the average performance line and the error bar. Please refer to the link https://www.dropbox.com/scl/fo/03g1ub7mvis64yucclqzm/h?rlkey=boyq188kpop55hj5ahyiyom16&dl=0. These experimental results indicate that the continuous approach has the potential to outperform their discrete counterparts. While this is quite preliminary, we do plan to do more thorough investigations along this line in the near future.\n\nWe would like to emphasize that our main objective (and contribution) is to provide a continuous approach (which appears under-developed), with rigorous analyses and provable results, both of which appear to be difficult to do in the discrete setting. On the numerical side, preliminary experimentation (as summarized above) has shown that the continuous approach is at least comparable to (or outperform) its discrete counterpart.\n\n2. The lack of implementation details. \n\nA: On p.8, l.279, we have referred the implementation details to Appendix D. In addition, we plan to post our codes on Github, so the reproducibility will not be an issue. As to the random seeds, we use 25 random seeds to run the algorithms. More details will be added to the revision.\n\n3. The evaluation of the proposed method is limited to...\n\nA: As mentioned above (in point 1), our focus here is to develop a continuous approach for policy optimization, supported by rigorous analyses and provable results. While numerical studies have shown promising performance of the continuous approach, as highlighted above in point 1, we agree that these are quite preliminary and limited in scope, and more thorough experimentation is needed, which we plan to do in the near future. \n\nIndeed we believe that our proposed algorithms can be used in many scenarios (financial trading, aviation control, etc) involving large-scale and high-frequency data. It will certainly warrant another independent study to implement our proposed algorithms and evaluate their performance in one of such applications. \n\n4. Connection of policy distance and reward performance difference\n\nA: As we mentioned on p.9, l.282-28, minimizing the KL-divergence (between the iterated policy and the optimal policy)\nis equivalent to minimizing the distance between the current policy objective and the optimal objective; and we referred to Appendix D.1 for further details. In addition, preliminary experiments indicate that our proposed algorithms do converge to the (local) optimum.\n\n5. The performance is compared with the proposed two methods...\n\nA: We have now conducted extra experiments and made more comparisons as summarized in point 1.\n\n6. Implementation details need to be included; what are the PPO-specific hyperparameters (e.g., clipping), for Example 2?\n\nA: We did not use the clipping technique. Details regarding the hyperparameter are spelled out in Appendix D (as mentioned  on p.9,  l. 297)."}, {"review_id": "FjR318Bh8J", "reviewer": "Reviewer_vuEi", "summary": "Reinforcement learning (RL) is a powerful tool for solving sequential decision making problems but has primarily been formulated for discrete-time Markov decision processes. However, many real-world systems are more naturally expressed in continuous time, and the proper choice of discretization time step may be challenging. Additionally, a controller operating in continuous time may be more suitable for high frequency applications. Prior work has formulated a continuous-time analog to the advantage function, called the q-value. Building on the concept of a q-value, the paper sets out to derive a continuous-time analog to the discounted occupation probability and performance difference lemma, two core concepts in RL. The authors then show how to compute a gradient of this performance difference lemma to give a continuous analog to the policy gradient. Next, the authors form a local approximation of the performance difference which takes an expectation over the current policy rather than the updated one. They provide bounds on the gap between the true performance metric and this local approximation. Then they use this local approximation to formulate a continuous analog to TRPO\\PPO. The paper evaluates the continuous-time policy gradient and PPO algorithms on synthetic linear-quadratic stochastic control problems. They also consider a two-dimensional optimal pair trading problem and show their CPPO algorithm performs best.", "questions": "- How do these continuous-time methods compare to their discrete-time counterparts when the time step discretization is uniform? How does the performance gap change with choice of discretization time step? What are the main advantages of this formulation?\n- How does this approach scale to harder, more realistic problems?\n- How exactly does the continuous-time policy gradient algorithm in this paper compare to previously proposed approaches?", "limitations": "There is no discussion of limitations of the continuous-time formulation or their specific algorithms. The paper would be stronger if it discussed these. This could include assumptions made in the proofs or the fact that the work assumes the dynamics follow an Ito SDE driven by Brownian motion.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Continuous-time RL is an under-explored area that has application in controlling systems at high frequency and using an adaptive discretization scheme with non-uniform time steps.\n- This appears to be the first work to propose a continuous-time analog in a stochastic setting to the performance difference lemma and derive a continuous PPO algorithm.  \n- The paper is well organized and clearly written. It does a good job explaining the results and provides enough information to support its claims.", "weaknesses": "- There is no discussion or experimental results which illustrate how the continuous-time formulation is advantageous over discrete-time RL. The proposed algorithms still ultimately require us to discretize. As such, there should be some comparison to existing discrete-time RL methods. The hope would be that formulating the problem in continuous time allows us to achieve better performance under certain choices of step size. Another potential benefit of continuous-time RL is the case of unevenly-spaced observations. Experiments which highlight these benefits would significantly strengthen the paper.\n- The problems considered are toy examples which do not tell us much about the scalability of the proposed methods. More complex scenarios would make the paper much more convincing. Especially ones in which the continuous-time formulation provides a clear advantage.\n- The discussion of related work could be better. It is still a little unclear to me how this work is positioned in the continuous-time RL literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments, especially on the experiments.\n\n1. There is no discussion or experimental results which illustrate how the continuous-time formulation is advantageous...\n\nA: We have conducted extra experiments to compare the CPG and CPPO to their discrete counterparts. (The results will be included in the revision). Specifically, we discretize the MDP in Example 1, and implement the classical PG and PPO algorithms. Our results show that in time discretization with step size $\\delta t=0.1$ and $\\delta t= 0.05$, the performance of CPG and CPPO is (at least) comparable to their discrete counterparts; in particular, for $\\delta t=0.1$, CPG outperforms PG. We have repeated the experiments for 25 random seeds, and plotted both the average performance line and the error bar. Please refer to the link https://www.dropbox.com/scl/fo/03g1ub7mvis64yucclqzm/h?rlkey=boyq188kpop55hj5ahyiyom16&dl=0. These experimental results indicate that the continuous approach has the potential to outperform their discrete counterparts. While this is quite preliminary, we do plan to do more thorough investigations along this line in the near future.\n\nWe would like to emphasize that our main objective (and contribution) is to provide a continuous approach (which appears under-developed), with rigorous analyses and provable results, both of which appear to be difficult to do in the discrete setting. On the numerical side, preliminary experimentation (as summarized above) has shown that the continuous approach is at least comparable to (or outperform) its discrete counterpart.\n\n\n2. More complex scenarios would make the paper much more convincing...\n\nA: As mentioned above (in point 1), our focus here is to develop a continuous approach for policy optimization, supported by rigorous analyses and provable results. While numerical studies have shown promising performance of the continuous approach, as highlighted above in point 1, we agree that these are preliminary and limited in scope, and more thorough experimentation is needed.\n\nWe believe that our proposed algorithms can be used in many scenarios (financial trading, aviation control, etc) involving large-scale and high-frequency data. It will certainly warrant another independent study (which we do plan to pursue as a next step) to implement our proposed algorithms and evaluate their performance in one of such applications. \n\n\n3. The discussion of related work could be better.\n\nA: To the best of our knowledge, this is the first paper that studies policy optimization in the continuous and stochastic setting. While this is a notably under-developed area in RL, we do have cited (on p.1-2) several related works and discussed how do they motivate or relate to our work. \n\n4. How does the performance gap change with the choice of discretization time step...\n\nA: We have implemented our proposed algorithms with different (time) step sizes: $\\delta t=0.02$, $\\delta t=0.05$ and $\\delta t=0.1$, and found that the performances are quite similar, suggesting that our CPG algorithm is insensitive to step size; please refer to the link https://www.dropbox.com/scl/fo/mkuqf1nux1ysiwhys9oqn/h?rlkey=8s4ui8i2wknd4za1o11alf1u5&dl=0 for details. \n\n5. Limitation of the Ito-SDE formulation...\n\nA: Similar to the classical MDP setting, we need to assume a Markovian and stationary setting in our continuous approach, which is represented by an It\\^o-SDE. Other more technical conditions are standard (and very mild), such as requiring the It\\^o-SDE to be well-posed, the continuity and growth condition on model parameters. In the revision, we will state these conditions more explicitly. "}]

Author rebuttal:

Meta-review: The reviewers all praised this work for the quality of its presentation and the strength of its contribution, with many commenting on the novelty and importance of this analysis in the continuous setting. All of the reviewers initially critiqued the empirical results, calling for further evaluation and comparison with discrete counterparts. The authors' rebuttal resolved these concerns, and consequently, the merits of this paper far outweigh the remaining weaknesses, and as such I recommend acceptance.
Decision: Accept (poster)