SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The above formula is indeed the continuous analogue to the well-known PG formula (without regularization) in the MDP setting, where  $\nabla_{\theta}\eta (\theta)|_{\theta = \theta_0} = \frac{1}{\beta}\mathbb{E}_{(x,a)}\left[\nabla_{\theta}\log \left(\pi^{\theta}(a\mid x)\right)A(x,a;\pi^{\theta_0})\right]$  ([53]), with  $A$  denoting the advantage function. Specifically, as a comparison, the formula in (19) replaces the visitation frequency by the occupation time, and the advantage function by the  $q$ -function, while keeping the same score function  $\nabla_{\theta}\log \left(\pi^{\theta}(a\mid x)\right)$ .
3.3 Continuous TRPO/PPO
Leveraging the performance-difference formula derived above, we can now move on to spell out the continuous counterpart of TRPO and PPO originally developed in [45, 47] for the discrete RL.
Local approximation function: Given a feedback policy  $\hat{\pi}$ , we define the local approximation function to  $\eta(\hat{\pi})$  by
$$
L ^ {\pi} (\hat {\pi}) = \eta (\pi) + \int_ {\mathbb {R} ^ {n}} d _ {\mu} ^ {\pi} (x) \left[ \int_ {\mathcal {A}} \hat {\pi} (a \mid x) \left(q (x, a; \pi) + p (x, a, \hat {\pi})\right) \mathrm {d} a \right] \mathrm {d} x. \tag {20}
$$
Comparing (20) to the formula (13), we see that the difference is to replace  $d_{\mu}^{\hat{\pi}}(s)$  with  $d_{\mu}^{\pi}(s)$ . Observe that
$$
\mathrm {(i)} L ^ {\pi} (\pi) = \eta (\pi), \quad \mathrm {(i i)} \nabla_ {\theta} \left(\eta (\pi^ {\theta})\right) | _ {\theta = \theta_ {0}} = \nabla_ {\theta} \left(L ^ {\pi^ {\theta_ {0}}} (\pi^ {\theta})\right) | _ {\theta = \theta_ {0}},
$$
i.e. the local approximation function and the true performance objective share the same value and the same gradient with respect to the policy parameters. Thus, the local approximation function can be regarded as the first order approximation to the performance metric. Furthermore, similar to [23, 45], we can apply simulation methods to evaluate the local approximation function only using the data generated from the current policy  $\pi$ :
$$
L ^ {\pi} (\hat {\pi}) = \eta (\pi) + \frac {1}{\beta (x , a) \sim (\beta d _ {\mu} ^ {\pi} , \pi (\cdot | x))} \left[ \frac {\hat {\pi} (a \mid x)}{\pi (a \mid x)} \left(q (x, a; \pi) + \gamma p (x, a, \hat {\pi})\right) \right]. \tag {21}
$$
Next, we provide analysis and bounds on the gap  $\eta (\hat{\pi}) - L^{\pi}(\hat{\pi})$ , which can then be used to ensure policy improvement (similar to approaches in [45, 64] for discounted/average reward MDP). First, we need some technical conditions on the model dynamics.
Assumption 1. Assume the following conditions for the state dynamics hold true:
(i) Global boundedness: There exists  $0 < \sigma_0 \leq \bar{\sigma}_0$  such that  $\sigma_0^2 \cdot I \leq \tilde{\sigma}^2(x, a) \leq \bar{\sigma}_0^2 \cdot I$  for all  $x, a$ ; (ii) Uniformly Lipschitz: There exists  $C_{\tilde{\sigma}} > 0$  such that  $\| \tilde{\sigma}(x, \pi) - \tilde{\sigma}(x', \pi) \|_F \leq C_{\tilde{\sigma}} \| x - x' \|_2$  for all  $\pi$  and  $x, x'$ ;
(iii) Monotonicity (for drift) or growth condition:
There exists  $C_{\tilde{b}} > 0$  such that  $(x - x')^\top (\tilde{b}(x, \pi) - \tilde{b}(x', \pi)) \leq C_{\tilde{b}} \| x - x' \|_2^2$  for all  $\pi$  and  $x, x'$ .
The following lemma provides a Wasserstein-2 bound between the discounted occupation measures  $d_{\mu}^{\pi}(\cdot)$  and  $d_{\mu}^{\tilde{\pi}}(\cdot)$  for two policies  $\pi$  and  $\pi'$ .
Lemma 4. Let  $\pi, \pi'$  be two feedback policies, and suppose the conditions in Assumption 1 hold. Define  $C_{\tilde{b},\tilde{\sigma}} := 2C_{\tilde{b}} + 1 + 2C_{\tilde{\sigma}}^2$  and  $C = \sup_{x,a} |b(x,a)|^2 + \frac{n\tilde{\sigma}_0^2}{2\sigma_0^2}$ . (Recall  $n$  is the dimension of the state.) Assume further that  $\beta > C_{\tilde{b},\tilde{\sigma}}$  and  $C < \infty$ . Then there is the bound
$$
W _ {2} \left(\beta d _ {\mu} ^ {\hat {\pi}}, \beta d _ {\mu} ^ {\pi}\right) \leq \frac {C}{\beta C _ {\tilde {b} , \tilde {\sigma}} (\beta - C _ {\tilde {b} , \tilde {\sigma}})} \cdot \max  \left(\sup  _ {x} \| \hat {\pi} (\cdot | x) - \pi (\cdot | x) \| _ {1}, \sup  _ {x} \| \hat {\pi} (\cdot | x) - \pi (\cdot | x) \| _ {1} ^ {\frac {1}{2}}\right). \tag {22}
$$
(The proof is deferred to Appendix B.3.) To derive a performance difference bound, define the Sobolev semi-norm as  $K \coloneqq \| f\|_{\dot{H}^1} \coloneqq \left(\int_{\mathbb{R}^n}|\nabla f(x)|^2\mathrm{d}x\right)^{\frac{1}{2}}$ , and its dual norm  $\| \cdot \|_{\dot{H}^{-1}}$  as  $\| \mu \|_{\dot{H}^{-1}} =$
$\sup \left\{\left|\langle g,\mu \rangle \right| \mid \| g\|_{\dot{H}^1}\leq 1\right\}$ . [32, 40] show the equivalence of this dual norm  $\| \mu -v\|_{\dot{H}^{-1}}$  to the Wasserstein-2 distance  $W_{2}(\mu ,v)$  for any probability measure  $\mu$  and  $v$ . Combining this fact with Lemma 4 yields the following result.
Theorem 5. Suppose the conditions in Lemma 4 hold, and further assume that  $d_{\mu}^{\hat{\pi}}(x), d_{\mu}^{\pi}(x)$  are upper bounded by  $M$  for all  $x \in \mathbb{R}^n$ . Define  $K \coloneqq \| f \|_{\dot{H}^1}$  with  $f(x; \pi, \hat{\pi}) \coloneqq \int_A \hat{\pi}(a | x) (q(x, a; \pi) + p(x, a, \hat{\pi})) \, \mathrm{d}a$  and  $C(\mu, \pi, \hat{\pi}) \coloneqq \frac{\sqrt{M}K}{2\beta^2C_{\bar{b},\bar{\sigma}}(\beta - C_{\bar{b},\bar{\sigma}})} \left( \sup_{x,a} \| b(x,a) \|^2 + \frac{n\bar{\sigma}_0^2}{2\sigma_0^2} \right)$ . Assuming  $C(\mu, \pi, \hat{\pi}) < \infty$ , we have  $\eta(\hat{\pi}) \geq \underline{L}^\pi(\hat{\pi})$ , where
$$
\underline {{L}} ^ {\pi} (\hat {\pi}) := L ^ {\pi} (\hat {\pi}) - C (\mu , \pi , \hat {\pi}) \cdot \max  \left(\sup  _ {x} D _ {\mathrm {K L}} (\hat {\pi} (\cdot | x) \| \pi (\cdot | x)), \sup  _ {x} \sqrt {D _ {\mathrm {K L}} (\hat {\pi} (\cdot | x) \| \pi (\cdot | x))}\right). \tag {23}
$$
Proof is given in Appendix B. By Theorem 5, we can use the minorization-maximization (MM) algorithm in [19, 23, 26], where  $\underline{L}^{\pi}(\hat{\pi})$  is taken as the surrogate function for  $\eta (\pi)$ . Specifically, given the policy  $\pi_{k}$ , if we can indeed solve the optimization problem  $\max_{\hat{\pi}}\underline{L}^{\pi_k}(\hat{\pi})$ , and designate its solution as  $\pi_{k + 1}$ . Then, we have
$$
\eta \left(\pi_ {k + 1}\right) \geq \underline {{L}} ^ {\pi_ {k}} \left(\pi_ {k + 1}\right) \geq \underline {{L}} ^ {\pi_ {k}} \left(\pi_ {k}\right) = \eta \left(\pi_ {k}\right) \tag {24}
$$
i.e., a guaranteed performance improvement. See also [26, Chapter 7] and [27] for the (global) convergence analysis of the MM algorithm (which exceeds the scope of this work). However, in general this optimization problem is not easy to solve directly since  $C(\mu, \pi, \hat{\pi})$  is unknown because of the unknown underlying dynamics, and we may also have to work with sample based estimates of the approximation functions. In the spirit of [45, 47], we provide algorithms in the next section that can be practically implemented by incorporating an adaptive penalty constant  $C_{\mathrm{penalty}}$  as an alternative to  $C(\mu, \pi, \hat{\pi})$ . Consequently, the resulting algorithms may no longer strictly preserve the increasing performance of  $\eta$  at each iteration, but overall increasing trend will be clear (as demonstrated in Figure 3).
4 Algorithms and Experiments
4.1 Sample-based Algorithms
Based on the analysis and results developed above, we provide sample-based estimates of the objective functions that lead to practical algorithms. Here we highlight several hyper-parameters: the learning rate  $\alpha$ ; the trajectory truncation parameter (time horizon)  $T$  (needs to be sufficiently large); the total sample size  $N$  or the sampling interval  $\delta_t$ , with  $N \cdot \delta_t = T$ . Also denote  $t_i := i \cdot \delta_t$ ,  $i = 0, \dots, N - 1$ , for the time points that we observe data from the environment.
Algorithm 1 CPG: Policy Gradient with  $\exp (\beta)$  random rollout
Input: Policy parameters  $\theta_0$ , critic net parameters  $\phi_0$ , batch/sample size  $J$
1: for  $k = 0,1,2,\dots$  until  $\theta_{k}$  converges do
2: Collect a truncated trajectory  $\{X_{t_i},a_{t_i},r_{t_i},p_{t_i}\}$ $i = 1,\dots ,N$  from the environment using  $\pi_{\theta_k}$ .
3: for  $i = 0, \dots, N - 1$  do: Update the critic parameters as in (8)
4: for  $j = 1, \dots, J$  do: Draw i.i.d.  $\tau_{j}$  from  $\exp(\beta)$ , round  $\tau_{j}$  to the largest multiple of  $\delta_{t}$  no larger than it, and compute the GAE estimator of  $q(X_{\tau_j}, a_{\tau_j})$
$$
\tilde {q} \left(X _ {\tau_ {j}}, a _ {\tau_ {j}}\right) := \left(r _ {\tau_ {j}} \delta_ {t} + e ^ {- \beta \delta_ {t}} V \left(X _ {\tau_ {j} + \delta_ {t}}\right) - V \left(X _ {\tau_ {j}}\right)\right) / \delta_ {t} \tag {25}
$$
5: Get an estimator of  $\nabla_j\eta (\theta_k)$  as
$$
\frac {1}{\beta} \left[ \nabla_ {\theta} \log \left(\pi^ {\theta_ {k}} \left(a _ {\tau_ {j}} \mid X _ {\tau_ {j}}\right)\right) \left(\tilde {q} \left(X _ {\tau_ {j}}, a _ {\tau_ {j}}\right) + \gamma p \left(X _ {\tau_ {j}}, a _ {\tau_ {j}}, \pi^ {\theta_ {k}}\right)\right) + \gamma \nabla_ {\theta} p \left(X _ {\tau_ {j}}, a _ {\tau_ {j}}, \pi^ {\theta_ {k}}\right) \right] \tag {26}
$$
6: Let  $\tilde{\nabla}\eta (\theta_k) = \frac{1}{J}\sum_{j = 1}^{J}\nabla_j\eta (\theta_k)$  and perform PG update:  $\theta_{k + 1} = \theta_{k} + \alpha \tilde{\nabla}\eta (\theta_{k})$
Continuous Policy Gradient (CPG). To estimate the policy gradient (19) from data, we first sample an independent exponential variable  $\tau \sim \exp (\beta)$  to get  $(X_{\tau}^{\pi},a_{\tau}^{\pi})\sim (d_{\mu}^{\theta_0},\pi^{\theta_0}(\cdot |x))$ . If there is a  $q$ -function oracle, then we can obtain an unbiased estimate of the policy gradient (of which the convergence analysis follows [63]). Lack of such an oracle, we employ the generalized advantage estimation (GAE) technique [46] to get  $q(X_{t},a_{t})\approx (Q_{\Delta t}(X_{t},a_{t};\pi) - V(X_{t};\pi)) / \delta_{t}\approx$ $(r_t\delta_t + e^{-\beta \delta_t}V(X_{t + \delta_t}) - V(X_t)) / \delta_t$ . This yields the policy gradient Algorithm 1.
Continuous PPO (CPPO). We now present Algorithm 2, a continuous version of the PPO, also as an approximation to the MM algorithm in Section 3.3. To do so, we need more hyperparameters: the tolerance level  $\epsilon$ , and the KL-divergence radius  $\delta$ . Moreover, we set  $\bar{D}_{\mathrm{KL}}(\theta \| \theta_k) := \mathbb{E}_{x \sim d_\mu^{\theta_k}} \sqrt{D_{\mathrm{KL}}(\pi_{\theta_k}(\cdot|x)) \|\pi_\theta(\cdot|x))}$ . (Empirically we find that taking average, instead of supremum, over  $x$  does not affect the algorithm performance while reducing computational burden, similar to what's observed in the discrete-time TRPO in [45].)
Algorithm 2 CPPO: PPO with adaptive penalty constant
Input: Policy parameters  $\theta_0$ , critic net parameters  $\phi_0$   
1: for  $k = 0,1,2,\dots$  until  $\theta_k$  converge do  
2: Follow the same as Steps 2-6 in Algorithm 1.  
3: Compute policy update (by taking a fixed  $s$  steps of gradient descent)  
 $\theta_{k + 1} = \arg \max_{\theta}\left\{L^{\theta_k}(\theta) - C_{\text{penalty}}^k\bar{D}_{\text{KL}}(\theta ||\theta_k)\right\}$   
4: if  $\bar{D}_{\mathrm{KL}}(\theta_{k + 1}||\theta_k)\geq (1 + \epsilon)\delta$  then  $C_{\text{penalty}}^{k + 1} = 2C_{\text{penalty}}^k$   
5: else if  $\bar{D}_{\mathrm{KL}}(\theta_{k + 1}||\theta_k)\leq \delta /(1 + \epsilon)$  then  $C_{\text{penalty}}^{k + 1} = C_{\text{penalty}}^k /2$
Algorithm 2 is essentially a continuous analogue of the TRPO/PPO methods. Note that in the penalty term we use the mean square-root of the KL-divergence, since we choose the radius  $\delta < 1$ ; hence, the square-root distance will dominate in the bound in (23). Moreover, interestingly, throughout our primary experiments, using the square-root KL-divergence outperforms (using the KL-divergence itself). Refer to Appendix C,D for more details.
LQ stochastic control. Consider an environment driven by an SDE with linear state dynamics and quadratic rewards, with  $b(x,a) = Ax + Ba$ ,  $\sigma (x,a) = Cx + Da$ , where  $A,B,C,D\in \mathbb{R}$ ,  $p(x,a,\pi) = -\log (\pi (a|x))$ , and  $r(x,a) = -\left(\frac{M}{2} x^2 +Rxa + \frac{N}{2} a^2 +Px + Qa\right)$ , where  $M\geq 0,N > 0,R,P,Q\in \mathbb{R}$ . Linear-quadratic (LQ) control problems play an important role in the control literature, not only because it has elegant and simple solutions but also because more complex, nonlinear problems can be approximated by LQ problems. In general, we do not know the model parameters (e.g.,  $A,B,\ldots$ ), and the idea is to use continuous RL methods to find the optimal policy.
Figure 1: Convergence of  $\theta$  in  $l_{2}$  distance
Figure 2: Convergence of  $\pi_{\theta}$  in KL-divergence
Here we adopt a Gaussian exploration parameterized by  $\theta$  as:  $\pi_{\theta}(\cdot \mid x) = \mathcal{N}(\theta_1x + \theta_2,\exp (\theta_3))$ , and we also parameterize the value function by  $\phi$  as  $V_{\phi}(x) = \frac{1}{2}\phi_{2}x^{2} + \phi_{1}x + \phi_{0}$ . (In fact, as shown in [58, Theorem 4], the optimal exploration and value functions are of this form, and constants such as  $\theta$  and  $\phi$  can be computed explicitly given the model dynamics.) We randomly choose a set of initial constants, and compute the optimal  $\theta^{*}$  and  $\phi^{*}$  with respect to these parameters; refer to Appendix D.1 for more details. Figure 1,2 show the convergence of algorithms for one certain realized trajectory.
In Figure 1, we compute the  $l_{2}$  distance between the current policy parameters and the optimal ones, i.e.  $\| \theta_k - \theta^*\| _2$ , which tracks the convergence of the policy parameters. In Figure 2, we plot the sample estimated KL divergence between the current policy  $\pi_{k}$  (specified by  $\theta_{k}$ ) and  $\pi^{*}$  (specified by  $\theta^{*}$ ), i.e.  $\mathbb{E}_{x\sim d_{\mu}^{\theta_k}}D_{\mathrm{KL}}(\pi_{\theta_k}(\cdot |x)\| \pi_\theta (\cdot |x))$ . The reason to consider the KL-divergence between  $\pi_{k}$  and  $\pi^{*}$  is that minimizing the KL-divergence to the optimal solution is equivalent to minimizing the distance between the current policy objective and the optimal objective (see Appendix D.1). The experiments illustrate that our proposed algorithms do converge to the (local) optimum.
We also compare the performance of CPO and CPPO to the approaches that directly discretize the time, and then apply the classical discrete-time PG and PPO algorithms. See the details in Appendix D.4. The experiments show that our proposed CPO and CPPO are comparable in terms of sample efficiency, and in many cases they outperform the discrete-time algorithms under a range of time discretization.
2-dimensional optimal pair trading. We also consider the 2-dimensional optimal pair trading problem formulated in [36]. The state space is  $X = (S, W) \in \mathbb{R}^2$  with  $X(0) = (s_0, w_0)$ , where  $S$  represents the spread between two stocks, and  $W$  denotes the corresponding wealth process. The trader intends to maximize the total discounted reward, with the reward function  $r(X, a) = \log(1 + W)$ . The state dynamics are:
$$
\mathrm {d} S _ {t} = k (\theta - S _ {t}) \mathrm {d} t + \eta \mathrm {d} B _ {t}, \quad \mathrm {d} W _ {t} = a _ {t} W _ {t} \left(k \left(\theta - S _ {t}\right) + \frac {1}{2} \eta^ {2} + \rho \sigma \eta + r _ {f}\right) \mathrm {d} t + \eta W _ {t} \mathrm {d} B _ {t}, \tag {28}
$$
We set  $p(x, a, \pi) \equiv 0$ , and add a constraint on the action:  $a_{t} \in [-\ell, \ell]$ . (The action  $a_{t}$  is the position taken on the first stock, which can be long/positive or short/negative.) Since the action space is bounded and continuous, we consider a beta distribution for policy parameterization:  $\pi_{\theta}(a \mid X) \coloneqq f\left(\frac{a + \ell}{2\ell}, \alpha_{\theta}(X), \beta_{\theta}(X)\right)$  with  $f(x, \alpha, \beta) \coloneqq \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)} x^{\alpha - 1} \left(1 - x^{\beta - 1}\right)$ . For  $\alpha_{\theta}$  and  $\beta_{\theta}$ , we use a 3-layer neural network (NN) parameterized by  $\theta$  for function approximation; and use another 3-layer NN for value function approximation. (More details are provided in Appendix D.2.)
Figure 3 shows that both algorithms, CPG and CPPO, converge to a local optimum (different between the two), and with an overall increasing trend over iterations. (Averaging is taken over 100 Monte Carlo estimates for each policy evaluation.)
Figure 3: Performance of both algorithms to the task
5 Conclusion and Further Works
We have developed in this paper the basic theoretical framework for policy optimization in continuous RL, and illustrated its potential applications using numerical experiments.
For further research, two topics are high on our agenda. First, we plan to study the convergence (rate) of the continuous policy gradient and TRPO/PPO, vis-a-vis the error due to the time increment  $\delta_t$ . Our conjecture is that it is likely to be polynomial-bounded under mild assumptions, similar to the analysis in [20, 16]), thus extending beyond the condition required by [63] and [60, 31]. Second, for the bounds on the statistical distance and the performance difference, we want to further develop a consistent bound like the one in [11] (for the discrete setting), i.e., one that remains meaningful when the discount factor  $\beta \rightarrow 0$ .
Wenpin Tang gratefully acknowledges financial support through NSF grants DMS-2113779 and DMS-2206038, and through a start-up grant at Columbia University. The works of Hanyang Zhao and David Yao are part of a Columbia-CityU/HK collaborative project that is supported by InnotHK Initiative, The Government of the HKSAR and the AIFT Lab. We also thank Yanwei Jia and Xun Yu Zhou for helpful discussions.
[1] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22-31. PMLR, 2017.
[2] Andrea Agazzi and Jianfeng Lu. Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regime. arXiv:2010.11858, 2020.
[3] Leemon C Baird. Reinforcement learning in continuous time: Advantage updating. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94), volume 4, pages 2448-2453. IEEE, 1994.
[4] Leemon C Baird III. Advantage updating. Technical report, WRIGHT LAB WRIGHT-PATTerson AFB OH, 1993.
[5] Matteo Basei, Xin Guo, Anran Hu, and Yufei Zhang. Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizon. Journal of Machine Learning Research, 23(178):1-34, 2022.
[6] Steven Bradtke. Reinforcement learning applied to linear quadratic regulation. Advances in neural information processing systems, 5, 1992.
[7] Gerard Brunick and Steven Shreve. Mimicking an Itô process by a solution of a stochastic differential equation. Ann. Appl. Probab., 23(4), 2013.
[8] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712, 2023.
[9] Hans Buehler, Lukas Gonon, Josef Teichmann, and Ben Wood. Deep hedging. Quantitative Finance, 19(8):1271-1291, 2019.
[10] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[11] Jim G Dai and Mark Gluzman. Refined policy improvement bounds for mdps. arXiv preprint arXiv:2107.08068, 2021.
[12] Jim G Dai and Mark Gluzman. Queueing network controls via deep reinforcement learning. Stochastic Systems, 12(1):30-67, 2022.
[50] Daniel W. Stroock and S. R. Srinivasa Varadhan. Multidimensional diffusion processes, volume 233 of Grundlehren der Mathematischen Wissenschaften. Springer-Verlag, Berlin-New York, 1979.
[51] Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3:9-44, 1988.
[52] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[53] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[54] Lukasz Szpruch, Tanut Treetanthiploet, and Yufei Zhang. Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models. arXiv:2112.10264, 2021.
[55] Coretin Tallec, Léonard Blier, and Yann Ollivier. Making deep q-learning methods robust to time discretization. In International Conference on Machine Learning, pages 6096-6104. PMLR, 2019.
[56] Wenpin Tang, Yuming Paul Zhang, and Xun Yu Zhou. Exploratory HJB equations and their convergence. SIAM Journal on Control and Optimization, 60(6):3191-3216, 2022.
[57] Kyriakos G Vamvoudakis. Q-learning for continuous-time linear systems: A model-free infinite horizon optimal control approach. Systems & Control Letters, 100:14-20, 2017.
[58] Haoran Wang, Thaleia Zariphopoulou, and Xun Yu Zhou. Reinforcement learning in continuous time and space: A stochastic control approach. J. Mach. Learn. Res., 21(198):1-34, 2020.
[59] Haoran Wang and Xun Yu Zhou. Continuous-time mean-variance portfolio selection: A reinforcement learning framework. Mathematical Finance, 30(4):1273-1308, 2020.
[60] Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence, 2019.
[61] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279-292, 1992.
[62] Jiongmin Yong and Xun Yu Zhou. Stochastic controls: Hamiltonian systems and HJB equations, volume 43. Springer Science & Business Media, 1999.
[63] Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Basar. Global convergence of policy gradient methods to (almost) locally optimal policies. SIAM Journal on Control and Optimization, 58(6):3586-3612, 2020.
[64] Yiming Zhang and Keith W Ross. On-policy deep reinforcement learning for the average-reward criterion. In International Conference on Machine Learning, pages 12535-12545. PMLR, 2021.
Appendix A Continuous RL: Formulation and Well-Posedness
A.1 Exploratory Stochastic-Control
For  $n,m$  positive integers, let  $b:\mathbb{R}^n\times \mathcal{A}\mapsto \mathbb{R}^n$  and  $\sigma :\mathbb{R}^n\times \mathcal{A}\mapsto \mathbb{R}^{n\times m}$  be given functions, where  $\mathcal{A}$  is a compact action space. A classical stochastic control problem [15, 62] is to control the state (or feature) dynamics governed by an Ito process, defined on a filtered probability space  $\left(\Omega ,\mathcal{F},\mathbb{P};\left\{\mathcal{F}_s^B\right\}_{s\geq 0}\right)$ , along with an  $\{\mathcal{F}_s^B\}$ -Brownian motion  $B = \{B_s,s\geq 0\}$ :
$$
\mathrm {d} X _ {s} ^ {a} = b \left(X _ {s} ^ {a}, a _ {s}\right) \mathrm {d} s + \sigma \left(X _ {s} ^ {a}, a _ {s}\right) \mathrm {d} B _ {s}, s \geq t, \quad X _ {t} = x, \tag {29}
$$
where  $a_{s}$  is the agent's action (control) at time  $s$ . The goal of the stochastic control (discounted objective over an infinite time horizon) is for any time-state pair  $(t,x)$  in (29), to find the optimal  $\left\{\mathcal{F}_s^B\right\}_{s\geq 0}$ -progressively measurable sequence of actions  $a = \{a_s,s\geq t\}$  (called the optimal policy) that maximizes the expected total  $\beta$ -discounted reward:
$$
\mathbb {E} \left[ \int_ {t} ^ {+ \infty} e ^ {- \beta (s - t)} r \left(X _ {s} ^ {a}, a _ {s}\right) \mathrm {d} s \mid X _ {t} ^ {a} = x \right], \tag {30}
$$
where  $r: \mathbb{R}^n \times \mathcal{A} \mapsto \mathbb{R}$  is the running reward of the current state and action  $(X_s^a, a_s)$ , and  $\beta > 0$  is a discount factor that measures the time-depreciation of the objective value (or the impatience level of the agent). Note that the state process  $X^a = \{X_s^a, s \geq t\}$  depends on the starting (initial) time-state pair  $(t, x)$ . For ease of notation, we denote by  $X^a$  instead of  $X^{t,x,a} = \{X_s^{t,x,a}, s \geq t\}$  the solution to the SDE in (29) when there is no ambiguity.