SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hanyang Zhao
Columbia University hz2684@columbia.edu
Wenpin Tang
Columbia University wt2319@columbia.edu
David D. Yao
Columbia University yao@columbia.edu
We study reinforcement learning (RL) in the setting of continuous time and space, for an infinite horizon with a discounted objective and the underlying dynamics driven by a stochastic differential equation. Built upon recent advances in the continuous approach to RL, we develop a notion of occupation time (specifically for a discounted objective), and show how it can be effectively used to derive performance-difference and local-approximation formulas. We further extend these results to illustrate their applications in the PG (policy gradient) and TRPO/PPO (trust region policy optimization/ proximal policy optimization) methods, which have been familiar and powerful tools in the discrete RL setting but under-developed in continuous RL. Through numerical experiments, we demonstrate the effectiveness and advantages of our approach.
Reinforcement Learning (RL, [52]) has been successfully applied to wide-ranging domains in the past decade, including achieving superhuman performance in games like Atari and Go [35, 48, 49], enhancing Large Language Models using human feedback [8, 10], and showing potentials in improving traditional model-based decisions in healthcare, inventory management, and finance [9, 30, 33]. Most existing works, including all references cited above, are formulated and solved as discrete-time sequential optimization problems such as Markov decision processes (MDPs, [42]). Yet in many applications, agents may need to monitor and interact with the random environment at an ultra-high frequency (e.g., autonomous driving, robot navigation, and high-frequency stock trading), which calls for a continuous-time/space approach.
Recent years have witnessed a fast growing body of research that has extended the frontiers of continuous RL in several important directions including, for instance, modeling the noise or randomness in the environment dynamics as following a stochastic differential equation (SDE), and incorporating an entropy-based regularizer into the objective function [58] to facilitate the exploration-exploitation tradeoff; designing model-free methods and algorithms, along with applications to portfolio optimization [18, 20, 21, 22]; studying regret bounds [5, 54], and so forth.
In this paper, we continue the above trend in continuous RL, focusing on an infinite horizon formulation with a discounted objective and the underlying dynamics driven by an SDE [24, 39]. We are specifically motivated by the following two questions.
(Q1) The visitation frequency in MDP (with a discounted objective) is defined as:  $\rho(s) = \sum_{t=0}^{\infty} \gamma^t \cdot \mathbb{P}(Y_t = s)$ , where  $\{Y_t\}$  is a Markov chain with state space  $S := \{s\}$ , and  $\gamma \in (0,1)$  is a discount factor. It plays an important role in many RL algorithms for MDP. So, a natural question is, what is the continuous counterpart of  $\rho(s)$ ?
(Q2) For continuous RL, how can we characterize the difference in performance between two policies? In particular, can we derive performance-difference formulas similar to those in the MDP
case [23, 45]? Can we adapt and apply the ideas and tools of the efficient policy optimization methods (e.g., [45, 47]) to the continuous RL setting?
Main contributions. We provide a unified theory/framework for policy optimization in continuous time and space. Specifically, we have addressed the above two questions (Q1) and (Q2) by developing the notion of occupation time/measure, specifically for a discounted objective, and focusing on its associated  $q$ -value. Based on these two quantities, we derive the performance-difference formula for continuous RL by means of perturbation analysis. Leveraging the performance-difference formula, we develop the continuous counterparts of the policy gradient (PG, [53]) and also propose the local approximation for the performance metric, for which we derive a bound on it and allow the development of a minorization-majorization (MM) algorithm. We further develop the continuous counterparts of trust region policy optimization/ proximal policy optimization (TRPO/PPO) methods in [45, 47], which have been familiar and powerful tools in the discrete RL setting but under-developed in continuous RL, as approximations to the previous algorithms. (What is worth mentioning is that these policy optimization algorithms do not require any a priori discretization of time and space.) Through numerical examples we show the convergence of these algorithms when applied to certain stochastic control tasks in continuous time and space.
Organization of the paper. In Section 2 we present the continuous RL formulation and develop necessary tools. The main results, the performance-difference formula (Theorem 2) and the bound (Theorem 5) are provided in Section 3. In Section 4 we propose two algorithms, policy gradient with random rollout and PPO with adaptive penalty, based on our analyses and theoretical results; and illustrate their performance via numerical experiments. Concluding remarks are summarized in Section 5.
Related works. One line of research on continuous RL focuses on modeling the underlying dynamics as a deterministic system, typically following a deterministic ordinary differential equation. Several papers [9, 37, 44] solve the problems via a priori discretization in either time or space; [13] develops a framework to apply the temporal difference to the continuous setting, and proposes algorithms that combine value iteration or advantage update as in [3, 4, 6] to avoid explicit discretization; [37] further investigates policy gradient methods, followed by more recent studies on model-free continuous RL methods [25, 29, 57] or model-based ones [14]; [55] studies the sensitivity of existing off-policy algorithms along with advantage updating to propose continuous RL algorithms that are robust to time discretization.
The formulation of continuous RL in a stochastic setting (i.e., with the state process driven by an SDE), can be traced back to [38], which however provides no data-driven solution. Recently, [58] develops an exploratory control model for the continuous RL. Built upon this approach and for a finite-horizon objective, [20] studies policy evaluation, and [21] policy gradient. Furthermore, [22] brings forth the notion of  $q$ -value, which leads to a continuous analogue of  $Q$ -learning. Also worth noting is [2, 28], which studies RL in the mean-field regime where continuous-time processes occur in the limit, and [16] extends the study to jump-diffusion processes.
In discrete-time MDPs, the body of research on bounding the performance difference between two policies also relates to our work: [1, 45] develop a policy improvement bound for the discounted total reward; [12, 64] studies the long-run average reward, and [11] proposes a bound that is continuous with respect to the discount factor.
Notation. For a measurable set  $\mathcal{A}$ , denote  $\mathcal{P}(\mathcal{A})$  for the set of probability distributions over  $\mathcal{A}$ . For a vector  $x$ , denote by  $\| x\|_2$  the Euclidean norm of  $x$ . For a matrix  $A$ , denote by  $\| A\|_F$  the Frobenius norm of  $A$ , and  $A^2 \coloneqq AA^\top$  where  $A^\top$  is the transpose of  $A$ . For a positive-definite matrix  $A$ , denote by  $A^{\frac{1}{2}}$  the square root matrix of  $A$ . For  $A, B$  two matrices of the same size, denote by  $A \circ B$  the inner product of  $A$  and  $B$ . For a function  $f$  on an Euclidean space,  $\nabla f$  (resp.  $\nabla^2 f$ ) denotes the gradient (resp. the Hessian) of  $f$ . For two distributions  $P, Q \in \mathcal{P}(A)$ , denote by  $W_2(P, Q)$  the Wasserstein-2 distance (or Quadratic Wasserstein distance) between  $P$  and  $Q$ :
$$
W _ {2} (P, Q) = \left(\inf _ {\gamma \in \Gamma (P, Q)} \mathbf {E} _ {(x, y) \sim \gamma} \| x - y \| ^ {2}\right) ^ {1 / 2},
$$
where  $\Gamma(P, Q)$  is the set of all couplings of  $P$  and  $Q$ ; and denote by  $D_{\mathrm{KL}}(P||Q)$  the KL-divergence between  $P$  and  $Q$ :  $D_{\mathrm{KL}}(P||Q) = \int p(x) \log \left(\frac{p(x)}{q(x)}\right) dx$ , in which  $p$  and  $q$  denote the probability densities of  $P$  and  $Q$ .
2 Formulation and Preliminaries
Continuous RL. We start with a quick formulation of the continuous RL, based on the same modeling framework as in [58]. Assume that the state space is  $\mathbb{R}^n$ , and denote by  $\mathcal{A}$  the action space. Let  $\pi (\cdot \mid x)\in \mathcal{P}(\mathcal{A})$  be a (state) feedback policy given the state  $x\in \mathbb{R}^n$ . A continuous RL problem is formulated by a distributional (or relaxed) control approach [62], which is motivated by the trial and error process in RL. The state dynamics  $(X_{s}^{a}, s\geq 0)$  is governed by the Ito process:
$$
\mathrm {d} X _ {s} ^ {a} = b \left(X _ {s} ^ {a}, a _ {s}\right) \mathrm {d} s + \sigma \left(X _ {s} ^ {a}, a _ {s}\right) \mathrm {d} B _ {s}, \quad X _ {0} ^ {a} \sim \mu \in \mathcal {P} (\mathbb {R} ^ {n}), \tag {1}
$$
where  $(B_{t}, t \geq 0)$  is the  $m$ -dimensional Brownian motion,  $b: \mathbb{R}^n \times \mathcal{A} \mapsto \mathbb{R}^n$ ,  $\sigma: \mathbb{R}^n \times \mathcal{A} \mapsto \mathbb{R}^{n \times m}$ , and the action  $a_{s}$  is generated from the distribution  $\pi(\cdot | X_{s}^{a})$  by external randomization. To avoid technical difficulties, we assume that the stochastic processes (1) (and (3), (9) below) are well-defined, see [24, Section 5.3] or [50, Chapter 6] for background.
From now on, write  $(X_s^\pi, a_s^\pi)$  for the state and action at time  $s$  given by the process (1) under the policy  $\pi = \{\pi(\cdot \mid x) \in \mathcal{P}(\mathcal{A}) : x \in \mathbb{R}^n\}$ . The goal here is to find the optimal feedback policy  $\pi^*$  that maximizes the expected discounted reward over an infinite time horizon:
$$
V ^ {*} := \max  _ {\pi} \mathbb {E} \left[ \int_ {0} ^ {+ \infty} e ^ {- \beta s} \left[ r \left(X _ {s} ^ {\pi}, a _ {s} ^ {\pi}\right) + \gamma p \left(X _ {s} ^ {\pi}, a _ {s} ^ {\pi}, \pi (\cdot | X _ {s} ^ {\pi})\right) \right] d s \mid X _ {0} ^ {\pi} \sim \mu \right], \tag {2}
$$
where  $r: \mathbb{R}^n \times \mathcal{A} \mapsto \mathbb{R}^+$  is the running reward of the current state and action  $(X_s^\pi, a_s^\pi)$ ;  $p: \mathbb{R}^n \times \mathcal{A} \times \mathcal{P}(\mathcal{A}) \mapsto \mathbb{R}$  is a regularizer which facilitates exploration (e.g., in [58],  $p$  is taken as the differential entropy defined by  $p(x, a, \pi(\cdot)) = -\log \pi(a)$ );  $\gamma \geq 0$  is a weight parameter on exploration (also known as the "temperature" parameter); and  $\beta > 0$  is a discount factor that measures the time-depreciation of the objective value (or the impatience level of the agent).
Performance metric. A standard approach to solving the problem in (2) is to find a sequence of policies  $\pi_k = \{\pi_k(\cdot \mid x):x\in \mathbb{R}^n\}$ ,  $k = 1,2,\ldots$  such that the value functions following the policies  $\pi_k$  will converge to  $V^{*}$ , or be at least increasing in  $k$ , i.e., demonstrating policy improvement.
Given a policy  $\pi (\cdot)$ , let  $\tilde{b} (x,\pi (\cdot))\coloneqq \int_{\mathcal{A}}b(x,a)\pi (a)\mathrm{d}a$  and  $\tilde{\sigma} (x,\pi (\cdot))\coloneqq \left(\int_{\mathcal{A}}\sigma^{2}(x,a)\pi (a)\mathrm{d}a\right)^{\frac{1}{2}}$ . Assume (for technical purpose) that  $\tilde{\sigma} (x,\pi (\cdot))$  is positive definite for every  $x\in \mathbb{R}^n$ . It is sometimes more convenient to consider the following equivalent SDE representation of (1):
$$
\mathrm {d} \tilde {X} _ {s} = \tilde {b} (\tilde {X} _ {s}, \pi (\cdot | \tilde {X} _ {s})) \mathrm {d} s + \tilde {\sigma} (\tilde {X} _ {s}, \pi (\cdot | \tilde {X} _ {s})) \mathrm {d} \tilde {B} _ {s}, \quad \tilde {X} _ {0} \sim \mu , \tag {3}
$$
in the sense that there exists a probability measure  $\tilde{\mathbb{P}}$  which supports the  $m$ -dimensional Brownian motion  $(\tilde{B}_s, s \geq 0)$ , and for each  $s \geq 0$ , the distribution of  $\tilde{X}_s$  under  $\tilde{\mathbb{P}}$  agrees with that of  $X_s$  under  $\mathbb{P}$  defined by (1), see Appendix A. Note that the dynamics in (3) does not require external randomization. Also set  $\tilde{r}(x,\pi) := \int_{\mathcal{A}} r(x,a)\pi(a)\mathrm{d}a$  and  $\tilde{p}(x,\pi) := \int_{\mathcal{A}} p(x,a,\pi)\pi(a)\mathrm{d}a$ . We formally define the (state) value function given the feedback policy  $\{\pi(\cdot | x): x \in \mathbb{R}^n\}$  by
$$
\begin{array}{l} V (x; \pi) := \mathbb {E} \left[ \int_ {0} ^ {+ \infty} e ^ {- \beta s} \left[ r \left(X _ {s} ^ {\pi}, a _ {s} ^ {\pi}\right) + \gamma p \left(X _ {s} ^ {\pi}, a _ {s} ^ {\pi}, \pi (\cdot | X _ {s} ^ {\pi})\right) \right] d s \mid X _ {0} ^ {\pi} = x \right] \tag {4} \\ = \mathbb {E} \left[ \int_ {0} ^ {\infty} e ^ {- \beta s} \left[ \tilde {r} \left(\tilde {X} _ {s} ^ {\pi}, \pi (\cdot | \tilde {X} _ {s} ^ {\pi})\right) + \gamma \tilde {p} \left(\tilde {X} _ {s} ^ {\pi}, \pi (\cdot | \tilde {X} _ {s} ^ {\pi})\right) \right] \mathrm {d} s | \tilde {X} _ {0} ^ {\pi} = x \right], \\ \end{array}
$$
which, under suitable conditions on model parameters  $(b,\sigma,r,p)$  and the policy  $\pi$ , is characterized by the Hamilton-Jacobi equation (see [21, 56]):
$$
\beta V (x; \pi) - \tilde {b} (x, \pi) \cdot \nabla V (x; \pi) - \frac {1}{2} \tilde {\sigma} ^ {2} (x, \pi) \circ \nabla^ {2} V (x; \pi) - \tilde {r} (x, \pi) - \gamma \tilde {p} (x, \pi) = 0. \tag {5}
$$
More technical details regarding the above formulation are spelled out in the Appendix.
We can now define the performance metric as follows:
$$
\eta (\pi) := \int_ {\mathbb {R} ^ {n}} V (x; \pi) \mu (d x), \tag {6}
$$
so  $V^{*} = \max_{\pi}\eta (\pi)$ . The main task of the continuous RL is to approximate  $\max_{\pi}\eta (\pi)$  by constructing a sequence of policies  $\pi_k, k = 1,2,\ldots$  recursively such that  $\eta (\pi_k)$  is non-decreasing.
Policy evaluation. Let's first recall a general approach in [20], which can be used to learn the state value function in (4) or the performance metric in (6) for a given policy  $\pi$ . The idea is that for any  $T > 0$  and a suitable test process  $(\xi_t, t \geq 0)$ ,
$$
\mathbb {E} \int_ {0} ^ {T} \xi_ {t} \left[ \mathrm {d} V \left(X _ {t} ^ {\pi}; \pi\right) + r \left(X _ {t} ^ {\pi}, a _ {t} ^ {\pi}\right) \mathrm {d} t + \gamma p \left(X _ {t} ^ {\pi}, a _ {t} ^ {\pi}, \pi (\cdot | X _ {t} ^ {\pi})\right) \mathrm {d} t - \beta V \left(X _ {t} ^ {\pi}; \pi\right) \mathrm {d} t \right] = 0. \tag {7}
$$
If we parameterize  $V(x;\pi) = V^{\phi}(x)$  and choose the special test function  $\xi_{t} = \frac{\partial V^{\phi}(X_{t}^{\pi})}{\partial\phi}$ , stochastic approximation leads to the online update:
$$
\phi \leftarrow \phi + \alpha \frac {\partial V ^ {\phi} \left(X _ {t} ^ {\pi}\right)}{\partial \phi} \left[ \mathrm {d} V ^ {\phi} \left(X _ {t} ^ {\pi}\right) + r \left(X _ {t} ^ {\pi}, a _ {t} ^ {\pi}\right) \mathrm {d} t + \gamma p \left(X _ {t} ^ {\pi}, a _ {t} ^ {\pi}, \pi (\cdot | X _ {t} ^ {\pi})\right) \mathrm {d} t - \beta V ^ {\phi} \left(X _ {t} ^ {\pi}\right) \mathrm {d} t \right], \tag {8}
$$
where  $\alpha > 0$  is the learning rate. This recovers the mean-squared TD error (MSTDE) method for policy evaluation in the discrete RL [51].
$q$ -value. The Q-value function [61] and the advantage function [3, 34] in discrete-time MDPs play a critical role in reinforcement learning theory and algorithms. However, as pointed out in [55], these concepts will not apply when the time interval shrinks to 0 (as in the continuous setting). To derive algorithms that fit the need of a continuous stochastic environment, [55, 22] proposed the advantage rate function. Namely, given a policy  $\pi$  and  $(t,x,a)\in [0,\infty)\times \mathbb{R}^n\times \mathcal{A}$ , consider a "perturbed" policy  $\hat{\pi}$  as follows. It takes the action  $a\in \mathcal{A}$  on  $[t,t + \Delta t)$  where  $\Delta t > 0$ , and then follows  $\pi$  on  $[t + \Delta t,\infty)$ . The corresponding state process  $X^{\hat{\pi}}$  given  $X_{t}^{\hat{\pi}} = x$  is broken into two pieces. On  $[t,t + \Delta t)$ , it is  $X^{a}$  which is the solution to
$$
\mathrm {d} X _ {s} ^ {a} = b \left(X _ {s} ^ {a}, a\right) \mathrm {d} s + \sigma \left(X _ {s} ^ {a}, a\right) \mathrm {d} B _ {s}, s \in [ t, t + \Delta t), \quad X _ {t} ^ {a} = x, \tag {9}
$$
while on  $[t + \Delta t, \infty)$ , it is  $X^{\pi}$  following (3) with the initial time-state pair  $(t + \Delta t, X_{t + \Delta t}^{a})$ . With  $\Delta t > 0$  as time discretization, the generalization of the conventional Q-function can be expressed as
$$
Q _ {\Delta t} (x, a; \pi) = V (x; \pi) + \left[ \mathcal {H} ^ {a} \left(x, \frac {\partial V}{\partial x} (x; \pi), \frac {\partial^ {2} V}{\partial x ^ {2}} (x; \pi)\right) - \beta V (x; \pi) \right] \Delta t + o (\Delta t). \tag {10}
$$
where  $\mathcal{H}^a (x,y,A)\coloneqq b(x,a)\cdot y + \frac{1}{2}\sigma^2 (x,a)\circ A + r(x,a)$  is the (generalized) Hamilton function in stochastic control theory [62]. This motivates the following definition.
Definition 1. [22] For a given policy  $\pi \in \Pi$  and  $(x,a)\in \mathbb{R}^n\times \mathcal{A}$ , define the  $q$ -value as
$$
q (x, a; \pi) := \lim  _ {\Delta t \rightarrow 0} \frac {Q _ {\Delta t} (x , a ; \pi) - V (x ; \pi)}{\Delta t} = \mathcal {H} ^ {a} \left(x, \frac {\partial V}{\partial x} (x; \pi), \frac {\partial^ {2} V}{\partial x ^ {2}} (x; \pi)\right) - \beta V (x; \pi), \tag {11}
$$
which represents the instantaneous advantage rate of an action in a given state under a given policy.
This section is concerned with theoretical developments. In Section 3.1, we define the (discounted) occupation time/measure which is the continuous analog of visitation frequency in MDPs. It is crucial in deriving the performance-difference formula in Section 3.2, which spins off two different algorithms - policy gradient and TRPO/PPO. In Section 3.3, we propose a local approximation for the performance metric, and derive a bound from which an MM algorithm is constructed.
3.1 Discounted Occupation Time
Here we first provide an answer to (Q1) by defining the notion of discounted occupation time for the continuous RL.
Definition 2. Let  $X = (X_{t}, t \geq 0)$  be governed by the SDE (3), and assume that it has a probability density function  $p^{\pi}(\cdot, t)$  at each time  $t$ . For each  $x \in \mathbb{R}^n$  and  $t \geq 0$ , define the  $\beta$ -discounted occupation time of  $X$  at the state  $x$  by
$$
d _ {\mu} ^ {\pi} (x) := \int_ {0} ^ {\infty} e ^ {- \beta s} p ^ {\pi} (x, s) d s. \tag {12}
$$
So  $d_{\mu}^{\pi}(\cdot)$  induces a finite measure on  $\mathbb{R}^n$  with a total mass of  $\beta^{-1}$ , which we call the discounted occupation measure.
In probability theory, the definition in (12) is referred to as the  $\beta$ -potential of  $X$ , which gives discounted visitation frequencies of the state process. We record the following result, which will be useful in the derivation of the performance-difference formula. It is a consequence of the occupation time formula [41, 43].
Lemma 1. Under the conditions in 2, we have  $\mathbb{E}\int_0^\infty e^{-\beta s}\varphi (X_s)\mathrm{d}s = \int_{\mathbb{R}^n}d_\mu^\pi (x)\varphi (x)\mathrm{d}x,$  for any measurable function  $\varphi :\mathbb{R}^n\mapsto \mathbb{R}_+$  for which the expectation exists.
3.2 Performance-Difference Formula
We are now ready to answer (Q2) by deriving the performance-difference formula between two policies in terms of the discounted occupation time in (12) and the  $q$ -values in (11).
Theorem 2. Given two feedback policies  $\hat{\pi} = \{\hat{\pi} (\cdot \mid x):x\in \mathbb{R}^n\}$  and  $\pi = \{\pi (\cdot \mid x):x\in \mathbb{R}^n\}$ , we have:
$$
\eta (\hat {\pi}) - \eta (\pi) = \int_ {\mathbb {R} ^ {n}} d _ {\mu} ^ {\hat {\pi}} (x) \left[ \int_ {\mathcal {A}} \hat {\pi} (a \mid x) \left(q (x, a; \pi) + \gamma p (x, a, \hat {\pi})\right) \mathrm {d} a \right] \mathrm {d} x. \tag {13}
$$
Proof sketch. The full proof is detailed in Appendix B.1. The essence of the proof is to use the perturbation theory and properties of the discounted occupation time. Define an operator  $\mathcal{L}^{\pi}: C^{2}(\mathbb{R}^{n}) \mapsto C(\mathbb{R}^{n})$  associated with the diffusion process as:
$$
\left(\mathcal {L} ^ {\pi} \varphi\right) (x) := - \beta \varphi (x) + \tilde {b} (x, \pi) \cdot \nabla \varphi (x) + \frac {1}{2} \tilde {\sigma} (x, \pi) ^ {2} \circ \nabla^ {2} \varphi (x). \tag {14}
$$
Then the Hamilton-Jacobi equation that characterizes the state value function can be expressed as:
$$
- \mathcal {L} ^ {\pi} V (x; \pi) = \tilde {r} (x, \pi) + \gamma \tilde {p} (x, \pi) \tag {15}
$$
Note that for any  $\varphi \in C^2 (\mathbb{R}^n)$ , we have  $\int_{\mathbb{R}^n}d_\mu^\pi (y)(-\mathcal{L}^\pi \varphi)(y)\mathrm{d}y = \int_{\mathbb{R}^n}\varphi (y)\mu (\mathrm{d}y)$ . This allows us to express the performance difference in model-dynamics related terms:
$$
\eta (\hat {\pi}) - \eta (\pi) = \int_ {\mathbb {R}} d _ {\mu} ^ {\hat {\pi}} (y) \left[ \left(\mathcal {L} ^ {\hat {\pi}} - \mathcal {L} ^ {\pi}\right) V (y; \pi) + \tilde {r} (y, \hat {\pi}) + \gamma \tilde {p} (y, \hat {\pi}) - \tilde {r} (y, \pi) - \gamma \tilde {p} (y, \pi) \right] d y. \tag {16}
$$
What remains is to reduce the above to the desired result in (13).
As discussed in Section 2, our main task is to construct (algorithmically) a sequence of policies  $\pi_{k}$  along which the performance improves. Here we illustrate how some well known approaches of policy improvement (from  $\pi$  to  $\hat{\pi}$ ) are instances of the performance difference formula (13).
$q$ -learning and soft  $q$ -learning. Since  $d_{\mu}^{\hat{\pi}} \geq 0$ , we only need to ensure that for all  $x \in \mathbb{R}^n$ ,  $\int_{\mathcal{A}} \hat{\pi}(a \mid x) (q(x, a; \pi) + \gamma p(x, a, \hat{\pi})) \, \mathrm{d}a \geq 0$ . This boils down to the problem that for any  $x$ , find  $v \equiv \hat{\pi}(\cdot \mid x)$  to maximize
$$
\int_ {\mathcal {A}} v (a) \left(q (x, a; \pi) + \gamma p (x, a, v)\right) \mathrm {d} a. \tag {17}
$$
There are two special cases:
(i) If  $p(x, a, v) \equiv 0$ , then  $v = \delta(a^{*})$  where  $a^{*} = \arg \max_{a} q(x, a, \pi)$ . This is essentially the counterpart of  $Q$ -Learning [61] in the discrete time, which we call  $q$ -learning.
(ii) If  $p(x, a, v) = -\log(v(a))$ , this is known as the entropy regularizer [17, 59]. Concretely, we need to solve
$$
\max  _ {v \in \mathcal {P} (\mathcal {A})} \int_ {\mathcal {A}} v (a) \left(q (x, a; \pi) - \gamma \log v (a)\right) \mathrm {d} a. \tag {18}
$$
which has a closed form solution with  $v^{*}(a) \propto \exp \left(\frac{q(x, a, \pi)}{\gamma}\right)$ , i.e.  $v^{*}$  is the Boltzmann policy for  $q$ -functions. This is a "soft" (à la [17]) version of the  $q$ -learning mentioned above.
Policy gradient. We may use function approximations to  $\pi$  by a parametric family  $\pi^{\theta}$ , with  $\theta \in \Theta \subseteq \mathbb{R}^{L}$ . For simplicity, we write  $d_{\mu}^{\theta_0}$  (resp.  $\eta(\theta)$ ) for  $d_{\mu}^{\pi^{\theta_0}}$  (resp.  $\eta(\pi^{\theta})$ ). Setting  $\hat{\pi} = \pi^{\theta}$  and  $\pi = \pi^{\theta_0}$  in (13) and taking derivative with respect to  $\theta$  on both sides, we get the following result.
Theorem 3 (Policy Gradient). The policy gradient at  $\pi^{\theta_0}$  is:
$$
\left. \nabla_ {\theta} \eta (\theta) \right| _ {\theta = \theta_ {0}} = \frac {1}{\beta} \mathbb {E} _ {(x, a)} \left[ \nabla_ {\theta} \log \left(\pi^ {\theta} (a \mid x)\right) \left(q (x, a; \pi^ {\theta_ {0}}) + \gamma p (x, a, \pi^ {\theta_ {0}})\right) + \gamma \nabla_ {\theta} p (x, a, \pi^ {\theta}) \right], \tag {19}
$$
where the expectation is w.r.t.  $(x,a)\sim (\beta d_{\mu}^{\theta_0},\pi^{\theta_0})$  , meaning  $x\sim \beta d_{\mu}^{\theta_0}(\cdot)$  and then  $a\sim \pi^{\theta_0}(\cdot \mid x)$