SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Listed below are the standard assumptions to ensure the well-posedness of the stochastic control problem in (29)-(30).
Assumption 2. The following conditions are assumed throughout:
(i)  $b, \sigma, r$  are all continuous functions in their respective arguments;
(ii)  $b, \sigma$  are uniformly Lipschitz continuous in  $x$ , i.e., there exists a constant  $C > 0$  such that for  $\varphi \in \{b, \sigma\}$ ,
$$
\left\| \varphi (x, a) - \varphi \left(x ^ {\prime}, a\right) \right\| _ {2} \leq C \| x - x ^ {\prime} \| _ {2}, \quad f o r a l l a \in \mathcal {A}, x, x ^ {\prime} \in \mathbb {R} ^ {n}; \tag {31}
$$
(iii)  $b, \sigma$  have linear growth in  $x$  and  $a$ , i.e., there exists a constant  $C > 0$  such that for  $\varphi \in \{b, \sigma\}$ ,
$$
\left\| \varphi (x, a) \right\| _ {2} \leq C \left(1 + \left\| x \right\| _ {2} + \left\| a \right\| _ {2}\right), \quad f o r a l l (x, a) \in \mathbb {R} ^ {n} \times \mathcal {A}; \tag {32}
$$
(iv)  $r$  has polynomial growth in  $x$  and  $a$ , i.e., there exists a constant  $C > 0$  and  $\mu \geq 1$  such that
$$
\left| r (x, a) \right| \leq C \left(1 + \| x \| _ {2} ^ {\mu} + \| a \| _ {2} ^ {\mu}\right) \quad f o r a l l (x, a) \in \mathbb {R} ^ {n} \times \mathcal {A}. \tag {33}
$$
The key idea underlying exploratory stochastic control is to use a randomized policy (or relaxed control), i.e., apply a probability distribution to the admissible action space. To do so, let's assume the probability space is rich enough to support a uniform random variable  $Z$  that is independent of the Brownian motion  $B = \{B_t\}$ . We then expand the original filtered probability space to  $\left(\Omega, \mathcal{F}, \mathbb{P}; \{\mathcal{F}_s\}_{s \geq 0}\right)$ , where  $\mathcal{F}_s = \mathcal{F}_s^B \vee \sigma(Z)$  (i.e., augment  $\mathcal{F}_s^B$  with the sigma field generated by  $Z$ ).
Let  $\pi : \mathbb{R}^n \ni x \mapsto \pi(\cdot \mid x) \in \mathcal{P}(\mathcal{A})$  be a stationary feedback policy given the state at  $x$ , where  $\mathcal{P}(\mathcal{A})$  is a suitable collection of probability distributions (with density functions). At each time  $s$ , an action  $a_s$  is generated from the distribution  $\pi(\cdot \mid X_s^a)$ , i.e. the policy only depends on the current state. In other words, we only consider stationary, or time-independent feedback control policies for the stochastic control problem (29)-(30).
Given a stationary policy  $\pi \in \mathcal{P}(\mathcal{A})$ , an initial state  $x$ , and an  $\{\mathcal{F}_s\}$ -progressively measurable action process  $a^\pi = \{a_s^\pi, s \geq 0\}$  generated from  $\pi$ , the state process  $X^\pi = \{X_s^\pi, s \geq 0\}$  follows:
$$
\mathrm {d} X _ {s} ^ {\pi} = b \left(X _ {s} ^ {\pi}, a _ {s} ^ {\pi}\right) \mathrm {d} s + \sigma \left(X _ {s} ^ {\pi}, a _ {s} ^ {\pi}\right) \mathrm {d} B _ {s}, s \geq t, \quad X _ {0} ^ {\pi} = x, \tag {34}
$$
defined on  $\left(\Omega, \mathcal{F}, \mathbb{P}; \{\mathcal{F}_s\}_{s \geq 0}\right)$ . It is easy to see that the dynamics in (34) define a time-homogeneous Markov process, such that for each  $t \geq 0$  and  $x$ :
$$
\left(X _ {s} ^ {\pi} \mid X _ {0} ^ {\pi} = x\right) \stackrel {{d}} {{=}} \left(X _ {s + t} ^ {\pi} \mid X _ {t} ^ {\pi} = x\right), s \geq 0.
$$
Consequently, the objective in (30) is independent of time  $t$ , and is equal to:
$$
\mathbb {E} \left[ \int_ {0} ^ {+ \infty} e ^ {- \beta s} r \left(X _ {s} ^ {\pi}, a _ {s} ^ {\pi}\right) \mathrm {d} s \mid X _ {0} ^ {\pi} = x \right]. \tag {35}
$$
Furthermore, following [58], we can add a regularizer to the objective function to encourage exploration (represented by the randomized policy), leading to
$$
V (t, x; \pi) := \mathbb {E} \left[ \int_ {t} ^ {\infty} e ^ {- \beta (s - t)} \left[ r \left(X _ {s} ^ {\pi}, a _ {s} ^ {\pi}\right) + \gamma p \left(X _ {s} ^ {\pi}, a _ {s} ^ {\pi}, \pi (\cdot | X _ {s} ^ {\pi}),\right) \right] d s \mid X _ {t} ^ {\pi} = x \right], \tag {36}
$$
where  $p: \mathbb{R}^n \times \mathcal{A} \times \mathcal{P}(\mathcal{A}) \mapsto \mathbb{R}$  is the regularizer, and  $\gamma \geq 0$  is a weight parameter on exploration (also known as the "temperature" parameter). For instance, in [58],  $p$  is taken as the differential entropy,
$$
p (x, a, \pi (\cdot)) := - \log \pi (a),
$$
and hence, the "entropy" regularizer. The same argument as before justifies that  $V(t,x;\pi)$  is independent of time  $t$ . That is, for all  $t\geq 0$ ,
$$
V (t, x; \pi) \equiv V (x; \pi) := \mathbb {E} ^ {\mathbb {P}} \left[ \int_ {0} ^ {\infty} e ^ {- \beta s} \left[ r \left(X _ {s} ^ {\pi}, a _ {s} ^ {\pi}\right) + \gamma p \left(X _ {s} ^ {\pi}, a _ {s} ^ {\pi}, \pi (\cdot \mid X _ {s} ^ {\pi})\right) \right] d s \mid X _ {0} ^ {\pi} = x \right]; \tag {37}
$$
which is the state-value function under the policy  $\pi$ ,  $V(x;\pi)$ , in (4), and which, in turn, leads to the performance function  $\eta(\pi)$  in (6). Moreover, recall the main task of the continuous RL is to find (or approximate)  $\eta^{*} = \max_{\pi} \eta(\pi)$ , where  $\max$  is over all admissible policies.
A.2 Controlled SDE and the HJ Equation
Note that the exploratory state dynamics in (34) is governed by a general Itô process. It is sometimes more convenient to consider an equivalent SDE representation—in the sense that its (weak) solution has the same distribution as the Itô process in (34) at each fixed time  $t$ . It is known ([58]) that when  $n = m = 1$ , the marginal distribution of  $\{X_s^\pi, s \geq 0\}$  agrees with that of the solution to the SDE, denoted by  $\{\tilde{X}_s, s \geq 0\}$ :
$$
\mathrm {d} \tilde {X} _ {s} = \tilde {b} \left(\tilde {X} _ {s}, \pi \left(\cdot \mid \tilde {X} _ {s}\right)\right) \mathrm {d} s + \tilde {\sigma} \left(\tilde {X} _ {s}, \pi \left(\cdot \mid \tilde {X} _ {s}\right)\right) \mathrm {d} \tilde {B} _ {s}, \quad \tilde {X} _ {0} = x,
$$
where  $\tilde{b}(x, \pi(\cdot)) = \int_{\mathcal{A}} b(x, a) \pi(a) \mathrm{d}a$  and  $\tilde{\sigma}(x, \pi(\cdot)) = \sqrt{\int_{\mathcal{A}} \sigma^2(x, a) \pi(a) \mathrm{d}a}$ . This result is easily extended to arbitrary  $n, m$ , thanks to [7, Corollary 3.7], with the precise statement presented below (assuming  $n = m$  for ease of exposition).
Theorem 6. Assume that for a policy  $\pi$  and for every  $x$ ,
$$
\int_ {\mathcal {A}} \sigma^ {2} (x, a) \pi (a) \mathrm {d} a \in \mathbb {R} ^ {n \times n},
$$
is positive definite. Then there exists a filtered probability space  $\left(\tilde{\Omega},\tilde{\mathcal{F}},\left\{\tilde{\mathcal{F}}_t\right\}_{t\geq 0},\tilde{\mathbb{P}}\right)$  that supports a continuous  $\mathbb{R}^n$ -valued adapted process  $\tilde{X}$  and an  $n$ -dimensional Brownian motion  $\tilde{B}$  satisfying
$$
\mathrm {d} \tilde {X} _ {s} = \tilde {b} (\tilde {X} _ {s}, \pi (\cdot | \tilde {X} _ {s})) \mathrm {d} s + \tilde {\sigma} (\tilde {X} _ {s}, \pi (\cdot | \tilde {X} _ {s})) \mathrm {d} \tilde {B} _ {s}, \quad \tilde {X} _ {0} = x, \tag {38}
$$
$$
\tilde {b} (x, \pi (\cdot)) = \int_ {\mathcal {A}} b (x, a) \pi (a) \mathrm {d} a, \quad \tilde {\sigma} (x, \pi (\cdot)) = \left(\int_ {\mathcal {A}} \sigma^ {2} (x, a) \pi (a) \mathrm {d} a\right) ^ {\frac {1}{2}}.
$$
For each  $s \geq 0$ , the distribution of  $\hat{X}_s$  under  $\mathbb{P}$  agrees with that of  $X_s^\pi$  under  $\mathbb{P}$  defined in (34).
As a consequence, the state value function in (37) is identical to
$$
V (x; \pi) = \mathbb {E} \left[ \int_ {0} ^ {\infty} e ^ {- \beta s} \int_ {\mathcal {A}} \left[ r (\tilde {X} _ {s}, a) + \gamma p \left(\tilde {X} _ {s}, a, \pi (\cdot | \tilde {X} _ {s})\right) \right] \pi (a | \tilde {X} _ {s}) \mathrm {d} a \mathrm {d} s | \tilde {X} _ {0} = x \right].
$$
Also define
$$
\tilde {r} (x, \pi) = \int_ {\mathcal {A}} r (x, a) \pi (a | s) \mathrm {d} a, \quad \tilde {p} (x, \pi) = \int_ {\mathcal {A}} p (x, a, \pi) \pi (a | x) \mathrm {d} a,
$$
so we can simplify the value function to
$$
V (x; \pi) = \mathbb {E} \left[ \int_ {0} ^ {\infty} e ^ {- \beta s} \left[ \tilde {r} \left(\tilde {X} _ {s}, \pi\right) + \gamma \tilde {p} \left(\tilde {X} _ {s} ^ {\pi}, \pi (\cdot \mid \tilde {X} _ {s})\right) \right] d s \mid \tilde {X} _ {0} = x \right]. \tag {39}
$$
Following the principle of optimality,  $V$  then satisfies the HJ equation:
$$
\beta V (x; \pi) - \tilde {b} (x, \pi) \cdot \nabla V (x; \pi) - \frac {1}{2} \tilde {\sigma} ^ {2} (x, \pi) \circ \nabla^ {2} V (x; \pi) - \tilde {r} (x, \pi) - \gamma \tilde {p} (x, \pi) = 0. \tag {40}
$$
To guarantee that the HJ equation in (40) characterizes the state-value function in (39), we need
Assumption 3. Assume the following conditions hold:
(i)  $b, \sigma, r, p$  are all continuous functions in their respective arguments.
(ii)  $b, r, p$  are uniformly Lipschitz continuous in  $x$ , i.e., there exists a constant  $C > 0$  such that for  $\varphi \in \{b, r\}$ ,
$$
\left\| \varphi (x, a) - \varphi \left(x ^ {\prime}, a\right) \right\| _ {2} \leq C \left\| x - x ^ {\prime} \right\| _ {2}, \quad f o r a l l a \in \mathcal {A}, x, x ^ {\prime} \in \mathbb {R} ^ {n},
$$
$$
| p (x, a, \pi) - p (x ^ {\prime}, a, \pi) | \leq C \| x - x ^ {\prime} \| _ {2}, \quad f o r a l l a \in \mathcal {A}, \pi \in \mathcal {P} (\mathcal {A}), x, x ^ {\prime} \in \mathbb {R} ^ {n}.
$$
(iii)  $\tilde{\sigma}$  is globally bounded, i.e., there exist  $0 < \sigma_0 < \bar{\sigma}_0$  such that
$$
\sigma_ {0} ^ {2} \cdot I \leq \tilde {\sigma} ^ {2} (x, a) \leq \bar {\sigma} _ {0} ^ {2} \cdot I, \quad f o r a l l a \in \mathcal {A}, x \in \mathbb {R} ^ {n}.
$$
(iv) the SDE (38) has a weak solution which is unique in distribution.
(v)  $\pi(a|x)$  is measurable in  $(x, a)$  and is uniformly Lipschitz continuous in  $x$ , i.e., there exists a constant  $C > 0$  such that
$$
\int_ {\mathcal {A}} | \pi (a | x) - \pi (a | x ^ {\prime}) | d a \leq C \| x - x ^ {\prime} \| _ {2}, \quad f o r a l l x, x ^ {\prime} \in \mathbb {R} ^ {n}.
$$
Theorem 7. Under Assumption 3, the state-value function in (39) is the unique (subquadratic) viscosity solution to the HJ equation in (40).
Proof. By [56, Section 3.1], the HJ equation in (40) has a unique (subquadratic) viscosity solution under the conditions (i)-(iii). Further by [21, Lemma 2], the viscosity solution is the state-value function.  $\square$
Appendix B Proofs of Main Results (in §3)
Recall in the proof sketch of the Theorem in §3, we have defined the operator  $\mathcal{L}^{\pi}: C^{2}(\mathbb{R}^{n}) \mapsto C(\mathbb{R}^{n})$  as
$$
\left(\mathcal {L} ^ {\pi} \varphi\right) (x) := - \beta \varphi (x) + \tilde {b} (x, \pi) \cdot \nabla \varphi (x) + \frac {1}{2} \tilde {\sigma} (x, \pi) ^ {2} \circ \nabla^ {2} \varphi (x),
$$
which leads to the following characterization of the HJ equation:
$$
- \mathcal {L} ^ {\pi} V (x; \pi) = \tilde {r} (x, \pi) + \gamma \tilde {p} (x, \pi). \tag {41}
$$
We need the following two lemmas concerning the operator  $\mathcal{L}^{\pi}$ .
Lemma 8. For any  $\varphi \in C^2 (\mathbb{R}^n)$ , we have
$$
\int_ {\mathbb {R} ^ {n}} d _ {x} ^ {\pi} (y) (- \mathcal {L} ^ {\pi} \varphi) (y) \mathrm {d} y = \varphi (x).
$$
Proof. The left hand side of the above equation is
$$
\begin{array}{l} { = } { \mathbb { E } \int _ { 0 } ^ { \infty } e ^ { - \beta s } \left( \beta \varphi ( \tilde { X } _ { s } ^ { \pi } ) - \tilde { b } ( \tilde { X } _ { s } ^ { \pi } , \pi ) \frac { \partial \varphi } { \partial x } ( \tilde { X } _ { s } ^ { \pi } ) - \frac { 1 } { 2 } \tilde { \sigma } ( \tilde { X } _ { s } ^ { \pi } , \pi ) ^ { 2 } \frac { \partial ^ { 2 } \varphi } { \partial x ^ { 2 } } ( \tilde { X } _ { s } ^ { \pi } ) \right) \mathrm { d } s } \\ = \mathbb {E} \int_ {0} ^ {\infty} e ^ {- \beta s} \left[ \left(\beta \varphi (\tilde {X} _ {s} ^ {\pi}) - \tilde {b} (\tilde {X} _ {s} ^ {\pi}, \pi) \frac {\partial \varphi}{\partial x} (\tilde {X} _ {s} ^ {\pi}) - \frac {1}{2} \tilde {\sigma} (\tilde {X} _ {s} ^ {\pi}, \pi) ^ {2} \frac {\partial^ {2} \varphi}{\partial x ^ {2}} (\tilde {X} _ {s} ^ {\pi})\right) d s - \tilde {\sigma} (\tilde {X} _ {s} ^ {\pi}, \pi) \frac {\partial \varphi}{\partial x} (\tilde {X} _ {s} ^ {\pi}) d B _ {s} \right] \\ = \mathbb {E} \int_ {0} ^ {\infty} \mathrm {d} \left(- e ^ {- \beta s} \varphi (\tilde {X} _ {s} ^ {\pi})\right) \\ = \lim  _ {s \rightarrow \infty} \left(- e ^ {- \beta s} \varphi \left(\tilde {X} _ {s} ^ {\pi}\right)\right) + \varphi \left(\tilde {X} _ {0} ^ {\pi}\right) \\ = \varphi (x), \\ \end{array}
$$
where the first equality follows from the definition of the occupation time and the third equality from Itô's formula.
Lemma 9. Let  $\pi, \hat{\pi}$  be two feedback policies. We have
$$
\left(\mathcal {L} ^ {\hat {\pi}} - \mathcal {L} ^ {\pi}\right) V (x; \pi) + \tilde {r} (x, \hat {\pi}) - \tilde {r} (x, \pi) - \gamma \tilde {p} (x, \pi) = \int_ {\mathcal {A} (x)} \hat {\pi} (a \mid x) q (x, a; \pi) d a. \tag {42}
$$
Proof. By definition of  $q(x, a; \pi)$  in (11), we have
$$
\begin{array}{l} \mathrm {R H S} = \int_ {\mathcal {A} (x)} \hat {\pi} (a \mid x) \left(\mathcal {H} ^ {a} \left(x, \frac {\partial V}{\partial x} (x; \pi), \frac {\partial^ {2} V}{\partial x ^ {2}} (x; \pi)\right) - \beta V (x; \pi)\right) d a \\ = \int_ {\mathcal {A} (x)} \hat {\pi} (a \mid x) \left(b (x, a) \cdot \frac {\partial V}{\partial x} (x; \pi) + \frac {1}{2} \sigma^ {2} (x, a) \circ \frac {\partial^ {2} V}{\partial x ^ {2}} (x; \pi) + r (x, a) - \beta V (x; \pi)\right) d a \\ = \tilde {r} (x, \hat {\pi}) + \mathcal {L} ^ {\hat {\pi}} V ^ {\pi} (x) \\ = \tilde {r} (x, \hat {\pi}) - \tilde {r} (x, \pi) - \gamma \tilde {p} (x, \pi) + \mathcal {L} ^ {\hat {\pi}} V ^ {\pi} (x) - \mathcal {L} ^ {\pi} V ^ {\pi} (x) \\ = \quad \text {L H S}. \\ \end{array}
$$
Proof of Theorem 2. Note that in (13), the equation to be proven, the right hand side can be written as  $\int_{\mathbb{R}} d_{\mu}^{\hat{\pi}}(y) f(x; \pi, \hat{\pi}) \mathrm{d}y$ , with
$$
f (x; \pi , \hat {\pi}) := \int_ {A} \hat {\pi} (a \mid x) \left(q (x, a; \pi) + \gamma p (x, a, \hat {\pi})\right) d a.
$$
From Lemma 9, we have
$$
f (x; \pi , \hat {\pi}) = \left(\mathcal {L} ^ {\hat {\pi}} - \mathcal {L} ^ {\pi}\right) V (x; \pi) + \tilde {r} (x, \hat {\pi}) + \gamma \tilde {p} (x, \hat {\pi}) - \tilde {r} (x, \pi) - \gamma \tilde {p} (x, \pi). \tag {43}
$$
On the other hand, for the left hand side of (13), we have
$$
\eta (\pi) = \int_ {\mathbb {R} ^ {n}} V (y; \pi) \mu (\mathrm {d} y) = \int_ {\mathbb {R} ^ {n}} d _ {\mu} ^ {\hat {\pi}} (y) (- \mathcal {L} ^ {\hat {\pi}}) V (y; \pi) \mathrm {d} y, \tag {44}
$$
with the second equality following from Lemma 8; and
$$
\eta (\hat {\pi}) = \int_ {\mathbb {R}} d _ {\mu} ^ {\hat {\pi}} (y) [ \tilde {r} (y, \hat {\pi}) + \gamma \tilde {p} (y, \hat {\pi}) ] \mathrm {d} y, \tag {45}
$$
following the definition of the discounted expected occupation time; moreover, from (41), we have
$$
0 = \int_ {\mathbb {R}} d _ {\mu} ^ {\hat {\pi}} (y) \left[ (- \mathcal {L} ^ {\pi}) V (y; \pi) - \tilde {r} (y, \pi) - \gamma \tilde {p} (y, \pi) \right] \mathrm {d} y. \tag {46}
$$
Hence, combining the last three equations (44,45,46), we have
$$
\eta (\hat {\pi}) - \eta (\pi) = \int_ {\mathbb {R}} d _ {\mu} ^ {\hat {\pi}} (y) \left[ \left(\mathcal {L} ^ {\hat {\pi}} - \mathcal {L} ^ {\pi}\right) V (y; \pi) + \tilde {r} (y, \hat {\pi}) + \gamma \tilde {p} (y, \hat {\pi}) - \tilde {r} (y, \pi) - \gamma \tilde {p} (y, \pi) \right] d y. \tag {47}
$$
Thus, we have shown LHS=RHS in (13).
Proof. It suffices to show the integral version of the theorem:
$$
\begin{array}{l} \left. \nabla_ {\theta} \left(\eta \left(\pi^ {\theta}\right)\right) \right| _ {\theta = \theta} = \int_ {\mathbb {R} ^ {n}} d _ {\mu} ^ {\pi^ {\theta}} (x) \left[ \int_ {A} \nabla_ {\theta} \pi^ {\theta} (a \mid x) \left(q (x, a; \pi^ {\theta}) + \gamma p (x, a, \pi^ {\theta})\right) + \right. \tag {48} \\ \left. \gamma \cdot \pi^ {\theta} (a \mid x) \nabla_ {\theta} p (x, a, \pi^ {\theta}) d a \right] d x. \\ \end{array}
$$
As before, we simplify notation by denoting  $\eta (\pi^{\theta})$  as  $\eta (\theta)$  and  $d^{\pi^{\theta}}$  as  $d^{\theta}$ . Then, by Theorem 2), we have
$$
\eta (\theta + \delta \theta) - \eta (\theta) = \int_ {\mathbb {R} ^ {n}} d _ {\mu} ^ {\theta + \delta \theta} (x) \left[ \int_ {A} \pi^ {\theta + \delta \theta} (a \mid x) \left(q (x, a; \theta) + \gamma p (x, a, \theta + \delta \theta)\right) \mathrm {d} a \right] \mathrm {d} x. \tag {49}
$$
$$
f (\delta \theta) = \int_ {\mathcal {A}} \pi^ {\theta + \delta \theta} (a \mid x) (q (x, a; \theta) + \gamma p (x, a, \theta + \delta \theta)) d a.
$$
Note that  $f(0) = 0$ , which follows from
$$
\begin{array}{l} f (0) = \int_ {A} \pi^ {\theta} (a \mid x) (q (x, a; \theta) + \gamma p (x, a, \theta)) d a \\ = \int_ {\mathcal {A}} \pi^ {\theta} (a \mid x) \left(\mathcal {H} ^ {a} \left(x, \frac {\partial V}{\partial x} (x; \pi), \frac {\partial^ {2} V}{\partial x ^ {2}} (x; \pi)\right) - \beta V (x; \pi) + \gamma p (x, a, \theta)\right) d a \\ = - \beta V (x; \pi) + \tilde {b} (x, \pi) \cdot \nabla V (x; \pi) + \frac {1}{2} \tilde {\sigma} ^ {2} (x, \pi) \circ \nabla^ {2} V (x; \pi) + \tilde {r} (x, \pi) + \gamma \tilde {p} (x, \pi) \\ = 0. \\ \end{array}
$$
$$
\begin{array}{l} \eta (\theta + \delta \theta) - \eta (\theta) = \langle d _ {\mu} ^ {\theta + \delta \theta}, f (\delta \theta) \rangle \\ = \left\langle d _ {\mu} ^ {\theta + \delta \theta}, f (\delta \theta) \right\rangle - \left\langle d _ {\mu} ^ {\theta + \delta \theta}, f (0) \right\rangle \\ = \left\langle d _ {\mu} ^ {\theta + \delta \theta}, f (\delta \theta) - f (0) \right\rangle \\ = \langle d _ {\mu} ^ {\theta + \delta \theta} - d _ {\mu} ^ {\theta}, f (\delta \theta) - f (0) \rangle + \langle d _ {\mu} ^ {\theta}, f (\delta \theta) - f (0) \rangle . \\ \end{array}
$$
Dividing both sides by  $\delta \theta$  completes the proof, as the first term on the last line above is of higher order than  $\delta \theta$ .
B.3 Proofs of Lemma 4 and Theorem 5
We need a lemma for the perturbation bounds.
Lemma 10. Assume that both  $\tilde{\sigma}^2 (x,\hat{\pi} (\cdot))$  and  $\tilde{\sigma}^2 (x,\pi (\cdot))$  are positive definite and
$$
\tilde {\sigma} ^ {2} (x, \pi (\cdot)), \tilde {\sigma} ^ {2} (x, \hat {\pi} (\cdot)) \geq \sigma_ {0} ^ {2} \cdot I.
$$
where  $\sigma_0 > 0$ , then we have that the difference between the square root matrix is bounded by
$$
\| \tilde {\sigma} (x, \hat {\pi}) - \tilde {\sigma} (x, \pi) \| _ {2} \leq \frac {1}{2 \sigma_ {0}} \| \tilde {\sigma} ^ {2} (x, \hat {\pi}) - \tilde {\sigma} ^ {2} (x, \pi) \| _ {2}.
$$
If we also assume that the upper bounds, i.e.
$$
\tilde {\sigma} ^ {2} (x, \pi (\cdot)), \tilde {\sigma} ^ {2} (x, \hat {\pi} (\cdot)) \leq \bar {\sigma} _ {0} ^ {2} \cdot I.
$$
by some  $\bar{\sigma}_0 > \sigma_0 > 0$ , then we have
$$
\| \tilde {\sigma} (x, \hat {\pi}) - \tilde {\sigma} (x, \pi) \| _ {2} \leq \frac {\bar {\sigma} _ {0}}{2 \sigma_ {0}} \| \hat {\pi} - \pi \| _ {1} ^ {\frac {1}{2}}.
$$
Proof. Consider a normalized vector  $x$  with  $\| x \|_2 = 1$  is an eigenvector of  $A^{\frac{1}{2}} - B^{\frac{1}{2}}$  with eigenvalue  $\mu$  then
$$
\begin{array}{l} x ^ {T} (A - B) x = x ^ {T} \left(A ^ {\frac {1}{2}} - B ^ {\frac {1}{2}}\right) A ^ {\frac {1}{2}} x + x ^ {T} B ^ {\frac {1}{2}} \left(A ^ {\frac {1}{2}} - B ^ {\frac {1}{2}}\right) x \\ = \mu x ^ {T} \left(A ^ {\frac {1}{2}} + B ^ {\frac {1}{2}}\right) x. \\ \end{array}
$$
thus, if  $A,B\geq \sigma_0^2 I$  , this implies
$$
\mu \leq \frac {\left| x ^ {T} (A - B) x \right|}{x ^ {T} \left(A ^ {\frac {1}{2}} + B ^ {\frac {1}{2}}\right) x} \leq \| A - B \| _ {2} \cdot \lambda_ {\min } \left(A ^ {\frac {1}{2}} + B ^ {\frac {1}{2}}\right) ^ {- 1} \leq \| A - B \| _ {2} / \left(2 \sigma_ {0}\right).
$$
$$
\tilde {\sigma} ^ {2} (x, \hat {\pi}) - \tilde {\sigma} ^ {2} (x, \pi) = \int_ {\mathcal {A}} \sigma^ {2} (x, a) (\tilde {\pi} (a | x) - \pi (a | x)) \mathrm {d} a.
$$
$$
\| \tilde {\sigma} ^ {2} (x, \hat {\pi}) - \tilde {\sigma} ^ {2} (x, \pi) \| _ {2} \leq \bar {\sigma} _ {0} ^ {2} \int_ {\mathcal {A}} | \tilde {\pi} (a | x) - \pi (a | x) | \mathrm {d} a = \bar {\sigma} _ {0} ^ {2} \cdot \| \tilde {\pi} (a | x) - \pi (a | x) \| _ {1}.
$$
Proof (of Lemma 4). Consider the Wasserstein-2 distance  $W_{2}(\mu, v)$  between distribution  $\mu$  and  $v$  as
$$
W_{2}(\mu ,\nu) = \Bigg(\inf_{\gamma \in \Gamma (\mu ,\nu)}\mathbf{E}_{(x,y)\sim \gamma}\| x - y\|_{2}^{2}\Bigg)^{1 / 2},
$$
where  $\Gamma (\mu ,\nu)$  is the set all probability measures on the product space  $\mathbb{R}^n\times \mathbb{R}^n$  with the marginal distributions being  $\mu$  and  $v$ , and  $\| \cdot \| _2$  is the standard Euclidean distance. Denote
$$
\bar {d} _ {\mu} ^ {\pi} := \beta d _ {\mu} ^ {\pi}.
$$
We want to get an upper bound on  $W_{2}(\bar{d}_{\mu}^{\pi}, \bar{d}_{\mu}^{\tilde{\pi}})$  in terms of the distance between two policies  $\pi$  and  $\hat{\pi}$ . Consider a specific coupling  $(X_{t}, Y_{t})$  below:
$$
\left\{ \begin{array}{l} \mathrm {d} X _ {s} = \tilde {b} \left(X _ {s}, \pi (\cdot | X _ {s})\right) \mathrm {d} s + \tilde {\sigma} \left(X _ {s}, \pi (\cdot | X _ {s})\right) \mathrm {d} B _ {s}, \\ \mathrm {d} Y _ {s} = \tilde {b} \left(Y _ {s}, \hat {\pi} (\cdot | Y _ {s})\right) \mathrm {d} s + \tilde {\sigma} \left(Y _ {s}, \hat {\pi} (\cdot | Y _ {s})\right) \mathrm {d} B _ {s}. \end{array} \right. \tag {50}
$$
with  $X_0 = Y_0$ , which leads to a joint distribution over  $\mathbb{R}^n\times \mathbb{R}^n$ :
$$
\tilde {\gamma} := \left\{\tilde {p} (x, y) = \int_ {0} ^ {\infty} \frac {1}{\beta} e ^ {- \beta t} f _ {(X _ {t}, Y _ {t})} (x, y) \mathrm {d} t \right\}.
$$
$$
W _ {2} ^ {2} \left(\bar {d} _ {\mu} ^ {\pi}, \bar {d} _ {\mu} ^ {\bar {\pi}}\right) \leq \mathbb {E} _ {(x, y) \sim \hat {\gamma}} \| x - y \| _ {2} ^ {2} = \int_ {0} ^ {\infty} \frac {1}{\beta} e ^ {- \beta s} \mathbb {E} \| X _ {s} - Y _ {s} \| _ {2} ^ {2} d s. \tag {51}
$$
It then boils down to estimating  $\mathbb{E}\| X_s - Y_s\| _2^2$  . By Ito's formula,
$$
\begin{array}{l} \mathrm {d} \| X _ {s} - Y _ {s} \| _ {2} ^ {2} = 2 \left(X _ {s} - Y _ {s}\right) ^ {\top} \left[ \left(\tilde {b} \left(X _ {s}, \pi\right) - \tilde {b} \left(Y _ {s}, \hat {\pi}\right)\right) \mathrm {d} s + \left(\tilde {\sigma} \left(X _ {s}, \pi\right) - \tilde {\sigma} \left(Y _ {s}, \hat {\pi}\right)\right) \mathrm {d} B _ {s} \right] \\ + \operatorname {T r} \left[ \left(\tilde {\sigma} \left(X _ {s}, \pi\right) - \tilde {\sigma} \left(Y _ {s}, \hat {\pi}\right)\right) ^ {2} \right] \mathrm {d} s. \\ \end{array}
$$
Taking expectation on both sides yields
$$
\frac {\mathrm {d}}{\mathrm {d} s} \mathbb {E} \| X _ {s} - Y _ {s} \| _ {2} ^ {2} = 2 \underbrace {\mathbb {E} \left[ \left(X _ {s} - Y _ {s}\right) ^ {\top} \left(\tilde {b} \left(X _ {s} , \pi\right) - \tilde {b} \left(Y _ {s} , \hat {\pi}\right)\right) \mathrm {d} s \right]} _ {(A)} + \underbrace {\operatorname {T r} \left[ \mathbb {E} \left(\tilde {\sigma} \left(X _ {s} , \pi\right) - \tilde {\sigma} \left(Y _ {s} , \hat {\pi}\right)\right) ^ {2} \right]} _ {(B)}, \tag {52}
$$
$$
\begin{array}{l} (\mathrm {A}) = \mathbb {E} \left[ \left(X _ {s} - Y _ {s}\right) ^ {\top} \left(\tilde {b} \left(X _ {s}, \pi\right) - \tilde {b} \left(Y _ {s}, \pi\right)\right) \mathrm {d} s \right] + \mathbb {E} \left[ \left(X _ {s} - Y _ {s}\right) ^ {\top} \left(\tilde {b} \left(Y _ {s}, \pi\right) - \tilde {b} \left(Y _ {s}, \hat {\pi}\right)\right) \mathrm {d} s \right] \\ \leq C _ {\tilde {b}} \cdot \mathbb {E} \| X _ {s} - Y _ {s} \| _ {2} ^ {2} + \frac {1}{2} \mathbb {E} \| X _ {s} - Y _ {s} \| _ {2} ^ {2} + \frac {1}{2} \mathbb {E} \| \tilde {b} (Y _ {s}, \pi) - \tilde {b} (Y _ {s}, \hat {\pi}) \| _ {2} ^ {2} \\ \leq \left(C _ {\tilde {b}} + \frac {1}{2}\right) \cdot \mathbb {E} \| X _ {s} - Y _ {s} \| _ {2} ^ {2} + \frac {1}{2} \| \tilde {b} (\cdot , \pi) - \tilde {b} (\cdot , \hat {\pi}) \| _ {2, \infty} ^ {2}; \\ \end{array}
$$