SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} (\mathbf {B}) = \mathbb {E} \| \tilde {\sigma} (X _ {s}, \pi) - \tilde {\sigma} (Y _ {s}, \hat {\pi}) \| _ {F} ^ {2} \\ \leq 2 \mathbb {E} \| \tilde {\sigma} (X _ {s}, \pi) - \tilde {\sigma} (Y _ {s}, \pi) \| _ {F} ^ {2} + 2 \mathbb {E} \| \tilde {\sigma} (Y _ {s}, \pi) - \tilde {\sigma} (Y _ {s}, \hat {\pi}) \| _ {F} ^ {2} \\ \leq 2 C _ {\tilde {\sigma}} ^ {2} \cdot \mathbb {E} \| X _ {s} - Y _ {s} \| _ {2} ^ {2} + 2 \sup  _ {x} \| \tilde {\sigma} (x, \pi) - \tilde {\sigma} (x, \hat {\pi}) \| _ {F} ^ {2} \\ := 2 C _ {\tilde {\sigma}} ^ {2} \cdot \mathbb {E} \| X _ {s} - Y _ {s} \| _ {2} ^ {2} + 2 \| \tilde {\sigma} (\cdot , \pi) - \tilde {\sigma} (\cdot , \hat {\pi}) \| _ {F, \infty} ^ {2}. \\ \end{array}
$$
Combining the above, we get
$$
\frac{\mathrm{d}}{\mathrm{d}s}\mathbb{E}\| X_{s} - Y_{s}\|_{2}^{2}\leq \underbrace{(2C_{\tilde{b}} + 1 + 2C_{\tilde{\sigma}}^{2})}_{C_{\tilde{b},\tilde{\sigma}}}\mathbb{E}\| X_{s} - Y_{s}\|_{2}^{2} + \underbrace{\|\tilde{b}(\cdot,\pi) - \tilde{b}(\cdot,\hat{\pi})\|_{2,\infty}^{2} + 2\|\tilde{\sigma} (\cdot,\pi) - \tilde{\sigma} (\cdot,\hat{\pi})\|_{F,\infty}^{2}}_{C(\pi ,\hat{\pi})}.
$$
By Gronwall's inequality, we have
$$
\mathbb {E} \| X _ {t} - Y _ {t} \| _ {2} ^ {2} \leq \frac {C (\pi , \hat {\pi})}{C _ {\tilde {b} , \tilde {\sigma}}} \left(e ^ {C _ {\tilde {b}, \tilde {\sigma}} t} - 1\right). \tag {53}
$$
Substituting back into (51), we obtain
$$
W _ {2} ^ {2} (\bar {d} _ {\mu} ^ {\pi}, \bar {d} _ {\mu} ^ {\hat {\pi}}) \leq \frac {C (\pi , \hat {\pi})}{C _ {\tilde {b} , \tilde {\sigma}}} \int_ {0} ^ {\infty} \frac {1}{\beta} e ^ {- \beta s} \left(e ^ {C _ {\tilde {b}, \tilde {\sigma}} s} - 1\right) \mathrm {d} s.
$$
Thus, if  $\beta > C_{\tilde{b},\tilde{\sigma}}$ , we have
$$
W _ {2} (\bar {d} _ {\mu} ^ {\pi}, \bar {d} _ {\mu} ^ {\hat {\pi}}) \leq \frac {C (\pi , \hat {\pi})}{C _ {\tilde {b} , \tilde {\sigma}} (\beta - C _ {\tilde {b} , \tilde {\sigma}}) \beta}.
$$
Concerning the term  $C(\pi, \hat{\pi})$ , we have
$$
\| \tilde {b} (\cdot , \pi) - \tilde {b} (\cdot , \hat {\pi}) \| _ {2, \infty} = \sup _ {x} \| \tilde {b} (x, \pi) - \tilde {b} (x, \hat {\pi}) \| _ {2} \leq \sup _ {x} \| \hat {\pi} (\cdot | x) - \pi (\cdot | x) \| _ {1} \cdot \sup _ {x, a} | b (x, a) |,
$$
$$
\| \tilde {\sigma} (\cdot , \pi) - \tilde {\sigma} (\cdot , \hat {\pi}) \| _ {F, \infty} = \sup  _ {x} \| \tilde {\sigma} (x, \pi) - \tilde {\sigma} (x, \hat {\pi}) \| _ {F} \leq \sqrt {n} \frac {\bar {\sigma} _ {0}}{2 \sigma_ {0}} \sup  _ {x} \| \hat {\pi} (\cdot | x) - \pi (\cdot | x) \| _ {1} ^ {\frac {1}{2}}.
$$
$$
\begin{array}{l} C (\pi , \hat {\pi}) = \| \tilde {b} (\cdot , \pi) - \tilde {b} (\cdot , \hat {\pi}) \| _ {2, \infty} ^ {2} + 2 \| \tilde {\sigma} (\cdot , \pi) - \tilde {\sigma} (\cdot , \hat {\pi}) \| _ {F, \infty} ^ {2} \\ \leq \left(\sup  _ {x, a} | b (x, a) | ^ {2} + \frac {d \cdot \bar {\sigma} _ {0} ^ {2}}{2 \sigma_ {0} ^ {2}}\right) \max  \left(\sup  _ {x} \| \hat {\pi} (\cdot | x) - \pi (\cdot | x) \| _ {1}, \sup  _ {x} \| \hat {\pi} (\cdot | x) - \pi (\cdot | x) \| _ {1} ^ {\frac {1}{2}}\right) \\ \end{array}
$$
which proves our upper bound.
Proof (of Theorem 5). We have that
$$
\begin{array}{l} \left| \eta^ {\hat {\pi}} - L ^ {\pi} (\hat {\pi}) \right| = \left| \left\langle d _ {\mu} ^ {\hat {\pi}} - d _ {\mu} ^ {\pi}, f \right\rangle \right| = \frac {\| f \| _ {\dot {H} ^ {1}}}{\beta} \left| \left\langle \bar {d} _ {\mu} ^ {\hat {\pi}} - \bar {d} _ {\mu} ^ {\pi}, \frac {f}{\| f \| _ {\dot {H} ^ {1}}} \right\rangle \right| \tag {54} \\ \leq \frac {K}{\beta} \| \bar {d} _ {\mu} ^ {\hat {\pi}} - \bar {d} _ {\mu} ^ {\pi} \| _ {\dot {H} ^ {- 1}} \leq \frac {K \sqrt {M}}{\beta} W _ {2} \left(\bar {d} _ {\mu} ^ {\hat {\pi}}, \bar {d} _ {\mu} ^ {\pi}\right). \\ \end{array}
$$
where  $K \coloneqq \sup_{\hat{\pi}} \| f \|_{\dot{H}^1} < \infty$  (more about  $K$  in the remarks below). Combining (54) with the estimate in (22) (of Lemma 4) yields the desired result in (23).
Remarks (on  $K$ ). In the performance-difference bound developed above, we assume  $K$  is finite:
$$
K := \| f \| _ {\dot {H} ^ {1}} := \left(\int_ {\mathbb {R} ^ {n}} | \nabla f (x) | ^ {2} \mathrm {d} x\right) ^ {\frac {1}{2}} <   \infty ,
$$
where  $f(x; \pi, \hat{\pi}) \coloneqq \int_{A} \hat{\pi}(a \mid x) (q(x, a; \pi) + p(x, a, \hat{\pi})) \, \mathrm{d}a$ . The famous Poincaré inequality can provide a lower bound on this quantity; but we need an upper bound as well, i.e.,
$$
K = \left(\int_ {\mathbb {R} ^ {n}} | \nabla f (x) | ^ {2} \mathrm {d} x\right) ^ {\frac {1}{2}} \leq C \left(\int_ {\mathbb {R} ^ {n}} | f (x) | ^ {2} \mathrm {d} x\right) ^ {\frac {1}{2}}.
$$
This above is essentially a reverse Poincaré Inequality, which is not likely to hold (in particular, the existence of the constant  $C$ ).
Should we indeed have a reverse Poincaré Inequality, then we can further bound  $f$  by
$$
\begin{array}{l} | f (x) | = \left| \int_ {\mathcal {A}} \left(\hat {\pi} (a \mid x) - \pi (a \mid x)\right) \left(q (x, a; \pi) + p (x, a, \hat {\pi})\right) d a \right| \\ \leq \int_ {\mathcal {A}} | \hat {\pi} (a \mid x) - \pi (a \mid x) | \cdot | q (x, a; \pi) + p (x, a, \hat {\pi}) | \mathrm {d} a \\ \leq 2 \sup  _ {a} | q (x, a; \pi) + p (x, a, \hat {\pi}) | D _ {\mathrm {T V}} (\pi (\cdot | x), \hat {\pi} (\cdot | x)), \\ \end{array}
$$
$$
\begin{array}{l} \left(\int_ {\mathbb {R} ^ {n}} | f (x) | ^ {2} \mathrm {d} x\right) ^ {\frac {1}{2}} \leq \left(\int_ {\mathbb {R} ^ {n}} 4 \sup _ {a} | q (x, a; \pi) + p (x, a, \hat {\pi}) | ^ {2} D _ {\mathrm {T V}} ^ {2} (\pi (\cdot | x), \hat {\pi} (\cdot | x)) \mathrm {d} x\right) ^ {\frac {1}{2}} \\ \leq \left(\int_ {\mathbb {R} ^ {n}} 2 \sup  _ {a} | q (x, a; \pi) + p (x, a, \hat {\pi}) | ^ {2} \mathrm {d} x\right) ^ {\frac {1}{2}} \sqrt {\sup  _ {x} D _ {\mathrm {K L}} (\pi (\cdot | x) , \hat {\pi} (\cdot | x))}, \\ \end{array}
$$
where the second inequality is from Pinsker's inequality. This way, we would have recovered a similar bound as in the discrete RL. Since we do not have the reverse Poincaré inequality, however, we have to assume that  $K$  is finite.
Appendix C Algorithms
C.1 Performance of CPPO with Square-root KL and Linear KL
Here we present a detailed version of the CPPO algorithm. For two probability distributions  $P$  and  $Q$  over the action space with density functions  $p$  and  $q$  correspondingly, recall that the KL-divergence between these two is defined as:
$$
D _ {\mathrm {K L}} (P \| Q) = \int_ {\mathcal {A}} \log \left(\frac {q (a)}{p (a)}\right) q (a) \mathrm {d} a,
$$
Denote  $D_{\mathrm{KL}}(\theta \| \theta_k) := \mathbb{E}_{x \sim d_\mu^{\theta_k}} D_{\mathrm{KL}}(\pi_{\theta_k}(\cdot | x) \| \pi_\theta(\cdot | x))$ , to distinguish it from  $\bar{D}_{\mathrm{KL}}(\theta \| \theta_k) := \mathbb{E}_{x \sim d_\mu^{\theta_k}} \sqrt{D_{\mathrm{KL}}(\pi_{\theta_k}(\cdot | x) \| \pi_\theta(\cdot | x))}$  which was used in CPPO Algorithm in 2.
Note that bounding the performance difference by the linear KL-divergence  $D_{\mathrm{KL}}(\theta, \theta_k)$ , instead of its square-root counterpart  $\bar{D}_{\mathrm{KL}}(\theta \| \theta_k)$ , will generally require stronger conditions (which may be difficult to satisfy). For completeness, we present the following algorithm as an important benchmark, the CPPO with linear KL-divergence:
Algorithm 3 CPPO: PPO with adaptive penalty constant (linear KL-divergence)
Input: Policy parameters  $\theta_0$ , critic net parameters  $\phi_0$   
1: for  $k = 0,1,2,\dots$  until  $\theta_{k}$  converge do  
2: Collect a truncated trajectory  $\{X_{t_i},a_{t_i},r_{t_i},p_{t_i}\}$ ,  $i = 1,\ldots ,N$  from the environment using  $\pi_{\theta_k}$ .  
3: for  $i = 0,\ldots ,N - 1$  do: Update the critic parameters as in (8)  
4: for  $j = 1,\ldots ,J$  do: Draw i.i.d.  $\tau_{j}$  from  $\exp (\beta)$ , round  $\tau_{j}$  to the largest multiple of  $\delta_t$  no larger than it, and compute the GAE estimator of  $q(X_{\tau_j},a_{\tau_j})$ $\tilde{q} (X_{\tau_j},a_{\tau_j})\coloneqq \big(r_{\tau_j}\delta_t + e^{-\beta \delta_t}V(X_{\tau_j + \delta_t}) - V(X_{\tau_j})\big) / \delta_t$ .  
5: Compute policy update (by taking a fixed  $s$  steps of gradient descent)  
 $\theta_{k + 1} = \arg \max_{\theta}L^{\theta_k}(\theta) - C^{k}_{\mathrm{penalty}}D_{\mathrm{KL}}(\theta \| \theta_k)$ .  
6: if  $D_{\mathrm{KL}}(\theta_{k + 1}\| \theta_k)\geq (1 + \epsilon)\delta$ , then  $C^{k + 1}_{\mathrm{penalty}} = 2C^{k}_{\mathrm{penalty}}$ .  
7: else if  $D_{\mathrm{KL}}(\theta_{k + 1}\| \theta_k)\leq \delta /(1 + \epsilon)$ , then  $C^{k + 1}_{\mathrm{penalty}} = C^{k}_{\mathrm{penalty}} / 2$ .
A comparison between the above and Algorithm 2 (using square-root KL divergence) is presented in §D.3 below, which clearly illustrates the advantage of square-root KL divergence.
C.2 KL-divergence
We elaborate here on the KL-divergence between the current policy and the optimal policy, along with the entropy regularizer. By the performance difference formula, we have
$$
\eta (\pi) - \eta (\pi^ {*}) = \int_ {\mathbb {R} ^ {n}} d _ {\mu} ^ {\pi} (x) \left[ \int_ {\mathcal {A}} \pi (a \mid x) \left(q (x, a; \pi^ {*}) - \gamma \log (\pi (a))\right) \mathrm {d} a \right] \mathrm {d} x.
$$
Notice that by the definition of KL-divergence we defined before, we have
$$
D _ {\mathrm {K L}} \left(\pi^ {*} (\cdot | x) \| \pi (\cdot | x)\right) = \int_ {\mathcal {A}} \log \left(\frac {\pi (a | x)}{\pi^ {*} (a | x)}\right) \pi (a | x) \mathrm {d} a.
$$
Similar as the previous discussion of soft  $q$ -learning,  $\pi^{*}$  is optimal implies that
$$
\pi^ {*} (a \mid x) \propto \exp (\frac {q (x , a , \pi^ {*})}{\gamma}),
$$
and the normalization constant is 1 can be proved through considering the exploratory HJB equation, see [22, 56]. Thus
$$
D _ {\mathrm {K L}} (\pi^ {*} (\cdot | x) \| \pi (\cdot | x)) = \int_ {\mathcal {A}} \log (\pi (a | x)) \pi (a | x) \mathrm {d} a - \int_ {\mathcal {A}} \frac {q (x , a , \pi^ {*})}{\gamma} \pi (a | x) \mathrm {d} a,
$$
$$
\eta (\pi) - \eta (\pi^ {*}) = - \gamma \cdot \mathbb {E} _ {x \sim d _ {\mu} ^ {\pi}} D _ {\mathrm {K L}} (\pi^ {*} (\cdot | x) \| \pi (\cdot | x)).
$$
This justifies our claim that the KL-divergence is essentially equivalent to the distance to the optimal performance.
Appendix D Experiments
D.1 Example 1
Recall, in the LQ control problem, the reward function is
$$
r (x, a) = - \left(\frac {M}{2} x ^ {2} + R x a + \frac {N}{2} a ^ {2} + P x + Q a\right),
$$
with  $M \geq 0, N > 0, R, Q, P \in \mathbb{R}$  and  $R^2 < MN$ , and we adopt the entropy regularizer as
$$
p (x, a, \pi) = - \log (\pi (a)).
$$
Furthermore, suppose that the discount rate satisfies  $\beta > 2A + C^2 + \max \left(\frac{D^2R^2 - 2NR(B + CD)}{N}, 0\right)$ .
The following results are readily derived from Theorem 4 of [58]. The value function of the optimal policy  $\pi^{*}$  is
$$
V (x) = \frac {1}{2} k _ {2} x ^ {2} + k _ {1} x + k _ {0}, \quad x \in \mathbb {R},
$$
$$
\begin{array}{l} k _ {2} := \frac {1}{2} \frac {\left(\rho - (2 A + C ^ {2})\right) N + 2 (B + C D) R - D ^ {2} M}{(B + C D) ^ {2} + (\rho - (2 A + C ^ {2})) D ^ {2}} \\ - \frac {1}{2} \frac {\sqrt {\left(\left(\rho - (2 A + C ^ {2})\right) N + 2 (B + C D) R - D ^ {2} M\right) ^ {2} - 4 \left((B + C D) ^ {2} + (\rho - (2 A + C ^ {2})) D ^ {2}\right) \left(R ^ {2} - M N\right)}}{(B + C D) ^ {2} + (\rho - (2 A + C ^ {2})) D ^ {2}}, \\ k _ {1} := \frac {P (N - k _ {2} D ^ {2}) - Q R}{k _ {2} B (B + C D) + (A - \rho) (N - k _ {2} D ^ {2}) - B R}, \\ \end{array}
$$
$$
k _ {0} := \frac {\left(k _ {1} B - Q\right) ^ {2}}{2 \rho \left(N - k _ {2} D ^ {2}\right)} + \frac {\gamma}{2 \rho} \left(\ln \left(\frac {2 \pi e \gamma}{N - k _ {2} D ^ {2}}\right) - 1\right)
$$
respectively. Moreover, the optimal feedback control is Gaussian, with density function
$$
\pi^ {*} (a; x) = \mathcal {N} \left(a \mid \frac {(k _ {2} (B + C D) - R) x + k _ {1} B - Q}{N - k _ {2} D ^ {2}}, \frac {\gamma}{N - k _ {2} D ^ {2}}\right).
$$
For a set of model parameters:  $A = -1, B = C = 0, D = 1, M = N = Q = 2, R = P = 1, \beta = 1, \gamma = 0.1$ , following the formulas and the parameterized policy  $\pi_{\theta}(\cdot \mid x) = \mathcal{N}(\theta_1x + \theta_2,\exp (\theta_3))$  and the corresponding value function  $V_{\phi}(x) = \frac{1}{2}\phi_{2}x^{2} + \phi_{1}x + \phi_{0}$ , we can derive the optimal parameters:
$$
\phi^ {*} = [ 0. 7 1 9 1 4 8 7 4, - 0. 1 0 5 5 5 1 2 8, - 0. 5 3 5 1 8 3 7 6 ],
$$
$$
\theta^ {*} = \left[ - 0. 3 9 4 4 4 8 7 2, - 0. 7 8 8 8 9 7 4 5, - 1. 4 0 4 0 0 9 4 4 \right].
$$
Table 1: Hyper-parameter values for Example 1
Alphabet	Description	Value
T	Trajectory Truncation Length	25
β	discount factor	1
δt	time interval	0.005
J	batch size for sampling exp(β)	100
α1	learning rate for policy iteration k	0.02 when k ≤ 50 and 0.02 × log(50/k) when k > 50
α2	learning rate for value iteration k	0.01 when k ≤ 50 and 0.01 × log(50/k) when k > 50
K	iteration threshold	2000
s	steps of gradient descent	10
δ	radius	0.0002
ε	tolerance level	0.5
D.2 Example 2
The model parameters are  $k = 0.01, \theta = 7, \eta = 0.1, \rho = 0.3, \sigma = 1, r_f = 0.01, \ell = 5$ . For both the value function and the policy parameterization, we use a 3-layer neural network, and with the initial parameters sampled from the uniform distribution over [-0.5, 0.5]. We use the tanh activation function for the hidden layer.
Table 2: Hyperparameter values for Example 2
Alphabet	Description	Value
T	Trajectory Truncation Length	25
β	discount factor	1
δt	time interval	0.005
J	batch size for sampling exp(β)	100
α1	learning rate for policy iteration k	0.005 when k ≤ 50 and 0.005 × log(50/k) when k > 50
α2	learning rate for value iteration k	0.01 when k ≤ 50 and 0.01 × log(50/k) when k > 50
K	iteration threshold	200
s	steps of gradient descent	10
δ	radius	0.025
ε	tolerance level	0.5
D.3 Performance of CPPO with Square-root KL and Linear KL
We compare the performance of CPPO with square-root KL-divergence (denote as CPPO), and linear KL-divergence (denoted as CPPO (nst) — non square-root) applied to the experiments in Example 1 and Example 2. Figure 4 compares the distance between the current policy parameters and the optimal parameters, with  $x$ -axis denoting the iteration times and  $y$ -axis denoting the  $L_{2}$  distance. Figure 5 compares the current expected return, with  $x$ -axis denoting the iteration times and  $y$ -axis denoting the current performance by taking the average of 100 times of Monte Carlo evaluation. In
Figure 4: Performance of CPPO and CPPO (nst) to the Example 1
Figure 5: Performance of CPPO and CPPO (nst) to the Example 2
both figures, the blue curve represents the algorithm with square-root KL-divergence as opposed to the orange one corresponding to the linear version. Both figures clearly demonstrate the advantage of the former. In particular, the linear version can suffer from getting stuck at the local optimum as demonstrated in Example 1.
D.4 Performance of CPG and CPPO compared to the classical discrete-time algorithms
We conduct experiments to compare the CPG and CPPO to their discrete counterparts. Specifically, we discretize the MDP in Example 1, and implement the classical PG and PPO algorithms. Our results show that in time discretization with step size  $\delta t = 0.1$  and  $\delta t = 0.05$ , the performance of CPG and CPPO is (at least) comparable to their discrete counterparts; in particular, for  $\delta t = 0.1$ , CPPO outperforms PG. We have repeated the experiments for 25 random seeds, and plotted both the average performance line and the error bar. These experimental results indicate that the continuous approach has the potential to outperform their discrete counterparts, which is worth further exploring in the future.
Figure 6: CPG in  $l_{2}$  distance  $(\delta_{t} = 0.05)$
Figure 7: CPG in KL distance  $(\delta_t = 0.05)$
Figure 8: DPG in  $l_{2}$  distance  $(\delta_t = 0.05)$
Figure 9: DPG in KL distance  $(\delta_t = 0.05)$
Figure 10: CPG in  $l_{2}$  distance  $(\delta_t = 0.1)$
Figure 11: CPG in KL distance  $(\delta_t = 0.1)$
Figure 12: DPG in  $l_{2}$  distance  $(\delta_t = 0.1)$
Figure 13: DPG in KL distance  $(\delta_t = 0.1)$
Figure 14: CPPO in  $l_{2}$  distance  $(\delta_{t} = 0.05)$
Figure 15: CPPO in KL distance  $(\delta_t = 0.05)$
Figure 16: DPPO in  $l_{2}$  distance  $(\delta_t = 0.05)$
Figure 17: DPPO in KL distance  $(\delta_t = 0.05)$
Figure 18: CPPO in  $l_{2}$  distance  $(\delta_t = 0.1)$
Figure 19: CPPO in KL distance  $(\delta_t = 0.1)$
Figure 20: DPPO in  $l_{2}$  distance  $(\delta_t = 0.1)$
Figure 21: DPPO in KL distance  $(\delta_t = 0.1)$