SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {L} _ {1, i, j} := C _ {1, i, j} \int_ {0} ^ {k h} e ^ {\left(\lambda_ {i} (k h - s) + \lambda_ {j} (l h - s)\right)} d s \\ = C _ {1, i, j} \frac {e ^ {\lambda_ {i} k h} + e ^ {\lambda_ {j}}}{\lambda_ {i} + \lambda_ {j}} \left(1 - e ^ {- (\lambda_ {i} + \lambda_ {j}) k h}\right) \\ \end{array}
$$
$$
\mathcal {L} _ {2, i, j} := C _ {2, i, j} \int_ {0} ^ {k h} e ^ {(\lambda_ {i} (k h - s) + \lambda_ {j} (l h - s))} (k h - s) ^ {\tilde {q} _ {i}} (l h - s) ^ {\tilde {q} _ {j}} d s,
$$
where  $C_{1,i,j}$ ,  $C_{i,j}$  are some constants and  $\tilde{q}_i\in \{0,\dots ,q_i - 1\}$ ,  $\tilde{q}_j\in \{0,\dots ,q_j - 1\}$ .
For the integral in  $\mathcal{L}_{2,i,j}$ , as  $\tilde{q}_i + \tilde{q}_j \leq n - 1$ , we can have the inequality:
$$
\begin{array}{l} \int_ {0} ^ {k h} e ^ {\left(\lambda_ {i} (k h - s) + \lambda_ {j} (l h - s)\right)} (k h - s) ^ {\tilde {q} _ {i}} (l h - s) ^ {\tilde {q} _ {j}} d s \\ \leq T ^ {n - 1} \int_ {0} ^ {k h} e ^ {\left(\lambda_ {i} (k h - s) + \lambda_ {j} (l h - s)\right)} d s. \tag {40} \\ \end{array}
$$
$$
\operatorname {t r} \left\{Q \mathbb {E} \left[ X (k h) X (l h) ^ {\top} \right] \right\} ^ {2} = \sum_ {i _ {1}, j _ {1}, i _ {2}, j _ {2}} \sum_ {k, l} \prod_ {l _ {1}, l _ {2} \in \{1, 2 \}} \mathcal {L} _ {l _ {1}, i _ {1}, j _ {1}} \mathcal {L} _ {l _ {2}, i _ {2}, j _ {2}},
$$
and all the terms are nonnegative. We drop all terms that include  $\mathcal{L}_{2,i,j}$  factor and only include the  $\mathcal{L}_{1,i,i}^{2}$  with  $k = l$  terms in the lower bound of part 1. That is to say, the lower bound of part 1 is  $\sum_{i=1}^{m} q_i \bar{C}_i \mathrm{Var}(h, \lambda_i)$ .
The upper bound of part 1 can be obtained by replacing all  $\mathcal{L}_{1,i,j}$  factors by  $\mathcal{L}_{2,i,j}$  and use the bound given in Equation (40). This leads to the upper bound for part 1 is  $\frac{\left(\bar{C}_2 + C_3(\lambda_1,\cdots,\lambda_m,T)\mathcal{O}(T)\right)\sigma^4 T^{2n + 5}}{Bh} +\mathcal{O}\left(\frac{1}{B}\right)$ .
For Part 2, let  $g(t) = \mathbb{E}\left[X(t)^{\top}QX(t)\right]$  on  $[0,T]$ . Then  $\mathbb{E}\left[\hat{V}_M\right]$  is the left Riemann sum approximation of  $g(t)$ , by the property of Riemann approximation,
$$
\left| \mathbb {E} \left[ \hat {V} _ {M} \right] - V _ {T} \right| \approx 2 h T g (T) + \mathcal {O} \left(h ^ {2}\right),
$$
$$
g (T) = \operatorname {t r} \left\{Q \mathbb {E} \left[ X (T) X (T) ^ {\top} \right] \right\} = \sigma^ {2} \operatorname {t r} \left\{Q \int_ {0} ^ {T} e ^ {A (t - s)} e ^ {A ^ {\top} (t - s)} d s \right\},
$$
which is a constant depends on  $\lambda_1, \dots, \lambda_m, T$ . Thus
$$
\left(\mathbb {E} \left[ \hat {V} _ {M} \right] - V _ {T}\right) ^ {2} \approx C _ {1} \left(\lambda_ {1}, \dots , \lambda_ {m}, T\right) \sigma^ {4} T ^ {2} h ^ {2} + \mathcal {O} \left(h ^ {3}\right),
$$
which has the same order in  $h$  as the scalar case in finite-horizon undiscounted setting. Thus the result in Equation (35) is obtained by combining part 1 bounds and part 2 approximation.
As we explained in the beginning of this proof, in the finite-horizon discounted setting, we will follow the similar arguments as in the proof of Equation (35) to obtain result Equation (36).
For the infinite-horizon discounted setting, the corresponding part 1 in the  $\mathrm{MSE}_{\infty}$  is the same as the part 1 in  $\mathrm{MSE}_T$  of Equation (36). The part 2 is approximated by using the decomposition Equation (12) and the fact that
$$
V _ {t, \infty} = \int_ {T} ^ {\infty} \gamma^ {t} \mathbb {E} [ X (t) ^ {\top} Q X (t) ] d t = \gamma^ {T} C (r, T, \lambda_ {1}, \dots , \lambda_ {m}).
$$
To verify  $V_{T,\infty}$  is  $\mathcal{O}(\gamma^T)$ , one can find the bounds of  $V_{T,\infty}$  by using the similar arguments in the above proof of Equation (35) and the following inequality:
$$
\begin{array}{l} \int_ {0} ^ {t} e ^ {(\lambda_ {i} + \lambda_ {j}) (t - s)} (t - s) ^ {\tilde {q} _ {i} + \tilde {q} _ {j}} d s \\ \leq t ^ {n - 1} \int_ {0} ^ {t} e ^ {(\lambda_ {i} + \lambda_ {j}) (t - s)} \mathrm {d} s = \frac {t ^ {n - 1}}{(\lambda_ {i} + \lambda_ {j})} \left(e ^ {(\lambda_ {i} + \lambda_ {j}) t} - 1\right). \\ \end{array}
$$
Then the components in  $\int_{T}^{\infty}\gamma^{t}\mathbb{E}\left[X(t)X(t)^{\top}\right]\mathrm{d}t$  is lower bounded by  $\int_{T}^{\infty}\frac{r^{t}}{(\lambda_{i} + \lambda_{j})}\left(e^{(\lambda_{i} + \lambda_{j})t} - 1\right)\mathrm{d}t$  and upper bounded by  $\int_{T}^{\infty}\frac{r^{t}t^{n - 1}}{(\lambda_{i} + \lambda_{j})}\left(e^{(\lambda_{i} + \lambda_{j})t} - 1\right)\mathrm{d}t.$  By the celebrated approximation of incomplete gamma function when  $T$  is large, we have
$$
\int_ {T} ^ {\infty} \frac {r ^ {t} t ^ {n - 1}}{(\lambda_ {i} + \lambda_ {j})} \left(e ^ {(\lambda_ {i} + \lambda_ {j}) t} - 1\right) d t \approx \frac {r ^ {T} T ^ {n - 1}}{(\lambda_ {i} + \lambda_ {j})} \left(e ^ {(\lambda_ {i} + \lambda_ {j}) T} - 1\right).
$$
Followed by
$$
V _ {T, \infty} = \operatorname {t r} \left\{Q \int_ {T} ^ {\infty} \gamma^ {t} \mathbb {E} [ X (t) X (t) ^ {\top} ]   \mathrm {d} t \right\},
$$
we can obtain that  $V_{T,\infty} = \gamma^T C(r,T,\lambda_1,\dots ,\lambda_m)$
This result leads to the fact that part 2 is
$$
\begin{array}{l} \left(C _ {8} \left(\lambda_ {1}, \dots , \lambda_ {m}, \gamma , T\right) + C _ {9} \left(\lambda_ {1}, \dots , \lambda_ {m}, \gamma , T\right) h\right) \sigma^ {4} \gamma^ {2 T} + \left(C _ {1 0} \left(\lambda_ {1}, \dots , \lambda_ {m}, \gamma , T\right) h ^ {2} \right. \\ \left. + C _ {1 1} (\lambda_ {1}, \dots , \lambda_ {m}, \gamma , T) h ^ {3}\right) \sigma^ {4} \gamma^ {T} + C _ {1 2} (\lambda_ {1}, \dots , \lambda_ {m}, T) \sigma^ {4} h ^ {4} + \mathcal {O} (h ^ {5}), \\ \end{array}
$$
which coincides with the  $\operatorname{Var}(h\lambda_i)$  in the infinite-horizon discounted scalar case. The results in (3) then follow.
D Complements to Numerical Simulations
The LQR experiments were run on a MacBook pro with an i9 CPU and 16GB of RAM.
D.1 Randomly-sampled Matrices
The matrices  $A$  in Figure 1(c) and Figure 1(d), corresponding to controlled multi-dimensional linear systems, are generated according to the procedure described below. It ensures the stability of the resulting system, i.e., that eigenvalues of the sampled matrix are negative.
The procedure works by random sampling an eigendecomposition. It starts with uniformly sampling two eigenvalues from disjoint, bounded intervals,  $\lambda_1 \in [-1.5, -1.0)$  and  $\lambda_3 \in (-1.0, -0.75]$ . The final eigenvalue is set to be  $\lambda_2 = -1.0$ . Note that since all the eigenvalues sampled are negative, any matrix whose eigenvalues are  $\lambda_1, \lambda_2, \lambda_3$  is said to be stable. Next, the eigenvectors are sampled randomly from the classical compact groups detailed in [Mezzadri, 2006]. For that, we randomly sample an orthogonal matrix  $L$  using a built-in ScIPY [Virtanen et al., 2020] routine, ORTHO_GROUP.RVS. Now let  $\Lambda = \mathrm{diag}(\lambda_1, \lambda_2, \lambda_3)$ , the random, dense, stable matrix  $A$  is obtained by computing the product  $A = L^\top \Lambda L$ .
D.2 Trade-off in LQR with Scaled Identity Matrices
In order to better understand the transition from scalar to vector case in the trade-off of the step-size due to the MSE in Section 3, numerical experiments for the case of identity matrices scaled by a constant are provided in this section. This allows to characterise the role played by the eigenvalues, with respect to the parameter  $a$  in the scalar case.
As expected, results in Figure 4 suggest that the trade-off for scaled identity matrices is very similar to the one in the scalar case. In this simple case the eigenvalues that are identical on all dimensions play the same role as the parameter  $a$ , i.e., by decreasing them, the trade-off shifts towards a smaller value for the optimal step-size, as suggested also by Figure 1(b).
D.3 Comparison of Empirical and Analytical MSEs in One-dimensional Langevin Systems
Figure 5 compares the empirical MSE in Figures 1(a) and 1(b) with the analytical MSEs, including both the exact and approximate versions. We can observe that the empirical results are consistent with the analysis. The error in approximate MSE becomes noticeable for large  $h$  especially when  $h > 1$ , as expected. Nevertheless, the optimal  $h^*$  remains consistent.
D.4 Implementation Details for Nonlinear Systems Experiments
We summarize the environment-specific parameters in Table 1 for the nonlinear-system experiments.
Compute Resources For training the stable policy for non-mujoco environments, we used a server with one GTX 1080. The training for MuJoCo environments was conducted on a cluster, using a single V100 Volta for each environment. For inference, since we need to run many episodes, we scaled things up on a cluster of CPUs.
Training Details We used the CDAU algorithm described in [Tallec et al., 2019] to train the policies since we find this algorithm to be robust to time discretization, especially when the environment runs at  $\delta t = 0.001$ . We closely followed the hyper-parameters setup described in Section 2 in the Appendix of [Tallec et al., 2019]. For more details, please refer to their paper, and the code in the supplementary materials.
Inference Details We run the policy for  $300k$  episodes at the finest time discretization  $\delta t = 0.001$  (600k in InvertedDoublePendulum and Pusher) and store the reward sequences. The number of episodes is chosen to be sufficient for 30 runs and the largest possible number of trajectories (depending on  $h$  and the data budget  $B$ ). These data get down-sampled offline for different choices of  $h$ . The episodes are randomly shuffled when we vary  $B$ .