SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The main characteristic of the finite-horizon setting is the trade-off between approximation and estimation error. Recall that in the infinite-horizon setting  $(\tau = \infty)$ , the estimation horizon  $T < \infty$  becomes a free variable that is chosen by the experimenter to define the measurement range  $[0, T]$ . Consequently the mean-squared error of the Monte-Carlo estimator suffers an additional truncation error from using a finite Riemann sum with  $N = T / h$  terms as an approximation to the infinite integral that defines the cost  $V_{\infty}$ . More precisely, we decompose the expected cost  $V_{\infty} = V_{T} + V_{T,\infty}$  where  $V_{T} = \int_{0}^{T} \gamma^{t} \mathbb{E}[x^{2}(t)] dt$  as before, and
$$
V _ {T, \infty} = \int_ {T} ^ {\infty} \gamma^ {t} \mathbb {E} [ x ^ {2} (t) ] \mathrm {d} t = \frac {\sigma^ {2} \gamma^ {T}}{2 a} \left(\frac {1}{\log (\gamma)} - \frac {e ^ {2 a T}}{\log (\gamma) + 2 a}\right). \tag {11}
$$
It is a direct calculation based on Lemma A.1 in Appendix. Thus the mean-squared error becomes
$$
\mathrm {M S E} _ {\infty} (h, B, T) = \mathbb {E} \left[ (\hat {V} _ {M} (h) - V) ^ {2} \right] = \mathrm {M S E} _ {T} (h, B) - 2 \mathbb {E} \left[ \hat {V} _ {M} (h) - V _ {T} \right] V _ {T, \infty} + V _ {T, \infty} ^ {2}, \tag {12}
$$
where  $\mathrm{MSE}_T(h,B) = \mathbb{E}\big[(\hat{V}_M(h) - V_T)^2\big]$  is the mean-squared error of discounted finite-horizon setting. Note that the term  $V_{T,\infty}^{2}$  is neither controlled by a small step-size  $h$  nor by a large data budget  $B$ , hence results in the truncation error from finite estimation. Fortunately, geometric discounting ensures that  $V_{T,\infty}^{2} = \mathcal{O}(\gamma^{2T})$ , which is not unexpected given that the term constitutes the tail of the geometric integral. In particular, setting  $T = c\cdot \log (B) / \log (1 / \gamma)$  for large enough  $c > 1$  ensures the truncation error is below the estimation variance. We summarize the result in the next theorem.
Theorem 3.6 (Infinite-horizon, discounted MSE). In the infinite-horizon, discounted setting, the mean-squared error of the Monte-Carlo estimator is
$$
\mathrm {M S E} _ {\infty} (h, B, T) = \sigma^ {4} T C (a, \gamma) \cdot \frac {1}{h B} + \frac {\sigma^ {4}}{1 4 4} \cdot h ^ {4} + \mathcal {O} \left(h ^ {5}\right) + \mathcal {O} \left(B ^ {- 1}\right), \tag {13}
$$
where we let  $C(a, \gamma) = \frac{1}{\log(\gamma)(a + \log(\gamma))(2a + \log(\gamma))^2}$  and assume that  $\gamma^T = o(h^4)$ .
The proof is provided in Appendix B.5. It follows that the optimal choice for the step-size is  $h^{*}(B,T) \approx (36TC(a,\gamma) / B)^{1 / 5}$ . The minimal mean-squared error is  $\mathrm{MSE}_{\infty}(h^{*},T,B) \leq \mathcal{O}\big((TC(a,\gamma) / B)^{4 / 5} + \gamma^{2T}\big)$ . Lastly, we remark that if  $\gamma^T$  is treated as a constant, the cross term  $\mathbb{E}\big[\hat{V}_M(h) - V_T\big]V_{T,\infty}$  in Equation (12) introduces a dependence of order  $\mathcal{O}(h\gamma^{2T})$  to the mean-squared error. In this case, the overall trade-off becomes  $\mathrm{MSE}_{\infty}(h,B,T) \approx \mathcal{O}\big(1 / (hB) + \gamma^{2T}(1 + h)\big)$ , and the optimal step-size is  $h^{*} \approx B^{-1 / 2}$ .
Vector Case Similar to the finite-horizon setting, we establish tight bounds for the MSE in the general case involving a stable matrix  $A$ . The detailed results are presented in Appendix C.3. As before, the MSE for the vector case can be computed in closed-form assuming that  $A$  is both diagonalisable and stable. The result reflects the same behaviour as in the scalar case. Conveniently, the MSE in Theorem 3.6 has been expressed with sharp terms in  $h$  and  $B$ , while confining the dependence on the system parameter  $a$  within the constant  $C$ , and the impact of higher-order terms in  $T$  within  $V_{T,\infty}$ . This allows us to state the vector case result in a similar form, where the constant will now depend on the eigenvalues of the matrix  $A$ , as well as the discount factor  $\gamma$ . These are provided in full detail in Appendix C.2.
Corollary 3.7. For  $A$  diagonalisable, with eigenvalues given by  $\Lambda$ , the mean-squared error of the Monte-Carlo estimator in the infinite-horizon, discounted setting is
$$
\mathrm {M S E} _ {\infty} (h, B, T) = C _ {3} \sigma^ {4} \frac {T}{h B} + \frac {\sigma^ {4}}{1 4 4} h ^ {4} + \mathcal {O} \left(h ^ {5} + \frac {1}{B}\right)
$$
with a constant  $C_3 = C_3(\Lambda, \gamma)$  and under the assumption that  $\gamma^T = o(h^4)$ .
The terms in Corollary 3.7 correspond to estimation error, approximation error and truncation error, mirroring the scalar scenario. The optimal step-size exhibits the same dependencies on  $T$  and  $B$  as in the scalar case, albeit with a different constant dependent on the eigenvalues.
4 From Linear to Non-Linear Systems: A Numerical Study
The trade-off identified in our analysis suggests that there exists an optimal choice for temporal resolution in policy evaluation. Our next goal is to verify the trade-off in several simulated dynamical systems. While our analysis assumes a linear transition and quadratic cost, we empirically demonstrate that such a trade-off also exists in nonlinear systems. For our experimental setup, we choose simple linear quadratic systems mirroring the setting of Section 2, as well as several standard benchmarks from Gym [Brockman et al., 2016] and MuJoCo [Todorov et al., 2012]. Our findings confirm the theoretical results and highlight the importance of choosing an appropriate step-size for policy evaluation.
4.1 Linear Quadratic Systems
We first run numerical experiments on the Langevin dynamical systems to examine the behaviour of the trade-off identified in our analysis. The results are shown in Fig. 1. For these experiments, we fix the noise  $\sigma^2 = 1$  and the cost  $Q = I$ . The lines in the plot represent the sample mean  $(\hat{V}_M(h) - V)^2$  and the shading represent the standard error of our sample means, computed over 50 independent runs. The plots in Fig. 1 exhibit a clear, U-shaped trade-off, as predicted by our theoretical results.
Fig. 1(a) shows the MSE in a one-dimensional system with  $T = 8$  and  $a = -1$ . The ground truth  $V$  is calculated analytically by using Eq. 21 in the Appendix. The figure illustrates how the error changes as we vary the data budget,  $B = \{2^{12}, 2^{13}, 2^{14}, 2^{15}, 2^{16}\}$ , and also illustrates the improvement that can be obtained by increasing the budget. As we increase  $B$ , both the error and the optimal step size  $h^*$ , decrease. This result strongly aligns with the analysis shown in Theorem 3.1 and Corollary 3.3. The same analysis is performed with respect to the parameter  $a$ , while fixing the data budget, in Fig. 1(b). By increasing the absolute value of the drift coefficient, the diffusion has a smaller impact, thus trajectories have less variability. This leads to a smaller  $h^*$  for the optimal point of the trade-off.
Fig. 1(c) and 1(d) present the experimental results for both undiscounted finite horizon and discounted infinite horizon multi-dimensional systems. For the finite-horizon setting,  $V$  is computed by numerically solving the Riccati Differential Equation; while in infinite-horizon, it is calculated through Lyapunov equation using a standard solver. In our multi-dimensional experiments, we set the
dimension  $n = 3$ . We fix all parameters and run our experiments for 5 randomly sampled  $3 \times 3$  dense, stable matrices in each setting. More details on the matrix structure can be found in Appendix D.1. Results in both plots suggest that the impact of the eigenvalues of  $A$  on  $h^*$  is mild and that the eigenvalue-dependent constant terms in Corollary 3.5 only marginally affect the optimal step-size  $h^*$ , similar to the trend observed in the scalar case with parameter  $a$ . In the infinite horizon system, the horizon needs to be large enough to manage truncation error while simultaneously being small enough to collect multiple trajectories. We choose  $\gamma$  large enough such that a good estimate of  $V$  can be obtained, and set  $T = 1 / (1 - \gamma)$ , which is referred to as the effective horizon in the RL literature.
4.2 Nonlinear Systems
Many nonlinear behaviors can be approximated by a high-dimensional linear system, which would be bounded by our theoretical results on the general case of  $n$ -dimensional systems, hinting that similar trade-offs could characterize nonlinear systems as well. We empirically show that the trade-off identified in linear quadratic systems carries over to nonlinear systems, with more complex cost functions. We demonstrate it in several simulated nonlinear systems from OpenAI Gym [Brockman et al., 2016], including Pendulum, BipedalWalker and six MuJoCo [Todorov et al., 2012] environments: InvertedDoublePendulum, Pusher, Swimmer, Hopper, HalfCheetah and Ant. We note that the original environments all have a fixed temporal discretization  $\delta t$ , pre-chosen by the designer. To measure the effect of  $h$ , we first modify all environments to run at a small discretization  $\delta t = 0.001$  as the proxy to the underlying continuous-time systems. We train a nonlinear policy parameterized by a neural network for each system, by the algorithm DAU [Tallec et al., 2019]. This policy is used to gather episode data from the continuous-time system proxy at intervals of  $\delta t = 0.001$  which are then down-sampled for different  $h$  based on the ratio of  $h$  and  $\delta t$ . The policy is stable in the sense that it produces reasonable behavior (e.g., pendulum stays mostly upright; Ant walking forward; etc.) and not cause early termination of episodes (e.g., BipedalWalker does not fall), in the continuous-time system proxy. The results of the MSE of Monte-Carlo policy evaluation are shown in Fig. 2. Similar to the linear systems case, we vary the data budget  $B$  and see how the MSE changes with the step-size  $h$ . The MSE shows a clear minimum for choosing the optimal step-size  $h^*$ , which generally decreases as the data budget increases. We slightly abuse notations by using  $V, \hat{V}$  to refer to the true and estimated sum of rewards instead of the cost. The true value of  $V$  is approximated by averaging the sum of rewards observed
at  $\delta t = 0.001$  from  $150k$  episodes. These environments fall under the finite horizon undiscounted setting. The system (and estimation) horizon  $T$  of our experiments is chosen to be the physical time of  $1k$  steps under the default  $\delta t$  in the original environments (with the exception of 200 steps for Pendulum and 500 steps for BipedalWalker). Please refer to Appendix D.4 for more implementation details, including the setup of  $B$ ,  $T$ ,  $\delta t$ ,  $h$ , training, and the compute resources. These systems are stochastic in the starting state, while having deterministic dynamics. Despite the different settings from our analysis, a clear trade-off is evident in all systems. This suggests that our findings may have broader applicability than the specific conditions under which our theoretical analysis was established.
4.3 Guidelines for Setting Step-Size  $h$
The precise characterization of the MSE in Section 3 can be exploited to set the step-size close to the optimal value without any prior knowledge of the system, provided experiments on a smaller data budget are performed beforehand. Although the optimal step-size  $h^*$  clearly depends on all quantities characterizing the dynamics and the policy, the technical analysis of the MSE accurately quantifies how  $h^*$  scales with respect to the data budget  $B$ . Specifically,  $h^*(B) \approx c_F B^{-1/3}$  for finite horizon and  $h^*(B) \approx c_I B^{-1/5}$  for infinite horizon, where  $c_F$  and  $c_I$  hide the dependencies on the system parameters, exposing only the order in  $B$ . This allows us to extrapolate the optimal step-size for the given data budget, using the constant estimated with a smaller one. By evaluating through numerical experiments the performances of different step-sizes on the reduced data budget, it is possible to identify an approximate  $h^*$ , and subsequently determine  $c_F$  or  $c_I$ , which then gives the whole range of optimal values of  $h$  with respect to  $B$  through the aforementioned relation. Note that this approach does not require prior knowledge of the dynamics, yet it provides a systematic way for setting the step size  $h$  for any given scenario.
Figure 3 plots the empirical  $h^*$  over  $B$  for all nonlinear environments, and fitted lines based on the relation from the analysis  $h^* = c_F B^{-1/3}$ , where the constant  $c_F$  varies with the environment. The plot shows that the scaling of  $h^*$  w.r.t.  $B$  predicted by our analysis is overall a good approximation of the trend observed in the experiments with nonlinear systems, except for negative cases like the Inverted-DoublePendulum. This suggests that setting the step-size according to the analysis can yield a value close to optimality.
We provide a precise characterization of the approximation, estimation and truncation errors incurred by Monte-Carlo policy evaluation in continuous-time linear stochastic dynamical systems with quadratic cost. This analysis reveals a fundamental bias-variance trade-off, modulated by the level of temporal discretization  $h$ . Moreover, we confirm in numerical simulations that the analysis accurately captures the trade-off in a precise, quantitative manner. We also demonstrate that the trade-off carries over to non-linear environments such as the popular MuJoCo physics simulation. These findings show that managing the temporal discretization level  $h$  can greatly improve the quality of Monte-Carlo policy evaluation under a fixed data budget  $B$ . These results have direct implications for practice, as it remains common to adopt a pre-set step-size regardless of the data resources anticipated, which we have seen is a highly sub-optimal approach for a given budget.
The present work serves as a first step toward understanding the effects of temporal resolution on RL. There are several limitations that we would like to address in future work. Our analysis is restricted to Monte-Carlo estimation, while there are more advanced techniques such as temporal difference learning and direct system identification, which may exhibit different behaviours. Also, our focus is policy evaluation. It is worth studying policy optimization to understand the full control setting, which might require relaxing the stability assumption on the closed-loop system since the controllers being iterated might not always be stable. Additionally, it would be interesting to explore non-uniform discretization schemes, such as through an adaptive sampling scheme. From the system's perspective, we have currently analyzed stochastic linear quadratic systems with additive Gaussian noise and noiseless observations. It remains to be determined if the exact characterization is still attainable with other types of noise, in the partially observable case, or with noisy observations. Finally, extending the analysis to non-linear systems would be valuable.
The authors would like to thank Csaba Szepesvari for the helpful discussions. Zichen Zhang gratefully acknowledges the financial support by an NSERC CGSD scholarship and an Alberta Innovates PhD scholarship during this project. He is thankful for the compute resources generously provided by Digital Research Alliance of Canada (and formerly Compute Canada), which is sponsored through the accounts of Martin Jagersand and Dale Schuurmans. Dale Schuurmans gratefully acknowledges funding from the Canada CIFAR AI Chairs Program, Amii and NSERC. Johannes Kirschner gratefully acknowledges funding from the SNSF Early Postdoc.Mobility fellowship P2EZP2_199781.
Y. Abbasi-Yadkori and C. Szepesvári. Regret bounds for the adaptive control of linear quadratic systems. In S. M. Kakade and U. von Luxburg, editors, Proceedings of the 24th Annual Conference on Learning Theory, volume 19 of Proceedings of Machine Learning Research, pages 1-26, Budapest, Hungary, 09-11 Jun 2011. PMLR.
Y. Abbasi-Yadkori, N. Lazic, and C. Szepesvári. Model-free linear quadratic control via reduction to expert prediction. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3108-3117. PMLR, 2019.
S. Bahl, M. Mukadam, A. Gupta, and D. Pathak. Neural dynamic policies for end-to-end sensory motor learning. Advances in Neural Information Processing Systems, 34, 2020.
L. C. Baird. Reinforcement learning in continuous time: Advantage updating. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94), volume 4, pages 2448-2453. IEEE, 1994.
M. Basei, X. Guo, A. Hu, and Y. Zhang. Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizon. Journal of Machine Learning Research, 23(178):1-34, 2022.
E. Bayraktar and A. D. Kara. Approximate Q-learning for controlled diffusion processes and its near optimality, 2023.
A Moment Calculations
Recall that the solution of the SDE in Equation (7), with  $x(0) = 0$ , takes the following form:
$$
x (t) = \sigma \int_ {0} ^ {t} e ^ {a (t - s)} \mathrm {d} w (s). \tag {14}
$$
An integral part of finding the mean-squared error of the Monte-Carlo estimator is the computation of the moments  $\mathbb{E}\left[x(t)^2\right]$ ,  $\mathbb{E}\left[x(t)^4\right]$  and  $\mathbb{E}\left[x(s)^2 x(t)^2\right]$  when  $s \leq t$ .
Lemma A.1. Let  $x(t)$  be the solution of Equation (7). The second moment of the state variable is
$$
\mathbb {E} \left[ x ^ {2} (t) \right] = \frac {\sigma^ {2}}{2 a} \left(e ^ {2 a t} - 1\right). \tag {15}
$$
For the fourth moment, we get:
$$
\mathbb {E} \left[ x (t) ^ {4} \right] = \frac {3 \sigma^ {4}}{4 a ^ {2}} \left(e ^ {2 a t} - 1\right) ^ {2} \tag {16}
$$
Assuming that  $s \leq t$ , we further get:
$$
\mathbb {E} \left[ x ^ {2} (s) x ^ {2} (t) \right] = \frac {\sigma^ {4}}{4 a ^ {2}} \left(e ^ {2 a s} - 1\right) e ^ {2 a t} \left\{\left(e ^ {- 2 a s} - e ^ {- 2 a t}\right) + 3 \left(1 - e ^ {- 2 a s}\right) \right\}. \tag {17}
$$
(1) We start with the second moment  $\mathbb{E}\left[x(t)^2\right]$ .
$$
\mathbb {E} \left[ x (t) ^ {2} \right] = \sigma^ {2} e ^ {2 a t} \mathbb {E} \left[ \left(\int_ {0} ^ {t} e ^ {- a s} d w (s)\right) ^ {2} \right] = \sigma^ {2} e ^ {2 a t} \int_ {0} ^ {t} e ^ {- 2 a s} d s = \frac {\sigma^ {2}}{2 a} (e ^ {2 a t} - 1)
$$
The calculation makes use of the Ito isometry, which can be stated as:
$$
\mathbb {E} \left[ \left(\int_ {0} ^ {t} z (s) \mathrm {d} w (s)\right) ^ {2} \right] = \mathbb {E} \left[ \int_ {0} ^ {t} z (s) ^ {2} \mathrm {d} s \right], \tag {18}
$$
for any stochastic process  $z(\cdot)$  adapted to the filtration induced by the Wiener process  $w(\cdot)$ .
(2) Next we compute  $\mathbb{E}\left[x(t)^4\right]$  through Itô's formula. Define  $y(t) := \int_0^t e^{-au} \, \mathrm{d}w(u)$ , so that  $\mathrm{d}y(t) = e^{-at} \, \mathrm{d}w(t)$ . Thus,
$$
\begin{array}{l} \mathrm {d} f (y (t)) = f ^ {\prime} (y (t)) \mathrm {d} y (t) + \frac {1}{2} f ^ {\prime \prime} (y (t)) (\mathrm {d} y (t)) ^ {2} \\ = f ^ {\prime} (y (t)) e ^ {- a t} d w (t) + \frac {1}{2} f ^ {\prime \prime} (y (t)) e ^ {- 2 a t} d t, \\ \end{array}
$$
for any  $f(\cdot)$ . By choosing  $f(y) = y^4$ :
$$
f ^ {\prime} (y) = 4 y ^ {3} \quad \text {a n d} \quad f ^ {\prime \prime} (y) = 1 2 y ^ {2}.
$$
Therefore, by integration and taking the expectation:
$$
\begin{array}{l} \mathbb {E} [ f (y (t)) ] = \mathbb {E} \left[ \int_ {0} ^ {t} f ^ {\prime} (y (u)) e ^ {- a u} d w (u) \right] + \frac {1}{2} \mathbb {E} \left[ \int_ {0} ^ {t} f ^ {\prime \prime} (y (u)) e ^ {- 2 a u} d u \right] \\ = \underbrace {\mathbb {E} \left[ \int_ {0} ^ {t} 4 \left(\int_ {0} ^ {u} e ^ {- a v} d w (v)\right) ^ {3} e ^ {- a u} d w (u) \right]} _ {= 0} + \frac {1}{2} \mathbb {E} \left[ \int_ {0} ^ {t} 1 2 \left(\int_ {0} ^ {u} e ^ {- a v} d w (v)\right) ^ {2} e ^ {- 2 a u} d u \right] \\ = 6 \mathbb {E} \left[ \int_ {0} ^ {t} \left(\int_ {0} ^ {u} e ^ {- a v} e ^ {- a u} d w (v)\right) ^ {2} d u \right] \\ = 6 \int_ {0} ^ {t} \mathbb {E} \left[ \left(\int_ {0} ^ {u} e ^ {- a v} e ^ {- a u} d w (v)\right) ^ {2} \right] d u \quad (\text {I tô i s o m e t r y}) \\ = 6 \int_ {0} ^ {t} \int_ {0} ^ {u} e ^ {- 2 a v} e ^ {- 2 a u} d v d u \\ = \int_ {0} ^ {t} e ^ {- 2 a u} \frac {1}{2 a} (1 - e ^ {- 2 a u}) d u \\ = \frac {3}{4 a ^ {2}} \left(1 - e ^ {- 2 a t}\right) ^ {2} \\ \end{array}
$$
From Equation (14) it holds  $x(t) = \sigma e^{at}y(t)$  so that the second part of the lemma follows.
(3) Lastly, we compute  $\mathbb{E}\left[x(s)^2 x(t)^2\right]$  for  $s \leq t$ .
$$
\begin{array}{l} \mathbb {E} \left[ x ^ {2} (s) x ^ {2} (t) \right] = \sigma^ {4} e ^ {2 a (s + t)} \mathbb {E} \left[ \left(\int_ {0} ^ {s} e ^ {- a u} \mathrm {d} w (u)\right) ^ {2} \left(\int_ {0} ^ {t} e ^ {- a u} \mathrm {d} w (u)\right) ^ {2} \right] \\ = \sigma^ {4} e ^ {2 a (s + t)} \mathbb {E} \left[ \left(\int_ {0} ^ {s} e ^ {- a u} d w (u)\right) ^ {2} \left(\int_ {0} ^ {s} e ^ {- a u} d w (u) + \int_ {s} ^ {t} e ^ {- a u} d w (u)\right) ^ {2} \right] \\ = \sigma^ {4} e ^ {2 a (s + t)} \left\{\underbrace {\mathbb {E} \left[ \left(\int_ {0} ^ {s} e ^ {- a u} d w (u)\right) ^ {4} \right]} _ {\text {(i)}} + \underbrace {\mathbb {E} \left[ \left(\int_ {0} ^ {s} e ^ {- a u} d w (u)\right) ^ {2} \right]} _ {\text {(i i)}} \mathbb {E} \left[ \left(\int_ {s} ^ {t} e ^ {- a u} d w (u)\right) ^ {2} \right] \right\} \\ \end{array}
$$
Note that we computed (i) before. For (ii) it holds:
$$
\begin{array}{l} \mathbb {E} \left[ \left(\int_ {0} ^ {s} e ^ {- a u} d w (u)\right) ^ {2} \right] = \int_ {0} ^ {s} e ^ {- 2 a u} d u \\ = \frac {1}{2 a} (1 - e ^ {- 2 a s}) \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ \left(\int_ {s} ^ {t} e ^ {- a u} d w (u)\right) ^ {2} \right] = \int_ {s} ^ {t} e ^ {- 2 a u} d u \\ = \frac {1}{2 a} \left(e ^ {- 2 a s} - e ^ {- 2 a t}\right) \\ \end{array}
$$
Therefore, assuming  $s \leq t$ , it holds that:
$$
\begin{array}{l} \mathbb {E} \left[ x ^ {2} (s) x ^ {2} (t) \right] = \sigma^ {4} e ^ {2 a (s + t)} \left\{\frac {1}{4 a ^ {2}} \left(1 - e ^ {- 2 a s}\right) \left(e ^ {- 2 a s} - e ^ {- 2 a t}\right) + \frac {3}{4 a ^ {2}} \left(1 - e ^ {- 2 a s}\right) ^ {2} \right\} \\ = \frac {\sigma^ {4}}{4 a ^ {2}} \left(e ^ {2 a t} - 1\right) e ^ {2 a s} \left\{\left(e ^ {- 2 a t} - e ^ {- 2 a s}\right) + 3 \left(1 - e ^ {- 2 a t}\right) \right\}. \\ \end{array}
$$
B Calculations of the Mean-Squared Error
B.1 Undiscounted, Finite-Horizon: Proof of Theorem 3.1
$$
\mathbb {E} [ \hat {V} _ {M} (h) ] = \frac {h}{M} \sum_ {i = 1} ^ {M} \sum_ {k = 0} ^ {N - 1} \mathbb {E} [ x _ {i} ^ {2} (k h) ] = h \sum_ {k = 0} ^ {N - 1} \mathbb {E} [ x ^ {2} (k h) ]
$$
where we denote  $x(t) = x_{1}(t)$  for simplicity. Next we expand the mean-squared error
$$
\begin{array}{l} \mathbb {E} \big [ (\hat {V} _ {M} (h) - V _ {T}) ^ {2} \big ] = \mathbb {E} \big [ \hat {V} _ {M} ^ {2} (h) \big ] - 2 V _ {T} \mathbb {E} \big [ \hat {V} _ {M} (h) \big ] + V _ {T} ^ {2} \\ = \frac {h ^ {2}}{M ^ {2}} \mathbb {E} \left[ \left(\sum_ {i = 1} ^ {M} \sum_ {k = 0} ^ {N - 1} x _ {i} ^ {2} (k h)\right) ^ {2} \right] - 2 V _ {T} \mathbb {E} [ \hat {V} _ {M} (h) ] + V _ {T} ^ {2} \\ = \frac {h ^ {2}}{M ^ {2}} \sum_ {i, j = 1} ^ {M} \sum_ {k, l = 0} ^ {N - 1} \mathbb {E} \left[ x _ {i} ^ {2} (k h) x _ {j} ^ {2} (l h) \right] - 2 V _ {T} \mathbb {E} [ \hat {V} _ {M} (h) ] + V _ {T} ^ {2} \\ = \frac {h ^ {2}}{M} \sum_ {k, l = 0} ^ {N - 1} \mathbb {E} [ x ^ {2} (k h) x ^ {2} (l h) ] + \frac {M ^ {2} - M}{M ^ {2}} \mathbb {E} [ \hat {V} _ {M} (h) ] ^ {2} - 2 V _ {T} \mathbb {E} [ \hat {V} _ {M} (h) ] + V _ {T} ^ {2} \\ \end{array}
$$
For the last equality, note that  $\mathbb{E}[\hat{V}_M(h)]^2 = h^2\sum_{k,l = 0}^{N - 1}\mathbb{E}[x^2 (kh)]\mathbb{E}[x^2 (lh)]$ . It remains to compute the expressions. By Lemma A.1 we have for the second moment of the state variable:
$$
\mathbb {E} \left[ x ^ {2} (t) \right] = \frac {\sigma^ {2}}{2 a} \left(e ^ {2 a t} - 1\right). \tag {19}
$$
Assuming that  $s \leq t$ , from the same lemma we get the following for the fourth moments: