Title: Managing Temporal Resolution in Continuous Value Estimation: A Fundamental Trade-off
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

Policy Evaluation

Temporal Discretization

Continuous Time

LQR
Abstract: A default assumption in reinforcement learning (RL) and optimal control is that observations arrive at discrete time points on a fixed clock cycle. Yet, many applications involve continuous-time systems where the time discretization, in principle, can be managed. The impact of time discretization on RL methods has not been fully characterized in existing theory, but a more detailed analysis of its effect could reveal opportunities for improving data-efficiency. We address this gap by analyzing Monte-Carlo policy evaluation for LQR systems and uncover a fundamental trade-off between approximation and statistical error in value estimation. Importantly, these two errors behave differently to time discretization, leading to an optimal choice of temporal resolution for a given data budget. These findings show that managing the temporal resolution can provably improve policy evaluation efficiency in LQR systems with finite data. Empirically, we demonstrate the trade-off in numerical simulations of LQR instances and standard RL benchmarks for non-linear continuous control.

Managing Temporal Resolution in Continuous Value Estimation: A Fundamental Trade-off

Abstract

A default assumption in reinforcement learning (RL) and optimal control is that observations arrive at discrete time points on a fixed clock cycle. Yet, many applications involve continuous-time systems where the time discretization, in principle, can be managed. The impact of time discretization on RL methods has not been fully characterized in existing theory, but a more detailed analysis of its effect could reveal opportunities for improving data-efficiency. We address this gap by analyzing Monte-Carlo policy evaluation for LQR systems and uncover a fundamental trade-off between approximation and statistical error in value estimation. Importantly, these two errors behave differently to time discretization, leading to an optimal choice of temporal resolution for a given data budget. These findings show that managing the temporal resolution can provably improve policy evaluation efficiency in LQR systems with finite data. Empirically, we demonstrate the trade-off in numerical simulations of LQR instances and standard RL benchmarks for non-linear continuous control.

1 Introduction

In many real-world applications of control and reinforcement learning, the underlying system evolves continuously in time [Eliasmith and Furlong, 2022]. For instance, a physical system like a robot is naturally modelled as a stochastic dynamical system. Nonetheless, sensor measurements are typically captured at discrete time intervals, which entails choosing the sampling frequency or measurement step-size . This step-size is usually treated as an immutable quantity based on prior measurement design, but it has a significant impact on data efficiency [Burns et al., 2023]. We will see that, from a data-cost perspective, learning can be far more data efficient if it operates at a temporal resolution that is allowed to differ from a prior step-size choice.

In this work, we investigate episodic policy evaluation with a finite data budget to provide a key initial step to addressing broader research questions on the impact of temporal resolution in reinforcement learning. We show that data efficiency can be significantly improved by leveraging a precise understanding of the trade-off between approximation error and statistical estimation error in value estimation - two factors that react differently to the level of temporal discretization. Intuitively, employing a finer temporal resolution leads to a better approximation of the continuous-time system from discrete measurements; however, under a fixed data budget, denser data within each trajectory results in fewer trajectories, leading to increased estimation variance due to system stochasticity. This implies that, for a given data cost, it can be beneficial to increase temporal spacing between recorded data points beyond a pre-set measurement step-size. This holds true for any system with stochastic dynamics, even if the learner has access to exact (noiseless) state measurements.

The main contributions of this work are twofold. First, we conduct a theoretical analysis focusing on the canonical case of Monte-Carlo value estimation in a Langevin dynamical system (linear dynamics perturbed by a Wiener process) with quadratic instantaneous costs, which corresponds to policy evaluation in linear quadratic control (LQR). To formalize the impact of time discretization on policy evaluation, we present analytical expressions for the mean-squared error that exactly characterize the approximation-estimation trade-off with respect to the step-size parameter. From this trade-off, we derive the optimal step-size for a given Langevin system and characterize its dependence on the data budget. Second, we carry out a numerical study that illustrates and confirms the trade-off in both linear and non-linear systems, including several MuJoCo control environments. The latter also highlights the practical impact of the choice of sampling frequency, which significantly affects the MSE, and we therefore provide recommendations to practitioners for properly choosing the step-size parameter.

1.1 Related Work

There is a sizable literature on reinforcement learning for continuous-time systems [e.g. Doya, 2000, Lee and Sutton, 2021, Lewis et al., 2012, Bahl et al., 2020, Kim et al., 2021, Yildiz et al., 2021]. These previous works largely focus on deterministic dynamics without investigating trade-offs in temporal discretization. A smaller body of work considers learning continuous-time control under stochastic [Baird, 1994, Bradtke and Duff, 1994, Munos and Bourgine, 1997, Munos, 2006], or bounded [Lutter et al., 2021] perturbations, but their objective is to make standard learning methods more robust to small time scales [Tallec et al., 2019], or develop continuous-time algorithms that unify classical methods in discrete-time [Jia and Zhou, 2022a,b], without explicitly addressing temporal discretization. However, we find that managing temporal resolution offers substantial improvements not captured by previous studies.

The LQR setting is a standard framework in control theory and it gives rise to a fundamental optimal control problem [Lindquist, 1990], which has proven to be a challenging scenario for reinforcement learning algorithms [Tu and Recht, 2019, Krauth et al., 2019]. The stochastic LQR considers linear systems driven by additive Gaussian noise with a quadratic cost, minimised using a feedback controller. Although this is a well-understood scenario with a known optimal controller in closed form [Georgiou and Lindquist, 2013], the statistical properties of the long-term cost have only recently been investigated [Bijl et al., 2016]. Our research closely relates to the now extensive literature on reinforcement learning in LQR systems [e.g. Bradtke, 1992, Krauth et al., 2019, Tu and Recht, 2018, Dean et al., 2020, Tu and Recht, 2019, Dean et al., 2018, Fazel et al., 2018, Gu et al., 2016]. These works uniformly focus on the discrete time setting, although the benefits of managing spatial rather than temporal discretization have also been considered [Sinclair et al., 2019, Cao and Krishnamurthy, 2020]. Wang et al. [2020] studies continuous-time LQR, focusing on the exploration problem. Basei et al. [2022] provides a regret bound depending on sampling frequency, for a specific algorithm based on least-squares estimation. Their analysis considers approximation and estimation errors independently, without identifying a trade-off.

There is compelling empirical evidence that managing temporal resolution can greatly improve learning performance [Lakshminarayanan et al., 2017, Sharma et al., 2017, Huang et al., 2019, Huang and Zhu, 2020, Dabney et al., 2021, Park et al., 2021], typically achieved through options [Sutton et al., 1999], a specific instance of which is action persistence, achieved by maintaining a fixed action over multiple time steps (also known as action repetition). Recently, these empirical findings have been supported by an initial theoretical analysis [Metelli et al., 2020], showing that temporal discretization plays a role in determining the effectiveness of fitted Q-iteration. Their analysis does not consider fully continuous systems, but rather remains anchored in a base-level discretization. Furthermore, it only provides worst-case upper bounds, without capturing detailed practical tradeoffs. Lutter et al. [2022] discusses the practical trade-off on time discretization but do not provide theoretical support. Bayraktar and Kara [2023] analyzes a trade-off between sample complexity and approximation error, which however requires the state and action spaces of the diffusion process to be discretized to yield an MDP. The two components in the trade-off are analyzed independently, unlike the unified statistical analysis provided in our work, and no exact characterization is presented.

2 Policy Evaluation in Continuous Linear Quadratic Systems

In the classical continuous-time linear quadratic regulator (LQR), a state variable X ( t ) ∈ R n evolves over time t ≥ 0 according to the following equation:

The dynamical model is fully specified by the matrices A ∈ R n × n , B ∈ R n × p and the diffusion coefficient σ . The control input U ( t ) ∈ R p is given by a fixed policy, and W ( t ) is a Wiener process. The state variable X ( t ) is fully observed. For simplicity, we assume that the dynamics start at X (0) = - → 0 ∈ R n [c.f. Abbasi-Yadkori and Szepesvári, 2011, Dean et al., 2020].

The quadratic cost J is defined for positive definite, symmetric matrices Q ∈ R n × n and R ∈ R p × p , a system horizon 0 < τ ≤ ∞ and a discount factor γ ∈ (0 , 1] :

In the following, we consider the class of controllers given by static feedback of the state, i.e.: U ( t ) = KX ( t ) where K ∈ R p × n is the static control matrix yielding the control input. It is well known that in infinite horizon setting, the optimal control belongs to this class. Given such an input, the LQR in Equation (1) can be further reduced to a linear stochastic dynamical system described by a Langevin equation. Using the definitions A := A + B K and Q := Q + K ⊤ RK , we express both the state dynamics and the cost in a more compact form:

The expected cost w.r.t. the Wiener process is V τ = E [ J τ ] . The policy plays a role in this work solely from its impact in the closed-loop dynamics A . Equation (3) is what we analyze in the following. We explicitly distinguish the finite-horizon setting where τ < ∞ , γ ≤ 1 and the cost is V τ , and the infinitehorizon setting where τ = ∞ , γ < 1 and the cost is V ∞ . In order not to incur infinite costs in either scenario, a stable closed-loop matrix A should be assumed. Note that the existence of a stabilizing controller is guaranteed under the standard controllability assumptions in LQR [Fazel et al., 2018, Abbasi-Yadkori et al., 2019, Dean et al., 2020]. Thus the closed-loop stability can be safely assumed.

Monte-Carlo Policy Evaluation Our main objective in policy evaluation is to estimate the expected cost from discrete-time observations. To this end, we choose a uniform discretization of the interval [0 , T ] with increment h , resulting in N = T/h time points t k := kh for k ∈ { 0 , 1 , . . . , N } . Here, the estimation horizon T , such that T < ∞ and T ≤ τ , is chosen by the practitioner (for simplicity assume that T/h is an integer). With the N points sampled from one trajectory, a standard way to approximate the integral in Equation (3) is through the Riemann sum estimator

To estimate V τ , we average M independent trajectories with cost estimates ˆ J 1 , . . . ˆ J M to obtain the Monte-Carlo estimator :

Our primary goal is to understand the mean-squared error of the Monte-Carlo estimator for a fixed system (specified by A , σ and Q ), to inform an optimal choice of the step-size parameter h for a predetermined data budget B = M · N .

Note that one degree of freedom remains in choosing M and N . For simplicity, we require that in the finite-horizon setting, the estimation grid is chosen to cover the full episode [0 , τ ] which leads to the constraint T = τ = N · h . We write the mean-squared error-surface as a function of h and B :

In the infinite horizon setting, i.e. τ = ∞ , the estimation horizon T is a free variable chosen by the experimenter that determines the number of trajectories through M = B N = Bh T . The mean-squared error for the infinite horizon setting is given as a function of h , B , and T :

3 Characterizing the Mean-Squared Error (MSE)

In the following our aim is to characterize the MSE of the Monte-Carlo estimator as a function of the step-size h and data budget B (and estimation horizon T in the infinite horizon setting). Our results uncover a fundamental trade-off for choosing an optimal step-size that leads to a minimal MSE.

One-Dimensional Langevin Process To simplify the exposition while preserving the main ideas, we will first present the results for the 1-dimensional case. The analysis for the vector case exhibits the same quantitative behavior but is significantly more involved. To distinguish the 1-dimensional from the n -dimensional setting described in Equation (3), we use lower-case symbols. Let x ( t ) ∈ R be the scalar state variable that evolves according to the following Langevin equation:

Here, a ∈ R is the drift coefficient and w ( t ) is a Wiener process with scale parameter σ > 0 . We assume that a ≤ 0 , i.e. the system is stable (or marginally stable).

The realized sample path in episode i = 1 , . . . , M is x i ( t ) (with starting state x (0) = 0 ) and t ∈ [0 , T ] . The expected cost is

where r i ( t ) = qx 2 i ( t ) is the quadratic cost function for a fixed q > 0 . The Riemann sum that approximates the cost realized in episode i ∈ [ M ] becomes ˆ J i ( h ) = ∑ N -1 k =0 hqx 2 i ( kh ) . Given data from M episodes, the Monte-Carlo estimator is ˆ V M ( h ) = 1 M ∑ M i =1 ˆ J i ( h ) . Since the square of the cost parameter q 2 factors out of the mean-squared error, we set q = 1 in what follows.

3.1 Finite-Horizon Setting

Recall that in the finite-horizon setting we set the system horizon τ and estimation horizon T to be the same. This implies that the estimation grid covers the full episode, i.e. hN = T = τ . Perhaps surprisingly, the mean-squared error of the Riemann estimator for the Langevin system (7) can be computed in closed form. The result takes its simplest form in the finite-horizon, undiscounted setting where γ = 1 and τ < ∞ . This result is summarized in the following theorem.

Theorem 3.1 (Finite-horizon, undiscounted MSE) . In the finite-horizon, undiscounted setting, the mean-squared error of the Monte-Carlo estimator is

While perhaps daunting at first sight, the result exactly characterizes the error surface as a function of the step-size h and the budget B for any given Langevin system. The proof involves computing the closed-form expressions for the second and fourth moments of the random trajectories x i ( t ) and is provided in Appendices A and B.1.

In the case of marginal stability ( a = 0 ), a simpler form of the MSE emerges that is easier to interpret. Taking the limit a → 0 of the previous expression yields the following result (refer to the discussion and proof in Appendix B.1):

Corollary 3.2 (MSE for marginally stable system) . Assume a marginally stable system, a = 0 . Then the mean-squared error of the Monte-Carlo estimator is

The first part of the expression can be understood as a Riemann sum approximation error controlled by the h 2 term. The second part corresponds to the variance term that decreases with the number of episodes as 1 M = T Bh . The remaining terms are of lower order terms for small h and large B . For a fixed data budget B , the step-size h can be chosen to balance these two terms:

where the approximation omits higher order terms in 1 /B . From this, we can compute the optimal number of episodes M ∗ ≈ Bh ∗ T = ( 2 3 ) 1 / 3 B 2 / 3 . We remark that under the assumption B ≫ 1 , we also obtain that M ∗ ≫ 1 . This is in agreement with the implicit requirement that h is big enough to consider at least one whole trajectory, i.e. h > T/B . Consequently, the mean-squared error for the optimal choice of h (up to lower order terms in 1 /B ):

In other words, the optimal error rate as a function of the data budget is O ( B -2 / 3 ) . We can further obtain a similar form for h ∗ for the general case where a ≤ 0 .

Corollary 3.3 (Approximate MSE) . The MSE is

for h → 0 and B →∞ , with system-dependent constants

Moreover, for any h > 0 and B > 0 ,

with c 1 = c 1 ( σ, a, T ) and c 2 = c 2 ( σ, a, T ) and the inequalities holds true up to a finite, lower-order polynomial expressions σ 4 h B poly ( h, a, T ) , given in Appendix B.2.

For the proof please see Appendix B.2. From the corollary, we can derive an optimal step-size, up to lower order terms in 1 /B :

Note that the same h ∗ ( B ) also minimizes the upper bound of the MSE up to a constant factor. The scaling MSE T ( h ∗ , B ) ≤ O ( B -2 / 3 ) cannot be improved given the lower bound on the MSE. The derivation is provided in Appendix B.3 where we also include a more precise expression of h ∗ .

Discounted Cost Adding discounting ( γ < 1 ) in the finite-horizon setting does not fundamentally change the results; however, it makes the derivation more involved (Appendix B.4).

Vector Case Addressing the general case ( n -dimensional Langevin systems with n > 1 ) for a stable matrix A requires forgoing the exact form of the MSE. We derive tight bounds on the MSE, both of which are convex functions of h , thereby narrowing down its behaviour with respect to the step size. The results are presented in Appendix C.3. Although the convex behaviour is proven only for Langevin systems, our experimental results in Section 4 exhibit a similar trade-off for general nonlinear stochastic systems.

Under the additional assumption that the matrix A is also diagonalisable, we are again able to exactly characterise the MSE with closed-form computations. Diagonalisability is a mild assumption since it can be achieved under a controllable system. Indeed, controllability allows for the free adjustment of the eigenvalues of the closed-loop matrix A through the choice of the controller K . The eigenvalues can effectively be chosen to be distinct from each other to ensure a diagonalisable A . While the explicit form of the MSE is computable, its lengthy formula is not easily interpretable and is thus deferred to Appendix C. The following theorem summarizes the result as a Taylor expansion for small h and large B .

Theorem 3.4 (Mean-squared error - vector case) . Assume A is diagonalisable, with eigenvalues Λ = { λ 1 , . . . , λ n } . The mean-squared error of the Monte-Carlo estimator in the finite-horizon, undiscounted setting, is

The proof, including the exact derivation of the constants C 1 , C 1 (Λ) , C 2 , C 2 (Λ) , can be found in Appendix C.1. Note that the terms composing the MSE closely resemble those obtained in the scalar analysis. In fact, when comparing them with the expressions in Equation (22) and Equation (23) (in Appendix B.3), the expression has the same order for h, B and T . The only difference is that in the vector case, cumbersome eigenvalue-dependent constants are involved, whereas in the scalar case, the result can more easily be expressed in terms of the system parameter a .

Since the optimal choice for h is determined by balancing the trade-off between the two terms above, E 1 for the approximation error and E 2 for the variance, its expression is analogous to the scalar case, as shown by the following corollary.

Corollary 3.5 (Optimal step size - vector case) . Under the assumption that B ≫ 1 , the optimal step-size for the vector case is given by

The constants in Corollary 3.5 are the same as in Theorem 3.4.

3.2 Infinite-Horizon Setting

The main characteristic of the finite-horizon setting is the trade-off between approximation and estimation error. Recall that in the infinite-horizon setting ( τ = ∞ ), the estimation horizon T < ∞ becomes a free variable that is chosen by the experimenter to define the measurement range [0 , T ] . Consequently the mean-squared error of the Monte-Carlo estimator suffers an additional truncation error from using a finite Riemann sum with N = T/h terms as an approximation to the infinite integral that defines the cost V ∞ . More precisely, we decompose the expected cost V ∞ = V T + V T, ∞ , where V = ∫ T γ t E [ x 2 ( t )] dt as before, and

T 0

It is a direct calculation based on Lemma A.1 in Appendix. Thus the mean-squared error becomes

where MSE T ( h, B ) = E [ ( ˆ V M ( h ) -V T ) 2 ] is the mean-squared error of discounted finite-horizon setting. Note that the term V 2 T, ∞ is neither controlled by a small step-size h nor by a large data budget B , hence results in the truncation error from finite estimation. Fortunately, geometric discounting ensures that V 2 T, ∞ = O ( γ 2 T ) , which is not unexpected given that the term constitutes the tail of the geometric integral. In particular, setting T = c · log( B ) / log(1 /γ ) for large enough c > 1 ensures the truncation error is below the estimation variance. We summarize the result in the next theorem.

Theorem 3.6 (Infinite-horizon, discounted MSE) . In the infinite-horizon, discounted setting, the mean-squared error of the Monte-Carlo estimator is

where we let C ( a, γ ) = 1 log( γ )( a +log( γ ))(2 a +log( γ )) 2 and assume that γ T = o ( h 4 ) .

The proof is provided in Appendix B.5. It follows that the optimal choice for the step-size is h ∗ ( B,T ) ≈ (36 T C ( a, γ ) /B ) 1 / 5 . The minimal mean-squared error is MSE ∞ ( h ∗ , T, B ) ≤ O ( ( T C ( a, γ ) /B ) 4 / 5 + γ 2 T ) . Lastly, we remark that if γ T is treated as a constant, the cross term E [ ˆ V M ( h ) -V T ] V T, ∞ in Equation (12) introduces a dependence of order O ( hγ 2 T ) to the meansquared error. In this case, the overall trade-off becomes MSE ∞ ( h, B, T ) ≈ O ( 1 / ( hB ) + γ 2 T (1 + h ) ) , and the optimal step-size is h ∗ ≈ B -1 / 2 .

Vector Case Similar to the finite-horizon setting, we establish tight bounds for the MSE in the general case involving a stable matrix A . The detailed results are presented in Appendix C.3. As before, the MSE for the vector case can be computed in closed-form assuming that A is both diagonalisable and stable. The result reflects the same behaviour as in the scalar case. Conveniently, the MSE in Theorem 3.6 has been expressed with sharp terms in h and B , while confining the dependence on the system parameter a within the constant C , and the impact of higher-order terms in T within V T, ∞ . This allows us to state the vector case result in a similar form, where the constant will now depend on the eigenvalues of the matrix A , as well as the discount factor γ . These are provided in full detail in Appendix C.2.

Corollary 3.7. For A diagonalisable, with eigenvalues given by Λ , the mean-squared error of the Monte-Carlo estimator in the infinite-horizon, discounted setting is

with a constant C 3 = C 3 (Λ , γ ) and under the assumption that γ T = o ( h 4 ) .

The terms in Corollary 3.7 correspond to estimation error, approximation error and truncation error, mirroring the scalar scenario. The optimal step-size exhibits the same dependencies on T and B as in the scalar case, albeit with a different constant dependent on the eigenvalues.

4 From Linear to Non-Linear Systems: A Numerical Study

The trade-off identified in our analysis suggests that there exists an optimal choice for temporal resolution in policy evaluation. Our next goal is to verify the trade-off in several simulated dynamical systems. While our analysis assumes a linear transition and quadratic cost, we empirically demonstrate that such a trade-off also exists in nonlinear systems. For our experimental setup, we choose simple linear quadratic systems mirroring the setting of Section 2, as well as several standard benchmarks from Gym [Brockman et al., 2016] and MuJoCo [Todorov et al., 2012]. Our findings confirm the theoretical results and highlight the importance of choosing an appropriate step-size for policy evaluation.

4.1 Linear Quadratic Systems

We first run numerical experiments on the Langevin dynamical systems to examine the behaviour of the trade-off identified in our analysis. The results are shown in Fig. 1. For these experiments, we fix the noise σ 2 = 1 and the cost Q = I . The lines in the plot represent the sample mean ( ˆ V M ( h ) -V ) 2 and the shading represent the standard error of our sample means, computed over 50 independent runs. The plots in Fig. 1 exhibit a clear, U-shaped trade-off, as predicted by our theoretical results.

Fig. 1(a) shows the MSE in a one-dimensional system with T = 8 and a = -1 . The ground truth V is calculated analytically by using Eq. 21 in the Appendix. The figure illustrates how the error changes as we vary the data budget, B = { 2 12 , 2 13 , 2 14 , 2 15 , 2 16 } , and also illustrates the improvement that can be obtained by increasing the budget. As we increase B , both the error and the optimal step size h ∗ , decrease. This result strongly aligns with the analysis shown in Theorem 3.1 and Corollary 3.3. The same analysis is performed with respect to the parameter a , while fixing the data budget, in Fig. 1(b). By increasing the absolute value of the drift coefficient, the diffusion has a smaller impact, thus trajectories have less variability. This leads to a smaller h ∗ for the optimal point of the trade-off.

Fig. 1(c) and 1(d) present the experimental results for both undiscounted finite horizon and discounted infinite horizon multi-dimensional systems. For the finite-horizon setting, V is computed by numerically solving the Riccati Differential Equation; while in infinite-horizon, it is calculated through Lyapunov equation using a standard solver. In our multi-dimensional experiments, we set the

10

3

10

2

10

1

10 0

10 1

h

10

2

10

1

10 0

10 1

(

VM

(

h

)

V

)

2

T=8, a=-1.0

B=4096

B=8192

B=16384

B=32768

B=65536

(a)

n

= 1

, finite horizon, vary

B

10

3

10

2

10

1

10 0

10 1

h

10

4

10

2

10 0

10 2

(

VM

(

h

)

V

)

2

T=8, B=16384

a=-16.0

a=-8.0

a=-2.0

a=-1.0

a=-0.5

a=0.0

(b)

n

= 1

, finite horizon, vary

a

10

3

10

2

10

1

10 0

h

10

2

10

1

10 0

(

VM

(

h

)

V

)

2

T=2, B=16384

A1

A2

A3

A4

A5

(c)

n

= 3

, finite horizon

10

3

10

2

10

1

10 0

h

10 0

10 1

(

VM

(

h

)

V

)

2

T=20, B=32768,

=0.95

A1

A2

A3

A4

A5

(d)

n

= 3

, infinite horizon

dimension n = 3 . We fix all parameters and run our experiments for 5 randomly sampled 3 × 3 dense, stable matrices in each setting. More details on the matrix structure can be found in Appendix D.1. Results in both plots suggest that the impact of the eigenvalues of A on h ∗ is mild and that the eigenvalue-dependent constant terms in Corollary 3.5 only marginally affect the optimal step-size h ∗ , similar to the trend observed in the scalar case with parameter a . In the infinite horizon system, the horizon needs to be large enough to manage truncation error while simultaneously being small enough to collect multiple trajectories. We choose γ large enough such that a good estimate of V can be obtained, and set T = 1 / (1 -γ ) , which is referred to as the effective horizon in the RL literature.

4.2 Nonlinear Systems

Many nonlinear behaviors can be approximated by a high-dimensional linear system, which would be bounded by our theoretical results on the general case of n-dimensional systems, hinting that similar trade-offs could characterize nonlinear systems as well. We empirically show that the trade-off identified in linear quadratic systems carries over to nonlinear systems, with more complex cost functions. We demonstrate it in several simulated nonlinear systems from OpenAI Gym [Brockman et al., 2016], including Pendulum, BipedalWalker and six MuJoCo [Todorov et al., 2012] environments: InvertedDoublePendulum, Pusher, Swimmer, Hopper, HalfCheetah and Ant. We note that the original environments all have a fixed temporal discretization δt , pre-chosen by the designer. To measure the effect of h , we first modify all environments to run at a small discretization δt = 0 . 001 as the proxy to the underlying continuous-time systems. We train a nonlinear policy parameterized by a neural network for each system, by the algorithm DAU [Tallec et al., 2019]. This policy is used to gather episode data from the continuous-time system proxy at intervals of δt = 0 . 001 which are then down-sampled for different h based on the ratio of h and δt . The policy is stable in the sense that it produces reasonable behavior (e.g., pendulum stays mostly upright; Ant walking forward; etc.) and not cause early termination of episodes (e.g., BipedalWalker does not fall), in the continuous-time system proxy. The results of the MSE of Monte-Carlo policy evaluation are shown in Fig. 2. Similar to the linear systems case, we vary the data budget B and see how the MSE changes with the step-size h . The MSE shows a clear minimum for choosing the optimal step-size h ∗ , which generally decreases as the data budget increases. We slightly abuse notations by using V, ˆ V to refer to the true and estimated sum of rewards instead of the cost. The true value of V is approximated by averaging the sum of rewards observed

10

3

10

2

10

1

h

10

4

10

3

10

2

(

VM

(

h

)

V

)

2

Pendulum-v1, T=10

10

3

10

2

10

1

h

10

3

10

2

BipedalWalker-v3, T=10

10

2

10

1

10 0

h

10

7

10

6

10

5

10

4

InvertedDoublePendulum-v2, T=50

10

2

10

1

10 0

h

10

2

10

1

10 0

10 1

Pusher-v2, T=50

10

2

10

1

h

10

4

10

3

(

VM

(

h

)

V

)

2

Swimmer-v3, T=40

10

3

10

2

10

1

h

10

6

10

5

10

4

10

3

10

2

10

1

h

10

1

10 0

HalfCheetah-v3, T=50

10

2

10

1

h

10

1

10 0

10 1

10 2

Sample Budget

B

0

2×

B

0

5×

B

0

10×

B

0

20×

B

0

40×

B

0

at δt = 0 . 001 from 150 k episodes. These environments fall under the finite horizon undiscounted setting. The system (and estimation) horizon T of our experiments is chosen to be the physical time of 1 k steps under the default δt in the original environments (with the exception of 200 steps for Pendulum and 500 steps for BipedalWalker). Please refer to Appendix D.4 for more implementation details, including the setup of B , T , δt , h , training, and the compute resources. These systems are stochastic in the starting state, while having deterministic dynamics. Despite the different settings from our analysis, a clear trade-off is evident in all systems. This suggests that our findings may have broader applicability than the specific conditions under which our theoretical analysis was established.

4.3 Guidelines for Setting Step-Size h

The precise characterization of the MSE in Section 3 can be exploited to set the step-size close to the optimal value without any prior knowledge of the system, provided experiments on a smaller data budget are performed beforehand. Although the optimal step-size h ∗ clearly depends on all quantities characterizing the dynamics and the policy, the technical analysis of the MSE accurately quantifies how h ∗ scales with respect to the data budget B . Specifically, h ∗ ( B ) ≈ c F B -1 / 3 for finite horizon and h ∗ ( B ) ≈ c I B -1 / 5 for infinite horizon, where c F and c I hide the dependencies on the system parameters, exposing only the order in B . This allows us to extrapolate the optimal step-size for the given data budget, using the constant estimated with a smaller one. By evaluating through numerical experiments the performances of different step-sizes on the reduced data budget, it is possible to identify an approximate h ∗ , and subsequently determine c F or c I , which then gives the whole range of optimal values of h with respect to B through the aforementioned relation. Note that this approach does not require prior knowledge of the dynamics, yet it provides a systematic way for setting the step size h for any given scenario.

Figure 3 plots the empirical h ∗ over B for all nonlinear environments, and fitted lines based on the relation from the analysis h ∗ = c F B -1 / 3 , where the constant c F varies with the environment. The plot shows that the scaling of h ∗ w.r.t. B predicted by our analysis is overall a good approximation of the trend observed in the experiments with nonlinear systems, except for negative cases like the InvertedDoublePendulum. This suggests that setting the step-size according to the analysis can yield a value close to optimality.

10 4

10 5

10 6

10

2

10

1

h*

Pendulum-v1

BipedalWalker-v3

InvertedDoublePendulum-v2

Pusher-v2

Swimmer-v3

HalfCheetah-v3

Fitted trade-off

cB

1/3

B

5 Conclusions

We provide a precise characterization of the approximation, estimation and truncation errors incurred by Monte-Carlo policy evaluation in continuous-time linear stochastic dynamical systems with quadratic cost. This analysis reveals a fundamental bias-variance trade-off, modulated by the level of temporal discretization h . Moreover, we confirm in numerical simulations that the analysis accurately captures the trade-off in a precise, quantitative manner. We also demonstrate that the trade-off carries over to non-linear environments such as the popular MuJoCo physics simulation. These findings show that managing the temporal discretization level h can greatly improve the quality of Monte-Carlo policy evaluation under a fixed data budget B . These results have direct implications for practice, as it remains common to adopt a pre-set step-size regardless of the data resources anticipated, which we have seen is a highly sub-optimal approach for a given budget.

The present work serves as a first step toward understanding the effects of temporal resolution on RL. There are several limitations that we would like to address in future work. Our analysis is restricted to Monte-Carlo estimation, while there are more advanced techniques such as temporal difference learning and direct system identification, which may exhibit different behaviours. Also, our focus is policy evaluation. It is worth studying policy optimization to understand the full control setting, which might require relaxing the stability assumption on the closed-loop system since the controllers being iterated might not always be stable. Additionally, it would be interesting to explore non-uniform discretization schemes, such as through an adaptive sampling scheme. From the system's perspective, we have currently analyzed stochastic linear quadratic systems with additive Gaussian noise and noiseless observations. It remains to be determined if the exact characterization is still attainable with other types of noise, in the partially observable case, or with noisy observations. Finally, extending the analysis to non-linear systems would be valuable.

Acknowledgments and Disclosure of Funding

The authors would like to thank Csaba Szepesvari for the helpful discussions. Zichen Zhang gratefully acknowledges the financial support by an NSERC CGSD scholarship and an Alberta Innovates PhD scholarship during this project. He is thankful for the compute resources generously provided by Digital Research Alliance of Canada (and formerly Compute Canada), which is sponsored through the accounts of Martin Jagersand and Dale Schuurmans. Dale Schuurmans gratefully acknowledges funding from the Canada CIFAR AI Chairs Program, Amii and NSERC. Johannes Kirschner gratefully acknowledges funding from the SNSF Early Postdoc.Mobility fellowship P2EZP2_199781.

References

Abbasi-Yadkori and C. Szepesvári. Regret bounds for the adaptive control of linear quadratic systems. In S. M. Kakade and U. von Luxburg, editors, Proceedings of the 24th Annual Conference on Learning Theory , volume 19 of Proceedings of Machine Learning Research , pages 1-26, Budapest, Hungary, 09-11 Jun 2011. PMLR.

Abbasi-Yadkori, N. Lazic, and C. Szepesvári. Model-free linear quadratic control via reduction to expert prediction. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 3108-3117. PMLR, 2019.

Bahl, M. Mukadam, A. Gupta, and D. Pathak. Neural dynamic policies for end-to-end sensory motor learning. Advances in Neural Information Processing Systems , 34, 2020.

C. Baird. Reinforcement learning in continuous time: Advantage updating. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94) , volume 4, pages 2448-2453. IEEE, 1994.

Basei, X. Guo, A. Hu, and Y. Zhang. Logarithmic regret for episodic continuous-time linearquadratic reinforcement learning over a finite-time horizon. Journal of Machine Learning Research , 23(178):1-34, 2022.

Bayraktar and A. D. Kara. Approximate Q-learning for controlled diffusion processes and its near optimality, 2023.

Bijl, J.-W. van Wingerden, T. B. Schön, and M. Verhaegen. Mean and variance of the LQG cost function. Automatica , 67:216-223, 2016.

Bradtke. Reinforcement learning applied to linear quadratic regulation. In S. Hanson, J. Cowan, and C. Giles, editors, Advances in Neural Information Processing Systems , volume 5, 1992.

J. Bradtke and M. O. Duff. Reinforcement learning methods for continuous-time Markov decision problems. In Advances in Neural Information Processing Systems , 1994.

Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI gym. arXiv preprint arXiv:1606.01540 , 2016.

Burns, T. Yu, C. Finn, and K. Hausman. Offline reinforcement learning at multiple frequencies. In Conference on Robot Learning , pages 2041-2051. PMLR, 2023.

Cao and A. Krishnamurthy. Provably adaptive reinforcement learning in metric spaces. Advances in Neural Information Processing Systems , 34, 2020.

Dabney, G. Ostrovski, and A. Barreto. Temporally extended ϵ -greedy exploration. In Proceedings of the International Conference on Learning Representations , 2021.

Dean, H. Mania, N. Matni, B. Recht, and S. Tu. Regret bounds for robust adaptive control of the linear quadratic regulator. Advances in Neural Information Processing Systems , 31, 2018.

Dean, H. Mania, N. Matni, B. Recht, and S. Tu. On the sample complexity of the linear quadratic regulator. Foundations of Computational Mathematics , 20(4):633-679, 2020.

Doya. Reinforcement learning in continuous time and space. Neural computation , 12(1):219-245, 2000.

Eliasmith and P. M. Furlong. Continuous then discrete: A recommendation for building robotic brains. In Conference on Robot Learning , pages 1758-1763. PMLR, 2022.

Fazel, R. Ge, S. Kakade, and M. Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. In International Conference on Machine Learning , pages 1467-1476. PMLR, 2018.

T. Georgiou and A. Lindquist. The separation principle in stochastic control, redux. IEEE Transactions on Automatic Control , 58(10):2481-2494, 2013. doi: 10.1109/TAC.2013.2259207.

Gu, T. Lillicrap, I. Sutskever, and S. Levine. Continuous deep Q-learning with model-based acceleration. In International conference on machine learning , pages 2829-2838. PMLR, 2016.

Huang and Q. Zhu. Infinite-horizon linear-quadratic-Gaussian control with costly measurements. arXiv preprint arXiv:2012.14925 , 2020.

Huang, V. Kavitha, and Q. Zhu. Continuous-time Markov decision processes with controlled observations. In 2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 32-39. IEEE, 2019.

Jia and X. Y. Zhou. Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach. Journal of Machine Learning Research , 23(154):1-55, 2022a.

Jia and X. Y. Zhou. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms. Journal of Machine Learning Research , 23(154):1-55, 2022b.

Kim, J. Shin, and I. Yang. Hamilton-Jacobi deep Q-learning for deterministic continuous-time systems with Lipschitz continuous controls. Journal of Machine Learning Research , 22, 2021.

Krauth, S. Tu, and B. Recht. Finite-time analysis of approximate policy iteration for the linear quadratic regulator. In Advances in Neural Information Processing Systems 32 . 2019.

S. Lakshminarayanan, S. Sharma, and B. Ravindran. Dynamic action repetition for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , 2017.

Lee and R. S. Sutton. Policy iterations for reinforcement learning problems in continuous time and space - fundamental theory and methods. Automatica , 126, 2021.

L. Lewis, D. Vrabie, and K. G. Vamvoudakis. Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers. IEEE Control Systems Magazine , 32(6):76-105, 2012. doi: 10.1109/MCS.2012.2214134.

Lindquist. Linear stochastic systems. SIAM Review , 32(2):325-328, 1990. doi: 10.1137/1032067.

Lutter, S. Mannor, J. Peters, D. Fox, and A. Garg. Value iteration in continuous actions, states and time. In Proceedings of the International Conference on Machine Learning , 2021.

Lutter, B. Belousov, S. Mannor, D. Fox, A. Garg, and J. Peters. Continuous-time fitted value iteration for robust policies. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

M. Metelli, F. Mazzolini, L. Bisi, L. Sabbioni, and M. Restelli. Control frequency adaptation via action persistence in batch reinforcement learning. In Proceedings of the International Conference on Machine Learning , 2020.

Mezzadri. How to generate random matrices from the classical compact groups. arXiv preprint math-ph/0609050 , 2006.

Munos. Policy gradient in continuous time. Journal of Machine Learning Research , 7, 2006.

Munos and P. Bourgine. Reinforcement learning for continuous stochastic control problems. Advances in neural information processing systems , 10, 1997.

Park, J. Kim, and G. Kim. Time discretization-invariant safe action repetition for policy gradient methods. Advances in Neural Information Processing Systems , 34, 2021.

Sharma, A. S. Lakshminarayanan, and B. Ravindran. Learning to repeat: Fine grained action repetition for deep reinforcement learning. In Proceedings of the International Conference on Learning Representations , 2017.

R. Sinclair, S. Banerjee, and C. L. Yu. Adaptive discretization for episodic reinforcement learning in metric spaces. In Proceedings of the ACM Conference on Measurement and Analysis of Computing Systems , 2019.

S. Sutton, D. Precup, and S. Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence , 112:181-211, 1999.

Tallec, L. Blier, and Y . Ollivier. Making deep Q-learning methods robust to time discretization. In International Conference on Machine Learning (ICML) , number 97, pages 6096-6104, 2019.

Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems . IEEE, 2012.

Tu and B. Recht. Least-squares temporal difference learning for the linear quadratic regulator. In International Conference on Machine Learning , pages 5005-5014. PMLR, 2018.

Tu and B. Recht. The gap between model-based and model-free methods on the linear quadratic regulator: An asymptotic viewpoint. In Conference on Learning Theory , pages 3036-3083. PMLR, 2019.

Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, ˙ I. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods , 17: 261-272, 2020. doi: 10.1038/s41592-019-0686-2.

Wang, T. Zariphopoulou, and X. Y. Zhou. Reinforcement learning in continuous time and space: A stochastic control approach. Journal of Machine Learning Research , 21(198):1-34, 2020.

Yildiz, M. Heinonen, and H. Lähdesmäki. Continuous-time model-based reinforcement learning. In International Conference on Machine Learning , pages 12009-12018. PMLR, 2021.

Appendix

A Moment Calculations

Recall that the solution of the SDE in Equation (7), with x (0) = 0 , takes the following form:

An integral part of finding the mean-squared error of the Monte-Carlo estimator is the computation of the moments E [ x ( t ) 2 ] , E [ x ( t ) 4 ] and E [ x ( s ) 2 x ( t ) 2 ] when s ≤ t .

Lemma A.1. Let x ( t ) be the solution of Equation (7) . The second moment of the state variable is

Assuming that s ≤ t , we further get:

Proof.

We start with the second moment E [ x ( t ) 2 ] .

The calculation makes use of the Itô isometry, which can be stated as:

for any stochastic process z ( · ) adapted to the filtration induced by the Wiener process w ( · ) .

(2) Next we compute E [ x ( t ) 4 ] through Itô's formula. Define y ( t ) := ∫ t 0 e -au d w ( u ) , so that d y ( t ) = e -at d w ( t ) . Thus,

for any f ( · ) . By choosing f ( y ) = y 4 :

For the fourth moment, we get:

Therefore, by integration and taking the expectation:

From Equation (14) it holds x ( t ) = σe at y ( t ) so that the second part of the lemma follows.

Note that we computed (i) before. For (ii) it holds:

Therefore, assuming s ≤ t , it holds that:

and

B Calculations of the Mean-Squared Error

B.1 Undiscounted, Finite-Horizon: Proof of Theorem 3.1

Proof. We first note that

where we denote x ( t ) = x 1 ( t ) for simplicity. Next we expand the mean-squared error

For the last equality, note that E [ ˆ V M ( h )] 2 = h 2 ∑ N -1 k,l =0 E [ x 2 ( kh )] E [ x 2 ( lh )] . It remains to compute the expressions. By Lemma A.1 we have for the second moment of the state variable:

Assuming that s ≤ t , from the same lemma we get the following for the fourth moments:

Note that by symmetry, a similar expression follows for s ≥ t .

Using these expressions, for the expected cost we get

We remark that a similar expression was previously obtained in [Bijl et al., 2016, Theorem 3]. Next, the expected estimated cost is

Lastly, it remains to compute the sum

The last equality is a cumbersome calculation that involves nested geometric sums. We verified the result using symbolic computation. For reference we provide the notebooks containing all calculations in the supplementary material. It remains to collect all terms to get the final result.

Proof of Corollary 3.2. When a = 0 , the Langevin equation 7 is reduced to d x ( t ) = σ d w ( t ) . The computation of MSE can be performed similarly to that in the proof of Theorem 3.1, by using the following moment results of Wiener process.

It remains to compute the sums, and collect terms, as in the proof of Theorem 3.1.

It is also worth pointing out that the result of Corollary 3.2 can also be computed by taking the limit of the MSE in Theorem 3.1 when a → 0 . And the resulting MSE from the limit matches the one computed directly as in the proof above. This shows that the MSE in Theorem 3.1 is continuous at a = 0 .

B.2 Undiscounted, Finite-Horizon: Approximate MSE

Proof of Corollary 3.3. For the asymptotic expansion, we use Theorem 3.1 to compute the leading terms in h of the mean-squared error:

Next, we compute explicit upper and lower bounds on the MSE that hold for any h and B . Note that for all x ≤ 0 ,

Hence we can directly bound the E 1 term from Theorem 3.1:

To bound E 2 note that for x ≤ 0 ,

Abbreviating E 3 = h ( e 2 aT -1 ) ( 4 e 2 ah + e 2 aT +1 ) -( e 2 ah -1 ) ( e 2 ah +4 e 2 aT +1 ) T , we have E 2 = σ 4 TE 3 2 a 2 (1 -e 2 ah ) 2 , and hence

To upper bound E 2 , we repetitively use that for all x ≤ 0 , 1 + x ≤ e x ≤ 1 + x + x 2 2 and 1 + x 2 x 3 x

Combining the last two displays yields the claimed upper bound. The lower bound follows along the same lines. Note that the bounds can be refined by including higher-order approximations of e x .

B.3 Undiscounted, Finite-Horizon: Optimal Step Size

Although the exact optimal step size h ∗ can be obtained from Theorem 3.1, such exact h ∗ doesn't have an explicit analytic solution in general. Numerically, we can find h ∗ by searching over step-sizes h m = T/m for m = 1 , . . . , B , provided knowledge of the system parameters a and fixed horizon T or, by finding the root between 0 and 1 of the following equation

where the equation is a simplified form of ∂ ∂h MSE T ( h, B ) = 0 .

From the analysis point of view, a trivial way to see the order of h ∗ in terms of B,a,T is finding the dominated term by using Taylor's expansion for exponential parts (which is true for any h ) in Theorem 3.1. Such asymptotic expansion is given in Corollary 3.3. It is immediate that for h ≥ 1 , both Equation (22) and Equation (23) will blow up. Thus, a small h < 1 is considered to minimize E 1 ( h, T, a ) + E 2 ( h,T,a ) B . Keeping the first term in both Equation (22) and Equation (23) and solving for the optimal h ∗ yields the result in Equation (10).

A more precise approximation of h ∗ than Equation (10) is a minimizer of E 1 ( h, T, a ) + E 2 ( h,T,a ) B truncated at O ( h 3 ) :

where

We can further express Equation (26) in terms of B , as

where the first term is exactly the result in Equation (10).

B.4 Finite-Horizon, Discounted

As stated in Section 3.1, adding discounting in the finite-horizon setting makes the mean-squared error more involved. In the regime where h is small and B is large, a Taylor expansion characterizes the error surface as follows:

The approximation shows only the lowest order terms for 1 / ( hB ) , γ T and h . The derivation is given in Lemma B.1 below. The results shows that main trade-off between h and B persists also for the discounted objective, as long as γ T is treated as a constant relative to h 2 and 1 /hB . In the limit where γ T becomes small (e.g. γ T = o ( h 4 ) ) the nature of the trade-off changes in that the approximation error improves to O ( h 4 ) . This can be understood from the fact that under geometric discounting combined with a decaying process, the sum of N = T/h estimation errors do not suffer a factor N , thereby removing a factor of 1 /h from the (non-squared) approximation error.

Lemma B.1 (Finite-horizon, discounted) . In the finite-horizon with a discount factor γ ∈ (0 , 1] setting, the mean-squared error of the Monte-Carlo estimator is

where

C 4 ( T, γ, a ) is some finite constant of ( T, γ, a ) that includes γ as a factor .

Proof. The proof follows the similar computations as those in the previous proof with a new expected cost as follows. In particular, using Lemma A.1, we get

Furthermore, the expected estimated cost is

Finally, the sum containing the fourth order cross-moments is

While not impossible to calculate on paper, a written derivation is beyond the scope of this work. Instead, we rely on symbolic computation to obtain the expression and corresponding Taylor approximations. The notebooks containing all derivations are provided in the supplementary material.

B.5 Infinite Horizon: Proof of Theorem 3.6

Proof. The proof relies on the decomposition provided in Equation (12). It only remains to compute the following cross term.

Thus, the mean-squared error MSE ∞ ( h, B, T, γ ) = E [ ( ˆ V M ( h ) -V ∞ ) 2 ] is obtained by combining the above computation with Equation (11) and Lemma B.1.

It is worth pointing out the trade-off always exists with or without the assumption γ T = o ( h 4 ) from MSE ∞ ( h, B, T, γ ) . For example, if γ T is constant with respect to h ,

From the above expression, MSE ∞ ( h, B, T, γ ) contains a constant term in this case. For the other cases of γ T , MSE ∞ ( h, B, T, γ ) can be obtained similarly by combining the cross term in the proof of Theorem 3.6, Equation (11) and Lemma B.1. The case where γ T = o ( h 4 ) is particularly interesting because even as γ T → 0 at such a fast rate, there is still a trade-off that never vanishes.

C Vector Case Analysis

C.1 Finite-Horizon, Undiscounted: Proof of Theorem 3.4

Proof. Consider the n-dimensional system that the solution of the trajectory of X ( t ) is

Since A is a diagonalizable matrix, we can decompose A as A = P -1 DP , where P is a invertible matrix (not necessarily to be orthogonal) and D is a diagonal matrix whose diagonal entries ( λ 1 , · · · , λ n ) are corresponding to the eigenvalues of the matrix A . Followed by which, we can decompose the matrix exponential of A as:

Define the 'diagonalized" process ˜ X ( · ) as:

where ˜ W ( s ) is a Wiener process (with dependent components when P is not orthogonal). This implies that X ( · ) = P -1 ˜ X ( · ) .

) ,

To see ˜ X i ( t ) clearly, we denote P = [ p ij ] n i,j =1 , and ˜ X i ( t ) = ( ϕ ( i ) 1 ( t ) , · · · , ϕ ( i ) n ( t )) ⊤ , then ϕ ( i ) l ( t ) = ∑ n j =1 p lj σ ∫ t 0 e λ l ( t -s ) d w ( i ) j ( s ) for each l ∈ { 1 , · · · , n } . Particularly, in such an expression, w ( i ) j ( s are independent Wiener processes for different i or j . Correspondingly, ˜ X ( t ) = ( ϕ 1 ( t ) , · · · , ϕ n ( t )) ⊤ and ϕ l ( t ) = ∑ n j =1 p lj σ ∫ t 0 e λ l ( t -s ) d w j ( s ) for each l ∈ { 1 , · · · , n } , where w j ( s ) are independent Wiener processes for different j .

By trace operation, we can rewrite ˆ V M ( h ) as follows:

where ˆ V M ( h ) = 1 M ∑ M i =1 ∑ N -1 k =0 h ˜ X ( t k ) ˜ X ( t k ) ⊤ ∈ R n × n . Similarly, V T = tr { P -⊤ QP -1 V T } , where V T = ∫ T 0 E [ ˜ X ( t ) ˜ X ( t ) ⊤ ] d t . Therefore, the MSE T ( h, B ) can be written as

For notional simplicity, we denote matrix P -⊤ QP -1 =: B = [ b lj ] n l,j =1 and ˆ V M ( h ) -V T =: C = [ c lj ] n l,j =1 .

Noting the fact that

it is sufficient to find MSE T by only computing E [ c l 1 j 1 c i 2 j 2 ] .

We first introduce the following expectations that are used in the computations. For any s ≤ t

By using the definitions of ˆ V M ( h ) and V T , it is trivial to see for any l, j ∈ { 1 , · · · , n }

̸

̸

where the last equation is due to the fact that for α = β

Thus, for any l 1 , l 2 , j 1 , j 2 ∈ { 1 , · · · , n } ,

̸

where

̸

and and

̸

Note that w ( i ) α and w ( i ) β are independent for α = β . By using the expectations Equations (31) and (33), we can further obtain I 2 ( M,h,T,λ l 1 , λ j 1 , λ l 2 , λ j 2 , α, β ) as

In the following computations, we will use ¯ C and C ( λ l 1 , λ j 1 , λ l 2 , λ j 2 ) to represent some constants that are not depending on h, T, B .

The expectation I 1 ( M,h,T,λ l 1 , λ j 1 , λ l 2 , λ j 2 , α ) is computed exactly the same way as in the proof of Theorem 1 by using the expectation results Equation (31) and Equation (32). Notice that the expectation result Equation (31) (when s = t ) has the same order in t as the expectation Equation (15). Moreover, the two expectations Equation (32) and Equation (17) have the same orders in s and t . Thus, I 1 ( M,h,T,λ l 1 , λ j 1 , λ l 2 , λ j 2 , α ) has the same orders in h, T, B as the scalar MSE, i.e.

The expectation I 2 ( M,h,T,λ l 1 , λ j 1 , λ l 2 , λ j 2 , α, β ) can be computed directly and has the result:

The expectation I 3 ( M,h,T,λ l 1 , λ j 1 , λ l 2 , λ j 2 , α, β ) can be computed as follows:

Thus, the final result is obtained by the expression of MSE in Equation (30), Equation (34) and the above computations. Again, we rely on symbolic computation to obtain the expression and corresponding Taylor approximations and include the notebooks of all derivations in the supplementary material.

The extension from Theorem 3.4 to the discounted finite-horizon results can be done in the same way as in the above proof (add the discount factor γ in ˆ V M ) by using the expectation cost for any λ 1 and λ 2 :

C.2 Proof of Corollary 3.7

Proof. We shall follow the similar proof as in the proof of Theorem 3.4 and the proof of Theorem 3.6. Continuing from Equation (34), in infinite-horizon discounted setting, we have

and

and

Similar arguments as in proof of Theorem 3.4, we can conclude I 1 ( M,h,T,λ l 1 , λ j 1 , λ l 2 , λ j 2 , α ) has the same orders in h, B, T as the MSE result in Theorem 3.6.

Moreover, let C i ( λ l 1 , λ j 1 , λ l 2 , λ j 2 , γ, T ) 's are some constants that depend on λ l 1 , λ j 1 , λ l 2 , λ j 2 , γ, T , then

and

The result in this Corollary is obtained by combining the above results. And we include the notebooks of all derivations in the supplementary material.

C.3 The case when A is a general stable matrix

Lemma C.1 (MSE when A is a general stable matrix ) . Let A be a stable n × n matrix with distinct eigenvalues λ 1 , · · · , λ m and corresponding multiplicities q 1 , · · · , q m . There exist some constants { ¯ C i } i m =1 , ¯ C 0 and C j ( λ 1 , · · · , λ m , γ, T ) 's, such that the mean-squared error of the Monte-Carlo estimator in different setting satisfies

(1) Finite-Horizon undiscounted setting:

where MSE T ( h, B, λ i ) is the mean-squared error of the Monte-Carlo estimator in Theorem 3.1 by replacing the drift a by λ i .

(2) Finite-Horizon discounted setting:

where MSE T ( h, B, γ, λ i ) is the mean-squared error of the Monte-Carlo estimator in Lemma B.1 by replacing the drift a by λ i .

(3) Infinite-Horizon discounted setting:

where MSE ∞ ( h, B, γ, λ i ) is the mean-squared error of the Monte-Carlo estimator in Theorem 3.6 by replacing the drift a by λ i .

Proof. As we can see the proof of Lemma B.1 is based on the proof of Theorem 3.1 with adding a discount factor γ , and the proof of Theorem 3.6 is based on the proof of Lemma B.1 with the decomposition Equation (12). By using the same flow direction, it is sufficient to show the result in case (1) and the results in case (2) and (3) follows.

Consider the decomposition of MSE T in finite-horizon undiscounted setting:

Before the analysis of part 1 and part 2, we will introduce the following mean-squared error notations for the finite-horizon undiscounted scalar case with drift λ i :

[

ˆ

2

M

]

[

ˆ

]

2

(

[

ˆ

]

)

2

where Var

(

For part 1:

h, λ

i

) =

E

V

-

E

V

M

(

and Approximation h, B, λ

i

) =

E

V

M

-

V

T

.

where the second equality is based on Isserlis' theorem and the trace operation.

Notice that E [ ˆ V M ] 2 = h 2 ∑ k,l E [ X ( kh ) ⊤ QX ( kh ) ] E [ X ( lh ) ⊤ QX ( lh ) ] , thus

To analyze the above form, we decompose the matrix A by it Jordan form, i.e. A = P -1 JP for some inevitable matrix P and J = diag ( J i , · · · , J m ) , where J i is the Jordan block corresponding to the eigenvalue λ i .

Notice that e J ( kh -s ) = diag ( e J 1 ( kh -s ) , · · · , e J m ( kh -s ) ) , where

Combining with the fact that for any k, l ,

we can conclude that for any k ≤ l , tr { Q E [ X ( kh ) X ( lh ) ⊤ ]} is a linear combination of L 1 ,i,j and L 2 ,i,j for all i, j , where

where C 1 ,i,j , C i,j are some constants and ˜ q i ∈ { 0 , · · · , q i -1 } , ˜ q j ∈ { 0 , · · · , q j -1 } . For the integral in L 2 ,i,j , as ˜ q i + ˜ q j ≤ n -1 , we can have the inequality:

Since

and all the terms are nonnegative. We drop all terms that include L 2 ,i,j factor and only include the L 2 1 ,i,i with k = l terms in the lower bound of part 1. That is to say, the lower bound of part 1 is ∑ m i =1 q i ¯ C i Var ( h, λ i ) .

The upper bound of part 1 can be obtained by replacing all L 1 ,i,j factors by L 2 ,i,j and use the bound given in Equation (40). This leads to the upper bound for part 1 is ( ¯ C 2 + C 3 ( λ 1 , ··· ,λ m ,T ) O ( T ) ) σ 4 T 2 n +5 Bh + O ( 1 B ) .

For Part 2, let g ( t ) = E [ X ( t ) ⊤ QX ( t ) ] on [0 , T ] . Then E [ ˆ V M ] is the left Riemann sum approximation of g ( t ) , by the property of Riemann approximation,

where

which is a constant depends on λ 1 , · · · , λ m , T . Thus

which has the same order in h as the scalar case in finite-horizon undiscounted setting. Thus the result in Equation (35) is obtained by combining part 1 bounds and part 2 approximation.

As we explained in the beginning of this proof, in the finite-horizon discounted setting, we will follow the similar arguments as in the proof of Equation (35) to obtain result Equation (36).

For the infinite-horizon discounted setting, the corresponding part 1 in the MSE ∞ is the same as the part 1 in MSE T of Equation (36). The part 2 is approximated by using the decomposition Equation (12) and the fact that

To verify V T, ∞ is O ( γ T ) , one can find the bounds of V T, ∞ by using the similar arguments in the above proof of Equation (35) and the following inequality:

Then the components in ∫ ∞ T γ t E [ X ( t ) X ( t ) ⊤ ] d t is lower bounded by ∫ ∞ T r t ( λ i + λ j ) ( e ( λ i + λ j ) t -1 ) d t and upper bounded by ∫ ∞ T r t t n -1 ( λ i + λ j ) ( e ( λ i + λ j ) t -1 ) d t . By the celebrated approximation of incomplete gamma function when T is large, we have

Followed by

we can obtain that V T, ∞ = γ T C ( r, T, λ 1 , · · · , λ m ) .

This result leads to the fact that part 2 is

which coincides with the Var ( hλ i ) in the infinite-horizon discounted scalar case. The results in (3) then follows.

D Complements to Numerical Simulations

The LQR experiments were run on a MacBook pro with an i9 CPU and 16GB of RAM.

D.1 Randomly-sampled Matrices

The matrices A in Figure 1(c) and Figure 1(d), corresponding to controlled multi-dimensional linear systems, are generated according to the procedure described below. It ensures the stability of the resulting system, i.e., that eigenvalues of the sampled matrix are negative.

The procedure works by random sampling an eigendecomposition. It starts with uniformly sampling two eigenvalues from disjoint, bounded intervals, λ 1 ∈ [ -1 . 5 , -1 . 0) and λ 3 ∈ ( -1 . 0 , -0 . 75] . The final eigenvalue is set to be λ 2 = -1 . 0 . Note that since all the eigenvalues sampled are negative, any matrix whose eigenvalues are λ 1 , λ 2 , λ 3 is said to be stable. Next, the eigenvectors are sampled randomly from the classical compact groups detailed in [Mezzadri, 2006]. For that, we randomly sample an orthogonal matrix L using a built-in SCIPY [Virtanen et al., 2020] routine, ORTHO_GROUP.RVS. Now let Λ = diag ( λ 1 , λ 2 , λ 3 ) , the random, dense, stable matrix A is obtained by computing the product A = L ⊤ Λ L .

D.2 Trade-off in LQR with Scaled Identity Matrices

In order to better understand the transition from scalar to vector case in the trade-off of the step-size due to the MSE in Section 3, numerical experiments for the case of identity matrices scaled by a constant are provided in this section. This allows to characterise the role played by the eigenvalues, with respect to the parameter a in the scalar case.

10

3

10

2

10

1

10 0

h

10

3

10

2

10

1

10 0

10 1

(

VM

(

h

)

V

)

2

T=2, B=4096

A

=-0.25

I

3

A

=-0.50

I

3

A

=-1

I

3

A

=-2

I

3

A

=-4

I

3

(a)

n

= 3

, finite horizon

10

2

10

1

10 0

h

10

2

10

1

10 0

10 1

10 2

(

VM

(

h

)

V

)

2

T=32, B=16384,

=0.9

A

=-0.25

I

3

A

=-0.50

I

3

A

=-1

I

3

A

=-2

I

3

A

=-4

I

3

(b)

n

= 3

, infinite horizon

As expected, results in Figure 4 suggest that the trade-off for scaled identity matrices is very similar to the one in the scalar case. In this simple case the eigenvalues that are identical on all dimensions play the same role as the parameter a , i.e., by decreasing them, the trade-off shifts towards a smaller value for the optimal step-size, as suggested also by Figure 1(b).

D.3 Comparison of Empirical and Analytical MSEs in One-dimensional Langevin Systems

Figure 5 compares the empirical MSE in Figures 1(a) and 1(b) with the analytical MSEs, including both the exact and approximate versions. We can observe that the empirical results are consistent with the analysis. The error in approximate MSE becomes noticeable for large h especially when h > 1 , as expected. Nevertheless, the optimal h ∗ remains consistent.

D.4 Implementation Details for Nonlinear Systems Experiments

We summarize the environment-specific parameters in Table 1 for the nonlinear-system experiments.

Compute Resources For training the stable policy for non-mujoco environments, we used a server with one GTX 1080. The training for MuJoCo environments was conducted on a cluster, using a single V100 Volta for each environment. For inference, since we need to run many episodes, we scaled things up on a cluster of CPUs.

10

3

10

2

10

1

10 0

10 1

h

10

2

10

1

10 0

10 1

(

VM

(

h

)

V

)

2

T=8, a=-1.0

B=4096

B=8192

B=16384

B=32768

B=65536

(a) exact MSE, vary

B

10

3

10

2

10

1

10 0

10 1

h

10

2

10

1

10 0

10 1

(

VM

(

h

)

V

)

2

T=8, a=-1.0

B=4096

B=8192

B=16384

B=32768

B=65536

(b) approximate MSE, vary

B

10

3

10

2

10

1

10 0

10 1

h

10

4

10

2

10 0

10 2

(

VM

(

h

)

V

)

2

T=8, B=16384

a=-16.0

a=-8.0

a=-2.0

a=-1.0

a=-0.5

a=0.0

(c) exact MSE, vary

a

10

3

10

2

10

1

10 0

10 1

h

10

5

10

4

10

3

10

2

10

1

10 0

10 1

10 2

(

VM

(

h

)

V

)

2

T=8, B=16384

a=-16.0

a=-8.0

a=-2.0

a=-1.0

a=-0.5

(d) approximate MSE, vary

a

Training Details We used the CDAU algorithm described in [Tallec et al., 2019] to train the policies since we find this algorithm to be robust to time discretization, especially when the environment runs at δt = 0 . 001 . We closely followed the hyper-parameters setup described in Section 2 in the Appendix of [Tallec et al., 2019]. For more details, please refer to their paper, and the code in the supplementary materials.

Inference Details We run the policy for 300 k episodes at the finest time discretization δt = 0 . 001 (600 k in InvertedDoublePendulum and Pusher) and store the reward sequences. The number of episodes is chosen to be sufficient for 30 runs and the largest possible number of trajectories (depending on h and the data budget B ). These data get down-sampled offline for different choices of h . The episodes are randomly shuffled when we vary B .

Peer reviews:
comments: SUMMARY:
The paper discusses the impact of time discretization on estimating cost functions for continuous-time stochastic optimal control problems. The authors consider a discrete-time Monte Carlo estimator based on the left-rule formula for numerical integration. Under the LQG assumption, the authors derive closed-form results for the mean square error (MSE), which depend on the sample size and the integration step size. The authors provide a numerical evaluation based on their modeling assumptions and propose hypotheses on how to extend their findings to general nonlinear systems.

STRENGTHS:
A large class of problems is naturally described as continuous-time systems, making investigations on this topic important. The paper effectively demonstrates how time discretization impacts the estimation of continuous-time quantities, such as the cost function. The work is well written and easy to follow. The results derived in this work are well presented and properly justified.

WEAKNESSES:
I believe the main weakness lies in the significance of the contribution itself. It is to me not particularly surprising that closed-form results for MSE under the LQG assumption can be found. Additionally, it is unclear to me when this method should be used. For instance, if one knows they have an LQG problem, they already know that the value function is a quadratic function and its coefficient can be computed by estimating the dynamics parameter.

Furthermore, the numerical evaluation for the nonlinear systems appears somewhat weak, and I am not sure whether the results can be easily transferred from the linear to the nonlinear regime.

QUESTIONS:
-

LIMITATIONS:
-
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFor the reinforcement learning (RL) setting, many problems are cast into a fixed discrete time sampling of the true underlying system. This paper investigates given some fixed data budget available, what is the optimal sampling rate in terms of temporal resolution to balance the trade-off between approximation and statistical estimation error. Theoretical results are obtained for the exact form of these errors is derived for the MSE of a Monte Carlo in a Langevin process for both finite and infinite horizon setting. This shows that there is an optimal choice in temporal resolution to balance these errors, and this is verified by simulated linear quadratic systems. Similar kind of trade-offs can be found in empirical (nonlinear) environments from gym and MuJoCo.\n\nSTRENGTHS:\nThis is a well-written paper which looks at a problem in value estimation that does not appear to have been explicitly studied on this level before. There is detailed theoretical analysis for the case of a Langevin process, and the results are interpreted in a practical way. The demonstration of these theoretical results in numerical simulations in Section 4.1 is very clear. Verification of similar effects on empirical environments is useful to see.\n\nOverall, this paper contributes to the literature on additional considerations that should be made in continuous time RL setting, and it would be useful for future developments in this area.\n\nWEAKNESSES:\nPractically the policy is varying at the same time, and perhaps the same step size should not be used over the course of training. Therefore it may be that the practical impact on implementation of this paper may be limited.\n\nFor Figure 3, it seems to be misleading to fit a line matching $h = \\mathcal{O}(B^{-1/3})$ for the empirical experiments. Particularly given lines like for that of the InvertedDoublePendulum, clearly a different gradient should be considered. The observation on these empirical experiments are interesting, but there is no need to force the results on these nonlinear systems to match the analysis in the previous section.\n\nIn setting the step-size $h$, this relies on the ability to perform experiments on a smaller data budget first, which may not always be possible in practice.\n\nQUESTIONS:\nHow might an adaptive sampling procedure affect the results?\n\nIt is assumed that the estimation horizon is the same length as the episode, this seems like a strong assumption which does not hold in a lot of applications (i.e. you do not know how long the episode is a priori, nor are the episodes the same lengths). How would the results in finite horizon change in this case?\n\nShould the $h$ on line 165 be $h^*$?\n\nCan you clarify where the MSE_T between line 168 and 169 come from?\n\nThe second term on the right hand side for the MSE_T has denominator $hB$, will this assume $h = \\mathcal{O}(B^{-1/3})$ therefore the second term is $\\mathcal{O}(B^{-2/3})$ hence till tends to 0?\n\nLIMITATIONS:\nThe theoretical results obtained seems to assume a fixed observation/estimation horizon T, that is pre-set, but since episode are not of fixed lengths in RL settings, and indeed may increase as the policy improves, this may affect the validity of the theoretical results.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the impact of time discretization on RL methods in order to improve data efficiency and show that that data efficiency can be significantly improved by leveraging a precise understanding of the trade-off between approximation error and statistical estimation error in value estimation. They conduct a theoretical analysis followed by numerical ones on MuJoCo environments to demonstrate value.\n\nSTRENGTHS:\nQuality and Clarity: The paper is well written and presents a clear and detailed theoretical framework to study the effect of time discretization in terms of data efficiency for control problems. The derivations are easy to follow with necessary background provided in the supplementary materials. \n\nOriginality: Marginal improvements are demonstrated by providing the necessary theoretical foundation which is missing from some prior work like Lutter et al as pointed out by the authors. \n\nSignificance: The paper addresses an important and relevant problem of improving data efficiency for data hungry RL algorithms.\n\nWEAKNESSES:\nSample efficiency is a huge pain point for solving control tasks in real world where collecting large amounts of data is costly, infeasible or even dangerous. A key weakness I found is the fidelity of this approach in real world scenario. I found the theory to be sound, but built on a lot of simplifying assumptions that do not hold true in real world. So although the work is interesting, I am not sure how strong will these results hold on real world data where the noise isn't additive, or the model isn't linear.\n\nQUESTIONS:\nNo specific questions. All concerns are addressed in the paper+supplementary materials.\n\nLIMITATIONS:\nAuthors have done a good job in stating the limitations that I agree with- (a) it would be interesting to see how this work extends to more advanced techniques that MC estimation, like TD learning. (b) Secondly, studying the full control setting that includes policy optimization and, (c) simplifying assumptions like linear models with additive noise are considered, but in real world these assumptions don't hold where the observations are noisy, and systems are non-linear. So how this analysis extends to real world sample efficiency for RL algorithms is unclear from the work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the optimal temporal discretization level to achieve the best Mean Square Error (MSE) in continuous value estimation, considering a fixed budget constraint (number of total Monte-Carlo samples). The paper provides theoretical analysis on a 1-dimensional Langevin dynamical system with quadratic costs and derives the asymptotic optimal step-size under both finite and infinite horizon settings. Extensive numerical studies, including non-linear MuJoCo environments, support the theoretical findings and reveal the bias-variance tradeoff associated with the choice of temporal discretization level.\n\nSTRENGTHS:\nMotivation and clarity:\n- The paper presents a clear motivation and an interesting problem setup regarding temporal resolution in continuous value estimation.\n- The paper is well-written with a clear structure. Key messages, such as the problem setup, objectives, key theoretical results, and numerical result figures, are easily understandable.\n\nQuality of the paper:\n- Section 3 delves deep into a special case of a 1-dimensional Langevin Process, fully characterizing the MSE and providing insights into the optimal step-size.\n- Section 4 builds upon the results from Section 3 and provides extensive numerical studies in various environments.\n- The conclusion of the paper effectively demonstrates the limitations and shows a deep understanding of the topic, as well as potential avenues for future research.\n\nWEAKNESSES:\nLimitation of the scientific impact:\n- My major concern lies in the limitation of the impact, as also mentioned in the paper's conclusion. The optimal resolution results are constrained to Naive Monte-Carlo estimation, which typically serves as a baseline for policy evaluation algorithms and cannot be directly extended to advanced methods such as temporal difference.\n- Although the numerical results demonstrate the bias-variance tradeoff for non-linear systems, the current theoretical results on the 1-dimensional linear dynamical system cannot be directly extended to more complex systems, where exact characterization no longer exists.\n\nQUESTIONS:\nSuggestions:\n1. If simplicity is not our primary criterion, it would be interesting to explore alternative discretization plans beyond uniform discretization.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

comments: SUMMARY:
This paper examines the time discretization for continuous value estimation. By analyzing Monte-Carlo value estimation for LQR systems for both finite-horizon and infinite discounted horizon settings, the authors finds that there is a fundamental trade-off between approximation error and statistical error in value estimation, which indicates there is an optimal choice for time discretization that depends on the data budget. The authors also demonstrate the trade-off in numerical simulation of LQR instances and non-linear mujoco environments.

STRENGTHS:
- The one dimension example presented in the paper is helpful for understanding
- The experiments in the paper clearly support the theoretical analysis and results.

WEAKNESSES:
- My only concern is the practise of the setting or scenarios: people may argue that the $\delta t$ is not randomly chosen and might be fixed or determined by the system itself. Can the authors illustrate some realistic scenarios that we can leverage the methods you proposed.

QUESTIONS:
My questions is the weakness point I listed.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-08-01