SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zichen Zhang*, Johannes Kirschner, Junxi Zhang, Francesco Zanini, Alex Ayoub, Masood Dehghan, Dale Schuurmans
{zichen2,jkirschn,junxi3,fzanini,aayoub,masood1,daes}@ualberta.ca
A default assumption in reinforcement learning (RL) and optimal control is that observations arrive at discrete time points on a fixed clock cycle. Yet, many applications involve continuous-time systems where the time discretization, in principle, can be managed. The impact of time discretization on RL methods has not been fully characterized in existing theory, but a more detailed analysis of its effect could reveal opportunities for improving data-efficiency. We address this gap by analyzing Monte-Carlo policy evaluation for LQR systems and uncover a fundamental trade-off between approximation and statistical error in value estimation. Importantly, these two errors behave differently to time discretization, leading to an optimal choice of temporal resolution for a given data budget. These findings show that managing the temporal resolution can provably improve policy evaluation efficiency in LQR systems with finite data. Empirically, we demonstrate the trade-off in numerical simulations of LQR instances and standard RL benchmarks for non-linear continuous control.
In many real-world applications of control and reinforcement learning, the underlying system evolves continuously in time [Eliasmith and Furlong, 2022]. For instance, a physical system like a robot is naturally modelled as a stochastic dynamical system. Nonetheless, sensor measurements are typically captured at discrete time intervals, which entails choosing the sampling frequency or measurement step-size. This step-size is usually treated as an immutable quantity based on prior measurement design, but it has a significant impact on data efficiency [Burns et al., 2023]. We will see that, from a data-cost perspective, learning can be far more data efficient if it operates at a temporal resolution that is allowed to differ from a prior step-size choice.
In this work, we investigate episodic policy evaluation with a finite data budget to provide a key initial step to addressing broader research questions on the impact of temporal resolution in reinforcement learning. We show that data efficiency can be significantly improved by leveraging a precise understanding of the trade-off between approximation error and statistical estimation error in value estimation — two factors that react differently to the level of temporal discretization. Intuitively, employing a finer temporal resolution leads to a better approximation of the continuous-time system from discrete measurements; however, under a fixed data budget, denser data within each trajectory results in fewer trajectories, leading to increased estimation variance due to system stochasticity. This implies that, for a given data cost, it can be beneficial to increase temporal spacing between recorded data points beyond a pre-set measurement step-size. This holds true for any system with stochastic dynamics, even if the learner has access to exact (noiseless) state measurements.
*All authors contributed equally
The main contributions of this work are twofold. First, we conduct a theoretical analysis focusing on the canonical case of Monte-Carlo value estimation in a Langevin dynamical system (linear dynamics perturbed by a Wiener process) with quadratic instantaneous costs, which corresponds to policy evaluation in linear quadratic control (LQR). To formalize the impact of time discretization on policy evaluation, we present analytical expressions for the mean-squared error that exactly characterize the approximation-estimation trade-off with respect to the step-size parameter. From this trade-off, we derive the optimal step-size for a given Langevin system and characterize its dependence on the data budget. Second, we carry out a numerical study that illustrates and confirms the trade-off in both linear and non-linear systems, including several MuJoCo control environments. The latter also highlights the practical impact of the choice of sampling frequency, which significantly affects the MSE, and we therefore provide recommendations to practitioners for properly choosing the step-size parameter.
There is a sizable literature on reinforcement learning for continuous-time systems [e.g. Doya, 2000, Lee and Sutton, 2021, Lewis et al., 2012, Bahl et al., 2020, Kim et al., 2021, Yildiz et al., 2021]. These previous works largely focus on deterministic dynamics without investigating trade-offs in temporal discretization. A smaller body of work considers learning continuous-time control under stochastic [Baird, 1994, Bradtke and Duff, 1994, Munos and Bourgine, 1997, Munos, 2006], or bounded [Lutter et al., 2021] perturbations, but their objective is to make standard learning methods more robust to small time scales [Tallec et al., 2019], or develop continuous-time algorithms that unify classical methods in discrete-time [Jia and Zhou, 2022a,b], without explicitly addressing temporal discretization. However, we find that managing temporal resolution offers substantial improvements not captured by previous studies.
The LQR setting is a standard framework in control theory and it gives rise to a fundamental optimal control problem [Lindquist, 1990], which has proven to be a challenging scenario for reinforcement learning algorithms [Tu and Recht, 2019, Krauth et al., 2019]. The stochastic LQR considers linear systems driven by additive Gaussian noise with a quadratic cost, minimised using a feedback controller. Although this is a well-understood scenario with a known optimal controller in closed form [Georgiou and Lindquist, 2013], the statistical properties of the long-term cost have only recently been investigated [Bijl et al., 2016]. Our research closely relates to the now extensive literature on reinforcement learning in LQR systems [e.g. Bradtke, 1992, Krauth et al., 2019, Tu and Recht, 2018, Dean et al., 2020, Tu and Recht, 2019, Dean et al., 2018, Fazel et al., 2018, Gu et al., 2016]. These works uniformly focus on the discrete time setting, although the benefits of managing spatial rather than temporal discretization have also been considered [Sinclair et al., 2019, Cao and Krishnamurthy, 2020]. Wang et al. [2020] studies continuous-time LQR, focusing on the exploration problem. Basei et al. [2022] provides a regret bound depending on sampling frequency, for a specific algorithm based on least-squares estimation. Their analysis considers approximation and estimation errors independently, without identifying a trade-off.
There is compelling empirical evidence that managing temporal resolution can greatly improve learning performance [Lakshminarayanan et al., 2017, Sharma et al., 2017, Huang et al., 2019, Huang and Zhu, 2020, Dabney et al., 2021, Park et al., 2021], typically achieved through options [Sutton et al., 1999], a specific instance of which is action persistence, achieved by maintaining a fixed action over multiple time steps (also known as action repetition). Recently, these empirical findings have been supported by an initial theoretical analysis [Metelli et al., 2020], showing that temporal discretization plays a role in determining the effectiveness of fitted Q-iteration. Their analysis does not consider fully continuous systems, but rather remains anchored in a base-level discretization. Furthermore, it only provides worst-case upper bounds, without capturing detailed practical trade-offs. Lutter et al. [2022] discusses the practical trade-off on time discretization but do not provide theoretical support. Bayraktar and Kara [2023] analyzes a trade-off between sample complexity and approximation error, which however requires the state and action spaces of the diffusion process to be discretized to yield an MDP. The two components in the trade-off are analyzed independently, unlike the unified statistical analysis provided in our work, and no exact characterization is presented.
2 Policy Evaluation in Continuous Linear Quadratic Systems
In the classical continuous-time linear quadratic regulator (LQR), a state variable  $X(t) \in \mathbb{R}^n$  evolves over time  $t \geq 0$  according to the following equation:
$$
\mathrm {d} X (t) = \mathbf {A} X (t) \mathrm {d} t + \mathbf {B} U (t) \mathrm {d} t + \sigma \mathrm {d} W (t). \tag {1}
$$
The dynamical model is fully specified by the matrices  $\mathbf{A} \in \mathbb{R}^{n \times n}$ ,  $\mathbf{B} \in \mathbb{R}^{n \times p}$  and the diffusion coefficient  $\sigma$ . The control input  $U(t) \in \mathbb{R}^p$  is given by a fixed policy, and  $W(t)$  is a Wiener process. The state variable  $X(t)$  is fully observed. For simplicity, we assume that the dynamics start at  $X(0) = \vec{0} \in \mathbb{R}^n$  [c.f. Abbasi-Yadkori and Szepesvári, 2011, Dean et al., 2020].
The quadratic cost  $J$  is defined for positive definite, symmetric matrices  $\mathbf{Q} \in \mathbb{R}^{n \times n}$  and  $\mathbf{R} \in \mathbb{R}^{p \times p}$ , a system horizon  $0 < \tau \leq \infty$  and a discount factor  $\gamma \in (0,1]$ :
$$
J _ {\tau} = \int_ {0} ^ {\tau} \gamma^ {t} \left[ X (t) ^ {\top} \mathbf {Q} X (t) + U (t) ^ {\top} \mathbf {R} U (t) \right] d t. \tag {2}
$$
In the following, we consider the class of controllers given by static feedback of the state, i.e.:  $U(t) = KX(t)$  where  $K \in \mathbb{R}^{p \times n}$  is the static control matrix yielding the control input. It is well known that in infinite horizon setting, the optimal control belongs to this class. Given such an input, the LQR in Equation (1) can be further reduced to a linear stochastic dynamical system described by a Langevin equation. Using the definitions  $A \coloneqq \mathbf{A} + \mathbf{B}K$  and  $Q \coloneqq \mathbf{Q} + K^{\top}RK$ , we express both the state dynamics and the cost in a more compact form:
$$
\mathrm {d} X (t) = A X (t) \mathrm {d} t + \sigma \mathrm {d} W (t), \quad J _ {\tau} = \int_ {0} ^ {\tau} \gamma^ {t} X (t) ^ {\top} Q X (t) \mathrm {d} t. \tag {3}
$$
The expected cost w.r.t. the Wiener process is  $V_{\tau} = \mathbb{E}\left[J_{\tau}\right]$ . The policy plays a role in this work solely from its impact in the closed-loop dynamics  $A$ . Equation (3) is what we analyze in the following. We explicitly distinguish the finite-horizon setting where  $\tau < \infty$ ,  $\gamma \leq 1$  and the cost is  $V_{\tau}$ , and the infinite-horizon setting where  $\tau = \infty$ ,  $\gamma < 1$  and the cost is  $V_{\infty}$ . In order not to incur infinite costs in either scenario, a stable closed-loop matrix  $A$  should be assumed. Note that the existence of a stabilizing controller is guaranteed under the standard controllability assumptions in LQR [Fazel et al., 2018, Abbasi-Yadkori et al., 2019, Dean et al., 2020]. Thus the closed-loop stability can be safely assumed.
Monte-Carlo Policy Evaluation Our main objective in policy evaluation is to estimate the expected cost from discrete-time observations. To this end, we choose a uniform discretization of the interval  $[0,T]$  with increment  $h$ , resulting in  $N = T / h$  time points  $t_k \coloneqq kh$  for  $k \in \{0,1,\dots,N\}$ . Here, the estimation horizon  $T$ , such that  $T < \infty$  and  $T \leq \tau$ , is chosen by the practitioner (for simplicity assume that  $T / h$  is an integer). With the  $N$  points sampled from one trajectory, a standard way to approximate the integral in Equation (3) is through the Riemann sum estimator
$$
\hat {J} (h) = \sum_ {k = 0} ^ {N - 1} \gamma^ {t _ {k}} h X (t _ {k}) ^ {\top} Q X (t _ {k}). \tag {4}
$$
To estimate  $V_{\tau}$ , we average  $M$  independent trajectories with cost estimates  $\hat{J}_1, \dots, \hat{J}_M$  to obtain the Monte-Carlo estimator:
$$
\hat {V} _ {M} (h) = \frac {1}{M} \sum_ {i = 1} ^ {M} \hat {J} _ {i} (h) = \frac {1}{M} \sum_ {i = 1} ^ {M} \sum_ {k = 0} ^ {N - 1} \gamma^ {t _ {k}} h X (t _ {k}) ^ {\top} Q X (t _ {k}).
$$
Our primary goal is to understand the mean-squared error of the Monte-Carlo estimator for a fixed system (specified by  $A$ ,  $\sigma$  and  $Q$ ), to inform an optimal choice of the step-size parameter  $h$  for a predetermined data budget  $B = M \cdot N$ .
Note that one degree of freedom remains in choosing  $M$  and  $N$ . For simplicity, we require that in the finite-horizon setting, the estimation grid is chosen to cover the full episode  $[0,\tau]$  which leads to the constraint  $T = \tau = N\cdot h$ . We write the mean-squared error-surface as a function of  $h$  and  $B$ :
$$
\mathrm {M S E} _ {T} (h, B) = \mathbb {E} \left[ \left(\hat {V} _ {M} (h) - V _ {T}\right) ^ {2} \right]. \tag {5}
$$
In the infinite horizon setting, i.e.  $\tau = \infty$ , the estimation horizon  $T$  is a free variable chosen by the experimenter that determines the number of trajectories through  $M = \frac{B}{N} = \frac{Bh}{T}$ . The mean-squared error for the infinite horizon setting is given as a function of  $h$ ,  $B$ , and  $T$ :
$$
\mathrm {M S E} _ {\infty} (h, B, T) = \mathbb {E} \left[ \left(\hat {V} _ {M} (h) - V _ {\infty}\right) ^ {2} \right]. \tag {6}
$$
3 Characterizing the Mean-Squared Error (MSE)
In the following our aim is to characterize the MSE of the Monte-Carlo estimator as a function of the step-size  $h$  and data budget  $B$  (and estimation horizon  $T$  in the infinite horizon setting). Our results uncover a fundamental trade-off for choosing an optimal step-size that leads to a minimal MSE.
One-Dimensional Langevin Process To simplify the exposition while preserving the main ideas, we will first present the results for the 1-dimensional case. The analysis for the vector case exhibits the same quantitative behavior but is significantly more involved. To distinguish the 1-dimensional from the  $n$ -dimensional setting described in Equation (3), we use lower-case symbols. Let  $x(t) \in \mathbb{R}$  be the scalar state variable that evolves according to the following Langevin equation:
$$
\mathrm {d} x (t) = a x (t) \mathrm {d} t + \sigma \mathrm {d} w (t). \tag {7}
$$
Here,  $a \in \mathbb{R}$  is the drift coefficient and  $w(t)$  is a Wiener process with scale parameter  $\sigma > 0$ . We assume that  $a \leq 0$ , i.e. the system is stable (or marginally stable).
The realized sample path in episode  $i = 1, \ldots, M$  is  $x_{i}(t)$  (with starting state  $x(0) = 0$ ) and  $t \in [0, T]$ . The expected cost is
$$
V _ {\tau} = \mathbb {E} \left[ \int_ {0} ^ {\tau} \gamma^ {t} r _ {i} (t) \mathrm {d} t \right] = \int_ {0} ^ {\tau} \gamma^ {t} q \mathbb {E} \left[ x _ {i} ^ {2} (t) \right] \mathrm {d} t, \tag {8}
$$
where  $r_i(t) = qx_i^2(t)$  is the quadratic cost function for a fixed  $q > 0$ . The Riemann sum that approximates the cost realized in episode  $i \in [M]$  becomes  $\hat{J}_i(h) = \sum_{k=0}^{N-1} hqx_i^2(kh)$ . Given data from  $M$  episodes, the Monte-Carlo estimator is  $\hat{V}_M(h) = \frac{1}{M} \sum_{i=1}^{M} \hat{J}_i(h)$ . Since the square of the cost parameter  $q^2$  factors out of the mean-squared error, we set  $q = 1$  in what follows.
3.1 Finite-Horizon Setting
Recall that in the finite-horizon setting we set the system horizon  $\tau$  and estimation horizon  $T$  to be the same. This implies that the estimation grid covers the full episode, i.e.  $hN = T = \tau$ . Perhaps surprisingly, the mean-squared error of the Riemann estimator for the Langevin system (7) can be computed in closed form. The result takes its simplest form in the finite-horizon, undiscounted setting where  $\gamma = 1$  and  $\tau < \infty$ . This result is summarized in the following theorem.
Theorem 3.1 (Finite-horizon, undiscounted MSE). In the finite-horizon, undiscounted setting, the mean-squared error of the Monte-Carlo estimator is
$$
\mathrm {M S E} _ {T} (h, B) = E _ {1} (h, T, a) + \frac {E _ {2} (h , T , a)}{B}, \quad \text {w h e r e}
$$
$$
E _ {1} (h, T, a) = \frac {\sigma^ {4} \left(- 2 a h + e ^ {2 a h} - 1\right) ^ {2} \left(e ^ {2 a T} - 1\right) ^ {2}}{1 6 a ^ {4} \left(e ^ {2 a h} - 1\right) ^ {2}},
$$
$$
E _ {2} (h, T, a) = \frac {\sigma^ {4} T \left[ h \left(e ^ {2 a T} - 1\right) \left(4 e ^ {2 a h} + e ^ {2 a T} + 1\right) - \left(e ^ {2 a h} - 1\right) \left(e ^ {2 a h} + 4 e ^ {2 a T} + 1\right) T \right]}{2 a ^ {2} \left(e ^ {2 a h} - 1\right) ^ {2}}.
$$
While perhaps daunting at first sight, the result exactly characterizes the error surface as a function of the step-size  $h$  and the budget  $B$  for any given Langevin system. The proof involves computing the closed-form expressions for the second and fourth moments of the random trajectories  $x_{i}(t)$  and is provided in Appendices A and B.1.
In the case of marginal stability  $(a = 0)$ , a simpler form of the MSE emerges that is easier to interpret. Taking the limit  $a\to 0$  of the previous expression yields the following result (refer to the discussion and proof in Appendix B.1):
Corollary 3.2 (MSE for marginally stable system). Assume a marginally stable system,  $a = 0$ . Then the mean-squared error of the Monte-Carlo estimator is
$$
\mathbf {M S E} _ {T} (h, B) = \frac {\sigma^ {4} T ^ {2}}{4} \cdot h ^ {2} + \frac {\sigma^ {4} T ^ {5}}{3} \cdot \frac {1}{h B} + \frac {\sigma^ {4} T ^ {2} (- 2 T ^ {2} + 2 h T - h ^ {2})}{3 B}.
$$
The first part of the expression can be understood as a Riemann sum approximation error controlled by the  $h^2$  term. The second part corresponds to the variance term that decreases with the number of episodes as  $\frac{1}{M} = \frac{T}{Bh}$ . The remaining terms are of lower order terms for small  $h$  and large  $B$ . For a fixed data budget  $\bar{B}$ , the step-size  $h$  can be chosen to balance these two terms:
$$
h ^ {*} (B) := \underset {h > 0} {\arg \min } \mathrm {M S E} _ {T} (h, B) \approx T \left(\frac {2}{3 B}\right) ^ {1 / 3}, \tag {9}
$$
where the approximation omits higher order terms in  $1 / B$ . From this, we can compute the optimal number of episodes  $M^{*}\approx \frac{Bh^{*}}{T} = \left(\frac{2}{3}\right)^{1 / 3}B^{2 / 3}$ . We remark that under the assumption  $B\gg 1$ , we also obtain that  $M^{*}\gg 1$ . This is in agreement with the implicit requirement that  $h$  is big enough to consider at least one whole trajectory, i.e.  $h > T / B$ .
Consequently, the mean-squared error for the optimal choice of  $h$  (up to lower order terms in  $1 / B$ ):
$$
\mathrm {M S E} _ {T} \left(h ^ {*}, B\right) \approx 3 \left(3 / 2\right) ^ {1 / 3} \sigma^ {4} T ^ {4} B ^ {- 2 / 3}.
$$
In other words, the optimal error rate as a function of the data budget is  $\mathcal{O}(B^{-2/3})$ . We can further obtain a similar form for  $h^*$  for the general case where  $a \leq 0$ .
Corollary 3.3 (Approximate MSE). The MSE is
$$
\mathbf {M S E} _ {T} (h, B) = c _ {1} (\sigma , a, T) h ^ {2} + \frac {c _ {2} (\sigma , a , T)}{h B} + \mathcal {O} \left(\frac {1}{B} + h ^ {3}\right)
$$
for  $h \to 0$  and  $B \to \infty$ , with system-dependent constants
$$
c _ {1} (\sigma , a, T) = \sigma^ {4} \frac {\left(e ^ {2 a T} - 1\right) ^ {2}}{1 6 a ^ {2}}, \quad c _ {2} (\sigma , a, T) = - \sigma^ {4} \frac {T \left(4 a T - e ^ {4 a T} + e ^ {2 a T} (8 a T - 4) + 5\right)}{8 a ^ {4}}.
$$
Moreover, for any  $h > 0$  and  $B > 0$ ,
$$
c _ {1} h ^ {2} + \frac {c _ {2}}{h B} \lesssim \mathrm {M S E} _ {T} (h, B) \lesssim 4 c _ {1} h ^ {2} + \frac {2 c _ {2}}{h B}
$$
with  $c_{1} = c_{1}(\sigma, a, T)$  and  $c_{2} = c_{2}(\sigma, a, T)$  and the inequalities hold true up to a finite, lower-order polynomial expressions  $\frac{\sigma^4 h}{B}$  poly(h, a, T), given in Appendix B.2.
For the proof please see Appendix B.2. From the corollary, we can derive an optimal step-size, up to lower order terms in  $1 / B$ :
$$
h ^ {*} (B) \approx \left(- \frac {T \left(4 a T - e ^ {4 a T} + e ^ {2 a T} (8 a T - 4) + 5\right)}{a ^ {2} \left(e ^ {2 a T} - 1\right) ^ {2}}\right) ^ {1 / 3} B ^ {- 1 / 3}. \tag {10}
$$
Note that the same  $h^*(B)$  also minimizes the upper bound of the MSE up to a constant factor. The scaling  $\mathrm{MSE}_T(h^*, B) \leq \mathcal{O}(B^{-2/3})$  cannot be improved given the lower bound on the MSE. The derivation is provided in Appendix B.3 where we also include a more precise expression of  $h^*$ .
Discounted Cost Adding discounting  $(\gamma < 1)$  in the finite-horizon setting does not fundamentally change the results; however, it makes the derivation more involved (Appendix B.4).
Vector Case Addressing the general case ( $n$ -dimensional Langevin systems with  $n > 1$ ) for a stable matrix  $A$  requires forgoing the exact form of the MSE. We derive tight bounds on the MSE, both of which are convex functions of  $h$ , thereby narrowing down its behaviour with respect to the step size. The results are presented in Appendix C.3. Although the convex behaviour is proven only for Langevin systems, our experimental results in Section 4 exhibit a similar trade-off for general nonlinear stochastic systems.
Under the additional assumption that the matrix  $A$  is also diagonalisable, we are again able to exactly characterise the MSE with closed-form computations. Diagonalisability is a mild assumption since it can be achieved under a controllable system. Indeed, controllability allows for the free adjustment of the eigenvalues of the closed-loop matrix  $A$  through the choice of the controller  $K$ . The eigenvalues can effectively be chosen to be distinct from each other to ensure a diagonalisable  $A$ . While the explicit form of the MSE is computable, its lengthy formula is not easily interpretable and is thus deferred to Appendix C. The following theorem summarizes the result as a Taylor expansion for small  $h$  and large  $B$ .
Theorem 3.4 (Mean-squared error - vector case). Assume  $A$  is diagonalisable, with eigenvalues  $\Lambda = \{\lambda_1, \dots, \lambda_n\}$ . The mean-squared error of the Monte-Carlo estimator in the finite-horizon, undiscounted setting, is
$$
\mathrm {M S E} _ {T} (h, B) = E _ {1} (h, T, \Lambda) + \frac {E _ {2} (h , T , \Lambda)}{B}, \quad \text {w h e r e}
$$
$$
E _ {1} (h, T, \Lambda) = (\bar {C} _ {1} + C _ {1} (\Lambda) \mathcal {O} (T)) \sigma^ {4} T ^ {2} h ^ {2} + \mathcal {O} (h ^ {3})
$$
$$
\frac {E _ {2} (h , T , \Lambda)}{B} = \left(\overline {{C}} _ {2} + C _ {2} (\Lambda) \mathcal {O} (T)\right) \sigma^ {4} \frac {T ^ {5}}{h B} + \mathcal {O} (1 / B).
$$
The proof, including the exact derivation of the constants  $\overline{C}_1$ ,  $C_1(\Lambda)$ ,  $\overline{C}_2$ ,  $C_2(\Lambda)$ , can be found in Appendix C.1. Note that the terms composing the MSE closely resemble those obtained in the scalar analysis. In fact, when comparing them with the expressions in Equation (22) and Equation (23) (in Appendix B.3), the expression has the same order for  $h$ ,  $B$  and  $T$ . The only difference is that in the vector case, cumbersome eigenvalue-dependent constants are involved, whereas in the scalar case, the result can more easily be expressed in terms of the system parameter  $a$ .
Since the optimal choice for  $h$  is determined by balancing the trade-off between the two terms above,  $E_{1}$  for the approximation error and  $E_{2}$  for the variance, its expression is analogous to the scalar case, as shown by the following corollary.
Corollary 3.5 (Optimal step size - vector case). Under the assumption that  $B \gg 1$ , the optimal step-size for the vector case is given by
$$
h ^ {*} (B) = \left(\frac {\overline {{C}} _ {1} + C _ {1} (\Lambda) \mathcal {O} (T)}{\overline {{C}} _ {2} (\Lambda) + C _ {2} (\Lambda) \mathcal {O} (T)}\right) ^ {1 / 3} T B ^ {- 1 / 3} + o \big (B ^ {- 1 / 3} \big).
$$
The constants in Corollary 3.5 are the same as in Theorem 3.4.
3.2 Infinite-Horizon Setting