SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 2 gives a bound on the regret of planning optimistically under unknown dynamics. The regret is proportional to the difference in the expected returns for  $\boldsymbol{x}_t$  and  $\boldsymbol{x}_t'$ . Note,  $\| \boldsymbol{x}_t - \boldsymbol{x}_t' \| \propto \beta_n(\delta) \sigma_{n-1}(\boldsymbol{x}_{t-1}, \pi_n(\boldsymbol{x}_{t-1}))$ . Hence, when we have low uncertainty in our predictions, planning optimistically suffers smaller regret. Next, we leverage Lemma 2 to give a sample complexity bound for the Bayesian and frequentist setting.
Bayesian Setting We start by introducing a measure of model complexity as defined by Curi et al. (2020).
$$
\mathcal {M C} _ {N} \left(\boldsymbol {f} ^ {*}\right) := \max  _ {\mathcal {D} _ {1}, \dots , \mathcal {D} _ {N} \subset \mathcal {Z} \times \mathcal {X}} \sum_ {n = 1} ^ {N} \sum_ {\boldsymbol {z} \in \mathcal {D} _ {n}} \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {z}) \| _ {2} ^ {2}. \tag {8}
$$
This complexity measure captures the difficulty of learning  $\pmb{f}^{*}$  given  $N$  trajectories. Mainly, the more complicated  $\pmb{f}^{*}$ , the larger the epistemic uncertainties  $\pmb{\sigma}_{n}$ , and in turn, the larger corresponding  $\mathcal{MC}_N(\pmb{f}^*)$ . Moreover, if the model complexity measure is sublinear in  $N$ , i.e.  $\mathcal{MC}_N(\pmb{f}^*) / N\to 0$  for  $N\rightarrow \infty$ , then the epistemic uncertainties also converge to zero in the limit, which implies
convergence to the true function  $f^*$ . We present our main theoretical result, in terms of the model complexity measure.
Theorem 1. Let Assumption 1 and 3 hold. Then, for all  $N \geq 1$ , with probability at least  $1 - \delta$
$$
\mathbb {E} _ {\mathcal {D} _ {1: N - 1}} \left[ \right. \max  _ {\pi \in \Pi} \mathbb {E} _ {\tau^ {\pi}} \left[ \right. I \left( \right.\boldsymbol {f} _ {\tau^ {\pi}} ^ {*}; \boldsymbol {y} _ {\tau^ {\pi}} \mid \mathcal {D} _ {1: N - 1}) \left. \right]\left. \right] \leq \mathcal {O} \left(\beta_ {N} T ^ {3 / 2} \sqrt {\frac {\mathcal {M C} _ {N} (\boldsymbol {f} ^ {*})}{N}}\right) \tag {9}
$$
Theorem 1 relates the maximum expected information gain at iteration  $N$  to the model complexity of our problem. For deterministic systems, the expectation w.r.t.  $\tau^{\pi}$  is redundant. The bound in Equation (9) depends on the Lipschitz constants, planning horizon, and dimensionality of the state space (captured in  $\beta_{N}$  and  $\mathcal{MC}_N(f^*)$ ). If the right-hand side is monotonically decreasing with  $N$ , Theorem 1 guarantees that the information gain at episode  $N$  is also shrinking with  $N$ , and the algorithm is converging. Empirically, Pathak et al. (2019) show that the epistemic uncertainties go to zero as more data is acquired. In general, deriving a worst-case bound on the model complexity is a challenging and active open research problem. However, in the case of GPs, convergence results can be shown for a very rich class of functions. We show this in the following for the frequentist setting.
Frequentist Setting with Gaussian Process Models We extend our analysis to the frequentist kernelized setting, where  $f^{*}$  resides in a Reproducing Kernel Hilbert Space (RKHS) of vector-valued functions.
Assumption 4. We assume that the functions  $f_{j}^{*}, j \in \{1, \dots, d_{x}\}$  lie in a RKHS with kernel  $k$  and have a bounded norm  $B$ , that is  $\pmb{f}^{*} \in \mathcal{H}_{k,B}^{d_{x}}$ , with  $\mathcal{H}_{k,B}^{d_x} = \{\pmb {f} \mid \| f_j\| _k \leq B, j = 1,\dots ,d_x\}$ .
In this setting, we model the posterior mean and epistemic uncertainty of the vector-valued function  $\pmb{f}^*$  with  $\pmb{\mu}_n(\pmb{z}) = [\mu_{n,j}(\pmb{z})]_{j\leq d_x}$ , and  $\pmb{\sigma}_n(\pmb{z}) = [\sigma_{n,j}(\pmb{z})]_{j\leq d_x}$ , where,
$$
\mu_ {n, j} (\boldsymbol {z}) = \boldsymbol {k} _ {n} ^ {\top} (\boldsymbol {z}) \left(\boldsymbol {K} _ {n} + \sigma^ {2} \boldsymbol {I}\right) ^ {- 1} \boldsymbol {y} _ {1: n} ^ {j},
$$
$$
\sigma_ {n, j} ^ {2} (\boldsymbol {z}) = k (\boldsymbol {x}, \boldsymbol {x}) - \boldsymbol {k} _ {n} ^ {\top} (\boldsymbol {z}) \left(\boldsymbol {K} _ {n} + \sigma^ {2} \boldsymbol {I}\right) ^ {- 1} \boldsymbol {k} _ {n} (\boldsymbol {x}), \tag {10}
$$
Here,  $\pmb{y}_{1:n}^{j}$  corresponds to the noisy measurements of  $f_{j}^{*}$ , i.e., the observed next state from the transitions dataset  $\mathcal{D}_{1:n}$ ,  $\pmb{k}_n = [k(\pmb{z},\pmb{z}_i)]_{i\leq nT}$ ,  $\pmb{z}_i\in \mathcal{D}_{1:n}$ , and  $\pmb{K}_n = [k(\pmb{z}_i,\pmb{z}_l)]_{i,l\leq nT}$ ,  $\pmb{z}_i,\pmb{z}_l\in \mathcal{D}_{1:n}$  is the data kernel matrix. It is known that if  $\pmb{f}^{*}$  satisfies Assumption 4, then Equation (10) yields well-calibrated confidence intervals, i.e., that Assumption 1 is satisfied.
Lemma 3 (Well calibrated confidence intervals for RKHS, Rothfuss et al. (2023)). Let  $\pmb{f}^{*} \in \mathcal{H}_{k,B}^{d_{x}}$ . Suppose  $\pmb{\mu}_n$  and  $\pmb{\sigma}_n$  are the posterior mean and variance of a GP with kernel  $k$ , c.f., Equation (10). There exists  $\beta_n(\delta)$ , for which the tuple  $(\pmb{\mu}_n, \pmb{\sigma}_n, \beta_n(\delta))$  satisfies Assumption 1 w.r.t. function  $\pmb{f}^{*}$ .
Theorem 2 presents our convergence guarantee for the kernelized case to the  $T$ -step reachability set  $\mathcal{R}$  for the policy class  $\pi \in \Pi$ . In particular,  $\mathcal{R}$  is defined as
$$
\mathcal {R} = \left\{\boldsymbol {z} \in \mathcal {Z} \mid \exists (\boldsymbol {\pi} \in \Pi , t \leq T), \text {s . t .}, p \left(\boldsymbol {z} _ {t} = \boldsymbol {z} \mid \boldsymbol {\pi}, \boldsymbol {f} ^ {*}\right) > 0 \right\}
$$
There are two key differences from Theorem 1; (i) we can derive an upper bound on the epistemic uncertainties  $\sigma_{n}$ , and (ii) we can bound the model complexity  $\mathcal{MC}_N(\pmb{f}^*)$ , with the maximum information gain of kernel  $k$  introduced by Srinivas et al. (2012), defined as
$$
\gamma_ {N} (k) = \max  _ {\mathcal {D} _ {1}, \dots , \mathcal {D} _ {N}; | \mathcal {D} _ {n} | \leq T} \frac {1}{2} \log \det  (\boldsymbol {I} + \sigma^ {- 2} \boldsymbol {K} _ {N}).
$$
Theorem 2. Let Assumption 3 and 4 hold, Then, for all  $N\geq 1$  , with probability at least  $1 - \delta$
$$
\max  _ {\boldsymbol {\pi} \in \Pi} \mathbb {E} _ {\boldsymbol {\tau} ^ {\pi}} \left[ \max  _ {\boldsymbol {z} \in \boldsymbol {\tau} ^ {\pi}} \sum_ {j = 1} ^ {d _ {x}} \frac {1}{2} \sigma_ {N, j} ^ {2} (\boldsymbol {z}) \right] \leq \mathcal {O} \left(\beta_ {N} T ^ {3 / 2} \sqrt {\frac {\gamma_ {N} (k)}{N}}\right). \tag {11}
$$
If we relax noise Assumption 3 to  $\sigma$ -sub Gaussian. Then, if Assumption 2 holds, we have for all  $N \geq 1$ , with probability at least  $1 - \delta$ ,
$$
\max  _ {\pi \in \Pi} \mathbb {E} _ {\tau^ {\pi}} \left[ \max  _ {\boldsymbol {z} \in \tau^ {\pi}} \sum_ {j = 1} ^ {d _ {x}} \frac {1}{2} \sigma_ {N, j} ^ {2} (\boldsymbol {z}) \right] \leq \mathcal {O} \left(\beta_ {N} ^ {T} T ^ {3 / 2} \sqrt {\frac {\gamma_ {N} (k)}{N}}\right). \tag {12}
$$
Moreover, if  $\gamma_N(k) = \mathcal{O}$  (polylog(N)), then for all  $z\in \mathcal{R}$ , and  $1\leq j\leq d_{x}$
$$
\sigma_ {N, j} (\boldsymbol {z}) \xrightarrow {\text {a . s .}} 0 f o r N \rightarrow \infty . \tag {13}
$$
We only state Theorem 2 for the expected epistemic uncertainty along the trajectory at iteration  $N$ . For deterministic systems, the expectation is redundant and for stochastic systems, we can leverage concentration inequalities to give a bound without the expectation (see Appendix A for more detail).
For the Gaussian noise case, we obtain a tighter bound by leveraging the change of measure inequality from Kakade et al. (2020, Lemma C.2.) (c.f., Lemma 6 in Appendix A for more detail). In the more general case of sub-Gaussian noise, we cannot use the same analysis. To this end, we use the Lipschitz continuity assumptions (Assumption 2) similar to Curi et al. (2020). This results in comparing the deviation between two trajectories under the same policy and dynamics but different initial states (see Lemma 2). For many systems (even linear) this can grow exponentially in the horizon  $T$ . Accordingly, we obtain a  $\beta_N^T$  term in our bound (Equation (12)). Nonetheless, for cases where the RKHS is of a kernel with maximum information gain  $\gamma_N(k) = \mathcal{O}(\mathrm{polylog}(N))$ , we can give sample complexity bounds and an almost sure convergence result in the reachable set  $\mathcal{R}$  (Equation (13)). Kernels such as the RBF kernel or the linear kernel (kernel with a finite-dimensional feature map  $\phi(x)$ ) have maximum information gain which grows polylogarithmically with  $n$  (Vakili et al. (2021)). Therefore, our convergence guarantees hold for a very rich class of functions. The exponential dependence of our bound on  $T$  imposes the restriction on the kernel class. For the case of Gaussian noise, we can include a richer class of kernels, such as Matern.
In addition to the convergence results above, we also give guarantees on the zero-shot performance of OPAX in Appendix A.5.
We evaluate OPAX on the Pendulum-v1 and MountainCar environment from the OpenAI gym benchmark suite (Brockman et al., 2016), on the Reacher, Swimmer, and Cheetah from the deep mind control suite (Tassa et al., 2018), and a high-dimensional simulated robotic manipulation task introduced by Li et al. (2020). See Appendix B for more details on the experimental setup.
Baselines We implement four baselines for comparisons. To show the benefit of our intrinsic reward, we compare OPAX to (1) a random exploration policy (RANDOM) which randomly samples actions from the action space. As we discuss in Section 3 our choice of objective in Equation (6) is in essence similar to the one proposed by Pathak et al. (2019) and Sekar et al. (2020). Therefore, in our experiments, we compare the optimistic planner with other planning approaches. Moreover, most work on active exploration either uses the mean planner or does not specify the planner (c.f., Section 6). We use the most common planners: (2) mean (MEAN-AE), and (3) trajectory sampling (TS-1) scheme proposed in Chua et al. (2018) (PETS-AE) as our baselines. The mean planner simply uses the mean estimate  $\mu_{n}$  of the well-calibrated model. This is also used in Buisson-Fenet et al. (2020). Finally, we compare OPAX to (4) H-UCRL (Curi et al., 2020), a single-task model-based RL algorithm. We investigate the following three aspects: (i) how fast does active exploration reduce model's epistemic uncertainty  $\sigma_{n}$  with increasing  $n$ , (ii) can we solve downstream tasks with OPAX, and (iii) does OPAX scale to high-dimensional and challenging object manipulation tasks? For our experiments, we use GPs and probabilistic ensembles (PE, Lakshminarayanan et al. (2017)) for modeling the dynamics. For the planning, we either the soft actor-critic (SAC, Haarnoja et al. (2018)) policy optimizer, which takes simulated trajectories from our learned model to train a policy, or MPC with the iCEM optimizer (Pinneri et al., 2021).
How fast does active exploration reduce the epistemic uncertainty? For this experiment, we consider the Pendulum-v1 environment. We sample transitions at random from the pendulum's reachable state-action space and evaluate our model's epistemic uncertainty for varying episodes and baselines. We model the dynamics with both GPs and PE. We depict the result in Figure 1. We conclude that the RANDOM agent is slower in reducing the uncertainty compared to other active exploration methods for both GP and PE models. In particular, from the experiment, we empirically validate Theorem 2 for the GP case and also conclude that empirically even when using PE models, we find convergence of epistemic uncertainty. Moreover, we notice for the PE case that OPAX reaches smaller uncertainties slightly faster than MEAN-AE and PETS-AE. We believe this is due to the additional exploration induced by the optimistic planner.
Can the model learnt through OPAX solve downstream tasks? We use OPAX and other active exploration baselines to actively learn a dynamics model and then evaluate the learned model on downstream tasks. We consider several tasks, (i) Pendulum-v1 swing up, (ii) Pendulum-v1 keep down (keep the pendulum at the stable equilibria), (iii) MountainCar, (iv) Reacher - go to target, (v) Swimmer - go to target, (vi) Swimmer - go away from target (quickly go away from the target position), (vii) Cheetah - run forward, (viii) Cheetah - run backward. For all tasks, we consider PEs, except for (i) where we also use GPs. Furthermore, for the MountainCar and Reacher, we give a reward once the goal is reached. Since this requires long-term planning, we use a SAC policy for these tasks. We use MPC with iCEM for the remaining tasks. We also train H-UCRL on tasks (i) with GPs, and (ii), (iii), (iv), (v), (vii) with PEs. We report the best performance across all episodes.
To make a fair comparison, we use the following evaluation procedure; first, we perform active exploration for each episode on the environment, and then after every few episodes we use the mean estimate  $\pmb{\mu}_n$  to evaluate our learned model on the downstream tasks.
Figure 2 shows that all active exploration variants perform considerably better than the RANDOM agent. In particular, for the MountainCar, the RANDOM agent is not able to solve the task. Moreover, PETS-AE performs slightly worse than the other exploration baselines in this environment. In general, we notice that OPAX always performs well and is able to achieve H-UCRL's performance on all the tasks for which H-UCRL is trained. However, on tasks that are new/unseen for H-UCRL, active exploration algorithms outperform H-UCRL. From this experiment, we conclude two things (1) apart from providing theoretical guarantees, the model learned through OPAX also performs well in downstream tasks, and (2) active exploration agents generalize well to downstream tasks, whereas H-UCRL performs considerably worse on new/unseen tasks. We believe this is because, unlike active exploration agents, task-specific model-based RL agents only explore the regions of the state-action space that are relevant to the task at hand.
Does OPAX scale to high-dimensional and challenging object manipulation tasks? To answer this question, we consider the Fetch Pick & Place Construction environment (Li et al., 2020). We again use the active exploration agents to learn a model and then evaluate the success rate of the learned model in three challenging downstream tasks: (i) Pick & Place, (ii) Throw, and (iii) Flip (see Figure 4). The environment contains a 7-DoF robot arm and four 6-DoF blocks that can be manipulated. In total, the state space is 58-dimensional. The 4-dimensional actions control the end-effector of the robot in Cartesian space as well as the opening/closing of the gripper. We compare OPAX to PETS-AE, MEAN-AE, a random policy as well as CEE-US (Sancaktar et al., 2022). CEE-US is a model-based active exploration algorithm, for which Sancaktar et al. (2022) reports state-of-the-art performance compared to several other active exploration methods. In all three tasks,
OPAX is at least on par with the best-performing baselines, including CEE-US. We run OPAX and all baselines with the same architecture and hyperparameter settings. See Appendix B for more details.
System identification is a broadly studied topic (Åström and Eykhoff, 1971; Schoukens and Ljung, 2019; Schön et al., 2011; Ziemann et al., 2022; Ziemann and Tu, 2022). However, system identification from the perspective of experiment design for nonlinear systems is much less understood (Chiuso and Pillonetto, 2019). Most methods formulate the identification task through the maximization of intrinsic rewards. Common choices of intrinsic rewards are (i) model prediction error or "Curiosity" (Schmidhuber, 1991; Pathak et al., 2017), (ii) novelty of transitions (Stadie et al., 2015), and (iii) diversity of skills (Eysenbach et al., 2018).
A popular choice for intrinsic rewards is mutual information or entropy (Jain et al., 2018; Buisson-Fenet et al., 2020; Shyam et al., 2019; Pathak et al., 2019; Sekar et al., 2020). Jain et al. (2018) propose an approach to maximize the information gain greedily wrt the immediate next transition, i.e., one-step greedy, whereas Buisson-Fenet et al. (2020) consider planning full trajectories. Shyam et al. (2019); Pathak et al. (2019); Sekar et al. (2020) and Sancaktar et al. (2022) consider general Bayesian models, such as BNNs, to represent a probabilistic distribution for the learned model. Shyam et al. (2019) propose using the information gain of the model with respect to observed transition as the intrinsic reward. To this end, they learn an ensemble of Gaussian neural networks and represent the distribution over models with a Gaussian mixture model (GMM). A similar approach is also proposed in Pathak et al. (2019); Sekar et al. (2020); Sancaktar et al. (2022). The main difference between Shyam et al. (2019) and Pathak et al. (2019) lies in how they represent mutual information. Moreover, Pathak et al. (2019) use the model's epistemic uncertainty, that is the
disagreement between the ensemble models as an intrinsic reward. Sekar et al. (2020) link the model disagreement (epistemic uncertainty) reward to maximizing mutual information and demonstrate state-of-the-art performance on several high-dimensional tasks. Similarly, Sancaktar et al. (2022), use the disagreement in predicted trajectories of an ensemble of neural networks to direct exploration. Since trajectories can diverge due to many factors beyond just the model epistemic uncertainty, e.g., aleatoric noise, this approach is restricted to deterministic systems and susceptible to systems with unstable equilibria. Our approach is the most similar to Pathak et al. (2019); Sekar et al. (2020) since we also propose the model epistemic uncertainty as the intrinsic reward for planning. However, we thoroughly and theoretically motivate this choice of reward from a Bayesian experiment design perspective. Furthermore, we induce additional exploration in OPAX through our optimistic planner and rigorously study the theoretical properties of the proposed methods. On the contrary, most of the prior work discussed above either uses the mean planner (MEAN-AE) or does not discuss the planner thoroughly or provide any theoretical results. In general, theoretical guarantees for active exploration algorithms are rather immature (Chakraborty et al., 2023; Wagenmaker et al., 2023) and mostly restrictive to a small class of systems (Simchowitz et al., 2018; Tarbouriech et al., 2020; Wagenmaker and Jamieson, 2020; Mania et al., 2020). To the best of our knowledge, we are the first to give convergence guarantees for a rich class of nonlinear systems.
While our work focuses on the active learning of dynamics, there are numerous works that study exploration in the context of reward-free RL (Jin et al., 2020; Kaufmann et al., 2021; Wagenmaker et al., 2022; Chen et al., 2022). However, most methods in this setting give guarantees for special classes of MDPs (Jin et al., 2020; Kaufmann et al., 2021; Wagenmaker et al., 2022; Qiu et al., 2021; Chen et al., 2022) and result in practical algorithms. On the contrary, we focus on solely learning the dynamics. While a good dynamics model may be used for zero-shot planning, it also exhibits more relevant knowledge about the system such as its stability or sensitivity to external effects. Furthermore, our proposed method is not only theoretically sound but also practical.
We present OPAX, a novel model-based RL algorithm for the active exploration of unknown dynamical systems. Taking inspiration from Bayesian experiment design, we provide a comprehensive explanation for using model epistemic uncertainty as an intrinsic reward for exploration. By leveraging the optimistic in the face of uncertainty paradigm, we put forth first-of-their-kind theoretical results on the convergence of active exploration agents in reinforcement learning. Specifically, we study convergence properties of general Bayesian models, such as BNNs. For the frequentist case of RKHS dynamics, we established sample complexity bounds and convergence guarantees for OPAX for a rich class of functions. We evaluate the efficacy of OPAX across various RL environments with state space dimensions from two to 58. The empirical results corroborate our theoretical findings, as OPAX displays systematic and effective exploration across all tested environments and exhibits strong performance in zero-shot planning for new downstream tasks.
We would like to thank Jonas Hübotter for the insightful discussions and his feedback on this work. Furthermore, we also thank Alex Hägele, Parnian Kassraie, Scott Sussex, and Dominik Baumann for their feedback.
This project has received funding from the Swiss National Science Foundation under NCCR Automation, grant agreement 51NF40 180545, and the Microsoft Swiss Joint Research Center.
Balcan, M.-F., Hanneke, S., and Vaughan, J. W. (2010). The true sample complexity of active learning. Machine learning, 80:111-139.
Berkenkamp, F., Turchetta, M., Schoellig, A., and Krause, A. (2017). Safe model-based reinforcement learning with stability guarantees. NeurIPS, 30.
Biagiotti, L. and Melchiorri, C. (2008). Trajectory Planning for Automatic Machines and Robots. Springer Publishing Company, Incorporated, 1st edition.
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. (2018). JAX: composable transformations of Python+NumPy programs.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym. arXiv preprint arXiv:1606.01540.
Buisson-Fenet, M., Solowjow, F., and Trimpe, S. (2020). Actively learning gaussian process dynamics. In Bayen, A. M., Jabbabaie, A., Pappas, G., Parrilo, P. A., Recht, B., Tomlin, C., and Zeilinger, M., editors, L4DC.
Chakraborty, S., Bedi, A., Koppel, A., Wang, M., Huang, F., and Manocha, D. (2023). STEERING: Stein information directed exploration for model-based reinforcement learning. pages 3949-3978.
Chaloner, K. and Verdinelli, I. (1995). Bayesian experimental design: A review. Statistical science, pages 273-304.
Chen, J., Modi, A., Krishnamurthy, A., Jiang, N., and Agarwal, A. (2022). On the statistical efficiency of reward-free exploration in non-linear rl. Advances in Neural Information Processing Systems, 35:20960-20973.
Chen, Y., Hassani, S. H., Karbasi, A., and Krause, A. (2015). Sequential information maximization: When is greedy near-optimal? In  $COLT$ .
Chiuso, A. and Pillonetto, G. (2019). System identification: A machine learning perspective. Annual Review of Control, Robotics, and Autonomous Systems, 2:281-304.
Chowdhury, S. R. and Gopalan, A. (2017). On kernelized multi-armed bandits. In ICML, pages 844-853. PMLR.
Chua, K., Calandra, R., McAllister, R., and Levine, S. (2018). Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In NeurIPS.
Cover, T. M. and Thomas, J. A. (2006). Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience.
Curi, S., Berkenkamp, F., and Krause, A. (2020). Efficient model-based reinforcement learning through optimistic policy search and planning. NeurIPS, 33:14156-14170.
Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. (2018). Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070.
García, C. E., Prett, D. M., and Morari, M. (1989). Model predictive control: Theory and practice - a survey. Automatica, pages 335-348.