# Learning Output Reference Model Tracking for Higher-Order Nonlinear Systems with Unknown Dynamics +
## 4 Validation Case Studies  
### ORM Tracking for a Linear Process  
A first introductory simple example of IMF-AVI for the ORM tracking of a first-order process motivates the more complex validation for the TITOAS process and offers insight into how the IMF-AVI solution scales up with the higher-order processes. Let a scalar discrete-time process discretized at \(T_{s}=0.1\)s be \(x_{k+1}=0.8187x_{k}+0.1813u_{k}\). The continuous-time ORM \(M(s)=1/(s+1)\) ZOH discretized at the same \(T_{s}\) leads to the extended process equivalent to (4), (output equations also given):  
\[ \begin{cases}x_{k+1}=0.8187x_{k}+0.1813u_{k},\\ x_{k+1}^{m}=0.9048x_{k}^{m}+0.09516r_{k},\\ r_{k+1}=r_{k},\\ y_{k}=x_{k},y_{k}^{m}=x_{k}^{m},\end{cases} \quad\Leftrightarrow\mathbf{x}_{k+1 }^{E}=\mathbf{E}(\mathbf{x}_{k}^{E},u_{k}) \tag{31}\]  
where a piece-wise constant reference input generative model is introduced to ensure that the extended process (31) has full measurable state. For data collection, the ORM's output \(y_{k}^{m}\) is collected along with: \(u_{k}\), \(x_{k}\) and the reference input \(r_{k}\). The measurable extended state vector is then \(\mathbf{x}_{k}^{E}=[x_{k},x_{k}^{m},r_{k}]^{\top}\). A discretized version of an integral controller with t.f. \(0.25/s\) at sampling period \(T_{s}=0.1\)s closes the loop of the control system and asymptotically stabilizes it, while calculating the control input \(u_{k}\) based on the feedback error \(e_{k}=r_{k}-y_{k}\). This CS setup is used for collecting transition samples of the form \(D=\{(\mathbf{x}_{k}^{E},\mathbf{u}_{k},\mathbf{x}_{k+1}^{E})\}\). Data is collected for 500 s, with normally distributed random reference inputs having variance \(\sigma_{r}^{2}=0.0951\), modeled as piece-wise constant steps that change their values every 20 s. Normally distributed white noise having variance \(\sigma_{u}^{2}=4.96\) is added on the command \(u_{k}\) at every time step to ensure a proper exploration by visiting as many combinations of states and actions as possible. Exploration has a critical role in the success of the IMF-AVI. A higher amplitude additive noise on \(u_{k}\) increases the chances of converging the approximate VI approach. The state transitions data collection is shown in Figure 1 for the first 1000 samples (100 s). Notice that a reference input modeled as a sequence of constant amplitude steps is used for exploration purposes, for which it may not be possible to write \(r_{k+1}=h^{m}(r_{k})\) as a generative model. To solve this, all transition samples that correspond to the switching times of the reference input are eliminated, therefore, \(r_{k+1}=r_{k}\) can be considered as the piece-wise constant generative model of the reference input. The control objective is to minimize \(J_{MR}^{\infty}(\mathbf{\theta})\) from (5) using the stage cost \(v(\mathbf{x}_{k}^{E})=(y_{k}^{m}-y_{k})^{2}\) (where the outputs obviously depend on the extended states as per (31)), with the discount factor  
Figure 1: Closed-loop state transitions data collection for Example 1: (top) \(y_{k}\) (black), \(r_{k}\) (blue), \(y_{k}^{m}\) (red); (bottom) \(u_{k}\). \(\gamma=0.9\). Thus the overall objective is to find the optimal state-feedback controller \(\mathbf{u}_{k}^{*}=\mathbf{C}^{*}(\mathbf{x}_{k}^{E})\) that makes the feedback CS match the ORM. The Q-function is linearly parameterized as \(Q(\mathbf{x}_{k}^{E},u_{k})=\boldsymbol{\Phi}^{\top}(\mathbf{x}_{k}^{E},u_{k})\boldsymbol {\pi}\), with the quadratic basis functions vector constructed by the unique terms of the Kronecker product of all input arguments of \(Q(\mathbf{x}_{k}^{E},u_{k})\) as  
\[\boldsymbol{\Phi}^{\top}(\mathbf{x}_{k}^{E},u_{k})=[x_{k}^{2},(x_{k}^{m})^{2}, r_{k}^{2},u_{k}^{2},x_{k}x_{k}^{m},x_{k}r_{k},x_{k}u_{k},x_{k}^{m}r_{k},x_{k}^{m} u_{k},r_{k}u_{k}] \tag{32}\]  
with \(\boldsymbol{\pi}\in\mathbb{R}^{10}\). The controller improvement step equivalent to explicitly minimizing the Q-function w.r.t.