# Learning Output Reference Model Tracking for Higher-Order Nonlinear Systems with Unknown Dynamics +
## 3 Solution to the ORM Tracking Problem
More explicitly, for a linear parameterization \(Q(\mathbf{x}_{k}^{E},\mathbf{u}_{k},\boldsymbol{\pi})=\boldsymbol{\Phi}^{\top} (\mathbf{x}_{k}^{E},\mathbf{u}_{k})\boldsymbol{\pi}\) using a set of \(n_{\Phi}\) basis functions of the form \(\boldsymbol{\Phi}^{\top}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})=[\Phi_{1}(\mathbf{ x}_{k}^{E},\mathbf{u}_{k}),...,\Phi_{n_{\Phi}}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})]\), the least squares solution to (11) is equivalent to solving the following over-determined linear system of equations w.r.t. \(\boldsymbol{\pi}_{j+1}\) in the least-squares sense (c 2019 IEEE [12]):  
\[ \begin{bmatrix}\boldsymbol{\Phi}^{\top}(\mathbf{x}_{1}^{E},\mathbf{u}_{1})\\...\\ \boldsymbol{\Phi}^{\top}(\mathbf{x}_{N}^{E},\mathbf{u}_{N})\end{bmatrix} \ \boldsymbol{\pi}_{j+1}= \begin{bmatrix}v(\mathbf{x}_{1}^{E},\mathbf{u}_{1})+\gamma\boldsymbol{\Phi}^{ \top}(\mathbf{x}_{2}^{E},\tilde{\mathbf{C}}(\mathbf{x}_{2}^{E},\boldsymbol{ \pi}_{j}))\boldsymbol{\pi}_{j}\\...\\ v(\mathbf{x}_{N^{E}}^{E},\mathbf{u}_{N})+\gamma\boldsymbol{\Phi}^{\top}( \mathbf{x}_{N+1}^{E},\tilde{\mathbf{C}}(\mathbf{x}_{N+1}^{E},\boldsymbol{ \pi}_{j}))\boldsymbol{\pi}_{j}\end{bmatrix} . \tag{12}\]  
Concluding, starting with an initial parameterization \(\boldsymbol{\pi}_{0}\), the IMF-AVI approach with linearly parameterized Q-function that allows explicit controller improvement calculation as in (10), embeds both VI steps into solving (12). Linearly parameterized IMF-AVI (LP-IMF-AVI) will be validated in the case study and compared to nonlinearly parameterized IMF-AVI (NP-IMF-AVI). Convergence of the generally formulated IMF-AVI is next analysed under approximation errors (c 2019 IEEE [12]). _IMF-AVI Convergence Analysis with Approximation Errors for ORM Tracking_  
The proposed iterative model-free VI-based Q-learning Algorithm 1 consists of the next steps (c) 2019 IEEE [12]). ```
S1: Initialize controller \(\mathbf{C}_{0}\) and the Q-function value to \(Q_{0}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})=0\), initialize iteration index \(j=1\) S2: Use one step backup equation for the Q-function as in (13) S3: Improve the controller using the Equation (14) S4: Set \(j=j+1\) and repeat steps S2, S3, until convergence
```  
**Algorithm 1** VI-based Q-learning. To be detailed as follows:  
S1. Select an initial (not necessarily admissible) controller \(\mathbf{C}_{0}\) and an initialization value \(Q_{0}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})=0\) of the Q-function. Initialize iteration \(j=1\). S2. Use one step backup equation for the Q-function  
\[ \begin{split} Q_{j}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})& =v(\mathbf{x}_{k}^{E},\mathbf{u}_{k})+\gamma Q_{j-1}(\mathbf{x}_ {k+1}^{E},\mathbf{C}_{j-1}(\mathbf{x}_{k+1}^{E}))\\ &=\min_{\mathbf{u}}\{v(\mathbf{x}_{k}^{E},\mathbf{u}_{k})+\gamma Q _{j-1}(\mathbf{x}_{k+1}^{E},\mathbf{u})\}\end{split}  \tag{13}\]  
S3. Improve the controller using the equation  
\[\mathbf{C}_{j}(\mathbf{x}_{k}^{E})=arg\min_{\mathbf{u}}Q_{j}(\mathbf{x}_{k}^{ E},\mathbf{u}). \tag{14}\]  
S4. Set \(j=j+1\) and repeat steps S2, S3, until convergence. **Lemma 1**. : _(_c_) _2019 IEEE [12]) For an arbitrary sequence of controllers \(\{\kappa_{j}\}\) define the VI-update for extended c.f. \(\xi_{j}\) as [48]_  
\[\xi_{j+1}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})=v(\mathbf{x}_{k}^{E},\mathbf{u}_{ k})+\gamma\xi_{j}(\mathbf{x}_{k+1}^{E},\mathbf{\kappa}_{j}(\mathbf{x}_{k+1}^{E})). \tag{15}\]  
_If \(Q_{0}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})=\xi_{0}(\mathbf{x}_{k}^{E},\mathbf{u }_{k})=0\), then \(Q_{j}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\leq\xi_{j}(\mathbf{x}_{k}^{E},\mathbf{ u}_{k})\)._  
Proof. : It is valid that  
\[ \begin{split}& Q_{1}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})=v(\mathbf{x}_{k}^{E}, \mathbf{u}_{k})+\gamma\overbrace{Q_{0}(\mathbf{x}_{k+1}^{E},\mathbf{C}_{0}( \mathbf{x}_{k+1}^{E}))}^{0}=\\ &=v(\mathbf{x}_{k}^{E},\mathbf{u}_{k})+\gamma\overbrace{\xi_{0 }(\mathbf{x}_{k+1}^{E},\mathbf{\kappa}_{0}(\mathbf{x}_{k+1}^{E}))}^{0}=\xi_{1} (\mathbf{x}_{k}^{E},\mathbf{u}_{k}).\end{split}