# Learning Output Reference Model Tracking for Higher-Order Nonlinear Systems with Unknown Dynamics +
## 4 Validation Case Studies
### Learning State-Feedback Controllers with Linearly Parameterized IMF-AVI  
Details of the LP-IMF-AVI applied to the ORM tracking control problem are next provided. The stage cost is defined \(v(\mathbf{x}_{k}^{E})=(y_{k,1}-y_{k,1}^{m})^{2}+(y_{k,2}-y_{k,2}^{m})2\) and the discount factor in \(J_{MR}^{\infty}\) is \(\gamma=0.95\). The Q-function is linearly parameterized using the basis functions [12]  
\[\begin{split}\mathbf{\Phi}^{\top}(\mathbf{x}_{k}^{E},\mathbf{u}_ {k})=&[(x_{k,1}^{m})^{2},(x_{k,2}^{m})^{2},r_{k,1}^{2},...,x_{k, 6}^{2},u_{k,1}^{2},u_{k,2}^{2},x_{k,1}^{m}x_{k,2}^{m},x_{k,1}^{m}r_{k,1},...,\\ & x_{k,1}^{m}u_{k,2},x_{k,2}^{m}r_{k,1},...,u_{k,1}u_{k,2}]\in \mathbb{R}^{78}.\end{split} \tag{39}\]  
This basis functions selection is inspired by the shape of the quadratic Q-function resulting from LTI processes with LQR-like penalties (see _Comment 7_). It is expected to be a sensible choice since the TITOAS process is a nonlinear one, therefore the quadratic Q-function may under-parameterize the true Q-function. Nevertheless, its computational advantage incentives the testing of such a solution. Notice that the controller improvement step at each iteration of the LP-IMF-AVI is based on explicit minimization of the Q-function. Solving the linear system of equations resulting after setting the derivative of \(Q(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\) w.r.t. \(\mathbf{u}_{k}\) equal to zero, it is obtained that (c) 2019 IEEE [12])  
\[\begin{split}\tilde{\mathbf{u}}_{k}^{*}=\begin{bmatrix}u_{k,1}^{* }\\ u_{k,2}^{*}\end{bmatrix}=\tilde{\mathbf{C}}^{*}(\mathbf{x}_{k}^{E},\mathbf{\pi}_{ j})=\begin{bmatrix}2\pi_{j,11}&\pi_{j,78}\\ \pi_{j,78}&2\pi_{j,12}\end{bmatrix}^{-1}\begin{bmatrix}F_{1}(\mathbf{x}_{k}^ {E})\\ F_{2}(\mathbf{x}_{k}^{E})\end{bmatrix},\\ F_{1}(\mathbf{x}_{k}^{E})=\pi_{j,22}x_{k,1}^{m}+\pi_{j,32}x_{k,2}^{m}+\pi_{j, 41}r_{k,1}+\pi_{j,49}r_{k,2}+\pi_{j,56}x_{k,1}+\\ \pi_{j,62}x_{k,2}+\pi_{j,67}x_{k,3}+\pi_{j,71}x_{k,4}+\pi_{j,74}x_{k,5}+\pi_{j, 76}x_{k,6}=\pi_{j,1}^{\top}\mathbf{x}_{k}^{E},\\ F_{2}(\mathbf{x}_{k}^{E})=\pi_{j,23}x_{k,1}^{m}+\pi_{j,33}x_{k,2}^{m}+\pi_{j, 42}r_{k,1}+\pi_{j,50}r_{k,2}+\pi_{j,57}x_{k,1}+\\ \pi_{j,63}x_{k,2}+\pi_{j,68}x_{k,3}+\pi_{j,72}x_{k,4}+\pi_{j,75}x_{k,5}+\pi_{j, 77}x_{k,6}=\pi_{j,2}^{\top}\mathbf{x}_{k}^{E}.\end{split} \tag{40}\]  
The improved controller is embedded in the system (12) of 70,000 linear equations with 78 unknowns corresponding to the parameters of \(\mathbf{\pi}_{j+1}\in\mathbb{R}^{78}\). This linear system (12) is solved in least squares sense, with each of the 50 iterations of the LP-IMF-AVI. The practical convergence results are shown in Figure 6 for \(\|\mathbf{\pi}_{j+1}-\mathbf{\pi}_{j}\|_{2}\) and for the ORM tracking performance in terms of a normalized c.f. \(J_{test}=1/N(\|y_{k,1}-y_{k,1}^{m}\|_{2}+\|y_{k,2}-y_{k,2}^{m}\|_{2})\) measured for samples over 200 s in the test scenario displayed in Figure 7. The test scenario consists of a sequence of piece-wise constant reference inputs that switch at different moments of time for the azimuth and pitch (\(y_{k,1}\) and \(y_{k,2}\), respectively), to illustrate the existing coupling behavior between the two control channels and the extent to which the learned controller manages to achieve the decoupled behavior requested but the diagonal ORM (c) 2019 IEEE [12]).  
Figure 6: The LP-IMF-AVI convergence on TITOAS (c) 2019 IEEE [12]).  
The best LP-IMF-AVI controller found over the 50 iterations results in \(J_{test}=0.0017\) (tracking results in black lines in Figure 7), which is more than 6 times lower than the tracking performance of the VRFT controller used for transition samples collection, for which \(J_{test}=0.0103\) (tracking results in green lines in Figure 7) (c 2019 IEEE [12]). The convergence of the LP-IMF-AVI parameters is depicted in Figure 8.