# Learning Output Reference Model Tracking for Higher-Order Nonlinear Systems with Unknown Dynamics +
## 3 Solution to the ORM Tracking Problem  
For unknown extended process dynamics (4), minimization of (5) can be tackled using an iterative model-free approximate Value Iteration (IMF-AVI). A c.f. that extends \(J_{MR}^{\infty}(\mathbf{x}_{k}^{E})\) called the Q-function(or action-value function) is first defined for each state-action pair. Let the Q-function of acting as \(\mathbf{u}_{k}\) in state \(\mathbf{x}_{k}^{E}\) and then following the control (policy) \(\mathbf{u}_{k}=\mathbf{C}(\mathbf{x}_{k}^{E})\) be [12; 46]  
\[Q^{\mathbf{C}}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})=v(\mathbf{x}_{k}^{E}, \mathbf{u}_{k})+\gamma Q^{\mathbf{C}}(\mathbf{x}_{k+1}^{E},\mathbf{C}(\mathbf{ x}_{k+1}^{E})). \tag{7}\]  
The optimal Q-function \(Q^{*}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\) corresponding to the optimal controller obeys [PERSON]'s optimality equation [12; 46]  
\[Q^{*}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})=\min_{\mathbf{C}(.)}\{v(\mathbf{x}_{ k}^{E},\mathbf{u}_{k})+\gamma Q^{\mathbf{C}}(\mathbf{x}_{k+1}^{E},\mathbf{C}( \mathbf{x}_{k+1}^{E}))\}, \tag{8}\]  
where the optimal controller and Q-functions are [12; 46]  
\[\mathbf{u}_{k}^{*}=\mathbf{C}^{*}(\mathbf{x}_{k}^{E})=arg\min_{\mathbf{C}}Q^{ \mathbf{C}}(\mathbf{x}_{k}^{E},\mathbf{u}_{k}),Q^{*}(\mathbf{x}_{k}^{E}, \mathbf{u}_{k})=\min_{\mathbf{C}(.)}Q^{\mathbf{C}}(\mathbf{x}_{k}^{E}, \mathbf{u}_{k}). \tag{9}\]  
Then, for \(J_{MR}^{\infty,*}=\min_{\mathbf{u}}J_{MR}^{\infty}(\mathbf{x}_{k}^{E},\mathbf{ u}_{k})\) it follows that \(J_{MR}^{\infty,*}=Q^{*}(\mathbf{x}_{k}^{E},\mathbf{u}_{k}^{*}=\mathbf{C}^{*}( \mathbf{x}_{k}^{E}))\). Implying that finding \(Q^{*}\) is equivalent to finding the optimal c.f. \(J_{MR}^{\infty,*}\). The optimal Q-function and optimal controller can be found using either Policy Iteration (PoIt) or Value Iteration (VI) strategies. For continuous state-action spaces, IMF-AVI is one possible solution, using different linear and/or nonlinear parameterizations for the Q-function and/or for the controller. NNs are most widely used as nonlinearly parameterized function approximators. As it is well-known, VI alternates two steps: the Q-function estimate update step and the controller improvement step. Several Q-function parameterizations allow for explicit analytic calculation of the improved controller as the following optimization problem (c 2019 IEEE [12])  
\[\tilde{\mathbf{C}}(\mathbf{x}_{k}^{E},\boldsymbol{\pi})=arg\min_{\mathbf{C}}Q ^{\mathbf{C}}(\mathbf{x}_{k}^{E},\mathbf{u}_{k},\boldsymbol{\pi}), \tag{10}\]  
by directly minimizing \(Q^{\mathbf{C}}(\mathbf{x}_{k}^{E},\mathbf{u}_{k},\boldsymbol{\pi})\) w.r.t. \(\mathbf{u}_{k}\), where the parameterization \(\boldsymbol{\pi}\) has been moved from the controller into the Q-function. (10) is the controller improvement step specific to both the PolIt and VI algorithms. In these special cases, it is possible to eliminate the controller approximator and use only one for the Q-function \(Q\). Then, given a dataset \(D\) of transition samples, \(D=\{(\mathbf{x}_{k}^{E},\mathbf{u}_{k},\mathbf{x}_{k+1}^{E})\},k=1\),\(N\) the IMF-AVI amounts to solving the following optimization problem (OP) at every iteration \(j\) (c 2019 IEEE [12])  
\[\boldsymbol{\pi}_{j+1}=arg\min_{\boldsymbol{\pi}}\sum_{k=1}^{N}(Q(\mathbf{x}_{ k}^{E},\mathbf{u}_{k},\boldsymbol{\pi})-v(\mathbf{x}_{k}^{E},\mathbf{u}_{k})- \gamma Q(\mathbf{x}_{k+1}^{E},\tilde{\mathbf{C}}(\mathbf{x}_{k+1}^{E}, \boldsymbol{\pi}_{j}),\boldsymbol{\pi}_{j}))^{2}, \tag{11}\]  
which is a [PERSON] residual minimization problem where the (usually separate) controller improvement step is now embedded inside the OP (11).