# Learning Output Reference Model Tracking for Higher-Order Nonlinear Systems with Unknown Dynamics +
## 3 Solution to the ORM Tracking Problem
For general nonlinear processes of type (1), different function approximators are employed for the Q-function, most commonly using NNs. Then the convergence of the VI Q-learning scheme is to a suboptimal controller and to a suboptimal Q-function, owing to the approximation errors. A generic convergence proof of the learning scheme under approximation errors is next shown, accounting for general Q-function parameterizations [49]. Let the IMF-AVI Algorithm 2 consist of the steps (c) 2019 IEEE [12]). ```
S1: Initialize controller \(\tilde{\mathbf{C}}_{0}\) and Q-function value \(\tilde{Q}_{0}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})=0,\forall(\mathbf{x}_{k}^{E}, \mathbf{u}_{k})\). Initialize iteration \(j=1\) S2: Update the approximate Q-function using Equation (24) S3: Improve the approximate controller using Equation (25) S4: Set \(j=j+1\) and repeat steps S2, S3, until convergence
```  
**Algorithm 2** IMF-AVI. To be detailed as follows:  
S1. Select an initial (not necessarily admissible) controller \(\tilde{\mathbf{C}}_{0}\) and an initialization value \(\tilde{Q}_{0}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})=0,\forall(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\) of the Q-function. Initialize iteration \(j=1\). S2. Use the update equation for the approximate Q-function  
\[\tilde{Q}_{j}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})=\upsilon(\mathbf{x}_{k}^{E}, \mathbf{u}_{k})+\gamma\tilde{Q}_{j-1}(\mathbf{x}_{k+1}^{E},\tilde{\mathbf{C}}_{ j-1}(\mathbf{x}_{k+1}^{E}))+\delta_{j}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\] \[=\min_{\mathbf{u}}\{\upsilon(\mathbf{x}_{k}^{E},\mathbf{u}_{k})+ \gamma\tilde{Q}_{j-1}(\mathbf{x}_{k+1}^{E},\mathbf{u})\}+\delta_{j} \tag{24}\]  
S3. Improve the approximate controller using  
\[\tilde{\mathbf{C}}_{j}(\mathbf{x}_{k}^{E})=arg\min_{\mathbf{u}}\tilde{Q}_{j}( \mathbf{x}_{k}^{E},\mathbf{u}) \tag{25}\]  
S4. Set \(j=j+1\) and repeat steps S2, S3, until convergence. _Comment 4_. : (c) 2019 IEEE [12]) In Algorithm 2, the sequences \(\{\tilde{\mathbf{C}}_{j}(\mathbf{x}_{k}^{E})\}\) and \(\{\tilde{Q}_{j}(\mathbf{x}_{k}^{E},\mathbf{u})\}\) are approximations of the true sequences \(\{\mathbf{C}_{j}(\mathbf{x}_{k}^{E})\}\) and \(\{Q_{j}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\}\). Since the true Q-function and controller parameterizations are not generally known, (24) must be solved in the sense of the OP (11) with respect to the unknown \(\tilde{Q}_{j}\), in order to minimize the residuals \(\delta_{j}\) at each iteration. If the true parameterizations of the Q-function and of the controller were known, then \(\delta_{j}=0\) and the IMF-AVIupdates (24), (25) coincide with (13), (14), respectively. Next, let the following assumption hold. A4. ([12]) There exist two positive scalar constants \(\underline{\psi},\overline{\psi}\) such that \(0<\underline{\psi}\leq 1\leq\overline{\psi}<\infty\), ensuring  
\[ \begin{split}\min_{\mathbf{u}}\{\underline{\psi}v(\mathbf{x}_{k}^{ E},\mathbf{u}_{k})+\gamma\tilde{Q}_{j-1}(\mathbf{x}_{k+1}^{E},\mathbf{u})\} \leq\tilde{Q}_{j}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\leq\\ \min_{\mathbf{u}}\{\overline{\psi}v(\mathbf{x}_{k}^{E},\mathbf{u} _{k})+\gamma\tilde{Q}_{j-1}(\mathbf{x}_{k+1}^{E},\mathbf{u})\}.\end{split}  \tag{26}\]  
_Comment 5_. : (c) 2019 IEEE [12]) Inequalities from (26) account for nonzero positive or negative residuals \(\delta_{j}\), i.e., for the approximation errors in the Q-function, since \(\tilde{Q}_{j}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\) can over- or under-estimate \(\min_{\mathbf{u}}\{v(\mathbf{x}_{k}^{E},\mathbf{u}_{k})+\gamma\tilde{Q}_{j-1 }(\mathbf{x}_{k+1}^{E},\mathbf{u})\}\) in (24). \(\underline{\psi},\overline{\psi}\) can span large intervals (\(\underline{\psi}\) close to \(0\) and \(\overline{\psi}\) very large). The hope is that, if \(\underline{\psi},\overline{\psi}\) are close to \(1\)--meaning low approximation errors--then the entire IMF-AVI process preserves \(\tilde{\delta_{j}}\approx 0\). In practice, this amounts to using high performance approximators.