# Learning Output Reference Model Tracking for Higher-Order Nonlinear Systems with Unknown Dynamics +
## 3 Solution to the ORM Tracking Problem
 \tag{16}\]  
Meaning that \(Q_{1}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\leq\xi_{1}(\mathbf{x}_{k}^{E},\mathbf{ u}_{k})\). Assume by induction that \(Q_{j-1}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\leq\xi_{j-1}(\mathbf{x}_{k}^{E}, \mathbf{u}_{k})\). Then  
\[ \begin{split}& Q_{j}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})=v(\mathbf{x}_ {k}^{E},\mathbf{u}_{k})+\gamma Q_{j-1}(\mathbf{x}_{k+1}^{E},\mathbf{C}_{j-1}( \mathbf{x}_{k+1}^{E}))\leq\\ &\leq v(\mathbf{x}_{k}^{E},\mathbf{u}_{k})+\gamma Q_{j-1}( \mathbf{x}_{k+1}^{E},\mathbf{\kappa}_{j-1}(\mathbf{x}_{k+1}^{E}))\leq\\ &\leq v(\mathbf{x}_{k}^{E},\mathbf{u}_{k})+\gamma\xi_{j-1}( \mathbf{x}_{k+1}^{E},\mathbf{\kappa}_{j-1}(\mathbf{x}_{k+1}^{E}))=\xi_{j}( \mathbf{x}_{k}^{E},\mathbf{u}_{k}),\end{split}  \tag{17}\]  
which completes the proof. Here, it was used that \(\mathbf{C}_{j-1}(\mathbf{x}_{k}^{E})\) is the optimal controller for \(Q_{j-1}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\) per (14), then, for any other controller \(\mathbf{C}(\mathbf{x}_{k}^{E})\) (in particular it can also be \(\kappa_{j-1}(\mathbf{x}_{k}^{E})\)) it follows that  
\[Q_{j-1}(\mathbf{x}_{k+1}^{E},\mathbf{C}_{j-1}(\mathbf{x}_{k+1}^{E}))\leq Q_{j-1 }(\mathbf{x}_{k+1}^{E},\mathbf{C}(\mathbf{x}_{k+1}^{E})). \tag{18}\]  
**Lemma 2**. : _(_c_) _2019 IEEE [12]) For the sequence {\(Q_{j}\)} from (13), under controllability assumption A1, it is valid that:_  
The expression above leads to \(Q_{j+1}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\geq\tilde{\varsigma}_{j}(\mathbf{x}_{k}^ {E},\mathbf{u}_{k})\). Since by Lemma 1 one has that \(Q_{j}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\geq\tilde{\varsigma}_{j}(\mathbf{x}_{k }^{E},\mathbf{u}_{k})\) it follows that \(Q_{j+1}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\geq Q_{j}(\mathbf{x}_{k}^{E}, \mathbf{u}_{k})\), proving first part of Theorem 1. Any non-decreasing upper bounded sequence must have a limit, thus \(\lim_{j\to\infty}\mathbf{C}_{j}=\mathbf{C}_{\infty}\) and \(\lim_{j\to\infty}Q_{j}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})=Q_{\infty}(\mathbf{x }_{k}^{E},\mathbf{u}_{k})\) with \(\mathbf{C}_{\infty}\) an admissible controller. For any admissible controller \(\boldsymbol{\eta}(\mathbf{x}_{k}^{E})=\mathbf{C}_{\infty}(\mathbf{x}_{k}^{E})\) that is non-optimal if follows from (20) that \(Q^{*}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\leq Q_{\infty}(\mathbf{x}_{k}^{E}, \mathbf{u}_{k})\). Still, part 2 of Lemma 2 states that \(Q_{j}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\leq Q^{*}(\mathbf{x}_{k}^{E}, \mathbf{u}_{k})\) implying \(Q_{\infty}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\leq Q^{*}(\mathbf{x}_{k}^{E}, \mathbf{u}_{k})\). Then from \(Q_{\infty}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\leq Q^{*}(\mathbf{x}_{k}^{E}, \mathbf{u}_{k})\leq Q_{\infty}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\) it must hold true that \(Q_{\infty}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})=Q^{*}(\mathbf{x}_{k}^{E}, \mathbf{u}_{k})\) and \(\mathbf{C}_{\infty}(\mathbf{x}_{k}^{E})=\mathbf{C}^{*}(\mathbf{x}_{k}^{E})\) which proves the second part of Theorem 1. _Comment 2_. : (c) 2019 IEEE [12]) (13) is practically solved in the sense of the OP (11) (either as a linear or nonlinear regression) using a batch (dataset) of transition samples collected from the process using any controller, that is in _off-policy_ mode. While the controller improvement step (14) can be solved either as a regression or explicitly analytically when the expression of \(Q_{j}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\) allows it. Moreover, (13) and (14) can be solved batch-wise in either online or offline mode. When the batch of transition samples is updated with one sample at a time, the VI-scheme becomes adaptive. _Comment 3_. : (c) 2019 IEEE [12]) Theorem 1 proves the VI-based learning convergence of the sequence of Q-functions \(\lim_{j\to\infty}Q_{j}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})=Q^{*}(\mathbf{x}_{k }^{E},\mathbf{u}_{k})\) assuming that the true Q-function parameterization is used. In practice, this is rarely possible, such as, e.g., in the case of LTI systems.