# Learning Output Reference Model Tracking for Higher-Order Nonlinear Systems with Unknown Dynamics +
## 2 Output Model Reference Control for Unknown Dynamics Nonlinear Processes
### Output Reference Model Control Problem Definition
to be minimized starting with \(\mathbf{x}_{0}\) be [6; 12; 46]  
\[J_{MR}^{\infty}(\mathbf{x}_{0}^{E},\mathbf{\theta})=\sum_{k=0}^{\infty}\gamma^{k} \|\mathbf{y}_{k}^{m}(\mathbf{x}_{k}^{E})-\mathbf{y}_{k}(\mathbf{x}_{k}^{E}, \mathbf{\theta})\|_{2}^{2}=\sum_{k=0}^{\infty}\gamma^{k}\|\mathbf{\epsilon}_{k}( \mathbf{x}_{k}^{E},\mathbf{\theta})\|_{2}^{2}. \tag{5}\]  
In (5), the discount factor \(0<\gamma\leq 1\) sets the controller's horizon, \(\gamma<1\) is usually used in practice to guarantee learning convergence to optimal control. \(\|\mathbf{x}\|_{2}=\sqrt{\mathbf{x}^{\top}}\mathbf{x}\) is the Euclidean norm of the column vector \(\mathbf{x}\). \(v_{MR}=\|\mathbf{y}_{k}^{m}(\mathbf{x}_{k}^{E})-\mathbf{y}_{k}(\mathbf{x}_{k}^ {E})\|_{2}^{2}\) is the stage cost where measurable \(\mathbf{y}_{k}\) depends via unknown \(\mathbf{g}\) in (1) on \(\mathbf{x}_{k}\) and \(v_{MR}\) penalizes the deviation of \(\mathbf{y}_{k}\) from the ORM's output \(\mathbf{y}_{k}^{m}\). In ORM tracking, the stage cost does not penalize the control effort with some positive definite function \(W(\mathbf{u}_{k})>0\) since the ORM tracking instills an inertia on the CS that indirectly acts as a regularizer on the control effort. Secondly, if the reference inputs \(\mathbf{r}_{k}\) do not set to zero, the ORM's outputs also do not. For most processes, the corresponding constant steady-state control will be non-zero, hence making \(J_{MR}^{\infty}(\mathbf{\theta})\) infinite when \(\gamma=1\)[12; 46]. Herein, \(\mathbf{\theta}\in\mathbb{R}^{\eta_{\theta}}\) parameterizes a nonlinear state-feedback admissible controller [6] defined as \(\mathbf{u}_{k}\xrightarrow{\Delta}\mathbf{C}(\mathbf{x}_{k}^{E},\mathbf{\theta})\), which used in (4) shows that all CS's trajectories depend on \(\mathbf{\theta}\). Any stabilizing controller sequence (or controller) rendering a finite c.f. is called admissible. A finite \(J_{MR}^{\infty}\) holds if \(\varepsilon_{k}\) is a square-summable sequence, ensured by an asymptotically stabilizing controller if \(\gamma=1\) or by a stabilizing controller if \(\gamma<1\). \(J_{MR}^{\infty}(\mathbf{\theta})\) in (5) is the value function of using the controller \(\mathbf{C}(\mathbf{\theta})\). Let the optimal controller \(\mathbf{u}_{k}^{*}=\mathbf{C}(\mathbf{x}_{k}^{E},\mathbf{\theta}^{*})\) that minimizes (5) be [12; 46]  
\[\mathbf{\theta}^{*}=arg\min_{\mathbf{\theta}}J_{MR}^{\infty}(\mathbf{x}_{0}^{E},\mathbf{ \theta}). \tag{6}\]  
Tracking a nonlinear ORM can also be used, however, tracking a linear one renders highly desirable indirect feedback linearization of the CS, where a linear CS's behavior generalizes well in wide operating ranges [1]. Then the ORM tracking control problem of this work should make \(v_{MR}\approx 0\) when \(\mathbf{r}_{k}\) drives both the CS and the ORM. Under classical control rules, following _Comment 1_, the process time delay and non-minimum-phase (NMP) character should be accounted for in \(\mathbf{M}(z)\). However, the NMP zeroes make \(\mathbf{M}(z)\) non-invertible in addition to requiring their identification, thus placing a burden on the subsequent VRFT IO control design [47]. This motivates the MP assumption on the process. Depending on the learning context, the user may select a piece-wise constant generative model for the reference input signal such as \(\mathbf{r}_{k+1}=\mathbf{r}_{k}\), or a ramp-like model, a sine-like model, etc. In all cases, the states of the generative model are known, measurable and need to be introduced in the extended state vector, to fulfill the Markov property of the extended state-space model. In many practical applications, for the ORM tracking problem, the CS's outputs are required to track the ORM's outputs when both the ORM and the CS are driven by the piece-wise constant reference input signal expressed by a generative model of the form \(\mathbf{r}_{k+1}=\mathbf{r}_{k}\). This generative model will be used subsequently in this paper for learning ORM tracking controllers.