## 2 Output Model Reference Control for Unknown Systems
To extend the process (1) with the ORM (2), we consider the reference input \(\mathbf{r}_{k}\) as a set of measurable exogenous signals that evolve according to \(\mathbf{r}_{k+1}=\mathbf{h}^{m}(\mathbf{r}_{k})\), with unknown \(\mathbf{h}^{m}:\Re^{m}\rightarrow\Re^{m}\) but measurable \(\mathbf{r}_{k}\). Piecewise constant \(\mathbf{r}_{k}\) can be modeled for example as \(\mathbf{r}_{k+1}=\mathbf{r}_{k}\) and it will be used throughout this paper. Then the extended state-space model with output equations is  
\[ \begin{split}\mathbf{x}_{k+1}^{E}=\left[\begin{array}{c}\mathbf{x }_{k+1}\\ \mathbf{x}_{k+1}^{m}\\ \mathbf{r}_{k+1}\end{array}\right]=\left[\begin{array}{c}\mathbf{f}(\mathbf{ x}_{k^{\prime}},\mathbf{u}_{k})\\ \mathbf{f}^{m}(\mathbf{x}_{k^{\prime}}^{m},\mathbf{r}_{k})\\ \mathbf{h}^{n}(\mathbf{r}_{k})\end{array}\right]=\mathbf{F}(\mathbf{x}_{k}^{E },\mathbf{u}_{k}),\mathbf{x}_{k}^{E}\in X^{E}=X\times X_{m}\times R_{m},\\ \mathbf{y}_{k}=\mathbf{g}(\mathbf{x}_{k}^{E}),\\ \mathbf{y}_{k}^{m}=\mathbf{g}^{m}(\mathbf{x}_{k}^{E}).\end{split}  \tag{3}\]  
The ORM tracking control problem is formulated in an optimal control framework. Let the infinite horizon cost function (c.f.) to be minimized starting with \(\mathbf{x}_{i}\) be [36]  
\[J(\mathbf{x}_{i}^{E},U_{i,\infty})=\sum_{k=i}^{\infty}\gamma^{k-i}V(\mathbf{x} _{k}^{E},\mathbf{u}_{k}),U_{i,\infty}=\{\mathbf{u}_{i},\ldots,\mathbf{u}_{ \infty}\}, \tag{4}\]  
where \(i\) indexes the starting time for \(\mathbf{x}_{i}^{E}\), the discount factor \(0<\gamma\leq 1\) ensures the convergence of \(J(\mathbf{x}_{i}^{E},U_{i,\infty})\)[23] and sets the controller's (or interacting agent's) horizon, the stage cost \(V>0\) depends on \(\mathbf{x}_{k}^{E}\) and \(\mathbf{u}_{k}\) and captures the distance relative to some pre-specified learning goal (target) usually constant in many applications. The unknown control inputs \(\mathbf{u}_{i}\), \(\mathbf{u}_{i+1}\),..., should minimize \(J(\mathbf{x}_{i}^{E},U_{i,\infty})\). A control sequence (or a controller) rendering a finite c.f. are called _admissible_. ORM tracking control requires that the undisturbed process output \(\mathbf{y}_{k}\) (also the control system output) tracks the ORM's output \(\mathbf{y}_{k}^{m}=\mathbf{M}(z)\mathbf{r}_{k}\). For stage cost \(V_{MR}=\left\|\mathbf{y}_{k}^{m}(\mathbf{x}_{k}^{E})-\mathbf{y}_{k}(\mathbf{x }_{k}^{E})\right\|_{2}^{2}\) in Equation (4) (measurable \(\mathbf{y}_{k}\) depends via unknown \(\mathbf{g}(\ )\) on \(\mathbf{x}_{k}\), but not on \(\mathbf{x}_{k+1}\)), we introduce the discounted infinite-horizon model reference tracking c.f. \[J_{MR}^{\infty}(\mathbf{x}_{0}^{E},\Theta)=\sum_{k=0}^{\infty}\gamma^{k} \big{\|}\mathbf{y}_{k}^{m}(\mathbf{x}_{k}^{E})-\mathbf{y}_{k}(\mathbf{x}_{k}^ {E},\Theta)\big{\|}_{2}^{2}=\sum_{k=0}^{\infty}\gamma^{k}\big{\|}\epsilon_{k}( \mathbf{x}_{k}^{E},\Theta)\big{\|}_{2}^{2}, \tag{5}\]  
where \(\epsilon_{k}(\mathbf{x}_{k}^{E},\Theta)\) is the model reference tracking error vector, \(\Theta\in\Re^{n_{\Theta}}\) is a parameterization of a nonlinear feedback admissible controller [23] defined as \(\mathbf{u}_{k}\stackrel{{ def}}{{=}}\mathbf{C}(\mathbf{x}_{k}^{E},\Theta)\), which used in Equation (5) reflects the influence of \(\Theta\) on all system trajectories outcomes. This controller coupled with Equation (3) ensures that the output of Equation (1) tracks the ORM's output. \(J_{MR}^{\infty}\) in (5) also serves as the value function of using the controller \(\mathbf{C}\). For finite \(J_{MR}^{\infty}\) when \(\gamma=1\), \(\epsilon_{k}\) must be a square-summable sequence which can be obtained with an asymptotically stabilizing controller that ensures \(\lim\limits_{k\rightarrow\infty}\left\|\mathbf{y}_{k}^{m}(\mathbf{x}_{k}^{E})- \mathbf{y}_{k}(\mathbf{x}_{k}^{E})\right\|_{2}^{2}=0\). In the general case when \(\gamma<1\), \(J_{MR}^{\infty}\) will be finite with any stabilizing controller that renders a finite upper bounded \(\epsilon_{k}\). Herein, admissible controller for Equations (4) and (5) means the controller that ensures a finite c.f.