## 5 Validation Case Study
### AAC Design for a MIMO Vertical Tank System
\begin{array}{cc}1.5576&-0.6065\\ 1&0\end{array} \right),\mathbf{B}_{1} =\left( \begin{array}{c}1\\ 0\end{array} \right),\mathbf{C}_{1} =\left( \begin{array}{c}0.0265\\ 0.0224\end{array} \right)^{\!\!\!T},\mathbf{D}_{1} =0, \tag{34}\] \[\mathbf{A}_{2} =\left( \begin{array}{cc}1.8097&-0.8187\\ 1&0\end{array} \right),\mathbf{B}_{2} =\left( \begin{array}{c}1\\ 0\end{array} \right),\mathbf{C}_{2} =\left( \begin{array}{c}0.0047\\ 0.0044\end{array} \right)^{\!\!\!T},\mathbf{D}_{2} =0,\]  
The ORM state will then be \(\mathbf{x}_{k}^{m}=[x_{k,1}^{m}\ x_{k,2}^{m}\ x_{k,3}^{m}\ x_{k,4}^{m}]^{T}\). The virtual reference \(\widetilde{\mathbf{r}_{k}}=\mathbf{M}(z)^{-1}\widetilde{\mathbf{y}_{k}}\) is used as input to the state-space models (34) to obtain the ORM's virtual states \(\widetilde{\mathbf{x}}_{k}^{m}\). The extended virtual state vector \(\widetilde{\mathbf{x}}_{k}^{E}=\left[\left(\widetilde{\mathbf{x}}_{k}\right) ^{T}\left(\widetilde{\mathbf{x}}_{k}^{m}\right)^{T}\left(\widetilde{\mathbf{r }}_{k}\right)^{T}\right]^{T}\in\mathbb{R}^{8}\) is used to offline compute the C-NN via VRFT by fitting the inputs \(\widetilde{\mathbf{x}}_{k}^{E}\) to the outputs \(\widetilde{\mathbf{u}}_{k}\). Note that using two second order ORM's produces four states in the extended state-space, which is disadvantageous. The ORM's orders should to be as low as possible (usually one) but a second order model offers greater flexibility in output response shaping. The VRFT C-NN architecture (Figure 1a) is a feedforward 8-10-2 fully connected one with biases having \(n_{tu}=10\) hidden neurons with \(\tanh(\,)\) activation function and the output activation functions are linear. The weights are initialized with random uniform numbers in \([-1.5,1.5]\). Since the NN training is performed offline, standard gradient backpropagation training with Leverhberg-Marquardt [48] is used for maximum 50 epochs to learn a stabilizing VRFT C-NN controller \(\mathbf{C}(\mathbf{x}_{k}^{E})\) for the MIMO control system, that minimizes \(J_{VR}^{N}\). 80% of the data is effectively used for training while the rest of 20% serves as validation data. Early stopping is used after six consecutive increases in the mean sum of squared error evaluated on the validation data. Other offline training algorithms such as Broyden-Fletcher-Goldfarb-Shanno [49; 50] and conjugate gradient [51; 52] may be similarly efficient while their computational burden is prohibitive for online real-time training. Results on a standard test scenario with the initial VRFT C-NN controller are shown in Figure 3. It is observed that the ORM tracking errors are bounded, since the VRFT controller is stabilizing (though not asymptotically) and validates the theoretical results of Theorem 1 and Corollary 1. Then it is an admissible controller for \(J_{\text{MR}}^{\infty}\) in Equation (5) with \(\gamma<1\). The initial controller tuning using VRFT is attractive also because it has learned a feature matrix \(\mathbf{\Phi}_{a}(\mathbf{x}_{k}^{E})\), so from this point onwards, the designer can choose to perform either output weights adaptation or full weights adaptation. The Q-function estimate of the VRFT C-NN (i.e., the critic Q-NN) is next learned in a policy evaluation step in order to serve as a good initial estimate of the Q-function that is needed for the following AAC learning and also to fulfil the requirements of Lemma 1. This step is possible since the VRFT controller is admissible and, for properly selected learning rate, the weights of the Q-NN will converge. The critic Q-NN approximating the Q-function has similar architecture with the C-NN, of size 10-25-1 (eight states and two controls), with \(n_{hc}=25\). The critic Q-NN output weights are randomly drawn from a zero-mean normal distribution with variance \(\sigma^{2}=90\) while the hidden layer weighs are uniformly randomly initialized in \([-1.5,1.5]\).