## 5 Validation Case Study
### AAC Design for a MIMO Vertical Tank System
Setting \(\gamma=0.95\), the learning rates \(\alpha_{c}=0.01\) in (27) and \(\alpha_{a}=0\) in (28) (no controller tuning), all the Q-NN weights are updated using the gradient back-propagation in (27), by driving the MIMO control system with a sequence of uniformly random piecewise constant steps in \(\tau_{k}=[r_{k,1}\,r_{k,2}]^{T}\in[0.05,0.25]\times[0.01,0.2]\). This procedure also serves as a tuning step for \(\alpha_{c}\). With \(r_{k,1}\) and \(r_{k,2}\) lasting 20 s and 33 s, respectively, we ensure they do not switch simultaneously, to better reveal the coupling effects between the control channels. After 500 s, the critic weights stabilize, the output weights being shown in Figure 4 for 4500 s. This pre-tuned Q-NN will be used as initialization to the following case studies. After this intermediate tuning step of the Q-NN, the designer can choose for full weight adaptation of the Q-NN or only for output weights' adaptation of the Q-NN while the features matrix \(\mathbf{\Phi}_{c}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})\) is kept constant. Figure 4: Q-NN output weights in Q-function learning of VRFT control. Figure 3: The initial VRFT controller (black dotted), the final adaptively learned controller (black solid), the BFQ controller (blue), the model-based BFQ controller (magenta) and the ORM outputs (red). The C-NN is now further tuned (in the architecture from Figure 1b to improve the ORM control performance. Setting \(\alpha_{a}=10^{-8}\) in (28) and \(\alpha_{c}=0.01\) in (27) (critic adaptation should generally be faster than actor adaptation), both the C-NN and Q-NN are adaptively trained online. Although carried out in an adaptive framework, the training unfolds on consecutive episodes where the feedback control system is driven by a sequence of random reference input steps for 700 s. The reference inputs are uniformly random piecewise constant steps in \(\tau_{k}=\left[\tau_{k,1}\right.\tau_{k,2}\big{]}^{T}\in\left[0.05,0.25\right] \times\left[0.01,0.2\right]\), with \(\tau_{k,1}\) and \(\tau_{k,2}\) lasting 20 s and 33 s, respectively. Updates (27), (28) are skipped when either \(\tau_{k,1}\) or \(\tau_{k,2}\) switch, to preserve the Markov property of the extended model. The controller parameters at the end of an episode are the initial ones for the following episode, the Q-NN weights following the same transfer rule. To ensure enhanced exploration of the state-action space, the C-NN controller output is perturbed every third sample time with probing noise according to:  
\[\mathbf{u}_{k}=\mathbf{C}(\mathbf{x}_{k}^{E})+\left( \begin{array}{c}rand\\ rand\end{array} \right)\cdot\Omega,\Omega= \begin{cases}1&\text{if }\text{mod}(k,3)\ =\ 0,\\ 0&\text{otherwise},\end{cases}  \tag{35}\]  
where _rand_ is a normally distributed random number with zero-mean and variance \(\sigma^{2}=3.56\) and \(\text{mod}(k,s)\) is the remainder after dividing \(k\) by \(s\). This is in fact a form of \(\varepsilon^{0}\)-greedy exploration strategy useful to try many actions in the vicinity of the current state. A typical learning episode is shown in Figure 5. After each learning episode, the learning is stopped and the C-NN performance is measured on the standard test scenario from Figure 3 and the decrease of a finite-time version of the c.f. \(J_{MR}^{\infty}\) from Equation (5), namely \(J_{MR}^{1400}\), is aimed. This standard test scenario is _not_ seen during training. The learning then resumes with the next episode. After maximum 30 learning episodes (meaning 21,000 s and 42,000 samples), the C-NN and Q-NN adaptations are stopped and the _learning trial_ (comprising of learning episodes) converges under _Theorem 3_. The final adaptively learned C-NN and the initial VRFT controller are shown performing in Figure 3. The episodic learning allows us to test the improvement in control performance between episodes. Figure 6 illustrates the \(J_{MR,AC}^{1400}\) decrease over episodes of a convergent learning trial.