## 5 Validation Case Study
### AAC Design for a MIMO Vertical Tank System
Throughout each learning episode, under the AAC update laws, the control system preserves its stability, ensured by _Theorem 2_. Figure 5: Typical training and learning episode. The perturbed control (black), the reference inputs to the CS (green) and the RM outputs (red). Figure 6: Evolution of \(J_{MR,AC}^{1400}\) with each episode for a typical trial of 30 episodes. For comparisons, a model-free approximated batch-fitted Q-learning (BFQ) controller [53; 54] is also proposed, using the same Q-NN and C-NN architectures with the same sizes. BFQ alternates offline training of the C-NN and the Q-NN, using 12,000 transition samples collected under the randomly perturbed model-free single-input single-output VRFT linear controllers \(C_{1}(z)=(2.6092+0.1184z^{-1}-2.3609z^{-2})/(1-z^{-1})\) and \(C_{2}(z)=(1.5735+0.2405z^{-1}-1.3547z^{-2})/(1-z^{-1})\), independently designed for the two tanks, respectively. BFQ implements a Value Iteration algorithm. The training settings assume that the weights of both NNs are initialized to uniform random numbers in \([-1.5,1.5]\). Maximum 200 epochs are used for training with Levenberg-Marquardt on 80% effectively training data and 20% validation data. Early stopping is employed to prevent overfitting after six maximum increases of the mean sum of squared errors on the validation data. 200 iterations of BFQ take about 2 hours and the best controller is saved. The value of \(J_{MR}^{1400}\) for the initial VRFT controller is \(J_{MR\_VFFT}^{1400}=1.58\), for the final adaptively learned controller is \(J_{MR\_A\_AC}^{1400}=0.45\) and for the BFQ controller is \(J_{MR\_MB\_0}^{1400}=0.32\). Additionally, a model-based approximated BFQ solution is also offered for comparisons. A first dimensionality reduction of the extended state space is performed by inspecting that, for both the ORMs, \(x_{k+1,2}^{m}=x_{k,1}^{m}\) and \(x_{k+1,A}^{m}=x_{k,3}^{m}\) from the state-space matrices in Equation (34) and, moreover, \(y_{k,1}^{m}=\mathbf{C_{1}}\big{[}x_{k,1}^{m},x_{k,2}^{m}\big{]}^{T}\approx 2 \mathbf{C_{1}}(1)x_{k,1}^{m},y_{k,2}^{m}=\mathbf{C_{2}}\big{[}x_{k,3}^{m}\ x_{k,4}^{m} \big{]}^{T}\approx 2\mathbf{C_{2}}(1)x_{k,3}^{m}\). Then \(x_{k,2}^{m},x_{k,4}^{m}\) are considered approximate duplicates of \(x_{k,1}^{m},x_{k,3}^{m}\) and removed from the extended state vector, now defined as a reduced extended state vector \(\mathbf{x}_{k}^{ER}=[x_{k,1}^{m},x_{k,2}^{m},x_{k,1}^{m},x_{k,3}^{m},r_{k,1}, \ r_{k,2}]^{T}\in\Re^{6}\) when used for feedback and controller learning. For \([0.05,0.25]\times[0.03,0.15]\times[0.5,4.5]\times[1,12]\times[0.05,0.25]\times [0.03,0.15]\) as the domain of \(\mathbf{x}_{k}^{ER}\) and \([0,1]\times[0.5,1]\) the domain of \(\mathbf{u}_{k}\), we generate a grid of \(NP=5\times 5\times 7\times 7\times 6\times 6\times 3\times 3=396900\) linearly spaced points. 5 points for each of \(x_{k,2}^{m},x_{k,4}^{m}\) would have led to \(NP\) of almost 10 million. Let the discrete domains be denoted \(X_{d}^{E}\) and \(U_{d}\), respectively. Domains of \(x_{k,1}^{m},x_{k,3}^{m}\) and \(r_{k,1},r_{k,2}\) are found by simulating the ORMs offline such that \(y_{k,1}^{m},y_{k,2}^{m}\) overlap the constrained domains of \(x_{k,1}=y_{k,1},x_{k,2}=y_{k,2}\). For a Q-NN of size 8-8-1, a C-NN of size 6-6-2 and \(\gamma=0.8\) found to ensure learning convergence, each iteration of the model-based BFQ trains both the Q-NN and the C-NN. For the Q-NN current iteration estimate (indexed by _iter_) denoted \(\hat{Q}_{iter}\big{(}\mathbf{x}_{k}^{ER},\mathbf{u}_{k}\big{)}\), the inputs patterns are \(\left\{\left[\big{(}\mathbf{x}_{k}^{ER}\big{)}^{T}\big{(}\mathbf{u}_{k}\big{)} ^{T}\right]^{T}\right\}\) and the target patterns are \(\left\{V_{\text{MR}}(\mathbf{x}_{k}^{ER})+\gamma\min_{\mathbf{u}\in U_{d}} \hat{Q}_{iter-1}\Big{(}\mathbf{F}(\mathbf{x}_{k}^{ER},\mathbf{u}_{k}),\mathbf{u }\Big{)}\right\}\), evaluated for all the points in \(X_{d}^{E}\times U_{d}\).