## 5 Validation Case Study
### Statistical Investigations of the AAC Control Performance  
Several thorough investigation case studies are considered, for which the initial C-NN tuned by VRFT and the initial Q-NN are the same. The investigation concerns full vs. partial tuning of the Q-NN and C-NN weights, while measuring the probing noise effect on the convergence. All statistics are measured on learning trials of maximum 50 episodes. The minimal and average \(J_{MR\_A\_AC}^{1400}\) values on 100 trials are measured along with the success percentage of convergent learning trials. The average number of episodes until reaching the minimal \(J_{MR\_AAC}^{1400}\) of a successful learning trial together with the standard deviation of the number of episodes in a successful learning trial, are both rendered in Table 1.  
_Case 1._ Under learning rates \(\alpha_{c}=0.01\) in (27) and \(\alpha_{a}=10^{-8}\) in (28), with full adaptation of the Q-NN and of the C-NN, the learning process convergence starting from the initial C-NN tuned by VRFT and initial Q-NN is investigated. For a convergent learning trial, for the constant learning rates being used, the best performance did never drop below \(J_{MR\_AAC}^{1400}=0.46\), which is inferior to the BFQ performance, suggesting that the proposed adaptive learning strategy is prone to getting stuck in local minima under the adaptive gradient-based update rules. In fact, BFQ is generally advertised as being more data-efficient, although actor-critic learning architectures also allow alternative updates of the C-NN and Q-NN for improving data usage efficiency. The learning trials converges in about 88% of the cases, comparable with other perturbed AAC designs [55; 56] given the wide operating range used for the controlled process. _Case 2._ For full weights adaptation of both Q-NN and C-NN, without random perturbation of the control action, with \(\alpha_{c}=0.01\) in (27) and \(\alpha_{a}=10^{-7}\) in (28), the convergence rate drops to 53% and the best performance is \(J_{MR\_AAC}^{1400}=0.45\). _Case 3._ In the case of output weights only tuning of both Q-NN and C-NN, with random perturbation of the control action, with \(\alpha_{c}=0.01\) in (27) and \(\alpha_{a}=10^{-6}\) in (28), 100% convergence rate was observed, but the performance never dropped below \(J_{MR\_AAC}^{1400}=0.59\). _Case 4._ With output weights only tuning of both the Q-NN and C-NN, starting from the initialized C-NN tuned by VRFT and the initial C-NN, in the absence of random perturbation of the control action, with \(\alpha_{c}=0.01\) in (27) and \(\alpha_{a}=10^{-5}\) in (28), the AAC learning is 100% convergent in all trials but the performance never drops below \(J_{MR\_AAC}^{1400}=0.50\). For \(\alpha_{a}=10^{-6}\), the average number of episodes per trial increases only to 11. The above four case studies are statistically characterized in Table 1. Concluding, full weight tuning of Q-NN and C-NN offers better performance (smaller \(J_{MR\_AAC}^{1400}\)) than when output weights only tuning is used. But full weight tuning lowers the convergence rate, as prone to stuck in local minima. In Case 1 vs. Case 2, the probing noise significantly improves the convergence, slightly improves the average \(J_{MR\_AAC}^{1400}\) and decreases the average number of episodes per convergent trial. While full weights adaptation is more sensitive since even small corrections in the input-to-hidden layer weights may lead to learning divergence. The output weights only tuning is more robust, with 100% convergence success rate to an improved solution, but with inferior achievable performance. The perturbing noise in this case worsens the average number of episodes per trial and the best achievable performance. Guaranteed convergence to an improved solution corresponding to a local minimum in Cases 3 and 4 is also caused by the good initial tuning offered by VRFT.