## 4 Adaptive Actor-Critic Learning for ORM Tracking Control
### Convergence of the AAC Learning Scheme with NNs  
While the results from Section 4.1 are formulated under generic functions for the Q-function and for the controller, the convergence to the optimal controller and optimal Q-function is not ensured. In the following, the convergence to of the AAC learning with NNs is shown. Linear parameterizationis the most widely used and supports tractable analysis. Let the output weight parameterization (29) of the Q-NN and C-NN lead to the update laws  
\[\begin{split}\mathbf{W}_{c}^{k}&=\mathbf{W}_{c}^{k-1}+ \alpha_{c}\delta_{k}\mathbf{\[PERSON]}_{c}^{k}(x_{k-1}^{E},\mathbf{u}_{k-1}),\\ \mathbf{W}_{a}^{k}&=\mathbf{W}_{a}^{k-1}-\alpha_{a} \mathbf{\[PERSON]}_{a}^{k}(x_{k-1}^{E})\big{(}\mathbf{W}_{c}^{k-1}\big{)}^{T} \frac{\partial\mathbf{\Phi}_{c}}{\partial\mathbf{u}}(x_{k-1}^{E},\mathbf{u}_{ k-1}),\end{split} \tag{30}\]  
be compactly written as  
\[\begin{split}\mathbf{W}_{c}^{k}&=\mathbf{W}_{c}^{k-1 }+\alpha_{c}\delta_{k}\mathbf{\[PERSON]}_{c}^{k-1},\\ \mathbf{W}_{a}^{k}&=\mathbf{W}_{a}^{k-1}-\alpha_{a} \mathbf{\[PERSON]}_{a}^{k-1}\big{(}\mathbf{W}_{c}^{k-1}\big{)}^{T}\mathbf{\[PERSON]}_{c,u}^{k-1},\end{split} \tag{31}\]  
where \(\delta_{k}=\underbrace{V(\mathbf{x}_{k-1}^{E},\mathbf{u}_{k-1})}_{\text{$ \sqcup$}^{k-1}}+\big{(}\mathbf{W}_{c}^{k-1}\big{)}^{T}\big{(}\gamma\mathbf{ \Phi}_{c}^{k}-\mathbf{\Phi}_{c}^{k-1}\big{)}\) Let the weights of the optimal Q-NN and optimal C-NN be \(\mathbf{W}_{c}^{*}\), \(\mathbf{W}_{a}^{*}\), the estimation errors being \(\tilde{\mathbf{W}}_{c}^{k}=\mathbf{W}_{c}^{k}-\mathbf{W}_{c}^{*}\tilde{ \mathbf{W}}_{a}^{k}=\mathbf{W}_{a}^{k}-\mathbf{W}_{a}^{*}\) that render the estimation error dynamics  
\[\begin{split}\tilde{\mathbf{W}}_{c}^{k}&=\tilde{ \mathbf{W}}_{c}^{k-1}+\alpha_{c}\delta_{k}\mathbf{\Phi}_{c}^{k-1},\\ \tilde{\mathbf{W}}_{a}^{k}&=\tilde{\mathbf{W}}_{a}^{k- 1}-\alpha_{a}\mathbf{\Phi}_{a}^{k-1}\big{(}\mathbf{W}_{c}^{k-1}\big{)}^{T} \mathbf{\[PERSON]}_{c,u}^{k-1},\end{split} \tag{32}\]  
Some assumptions follow:  
\(A7\). Let the estimation error of the critic be denoted as \(\zeta_{c}^{k-1}=\Big{(}\tilde{\mathbf{W}}_{c}^{k-1}\Big{)}^{T}\mathbf{\Phi}_{ c}^{k-1}\), let the critic's and actor's hidden activation layers be bounded as \(\|\mathbf{\Phi}_{c}^{k-1}\|^{2}\leq\overline{\varphi}_{c,l}\|\mathbf{\Phi}_{a} ^{k-1}\|^{2}\leq\overline{\varphi}_{a}\) and let the critic's activation layer derivative w.r.t. \(\mathbf{u}\) be bounded as \(\|\mathbf{\Phi}_{c,u}^{k-1}\|^{2}\leq\overline{\varphi}_{c,u}\) where Frobenius norm was used, which is equivalent to the Euclidean norm when it is applied to vectors. The above upper bounds follow since the activation functions are bounded and so are their derivatives.  
Under \(A7\), the AAC learning scheme converges to a vicinity of the optimal controller and optimal Q-function since the estimation errors \(\tilde{\mathbf{W}}_{c}^{k},\tilde{\mathbf{W}}_{a}^{k}\) are uniformly ultimately bounded provided that \(\alpha_{c}>4\overline{\varphi}_{c}^{2}-2\overline{\varphi}_{c}+1\), for \(\overline{\varphi}_{c}>1\).  
Proof.: See Appendix B.  
_Comment 5._ Notice that the temporal difference error \(\delta_{k}\) is calculated in terms of the Q-NN's output. Then \(\delta_{k}\) is backpropagated to correct the Q-NN weights. Moreover, \(\delta_{k}\) is further backpropagated to correct the C-NN weights, since the C-NN output is an input to the Q-NN. Hence, the resulted AAC architecture belongs to the deep learning approaches (the architecture is presented in Figure 1b of the next Section).  
Figure 1: (**a**) control system with the VRFT controller; (**b**) control system with the VRFT NN controller further tuned by AAC design in a deep learning architecture; (**c**) the vertical tank system.