## 4 Adaptive Actor-Critic Learning for ORM Tracking Control
### AAC Using Neural Networks As Approximators  
In practice, specific FAs such as NNs are employed as approximators for the Q-function and for the controller, respectively. Using NNs as FAs (implying nonlinear features parameterization), the convergence of the learning scheme depends on a large extent to the selection of the learning parameters. However, the advantage of using generic NN architectures is that no manual or automatic feature selection is needed for parameterizing the Q-function and controller estimates. The proposed AAC design herein uses two NNs to approximate the Q-function (critic, referred to herein as Q-NN), and the controller (actor, referred to herein as C-NN), respectively. Assume an initial admissible NN VRFT state-feedback controller exists. Let the critic NN FA and controller NN FA be parameterized as \(\hat{Q}_{k}(\mathbf{x}_{k}^{E},\mathbf{u}_{k},\mathbf{\sigma}_{k}^{c})\) and \(\mathbf{C}_{k}(\mathbf{x}_{k}^{E},\mathbf{\sigma}_{k}^{a})\), respectively. With three-layer feed-forward NNs having one hidden layer, fully connected with bias, the critic and the controller are modeled by:  
\[\hat{Q}_{k}=W_{c,n_{hc}+1}^{k}+\sum_{i=1}^{n_{hc}}W_{c,i}^{k}\sigma_{i}(\sum_{ j=1}^{n_{ic}+1}V_{c,ji}^{k}l_{j}^{k}), \tag{25}\]  
\[u_{k}^{l}=W_{a,n_{ha}+1}^{k,l}+\sum_{i=1}^{n_{ha}}W_{a,i}^{k,l}\sigma_{i}(\sum_ {j=1}^{n_{ia}+1}V_{a,ji}^{k}l_{j}^{k}),\,l=\overline{1,m}, \tag{26}\]  
with \(\mathbf{W}_{c}=\left[W_{c,1}^{k}\,\dots W_{c,n_{hc}+1}^{k}\right]^{T}\)--the critic output layer weights having \(n_{hc}\) hidden neurons, \(\mathbf{V}_{c}=\left[V_{c,ji}^{k}\right]_{i=1\dots n_{hc},j=1\dots n_{ic}+1}\)--the critic hidden layer weights matrix, \(\mathbf{I}_{c}^{k}=\left[I_{1}^{k}\dots I_{n_{ic}+1}^{k}\right]^{T}=\left[ \left(\mathbf{x}_{k}^{E}\right)^{T}(\mathbf{u}_{k})^{T}\,1\right]\)--the critic input vector of size \(n_{ic}+1\) (the bias input is constant 1), \(\mathbf{W}_{a}^{k,l}=\left[W_{a,1}^{k,l}\,\dots W_{a,n_{ha}+1}^{k,l}\right]^{T},l=1..m\) - the output layer weights of the \(l\)-th controller output \(\mathbf{u}_{k}=\left[u_{k}^{1}\dots u_{k}^{m}\right]^{T}\), having \(n_{ha}\) hidden neurons, and \(\mathbf{V}_{a}=\left[V_{a,ji}^{k}\right]_{i=1\dots n_{ha},j=1\dots n_{ia}+1}\)--the controller hidden layer weights matrix, \(\mathbf{I}_{a}^{k}=\left[I_{1}^{k}\dots I_{n_{ia}+1}^{k}\right]^{T}=\left[ \left(\mathbf{x}_{k}^{E}\right)^{T}1\right]^{T}\)--the controller input vector of size \(n_{ia}+1\) (bias input 1 included). \(\sigma_{i}=\tanh_{i}\), is the hyperbolic tangent activation function at the output of \(i^{\text{th}}\) hidden neuron. The controller and critic are cascaded, with the actor output \(\mathbf{u}_{k}\) as a part of the critic input alongside \(\mathbf{x}_{k}^{E}\) The actor and critic weights are _formally_ parameterized as \(\mathbf{\theta}_{a}^{k}=\left[\left(\mathbf{W}_{a}^{k,l}\right)^{T}\left( \mathbf{V}_{a}^{k}\right)^{T}\right]^{T}\) and \(\mathbf{\theta}_{c}^{k}=\left[\left(\mathbf{W}_{c}^{k}\right)^{T}\left( \mathbf{V}_{c}^{k}\right)^{T}\right]^{T}\), respectively. The AAC's gradient descent tuning rules for the critic (25) and controller (26) as parameterized variants of (15) and (16) are  
\[\left\{ \begin{array}{l}\theta_{k}^{c}=\theta_{k-1}^{c}+\alpha_{c} \cdot\frac{\partial\hat{Q}(\mathbf{x},\mathbf{u},\Theta)}{\partial\theta}\Big{|} _{(\mathbf{x}_{k-1}^{E},\mathbf{u}_{k-1},\Theta_{c}^{k-1})}\cdot\delta_{k}, \text{ detailed as }:\\ W_{c,i}^{k}=W_{c,i}^{k-1}+\delta_{k}\alpha_{c}\times\left\{\begin{array}{cc} \sigma_{i}(\sum\limits_{j=1}^{n_{ic}+1}V_{c,ji}^{k-1}t_{j}^{k-1}),&\text{if }i\
eq n_{hc}+1,\\ 1,&\text{if }i=n_{hc}+1,\end{array}\right.\\ V_{c,ji}^{k}=V_{c,ji}^{k-1}+\delta_{k}\alpha_{c}W_{c,i}^{k-1}t_{j}^{k-1}\sigma_ {i}^{\prime}(\sum\limits_{j=1}^{n_{ic}+1}V_{c,ji}^{k-1}t_{j}^{k-1}),\end{array} \right. \tag{27}\]  
\[\left\{