## 4 Adaptive Actor-Critic Learning for ORM Tracking Control
### AAC Using Neural Networks As Approximators
\begin{array}{l}\theta_{k}^{a}=\theta_{k-1}^{a}-\alpha_{a} \cdot\frac{\partial\hat{Q}(\mathbf{x},\mathbf{u},\Theta)}{\partial\mathbf{u}} \Big{|}_{(\mathbf{x}_{k-1}^{E},\mathbf{u}_{k-1},\Theta_{c}^{k-1})}\cdot\frac{ \partial\mathbf{u}(\mathbf{x},\Theta)}{\partial\Theta}\Big{|}_{(\mathbf{x}_{k -1}^{E},\Theta_{a}^{k-1})},\text{as }:\\ W_{a,i}^{k,l}=W_{a,i}^{k-1,l}-\alpha_{a}\underbrace{\left(\sum \limits_{i=1}^{n_{hc}}W_{c,i}^{k-1}V_{c,\xi i}^{k-1}\sigma_{i}^{\prime}(\sum \limits_{j=1}^{n_{ic}+1}V_{c,ji}^{k-1}t_{j}^{k-1})\right)}_{\triangleq\hat{Q}( \mathbf{x},\mathbf{u},\Theta)}\left[{}_{(\mathbf{x}_{k-1}^{E},\mathbf{u}_{k-1 },\[PERSON]_{c}^{k-1})}\right]}_{(\mathbf{x}_{k-1}^{E},\mathbf{u}_{k-1}^{E}, \mathbf{u}_{k-1}^{E-1})},\text{as }:\\ W_{a,i}^{k,l}=W_{a,i}^{k-1,l}-\alpha_{a}\underbrace{\left(\sum \limits_{i=1}^{n_{hc}}W_{c,i}^{k-1}V_{c,\xi i}^{k-1}\sigma_{i}^{\prime}(\sum \limits_{j=1}^{n_{ic}+1}V_{c,ji}^{k-1}t_{j}^{k-1})\right)}_{\triangleq\hat{Q}( \mathbf{x},\mathbf{u},\Theta)}\left[{}_{(\mathbf{x}_{k-1}^{E},\mathbf{u}_{k-1 },\[PERSON]_{c}^{k-1})}\right]}_{(\mathbf{x}_{k-1}^{E},\mathbf{u}_{k-1},\[PERSON]_{ c}^{k-1})}\times\\ V_{a,ji}^{k}=V_{a,ji}^{k}-\alpha_{a}\left.\frac{\partial\hat{Q}(\mathbf{x}, \mathbf{u},\[PERSON])}{\partial\mathbf{u}}\right|_{(\mathbf{x}_{k-1}^{E},\mathbf{ u}_{k-1},\[PERSON]_{c}^{k-1})}W_{a,i}^{k-1,l}t_{j}^{k-1}\sigma_{i}^{\prime}(\sum \limits_{j=1}^{n_{ic}+1}V_{a,ji}^{k-1}t_{j}^{k-1}),\end{array} \right. \tag{28}\]  
with \(\alpha_{a}\), \(\alpha_{c}\)--the learning rate magnitudes of the critic and controller training rules, respectively and \(\sigma_{i}^{\prime}\) is the derivative of \(\sigma_{i}\) w.r.t its argument and \(\xi\), is the index of \(u_{k}^{l}\) in the \(\mathbf{I}_{c}^{k}\). In many practical applications, the designer chooses to perform either full or partial adaptation of the NNs' weights, the latter implying only output weights adaptation. In this latter case, the Q-NN and C-NN parameterizations are:  
\[\hat{Q}_{k}(\mathbf{x}_{k}^{E},\mathbf{u}_{k})=\big{(}\mathbf{W}_ {c}^{k}\big{)}^{\!\!T}\sigma(\mathbf{V}_{c}^{k}\big{[}\!\left(\mathbf{x}_{k}^{ E}\right)^{\!\!T}\left(\mathbf{u}_{k}\right)^{\!\!T}1\big{]}^{\!\!T})=\big{(} \mathbf{W}_{c}^{k}\big{)}^{\!\!T}\mathbf{\Phi}_{c}^{k}(\mathbf{x}_{k}^{E}, \mathbf{u}_{k}), \tag{29}\] \[\mathbf{u}_{k}=\mathbf{C}_{k}(\mathbf{x}_{k}^{E})=\big{(}\mathbf{ W}_{a}^{k}\big{)}^{\!\!T}\sigma(\mathbf{V}_{a}^{k}\big{[}\!\left(\mathbf{x}_{k}^{ E}\right)^{\!\!T}1\big{]}^{\!\!T})=\big{(}\mathbf{W}_{a}^{k}\big{)}^{\!\!T} \mathbf{\Phi}_{c}^{k}(\mathbf{x}_{k}^{E}),\]  
where \(\mathbf{\Phi}_{c}^{k}(\mathbf{x}_{k}^{E},\mathbf{u}_{k}),\mathbf{\Phi}_{a}^{k} (\mathbf{x}_{k}^{E})\) are the matrices of basis functions (or input features) and \(\mathbf{W}_{c}^{k}\), \(\mathbf{W}_{a}^{k}\) are the tunable output weights parameters, rendering \(\hat{Q}_{k},\mathbf{u}_{k}\) as linear combination of basis functions. This linear parameterization simplifies the convergence analysis but also requires manual features selection and training as a disadvantage. As it is well-known, the AAC architecture performs online with the C-NN sending controls to the process and the Q-NN serving both to estimate the Q-function and to adaptively tune the C-NN. The closed-loop control system with the process (3) combined with the AAC tuning rules (27), (28) has the unique property that its dynamics is mainly driven by the reference input \(\mathbf{r}_{k}\) viewed as a particular state of the extended state vector and, possibly, by exogenous unknown disturbances. Since \(\mathbf{r}_{k}\) is user selectable, it can be used to drive the control system in a wide operating range to ensure efficient exploration of the state space. Enhanced exploration of the domain \(X^{E}\times U\) can be performed by trying random actions in every state, usually as additive uniform random actions.