## 4 Adaptive Actor-Critic Learning for ORM Tracking Control
### Adaptive Actor-Critic Design  
The proposed AAC design is a gradient-based scheme designed to converge to the optimal Q-function and optimal controller estimates. Let the temporal-difference error be measured from data as  
\[\delta_{k}(\mathbf{x}_{k-1}^{E},\mathbf{u}_{k-1})=V(\mathbf{x}_{k-1}^{E},\mathbf{ u}_{k-1})+\gamma\hat{Q}_{k-1}(\mathbf{x}_{k}^{E},\mathbf{C}_{k}(\mathbf{x}_{k}^{E}))- \hat{Q}_{k-1}(\mathbf{x}_{k-1}^{E},\mathbf{u}_{k-1}) \tag{14}\]  
where the continuous function \(\hat{Q}_{k}(\cdot,\cdot)\) in its arguments is the Q-function estimate at time \(k\), time at which some controller \(\mathbf{C}_{k}(\mathbf{x}_{k})\) is also available. _From this point onward, for notation simplicity_, \(\mathbf{x}_{k}\)_or plain_\(\mathbf{x}\)_are used instead of_\(\mathbf{x}_{k}^{E}\) The proposed AAC design attempts the Q-function update to online minimize the c.f. \(E_{c,\,k}=0.5\ \delta_{k}^{2}\), while the controller attempts to online minimize the Q-function using gradient descent. Taxonomically, the proposed AAC belongs to the online Policy Iteration schemes where the policy evaluation step (of Bellman error residual minimization type) interleaves with the policy improvement step. The update laws for the AAC design from input-state data are:  
\[\mathbf{u}_{k}=\mathbf{C}_{k}(\mathbf{x})=\mathbf{C}_{k-1}(\mathbf{x})-\alpha_ {a}\cdot\left.\frac{\partial\hat{Q}_{k-1}(\mathbf{x},\mathbf{u})}{\partial \mathbf{u}}\right|_{\mathbf{u}_{k-1}},\forall\mathbf{x}, \tag{15}\]  
\[ \begin{array}{l}\hat{Q}_{k}(\mathbf{x}_{k-1},\mathbf{u}_{k-1})=\hat{Q}_{k-1 }(\mathbf{x}_{k-1},\mathbf{u}_{k-1})+\alpha_{c}\delta_{k}=\\ =\hat{Q}_{k-1}(\mathbf{x}_{k-1},\mathbf{u}_{k-1})+\alpha_{c}\big{(}V(\mathbf{ x}_{k-1},\mathbf{u}_{k-1})+\gamma\hat{Q}_{k-1}(\mathbf{x}_{k},\mathbf{u}_{k}=C_{k}( \mathbf{x}_{k}))-\hat{Q}_{k-1}(\mathbf{x}_{k-1},\mathbf{u}_{k-1})\big{)},\end{array}  \tag{16}\]  
where \(\alpha_{a}>0\), \(\alpha_{c}>0\) are learning rates. The controller \(\mathbf{C}(\mathbf{x}_{k})\) is imagined as a function (or as an infinitely dense table) mapping any state to a control action. _Comment 1_: In particular, for any admissible controller \(\mathbf{C}_{0}\), repeated calls of (16), under proper exploration (translated to visiting all the pairs \((\mathbf{x}_{k-1},\mathbf{u}_{k-1})\in X^{E}\times U\) often and to generating the sample \(\mathbf{x}_{k}\)), and under proper selection of \(\alpha_{c}>0\), will update \(\hat{Q}_{k}(\mathbf{x},\mathbf{u})\), \(\forall\mathbf{x}\in X^{E}\) until \(\delta_{k}=0\) at which point (11) must hold and the converged \(Q^{\mathbf{C}_{0}}(\mathbf{x}_{k},\mathbf{u}_{k})\) evaluates \(\mathbf{C}_{0}\). This is an _online off-policy model-free_ policy evaluation step. Then \(Q_{0}(\mathbf{x},\mathbf{u})=Q^{\mathbf{C}_{0}}(\mathbf{x},\mathbf{u})\) can be an initialization for AAC. Whereas, an initial admissible controller can be obtained for example using VRFT as shown later in the case study. _Comment 2_: The converged Q-function of an admissible control \(\mathbf{C}_{0}\), be it \(Q^{\mathbf{C}_{0}}(\mathbf{x}_{k},\mathbf{u}_{k})\), is positive by definition since it accumulates stage costs \(V>0\). Moreover, it is always greater than the optimal Q-function, i.e., \(Q^{\mathbf{C}_{0}}(\mathbf{x}_{k},\mathbf{u}_{k})\geq Q^{{}^{\prime}}(\mathbf{ x}_{k},\mathbf{u}_{k})>0\) and obeys the Bellman equation. _Comment 3_: From (15), it follows that  
\[Q_{k-1}(\mathbf{x},\mathbf{C}_{k}(\mathbf{x}))\leq Q_{k-1}(\mathbf{x},\mathbf{ C}_{k-1}(\mathbf{x})),\forall\mathbf{x}\in X^{E} \tag{17}\]  
for a small enough \(\alpha_{a}>0\). _Lemma 1_. : Starting from an admissible controller \(\mathbf{C}_{0}\) with corresponding Q-function initialization \(\hat{Q}_{0}(\mathbf{x},\mathbf{u})=Q^{\mathbf{C}_{0}}(\mathbf{x},\mathbf{u})\), the sequence \(\left\{\hat{Q}_{k}(\mathbf{x},\mathbf{u})\right\}\) is monotonic and non-increasing ensuring that  
\[\hat{Q}_{k}(\mathbf{x},\mathbf{u})\leq\hat{Q}_{k-1}(\mathbf{x},\mathbf{u}),( \mathbf{x},\mathbf{u})\in X^{E}\times U \tag{18}\]  
Proof.