## 4 Adaptive Actor-Critic Learning for ORM Tracking Control
### Adaptive Actor-Critic Design
It is valid for the first three time steps that  
\[\hat{Q}_{1}(\mathbf{x}_{0},\mathbf{u}_{0})\stackrel{{(\ref{eq:201 })}}{{=}}\hat{Q}_{0}(\mathbf{x}_{0},\mathbf{u}_{0})+\alpha_{c}[V(\mathbf{x}_{0},\mathbf{u}_{0})+\gamma\hat{Q}_{0}(\mathbf{x}_{1},\mathbf{u}_{1}=\mathbf{C}_{ 1}(\mathbf{x}_{1}))-\hat{Q}_{0}(\mathbf{x}_{0},\mathbf{u}_{0})]\stackrel{{ (\ref{eq:2011})}}{{\leq}}\hat{Q}_{0}(\mathbf{x}_{0},\mathbf{u}_{0}). \tag{22a}\] \[\hat{Q}_{2}(\mathbf{x}_{1},\mathbf{u}_{1})\stackrel{{(\ref{eq:201 })}}{{=}}\hat{Q}_{1}(\mathbf{x}_{1},\mathbf{u}_{1})+\alpha_{c}[V(\mathbf{x}_{1},\mathbf{u}_{1})+\gamma\hat{Q}_{1}(\mathbf{x}_{2},\mathbf{u}_{2}=\mathbf{C}_{ 2}(\mathbf{x}_{2}))-\hat{Q}_{1}(\mathbf{x}_{1},\mathbf{u}_{1})]\stackrel{{ (\ref{eq:2011})}}{{\leq}}\hat{Q}_{1}(\mathbf{x}_{1},\mathbf{u}_{1}).\] (22b) \[\hat{Q}_{3}(\mathbf{x}_{2},\mathbf{u}_{2})\stackrel{{(\ref{eq:201 })}}{{=}}\hat{Q}_{2}(\mathbf{x}_{2},\mathbf{u}_{2})+\alpha_{c}[V(\mathbf{x}_{2},\mathbf{u}_{2})+\gamma\hat{Q}_{2}(\mathbf{x}_{3},\mathbf{u}_{3}=\mathbf{C}_{ 3}(\mathbf{x}_{3}))-\hat{Q}_{2}(\mathbf{x}_{2},\mathbf{u}_{2})]\stackrel{{ (\ref{eq:2011})}}{{\leq}}\hat{Q}_{2}(\mathbf{x}_{2},\mathbf{u}_{2}). \tag{22c}\]  
Cancelling the same terms in both sides of Equation (22a-c), since \(\alpha_{c}>0\), it follows that the sums in square parentheses are negative. These sums are further refined using _Lemma 1_ as  
\[V(\mathbf{x}_{0},\mathbf{u}_{0})+\gamma\hat{Q}_{0}(\mathbf{x}_{1},\mathbf{u}_{ 1})\leq\hat{Q}_{0}(\mathbf{x}_{0},\mathbf{u}_{0}), \tag{23a}\] \[V(\mathbf{x}_{1},\mathbf{u}_{1})+\gamma\hat{Q}_{1}(\mathbf{x}_{2},\mathbf{u}_{ 2})\leq\hat{Q}_{1}(\mathbf{x}_{1},\mathbf{u}_{1})\stackrel{{(\ref{eq:201 })}}{{\leq}}\hat{Q}_{0}(\mathbf{x}_{1},\mathbf{u}_{1}),\] (23b) \[V(\mathbf{x}_{2},\mathbf{u}_{2})+\gamma\hat{Q}_{2}(\mathbf{x}_{3},\mathbf{u}_{ 3})\leq\hat{Q}_{2}(\mathbf{x}_{2},\mathbf{u}_{2})\stackrel{{(\ref{eq:201 })}}{{\leq}}\hat{Q}_{1}(\mathbf{x}_{2},\mathbf{u}_{2}), \tag{23c}\]  
Using (23c) in (23b) it follows that \(V(\mathbf{x}_{1},\mathbf{u}_{1})+\gamma V(\mathbf{x}_{1},\mathbf{u}_{1})+\gamma^ {2}\hat{Q}_{2}(\mathbf{x}_{3},\mathbf{u}_{3})\leq\hat{Q}_{0}(\mathbf{x}_{1}, \mathbf{u}_{1})\), which used in (23a) results in\(V(\mathbf{x}_{0},\mathbf{u}_{0})+\gamma V(\mathbf{x}_{1},\mathbf{u}_{1})+ \gamma^{2}V(\mathbf{x}_{2},\mathbf{u}_{2})+\gamma^{3}\hat{Q}_{2}(\mathbf{x}_{ 3},\mathbf{u}_{3})\leq\hat{Q}_{0}(\mathbf{x}_{0},\mathbf{u}_{0})\). Extending the exemplified reasoning backwards from infinity it follows that  
\[\lim_{N\to\infty}\left(\sum_{i=0}^{N-1}\gamma^{i}V(\mathbf{x}_{i},\mathbf{u}_{i})+ \gamma^{N}\hat{Q}_{N-1}(\mathbf{x}_{N},\mathbf{u}_{N})\right)\leq\hat{Q}_{0}( \mathbf{x}_{0},\mathbf{u}_{0}). \tag{24}\]  
Since \(\lim_{N\to\infty}\gamma^{N}\hat{Q}_{N-1}(\mathbf{x}_{N},\mathbf{u}_{N})=0\) because \(\hat{Q}_{N-1}(\mathbf{x}_{N},\mathbf{u}_{N})\) is bounded since resulting from a non-increasing sequence, it follows that \(\lim_{N\to\infty}\sum_{i=0}^{N-1}\gamma^{i}V(\mathbf{x}_{i},\mathbf{u}_{i})\leq \hat{Q}_{0}(\mathbf{x}_{0},\mathbf{u}_{0})\) is finite. The left term in the inequality is the cost of using the controller \(\mathbf{u}_{0}=\mathbf{C}_{0}(\mathbf{x}_{0}),\mathbf{u}_{1}=\mathbf{C}_{1}( \mathbf{x}_{1}),\ldots\). Then it follows that the control system remains stable under the time-varying control of the AAC updates. Moreover, for \(\gamma=1\), the sequence \(\left\{\sqrt{V(\mathbf{x}_{i},\mathbf{u}_{i})}\right\}\), \(i=\overline{0,\infty}\) must be square-summable which implies that the control system is asymptotically stabilized by \(\mathbf{C}_{k}\), thus proving the claim of the theorem. **Comment 4**: The above proof resembles the stabilizing action-dependent value iteration of [30], but here relies on gradient-based updates of the Q-function estimate and of the controller, rather than on its minimization. The stability result of the _Theorem 2_ is valid under continuous updates of the AAC laws (15), (16) under no exploration.