## 3 Controller Design
### Reinforcement Learning-Based Control Design
Combining (20) and (23), it can be derived that  
\[V_{s}+\
abla V_{s}^{*\top}\mathbf{s}+\lambda^{2}\bar{\omega}\text{ln}[\mathbf{1}_{ c}-\text{tanh}^{2}(\bar{M}_{o}^{*})]=0. \tag{24}\]  
The nearly optimal control \(M_{o}^{*}\) can be obtained by solving (24) if \(\
abla V_{s}^{*}\) is available. Inspired by the NN-based control scheme, the following approximation of \(V^{*}(\mathbf{s})\) can be established  
\[V^{*}(\mathbf{s})=\mathbf{W^{*}}^{\top}\mathbf{\sigma}[\mathbf{s}(t)]+\epsilon[\mathbf{s}(t)] \tag{25}\]  
where \(\mathbf{W}^{*}\) is the optimal weight vector, \(\mathbf{\sigma}[\mathbf{s}(t)]\) is the base vector, and \(\epsilon[\mathbf{s}(t)]\) is the approximation error of NN. Subsequently, the gradient of \(V^{*}(s)\) is  
\[\
abla V_{s}^{*}=\mathbf{W^{*}}^{\top}\
abla\mathbf{\sigma}_{s}[\mathbf{s}(t)]+\
abla\mathbf{ e}_{s}[\mathbf{s}(t)] \tag{26}\]  
where \(\
abla\mathbf{\sigma}_{s}=\partial\mathbf{\sigma}/\partial\mathbf{s}\), \(\
abla\mathbf{e}_{s}=\partial\mathbf{\epsilon}/\partial\mathbf{s}\). In light of the universal approximation property of NN for smooth functions on prescribed compact sets, the approximation errors \(\epsilon[\mathbf{s}(t)]\) and \(\
abla\mathbf{e}_{s}[\mathbf{s}(t)]\) are bounded with a finite dimension of \(\mathbf{\sigma}[\mathbf{s}(t)]\)[27]. Moreover, it is assumed that \(||\mathbf{W}^{*}||\), \(\mathbf{\sigma}[\mathbf{s}(t)]\) and \(\
abla\mathbf{\sigma}_{s}[\mathbf{s}(t)]\) are bounded [13; 14; 16; 17; 21]. Recalling (22), the NN-based nearly optimal control law can be formulated as:  
\[ \begin{cases}\mathbf{M}=\hat{\mathbf{M}}_{o}^{*}-\mathbf{B}_{1}^{-1}\Big{\{}\mathbb{C} \Big{[}\lambda_{1i}\text{sig}^{1/2}(s_{i})\Big{]}-\mathbf{M}_{d}\Big{\}}\\ \hat{\mathbf{M}}_{o}^{*}=-\lambda\text{tanh}\Big{[}\frac{1}{2\lambda}\mathbf{\omega}^ {-1}\mathbf{B}_{1}^{\top}\Big{(}\
abla\mathbf{\sigma}_{s}^{\top}\mathbf{W}^{*}+\
abla\mathbf{ e}_{s}\Big{)}\Big{]}\end{cases} . \tag{27}\]  
Defining the Bellman error as [14]:  
\[B_{\epsilon}=\lambda^{2}\bar{\omega}\Big{\{}\text{ln}\Big{[}\mathbf{1}_{c}- \text{tanh}^{2}(\bar{M}_{o}^{*})\Big{]}-\text{ln}\Big{[}\mathbf{1}_{c}-\text{ tanh}^{2}\Big{(}\hat{\mathbf{M}}_{o}^{*}\Big{)}\Big{]}\Big{\}} \tag{28}\]  
with \(\hat{\mathbf{M}}_{o}^{*}=\frac{1}{2\lambda}\mathbf{\omega}^{-1}\mathbf{B}_{1}^{\top} \
abla\mathbf{\sigma}_{s}^{\top}\mathbf{W}^{*}\), (20) can be rewritten as:  
\[V_{s}+\mathbf{W}^{*\top}\
abla\mathbf{\sigma}_{s}\mathbf{\Gamma}+\lambda^{2}\bar{\omega} \text{ln}\Big{[}\mathbf{1}_{c}-\text{tanh}^{2}(\hat{\mathbf{M}}_{o}^{*})\Big{]}+ \epsilon_{H}=0 \tag{29}\]  
where \(\epsilon_{H}=\
abla\mathbf{\epsilon}_{s}^{\top}\mathbf{\Gamma}+B_{\epsilon}+\mathbf{W}^{* }\
abla\mathbf{\sigma}_{s}\mathbf{D}_{2}\) is the bounded HJB error [10]. In this paper, the optimal weight \(\mathbf{W}^{*}\) is generated by the online RL scheme with the AC structure. In this context, the nearly optimal control policy is formulated as:  
\[ \begin{cases}\mathbf{M}=\hat{\mathbf{M}}_{a}^{*}-\mathbf{B}_{1}^{-1}\Big{\{}\mathbb{C} \Big{[}\lambda_{1i}\text{sig}^{1/2}(s_{i})\Big{]}-\mathbf{M}_{d}\Big{\}}\\ \hat{\mathbf{M}}_{a}^{*}=-\lambda\text{tanh}\Big{(}\hat{\mathbf{M}}_{a}^{*}\Big{)}, \hat{\mathbf{M}}_{a}^{*}=\frac{1}{2\lambda}\mathbf{\omega}^{-1}\mathbf{B}_{1}^{\top}\
abla \mathbf{\sigma}_{s}^{\top}\hat{\mathbf{W}}_{a}\end{cases}  \tag{30}\]  
where \(\hat{\mathbf{W}}_{a}\) is the weight of the actor network. Moreover, the performance index (16) can be estimated as:  
\[\hat{V}_{s}=\hat{\mathbf{W}}_{c}^{\top}\mathbf{\sigma}_{s}[\mathbf{s}(t)] \tag{31}\]where \(\hat{W}_{c}\) is the weight of the critic network.