## 3 Adaptive-Dynamic-Programming-Based Robust Control Design
### ADP-Based Nominal Optimal Control Design  
Consider the nominal systems of (7):  
\[\dot{\mathbf{e}}_{i}=\mathbf{f}(\mathbf{e}_{i})+\mathbf{g}_{i}\mathbf{\mu}_{i}.\]  
where \(\mathbf{\mu}_{i}\) is the approximated optimal control policies to be designed. According to [17; 26; 33], the infinite horizon cost functions can be defined as  
\[V_{i}(\mathbf{e}_{i}(0))=\int_{0}^{\infty}\Bigl{(}\mathbf{e}_{i}^{\text{T}}\mathbf{Q}_{i} \mathbf{e}_{i}+\mathbf{\mu}_{i}^{\text{T}}\mathbf{R}_{i}\mathbf{\mu}_{i}\Bigr{)}dt, \tag{17}\]  
where \(\mathbf{Q}_{i}\) and \(\mathbf{R}_{i}\) are assumed to be positive definite diagonal constant matrices. Assuming that the optimal policies \(\mathbf{\mu}_{i}^{*}\) exist, the corresponding optimally cost functions of (17) are formulated as  
\[V_{i}^{*}(\mathbf{e}_{i}(0))=\min_{\mathbf{\mu}_{i}\in\mathbf{\
abla}_{i}(\mathbf{\omega}_{i} )}\left(\int_{0}^{\infty}\Bigl{(}\mathbf{e}_{i}^{\text{T}}\mathbf{Q}_{i}\mathbf{e}_{i}+ \mathbf{\mu}_{i}^{\text{T}}\mathbf{R}_{i}\mathbf{\mu}_{i}\Bigr{)}dt\right), \tag{18}\]  
where \(\mathbf{\
abla}_{i}(\mathbf{\omega}_{i})\) are the admissible sets on the compact sets \(\mathbf{\omega}_{i}\). Taking the time derivative of (18), one can obtain the Hamilton-Jacobi-Bellman (HJB) equations:  
\[H_{i}(\mathbf{e}_{i},\mathbf{\mu}_{i}^{*},\
abla V_{i}^{*}(\mathbf{e}_{i}))=\
abla^{\text{ T}}V_{i}^{*}(\mathbf{f}(\mathbf{e}_{i})+\mathbf{g}_{i}\mathbf{\mu}_{i}^{*})+\mathbf{\mu}_{i}^{\text{ $\ast$}\text{T}}\mathbf{R}_{i}\mathbf{\mu}_{i}^{*}+\mathbf{e}_{i}^{\text{T}}\mathbf{Q}_{i}\mathbf{ e}_{i}=0, \tag{19}\]  
where \(\
abla=\partial/\partial\mathbf{e}_{i}\) are the Jacobian matrices. Then, the optimal control policies \(\mathbf{\mu}_{i}^{*}\) can be derived by taking the partial derivative for (19) with respect to \(\mathbf{\mu}_{i}^{*}\):  
\[\mathbf{\mu}_{i}^{*}=-\frac{1}{2}\mathbf{R}_{i}^{-1}\mathbf{g}_{i}^{\text{T}}\
abla V_{i}^ {*}(\mathbf{e}_{i}). \tag{20}\]  
Substituting (20) into (19), the HJB equations can be further written as  
\[0=\
abla{V_{i}^{\text{$\ast$}\text{T}}}(\mathbf{e}_{i})f(\mathbf{e}_{i})+\mathbf{e}_{i}^{ \text{T}}\mathbf{Q}_{i}\mathbf{e}_{i}-\frac{1}{4}\
abla{V^{\text{$\ast$}\text{T}}}(\bm {e}_{i})\mathbf{g}_{i}\mathbf{R}_{i}^{-1}\mathbf{g}_{i}^{\text{T}}\
abla V_{i}^{*}(\mathbf{e}_ {i}). \tag{21}\]  
However, for the HJB Equation (21), it is difficult to obtain the analytical solutions owing to their nonlinearity. Therefore, the AC NN technique will be introduced to solve this problem. For all \(\mathbf{e}_{i}\in\mathbf{\omega}_{i}\), the optimal cost functions \({V_{i}}^{*}(\mathbf{e}_{i})\) can be approximated by  
\[{V_{i}}^{*}(\mathbf{e}_{i})=\mathbf{W}_{i}^{\text{T}}\mathbf{\sigma}_{i}(\mathbf{e}_{i})+\mathbf{ \varepsilon}_{i},\]where \(W_{i}\in\mathcal{R}^{n}\) denote the unknown ideal weight vectors, and \(\sigma_{i}(\mathbf{e}_{i})\in\mathcal{R}^{n}\) are the basis function vectors with \(n\) being the number of hidden layer neurons; \(\varepsilon_{i}\) is the approximation error. In addition, the optimal control policies \(\mathbf{\mu}_{i}^{*}\) can be reconstructed in the form of  
\[\mathbf{\mu}_{i}^{*}=-\frac{1}{2}\mathbf{R}_{i}^{-1}\mathbf{g}_{i}^{T}(\
abla^{T}\mathbf{e}_{i }(\mathbf{e}_{i})\mathbf{W}_{i}+\
abla\varepsilon_{i}).\]  
Two NNs (i.e., AC NNs) are usually utilized to approximate the control policies and optimal cost functions, respectively. The approximation results can be given by  
\[V_{i}(\mathbf{e}_{i})=\hat{\mathbf{W}}_{ic}^{T}\mathbf{\sigma}_{i}(\mathbf{e}_{i}), \tag{22}\] \[\mathbf{\mu}_{i}=-\frac{1}{2}\mathbf{R}_{i}^{-1}\mathbf{g}_{i}^{T}\
abla^{T} \mathbf{\sigma}_{i}(\mathbf{e}_{i})\hat{\mathbf{W}}_{ia}, \tag{23}\]  
where \(\hat{\mathbf{W}}_{ia},~{}\hat{\mathbf{W}}_{ic}\in\mathcal{R}^{n}\) denote the weight vectors of the AC NNs.