## 3 Controller Design
### Reinforcement Learning-Based Control Design  
Firstly, the following sliding variable \(\mathbf{s}\) is defined as:  
\[\mathbf{s}=\mathbf{\eta}_{2}+c\mathbf{\eta}_{1} \tag{12}\]where \(\mathbf{c}\) is a positive diagonal matrix. Taking the first time derivative of \(\mathbf{s}\) along (11) yields  
\[\dot{\mathbf{s}}=\Xi+\mathbf{B}_{1}\mathbf{M}+\mathbf{D}_{1} \tag{13}\]  
where \(\Xi=c\dot{\mathbf{\eta}}_{1}-\mathbf{\Lambda}\). In order to achieve a satisfactory tracking performance, a traditional super-twisting controller based on (12) and (13) is developed as follows:  
\[\mathbf{M}=-\mathbf{B}_{1}^{-1}\mathbf{\Gamma},\mathbf{\Gamma}=\Xi+\mathbb{C}\Big{[}\lambda_{ 1i}\text{sig}^{1/2}(s_{i})\Big{]}-\mathbf{M}_{d}, \tag{14}\]  
where \(\text{sig}^{1/2}(s_{i})=|s_{i}|^{1/2}\text{sign}(s_{i})\), \(\dot{\mathbf{M}}_{d}=-\mathbb{D}(\lambda_{2i})\text{sign}(s_{i})\), and \(\lambda_{1i}\) and \(\lambda_{2i}\) are positive constants. Nevertheless, the feasibility of (14) may not be guaranteed with limited control inputs, and the time-fuel performance index cannot be approximately optimized. To this end, an online RL-based, nearly optimal, controller is proposed by integrating the AC structure into the super-twisting controller for a comprehensive solution to the issue mentioned above. Before elaborating the detailed design procedure, it is assumed that there exists a group of admissible control strategies [10]  
\[\mathbf{M}=\mathbf{M}_{o}-\mathbf{B}_{1}^{-1}\Big{\{}\mathbb{C}\Big{[}\lambda_{1i}\text{ sig}^{1/2}(s_{i})\Big{]}-\mathbf{M}_{d}\Big{\}}\in\mathbf{\Omega}_{u}. \tag{15}\]  
Moreover, \(\mathbf{M}\) can achieve the control objective with the following time-fuel performance index being satisfied  
\[V(\mathbf{s})=\int_{t}^{\infty}\{V_{s}(\tau)+J[\mathbf{M}_{o}(\tau)]\}\text{d}\tau \tag{16}\]  
where \(V_{s}=\mathbf{s}^{\top}\mathbf{Q}\mathbf{s}\), \(\mathbf{Q}\) is a positive diagonal matrix, and \(J(\mathbf{M}_{o})\) is chosen as [26]:  
\[J(\mathbf{M}_{o})=2\int_{0}^{\mathbf{M}_{o}}\!\Big{[}\lambda\text{tanh}^{-1}\Big{(} \frac{\mathbf{v}}{\lambda}\Big{)}^{\top}\mathbf{\sigma}\Big{]}\text{d}\sigma \tag{17}\]  
where \(\mathbf{\sigma}\) is selected as a positive diagonal matrix, \(\mathbf{v}\) represents the variable of integration, and the upper and lower bounds of \(\mathbf{v}\) are \(\mathbf{M}_{o}\) and \(\mathbf{0}\), respectively. The Lyapunov equation of (16) can be calculated as:  
\[V_{s}(t)+J[\mathbf{M}_{o}(t)]+\
abla V_{s}^{\top}\mathbf{s}=0 \tag{18}\]  
where \(\
abla V_{s}=\partial V(\mathbf{s})/\partial\mathbf{s}\). The optimal value function is defined as:  
\[V^{*}(\mathbf{s})=\min_{\mathbf{M}_{o}}\int_{t}^{\infty}\{V_{s}(\tau)+J[\mathbf{M}_{o}( \tau)]\}\text{d}\tau, \tag{19}\]  
and the corresponding HJB equation can be formulated as:  
\[\min_{\mathbf{M}_{o}}\Big{\{}V_{s}+J(\mathbf{M}_{o})+\
abla V_{s}^{\ast\top}\mathbf{s} \Big{\}}=0, \tag{20}\]  
where \(\
abla V_{s}^{*}=\partial V^{*}(\mathbf{s})/\partial\mathbf{s}\). Equation (20) is equivalent to  
\[\frac{\partial}{\partial\mathbf{M}_{o}^{*}}\Big{[}V_{s}+J(\mathbf{M}_{o}^{*})+\
abla V _{s}^{\ast\top}\mathbf{s}\Big{]}=0. \tag{21}\]  
Solving the partial derivative of (21) yields the following optimal control strategy  
\[\dot{\mathbf{s}}=\Xi+\mathbf{B}_{1}\mathbf{M}+\mathbf{D}_{1} \tag{13}\]0Substituting \(M_{o}^{*}\) into (17), one obtains  
\[ \begin{split} J(M_{o}^{*})&=\lambda\
abla V_{s}^{*\top} B_{1}\text{tanh}(\bar{M}_{o}^{*})+\lambda^{2}\bar{\omega}\text{ln}\Big{[} \mathbf{1}_{c}-\text{tanh}^{2}(\bar{M}_{o}^{*})\Big{]}\\ &=-\
abla V_{s}^{*\top}B_{1}\mathbf{M}_{o}^{*}+\lambda^{2}\bar{ \omega}\text{ln}\Big{[}\mathbf{1}_{c}-\text{tanh}^{2}(\bar{M}_{o}^{*})\Big{]}, \end{split}  \tag{23}\]  
where \(\mathbf{1}_{c}\) is a column vector with all elements being one, and \(\bar{\omega}\) is a row vector generated by the elements on the main diagonal of \(\bar{\omega}\).