## 3 Controller Design and Stability Analysis
### Stability Analysis
The \(n\)-th fuzzy rule in fuzzy Q learning is defined as follows:  
\(R_{\mathbf{\eta}_{1},i}\): IF \(\zeta_{1,i}^{(\mathbbm{k})}\) is \(L_{1,i}^{(\mathbbm{k})}\) and \(\zeta_{2,i}^{(\mathbbm{k})}\) is \(L_{2,i}^{(\mathbf{\eta}_{1})}\) and \(\ldots\ldots\) and \(\zeta_{m,i}^{(\mathbbm{k})}\) is \(L_{m,i}^{(\mathbf{\eta}_{1})}\), \(\text{THEN}\ \text{u}_{i}^{(\mathbf{\eta}_{1})}\in\mathfrak{U}_{i,\mathbf{\eta}_{1}}\) that \(\text{u}_{i}^{(\mathbf{\eta}_{1})}=\text{u}_{i,1}^{(\mathbf{\eta}_{1})}\) with \(q_{i}^{(\mathbbm{k})}(\mathbf{\eta},1)\) or \(\text{u}_{i}^{(\mathbf{\eta}_{1})}=\text{u}_{i,2}^{(\mathbf{\eta}_{1})}\) with \(q_{i}^{(\mathbf{\eta}_{1})}(\mathbf{\eta},2)\) or \(\text{u}_{i}^{(\mathbf{\eta}_{1})}=\text{u}_{i,3}^{(\mathbf{\eta}_{1})}\) with \(q_{i}^{(\mathbbm{k})}(\mathbf{\eta},3),\ldots,\ \text{u}_{i}^{(\mathbf{\eta}_{1})}=\text{u}_{i,p}^{(\mathbf{ \eta}_{1})}\) with  
\[|e_{i}|\leq\sqrt{\frac{2}{k_{pi}}}\max\{V_{i}(0),\ (\overline{\Omega}_{i}+\frac{1}{2\psi_{i}} \big{(}2\Gamma_{i}^{*})^{2})i=1,2..,n \tag{21}\]9  
\[||e(t)||\leq\sqrt{\sum\limits_{i=1}^{n}\frac{2}{k_{pi}}}\max\{V_{i}(0),[ \overline{\Omega}_{i}+\frac{1}{2\psi_{i}}\big{(}2\Gamma_{i}^{*})^{2}] \tag{22}\]0  
\[||e(t)||\leq\sqrt{\sum\limits_{i=1}^{n}\frac{2}{k_{pi}}}\max\{V_{i}(0),[ \overline{\Omega}_{i}+\frac{1}{2\psi_{i}}\big{(}2\Gamma_{i}^{*})^{2}] \tag{22}\]1  
\[||e(t)||\leq\sqrt{\sum\limits_{i=1}^{n}\frac{2}{k_{pi}}}\max\{V_{i}(0),[ \overline{\Omega}_{i}+\frac{1}{2\psi_{i}}\big{(}2\Gamma_{i}^{*})^{2}] \tag{22}\]2  
are \(e_{i}\) and \(\dot{q}_{i}\) such that \(\text{s}_{i}^{(\mathbbm{k})}=\left\{e_{i}(\mathbbm{k}),\ \dot{q}_{i}(\mathbbm{k})\right\}\). The set of fuzzy inputs \(\text{s}_{i}^{(\mathbbm{k})}=\left\{\text{s}_{1,i}^{(\mathbbm{k})},\ \ldots,\ \text{s}_{m,i}^{(\mathbbm{k})}\right\}\) is fuzzified by the membership function shown in Figure 2 and then matched with the rule antecedents (19), providing the firing rate vector \(\varphi\bigg{(}\text{s}_{i}^{(\mathbbm{k})}\bigg{)}=\bigg{[}\varphi_{1} \bigg{(}\text{s}_{i}^{(\mathbbm{k})}\bigg{)},\varphi_{2}\bigg{(}\text{s}_{i}^ {(\mathbbm{k})}\bigg{)}...\varphi_{n}\bigg{(}\text{s}_{i}^{(\mathbbm{k})} \bigg{)}\bigg{]}\). \(n\) is the amount of fuzzy rules (There are \(n\)- fuzzy rules). For the \(\mathbf{\eta}^{th}\) rule in the \(i^{th}\) subsystem (\(R_{\mathbf{\eta},i}\)), the optimal action at the \(\mathbbm{k}\) moment is defined as the action with the maximum \(q_{i}^{(\mathbbm{k})}(\mathbf{\eta},p)\), \(p\in\{1,2,..,P\}\) among \(P\) action candidates. \[||e(t)||\leq\sqrt{\sum\limits_{i=1}^{n}\frac{2}{k_{pi}}}\max\{V_{i}(0),[ \overline{\Omega}_{i}+\frac{1}{2\psi_{i}}\big{(}2\Gamma_{i}^{*})^{2}] \tag{22}\]3  
\[||e(t)||\leq\sqrt{\sum\limits_{i=1}^{n}\frac{2}{k_{pi}}}\max\{V_{i}(0),[ \overline{\Omega}_{i}+\frac{1}{2\psi_{i}}\big{(}2\Gamma_{i}^{*})^{2}] \tag{22}\]4 Membership function of fuzzy input of the \(i^{th}\) subsystem. To prevent the selection of \(\mathfrak{u}\) from the local optimum in the learning process, we introduce a greed mechanism:  
\[\hat{\mathfrak{u}}_{i}^{(\mathbf{\mathcal{H}})}=\left\{ \begin{array}{l}\mathfrak{u }^{+(\mathbf{\mathcal{H}})}_{i}\text{, with probablity }\rho\\ \mathfrak{u}^{*(\mathbf{\mathcal{H}})}_{i}\text{ with probablity }1-\rho\end{array} \right. \tag{37}\]  
where \(\mathfrak{u}^{+(\mathbf{\mathcal{H}})}_{i}\) is an action randomly selected from \(\mathfrak{U}_{i,\mathbf{\mathcal{H}}_{i}}\). \(0<\rho<1\) is the probability to explore random actions. The numerical value of \(\Lambda_{i}\) is calculated by firing rates and the selected actions:  
\[\Lambda_{i}\left(\mathbf{s}_{i}^{(\mathbf{k})}\right)=\frac{\sum_{\mathbf{\mathcal{H}} =1}^{N}\varphi\mathbf{n}\bigg{(}\mathbf{s}_{i}^{(\mathbf{k})}\bigg{)}\hat{\mathfrak{u }}_{i}^{(\mathbf{\mathcal{H}}_{i})}}{\sum_{\mathbf{\mathcal{H}}=1}^{N}\varphi\mathbf{n} \bigg{(}\mathbf{s}_{i}^{(\mathbf{k})}\bigg{)}} \tag{38}\]  
The update principle of \(\mathpzc{q}\) is the most important part in the whole learning process.