## 3 Controller Design and Stability Analysis
### Stability Analysis
The \(\mathpzc{q}\)-values are updated according to the rewards of the selected actions; the optimal action can achieve higher rewards, therefore, we can finally learn the optimal action. The Q value at the \(\mathbf{k}\) moment can be designed as follows:  
\[Q\bigg{(}\mathbf{s}_{i}^{(\mathbf{k})}\bigg{)}=\frac{\sum_{\mathbf{\mathcal{H}}=1}^{N }\varphi\mathbf{n}\bigg{(}\mathbf{s}_{i}^{(\mathbf{k})}\bigg{)}\mathpzc{q}_{i}^{( \mathbf{k})}\bigg{(}\mathbf{n},\ \hat{\mathfrak{u}}_{i}^{(\mathbf{\mathcal{H}}_{i})} \bigg{)}}{\sum_{\mathbf{\mathcal{H}}=1}^{N}\varphi\mathbf{n}\bigg{(}\mathbf{s}_{i}^{( \mathbf{k})}\bigg{)}} \tag{39}\]  
The target value at the state \(\mathbf{s}_{i}^{(\mathbf{k})}\) is calculated as:  
\[\mathcal{V}\bigg{(}\mathbf{s}_{i}^{(\mathbf{k})}\bigg{)}=\frac{\sum_{\mathbf{\mathcal{ H}}=1}^{N}\varphi\mathbf{n}\bigg{(}\mathbf{s}_{i}^{(\mathbf{k})}\bigg{)}\mathpzc{q}_{i}^{( \mathbf{k})}\bigg{(}\mathbf{n},\ \mathfrak{u}^{*(\mathbf{\mathcal{H}})}_{i}\bigg{)}}{\sum_{\mathbf{ \mathcal{H}}=1}^{N}\varphi\mathbf{n}\bigg{(}\mathbf{s}_{i}^{(\mathbf{k})}\bigg{)}} \tag{40}\]  
When the system state \(\mathbf{s}_{i}^{(\mathbf{k})}\) is driven to the next state \(\mathbf{s}_{i}^{(\mathbf{k}+1)}\), temporal difference (TD) is calculated according to the reward obtained at the \(k\) moment \(\mathbf{r}_{i}^{(\mathbf{k})}\):  
\[\Delta Q_{i}^{(\mathbf{k})}=\mathbf{r}_{i}^{(\mathbf{k})}+\alpha_{i}\mathcal{V} \bigg{(}\mathbf{s}_{i}^{(\mathbf{k}+1)}\bigg{)}-Q\bigg{(}\mathbf{s}_{i}^{(\mathbf{k}) }\bigg{)} \tag{41}\]  
where \(\alpha_{i}\in[0,1]\) is the discount factor reflecting the contribution of the future reward. The reward \(\mathbf{r}_{i}^{(\mathbf{k})}\) at the \(\mathbf{k}\) moment of the \(i^{th}\) subsystem is designed in (42) and the meaning of (42) is explained in Remark 9. \[\mathbf{r}_{i}^{(\mathbf{k})}=\left\{ \begin{array}{c}0\text{, if }|\hat{ \mathfrak{S}}_{i}(\mathbf{k})|>\sigma_{i}\\ \cos\bigg{(}\frac{\pi}{2}\frac{|\hat{\mathfrak{S}}_{i}(\mathbf{k})|}{\sigma_{ i}}\bigg{)}\text{, if }|\hat{\mathfrak{S}}_{i}(\mathbf{k})|\leq\sigma_{i}\end{array} \right. \tag{42}\]  
where \(\hat{\mathfrak{S}}_{i}(\mathbf{k})=\hat{q}_{i}(\mathbf{k})+\Lambda_{i}tanh \bigg{(}\frac{\varepsilon_{i}(\mathbf{k})}{\bar{\mathbf{\Phi}}_{i}}\bigg{)}\) and \(\sigma_{i}>0\) is a positive constant. Finally, the adaptive law of \(\mathfrak{q}\)-values is:  
\[\mathpzc{q}_{i}^{(\mathbf{k}+1)}\bigg{(}\mathbf{n},\hat{\mathfrak{u}}_{i}^{(\mathbf{ \mathcal{H}}_{i})}\bigg{)}=\mathpzc{q}_{i}^{(\mathbf{k})}\bigg{(}\mathbf{n},\hat{ \mathfrak{u}}_{i}^{(\mathbf{\mathcal{H}}_{i})}\bigg{)}+\Gamma_{i}\Delta Q_{i}^{( \mathbf{k})}\cdot\frac{\varphi\mathbf{n}\bigg{(}\mathbf{s}_{i}^{(\mathbf{k})}\bigg{)} }{\sum_{\mathbf{\mathcal{H}}=1}^{N}\varphi\mathbf{n}\bigg{(}\mathbf{s}_{i}^{(\mathbf{k})} \bigg{)}} \tag{43}\]where \(\Gamma_{i}\in[0,1]\) is the learning rate. **Remark 9**. : _The reward function (42) indicates that higher values of reward \(\mathbf{\sigma}_{i}^{(\mathbbm{k})}\) can be obtained by the smaller \(\hat{\mathfrak{S}}_{i}(\mathds{k})\), which means the the action that satisfies (24) will be given with the highest reward (\(\mathbf{\sigma}_{i}^{(\mathbbm{k})}=1\))._  
The aforementioned process of fuzzy Q learning to tune \(\Lambda_{i}\) can be also concluded in Figure 3. The proposed control scheme can be concluded in Figure 4.  
**Remark 10**. : _All the system states including the tracking errors are proven to be bounded with \(\Lambda_{i}>0\). Therefore, the instability will not occur even if an inappropriate positive value of \(\Lambda_{i}\) is calculated by the fuzzy Q learning mechanism trying some bad action candidates. Hence, the designed control law (7), (9)-(14) offers the fuzzy Q learning mechanism (32)-(43) a safe environment to learn a optimal \(\Lambda_{i}\).