## 4 Simulation Results and Analysis  
In this section, similar to works [38; 39; 40; 41; 42], we use the simulation to verify the effectiveness of the proposed controller. A 2-rigid-link robotic manipulator shown in Figure 5 is carried in Matlab 2018a. The sampling time of simulating the real dynamics of robotic manipulators is set as \(1\times 10^{-4}\) s. The sampling time of TDE and controller is set as \(1\times 10^{-3}\) s (\(10\) times to \(1\times 10^{-4}\)) to show the discrete nature of using controllers in practice. The sampling time of fuzzy Q learning is set as \(0.01\) s according to Remark 10. The dynamic model of a 2-rigid-link robotic manipulator is given as follows, which can be also found in [26]. \[ \begin{array}{c}\boldsymbol{M}(\boldsymbol{q})=\begin{bmatrix}m_{2}l_{2}^{2 }+2l_{1}l_{2}m_{2}\cos(q_{2})+(m_{1}+m_{2})l_{1}^{2}&m_{2}l_{2}^{2}+l_{1}l_{2} m_{2}\cos(q_{2})\\ m_{2}l_{2}^{2}+l_{1}l_{2}m_{2}\cos(q_{2})&m_{2}l_{2}^{2}\end{bmatrix}\\ \\ \boldsymbol{C}(\boldsymbol{q},\dot{\boldsymbol{q}})\dot{\boldsymbol{q}}= \begin{bmatrix}-m_{2}l_{1}l_{2}\sin(q_{2})\dot{q}_{2}^{2}-2m_{2}l_{1}l_{2} \sin(q_{2})\dot{q}_{1}\dot{q}_{2}\\ m_{2}l_{1}l_{2}\sin(q_{2})\dot{q}_{1}^{2}\end{bmatrix}\\ \\ \boldsymbol{G}(\boldsymbol{q})=\begin{bmatrix}(m_{1}+m_{2})l_{1}cos(q_{2})g+m_{ 2}l_{2}cos(q_{1}+q_{2})g\\ m_{2}l_{2}\cos(q_{1}+q_{2})g\end{bmatrix}\\ \\ \boldsymbol{F}(\dot{\boldsymbol{q}})=\begin{bmatrix}F_{1}\\ F_{2}\end{bmatrix}\\ \\ \boldsymbol{\tau}_{d}=\begin{bmatrix}\tau_{d1}\\ \tau_{d2}\end{bmatrix}\\ \end{array} \]  
Figure 4: Proposed RL-based adaptive control scheme. The system parameters are given in Table 1, which are the same as [26]. The initial angular positions and velocities of two joints are set as: \(q_{1}=0^{\circ}\), \(q_{2}=0^{\circ}\), \(\dot{q}_{1}=0^{\circ}/s\) and \(\dot{q}_{2}=0^{\circ}/s\). The desired angular positions of two joints are set as: \(q_{r1}=30^{\circ}\) and \(q_{r2}=45^{\circ}\). The upper bounds of the angular position and angular velocity for the \(i^{th}\) joint of manipulator (\(\varepsilon_{i}\) and \(\Lambda_{i}\)) are selected by the user dependent on the specific mission requirement. To successfully implement the proposed controller, users can select any values satisfying \(\varepsilon_{i}>|q_{ri}|\) and \(\Lambda_{i}>0\). Therefore, in the simulation of this paper, we select \(\varepsilon_{1}=50^{\circ}\), \(\varepsilon_{2}=60^{\circ}\), \(\Lambda_{1}=10^{\circ}/s\) and \(\Lambda_{2}=12^{\circ}/s\). **Remark 12**. : \(k_{pi}\) _and \(k_{di}\) are the proportional coefficient and differential coefficient, respectively. The great \(k_{pi}\) could decrease steady state error, but an excessively great could result in a significant overshoot. While the great \(k_{di}\) could improve the robustness to disturbance/uncertainty, but an over-great \(k_{di}\) could compromise the tracking accuracy. The selection of \(k_{pi}\) and \(k_{di}\) can be based on the tuning rules of PID controllers._  
**Remark 13**. : _Great \(k_{si}\) can amplify the effect of the terms \(A_{i}\) and \(B_{i}\) in (10) to keep the angular positions from hitting their constraints. An inappropriately small \(k_{si}\) could lead to a \(|q_{i}|\) too close to \(\varepsilon_{i}\) and then result in an over-great \(u_{i}\) in (10). Therefore, the selection of \(k_{si}\) could be started at a small value, and then users could gradually increase \(k_{si}\) until the magnitude of \(u_{i}\) is acceptable. Great \(k_{yi}\) can amplify the effect of the term \(\mathcal{C}_{i}\) in (10) to keep the angular velocities from hitting their constraints. An excessively great \(k_{yi}\) could lead to an over great \((1+k_{yi}\mathcal{C}_{i})\) in (10) to decline the converging rate of the tracking error even if the angular velocity is not close to its constraint, which