# Fuzzy Q-Learning Agent for Online Tuning of PID Controller for DC Motor Speed Control
## 4 Simulation Results  
The simulation will compare the conventional PID tuned by Z-N with the proposed method. This comparison aims to highlight the improvement of the conventional PID through the supervision of the fuzzy Q-Learning algorithm. The simulation time is set to 4500 s with a simulation step time of 0.001 s. The round of the fuzzy Q-Learning agent is 0.1 s. The value of the learning rate and the discount factor equals to \(\eta=0.1\) and \(\gamma=0.9\) respectively. During the simulation, various set points are applied in the range of [10, 250] rad/s with sample time of 30 s. Figure 4 presents the set point of the motor's velocity; the velocity of the motor which is controlled by the conventional PID controller, tuned by the Z-N method and by the fuzzy Q-Learning PID controller. Figure 5 presents the control signal of both controllers. In the beginning of the simulation, where the fuzzy Q-Learning agent mainly explores, we can see that the rising time is the same for both controllers. On the contrary, the fuzzy Q-Learning PID controller increases the amplitude of the oscillations in the transient states, and it also increases the settling time. During the exploitation phase, we can see that the fuzzy Q-Learning PID controller performs better. The rising time is again the same for both controllers, but now the oscillations of the fuzzy Q-Learning PID have lower amplitude values in the transient states and the settling time is less compared to the conventional PID controller. Characteristics of the output response for both controllers can be found in Table 2 for two different time points. These two points are at 60 s (exploration phase) and at 4260 s (exploitation phase).  
Figure 4: (**a**) Velocity of DC motor (\(\omega\)) with conventional PID control strategy (blue line) and velocity set point (red line), and (**b**) velocity of DC motor (\(\omega\)) with fuzzy Q-Learning PID control strategy (blue line) and velocity set point (red line).  
Figure 5: (**a**) Control signal of conventional PID control strategy and (**b**) control signal of fuzzy Q-Learning PID control strategy.  
The better performance of the fuzzy Q-Learning PID after the exploration phase is also indicated by the values of the Integral Absolute Error (\(IAE\)) and the Integral Time Absolute Error (\(ITAE\)). The \(IAE\) and the \(ITAE\) are defined as:  
\[IAE=\int\lvert e\rvert dt \tag{18}\]  
\[\Pi\text{TAE}\int t\lvert e\rvert dt \tag{19}\]  
At this point, it is worth mentioning that maximizing the value of \(REW\) does not mean that the values of \(IAE\) and \(ITAE\) are minimized. Both the values of the \(IAE\) and the \(ITAE\) are minimized when the error is zero. If the values of the error in both states are set to zero, then the value of the \(REW\) is not maximized but becomes zero. On the other hand, when the value of the error in the current state is very big and the value of the error in the next state is zero, the value of the \(REW\) is maximized, while the \(IAE\) and the \(ITAE\) are not minimized. Figures 6 and 7 present the values of \(IAE\) and \(ITAE\) for both control strategies through time. It is obvious that the conventional PID performs better in the beginning (\(IAE\) and \(ITAE\) have lower values), but as the agent of the fuzzy Q-Learning PID controller turns from the exploration to exploitation, the values of the \(IAE\) and \(ITAE\) turn to be lower. Finally, Figure 8 presents how the three gains of the proposed control strategy are changed through time in respect to the various changes of the set point.