# Fuzzy Q-Learning Agent for Online Tuning of PID Controller for DC Motor Speed Control
## 3 Control Strategy  
The block diagram of the control strategy is depicted in Figure 2. The set point (\(r\)) is the desired angular velocity of the rotor. The measured value of the rotor velocity, through the feedback control loop, is subtracted from the set point and the value of the error (e) arises. The error is the input of both the PID controller and the fuzzy Q-Learning agent. The PID gains are set offline by the Z-N method and then they are tuned online by the fuzzy Q-Learning algorithm. The control process is the speed of the motor, which has as input the control signal of the PID controller (voltage) and as output the angular velocity of the rotor. Figure 2: Block diagram of the control strategy. The parameters of the DC motor can be seen in Table 1.  
The transfer function of the given motor is:  
\[G(s)=\frac{0.067}{0.00113\;s^{2}+0.0078854\;s+0.0171} \tag{14}\]  
and the transfer function of the PID controller tuning by the Z-N method is:  
\[G_{PID}(s)=2.42+14.89\frac{1}{s}+0.1s \tag{15}\]  
More information about the DC motor model and the transfer function of the PID controller can be found in [20]. The input signal (error) of the fuzzy Q-Learning agent defines the value of its only one state variable and is conditioning in the range of [-1, 1]. For this input, we use seven Membership Functions (MFs) (Figure 3). The PB, PM, PS, Z, NS, NM and NB denote Positive Big, Positive Medium, Positive Small, Zero, Negative Small, Negative Medium and Negative Big, respectively. The one input with the seven MF's results to 7 states represented by an equal number of fuzzy rules. The output of the agent is a vector with three output variables, \(A=[A^{p}\;A^{i}\;A^{d}]\) each one for each gain of the PID controller. Each output variable has a group of five fuzzy actions (fuzzy singletons). For the output variable \(A^{p}\) the group of the fuzzy singletons is: \(A^{p}=\left\{\frac{1}{-0.5}+\frac{1}{-0.25}+\frac{1}{0}+\frac{1}{0.25}+\frac{1 }{0.5}\right\}\). Where \"+\" is the union operator and \"-\" denotes a membership degree to a value of the \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline \(J\) & 0.0113 N\(\cdot\)m\(\cdot\)s\({}^{2}\)/rad \\ \(b\) & 0.028 N\(\cdot\)m\(\cdot\)s/rad \\ \(L_{a}\) & 0.1 H \\ \(R_{a}\) & 0.45 \(\Omega\) \\ \(K_{T}\) & 0.067 N\(\cdot\)m/A \\ \(K_{b}\) & 0.067 V\(\cdot\)s/rad \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Motor parameters. Figure 3: Input membership functions. membership function domain. For the output variable \(A^{i}\) the group of the fuzzy singletons is: \(A^{i}=\left\{\frac{1}{-0.5}+\frac{1}{-0.25}+\frac{1}{0}+\frac{1}{0.25}+\frac{1}{ 0.5}\right\}\) and for the output variable \(A^{d}\) the group of the fuzzy singletons is: \(A^{p}=\left\{\frac{1}{-0.5}+\frac{1}{-0.25}+\frac{1}{0}+\frac{1}{0.25}+\frac{1} {0.5}\right\}\). Each value of the output variable represents the percentage change of each gain according to its initial value, which has arisen by the Z-N method. The action space of the agent consists of 125 actions and the total state-action space consists of 875 combinations. As mentioned before, the agent learns through the interaction of the environment. In order to acquire knowledge through its experience, an exploration/exploitation algorithm must be used in order to define whether the agent is to perform exploration or exploitation. In order to perform high exploration when the agent goes to a new state, agents explore for a certain number of rounds per state (500 rounds/state). The agent then checks and performs the action that has not been performed at all (if there are any). This allows all possible actions to be performed at least one time for each state. After that, the agent performs exploitation for 99% and 1% exploration for a given state. The agent reward (\(R\)) is defined as follows:  
\[R=\frac{1}{1+\left|e^{\prime}\right|}-\frac{1}{1+\left|e\right|} \tag{16}\]  
where \(e^{\prime}\) is the error in the next state. The fraction \(\frac{1}{1+\left|e\right|}\) takes value in the range [0, 1].