# Fuzzy Q-Learning Agent for Online Tuning of PID Controller for DC Motor Speed Control
## 2 Preliminaries
### Reinforcement Learning
#### 2.4.1 Q-Learning  
The Q-Learning is a reinforcement learning method. The agent computes the Q-Function, which estimates the future discounted rewards of possible actions applied in states [26]. The output of the Q-Function for a state \(\textbf{x}\) and an action \(a\) is represented as \(Q(\textbf{x},a)\). The \(Q\) value of each action \(a\) when performed in a state \(\textbf{x}\) can be calculated as follows:  
\[Q^{\prime}(\textbf{x},a):=Q(\textbf{x},a)+\eta\big{(}R\big{(}\textbf{x},a, \textbf{x}^{\prime}\big{)}+\gamma\overset{max}{a}Q\big{(}\textbf{x}^{\prime},a \big{)}-Q(\textbf{x},a)\big{)} \tag{9}\]  
where \(Q^{\prime}(\textbf{x},a)\) is the updated value of the state-action combination when the agent obtains the reward \(R(\textbf{x},a,\textbf{x}^{\prime})\) after executing the action \(a\) in the state \(\textbf{x}\). Q-Learning assumes that the agent continues for state \(\textbf{x}\) by performing the optimal policy, thus \(\overset{max}{a}Q(\textbf{x}^{\prime},a)\) is the maximum value of the execution of the best action in the state \(\textbf{x}^{\prime}\) (the state comes out after performing \(a\) in state \(\textbf{x}\)). The learning rate \(\eta\) defines in which degree the new information overrides the old one [27]. The discount factor \(\gamma\) defines the significance of the future rewards [28]. Overall, the Q-Learning agent chooses the action \(a\) to be applied in the current state \(\textbf{x}\) and identifies the next state \(\textbf{x}^{\prime}\). It acquires the reward from this transition i.e., \(R(\textbf{x},a,\textbf{x}^{\prime})\), and updates the value for the pair \((\textbf{x},a)\) assuming that it performs the optimal policyfrom state \(x\) and on. A Q-Learning agent applies an exploration/exploitation scheme in order to compute a policy that maximizes the sum of future discounted rewards (\(REW\)).  
\[REW=\sum_{i=0}^{t}\gamma^{t}R_{i} \tag{10}\]  
where \(t\) is the number of the time steps. The main characteristic which makes Q-Learning suitable to be applied in our study is its model-free approach. On the other hand, Q-Learning cannot be applied in a straightforward manner to the continuous action-state spaces that our problem involves. The simplest solution in order for Q-Learning to be applied in continuous action-state spaces is to discretize the action-state space. Discretizing the space may lead to sudden changes in actions, even for smooth changes in states. Making the discretization more fine-grained will smooth these changes, but the state-action combination increases. In problems with multidimensional state space, the number of the combinations grows exponentially. On the other hand, by using other approximators such as neural networks, two slow processes are combined. Q-Learning and neural networks are known for their slow learning rate [29]. In order to overcome these limitations, fuzzy function approximation can be used in combination with Q-Learning, as fuzzy systems are universal approximators [30]. Knowledge of the expert can be embedded in order to assist the learning mechanism. Through this method, actions change smoothly in response to smooth changes in states, without the need for discretizing the space in a fine-grained way [31].