# Fuzzy Q-Learning Agent for Online Tuning of PID Controller for DC Motor Speed Control
## 2 Preliminaries
### Reinforcement Learning
#### 2.4.2 Fuzzy Q-Learning  
In Q-Learning, Q-values are stored and updated for each state-action pair. In cases where the number of state-action pairs is very large, the implementation might be impractical. Fuzzy Inference Systems have the advantage of achieving good approximations [32] in the Q-function, and simultaneously make possible the use of the Q-Learning in continuous states-space problems (Fuzzy Q-Learning) [29]. In fuzzy Q-Learning, \(x\) is the crisp set of the inputs defining the state of the agent. These are converted into fuzzy values and each fuzzy rule corresponds to a state. In other words, the firing strength of each rule defines the degree to which the agent is in a state. Furthermore, the rules do not have fixed consequents; meaning there are no fixed (predefined) state-action pairs but through the exploration/exploitation algorithm arise the consequents of each rule (state-action pair). Thus, the FIS has competing actions for each rule and the rules have the form:  
\[\begin{array}{ll}\textbf{if}\;\textbf{\emph{x}}\;\textbf{\emph{is}}\;S_{i}& \textbf{\emph{then}}&\alpha[i,1]\\ &\text{or}&\alpha[i,2]\\ &\vdots&\\ &\text{or}&\alpha[i,k]\end{array}\]  
where \(\alpha[i,k]\) is the \(k\)th possible action in rule \(i\). For each possible action in rule \(i\) there is a corresponding value \(q[i,k]\). This corresponding value determines how \"good\" the corresponding action is, when the rule \(i\) is fired. The state \(S_{i}\) is defined by \((x_{1}\)_is \(S_{i,1}\) and \(x_{2}\) is \(S_{i,2}\dots\) and \(x_{n}\) is \(S_{i,n})\)_, where \(S_{i,j}\), \(j=1,\dots,n\) are fuzzy sets. Specifically, the algorithm of the fuzzy Q-Learning is briefly presented below.  
* Observe state \(x\)
* Take an action \(\alpha_{i}\) for each rule \(i\) according to exploration/exploitation algorithm
* Calculate the global output \(a(x)\) according to Equation (10)
* Calculate the corresponding value \(Q(x,a)\) according to Equation (11)
* Capture the new state information
* Calculate reward \(R(x,a,x^{\prime})\)
* Update q-values according to Equation (12)The fuzzy Q-Learning algorithm and its equation are described extensively in seven steps.  
1. Observation of the state \(\mathbf{x}\)
2. For each fired rule, one action is selected according to the exploration/exploitation strategy.
3. Calculation of the global output \(a(\mathbf{x})\) \[a(\mathbf{x})=\frac{\sum_{i=1}^{N}w_{i}(\mathbf{x})\;a_{i}}{\sum_{i=1}^{N}w_{i}(\mathbf{x})}\] (11) where \(a_{i}\) is as specified in Section 2.4.2 (consequents of the rule \(i\)) and corresponds to the selected action of rule \(i\).
4. Calculation of the corresponding value \(Q(\mathbf{x},a)\). \[Q(\mathbf{x},a)=\frac{\sum_{i=1}^{N}w_{i}(\mathbf{x})\;a_{i}\;q\big{[}i,i^{\dagger} \big{]}}{\sum_{i=1}^{N}w_{i}(\mathbf{x})}\] (12) where \(q\big{[}i,i^{\dagger}\big{]}\) is the corresponding q-value of the fired rule \(i\) for the selection of the action \(i^{\dagger}\) by the exploration algorithm.
5. Application of the action \(a(\mathbf{x})\) and observation of the new state \(\mathbf{x}^{\prime}\).
6. Calculation of the reward \(R(\mathbf{x},a,\mathbf{x}^{\prime})\).
7. Updating the q values according to: \[\Delta q\Big{[}i,i^{\dagger}\Big{]}=\eta\;\Delta Q\frac{w_{i}(\mathbf{x})}{\sum_{i =1}^{N}w_{i}(\mathbf{x})}\] (13) where \(\Delta Q=R(\mathbf{x},a,\mathbf{x}^{\prime})+\gamma\cdot Q(\mathbf{x}^{\prime},a^{*})-Q( \mathbf{x},a)\), \(Q(\mathbf{x}^{\prime},a^{*})=\frac{\sum_{i=1}^{N}w_{i}(\mathbf{x}^{\prime})\;a_{i}\;q \big{[}i,i^{*}\big{]}}{\sum_{i=1}^{N}w_{i}(\mathbf{x}^{\prime})}\) and \(q[i,i^{*}]\) is the selection of the action \(i^{*}\) that has the maximum \(Q\) value for the fired rule \(i\).