## 3 RL-Inspired Controller  
The reinforcement learning approach consists of an environment, an agent and an interpreter. The agent, based on the state perceived by the interpreter \(s_{t}\) and the previous rewards provided by the interpreter \(r_{1\ldots t}\), selects the best action to be carried out. This action, \(a_{t}\), produces an effect on the environment. This fact is observed by the interpreter who provides information to the agent about the new state, \(s_{t+1}\), and the reward of the previous action, \(r_{t+1}\), closing the loop [38; 39]. Some authors consider that the interpreter is embedded in either the environment or the agent; in any case, the function of the interpreter is always present. Discrete reinforcement learning can be expressed as follows [40]:  
* \(S\) is a finite set of states perceived by the interpreter. This set is made with variables of the environment, which must be observable by the interpreter and may be different to the state variables of the environment. * \(A\) is a finite set of actions to be conducted by the agent. * \(s_{t}\) is the state at \(t\)
* \(a_{t}\) is the action performed by the agent when the interpreter perceives the state \(s_{t}\)
* \(r_{t+1}\) is the reward received after action \(a_{t}\) is carried out
* \(s_{t+1}\) is the state after action \(a_{t}\) is carried out
* The environment or world is a Markov process: \(MDP=\langle s_{0},a_{0},r_{1},s_{1},a_{1},r_{2},s_{2},a_{2}\ldots\rangle\)
* \(\pi:S\times A\rightarrow[0,1]\) is the policy; this function provides the probability of selection of an action \(a\) for every pair \((s,a)\)
* \(p^{a}_{ss^{\prime}}=\Pr[s_{t+1}=s^{\prime}|s_{t}=s\wedge a_{t}=a]\) is the probability that a state changes from \(s\) to \(s\prime\) with action \(a\)
* \(p^{\pi}(s^{\prime},a^{\prime})\) is the probability of selecting action \(a^{\prime}\) at state \(s^{\prime}\) under policy \(\pi\)
* \(r^{a}_{s}=E[r_{t+1}|s_{t}=s\wedge a_{t}=a]\) is the expected one-step reward
* \(Q^{\pi}_{(s,a)}=r^{a}_{s}+\gamma\sum_{s^{\prime}}p^{a}_{ss^{\prime}}\sum_{a^ {\prime}}p^{\pi}(s^{\prime},a^{\prime})Q^{\pi}_{(s^{\prime},a^{\prime})}\) is the expected sum of discounted rewards  
The scheme of the designed controller inspired by this RL approach is presented in Figure 1. It is composed by a state estimator, a reward calculator, a policy table, an actuator, and a method to update \begin{table}
\begin{tabular}{c l l} \hline \hline
**Parameter** & **Description** & **Value/Units** \\ \hline \(L_{a}\) & Inductance of the armature & 13.5 mH \\ \(K_{g}\) & Constant of the generator & 23.31 \\ \(K_{\phi}\) & Magnetic flow coupling constant & 0.264 V/rad/s \\ \(R_{a}\) & Resistance of the armature & 0.275 \(\Omega\) \\ \(R_{L}\) & Resistance of the load & 8 \(\Omega\) \\ \(\int\) & Inertia & 6.53 kg m\({}^{2}\) \\ \(R\) & Radio of the blade & 3.2 m \\ \(\rho\) & Density of the air & 1.223 kg/m\({}^{3}\) \\ \(K_{f}\) & Friction coefficient & 0.025 N m/rad/s \\ \([c_{1},c_{2},c_{3}]\) & \(C_{p}\) constants & [0.73, 151, 0.58] \\ \([c_{4},c_{5},c_{6}]\) & \(C_{p}\) constants & [0.002, 2.14, 13.2, 18.4] \\ \([c_{7},c_{8},c_{9}]\) & \(C_{p}\) constants & [18.4, \(-\)0.02, \(-\)0.003] \\ \([K_{\phi},T_{0}]\) & Pitch actuator constants & [0.15, 2] \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Parameters of the wind turbine model. the policy. The state estimator receives the output power error (\(P_{err}\)), defined as the difference between the rated power \(P_{ref}\) and the current output power \(P_{out}\), and its derivative, \(\hat{P}_{err}\). These signals are discretized and define the state \(s_{t}\in S\), where \(t\) is the current time. The interpreter is implemented by the state estimator and the reward calculator. The agent includes the policy, the policy update algorithm and the actuator. Both interpreter and agent form the controller (Figure 1). The policy is defined as a function \(\pi:S\to A\) which assigns an action \(a_{t}\in A\) to each state in \(S\).