## 3 RL-Inspired Controller
### Exploring Reward Strategies
#### 3.2.1 Only Positive (O-P) Reward Strategies  
The most intuitive approach seems to be rewarding the relative position of the system output to the rated (reference) value. The closer the output to the desired value, the bigger the reward. However, considering the distance (absolute value of the error), the reward grows with the error. To avoid this problem, a maximum error is defined, \(P_{err_{MAX}}\), and the absolute error is subtracted from it. This is called \"Position Reward Strategy\" (PRS), Equation (16). \[PRS:r_{t_{i}}=P_{err_{MAX}}-\left|P_{err}(t_{i})\right| \tag{16}\]  
This strategy only provides positive rewards and no punishments. Thus it belongs to the category only positive (O-P) reinforcement. As it will be seen in the results section, this is the cause of its lack of convergence when individually applied. The main drawback of O-P rewards is that the same actions are selected repeatedly, and many others are not explored. This means that the optimal actions are rarely visited. To solve it, exploration can be externally forced by greedy-methods [40] or O-P rewards can be combined with positive-negative reinforcement (P-N rewards). To illustrate the problem, let \(T^{\pi}_{(s_{\theta})}\) be initialized to 0 for all the states and actions, \(T^{\pi}_{(s_{\theta})}(t_{0})=0,\ \forall(s,a)\). At \(t_{0}\) the system is in the state \(s_{0}=s\). Since all actions have the same value in the table, any action \(a_{0}=a_{s_{0}}\) is at random selected. At the next control time \(t_{1}\) the state is \(s_{1}\), which can be different or equal to \(s_{0}\). The reward received is \(r_{1}>0\). The policy update algorithm modifies the value of the table associated to the previous pair (state, action) \(T^{\pi}_{(s_{0}\theta_{0})}(t_{1})=f_{\pi}(r_{1})>0\). Now a new action \(a_{1}\) associated with state \(s_{1}\) must be selected. If \(s_{1}\
eq s_{0}\) the action is randomly selected again because all the actions in the row have the value 0. However, if the state is the same as the previous one, the selected action is the same as in \(t_{0}\), \(a_{1}=a_{0}=a_{s_{0}}\) because \(f_{\pi}(r_{1})>0\) is the maximum value of the row. In that case, at the next control time \(t_{2}\) the table is updated \(T^{\pi}_{(s_{0},a_{0})}(t_{2})=f_{\pi}(r_{1},r_{2})\). With this O-P rewards this value always tends to be greater than 0, forcing the same actions to be selected. Only some specific QL configurations may give negative values. This process is repeated every control period. If the state action is different from all previous states, a new cell in the table is populated. Otherwise, the selected action will be the first action selected in that state. If the initialization of the table \(T^{\pi}_{(s,a)}\) is high enough (regarding the rewards), we can ensure that all actions will be visited at least once for OR, MAR and QL update policies. This can be a solution if the system is stationary, because the best action for each state does not change, so that once all the actions have been tested, the optimum one has necessarily been found. However, if the system is changing this method is not enough. In these cases, \(\epsilon\)-greedy methods have shown successful results [40]. In each control period, the new action is randomly selected with a probability \(\epsilon\) (forced exploration) or selected from the table \(T^{\pi}_{(s,a)}\) with a probability (\(1-\epsilon\)) (exploitation). Another possible measure that can be used with the O-P strategy to calculate the reward is the previous MSE. In this case, the reward is calculated by applying a time window to capture the errors prior to the current moment. We have called this strategy \"MSE reward strategy\" (MSE-RS) and it is calculated by Equation (17):  
\[MSE-RS:r_{t_{i}}=\frac{1}{\sqrt{\frac{1}{T_{w}}\sum_{j=i-k}^{i}\left[P_{err} \left(t_{j}\right)^{2}T_{sj}\right]}} \tag{17}\]  
where \(T_{w}\) is the time length window; \(T_{sj}\) is the variable step size at \(t_{j}\), and \(k\) is calculated so \(t_{i}-t_{i-k}=T_{w}\).