## 3 RL-Inspired Controller
### Exploring Reward Strategies
#### 3.2.2 Positive Negative (P-N) Reward Strategies  
Unlike O-P reinforcement, P-N reward strategies encourage the natural exploration of actions that enable convergence of learning. The positive and negative rewards compensate the values of the table \(T_{(s,s)}^{\pi}\), which makes it easier to carry out different actions even if the states are repeated. An advantage of natural exploration over \(\epsilon\)-greedy methods is that their behavior is more deterministic. This provides more repeatable results with fewer iterations. However, the disadvantage is that if rewards are not well balanced, the exploration may be insufficient.  
To ensure that the rewards are well balanced, it is helpful to calculate the rewards with some measure of the error variance. PRS, MSE-RS, Mean-RS perform an error measurement in a specific period of time. They do provide neither a measure of the variation nor how quickly its value changes. A natural evolution of PRS is to use speed rather than position to measure whether we are getting closer to or away from rated power and how fast we are doing so. We call it \"velocity reward strategy\" (VRS), Equations (19) and (20).  
\[r_{v}=\left\{\begin{array}{cc}-\dot{P}_{err}(t_{i})&P_{err}(t_{i})>0\\ \dot{P}_{err}(t_{i})&P_{err}(t_{i})\leq 0\end{array}\right., \tag{19}\]  
\[VRS:r_{t_{i}}=\begin{cases}-r_{v}&sgn\Big{(}\dot{P}_{err}(t_{i})\Big{)}\
eq sgn \Big{(}\dot{P}_{err}(t_{i}-1)\Big{)}\wedge\Big{|}\dot{P}_{err}(t_{i})\Big{|}< \Big{|}\dot{P}_{err}(t_{i}-1)\Big{|}\\ r_{v}&\text{otherwise}\end{cases} \tag{20}\]  
The calculation is divided into two parts. First, \(r_{v}\) is calculated to indicate whether we are getting closer to or away from the nominal power, Equation (19). If the error is positive and decreases, we are getting closer to the reference. The second part is to detect when the error changes sign and the new absolute error is less than the previous one. It would be a punished action according to Equation (19) but when this case is detected, the punishment becomes a reward, Equation (20).  
A change in the MSE-RS can also be measured as Equation (17). This produces a new P-N reward, the \(\Delta\)MSE-RS, which is calculated by Equation (21):  
\[\Delta MSE-RS:r_{t_{i}}=\sqrt{\frac{1}{T_{w}}\sum_{j=i-1-k}^{i-1}\Big{[}P_{err }\Big{(}t_{j}\Big{)}^{2}T_{sj}\Big{]}}-\sqrt{\frac{1}{T_{w}}\sum_{j=i-k}^{i} \Big{[}P_{err}\Big{(}t_{j}\Big{)}^{2}T_{sj}\Big{]}} \tag{21}\]  
Comparing Equation (17) and Equation (21) it is possible to observe how the reward is calculated by subtracting the inverse of MSE-RS at \(t_{i}\) from the inverse of MSE-RS at \(t_{i-1}\). In this way, if the MSE is reduced, the reward is positive; otherwise it is negative.  
Similar to MSE, a P-N reward strategy can be obtained based on the mean value of the error. It is calculated with Equation (22) and is called \(\Delta\)_Mean - RS_.  
\[\Delta\)_Mean - RS: \(r_{t_{i}}=\left|\frac{1}{T_{w}}\sum_{j=i-1-k}^{i-1}\Big{[}P_{err} \Big{(}t_{j}\Big{)}T_{sj}\Big{]}\right|-\left|\frac{1}{T_{w}}\sum_{j=i-k}^{i} \Big{[}P_{err}\Big{(}t_{j}\Big{)}T_{sj}\Big{]}\right|\)_  
As will be shown in the results, the \(\Delta\)_Mean - RS_ improve the mean value of the error compared to other reward strategies.  
In addition to these P-N rewards, it is possible to define new rewards strategies by combining O-P with P-N rewards, such as PRS-VRS. However, the combination of P-N rewards between them does not generally provide better results; even in the same cases this combination can produce an effective O-P reward if they always have the same sign.