## 4 Case Study
### SGD SARSA in the HKT System  
A common method of controlling the turbine system is through a quadratic feedback control law called k-omega-squared torque control [38]. If \(k\) is tuned properly, the HKT will track the optimal tip speed ratio \(\lambda^{*}\) for all flow velocities below \(u_{rated}\). The goal of the proposed RL method in the HKT system is to find a value or profile of values for \(k\) to ensure ample energy generation while reducing \(LC\) over the deployment. A high-level block diagram showing how the proposed RL-based control scheme is integrated with the k-omega-squared framework is shown in Figure 6, and a brief overview of the function of the proposed control system is as follows. At each time step, the RL agent samples the current state [\(k\), \(u_{0}\), \(T_{amb}\), \(c_{e}\)] and the reward for taking the previous action \(R\). From these sampled values, the RL agent updates the weight vector \(\mathbf{W}\) following (19) and selects a new action following the current policy \(\pi\). These actions result in a change in the feedback gain \(k\). The new value of \(k\) changes the generator torque command \(T_{gen}\) following the k-omega-squared framework and is realized by changing the q-axis current \(i_{q}\) following (6). From the new values of \(T_{gen}\) and \(i_{q}\), the operating conditions of the turbine and the power electronics are changed following the methods outlined in Section 2. The reward function block then samples the generator power \(P_{gen}\), the junction temperature of the power electronic device \(T_{j}\), and the cost of energy \(c_{e}\) to provide the RL agent with a meaningful representation of the quality of the last action selected following the current policy \(\pi\). In terms of the RL framework, the RL agent considers four states: the current value of \(k\), \(u_{0}\), \(T_{amb}\), and the current cost of energy \(c_{e}\). Each state is then normalized by some base value to represent the values of each state between 0 and 1. These base values are represented as \(k_{b}\), \(u_{b}\), \(T_{b}\), and \(c_{b}\), respectively. The states have been normalized such that the value of \(\sigma\) in (17) was suitable for all state/action dimensions, simplifying the design space. The actions the RL agent can take are incremental changes to the current value of \(k\), \(\Delta k\). Although the state space is continuous, the action space is discretized and normalized between \(-0.5\) and \(0.5\) using a normalization variable \(\Delta k_{b}\). The agent takes actions following an \(\varepsilon\)-greedy policy, where random actions are taken with some probability \(\varepsilon\), and actions that correspond to the highest action value (called \"greedy\" actions) are taken otherwise. Figure 6: A block diagram highlighting the interaction between the RL agent and the HKT system. Descriptions of the environment block (blue), reward function block (pink), and RL agent block (green) can be found in Sections 2 and 3.2. After each training epoch, the value of \(\epsilon\) is reduced using a linear decay parameter \(\epsilon_{d}\) as shown in (20). The \(\epsilon\)-greedy policy was chosen over other policies due to its popularity, simplicity, and ability to effectively manage the exploration-exploitation problem that is common in RL. \[\epsilon\leftarrow\epsilon\cdot\epsilon_{d} \tag{20}\]  
A common downfall of applying RL methods to continuous problems with multiple tasks is forgetfulness [39]. Forgetfulness relates to the phenomenon in which the learned parameters from one task are forgotten while the agent is being trained on a new task. Agents are especially prone to this issue when there are significant differences between rewards for each task. With the variability in environmental conditions between training years, the proposed RL method is highly susceptible to this unfortunate side effect. To avoid this problem, this work considers a dynamic definition of \(\alpha\) that is incrementally reduced as the agent is trained.