## 4 Case Study
### SGD SARSA in the HKT System
The value of \(\alpha\) begins at some initial value \(\alpha_{0}\) and is reduced by the number of training epochs \(n_{e}\), as follows:  
\[\alpha=\frac{\alpha_{0}}{n_{e}} \tag{21}\]  
This ensures that, the more the RL agent learns, the amount in which the weights vary from task to task is reduced to prevent the agent from overwriting what has previously been learned. As described above, the RBF function approximation method is defined by \(\mathbf{C}\) and \(\sigma\). For this work, \(n_{c}\) RBF center points are designated for each dimension of the state and action space. The center points in the state space are equally spaced values from 0 to 1. The possible actions the RL agent can take are defined directly by the center points of the RBF in the action dimension such that the possible actions the agent can take are defined as the set \(k_{b}\cdot\{-0.5,-0.25,0,0.25,0.5\}\). One of the novelties of this work is the definition of the reward function \(R\). Considering that the RL agent in this system should be maximizing energy generation while simultaneously minimizing the aging effects on the power electronic converter, this problem presents itself as a multi-objective reinforcement learning problem. It is typical for an RL agent to only handle problems with one reward. In these scenarios, it is common to use a linear combination of multiple reward functions, termed scalarization [40]. Scalarization often includes weighting of \(n\) reward functions depending on the application, where the values of weights and reward functions depend on the application, as shown in (22). However, it is quite challenging to determine meaningful weights for each reward signal. \[R=w_{1}\cdot r_{1}+w_{2}\cdot r_{2}+...w_{n}\cdot r_{n} \tag{22}\]  
In this work, an intuitive approach to determining the weighting of the respective rewards is developed using a monetary-based reward function. This methodology is similar to that presented in [15]. However, this work considers an estimate of the costs associated with operating the power electronic converter at a given value of \(T_{j}\), while other system-level costs are excluded (such as regular maintenance costs, environmental protection costs, and depreciation costs). First, a term related to the revenue from energy generation \(r_{g}\) is defined in (23), where \(T_{s}\) is the sample time for the control algorithm and \(c_{e}\) is the cost of energy in \(USD/kWh\). Many areas in the United States have energy prices that are dictated by a local energy market. Therefore, \(c_{e}\) becomes a time-varying function that depends on the amount of load and generation at a given time. \[r_{g}=\frac{P_{gen}\cdot T_{s}}{3.6\cdot 10^{6}}\cdot c_{e} \tag{23}\]Next, the costs \(r_{c}\) associated with operating at some \(T_{j}\) are approximated using (24). This cost-based reward considers the cost of the power electronic converter \(c_{c}\) and the expected lifetime of the converter \(L_{c}\), which are dependent on the components selected for the converter and the given application. This cost model utilizes the Arrhenius lifetime acceleration model \(A_{f}(T_{j})\) to estimate how the lifetime estimate is impacted when operating at temperatures other than the reference temperature \(T_{ref}\), where \(T_{j}\) and \(T_{ref}\) are provided in Kelvin [41]. The definition of \(A_{f}(T_{j})\) can be found in (25), where \(E_{a}\) is the activation energy for a given failure mechanism and \(k_{B}\) is [PERSON]'s Constant. \[r_{c}=A_{f}(T_{j})\cdot\frac{T_{s}}{L_{c}}\cdot c_{c} \tag{24}\]  
\[A_{f}(T_{j})=e^{\frac{E_{a}}{E_{B}}\cdot(\frac{1}{T_{ref}}-\frac{1}{T_{j}})} \tag{25}\]  
Lastly, a penalty term \(r_{p}\) is developed to deter the agent from applying \(\Delta k\), which would result in negative values of \(T_{gen}\), or from exceeding some pre-defined maximum feedback coefficient \(k_{max}\).