## 2 Related Works
### RL-Based Charging Coordination for Home Batteries  
Unlike [14; 15; 16; 17; 18; 19; 20], our previous work [13] presented an RL approach for optimizing consumer-side battery storage integrated with renewable energy sources in smart grids. The proposed approach consists of two components: the agent, a trained neural network that determines optimal charging decisions, and the environment, which includes the home storage batteries. Their interactions are represented as tuples of state \((s_{t})\), action \((a_{t})\), and reward \((r_{t})\) at each time step \((t)\), defined as follows. * \(C_{\max}\): The maximum available power for charging, which represents the upper limit of power allocation to prevent grid overload. - \(SoC_{t}^{(t)}\): The state of charge for each consumer \(i\) at time \(t\). Consumers with lower SoC values have a higher priority for charging, while those with higher SoC values receive less priority to maintain fairness and efficiency. * Action \((a_{t})\): The action, denoted by \(a\), represents the power allocation decisions made by the RL agent for each customer's battery at a given time step \(t\). The action for each customer is a value between \((0,1)\), and the action space defines all possible power allocation combinations with a sum equal to or less than (\(\sum_{i=0}^{n}a_{i}\leq 1\)) that the agent can choose from. Each action element \(a_{i}\) corresponds to the amount of power allocated to customer \(i\) during the current time step \(t\). The agent aims to select an action that maximizes the expected cumulative reward over time. By effectively selecting actions within the action space, the RL agent can learn an optimal policy for power allocation. * Reward (\(r_{t}\)): The reward function provides feedback to the RL agent, guiding it toward optimal behavior. The reward function is a weighted sum of three components: energy usage efficiency (\(r_{g}\)), fairness (\(r_{f}\)), and consumer satisfaction (\(r_{c}\)). Each component addresses a specific objective of the charging coordination system, with their combined effect encouraging the agent to balance multiple goals effectively. \[r=r_{g}+r_{f}+r_{c}\] (1) The detailed descriptions of the three components are as follows:
* Energy Usage Reward (\(r_{g}\)): This reward ensures that the total power allocated remains within the available capacity \(C_{\text{max}}\). If the agent allocates more power than allowed, it incurs a large penalty. Otherwise, the reward increases as the allocated power decreases, so that we encourage the agent to save energy: \[r_{g}= \begin{cases}-10&\text{if }R_{t}>C_{\text{max}}\\ \dfrac{C_{\text{max}}-R_{t}}{C_{\text{max}}}&\text{if }R_{t}\leq C_{\text{ max}}\end{cases} \] (2) Here, \(R_{t}\) is the total power allocated at time \(t\). * Fairness Reward (\(r_{f}\)): This reward component incentivizes actions that lead to equal battery levels among consumers, promoting fairness in power distribution and minimizing disparities. It is defined as follows: \[r_{f}=-\text{CV}_{t}+1\] (3) where the Coefficient of Variation (CV) quantifies the deviation of battery levels from an equal distribution. It is calculated using the mean and standard deviation of the SoC for all consumers as: \[\text{SoC}_{\text{mean}}=\dfrac{1}{N}\sum_{i=1}^{N}\text{SoC}_{i}\] (4) \[\text{SD}=\sqrt{\dfrac{1}{N}\sum_{i=1}^{N}(\text{SoC}_{i}-\text{SoC}_{\text{ mean}})^{2}}\] (5) \[\text{CV}=\dfrac{\text{SD}}{\text{SoC}_{\text{mean}}}\] (6) Using Equation (3), the reward \(r_{f}\) is calculated such that a CV of 0 indicates perfect equality in battery levels, while a CV of \(-1\) represents the greatest disparity. To ensure the reward remains within the range \([0,1]\), we shift and scale it by applying a negative sign to \(CV\) and adding 1. This normalization yields:
* \(r_{f}=1\): Achieved when \(\text{CV}=0\), indicating all consumers have the same battery level. * \(r_{f}=0\): Achieved when \(\text{CV}=-1\), indicating significant disparities in battery levels.