## 4 Machine Learning in Energy Trading
### Machine Learning Solution Strategies
\begin{table}
\begin{tabular}{p{128.0 pt} p{128.0 pt} p{128.0 pt} p{128.0 pt} p{128.0 pt}} \hline \hline
**Ref.** & **RL Algorithm** & **State** & **Action** & **Reward** \\ \hline \multirow{23}{*}{[233]} & \multirow{2}{*}{Q-learning, SARSA} & Current energy supply and demand & Buy/sell energy & Profit/loss from energy trading \\ \hline \multirow{23}{*}{[234]} & \multirow{2}{*}{Q-learning, DQN, DDPG} & Energy market prices & Adjust energy production & Cost savings/penalties \\ \hline \multirow{2}{*}{[235]} & \multirow{2}{*}{Fitted Q-iteration} & Battery SoC & Charge/discharge battery & Profit from battery usage \\ \hline \multirow{2}{*}{[236]} & \multirow{2}{*}{DQN, DDPG} & Weather conditions & Modify energy generation & Increased renewable energy use \\ \hline \multirow{2}{*}{[237]} & \multirow{2}{*}{PG, A3C} & Historical energy consumption & Optimize energy trading strategy & Improved performance \\ \hline \multirow{2}{*}{[238]} & \multirow{2}{*}{Q-learning, SARSA} & Market demand and supply trends & Adjust energy trading volume & Higher market share \\ \hline \multirow{2}{*}{[239]} & \multirow{2}{*}{DQN, SAC} & Competitor actions and strategies & Modify energy pricing & Increased market competitiveness \\ \hline \multirow{2}{*}{[240]} & \multirow{2}{*}{DDPG, SAC} & Grid condition and balancing services & Change trading strategy & Grid-balancing services, cost reduction \\ \hline \multirow{2}{*}{[241]} & \multirow{2}{*}{PPO} & Energy market regulations and policies & Comply with regulations or exploit loopholes & Avoid penalties or take advantage of incentives \\ \hline \multirow{2}{*}{[242]} & \multirow{2}{*}{SAC, DQN, TD3, PPO} & Customer demand patterns and preferences & Offer tailored energy products or services & Increased customer satisfaction or retention \\ \hline \multirow{2}{*}{[243]} & \multirow{2}{*}{Q-learning} & Market trends and competitor behavior & Adjust pricing or marketing strategies & Increased market share or competitive advantage \\ \hline \multirow{2}{*}{[244]} & \multirow{2}{*}{Q-learning, DQN, PPO} & Energy storage capacity and charging/discharging rates & Control energy storage systems & Efficient utilization and optimal charging and discharging \\ \hline \hline \end{tabular}
\end{table} 
Table 6: RL algorithm based on state, action, and reward functionality in energy trading. \begin{table}
\begin{tabular}{p{128.0 pt} p{128.0 pt}} \hline \hline
**Application** & **RL Algorithm** \\ \hline Bidding strategies [217] & SARSA \\ \hline Pricing mechanism [218] & DDPG \\ \hline Price optimization [219] & Deep double Q-learning \\ \hline Bidding strategies [220] & SAC \\ \hline P2P energy trading [221] & Fuzzy Q-learning \\ \hline Bidding strategies [222] & DDPG \\ \hline Bidding strategies, P2P energy trading [223] & Actor-critic \\ \hline P2P energy trading [224] & Q-learning \\ \hline P2P energy trading [225] & DQN \\ \hline P2P energy transaction [226] & Q-learning \\ \hline Demand response [227] & Q-learning \\ \hline Bidding strategies [228] & DQN \\ \hline Energy prices [229] & TD3 \\ \hline Demand response, bidding strategies [230] & Q-learning \\ \hline Trading strategies [231] & DQN \\ \hline Energy price optimization [232] & TRPO \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Application strategies for energy transactions in different RL algorithms. The policy optimization approaches mostly operate in continuous state and action spaces to find an optimal policy for an agent to maximize its cumulative reward in an environment. PPO is a state-of-the-art policy optimization algorithm trust region approach to ensure that the policy updates stay within a safe region to prevent large policy changes. PPO offers the best sample efficiency and good convergence properties. TRPO algorithm uses a similar trust region approach to ensure that policy updates are not too aggressive. It offers more stability and ensures that the policy changes are gradual. DDPG is a policy optimization algorithm specifically designed for continuous action spaces.