## 4 Machine Learning in Energy Trading
### Machine Learning Solution Strategies
It uses an AC architecture with a deterministic policy, which is suitable for problems with continuous action spaces. In RL algorithms, the action space represents the set of possible actions that an agent can take in each state. It can be discrete or continuous, depending on the nature of the problem. Discrete action spaces consist of a finite number of actions, while continuous action spaces are characterized by a range of possible action values. In deterministic policy function, all the actions remain the same until the policy changes from the set of environment state to the set of actions. A stochastic policy function is a conditional probability distribution from the set of states to the set of actions. As an action selection mechanism, the epsilon-greedy policy selects the action with the highest Q-value most of the time (greedy exploitation), but with a small probability epsilon, it chooses a random action (exploration). The tree search approach represents various scenarios, decision points, or states in a problem-solving domain. A decision tree strategy uses decision structures for problem-solving and decision-making tasks. Decision trees can be used for both classification and regression tasks. They are particularly useful for tasks that involve making a sequence of decisions based on multiple attributes or features. The SoftMax policy assigns probabilities to each action based on their respective Q-values. \begin{table}
\begin{tabular}{p{42.7 pt} p{42.7 pt} p{42.7 pt} p{42.7 pt} p{42.7 pt} p{42.7 pt} p{42.7 pt} p{42.7 pt} p{42.7 pt} p{42.7 pt} p{42.7 pt} p{42.7 pt}} \hline \hline
**Ref.** & **RL Algorithms** & **Objectives** & **Agent Learning Method** & **Learning Updates** & **Value/Policy** & **Policy Type** & **State Space** & **Action Space** & **Action Selection** \\ \hline
[245] & Q-learning & Energy trading & TD learning & Bellman equation & Value & OFF & Deterministic & Discrete & Discrete &
\begin{tabular}{} \end{tabular} \\ \hline
[246] & DQN & \
\begin{tabular}{} \end{tabular} & Neural network & Bellman equation & Value & OFF & Deterministic & Continuous & Discrete &
\begin{tabular}{} \end{tabular} \\ \hline
[247] & SARSA & Maximizing reward & TD learning & SARSA algorithm’s & Policy & ON & Stochastic & Continuous & Continuous &
\begin{tabular}{} \end{tabular} \\ \hline
[248] & AC & Demand flexibility & Gradient ascent & A2C algorithm & Policy+ Value & N/A & Stochastic & Continuous &
\begin{tabular}{} \end{tabular} \\ \hline
[249] & SAC & Demand flexibility & Gradient descent & Entropy realization method & Policy+ Value & N/A & Stochastic & Continuous &
\begin{tabular}{} \end{tabular} \\ \hline
[250] & TD & \
\begin{tabular}{} \end{tabular} & Bootstrapping approach & Monte Carlo methods & Value & OFF & Stochastic & Continuous & Discrete &
\begin{tabular}{} \end{tabular} \\ \hline
[251] & PPO & Energy trading & Gradient descent & Monte Carlo estimation & Policy & ON & Stochastic & Continuous &
\begin{tabular}{} \end{tabular} \\ \hline
[252] & TRPO & Energy trading & Monte Carlo estimation & Neutral networks & Policy & ON & Stochastic & Continuous &
\begin{tabular}{} \end{tabular} \\ \hline
[253] & PG & \
\begin{tabular}{} \end{tabular} & Gradient ascent & Monte Carlo algorithm & Policy & ON & Stochastic & Continuous &
\begin{tabular}{} \end{tabular} \\ \hline
[254] & DDFG &
\begin{tabular}{} \end{tabular} & Neural network & Bellman equation & Policy & ON & Deterministic & Continuous & Continuous & Decision tree \\ \hline
[255] & DDGP &
\begin{tabular}{} \end{tabular} & Deep neural market & Bellman network & Policy & ON & Deterministic & Continuous & Continuous & Decision tree \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Summary of RL algorithms based on single agent type, policy, state space, action space, and action selection.