## 4 Machine Learning in Energy Trading
### RL Approaches and Algorithms  
RL, as an ML strategy, can be harnessed to address energy trading-related challenges by optimizing the decisions made by participants or agents in dynamic and uncertain environments. Practically, the game player/agent(s) need to select an action from a set of possible actions. The chosen action is based on the agent's policy and strategy. The agent updates the policy without a value function, improving with each learning step by adjusting weights. The basic RL model, including policy evaluation and reward, is illustrated in Figure 12.  
The core concept of RL is to learn a policy that maximizes the cumulative reward [211]. The agent or game player observes the current state of the environment, takes an action, and receives a reward based on the outcome of that action. The agent then uses this feedback to update its policy and maximize its expected cumulative reward. The RL approach proves effective, particularly when the goal is to sustain equilibrium in energy trading games across various computations and when assessing financial payouts within an environment characterized by incomplete information [212]. Nevertheless, the RL approach encounters challenges in energy trading games when prosumers opt to keep their strategies hidden from trading partners and competitors, adopting a non-cooperative stance. Additionally, the distribution of payoffs is often not discussed, finalized, or shared in advance. RL algorithms are strategic trade-offs between exploring new actions and exploiting those actions that are optimal in the specific domain. RL algorithms constantly adapt to a changing environment, which makes exploration an important consideration. Thus, RL is an equilibrium between exploration and exploitation. In RL, learning is normally classified based on the model-based and model-free algorithms. Figure 13 illustratively shows the classification of RL algorithms. Model-based RL draws on control theory, assumptions, and approximations, which can result in a complex framework [213]. This approach learns optimal behavior indirectly by utilizing an internal environmental model, employing search or planning techniques, and subsequently executing a sequence of appropriate actions while observing the resulting transitions and outcomes, including the next state and the immediate action reward function. Model-free RL algorithms do not estimate the transition probability model and action reward function strategies [214]. These algorithms use a trial-and-error approach to learn directly from the state/action values or policies to achieve optimal performance but without using estimation, approximation, or real-world models. These algorithms  
Figure 12: RL model with the policy evaluation and reward. are adaptable, trajectable, linear, and broader in learning domains, but are still simple and faster [215]. In this work, more focus will be given to model-free RL algorithms specially used for energy trading applications in the LEM domain. The two major variants of model-free learning approaches are value-based and policy-based. In value-based learning, the players learn from trajectory sampling of the same environment by updating and improving the action (Q) value for the action-value function at each defined iteration until it optimally converges [216]. Q-learning (action(s)) is the most popular example that learns from another policy. Fitted learning and deep Q-learning (multiple actions) are the two major variants of Q-learning. In the Q-learning approach, the learning agent chooses the activities based on state-action values. After every learning step, all the Q-action values are updated using the Bellman equation; this means the action of learning is dependent on any value function. To define and determine the optimum action at a required state, all the Q-values must be initialized at the input state.