## 2 Principles of MDP and Reinforcement Learning
### Principles of RL  
RL provides a promising approach to solving the MDP problem and addresses the real-time decision-making/control problem in a complex, stochastic, and highly dynamic system environment. A general interaction process between the RL agent and the environment is presented in Figure 1. After receiving the current states from the environment, an RL agent generates a corresponding action using its policy; then, the environment provides the state at the next time step (\(s^{\prime}\)) and calculates the corresponding reward (\(r^{\prime}\)) after the control action is executed in the environment. Through such interactions, the RL agent optimizes its policy (typically modeled as neural networks) to maximize the accumulated rewards. In this way, the RL agent will gradually master the control problem after a certain training period. Successful applications of RL include DeepMind's AlphaGo, AlphaStar, ATARI games, self-driving cars, and many others [22]. A soft actor-critic (SAC) RL algorithm that also uses the value function and Q function is chosen in this work. Traditional RL algorithms are designed to maximize the expected reward \(\sum\limits_{t}E_{(s_{t};a_{t})\sim p_{\pi}}[R(s_{t},\ a_{t})]\) only; however, the SAC algorithm trains stochastic policies using neural networks with entropy regularization. In other words, SAC maximizes both the expected reward and the control policy entropy value [23]. The SAC's optimal policy, denoted as \(\pi^{*}\), can be formulated in Equation (4):  
\[\pi^{*}=\arg\underset{\pi}{\max}\underset{t}{\sum}E_{(s_{t};a_{t})\sim p_{ \pi}}[R(s_{t},\ a_{t})+\alpha H(\pi\cdot|s_{t})] \tag{4}\]  
where \(H(\pi\cdot|s_{t})\) is the entropy value of the control policy at the state \(s_{t}\), and \(a\) is defined as the temperature coefficient that balances exploration and exploitation during the training process in providing more effective control actions of the agent. The policy evaluation and updates of the SAC RL agent can be achieved by training NNs with stochastic gradient descent values. The soft value function networks are trained using Equation (5):  
\[J_{V}(\psi)=E_{s_{t}\sim D}\Big{[}(V_{\psi}(s_{t})-V_{soft}(s_{t})\Big{]}^{2} \tag{5}\]  
where  
\[V_{soft}(s_{t})=E_{a_{t}\sim\pi}\Big{[}Q_{soft}(s_{t},\ a_{t})-\alpha log\pi(a_ {t}|s_{t})\Big{]} \tag{6}\]  
The soft Q function can be trained by minimizing Equation (7):  
\[J_{\theta}(Q)=E_{(s_{t},\ a_{t})\sim D}\big{[}Q_{\theta}(s_{t},\ a_{t})-\hat{ Q}(s_{t},\ a_{t})\big{]}^{2} \tag{7}\]  
where  
\[\hat{Q}(s_{t},\ a_{t})=R(s_{t},\ a_{t})+\gamma E_{s_{t+1}\sim p}\big{[}V_{ \hat{\psi}}(s_{t+1})\big{]} \tag{8}\]  
Figure 1: Interaction between RL agent and power system environment. \(V_{\psi}(s_{t+1})\) represents the target value network of SAC. The policy parameters are then updated by minimizing the expected Kullback-Leibler (KL) divergence, given in Equation (9):  
\[ \begin{split} J_{\pi}(\phi)=D_{KL}(\pi(\cdot|s_{t})||(\exp\Big{(} \frac{1}{\pi}Q_{\theta}(s_{t},\cdot)\Big{)}-\text{log}Z(s_{t}))\\ =E_{s_{t}\sim D}\Big{[}E_{a_{t}\sim\pi_{\phi}}\big{[}\text{alog} \big{(}\pi_{\phi}(\text{a}_{t}|\text{s}_{t})\big{)}-Q_{\theta}(a_{t},\ s_{t}) \big{]}\Big{]}\end{split}  \tag{9}\]  
Compared with other RL algorithms, SAC performs better regarding sampling efficiency and stability. In the objective function of the control strategy, the temperature coefficient \(\alpha\) determines the weight of the entropy and the reward value, thereby controlling the random sampling degree of the optimal strategy. It is worth noting that if a fixed temperature coefficient \(\alpha\) is used, the SAC agent becomes unstable as the number of training samples increases. In order to solve this problem and improve the convergence speed of agent training, this paper adopts the method of automatically updating the temperature coefficient. The temperature coefficient can be changed automatically as the control strategy is updated to explore more feasible solutions.