## 4 Applying the GDCAC Algorithm for EM Modeling Considering Renewable Power Penetration
### Introduction of Gradient Descent Continuous Actor-Critic Algorithm  
The GDCAC algorithm is a modified policy search actor-critic-based reinforcement learning method that can rapidly solve Markov decision-making problems with continuous state and action spaces. Based on the actor-critic structure [40], state and action spaces can be made continuous by using a linear combination of many basis functions. The detailed mathematical principle of GDCAC algorithm can be described as follows:  
By using a linear function [40], we estimate and repeatedly update in an agent's critic part a value function defined by the continuous state space\(\mathbf{X}\):  
\[\hat{V}(\mathbf{x})=\sum_{h=1}^{n}\phi_{h}(\mathbf{x})\theta_{h}=\mathbf{\phi}(\mathbf{x})^{T} \mathbf{\theta}\mathbf{x}\in\mathbf{X}, \tag{21}\]  
where \(\phi_{h}:\mathbf{X}\rightarrow\mathbf{R}(h=1,2,\ldots,n)\) represents the \(h\)th basis function of state \(x\in\mathbf{X}\). Then, the fixed basis function vector of state \(\mathbf{x}\in\mathbf{X}\) can be described as: \(\mathbf{\phi}(\mathbf{x})=(\phi_{1}(\mathbf{x}),\phi_{2}(\mathbf{x}),\ldots,\phi_{n}(\mathbf{x}) )^{T}\in\mathbf{R}^{n}\). The linear parameter vector \(\mathbf{\theta}\) can be described as: \(\mathbf{\theta}=(\theta_{1},\theta_{2},\ldots,\theta_{n})^{T}\in\mathbf{R}^{n}\). By using a linear function [40], we estimate and repeatedly update in an agent's actor part an optimal policy function \(\hat{A}:\mathbf{X}\rightarrow\mathbf{U}\) defined by the continuous state space \(\mathbf{X}\):  
\[u_{\mathbf{x}}{}^{(optimal)}=\hat{A}(\mathbf{x})=\mathbf{\phi}(\mathbf{x})^{T}\mathbf{ \omega}\quad\mathbf{x}\in\mathbf{X}, \tag{22}\]  
where \(\mathbf{U}\) represents the continuous action space of an agent, \(u_{\mathbf{x}}{}^{(optimal)}\in\mathbf{U}\) represents the optimal action in face of state \(\mathbf{x}\). The linear parameter vector \(\mathbf{\omega}\) can be described as:  
\[\mathbf{\omega}=(\omega_{1},\omega_{2},\ldots,\omega_{n})^{T}\in\mathbf{R}^{n}.\]  
An agent must generate a corresponding action \(u\in\mathbf{U}\) in the face of any state \(\mathbf{x}\in\mathbf{X}\) based on the policy maintained and repeatedly updated by its actor part. During the reinforcement learning process, in order to balance the exploration and exploitation [2; 16; 18; 20; 21; 25; 40], the policy must be established as an action-generating model that has the ability to explore. That is to say, the probabilities of selecting sub-optimal actions in the face of any state \(\mathbf{x}\in\mathbf{X}\) are non-zero. This paper employs a Gaussian distribution function as the policy corresponding to the actor part:  
\[\rho(\mathbf{x},u)=\frac{1}{\sqrt{2\pi}\sigma}\exp\biggl{\{}-\frac{1}{2\sigma^{2} }(u-\mathbf{\phi}(\mathbf{x})^{T}\mathbf{\omega})^{2}\biggr{\}}, \tag{23}\]  
where \(\sigma>0\) is a standard deviation parameter that represents the exploring ability of the algorithm. In order to determine the linear parameter vector \(\mathbf{\theta}\), the Mean-Squared Error (MSE) function of \(\mathbf{\theta}\) is defined as [40]:  
\[MSE(\mathbf{\theta})=\frac{1}{2}\int\limits_{\mathbf{x}\in\mathbf{X}}P^{(\rho)}(\mathbf{x })[V^{(\rho)}(\mathbf{x})-\mathbf{\phi}(\mathbf{x})^{T}\mathbf{\theta}]^{2}d\mathbf{x}, \tag{24}\]  
where \(P^{(\rho)}(\mathbf{x})\) is the probability density function of the state under policy \(\rho\). Hence, the global optimal value of \(\mathbf{\theta}\) defined as \(\mathbf{\theta}^{*}\) must satisfy [40]:  
\[MSE(\mathbf{\theta}^{*})\leq MSE(\mathbf{\theta}). \tag{25}\]Because there is no a priori knowledge about \(V^{(\rho)}(x)\), minimizing \(MSE(\mathbf{\theta})\) directly is impossible.