## 4 Applying the GDCAC Algorithm for EM Modeling Considering Renewable Power Penetration
### The Proposed GDCAC-Based EM Procedure Considering Renewable Power Penetration  
According to the mathematical principle of GDCAC algorithm introduced in Section 4.1, the step-by-step procedure of implementing the GDCAC algorithm for bilateral hour-ahead EM modeling considering renewable power penetration is described as follows:1. **Input:** for NRGenCOi (\(i=1,2,\ldots,N_{g1}\)) \(\mathbf{\phi}_{g}\): \(\mathbf{X}\rightarrow\mathbb{R}^{n}\), step length parameter series \(\left\{\alpha_{t}(g)\right\}_{t=1}^{\infty}\) and \(\left\{\beta_{t}(g)\right\}_{t=1}^{\infty}\); for DisCOj (\(j=1,2,\ldots,N_{d}\)) \(\mathbf{\phi}_{d}\): \(\mathbf{X}\rightarrow\mathbb{R}^{n}\), step length parameter series \(\left\{\alpha_{t}(d)\right\}_{t=1}^{\infty}\) and \(\left\{\beta_{t}(d)\right\}_{t=1}^{\infty}\); and parameters \(\sigma,m\) for every NRGenCO and DisCO. 2. \(T=1\). 3. Initialize the linear parameter vectors \(\mathbf{\theta}_{1}(^{gi})\) and \(\mathbf{\omega}_{1}(^{gi})\) for NRGenCOi (\(i=1,2,\ldots,N_{g1}\)), linear parameter vectors \(\mathbf{\theta}_{0}(^{dj})\) and \(\mathbf{\omega}_{0}(^{dj})\) for DisCOj (\(j=1,2,\ldots,N_{d}\)). 4. Random state generation: in iteration \(t\), a random point, \(\mathbf{x}_{t}=\left(x_{1,t},x_{2,t},\ldots,x_{v,t},\ldots,x_{N_{g2,t}}\right)\), is generated in the continuous state space \(\mathbf{X}\), which represents the continuous state space of power productions by all RPGenCOs. 5. In iteration \(t\), NRGenCOi (\(i=1,2,\ldots,N_{g1}\)) chooses and implements an action \(u_{gi,t}\sim N(\mathbf{\phi}_{g}(x_{t})^{T}\mathbf{\omega}_{t}(^{gi}),\sigma^{2})\) (\(u_{gi,t}\in\left[k_{gi,\min},k_{gi,\max}\right]\)) from state \(x_{t}\), DisCOj (\(j=1,2,\ldots,N_{d}\)) chooses and implements an action \(u_{dj,t}\sim N(\mathbf{\phi}_{d}(x_{t})^{T}\mathbf{\omega}_{t}(^{dj}),\sigma^{2})\) (\(u_{dj,t}\in\left[k_{dj,\min},k_{dj,\max}\right]\)) from state \(x_{t}\), and then ISO implements the congestion management model considering renewable power penetration represented by Equations (5)-(11). 6. NRGenCOi (\(i=1,2,\ldots,N_{g1}\)) observes the immediate reward \(r_{gi,t}\) using Equation (19) and DisCOj (\(j=1,2,\ldots,N_{d}\)) observes the immediate reward \(r_{dj,t}\) using Equation (20). 7. Discount factor setting: because \(\mathbf{x}_{t}\) is a stochastic variable independent from \(u_{gi,t}\) (\(i=1,2,\ldots,N_{g1}\)) and \(u_{dj,t}\) (\(j=1,2,\ldots,N_{d}\)), every NRGenCO and DisCO has no idea what the true value of \(\mathbf{x}_{t+1}\) while in iteration \(t\). Therefore, similar to [2], we assume that the discount factor \(\gamma_{t}\) for every NRGenCO and DisCO in iteration \(t\) equals 0. 8. Learning: in this step, \(\mathbf{\theta}_{t}(^{gi})\) and \(\mathbf{\omega}_{t}(^{gi})\) for NRGenCOi (\(i=1,2,\ldots,N_{g1}\)) as well as \(\mathbf{\theta}_{0}(^{dj})\) and \(\mathbf{\omega}_{0}(^{dj})\) for DisCOj (\(j=1,2,\ldots,N_{d}\)) are updated using TD(0) error and the gradient descent method: NRGenCOi: \[\delta_{gi,t}=r_{gi,t}+\gamma_{t}\mathbf{\phi}_{g}(\mathbf{x}_{t+1})^{T}\mathbf{\theta}_{ t}(^{gi})-\mathbf{\phi}_{g}(\mathbf{x}_{t})^{T}\mathbf{\theta}_{t}(^{gi})\] (35) \[\mathbf{\theta}_{t+1}(^{gi})=\mathbf{\theta}_{t}(^{gi})+\alpha_{t}(^{g}) \delta_{gi,t}\mathbf{\phi}_{g}(\mathbf{x}_{t})\] (36) \[\mathbf{\omega}_{t+1}(^{gi})=\mathbf{\omega}_{t}(^{gi})+\beta_{t}(^{g}) \frac{1}{1+e^{-m\delta_{gi,t}}}(u_{g,t}-\mathbf{\phi}_{g}(\mathbf{x}_{t})^{T}\mathbf{\omega }_{t}(^{gi}))\mathbf{\phi}_{g}(\mathbf{x}_{t}).\] (37) DisCOj: \[\delta_{dj,t}=r_{dj,t}+\gamma_{t}\mathbf{\phi}_{d}(\mathbf{x}_{t+1})^{T}\mathbf{\theta}_{ t}(^{dj})-\mathbf{\phi}_{d}(\mathbf{x}_{t})^{T}\mathbf{\theta}_{t}(^{dj})\] (38) \[\mathbf{\theta}_{t+1}(^{dj})=\mathbf{\theta}_{t}(^{dj})+\alpha_{t}(^{d}) \delta_{dj,t}\mathbf{\phi}_{d}(\mathbf{x}_{t})\] (39) \[\mathbf{\omega}_{t+1}(^{dj})=\mathbf{\omega}_{t}(^{dj})+\beta_{t}(^{d}) \frac{1}{1+e^{-m\delta_{dj,t}}}(u_{dj,t}-\mathbf{\phi}_{d}(\mathbf{x}_{t})^{T}\mathbf{ \omega}_{t}(^{dj}))\mathbf{\phi}_{d}(\mathbf{x}_{t}).\] (40)
9. \(T=t+1\). 10. If \(t\leq T\), return to (4).