## 5 Results
### Network Structure  
The structures of the actor network and critic network designed in this paper are shown in Figure 4. In order to prevent overfitting of the network, layer normalization (LN) is applied to the output vector of the hidden layer. The activation function of the hidden layer of the actor network was a rectified linear unit (ReLU), and the activation function of the output layer was a hyperbolic tangent function (tanh). The activation function of the output layer of the critic network was still ReLU. The optimizer used in the gradient descent process was the adaptive moment estimation optimizer (Adam). \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Parameter** & **Value** & **Parameter** & **Value** \\ \hline \(\varepsilon_{m}\) & 1.1 & \(\int_{base}^{d}\)/\(\Psi\) & 1 \\ \(\varphi_{base}^{d}\)/kW & 100 & \(p^{st}\)/kW & 200 \\ \(P_{thr}\)/kW & 50 & \(\gamma\) & 0.9 \\ \(a_{j}\)/(\(\Psi\)/kW\({}^{2}\)h) & [0.0015.0.003] & \(b_{j}\)/(\(\Psi\)/kWh) & [0.001,0.026] \\ \(c_{j}\)/(\(\Psi\)/h) & [1,1.8] & \(P_{j}^{g,min}\)/kW & [0] \\ \(p_{j}^{g,max}\)/kW & [60,80] & \(k_{B}\)/(\(\Psi\)/kWh) & 0.02 \\ \(exp\)/kWh & 80 & \(p^{8,min}\)/kW & \(-50\) \\ \(p^{8,min}\)/kW & 50 & \(SOC^{min}\) & 0.2 \\ \(SOC^{max}\) & 0.8 & \(\varepsilon_{c}\)/\(\varepsilon_{disc}\) & 0.9 \\ \(\alpha_{1}\)/(\(\Psi\)/kWh) & 0.14 & \(\Delta\)/(\(\Psi\)/kWh) & 0.0035 \\ \(K_{d}\) & 0.3 & \(\int_{base}^{m}\)/\(\Psi\) & 2 \\ \(\varphi_{base}^{m}\) & 1 & \(\beta_{1}\) & 0.6 \\ \(\beta_{2}^{m}\) & 0.4 & \(-\) & \(-\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Parameters. Figure 4: The structures of actor and critic networks: (**a**) actor network; (**b**) critic network. It can be seen in Figure 5 that after 200 episodes of training, all agents could quickly learn their optimal strategies and reach the equilibrium point. As shown in Figure 5a, the reward of the DSO agent was zero at the initial stage, but it could be stabilized to about 100 after 150 episodes. At the initial stage, the reward of the DSO agent rose sharply, and the reward of the MG agents dropped rapidly. This is because DSO makes full use of its dominant position before MG agents learn the best strategy. Once all MG agents learn their optimal strategies, the dominant position of the DSO will be restrained. Compared with the other MGs, the reward of MG3 is larger, which is due to its higher proportion of renewable energy. Differently from the agents in Figure 5, the agents in Figure 6 experienced a long period of fluctuation and did not reach a stable stage until 400 rounds. However, the fluctuations were still large. This fully shows that the environment of multi-agent game is unstable. The introduction of centralized training and priority sampling strategy can significantly improve the learning efficiency of multiple agents. It can be seen in Equation (21) that the weight factor \(\alpha\) in the DSO's reward is used to maintain the balance between the DSO's profits and the stationary degree of power exchange at the PCC (the stationary degree is the opposite of Equation (3)). The retail electricity price strategy and reward of the DSO agent will be impacted by \(\alpha\). Figure 7 shows the optimal retail electricity price under different \(\alpha\) values. It can be seen in Figure 7a that the DSO tends to reduce retail electricity price as the weight factor \(\alpha\) increases. At time step 16, the retail electricity price changed from light blue, representing the lowest prices, to dark blue, representing low prices, and then to light blue, as \(\alpha\) increased from 0.4 to 1. At time step 48, the retail electricity price changed from white to dark blue, representing low prices. At time step 70, the retail electricity price changed from red, representing high prices, to blue, representing low prices. This is because the DSO will pay less attention to the power-selling profit when \(\alpha\) is increasing.