# A Deep Reinforcement Learning Approach for Microgrid Energy Transmission Dispatching
## 4 The Proposed HDQN Approach
### Implementation of HDQN
Figure 7 shows the power loss within 24 h, and the results show that power loss is worth paying attention to in the economical operation of microgrids. The total power losses of the SA and the ADP are 199.91 kW and 130.87 kW, respectively. Using the proposed approach, the total power losses are 113.77 kW, saving 86.14 kW and 17.1 kW in one day, respectively. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Actor learning rate & 0.0005 \\ Critic learning rate & 0.0001 \\ Batch size & 256 \\ Steps in one episode (T) & 24 \\ Discount factor & 0.95 \\ Iterates & 1000 \\ Calculation accuracy & 0.0001 \\ Replay memory & 50,000 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Parameters of the HDQN for the Microgrid Energy Dispatching Problem. Figure 4: The phase cost within 24 h for three approaches. Figure 5: The energy output of PV within 24 h.

Figure 3: The total cost within 24 h for three approaches. Figure 6: The energy output of DG within 24 h.

The discount factor in the approach indicates the impact of the current behavior on the future, and the larger the discount factor, the more the model focuses on the impact generated by the current behavior. Therefore, the discount factor has an important impact on the convergence of the approach. In Figure 8, it is clear that different discount factors have certain influences on the system, but the whole system is within the error range. Different discount factors can make the approach converge to the optimal value and have the same convergence performance. It can be seen that the approach is effective for solving the problem. As can be seen above, when the total power supply of the microgrid is greater than the total demand, the proposed approach can achieve optimization of the operation and maintenance cost. But, if there is a lack of diesel generators in the microgrid (or if the output power of the diesel generators is zero), the total power supply may not fully meet the total demand, and the total power quality will be decreased. In this case, it is necessary to dispatch the power supply for different loads to ensure that the overall operation of the microgrid is satisfied to the greatest extent in the case of power shortages. Figures 9 and 10 illustrate the total cost and step cost of using the SA, ADP and proposed approach without a diesel generator. It is still clear that in the absence of a diesel generator, the total cost and cost per time step obtained by using the proposed approach are lower than the SA and the ADP. The total cost of the ADP is USD 80.4 per day, while the total cost of the SA is USD 122.5 per day. The proposed approach saves USD 12 and USD 54.5 per day, with savings rates of about 15% and 44%, respectively. When a microgrid system does not have a diesel generator, the most important costs during the system operation are the line loss and the loss of energy storage batteries. Figure 8: The cost error of each time step to different discount factors. Figure 7: The power loss of the microgrid within 24 h.

Without a diesel generator, the total cost is reduced, but the quality of the power supply is greatly affected, and the total power supply cannot fully meet the load demand. ## 6 Conclusions

In this paper, the optimal energy transmission dispatching approach of the microgrid is introduced. An optimal approach for energy transmission dispatching based on an HDQN is proposed to achieve energy storage battery management and transmission energy allocation of the microgrid. To reduce complexity, the system is designed in two layers, each represented by an agent. The upper layer first makes charging and discharging decisions for the battery, and the lower layer uses the power on each transmission line as a decision variable, based on the decision results of the upper layer, to obtain a decision result that minimizes the overall operation cost of the system. Simulation results show that the proposed method combines the advantages of the HDQN to solve the problem of power allocation optimization in the microgrid.