# A Deep Reinforcement Learning Approach for Microgrid Energy Transmission Dispatching
## 4 The Proposed HDQN Approach
### Implementation of HDQN  
To realize the optimal charging and discharging management and energy transmission dispatching, this paper uses the HDQN approach to control the actions of various devices in the microgrid. The upper-layer agent performs calculations based on the observed environmental state and its strategy at time \(t\), thereby selecting a corresponding action and transmitting the action to the lower-layer agent at the same time. Based on this, the lower-layer agent calculates according to its strategy to obtain a set of determined actions. All actions of the upper and lower layers of the agent act on the running environment and obtain the next environment state. The algorithm steps are shown in Algorithm 1. ```
1: Randomly initialize the Q-network, the Actor network and the Critic network with weights \(\theta^{q},\theta^{u},\theta^{c}\). 2: Initialize the target network Q-target network, the Actor-target network and the Critic-target network with weights \(\theta

## 5 Experimental Analysis

This experiment considers a typical microgrid system, including a photovoltaic system, a diesel generator, two batteries and two loads [21]. The time interval is \(\Delta t=1\) h. Reference [30] for the typical daily load and irradiation data. The batteries capacities are set to 100 kWh in the experiment. The rated power output and maximum charging/discharging rate of the batteries are 16 kW. The initial energy of the batteries is 100 kWh. The rated power of PV and DG are 210 kW and 120 kW, respectively. In this experiment, the Q-network of the upper-layer agent has three fully connected hidden layers. Through a large number of simulation experiments, the number of neurons is set to 300, 300 and 150 respectively. The activation function is the M-ReLU, whose hidden layer has 200 and 150 M-ReLU neurons [12]. The upper and lower layers use the same network parameters. The power transmission dispatching in this paper is a periodic task with a 24 h cycle. In this experiment, some initial parameters are summarized in Table 1.

Figures 3 and 4 show the cost curves of the SA, ADP and proposed approach under the same conditions. Figure 3 shows the total cost generated by the three dispatching methods. From the experimental results, it can be seen that the total cost does not increase substantially between 09:00 and 16:00. The reason is that photovoltaic is the main power supply method during this period, and the total cost mainly comes from transmission line losses. At other times, the photovoltaic output drops to zero. At the same time, the diesel generator and the batteries serve as the main power supply for the load. Therefore, the cost of power supply is relatively high. Figure 4 shows the cost per time step of the SA, ADP and proposed approach. These experimental results show that the proposed method spends less on each time step than the SA and ADP, and the dispatching effect is better. The results show that the proposed approach costs less than the SA and ADP in each time step and has a better dispatching effect. The cost of the SA, ADP and proposed approach to provide services for all loads in one day is USD 1309.49, USD 1091.92 and USD 1050.58, respectively. Compared with the other two approaches, the proposed approach saves USD 258.91 and USD 41.34, respectively. Obviously, the proposed approach provides the optimal dispatching scheme, which can support the operation of the microgrid with minimal operation cost. Figures 5 and 6 show the energy outputs of PV and DG at each time period. Through the analysis of the results, the energy outputs of PV and DG have a strong correlation. DG does not operate between 11:00 and 15:00, while the output power of PV is in a high operation state. At the same time, the energy output of PV can meet the energy demand of the load and provide energy for charging the energy storage battery. At other times, PV and DG work together to meet the energy demand of the load.