# Research on High-Efficiency Routing Protocols for HWSNs Based on Deep Reinforcement Learning
## 4 Improved DQN Algorithm
### Design of HDQN Algorithm  
To enhance the energy efficiency of HWSNs, this paper proposes the HDQN optimization algorithm for information transmission efficiency in such networks. The algorithm utilizes a short-distance, multi-hop information transmission path selection method based on the free-space energy model as much as possible. Assuming that in the HWSNs application scenario, the ordinary node \(node_{or\_a}\) forwards the sensed data and packet header information to the cluster head \(node_{CH-A}\). Then, it selects a super node as the Agent to exchange them with the environment. In addition, the cluster head \(node_{CH-A}\) broadcasts the packet header information redefined by the HDQN algorithm (including reward value, remaining energy, and distance to neighboring nodes) to the neighboring cluster head \(node_{CH-B}\) and the cluster head \(node_{CH-C}\). If cluster head \(node_{CH-C}\) does not receive the sensed data from the cluster head \(node_{CH-A}\) next, it will discard the header information content. Once the corresponding cluster head \(node_{CH-B}\) receives complete sensed data, itcontinues to transmit the sensed data and packet header information to the next relay node. The flowchart of the HDQN algorithm is shown in Figure 5.  
Figure 5 describes the flowchart of the HDQN algorithm, which addresses the information transmission path selection problem in HWSNs. The design of this algorithm has two main objectives. First, it aims to balance energy consumption across the entire HWSNs by assigning high-energy super nodes to act as Agents, cluster heads, and relay nodes, thereby handling more energy-intensive tasks. Second, it seeks to optimize information transmission by utilizing a short-distance, multi-hop method as much as possible. The first step of the HDQN algorithm involves the super node acting as an Agent to perceive the state \(s_{t}\) of the HWSNs environment. This state information is then fed into the DQN evaluation network. The evaluation network generates corresponding reward \(Q(s_{t},a_{t})\) values for all possible information transmission actions based on the current state \(s_{t}\) and selects the action \(a_{t}\) with the maximum Q-value for execution. Subsequently, the process of state, action, reward, and transition to the next state is stored as experience \(\{s_{t},a_{t},r_{t},s_{t+1}\}\) in the experience replay pool. To continuously train and improve the evaluation network, the HDQN algorithm randomly samples \(M\) mini-batches from the experience replay pool for training the evaluation network. Specifically, the loss function is computed according to Equation (20) to update the parameters of the evaluation network. \[L_{i}(\delta_{i})=\mathbb{E}_{s_{t},a_{t}}\Bigg{[}\bigg{(}r_{t}+\gamma\underset {a_{t+1}}{\max}Q(s_{t+1},a_{t+1}|\delta_{i-1}\,)-Q(s_{t},a_{t}|\delta_{i})\, \bigg{)}^{2}\Bigg{]} \tag{20}\]  
where \(\delta\) represents the parameters of the DNN and \(i\) denotes the iteration index. Through this method, the HDQN algorithm can determine the optimal path for information transmission in HWSNs. As shown in Figure 6, the HDQN algorithm improves the packet header structure by incorporating information relevant to DQN learning. Figure 5: Flowchart of HDQN algorithm. As illustrated in Figure 6, the original packet header structure contains the following key pieces of information relevant to data transmission:  
Data packet ID: This field serves as a unique identifier for the packet, ensuring that each packet can be uniquely recognized and tracked within the network;  
Source address: This indicates the network location of the sender node, specifying where the data packet originates;  
Cluster header address: This represents the address of the cluster head node to which the source node belongs. The cluster head acts as a local controller or aggregator for data coming from nodes within its cluster;  
BS address: This refers to the address of the BS where the packet is ultimately intended to be delivered.