# Research on High-Efficiency Routing Protocols for HWSNs Based on Deep Reinforcement Learning
## 4 Improved DQN Algorithm
### Design of HDQN Algorithm
The BS serves as the final destination for the data in most cases, especially in hierarchical network structures. This packet header structure is critical for ensuring that data packets are correctly routed through the network to their intended destinations, with clear identification of the source, the relevant cluster head, and the BS. Additionally, to enhance the learning efficiency of the HDQN algorithm, the following DQN learning-related information has been innovatively included:  
1. R value: This represents the cumulative expected reward value for performing a specific action in a given state, helping to evaluate the long-term benefits of actions taken by the Agent;
2. Remaining energy of current heterogeneous node: This indicates the remaining energy level of the current heterogeneous node, which is crucial for assessing its capability to perform tasks and transmit data;
3. Previous hop packet ID: This field denotes the packet ID and address of the previous hop's source node, aiding in tracking the packet's origin and its route through the network;
4. Distance to neighboring nodes: This represents the physical distance between the current node and its neighboring nodes, which is important for determining optimal routing paths and connectivity;
5. Distance from receiving node to BS: This indicates the distance from the current node to the BS, providing essential information for optimizing the transmission path and efficiently managing energy resources. The improved packet header structure facilitates the efficient and real-time learning of the HDQN algorithm, thereby enhancing the overall energy efficiency of the HWSNs. Algorithm 1 describes the pseudo-code for solving the information transmission path selection problem in HWSNs using the HDQN algorithm. The input to the algorithm is  
Figure 6: Improved HDQN algorithm packet header. the current state information of the HWSNs, and the output is the optimized information transmission path selection policy. The process begins by initializing the parameters for the HDQN algorithm, the evaluation network, the target network, and the experience replay pool. Typically, the Agent selects the action with the highest Q-value. However, to ensure exploration of the environment in the current HWSNs, there is also a certain probability of randomly selecting an action. During execution, ordinary nodes directly transmit sensed information to the cluster head node. Subsequently, the Agent calculates the corresponding reward based on the reward function and updates it to the next state. After the state update, the learning process begins. To obtain new network inputs and improve its learning efficiency, the HDQN algorithm transfers previously accumulated stored state from experience replay memory to the experience replay pool. It then randomly samples mini-batches from this pool and uses them to update the evaluation network, calculating the target Q-value for the current state. This entire process is iteratively performed across multiple training and learning cycles, ultimately yielding the optimized information transmission path policy for the HWSNs. ```
Input: Node state information Output: Routing policy Initialize the evaluation network and target network parameters using parameter \(\delta\). Initialize the experience replay memory \(D\). Set the super node as the Agent to interact with the environment. Iterate over episodes \(\{1,2,\cdots,N^{\text{eps}}\}\): Initialize state information \(s_{t}\). Iterate over time steps \(t=\{1,2,\cdots,T\}\). Obtain the current state \(s_{t}\). Select action \(a_{t1}=\text{argmax}Q(s_{t},a_{t})\) according to the \(\epsilon\)-greedy strategy. Randomly select action based on the \(1-\epsilon\) strategy. Ordinary nodes forward data packets to the higher-level node, and obtain the corresponding reward \(r_{t}\) and next state information \(s_{t+1}\) according to Equation (18). Update the current state to the new state to acquire updated network input. Store action \(\{s_{t},a_{t},r_{t},s_{t+1}\}\) in the experience replay memory.