## 3 Proposed Algorithm
### Proposed Algorithm Description  
The proposed real-time data transmission scheduling algorithm based on deep Q-learning (RS-DQL) comprehensively considers the influences of communication constraints and interference between sensor nodes in WSNs, the remaining cutoff time, and the variation of the remaining hop count in the process of data transmission and scheduling. It uses deep neural networks to evaluate state-action mapping relationships and is updated by Q-learning methods. In addition, the empirical replay was introduced to reduce data relevance and adapt to the randomness of the training process. The idea of RS-DQL algorithm is as follows: Firstly, the data transmission and communication interference model of WSNs is constructed to determine the concurrent node-set based on the most urgent data. The Q-learning algorithm is used to acquire partial state transfer information (including the current state, the action to be performed, the reward to be obtained, and the next state) and store it into the experience pool after a certain time slot. This process does not train the SAE network model. After a period of time, samples are extracted from the experience pool and combined with the DQN network model for supervised training of the SAE network. In the process of network model learning and training, the system gradually rewards the actions with less packet loss for data transmission to achieve an approximate optimal scheduling algorithm. The scheduling algorithm flow of RS-DQL is shown in Figure 5. Firstly, before the experience pool is full, the SAE network model randomly selects actions. The Q-learning algorithm learns a part of the state and action data based on the selected actions. After the experience pool is full, the SAE network is gradually trained. Its parameters are updated with supervision during the learning process. When the system shifts to the hidden state, the SAE network recommends the system's actions in this state, performs the actions, updates the Q value network, etc. Repeat the learning process until the loss function reaches the target accuracy or the expected number of training sessions. Finally, the data transmission scheduling of the system is carried out by the state-action mapping in the trained SAE network model. The process description of the real-time data transmission scheduling algorithm based on deep Q-learning can be obtained from the above system model and scheduling strategy. However, this algorithm studies the scheduling problem of multiple concurrent data transmission in WSNs. How to allocate data transmission tasks in each time slot is related to behavior acquisition and training effect in the process of deep Q-learning. According to the exploration and development strategy, the most urgent data are first determined as the current time slot execution action \(a\). Based on the most urgent data and system status,  
Figure 5: Flow chart of the RS-DQL algorithm. multiple data that can execute the transmission task simultaneously in the current time slot are determined. Add \(a\) to the waiting queue, and then select the data that do not conflict with all data transmission in the waiting queue according to the degree of urgency. The data that do not conflict with other data transmission in the waiting queue are selected repeatedly until the maximum amount of data transmission is reached. The state process of system state information is guided by action strategy. SAE network model establishes the mapping relationship between states and actions. Finally, the DQN network model is used for training to obtain the final node scheduling strategy. The specific algorithm description is shown in Algorithm 1.