## 2 System Model
### Q-Learning Model  
The data transmission scheduling in WSNs mainly solves the problem of deciding which nodes are scheduled for transmission in each time slot. The transmission status and the location of data need to be considered. The Q-learning model of WSNs data transmission scheduling problem can be represented by \((S,A,\pi,R)\). \(S\) is the state space, representing the state set of all data in WSNs. \(A\) is the action space, representing the action set of WSNs. \(\pi\) is the learning strategy of the agent, and represents the slot allocation of WSNs data transmission. \(R\) is the agent's reward, indicating the feedback of the agent's action in the current time slot.  
(1) System space model  
The state space of the whole WSNs consists of the current state of data generated by all nodes in the network, which can be expressed as \(S=\{s_{1},s_{2},\ldots\ldots,s_{M-1},s_{M}\}\). The current state of data generated by any node is \(s_{i}=\{C_{i},t_{i},h_{i}\}\). Thus, the size of the current state space of data generated by any node is \(T_{i}{(H_{i})}^{2}\). The size of the state space of the entire system can be expressed as \(\prod\limits_{i=1}^{M}{T_{i}{(H_{i})}^{2}}\).  
In WSNs, all possible situations of nodes conducting data transmission scheduling in each time slot constitute the action space of the system, which can be expressed as \(A=\{a_{1},a_{2},\ldots,a_{M-1},a_{M}\}\). For the data generated by any node in the current time slot, if the data are transmitted, the corresponding action \(a_{i}\) is 1; otherwise, it is 0. Regardless of WSNs sensor node transmission constraints and assumptions, the maximum movement space of the whole system is \(2^{M}\) in theory. Thus, the size of the system can be expressed as the product of the size of the state space and the size of the action space. In such an ample system space, reinforcement learning cannot get an effective scheduling strategy. The introduction of deep reinforcement learning can be a good solution to the time slot allocation and network control problems of large-scale systems.  
(2) Value function model  
The goal of reinforcement learning is to achieve mapping strategy from environment state to action \(\pi:S\to A\). The Q-learning algorithm can be regarded as a random expression of value iteration algorithm. Value iteration can be expressed by action value function  
In the Q-learning model, the Q value update of the system is defined as follows:  
\[Q_{t}(s_{t},a)=(1-\partial)Q_{t}(s_{t},a)+\partial[R(s^{\prime}|s,a)+\gamma\sum \limits_{s^{\prime}\in S}P(s^{\prime}|s,a)\underset{a\in A}{\max}Q_{t}(s^{ \prime}|s,a)] \tag{4}\]  
where \(\partial\in(0,1)\) is the learning rate factor. The larger \(\partial\) is, the more the system learning process depends on the reward function and the value function. The smaller \(\partial\) is, the more the system relies on the accumulated learning experience, and the slower the learning rate. The Q-learning algorithm maximizes the system utility by calculating and updating the Q value, but \(P(s^{\prime}|s,a)\) is usually unknown, and in the Q-learning algorithm, \(R(s,a)+\sum\limits_{s^{\prime}\in S}P(s^{\prime}|s,a)\underset{a\in A}{\max}Q _{t}(s^{\prime}|s,a)\) can be directly replaced by an unbiased estimate constructed from the current transformation \(R_{t+1}+\underset{f\in A}{\max}Q(s_{t+1},a_{t})\), so as to obtain the final Q The value function updates the formula as:  
\[Q_{t}(s_{t},a)\leftarrow(1-\partial)Q_{t}(s_{t},a)+\partial[R(s^{\prime}|s,a) +\gamma\underset{a\in A}{\max}Q_{t}(s^{\prime}|s,a)] \tag{5}\]