# An Improved Q-Learning-Based Sensor-Scheduling Algorithm for Multi-Target Tracking
## 4 Detailed Description of the Proposed Algorithm
### Dynamic Clustering  
Cluster, as the most commonly used network structure, is widely used to reduce energy consumption and prolong network lifetime. In the clustering structure, sensors are classified into the cluster head (CH) and cluster members (CMs). CH has stronger abilities than CMs; it can collect the data from its CMs, perform data fusion to remove transmission redundancy, and forward the data to the sink node. Additionally, CH can provide necessary target information for its member nodes. Therefore, in the first step of our scheduling algorithm, sensors are organized into clusters to facilitate collaborative data processing in the network. The traditional clustering method is that the sink node selects several sensors with strong capability as the CHs, and determines their subordinate CMs. However, this approach is not suitable in target-tracking scenarios, since most of the sensors in the network do not need to participate in the tracking tasks. Therefore, we employ a delayed broadcasting mechanism in which CH voluntarily recruits sensors to form a dynamic cluster. In this way, we can reduce the number of nodes forming clusters in the network, minimizing unnecessary energy consumption. First, the sensors that can sense the target will set a back-off timer and broadcast its competing message when the timer expires. If by the time the back-off timer expires, the sensor receives a competing message from other sensors, it cancels the timer. Specifically, the back-off time, \(T_{back-off}\), for which sensor \(i\) uses is defined as [18; 20]:  
\[T_{back-off}(i)=T_{\text{min}}+(T_{\text{max}}-T_{\text{min}})*(\mu\frac{ \overline{d_{i}}}{e_{re}(i)})+T_{rand}, \tag{4}\]  
where \(T_{\text{min}}\) and \(T_{\text{max}}\) denote the minimum and maximum back-off timer values; \(\overline{d_{i}}\) is the average distance from node to targets; \(e_{re}(i)\) is the residual energy of sensor; and \(\mu\) is a parameter which can nondimensionalize the second term in Equation (4). \(\mu\) is given by  
\[\mu=e_{0}/r_{s}, \tag{5}\]  
\begin{table}
\begin{tabular}{c c c c} \hline Network Scale & 100 m \(\times\) 100 m & \(E_{elec}\) & 50 nJ/bit \\ \hline Sink coordinates & (50 m, 50 m) & \(\varepsilon_{fs}\) & 10 pJ/bit/m\({}^{2}\) \\ \hline Sensing radius & 20 m & \(\varepsilon_{amp}\) & 0.0013 pJ/bit/m\({}^{4}\) \\ \hline Communication & 40 m & Learning rate \(\beta\) & 0.8 \\ \hline Size of control packet & 8 bytes & Discount factor \(\gamma\) & up and right at an angle of \(\pi/6\) with the horizontal axis \\ \hline Size of data packet & 256 bytes & Weight coefficient \(\lambda\) & 0.4 \\ \hline Time step & 1 s & The length of the data packets & 600 \(\times\) 8 bits \\ \hline Number of sensors & 200 & Initial energy & 0.05 J \\ \hline Number of targets & (2, 4, 6, 8, 10) & Random motion parameter \(\eta\) & random number \(\in(0,1)\) \\ \hline Initial speed of target & 10 m/s & Localization parameter \(\xi\) & 1.5 \\ \hline \end{tabular}
\end{table}
Table 1: Simulation parameter settings.  
where \(e_{0}\) is the initial energy of the sensor and \(r_{s}\) is the node sensing range, which denotes the maximum distance from the node to targets. Moreover, the third term in Equation (4) is a random part.  
According to Equation (4), the delay determined by the back-off timer value can ensure that the sensor with a shorter average distance to targets and more residual energy will broadcast the competing message earlier, and thus have a higher chance of serving as the CH.