## 4 A Multi-Reporting Nodes Selection Algorithm Based on Lightweight Q-Learning for Various Services
Considering that the number of nodes within the segment after dynamic adjustment is:  
\[N_{\text{Vi}}^{\prime}=\frac{N_{\text{P2}}}{Y}+\Delta=\frac{N_{\text{P2}}}{Y}+ \frac{N_{\text{Vi}}}{Y+x}=\frac{N_{\text{P2}}*(Y+x+1)}{Y(Y+x)}, \tag{14}\]  
when Y is fixed, \(\frac{\text{d}(N_{\text{Vi}}^{\prime})}{\text{d}x}=-\frac{1}{Y(Y+x)^{2}}\), which means \(N_{\text{Vi}}^{\prime}\) is monotonically decreasing with respect to x. Therefore, to ensure the search scope and maximize both the floating range and the number of segment nodes, the floating range can be set as follows:  
\[\Delta\leq\frac{N_{\text{Vy}}}{Y+1}. \tag{15}\]  
At the same time, by dynamically adjusting the number of reporting nodes Y, repeating the virtual boundary partitioning and dynamic floating process mentioned above, the optimization of the Q-learning search process can be achieved. Taking the division of the virtual boundary as state \(W\). The increase or decrease in the number of segments and the changes to the virtual boundaries are regarded as the action space \(A\). The reward function based on the awareness evaluation function can be constructed as:  
\[\mathrm{R}(w,a)=-[\mathrm{Ev}\,\mathrm{a}\left(w^{\prime}\right)+\mu\cdot w^{ \prime}]\,, \tag{16}\]  
where \(\mathrm{Ev}(w^{\prime})\) is the awareness evaluation function under the new state \(w^{\prime}\), where \(\mu\) is the regularization penalty coefficient, and \(w^{\prime}\) represents the number of segments and the division of virtual boundaries in the new state. The Q-values are updated using R, and the update formula is as follows:  
\[\mathrm{Q}(w,a)\leftarrow\mathrm{Q}(w,a)+\alpha\left[r+\gamma\max_{a^{ \prime}}\mathrm{Q}(w^{\prime},a^{\prime})-\mathrm{Q}(w,a)\right], \tag{17}\]  
where \(\alpha\) is the learning rate; \(\gamma\) is the update rate, and \(0\leq\alpha\leq 1\), \(0\leq\gamma\leq 1\). The pseudocode of the algorithm is presented in Algorithm 1:  
```
Input: Number of services, \(N_{z}\); transmission path of each service, \(P_{z}\); maximum number of segments for each service, \(Y_{z}\);
1 for\(i\gets 1\)to\(N_{z}\)do
2 for\(j\gets 1\)to\(Y_{z}\)do (adjust the number of segments)
3 Divide \(P_{i}\) equally into segments with virtual boundary \(\mathrm{V}=\{\mathrm{V}_{1},\mathrm{V}_{2},\ldots,\mathrm{V}_{Y_{j}}\}\);
4 Calculate floating range \(\Delta\) with Equation (15), and the number of potential combinations of virtual boundary is \(N_{\Delta}=(2\Delta+1)^{Y_{j}}\);
5 for\(k\gets 1\)to\(N_{\Delta}\)do
6 Adjust the virtual boundary \(\mathrm{V}=\{\mathrm{V}_{1},\mathrm{V}_{2},\ldots,\mathrm{V}_{Y_{j}}\}\) with \(\Delta\) to generate new V\({}^{\prime}=\{\mathrm{V}_{1}+\Delta,\mathrm{V}_{2}-\Delta,\ldots, \mathrm{V}_{Y_{j}}\}\), which is the state \(W\);
7 for\(h\gets 1\)to\(Y_{j}\)do (using APINT model for each segment)
8 Find potential reporting nodes within segment \(\mathrm{V}_{h}\) using Equation (6);
9 Identify the optimal reporting nodes in subsegment \(\mathrm{V}_{h}\) based on Equations (8) and (9), which is the action \(A\);
10 Calculate telemetry and reporting time, as well as awareness overhead within subsegment;
11 end
12 Calculate the awareness evaluation function as the reward \(R\) based on Equation (10);
13 Update the Q value table by using Equation (17);
14 end
15 Find the best virtual boundary division range and the best reporting node location;
16 end
17 Compare the maximum Q-values associated with the optimal virtual boundary divisions in scenarios with different number of segments;
18 Output the optimal reporting nodes' number, location, awareness time, awareness overhead of current service;
19 end
```  
**Algorithm 1**Dynamic selection of multiple reporting node based on the improved Q-learning  
**Output:** Generate optimal reporting node selections for various services.