## 3 System Design
### Feedforward Processes for Actor and Critic Models  
#### 3.3.1 Feedforward Calculation of Actor Neural Network  
First, we give the feedforward process of the graph convolutional layer. We use \(A_{i,j}\) and \(X\) to denote the topological adjacency matrix of the sensor network and the state of network observed by the controller, respectively. The number of nodes and the number of features are denoted by \(N\) and \(F\), respectively. Apparently, we have \(A_{i,j}\in R^{N\times N}\) and \(X\in R^{N\times F}\). We define an adjacency matrix with self-loop \(\tilde{A}_{i,j}=I_{N}+A_{i,j}\), where \(I_{N}\) is the unit matrix of order \(N\). Thus, the degree matrix \(D_{i,j}\in R^{N\times N}\) is  
\[D_{i,j}=\left\{\begin{array}{ll}\sum_{j}\tilde{A}_{i,j}&\text{if }i=j\\ 0&\text{else}\end{array}\right. \tag{2}\]  
In the process of graph convolution calculation, nodes with too many neighbor nodes will have a large gap with other nodes when aggregating features. To avoid this problem, we define the standardized adjacency matrix as \(\tilde{A}_{i,j}=D_{i,j}^{-1/2}\tilde{A}_{i,j}D_{i,j}^{-1/2}\). After two graph convolutional layers, the output tensor \(H\) is obtained by  
\[H=ReLU(\tilde{A}_{i,j}(ReLU(\tilde{A}_{i,j}XW_{g}^{(0)}+b_{g}^{(0)}))W_{g}^{( 1)}+b_{g}^{(1)}) \tag{3}\]  
where \(W_{g}^{(0)}\in R^{F\times C}\), \(b_{g}^{(0)}\in R^{C\times 1}\), \(W_{g}^{(1)}\in R^{C\times Z}\), \(b_{g}^{(1)}\in R^{Z\times 1}\) are the weights that need to be trained in the Actor model. C and Z are the characteristic dimensions of the output after graph convolution.  
In addition to the adjacency matrix \(A_{i,j}\) and the state matrix \(X\), the Actor model also includes the position vector \(l\in R^{N\times 1}\) of the current node and the vector \(d\in R^{N\times 1}\) of the target node. Both vectors are composed of one-hot codes, which represents the unique ID of nodes.  
The state vector of the current environment \(S\) can be obtained by  
\[S=Concatenate(Flatten(H),l,d) \tag{4}\]  
Figure 3: Model Structure of the Actor neural network and the Critic neural network.  
In (4), the dimension of \(S\) is \(N\times Z+2N\).  
The state vector \(S\) will be used as the input of the fully connected neural network in the Actor model, and finally the corresponding action vector \(\mu\in R^{N\times 1}\) is obtained at the output of the Actor. This vector determines the next hop after the current node receives the data packet. Since the network topology is not fully connected in most cases, it is necessary to use the mask vector to filter the action vector output by the Actor fully connected layer. The vector \(mask\) represents the adjacency information of the node where the packet is located. It can be calculated by  
\[mask=A_{i,j}^{T}l \tag{5}\]  
where \(A_{i,j}^{T}\) is the transpose of the adjacency matrix. Specifically, the purpose of setting the mask vector is to limit the decision space and avoid forwarding data packets to non-neighbor nodes.  
The final output of the Actor model can be obtained by  
\[\mu=softmax(mask\bigodot(W_{a}S+b_{a})) \tag{6}\]  
where \(\bigodot\) represents the Hadamard product, and \(W_{a}b_{a}\) are the weights of the fully connected layer.