# Dynamic Cooperative Spectrum Sensing Based on Deep Multi-User Reinforcement Learning
## 3 The Proposed Deep Multi-User Reinforcement Learning
### Architecture of the Proposed DMRL  
The network structure of DMRL is illustrated in Figure 4. The input of the network is the state of each user, which contains a vector of size \(2K+2\). The first \(K+1\) elements represent the user's transmission status. The next \(K\) elements from \(K+1\) present the remaining capacity of channels. The last element represents the \(ACK\) signal. In the hidden layer, the outputs are calculated according to the Matmul function and the Relu function. Matmul function [41] represents the weighted sum of nodes at each level. The ReLU [41] is the linear activation function that will output the input directly if it is positive, otherwise, it will output zero. In the neural network, the activation function is responsible for transforming the summed weighted input of the node into the activation of the node or output. It is formulated as:  
\[h_{j}=Relu(Matmul(s,\omega_{j})+b_{j}), \tag{9}\]  
where \(\omega_{j}\) represents the weight matrix of the \(j\)-th hidden layer in the network structure; \(b_{j}\) represents the offset matrix of the \(j\)-th hidden layer. In the DMRL structure, there is a layer of value and advantage layer. Among them, \(V(s_{n}(t))\) represents the value of the static state environment itself, \(s_{n}(t)\) represents the state in the time slot \(t\), \(A(a_{n}(t))\) represents the additional value of selecting an action and \(a_{n}(t)\) represents the action in the time \(t\). Then, finally these two come together and get the \(Q\) value for each action. Therefore, we can change all the \(Q\) values in a certain state by just changing the value of \(V(s_{n}(t))\). We calculate the value \(V(s_{n}(t))\) and the advantage \(A(a_{n}(t))\) separately based on the output \(h_{j}\) of the hidden layer. The related formulas are shown as (10) and (11),  
\[V(s_{n}(t)) =Matmul(h_{j},\omega_{j,1})+b_{j,1}, \tag{10}\] \[A(a_{n}(t)) =Matmul(h_{j},\omega_{j,2})+b_{j,2}. \tag{11}\]  
To avoid the case that only the parameters \(\omega_{j,2}\) and \(b_{j,2}\) are updated in the proposed DMRL network, (11) is rewritten as:  
\[A(a_{n}(t))=Matmul(h_{j},\omega_{j,2})+b_{j,2}. \tag{12}\]  
Figure 4: The network structure of deep multi-user reinforcement learning (DMRL).  
From (12), it can be found that matrix \(A(a_{n}(t))\) has a zero-sum characteristic. That is, the sum of matrix \(A(a_{n}(t))\) is 0. Finally, adding the value \(V(s_{n}(t))\) and the advantage \(A(a_{n}(t))\) to output layer, the \(Q\)-value matrix can be obtained, which is formulated as:  
\[Q(a_{n}(t))=V(s_{n}(t))+A(a_{n}(t)). \tag{13}\]  
If the value of the first element in the \(Q\)-value matrix is the largest, it means that the current user does not send data packets. If the value of the \(K+1\) element in the \(Q\)-value matrix is the largest, it means that the current user will send data packets. To facilitate the channel selection for the current users, the \(Q\)-value matrix in (13) is normalized to a probability matrix as shown in:  
\[P=\frac{e^{Q(s)}}{\sum e^{Q(s)}}. \tag{14}\]  
Therefore, for channel access, it is necessary to select the subscript with the highest probability value to transmit packets.