## 4 Channel Allocation Based on Multi-Agent Deep Reinforcement Learning
### Training Algorithm for MDRLNN  
A random search strategy is used to train the proposed MDRLNN to improve its performance. The detailed training steps are as follows:  
First, an experience pool is established to store the sample data needed for training.  
Second, an initial JA \(A_{\text{generate}}^{t}\) based on the current state \(s^{t}\) of the environment is generated by the agents, and a batch of random JAs \(\left\{A_{\text{random},1}^{t},\cdots,A_{\text{random},B}^{t}\right\}\) are generated. Then, the JAs of \(A_{\text{generate}}^{t}\) and \(\left\{A_{\text{random},1}^{t},\cdots,A_{\text{random},B}^{t}\right\}\) are transformed into their corresponding channel allocation schemes \(G_{\text{generate}}^{t}\) and \(\left\{G_{\text{random},1}^{t},\cdots,G_{\text{random},B}^{t}\right\}\). In addition, the JAs of \(A_{\text{generate}}^{t}\) and \(\left\{A_{\text{random},1}^{t},\cdots,A_{\text{random},B}^{t}\right\}\) with the current state \(s^{t}\) of the environment are, respectively, fed into the power allocation network (i.e., ULNN in Section 5) to obtain their corresponding power allocation schemes, \(P_{\text{generate}}^{t}\) and \(\left\{P_{\text{random},1}^{t},\cdots,P_{\text{random},B}^{t}\right\}\).  
Third, \(G_{best}^{t}=\underset{G}{\text{argmax}}F_{\text{sum}}\) and \(r^{t}=\max E_{\text{sum}}^{t}\) are calculated, where  
\[E_{\text{sum}}^{t}=\left\{E_{\text{sum}}(H,G_{\text{generate}}^{t},P_{\text{ generate}}^{t}),E_{\text{sum}}(H,G_{\text{random},1}^{t},P_{\text{random},1}^{t}), \cdots,E_{\text{sum}}(H,G_{\text{random},B}^{t},P_{\text{random},B}^{t})\right\} \tag{13}\]  
Finally, the sample data \((s^{t},A_{\text{best}}^{t},G_{\text{best}}^{t},r^{t})\) is stored in the experience pool, where \(A_{\text{best}}^{t}\) is the JA corresponding to the channel allocation scheme \(G_{\text{best}}^{t}\).  
The MDRLNN for the channel allocation is described in detail in Algorithm 1, shown as follows.  
```
1:State space \(S\); The experience pool; The initialized MDRLNN
2:The well-trained MDRLNN; joint action; channel allocation scheme
3:for\(t\)do
4:for each cell \(m\)do
5:for each channel \(n\)do
6:\(a_{m}^{n}\leftarrow\epsilon\)-greedy strategy based on the output of Agent\({}_{m,n}\)
7:endfor
8:endfor
9:\(C_{\text{generate}}^{t}\), \(P_{\text{generate}}^{t}\gets A_{\text{generate}}^{t}\leftarrow[a_{1}^{1},a_{1}^{2} \ldots,a_{1}^{N},\ldots,a_{m^{\prime}}^{n}\ldots,a_{M}^{N}]\)
10:\(G_{\text{random},1}^{t}\ldots,G_{\text{random},B}^{t}\), \(\{P_{\text{random},1}^{t}\ldots,P_{\text{random},B}^{t}\}\leftarrow\{A_{ \text{random},1}^{t},\ldots,A_{\text{random},B}^{t}\}\leftarrow\) randomly generated by the system
11:\(r^{t}=\text{max}E_{\text{sum}}^{t}\)
12:\(A_{\text{best}}^{t}\leftarrow G_{\text{best}}^{t}\leftarrow\text{argmax}E_{ \text{sum}}^{t}\)
13: the sample data \((s^{t},A_{\text{best}}^{t},G_{\text{best}}^{t},\ r^{t})\) is stored in the experience pool
14:if the number of the sample data in the experience pool reaches a certain level then
15: A batch \({}^{\text{X}}\) of sample data \((s^{t},A_{\text{best}}^{t},C_{\text{best}}^{t},r^{t})\) is randomly selected
16:for each cell \(j\)do
17:for each subchannel \(i\)do
18:\(\text{Loss}(\theta)=\frac{1}{2\sqrt{s}}\sum(r^{t}-Y_{m,a_{j}^{t}})^{2}\)
19:\(\theta\gets Adam(\theta,\
abla_{\theta}\text{Loss}(\theta))\)
20:endfor
21:endfor
22: test the MDRLNN
23:endif
24:endfor
```  
**Algorithm 1** MDRLNN for channel allocation