## 4 Channel Allocation Based on Multi-Agent Deep Reinforcement Learning
### Proposed Multi-Agent Deep Reinforcement Learning Neural Network  
The structure of the proposed multi-agent deep reinforcement learning neural network (MDRLNN) is shown in Figure 3.  
Figure 2: Schematic diagram of the multi-agent deep reinforcement learning for channel allocation aided by the power allocation.  
As shown in Figure 3, the proposed MDRLNN consists of \(M\times N\) single-agent networks, each of which corresponds to a subchannel. This is because there are \(N\) available subchannels in each cell, and each cell contains \(N\) agents corresponding to the available subchannels.  
When the global channel gain information is obtained from the environment, it is normalized first. The normalization \(h\) is shown in Equation (8).  
\[h^{t}=\frac{-\log_{10}(\overline{H}^{t})-\mathrm{E}\!\left[-\log_{10}( \overline{H}^{t})\right]}{\sqrt{\mathrm{E}\!\left\{\left(-\log_{10}(\overline{ H}^{t})-\mathrm{E}\!\left[-\log_{10}\!\left(\overline{H}^{t}\right)\right] \right)^{2}\right\}}} \tag{9}\]  
In (8), both \(h^{t}\) and \(\overline{H}^{t}\) are vectors of \(M\times N\times K\) dimensions.  
After the normalization, the system will distribute the information to the single-agent networks. It means that the single-agent networks have the same input dimension, determined by the size of the global channel gain. However, the output dimension of the single-agent networks is determined by the number of users in the cell. If there are \(K_{m}\) users in the cell \(m\), the single-agent network output dimension in the cell \(m\) is \(\left|A_{m}\right|\), \(\left|A_{m}\right|\)\(=C_{K_{m}}^{2}\), representing the number of combinations of two different user in the NOMA system. All of this implies that the \(N\) agents in the same cell have the same single-agent neural network structure. The single-agent neural network structure of agents of Agent\({}_{m,1}\) to Agent\({}_{m,N}\) in the cell \(m\) is shown in Figure 4.  
Figure 4: Single-agent neural network structure of agents in the cell \(m\).  
Figure 3: Proposed MDRLNN for channel allocation.  
Furthermore, as can be seen in Figure 3, each agent adopts an \(\varepsilon\)-greedy strategy to select actions. In this process, the base station retrieves user clusters and prioritizes allocating channels to those containing unassigned users, ensuring that all users obtain a subchannel.  
Specifically, assuming there are \(K_{m}\) users in cell \(m\), the base station generates the user cluster index set \(\left[A_{m,u_{1},\cdots}\,A_{m,u_{k_{m}}},\cdots\,A_{m,u_{k_{m}}}\right]\) based on the mapping function \(A_{m}\), defined in Section 4.2. When \(k_{m}=1\), \(A_{m,u_{1}}=[0,1,\cdots,K_{m}-2]\); when \(k_{m}=K_{m}\), \(A_{m,u_{K_{m}}}=[K_{m}-2,(K_{m}-2)+(K_{m}-2),\cdots,(K_{m}-2)+\cdots+1]\); and when \(k_{m}\
otin\{1,K_{m}\}\), \(A_{m,u_{k_{m}}}\) is defined as follows.  
\[A_{m,u_{k_{m}}}=\left[\begin{array}{c}(k_{m}-1)-1,\\ K_{m}-1+(k_{m}-2)-1,\\ 2K_{m}-(1+2)+(k_{m}-3)-1,\\ \cdots,\\ (k_{m}-1)K_{m}-(1+2+\cdots+(k_{m}-1))\\ \cdots,\\ (k_{m}-1)K_{m}-(1+2+\cdots+(k_{m}-1))+(K_{m}-k_{m})-1\end{array}\right],\;\;k_{ m}\
otin\{1,K_{m}\} \tag{10}\]  
During the agent's action-selection process, the base station collects the cluster indices of users not yet assigned to any subchannel, shown as follows:  
\[A_{m}^{n}=\left[A_{m,u_{k_{m}}},\cdots,A_{m,u_{k_{m}}}\right] \tag{11}\]  
Based on this list, the optimal action \(a_{m}^{n}\) will be selected. After all users are allocated to a subchannel, subsequent agents will select user clusters within the original action space \(A_{m}\).  
Finally, the actions selected by all agents are combined into a joint action \(A^{t}\), as written below:  
\[A^{t}=\left[a_{1}^{1},\cdots,a_{1}^{N},\cdots,a_{m}^{n},\cdots,a_{M}^{1}, \cdots,a_{M}^{N}\right] \tag{12}\]  
where \(a_{m}^{n}\) is the action of Agent\({}_{m,n}\).