## 4 Channel Allocation Based on Multi-Agent Deep Reinforcement Learning
### Channel Allocation Using Multi-Agent Deep Reinforcement Learning  
In this subsection, the optimization constraint problem of channel allocation is modeled as a reinforcement learning task. Due to the large scale of the channel allocation scheme, using a single-agent structure for channel allocation may lead to system overload. Hence, we adopt a multi-agent structure, where each available subchannel in the base stations is regarded as an agent, and the downlink multi-cell NOMA system is viewed as the environment. Assume that there are \(M\) cells in the environment and the number of available subchannels in each cell is \(N\); thus, the total number of agents in the environment is \(M\times N\). A schematic diagram of the multi-agent deep reinforcement learning for channel allocation aided by the power allocation is shown in Figure 2.  
As shown in Figure 2, the agents in each cell first select an action \(a_{m}^{n}\), respectively, from the action space \(A_{m}\) based on the current state \(s^{t}\) of the environment until all agents in the environment have completed their actions. Then, the actions of all agents are combined to form a joint action (JA). After that, both the JA and the global channel gain are fed into the power allocation network (i.e., ULNN in Section 5) to obtain a power allocation scheme, and the JA is transformed into a binary channel allocation scheme. With the operation of the power allocation scheme and the binary channel allocation scheme on the downlink multi-cell NOMA environment, a reward \(r^{t}\) is obtained, and the system transitions to the next state \(s^{t+1}\).  
For convenience, the multi-agent deep reinforcement learning for the channel allocation is denoted as MDRL-CA. State, action, and reward of the MDRL-CA are defined as follows:  
1. State: In this paper, the energy efficiency is taken as the optimization goal in the NOMA system, each agent should consider the global information of the environment when selecting actions. Therefore, this NOMA system adopts a centralized architecture, and the state of the environment is characterized by the global channel gain information \(H^{t}\), \(H^{t}=[H_{1,1}^{1,t},\cdots H_{1,K^{t}}^{1,t},\cdots,H_{M,K^{t}}^{1,t},\cdots,H _{M,K}^{N,t}]\).
2. Action: Since a channel should be allocated to two different users, the action represents a combination of two different users. The size of the action space of the agent in the cell \(m\) is \(\left|A_{m}\right|=C_{Km}^{2}\). It is noted that, since the number of users in each cell may be different, the action space of the agents in different cells may also be different. If there are 3 users in the cell \(m\), the size of the action space of the agent in the cell \(m\) is \(\left|A_{m}\right|=C_{3}^{2}=3\), and the action space is shown as \(A_{m}=\left\{0:(1,2),\ 1:(1,3),\ 2:(2,3)\right\}\), where action 0 indicates that the agent has selected the 1 st and 2 nd users, action 1 indicates that the agent has selected the 1 st and 3 rd users, and action 2 indicates that the agent has selected the 2 nd and 3 rd users.
3. Reward: Since the NOMA system uses the energy efficiency as the optimization goal, this paper adopts the energy efficiency as the reward function.