## 1 Introduction
Through unsupervised learning, the system is able to autonomously learn and adjust the power allocation strategy based on the characteristics of the channels without explicitly labeled data. This approach not only reduces the dependence on manually labeled data, but also better adapts to the complex and changing wireless communication environment. Inspired by the above, this paper divides the resource allocation problem of the multi-cell NOMA systems into two sub-problems of channel allocation and power allocation and uses the methods of deep reinforcement learning and unsupervised learning to solve them. The contributions of this paper are listed as follows. First, this paper treats the channel allocation problem as a process of selecting users for channels. In each environmental state, the system allocates channels to corresponding users until all available channel resources are exhausted. Our goal is to find a near-optimal allocation scheme that maximizes the overall system performance. Due to the lack of near-optimal label data, conventional supervised learning methods cannot be directly applied to the channel allocation. Hence, this paper proposes a channel allocation method based on multi-agent deep reinforcement learning, motivated by the advantage of deep reinforcement learning in subchannel selection [36]. This method leverages the deep reinforcement learning and the collaborative characteristics among multiple agents to explore various channel allocation solutions and obtains a near-optimal channel allocation solution by training the neural network. Second, since power control in NOMA systems is a continuous non-convex optimization problem and considers the timeliness and accuracy of the system, it is very difficult to obtain the near-optimal solution by using supervised learning or discretization methods. Therefore, this paper proposes a power allocation method based on unsupervised learning, motivated by the fact that unsupervised learning is more efficient and effective than reinforcement learning in solving continuous power optimizations [24]. Meanwhile, to better exploit the hidden information of the channel allocation schemes to improve solutions of power allocation, the proposed unsupervised network for power allocation adopts a transformer architecture. In order to reduce the training burden of the power allocation network, it only uses the attention mechanism to deeply analyze the channel allocation scheme. Note that unsupervised learning can mine hidden information without labeling, but traditional neural networks (e.g., CNN, RNN) are difficult to model complex dependencies. The attention mechanism explicitly quantifies the interactions among data points through QKV (query-key-value) dynamic weight assignment, and transforms the implicit correlations discovered by unsupervised learning into interpretable weight matrices. Meanwhile, since the unsupervised objective (e.g., energy efficiency maximization in this paper) is usually non-convex, traditional neural networks are prone to fall into local optimum. In addition, the attention mechanism using residual connection and layer normalization can stabilize the gradient flow of unsupervised training and combine with multi-head attention to explore multiple solution spaces in parallel and reduce the sensitivity of non-convex optimization, which has been proven to be able to achieve good performance with reasonable training time. Finally, we have made comparisons of the proposed framework with the other approaches, such as the single-agent DQL (DRLA) and the multi-agent deep reinforcement learning (MDRLA) proposed [13], the DQN-DDPG dual-layer deep learning resource allocation framework [33], and the resource allocation algorithm based on the deep deterministic policy gradient and unsupervised learning (DDPG-UL) [32]. Simulation results indicate that our proposed framework achieves better system performance than these algorithms.