## 5 Unsupervised Learning for Power Allocation
### Attention-Based ULNN for Power Allocation  
In this subsection, we first introduce an attention-based ULNN to achieve channel power allocation. Then, the power of each subchannel is further allocated to the corresponding users. Note that the JA obtained by the agents in the channel allocation can be viewed as the representations of the channel allocation scheme, and it contains more information than the binary channel allocation scheme. Based on this, the JA obtained by the agents in the channel allocation and the current state \(s^{t}\) of the environment are fed into the proposed attention-based ULNN to obtain the near-optimal power allocation scheme \(P\). Inspired by the transformer model [37], the encoder and decoder structures are used in the proposed ULNN, as shown in Figure 5.  
(1) Encoder: At this layer, the critical information from the JA will be captured and preserved. Specifically, in the multi-cell NOMA system, the same action in a JA may correspond to different user clusters, since the number of users in each cell is not necessarily the same. To address this challenge, despite the relatively simple architecture of the traditional feedforward neural network, its fixed-dimensional input-output structure  
Figure 5: Structure of the proposed attention-based ULNN for power allocation. fails to effectively model long-range dependencies between sequence elements and lacks sensitivity to the input sequence order. In order to distinguish the semantic differences caused by positional variations, the encoder adds positional encoding \(PE\) to each action in the JA. Then, a multi-head attention mechanism is employed to capture interference relationships among user clusters, channel correlations, and other critical factors, thereby enhancing overall system performance. One attention head might focus on the power allocation of the subchannels within a cell, while the other attention heads are used to model the inter-cell interference. The specific process is shown as follows. First, the JA output from the channel allocation is used as the input of the encoder, with the size of \(M\times N\). For each action in the JA, the encoder first linearly projects it into the initial \(d_{e}\)-dimensional output (\(d_{e}\) is 512 in this paper). Since the same actions may represent the different combinations of users in different cells, the position encoding \(PE\) is added to this initial \(d_{e}\)-dimensional output to distinguish the actions, as shown below:  
\[ \begin{split} PE_{(pos,2i)}=\sin(pos/10000^{2i/d_{e}})\\ PE_{(pos,2i+1)}=\cos(pos/10000^{2i/d_{e}})\end{split}  \tag{15}\]  
where \(pos\) is the position of the action in the JA. Then, the initial output is encoded through \(L\) identical subsequent layers, where each layer consists of two sub-layers. The first sub-layer is a multi-head attention layer, which contains 8 parallel attention heads (\(h_{d}=8\)). The input of this layer is denoted by \(X\), and is linearly projected to obtain the query (Q), the key (K), and the value (V), which can be expressed as  
\[ \begin{split} Q_{i}&=XW_{i}^{Q}\\ K_{i}&=XW_{i}^{K}\\ V_{i}&=XW_{i}^{V}\end{split}  \tag{16}\]  
where \(W_{i}^{Q}\), \(W_{i}^{K}\in R^{d_{e}\times d_{k}}\), and \(W_{i}^{V}\in R^{d_{e}\times d_{p}}\), \(i=1,2,3,\cdots h_{d}\), and \(d_{k}=d_{v}=d_{e}/h_{d}=64\). Perform the attention operation on the result of each projection as follows:  
\[head_{i}=\text{Attention}(Q_{i},K_{i},V_{i})=\text{softmax}(\frac{Q_{i}K_{i}^ {T}}{\sqrt{d_{k}}})V_{i} \tag{17}\]  
The results are concatenated and then linearly transformed into a smaller dimensional output, as expressed below:  
\[MultiHead(Q,K,V)=Concat(head_{1},\cdots,head_{h_{d}})W^{O} \tag{18}\]  
where \(W^{O}\in R^{h_{d_{v}}\times d_{e}}\) and \(h_{d_{v}}=h_{d}\times h_{v}\). The second sublayer is the Feed-Forward Neural Network layer, which consists of two linear layers and an intermediate RELU activation layer. After each sublayer, residual connections and layer normalization operations are added sequentially.