## 3 Resource Allocation Considering NS and D2D
### System Model and Problem Formulation  
The water-filling method aims to allocate the BS signal power to the UEs in such a way as to maximize the sum of the network data rate and meet the constraint of allocating power with positive values to the UEs, satisfying the BS power availability [18]. The water-filling method was chosen because it solves the optimization problem described by Equations (7)-(9) with linear computational complexity by solving a system with \((Z+1)\) equations and \((Z+1)\) unknowns. \[\max_{p} \sum_{j=1}^{Z}w_{j}\times log\bigg{(}1+p_{j}\times\frac{G_{Bj}}{ \sigma^{2}+\sum_{d=1}^{D}p_{d}\times G_{jd}}\bigg{)},\] (7) Subject to \[\sum_{i=j}^{Z}p_{j}\leq P_{bs}, \tag{8}\] \[p_{j}\geq 0,\quad i=1,2,...,Z.\quad P_{bs}\geq 0, \tag{9}\]  
where \(Z\) is the total number of UEs, \(w_{j}\) is the bandwidth allocated to the UE \(j\), \(P_{bs}\) is the BS power and \(p_{j}=p_{1}\), \(p_{2}\),..., \(p_{Z}\) is the power allocated to the UEs. The expression \(\frac{G_{Bj}}{\sigma^{2}+\sum_{d=1}^{D}p_{d}\times G_{jd}}\) represents the channel state information (csi). By applying [PERSON]'s dual method \((\mathcal{L})\), the objective function (7) can be reduced to  
\[\mathcal{L}(p,\lambda)=\sum_{j=1}^{Z}w_{j}\times log\bigg{(}1+p_{j}\times\frac {G_{Bj}}{\sigma^{2}+\sum_{d=1}^{D}p_{d}\times G_{jd}}\bigg{)}-\lambda(\sum_{j= 1}^{Z}p_{j}-P_{bs}), \tag{10}\]  
where \(\lambda\) is the Lagrange multiplier. A solution to the dual problem in Equation (10) is obtained by setting the gradient to zero, i.e., \(\
abla\mathcal{L}(p,\lambda)=0\), as follows:  
\[\frac{\partial\mathcal{L}(p,\lambda)}{\partial p} = \frac{\frac{G_{Bj}}{\sigma^{2}+\sum_{d=1}^{D}p_{d}\times G_{jd}}} {1+p_{j}\cdot\frac{G_{Bj}}{\sigma^{2}+\sum_{d=1}^{D}p_{d}\times G_{jd}}}- \lambda=0, \tag{11}\] \[p_{j} = \frac{1}{\lambda}-\frac{\mathbf{1}}{\frac{G_{Bj}}{\sigma^{2}+ \sum_{d=1}^{D}p_{d}\times G_{jd}}}. \tag{12}\]  
Therefore, the power allocation for all UEs can be rewritten as follows:  
\[p_{j}(t)=\max\bigg{\{}0,\frac{1}{\lambda_{t}}-\frac{\sigma^{2}+\sum_{d=1}^{D}p _{d}\times G_{jd}}{G_{Bj}}\bigg{\}},\forall j=1,2,...,Z. \tag{13}\]  
The choice for the \(\lambda\) value determines the \"water level\" for the constraint of the sum of power allocated to the UEs, i.e., the total power consumed by the BS as shown in the Figure 2. The algorithm considers it unnecessary to allocate power to some UEs given the considerably high noise level, by limiting communication only to UEs with better channel quality so that the data rate is as high as possible. This mechanism is especially crucial for providing some QoS requirements in wireless networks. Traditional approaches to solving the water-filling method consist of choosing the best lambda value for allocating power to UEs to maximize throughput. In a modern wireless network scenario with network slicing and D2D communication, the power allocation problem is not only concerned about maximizing total throughput, but it also becomes a part of the resource allocation, whose main objective is to meet the SLA requirements of each slice. The choice of \(\lambda\) value, the main parameter for power allocation, is made by an intelligent agent according to a reward function. Using the value of \(\lambda\), we apply the water-filling algorithm to allocate power to the UEs. In this paper, we propose modeling the problem of power and bandwidth allocation through state transition, where the RL agent decides which action to take by exploring the environment and maximizing the long-term reward [19]. The RL environment is represented by three components, state, action, and reward, which are described below:  
State: The system state \(s(t)\) is represented by the number of packets received \(NPR\) in each slice \(i\) and by the _csi_ each UE \(j\).