## 3 Resource Allocation Considering NS and D2D  
For the sake of readability, all the symbols and optimization variables are summarized in Table 1. This section discusses the state and action spaces and the reward function for the proposed scenario. We considered that a single BS provides the radio resource for network slices running different services (i.e., URLLC and video). D2D communication occurs in each slice of the network, as shown in Figure 1. The considered wireless communication system model refers to the downlink direction and comprises a small cell with a BS at its center. We used the inband category and the underlay subcategory to reuse the spectrum while considering the interference effect on the resource allocation to any UE connected to the D2D pair. However, we assume that UEs do not reuse spectra from other UEs. \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Notations** & **Description** & **Notations** & **Description** \\ \hline \(\Gamma\) & Signal to interference plus noise ratio & \(Q(s(t),a(t))\) & Q-value function \\  & (SINR) & \(s(t)\) & System state \\ \(\Gamma_{j}\) & SINR of UE \(j\) & \(a(t)\) & System action \\ \(G_{Bj}\) & Channel gain between the BS and the UE \(j\) & \(r(s(t),a(t))\) & Reward function \\ \(G_{jd}\) & Channel gain between the UE \(j\) and the D2D pair \(d\) & \(\alpha\), \(\beta\) & Constants used to add weight to values and maintain them with the same number of digits \\ \(G_{dd}\) & Channel gain between the D2D pair \(d\) & \(R_{b}\) & Replay buffer \\  & devices & & \\ \(G_{uv}\) & Channel gain between two devices & \(PDR\) & Package delivery rate \\ \(G_{Bd}\) & Channel gain between the BS and the D2D pair \(d\) & \(NPR\) & Number of packets received \\ \(p_{d}\) & Transmission signal power from D2D & \(NPQ(t)\) & Number of packets in the queue at time \(t\) \\  & pair \(d\) & & \\ \(p_{j}\) & Transmission signal power from UE \(j\) & \(NPT\) & Number of packets transmitted \\ \(\sigma_{z}^{2}\) & Noise power & \(R\) & Discounted cumulative reward \\ \(P_{bs}\) & BS power & \(\gamma\) & discount factor \\ \(d_{uv}\) & Distance between the transmitter \(u\) and the receiver \(v\) & \(\pi(s)^{*}\) & Policy \\ \(K_{uv}\) & Normalization constant & \(\pi(s)^{*}\) & Optimal policy \\ \(\delta\) & Loss exponent & \(\mu(s\,|\,\theta^{\mu})\) & Deterministic policy \\ \(PL\) & Path loss & \(L\) & Bellman loss function \\ \(w_{d}\) & Bandwidth of the D2D pair shared with the UE & \(J\) & Expected return from the start of the distribution \\ \(w_{j}\) & Bandwidth allocated to the UE \(j\) & \(T\) & Represents how many time steps are still left in the training session or episode \\ \(Z\) & Total number of UEs & \(\eta\) & Noise in the actor output \\ \(csi\) & Channel state information & \(\rho\) & Smoothing factor \\ \(C\) & Shannon capacity & \(\Theta^{\prime}\) & Weights of the target Q-network \\ \(\mathcal{L}\) & Lagrange’s dual method & \(\Theta\) & Weights of the main Q-network \\ \(\lambda\) & Lagrange multiplier & \(\theta^{\mu}\) & Weights of the actor main network \\ \(E[\cdot]\) & Expectation operator & \(\theta^{Q}\) & Weights of the critic main network \\ \(\
abla\) & Gradient & \(\theta^{\mu^{\prime}}\) & Weights of the actor target network \\ \(EE\) & Energy efficiency & \(\theta^{Q^{\prime}}\) & Weights of the critic target network \\ \(D\) & Total number of D2D pairs & \(ActionMax\) & Total number of available slots for allocation \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Notations. Similar to that in [1], the traffic demand of the UEs was generated by using the probabilistic distributions described in Table 2 based on their respective streaming model. Accordingly, in this work, resource allocation is performed at each transmission time interval (TTI), and orthogonal frequency division multiplexing (OFDM) is considered the technique utilized in the long-term evolution (LTE) transmission downlink, which allows a simultaneous transmission of different data packets by assigning different subcarriers to the user [16].