## 5 Evaluation Results
### Results and Discussion  
After the agents performed the resource allocation for the slices, the power and bandwidth allocations were shared equally for the UEs in each slice. The performance of the RL algorithms was analyzed for over 3000 episodes in training. For a larger amount of episodes, we observed no significant changes in the results obtained with the considered agents. Figure 5a-c present the learning process of the agents in radio resource management. We plotted the reward values as a function of the number of episodes in the training phase. The sub-figures depict that the discounted long-term reward tends to gradually increase as the number of training episodes increases. The mean and immediate rewards presented random values due to the stochastic behavior of the environment. Figure 5a illustrates that the discounted long-term reward of the DQN agent becomes constant after 500 episodes due to the Q-value overestimation. In Figure 5b, the DDQN agent improves the result related to the other approaches by employing two evaluation functions in its learning method. In Figure 5c, DDPG-KRP shows faster convergence in training and higher reward values compared to the other considered approaches, which indicate better learning, as we expected, due to good learning ability in high-dimensional action space. \begin{table}
\begin{tabular}{c l} \hline \hline
**Parameters** & **Values** \\ \hline BS power (\(P_{bs}\)) & 10 W \\ Transmission power D2D & 0.1 W \\ Noise power & \(-\)114 dBm \\ Small cell radius & 300 m \\ Maximum distance between a D2D pair & 30 m \\ TTI & 1 ms \\ Number of OFDM symbols per TTI & 7 \\ Simulation time & 1000 TTI \\ RB bandwidth & 180 KHz \\ Bandwidth & 10 MHz \\ RBs per TTI & 100 \\ Mobility model & UEs and D2D pairs are uniformly distribute \\  & in the small cell to each TTI \\ Replay buffer (\(R_{b}\)) & 10,000 \\ Weight on channel capacity (\(\alpha\)) & 0.1 \\ Weight on PDR (\(\beta\)) & 10,000 \\ Q-value update & 1 s (2000 scheduling slots) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Simulation parameters. Figures 6-10 show a detailed comparison of the performance and results for the DRL agents after training. According to Table 2, a high weight \(\beta\) is assigned in the reward function to increase the number of digits and increase the priority of the PDR in each slice. Figure 6 shows that the resource allocation using the DQN algorithm does not allocate all available resources generating a low PDR. The DDQN algorithm can allocate almost all available resources but does not meet the PDR of all services. Compared to the DQN and DDQN algorithms, the proposed DDPG-KRP algorithm allows the agent to act by penalizing actions that do not meet the SLA requirements of each slice or by exceeding the total amount of available resources. DDPG-KRP generates a deterministic action in continuous space and then this action is discretized by the KNN algorithm. Figure 6 corroborates our belief that the DDPG-KRP presents better learning of the complicated relationship between demand and PDR than the other agents since all available resources are used and better performance in terms of the PDR is achieved compared to the other considered approaches. Figure 7 compares the throughput by each DRL agent with the increase in UEs for each service. The traffic demands of the URLLC service are high because of the packet size. Given the higher transmission volume and the strictly lower latency requirement, the DRL agent learns to allocate a larger amount of resources to these slices, which shows a higher throughput. We observe that the proposed DDPG-KRP achieves a significant gain over the other methods. The highest throughput values for each service are because of its learning ability to allocate all available resources to the slices. We have applied a \(t\)-test to verify if the difference between the throughput values of Figure 7 for each approach is zero. We have found that the test does reject the null hypothesis at the 5% significance level.