## 5 Performance Evaluation  
### Simulation Environment and Parameters  
We validated our proposed system model using MATLAB-based simulation. In the proposed model, the DRL-based learning algorithm is trained by its agent, i.e., DUEs, instead of training eNBs/SCeNBs, which helps to better coordinate between network elements. During simulation, we considered a single eNB located at the center of a D2D-enabled HetNet with a 500 m cell radius and CUEs are randomly distributed. DUEs and SUEs are realized according to two independent PPPs with \(\lambda_{d}\) and \(\lambda_{s}\) densities, respectively. We have assumed that one RB has been reused by multiple D2D pairs and SUEs which are already occupied by one CUE. Therefore, the number of CUEs is selected equal to the number of RBs. A DQN-based small neural network (single input, three hidden and single output layers) is considered for quicker decision-making of each agent. The number of neurons in the hidden layers is 200, 100, and 50 respectively. DQN is trained with stochastic gradient descent with a batch size of 32. In addition, \(\theta^{\prime}\) is updated with \(\theta\) in every 20-time steps. Batch size and epochs have significant effects on optimal performance like convergence, accuracy, and stability of the DQN model. To maintain the reasonable training time, memory requirements, and computation cost, we have chosen these values as shown in Table 4. As in our system model, micro cells are deployed with \(10^{-5}\) density. we have chosen a moderate shadowing standard deviation of 8 dB with path loss coefficients 3 and 4 to make the test scenario as realistic as possible. Rest of the parameters like mobile maximum transmit power (23 dBm), RB Bandwidth (180 KHz) and Noise power (\(-118\) dBm) are chosen as per the standard industry specifications or used by other similar research so that we can compare our model outcomes with their results effectively. Table 4 represents the list of parameters used in the network as well as DRL simulations for a D2D-enabled cellular HetNet. \begin{table}
\begin{tabular}{l c} \hline \hline
**Simulation Parameter** & **Value** \\ \hline Cell Radius & 500 m \\ \hline SCeNBs density, \(\lambda_{s}\) & \(10^{-5}\) \\ \hline DUEs density, \(\lambda_{d}\) & \(10^{-3}\) \\ \hline CUEs maximum transmission power, \(P_{i}\) & 23 (dBm) \\ \hline DUEs maximum transmission power, \(P_{j}\) & 10–20 (dBm) \\ \hline SUEs maximum transmission power, \(P_{k}\) & 10–20 (dBm) \\ \hline Path loss exponent, \(\alpha\) & 3 and 4 \\ \hline SINR threshold for CUE, SUE and DUE, \(\gamma_{th}\) & 8 dB \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Parameters used in the simulation. Figure 6 shows the sum throughput of DUEs as a function of the distance between D2D pairs in the system. With the increasing of distance between D2D pairs, the achieved sum throughput of DUEs for all schemes gradually decreases. With increasing the distances between D2D pairs from 10 m to 20 m, the sum throughput of all DUEs decreases by 14.7%, 19.4%, and 20.7% in proposed DRL, D3 QN DRL, and Only Power Optimize schemes, respectively. Moreover, a 14.7% reduction indicates the proposed DRL scheme has a better sum throughout at higher distances between D2D pairs compared to others. This is happening due to two major reasons. First, the average channel gain between the Tx and Rx of D2D pairs decreases with increasing distances, and second, to ensure the minimum QoS requirements of D2D pairs, DUE Tx needs to increase its transmit power, which increases interference between other DUEs and CUEs. Figure 7 represents the influence of the cell radius of eNB/SCeNB on the sum throughput for D2D pairs. Due to the increase in cellular cell radius, the mutual interference between users, and the interference severity impact of eNBs to D2D pairs reduces.