## 4 Simulations  
In this section, after analytical modeling, we evaluate the performance of our proposed algorithm. The simulation parameters are provided in Table 2. For the performance evaluation, we utilize a real human mobility trajectory, which is collected in Korea Advanced Institute of Science & Technology (KAIST), Korea [42,43]. For evaluation purposes, we set \(\epsilon=0.5\) in order to ensure that all actions are explored sufficiently before selecting a particular action for reward maximization. In addition, we set \(\psi\) and \(\alpha\) to 0.6 and 0.7, respectively. The experiments were performed using MATLAB R2017a on an X64-based processor system that is equipped with an Intel(R) Core(TM) i7-7500 CPU running at 2.90 GHz and 16 GB RAM. Simulations were executed for 500 time periods and 800 iterations were performed for each simulation setup. For performance validation, we compare the performance of our proposed caching approach with several benchmark approaches, including most popular caching (MPC), random caching (RC), least frequently used (LFU) caching, and no-D2D. The outlined benchmark approaches are discussed below:  
* Most popular caching (MPC): MPC only caches popular content instead of caching all content. This provides cache resource conservation [44]. * Random caching (RC): In this scheme, random data caching decisions are made, resulting in a reduction in complexity without compromising data service quality [45]. * Least frequently used (LFU): This scheme focuses on data demand frequency, where data demands follows data popularity. This scheme incorporates data popularity and data demand frequency to optimize data caching [46]. * no-D2D: As its name implies, no D2D communication is used in this scheme. Moreover, data caching is performed only at the SBS level by exploiting Q-learning. Figure 2 illustrates the impact of varying amounts of content on network delay. The obtained results demonstrate that the delay increases with increasing amounts of content. The increase in the delay shows the unavailability of the requested content within the cache resources. The absence of requested content in the cache incurs an additional delay due to fetching requested content from the core network, substantially increasing data access delay. The obtained results demonstrate that when the amount of content is equal to 600, our proposed caching scheme attains 29%, 30.7%, 16.6%, and 24.8% lower delay than MPC, RC, LFU, and no-D2D schemes, respectively. These results highlight that our proposed caching approach can flexibly perform data caching and improve cache access. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Parameter** & **Notation** & **Value** \\ \hline Small cell size & \(A_{sc}\) & 250 m \(\times\) 250 m \\ \hline Millimeter-wave bandwidth & \(W_{mm}\) & 2.16 GHz \\ \hline Microwave bandwidth & \(W_{\mu}\) & 20 MHz \\ \hline SBS transmission power & \(P_{i}\) & 30 dBm \\ \hline UE transmission power & \(P_{\
u}\) & 23 dBm \\ \hline mmWave noise spectral density & \(\sigma_{mm}^{2}\) & \(-\)134 dBm/MHz \\ \hline Microwave noise spectral density & \(\sigma_{\mu}^{2}\) & \(-\)174 dBm/Hz \\ \hline Discount factor & \(\alpha\) & 0.7 \\ \hline Learning rate & \(\psi\) & 0.8 \\ \hline Half-power beamwidth & \(\omega_{m}\) & 30\({}^{\circ}\) \\ \hline Popularity skewness & \(\gamma\) & 1 \\ \hline Content size & \(f_{m}\) & 100 Mbit \\ \hline \hline \end{tabular}
\end{table} 
Table 2: System parameters. Figure 3 illustrates the influence of the UE's density on the delay. There is a direct relation between the number of sets of UE and data requests. When the amount of UE increases within a small cell, it constitutes more data traffic. However, our proposed approach exploits the UE's density to reduce data access by leveraging D2D communications. This enables the UE to fulfill the data demands through D2D communication.