# Reinforcement Learning for Energy Optimization with 5G Communications in Vehicular Social Networks
## 5 Results and Discussion  
A single-cell H-CRAN environment was considered in this work, and the parameters used in the simulation are summarized in Table 1. We set the parameters and speed requirements for mobility dataset according to the 3 GPP (3 rd Generation Partnership Project) specifications release 16 [22; 27]. Four RRHs and vehicles were distributed randomly in a single macro cell with an intercell distance of 500 m. The mobility dataset used in the simulation was a dataset in an urban area created using a simulation of urban mobility (SUMO) simulator [28]. In the dataset, all vehicles had mobility with a random trip model at the Seoul City Hall in South Korea according to the 3 GPP specification release 15 [29]. The datasets consisted of two types of scenarios: light traffic and heavy traffic. The two types were vehicular mobility datasets for urban areas with light traffic and heavy traffic scenarios. In a light traffic scenario, vehicles can move fast and the distance between vehicles increases. This means that fewer vehicles can select the D2D mode because of the D2D distance threshold. In a heavy traffic scenario, vehicles move slowly because of the traffic jam, and the distance between vehicles becomesshorter. This allows the vehicles to select a D2D mode more often. Compared to heavy traffic scenarios, fewer vehicles move and require resources at the same system load situation in light traffic scenarios. The D2D distance threshold affects the system energy efficiency, which can also be affected by the mode selection. To consider and present the effects of the D2D distance threshold, we simulated experiments with various D2D distances, as reported in [27]. We considered Rayleigh fading, log-normal shadowing, and the path-loss model \(140.7+36.7\log(distance(km))\), based on [30]. The D2D distance threshold was varied between 250 and 350 m, according to [31; 32]. The Q-learning parameters were \(\alpha=0.01\), \(\beta=0.9\), and \(\varepsilon=0.01\). We set the time for the D2D link establishment as 6 ms, as reported in [33], and the size of time unit is 3 s.  
We performed simulations while changing the load of the system and the D2D distance with different traffic scenarios and compared the proposed algorithm with other existing algorithms. First, we denote our proposed algorithm as \"proposed algorithm\". Second, the condition where no algorithm was applied was used to create a baseline, and we denoted the baseline as \"BA\". It selects the communication mode by comparing the strength of the expected SINR between cellular and D2D modes. Third, for a comparison with a mode-selection and power-control algorithm with Q-learning, the algorithm in [21] was used. It learned to maximize the device's energy efficiency with a fixed target SINR. The objective that maximizes device energy efficiency can also maximize system energy efficiency by selecting more D2D modes. We denote that as \"Compare1\". Fourth, we used the algorithm in [20] to compare power control with Q-learning. It also learned to maximize device energy efficiency using a fixed target SINR. We denote that as \"Compare2\". The energy efficiencies for different D2D distance scenarios are compared as functions of the system loads with different mobility scenarios, as shown in Figure 3. Figure 3a shows that the system energy efficiency of the proposed algorithm is 83% and 89% higher than those of BA and Compare2, respectively. Compared to Compare1, it presents high energy efficiency, with a difference of approximately 26%. In Figure 3b it can be seen that the proposed algorithm outperformed BA, Compare1, and Compare2 by more than 121%, 26%, and 126%, respectively. Figure 3c shows that the system energy efficiency of the proposed algorithm is 102% and 108% higher than that of BA and Compare2, respectively. Compared to Compare1, it presents higher energy efficiency, with a difference of approximately 21%. Figure 3d shows that the proposed algorithm performed approximately 128%, 27%, and 133% better than BA, Compare1, and Compare2, respectively.