# Reinforcement Learning for Energy Optimization with 5G Communications in Vehicular Social Networks
## 3 System Model and Problem Definition  
In this work, we consider the single-cell scenario in H-CRANs where cellular-mode vehicles and D2D-mode vehicles coexist. The vehicles are expressed as devices in this study. H-CRANs comprise a BBU pool and multiple RRHs, as shown in Figure 1. The total device set can be expressed as \(U\), and it consists of a cellular device set \(C\) and D2D device set \(D\). The cellular devices, \(C=\{1,\ldots,c\}\), and the number of D2D devices are distributed randomly within the set \(D=\{1,\ldots,d\}\). We denote the set of RRHs as \(S=\{1,\ldots,s\}\) and RBs as \(K=\{1,\ldots,k\}\). We assume that one RB can be allocated to one cellular device and shared with multiple D2D devices. Each device can select the transmission mode between the cellular mode, \(m_{c}\), and D2D mode, \(m_{d}\); to select the D2D mode, the distance between the D2D pairs must satisfy the D2D distance threshold. The SINR of cellular device \(c\) at the \(k\)-th RB, as reported in [20], is  
\[\gamma_{k}^{c}=\frac{p_{tr}^{0}.\xi_{k}^{sc}}{N+\sum_{d\in D}p_{k}^{d}.\xi_{k} ^{ad}+\sum_{s^{\prime}GRH,\;s^{\prime}\
eq s}p_{tr}^{0}.\xi_{k}^{s^{\prime}c}}\,, \tag{1}\]  
where \(p_{tr}^{0}\) is the transmission power of the RRH; \(\xi_{k}^{sc}\) is the channel gain between the associated RRH \(s\) and cellular device \(c\), and \(N\) is the noise power spectral density. The channel gain between cellular device \(c\) and the associated RRH \(s\) can be calculated as \(g_{k}^{sc}=G\theta^{j\epsilon}\xi^{s,c}L^{s,c-}\), where \(G\) is the constant gain from the antenna and amplifier, \(\theta^{s,c}\) is the multipath fading gain with a log-normal distribution, \(L^{s,c}\) is the distance between the cellular device \(c\) and the associated RRH \(s\), and is the path-loss exponent. The SINR of D2D device \(d\) at the \(k\)-th RB, as reported in [16], is  
\[\gamma_{k}^{d}=\frac{p_{k}^{d}.\xi_{k}^{dd^{\prime}}}{N+p_{k}^{c}.\xi_{k}^{ad} +\sum_{j\in D,\;j\
eq d}p_{k}^{j}.\xi_{k}^{j^{d}}}\,, \tag{2}\]  
Figure 1: System model. where \(p_{k}^{d}\) is the transmission power of D2D device \(d\), \(g_{k}^{dd^{\prime}}\) is the channel gain between another D2D pair \(d^{\prime}\), \(p_{k}^{c}\) is the transmission power of cellular device \(c\) which shares the \(k\)-th RB, and \(g_{k}^{cd}\) is the channel gain between cellular device \(c\) and D2D device \(d\). The total system capacity, according to the Shannon capacity, can be expressed as  
\[=W^{total}\sum\
olimits_{c\in C}\sum\
olimits_{k\in K}\{\log_{2}\!\left(1+ \gamma_{k}^{c}\right)+\sum\
olimits_{d\in D}\log_{2}\!\left(1+\gamma_{k}^{d} \right)\}, \tag{3}\]  
where \(W^{total}\) is the total bandwidth of the system. The power consumption of the system includes the power consumed by the RRHs and fronthaul devices. The power consumption model of the RRH can be expressed as  
\[P_{rm}=\sum\
olimits_{s=1}^{S}\left(\phi_{rm}+\Delta slope\sum\
olimits_{c=1} ^{C}a_{c}^{s}p_{tr}^{0}\right), \tag{4}\]  
where \(\phi_{rm}\) is the circuit power of the RRH, \(\Delta slope\) is the slope of the load-dependent power consumption of RRH, as reported in [22], and \(a_{c}^{s}\) is the association indicator of cellular device \(c\), with values of 1 for an association and 0 for a nonassociation. The power consumption model for the fronthaul links was reported in [23] and can be expressed as  
\[P_{fronthaul}=\sum\
olimits_{s=1}^{S}\left(\phi_{fronthaul}+\ell\cdot t_{s}\right), \tag{5}\]  
where \(\phi_{fronthaul}\) is the circuit power from the fronthaul transceiver and switch, \(\ell\) is the power consumption per bit/s, and \(t_{s}\) is the traffic associated with RRH \(s\). The macro base station provides only voice services; thus, its power consumption is not considered.