## 4 Proposed Algorithm
### Deep Q-Learning Based Resource Allocation Method  
The sharing of communication resource among V2V VEs, can cause high complexity in interference controls, continuous-value state, and large action space. Therefore, the resource allocation problem was formulated as a Markov decision process. MDP can be defined by a tuple \((S,A,p,r)\) where \(S\) is a set of states; \(A\) is a set of actions; \(p\) is a transition probability from state \(s\) to \(s^{\prime}(s,s^{\prime}\in\mathbb{S})\); and \(r\) is the immediate reward. At each time \(t\), the V2I mode VE makes a request to allocate communication resources to the serving RRH. RRH observes the state, \(s_{t}\), and selects an action \(a_{t}\), accordingly. After the action is executed, the state \(s_{t}\) of the environment changes to a new state \(s^{\prime}_{t}\) and the agent receives a reward \(r_{t}\). In DQN, updates of Q-table are converted into the updates of network weights. The experience replay and fixed target network are developed to accelerate training process, so  
Figure 1: Cluster formation.  
it can improve the convergence. In fixed target network, the update of the target Q-network is adopted to accelerated the training process. Figure 2 is illustrated to describe DQN.  
The details of the tuples in our proposed method are defined as follows:  
1. The state of V2I VE \(c\), the observed state \(s_{t}\) at time unit \(t\), consists of eight pieces of information: The interference received from CH \(I_{t-1}^{v,c}\); the interference received from the CMs in vehicle cluster \(i\)\(I_{t-1}^{v,j}\); the size of cluster \(I_{t}^{i}\); the load of serving RRH \(\lambda_{t}^{s}\); the rate of RB \(k\) usage \(R_{t}^{k}\); the number of VEs sharing the same RB \(k\)\(L_{t}^{k}\); the queue length of VE \(c\)\(L_{t}^{c}\); and the number of clusters in range of RRH \(s\)\(L_{t}^{s}\). Thus, the state can be described as follows: \[s_{t}=\{I_{t-1}^{v,c},I_{t-1}^{v,i},I_{t}^{i},\lambda_{t}^{s},R_{t}^{k},L_{t}^ {k},L_{t}^{c},L_{t}^{s}\}\] (13)
2. The action of each V2I VE \(c\) is defined as \[a_{t}=\{p,k\}\] (14) where \(p\) is the transmit power level, which is divided into discrete intervals, \([0,p_{max}]\) and \(k\) is RB.
3. To maximize the system capacity and guarantee the QoS requirements, the reward is defined as \[r_{t}=\left\{\begin{array}{ll}\omega\cdot\gamma_{k_{t}}+(1-\omega)\cdot(T_{0 }-T_{t-1}),&\sigma_{t-1}\leq\sigma_{0}\\ -1,&\sigma_{t}>\sigma_{0}\end{array}\right.\] (15) where \(T_{0}\) is the maximum latency constraint; \(T_{t-1}\) is the latency at time \(t-1\); \(T_{0}-T_{t-1}\) represents the latency penalty; \(\sigma_{t-1}\) is the outage probability at time \(t-1\).  
For optimal policy, the deep Q-network is trained to approximate the Q-function. The deep Q-learning agents deploy each RRH group, and the optimal policy is determined based on the following update equation.  
\[\begin{split} Q_{t+1}(s_{t},a_{t})=Q_{t}(s_{t},a_{t})\\ +\alpha\Big{[}r_{t+1}+\beta\cdot\max_{a}Q_{t}(s_{t+1},a_{t+1})-Q_ {t}(s_{t},a_{t})\Big{]}\end{split} \tag{16}\]  
where \(\alpha\) is the learning rate; \(\beta\) is the discount factor [22]. The optimal policy \(\pi^{*}\) for state \(s_{t}\) can be expressed as  
\[\pi^{*}(s_{t})=\max_{a_{t}}Q_{t+1}(s_{t},a_{t}) \tag{17}\]  
The deep Q-network improves the Q-learning by combining DNN with Q-learning, to approximate the Q-function by a DNN. The deep Q-learning updates the Q-network with  
Figure 2: The structure of DQN.  
weight \(\{\theta\}\)[14]. When \(\{\theta\}\) is determined, \(Q(s_{t},a_{t})\) becomes the output of the DNN. The Q-network updates the weight \(\theta\) to minimize the following loss function.  
\[Loss(\theta)=\sum_{(s_{t},a_{t})}(y-Q(s_{t},a_{t},\theta))^{2} \tag{18}\]  
where  
\[y=r_{t}+\max_{a\in A}Q(s_{t},a_{t},\theta) \tag{19}\]  
where \(r_{t}\) is the corresponding reward.