# Energy-Efficient Resource Allocation Based on Deep Q-Network in V2V Communications
## 4 Deep Q-Network for Energy-Efficient Resource Allocation  
### Reinforcement Learning  
In RL, an agent observes a state in an environment that satisfies the Markov decision process (MDP). Then, the optimal action is selected according to the given policy. Depending on the selected action, the agent interacts with the environment, receives a reward from the environment, and transitions to the next state.  
The goal of RL is to maximize the expected return value after the episode ends. The return formula is given as follows:  
\[R_{t:T}=r_{t}+\gamma r_{t+1}+\gamma^{2}r_{t+2}+\ldots+\gamma^{T}r_{t+T-1}, \tag{7}\]  
where \(r_{t}\) denotes the reward obtained immediately at time \(t\), \(T\) is the time step, and \(\gamma\) denotes the discount factor. The structure of our RL is shown in Figure 2. The agent observes the state of the environment at the time (\(t\)) and selects the best action according to the given policy. When the agent selects an action for all V2V links, the actions are stored in the joint action group and interact with the environment at the same time, and the agent receives a reward. In our system model, we develop the RL with the following parameters:  
1. _State space:_ We use the following state, similar to the unicast scenario of [11]. \[\mathbf{s}_{t}=\{\mathbf{H}_{t},\mathbf{I}_{t-1},\mathbf{G}_{t},\mathbf{N}_{t -1},U_{t},L_{t}\},\] (8) where \(\mathbf{H}_{t}\) is the CSI of V2I links at time \(t\); \(\mathbf{I}_{t-1}\) is the interference power to the link at time \(t-1\); \(\mathbf{G}_{t}\) is the instantaneous CSI of the corresponding V2V link at time \(t\); \(\mathbf{N}_{t-1}\) is the information of RBs selected by surrounding vehicles at time \(t-1\); \(U_{t}\) is the time remaining to satisfy the latency constraints at time \(t\); and \(L_{t}\) is the remaining data to be received from the transmitter of the V2V link at time \(t\). \(\mathbf{H}_{t}\), \(\mathbf{I}_{t-1}\), \(\mathbf{G}_{t}\), and \(\mathbf{N}_{t-1}\) are vectors containing the state information of the corresponding RBs, and \(U_{t}\) and \(L_{t}\) are scalar values that are the time remaining to satisfy the latency constraints and the remaining data, respectively. Therefore, the dimension of the state space is given by \(D_{\text{state}}=(4\times N_{\text{RB}})+2\).
2. _Action space:_ The action determines the transmit power and the allocation of RBs. Hence, the dimensions of the action space are given by \(D_{\text{action}}=N_{\text{pwr}}\times N_{\text{RB}}\), where \(N_{\text{pwr}}\) is the number of transmit power levels in the V2V link and \(N_{\text{RB}}\) is the number of RBs.
3. _Reward:_ We formulate the following reward function taking two penalties into account, the transmission time and the power consumption: \[r_{t}=\lambda_{\text{V2I}}\sum_{l\in\mathcal{L}}C_{l}+\lambda_{\text{V2V}} \sum_{k\in\mathcal{K}}C_{k}-\lambda_{\text{latency}}(T_{0}-U_{t})-\lambda_{ \text{pwr}}\frac{1}{K}\sum_{k\in\mathcal{K}}\frac{p_{k}}{p_{\text{max}}},\] (9) where \(T_{0}\) is the maximum tolerable latency, and therefore, \((T_{0}-U_{t})\) means the transmission time. Moreover, \(p_{\text{max}}\) is the maximum transmit power in the V2V link. \(\lambda_{\text{V2I}}\) and \(\lambda_{\text{V2V}}\) represent the weight for the sum rate of the V2I links and the sum rate of the V2V links, respectively. \(\lambda_{\text{latency}}\) and \(\lambda_{\text{pwr}}\) represent the weight of the penalty according to an increase in the transmission time and the penalty according to an increase in the transmit power, respectively. As the sum rate of the V2I or V2V links increases, a positive factor is added. However, as the transmission time or power consumption increases, a negative factor is added.