## 4 DRL for Resource Management
Therefore, we are motivated to consider designing a partial CSI resource allocation mechanism using interference power measurement. Compared with the complete CSI version designed as (9), the partial CSI observation space designed as (8) can help reduce the signaling overhead. (1) State Space: As stated above, to reduce signaling overhead in a centralized solution, in our solution, each V2V link feeds interference and delay information back to the BS. Hence, BS considers the partial CSI feedback from all V2V pairs as the current state. Thus, the state space of the BS is described as follows  
\[\mathbf{s}_{t}=\{\mathbf{o}_{1,t},\mathbf{o}_{2,t},\mathbf{o}_{3,t},\dots, \mathbf{o}_{\text{N},t}\} \tag{10}\]  
(2) Action Space: In this paper, we focus on the subcarrier assignment issues in the V2V communication network. Hence, the action is defined as  
\[\mathbf{a}_{t}=\{b_{1,t},b_{2,t},b_{3,t},\dots,b_{n,t}\} \tag{11}\]  
where \(b_{n,t}=m\), \(\forall m\in\text{M}\) represents that the \(m\)th subcarrier has been selected by the \(n\)th V2V pair. (3) Reward: In the RL framework, the training process is driven by the reward function. An agent searches decision-making policy by maximizing reward under the interaction with the environment. Hence, to meet the different requirements of communication devices, corresponding reward functions need to be formulated. In vehicular communications, V2V links exchange critical safety information and have strict latency requirements, whereas V2I links support high-rate data transmission [32]. Our objective is to improve the sum throughput of V2I links while meeting the latency requirements of V2V links. Therefore, we propose the following reward function  
\[r_{t}=\lambda_{c}{\sum_{m=1}^{\text{M}}R_{m}^{m}[t]+\lambda_{c}{\sum_{m=1}^{ \text{N}}G_{n}[t]}} \tag{12}\]  
where \(\lambda_{c}\), and \(\lambda_{c}\) are the weight factors for the contributions of these two parts to the reward function composition. In response to the first objective, we simply include the sum capacity of all V2I links. To achieve V2V link low latency requirements, for each V2V link, we set the reward \(G_{n}\) equal to the effective V2V transmission rate until the V2V links deliver the payload successfully within the delay constraint, after which the reward is set to a constant number, \(c\), that is greater than the largest possible V2V transmission rate. Therefore, the sooner the V2V link completes the transmission, the more rewards can be obtained. As such, the V2V-related reward at each time step \(t\) is  
\[G_{n}(t)=\left\{ \begin{array}{ll}R_{n}^{m}(t)&L_{n,t}>0\\ c&L_{n,t}\leq 0\end{array} \right. \tag{13}\]  
Q-learning [33] is a typical RL algorithm, which can be used to learn the optimal strategy when the state-action space is small. In the Q-learning algorithm, the action-value function is used to calculate the expected accumulative rewards for starting from a state \(s\) by taking action \(a\) under policy \(\pi\), which can be expressed by  
\[Q^{\pi}(s,a)=\mathrm{E}[R_{t}|s_{t}=s,a_{t}=a] \tag{14}\]  
Similarly, the optimal action-value function is obtained by  
\[Q^{*}(s,a)=\mathrm{E}\bigg{[}r_{t+1}+\gamma\underset{a_{t+1}}{\max}Q^{*}(s_{t +1},a_{t+1})|s_{t}=s,a_{t}=a\bigg{]} \tag{15}\]  
where \(s_{t+1}\) is the new state after taking action \(a\). The action-value function is updated by  
\[Q(s_{t},a_{t})\gets Q(s_{t},a_{t})+\alpha\bigg{[}r_{t+1}+\gamma\underset {a_{t+1}}{\max}Q(s_{t+1},a_{t+1})-Q(s_{t},a_{t})\bigg{]} \tag{16}\]  
where \(\alpha\) is the step-size parameter. Moreover, the choice of action \(a_{t}\) in state \(s_{t}\) follows exploitation and exploration policies, a widely used algorithm is the \(\epsilon\) -greedy algorithm [34], which is defined as  
\[a\leftarrow\left\{ \begin{array}{ll}\underset{a}{\mathrm{argmax}}Q(s,a)& \text{with probability }1{-}\epsilon\\ \text{random action}&\text{with probability }\epsilon\end{array} \right. \tag{17}\]  
Here, \(\epsilon\) is the exploration rate.