## 4 DRL for Resource Management  
In this section, we describe how to use the DRL method within the proposed framework to solve the spectrum allocation, including the proposed partial CSI observations and double DQN learning algorithm. For mode 3 resource allocation, our objective is to find the optimal resource allocation policy that can meet the latency demand of V2V links without excessively affecting the V2I links data rate under low signaling overhead. In addition, the vehicular communication network needs an intelligent resource management framework to make decisions intelligently, therefore, we adopt DRL architecture to make resource-sharing decisions. In the RL, an agent connects its environment, at each moment, the agent takes action based on the current environment state. The action will impact the state of the environment, causing certain changes in the environment. Then the agent receives numerical rewards and observes a new environment state. Through continuously exploiting the environment, agents learn how the environment generates rewards and then update their policy accordingly. In RL, the agent's objective is to maximize the expected sum of rewards, which is defined as  
\[R_{t}=\sum_{k=0}^{\infty}\gamma^{k}r_{t+k+1} \tag{6}\]  
where \(\gamma\in\) [0,1] is called the discounter factor. We adopt double DQN with experience replay architecture for subcarrier assignment, as shown in Figure 2. BS is regarded as a learning agent and the vehicular network acts as the environment state. Three key elements of the proposed model, i.e., state, action and reward, are defined, respectively, as follows. As shown in Figure 2, each V2V pair observed local information, which includes communication interference channel power gain and latency information. We consider that it is difficult to obtain a complete CSI in the vehicle network, so only part of the CSI is considered in our observation space design. This helps to reduce the signaling overhead caused by channel estimation and CSI feedback. To consider the low latency communication  
Figure 2: Proposed framework for resource allocation. requirements, key elements related to latency should be involved. In Figure 2, \(\mathbf{I}_{n,t-1}\), \(L_{n,t}\), \(T_{n,t}\) represents the interference power gain received by each sub-channel of the \(n\)th V2V link in the previous time slot, remaining payload size for transmission and the time left before violating the latency constraint; among them, \(I_{n,t}^{m}\) can be written as follows  
\[I_{n,t}^{m}=\sum_{l\
eq n}^{\text{N}}p_{l,t}^{m}p_{l}g_{l,n,t}^{m}+p_{m}g_{m,n,t} ^{m} \tag{7}\]  
In summary, the partial CSI observation space of the \(n\)th V2V pair can be expressed as  
\[\mathbf{o}_{n,t}=\{\mathbf{I}_{n,t-1},\ L_{n,t},\ T_{n,t}\} \tag{8}\]  
where \(\mathbf{I}_{n,t}\)=(\(I_{n,t}^{1}\), \(I_{n,t}^{2}\), \(I_{n,t}^{3}\),..., \(I_{n,t}^{M}\)). Specifically, the complete CSI observation space is defined as  
\[\widetilde{\mathbf{o}}_{n,t}=\{\mathbf{G}_{n,t},\ \mathbf{I}_{n,t-1},\ L_{n,t},\ T_{n,t}\} \tag{9}\]  
where \(\mathbf{G}_{n,t}\)=(\(\mathbf{g}_{n,t}\), \(\mathbf{g}_{n,n,t}\), \(\mathbf{g}_{n,t}\), \(\mathbf{g}_{n,t}\)). Feeding back complete CSI to the BS will generate a large signaling overhead, and it is unrealistic in the dynamic environment of the vehicle, especially the links not connected to the BS, such as the V2V communication links and the interference of other vehicles on the V2V links. To obtain the \(\mathbf{g}_{n,t}\), \(\mathbf{g}_{m,n,t}\) and \(\mathbf{g}_{l,n,t}\), channel estimation at the receiver side is necessary, after which the estimation results can be fed back to the transmitter. Then the transmitter feeds back to BS. Therefore, we consider not using complete CSI. On the other hand, in mode 4, the user selects the spectrum resource with sensing channel measurements, the level of interference that will be experienced if the sensing user transmits in the corresponding spectrum resource [31].