# An Integrated DQN and RF Packet Routing Framework for the V2X Network
## 3 Design of the IDRF System
### Offline Phase
#### 3.1.5 Deep Reinforcement Learning Training Unit  
In this study, the DQN approach was employed to make decisions about the packet forwarding transmission path to achieve the shortest end-to-end delay from the source vehicle S to the destination vehicle D. This decision-making process takes into account the presence of intermediate relay nodes up to M hops (Figure 15). The V2X vehicular network environment consists of vehicles. Within its communication radius, the source vehicle S has \(N_{1}\) neighboring candidate vehicles at Hop 1 that are capable of relaying packets. This implies that, in the current state \(s_{1}\), there are \(N_{1}\) possible actions to choose from. Vehicle \(v_{i}\) represents the i-th candidate node within the communication radius of the current node \(v_{carry}\), which is currently carrying the packet. Importantly, \(v_{i}\) is not chosen as the i-th relay node. \(D_{i}\) represents the total delay required for source vehicle S to relay the packet to \(v_{i}\) successfully. Figure 14: Predicted visited intersections and corresponding arrival times determined using the RF prediction unit (the red frame shows the next intersection, the intersection after the next, and the corresponding arrival times). Figure 15: Network model of the decision-making process. First, we designed the DQN state as follows: \(State=\{\)Vehicle ID \(V_{i}\), road segment ID \(S_{i}\), intersection ID \(I_{i}\), vehicle's driving speed \(speed_{i}\), vehicle's driving angle \(angle_{i}\), whether \(v_{carry}\) is the packet-sending node \(v_{src}\), whether \(v_{carry}\) is a relay node \(v_{relay}\), \((S_{i},I_{i})\), \((speed_{i},angle_{i})\), and (whether \(v_{carry}\) is \(v_{src}\), whether \(v_{carry}\) is \(v_{relay}\)), resulting in a total of 10 dimensions for the DQN neural network. The indication of whether \(v_{carry}\) is a relay node \(v_{relay}\) signifies that the current node \(v_{carry}\), which is carrying the packet, is neither the sending node \(v_{src}\) nor the target node \(v_{dst}\), but rather, is sending other vehicle nodes. The combination of data with similar attributes can increase the sample count, providing more diverse information for the DQN while making decisions and enhancing the decision quality. The _Action_ refers to the selection of a neighboring candidate vehicle within the communication radius of the vehicle carrying the packet, which is also the output dimensionality of the DQN network. The _Reward_ is designed according to Equation (5), and it encompasses three distinct scenarios: (1) If the DQN selects \(V_{i}\) as a relay node from the candidate vehicles and the packet transmission is complete, the _Reward_ is given as \(Reward=\frac{1-DR_{i}}{\sum_{j=1}^{candidate}1-DR_{j}}\), where \(DR_{i}=\frac{D_{i}}{\sum_{j=1}^{candidate}D_{j}}\) is equal to the proportion of the transmission delay \(D_{i}\) to \(\sum_{j=1}^{candidate}D_{j}\). The objective of the DQN is to determine the packet transmission path with the shortest end-to-end delay. Therefore, assigning higher rewards to candidate vehicles with lower transmission delays \(D_{i}\) (but with higher \(1-DR_{i}\)) is essential. In the end, all candidate vehicle values are normalized within the range [0-1], yielding the final reward \(\frac{1-DR_{i}}{\sum_{j=1}^{candidate}1-DR_{j}}\). (2) If \(V_{i}\) is not chosen as a relay node among all candidate vehicles, the reward is set to 0. (3) If \(V_{i}\) is selected as a relay node among all candidate vehicles, the transmission to \(V_{i}\) fails, and the reward is assigned a value of \(-1\). Based on the rewards provided during the DQN training process, we can obtain weights [\(W_{s,1}(1)\), \(W_{s,2}(1)\), \(\ldots\), \(W_{s,N_{1}}(1)\)] for the selection of actions [1, 2, \(\ldots\), \(N_{1}\)]. The packet forwarding path must be chosen by using the candidate neighboring vehicle with the highest trained weight at each hop to achieve the shortest end-to-end delay for packet transmission from the source vehicle to the destination vehicle.