# An Integrated DQN and RF Packet Routing Framework for the V2X Network
## 4 System Simulations and Performance Evaluations
### DQN Hyperparameter Configuration and Testing  
In the following section, we discuss the hyperparameter configuration used for DQN training in this paper. The vehicle trajectory data of the Beijing city taxi dataset used for training is collected, as described in Section 3.1. The vertical axis primarily focuses on the cumulative reward, defined by Equation (13), while the horizontal axis represents training epochs. This paper defines the cumulative reward as the average of all packet forwarding paths. To begin, the cumulative reward of packet \(i\) in the forwarding path of packet \(i\) is calculated. Whenever the packet passes through relay node \(j\) in the forwarding path of packet \(i\), a reward \(\textit{Reward}_{j}^{i}\) is obtained. Thus, \(\sum_{j=1}^{h}\textit{Reward}_{j}^{i}\), where \(h\) is the number of relay nodes traversed by packet \(i\) along its forwarding path from the source to the destination. Subsequently, the sum of the rewards across all packet forwarding paths \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Parameter Value** \\ \hline Map size & 3880 M \(\times\) 5636 M \\ \hline Simulation time & 12:00–13:00 (3600 s for congestion periods), 13:00–17:00 (7200 s for sparse periods) \\ \hline MAC protocol & IEEE 802.11p \\ \hline Radio propagation model & Log Distance Propagation Loss Model \\ \hline Packet size & 1024 Bytes \\ \hline Buffer size & 10 MBytes \\ \hline Number of trained vehicles & 8456 (congestion periods), 3481 (sparse periods) \\ \hline Number of untrained vehicles & 1212 (congestion periods), 584 (sparse periods) \\ \hline Transmission range (M) & 300, 425 (default value), 550, 675 \\ \hline Number of pairs & 4, 8, 16, 32 (default value) \\ \hline Message Time & 1 \\ \hline TTL & 90 \\ \hline Learning rate \(\alpha\) & 0.001 (default value), 0.05, 0.01 \\ \hline Discount factor \(\gamma\) & 0.9, 0.95, 0.99 (default value) \\ \hline RF threshold & Gini \\ \hline \hline \end{tabular}
\end{table} 
Table 7: NS-3 Simulation Parameters. is computed as \(\sum_{i=1}^{Number\ of\ transmitted\ packets\ by\ source\ \sum_{j=1}^{h}\ Reward_{j}^{i}\). Finally, the cumulative reward is determined using Equation (13). \[\textit{Cumulative\ reward}=\frac{\sum_{i=1}^{Number\ of\ transmitted\ packets\ by\ source\ \sum_{j=1}^{h}\ Reward_{j}^{i}}}{Number\ of\ transmitted\ packets\ by\ source} \tag{13}\]  
Figures 28 and 29 illustrate the cumulative rewards obtained from training for the sparse and congested periods with three different learning rates (0.001, 0.005, 0.01). For sparse and congested periods, a learning rate of 0.01 achieves the highest cumulative reward at epochs 3000 and 5000, respectively. However, declines in the cumulative rewards appear as the epochs progress to 12,000 and 15,000, respectively, indicating that an excessively high learning rate of 0.01 is detrimental to the environment. With a learning rate of 0.001, the highest cumulative rewards are achieved stably after epochs 8000 and 11,000, respectively. Comparing the outcomes with learning rates of 0.005 and 0.001, the latter yields higher cumulative rewards, albeit demanding more extensive training durations. Figures 30 and 31 depict the cumulative rewards from training for the sparse and congested periods with three different discount factors (0.9, 0.95, and 0.99). For the sparse and congested periods, a discount factor of 0.9 achieves the highest cumulative reward at epochs 4000 and 5000 and remains stable as the epochs advance to 12,000 and 15,000, respectively. This indicates that a lower discount factor of 0.9 leads to quicker cumulative  
Figure 28: Cumulative rewards of different learning rates for the sparse period. Figure 29: Cumulative rewards of different learning rates for the congested period. reward accumulation but limits the peak value. A discount factor of 0.95 yields the highest cumulative rewards during epochs 6000-7000 and 7000-8000, maintaining stability as the epochs advance to 12,000 and 15,000, respectively.