# An Integrated DQN and RF Packet Routing Framework for the V2X Network
###### Abstract
To reduce the size of the Q-table in Q-learning, we proposed the Software-Defined Directional QGRID (SD-QGRID) routing architecture [11], which uses the grid-based concept to partition the entire environment into uniformly sized grids. By deploying the SDN CN and enabling all vehicles to exchange HELLO messages, a V2X centralized control is established. Hence, the SD-QGRID routing algorithm improves the computation method employed by the Q-table and determines the macroscopic packet transmission direction between grids by considering the real-time locations of vehicles and their historical trajectory records involving movement to adjacent grids. Using fuzzy Q-learning, ITAR-FQ [12], reduces potential signal propagation obstruction by buildings, which may lead to rapid signal decay. ITAR-FQ consists of two components: RTAP-RE and RDP-FQ. RTAP-RE is designed to handle traffic information and estimate the road quality (RQ) through a road assessment. On the other hand, RDP-FQ is responsible for exchanging traffic information at intersections between adjacent roads. However, accurate vehicle trajectory data and the issue of end-to-end delay have not been considered in experiments involving ITAR-FQ. As  
Figure 1: The SDN networking architecture used in this study. a result, vehicles carry packets for extended periods, impeding their ability to efficiently transmit them to the destination node through V2X networks. Additionally, the algorithm does not promptly update the optimal routing paths when new vehicles join the network. QAGR [13] optimizes the routing scheme for global message forwarding by categorizing the environment into two parts, aerial UAVs and ground vehicles, with UAVs being responsible for the collection of global road traffic information. Vehicles maintain a fixed-size Q-table and converge towards the reward function proposed by the author. A search of the Q-table filtered by global routing paths allows route requests to be relayed to the optimal nodes. PFQ-AODV [14] incorporates Fuzzy Logic and learns the optimal routing path through the Q-learning method, thereby addressing the challenge of route selection in VANET environments. In summary, these Q-learning-based routing protocols suffer from two apparent problems. First, they do not utilize trajectory data from new vehicles in real-time or consider the reliability of inter-vehicle links, end-to-end delay, vehicle travel directions, or the latest packet forwarding paths. The absence of past historical data for new vehicles presents a challenge when training the Q-learning model, as the optimal routing paths for these vehicles cannot be updated promptly. Second, the excessive number of executed actions and generated values in the Q-table can lead to significant storage space consumption during Q-learning. Deep reinforcement learning (DRL) [15; 16] addresses the issues encountered in complex environments by substituting the Q-table used in Q-learning with neural network layers. TDRL-RP [17] introduces a software-defined trust-based DRL framework. Since the vulnerability of vehicle infrastructure communication to attacks from malicious nodes results in degradation of the overall network performance, the algorithm uses DRL on top of SDN to learn the optimal routing paths in the VANET environment. Its objective is to enhance the network's scalability, flexibility, and self-learning capabilities. Compared to the initial neural network approach, faster DRL convergence can be achieved by integrating a CNN model with SDN. Regarding the packet forwarding strategy, a neighbor behavioral trust module is employed with TDRL-RP. However, this still does not account for the latest forwarding paths, end-to-end delay, or vehicle travel direction. VRDRT [18] integrates DRL into the VANET environment. Due to the high vehicle mobility and consequent rapid changes in the communication links and network topology, algorithm design must address packet transmission delay and stability issues in the networks.