# An Integrated DQN and RF Packet Routing Framework for the V2X Network
###### Abstract  
With the development of artificial intelligence technology, deep reinforcement learning (DRL) has become a major approach to the design of intelligent vehicle-to-everything (V2X) routing protocols for vehicular ad hoc networks (VANETs). However, if the V2X routing protocol does not consider both real-time traffic conditions and historical vehicle trajectory information, the source vehicle may not transfer its packet to the correct relay vehicles and, finally, to the destination. Thus, this kind of routing protocol fails to guarantee successful packet delivery. Using the greater network flexibility and scalability of the software-defined network (SDN) architecture, this study designs a two-phase integrated DQN and RF Packet Routing Framework (IDRF) that combines the deep Q-learning network (DQN) and random forest (RF) approaches. First, the IDRF offline phase corrects the vehicle's historical trajectory information using the vehicle trajectory continuity algorithm and trains the DQN model. Then, the IDRF real-time phase judges whether vehicles can meet each other and makes a real-time routing decision to select the most appropriate relay vehicle after adding real-time vehicles to the VANET. In this way, the IDRF can obtain the packet transfer path with the shortest end-to-end delay. Compared to two DQN-based approaches, i.e., TDRL-RP and VRDRT, and traditional VANET routing algorithms, the IDRF exhibits significant performance improvements for both sparse and congested periods during intensive simulations of the historical GPS trajectories of 10,357 taxis within Beijing city. Performance improvements in the average packet delivery ratio, end-to-end delay, and overhead ratio of the IDRF over TDRL-RP and VRDRT under different numbers of pairs and transmission ranges are at least 3.56%, 12.73%, and 5.14% and 6.06%, 11.84%, and 7.08%, respectively. to relay packets to other vehicles through Roadside Units (RSUs) in the network. Typically, there are physical wired connections between RSUs and the network infrastructure, and IEEE 802.11p [4] is adopted for communication between vehicles. Despite this, a loss of signal strength is caused by buildings or obstacles in the environment, leading to unstable communication. Therefore, a framework is required to gather information about the current state of the V2X network to determine the optimal source-to-destination packet relay path. Software-defined networking (SDN) is a new technology [5] that allows dynamic changes in the network. SDN is divided into two main components within the network environment: the Control Node (CN) and the Data Node (DN). The CN controls the vehicles when SDN is applied to V2X, and the DN is responsible for communication between vehicles, RSUs, and SDN/OpenFlow switches. RSUs are installed at each intersection to collect information about the vehicles. By utilizing the SDN architecture, the CN can obtain the statuses of all vehicles through RSUs and make optimal decisions on packet forwarding paths in the V2X environment (Figure 1). However, vehicle communication is often unstable due to the high vehicle mobility, insufficient reliability of the inter-vehicle links, and inadequate infrastructure. Therefore, a well-designed packet routing protocol for V2X is crucial. Recently, the integration of artificial intelligence has optimized routing decision-making [6; 7]. The three main components of reinforcement learning (RL) [8; 9] are environment, decision, and reward. Based on the current state of the environment, the agent selects an action and receives a corresponding reward. Consequently, the environment transitions into a new state. Using Q-learning [10] as an example, the values generated from performing actions in different states are recorded and stored in the Q-table during the reinforcement learning process. Having an excessive number of executed actions and their associated values can lead to excessive occupation of the storage space in the Q-table.