## 2 Related Work  
Routing is a fundamental function in the Internet, responsible for forwarding data packets from source to destination addresses. It plays a crucial role in ensuring Quality of Service (QoS) [6]. Traditional routing strategies, like Open Shortest Path First (OSPF) routing [7], may lead to issues such as network congestion and low link utilization. Comparedto optimal routing methods, traditional approaches may exhibit performance differences of up to 5000 times [8]. Routing optimization has been a topic of in-depth research [9]. Existing solutions include those based on analytical optimization [10] and those leveraging machine learning [11; 12] and deep learning [13; 14]. Papers [9; 15; 16; 17] also proposed various SDN routing optimization methods. In this section, we review SDN routing optimization work based on Reinforcement Learning (RL) and Deep Reinforcement Learning (DRL). Unlike model-based routing optimization algorithms, RL-based methods are model-free. Moreover, while machine learning algorithms require dataset labeling and training, RL algorithms directly interact with the network environment to learn strategies. The key application of reinforcement learning in routing problems focuses on modeling the forwarding process using the Markov Decision Process (MDP). This process involves setting appropriate states, actions, and reward functions based on specific network scenarios. The intelligent agent adapts to various dynamic changes through interaction with the environment [18]. RL can play a crucial role in SDN routing optimization [19]. As early as 1993, [PERSON] et al. [20] proposed using the Q-routing algorithm to avoid network congestion, marking the first academic use of reinforcement learning for routing optimization. Although this algorithm was not applied in an SDN architecture, it provided a new reference direction for subsequent routing optimization algorithms. [PERSON] et al. [21] introduced multi-layer SDN QoS-aware adaptive routing, combining the work of [PERSON] [22] and [PERSON] [23], introducing a distributed control plane architecture, and achieving a reliable SDN infrastructure with minimal signal latency. [PERSON] et al. [24] proposed the QR-SDN algorithm to create multiple paths between source and destination addresses, maintaining flow integrity using Q-Learning, and employing the softmax function as an exploration-exploitation strategy. However, this algorithm can only guarantee lower flow latency than Shortest Path First (SPF) in small networks. [PERSON] et al. [25] used the Q-Learning algorithm to address the delay minimization problem in multipath SDN networks. They employed two different exploration strategies, namely \(e\)-greedy, and softmax, and evaluated the performance of the Q-Learning algorithm based on average flow latency and convergence time for different load levels. [PERSON] et al. [26] proposed the RSIR algorithm, which defined an active routing algorithm based on link-state, exhibiting lower overall latency and better load balancing compared to the traditional Dijkstra algorithm. Building on this work, they also introduced Deep Reinforcement Learning (DRL) and defined the DRSIR algorithm [27] in SDN. DRSIR considers path state indicators to generate active, efficient, and intelligent routing to adapt to dynamic traffic changes. The evaluation of DRSIR was conducted through simulation using real and synthetic traffic matrices. [PERSON] et al. [9] also introduced DRL into the routing process and proposed the RL-Routing algorithm to address Traffic Engineering (TE) issues in SDN regarding throughput and latency. They performed comprehensive experiments based on Fattree, NSFNet, and ARPANet topologies, demonstrating that experience-driven artificial intelligence has advantages over traditional algorithms in solving TE problems. [PERSON] et al. [28] proposed a DDPG routing optimization solution, which, unlike the previous Q-table-based mechanism, saves time and storage by using a neural network. [PERSON] et al.