# Multi-Path Routing Algorithm Based on Deep Reinforcement Learning for SDN
## 2 Related Works
### Reinforcement Learning in SDN  
SDN has shifted traditional network operations from hardware to software, both simplifying and integrating the functionality of the network control plane while also enhancing the reliability of network hardware devices. With the continuous growth in network complexity and traffic demands, traditional shortest path algorithms suffer from drawbacks such as slow convergence and network congestion. Researchers have turned to machine learning, specifically deep reinforcement learning, to optimize SDN route selection. Compared to traditional methods, DRL may introduce some additional overhead. Deep learning models have a large number of parameters, requiring more storage space to save and load the model. DRL models need to be trained, which might require a significant amount of time and data. Traditional methods are usually based on fixed algorithms and do not require a training process. DRL might experience unstable phases when learning and adapting to new environments, which could lead to short-term instability in routing policies. Some DRL approaches use an experience replay mechanism, necessitating the storage and processing of vast amounts of historical data. Despite these overheads, DRL remains attractive for SDN routing decisions because it can learn and adapt to complex network environments and often outperforms traditional methods in many scenarios. Various studies have explored the application of deep reinforcement learning in SDN route optimization, focusing on intelligent routing, customization, and fine-grained management. Table 1 presents a succinct overview of recent research employing DRL in SDN. The above table summary elucidates the learning techniques applied, the formulation of actions, and the criteria used for assessment. In the studies [43; 44], the authors employed the RL method Q-Learning, which relies on Q-tables and demands significant memory, data, and time resources. Conversely, refs. [25; 45; 49] used the DRL method DQN, and [46] employed the Dueling Double DQN method, which optimizes DQN. These methods utilize deep learning networks to approximate values instead of Q-tables, making them more practical and scalable than Q-Learning. Additionally, refs. [26; 47] respectively used the Advanced AC (A2C) and PPO methods, which employ policy gradients instead of value approximation. These methods can enhance convergence compared to DQN. In studies [26; 43; 44; 45; 46], the choice of the next hop is used as the action. These studies reveal that this approach fulfills end-to-end performance requirements only when the appropriate next hop is selected. An incorrect choice can lead to significant performance degradation and even routing loops. In contrast, ref. [47] employs the adjustment of graph neural network parameters as the action, which requires reinforcement learning assisted by a specific graph neural network, making it less generalizable. Furthermore, the concept of the takeover decision of the switches was employed as the action in [48] to adaptively mitigate the propagation of attacks, thereby enhancing the resilience of Software Defined Industrial Networks (SDIN). The authors in [25] select routes from a preselected route group, which eliminates the possibility of routing loops and provides stable connections. However, as the number of nodes increases, the count of preselected paths may grow rapidly, introducing security vulnerabilities. Therefore, there is a compelling need to devise high-efficiency DRL algorithms capable of addressing these concerns. Contrary to what is described in Section 2.1, these studies only considered single-route optimization for the entire route without taking into account multi-route performance. More critically, they did not consider the security performance under a DDoS attack environment, thereby overlooking the potential adverse impacts, such as network congestion, that the crucial factor of security might bring to the network.