# Multi-Path Routing Algorithm Based on Deep Reinforcement Learning for SDN
## 1 Introduction
[26] introduced a Proximal Policy Optimization (PPO)based deep reinforcement learning technique to improve SDN network routing, allowing for real-time intelligent control and administration. However, these efforts often neglect security considerations in real-time communication. Therefore, there is a pressing need for an efficient deep learning algorithm that offers strong privacy protection, real-time communication capabilities, and high efficiency. In this study, a novel multi-path routing approach called Trust-Based Proximal Policy Optimization (TBPPO) is introduced for SDN. TBPPO leverages a trust value mechanism based on Kullback-Leibler divergence (KL divergence) [27] and a node diversity assessment mechanism to enhance SDN robustness, address congestion problems, and notably fortify defenses against Distributed Denial of Service (DDoS) attacks. Furthermore, an improved PPO algorithm is tailored for optimizing multi-path routing in SDN, while also considering security, network delay, and variations in multi-path delays. This advancement is crucial for improving SDN routing optimization. To enhance computational efficiency, an enhanced Depth-First Search (DFS) algorithm is incorporated, simplifying the action space. The experimental results validate TBPPO's superior performance in terms of convergence and overall effectiveness when compared to the traditional methods. The contribution of this work can be summarized as follows:  
1. We present TBPPO, a multi-path routing algorithm designed specifically for SDN. This innovative approach integrates a trust value mechanism based on KL-divergence and node diversity as a key security assessment criterion. TBPPO addresses network fluctuations, enhances robustness, and mitigates congestion issues, with a primary focus on countering DDoS attacks. 2. We present an enhanced PPO algorithm to optimize security and efficiency in SDN routing. This novel algorithm involves the optimization of multi-path routing, con sidering factors like security, network delay, and variations in multi-path delays. To address the recurrent gradient explosion issues observed during the experimental process, we introduced learning rate decay and layer normalization. Furthermore, we incorporated a trust value-based routing selection approach, resulting in enhanced security stability and a reduced delay performance. 3. To avoid routing loops, we abandoned the traditional Next Hop routing mechanism and adopted a path selection approach. The improved DFS uses a path selection method to choose a set of promising routes, which are called routing groups. Then, we search for the best multi-path route within these routing groups. This enhancement significantly reduces packet loss and improves the overall efficiency and practicality of the algorithm. 4. Finally, we conducted experiments using the NSFNet-14 and Germany-50 network topologies. The results demonstrate that our TBPPO technique outperforms traditional approaches in terms of both convergence and overall performance. Additionally, the obtained findings emphasize TBPPO's potential as an effective solution for improving SDN security and routing efficiency. The subsequent sections of this paper will be organized as follows: Section 2 provides an overview of related works. Section 3 offers a detailed description of the system model, encompassing crucial elements such as the SDN network environment. Section 4 delves into the design and specific details of the TBPPO algorithm. This includes its operational principles and key algorithms. Section 5 presents simulation results and corresponding analyses to validate and evaluate the performance of the TBPPO algorithm in various scenarios. Section 6 summarizes the experimental conclusions.