# Multi-Path Routing Algorithm Based on Deep Reinforcement Learning for SDN
## 6 Conclusions  
This paper presents the TBPPO algorithm, a deep reinforcement learning-based approach that addresses the limitations of existing intelligent routing algorithms in SDN environments. TBPPO leverages a unique trust value mechanism based on KL divergence and optimizes DFS and PPO algorithms to establish secure, low-delay routing solutions. Initially, it refines the DFS algorithm for route selection, effectively reducing the complexity of the reinforcement learning network's action space and avoiding routing loops. Furthermore, it employs an improved KL divergence algorithm to estimate the trust values related to potential node attacks and a node diversity assessment method to estimate the ISP balance, providing an easy-to-calculate security assessment method for this algorithm. Lastly, the algorithm enhances PPO to explore multiple performance-balanced paths, enabling diverse network configurations and communication pairs to select routes that align with the network characteristics. We demonstrated the effectiveness of the TBPPO algorithm compared to cutting-edge methods in medium to large network topologies. The result show that, in large networks, TBPPO with security mechanisms experienced fewer attacks in route selection compared to PPO. Both its delay and variation performance were reduced by about 100 ms, its trust value led by 0.5, and the diversity of nodes was ahead by 20. Moreover, when comparing TBPPO with PPO and TBDRSIR, PPO still converges stably under complex multi-objective dynamic conditions. This result can be attributed to the excellent stability of PPO. In multi-route performance tests, we discovered that when TBPPO is used with more than four routes, its performance deteriorates significantly; therefore, we recommend applying TBPPO to three or four route path tasks to showcase its best capabilities. We hope that TBPPO can offer a better multi-path secure routing optimization method for real-time communication, such as in scenarios like online classes, streaming videos, live web broadcasts, and real-time meetings. In future research, we will focus on the application of the algorithm in heterogeneous network scenarios, making it adaptable to a wider range of practical application scenarios, like smart grid. We plan to introduce the graph convolutional networks (GCNs) into the DRL process for a better performance.  
Conceptualization, Y.Z., L.Q. and Z.W.; methodology, Z.W.; software, Y.X.; validation, X.W. and S.W.; formal analysis, A.P.; investigation, Z.W.; resources, Y.Z.; data curation, S.W.; writing--original draft preparation, S.W.; writing--review and editing, L.Q. and A.P.; visualization, Y.X. and X.W.; supervision, Z.W.; project administration, Y.Z.; funding acquisition, Y.Z. All authors have read and agreed to the published version of the manuscript.  
This research was funded by Science and Technology Project of State Grid Zhejiang Electric Power Co., grant number B311 XT230019.  
Not applicable.  
The data presented in this study are available on request from the corresponding author. The data are not publicly available due to privacy.  
The authors [PERSON], [PERSON], [PERSON] and [PERSON] were employed by the company Information Communication Branch of State Grid Zhejiang Electric Power Co., Hangzhou 310007, China. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. The funder had no role in the design of the study; in the collection, analysis, or interpretation of data, in the writing of the manuscript, or in the decision to publish the results.