# Multi-Path Routing Algorithm Based on Deep Reinforcement Learning for SDN
## 5 Results and Validation
### Performance Comparison
#### 5.3.4 Delay Comparison  
Figure 5 illustrates the delay performance comparison in the NSFNet-14 and Germany-50 topologies conducted following the simulation of 3000-episode iterations. Figure 5a shows the delay performance in the NSFNet-14 topology. After 3000-episode iterations, TBPPO, TBDRSIR, PPO, and DRSIR divided packets into multiple routes, while Dijkstra only used the shortest path with a fixed delay of 160 ms. TBPPO converged to 223.3 ms after 600-episode iterations, and PPO reached 216.67 ms after 500-episode iterations in terms of average delay. DRSIR displayed gradual convergence, approaching around 260 ms after 1200-episode iterations, albeit with some noticeable fluctuations. In contrast, TBDRSIR initially demonstrated convergence to 300 ms, but experienced a sudden divergence around 2700-episode iterations, ultimately failing to converge. These findings indicate that in the context of low-delay network exploration, PPO outperforms DQN.  
Figure 4: Performance comparison of reward of DRL algorithms: (**a**) NSFNet-14, and (**b**) Germany-50.  
Additionally, TBPPO, which needs to balance both exploration and security considerations while also not significantly surpassing PPO, remains competitive. Furthermore, when compared to the theoretically optimal Dijkstra, both TBPPO and PPO exhibit a relatively similar performance.  
In Figure 5b, the delay performance in the Germany-50 network topology is presented. This network comprises 50 nodes and 176 edges, resulting in a larger number of possible routes compared to NSFNet-14. To address resource constraints, the paths were sorted by delay, with the top 10,000 routes retained. The action space for testing the algorithm consisted of these 10,000 routes. After a 3000-cycle simulation, TBPPO, TBDRSIR, PPO, and DRSIR employed a data packet division strategy using three distinct paths for the data transmission. In contrast, the Dijkstra algorithm, in its quest for the shortest route within the network topology, funneled all traffic along that singular path. Within the simulated network configuration, only one shortest path with a 160 ms delay was available, which ultimately led to Dijkstra maintaining a consistent average delay of 250 ms. According to the delay comparison in the figure, the convergence patterns of various algorithms can be observed. TBPPO starts to approach 423.3 ms after roughly 1400-episode iterations, while PPO achieves convergence at 503.3 ms after 1200-episode iterations. DRSIR begins to approach the 657 ms mark after 300-episode iterations, but its performance continues to fluctuate. In contrast, TBDRSIR does not reach convergence. It is evident that TBPPO consistently performs well on large networks, while TBDRSIR, in our analysis, seems to exhibit an inferior performance, possibly due to the inclusion of the trust mechanism, which appears to negatively impact DQN's performance. When compared to NSFNet-14, PPO and TBPPO still showcase superior convergence and optimization capabilities. Moreover, as the network scales up, PPO's advantage in securing routing becomes more apparent, resulting in lower delay due to fewer instances of attacks.