# Multi-Path Routing Algorithm Based on Deep Reinforcement Learning for SDN
## 4 Trust-Based Proximal Policy Optimization (TBPPO)
### Security Module  
DDoS attacks represent a prevalent and disruptive network threat. These attacks involve malicious nodes intentionally discarding messages from legitimate nodes, causing severe disruptions to the data transmission. What makes these attacks particularly insidious is that these malicious nodes can mimic legitimate behavior when not receiving data packets, making them hard to detect and highly destructive [51; 52; 53]. To address the challenge of DDoS attacks and establish trust levels for network nodes, we introduce a KL divergence as a trust mechanism to counter network attacks. Initially, specific features are carefully selected as the KL benchmark, drawing from the normal network operation records. The choice of these features can be tailored to suit specific circumstances. The records are configured to combat DDoS attacks and take the following form:  
\[Group(0,v_{i})=(\text{SIP},\text{DIP},\text{SYN},\text{DP}) \tag{5}\]  
where SIP, DIP, SYN, and DP represent the frequency of the source IP, the destination IP, the SYN packets, and the different destination ports in the records of node \(v\). At the end of each predefined time interval denoted as \(\tau\), we compute the KL divergence between the records of the node within that time frame and the baseline. A higher KL divergence value serves as an indicator of a higher probability that the node is currently under attack, consequently leading to a reduced security level. The formula for computing this divergence is given as follows:  
\[trust(\tau,v_{i})=KL(Group(0,v_{i}),Group(\tau,v_{i})) \tag{6}\]  
where \(Group(0,v_{i})\) represents the KL baseline, \(Group(\tau,v_{i})\) corresponds to the records of node \(v_{i}\) within the time window \(\tau\), \(Trust(\tau,v_{i})\) signifies the trust value of node \(v_{i}\) at time \(\tau\), and \(Trust(\tau)\) denotes the trust values of all nodes at time \(\tau\). The term \(KL\) refers to the enhanced iterative KL divergence calculation formula, which is enhanced from the equation in [23] and defined as follows:  
\[KL\Big{(}p_{\parallel}q\Big{)}=\sum_{i=1}^{I}\beta_{i}p(x_{i})\Big{(}log\frac {p(x_{i})}{q(x_{i})}\Big{)} \tag{7}\]  
where \(I\) represents the total data volume in the records. It is worth noting that the size of \(I\) may differ among nodes due to the varying number of destination ports. Moreover, \(\beta_{i}\) signifies the weights assigned to different parameters, and these weights are allocated based on specific mechanisms tailored to different attack models. Next, we introduce a novel node-type mechanism to enhance the management of various nodes within the system. Currently, different Internet ISP (ISP) are responsible for handling distinct nodes. However, depending heavily on a single ISP can lead to critical security issues, including server outages, attacks on ISPs, and potential data breaches by ISPs [54]. In response to these challenges, we present a solution that focuses on diversifying the selection of multiple paths to reduce dependence on a single ISP. This approach aims to minimize security vulnerabilities associated with ISP-related issues. We design a metric Node Diversity presented as the ISP variance of an individual route to describe the node diversity of this route. In this context, the variable \(t\) is used to represent different ISP, and the Node Diversity in a single path is formulated as:  
\[var(a_{l})=\sum_{t\in T}\bigl{(}Count_{a_{l}}(t)-\overline{Count_{a_{l}}} \bigr{)}^{2} \tag{8}\]  
where \(t\) represents the node type currently under consideration. For all \(t\) belonging to the set \(T\), \(Count_{a_{l}}(t)\) is used to signify the number of nodes of this ISP along route \(a_{l}\). Here, \(a_{l}\) represents the \(l\)-th route within the currently selected route group \(A(\tau)\). \(T\) denotes the set encompassing all node types, expressed as follows:  
\[T=[ISP\,1,ISP\,2,ISP\,3] \tag{9}\]  
In the following, we use the notation \(\delta_{v_{i}}\) to represent the node type of any given node \(v_{i}\), and it is expressed as \(\delta_{v_{i}}=t\).