# Multi-Path Routing Algorithm Based on Deep Reinforcement Learning for SDN
## 4 Trust-Based Proximal Policy Optimization (TBPPO)
### Enhanced PPO Module  
In traditional policy gradient algorithms, policy weights are typically updated by calculating the gradient of the objective function and applying it with a step size. However, this update process may encounter problems like overshooting or undershooting. To address these issues, we adopt the PPO algorithm. PPO is a policy gradient method in reinforcement learning, which enhances the policy by optimizing a surrogate objective function using stochastic gradients obtained by sampling the data from interactions with the environment. It allows for multiple small-batch updates, as opposed to a single gradient update for each data sample. The specific PPO variant employed in this research is the PPO-clip algorithm, which relies on a clipping mechanism. To prevent the importance sampling function from exceeding the predefined upper or lower bounds, a truncation function denoted as \(J_{PPO}^{\theta^{k}}(\theta)\) is applied. This function automatically limits the importance sampling values when they go beyond the specified upper or lower limits. This can be expressed as Equation:  
\[\small \begin{split} J_{PPO}^{\theta^{k}}(\theta)\approx\sum_{(s_{\tau},a _{\tau})}\min\biggl{(}\frac{p_{\theta}(a_{\tau}\mid s_{\tau})}{p_{\theta^{k}}(a _{\tau}\mid s_{\tau})}A^{\theta^{k}}(s_{\tau},a_{\tau}),\ clip\biggl{(}\frac{p_{ \theta}(a_{\tau}\mid s_{\tau})}{p_{\theta^{k}}(a_{\tau}\mid s_{\tau})},1- \varepsilon,1+\varepsilon\biggr{)}A^{\theta^{k}}(s_{\tau},a_{\tau})\biggr{)} \end{split}  \tag{19}\]  
Here, \(J_{PPO}^{\theta^{k}}(\theta)\) is used to assess the expected cumulative reward (performance) of the policy, \(\theta\) is the current policy's parameter, and \(\theta^{k}\) is the policy parameters at a previous time step or iteration step k. \(s_{\tau}\) represent S(\(\tau\)). \(a_{\tau}\) represents the action taken at time step \(\tau\), and \(a_{\tau}\in A(\tau)\). \(clip\biggl{(}\frac{p_{\theta}(a_{\tau}|s_{\tau})}{p_{\theta^{k}}(a_{\tau}|s_{ \tau})},1-\varepsilon,1+\varepsilon\biggr{)}\) is a clipping function used to limit the ratio between \(1-\varepsilon\) and \(1+\varepsilon\), where \(\varepsilon\) is a small positive value usually employed to ensure that policy updates are bounded. The function \(A^{\theta^{k}}\) represents the function, which provides an estimate of the advantage when taking action \(a_{\tau}\) with the parameters set \(\theta^{k}\):  
\[\small \begin{split}\hat{A}_{t}^{GAE(\gamma,A)}=\sum_{l=0}^{\infty }\Bigl{(}\gamma\lambda\Bigr{)}^{l}\delta_{t+l}^{V}=\delta_{t}^{V}+(\gamma \lambda)\delta_{t+1}^{V}+(\gamma\lambda)^{2}\delta_{t+2}^{V}+\cdots+(\gamma \lambda)^{T-t+1}\delta_{T+1}\end{split}  \tag{20}\]  
Here, \(\gamma\) is the discount factor, \(\lambda\)_is_ the GAE parameter, and \(\delta_{t}^{V}\) is the temporal difference function:  
\[\delta_{t}^{V}=r_{t}+\gamma V_{\omega}(s_{t+1})-V_{\omega}(s_{t}) \tag{21}\]  
We utilize its gradient as the loss function for parameter \(\theta\):  
\[\bigtriangledown J^{\theta^{\prime}}(\theta)=E_{\pi_{\theta^{\prime}}} \biggl{[}\frac{\pi_{\theta}(s,a)}{\pi_{\theta^{\prime}}(s,a)}R(s,a)\bigtriangledown log \pi_{\theta^{\prime}}(s,a)\biggr{]} \tag{22}\]  
In our experiments, we observed that fully connected layers exhibit relatively weak fitting capabilities for the relationship between states and actions and they can often lead to the problem of gradient explosion. Therefore, we made some improvements to the PPO algorithm. We introduced a learning rate decay technique, which can enhance the stability of training in the later stages and improve the training effectiveness. Here, we employed linear learning rate decay, where the Actor Network's learning rate decreases linearly from an initial value of \(1\times 10^{-4}\) to 0 as the training steps progress. This is formulated as:  
\[\small \begin{split}\alpha_{new}=\alpha_{initial}\times(\text{ decay factor})^{\frac{\text{iteration}}{\text{decay step}}}.\end{split}  \tag{23}\]  
where \(\alpha\) presents the parameter operated.