# Multi-Path Routing Algorithm Based on Deep Reinforcement Learning for SDN
## 3 System Model
### Attack Module  
The system aims to assess the vulnerability of a network over a series of time windows \(T\) in the presence of external attacks characterized by the strategy \(\pi_{att}\). Each node in the network, denoted as \(v_{i}\), is assigned a probability of being attacked, represented as  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Paper** & **Description** & **Learning Approach** & **Action** & **Performance Metrics** \\ \hline
[25] &
\begin{tabular}{c} DRSIR: A DQN-based solution for intelligent routing \\ in SDN based on path-state metrics. \\ \end{tabular} & DQN & Path Selection & Delay, Loss and Throughput \\ \hline
[26] & \
\begin{tabular}{c} A DRL-based method, which can maximize \\ numerous objectives to dynamically update the \\ routing strategy. \\ \end{tabular} & PPO & Next Hop &
\begin{tabular}{c} Average Delay, \\ Maximum Delay \\ \end{tabular} \\ \hline
[43] &
\begin{tabular}{c} RLBeep: A RL-based method to increase the lifetime \\ in wireless sensor networks. \\ \end{tabular} & Q-Learning & Next Hop & First node death time \\ \hline
[44] & \
\begin{tabular}{c} RLMR: A Q-Learning based method performs routing \\ for different flows, based on the real-time information \\ of network state and flow characteristics. \\ \end{tabular} & Q-Learning & Next Hop &
\begin{tabular}{c} Forwarding Efficiency, \\ Loss rate \\ \end{tabular} \\ \hline
[45] &
\begin{tabular}{c} NetworkAI: A network architecture using network \\ monitoring technologies and artificial intelligence for \\ generating control policies. \\ \end{tabular} & DQN & Next Hop & Delay \\ \hline
[46] & \
\begin{tabular}{c} RLRouting: a reinforcement learning routing \\ algorithm solving traffic engineering (TE) problem of \\ SDN in terms of throughput and delay. \\ \end{tabular} & Dueling DDQN & Next Hop &
\begin{tabular}{c} Utilization rate, Delay and \\ CPU Usage \\ \end{tabular} \\ \hline
[47] & \
\begin{tabular}{c} DRL-GS: A DRL-based solution to optimize topology \\ in SDN \\ \end{tabular} & A2C,PPO,GNN & Parameter of GNN &
\begin{tabular}{c} Entropy Loss, Training \\ Accuracy and \\ Testing Accuracy \\ \end{tabular} \\ \hline
[48] & DQSP: A DRL-based solution concerning QoS in \\ SDN-IoT & DDPG & Next Hop &
\begin{tabular}{c} Packet Delivery Ratio, Delay, \\ Probability of passing \\ attacked nodes \\ \end{tabular} \\ \hline
[49] & \
\begin{tabular}{c} An attack mitigation scheme based on DRL to \\ adaptively prevent the spread of attacks. \\ \end{tabular} & DQN &
\begin{tabular}{c} Takeover Decision of \\ the Switches \\ \end{tabular} & Request Arrival Rate \\ \hline \hline \end{tabular}
\end{table}
Table 1: Reinforcement Learning in SDN related research.  
\(p(v_{i})=\pi_{att}(v_{i})\). Within each time window \(\tau\), node \(v_{i}\) generates a set of send-receive records that, to some extent, indicate the node's susceptibility to attacks:  
\[Generator_{\pi_{att}}(\tau,v_{i})=Group(\tau,v_{i}) \tag{2}\]  
where \(Generator_{\pi_{att}}\) represents the mapping of the external environment generating attack records using the strategy \(\pi_{att}\). \(Group(\tau,v_{i})\) represents the records of node \(v_{i}\) at time \(\tau\), and \(Group(\tau)\) represents the record group of all nodes at time \(\tau\). When defining the time window \(\tau\) and using \(M(\tau)\) to represent the delay matrix, the changes in the delay matrix caused by \(Group(\tau,v_{i})\) are described by the mapping relationship \(Trans\) as follows:  
\[Trans_{Group(\tau)}(M(\tau-1),\tau)\to M(\tau) \tag{3}\]