# Ship Network Traffic Engineering Based on Reinforcement Learning
## 3 Experiment
### Complex Networks  
The Abilene dataset is commonly used in traffic engineering. The network structure of this dataset is 12 nodes and 30 paths. The network structure diagram, weight coefficient, and bandwidth of each path are shown in Figure 6 and Table 3.  
According to the weight of each edge, the shortest path between each node can be calculated, as shown in Table 4. \begin{table}
\begin{tabular}{c c} \hline \hline
**Source -> Destination** & **Shortest Path** \\ \hline \(0\) - \(>1\) & \(0,1\) \\ \(0\) - \(>2\) & \(0,1,5,2\) \\ \(0\) - \(>3\) & \(0,1,5,6,3\) \\ \(0\) - \(>4\) & \(0,1,4\) \\ \(0\) - \(>5\) & \(0,1,5\) \\ \(0\) - \(>6\) & \(0,1,5,6\) \\ \(\cdots\) & \(\cdots\) \\ \(11\) - \(>8\) & \(11,8\) \\ \(11\) - \(>9\) & \(11,8,2,5,6,3,9\) \\ \(11\) - \(>10\) & \(11,8,2,5,6,3,10\) \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Shortest path in Abilene dataset. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Link\_Index** & **Source** & **Destination** & **OSPF** & **Capacity (kbps)** \\ \hline \(0\) & \(0\) & \(1\) & \(1\) & 9,920,000 \\ \(1\) & \(1\) & \(0\) & \(1\) & 9,920,000 \\ \(2\) & \(1\) & \(4\) & \(1176\) & 9,920,000 \\ \(3\) & \(1\) & \(5\) & \(587\) & 2,480,000 \\ \(\cdots\) & \(\cdots\) & \(\cdots\) & \(\cdots\) & \(\cdots\) \\ \(26\) & \(10\) & \(3\) & \(2095\) & 9,920,000 \\ \(27\) & \(10\) & \(9\) & \(861\) & 9,920,000 \\ \(28\) & \(11\) & \(1\) & \(846\) & 9,920,000 \\ \(29\) & \(11\) & \(8\) & \(233\) & 9,920,000 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Abilene dataset weight coefficient and bandwidth. Figure 6: Abilene dataset network structure diagram. As mentioned in Section 3.1, transmitting all flows through the shortest path causes partial path congestion and reduces the network's overall performance. Therefore, we employed reinforcement learning algorithms to reroute flows in the network and improve its overall performance. First, we calculated the reachable path between each node, as shown in Table 5.  
The Abilene dataset collects the traffic matrix from the network path every 5 min, generating 288 pieces of data daily. Using datasets from 1 March 2004 to 4 March 2004, we simulated the transmission of 2016 pieces of data in the network. We calculated the utilization of each path and used average utilization as an evaluation indicator of the network's overall efficiency. For ease of presentation, only the experimental results of the three-node network are shown below. In the experiment, three nodes codenamed '1', '2', and '3' sent messages to each other. In reinforcement learning, there are six actions, '1 -> 2', '1 -> 3', '2 -> 1', '2 -> 3', '3 -> 1', '3 -> 2', that represent the packets to be rerouted. There are two paths between each node, as shown in Table 6. \begin{table}
\begin{tabular}{c c} \hline \hline
**Source -> Destination** & **Path** \\ \hline \(1\) -> 2 & '1’,’3’,'2’ \\ \(1\) -> 3 & '1’,’3’,’2’ \\ \(2\) -> 1 & '1’,’2’,'3’ \\ \(2\) -> 3 & '2’,’1’,'3’ \\ \(3\) -> 1 & '3’,’2’,’1’ \\ \(3\) -> 2 & '3’,’1’,’2’ \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Reachable paths between nodes. \begin{table}
\begin{tabular}{c c} \hline \hline
**Source -> Destination** & **Shortest Path** \\ \hline \(0\) -> 1 & '0’,’1’ \\  & '0’,’1’,’4’,’6’,’5’,’2’ \\  & '0’,’1’,’4’,’7’,’9’,’3’,’6’,’5’,’2’ \\
0 -> 2 & '0’,’1’,’4’,’7’,’9’,’10’,’3’,’6’,’5’  
Reinforcement learning has a total of [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1]...[1, 1, 1, 1, 1] 64 states using '0' and '1' to represent whether the messages between nodes are routed through the first or second path, respectively. The partial Q table after 600,000 iterations is shown in Table 7.  
As seen in the Q table, 63 states are covered from state [0, 0, 0, 0, 0, 0] to state [1, 1, 1, 1, 1, 1], indicating that training was relatively sufficient. The occupancy rate of the edge with the highest occupancy rate in the network is 80.57% in the state [0, 0, 0, 0, 0]. In the Q table, the 3 -> 2 packets rerouted from the '1' channel are transferred from the '2' channel.