# Ship Network Traffic Engineering Based on Reinforcement Learning
## 3 Experiment
### Importance of Instructions  
Control and command data play a pivotal role in remotely managing vital vessel systems, including propulsion, navigation, and safety mechanisms. Given the critical nature of these operations, certain instructions hold greater significance than others. To ensure swift response times and precise control over vessel systems, control and command data typically demand low-latency transmission. Although individual control commands \begin{table}
\begin{tabular}{c c c} \hline
**State** & **Left** & **Right** \\ \hline
0, 0, 0 & 4.03361 & 7.15169 \\
1, 0, 1 & 5.72410 & 4.65601 \\
2, 1, 0 & 4.38533 & 5.24900 \\
3, 0, 1 & 3.61000 & 3.03399 \\
4, 1, 0 & 2.53626 & 2.90000 \\
5, 0, 1 & 1.00000 & 0.80855 \\ Terminal & 0.00000 & 0.00000 \\
1, 1, 0 & 0.27307 & 4.47023 \\
2, 0, 1 & 3.72987 & 0.14271 \\
3, 1, 0 & 0.32789 & 3.81745 \\
4, 0, 1 & 2.53300 & 0.48155 \\
5, 1, 0 & 0.00000 & 1.94437 \\ \hline \end{tabular}
\end{table} 
Table 1: Traffic balancing Q table. Figure 5: Traffic balancing and instruction priority experimental results. may be compact, the need for frequent transmission and adherence to low-latency standards necessitates allocating moderate to high bandwidth resources. Incorporating importance into the reward function involves assigning different weights or coefficients to rewards based on the importance level of instructions. Trade-off with network load balancing penalizing actions that lead to network congestion or imbalances in traffic distribution [23]. In Q-learning, Q(s1, a2) contains a maximum estimate of Q(s2). If the agent selects action a2 in state s1 and reaches state s2, no matter which action is selected in state s2, it will obtain the maximum estimated value in state s2. This estimated value is calculated from the maximum reward in state s2. In other words, only the reward in the desired final state must be set to 1, while the rewards in other states are set to 0. After 600 iterations, the reward in the final state will be continuously transferred with each calculation of Q(s, a), thereby updating the Q table. We used the above principles to design a new Q-learning algorithm. In this algorithm, we assume that the third instruction is more important and must be transmitted from path 2. To balance the flow and ensure that high-importance instructions are transmitted preferentially, we set the reward of the final state [2, 1, 2, 1, 2, 1] to 1 and the rewards of the other states to 0. Among them is s_n is 2, indicating that the nth instruction was transferred through channel 2, and s_n is 1. In other words, the nth instruction is transferred through channel 1 and s_n is 0, suggesting that the nth instruction was not transferred. The Q-value calculated after 600 iterations is shown in Table 2.  
When the agent is in state [2, 1, 2, 1, 2, 0] and the reward [2, 1, 2, 1, 2, 1] is 1, the Q-target with the left action is 1. Therefore, the sixth instruction is transmitted through path 1 and reaches the target state [2, 1, 2, 1, 2, 1]. If the agent chooses the right action in the state [2, 1, 2, 1, 0, 0], it will receive a maximum estimated value of 1 [2, 1, 2, 1, 2, 0]. If the agent chooses the left action, it will receive a maximum estimated value of 0 [2, 1, 2, 1, 1, 0]. Therefore, the calculated Q-target of the right path is 0.88270, which is higher than the left path at 0.64075. When the fifth instruction passed through path 2, it reached state [2, 1, 2, 1, 2, 0]. Since the third instruction has higher priority than the other instructions, its routing weight coefficient was set to 2, whereas the other instructions were set to 1. As mentioned in Section 3.1, the transmission times of the right and left channels were 1 ms and 2 ms, respectively. As shown in Figure 5, the maximum transmission consumption is in the state [1, 2, 1, 2, 1, 2]: 11. The state [2, 1, 2, 1, 2, 1] is the transmission consumption of reinforcement learning: 10, which is reduced by 9.1%.