# Ship Network Traffic Engineering Based on Reinforcement Learning
## 3 Experiment  
### Traffic Balancing  
Assume that there are two paths, 'left' and 'right', with the 'right' path being shorter than the 'left' path, and six instructions packaged into data packets, as shown in Figure 4. Firstly, for a task, only the left and right pathways are available for selection, forming the action space of reinforcement learning. Although individual actions are very easy to determine, the continuous selection process of different tasks forms an exponential optimization space. In the traditional OSPF algorithm, all six instructions select the right path, which blocks it and reduces the instructions' QoS transmission efficiency. As mentioned in Section 2, reinforcement learning requires an environment, state, action, and reward. In traffic engineering, we can obtain the instruction id (0-5) and detect the busyness of the 'left' and 'right' paths. The states are combined into a state of reinforcement learning, as shown in the first column of the table, where '0' means the channel is idle and '1' means it is busy. When receiving an instruction, we must select an action from the left and right paths as the reinforcement learning action. We designed the environment and rewards as follows: When the agent chooses a left (or right) action, we judge the left (or right) path in the current state. If the left (or right) channel is idle, a reward of '1' (or '2') is assigned because the right channel is shorter than the left. To achieve higher QoS, we generally hoped the instruction would pass through the right channel as much as possible without blocking. If the left (or right) path is busy, a reward of '0' is assigned. Table 1 shows the Q table after 600 iterations. When the first instruction (id is '0') is obtained, the Q-target value for selecting the left path is 4.03361, which is lower than the right path at 7.15169. The Q-learning algorithm will select the right path with greater probability. Since the right channel is idle, the first instruction is transmitted through the right channel. The agent receives the second instruction (id is '1') but the right channel is occupied. Therefore, the agent reaches the status of the second row in the table. In the same way, when the second instruction is received (id is '1'), the Q-target value for selecting the left path is 5.72410, which is higher than the right path at 4.65601. The Q-learning algorithm will select the left path with greater probability. Since the left channel is idle, the second instruction will be transmitted through the left channel. When the agent receives the third instruction (id is '2'), the left channel is occupied. At the same time, the right channel completes the first instruction and returns to the idle state. The agent then reaches the status of the second row in the table. The agent only reaches the 'terminal state' when the six instructions are transmitted. Figure 4: Schematic diagram of traffic balancing. Due to the reward function settings in the Q table, the agent will prioritize the right channel when the left and right channels are both idle. Similarly, when the left (or right) channel is busy, the right (or left) channel will be selected. As shown in Figure 5, assuming that the transmission time of each instruction is 1 ms, the OSPF algorithm requires 6 ms to complete these instructions. By contrast, the reinforcement learning algorithm requires only 3 ms, indicating a 50% improvement in QoS compared to the OSPF algorithm. For more complex network topologies, defining the action space becomes challenging due to the increased number of decision points and potential actions at each point. In more complex scenarios, actions might be continuous variables representing parameters of a policy or control function. For instance, the agent could adjust the allocation of network resources along different paths in a continuous manner, such as by dynamically adjusting bandwidth allocation. However, this Q-learning algorithm only manages traffic balancing. Some ship system instructions are more important than others.