# Ship Network Traffic Engineering Based on Reinforcement Learning
## 2 Materials and Methods
### Q-Learning  
Q-learning [17] is a reinforcement learning algorithm used to train agents in optimal environmental interaction strategies. Reinforcement learning is a branch of machine learning whose goal is to teach agents from trial and error and maximize accumulated rewards through environmental interaction [18].  
Q-learning [18] was proposed by computer scientist [PERSON] in 1989. Its core idea is to represent the long-term cumulative reward of each state and action pair by learning a value function (Q-value function). The Q-value function is a measure of how good or bad it is to perform an action in a given state. The basic update rules of the algorithm are as follows:  
\[Q(s,a)\leftarrow(1-\alpha)\cdot Q(s,a)+\alpha\cdot\left(r+\gamma\cdot\max_{a^{ \prime}}Q(s^{\prime},a^{\prime})\right)\]  
where:  
* \(Q(s,a)\) is the Q-value of performing action \(a\) in state \(s\).
* \(\alpha\) (the learning rate) controls the extent to which newly acquired information overrides the old Q-value. A low value of \(\alpha\) makes the agent more conservative and less likely to change its Q-values significantly based on new experiences, while a high value of \(\alpha\) makes the agent more prone to quick changes in its Q-values.
* \(\gamma\) (the discount factor) determines the importance of future rewards. It represents the agent's preference for immediate rewards over future rewards. A value of \(\gamma\) close to 1 implies that future rewards are highly valued, while a value close to 0 implies that future rewards are discounted.  
* \(s^{\prime}\) is the new state the agent transitions to after performing action \(a\).
* \(\max_{a^{\prime}}Q(s^{\prime},a^{\prime})\) represents the Q value corresponding to selecting the optimal action \(a^{\prime}\) under the new state \(s^{\prime}\).
* The term \(R+\gamma\cdot\max_{a^{\prime}}Q(s^{\prime},a^{\prime})\) represents the estimated total reward the agent expects to receive from the current state \(s\) after taking action \(a\) and transitioning to the next state \(s^{\prime}\), considering the immediate reward \(R\) received and the maximum expected future reward \(\gamma\cdot\max_{a^{\prime}}Q(s^{\prime},a^{\prime})\) achievable from state \(s^{\prime}\). The learning rate \(\alpha\) controls how much the new estimate of the Q-value (the right-hand side of the equation) updates the existing Q-value. It balances the trade-off between exploration and exploitation by determining the extent to which the agent relies on new information versus its previous knowledge. The discount factor \(\gamma\) discounts the value of future rewards, preventing the sum of rewards from becoming infinite in environments with infinite horizons or ensuring convergence in environments with finite horizons.  
Q-learning trains the agent to learn optimal environmental interaction strategies and update the Q-value according to the reward signal, in other words, choose the action that maximizes the cumulative reward in each state. This strategy makes Q-learning excellent at solving a variety of reinforcement learning problems, including games, control problems, etc.