## 3 Dynamic Routing Algorithm for Inter-Satellite Network Based on Q-Learning
### Dynamic Routing Algorithm Based on Q-Learning  
In this section, a Markov decision process is established for the path selection problem of the inter-satellite link. The arrival and departure of business flows are regarded as random processes, and different types of business flows have different statistical characteristics. We assume that business flow requests arrive independently. Furthermore, the probability of the business type follows a pre-specified Poisson distribution. The SDN controller must make a decision at each discrete time interval to accept or reject new service requests based on the current system state. In the case of acceptance, it is decided to allocate the best transmission path for the traffic flow. If declined, there is no need to allocate resources for it. The Markov decision process (MDP) provides a mathematical framework for modeling Q-learning systems, represented by a quadruple \((S,A,P,R)\), where \(S\) represents a finite state set, \(A\) a finite action set, \(P\) the state transition probability, and R a reward set. The relationship among state, action and reward function can be expressed by Equation (2f). The state transition function: \(S\times A\to P(s)\), where \(P(s)\) is the probability distribution for state \(s\), and the reward function returns a reward \(R(s,a)\) after a given action \(a\in A(s)\), where \(A(s)\) is the set of available actions for state \(s\). The learning task in MDP is to find the policy \(\pi\): maximizing the cumulative reward. To maximize the reward received across all interactions, the agent must choose each action according to a strategy that balances exploration (acquisition of knowledge) and exploitation (use of knowledge). Store the value function (or Q value) of each state-action pair in the MDP. \(Q(s,a)\) is the expected discounted reward when executing action \(a\) in state \(s\) following policy \(\pi\). \[S\times A\to R \tag{2f}\]  
In addition to agents/decision makers and dynamic systems, Q-learning also includes decision functions, value functions, long-term rewards, etc. [18]. The decision function refers to the policy that the agent will take, which maps from the perceived system state to the corresponding action, guiding the action of Q-learning. The exploration strategy we use in this chapter is \(\varepsilon-decreasing\), where the probability of choosing a random action (exploration) is \(\varepsilon\in[0,1]\) and choosing the best action (exploitation) is \(1-\varepsilon\). In this way, at the beginning, maintain a high exploration rate, and each subsequent episode \(\tau\in\Lambda\) will decrease according to \(\varepsilon_{\tau}=\sqrt{1-\left[\tau/(4\times\left|\Lambda\right|)\right]^{2}}\). The state-action value function characterizes the value of a state-action pair, indicating the difference between the current state and the stable state. The Q value function is updated as shown in Formula (2g). \[Q(s_{t},a_{t})=(1-\alpha)Q(s_{t},a_{t})+\alpha\Big{[}R(s_{t},a_{t})+\gamma max _{a}Q(s_{t+1},a_{t+1})\Big{]} \tag{2g}\]  
\(\alpha\in[0,1)\) is the learning factor of Q-learning, which represents the rate of the newly acquired training information covering the previous training, and \(R(s_{t},a_{t})\) is the reward at time t. \(\gamma\in[0,1]\) is the discount factor that determines the importance of future rewards. In Equation (2g), specifically, at time t, the agent performs action \(a_{t}\) on the current state \(s_{t}\) of the routing optimization model and then reaches state \(s_{t+1}\), and at the same time, the feedback loop will report the reward function \(R(s_{t},a_{t})\) to the agent, and update the action value function \(Q(s_{t},a_{t})\) and Q matrix accordingly.