# Q-Learning Empowered Highly Dynamic and Latency-Aware Routing Algorithm for Flying Ad-Hoc Networks
## 3 Modeling Analysis
### Reinforcement Learning  
Reinforcement learning [29] is an unsupervised artificial intelligence approach that enables agents to observe and comprehend their environment without external guidance and subsequently determine optimal or near-optimal action selection to achieve optimal system performance. Figure 3 illustrates a simplified reinforcement learning process.  
If an agent's action in a given state results in a positive reward, the agent's inclination to adopt this action in the future will be strengthened. Conversely, if the action leads to a negative reward, the agent's inclination to adopt the action will be weakened. Since the agent does not have prior knowledge of which action is most beneficial to achieve the goal, it must actively test the environment. The environment will respond to the agent's actions and provide feedback. Based on this feedback, the agent modifies its action strategy to  
Figure 3: Reinforcement learning process.  
Figure 2: A simple FANET structure.  
adapt to the environment and then sends out probes to obtain new feedback, thus further optimizing its behavior to achieve the ultimate goal.  
The Q-learning algorithm is a reinforcement learning technique that enables an agent to identify the optimal path to a target node with the highest return value. This is achieved by periodically updating the state activity value (Q-value) at different states. The algorithm is an unsupervised active learning approach that does not require a specific system model and can be adapted to different environments through real-time interactions.  
Unlike other methods, Q-learning does not require the estimation of the environment model or the evaluation of intermediate costs. Instead, it directly optimizes an iteratively calculated Q-function. The Q-value is the result of long-term learning, which summarizes all the required information and stores it in a two-dimensional table in terms of state and action. When a decision is needed, the agent selects the action that can obtain the maximum benefit according to the Q-value, thus conducting a straightforward and efficient decision-making process. The core idea of the algorithm is to continuously update the Q-value. The state-action pair (\(s\), \(a\)) at time \(t\) is denoted by (\(s\), \(a\), \(t\)), where \(s\) represents the state and \(a\) represents the action taken in the state. The Q-value for a state-action pair (\(s\), \(a\)) is obtained by applying an action \(a\) in the state \(s\), where \(a\) and \(s\) are bounded within the range [0, 1]. Upon performing an action in the state, the agent transitions to a new state \(s^{\prime}\) and receives a cumulative reward value denoted by \(\max_{a^{\prime}}Q(s^{\prime},a^{\prime})\) for performing the best action at this time. The Q-learning algorithm comprises the following steps:  
Step 1: For each s and a, initialize \(Q(s,a)\) to 0;  
Step 2: Select an action \(a\) according to the Q table and execute it;  
Step 3: Get rewards and observe the new state \(s^{\prime}\);  
Step 4: Update \(Q(s,a)\) based on rewards and \(Q(s^{\prime},a^{\prime})\);  
Step 5: Set the new state \(s^{\prime}\) to the current state \(s\);  
Step 6: Return to Step 1 to continue execution.