# Automated Enemy Avoidance of Unmanned Aerial Vehicles Based on Reinforcement Learning
## 6 Implementation and Results
### Learning with the Sarsa Based Framework  
First, we use the newly constructed framework based on the Sarsa method and the CMAC function approximator to learn the policy for the enemy avoidance problem in the HITL simulation environment.  
As in reinforcement learning, the goal is to maximize the expected accumulate rewards, thus the action-value function \(Q\) is an effective metric to measure the performance of the policy learning.  
Considering the mission of the decision making UAV, it has to reach the goal point \(G_{f}\) successfully as soon as possible. On the other hand, the number of steps that an episode lasts can partially indicates how long it takes the decision making UAV to reach the goal point \(G_{f}\). Therefore, the number of steps for an episode can be used as a metric to measure the performance to some extend.  
However, there are several ending reasons. It can take very few steps to end the episode if the decision making UAV collides with or attacks the enemy UAV at a very early stage of an episode. Therefore, the number of steps alone is not enough to measure the performance of the learning in the enemy avoidance problem. To this end, the numbers of episodes that ending for different reasons are calculated, so as to measure more accurately how the learning changes the episode ending reasons.  
The experiment for learning the policy with the proposed framework has been run for 14,362 episodes. The related parameter settings are: \(\epsilon=0.01\), \(\alpha=0.1\), \(\gamma=0\), and the number of tiles laid for each variable is \(n=32\). The results of the above three metrics are presented as follows.  
Figure 7 illustrates how the \(Q\) -value changes with the increase of the episodes during the policy learning process. The figure is drawn with data filtered by a window of 1000 episodes. From Figure 7, it can be seen that the \(Q\)-value increases as the number of episodes increases, and becomes stable after around 10,000 episodes.  
Figure 6: The simulation process.  
Figure 8 shows how the number of steps for each episode changes as the number of policy learning episodes increases. The figure is also drawn with data filtered by a window of 1000 episodes. In Figure 8, the number of steps fluctuates very much at the beginning, but converges to a relatively small number as the number of episodes increases, and it comes to a plateau after around 10,000 episodes.  
Figure 9 presents how the accumulated numbers of different ending reasons change during the policy learning process. From Figure 9, it can be see that the episodes are mostly ended for collision with the enemy UAV at the initial period of the learning process. After about 1500 episodes, the number of episodes ending for reaching the goal begins to increase rapidly, which surpasses those of other ending reasons very soon. The number of episodes ending for successfully attacking the enemy UAV has a rise from about the 4000 th episode to about the 8000 th episode, and surpasses the number of episodes ending for collision with the enemy UAV during this period. Except for ending for reaching the goal point, the numbers of other ending reasons all stop to increase after about 9000 episodes. These results mean that the learning agent gradually learns to avoid the enemy UAV and attack the enemy UAV, and then it learns to limit the usage of the attacking action, and finally it has learned a very good policy to reach the goal point.  
Figure 8: The number of steps for each episode in the policy learning.  
Figure 7: The \(Q\)-value for each episode in the policy learning.