## 1 Smart Decision-Making Model 1
* **For the end of each epoch:** At the end of each era, either because the agent has reached the target, because a maximum number of iterations have been given or because there is a situation contemplated for this to happen, a reward is established that varies depending on whether the target has been reached or not and, depending on the distance to it. So, if the destination is reached, a reward is generated that follows Equation (1), while if the target is not reached the generated reward follows Equation (2). This rewards you, whether the target has been achieved or not, proximity to the destination location. This solution allows the agent to have an understanding of how good this sequence of actions has been with respect to the previous ones, even if the objective is not achieved, since, by entering the parameter of the distance to the objective, even if the training is not satisfactory at that time, the agent can know if it is better than other sequences where the path point in question has not been reached and, in this way, improve their knowledge and learning. \[\gamma=200-0.1\cdot d\] (1) \[\gamma=-200-0.1\cdot d,\] (2) where \(\gamma\) corresponds to the reward value and \(d\) to the distance to the target. To improve learning and accelerate the convergence of training, several cases have been established for which the time is over, the environment is restarted and started again. These cases are determined by the observation space and the interpretation of these by the algorithm. Thus, the time is terminated if the UAV has collided with an obstacle or is in a situation incompatible with the flight stability of the system. To do this, the value of the sensor is first checked to sound on board, so that if that value is below 0.5 m it is assumed that the aircraft will collide with an obstacle of the environment. Secondly, pitch and roll angles are checked and, if they are above 90\({}^{\circ}\), the UAV is considered to have reached a position incompatible with the stability and flight of the system. Finally, to speed up the learning process, during training a work space is established that delimits an area through which the agent can be moved, ending the epoch, and starting a new one in the event that an action leads the UAV to a location outside the bounded area. As shown in Figure 15, as the training progresses, it can be seen how the learning of the network improves and the decision-making model generated approaches a policy that manages to maximize the value returned, showing that the structure of DQN used in this model is satisfactory. Figure 15 shows the total reward obtained at the end of each training season, and it shows the essence of DRL algorithms based on trial-error learning. In the early days, when the scanning rate is higher, the agent establishes sequences of actions that are not satisfactory by accumulating, at the end of each era, a negative total reward. As training progresses and, the agent first learns what are the best actions to take and, secondly, the exploration rate is reduced to prioritize the exploitation of the best actions, it can be observed how the reward accumulated at the end of each era is being more positive, increasing with training and achieving some stability around the maximum reward value at the end of the training. This analysis is completed with Figure 16, which shows the total reward accumulated as training progresses. In it, you can see how at the beginning of the training, the lack of knowledge and learning and, the configuration of the training to perform a high exploration at the beginning in search of the sequence of actions that maximizes the returned value causes the accumulated reward to decay towards negative values in the first half of the training. As the agent gains learning, the returned value improves  
Figure 15: Accumulated reward per training epoch for decision making model 1.  
and increases, and the abandonment of exploration to the detriment of exploration causes the slope of the curve to soften the accumulated reward.