## 5 Experiments  
Given the complexity of drone control in three-dimensional space, we opted to conduct experimental tests on a two-dimensional plane. We utilized the experimental environment described in the literature [30], which consists of three marked square areas. For detailed information about the specific experimental setup, please visit the following website: [[https://github.com/marek-robak/Drone-2d-custom-gym-env-for-reinforcement-learning](https://github.com/marek-robak/Drone-2d-custom-gym-env-for-reinforcement-learning)]([https://github.com/marek-robak/Drone-2d-custom-gym-env-for-reinforcement-learning](https://github.com/marek-robak/Drone-2d-custom-gym-env-for-reinforcement-learning)) (accessed on 11 December 2023). The smallest square represents the initial space where the drone may appear, while the larger square defines the area where the target point may be located. Here, the agent needs to learn to navigate the drone to the target point. The largest square represents the boundaries of the drone's flyable space, and if the drone flies out of this range, the current round ends. The drone model utilized in our study is constructed from a rigid body that comprises three sections, two of which function as motors. These motors are capable of producing lift, as depicted by the elongated red lines, while the grey lines serve as a reference scale. At the initiation of each trial, the drone is propelled in an arbitrary direction, with the motors temporarily disabled for a predetermined sequence of time steps. This arrangement aims to forge a scenario in which the reinforcement learning agent is tasked with regaining control and stabilizing the drone during its descent. The portion of the trajectory delineated in red illustrates the phase of flight where the agent lacks control. Additionally, the simulation of the drone's environment is conducted at a temporal resolution of 60 frames per second, as illustrated in Figure 7.  
In this experiment, the hardware setup used was as follows:  
* CPU: Manufacturer: Mechrevo,Country: China,12 th Gen Intel(R) Core(TM) i7-12650H, with a clock speed of 2300 Mhz, with 10 cores and 16 logical processors. * GPU: Manufacturer: Mechrevo,Country: China,NVIDIA GeForce RTX 4050. For detailed configurations of the hyperparameters, refer to Table 2.  
In our study, we strategically simplified the drone's operational context by confining its movements to the xoz plane, thereby significantly mitigating the complexities associated with navigating a fully three-dimensional space. Within this two-dimensional framework, the drone utilizes its advanced control system to navigate from an initial point to a specified target location. This experimental setup enables a concentrated examination of the drone's control and navigational efficacy on a plane, alongside an assessment of its stability and task execution proficiency. This methodological choice not only facilitated a comprehensive analysis of the drone's control system efficiency but also established a foundational basis for future explorations into three-dimensional spatial navigation, yielding crucial data and insights. Figure 8 simulates the drone's situation in three-dimensional space:  
Figure 7: Two-dimensional experimental environment space. Furthermore, we have depicted our strategy through illustrations in Figures 9 and 10. Given the original eight-dimensional state space, we employed Principal Component Analysis (PCA) to compress it into a two-dimensional space for the sake of clarity in visualization. In the derived visual representation, the orientation of the arrows signifies the actions the model is inclined to undertake in specific states. A concentration of arrows pointing in a uniform direction within certain regions implies a degree of consistency in the model's action choices in those states. In contrast, a varied spread of arrow directions may indicate the states' inherent uncertainty or highlight the exploratory behavior of the model during its learning phase.