# An Aerial Mixed-Reality Environment for First-Person-View Drone Flying
## 1 Introduction  
As cameras mounted on drones can easily capture photographs of places that are inaccessible to people, drones have been recently applied in various fields, such as aerial photography [1], search and rescue [2], disaster management [3], and entertainment [4]. Commercial drones are controlled in the third-person or first-person view (FPV), i.e., the view of an unmanned aircraft. In the third-person view, a pilot can view the entire drone in relation to its surroundings during a flight. However, it is difficult to estimate the position and direction of a drone relative to nearby objects to avoid obstacles when flying a drone from a third-person view because the range of human stereoscopic vision is limited to approximately 90 m, and the effective range is considerably smaller. More advanced drone control systems allow pilots to visualize that they are onboard their drones. FPV drones, such as FatShark [5] and Skyzone [6], are remotely controlled using an onboard camera and FPV goggles. These drones send monocular video feeds via analog signals from the camera to the pilot's goggles with a limited field of view of \(35^{\circ}\)-\(43^{\circ}\) and resolutions of a video graphics array (\(640\times 480\)) or wide video graphics array (\(800\times 600\)). Professional drone pilots can safely fly an FPV drone by estimating distances between the drone and obstacles depending on their experience; however, it is not easy for novice pilots to fly an FPV drone while avoiding collisions with obstacles due to the low transmission rate, poor reception, fast speed, and obstacles. According to a remote robot control experiment [7], FPV control using a stereo camera yields fewer errors than using mono images. Hence, the first-person experience could be improved by utilizing more advanced techniques, including stereo views and wide fields of views. Moreover, safer and more exciting environments could be provided to prevent drone crashes and reduce collisions with obstacles, thereby protecting drones and their surroundings. Our research was motivated by the need to providesafe, inexpensive, and fun environments for flying a real drone, rather than using a virtual reality (VR) drone flight simulator. To achieve this purpose, an FPV drone can be combined with immersive mixed-reality (MR) technology, where physical and virtual objects co-exist and interact in real time. Stimulating drone racing or training tracks can be created by mixing free-form virtual objects with simple physical obstacles, which may be difficult or expensive to make in the real world. A novel aerial MR environment based on stereo vision for flight training is proposed in this study. We utilized a stereo camera to capture real-world scenes during flight to improve depth perception, and introduced an MR environment that combines virtual three-dimensional (3D) objects with a stereo video stream from the drone camera in real time to control the difficulty level depending on a pilot's skill. The proposed aerial MR environment enables more accurate depth perception, which is important for determining the direction and orientation of the drone when passing through an obstacle, and facilitates engaging interactions with physical and virtual objects. The main contributions of this study are as follows. * We propose a new aerial MR environment using an FPV drone with a stereo camera to provide users with a safe and engaging flight environment for flight training. The stereo camera improves the accuracy of the pilot's distance estimation from binocular disparities, and a head-mounted display (HMD) with a wide field of view increases immersion. * Different MR environments for drone flying can be created easily by placing several drone flags in the real world. The flags are recognized by machine learning techniques and virtual objects are synthesized with a real scene when detecting the flags during flight. * The proposed system is evaluated on the basis of user studies to compare the experiences of using our FPV-MR drone system with a VR drone flight simulator.