# An Aerial Mixed-Reality Environment for First-Person-View Drone Flying
## Mixed Reality for Drones
### Flight in a Mixed Reality Environment  
To generate virtual objects dynamically using real objects, the locations of real objects must be determined based on the images obtained from the stereo camera. In this study, predefined objects were recognized by the transfer learning of a faster R-CNN, as described in Section 2.3. Transfer learning is a popular method in computer vision because it can build accurate models in a timesaving manner [44]. Figure 3: (**a–d**) First-person-view of a drone in mixed reality combining a real world with a virtual forest. Figure 2: A hexacopter with a stereo camera. Transfer learning refers to the use of a pre-trained model that was trained on a large benchmark dataset for a new problem. Due to the computational cost of training, it is typical to use models from the published literature (e.g., Visual Geometry Group, Inception, MobileNet). A comprehensive review of pre-trained models' performance using data from the ImageNet [45] challenge was presented by [PERSON] et al. [46]. For transfer learning in this study, the Google Inception V2 module [47] was used as a backbone network, which is the most typically used CNN since the release of TensorFlow. Subsequently, a faster R-CNN model pre-trained on the COCO dataset [48] was used. The data of the objects to be recognized were collected through images captured during the actual FPV drone flight. The parameters of the model were adjusted appropriately for this system using the captured images as learning data. As a result of learning, flags were identified with an average performance time of 300 ms and an average accuracy of 97% from real-time video frames. Here, the accuracy was calculated as a ratio of the number of correctly identified images to the total number of test images with flags acquired from an experimental flight lasting around 1 min. Virtual objects can be created interactively using physical objects. For example, virtual large rings can be generated next to drone racing flags, as shown in Figure 5, which can be passed through to practice drone flying. As another example, trees in a forest can be generated using water bottles for practicing obstacle dodging, as shown in Figure 6.  
Figure 4: (**a,b**) First-person-view of a drone in mixed reality combining a real world with a virtual city. Figure 5: (**a**) A detected flag, (**b**) A created virtual ring next to the flag. Figure 6: (**a**) Identified water bottles, (**b**) Created virtual trees using the water bottles. Target objects such as flags or bottles are identified by a faster R-CNN, and then the center point is calculated using the start and end points of the 2D coordinates of objects identified in the 2D image. The ZED camera calculates the distance between a specific pixel and the camera using triangulation [49]. After obtaining the distance \(z\) between the center point of the identified object and the camera, it is mapped to a 3D virtual space. The 3D virtual space is first defined by the viewpoint vector of the camera. The viewpoint of the camera is the \(z\)-axis, the width is the \(x\)-axis, and the height is the \(y\)-axis. The locations of virtual objects are mapped in the Unity3D engine in the following manner. The center point calculated for 3D mapping must be adjusted according to the size of the screen to be rendered to the user. The equation for the adjustment is as follows:  
\[(x^{\prime},\ y^{\prime})=((p(x,\ y))/h) \tag{1}\]  
where \((x,y)\) is the center of the detected object, \(p\) is the size of the rendered plane, and \(h\) is the size of the image obtained from the camera. Furthermore, the Unity3D engine calculates the virtual world coordinates by converting through a reverse projection from the current projection plane to the 3D space using \(x^{\prime}\) and \(y^{\prime}\) from Equation (1) and the depth value \(z\) from the ZED camera. Finally, the virtual objects are moved appropriately to the \(x\)- and \(y\)-axis directions such that the real and virtual objects do not overlap on the 3D coordinates of the converted target object.