# Integration of the 3D Environment for UAV Onboard Visual Object Tracking
## 3 Single Visual Object Tracking Pipeline for UAV
### Tracking Cycle and Occlusion Handling  
During initialization, the visual tracker learns an appearance model of the object based on the initial bounding box. At the same time, the position of the object in the image is projected onto the estimated ground plane of the 3D reconstruction, i.e., the scene space. To this end, we rely on the pinhole camera model for estimating the depth value of the object in the scene space. Supposing that the object does not leave the ground in the real world, we can assume that the object is bound to only move on the plane ground \((\mathbf{x}_{g},\mathbf{y}_{g})\). Let \(\mathbf{n}_{c}=(a_{c},b_{c},c_{c})\) describe the normal vector of the plane ground and \(\mathbf{t}_{t}^{c}=(x_{c}^{o},y_{c}^{o},z_{c}^{o})_{t}\) describe the position of the object on the plane--expressed in camera frame coordinates. In Equation (4), we describe the plane in standard form for a generic point on the plane. In Equation (5), we define \(x_{c}^{o}\) and \(y_{c}^{o}\), where \((x_{i}^{o},y_{i}^{o})_{t}\) are the image coordinates of the object inferred by the visual tracker and \(f\) is the focal length of the camera. By replacing \(x_{c}^{o}\) and \(y_{c}^{o}\) in Equation (4) and simplifying it, we obtain Equation (6), allowing us to infer the depth value for the object based on its image coordinates \((x_{i}^{o},y_{i}^{o})_{t}\). Now that the depth value \(z_{c}\) is estimated, we can determine the missing coordinates \(x_{c}^{o}\) and \(y_{c}^{o}\) of \(\mathbf{t}_{t}^{c}\) based on Equation (5). By applying a rigid body transformation \(\mathbf{H}_{w}^{c}\) from the camera to the world frame of reference, we obtain the position of the object in the scene space as \(\mathbf{t}_{t}^{w}=(x_{w}^{o},y_{w}^{o},z_{w}^{o})_{t}\). \[a_{c}x_{c}^{o}+b_{c}y_{c}^{o}+c_{c}z_{c}^{o}-d_{c}=0, \tag{4}\]  
\[x_{c}^{o}=\frac{x_{i}^{o}z_{c}^{o}}{f}\quad\text{and}\quad y_{c}^{o}=\frac{y_ {i}^{o}z_{c}^{o}}{f}, \tag{5}\]  
\[z_{c}^{o}=\frac{fdc}{a_{c}x_{i}^{o}+b_{c}y_{i}^{o}+c_{c}f}. \tag{6}\]  
The projected position \(\mathbf{t}_{t}^{w}\) for the first frame \(t=1\), expressed in 3D coordinates, is considered as the initial position for the state \(s_{-1}\). In the following frame, the _visual tracker_ component computes a bounding box delimiting the position and size of the object in the frame. Additionally, we extract the current search area and the similarity score map. Particles are generated uniformly onto the ground of the 3D reconstruction but are delimited on the projected surface of the search area. Particles are then weighted accordingly to the similarity score map of the _visual tracker_ component. Following is a resampling step, which shrinks the possible locations where the object might be located by regenerating new particles where previous high-weighted particles were located. In consequence, particles are mostly located around a high similarity response, giving us an estimated position of the object in 3D. The current 3D position \((x_{w},y_{w},z_{w})_{t}\) along with the previous 3D position of state \(s_{-1}\) are used to determine an initial velocity. As a result, an initial state \(s_{0}\) for the object is estimated with velocity and position. In the following frame, the particle filter can be used to predict the position of the object in the scene space as \((x_{w}^{p},y_{w}^{p},z_{w}^{p})_{t}\). To deal with the uncertainties in the transition model--i.e., the constant velocity model--Gaussian noise term is added along \(\mathbf{x}_{g}\) and \(\mathbf{y}_{g}\). After initialization, our tracking framework enters an online tracking cycle. Figure 4 presents the essential architecture of the framework during online tracking. On an incoming frame, the _visual tracker_ component defines a search area and produces a similarity score map along with estimating a bounding box for the object on frame \(t\). During the _3D Context_ computation step, we estimate the 3D position of the object in the scene space, distinguish the object from distractors, and recognize occlusions.