# Integration of the 3D Environment for UAV Onboard Visual Object Tracking
## 3 Single Visual Object Tracking Pipeline for UAV
### Tracking Cycle and Occlusion Handling
The new 3D position of the object \((x_{w},y_{w},z_{w})_{t}\) expressed in the scene space is projected back onto the frame \(t\), corresponding to the new estimated position \((x_{t}^{n},y_{i}^{n})_{t}\) of the object in image space. Figure 9 displays the different building blocks of the _3D Context_ component, which contains the particle filter that estimates the object state, i.e., the 3D position and 3D velocity in the scene space. In the first step, the particle filter predicts particles on an incoming frame. The predicted particles areclustered in the image space. We then identify the cluster representing the object based on how close each cluster is to the previously estimated position of the object in the scene space. The _occlusion identifier_ component identifies occluded particles composing the object cluster through the depth map approximation. To this end, we compare the depth value of the predicted particles from the perspective of the UAV and the depth value from the depth map approximation. We identify particles hidden by structures if there is a discrepancy between the predicted depth and the observed depth. We considered that if 50% of the particles from the object cluster are occluded, then the object is also occluded. Similarly, the object is automatically considered as hidden if the similarity score map is flat and widely spread over the search area. Should the object be identified as occluded, the tracking framework would rely on the predictions provided by the particle filter \((x_{w}^{p},y_{w}^{p},z_{w}^{p})_{t}\). To identify the reappearance of the object, a high similarity score close to the expected position is required. When more than 50% of the predicted particles are not labeled as occluded, we consider that the object is potentially visible again. A specific threshold for the similarity score map is set to consider the object as being truly visible again. A similarity score greater-than or equal-to this threshold must be reached to consider updating the particle filter with the observation, i.e., the similarity score map. If distractors are present when the object reappears, then the group of particles closest to the prediction is considered to represent the object. When the object is visible, a second step is to update the belief of the particle filter with the observation, i.e., similarity score map. However, before updating a small percentage, i.e., 10%, of particles are uniformly redistributed across the projected search area on the ground. This redistribution ensures that we maintain a multimodal distribution. Without redistributing a portion of the particles, particles would clump around the object and only a small portion of the similarity score map would be considered for updating the weights. The weights of the particles are updated by projecting their 3D position in the image space, allowing us to weigh them accordingly to their 2D location in the similarity score map. We then resample particles through stratified resampling [61]. By using a particle filter, we can model multimodal tracking, preventing instantaneous switching from object to distractors as illustrated in Figure 10.  
Figure 9: Detailed view of the architecture with the _Context 3D_ component. The prediction of the particle filter and the depth map approximation are used to identify occluded particles. Particles are clustered depending on their position in the image to identify potential groups, such as the object and distractors. In case the object is considered as occluded, the 3D coordinate of the prediction provided by the particle filter \((x_{w}^{p},y_{w}^{p},z_{w}^{p})_{t}\) is used as the presumed position of the object in the scene space and is reprojected in the image as \((x_{i}^{a},y_{i}^{a})_{t}\). In case the object is not identified as occluded, an update and resampling are performed on the particles before clustering them for a second time. Following is the determination of the cluster representing the object.