# Pose Tracking and Object Reconstruction Based on Occlusion Relationships in Complex Environments
###### Abstract
EPro-PnP [11] achieves end-to-end training of camera poses by transforming the perspective-n-point (PnP) problem into a probability density prediction task. By learning the 2D-3D correlation based on ground truth poses, this method can stably and flexibly train pose estimation networks, surpassing the performance of traditional methods. Additionally, the concept of EPro-PnP can be applied to other geometric optimization problems and has certain theoretical generalizations. The latest research, BundleSDF [12], links the pose graph optimization process with the geometric representation of objects using a key-frame pool, making it highly adaptable to significant motion, partial occlusion, and textureless surfaces. Furthermore, with online learning of the neural object field, BundleSDF enables pose tracking for arbitrary objects. Although current pose tracking achieves considerable accuracy in scenarios with no occlusion or occasional small occlusions, it still faces significant challenges in situations where the target interacts frequently with the environment (such as hands, robotic arms, etc.) or in complex scenes with long-term random occlusions. Existing pose tracking solutions [13; 14; 15; 16], especially those that rely on modeling [17], face severe challenges in such scenarios. These challenges mainly manifest in the following aspects:  
1. **Difficulties in Segmentation. ** For tracking unknown objects, accurate segmentation of the target is crucial. The segmentation difficulty increases significantly when occluding objects are moving relative to the target object. 2. **Boundary Ambiguity. ** When occluding objects appear in front of the target, even if the segmentation module can accurately separate them in a timely manner, the mask recorded at the boundary between the two objects will only represent the boundary of the occluding object. If not properly handled, this situation may introduce ambiguity. 3. **Feature Matching Problems. ** Traditional tracking methods typically rely on matching relationships between key points in adjacent frames to estimate relative displacement. However, this approach heavily relies on the texture features of the target object's surface. When an occlusion occurs in the scene, some selected feature points from the previous frame may be lost due to occlusion. Our method is an improved version based on BundleSDF [12], and the core focus of this method lies in deep learning-based object pose tracking, aimed at enhancing the performance of object pose estimation in complex environments. We found that frequent occlusions can severely affect pose tracking. The baseline method uses segmentation masks to separate the target object from the environment to eliminate environmental interference, and in subsequent geometric shape analysis, the segmentation mask is assumed to represent the geometric boundary of the object in the current view. However, the occurrence of foreground occlusion may mislead this process, leading to tracking failure. Therefore, addressing the challenge of effectively tracking targets in complex environments and dealing with frequent occlusions, our research introduces the concept of dual-layer masks and designs a target object pose-tracking algorithm based on the occlusion relationship between the foreground and the target object. Firstly, the target video segmentation network is applied to segment the target object area in the current frame, then, different from the baseline method, the depth relationship is analyzed to calculate the dual-layer mask of the current frame, effectively detecting foreground occluding objects while segmenting the target object. Secondly, a feature-matching network is used to roughly estimate the pose and perform pose graph optimization with historical frames. During this process, like BundleSDF, a neural object field for the object is trained using multi-view images, recording the geometric appearance of the target to assist the subsequent optimization process.