# Tracking and Registration Technology Based on Panoramic Cameras
###### Abstract  
Augmented reality (AR) has become a research focus in computer vision and graphics, with growing applications driven by advances in artificial intelligence and the emergence of the metaverse. Panoramic cameras offer new opportunities for AR due to their wide field of view but also pose significant challenges for camera pose estimation because of severe distortion and complex scene textures. To address these issues, this paper proposes a lightweight, unsupervised deep learning model for panoramic camera pose estimation. The model consists of a depth estimation sub-network and a pose estimation sub-network, both optimized for efficiency using network compression, multi-scale rectangular convolutions, and dilated convolutions. A learnable occlusion mask is incorporated into the pose network to mitigate errors caused by complex relative motion. Furthermore, a panoramic view reconstruction model is constructed to obtain effective supervisory signals from the predicted depth, pose information, and corresponding panoramic images and is trained using a designed spherical photometric consistency loss. The experimental results demonstrate that the proposed method achieves competitive accuracy while maintaining high computational efficiency, making it well-suited for real-time AR applications with panoramic input. augmented reality; panoramic vision; tracking and registration 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vol., pp. 2020-2021, 2020,  
algorithms to simultaneously optimize the camera pose and 3D scene map to achieve tracking and registration, but the 3D point cloud maps usually obtained are sparse. Borrowing the idea of the direct method in traditional visual SLAM and combining it with deep learning techniques for camera pose estimation, end-to-end learning from the input images to the depth of the dense scene and the accurate camera pose can be realized. Although supervised learning methods can achieve higher accuracy in camera pose estimation [14; 15], they require many data with high-quality pose labels for training, and large-scale labeled data in this regard are currently relatively scarce. In order to circumvent the limitation of the lack of data labels, researchers have generally turned to unsupervised deep learning methods to learn scene depth and camera pose information directly from unlabeled image data [16; 17]. To this end, some scholars have attempted to directly apply the camera pose estimation models designed for perspective images to cubic panoramas [18] or equirectangular panoramas [19; 20], but these methods do not adequately take into account the discontinuity of cubic panoramas and the distortion of equirectangular panoramas, which limits the accuracy of the pose estimation. Meanwhile, another class of pose estimation methods is used to assist in acquiring depth information by using the pose estimation information as a supervisory signal for depth estimation [21; 22]. Although these methods consider the distortion problem of panoramic images, the model complexity is high, and they still lack an in-depth analysis of pose estimation. In summary, our contributions are as follows:  
* We propose an unsupervised deep learning-based pose estimation model for panoramic cameras, aiming to address the dependence of model training on large-scale panoramic pose-labeled data and to quickly and accurately estimate the pose information of panoramic cameras. * To reduce the complexity of the model and improve the computational efficiency, we design a lightweight depth estimation sub-network and a pose estimation sub-network. * To improve the accuracy of the pose estimation, especially to deal with the effect of relative motion, we introduce a learnable occlusion mask module in the pose estimation sub-network.