# Tracking and Registration Technology Based on Panoramic Cameras
## 2 Related Work  
Tracking and registration are the primary technologies for implementing augmented reality (AR) systems and serve as essential prerequisites for the integration of virtual and real-world scenes. This technology enables the overlay of virtual objects onto real environments by continuously tracking the position and orientation (pose) of the camera within the real-world coordinate system and using the obtained pose information to accurately register virtual elements into the real scene. Traditional methods estimate camera pose from image sequences based on geometric constraints and can be categorized into marker-based and markerless tracking methods [23]. In addition, learning-based camera tracking methods estimate camera pose by extracting high-dimensional image features from image sequences [24]. Early augmented reality (AR) systems primarily relied on marker-based camera tracking algorithms to achieve tracking and registration. This process involves identifying specific markers or feature points on natural markers in the real world and computing a homography matrix from the 3D space to the image plane based on these features, thereby determining the camera's pose. Special markers include artificially designed elements such as QR codes, specific patterns, or objects, while natural markers refer to pre-existing elements in the environment, such as buildings, landmarks, or objects with rich textures. ARToolkit [1] is a classic marker-based AR platform that estimates camera pose by detecting square fiducial markers and computing the homography. Voforia [2] is another popular augmented reality platform based on natural markers, which supports multiple types of markers, including 2D codes, image targets, 3D object targets, etc. Similar to ARToolkit, Voforia performs camera tracking by estimating a monotonicity matrix, but it is able to use more feature point matches, and the distribution of feature points is not limited to fixed positions. Compared with marker-based tracking, markerless camera tracking relies solely on image data captured by the camera, without requiring any predefined markers. It is widely regarded as a key enabler for the future of augmented reality due to its adaptability to diverse environments and its robustness against marker occlusion. Markerless tracking is typically achieved through visual SLAM (Simultaneous Localization and Mapping), which performs localization and mapping simultaneously in unknown environments. Depending on the type of visual sensor, SLAM can be classified into monocular, stereo, and RGB-D approaches. From the perspective of 3D scene reconstruction, SLAM methods are further divided into feature-based methods that yield sparse maps and direct methods that generate semi-dense reconstructions. PTAM [3] pioneered parallel processing in visual SLAM with separate tracking and mapping threads, enabling real-time performance. Building on this, ORB-SLAM2 [4] introduced robust loop closure and global optimization. DTAM [5] and LSD-SLAM [6] shifted toward direct methods for dense and semi-dense mapping, improving robustness under challenging conditions. DSO [7] further enhanced photometric calibration and tracking accuracy, while SVO [8] combined direct and feature-based approaches for efficient and precise pose estimation with improved depth filtering. Feature-based SLAM methods offer accurate tracking by minimizing reprojection error but struggle in low-texture environments and incur high computational costs for feature extraction. In contrast, direct methods optimize photometric error without explicit feature detection, offering better performance in textured scenes, but are sensitive to illumination changes. To improve robustness, wide-FOV cameras such as fisheye and panoramic lenses have been integrated into SLAM systems. Extensions of LSD-SLAM, DSO, and ORB-SLAM adapt SLAM to these camera models [9; 10; 11; 12; 13; 25; 26; 27]. While a wider FOV enhances tracking stability, it also increases computational demand.