# Tracking and Registration Technology Based on Panoramic Cameras
## 3 Method
### Panoramic Depth Estimation Sub-Network  
Panoramic depth estimation is a key component in enabling unsupervised panoramic camera pose estimation. Considering the real-time requirements of 3D tracking and registration in augmented reality systems, this chapter proposes a lightweight panoramic depth estimation sub-network tailored for panoramic camera pose estimation. As illustrated in Figure 2 (where C denotes 2D convolution, D denotes dilated convolution, T denotes transposed convolution, and 2\(\times\) represents upsampling), the network adopts an encoder-decoder architecture with skip connections. The sub-network consists of an encoder, a decoder, and skip connections between them. It takes a single equirectangular panoramic image as input and outputs dense panoramic depth maps at two different scales. The encoder extracts image features using multi-scale rectangular convolution modules, dilated convolutions, and standard convolutional operations. The decoder reconstructs multi-scale depth maps from the encoded features through transposed convolutions and other upsampling techniques, thus achieving the transformation from pixel-level image information to depth information. Meanwhile, skip connections integrate shallow detail features from the encoder with deep semantic features in the decoder, enhancing the network's robustness and preserving fine-grained details from the input image, thereby producing depth maps with rich structural information. **Encoder**: The encoder mainly consists of multi-scale rectangular convolution modules, dilated convolution modules, and convolutional downsampling operations. The output feature dimensions and corresponding hyperparameters are summarized in Table 1. The hyperparameters of the convolutional operations include kernel size, stride, padding, and dilation rate. The output feature dimensions are represented as channel \(\times\) height \(\times\) width. Equirectangular panoramic images typically suffer from significant distortions, especially in the polar regions. While conventional square convolutional filters are effective at extracting features from undistorted perspective images, they are less effective when applied to distorted panoramic images. To address this issue, a multi-scale rectangular convolution module is introduced to mitigate the impact of distortion. According to [28], we employ only two layers of rectangular convolution filters; this design effectively addresses the distortion in ERP images. The most severe geometric distortions occur near the poles and primarily affect low-level features such as edges. By applying distortion-aware filters early, we compensate for these effects before deeper semantic features are extracted. Each rectangular filter block uses multiple aspect ratios with approximately constant area, enabling adaptive feature extraction across image rows. Combined with later dilated  
Figure 2: Panoramic depth estimation sub-network architecture. convolutions and limited downsampling, which provide large receptive fields, two such layers are sufficient to model the spatially varying distortions without further overhead. Given the varying degrees of distortion across different regions of a panoramic image, each layer of the multi-scale rectangular convolution module adopts four rectangular convolution filters with different aspect ratios to extract features, thereby reducing the impact of distortion on feature extraction. Since shallow features are helpful for the decoder to reconstruct meaningful representations, a skip connection is introduced after the second layer of the rectangular convolution module. The shallow features are concatenated and passed to the decoder to enhance its feature decoding capability. To improve computational efficiency, the encoder performs only two downsampling operations on the input image, instead of applying multiple downsampling steps and stacking a large number of convolutional filters. This reduces the number of parameters and makes the network more lightweight.