# Tracking and Registration Technology Based on Panoramic Cameras
## 3 Method
### Panoramic View Reconstruction  
Panoramic view reconstruction synthesizes novel views of the scene by using the input panoramic image and its corresponding depth map, combined with camera poses from different viewpoints. This process involves two main steps: pixel coordinate projection and pixel value interpolation filling, as illustrated in Figure 4.  
During the motion of the panoramic camera, suppose the panoramic image captured at time \(t\) is denoted as \(I_{t}\), with its corresponding depth map \(D_{t}\), and the image captured at time \(t+1\) is denoted as \(I_{t+1}\). The relative camera pose from time \(t\) to \(t+1\) is represented as \(T_{t,t+1}\). Through pixel coordinate projection, any pixel \(p_{t}\) in image \(I_{t}\) is projected to a corresponding pixel \(p_{t+1}\) in image \(I_{t+1}\). Then, via pixel value interpolation, the pixel value at \(p_{t}\) in image \(I_{t}\) is used to fill the corresponding location \(p_{t+1}\) in image \(I_{t+1}\), generating a synthesized view \(I^{\prime}_{t+1}\), where the pixel value at \(p_{t+1}\) corresponds to that of \(p_{t}\). Figure 4: Illustration of the panoramic view reconstruction process, which involves pixel coordinate projection and value interpolation based on depth and camera pose information. \begin{table}
\begin{tabular}{c c c c c c} \hline
**Name** & **Operations** & **Kernel Size** & **Stride** & **Padding** & **Dilation Rate** & **Output Dimensions** \\ \hline Input & & & & & & \(9\times 256\times 512\) \\ C1 & Conv2d & \(7\times 7\) & 2 & \(3\times 3\) & 1 & \(64\times 128\times 256\) \\  & Maxpool & & & & & \(64\times 64\times 128\) \\ Layer1 & BasicBlock & \(3\times 3\) & 1 & \(1\times 1\) & 1 & \(64\times 64\times 128\) \\  & Maxpool & & & & & \(64\times 32\times 64\) \\ Layer2 & BasicBlock & \(3\times 3\) & 1 & \(1\times 1\) & 1 & \(128\times 32\times 64\) \\  & Maxpool & & & & & \(128\times 16\times 32\) \\ Layer3 & BasicBlock & \(3\times 3\) & 1 & \(1\times 1\) & 1 & \(256\times 16\times 32\) \\  & Maxpool & & & & & \(256\times 8\times 16\) \\ Layer4 & BasicBlock & \(3\times 3\) & 1 & \(1\times 1\) & 1 & \(512\times 8\times 16\) \\ C2 & Conv2d & \(3\times 3\) & 1 & \(1\times 1\) & 1 & \(256\times 8\times 16\) \\ C3 & Conv2d & \(3\times 3\) & 1 & \(1\times 1\) & 1 & \(12\times 8\times 16\) \\ AvgPool2d & & & & & & \(12\times 1\times 1\) \\ (\(T_{t,t-1},T_{t,t+1}\)) & reshape & & & & & \(2\times 6\) \\ C4 & Conv2d & \(3\times 3\) & 1 & \(1\times 1\) & 1 & \(2\times 64\times 128\) \\ M\({}_{2}\) & Upsample & & & & & \(2\times 256\times 512\) \\ C5 & Conv2d & \(3\times 3\) & 1 & \(1\times 1\) & 1 & \(2\times 32\times 64\) \\ M\({}_{1}\) & Upsample & & & & & \(2\times 128\times 256\) \\ \hline \end{tabular}
\end{table} 
Table 3: Output dimensions and hyperparameters of the pose estimation sub-network. In the process of pixel coordinate mapping, the projected pixel \(p_{t+1}\) in image \(I_{t+1}\) may fall into one of the following three cases: if the image coordinates of \(p_{t+1}\) fall outside the valid range of image \(I_{t+1}\), the corresponding pixel value in the reconstructed image \(I^{\prime}_{t+1}(p_{t})\) is set to zero; if the image coordinates of \(p_{t+1}\) lie within the valid image range and are integers, the corresponding pixel value in the reconstructed image is directly assigned as \(I^{\prime}_{t+1}(p_{t})=I_{t+1}(p_{t+1})\); if the image coordinates of \(p_{t+1}\) are within the valid image range but contain fractional values, the exact pixel value at \(p_{t+1}\) in image \(I_{t+1}\) cannot be directly retrieved. In this case, bilinear interpolation is employed using the pixel values of the four neighboring pixels--top left \(p^{ll}_{t+1}\), top right \(p^{tr}_{t+1}\), bottom left \(p^{bl}_{t+1}\), and bottom right \(p^{br}_{t+1}\).