# CAPDepth: 360 Monocular Depth Estimation by Content-Aware Projection
## 3 Method
### Geometry-Aware Fusion  
The simplicity of predicting depth maps from tangential images comes at a cost: Since depth estimation for each patch is conducted independently, a globally consistent depth scale is no longer guaranteed. Additionally, an object projected from different angles onto  
Figure 5: Illustration of line curvature (LC), line inclination (LI), and Tissot’s indicatrix measurement methods: (**a**) The curved projection line \(L_{1}\) (represented by the green solid line) after CAP projection, the dotted line \(s\) (represented by the blue dotted line) connecting the endpoints of the projection line \(L_{1}\), and the line curvature LC is defined as the maximum distance between \(L_{1}\) and \(s\). (**b**) The straight line \(L_{2}\) (represented by the red solid line) obtained by straight line projection, the curved projection line \(L_{3}\) (represented by the green solid line) after CAP projection, and the endpoints of \(L_{2}^{\prime}\) and \(L_{3}\) are connected by translation. The line inclination LI is defined as the maximum distance between \(L_{2}^{\prime}\) and \(s\). (**c**) Left: the infinitesimal unit circle defined on the sphere; right: the corresponding Tissot index after projection on the plane. Figure 6: (**a**) ERP RGB input image. (**b**) Predicted ERP depth map. (**c**) Tangent image generated from (**a**). (**d**) Block-wise predicted depth map. multiple tangential images may exhibit variations in geometric position across images. As shown in Figure 7(Left), differences in depth estimates in overlapping areas between adjacent patches can lead to errors during the final merge into the ERP format, resulting in overlapping artifacts and misaligned spatial positions. To compensate for feature differences caused by geometric disparities in tangential images, [PERSON] et al. [16] uses a geometric embedding module to provide additional geometric information. First, for a pixel point \(P_{t}(x_{t},y_{t})\) on the tangential image, its corresponding 3D spherical coordinates \(P_{s}(\lambda,\phi,\rho)\) on the unit sphere and the viewpoint center \(P_{c}(\lambda_{c},\phi_{c})\) are found as input attributes for the geometric embedding network. The spherical coordinates link the tangential-image plane coordinates to associate global and local geometric feature information. In contrast, the viewpoint coordinates link the spatial location of the tangential image on the sphere. The geometric embedding consists of a two-layer MLP, encoding the five-channel spherical attributes and fusing them with the corresponding spatial position image features through summation. This early fusion is then added to layer 1 of the encoder. During network iterations, the geometric features of the first iteration are fixed and associated with the tangential image-projection parameters. In the second iteration, the newly calculated geometric features are applied within the network. In our case, this study redesigns the feature fusion and image fusion algorithms (to ERP format). Specifically, this study records an index map of tangential images in the ERP space, storing the spatial position and mapping relationship of each patch in the ERP. An extra overlapping region of 1/32 of the patch width and 1/16 of the patch height is set between adjacent patches. Mean filtering works on this region during de-blocking to achieve smoothing and de-duplication. Feature fusion and image fusion (Figure 7(Right)) and time efficiency (specific data are given in Section 4.3) have been significantly improved. To ensure a globally consistent depth scale, this research uses a Transformer to aggregate information from the patches. Similar to the method proposed in [34], CAPDepth retains positional information. The structure of the multi-head attention Transformer is as described in [35]. Through the self-attention architecture, the Transformer learns to globally  
Figure 7: Effectiveness of geometric feature fusion. (**Left**): Overlapping depth and misaligned geometry merged. (**Right**): Smoother, more consistent depth map.