# NTPP-MVSNet: Multi-View Stereo Network Based on Neighboring Tangent Plane Propagation
## 3 Methods
### Depth Hypothesis
#### 3.2.1 Normal Estimation  
The estimation of the normals is not a primary function of the multi-view stereo network in general. Rather, the emphasis is on the accuracy and completeness of the spatial position calculation for each pixel. As such, the network often fails to fully consider the geometric properties of the resulting 3D model. This issue has been addressed by related studies, such as monodepth, which have augmented the loss function with smoothing terms to facilitate the coherent transformation of the surface depth. Through the incorporation of a smoothing constraint into the loss function, the noise and discontinuities in the depth can be mitigated while still preserving the detailed information.  
\[Loss_{smooth}=|\partial_{x}d_{i}|e^{-|\partial_{x}h|}+|\partial_{y}d_{i}|e^{- |\partial_{y}h|}. \tag{1}\]  
It has been noted that the neighborhood depth information and the current pixel's depth information frequently demonstrate a linear transformation pattern as opposed to an identity transformation pattern within this method. Consequently, the enforcement of a strong consistency constraint on the neighborhood depth information stands in contradiction to the observed objective facts.  
In this presented study, the estimation of surface normal vectors within a multi-view stereo algorithm was suggested. The surface normal vectors were expected to furnish structural cues conducive to depth estimation, thereby optimizing the feature volume of surface normal vectors simultaneously with the regression of the depth map. The combined estimation of depth and surface normals can engender greater accuracy and robustness concerning the 3D reconstruction while also promoting better comprehension and significance of the scene geometry.  
Related works, such as the NNet [29], greatly consume memory. Furthermore, there is a logical deduction between the normal and depth information, and this method does not fully utilize the depth cost aggregation process but rather starts training the normal vector map again from the originally extracted feature map, resulting in lower algorithm efficiency.  
In Figure 6, it is proposed to simultaneously produce the depth prediction module as well as the normal vector prediction module of the model utilizing the cost aggregation method. The 3D U-Net structure, in accordance with the conventional multi-view stereo networks, is leveraged to standardize the cost aggregation and expunging the effect of individual outliers in the matching. In addition, to acquire depth maps and normals at the same time, a convolutional neural network was integrated at the terminal end of the 3D U-Net structure for dimension transformation of the output cost volume. Lastly, the final cost volume \(V\) measures \(R^{H\times W\times D\times 3}\). The regression formula for the depth map is demonstrated below, with \"\(L\)\" indicating the stage level.  
\[P^{L}_{(i,\ j)}(d_{m})=\frac{e^{V(i,\ j,\ m,\ 0)}}{\sum_{m=0}^{M-1}e^{V(i,\ j,\ m,\ 0)}} \tag{2}\]  
\[D^{L}(p)=\sum_{m=0}^{M-1}d_{m}\times P^{L}_{p}(d_{m})\quad\text{where }d_{m} \in\{d_{i}|i=1,...,M\}. \tag{3}\]  
Contingent upon the confidence level of each depth sampling, the confidence in the normal vector calculation is simultaneously influenced. Given that the normal vector calculation's degree of freedom is 2, salient characteristics pertaining to the last two dimensions of the volume were deployed to calculate the normal vector information under each depth sampling scenario. In congruence with the depth regression method, the normal vector estimation map for the targeted viewpoint materializes via a weighted averaging technique coupled with a subsequent re-normalization process, both under the guidance ofthe confidence conditions. The specific formula characterizing this calculation process is enumerated in the following.  
\[n^{L}(p)=\sum_{m=0}^{M-1}n_{m}\times P_{p}^{L}(n_{m})\quad\text{where }n_{m}\in\{n_{i}|i=1,...,M\}. \tag{4}\]