# NTPP-MVSNet: Multi-View Stereo Network Based on Neighboring Tangent Plane Propagation
## 3 Methods
### Depth Hypothesis
#### 3.2.3 Propagation  
Subsequent to acquiring the low-resolution depth and surface normal prediction maps \(D_{l}\), \(N_{l}\) (\(l<3\)), the next imperative step pertains to ascertaining the neighborhood tangent plane information on these maps, followed by an assessment of the depth hypothesis values via homographic transformations. To this end, the projection of the tangent plane, which is specified in accordance with the object's surface, onto the image plane constitutes a pivotal undertaking.  
In order to accomplish scale transformation, the depth and surface normal prediction values are initially subjected to upsampling operations, with bilinear methods being a popular choice. Subsequently, based on the pinhole imaging principle, every pixel is re-projected into the world coordinate system. The calculation formula for this procedure is enunciated below.  
\[\overrightarrow{n_{(i,j)}}^{T}p=\overrightarrow{n_{(i,j)}}^{T}(KR^{-1} \left[\begin{array}{c}i*d_{(i,j)}\\ j*d_{(i,j)}\\ d_{(i,j)}\\ \end{array}\right]-R^{-1}t). \tag{5}\]  
Here, \(K\) represents the intrinsics matrix of the camera, and \(R\) and \(t\) represent the rotation and translation matrix for transforming from the world coordinate system to the camera coordinate system.\(\left[\begin{array}{cc}X_{(i,j)}&Y_{(i,j)}&Z_{(i,j)}&1.\\ \end{array}\right]^{T}\) expresses the homogeneous coordinates of the 3D point that corresponds to the pixel \((i,j)\).  
Based on the surface normal prediction value of pixel \((i,j)\) output by the regularization module, it is possible to estimate the tangent plane in space that corresponds to this point. Specifically, the calculation formula for this is as follows:  
\[\left[\begin{array}{cc}\overrightarrow{n_{(i,j)}}^{T}&-\overrightarrow{n_{( i,j)}}^{T}\left[\begin{array}{c}X_{(i,j)}\\ Y_{(i,j)}\\ Z_{(i,j)}\\ \end{array}\right]\\ \end{array}\right]\left[\begin{array}{c}X\\ Y\\ Z\\ 1\\ \end{array}\right]=0. \tag{6}\]  
According to prior assumptions, we approximate the tangent planes of neighboring pixels. Similarly, based on the above calculations, the equation for the tangent plane in space for neighboring point \((i,j)\) under the predicted value \(\overrightarrow{n_{(i,j)}}\) is as follows:  
Figure 7: The calculation and implementation of spatially continuous neighborhood positions.  
\[\left[\begin{array}{cc}\overrightarrow{n_{(\Delta i,\Delta j)}}^{\top}&- \overrightarrow{n_{(\Delta i,\Delta j)}}^{\top}T\left[\begin{array}{c}X_{( \Delta i,\Delta j)}\\ Y_{(\Delta i,\Delta j)}\\ Z_{(\Delta i,\Delta j)}\end{array}\right]\\ \end{array}\right]\left[\begin{array}{c}X\\ Y\\ Z\\ 1\end{array}\right]=0. \tag{7}\]  
Substituting the above pinhole imaging equation, we obtain the following formula:  
\[\left[\begin{array}{cc}\overrightarrow{n_{(\Delta i,\Delta j)}}^{\top}&- \overrightarrow{n_{(\Delta i,\Delta j)}}^{\top}T\left[(KR)^{-1}\left[ \begin{array}{c}\Delta i\widehat{d}_{(\Delta i,\Delta j)}\\ \Delta j\widehat{d}_{(\Delta i,\Delta j)}\\ \widehat{d}_{(\Delta i,\Delta j)}\end{array}\right]-R^{-1}t\right]\\ \end{array}\right]\left[\begin{array}{cc}KR&Kt\\ 0&1\end{array}\right]^{-1}\left[\begin{array}{c}i\widehat{d}_{(i,j)}\\ j\widehat{d}_{(i,j)}^{\top}\\ \widehat{d}_{(i,j)}\\ 1\end{array}\right]=0 \tag{8}\]  
After rearranging the formula, the final depth hypothesis for pixel \((i,j)\) is:  
\[\widetilde{d}_{(i,j)}=\frac{\overrightarrow{n_{(\Delta i,\Delta j)}}^{\top}T \left(KR\right)^{-1}\left[\begin{array}{c}\Delta i\\ \Delta j\\ 1\end{array}\right]\widehat{d}_{(\Delta i,\Delta j)}}{\overrightarrow{n_{( \Delta i,\Delta j)}}^{\top}T\left(KR\right)^{-1}\left[\begin{array}{c}i\\ j\\ 1.\end{array}\right]} \tag{9}\]