# A Sparsity-Invariant Model via Unifying Depth Prediction and Completion
## 4 The Proposed Method
### Training Loss
#### 4.3.2 Mean Normal Loss  
The mean normal loss [34] measures the disparity between the estimated normal of a pixel, which is determined by considering its neighboring pixels, and the ground truth normal of that pixel.  
We depict the process of inferring surface normal from the depth map as a least-squares problem. Specifically, for any pixel \(i\), assuming its depth is \(z_{i}\), we first compute its three-dimensional coordinates \((x_{i},y_{i},z_{i})\) from its two-dimensional coordinates \((u_{i},v_{i})\) based on the pinhole camera model, as described by Equation (1). Then, to compute the surface normal of pixel \(i\), we first determine the tangent plane passing through pixel \(i\) in 3D space. We proceed with the assumption that pixels within the local neighborhood of pixel \(i\) lie on the same tangent plane. Specifically, we define the set of neighboring pixels \(\mathcal{N}_{i}\), including pixel \(i\) itself.  
With these pixels lying on the tangent plane, the estimated surface normal vector \(\vec{n}=(n_{x},n_{y},n_{z})\) should satisfy the following equation:  
\[A\vec{n}=\vec{b},\quad subject\ to\ ||\vec{n}||_{2}^{2}=1 \tag{8}\]  
where \(A\) is the data matrix formed by stacking the 3D points within the patch as shown in Equation (9).  
\[A=\begin{bmatrix}x_{1}&y_{1}&z_{1}\\ x_{2}&y_{2}&z_{2}\\ \vdots&\vdots&\vdots\\ x_{K}&y_{K}&z_{K}\end{bmatrix}\in\mathcal{R}^{K\times 3} \tag{9}\]  
Figure 3: Diagram illustrating the upsampling of plane coefficients. Regions with the same color and numerical values represent the same plane coefficients, indicating that the upsampled results maintain consistency in plane coefficients within local regions.  
\(\vec{b}\in\mathcal{R}^{k\times 1}\) is a constant vector consisting of all ones. \(K\) is the size of \(\mathcal{N}i\). The least-squares solution to this problem, i.e., minimizing \(||A\vec{n}-\vec{b}||^{2}\), has a closed-form solution. Thus, we can obtain  
\[\vec{n}=\frac{(A^{T}A)^{-1}A^{T}\vec{b}}{\|(A^{T}A)^{-1}A^{T}\vec{b}\|_{2}}. \tag{10}\]  
In this loss function, to estimate the normal vector \(\vec{n}\), we construct a matrix \(A\) by stacking the 3D points within a \(k\times k\) patch centered around the pixel from \(\mathcal{N}i\) and solve the equation \(A\vec{n}=\vec{b}\). To compute the mean normal loss, we first estimate surface normal vectors for all \(K\) non-overlapping patches in the predicted depth map \(Z\) and the ground truth depth map \(Z^{*}\), and then we calculate the loss by penalizing their differences. Therefore, the mean normal loss is defined as  
\[L_{MNL}=\sum_{i}\|\vec{n}_{i}-\vec{n_{i}}^{*}\|_{1}, \tag{11}\]  
where \(\vec{n_{i}}\) is the normal vector calculated from the predicted depth map, and \(\vec{n_{i}}^{*}\) is the normal vector computed from the ground truth depth map. The difference between the two vectors is measured by subtraction.  
As the value of \(K\) increases, the selected patch may no longer satisfy the plane assumption due to depth discontinuities within the patch. Even in such cases, the mean normal loss still provides an effective supervision, because it penalizes inconsistencies between the local tangent plane within the patch and \(n_{i}\). According to Equation (10), we can also observe that the loss directly influences the depth of all points within the patch through \(A\).