# Depth Image Completion through Iterative Low-Pass Filtering
## 2 Approach
### Iterative Low-Pass Filter (ILF)
\[\sigma_{i}<S_{criteria}\parallel i<1500,\ S_{criteria}=0.005, \tag{1}\]  
\[\sigma_{i}=\sqrt{\frac{\sum(P_{i}-P_{i-1})^{2}}{N}}, \tag{2}\]where \(P_{i}\) is the expanded depth matrix after the \(i\)-th iteration, \(\sigma_{i}\) is the deviation between \(P_{i}\) and \(P_{(i-1)}\), and \(S_{criteria}\) is the stop criterion for the minimum deviation, set to 0.005 m. This deviation is aligned with the precision of the depth camera. The results are presented below. Figure 2 illustrates the depth differences between each iteration, highlighting the nuanced evolution during the Iterative Low-Pass Filter (ILF) process. Notably, we employed a weighted spectrum derived from a single RGB image to stabilize the ILF process. The spatial frequency of the RGB image was extracted and utilized as a weight in the initial iterations. This strategic approach stems from the observation that, under certain illumination conditions, the spatial frequency of the depth image can be discerned in RGB images. The spatial frequency of the surface at low and mid frequencies serves as an insightful initial approximation. To account for noise in the high spatial frequencies present in RGB images, the weighted spectrum was incorporated for a maximum of 150 iterations in the initial stage of the ILF. This precaution was taken to prevent errors from emerging in the subsequent high spatial frequency information. During the initial process stage, the low-pass filter was multiplied by the Fourier spectrum of the depth matrix, steering the process towards generating depth data that aligned with the structural features of the RGB images. After a maximum of 150 iterations, the depth data accurately reflected the surface's trend. The weight of the RGB spatial frequency was then removed from the iterative process to mitigate potential side effects, such as overfitting and spatial noise from the RGB images. We further delved into the convergence processes of the ILF by calculating the L1 norm value within operation blocks. To establish a reference to the ground truth data, RGB-D data extracted from the NYU Depth V2 dataset [38] was employed. In compressed sensing, the L1 norm is widely utilized for its capacity to encourage sparse representations of signals. The optimization objective is to ascertain a sparse representation by minimizing the L1 norm of the signal while adhering to the measurement constraints. The results demonstrated a decreasing trend in the L1 norm value throughout the iteration process, indicating the convergence of the expanded depth matrix during the ILF process. The L1 norm values of the depth matrices at each epoch are presented in the results. Figure 3a illustrates a notable decrease in the L1 norm during the iteration steps, showcasing the convergence of depth matrices throughout the Iterative Low-Pass Filter (ILF) process. Meanwhile, Figure 3b presents the error between the depth matrix and the ground truth. The results indicate that, during the ILF process, depth matrices converge and are consistently updated with reference to residual data. It is worth noting the presence of several glitches in Figure 3a, which were attributed to the updated low-pass filters. Figure 2: Standard deviation (log \(Y\)-axis)-iteration plot. When the ILF transitions to an updated low-pass filter, the L1 norm value experiences a temporary rise. This increase is a consequence of the updated low-pass filter providing additional frequency information due to its larger radius. As the new data, influenced by the updated low-pass filter, is incorporated, the L1 norm value initially rises and subsequently declines to a lower value in the subsequent epochs. This dynamic can be explained by the fact that the updated low-pass filter redirected the ILF towards the next optimization goal, thereby preventing local extrema that might lead to the ILF process being prematurely halted due to satisfaction of the stop criteria. Consequently, the ILF was adept at updating the depth matrix with appropriate values.