## 2 Proposed inpainting method
### Disocclusions Filling  
Disocclusions are caused by the discontinuity of depth value, and usually belong to the background regions. Therefore, the hole filling is preferred to start from the background side. Pixels on the background boundaries should be given higher priority, while the confidence term and data term cannot be ignored. For depth image with multiple depth layers or many foreground objects, using a global threshold to distinguish foreground and background is a difficult task. In this paper, a local background-first hole filling method is proposed to handle the disocclusions. The pixels around each hole are divided into foreground and background by applying an adaptive threshold, as well as depth information is used to guide the priority computing and the search of the best matching patch. Figure 9: Schematic diagram of the modified 3D warping. Through the 3D warping, both the color image and the depth image of the virtual view are obtained, and the positions of holes in the two images are corresponding. In the virtual depth image, based on the depth distribution, the foreground and background pixels are determined within the minimum-bounding rectangle of each hole. In this case, we use the Otsu thresholding method to set the separation threshold for each hole. In order to eliminate the effect of outliers, 5% valid pixels in each rectangle with highest and lowest depth values are removed before the computing of threshold. Then the threshold is applied to distinguish the foreground and background pixels near each hole as follows:  
\[BG(u_{v},v_{v})=\left\{ \begin{array}{l}1,d(u_{v},v_{v})\leq\text{Otsu}(Hole_ {i})\\ 0,d(u_{v},v_{v})>\text{Otsu}(Hole_{i})\end{array} \right., \tag{13}\]  
where \((u_{v},v_{v})\) is the point within the minimum bounding rectangle of hole \(i\). The threshold of each hole is also applied to the priority calculation for disocclusion filling. Since the original inpainting algorithm causes the foreground and background pixels to spread simultaneously in the disocclusion, we refine the priority calculation based on the depth information as follows:  
\[P(p)=[C(p)+Z(p)+B(p)]\cdot D(p), \tag{14}\]  
where \(Z(p)\) is depth term and \(B(p)\) is background term. They are given by:  
\[Z(p)=\frac{d_{\text{max}}-\frac{\sum_{q\in\Psi_{p}\cap(l_{p}-\Omega)}d(q)}{ \left|\Psi_{p}\cap(l_{p}-\Omega)\right|}}{d_{\text{max}}-d_{\text{min}}}, \tag{15}\]  
\[B(p)=\frac{\sum_{q\in\Psi_{p}\cap(l_{p}-\Omega)}B(q)}{\left|\Psi_{p}\cap(l_{p} -\Omega)\right|}, \tag{16}\]  
where \(\left|\Psi_{p}\cap(l_{p}-\Omega)\right|\) is the number of valid pixels in \(\Psi_{p}\). \(d_{\text{max}}\) and \(d_{\text{min}}\) are the highest and lowest nonzero depth values in the virtual depth image, respectively. The depth term reflects the average depth value of the valid pixels in \(\Psi_{p}\). Since disocclusions should be filled with background pixels, the patch, which has lower depth values, is given a higher priority. The background term represents the percentage of the local background pixels in the patch, and the patch with more background pixels is given a higher priority. The improved priority calculation ensures that when the confidence and data terms of the two patches are close, the one closer to the background can be preferentially filled. Especially for the depth image with multiple depth layers, the background term helps to correctly identify the foreground and background objects. After determining the filling priorities, we search for the best matching patch in both the reference image and virtual image.