## 2 Proposed inpainting method
### Depth Image Preprocessing  
The quality of the depth image has a great impact on the rendering of the virtual view. The depth value usually mutates at the junction of the foreground and the background in the depth image. The ghosts are mainly caused by the inaccurate depth values of foreground boundary, causing the foreground pixels to be projected into the background region. In addition, the disocclusion is usually the background region occluded by the foreground in the reference view, which becomes visible in the virtual view. It often appears between the foreground and the background in the virtual image, depending on the location of the virtual view. Because of artifacts, both sides of the disoccluded region may be foreground pixels. Subsequent hole filling algorithm will further spread the error and seriously reduce the quality of the virtual view. Figure 6: Inpainting results of extended reference image and its depth image: (**a**) Inpainting result of extended reference image; and (**b**) inpainting result of extended depth image. In order to remove the ghosts and project them to the correct space, in this paper, we propose a depth image preprocessing method to refine the depth values of the pixels near the boundary of the foreground. Since only several pixels in the depth image are processed, the 3D perception of the synthesized view can be effectively improved without significantly reducing the quality of the depth image. In the preprocessing algorithm, we first determine the foreground boundary to be refined based on the external parameters of the virtual view. In general, ghosts and disocclusions occur on the right side of the foreground for the right synthesized view, and similar situations for the other locations of virtual view. Therefore, only one side of the foreground boundary needs to be refined. In this paper, we take the synthesis of the right virtual view for example. The detection of the foreground boundary is performed in the depth image. All pixels are scanned to determine whether adjacent points in the horizontal direction are still adjacent in the virtual view. If not, holes or overlap of pixels will appear in the virtual image. The depth image can reflect the distance from the object to the camera, which is expressed as the depth value in the image. The foreground and adjacent background usually have different depth values, and disocclusion occur on the region with sharp depth transition. Therefore, the boundaries to be processed are detected according to the degree of depth discontinuity and marked as follows:  
\[Mask(u,v)=\left\{ \begin{array}{l}1,d(u,v)-d(u+1,v)>T\&\&\mid d(u+k,v)-d(u+1,v )\right\rvert\leq T\\ 0,\text{otherwise}\end{array} \right., \tag{11}\]  
where \(d(u,v)\) means the depth value of the pixel \((u,v)\) in reference image, \(\left|d(u+k,v)-d(u+1,v)\right|\leq T\) is used to avoid the depth value of other foreground being changed. \(k=\left\{1,2,\cdots,\frac{M-1}{2}\right\}\), and \(M\) is the width of the preprocessing template and is set to 7 in our experiment. The threshold \(T\) is set according to the depth distribution of the depth image. After obtaining the mask image, the overlap of the marked foreground pixel with other background pixels in the virtual image is detected. If the overlap appears, the related foreground pixel will not be refined to avoid two types of backgrounds appear at the boundary of foreground in the virtual image. Overlap detection is achieved through the original 3D warping process. Each marked pixel is assigned a label to record the overlap of the pixels. If the pixel \((u,v)\) overlaps with other pixels in the virtual image and the depth difference between them is greater than \(T\), the value of label is set to 1, and vice versa is 0. The foreground pixels that need not be refined in the mask image are filtered as follows:  
\[Mask\_filtered(u,v)=\left\{ \begin{array}{l}1,Mask(u,v)=1\&\&\mid label(u,v)= 0\\ 0,\text{otherwise}\end{array}