# Reliability-Based View Synthesis for Free Viewpoint Video
## 3 Proposed Framework
### Depth Map Processing Method  
After weighted blending is completed, the warped texture image and depth map become entirely filled. However, in the previous process, cracks and pinholes could be observed in the rendered image. With the previous method, a classical median filter was applied to smooth the texture image or remove these artifacts. In our framework, a depth map processing method is proposed. Not only the above-mentioned artifacts, but also the background pixels in cracks of foreground regions (shown in Figure 4a,b) can be removed. This method has advantages in preserving the texture details, since it is only performed on the detected coordinates.  
The main idea of DMPM is based on the fact that pixel value in depth maps always changes smoothly in a large area, except in the case of sharp edges in the boundary area between foreground objects and background. These features allow easy detection of noise in depth maps. In fact, most artifacts and noise caused by inaccurate depth values are reduced because of the previously introduced depth refinement, but some unreliable or undetected depth values remain in the reference depth map, most of them in out-of-boundary areas and occluded areas. Therefore, DMPM is still necessary. Details of the depth map processing method are as follows:  
1. A conventional median filter is proposed to apply to the coarse depth map \(d_{in}(x,y)\) to obtain an improved depth map \(d^{\prime}(x,y)\). It is capable of removing the existing noise and preserves the sharp boundary information.
2. The texture image \(I_{in}(x,y)\) is refined according to the improvement of its associated depth map. If the condition \(|d^{\prime}(x,y)-d_{in}(x,y)|>\varepsilon\) is satisfied (\(\varepsilon\) is a threshold value for depth difference), this indicates that the depth value of the pixel is unreliable and it is renewed after the median filter. An inverse mapping process using the updated depth value is employed to find an appropriate texture pixel. A depth range \(d^{\prime\prime}\in[d^{\prime}-\varepsilon,\;\;d^{\prime}+\varepsilon]\) is used as a candidate to find its corresponding pixel in two reference views. In Equations (11) and (12), we can get a corresponding reference pixel location (\(u_{r}\), \(v_{r}\)) through pixel (\(x\), \(y\)) and the associated depth values \(z_{v}\) and \(z_{r}\); \(\mathbf{A}\) and \(\mathbf{b}\) denote rotation matrix and translation matrix, respectively. Several measurements are used to make sure a highly reliable pixel is obtained by using backward warping. First, the depth value of the obtained pixel should be close to the updated depth value \(d^{\prime}(x,y)\). Second, the disparity between (\(x\), \(y\)) and (\(u_{r}\), \(v_{r}\)) should not be too large according to the alignment of the reference viewpoint and virtual viewpoint: \[z_{v}\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]=z_{r}\mathbf{A}\left[\begin{array}{c}u_{r}\\ v_{r}\\ 1\end{array}\right]+\mathbf{b},\] (17) \[z=\frac{255z_{near}z_{far}}{d^{\prime\prime}\left(z_{far}-z_{near}\right)+2 55z_{near}}.\] (18)  
In our previous method, we simply used a median filter on (\(x\), \(y\)), and this turned out to be very effective when the texture of this area was smooth. However, a median filter easily produces blurring effects when the scene has detailed textures. Unlike the texture images, the smooth regions in the depth map are invulnerable to the filter with gray value distributions. After the renovation is conducted, the associated texture image is updated according to the improvement of its depth map.  
Figure 4 shows the updated version of the integrated depth map, where the infiltration errors and unnatural depth distribution are eliminated by the classical median filter, while the sharp edges are preserved. Comparing Figure 4,c, the DMPM generates desirable improvement and avoids filtering of the entire image at the same time.