# Reliability-Based View Synthesis for Free Viewpoint Video
## 3 Proposed Framework
### Depth Refinement  
There are two steps in the depth refinement process, as illustrated in Figure 1a. In the first step, depth consistency cross-checking is used to check whether each pixel's depth value is reliable or unreliable. Second, depth refinement is employed to interpolate the depth values of unreliable pixels. The details of the first step are as follows. For depth consistency cross-checking of the left reference view: let \((u,v)\) be the coordinate of one pixel from the left reference view, then its corresponding pixel \((u^{w},v^{w})\) in the right reference view is obtained through the classical DIBR technology [2]. The texture value \(I\) and depth value \(D\) of these two pixels are verified; the subscript \(L\) and \(R\) indicate left view and right view, respectively. \(I_{th}\) is a large preset threshold value for texture comparison and \(D_{th}\) is a small preset threshold value for depth comparison. The consistency checking produces five results, as follows:  
1. If \(||I_{L}(u,v)-I_{R}(u^{w},v^{w})||_{2}^{2}\leq I_{th}\) and \(|D_{L}(u,v)-D_{R}(u^{w},v^{w})|\leq D_{th}\) are both satisfied, this implies that these two pixels are matched. This depth pixel in the left reference depth map is reliable only in this situation, and it is marked as black in its cross-checking mask. 2. If \(||I_{L}(u,v)-I_{R}(u^{w},v^{w})||_{2}^{2}>I_{th}\) and \(|D_{L}(u,v)-D_{R}(u^{w},v^{w})|>D_{th}\) are both satisfied, this implies that these two pixels fail to match. In this situation, there is a high probability that the pixel belongs to the occlusion area and the depth pixel in the left reference depth map fails to check whether it is reliable or not, then it is marked as blue in its cross-checking mask. 3. If \(||I_{L}(u,v)-I_{R}(u^{w},v^{w})||_{2}^{2}>I_{th}\) and \(|D_{L}(u,v)-D_{R}(u^{w},v^{w})|\leq D_{th}\) are both satisfied, this implies that these two pixels fail to match. Either an erroneous texture pixel or an unreliable depth value causes this situation. We will check its surrounding depth distribution to find the real reason in the second step. The depth pixel in the left reference depth map is unreliable and it is marked as red in its cross-checking mask. 4. If \(||I_{L}(u,v)-I_{R}(u^{w},v^{w})||_{2}^{2}\leq I_{th}\) and \(|D_{L}(u,v)-D_{R}(u^{w},v^{w})|>D_{th}\) are both satisfied, this also implies that the depth pixel is unreliable, and it is marked as green in its cross-checking mask. 5. Some pixels in the left reference view are not able to project into the right reference view, because their corresponding pixels are located outside the image boundary. These areas are marked as white. Figure 1: Framework of the proposed view synthesis: (**a**) illustration of Depth Refinement; (**b**) the framework of proposed approaches using refined depth information. Figure 2 shows a result of the depth consistency check; because pixels in the white and blue regions fail to get a chance to verify their reliability, they are all determined to be unreliable and a specific weight is given when they are interpolated into virtual view. Several measures are implemented to refine other unreliable pixels, especially for the red and green regions. The main idea for the refinement is to find the most appropriate reliable pixel value to interpolate the depth value of unreliable pixels. Neighboring pixels from four directions are utilized here, and both the inverse proportion of distance and the reliability of the depth value are considered in calculating the weighting factors. If the reliable depth pixel maps to a reliable pixel in the other view, this indicates that this depth pixel is highly reliable. On the contrary, if the corresponding pixel in the other view is unreliable, the reliability of the pixel is lower. Let \(WD_{t}\), \(WD_{b}\), \(WD_{l}\), and \(WD_{r}\) be the weighting factors calculated by the distance from the current unreliable depth value to the nearest reliable depth pixel in top, bottom, left, and right directions, respectively.