# Reliability-Based View Synthesis for Free Viewpoint Video
## 3 Proposed Framework
### Reliability-Based Weighted Blending  
As the background distribution for each reference view is obtained by the proposed background modeling method discussed in Section 3.2, two background images are projected into virtual viewpoint and then blended into one background image in virtual viewpoint (represented by \(I_{B}\)). Previous research shows that GMM has an inherent capacity to capture background and foreground pixel intensities; missing pixel intensities of an occluded area are successfully recovered by exploiting temporal correlation.  
In our proposed method, weighting factors are also applied to blend two reference views and one background image into a synthesized image. Two reference texture images are projected to virtual view using their corresponding refined depth maps, and two intermediate texture images \(I_{L}\), \(I_{R}\) and depth images \(D_{L}\), \(D_{R}\) are obtained. The reliability-based weighted blending process to produce a virtual image \(I_{V}\) is as follows:  
1. If a pixel is filled in both \(I_{L}\) and \(I_{R}\), first two depth values are compared. If the depth value of one pixel is much bigger than the other, this indicates that one pixel is obviously nearer to the capturing device. \(I_{V}\) is filled by the pixel with a bigger associated depth value. If two depth values are very close, weighting factors are utilized. \(I_{V}\) is formulated as follows: \[I_{V}=W_{L}\cdot I_{L}+W_{R}\cdot I_{R},\] (15) \[W_{i}^{\prime}=WD_{i}\times WR_{i},i=L,\ R,\] (16) where \(WD\) is the weighting factor for the inversely proportional distance between reference view and virtual view, and \(WR\) is the weighting factor for the previously defined reliability of depth value. One of three values (\(r_{H}\), \(r_{M}\), or \(r_{L}\)) is assigned to \(WR\) when a pixel in this reference intermediate image is mapped by a reliable, refined, or unreliable depth value, respectively. It should be noted that \(W_{L}\) and \(W_{R}\) need to be normalized by \(W_{i}^{\prime}\) so that \(W_{L}+W_{R}=1\).
2. If only one pixel is filled in two reference views, for example only \(I_{L}\) is filled, the reliability of \(I_{L}\) is taken into consideration. If \(I_{L}\) is mapped by a reliable depth value, \(I_{V}\) can simply be filled with \(I_{L}\) (\(I_{V}=I_{L}\)). Otherwise, background information is used to generate \(I_{V}\). If \(D_{L}\) is close to the background depth value, then \(I_{V}=(I_{L}+I_{B})/2\); if \(D_{L}\) is much bigger than \(D_{B}\), \(I_{V}=I_{L}\).
3. If pixels in both reference views are not filled, we use the constructed background image to deal with the hole-filling challenge. First, we check the surrounding depth value of \(I_{V}\) and find the filled depth value to determine a proper depth value range. Then \(I_{V}\) is filled by the  
Figure 3: Adaptive background modeling results: (**a**) _Ballet_ background image; (**b**) _Breakdancers_ background image.  
background pixel if its depth value is in the obtained range. Otherwise, inverse warping and classical inpainting are applied to fill \(I_{V}\).  
We propose this hole-filling method to ensure that background pixels are appropriate to fill the remaining hole regions. Because they adopt depth information, background pixels can be chosen to improve the rendered image quality even when the hole is surrounded by foreground objects.