# Reliability-Based View Synthesis for Free Viewpoint Video
## 3 Proposed Framework
### Adaptive Background Modeling  
In the previous step, a refined depth map was obtained. In Section 3.2, we propose to apply an adaptive background modeling method evolving from Gaussian mixture model (GMM) to generate a reference image. GMM is commonly used in video processing to detect moving objects because of its capacity to identify foreground and background pixels [7]. In previous research, GMM was utilized to construct a stable background image aiming to fill large empty regions. However, GMM is not suitable for scenes that contain periodic or reciprocating foreground objects; these foreground moving objects are easily detected as erroneous background pixels, thus generating an inaccurate background image. In addition, some background pixels might have slight changes, for example,  
Figure 2: Result of depth consistency cross-check. pixel densities are different while shadows caused by foreground objects appear or move. Thus, the stable background images generated by previous approaches always had blurring effects and were not accurate. In our proposed adaptive background modeling method, both the texture images and their associated depth maps are utilized to explore the temporal correlation. In addition, we propose to apply a reliability-based view synthesis method using background information to interpolate the intermediate image and fill the disocclusions. The proposed method works at the pixel level, and every pixel is modeled independently by a mixture of \(K\) Gaussian distributions, where \(K\) is usually between 3 and 5. By using this distribution, pixel values that have a high probability of occurring are saved if their associated depth values show they belong to the background. The Gaussian mixture distribution with \(K\) components can be written as [25]:  
\[p\big{(}x_{j,t}\big{)}=\sum\limits_{i=1}^{K}\omega_{j,t}^{i}\cdot\eta\left(x_ {j,t},\mu_{j,i,t},\sigma_{j,i,t}^{2}\right), \tag{3}\]  
where \(p\big{(}x_{j,t}\big{)}\) denotes the probable density of value \(x_{j,t}\) of pixel \(j\) at time \(t\); \(\eta\) is the Gaussian density function with three dependent variables: \(x_{j,t}\), \(\mu_{j,i,t}\), and \(\sigma_{j,i,t}^{2}\), where \(\mu_{j,i,t}\) denotes the mean value of pixel \(x_{j}\); and \(\sigma_{j,t}^{2}\) is the variance value of the pixel. Further, \(\omega_{j,t}^{i}\) is the weight of the \(i\)th Gaussian distribution at time \(t\) of pixel \(j\), with \(\sum\limits_{i=1}^{K}\omega_{j,t}^{i}=1\). The function \(\eta\) is given by:  
\[\eta\left(x_{j},\mu_{j,t},\sigma_{j,t}^{2}\right)=Q\cdot exp\left[-\frac{1}{2} \left(\sigma_{j,t}^{2}\right)^{-1}\left(x_{j}-\mu_{j,t}\right)^{2}\right], \tag{4}\]  
\[Q=\frac{1}{\left(2\pi\right)\cdot d/2\cdot\left|\sigma_{j,t}^{2}\right|^{1/2}}, \tag{5}\]  
Before texture information is modeled by Gaussian distribution, we propose to verify each novel pixel to ensure that it is not from a foreground region. If the depth value is much bigger than the stored depth buffer (which means this pixel is nearer to a captured device), the pixel is considered as a foreground pixel. Otherwise, if the depth value is much smaller than the stored buffer, the pixel is considered as a background pixel, and the modeled distribution is not reliable and should be restarted. The detailed process to generate the reference background distribution is as follows:  
1. **Initialization. ** The model is initialized at the beginning of the generation (time \(t_{0}\)): \[\omega_{j,t_{0}}^{j}=\left\{ \begin{array}{ll}1,&\text{if }i=1\\ 0,&\text{else}\end{array} \right.,\] (6) \[\mu_{j,i,t_{0}}=\left\{ \begin{array}{ll}x_{j,t_{0}},&\text{if }i=1\\ 0,&\text{else}\end{array} \right.,\] (7) \[\sigma_{j,i,t_{0}}^{2}=\sigma_{j}^{2},\] (8) \[d_{j}=d_{j,t_{0}},\] (9) where the variance value \(\sigma_{j}^{2}\) is set to a certain large number, \(d_{j}\) is the stored depth buffer for pixel \(j\), and \(d_{j,t_{0}}\) is the depth value of pixel \(j\) at time \(t_{0}\). 2. **Update.