# Reliability-Based View Synthesis for Free Viewpoint Video
## 3 Proposed Framework
### Adaptive Background Modeling
** In the next frame, i.e., at time \(t_{1}\), we first check the depth level of this pixel, and \(d_{j,t1}\) is compared with the existing depth buffer \(d_{j}\). There are three situations for the depth comparison results:1. If the condition \(d_{j,t1}-d_{j}\ >t_{d}\) is satisfied (\(t_{d}\) is a predefined threshold depth value), this indicates that the new pixel \(x_{j,t1}\) belongs to the foreground objects, it will be discarded, and background distribution will not be updated. 2. If \(\left|d_{j,t_{1}}-d_{j}\ \right|<t_{d}\) is verified, \(x_{j,t1}\) is searched to match with \(K\) Gaussian models. From each model \(i\) from 1 to \(K\), if the condition \(\left|x_{j,t_{1}}-\mu_{j,i,t_{0}}\right|\leq 2.5\left|\sigma_{j,i,t_{0}}\right|\) is satisfied, the matching process will stop, and the matched Gaussian model will be updated as follows: \[\omega_{j,t_{1}}^{i}=(1-\alpha)\omega_{j,t_{0}}^{i}+\alpha,\] (10) \[\mu_{j,i,t_{1}}=(1-\rho)\mu_{j,i,t_{0}}+\rho\cdot x_{j,t_{1}},\] (11) \[\sigma_{j,i,t_{1}}^{2}=(1-\rho)\sigma_{j,i,t_{0}}^{2}+\rho\big{(}x_{j,t_{1}}- \mu_{j,i,t_{1}}\big{)}^{2},\] (12) \[d_{j}=\frac{d_{j}+d_{j,t_{1}}}{2},\] (13) where \(\alpha\) is the model learning rate (\(\alpha=0.01\)), and \(\rho=\alpha/\omega_{j,t_{0}}^{i}\). The other parameters of the Gaussian models remain unchanged except: \[\omega_{j,t_{1}}^{i}=(1-\alpha)\omega_{j,t_{0}}^{i},\] (14) These two parameters reflect the rate of model convergence. If pixel \(x_{j,t_{1}}\) fails to match all the current Gaussian models, a new Gaussian model is introduced to evict the Gaussian model with the smallest \(\omega/\sigma\) value. The mean and variance values of the other Gaussian models remain unchanged, while the new model is set with \(\mu_{j,t_{1}}=x_{j,t_{1}},\ \ \sigma_{j,t_{1}}=30\), \(\omega_{j,t_{1}}=0.01\). Finally, the weights of \(K\) Gaussian models are normalized to \(\sum\limits_{i=1}^{K}\omega_{j,t_{1}}^{i}=1\). 3. In the third situation, if the condition \(d_{j}-d_{j,t1}\ >t_{d}\) is satisfied, this indicates that the new input pixel \(x_{j,t1}\) belongs to the background and the previous Gaussian distributions need to be abandoned. The first step is executed for \(x_{j,t1}\). 3. **Convergence. ** The remaining frames are processed by repeating step 2. The value of background pixels is derived by \(\mu\), and the most stable pixels in the time domain are modeled as background image; meanwhile, the number of Gaussian models of each pixel is obtained to determine whether the pixel experiences similar intensities over time or not. Figure 3 shows two examples of adaptive background modeling. Figure 3a presents the _Ballet_ background image generated with a small baseline, where _cam03_ is chosen as a target virtual viewpoint that is interpolated by the reference viewpoints _cam02_ and _cam04_. Figure 3b presents the _Breakdancers_ modeling result, where the background image at virtual viewpoint _cam04_ is projected from reference viewpoints _cam02_ and _cam06_. Although some foreground objects are stored in a stable temporal background reference using the mechanism of the proposed framework, these effects would not affect the quality of the final synthesized image, since the filling of remaining empty regions always occurs in the background areas. Thus, the temporal stable background information can be obtained by both large and small baseline instances. This adaptive background modeling approach can be widely adopted in applications with unchanged scenes.