## 3 Proposed Framework
### Reference View Extension
#### 3.1.2 Extended Region Filling  
[PERSON]'s algorithm is an effective image inpainting method with three steps: priority calculation, best matching patch searching and priority updating. In this paper, the [PERSON]'s algorithm is optimized to fill different types of holes. In this section, the extended region in the depth image is filled first. Then the depth information is used to fill the extended region in the reference image.  
1 [PERSON]'s inpainting algorithm  
As shown in Figure 5, \(I\) represents the entire image, \(\Omega\) is the hole region to be filled, \(\Phi\) the source region and define as \(\Phi=I-\Omega\). \(\delta\Omega\) indicates the boundary of the hole. In order to determine the fill order, the priority of all pixels needs to be computed for all pixels on the boundary \(\delta\Omega\). In the original algorithm, the computation of the priority takes into account two factors: the amount of known information and the amount of structural information contained in the patch. For pixel \(p\) on the \(\delta\Omega\), \(\Psi_{p}\) is the square patch centered at \(p\). The priority of \(p\) is given by:  
\[P(p)=C(p)\cdot D(p), \tag{3}\]  
where \(C(p)\) is the confidence term and \(D(p)\) is the data term. They are defined as follows:  
\[C(p)=\frac{\sum_{q\in\Psi_{p}\cap\Phi}C(q)}{\left|\Psi_{p}\right|}, \tag{4}\]  
Figure 4: Extended reference image and its depth image: (**a**) Extended reference image; and (**b**) extended depth image.  
\[D(p)=\frac{\left|\
abla I_{p}^{\perp}\cdot n_{p}\right|}{\alpha}, \tag{5}\]  
where \(\left|\Psi_{p}\right|\) is the area of \(\Psi_{p}\), which means the number of pixels within the patch \(\Psi_{p}\), \(\alpha\) is a normalization factor, for a typical gray-level image \(\alpha=255\), \(n_{p}\) is a unit vector orthogonal to the front \(\delta\Omega\) in the point \(p\), \(\
abla^{\perp}\) is the direction of the isophote, and \(\perp\) denotes the orthogonal operator. It can be seen that the confidence term gives higher priority to the patches, which have more non-hole pixels. The isophote reflects the structure of the image, and therefore, the data term gives higher priority to the patches, which are on the continuation of strong edges to encourage linear structures to be synthesized first. During initialization, the function \(C(p)\) is set to \(C(p)=0\) for hole pixels in \(\Omega\), and \(C(p)=1\) for else.  
After the point \(\hat{p}\) with the highest priority is found, the patch \(\Psi_{\hat{p}}\) is ready to be filled. The original algorithm traverses the source region and searches for the patch \(\Psi_{\hat{q}}\) with the highest similarity to \(\Psi_{\hat{p}}\) by using the Sum of Squared Difference (SSD) between the two patches as the matching criterion. \(\Psi_{\hat{q}}\) is determined as follows:  
\[\Psi_{\hat{q}}=\arg\min_{\Psi_{\hat{q}}\in\Phi}\text{SSD}\big{(}\Psi_{\hat{p}},\Psi_{\hat{q}}\big{)}, \tag{6}\]  
where the calculation of SSD is done in the RGB color space and only contains non-hole pixels in the two patches.  
When the hole region of \(\Psi_{\hat{p}}\) is filled, the confidence term \(C(p)\) is updated in the region limited by \(\Psi_{\hat{p}}\) as follows:  
\[C(p)=C(\hat{p}),\forall p\in\Psi_{\hat{p}}\cap\Omega, \tag{7}\]  
the updated confidence term is used for the next calculation of priority. Then repeat the above process until all holes in the extended region are filled.