# Hole Concealment Algorithm Using Camera Parameters in Stereo 360 Virtual Reality System
## 3 Algorithm Proposed to Conceal a Hole
### Prediction for Camera Parameters  
This subsection explains the method used to predict \(K(B_{R}^{c})\), \(\text{R}(B_{R}^{c})\), \(\text{R}(B_{L}^{c})\), and \(K(B_{L}^{c})\) in order to derive \(H(B_{L}^{c}\to B_{R}^{c})\) in (11). As shown in Figure 2, because \(B_{L}^{c}\) includes a hole, \(\text{R}(B_{L}^{c})\) and \(K(B_{L}^{c})\) are unknown. In addition, because \(B_{R}^{c}\) consists of fractions of multiple pictures in the right view, \(K(B_{R}^{c})\) and \(\text{R}(B_{R}^{c})\) should be recalculated instead of using those resulting from the process to make the right ERP. In order to predict \(\{K(B_{R}^{c})\), \(\text{R}(B_{R}^{c})\), \(\text{R}(B_{L}^{c})\), \(K(B_{L}^{c})\}\), the following operations are executed. First, we estimate homography matrixes \(H(B_{L}^{l}\to B_{R}^{l})\) and \(H(B_{L}^{r}\to B_{R}^{r})\) for pairs \(\{B_{L}^{l}\), \(B_{R}^{l}\}\) and \(\{B_{L}^{r}\), \(B_{R}^{r}\}\), respectively. \(H(B_{L}^{l}\to B_{R}^{l})\) and \(H(B_{L}^{r}\to B_{R}^{r})\) are derived by applying \"feature extraction,\" RANSAC, and \"bundle adjustment\" to those pairs. Figure 6 shows the examples used to estimate \(H(B_{L}^{l}\to B_{R}^{l})\) and \(H(B_{L}^{r}\to B_{R}^{r})\), which have the following relationships. \[H(B_{L}^{l}\to B_{R}^{l}) =K(B_{R}^{l})\text{R}(B_{R}^{l})^{-1}\text{R}(B_{L}^{l})K(B_{L}^{ l})^{-1} \tag{13}\] \[H(B_{L}^{r}\to B_{R}^{r}) =K(B_{R}^{r})\text{R}(B_{R}^{r})^{-1}\text{R}(B_{L}^{r})K(B_{L}^{ r})^{-1} \tag{14}\]be represented as a combination of rotational components \(\mathrm{R}_{x}\), \(\mathrm{R}_{y}\), \(\mathrm{R}_{z}\) on the x, y, z axes as follows. \[\mathrm{R}=\mathrm{R}_{z}(\theta)\mathrm{R}_{y}(\sigma)\mathrm{R}_{x}(\phi)= \left[ \begin{array}{ccc}\cos\ \theta&-\sin\ \theta&0\\ \sin\ \theta&\cos\ \theta&0\\ 0&0&1\end{array} \right]\left[ \begin{array}{ccc}\cos\ \sigma&0&\sin\ \sigma\\ 0&1&0\\ -\sin\ \sigma&0&\cos\ \sigma\end{array} \right]\left[ \begin{array}{ccc}1&0&0\\ 0&\cos\ \phi&-\sin\phi\\ 0&\sin\ \phi&\cos\phi\end{array} \right] \tag{15}\]  
where \(\phi\), \(\sigma\), and \(\theta\) are the Euler angles about the x, y, and z axes, respectively. Applying the property to \(\{\mathrm{R}(B^{l}_{L}),\ \mathrm{R}(B^{c}_{L}),\ \mathrm{R}(B^{r}_{L}),\ \mathrm{R}(B^{l}_{R}),\ \mathrm{R}(B^{c}_{R}),\ \mathrm{R}(B^{r}_{R})\}\) gives  
\[\mathrm{R}(B^{i}_{j})\mathrm{=R}_{z}(\theta(B^{i}_{j}))\mathrm{R}_{y}(\sigma (B^{i}_{j}))\mathrm{R}_{x}(\phi(B^{i}_{j})) \tag{16}\]  
\[\mathrm{R}_{x}(\phi(B^{i}_{j}))=\left[ \begin{array}{ccc}1&0&0\\ 0&\cos\phi(B^{i}_{j})&-\sin\phi(B^{i}_{j})\\ 0&\sin\phi(B^{i}_{j})&\cos\phi(B^{i}_{j})\end{array} \right] \tag{17}\]  
\[\mathrm{R}_{y}(\sigma(B^{i}_{j}))\mathrm{=}\left[ \begin{array}{ccc}\cos\ \sigma(B^{i}_{j})&0&\sin\ \sigma(B^{i}_{j})\\ 0&1&0\\ -\sin\ \sigma(B^{i}_{j})&0&\cos\ \sigma(B^{i}_{j})\end{array} \right] \tag{18}\]  
\[\mathrm{R}_{z}(\theta(B^{i}_{j}))\mathrm{=}\left[ \begin{array}{ccc}\cos\ \theta(B^{i}_{j})&-\sin\ \theta(B^{i}_{j})&0\\ \sin\ \theta(B^{i}_{j})&\cos\ \theta(B^{i}_{j})&0\\ 0&0&1\end{array} \right] \tag{19}\]  
where i = {l, c, r} and j = {l, R}. In (16)-(19), \(\phi(B^{i}_{j})\), \(\sigma(B^{i}_{j})\), and \(\theta(B^{i}_{j})\) are the Euler angles about the x, y, and z axes in \(\mathrm{R}(B^{i}_{j})\), respectively. As observed in Figure 2, because \(B^{c}_{L}\) and \(B^{c}_{R}\) are located in the centers of \(\{B^{l}_{L}\) and \(B^{r}_{L}\}\) and \(\{B^{l}_{R}\) and \(B^{r}_{R}\}\), respectively, we can assume that their Euler angles have the following relationships. \[\phi(B^{c}_{j})=\frac{\phi(B^{l}_{j})+\phi(B^{r}_{j})}{2} \tag{20}\]  
\[\sigma(B^{c}_{j})=\frac{\sigma(B^{l}_{j})\mathrm{+}\sigma(B^{r}_{j})}{2} \tag{21}\]  
\[\theta(B^{c}_{j})=\frac{\theta(B^{l}_{j})\mathrm{+}\theta(B^{r}_{j})}{2} \tag{22}\]  
where \(j=\{L,\ R\}\). Substituting (20)-(22) into (Third, \(K(B_{L}^{c})\) and \(K(B_{R}^{c})\) are estimated from {\(K(B_{R}^{I})\), \(K(B_{L}^{I})\)} and {\(K(B_{R}^{r})\), \(K(B_{L}^{r})\)}, respectively, which were derived in the first step.