# Hole Concealment Algorithm Using Camera Parameters in Stereo 360 Virtual Reality System
## 2 Problem Formulation  
The Figure 1 shows the process used to make the stereo 360 VR images, which consist of left and right equi-rectangular projections (ERPs). In this configuration, the left and right cameras are mounted on a rig and capture pictures in the same direction, where general purpose cameras (e.g., built-in cameras in smartphones) are used to capture pictures. The left and right ERPs are generated by applying several techniques to the sets of pictures with the left and right views. For example, the left ERP is made by consecutively applying feature extraction, mapping between features, a calculation of the homography matrix to represent the relationship, bundle adjustment to optimize the camera parameters, seam finding, and blending. The right ERP image is also made by a series of the same techniques. In this paper, APIS [5] is used to make the left and right ERPs, because it is known as one of the most efficient algorithms. APIS is applied independently to the left and right sets of pictures that have been captured by two built-in smartphone cameras. In this scenario, no mechanical equipment is used, other than a tripod. Thus, the pictures are captured in arbitrary directions and positions. Note that APIS does not include a function to conceal the holes in the stitched image. APIS is the basic style algorithm to stitch the pictures.  
Figure 2 shows an example of the left and right ERPs that are generated using the procedure shown in Figure 1. As observed in Figure 2, we find some holes in these pictures. In the scenario illustrated in Figure 1, given that general-purpose cameras (instead of specialized equipment) are used, their narrow field-of-view (FOV) and the fact that pictures are often captured in arbitrary directions may produce non-overlapping regions and distortion (e.g., ghost effect or misalignment) in stitched 360\({}^{\circ}\) VR images. Non-overlapping regions also show up owing to the limitations of the stitching module, because the derivation of the relationship between spatially neighboring pictures is based on the RANSAC and feature points of each picture. The non-overlapping regions become holes in the stitched image, where the shape of the hole is dependent on the structure of the neighboring warped pictures. The shape of the holes may look like a square or a lozenge because those holes are surrounded by the neighboring warped pictures. Examples of the holes are shown in Section 4, where we can see the various shapes of holes.  
Figure 1: Procedure to make stereo 360 VR images consisting of left and right equi-rectangular projections (ERPs).  
In Figure 2, \(B_{L}^{c}\) is a block overlapping a hole in the left ERP. \(B_{L}^{l}\) and \(B_{L}^{r}\) are the left and right sides neighboring blocks of \(B_{L}^{c}\), respectively. \(B_{R}^{c}\), \(B_{R}^{l}\), and \(B_{R}^{r}\) in the right ERP are blocks corresponding to \(B_{L}^{c}\), \(B_{L}^{l}\), and \(B_{L}^{r}\), respectively. The corresponding blocks in the left and right ERPs are located in the same positions.  
This paper proposes an efficient algorithm to conceal the holes, where pixels in the holes are filled with pixels in other ERPs and the location of the pixel used is derived from the camera parameters of the neighboring blocks \(\{B_{R}^{c}\), \(B_{R}^{l}\), \(B_{R}^{r}\), \(B_{L}^{l}\), \(B_{L}^{r}\}\).