# Hole Concealment Algorithm Using Camera Parameters in Stereo 360 Virtual Reality System
## 1 Introduction  
Virtual reality (VR) has been one of the most important topics in the field of multimedia signals and systems for approximately 10 years. There are many applications for 360 VR systems, such as broadcasting, movies, social media communication, remote education, and virtual tourism [1]. 360 VR images and videos are generated using a 360 VR camera rig or a spinning VR camera, which is usually expensive or consists of heavy equipment. As the technologies related to VR have been studied, methods that use a general purpose camera or cheap smartphone instead of the technical equipment to generate a 360 VR image have been developed [2; 3; 4; 5]. In [2], the authors explained a method to stitch images using direct alignment or feature-based alignment. When the feature-based alignment approach is employed, feature points are searched in each picture to be stitched using one of the related algorithms [6; 7; 8; 9; 10; 11]. Then, the feature points are matched by RANSAC [12], which provides homography matrixes to represent the relationship between neighboring pictures with overlapping common regions. Intrinsic and extrinsic matrixes of each picture can be derived from the homography matrix. The intrinsic and extrinsic matrixes of all pictures to be used for generating a VR image are jointly optimized through a bundle adjustment procedure [13]. In [3], a real time algorithm was proposed to make a panoramic image from pictures taken by the camera of a mobile phone, where the resulting image is shown on a cylindrical surface. The pictures taken by a mobile phone have different colors and brightness values compared to other pictures because these depend on the direction and position of the camera used in taking the pictures. This difference produces various artifacts in the VR image. In order to reduce the degradation, an algorithm to find the seams and blend the overlapping regions was proposed in [4]. Automatic Panoramic Image Stitching (APIS) [5] is one of the most efficient methods to make the panorama image or VR pictures. APIS consists of a variety of modules explained in [2; 7; 8; 11; 12] which include the feature-based alignment, Scale Invariant Feature Transform (SIFT), Speed-Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), Random Sample Consensus (RANSAC), bundle adjustment, straightening, and multiband blending. In order to increase the quality of the stitched VR image, we can use high-end specialized equipment, for example GoPro VR cameras. When high-end specialized equipment is used to generate 360\({}^{\circ}\) VR images, it is considerably easier to align pictures without notable seam artifacts than that when using conventional cameras. Although VR images obtained from high-end specialized equipment have higher quality than those obtained from conventional cameras, conventional cameras are more affordable and simpler to use than specialized equipment. In fact, specialized equipment is bulky and expensive, thereby hindering its use in daily life. With the increase in the viewer requirement for a sense of immersion, stereo 360 VR systems have been introduced to meet this need [14; 15]. A stereo 360 VR image provides a more realistic and immersive sense to the viewer than a monocular 360 VR image, but the cost to make the contents increases further. The high cost of the equipment to generate 360 VR images may be one of the things preventing the spread of applications based on VR technology. This paper considers a system to generate a stereo 360 VR image using two cheap smartphones or two general purpose cameras attached to a rig. In our scenario, because the general-purpose cameras are used to capture pictures, the pictures taken in arbitrary directions may produce non-overlapping regions and those become holes in the stitched image. In order to conceal the holes in stereo 360 VR images, various inpainting algorithms [16; 17; 18; 19; 20; 21; 22; 23; 24] can be considered as solutions, even though these were originally proposed for non-VR images. [PERSON] et al.