View synthesis with sparse light field for 6DoF immersive video

ETRI JournalVolume 44, Issue 1 p. 24-37 SPECIAL ISSUEFree Access View synthesis with sparse light field for 6DoF immersive video Sangwoon Kwak, Sangwoon Kwak orcid.org/0000-0002-9295-8966 Media Research Division, Electronics and Telecommunications Research Institute, Daejeon, Rep. of KoreaSearch for more papers by this authorJoungil Yun, Corresponding Author Joungil Yun sigipus@etri.re.kr orcid.org/0000-0002-9848-3041 Media Research Division, Electronics and Telecommunications Research Institute, Daejeon, Rep. of Korea Correspondence Joungil Yun, Media Research Division, Electronics and Telecommunications Research Institute. Daejeon, Rep. of Korea. Email: sigipus@etri.re.krSearch for more papers by this authorJun-Young Jeong, Jun-Young Jeong Media Research Division, Electronics and Telecommunications Research Institute, Daejeon, Rep. of KoreaSearch for more papers by this authorYoungwook Kim, Youngwook Kim Department of Computer Science and Engineering, Sogang University, Seoul, Rep. of KoreaSearch for more papers by this authorInsung Ihm, Insung Ihm Department of Computer Science and Engineering, Sogang University, Seoul, Rep. of KoreaSearch for more papers by this authorWon-Sik Cheong, Won-Sik Cheong Media Research Division, Electronics and Telecommunications Research Institute, Daejeon, Rep. of KoreaSearch for more papers by this authorJeongil Seo, Jeongil Seo Media Research Division, Electronics and Telecommunications Research Institute, Daejeon, Rep. of KoreaSearch for more papers by this author Sangwoon Kwak, Sangwoon Kwak orcid.org/0000-0002-9295-8966 Media Research Division, Electronics and Telecommunications Research Institute, Daejeon, Rep. of KoreaSearch for more papers by this authorJoungil Yun, Corresponding Author Joungil Yun sigipus@etri.re.kr orcid.org/0000-0002-9848-3041 Media Research Division, Electronics and Telecommunications Research Institute, Daejeon, Rep. of Korea Correspondence Joungil Yun, Media Research Division, Electronics and Telecommunications Research Institute. Daejeon, Rep. of Korea. Email: sigipus@etri.re.krSearch for more papers by this authorJun-Young Jeong, Jun-Young Jeong Media Research Division, Electronics and Telecommunications Research Institute, Daejeon, Rep. of KoreaSearch for more papers by this authorYoungwook Kim, Youngwook Kim Department of Computer Science and Engineering, Sogang University, Seoul, Rep. of KoreaSearch for more papers by this authorInsung Ihm, Insung Ihm Department of Computer Science and Engineering, Sogang University, Seoul, Rep. of KoreaSearch for more papers by this authorWon-Sik Cheong, Won-Sik Cheong Media Research Division, Electronics and Telecommunications Research Institute, Daejeon, Rep. of KoreaSearch for more papers by this authorJeongil Seo, Jeongil Seo Media Research Division, Electronics and Telecommunications Research Institute, Daejeon, Rep. of KoreaSearch for more papers by this author First published: 24 February 2022 https://doi.org/10.4218/etrij.2021-0205 Funding information: Institute for Information Communications Technology Planning and Evaluation (IITP), Grant/Award Number: 2017-0-00072 AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Virtual view synthesis, which generates novel views similar to the characteristics of actually acquired images, is an essential technical component for delivering an immersive video with realistic binocular disparity and smooth motion parallax. This is typically achieved in sequence by warping the given images to the designated viewing position, blending warped images, and filling the remaining holes. When considering 6DoF use cases with huge motion, the warping method in patch unit is more preferable than other conventional methods running in pixel unit. Regarding the prior case, the quality of synthesized image is highly relevant to the means of blending. Based on such aspect, we proposed a novel blending architecture that exploits the similarity of the directions of rays and the distribution of depth values. By further employing the proposed method, results showed that more enhanced view was synthesized compared with the well-designed synthesizers used within moving picture expert group (MPEG-I). Moreover, we explained the GPU-based implementation synthesizing and rendering views in the level of real time by considering the applicability for immersive video service. 1 INTRODUCTION The COVID-19 pandemic has brought us a rapid transition toward a nonface-to-face life environment, such as video conferencing and distance learning, that is built by fusing various digital techniques. Moreover, as indoor leisure time inevitably increases, the demand for multimedia services, such as movies and performance sport, has also explosively grown. Recently, a distinct type of media platform called "Metaverse," which supports a virtual interactive environment with another level of immersion by combining virtual reality (VR), 3D modeling, and so on, has gained worldwide attention. Thus, there has been a rapid interest in immersive videos that provide more real-life media experiences compared with traditional 2D videos. Along with such a trend, key technologies for acquiring, compressing, and generating an immersive video are being actively studied more than ever before [1-6]. Two essential functionalities of an immersive video are stereopsis and motion parallax. Stereopsis enables perceiving object structure as 3D by separately delivering two slightly offset images to the left and right eyes, whereas motion parallax is the feature that displays proper viewpoint images by random movement alike in real life. To support media experience with such features, densely collected images corresponding to all viewpoints are needed. However, since it is practically impossible to acquire and store such huge data in advance, the virtual view synthesis technique that produces virtual images corresponding to any position and orientation using a sparse set of acquired images is very necessary. Shum and Kang [7] classified virtual view synthesis within the field of image-based rendering (IBR) into two types based on whether geometry information is employed or not. The first type is the ray-based approach, such as light field rendering (LFR) [8] or lumigraph [9], which does not use any geometry but only interpolates rays from acquired sample sets. The second type is the warping-based approach, such as view-dependent texture mapping or depth image-based rendering (DIBR), which warps input images to specific positions using their corresponding depth maps. The original LFR proposed by Levoy and Hanrahan [8] mainly focused on efficient parameterization and accurate reconstruction of rays only from the set of light field images. However, throughout later studies, there have been several attempts to enhance the rendering performance of sparsely acquired light field images using geometry proxies [2, 10, 11]. Therefore, the separation criteria described in Shum and Kang [7] may not be fully valid currently, but notable distinctions still exist between the ray-based and warping-based approaches in photo-realism and degrees of freedom (DoF). The photo-realistic feature of virtual views, such as expressing the effects of light reflection, is essential for providing a distinctive media experience. The ray-based approach that interprets the scene information with the unit of light in 3D space rather than pixels in 2D images has been the more preferred method for photo-realism. Buehler and others [10] considered both the equivalent ray consistency and minimal angular deviation of the ray as substantial properties of their algorithm. Also, Overbeck and others [2] introduced a disk-based method that employs a weighted kernel parametrized by ray position and direction. However, such a ray-based method requires a large volume of light information to sufficiently cover a fair amount of viewing space. Therefore, the range for experiencing motion parallax is considerably limited compared with the data size to be acquired and processed. Recently, there is remarkable progress in learning-based volume rendering using multilayer perceptrons (MLPs) to represent the scene as a 5D radiance function, for example, NeRF [6] and its follow-ups [12, 13]. This method has shown distinguishing results on photo-realistic virtual view synthesis. However, this approach takes tens of hours of GPU computation time to train the MLPs per scene and several minutes to render only a single frame image with the trained model. This time aspect prevents the real-time rendering of a high-resolution immersive video with equal or higher than full HD quality. Although recent work [14] has improved the rendering speed by subdividing the scene into a grid of voxels and using thousands of smaller MLPs for each voxel, it remains short of real-time applications. DIBR, however, can synthesize virtual views using a relatively small number of input images and their corresponding geometry. This method, which operates using depth-based 3D warping, was initially developed along with the growth of technical maturity of free-viewpoint TV (FTV) standard conducted by the moving picture expert group (MPEG) [15-18]. FTV aimed to provide limited motion parallax with stereopsis and unidirectionality. Thus, DIBR for providing motion parallax with a multibaseline camera has been mainly studied with respect to pixel-wise warping, blending, and hole filling [19, 20]. To go one step further from FTV, under the project scope of MPEG immersive (MPEG-I), the MPEG video coding group has been working on the standard named MPEG immersive video (MIV) since 2018. MIV is the compression standard developed for handling the data format expressed in multiview video + depth (MVD), which has been considered a potential candidate for supporting 6DoF immersion within a limited space [21]. However, since MVD consists of multiple pairs of videos with color and depth, its data size hinders appropriate handling by existing multimedia service workflows. By tackling this issue, MIV has been developed as the primary standard for allowing efficient compression of MVD with a suitable amount of data with minimal quality loss. Besides compression, another essential factor affecting the immersion level is the view synthesis technique. As MVD only contains sampled data of the 3D scene acquired using sparsely deployed cameras, it is impossible to support continuous and seamless experiences using those data. Thus, intermediate views are generated with the given information. Since the degree of immersion strongly relies on the quality of synthesized views, a significant effort is required for properly designing the view synthesizer. Therefore, multiple view synthesis packages have been proposed as reference tools and used by the MIV group. During the initial stage of standardization, two tools, reference view synthesizer (RVS) and versatile view synthesizer (VVS), were used [22-24]. Both tools have the advantage that more robust views can be synthesized by accepting an unlimited number of input views acquired using unrestricted camera arrangement. However, the view weighting synthesizer (VWS), which employs a unique method for determining camera weight, was chosen as the representative synthesizer in MIV [25]. The view synthesizers listed above are all built upon the DIBR based on patch-wise warping. This solves the crack artifact problem caused by pixel-by-pixel warping from constructing a patch from the input view image and warping each patch in the case when 6DoF motion is performed. As an early approach of patch-wise warping, the concept of superpixels, which are enlarged segments composed of adjacent pixels, was considered [26, 27]. However, since superpixels are oversegmented, the geometry between pixels cannot be accurately interpolated. To properly deal with this point, triangle-based 3D warping was proposed [22, 28]. This constructs a triangular region using three adjacent pixels, warps each triangle, and performs trilinear interpolation to fill the internal color and depth. This method avoids crack-like holes from pixel-wise warping and allows more geometrically accurate interpolation than superpixels. Thus, our algorithm was built using the DIBR structure with triangle-based 3D warping similar to RVS. In this paper, we propose an enhanced virtual view synthesis algorithm using triangle-base warping, which considers both directional similarities of rays and depth distribution of warped triangles during the blending procedure. Using the proposed algorithm, it was confirmed that the quality of synthesized views improved both objectively and subjectively compared with the results of the synthesizers used in MPEG-I. Furthermore, we discussed the GPU-based implementation of the proposed synthesizer that renders views in real time. 2 BASELINE ARCHITECTURE The overall baseline architecture of the virtual view synthesis algorithm using patch-wise warping is shown in Figure 1. Color, depth, and camera parameters corresponding to a set of sparse light field images I 0 , I 1 , . . . , I N − 1 are fed as input. The entire process consists of three sequential steps, namely, triangle-based 3D warping, blending, and inpainting. The base structure of the view synthesis algorithm introduced in this section is closely aligned with RVS [23]. FIGURE 1Open in figure viewerPowerPoint Overall baseline architecture of virtual view synthesis algorithm 2.1 Triangle-based 3D warping First, a triangle defined with three vertices each corresponding to an adjacent pixel within an input view image is warped to a virtual viewpoint. Depending on the projection format of the input image, each pixel of the triangle is unprojected to the 3D space differently. Here, two projection formats are considered: perspective projection and equirectangular projection (ERP). For the image expressed in perspective projection, the unprojection equation of a pixel m = u , v in image coordinates toward a corresponding 3D point m c = X c , Y c , Z c T in camera coordinates is simply defined as follows: m c = d u , v · K c − 1 · u v 1 T , (1)where d u , v is a depth value of the pixel m c and K c is an intrinsic matrix of a given camera in homogeneous coordinates. For the ERP format, a pixel in image coordinates is converted to longitude and latitude values in spherical coordinates. Suppose that the image size is w × h and the angular ranges are θ m , θ M and ϕ m , ϕ M for the vertical and horizontal directions, respectively, then the spherical coordinates θ , ϕ of a pixel m can be calculated as follows: θ = θ m + θ M − θ m w − u w , (2) ϕ = ϕ m + ϕ M − ϕ m h − v h . (3)Thus, the ray directed from the projection center of the camera to the pixel m is obtained as r ̂ m = cos ϕ cos θ cos ϕ sin θ sin ϕ . (4)Using this ray direction, the 3D point with respect to camera coordinates can be expressed as follows: m c = r m c r ̂ m , (5)where r m c is a distance from the camera center to the 3D point m c . After the unprojection, the 3D point defined with respect to the input view's camera coordinates is transformed to world coordinates with the rotation R c and translation t c of the camera by m W = R c m c + t c . This can be represented by an extrinsic matrix E c in homogeneous coordinates, as follows: m c = R c T m W − R c T t c = E c · X W Y W Z W 1 T . (6)The sequential transform of coordinates from an input camera to the virtual camera is combined into a single affine transformation: m v = E v E c − 1 m c , where E v is an extrinsic matrix of the virtual camera. Finally, the 3D point in the virtual camera coordinates m v is projected to the image coordinates of the virtual view according to the projection format using the inverse process of the unprojection. We define the function of the entire 3D warping from the i-th input camera to the virtual camera V as τ i · : I i → V, which depends on the projection format of the input and virtual images. After the three vertices of a triangle in an input image are warped to the virtual image, the color and depth values of pixels inside the warped triangle are occupied by trilinear interpolating the color and depth values of the vertices. As shown in Figure 1, the color value of a pixel p in the i-th warped image laying inside the triangle composed of three vertices τ i m 0 , τ i m 1 , and τ i m 2 can be obtained as follows: c i warp p ⊿ m 0 m 1 m 2 = ∑ k = 0 2 s k p · c I i m k , (7)where ⊿ m 0 m 1 m 2 denotes a triangle composed of the three pixels, c I i m k is a color value of the pixel m k in the i-th input image, and s k p is a normalized area ratio of the subdivided triangle by pixel p, that is, s 0 + s 1 + s 2 = 1. Also, the depth value of the pixel d i warp p ⊿ m 0 m 1 m 2 is calculated the same way as 7. Suppose a projected triangle identical to the position of the pixel exists. The pixel quality defined by the shape of the triangle that it belongs to and its depth value are then compared to select the most suitable pixel at the position. Suppose that another triangle with three vertices τ i m 0 ′ , τ i m 1 ′ , and τ i m 2 ′ is projected to the same pixel position p in the i-th warped image. Then its color value is determined by c i warp p = c i warp p ⊿ m 0 ′ m 1 ′ m 2 ′ if q i ⊿ m 0 ′ m 1 ′ m 2 ′ p > q i ⊿ m 0 m 1 m 2 p , c i warp p ⊿ m 0 m 1 m 2 otherwise , (8)where q i ⊿ p is defined as q i ⊿ m 0 m 1 m 2 p = K − l max ⊿ τ i m 0 τ i m 1 τ i m 2 d i warp p ⊿ m 0 m 1 m 2 α . (9) l max ⊿ is the maximum side length of a triangle and K and α are the preset constant positive number. q i ⊿ p indicates a quality of the pixel p in the triangle ⊿, defined by considering the amount of triangle elongation. It is designed to have a lower value when the shape of the triangle is more stretched. This value is used during the blending process as the metric for measuring the quality of a pixel. 2.2 Blending While blending multiple warped images, a color value of each pixel within the virtual view is calculated using the weighted average of the color information of warped pixels. Let c v p denote the color value of a pixel p in a virtual image. It can be computed as follows: c v p = ∑ i = 0 N − 1 w i p · c i warp p ∑ i = 0 N − 1 w i p , (10)where w i p is a blending weight of a pixel p in the i-th warped image. Similar to the quality of a pixel described in (9), a baseline blending weight is set to w i q p , which evaluates the elongation of the triangle using the pixel in and warped depth values as follows: w i q p = K − l max ⊿ τ i m 0 τ i m 1 τ i m 2 d i warp p α . (11)The degree of triangle's elongation can be used in the blending process to select the most suitable pixel for a certain position among multiple pixels warped from different viewpoints. When input images, such as that shown in Figure 2A, are warped to a virtual viewpoint so that warped images, such as that shown in Figure 2B, are generated, some elongated triangles near the boundary of the object exist where the depth values of the triangle's vertices are different. By assigning lower weights on such elongated triangles in the subsequent blending process, the blended image with minimal artifacts, such as that shown in Figure 2C, can be obtained. Two positive effects are expected with this weight design. First, the disoccluded area near the boundary of each warped image caused by the viewpoint change can be suppressed. Second, the amount of information can be effectively maintained by trusting the area with higher pixel density than the area that is excessively interpolated. FIGURE 2Open in figure viewerPowerPoint Blending process based on triangle's elongation: (A) input images, (B) warped images, and (C) blended image 2.3 Inpainting In this paper, as a postprocess, a simple and lightweight inpainting algorithm is applied to fill the holes that still exist after the blending process. This is because the proposed view synthesis algorithm maximizes the quality of the virtual view by wisely blending a fair number of warped images, rather than using sophisticated inpainting techniques. Additionally, applying a less complex algorithm to inpainting is suitable for providing GPU-supported real-time view synthesis by minimizing the computational burden. 3 PROPOSED IMPROVEMENTS This section introduces our contributions to improving the quality of the synthesized view. In Section 3.1, we present a calibration method of triangle colorization in the warping process for virtual view images in ERP format. In Section 3.2, we present a novel blending weight using the ray directional similarity to enhance photo-realism of virtual views, such as light reflection. In the last subsection, we present another blending weight using warped density triangles along a frustum of the virtual viewpoint to remove artifacts due to an inconsistency between depth values of warped pixels. 3.1 Calibrated triangle colorization for ERP format The shape of warped triangle is determined by the spatial relationship among three vertices and the degree of position and orientation difference between input and virtual views. The baseline algorithm described in Section 2 equivalently uses triangles' elongation as a key feature for selecting the most appropriate pixels for warped images using both the perspective projection and ERP formats. However, unlike that in the perspective projection format, the triangle shape is more stretched or shrunk in the ERP format along the latitude location where the triangle is projected at. Consequently, when determining the color of triangles within the warped image, inappropriate pixels from certain background region can be selected, thereby causing artifacts near the pole region. To resolve this issue, the calibrated quality of triangles, q i calib ⊿ p , for triangle colorization was employed. Also, the sinusoidal projection [29] can be applied to obtain the modified vertices of projected triangles for calculating q i calib ⊿ p . Sinusoidal projection is one of the projection methods designed for a sphere, which projects areas of the same size on the sphere regardless of the projection location, and this nature can be used to solve the problem that occurs in ERP format; that is, the size of projected triangles varies along the latitude. Therefore, q i calib ⊿ p is calculated using the sinusoidal projection centered at the longitude of each triangle, and the comparison inequality in (8) is adjusted as follows: c i warp p = c i warp ( p | ⊿ { m 0 ′ , m 1 ′ , m 2 ′ } ) if q i calib ( ⊿ { m 0 ′ , m 1 ′ , m 2 ′ } | p ) > q i calib ⊿ m 0 m 1 m 2 p c i warp p ⊿ m 0 m 1 m 2 otherwise . (12)Note that the proposed quality measurement of the triangle is applied only during triangle colorization, not in the blending procedure. This is because the blending weight for determining the pixel color is inversely proportional to the area of the triangle projected on the warped image. Thus, the proposed method applies (12) to triangle colorization to minimize the quality variation due to the change of the latitude location in ERP format, while the quality in blending weight is used as before (11) according to the elongation of triangles after projection. 3.2 Blending weight based on directional similarity of rays As illustrated in Figure 3A, the blending process can be interpreted as the reconstruction process of a virtual ray Ray V from the center of the virtual camera V to a 3D point P using a set of rays acquired from input cameras Ray i | i = 0 , . . . , N that direct to the same point. However, in this example, Ray V is a ray equivalent to Ray 1 , so it is more valid to trust Ray 1 's information when reconstructing Ray V , rather than the entangling information from multiple rays. This feature, called epipole consistency, is the desired property that an ideal IBR algorithm should have [10]. FIGURE 3Open in figure viewerPowerPoint Ray analysis for blending weight: (A) motivation, (B) concept, and (C) expected effects in blending process From the above motivation, a true ray is defined as a ray for the virtual view, which is identical to one directly acquired by a real physical camera. However, it is practically impossible to retain all true rays existing in the scene geometry. Only sparsely sampled rays are available in sampled views. Therefore, true rays are used to reflect the amount of directional similarity when calculating blending weights. As shown in Figure 3B, the concept of the proposed ray-based blending weight is to assign higher blending weight to the rays whose directions were more similar to that of the true ray and relatively lower blending weight to those with dissimilar directions. Two advantages are expected from such applications: First, elongated triangles near edges caused by viewpoint changes can be suppressed. Second, photo-realistic aspects, such as the light reflection effect in synthesized views, can be more realistically expressed. Figure 3C shows the expected effects of the proposed blending method. For the three points P , Q , and R in the scene geometry, suppose the rays of input cameras C 2 , C 3 , and C 4 have the highest directional similarity with those of the virtual view, respectively. Then, in the blending process for obtaining the color values of the pixel virtual views p v , q v , and r v , warped pixels p 2 , q 3 , and r 4 , which correspond to C 2 , C 3 , and C 4 , respectively, have the highest blending weight based on the directional similarity. Several approaches exist for estimating the directional similarity between rays; however, we computed the area of the plane defined with three points V , C i , and P for each input camera and assigned blending weight inversely proportional to those areas. Figure 4 shows the comparison of the proposed approach and other possible approaches. The example shown in Figure 4B assigns higher blending weight to P 1 captured by C 1 than that to P 2 captured by C 2 . Suppose there was an actual camera at V; then, the color value of the ray is determined by that of P 1 . Here also, two alternative approaches were compared: an angular similarity between the ray of an input camera and the virtual ray at point P and a vertical distance from the center of each input camera to the virtual ray. If the angle is used, P 2 in the background has higher weight because θ 2 < θ 1 . If the vertical distance is considered, P 1 and P 2 are assigned with the same weight because d 1 = d 2 . However, with the proposed area-based weight, P 1 has a higher weight, which is closer to the ground truth. FIGURE 4Open in figure viewerPowerPoint Illustration of ray-based blending weight: (A) proposed approach and (B) other analogous approaches To apply the proposed ray-based blending weight to the existing triangle quality-based weight, we use simple multiplication as below: w i p = w i q p · w i r p . (13) w i r p is the blending weight based on the directional similarity of rays and is defined as follows: w i r p = 1 − λ + λ · F ray α , (14)where F ray α = 1 / exp b · A ray , b is a baseline length between cameras, and A ray is the area of the plane defined in Figure 4A. λ and α are parameters for adjusting the ratio between the weights and controlling the intensity of the proposed ray-based weight, respectively. 3.3 Blending weight based on the depth distribution of triangles When multiple input images are warped to a single virtual viewpoint, depth values of multiple pixels warped to the identical pixel position in the virtual view image are probably inconsistent with each other. For depth images obtained using multiview depth estimation, it is difficult to accurately match depth values between distant input views. Therefore, at a pixel position in a virtual view image, depth values of warped pixels are distributed along the ray corresponding to the pixel position. If warped pixels are blended without considering distribution characteristics, the synthesis quality degrades owing to inconsistencies among depth values or outliers. The problem of determining the color of a ray from the distribution of 3D points, that is, object density, has been studied in previous works on volume rendering [6, 30]. From Max [30],