Overview of current development in depth map coding of 3D video and its future

IET Signal ProcessingVolume 14, Issue 1 p. 1-14 Review Article Free Access Overview of current development in depth map coding of 3D video and its future Yui-Lam Chan, Corresponding Author enylchan@polyu.edu.hk Department of Electronic and Information Engineering, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, People's Republic of ChinaSearch for more papers by this authorChang-Hong Fu, School of Electronic and Optical Engineering, Nanjing University of Science and Technology, No. 200 Xiao Ling Wei Street, Nanjing, People's Republic of ChinaSearch for more papers by this authorHao Chen, School of Electronic and Optical Engineering, Nanjing University of Science and Technology, No. 200 Xiao Ling Wei Street, Nanjing, People's Republic of ChinaSearch for more papers by this authorSik-Ho Tsang, Department of Electronic and Information Engineering, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, People's Republic of ChinaSearch for more papers by this author Yui-Lam Chan, Corresponding Author enylchan@polyu.edu.hk Department of Electronic and Information Engineering, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, People's Republic of ChinaSearch for more papers by this authorChang-Hong Fu, School of Electronic and Optical Engineering, Nanjing University of Science and Technology, No. 200 Xiao Ling Wei Street, Nanjing, People's Republic of ChinaSearch for more papers by this authorHao Chen, School of Electronic and Optical Engineering, Nanjing University of Science and Technology, No. 200 Xiao Ling Wei Street, Nanjing, People's Republic of ChinaSearch for more papers by this authorSik-Ho Tsang, Department of Electronic and Information Engineering, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, People's Republic of ChinaSearch for more papers by this author First published: 01 February 2020 https://doi.org/10.1049/iet-spr.2019.0063Citations: 2 AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract 3D videos have attracted attention from academia and industry after great success in the film industry. Multiview video plus depth (MVD) is the most popular 3D video format to provide vivid 3D feeling and has been adopted as an international 3D video coding standard, namely 3D extension of high efficiency video coding (HEVC). MVD includes a limited number of textures and depth maps to synthesise virtual views. In MVD, depth samples describe the distance between a camera and an actual object as a grey-level image. The characteristics of depth maps are quite different from texture images. Consequently, new coding tools are designed for depth maps in 3D-HEVC to improve the coding efficiency at the expense of high-computational complexity, which faces great challenges in coding systems. Depth map coding is also an important technique in immersive media to support three degrees of freedom 3DoF+/6DoF applications such as virtual reality/augmented reality. The study starts with an overview of what has been done over the last decade in 3D-HEVC, especially depth map coding, regarding theories, methodologies, current research and state-of-the-art fast approaches. Following this, a comprehensive comparison of the reviewed techniques is presented, and an outlook on its future trends is provided. 1 Introduction 3D videos provide an exciting viewing experience with the illusion of depth perception. It is not limited to a theatre nowadays. It also becomes increasingly interesting for home entertainment due to the proliferation of 3D television, Blu-ray 3D, virtual reality (VR) headsets etc. Pervasive applications include immersive video conference developed by Heinrich Hertz Institute [1], 3D movie, Microsoft motion-sensing game (XBOX) [2] etc. In principle, 3D scene perception can be achieved by presenting two different videos to the viewer's left and right eyes simultaneously via 3D glasses [3]. Owing to the user habit, the requirements of the 3D video in the home, however, are quite different from the theatre scenario, where an audience sits in a chair for a limited amount of time and paying full attention to the movie. In the theatre, stereo content is sufficient enough and the audience wears 3D glasses for watching. In contrast, there is a great variety of 3D displays designed for home users, starting from traditional two-view stereo displays with glasses to more sophisticated auto-stereoscopic displays without glasses. In the auto-stereoscopic display, more than two views are displayed at a time. Recent auto-stereoscopic displays are capable of displaying N different views at the same time, of which only a stereo pair is visible from a specific location. We believe the auto-stereoscopic display will become dominant in a living home environment in the coming future. To support this, multiple views are either provided as input to the display or these views are synthesised locally with the help of depth maps at the display. Owing to limitations in the production environment and bandwidth constraints [4], the latter approach is the current trend where only limited views and their corresponding depth maps are captured, compressed, and transmitted for 3D video applications [5, 6], and this 3D video format is referred to as multi-view video plus depth (MVD) [7, 8]. With the aid of depth information, it can synthesise an arbitrary number of views from these limited views via the depth-image-based rendering (DIBR) technology [9] at the decoder side. In March 2011, MPEG issued a call for proposals (CfP) on 3D video technology [6] based on the state-of-the-art high-efficiency video coding (HEVC) standard [10, 11]. The responses and subjective tests demonstrated the superiority of MVD, and it was then adopted as the 3D extension of HEVC (3D-HEVC). Consequently, the ISO/IEC MPEG and ITU-T Video Coding Experts Group standardisation bodies established the Joint Collaborative Team on 3D Video Coding Extension (JCT-3V) to develop the next generation of an international 3D video coding standard in July 2012. Based on the HEVC [11], the first 3D-HEVC reference software was contributed by proponents that achieve the best performance to the CfP [6, 12]. In February 2015, 3D-HEVC was finally included in the third version of HEVC which provides the joint coding tools of texture and depth map for various advanced 3D displays [13]. In MVD, a depth map is represented by a grey-level image, as shown in Fig. 1, which captures the distance between a camera and an actual object. By representing the depth information, a depth map plays a key role in the DIBR process and should be carefully coded. It is observed that the characteristics of a depth map are quite different from those of a texture image. First, a depth map has mostly smooth regions delimited by sharp edges [7, 14]. Second, the distortion of sharp edges induces ringing artefacts at object boundaries in synthesised views [15]. Different from the 2D video, preserving the sharp edges rather than the visual quality becomes the most critical task for depth map coding in 3D-HEVC. Fig. 1Open in figure viewerPowerPoint Example of MVD video (a) Texture video, (b) Depth map Depth map coding in 3D-HEVC adopts a quadtree-based coding tree unit (CTU) structure in HEVC. The CTU is a basic unit and is divided into several coded units (CUs) that can be represented by a recursive quadtree structure. An optimal CU partition within a CTU is a combination of different square sizes or depth level (DL) ranging from 64 × 64 (DL = 0) to 8 × 8 pixels (DL = 3). The CU is then split into several prediction units (PUs), where the best prediction mode for each PU is selected from many candidates. In addition to the coding framework of HEVC, 3D-HEVC introduces many new coding tools especially for depth maps, which will be described in Sections 2 and 3. These new coding techniques in 3D-HEVC yield significant coding efficiency and provide the outstanding perceptual quality of synthesised views. However, the coding complexity for depth maps is increased drastically, and fast depth map coding techniques are desirable for making 3D-HEVC practice. The rest of this paper is organised as follows. The basic concept of the 3D-HEVC is given in Section 2. Section 3 then describes various coding techniques for depth maps in 3D-HEVC in detail. Section 4 introduces the recent fast algorithms for depth map coding. The emerging learning-based fast coding methods are then discussed in Section 5, together with our proposed decision tree-based depth coding algorithm. Finally, Section 6 depicts an outlook on the use of depth maps in immersive video coding. 2 System structure of 3D-HEVC In this section, the MVD data format representing 3D video as well as the corresponding transmission system is described in Section 2.1. The 3D-HEVC coding system designed for MVD videos is then introduced in Section 2.2. 2.1 Transmission of 3D video The multi-view video (MVV) data format is always used for 3D video representing, where each view requires one camera to capture the scene independently. Although the MVV data format can effectively represent one N-view video, the amount of data is extremely large to represent a remarkable number of views. In recent years, DIBR technology is designed and introduced to synthesise virtual views. With DIBR, not all views must be transmitted since additional views can be synthesised by the limited textures and depth maps. As a result, MVD, which is composed of a small number of captured views with their associated depth maps, has replaced MVV due to its superior coding efficiency for 3D video representation. For one MVD video, a limited number of textures and their associated depth maps, as well as the camera parameters, are coded and multiplexed into one bit stream. After transmission, the additional views can then be synthesised at the decoder side. The basic structure of the 3D video transmission system using the MVD data format is illustrated in Fig. 2. In general, one MVD video consists of multiple textures, the corresponding depth maps, and the camera parameters. These signals are all set as input signals of the 3D-HEVC encoder. After the coding process, the bit stream referring to the whole MVD is then sent to the transmission system. It is noted that each transmitted bit stream packet contains a header that signals the type of information of the current packet. Therefore, based on header information, the 3D transmitted bit stream can be extracted and used for heterogeneous displays such as a 2D display, a stereo display, and an N-view display. This system structure facilitates the format scalability. For instance, if the bit stream is received by an N-view display, the input textures, depth maps, and camera parameters are reconstructed by the 3D-HEVC decoder. The DIBR technology using the reconstructed signals can then synthesise the virtual intermediate views for the N-view display. Here, the N-view display can be considered as an auto-stereoscopic display, where the two views for the stereo video can be selected by an audience. On the other hand, if the 3D receiver is replaced by a stereo or 2D receiver, the required bit stream can be extracted for the corresponding stereo or 2D display. Fig. 2Open in figure viewerPowerPoint Overview of the system structure and the data format for the transmission of 3D video 2.2 3D-HEVC codec structure The 3D-HEVC encoder in Fig. 2 is designed to encode MVD data. It deals with several views and associated depth maps at the same time instant rather than one single video picture in HEVC. Fig. 3 describes the basic codec structure of 3D-HEVC. In principle, each texture or depth signal is coded by an encoder inherited from the technology of HEVC. The bit stream packets referring to each signal component are then multiplexed together to form the 3D-HEVC bit stream. There is a base/independent view among all views, which is first coded by a fully HEVC compliant codec in order to support format scalability. Other views are dependent views and coded by a modified HEVC codec. As shown in Fig. 3, the independent view can be used as the reference for the dependent views, which is called the inter-view prediction technique. Besides, at each view, the texture picture has an associated depth map capturing the same scene from the same camera position. Thus, 3D-HEVC designs several inter-component techniques for reducing the redundancy between the textures and associated depth maps. Both inter-view techniques and inter-component techniques are marked as the red lines in Fig. 3. Owing to the special characteristics of depth maps, several additional coding tools are designed for depth maps in 3D-HEVC. Detailed descriptions of the depth map coding techniques in 3D-HEVC will be discussed in the next section. Fig. 3Open in figure viewerPowerPoint Basic codec structure of 3D-HEVC with the inter-component prediction (red arrows) 3 Depth map coding techniques Depth map coding is the key factor of 3D-HEVC, and its framework is mostly inherited from the texture video coding in HEVC or 3D-HEVC, such as quadtree coding structure, inter-view coding tools etc. The inter-view coding tools, including disparity-compensated prediction, inter-view motion prediction, are designed for reducing the redundancy between different views. These inherited techniques will not be described in this study and the interested reader is referred to [16]. In the following, we will focus on the additional coding tools for depth maps in 3D-HEVC. Instead of traditional rate-distortion optimisation (RDO) used in HEVC, a new optimisation method considering the synthesised view quality for depth coding in 3D-HEVC will be described in Section 3.1. The inter-component techniques that reduce the redundancy between depth maps and their associated texture videos will be discussed in Section 3.2. In Section 3.3, we will describe the new depth intra coding tools for better representing the characteristics of depth maps. 3.1 View synthesis optimisation (VSO) Depth maps are transmitted for view synthesis instead of directly viewed by audiences. In this situation, the reconstructed depth map quality cannot guarantee the synthesised view quality from the traditional RDO. Therefore, the VSO scheme [17-20] was designed in 3D-HEVC to look for the optimal mode in depth map coding. Different from RDO, the VSO scheme considers both the synthesised view and depth map quality. The VSO cost is computed by the distortion plus the Lagrangian multiplier times the coding rate as follows: (1)where is one of the possible candidate modes at the DL. In VSO, takes both the distortion of the synthesised view, , and the distortion of the depth map, , into account which is formulated as (2)where and are the weighting factors of and , respectively. is calculated by the sum of squared error or sum of absolute Hadamard transform difference between the original and reconstructed depth maps with the chosen mode. The way to compute can either be computed by the rendering approach or the non-rendering approach. The synthesised view distortion change (SVDC)-based VSO defined in [21] is considered as the rendering approach, which directly performs view synthesis using the encoded data. As shown in Fig. 4, is represented by the change of overall distortion of a synthesised view depending on the change of the depth data within a coding block being tested (3)where refers to the whole original synthesised view, and means the sample position in . is synthesised from the original video, D, and depth data, T. is the synthesised view constructed by the reconstructed video and depth values for already encoded block before the current block is determined, and D', respectively, in Fig. 4. is similar except that the reconstructed depth values from the mode, , under test are used for view synthesis. Since the view synthesis is included in SVDC, it induces high-computational complexity. Fig. 4Open in figure viewerPowerPoint Definition of the SVDC related to the distorted depth data of the block depicted by the hatched area which is the candidate mode under test; DIBR is used for view synthesis and SSD stands for the sum of squared differences The non-rendering approach, on the other hand, is the model-based view synthesis distortion (VSD) [16], which weights the depth distortion with the sum of absolute horizontal gradients of the co-located texture as (4)where means the sample position in a PU. and indicate the original and reconstructed depth data in a PU, respectively. indicates the reconstructed texture data. Also, is the proportional coefficient determined by the depth distance from the camera. In comparison with the SVDC-based VSO, the VSD-based VSO requires less computational complexity, but only gives lower accuracy because the calculation of does not actually carry out view synthesis. 3.2 Inter-component techniques In 3D-HEVC, the inter-component techniques allow the depth coding to use the already coded data from the associated texture video from the same view. The motion parameter inheritance (MPI) [22, 23] and the depth quadtree prediction (DQP) [24] can be classified in this category. 3.2.1 Motion parameter inheritance (MPI) The motion characteristics of the depth map and their associated texture video are very similar because they project the same scene from the same viewpoint at the same time. MPI is therefore introduced in 3D-HEVC, where the motion parameters from the texture video are allowed to be inherited as one candidate of merge mode in the corresponding depth block. For each depth block, it is adaptively determined whether the co-located block of the associated video is added into the candidate list of merge mode or not. In other words, MPI mode is used to extend the candidate list of merge mode in which the first candidate refers to merging the co-located block from the associated video. The advantage of using the merge mode syntax is to allow efficient signalling in the case where MPI is adopted without coding a residual signal. It is noted that quarter-pixel accuracy is used in the motion vectors of texture videos, while the depth map only utilises full-pixel accuracy. Therefore, the inherited MPI candidates are quantised to their nearest full-sample position in the depth map coding. 3.2.2 Depth quadtree prediction (DQP) In spite of representing the same scene from the same camera, the texture video always contains more details than its associated depth map. Consequently, the corresponding quadtree structure of texture video tends to be more complicated than that of the depth map. DQP performs a prediction of the depth quadtree from the corresponding texture quadtree. In general, the partition level of each CU in a depth map cannot be larger than that of the collocated texture blocks. Owing to this observation, the possible depth PU partitions are limited to a small range compared with texture PU partitions. Without DQP, there are a totally eight possible partitions for each depth PU: 2N × 2N, N × N, N × 2N, 2N × N, 2N × nU, 2N × nD, nL × 2N, nR × 2N. As shown in Fig. 5, the limited possible depth PU partitions are listed in the right according to the collocated texture PU partitions that have been determined in the left. For example, if the collocated texture PU is selected as 2N × 2N, the depth PU is decided as 2N × 2N without checking all other PU partitions. With DQP, the coding complexity from PU or CU partition can be remarkably reduced. At the same time, the optimal partition may be skipped in some cases resulting in an inaccurate prediction. In order to avoid quality degradation in key reference frames, DQP is only applied in inter slices that do not belong to random access pictures. Fig. 5Open in figure viewerPowerPoint Texture partitions and corresponding possible candidates of depth partitions 3.3 Depth intra coding tools As mentioned in Section 1, depth maps have large flat areas delimited by sharp edges. Preserving sharp edges in depth maps is the most critical task. Consequently, the investigations into alternative depth intra coding tools were carried out. The new tools include depth intra skip (DIS) mode [25], depth modelling mode (DMM) [18], and segment-wise depth coding (SDC) [26]. DIS is designed for flat areas, while DMM and SDC help to preserve the sharp edges in depth maps. 3.3.1 Depth intra skip (DIS) DIS mode is a new intra mode especially for depth map coding, which is quite useful in coding flat regions. As shown in Fig. 6, DIS directly uses the reconstructed value(s) of some designated spatial neighbouring pixel(s) to represent the current CU. In DIS, no prediction residual is encoded, and this is the major difference between DIS and other intra modes. Fig. 6Open in figure viewerPowerPoint Examples of CUs predicted by DIS (a) Vertical mode (Type 0), (b) Vertical mode (Type 1), (c) Horizontal mode (Type 2), (d) Horizontal mode (Type 3) There are two vertical modes and two horizontal modes of DIS, which is signalled by an index (DIS type 0–3). The vertical DIS mode indexed by type 0 and type 1 is shown in Figs. 6 a and b, where the neighbouring reference pixels are above the current block. Type 0 in Fig. 6 a constructs a predicted CU with one single depth value from the mid-above pixel, while Type 1 in Fig. 6 b is the same as the vertical angular mode in terms of block prediction. Besides, the two horizontal DIS types with left-neighbouring reference pixels are also described in Figs. 6 c and d. Type 2 in Fig. 6 c constructs a predicted CU with one single depth value from the mid-left pixel, while Type 3 in Fig. 6 d is the same as the horizontal angular mode in terms of block prediction. For each DIS mode type, the VSO cost is separately calculated without residual coding. Finally, the DIS type with the minimum VSO cost using (3) is determined as the best DIS type for the current CU. 3.3.2 Depth modelling mode (DMM) Different from the conventional HEVC intra modes (CHIMs) consisting of planar, DC, and 33 angular modes as illustrated in Fig. 7 a, DMM does not use the reconstructed neighbouring pixels. DMM in Fig. 7 b is a new intra mode to better describe the sharp edges in a depth map. Two regions (P1 and P2) are generated from flexible non-rectangle partition, by varying the wedgelet separation line LSE. Each segment is predicted by a constant pixel value (CPV). As shown in Fig. 7 b, for each partition of DMM, S and E denote the start and end points of LSE, respectively. As S and E move around the block boundary, the total number of possible divisions' increases significantly, which results in a huge number of wedgelet patterns in DMM. Since it is necessary to find the best wedgelet pattern as a prediction mode, the intra mode decision obviously becomes very time-consuming. Fig. 7Open in figure viewerPowerPoint Depth intra modes in 3D-HEVC (a) 35 HEVC traditional intra modes, (b) DMM 3.3.3 Segment-wise depth coding (SDC) After prediction, residuals are encoded by subtracting the prediction from the original pixel values. In residual coding, SDC is raised as an alternative coding method in 3D-HEVC. For natural videos, discrete cosine transformation and quantisation (QDCT) is applied to residual blocks after inter or intra prediction. In depth map coding, some non-zero residuals are always distributed along the sharp edge and the traditional QDCT may degrade the sharp edge due to the loss of high-frequency QDCT coefficients. SDC allows using only a CPV to represent each segment instead of QDCT. It is performed in the pixel-domain and can be applied to CHIMs and DMM. The residual block is regarded as one segment for CHIMs, while it is composed of two segments for DMM. In SDC, the residual signals of each segment are calculated by the delta CPV, , which is defined as (5)where denotes the original CPV (i.e. DC value of original block) and is the predicted CPV. For CHIMs, is the mean value of four corner pixels of the predicted block, while it is predicted from the neighbouring pixels for DMM. An offset around (five candidates with offsets of 0, −1, 1, 2, −2) is searched based on rendering-based VSO to provide an accurate description. After that, the rectified delta CPVs are signalled in the bit stream. Moreover, SDC uses the depth lookup table (DLT) to encode each CPV and its predicted CPV for reducing bitrate further [26]. Owing to the characteristics of depth maps, only a small amount of depth values is utilised from the full depth range of 0–255. The DLT only uses a restricted set of the valid depth values, and they are mapped into an index table. 4 Fast algorithms for depth maps With the additional coding tools in 3D-HEVC, depth map coding induces huge computational complexity. Therefore, the development of fast algorithms becomes crucial in 3D-HEVC. In this section, two main categories of the current fast algorithms are introduced. They are fast PU mode decision and fast CU size decision. 4.1 Fast algorithms for PU mode decision For PU inter mode decision, the coding techniques for depth maps do not have much difference from those for texture videos in 3D-HEVC. We thus do not introduce the techniques related to PU inter mode decision in this study. Interested readers may refer to [16]. In contrast, a lot of current research works pay more attention to the PU intra mode decision for depth maps, where some additional coding tools are added. In Section 4.1.1, the PU intra mode decision process for depth maps in 3D-HEVC, as well as a rough complexity analysis based on the reference software HTM-16.0 [27], are described. Then, the recent fast algorithms designed for depth intra mode decision are introduced in Section 4.1.2. 4.1.1 PU intra mode decision The implementation of PU intra mode decision includes three major parts: DIS, intra 2N × 2N and intra N × N, as shown in Fig. 8. In DIS, the DIS type with the minimum using SVDC-based VSO using (3) is determined as the best DIS type for the current 2N × 2N PU. In intra 2N × 2N or intra N × N, the best mode from CHIMs and DMM mentioned above are selected, in which the detailed process can be summarised in the following five steps: Step 1: Rough mode decision (RMD) [16] is used to choose a number of effective candidates from CHIMs by minimising in (1), where is calculated by the model-based VSD in (4) and includes only the bits for each prediction mode. These effective candidates (three modes for 64 × 64, 32 × 32 and 16 × 16 PUs, and eight modes for 8 × 8 and 4 × 4 PUs) are then put into the candidate pool for further evaluation. In addition, the three most probable modes (MPMs) from the left and up neighbouring PUs are appended as candidates in the pool. Fig. 8Open in figure viewerPowerPoint PU intra mode decision for depth map in 3D-HEVC Step 2: The optimal wedgelet partition among all the DMM candidates is sought by minimising using model-based VSO in (3). Then the optimal DMM wedgelet partition is added into the candidate pool. Step 3: using SVDC-based VSO in (3) is computed for each candidate in the candidate pool from steps 1 and 2. At this stage, includes the bits for both the prediction mode and residuals. It is noted that the residuals are coded with the traditional QDCT with limited residual quad-tree (RQT) [16]. Step 4: using SVDC-based VSO in (3) is calculated again for each candidate in the pool with SDC as the residual coding approach. Step 5: from steps 3and 4 are compared to select the final optimal intra mode. With the optimal intra mode, the full RQT is searched to determine an optimal transform kernel size. As shown in Fig. 8, the corresponding optimal cost in step 5 is further compared with the VSO cost of DIS to obtain the final optimal intra mode. From step 1 to step 5, the main idea is to construct a candidate pool and then select the optimal intra mode from the pool that is inherited from the texture intra coding. However, due to the VSO calculation and DMM or SDC tools designed for depth maps, depth intra coding is far more complicated than texture intra coding. To study the coding complexity of depth intra coding in detail, we did some statistical analysis of the encoding time occupation for each step of depth intra decision in Fig. 9. It is noted that all the experiments were conducted with HTM-16.0 under all-intra configuration. The test sequences include all eight sequences recommended in the common test condition (CTC) [28]. As shown in Fig. 9, DIS calculation only occupies about 4.48% of the total depth intra coding time, while the RMD and MPM search also takes up a small proportion of 6.02%. DIS only has four types and does not need residual coding. RMD has 35 intra modes to consider but employs the non-rendering method for calculating . Also, the MPM search does not include the ca