Depth map super‐resolution via shape‐adaptive non‐local regression and direction‐based local smoothness

Electronics LettersVolume 57, Issue 12 p. 475-477 Image and vision processing and display technologyOpen Access Depth map super-resolution via shape-adaptive non-local regression and direction-based local smoothness Yingying Zhang, Yingying Zhang orcid.org/0000-0002-3718-9826 School of Communication and Information Engineering, University of Electronic Science and Technology of ChinaSearch for more papers by this authorHonggang Chen, Honggang Chen College of Electronics and Information Engineering, Sichuan University, Chengdu, ChinaSearch for more papers by this authorChao Ren, Chao Ren College of Electronics and Information Engineering, Sichuan University, Chengdu, ChinaSearch for more papers by this authorCe Zhu, Corresponding Author Ce Zhu eczhu@uestc.edu.cn School of Communication and Information Engineering, University of Electronic Science and Technology of China E-mail: eczhu@uestc.edu.cnSearch for more papers by this author Yingying Zhang, Yingying Zhang orcid.org/0000-0002-3718-9826 School of Communication and Information Engineering, University of Electronic Science and Technology of ChinaSearch for more papers by this authorHonggang Chen, Honggang Chen College of Electronics and Information Engineering, Sichuan University, Chengdu, ChinaSearch for more papers by this authorChao Ren, Chao Ren College of Electronics and Information Engineering, Sichuan University, Chengdu, ChinaSearch for more papers by this authorCe Zhu, Corresponding Author Ce Zhu eczhu@uestc.edu.cn School of Communication and Information Engineering, University of Electronic Science and Technology of China E-mail: eczhu@uestc.edu.cnSearch for more papers by this author First published: 19 May 2021 https://doi.org/10.1049/ell2.12129AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract In this letter, a novel single depth map super-resolution algorithm is proposed, which combines the non-local prior and local smoothness prior. Unlike the color-guided methods, the proposed method does not need a corresponding color image to aid the depth map super-resolution. To explore the non-local self-similarity in the depth map, a shape-adaptive adjusted non-local regression is constructed using the shape-adaptive similar patch groups. This prior can make full use of the non-local information of the depth map and alleviate the effect of irrelevant pixels. To construct different local structures, the direction-based local smoothness prior to model the different directional information is proposed, which can preserve the different structures in the depth map. Compared with the state-of-the-art methods, experimental results indicate that the proposed approach can achieve better reconstruction performance. Introduction To acquire high quality depth maps, depth map super-resolution (SR) methods have been proposed. These methods can be classified into colour-guided [1-3] and non colour-guided methods [4-6]. Colour-guided methods recover depth maps by using the corresponding colour image as a powerful aid. However, in many cases, the corresponding colour images are not always available. Non colour-guided methods can infer a high-resolution (HR) depth map from a single low-resolution (LR) input without the assistance of additional intensity images. Benefit from this merit, in this letter, we focus on the non colour-guided method. Non-local self-similarity is an important feature of the depth map, which exploits the correlation between patches having similar patterns. Based on this character, many effective methods have been proposed [7, 8]. Among these methods, the non-local mean (NLM) is one of the representative works. Although the above methods exploit the correlation between patches, the non-local feature is formulated only by using the center pixel of each patch, which results in that the non-local self-similarity cannot be fully exploited. Furthermore, for the region that does not locally smooth, there may be great differences in grey value and neighborhood structure between the current pixel and neighbouring pixels. Although small weights are assigned to these neighbouring pixels, the estimated depth map will be blurred because of these irrelevant pixels. Edges play an important role in the texture-less depth map. To restore these structures, most of the methods penalise local smoothness along with horizontal and vertical directions(e.g., [4, 6]). Although these methods can obtain good results, the depth map has different edge structures and these methods will lead to a lack of directional information. Besides, they usually only consider the local structure of the depth map and ignore the patch correlation in the depth maps. In this letter, we propose a novel single depth map SR method without the guidance of colour image. The proposed method combines the advantage of the non-local self-similarity and the local directional smoothness. To fully exploit the non-local self-similarity of the depth map, we introduce the adjusted non-local regression (ANLR) prior [9], which can make full use of the non-local information of the depth map. However, this prior gives each similar pixel a weight even for the irrelevant pixel. When the patch contains considerable variations, the estimated result will be blurred. To solve this problem, we proposed the shape-adaptive (SA) ANLR prior, which can alleviate the effect of irrelevant pixels and suppress the blurring artifacts. To construct different directional information, we first extract directional features of depth maps using the curvelet transform and then build a direction-based local smoothness prior using these directional features. The directional feature can effectively restore a sharper edge in the reconstructed depth map. Optimisation algorithm Depth map SR can be formulated as the following minimisation problem: argmin X ∥ Y − D H X ∥ 2 2 + γ R ( X ) (1)Where D and H stand for downsampling and the blurring operator, γ is the regularization parameter. R ( X ) is the image prior. The effective prior is important for the SR performance. The proposed method combines two effective priors of non-local self-similarity and local smoothness for the depth map recovery. Non-local self-similarity Since the ANLR prior uses the patchwise strategy, it can fully exploit the non-local self-similarity and fully utilize the non-local information of the depth map. However, when the target pixel is in the edge region, the estimated value of it will be inaccurate because of the irrelevant pixel in the similar pixel group. To solve this problem, SA-ANLR prior is adopted. In SA-ANLR prior, the SA patch is used instead of the fixed-size square patch. Since the shape of the patch is adaptive to the image structures [10], the SA patch can avoid smoothing across edges. To extract SA patches efficiently, the anisotropic local polynomial approximation-intersection of confidence intervals (LPA-ICI) technique [11] is used. For each target pixel x ∈ X , we extract a target patch of size b × b centered at the target pixel and an SA mask is obtained for the target patch by its shape-adaptive neighborhood. For each target patch, its N closest patches p x n , n = 1 , … , N are searched within a search window W. The SA similar patch group G x = [ M x ( p x 1 ) , M x ( p x 2 ) , … , M x ( p x N ) ] is gotten by masking these similar patches. M ( · ) is the operator of extracting homogeneous pixels in the SA mask. The SA similar patch group (SA-SPG) is shown in Figure 1. Because of the patchwise strategy [9], each pixel belongs to B overlapped patches B = ∑ m = 1 b 2 M x m b 2 − m + 1 ), which is an overlap-based scheme. M x m is the SA mask of the neighbor m of the target pixel x, b 2 − m + 1 is the corresponding position of the target pixel in the patch centered at m. Because there are N similar patches for each overlapped patch, so each pixel has B N similar pixels. We call these B N similar pixels an SA overlap-based similar pixel group (SA-OSPG). Let Ω x = { R b 2 − m + 1 ( G x m ) } 1 b 2 is the similar pixel set of x, where G x m is the SA similar patch group of neighbor m of the target pixel x, R b 2 − m + 1 is the operation of extracting the b 2 − m + 1 th pixel of each patch in G x m . The SA-OSPG Ω x is shown in Figure 1, where the entries in black mean irrelevant pixels. Fig. 1Open in figure viewerPowerPoint SA-SPG and SA-OSPG Before constructing the SA-ANLR prior, the similarity weight of each pixel in one SA-OSPG needs to be calculated. However, for each similar pixel group, the similarity weight computed from original pixel pairs may be unstable in some cases. To improve the performance, we first filter each pixel in X. For the pixel x, we redefine it by the weighted average of the pixels in its shape-adaptive neighborhood. Then, the weighted pixel x w is given by: x w = ∑ j ∈ u x + w ( j ) x j (2)where w ( j ) is the weight of the neighbor x j , u x + is the shape-adaptive neighborhood of x. Using these redefined pixels, the similarity weight is defined by: w ∼ j = 1 ∑ j ∈ Ω x exp − x w − x w j 2 2 σ 2 exp − x w − x w j 2 2 σ 2 (3)Finally, we can formulate the SA-ANLR prior as: J S A − ANLR = ∑ x ∈ X w x x − ∑ j ∈ Ω x w j ∼ x j 2 2 = Λ I − W N L X 2 2 (4)where Λ is the diagonal matrix, and w x is the element on the main diagonal. w x = α / max ( α , var ( Ω x ) β ) , α and β are positive constants, var ( Ω x ) is the variance of the SA-OSPG of x. The weight w x penalizes the reliability of the SA-OSPG, which enhances the robustness of data estimation. When the standard deviation is small, it means the SA-OSPG is reliable and a large weight can be imposed. Local smoothness The edge plays an important role in depth maps and has significant directional characteristics. To preserve the directional characteristics, most of the methods force local smoothness along with horizontal and vertical directions. These methods will lead to a lack of directional information since the depth map has different edge structures. In this letter, we use local directional smoothness prior to model this character. The directional smoothness means that in the depth map the intensity difference between adjacent pixels in a specific direction is small. The directional smoothness represents structures of the depth map at the pixel level and is a complement of the non-local prior. The curvelet transform [12] as a multi-scale directional transform possesses very high directional sensitivity [13]. Since it can represent edges more precisely, we use curvelet transform to extract directional features of depth maps. For a given depth map X, the curvelet coefficients can be obtained by Q = Γ ( X ) (5)where Γ ( . ) is the curvelet transform. Q is a set of curvelet coefficient matrices of X and can be expressed as { Q s , l } . s and l are scales and directions, respectively. The depth map is decomposed into five scales. The curvelet coefficient matrixes on the first scale are smooth details in the depth map. The curvelet coefficient matrices from the second to fifth scales are high-frequency coefficients, and each coefficient matrix represents a direction matrix. Since there is more directional information at finer scales, we only consider the finest scale. There are 64 curvelet coefficient matrices on the finest scale. Due to the symmetry of curvelet coefficients, we only use half the matrices on the finest scale and divide them into 16 direction subsets Z k , k = 1 , … , 16 . The multi-directional matrices A k of the input image of the kth direction can be formulated as A k = Γ − 1 Z k , k = 1 , … , 16 (6)where Γ − 1 is the curvelet inverse transform. The directional features A k represent the direction intensity of X in the kth direction. When the value of A k ( i , j ) is large, it means that X ( i , j ) has more feature information in the the kth direction. In this letter, we defined the direction-based local smoothness prior as: ∑ k W k L G k X 1 (7) G k are the convolution matrices for the kth direction. W k L is a diagonal weighting matrix. The weight is defined as W k L = e − G k X 2 / σ + H A k / η (8) H A k = A k , if e − G k X 2 / σ < τ 0 , otherwise. (9)where the first term is the depth weight. However, the depth weight cannot reflect the rich structures of depth maps. To get the directional information, we introduce the directional feature A k to the weight W k L . When the value of A k ( i , j ) is large, it means the kth direction is the principal direction of X ( i , j ) . According to the directional feature, we can get the principal direction of the image. In the principal direction, the adjacent pixels is more similar and a higher weight should be given G k X . To achieve this, we add weighting parameter H ( A k ) to the depth weight. H ( A k ) is the operation of extracting the directional feature value for the pixel in edge region. Finally, by combining the non-local self-similarity prior and local smoothness prior, we have a joint prior model: argmin X ∥ Y − D H X ∥ 2 2 + γ Λ I − W N L X 2 2 + λ ∑ k W k L G k X 1 (10)Equation (10) can be rewritten as: argmin X ∥ Y ̂ − K X ∥ 2 2 + λ ∑ k W k L G k X 1 (11)where Y ̂ = ( Y 0 ) K = ( D H γ Λ ( I − W N L ) ) Equation (11) is a l − 1 related minimization problem. Here, we take a Split Bregman iteration approach to solve it [14]. Experimental results We use six data sets from the Middlebury benchmark database, that is Art, Book, Moebius, Reindeer, Laundry, and Dolls, to evaluate the performance. Because the test images are too big and time-consuming, we only use a small part of the whole picture. To generate LR depth maps, the ground truth images are blurred by a 7 × 7 Gaussian kernel with a standard deviation 1.15 and then downsampled to the desired resolution. We test the up-sampling performance at factors of 2 and 4. To evaluate our method, we compare the proposed method with four methods, including the color-guide methods, that is anisotropic total generalized variation (TGV) [15] and static and dynamic guided filtering (SDF) [16], non colour-guide methods, that is edge-guided (EG) [5] and ScSR [17]. The number of similar patches N is 10, the patch size is 7 × 7, and the size of the search window is 13 × 13. The control parameter γ and λ are set to 0.074 and 0.016, respectively. Table 1 presents the up-sampling results on quantitative evaluation and mean absolute error (MAE) is used as the evaluation metric. The first two methods in the tables are colour-guide methods. For the scaling factor 2, our proposed method obtains the lowest MAE and performs even better than the color-guided methods. For the scaling factor 4, our proposed method performs better than the single depth SR methods as well. Table 1. Quantitative up-sampling results on the Middlebury dataset MAE 2 × Art Book Moebius Laundy Reindeer Dolls SDF 2.5788 0.5194 0.7634 1.3214 1.6284 0.5638 TGV 3.1718 0.4841 0.7916 1.3446 1.7964 0.5252 ScSR 2.0117 0.3887 0.6139 1.0351 1.2674 0.4111 EG 2.2110 0.5686 0.8492 1.3162 1.5168 0.6030 Ours 0.9769 0.3740 0.4359 0.3175 0.3735 0.3832 MAE 4 × Art Book Moebius Laundy Reindeer Dolls SDF 3.0475 0.7072 0.8008 1.3816 2.6614 0.8272 TGV 3.7260 0.6951 0.9841 1.7806 3.4150 0.7531 ScSR 3.1917 0.9550 1.5372 2.4257 4.2544 1.0115 EG 2.5512 0.7633 1.1965 1.9093 3.1944 0.7747 Ours 2.0327 0.7507 1.0141 1.2466 2.1041 0.7262 Figure 2 shows visual comparison results at 4 × up-sampling for image Art. From the figures, we can see that the TGV [15] method produces texture copy artifacts in some regions where the inconsistency between colour and depth maps is not considered. This fact is clearly confirmed by corresponding error maps. In SDF [16], static and dynamic guidance is utilized to eliminate the inconsistency, but it fails in some cases. The result of ScSR [17] introduced displeasing artifacts. Its error map shows that there are also noticeable errors near edges because of the ringing artifact. The result of the EG [5] has jagging artifacts when an accurate edge is not available. From its error map, we can see that there are large errors around the depth discontinuities. This is because EG method is highly dependent on an accurate edge. Our method has much smaller errors around the depth map edges. Fig. 2Open in figure viewerPowerPoint Visual comparison of “Art' (scaling factor = 4) and the corresponding error maps. (a) Ground truth. (b) SDF. (c) TGV. (d) ScSR. (e) EG. (f) Our method To evaluate the effectiveness of the proposed method, we also compare our method with DCLS (only direction-based local smoothness prior is used), ANLR, SA-ANLR. Table 2 presents the quantitative depth recovery results for 2 up-sampling. Figure 3 gives the results of a visual comparison (the patches in the red boxes are highlighted regions). From the results shown in Table 2 and Figure 3, we can see that the results of DCLS are the worst. The SA-ANLR produces better results than ANLR, which proves the effectiveness of the SA-ANLR. The proposed method obtains the best performance and reconstructs the depth map with a sharp edge, which shows that the combination of the SA-ANLR and DCLS is effective for restoring accurate depth maps. Table 2. Average MAE if different extended version on 6 test image Method DCLS ANLR SA-ANLR Ours MAE 2× 1.0487 0.7573 0.6098 0.4768 Fig. 3Open in figure viewerPowerPoint Visual comparison of “Laundry.” (a) Ground truth. (b) DCLS. (c) ANLR. (d) SA-ANLR. (e) Our method In general, the proposed method can significantly improve the SR performance. The drawback of it is the computational complexity since the local structure analyzing, the non-local similarly pixels searching and iterative optimization of the objective equation are time-consuming. In our future work, we will try to improve the efficiency of the proposed method. Conclusion In this letter, we have introduced a novel method for a single depth map recovery without a corresponding color image as an assistant. The proposed method introduces a joint non-local self-similarity and local smoothness strategy. To fully exploit non-local self-similarity in the depth map, we build an SA-ANLR prior, which can alleviate the effect of irrelevant pixels and recovery sharp edges. To character different local edge information, we use the direction-based local smoothness. The proposed method combines the advances of the two priors and can recover fine structures. In both quantitative and qualitative evaluations, the proposed method can achieve good performance. Conflict of Interest The authors declare no conflict of interest. References 1Ren, Y., et al.: Edge-guided with gradient-assisted depth up-sampling. Electron. Lett. 53(21), 1400– 1402 (2017) Wiley Online LibraryWeb of Science®Google Scholar 2Dong, W., et al.: Color-guided depth recovery via joint local structural and nonlocal low-rank regularization. IEEE Trans. Multimedia 19(2), 293– 301 (2017) CrossrefWeb of Science®Google Scholar 3Liu, X., et al.: Depth Super-Resolution via Joint Color-Guided Internal and External Regularizations. IEEE Trans. Image Process. 28(4), 1636-1645 (2019) CrossrefPubMedWeb of Science®Google Scholar 4Ferstl, D., Rüther, M., Bischof, H.: Variational Depth Superresolution Using Example-Based Edge Representations. Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Santiago, Chile, pp. 513– 521 (2015) Google Scholar 5Xie, J., Feris, R.S., Sun, M.: Edge-Guided Single Depth Image Super Resolution. Trans. Image Process. 25(1), 428– 438 (2016) CrossrefPubMedWeb of Science®Google Scholar 6Mandal, S., Bhavsar, A., Sao, A.K.: Hierarchical example-based range-image super-resolution with edge-preservation. IEEE Int. Conf. Image Process. (ICIP), Paris, France, pp. 3867– 3871 (2014) Google Scholar 7Favaro, P.: Recovering thin structures via non-local-means regularization with application to depth from defocus. Proc. Comput. Vis. Pattern Recognit. (CVPR) pp. 1133– 1140 (2010) Google Scholar 8Park, J., et al.: High-Quality Depth Map up-sampling and Completion for RGB-D Cameras. IEEE Trans. Image Process. 23(12), 5559– 5572 (2014) CrossrefPubMedWeb of Science®Google Scholar 9Ren, C., He, X., Nguyen, T.Q.: Adjusted non-local regression and directional smoothness for image restoration. IEEE Trans. Multimedia 21(3), 731– 745 (2019) CrossrefWeb of Science®Google Scholar 10Chang, K., et al: Data-Adaptive Low-Rank Modeling and External Gradient Prior for Single Image Super-Resolution. Signal Process. 161, 36– 49 (2019) CrossrefWeb of Science®Google Scholar 11Katkovnik, V., et al.: Adaptive window size image de-noising based on intersection of confidence intervals (ICI) rule. J. Math. Imag. 16(3), 223– 235 (2002) CrossrefWeb of Science®Google Scholar 12Ma, J., Plonka, G.: The curvelet transform. IEEE Signal Process. Mag. 27(2), 118– 133 (2010). CrossrefWeb of Science®Google Scholar 13Li, X., et al.: Single image super-resolution via directional group sparsity and directional features. IEEE Trans. Image Process. 24(9), 2874– 2888 (2015) CrossrefPubMedWeb of Science®Google Scholar 14Goldstein, T., Osher, S.: The split Bregman method for L1-regularized problems. SIAM J. Imag. Sci. 2(2), 323– 343 (2009) CrossrefWeb of Science®Google Scholar 15Ferstl, D., et al.: Image guided depth up-sampling using anisotropic total gGeneralized variation. Proc. Comput. Vis. Pattern Recognit. (CVPR), Sydney, NSW, pp. 993– 1000 (2013) Google Scholar 16Ham, B., Cho, M., Ponce, J.: Robust image filtering using joint static and dynamic guidance. Proc. Comput. Vis. Pattern Recognit. (CVPR), Boston, MA, 4823– 4831 (2015) Google Scholar 17Yang, J., et al.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861– 2873 (2010) CrossrefPubMedWeb of Science®Google Scholar Volume57, Issue12June 2021Pages 475-477 FiguresReferencesRelatedInformation