# Efficient Stereo Matching Based on Pervasive Guided Image Filtering

Authors: Chengtao Zhu; Yau-Zen Chang
Journal: Mathematical Problems in Engineering
 (2019)

---

## Abstract

This paper presents an effective cost aggregation strategy for dense stereo matching. Based on the guided image filtering (GIF), we propose a new aggregation scheme called Pervasive Guided Image Filtering (PGIF) to introduce weightings to the energy function of the filter which allows the whole image pair to be taken into account. The filter parameters of PGIF are calculated as two-dimensional convolution using the bright and spatial differences between the corresponding pixels, which can be incrementally calculated for efficient aggregation. The complexity of the proposed algorithm is O(N), which is linear to the number of image pixels. Furthermore, the algorithm can be further simplified into O(N/4) without significantly sacrificing accuracy if subsampling is applied in the stage of parameter calculation. We also found that a step function to attenuate noise is required in calculating the weights. Experimental evaluation on version 3 of the Middlebury stereo evaluation datasets shows that the proposed method achieves superior disparity accuracy over state-of-the-art aggregation methods with comparable processing speed.


---

## Body

## 1. Introduction

Stereoscopic vision is less invasive and valuable for many applications, such as 3D reconstruction and environmental detection of autonomous vehicles, as it relies only on pairs of images captured from different perspectives. In stereo vision systems, stereo matching algorithms are critical for correct and accurate disparity estimation, finding corresponding pixels in matching images for each pixel in the reference image.According to [1], dense stereo matching algorithms fall into two categories: local methods and global methods. The global methods treat disparity calculations as a minimization problem, where the objective function consists of a measurement part and a penalty part. The measurement part indicates the similarity between the slice on the pair of images and the penalty part to suppress the change in disparity. Representative global methods include Belief Propagation [2], Graph Cut [3], and Dynamic Programming [4]. These techniques require a lot of calculations and are not applicable for real-time applications.In contrast, the local methods are popular for fast disparity calculations. Local approaches tackle the effects of light changes through local windows and are categorized into parametric [5] and nonparametric methods [6]. Local algorithms generally perform four stages [1]: (1) preliminary matching cost calculations, (2) cost aggregation over support regions, (3) disparity estimation, and (4) disparity refinement.Cost aggregation plays a key role in the local stereo matching algorithms. Early approaches, such as in [7], achieved limited performances, especially in discontinues and occlusion regions. To accommodate the assumption that pixels are of similar disparity within the same aggregation windows [8], many adaptive weighting aggregation methods, such as in [9–11], were presented.Recent years have seen a trend to treat the cost aggregation as image filtering. The guided image filter [12] (GIF) is able to provide superior edge profiles without gradient-reversal artifacts, and was successfully applied to cost aggregation in [13]. In order to reduce the computational load of the GIF, [14] recommends subsampling the cost and the guidance image to calculate the coefficients. Ref. [15] proposed the Weighted Guided Image Filtering (WGIF) to improve GIF by modifying the regularization term of the energy function. Besides, [16, 17] applied WGIF for stereo matching with limited performance because of the lack of pixel information outside the fixed windows.In order to improve the precision of stereo matching, a series of approaches with adaptive guided filters were proposed [18–20] to remove the limitation of the fixed-window formulation. Among them, [20] adaptively tunes the size of rectangular support windows based on both the intensity difference and distance between the neighboring pixel and the central pixel. However, these approaches still suffer from the loss of information outside the windows. Ref. [21] uses whole image for matching cost calculation, where the weights of aggregation are computed according to a measure of similarity between neighboring pixels in the guidance image. In addition, a new scheme called weight propagation is proposed to efficiently calculate weights.In this work, we extend the scheme of GIF [12] for disparity cost aggregation, as suggested in [13], by introducing bilateral weights that take distance and intensity differences into account, as suggested in [21]. This approach uses the whole image for aggregation. Similar to the convolution procedure of [21], the complexity of the proposed algorithm is O(N), which is linear to the image resolution. We call our approach Pervasive Guided Image Filtering, denoted as PGIF.The main contribution of this paper is a new cost aggregation algorithm for stereo matching which can be summarized as follows:(1) An innovative aggregation scheme is proposed that weights the cost function of the GIF to allow for consideration of the entire image pair.(2) We demonstrate that a constraint modification of the aggregation weight by a step function is crucial to avoid value reduction in the weight propagation process. This minor modification further improves the accuracy of disparity.(3) The proposed aggregation algorithm can be calculated as two-dimensional convolution with complexity O(N) using the weight-propagation method of [21]. Besides, the algorithm can be further simplified into O(N/4) without sacrificing accuracy if subsampling is applied in calculating the parameters of the guided image filtering, as suggested by [14].(4) A performance evaluation of version 3 of the Middlebury Stereoscopic Evaluation Dataset demonstrates the superiority of the proposed method to most state-of-the-art aggregation methods in terms of disparity accuracy and processing speed.


## 2. Aggregations of Preliminary Cost

### 2.1. Cost Function Definition and GIF Aggregation

In a local stereo matching procedure, a disparity map is obtained through four steps: (1) computation of the preliminary matching cost, (2) aggregation of the cost via volumetric filtering, (3) disparity selection via winner-take-all (WTA), and (4) postprocessing for disparity refinement. In the first step, the preliminary matching cost is a three-dimensional array of dissimilarity measures for every pixel within potential disparities. When a pixel atp=(x,y) is assigned a disparity valued, we use the notation CL(p,d) as the preliminary cost based on the left image.There are many metrics that can be used to measure the degree of matching between image patches, such as sum of squared difference and normalized cross correlation. In the following investigations, we use the basic metric, the truncated absolute difference of the gradient, for the cost:(1)CLp,d=min∇ILp-∇IRp+d,τ,where IL and IR are the left and right images of the stereo pair and τ is a truncation threshold, normally assigned as 2, which is applied to reduce mismatch in noisy or obscured regions. This concise arrangement also reflects the effectiveness of the proposed scheme.The general form of cost aggregation in the traditional local stereo matching algorithms can be written as a weighted sum of the preliminary matching cost:(2)C′Lp,d=∑q∈ΩpWp,q·CLq,d,where C′L(p,d) is the aggregated cost and q are pixels within a window Ω centered at pixel p.Equation (2) is a general form of window-based cost aggregation. When the aggregation weight Wp,q is set to 1, we have the simplest stereo matching algorithm, such as in [1]. The performance of the window-based approaches depends on the correct choice of the size of the support window, since the effects of external pixels are ignored.Full Image-Guided Filtering [21] (FIF) is one of the methods using the entire image as a supporting window, which implements a scheme called weight propagation for the cost aggregation:(3)CLFIGFp,d=∑q∈IL∏m,n∈Lp,qexp-ILm-ILnσ·CLq,d=∑q∈ILWp,qFIGF·CLq,d.In (3), σ is a constant, Lp,q is the path between a pixel pair (p,q), and IL(m)-IL(n) is the Euclidian distance between the intensities of two pixels, m and n, along the path. Note that the guided image filtering [12] (GIF) scheme is not employed in this method.On the other hand, Fast Cost-Volume Filtering method [13] (FCVF) achieves significant performance compared to most local stereo matching methods. In this method, the weights are not explicitly calculated, as shown in (2). Instead, based on the principle of GIF [12], the filtered cost assumes a local linear model in terms of the guidance image IL such that(4)CLFCVFp,d=ap,d·ILq+bp,d,∀q∈Ωp,where a(p,d) and b(p,d) are obtained by minimizing an energy function E(p,d) within the supporting window Ω. The energy function for the parameters,a andb, is defined as(5)Ep,d=∑q∈Ωpap,d·ILq+bp,d-CLq,d2+ε·ap,d2,where ε is the normalization parameter used to limit the magnitude of a(p,d), usually chosen to be 0.0001. The optimal values of the linear parameters in (5), denoted as a∗(p,d) and b∗(p,d), are(6)a∗p,d=∑q∈ΩpILq·CLq,d/Ωp-∑q∈ΩpILq/Ωp·∑q∈ΩpCLq,d/Ωp∑q∈ΩpILq2/Ωp-∑q∈ΩpILq/Ωp·∑q∈ΩpILq/Ωp+ε,b∗p,d=∑q∈ΩpCLq,dΩp-a∗p,d·∑q∈ΩpILqΩpwhere Ωp is the number of pixels in the support window Ω centered on p. This GIF-based aggregation can also be rearranged into the form of (2) by replacing the parameters of (4) with the best values of (6): (7)CLFCVFp,d=∑q∈Ωp∑k∈Ωp∩Ωq1Ωp·Ωk1+ILp-μILk·ILq-μILkσ2ILk+ε·CLq,d=∑q∈ΩpWp,qFCVF·CLq,d,where μ(IL(k)) is the mean values of IL and σ2(IL(k)) is the variance of IL:(8)μILk=∑s∈ΩkILsΩkσ2ILk=∑s∈ΩkILs2Ωk-∑s∈ΩkILsΩk·∑s∈ΩkILsΩk.


### 2.2. The Proposed Aggregation Scheme: Pervasive Guided Image Filtering (PGIF)

According to the investigation of the last section, we have that a proper image filtering algorithm is critical for the accuracy of stereo matching. However, most GIF-based local stereo matching methods suffer from the same problem of missing information outside the supporting window.In this section, we propose a GIF-based scheme that uses the full image for the filtering and call it the Pervasive Guided Image Filtering (PGIF). Similar to (4), the scheme also approximates the filtered cost as a linear model in terms of the guidance image IL. However, the corresponding energy function for the linear parameters is defined as a weighted version of (5):(9)Ep,d=∑q∈ILωPGIFp,q·ap,d·ILq+bp,d-CLq,d2+ε·ap,d2,where ε is a regularization parameter to restrict the value of a(p,d) and the weight ωPGIF(p,q) reflects the relative importance of the pixelq = (i,j) with respect to the pixelp = (x,y). Importantly, the weight is calculated as a multiplication of horizontal and vertical weighting factors, W¯i,xH and W¯j,yV:(10)ωPGIFp,q=ωPGIFx,y,i,j=W¯i,xH·W¯j,yV,where(11)W¯i,xH=∏k=mini,x+1maxi,xexpfILk,j-ILk-1,j-β,i≠x1,i=xW¯j,yV=∏k=minj,y+1maxj,yexpfILx,k-ILx,k-1-β,j≠y1,j=yand(12)fz=0,ifz<11,ifz≥1.The parameterβ is a constant filter factor, where the larger value corresponds to a higher weight. As 0<W¯i,xH,W¯j,yV≤1, successive multiplication of them ensures smaller weights for longer distances. Furthermore, the step function f(z) defined in (11) is introduced to avoid the loss of information when there is a significant local density difference along the path from pixel q to pixel p. By this arrangement, the values of the weightings stay within the interval [exp(-1/β),1], which are proportional to density similarity.The necessity of introducing the functionf(z) is illustrated in Figure 1, where all pixel values are 10, except p3, which is 99. Assuming β=4, we have that ωPGIF(p7,p1)=ωPGIF(p10,p1)=0.3678. However, if f(z) is not applied, we have that ωPGIF(p7,p1)=ωPGIF(p10,p1)=1.624×10-4, making the intensity of p1 invisible to both p7 and p10.Figure 1
A case indicating the necessity of introducing a step function to avoid the effects of sudden changes in intensity.Optimal values of the linear parameters of the filter,a(p,d) and b(p,d), are obtained by minimizing the energy function E(p,d). This can be achieved by assigning the first derivatives of it to zero:(13)∂Ep,d∂ap,d=0∂Ep,d∂bp,d=0.Solving (13), the best estimate of the parameters is(14)a∗p,d=μ-ILp·CLp,d-μ-ILp·μ-CLp,dσ-ILp+εb∗p,d=μ-CLp,d-a∗p,d·μ-ILp,where(15)μ-ILp·CLp,d=∑q∈ILωPGIFp,q·ILp·CLp,d∑q∈ILωPGIFp,qμ-CLp,d=∑q∈ILωPGIFp,q·CLq,d∑q∈ILωPGIFp,qμ-ILp=∑q∈ILωPGIFp,q·ILq∑q∈ILωPGIFp,qσ-ILp=∑q∈ILωPGIFp,q·ILq·ILq∑q∈ILωPGIFp,q-μ-ILp2.To reduce the computational complexity of (15), these equations can be decomposed into four one-dimensional convolutions using the principle of weight propagation, as developed in [21]. Let us take the numerator of μ-(IL(p)·CL(p,d)), denoted as Z(p,d), as an example:(16)Zp,d=∑q∈ILωPGIFp,q·ILp·CLp,d=∑i=1s1∑j=1s2W¯i,xH·W¯j,yV·ILi,j·CLi,j,d,where the weight ωPGIF is calculated according to (10) and s1 and s2 are the width and height of the guidance image, respectively. Firstly, we define the left-to-right and the right-to-left weighted sums, ML(x,j,d) and MR(x,j,d), as(17)MLx,j,d=∑i=1x-1W¯i,xH·ILi,j·CLi,j,d+ILx,j·CLx,j,dMRx,j,d=∑i=x+1s1W¯i,xH·ILi,j·CLi,j,d+ILx,j·CLx,j,d.Since W¯i,xH=W¯i,x-1H·W¯x-1,xH,ML(x,j,d) can be written in recursive form:(18)MLx,j,d=∑i=1x-2W¯i,xH·ILi,j·CLi,j,d+W¯x-1,xH·ILx-1,j·CLx-1,j,d+ILx,j·CLx,j,d=W¯x-1,xH·∑i=1x-2W¯i,x-1H·ILi,j·CLi,j,d+ILx-1,j·CLx-1,j,d+ILx,j·CLx,j,d=W¯x-1,xH·MLx-1,j,d+ILx,j·CLx,j,d.Similarly,(19)MRx,j,d=W¯x,x+1H·MRx+1,j,d+ILx,j·CLx,j,d.Secondly, we define the horizontal weighted sum, denoted asMH(x,j,d):(20)MHx,j,d=MLx,j,d+MRx,j,d-ILx,j·CLx,j,d.Based on a similar procedure, we define the top-to-bottom and bottom-to-top weighted sums, MT(x,y,d) and MB(x,y,d), as(21)MTx,y,d=W¯y-1,yV·MTx,y-1,d+ILx,y·CLx,y,d=∑j=1y-1W¯j,yV·MHx,j,d+ILx,y·CLx,y,d(22)MBx,y,d=W¯y,y+1V·MBx,y+1,d+ILx,y·CLx,y,d=∑j=y+1s2W¯j,yV·MHx,j,d+ILx,y·CLx,y,d.Hence, Z(p,d) can be calculated as(23)Zp,d=Zx,y,d=MTx,y,d+MBx,y,d-ILx,y·CLx,y,d.In sum, we can computeZ(p,d) using (23), where MT(x,y,d) and MB(x,y,d) are obtained by (21) and (22). Next, MH(x,j,d) in (21) and (22) are calculated from (20). Finally, ML(x,j,d) and MR(x,j,d) of (20) are recursively calculated using (18) and (19).We have that all of the two-dimensional convolutions in (15) can be decomposed into one-dimensional convolution operations in four directions with computational complexity O(N), where N is the total number of pixels of the guidance image.Once we have the optimal filter parametersa∗(p,d) and b∗(p,d), the aggregated matching cost volume is calculated as(24)CLPGIFp,d=a∗p,d·ILp+b∗p,d.We may substitute the parameters of (24) by the optimal values of (14) and rearrange them into the following form:(25)CLPGIFp,d=∑q∈ILωPGIFp,q∑q∈ILωPGIFp,q1+ILp-μ-ILp·ILq-μ-ILpσ-ILp+ε·CLq,d=∑q∈ILWp,qPGIF·CLq,d.The disparity map, denoted asDL(p), can then be obtained by the winner-take-all operation on the aggregated cost volume CLPGIF(p,d):(26)DLp=argmind∈RCLPGIFp,d,where R={dmin,…,dmax} is the range of disparity values.A close comparison between (7) and (25) indicates that the effective region of (25) extends from the local aggregation window Ω of (7) to the entire guided image IL, where the weight of (25) includes the weight ωPGIF of the energy function defined in (9).In addition, in order to speed up the calculation, [14] suggests subsampling both the cost volume and the guidance image to calculate the linear coefficients. We can follow the same process by subsampling IL and CL to calculate the optimal filter parameters and then use them for normal aggregation. This will accelerate the algorithm to O(N/4).


## 2.1. Cost Function Definition and GIF Aggregation

In a local stereo matching procedure, a disparity map is obtained through four steps: (1) computation of the preliminary matching cost, (2) aggregation of the cost via volumetric filtering, (3) disparity selection via winner-take-all (WTA), and (4) postprocessing for disparity refinement. In the first step, the preliminary matching cost is a three-dimensional array of dissimilarity measures for every pixel within potential disparities. When a pixel atp=(x,y) is assigned a disparity valued, we use the notation CL(p,d) as the preliminary cost based on the left image.There are many metrics that can be used to measure the degree of matching between image patches, such as sum of squared difference and normalized cross correlation. In the following investigations, we use the basic metric, the truncated absolute difference of the gradient, for the cost:(1)CLp,d=min∇ILp-∇IRp+d,τ,where IL and IR are the left and right images of the stereo pair and τ is a truncation threshold, normally assigned as 2, which is applied to reduce mismatch in noisy or obscured regions. This concise arrangement also reflects the effectiveness of the proposed scheme.The general form of cost aggregation in the traditional local stereo matching algorithms can be written as a weighted sum of the preliminary matching cost:(2)C′Lp,d=∑q∈ΩpWp,q·CLq,d,where C′L(p,d) is the aggregated cost and q are pixels within a window Ω centered at pixel p.Equation (2) is a general form of window-based cost aggregation. When the aggregation weight Wp,q is set to 1, we have the simplest stereo matching algorithm, such as in [1]. The performance of the window-based approaches depends on the correct choice of the size of the support window, since the effects of external pixels are ignored.Full Image-Guided Filtering [21] (FIF) is one of the methods using the entire image as a supporting window, which implements a scheme called weight propagation for the cost aggregation:(3)CLFIGFp,d=∑q∈IL∏m,n∈Lp,qexp-ILm-ILnσ·CLq,d=∑q∈ILWp,qFIGF·CLq,d.In (3), σ is a constant, Lp,q is the path between a pixel pair (p,q), and IL(m)-IL(n) is the Euclidian distance between the intensities of two pixels, m and n, along the path. Note that the guided image filtering [12] (GIF) scheme is not employed in this method.On the other hand, Fast Cost-Volume Filtering method [13] (FCVF) achieves significant performance compared to most local stereo matching methods. In this method, the weights are not explicitly calculated, as shown in (2). Instead, based on the principle of GIF [12], the filtered cost assumes a local linear model in terms of the guidance image IL such that(4)CLFCVFp,d=ap,d·ILq+bp,d,∀q∈Ωp,where a(p,d) and b(p,d) are obtained by minimizing an energy function E(p,d) within the supporting window Ω. The energy function for the parameters,a andb, is defined as(5)Ep,d=∑q∈Ωpap,d·ILq+bp,d-CLq,d2+ε·ap,d2,where ε is the normalization parameter used to limit the magnitude of a(p,d), usually chosen to be 0.0001. The optimal values of the linear parameters in (5), denoted as a∗(p,d) and b∗(p,d), are(6)a∗p,d=∑q∈ΩpILq·CLq,d/Ωp-∑q∈ΩpILq/Ωp·∑q∈ΩpCLq,d/Ωp∑q∈ΩpILq2/Ωp-∑q∈ΩpILq/Ωp·∑q∈ΩpILq/Ωp+ε,b∗p,d=∑q∈ΩpCLq,dΩp-a∗p,d·∑q∈ΩpILqΩpwhere Ωp is the number of pixels in the support window Ω centered on p. This GIF-based aggregation can also be rearranged into the form of (2) by replacing the parameters of (4) with the best values of (6): (7)CLFCVFp,d=∑q∈Ωp∑k∈Ωp∩Ωq1Ωp·Ωk1+ILp-μILk·ILq-μILkσ2ILk+ε·CLq,d=∑q∈ΩpWp,qFCVF·CLq,d,where μ(IL(k)) is the mean values of IL and σ2(IL(k)) is the variance of IL:(8)μILk=∑s∈ΩkILsΩkσ2ILk=∑s∈ΩkILs2Ωk-∑s∈ΩkILsΩk·∑s∈ΩkILsΩk.


## 2.2. The Proposed Aggregation Scheme: Pervasive Guided Image Filtering (PGIF)

According to the investigation of the last section, we have that a proper image filtering algorithm is critical for the accuracy of stereo matching. However, most GIF-based local stereo matching methods suffer from the same problem of missing information outside the supporting window.In this section, we propose a GIF-based scheme that uses the full image for the filtering and call it the Pervasive Guided Image Filtering (PGIF). Similar to (4), the scheme also approximates the filtered cost as a linear model in terms of the guidance image IL. However, the corresponding energy function for the linear parameters is defined as a weighted version of (5):(9)Ep,d=∑q∈ILωPGIFp,q·ap,d·ILq+bp,d-CLq,d2+ε·ap,d2,where ε is a regularization parameter to restrict the value of a(p,d) and the weight ωPGIF(p,q) reflects the relative importance of the pixelq = (i,j) with respect to the pixelp = (x,y). Importantly, the weight is calculated as a multiplication of horizontal and vertical weighting factors, W¯i,xH and W¯j,yV:(10)ωPGIFp,q=ωPGIFx,y,i,j=W¯i,xH·W¯j,yV,where(11)W¯i,xH=∏k=mini,x+1maxi,xexpfILk,j-ILk-1,j-β,i≠x1,i=xW¯j,yV=∏k=minj,y+1maxj,yexpfILx,k-ILx,k-1-β,j≠y1,j=yand(12)fz=0,ifz<11,ifz≥1.The parameterβ is a constant filter factor, where the larger value corresponds to a higher weight. As 0<W¯i,xH,W¯j,yV≤1, successive multiplication of them ensures smaller weights for longer distances. Furthermore, the step function f(z) defined in (11) is introduced to avoid the loss of information when there is a significant local density difference along the path from pixel q to pixel p. By this arrangement, the values of the weightings stay within the interval [exp(-1/β),1], which are proportional to density similarity.The necessity of introducing the functionf(z) is illustrated in Figure 1, where all pixel values are 10, except p3, which is 99. Assuming β=4, we have that ωPGIF(p7,p1)=ωPGIF(p10,p1)=0.3678. However, if f(z) is not applied, we have that ωPGIF(p7,p1)=ωPGIF(p10,p1)=1.624×10-4, making the intensity of p1 invisible to both p7 and p10.Figure 1
A case indicating the necessity of introducing a step function to avoid the effects of sudden changes in intensity.Optimal values of the linear parameters of the filter,a(p,d) and b(p,d), are obtained by minimizing the energy function E(p,d). This can be achieved by assigning the first derivatives of it to zero:(13)∂Ep,d∂ap,d=0∂Ep,d∂bp,d=0.Solving (13), the best estimate of the parameters is(14)a∗p,d=μ-ILp·CLp,d-μ-ILp·μ-CLp,dσ-ILp+εb∗p,d=μ-CLp,d-a∗p,d·μ-ILp,where(15)μ-ILp·CLp,d=∑q∈ILωPGIFp,q·ILp·CLp,d∑q∈ILωPGIFp,qμ-CLp,d=∑q∈ILωPGIFp,q·CLq,d∑q∈ILωPGIFp,qμ-ILp=∑q∈ILωPGIFp,q·ILq∑q∈ILωPGIFp,qσ-ILp=∑q∈ILωPGIFp,q·ILq·ILq∑q∈ILωPGIFp,q-μ-ILp2.To reduce the computational complexity of (15), these equations can be decomposed into four one-dimensional convolutions using the principle of weight propagation, as developed in [21]. Let us take the numerator of μ-(IL(p)·CL(p,d)), denoted as Z(p,d), as an example:(16)Zp,d=∑q∈ILωPGIFp,q·ILp·CLp,d=∑i=1s1∑j=1s2W¯i,xH·W¯j,yV·ILi,j·CLi,j,d,where the weight ωPGIF is calculated according to (10) and s1 and s2 are the width and height of the guidance image, respectively. Firstly, we define the left-to-right and the right-to-left weighted sums, ML(x,j,d) and MR(x,j,d), as(17)MLx,j,d=∑i=1x-1W¯i,xH·ILi,j·CLi,j,d+ILx,j·CLx,j,dMRx,j,d=∑i=x+1s1W¯i,xH·ILi,j·CLi,j,d+ILx,j·CLx,j,d.Since W¯i,xH=W¯i,x-1H·W¯x-1,xH,ML(x,j,d) can be written in recursive form:(18)MLx,j,d=∑i=1x-2W¯i,xH·ILi,j·CLi,j,d+W¯x-1,xH·ILx-1,j·CLx-1,j,d+ILx,j·CLx,j,d=W¯x-1,xH·∑i=1x-2W¯i,x-1H·ILi,j·CLi,j,d+ILx-1,j·CLx-1,j,d+ILx,j·CLx,j,d=W¯x-1,xH·MLx-1,j,d+ILx,j·CLx,j,d.Similarly,(19)MRx,j,d=W¯x,x+1H·MRx+1,j,d+ILx,j·CLx,j,d.Secondly, we define the horizontal weighted sum, denoted asMH(x,j,d):(20)MHx,j,d=MLx,j,d+MRx,j,d-ILx,j·CLx,j,d.Based on a similar procedure, we define the top-to-bottom and bottom-to-top weighted sums, MT(x,y,d) and MB(x,y,d), as(21)MTx,y,d=W¯y-1,yV·MTx,y-1,d+ILx,y·CLx,y,d=∑j=1y-1W¯j,yV·MHx,j,d+ILx,y·CLx,y,d(22)MBx,y,d=W¯y,y+1V·MBx,y+1,d+ILx,y·CLx,y,d=∑j=y+1s2W¯j,yV·MHx,j,d+ILx,y·CLx,y,d.Hence, Z(p,d) can be calculated as(23)Zp,d=Zx,y,d=MTx,y,d+MBx,y,d-ILx,y·CLx,y,d.In sum, we can computeZ(p,d) using (23), where MT(x,y,d) and MB(x,y,d) are obtained by (21) and (22). Next, MH(x,j,d) in (21) and (22) are calculated from (20). Finally, ML(x,j,d) and MR(x,j,d) of (20) are recursively calculated using (18) and (19).We have that all of the two-dimensional convolutions in (15) can be decomposed into one-dimensional convolution operations in four directions with computational complexity O(N), where N is the total number of pixels of the guidance image.Once we have the optimal filter parametersa∗(p,d) and b∗(p,d), the aggregated matching cost volume is calculated as(24)CLPGIFp,d=a∗p,d·ILp+b∗p,d.We may substitute the parameters of (24) by the optimal values of (14) and rearrange them into the following form:(25)CLPGIFp,d=∑q∈ILωPGIFp,q∑q∈ILωPGIFp,q1+ILp-μ-ILp·ILq-μ-ILpσ-ILp+ε·CLq,d=∑q∈ILWp,qPGIF·CLq,d.The disparity map, denoted asDL(p), can then be obtained by the winner-take-all operation on the aggregated cost volume CLPGIF(p,d):(26)DLp=argmind∈RCLPGIFp,d,where R={dmin,…,dmax} is the range of disparity values.A close comparison between (7) and (25) indicates that the effective region of (25) extends from the local aggregation window Ω of (7) to the entire guided image IL, where the weight of (25) includes the weight ωPGIF of the energy function defined in (9).In addition, in order to speed up the calculation, [14] suggests subsampling both the cost volume and the guidance image to calculate the linear coefficients. We can follow the same process by subsampling IL and CL to calculate the optimal filter parameters and then use them for normal aggregation. This will accelerate the algorithm to O(N/4).


## 3. Experimental Study

Extensive experiments were conducted to verify the effectiveness of the proposed scheme in calculating disparity maps. We studied five representative stereo matching algorithms and two versions of the proposed scheme:(i)
The Fast Cost Volume Filtering, denoted as FCVF [13](ii)
The Full-Image Guided Filtering for Fast Stereo Matching, denoted as FIF [21](iii)
The Fast Guided Image Filtering, denoted as FGIF [14](iv)
The Adaptive Guided Image Filter, denoted as AGIF [20](v)
The Weighted Guided Image Filtering, denoted as WGIF [17](vi)
The proposed scheme, which is of computational complexity O(N), denoted as PGIF(N)(vii)
The proposed scheme with downsampling in calculating the linear parameters, which is of computational complexity O(N/4), denoted as PGIF(N/4).It is worth noting that there is only one design parameter,β of (11), in our approach. For all of these computational experiments, we fixed its value as 4. Besides, the experiments are performed in MATLAB 2017b using Intel Core I7 3610 with 16 GB RAM.We tested these frameworks using the Middlebury (version 3) benchmark stereo database [22], which has a collection of stereo pairs for matching performance evaluation. Among them, 15 pairs were used, from “Adirondack” to “Vintage”, in the “trainingQ” image set. The Middlebury defines two measures for evaluation, including nonoccluded (non-occ), and all regions. The “error rates” to be presented below are denoted as “bad 1.0” in the benchmark, which is the percentage of “bad” pixels with a disparity error greater than 1.0. Furthermore, the “weighted average error” is calculated using different weights for different image pairs such that the image pairs “PianoL”, “Playroom”, “Playtable”, “Shelves”, and “Vintage” contribute only half of the error values. This arrangement of weights is applied, according to the remarks on the benchmark website, “to compensate for the varying difficulty of the different datasets.”Figure2 shows the disparity maps obtained by these algorithms without disparity refinement. The corresponding error rates in the nonoccluded region and the all-region are shown in Tables 1 and 2, respectively. The erroneous pixels in the nonoccluded area are marked in red, while those in the occluded area are marked in green.Table 1
Comparison of the error rates in the non-occluded (non-occ) region without refinements using different matching algorithms (%).
Image Sets FCVF[13] FIF[21] FGIF[14] AGIF[20] WGIF[17] PGIF(N) PGIF(N/4) Adirondack 8.78 8.29 9.19 7.81 7.87 6.43 6.51 ArtL 12.22 12.12 13.21 11.75 11.89 10.90 11.07 Jadeplant 21.81 24.18 23.12 20.87 21.45 20.03 20.36 Motorcycle 9.87 11.96 10.49 9.70 9.73 9.01 9.54 MotorcycleE 9.72 11.80 10.03 9.27 9.62 8.93 9.00 Piano 16.20 15.09 15.96 15.17 15.60 14.01 14.27 PianoL 33.44 30.81 32.78 32.84 32.99 30.25 30.76 Pipes 10.45 11.40 10.74 10.05 10.16 9.96 10.11 Playroom 22.68 19.10 23.48 20.99 21.39 16.62 18.63 Playtable 41.89 41.26 40.96 41.03 41.02 39.29 39.47 PlaytableP 13.81 16.42 14.18 12.93 13.06 13.08 13.20 Recycle 10.52 8.26 10.72 10.13 10.39 8.38 9.03 Shelves 39.52 32.48 39.60 39.32 39.76 36.11 38.02 Teddy 7.18 6.80 7.30 7.00 7.09 6.33 6.49 Vintage 33.22 37.59 35.74 32.53 32.55 30.05 31.57 Weighted Average 16.47 16.56 16.90 15.84 16.06 14.66 15.10Table 2
Comparison of the error rates in the all-region without refinement using different matching algorithms (%). The best score for the same image set is marked in bold.
Image Sets FCVF[13] FIF[21] FGIF[14] AGIF[20] WGIF[17] PGIF(N) PGIF(N/4) Adirondack 10.09 10.22 10.53 9.57 9.68 8.39 8.38 ArtL 21.90 21.96 22.93 21.81 22.02 21.64 21.22 Jadeplant 34.51 36.88 35.60 34.09 34.46 33.45 33.49 Motorcycle 13.50 15.80 14.16 13.59 13.64 13.23 13.47 MotorcycleE 13.68 15.63 13.78 13.24 13.55 13.25 13.09 Piano 19.99 19.49 19.81 19.34 19.45 18.29 18.38 PianoL 36.43 34.44 35.87 36.04 36.14 33.91 34.22 Pipes 21.45 22.66 21.63 21.17 21.30 21.52 21.44 Playroom 30.61 27.78 31.45 29.07 29.34 25.38 27.18 Playtable 44.65 44.13 43.86 43.98 44.00 42.63 42.65 PlaytableP 17.29 19.81 18.35 15.92 16.39 17.55 17.37 Recycle 12.30 10.22 12.50 12.42 12.72 10.50 11.21 Shelves 40.14 33.62 40.29 39.85 40.27 36.89 38.66 Teddy 12.59 12.44 12.72 12.48 12.57 11.94 12.09 Vintage 37.18 40.99 39.55 36.38 36.39 34.01 35.41 Weighted Average 21.74 22.05 22.20 21.30 21.51 20.49 20.74Figure 2
Comparison of disparity maps obtained by different algorithms without refinement. The erroneous pixels in the nonoccluded area are marked in red, while those in the occluded area are marked in green.
(a)
Color images selected from the benchmark: Adirondack, Piano, Playroom, Recycle, and Vintage (b)
Ground truth disparity maps (c)
Disparity maps generated by FCVF [13] (d)
Disparity maps generated by FIF [21] (e)
Disparity maps generated by FGIF [14] (f)
Disparity maps generated by AGIF [20] (g)
Disparity maps generated by WGIF [17] (h)
Disparity maps generated by PGIF(N) (i)
Disparity maps generated by PGIF(N/4)According to the results, FIF [21] performs best for Recycle and Shelves, and AGIF [20] performs best for PlaytableP in the nonoccluded (non-occ) region. Besides, FIF [21] performs best for Recycle and Shelves, and AGIF [20] performs best for MotorcycleE, Pipes, and PlaytableP in the all-region. However, PGIF(N) performs best in all other situations, including both the nonoccluded region and the all-region. It is clear that PGIF(N) has the best ranking for the weighted average error performance, which is 14.66% for the nonoccluded region and 20.49% for the all-region, while PGIF(N/4) is just second to it, which is 15.10% for the nonoccluded region and 20.74% for the all-region, respectively.The times required to run these stereo matching algorithms are shown in Table3. As expected, PGIF(N/4) only needs to calculate a quarter of the time required for PGIF(N). Furthermore, if we check the time required to run the stereo matching algorithm, we have that PGIF(N) and FIF [21] belong to the same level, while PGIF(N/4) and FGIF [14] belong to the same level. We may conclude that PGIF(N/4) is the best scheme when considering both the correctness of the matching and the efficiency of the calculation.Table 3
Comparison of the time required to run the stereo matching algorithms (s). The smallest value for the same image set is marked in bold.
Methods Adirondack Piano Playroom Recycle Vintage FCVF[13] 16 12 17 13 37 FIF[21] 31 24 33 26 74 FGIF[14] 4 4 5 4 10 AGIF[20] 55 48 57 49 106 WGIF[17] 18 14 20 16 41 PGIF(N) 33 26 35 27 77 PGIF(N/4) 8 6 9 7 17Similar to the proposed scheme, FIF [21] also uses the entire image for aggregation. However, in addition to the difference between the methods in cost-volume filtering, a step function f(z), defined in (12), is introduced in the proposed scheme to reduce the effects of noise. It is important to clarify whether the performance improvement of the proposed solution is mainly due to this function. Therefore, extensive experiments were conducted on this clarification and are summarized in Figure 3, in which the same color convention as used in Figure 2 is applied. Based on these results, we have that using the step function does improve the performance of FIF [21]. Importantly, our method is still better than the modified FIF [21].Figure 3
Performance comparison of error rates in the all-region of disparity maps using different schemes. These experiments were performed to verify the importance of using a step function (the same color convention as in Figure2 is applied here).
(a)
Color images selected from the benchmark (from left to right): Jadeplant, Motorcycle, Pipes, and PlaytableP (b)
Ground truth disparity maps (c)
Error rates in the all-region using the original FIF [21] (d)
Error rates in the all-region with the step functionf(z) in applying FIF [21] (e)
Error rates in the all-region using PGIF(N)In addition, there are several parameters that may affect the performance of the algorithms under study, including the parameterr of FCVF [13], the parameter σ of FIF [21], and the parameter β of the PGIF(N). With a close look at the filter kernel of FCVF [13], included in (7), we have that the resulting disparity map is smoother with the increase of the parameter r. The parameter σ in the filter kernel of FIF [21], presented in (3), and the parameter β of the proposed scheme both have the similar behaviors.We studied their effects on error rates through extended experimental calculations and the results are summarized in Figure4. Based on the results, we have that the best values of these parameters are r = 5, σ = 0.11, and β = 4. These values were applied to the experimental calculations presented above.Figure 4
Effects of parameter values on the weighted average error rate.
(a)
The effect of parameterr on the weighted average error rate of FCVF [13] (b)
The effect of parameterσ on the weighted average error rate of FIF [21] (c)
The effect of parameterβ on the weighted average error rate of the PGIF(N)


## 4. Conclusions

Inspired by FIF [21] and FCVF [13], we propose a new GIF-based aggregation scheme for the calculation of dense disparity maps. The scheme uses the entire image for cost calculation and exploits the weight propagation [21] method for efficient computation.We redesign the energy function of the guided image filter [12] so that we can not only preserve the original weight propagation structure [21] but also embed the effects of distance and intensity differences in the weights. This arrangement allows efficient use of the entire image when the GIF [12] scheme is used for cost aggregation. Besides, there is only one design parameter of the proposed scheme. We also suggest the introduction of a step function in the calculation of weights to attenuate noise.A performance evaluation using the Middlebury (version 3) benchmark stereo database [22] shows that the proposed solution provides superior disparity accuracy and comparable processing speed compared to representative aggregation methods. The experimental results also justify the use of a step function when calculating the aggregate weights.



---

*Source: 3128172-2019-04-17.xml*