# Semi-Global Stereo Matching Algorithm Based on Multi-Scale Information Fusion
## 2 SGM-MSIF Algorithm
### Improved Cross-Scale Multi-Path Cost Aggregation
Considering both the matching accuracy and real-time performance of the algorithm, a four-path cost aggregation is used, as shown in Figure 3.  
In CSCA [16], the cost aggregation process is regarded as a weighted least squares (WLS) optimization problem for the initial matching cost volume. The optimization function at a single scale is defined as:  
\[\widetilde{C}(i,l)=\underset{z}{\text{argmin}}\frac{1}{Z_{i}}\sum_{j\in N_{i}}K (i,j)\left\|z-C(j,l)\right\|^{2} \tag{8}\]  
where \(\widetilde{C}(i,l)\) represents the aggregated cost value of pixel \(i\) under disparity \(l\), \(N_{i}\) represents the neighborhood pixels of pixel \(i\), \(\widetilde{C}(j,l)\) is the cost value of pixel \(j\) in neighborhood \(N_{i}\) under disparity \(l\), \(K(i,j)\) is the kernel function to measure the similarity between pixel \(i\) and \(j\), \(Z_{i}=\sum_{j\in N_{i}}K(i,j)\) is the regularization constant, and \(z\) represents the target cost value. The optimal solution is obtained by calculating the partial derivative of \(z\) in Equation (8), which is shown in Equation (9):  
\[\widetilde{C}(i,l)=\frac{1}{Z_{i}}\sum_{j\in N_{i}}K(i,j)C(j,l) \tag{9}\]  
In order to overcome the issue of high EMR of single scale stereo matching algorithm in low-texture regions, as in CSCA [16], constraint items are added between different scales, and the aggregated cost volumes under \(S+1\) scales are connected together. The final multi-scale optimization function is expressed in Equation (10):  
\[\hat{V}=\underset{\{z^{s}\}_{s=0}^{S}}{\text{argmin}}(\sum_{s=0}^{S}\frac{1}{Z _{i^{p}}^{s}}\sum_{j^{p}\in N_{i^{p}}}K(i^{s},j^{s})\left\|z^{s}-C^{s}(j^{s},l^ {s})\right\|^{2}+\gamma\sum_{s=1}^{S}\left\|z^{s}-z^{s-1}\right\|^{2}) \tag{10}\]  
where, \(s\in\{0,1,2,\ldots,S\}\) represents the scale parameter of the left and right images at different scales obtained by Gaussian pyramid down-sampling, \(i^{s}\) and \(j^{s}\) represent the pixels \(i\) and \(j\) under scale \(s\), \(N_{i^{p}}\) represents the neighborhood pixels of \(i^{s}\), \(l^{s}\) represents the disparity \(l\) under scale \(s\), \(C^{s}\) represents the cost value at scale \(s\) and \(C^{0}\) is the cost value at the finest scale. \(K(i^{s},j^{s})\) is the kernel function to measure the similarity between pixels \(i^{s}\) and \(j^{s}\), and \(Z_{i^{p}}^{s}=\sum_{j^{p}\in N_{i^{p}}}K(i^{s},j^{s})\) is the regularization constant. Considering the relationship between adjacent scales, constraint parameter \(\gamma(\gamma>0)\) is introduced to control the regularization intensity. \(\{z^{s}\}_{s=0}^{S}\) represents the target cost value at each scale. The \(S+1\) elements in vector \(\hat{V}=\left[\hat{C}^{0}(i^{0},l^{0}),\hat{C}^{1}(i^{1},l^{1}),\ldots,\hat{ C}^{S}(i^{s},l^{S})\right]^{T}\) represent the cost value of multi-scale information fusion in each scale. In order to obtain the optimal solution, setting \(F(\{z^{s}\}_{s=0}^{S})=\hat{V}\), and Equation (11) is obtained by calculating partial derivative of \(z^{s}\) when \(s\in\{1,2,\ldots,S-1\}\):  
\[\frac{\partial F}{\partial z^{s}}=\frac{2}{Z_{i^{p}}^{s}}\sum_{j^{p}\in N_{i^{ p}}}K(i^{s},j^{s})(z^{s}-C^{s}(j^{s},l^{s}))+2\gamma(z^{s}-z^{s-1})-2\gamma(z^{s +1}-z^{s}) \tag{11}\]  
Figure 3: 4-path cost aggregation. Combining Equation (9), Equation (11) can be simplified as:  
\[\frac{\partial F}{\partial z^{s}}=2(-\gamma z^{s-1}+(1+2\gamma)z^{s}-\gamma z^{s+ 1}-\widetilde{C}^{s}(\tilde{t}^{s},l^{s})) \tag{12}\]  
\(\frac{\partial F}{\partial z^{s}}=0\) gives:  
\[-\gamma z^{s-1}+(1+2\gamma)z^{s}-\gamma z^{s+1}=\widetilde{C}^{s}(l^{s},l^{s}) \tag{13}\]  
Similarly, when \(s=0\) and \(s=S\), we get:  
\[(1+\gamma)z^{0}-\gamma z^{1}=\widetilde{C}^{0}(i^{0},l^{0}) \tag{14}\]  
Figure 3.0  
where \(\widetilde{C}^{s}(i^{s},l^{s})\) represents the result of multi-path cost aggregation under scale \(s\), which is known.