# A Semantically Aware Multi-View 3D Reconstruction Method for Urban Applications
## 2 Materials and Methods
### Semantic-Based SGM
#### 2.4.2 Cost Aggregation  
Cost aggregation employs a global stereo matching algorithm which involves finding the optimal disparity for each pixel such that the overall energy is minimized. The energy equation is as follows:  
\[energy(X,P)=\sum_{i}DataCost(_{i},x_{i})+\sum_{j=neighbor(i)}SmoothCost\big{(} x_{i},x_{j}\big{)} \tag{12}\]  
This is a two-dimensional optimization problem. To enhance optimization efficiency, SGM (Semi-Global Matching) is used to convert the problem into an approximation of two-dimensional optimality using one-dimensional path aggregation. This approach not only improves efficiency but also ensures effectiveness. The SGM energy equation is as follows:  
\[E(D)=\sum_{x_{b}}C(x_{b},D(x_{b}))+\sum_{x_{N}}P_{1}T[\|D(x_{b})-D(x_{N})\|=1 ]+\sum_{x_{N}}P_{2}T[\|D(x_{b})-D(x_{N})\|>1] \tag{13}\]  
In Formula (13), \(C(x_{b},D(x_{b}))\) represents the matching cost based on WZNCC, and the second and third terms are the smoothing terms. Since we expect the disparity of pixels to be continuous, if the disparity of the current pixel differs slightly from its neighboring pixels (equal to 1 pixel), we assign a smaller penalty \(P_{1}\). If it is greater than one pixel, a larger penalty \(P_{2}\) is assigned. Throughthe implementation of these two penalty terms, we aim to ensure that the global disparity is as consecutive as possible. This approach is a common method in machine learning that is known as regularization constraints. Depth in real-world scenes is not completely consecutive and there are numerous abrupt depth changes. As a result, there is a need for a certain level of tolerance toward situations where the disparity exceeds one pixel. Therefore, \(P_{2}\) needs to be dynamically adjusted to account for discontinuity in disparity. Consequently, the formula for the optimized \(P_{2}\) chosen in the openMVS is as follows:  
\[P_{2}=P^{\prime}{}_{2}\times(1+\alpha\times e^{(-(I_{x_{b}}-I_{x_{b}})^{2}/(2 \times\beta^{2}))}) \tag{14}\]  
where \(I_{x_{b}}\) and \(I_{x_{N}}\) represent the brightness values of adjacent pixels. In this equation, the value of \(P^{\prime}{}_{2}\) is 4, the value of \(\alpha\) is 14, and the value of \(\beta\) is 38 [28; 32]. The purpose of implementing this formula is to determine whether two pixels are discontinuous parts based on the difference in brightness values. If it is actually a foreground-background relationship, then we reduce the penalty intensity of \(P_{2}\) for cases where the disparity exceeds one pixel. However, this method cannot completely determine whether the disparity is continuous. Therefore, we incorporate a semantic term and calculate the final value of \(P_{2}\) using a weighted computation as follows:  
\[P_{2}=P^{\prime}{}_{2}\times[0.8\times\gamma\times(T[S(x_{b})=S(x_{N})])+0.2 \times(1+\alpha\times e^{(-(I_{x_{b}}-I_{x_{N}})^{2}/(2\times\beta^{2}))})]. \tag{15}\]  
In the above equation, the value of \(\gamma\) is 48, and the other parameters are the same as in Equation (10). The weights of the grayscale criterion and the semantic criterion are 0.8 and 0.2, which are summarized through the experiments. Our purpose is to set an accurate \(P_{2}\) penalty term for the path pointed by the red arrow in Figure 4 through semantics, and then to obtain the accurate disparity of the pixel. Compared with using only the grayscale criterion to calculate the penalty term \(P_{2}\) and judging the continuity of depth, using semantic weighted Formula (15) can calculate a more accurate disparity. In the specific solution process, the idea of path cost aggregation in SGM is as follows: the matching cost under all disparities for a pixel is aggregated over all possible paths around the pixel in a one-dimensional manner to obtain the path cost value under each path. The calculation is as follows:  
\[L_{r_{i}}(x_{b},d)=C(x_{b},d)+\min \begin{bmatrix}L_{r}(x_{b}-r_{i},d),\\ L_{r_{i}}(x_{b}-r_{i},d-1)+P_{1r}\\ L_{r_{i}}(x_{b}-r_{i},d+1)+P_{1r}\\ L_{r_{i}}(x_{b}-r_{i},i)+P_{2}\end{bmatrix}