# Semantic Structure from Motion for Railroad Bridges Using Deep Learning
## 2 Background
### SfM  
SfM is one of the most popular methods for generation of a 3D model using image data [27; 28; 29]. The first step in SfM is to identify the correspondence between the images. SfM uses scale-invariant feature transform (SIFT) and speeded-up robust feature (SURF) algorithms to find feature points in photographic data and identify correspondence between feature points [30; 31]. However, not all of the obtained feature points belong to a correspondence relationship. An outlier in which the feature points do not coincide can exist. In SfM, random sample consensus (RANSAC) was used, which is a method of randomly selecting sample data to remove outliers and selecting data with the largest consensus. As shown in Figure 4, a feature model of each photograph was obtained by applying RANSAC and a feature matching process was performed to compare the feature models. In this process, if the points match through feature matching, it is determined as a correspondence relationship; if not, it is regarded as an outlier and removed.  
Figure 3: Original convolution layer and residual learning block. (a) Original convolution layer; (b) residual learning block.  
When the correspondence between the pictures is grasped through the above process, the position of the camera can be estimated, and the feature points can be expressed as 3D PCD. However, it is not possible to express all shapes of an object with 3D PCD generated only with feature points. Therefore, the 3D dense reconstruction technique is used to obtain dense 3D PCD. The 3D dense reconstruction technique is a method of interpolation using pixels around feature points. Normalized cross correlation (NCC) and inverse distance weighted techniques are used.  
NCC is a technique for measurement of geometric similarity by comparing the red-green-blue (RGB) values of pixels. The 3D dense reconstruction technique refers to the surrounding pixels of the feature point. The RGB value of the surrounding pixels is extracted using a filter on the pixel size. If the RGB obtained from the 3D dense reconstruction technique is calculated using Equation (1), the NCC can be obtained.  
\[\frac{1}{n}\sum_{x,y}\frac{\Big{(}f(x,y)-\overline{f}\Big{)}\big{(}t(x,y)- \overline{t}\big{)}}{\sigma_{f}\sigma_{t}} \tag{1}\]  
where \(n\) is the size of the filter, \(f(x,y)\) is the RGB at the coordinates of the filter, \(t(x,y)\) is the RGB at the \(x\) and \(y\) coordinates of the comparison filter, \(\overline{f}\) is the average RGB of the filter, \(\overline{t}\) is the average RGB of the comparison filter, \(\sigma_{f}\) is the standard deviation of the RGB of the filter, and \(\sigma_{t}\) is the standard deviation of the RGB of the comparison filter. As the RGB is nonnegative, NCC has a value between \(-1\) and \(1\). If NCC is close to \(1\), the two filters are similar. However, because the NCC obtained from the process does not include the distance information of the photograph, a high reliability cannot be ensured. Therefore, the process of assigning a weight to a distance is required. The NCC uses the inverse distance weighting method, expressed by  
\[Z_{p}=\frac{\sum_{i=1}^{n}Z_{i}W_{i}}{\sum_{i=1}^{n}W_{i}} \tag{2}\]  
where \(Z_{p}\) is the estimated value (interpolation value) of the estimation point, \(Z_{i}\) is the reference value of the position (\(x_{i}\), \(y_{i}\)), \(W_{i}\) is the weight, and \(n\) is the number of reference values. The weight \(W_{i}\) can be calculated using Equation (3), where \(d_{i}\) represents the distance,  
\[W_{i}=\frac{1}{d_{i}^{2}} \tag{3}\]  
Through this process, interpolation is performed on a section in which no feature point exists, and dense 3D PCD can be generated based on the result.  
Figure 4: Feature matching.