# Data and Knowledge-Driven Bridge Digital Twin Modeling for Smart Operation and Maintenance
## 5 Bridge Digital Twin Modeling Using Multi-Source Heterogeneous Data
### Spatial Modeling of Bridge Digital Twin  
The spatiotemporal alignment of various bridge data sources is critical for (1) establishing correlations among various bridge defects and deformations and (2) explaining root causes of the identified structural health issues. Hence, the spatial modeling process begins with the spatiotemporal alignment of 2D image data, 3D point cloud data, and sensor data into a unified coordinate system, forming the foundation for spatial modeling of the bridge digital twin. Given that the spatial coordinates of sensor data are relatively straightforward to obtain, the critical challenge lies in aligning the 2D image data with the 3D point cloud data, which is key to achieving precise spatial modeling.  
The correspondence between 2D image data and 3D point cloud data can be established using the speeded-up robust features (SURF) algorithm, which extracts key feature points [52]. This is followed by initializing transformation matrices and parameters, with refinement achieved through the iterative closest point (ICP) algorithm to enhance alignment accuracy.  
First, SURF is applied to extract key feature points from the 2D image data. For faster computation, SURF employs the integral image method, where the integral image is defined as in Formula (22):  
\[I_{integral}(x,y)=\sum_{i=0}^{x}\sum_{j=0}^{y}I(i,j) \tag{22}\]  
Figure 8: Integration of data and knowledge-driven modules for smart bridge operation and maintenance.  
where \(I(i,j)\) represents the pixel intensity at point \((i,j)\), and \(I_{integral}(x,y)\) is the cumulative sum of pixel intensities from the top-left corner \((0,0)\) to \((x,y)\).  
SURF detects key points using the Hessian matrix, defined as in Formula (23):  
\[H(x,y,\sigma)=\begin{bmatrix}L_{xx}(x,y,\sigma)&L_{xy}(x,y,\sigma)\\ L_{xy}(x,y,\sigma)&L_{yy}(x,y,\sigma)\end{bmatrix} \tag{23}\]  
where \(L_{xx}\), \(L_{xy}\), and \(L_{yy}\) are second-order partial derivatives in the \(x\) and \(y\) directions, and \(\sigma\) represents the scale. The key feature points are identified based on the response of the Hessian matrix. Subsequently, feature points from the 2D image and geometric feature points from the 3D point cloud data are extracted and matched to form corresponding pairs. The next step involves initializing the rotation matrix \(R\) and the translation vector \(T\) between the two datasets. A least-squares method is then applied to solve for the transformation matrix, ensuring that the transformed 2D image data align as closely as possible with the corresponding points in the 3D point cloud data as in Formula (24):  
\[\min_{R,T}\sum_{i=1}^{N}\|p_{i}^{3D}-\left(R\cdot p_{i}^{2D}+T\right)\|^{2} \tag{24}\]  
where \(p_{i}^{3D}\) represents the corresponding point in the 3D point cloud data, while \(p_{i}^{2D}\) denotes the feature point in the 2D image data. \(N\) refers to the total number of matching points. The initial alignment is achieved by calculating the initial rotation matrix \(R_{0}\) and translation vector \(T_{0}\) using singular value decomposition. The alignment is then iteratively refined using the iterative closest point (ICP) algorithm. This process continuously reduces the distance between the two point sets, progressively optimizing the rotation matrix and translation vector. The iteration continues until the error converges to a predefined threshold or the maximum number of iterations is reached, yielding the final optimized rotation matrix \(R\) and translation vector \(T\).  
Upon completing the geometric alignment of the 2D image data and 3D point cloud data, the geometric alignment of the sensor data is accomplished by measuring the actual installation locations of the sensors to obtain their spatial coordinates. These coordinates are then used to map the sensor data into the unified coordinate system.