# An Operational Image-Based Digital Twin for Large-Scale Structures
## 5 AI-Based Surface Defect/Damage Detection  
The domain of defect/damage detection on TP surfaces falls under the category of either semantic segmentation or object detection. Given recent advancements in object detection with YOLO-based algorithms, which include benefits such as high classification accuracy and real-time throughput [11], we applied object detection to the task at hand. Incoming images originated from drone footage using a high-resolution camera (\(\sim\)45 MP), reinforcing the requirement for highly efficient neural networks. A YOLO-based architecture was chosen, due to the high network throughput and the availability of open-source implementations in common machine-learning libraries. Given the large image sizes in this project (approx. 45 MP), a network with high throughput was a key requirement. The availability of common machine-learning libraries allowed for a wider range of and easier deployment on GPU-ready machines. The neural network architecture used was YOLOv5 (see reference [11]), which, in turn, is based on the seminal YOLO (You Only Look Once) network established by [PERSON] [11]. In this family of architectures, input images are effectively divided into a grid of cells, where each cell outputs a likelihood constrained on the interval [0, 1] for the presence of one or more objects (called \"objectness\"), values describing those bounding boxes (including height, width, and box center), and their associated class likelihoods. The loss function contains terms for each of these values, where the values from cells that have objectness lower than a given threshold are ignored and not penalized during loss calculation. All of the available labeled data amounted to \(\sim\)1200 images from handheld cameras and manual drone flights, totaling \(\sim\)3500 objects of interest. Of these images, \(\sim\)230 were reserved as a validation set. We expanded the training set by applying augmentations, including rotation, translation, HSV value shifts, perspective shifts, horizontal/vertical flips, and more. A widely used performance metric for object detection is mean average precision (mAP). A target performance of 0.7 mAP was defined in the early stages of this study, which corresponded with a high-performing model on open source datasets during that time. The best performing model reached only 0.31 mAP on a hold-out set. The reason for this  
Figure 9: (**a**) The color on the CAD model represents the geometric distance between the CAD model and the LiDAR-based reconstructed model. (**b**) The histogram shows this relation, together with probabilities of the distance differences measured in meters. Green color: distances close to zero. [PERSON] and blue colors correspond to positive and negative distances, respectively. A logarithmic probability scale is used. Note that the histogram for the first peak is relatively narrow, corresponding to a low standard deviation of the geometric differences. -star contrast in the expected performance and the realized performance is attributed to the subjectivity in labeling. Contrary to most publicly used datasets, where object granularity is often not an issue, this work demonstrated data where a single object of interest may constitute many smaller objects (see Figures 10 and 11), or conversely where many smaller objects may constitute a single larger object. This decision-making is somewhat subjective and depends on the pragmatism of the labeler. As an example, the class \"Flying Rust\" often comprises 100 or more small flecks of rust on the TP surface, grouped in heterogeneous clusters. Given that the object granularity bore little consequence in our application field, we developed a performance metric that calculated the mean intersect over union (IoU) for each given class, irrespective of the number of bounding boxes that intersected with a ground truth bounding box (or vice versa). With this metric (which we abbreviated to mIoU), our model performed at 0.59. For comparison, a high performing model evaluated on the COCO 2017 dataset reached 0.69 mIoU.