# Detection and characterization of defects on mechanical structures by using 3D vision

Zakaria Belbacha, Hamdi Ben Abdallah, Igor Jovančević, Jean-José Orteu, Ludovic Brethes

► To cite this version:

Zakaria Belbacha, Hamdi Ben Abdallah, Igor Jovančević, Jean-José Orteu, Ludovic Brethes. Detection and characterization of defects on mechanical structures by using 3D vision. QCAV'2023 - the 16th international conference quality control by artificial vision, Jun 2023, Albi, France. 7 p., 10.1117/12.3000509. hal-04168243

HAL Id: hal-04168243

https://imt-mines-albi.hal.science/hal-04168243

Submitted on 21 Jul 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Detection and characterization of defects on mechanical structures by using 3D vision

Zakaria Belbacha<sup>a</sup>, Hamdi Ben Abdallah<sup>a</sup>, Igor Jovančević<sup>b</sup>, Jean-José Orteu<sup>a</sup>, and Ludovic Brèthes<sup>c</sup>

<sup>a</sup>Institut Clément Ader (ICA) ; Université de Toulouse ; CNRS, IMT Mines Albi, INSA, UPS, ISAE ; Campus Jarlard, 81013 Albi, France

<sup>b</sup>Faculty of Natural Sciences and Mathematics, University of Montenegro, Cetinjska 2, 81000 Podgorica, Montenegro

<sup>c</sup>Diota, 231 Pierre and Marie Curie Street, 31670 Labège, France

## ABSTRACT

This paper deals with the detection and characterization of surface damages (a dent, a crack, etc.) on mechanical surfaces using 2D/3D vision (3D scanner and/or 2D RGB camera). The main innovative aspect lies in the exploitation of the Computer Aided Design model, when it is available, with two possible scenarios: "manual control" via a hand-held 3D scanner carried by an operator, or "automated control" via a 3D scanner carried by a cobot. This research work has been carried out within the joint research laboratory "Inspection 4.0" between IMT Mines Albi/ICA and the DIOTA company, specialized in the development of numerical tools for Industry 4.0.

**Keywords:** aerospace, visual inspection, defect detection, defect characterization, 3D point cloud, deep learning

## 1. INTRODUCTION

In the field of aeronautics and automotive industry, it is essential to perform periodic inspections, whether on structural parts before the assembling phase, on assembled sections before delivery, or a pre-flight inspection of an aircraft in service. We distinguish two types of control: (1) "conformity control": verifying that a part or a mechanical assembly conforms to its Computer Aided Design (CAD) model (digital model) considered as a reference. (2) "damage detection and characterization": detecting potential "aesthetic" defects such as dents, cracks, and characterizing them. In this work, we will focus on the second problem (damage detection and characterization) that has been tackled by using a 3D scanner carried by an operator (manual control) or by a cobot (automated control). See Figure 1. The damage detection is performed by a deep learning approach on 2D images (provided by the scanner or by an additional 2D camera mounted on the robot) and the damage characterization is performed by processing the 3D point cloud provided by the scanner.

<figure><img src="image_2.png" /><figcaption>Figure 1: (a) Our inspection approach. (b) Control with robotic platform.</figcaption></figure>
## 2. RELATED WORK

Machine-based approaches offer significant benefits in the area of industrial inspection, such as performance, enhanced safety, and increased automation. Despite the importance of inspection and maintenance in the aerospace domain, the applications of emerging fields of deep learning and computer vision in this area remain quite limited. According to the literature, only a few researchers are focused on the automation of aircraft visual inspection.

One such work is an automatic aircraft fuselage defect detection method¹ based on DNN that was pre-trained on ImageNet as a feature extractor and applied transfer learning with a binary mask for each image. The authors proposed to boost the defect detection algorithm using the region of interest detected by using SURF, and this has achieved a 6x speedup compared to a baseline approach that tests all patches. The authors have used data balancing techniques to decrease the number of no-defect samples and to augment the class of rare defects. The proposed algorithm achieves about 96.37% accuracy and it is able to detect almost all the defects.

In,² a new visual method with object detection to perform airplane external screw inspection with UAV is presented. A CNN approach is used to characterize zones of interest and extract screws from the images with a prior model used for matching. Then, computer vision algorithms are employed to evaluate the status of each screw and detect missing and loose ones. The authors have trained a 2D pattern generative model via unsupervised learning using Generative Adversarial Network (GAN) architecture to create a pattern associated with the screw through a bipartite graph approach.

³ presents an analysis of different machine-learning approaches that detect rare classes of defects due to extremely imbalanced datasets. The problem with a model trained on this kind of dataset is that the model learns to achieve high accuracy by consistently predicting the majority class, even if recognizing the minority class is of equal or more importance when applying the model to a real-world scenario. To address this problem, the authors compared several methods with different training dataset sizes with balanced classes. Then, a hybrid approach combines classic deep learning models and few-shot learning approaches such as matching network and Prototypical network which can learn from a few samples.

The visual inspection process by using a drone-based system and sensors in a smart hangar, or by using mobile platforms⁴ can have a major impact on reducing inspection time and also the Aircraft on Ground time (AOG: a plane can not fly for technical reasons), especially if the automated inspection system is able to assess the criticality and severity of the damages according to the Aircraft Maintenance Manual and Structure Repair Manual.

The automated drone inspection reduces the AOG and preparation time up to 90⁵%, since the Remote Automated Plane Inspection and Dissemination System (RAPID⁵) can quickly reach difficult areas such as the empennage and both wings, without intensive labor or dedicated facilities and infrastructure. It greatly increases workplace safety and cost savings, as there is no heavy or cumbersome equipment that may further damage the aircraft or place engineers in potentially dangerous situations. For instance, the time to locate lightning damage using aerial work platforms is between 8 and 12 hours. This time can be reduced by 75% if MainBlades⁶ system inspection is used.

Our defect detection phase is based on a deep learning object detection approach. Improving the accuracy of object detection in computer vision is an ongoing research area. When the error rate in object detection decreases, it means that the machine is able to detect and classify objects more effectively. For instance, the machine error went from 26% in 2011, to only 3% in 2016 which is less than human error claimed to be 5⁷%.

# 3. DETECTION OF DAMAGES FROM 2D IMAGES

## 3.1 Methodology

The developed damage detection method processes 2D images. These images are acquired either by an additional 2D camera or directly by a 3D scanner and its SDK if it provides 2D textured images. It should be noted that the 3D scanner is mounted on a robot effector arm. The proposed method for 2D defect detection, based on supervised deep learning, consists of a pre-trained neural network model able to detect and localize one or several damages. In this work, we refer to the ground truth which is a result of a human annotation process. As it can
be seen in Figure 2, we mainly use 3 mechanical parts to provide an annotated dataset of real damages. We manually label the "dent" defects as class 1 (red color), and "crack" defects as class 2 (pink color) as shown in Figure 2. In the following section, we explain our choice of CNN-based detector architecture. Afterwards, we introduce a technique to improve the quality of the prediction.

<figure><img src="image_3.png" /><figcaption>Figure 2: Annotating images of damaged parts, by a human: (a) composite vein panel 1, (b) composite vein panel 3, (c) metal sheet, (d) "dent" defects, (e) "crack" defects.</figcaption></figure>

## 3.2 Overview of YOLOv5 based on Transformer Prediction Head (TPH)

There are many CNN-based object detectors, whether one-stage or multi-stage detectors with high inference speed, precise localization, and recognition accuracy, like SSD,⁸ R-CNN,⁹ Fast R-CNN¹⁰ and Faster R-CNN.¹¹ We have chosen YOLOv5, a single-stage object detector model which is mainly composed of three components (like all models of this type): backbone, neck, and a head to make dense predictions from the input information as shown in Figure 3.

<figure><img src="image_4.png" /><figcaption>Figure 3: Single-stage detector architecture.</figcaption></figure>

First, YOLOv5 resizes the incoming images of different size to 640 x 640 resolution, combines 4 images into a single to complete the tasks of mosaic and mixup¹² data augmentation, anchor frame with image scaling calculation, and then sends them to the backbone network. The specific architecture of the backbone in YOLOv5 is based on a variant of the EfficientNet model, which is known for its efficient use of resources while maintaining high performance. This backbone is designed to strike a balance between accuracy and computational efficiency.

The backbone in YOLOv5 consists of a series of downsample blocks, which are composed of convolutional layers, activation functions, and downsampling operations. These blocks progressively reduce the spatial dimensions of the feature maps while increasing the number of channels, capturing both local and global context. By
reducing the spatial dimensions, the backbone enables the subsequent detection layers to have a larger receptive field, allowing them to capture information from a wider region of the input image.

For the neck, it uses the *dynamic anchor boxes* method for generating the anchor boxes, with a clustering algorithm to group the ground truth bounding boxes into clusters, then using the centroids of the clusters as the anchor boxes. This allows the anchor boxes to be more closely aligned with the detected objects size and shape by introducing *Spatial Pyramid Pooling* (SPP) layers to lower the spatial resolution of the feature maps.¹³

In order to make the prediction more stronger to better detect different damages at various scales, and particularly tiny-scale, we add an additional prediction head and we replace CNN-based prediction heads with *Transformer Prediction Heads (TPH)*, constructing the TPH-YOLOv5 model. The transformer encoder block was inspired by the vision transformer,¹⁴ and modified by replacing some convolutional blocks and CSP bottle neck in the original version of YOLOv5 with these transformer encoder blocks. Each transformer encoder block includes two sub-layers. The first sub-layer is a multi-head attention layer and is followed by the second sub-layer MLP. The MLP sub-layer is a fully-connected layer. Each sub-layer implements the residual connection in order to increase the ability to capture different local information. TPH gives a performance gain of about 7%.¹⁵

## 3.3 Results of training

As illustrated in Figure 4, the output is one bounding box - region of interest (ROI) around each detected defect. We mainly use 3 mechanical parts to provide an annotated dataset of real damages (cracks and dents) with manual annotation. We generated more than 3000 annotated and augmented images. Training and validation data represent 81% and 19% of the dataset, respectively. After training, the model achieves an accuracy of 99% with 0.995 mAP (mean average precision). Real time tests have been made to evaluate the robustness of our model. Currently, we are working on expanding our dataset to make our model more general and cover a wider range of industrial parts. We are also considering the integration of a new additional layer with instance segmentation capability. For new and external datasets, we apply transfer learning (reuse of the pre-trained model) which offers a reduction in training time and improving neural network performance.

<figure><img src="image_5.png" /><figcaption>Figure 4: Detection of damages on 2D images: (a) composite panel 3, (b) metal sheet, (c) damage number 7 on the metal sheet (scanner texture image).</figcaption></figure>

# 4. CHARACTERIZATION OF DAMAGES FROM 3D POINT CLOUDS

## 4.1 Characterization using the CAD model if it is available

When the CAD model is available, the characterization algorithm works in two steps as shown in Figure 5: the first step is to align the CAD model and the scanned point cloud by applying a global alignment based on the 4PCS algorithm,¹⁶ then a local alignment based on Anderson acceleration approach¹⁷ and robust error metric based on Welch's function¹⁸ which is less sensitive to outliers and partial overlaps. The second step is the measurement of the deviation between the aligned point cloud and the CAD model. The technique¹⁹ is based on the calculation of the distance from a point to a triangle, i.e. from a point to the plane of the triangle if its projection on it is inside the triangle, and from the point to the nearest edge otherwise.
<figure><img src="image_6.png" /><figcaption>Figure 5: Characterization of damages using the CAD model: (a) reference part in blue and scanned point cloud in green, (b) Point cloud registration and characterization of damages (depth, width, length).</figcaption></figure>

## 4.2 Characterization without using the CAD model

When the CAD model of the part is not available (Figure 6), the characterization algorithm works in two steps:

* Step 1: Creation of a virtual model by surface approximation according to the prior information: The chosen model is a plane when we are inspecting a simple surface, and it is a 2.5D quadratic model when the inspected surface is slightly curved. In the case when the surface is simple, the model is adjusted by the least squares method, whereas when the inspected surface is complex, a B-Spline is fitted to the point cloud, using principal component analysis (PCA), followed by a refinement and triangulation of the fitted B-Spline surface, to obtain a smooth and parametric surface representation.

<figure><img src="image_7.png" /><figcaption>Figure 6: Surface approximation for virtual model creation: (a) simple planar surface, (b) simple curved surface, (c) complex curved surface, (d) Complex free-form surface.</figcaption></figure>

* Step 2: The measurement of the deviation between the scanned point cloud and the virtual model.

<figure><img src="image_8.png" /><figcaption>Figure 7: (a) Composite part panel 1 with damage (b) Scanned point cloud of damage (c) Characterization of the damage.</figcaption></figure>

## 5. INTERCOMPARISON WITH SEVERAL SYSTEMS FOR DAMAGE ANALYSIS

In order to evaluate the methods we have developed, in particular our DECADOM Smart Inspection software dedicated to characterization, we opted to carry out an intercomparison matrix of damage analysis and post-processing between the results of different inspection softwares and 3D numerisation systems used for quality control and dimensional analysis.

When comparing these systems, several factors can be considered:

* Accuracy and precision: Evaluate the accuracy and precision of each system in capturing and measuring 3D data. This includes assessing their ability to capture fine details, handle different surface types, and maintain measurement consistency.
* Compatibility and integration: Consider the compatibility of each system with different hardware devices, file formats, and industry standards. Integration with other software tools or platforms used in the production workflow is also important.

* Feature set: Compare the features and functionalities offered by each system: alignment, filtering, mesh generation, surface deviation analysis, Geometric Dimensioning and Tolerancing GD&T analysis, and reporting capabilities.

* User interface and ease of use: Assess the user interface and ease of use of each system. A user-friendly interface with intuitive workflows can enhance productivity and efficiency.

* Cost: Consider the cost of each system, including initial investment, licensing, and ongoing maintenance fees. Evaluate the value for money based on the specific needs of the particular application.

* Support and training: Evaluate the technical support and training resources provided by each system's developer. Access to documentation and responsive customer support to maximize the benefits of the software.

We compared several inspection and damage analysis systems, such as SmartDent3D, Polyworks inspector, DentCheck, GelSight, GOM Inspect etc. Below is a comparison table for some of the damages present on composite panel 1 (Figure 2). The table contains results obtained with our software DC Smart Inspection, and GOM Inspect.

<table><caption>Table 1: Damage inspection results of composite panel 1 with DC Smart inspection and GOM Inspect software.</caption><thead><tr><th rowspan="2">Part name</th><th rowspan="2">Damage</th><th colspan="2">DC Smart Inspection</th><th colspan="2">GOM Inspect</th></tr><tr><th>Surface( w × I)</th><th>Depth(mm)</th><th>Surface (w × l)</th><th>Depth (mm)</th></tr></thead><tbody><tr><td rowspan="3">Composite part panel 1</td><td>D1</td><td>8.10 × 12.75</td><td>3.5</td><td>6.95 × 11.43</td><td>3.75</td></tr><tr><td>D4</td><td>9.42 × 13.19</td><td>3.13</td><td>12.29 × 15.14</td><td>3.19</td></tr><tr><td>D5</td><td>15.78 × 15.28</td><td>1.21</td><td>18.55 × 16.54</td><td>1.26</td></tr></tbody></table>

It can be seen that our results are comparable to the results obtained with GOM Inspect, notably in estimating depth of the defect. It should be noted that the majority of the systems require manual detection and selection by the user, in order to launch inspection process, while automatic detection and characterization algorithm is implemented in our software. Hence, there is a slight difference in damaged surface area estimation.

## 6. CONCLUSION

In this article, we propose an approach for the detection and characterization of damage on mechanical parts, more specifically in the aerospace field. The aim is to detect the presence of potential damage such as dents or cracks, then characterize it by assessing its extent (surface, depth) and its degree of criticality. We address this in two stages.

First, we propose defect detection based on 2D image analysis. 2D images are acquired by a camera or a scanner carried by a robot. We have developed a neural network model based on the YOLOv5-TPH architecture able to identify and localize (ROI) damages on aeronautical parts. In order to bring and enhance the variability of the training data set, we set up a tool for data augmentation and transformation, as well as automatic labeling. Experimental results show that the model is highly accurate in detecting defects.

Next, we treat the point clouds acquired by a 3D scanner to characterize the damage. Using CAD model as a reference, we can carry out the registration between the scanned point cloud with potential damages and the reference model, then compute the surface deviation. When a CAD model is not available, we make a surface approximation based on a scanned point cloud to create a virtual CAD model. In order to evaluate and validate our analysis and inspection results, we conducted an intercomparison campaign with several systems for damage analysis.

## ACKNOWLEDGMENTS

This research work is co-funded by the French "Région Occitanie." We would like to thank the "Région Occitanie" for the funding of this research, and CETIM Sud Ouest for the fruitful collaboration.
REFERENCES

[1] Malekzadeh, T., Abdollahzadeh, M., Nejati, H., and Cheung, N.-M., "Aircraft fuselage defect detection using deep neural networks," *arXiv preprint arXiv:1712.09213* (2017).

[2] Miranda, J., Larnier, S., Herbulot, A., and Devy, M., "Uav-based inspection of airplane exterior screws with computer vision," in [*14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications.*], (2019).

[3] Miranda, J., Veith, J., Larnier, S., Herbulot, A., and Devy, M., "Machine learning approaches for defect classification on aircraft fuselage images acquired by an uav," in [*Fourteenth International Conference on Quality Control by Artificial Vision*], **11172**, 49–56, SPIE (2019).

[4] Jovančević, I., Larnier, S., Orteu, J.-J., and Sentenac, T., "Automated exterior inspection of an aircraft with a pan-tilt-zoom camera mounted on a mobile robot," *Journal of Electronic Imaging* **24**(6), 061110–061110 (2015).

[5] MRO Drone: RAPID (Remote Automated Plane Inspection and Dissemination System). Available online: https://www.mrodrone.net/.

[6] MainBlades: automated aircraft drone inspections. Available online: https://mainblades.com/.

[7] Gina Smith. Google Brain chief: AI tops humans in computer vision, and healthcare will never be the same. Aeronews, available online: https://siliconangle.com/2017/09/27/google-brain-chief-jeff-dean-ai-beats-humans-computer-vision-healthcare-will-never/.

[8] SSD official paper: https://arxiv.org/pdf/1512.02325.pdf.

[9] RCNN official paper: https://arxiv.org/pdf/1311.2524.pdf.

[10] Fast RCNN official paper: https://arxiv.org/pdf/1504.08083.pdf.

[11] Faster RCNN official paper: https://arxiv.org/pdf/1506.01497.pdf.

[12] Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D., "mixup: Beyond empirical risk minimization," *arXiv preprint arXiv:1710.09412* (2017).

[13] Redmon, J., Divvala, S., Girshick, R., and Farhadi, A., "You only look once: Unified, real-time object detection," in [*Proceedings of the IEEE conference on computer vision and pattern recognition*], 779–788 (2016).

[14] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al., "An image is worth 16x16 words: Transformers for image recognition at scale," *arXiv preprint arXiv:2010.11929* (2020).

[15] Zhu, X., Lyu, S., Wang, X., and Zhao, Q., "Tph-yolov5: Improved yolov5 based on transformer prediction head for object detection on drone-captured scenarios," in [*Proceedings of the IEEE/CVF International Conference on Computer Vision*], 2778–2788 (2021).

[16] Aiger, D., Mitra, N. J., and Cohen-Or, D., "4-points congruent sets for robust pairwise surface registration," in [*ACM SIGGRAPH 2008 papers*], 1–10 (2008).

[17] Pavlov, A. L., Ovchinnikov, G. W., Derbyshev, D. Y., Tsetserukou, D., and Oseledets, I. V., "Aa-icp: Iterative closest point with anderson acceleration," in [*2018 IEEE International Conference on Robotics and Automation (ICRA)*], 3407–3412, IEEE (2018).

[18] Holland, P. W. and Welsch, R. E., "Robust regression using iteratively reweighted least-squares," *Communications in Statistics-theory and Methods* **6**(9), 813–827 (1977).

[19] Girardeau-Montaut, D., *Détection de changement sur des données géométriques tridimensionnelles*, PhD thesis, Télécom ParisTech (2006).