<figure><img src="image_1.png" /></figure>

HAL
open science

# Integrated design of an aerial soft-continuum manipulator for predictive maintenance

Xinrui Yang, Mouad Kahouadji, Othman Lakhal, Rochdi Merzouki

<figure><img src="image_2.png" /></figure>

## To cite this version:

Xinrui Yang, Mouad Kahouadji, Othman Lakhal, Rochdi Merzouki. Integrated design of an aerial soft-continuum manipulator for predictive maintenance. Frontiers in Robotics and AI, 2022, 9, pp.980800. 10.3389/frobt.2022.980800. [https://hal.science/hal-03819931v1](https://hal.science/hal-03819931v1)

HAL Id: hal-03819931

https://hal.science/hal-03819931v1

Submitted on 15 May 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_4.png" /></figure>

Check for updates

OPEN ACCESS

EDITED BY

S.M.Hadi Sadati,
King's College London, United Kingdom

REVIEWED BY

Mahboubeh Keyvanara,
Eindhoven University of Technology,
Netherlands
Zicong Wu,
King's College London, United Kingdom
Mohammad Sheikh Sofla,
University of Tehran, Iran

*CORRESPONDENCE

Xinrui Yang,
xinruiyang@univ-lille.fr

SPECIALTY SECTION

This article was submitted to Soft Robotics, a section of the journal Frontiers in Robotics and AI

RECEIVED 28 June 2022

ACCEPTED 29 August 2022

PUBLISHED 20 September 2022

CITATION

Yang X, Kahouadji M, Lakhal O and
Merzouki R (2022), Integrated design of
an aerial soft-continuum manipulator
for predictive maintenance.
Front. Robot. AI 9:980800.
doi: 10.3389/frobt.2022.980800

COPYRIGHT

© 2022 Yang, Kahouadji, Lakhal and
Merzouki. This is an open-access article
distributed under the terms of the
Creative Commons Attribution License
(CC BY). The use, distribution or
reproduction in other forums is
permitted, provided the original
author(s) and the copyright owner(s) are
credited and that the original
publication in this journal is cited, in
accordance with accepted academic
practice. No use, distribution or
reproduction is permitted which does
not comply with these terms.

Integrated design of an aerial
soft-continuum manipulator for
predictive maintenance

Xinrui Yang*, Mouad Kahouadji, Othman Lakhal and
Rochdi Merzouki

Laboratory CRISTAL, University of Lille, Lille, France

This article presents an integrated concept of an aerial robot used for predictive maintenance in the construction sector. The latter can be remotely controlled, allowing the localization of cracks on wall surfaces and the adaptive deposit of the material for *in situ* repairs. The use of an aerial robot is motivated by fast intervention, allowing time and cost minimizing of overhead repairs without the need for scaffolding. It is composed of a flying mobile platform positioned in stationary mode to guide a soft continuum arm that allows to reach the area of cracks with different access points. Indeed, some constructions have complex geometries that present problems for access using rigid mechanical arms. The aerial robot uses visual sensors to automatically identify and localize cracks in walls, based on deep learning convolutional neural networks. A centerline representing the structural feature of the crack is computed. The soft continuum manipulator is used to guide the continuous deposit of the putty material to fill the microscopic crack. For this purpose, an inverse kinematic model-based control of the soft arm is developed, allowing to estimate the length of the bending tubes. The latter are then used as inputs for a neural network to predict the desired input pressure to bend the actuated soft tubes. A set of experiments was carried out on cracks located on flat and oblique surfaces, to evaluate the actual performances of the predictive maintenance mechatronic robot.

KEYWORDS

aerial soft manipulator, 3D printing in construction, deep learning, crack detection, neural network

# 1 Introduction

The construction domain faces many constraints; one of them is the lifespan of the structures which depends on the type of materials used in these structures. The used materials (concrete, metal, wood, etc.) are subject to external conditions such as climate changes and humidity. These latter cause a lot of damage and degradation that can produce cracks and it may lead to a breakdown of the building. Repairing these cracks at the first degradation stage is a subject of predictive maintenance. It can be less costly, with less mobilization time and safety for workers on site. Therefore, this topic is recent original
research in designing and controlling autonomous robots for
crack repair to anticipate important structure degradation.

Concerning concrete surfaces, it exists and works in relation
to the identification and characterization of cracks and other
structural damages. Aggelis et al. (2011) worked on a robust
methodology for damage evaluation in concrete before it
becomes visible. Also, Albishi et al. (2012) worked on metal
surfaces and proposed a new sensor based on complementary
split-ring resonators to detect surface cracks.

In addition, the research area for automation of predictive
maintenance has become wider in these last years because of the
electronic and robotic revolution. There exist various concepts
and works related to the use of robots for crack repair. In this
topic, Zhu et al. (2019) presented an arm system for roadway
crack sealing, and Tsai et al. (2013) developed an automatic
method using a mobile robot for crack detection and sealing for
roadways.

The most proposed concepts of predictive maintenance are
based on vision systems. The image processing technology is the
main step for structural health monitoring tasks, including crack
detection, quality, and safety for both modern and old
constructions. For a crack detection task, various approaches
which did not require human involvement were proposed. Based
on whether the detection system has contact with a wall surface,
the methods can be divided into two main categories: based on
contact and based on vision (without contact). Contact-based
methods usually use embedded sensors to detect and characterize
mechanical fractures based on the data recorded from the
sensors. Palermo et al. (2020) proposed a customized design
of an integrated tactile and proximity sensor. Lau (2003) used an
optical fiber to assess concrete structures. Vision-based methods
(Yao et al., 2019) rely on image structure features and they detect
the surface defects by analyzing the texture, skeleton, edge
(Canny, for example), and spectrum of the image.

Recently, deep learning convolutional neural networks (CNNs) have been used in image-related tasks such as target tracking (Feng et al., 2019) and semantic segmentation (Li et al., 2019). In addition to the fast improvement of computational ability, a CNN also has a strong ability in feature extraction, which is a key point in this kind of detection task. He et al. (2017) proposed a region-based convolutional neural network architecture for instance segmentation. Mohan and Poobal (2018) and Silva and Lucena (2018) studied and reviewed multiple studies in the field of deep learning based on crack detection and compared the performances in different experimental conditions.

Researchers have validated the feasibility and satisfying
performance of deep learning CNN approaches for surface
defect detection. Dorafshan et al. (2018a) investigated the
feasibility of using a CNN based on an AlexNet dataset
(Krizhevsky et al., 2017), in the inspection of concrete
buildings using small unmanned aerial vehicles. Dorafshan
et al. (2017) and Dorafshan and Maguire (2018) proved the

feasibility and challenges of fatigue crack detection in bridge
structures as well. As for the comparative study, the CNN shows
very robust performance compared to the traditional, well-
known edge detection methods. Dorafshan et al. (2018b)
compared the performance of common edge detectors and
deep CNNs for image-based crack detection in concrete
structures. It shows the advantage of CNNs to perform crack
detection, compared to edge detection algorithms which are
more influenced by external noises. In addition to crack
detection on concrete wall surfaces, deep CNNs can also be
used in the detection of other categories of surface imperfections,
with rather good performance. Perez et al. (2019), proposed a
CNN to detect the dampness of buildings, while discussing the
cause and effects of this type of defect.

With the development and massive implementation of
deep learning convolutional neural networks in defect
detection tasks, we focus, in this work, on developing a
model based on a CNN to identify cracks on concrete
surfaces. The transfer learning method is used in the
development of a crack detection model, i.e., we fine-tune
the hyperparameters of the crack detection model based on a
pre-trained model for multi-class instance segmentation.
When a crack is detected and located, it is then classified
in the following classes: microscopic (width < 5 mm),
mesoscopic (5 mm < width < 10 mm), and macroscopic
(width > 10 mm). In the following work, we were interested
in microscopic cracks. Then, their shapes and structural
features can be computed, extracted, and sent to the crack
reparation mechatronic robot to perform the predictive
maintenance process.

A soft-continuum manipulator has been introduced these
last years for inspection activities in the construction domain.
The monitoring of the surface state can be automated to detect
any appearance of cracks. Several systems have been developed to
automate the inspection process in some specific environments,
such as subway tunnels (Zhang et al., 2014), flexible pavement
surfaces (Oliveira and Correia, 2013) (Shi et al., 2016), bridge
decks (Prasanna et al., 2016), and buildings (Yan et al., 2017)
(Chen et al., 2016). De Paz et al. (2013) used a hexapod robot,
named Hex-piderix, with a stereo camera to inspect the cracking
on surfaces such as walls or roofs of buildings. Access to these
places is sometimes difficult for the operators. Some used
unmanned aerial vehicle (UAV) systems to diagnose cracks
on surfaces of the buildings. Crack detection is essential for
health monitoring of built infrastructure. An integrated system
that enables revisiting crack locations during building
inspections by means of a quadrature UAV is presented by
Kucuksubasi and Sorguc (2018). Phung et al. (2017) used a
UAV the data collection to create a 3D model of the targeted
structure by using laser scanners. For kinematic modeling of a
soft continuum arm, data-driven approaches have been
considered to improve the performance model reconstruction,
such as by Karlik and Aydin (2000) and Daachi et al. (2012),
where neural networks were used to approximate the behavior of
a large number of Degrees of Freedom (DoF) robot manipulators,
allowing to develop adaptive controllers in the presence of
nonlinearities. A hybrid controller combining both the model-
based and neural network approaches has been proposed by
Jiang et al. (2017) and Reinhart et al. (2017). Jiang et al. (2017)
presented a control algorithm for a manipulator that can be
extended to several segments in a 2D plane. Its kinematics is
solved in two levels. In the first level, an analytical model based on
the gradient descent method is used to determine the optimized
positions of all segment peaks based on predefined
measurements. The second level uses neural networks to
determine the pressures of each segment, taking into account
the viscoelasticity property. Reinhart et al. (2017) proposed a
hybrid, data-driven analytical approach for the early control of a
bionic manipulator. The effectiveness of this approach was
demonstrated by a reduction in tracking errors, which
represents less than half of the errors observed when using
the analytical model. Lakhal et al. (2015) used a multi-layer
neural network to approximate the solutions of the inverse
kinematic equations of the soft continuum robot, where the
time-allocation for the learning process was considerable, due
to the complex kinematic of the robot. A data-driven approach
has also been used to approximate the soft arm kinematics by
MELINGUI et al. (2014), to develop a kinematic controller
(Melingui et al., 2015). A hybrid approach is introduced by
Lakhal et al. (2015) for real-time solving of the soft arm
model. It consists of a real-time resolution and
implementation in case of trajectory tracking. A methodology
of kinematic modeling and synthesis of the approximate
solutions is proposed for the case of a continuum
manipulator. The modeling approach is issued from a
modeling of a series of parallel rigid manipulators, where the
resolution of the nonlinear model is approximated using a neural
network technique.

## 1.1 Study contribution

Repair of wall surfaces can require time and high
precision. The automation of this operation allows for
reducing the intervention time and keeping a good repair
quality. Currently, the literature does not report on integrated
solutions for all the following steps for crack repair, namely,
detection, localization, preparation, and repair. In this study,
we propose an integrated concept for automatic detection,
localization, and in situ repair of microscopic cracks with a
local intervention on the surface. It uses cooperative behavior
between an unmanned aerial vehicle (UAV) and a soft
continuum arm to facilitate the task of predictive
maintenance of cracks on flat or oblique surfaces. The
advantage of using a UAV is that it can scan a large area
on the wall surface and reach the microscopic crack location in

less time, instead of using a scaffold. The UAV control can be
autonomous or with the operator in the loop. This last option
is considered in this development, to consider the safety
requirement of using the UAV in a confined space. Then,
the soft arm allows guiding a nozzle to realize a continuous
deposit of putty materials to fill the cracks. The choice comes
from the fact that it can be operated on congested and narrow
surfaces, due to its flexibility, after maintaining the UAV
stable at its stationary position. This inherent flexibility
makes them suitable for various applications, including
tracking the shape of the cracks and operating in complex
and congested environments. The use of continuum
manipulators not only reduces the intervention time but
they are also light and easy to install and transport. This
work explains the steps of detection, localization, and repair of
microscopic cracks in terms of local intervention with a
continuous deposit of a putty material.

The article is organized as follows: Section 2 presents the predictive maintenance system for microscopic cracks in which the different components of the system are described. It also presents the deep learning approach for the identification and localization of cracks. The quantitative approach used to control the integrated mechatronic system is developed. The implementation and the obtained results are presented in section 3 followed by a discussion in section 4. The conclusion of the results and future work are provided in section 5.

## 2 Materials and methods

The aim of our concept is to repair cracks automatically
using robots and developed technologies. This task is achieved
by performing several steps. The process of this operation
called predictive maintenance is shown in Figure 1. The first
step is to scan the structures with a camera embedded on the
UAV and send the images and the current position of the
system to an on-board computer to detect cracks. Once a crack
is detected, according to the size of the crack and its location,
the reparation decision will be taken and the system generates
the shape of the crack and its centerline and they will be sent to
the mobile platform and soft robot. The next step is to
calculate the coordinates of this crack in the absolute frame
of the UAV; this operation allows the mobile platform to move
to the location of the crack. In this study, the movement of the
mobile platform to the location of the crack is done manually
using a joystick, and the UAV then keeps the same direction
and altitude. Once the mobile platform is in a stationary
position, the soft arm guides the nozzle along the
centerline of the crack when it starts pumping the material.
However, the accuracy of maintaining the UAV position can
be affected by weather conditions, external contacts, GPS
signal strength, etc.
<figure><img src="image_5.png" /><figcaption>FIGURE 1
Principal of the predictive maintenance process.</figcaption></figure>

## 2.1 System description

This part presents a general description of the predictive maintenance system, followed by a description of the methods developed for detecting and repairing microscopic cracks.

The experimental platform is a prototype for predictive maintenance, composed of an unmanned aerial vehicle (UAV) and a soft arm, as shown in Figure 2. The idea is to design an industry-scale prototype that can automatically detect, locate, and repair microscopic cracks after filling putty materials from an embedded nozzle.

### 2.1.1 Soft arm

The soft arm describes one section of the continuum manipulator called CBHA for compact bionic handling arm (Escande et al., 2015) as shown in Figure 3. It is a soft bionic manipulator inspired by an elephant trunk. It is a continuum robot and can be classified as a soft robot with intrinsic actuation. The rigidity of the soft arm is provided first by the polyamide material manufactured through a rapid prototyping process, and secondly, by the high number of interconnected bionic backbones, which provide it wide maneuverability compared
<figure><img src="image_6.png" /><figcaption>**FIGURE 2**
Predictive maintenance robot system.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**FIGURE 3**
Two serial sections of the soft continuum manipulator.</figcaption></figure>

to a rigid manipulator. Due to its flexible material, one section of the soft arm is lightweight (0.460 kg) and can carry out a maximal payload of 0.140 kg. From a macroscopic level, the bending section is composed of three continuous flexible backbone tubes (soft actuator). From a microscopic level, each tube is composed of bionic backbones. It is composed of three electro-pneumatic valves to control the pressure of each backbone tube. A total of three sensors are used to estimate the bending length from the three wire-potentiometers shown in Figure 3, which allow measuring the instantaneous lengths of the tube during bending. Also, reflective markers are installed on the tip and along the arm for shape control based on the Optitrack vision system shown in Figure 13.

### 2.1.2 Flying platform (unmanned aerial vehicle)

It is a four-rotor mobile flying platform equipped with four brushless DC motors. It is controlled by a Pixhawk autopilot, equipped with a GPS for outdoor missions and a
<figure><img src="image_8.png" /><figcaption>FIGURE 4
Architecture of the Mask RCNN.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>FIGURE 5
Results of the segmentation model applied on different images of cracks.</figcaption></figure>

PX4 flow sensor for indoor missions. Concerning indoor applications of the flying soft continuum manipulator technology, the miniature crack repair on interior surfaces of tall buildings and edifices of historical heritage is the main challenge. For that, a 3D scan of the indoor surfaces can be done a priori in offline, for the automatic detection and localization of the target cracks. The advantage of the optical flow is the position estimation using ultrasonic data and the image flow. For this considered UAV, the air supply

unit and the power supply are placed on the ground. They are connected to the UAV through wires of 10 m. This is to reduce the overall weight of the system. The UAV is used as a guide platform for the soft arm. First, the UAV is stabilized in the stationary mode in front of the crack with the operator control in the loop. A wheeled contact (Figure 2) between the UAV and the wall has allowed to maintain the system stable. The parameters of the UAV are mass = 3.1 kg and size = 0.65 m. The UAV can carry out a load of 2 kg in the presented version.
<figure><img src="image_10.png" /><figcaption>FIGURE 6
Crack shape reconstruction in the image frame.</figcaption></figure>

The soft arm, the nozzle system, and the piezo-electric valves have a weight of 1.25 kg. The fill of the crack is realized with the putty material traveling inside the nozzle. The Optitrack system is used to improve the stability of the UAV, where the tests are performed indoors, in absence of external disturbances. The current UAV system has been designed to hold a nozzle of 5 mm diameter, containing a maximum of 0.125 L of putty material (Figure 2). The nozzle is operated as a pre-loaded needle where the mandarin is replaced by a pressure-driven catheter through an electro-pneumatic valve shown in Figure 3. The vision guidance sensor of the UAV (Figure 2) collects image data through its embedded cameras and distance data through its ultrasonic sensors.

## 2.2 Crack detection and localization

This section describes a deep learning-based approach for the detection and localization of cracks. The approach explores the images captured by the embedded visual sensor (camera) of the UAV and detects existing cracks. Furthermore, a centerline (i.e., skeleton) of the crack is extracted to generate a tool path for the repairing task. Then the location of the crack within the frame of the drone and world frame is calculated. A deep learning instance segmentation model is adopted for the detection of cracks, followed by a morphological medial axis transform for the extraction of the centerline. The localization is realized by applying frame transformation to the calculated crack location.

### 2.2.1 Deep learning for crack detection

The detection of an existing crack on a wall surface is an essential step for structural health monitoring and for the procedure of predictive maintenance. Visual-based deep learning models are a set of widely implemented methods for this task. Some research works have implemented image classification (Flah et al., 2020) or object detection models (Park et al., 2020) for the detection and localization of cracks or cracked areas on concrete surfaces; however, those models do not meet our requirement for the purpose of automatic crack repair, because the tool path of the robot arm should overlap the structural shape of the crack. On the basis of the 2D image data available from visual sensors, we adopted an instance segmentation model, namely, Mask RCNN (He et al., 2017) for the detection and segmentation of cracks. Taking an image as input, the model is able to segment the crack area at the pixel level in the image frame.

Mask RCNN has been proven to be efficient and accurate in other single-target-single-class instance segmentation tasks in multiple fields including medical science (Anantharaman et al., 2018) and agriculture (Jia et al., 2020; Hameed et al., 2022).

An image dataset is prepared based on two public concrete crack image datasets containing over 100,000 images in total (Özgenel, 2017; Dorafshan et al., 2018c; Maguire et al., 2018). The images in those datasets are classified into two categories according to the existence of cracks, but without annotation. Since we need to train an instance segmentation model, instead of using directly the datasets which are ready for training a binary classification model, we select certain images to create an
<figure><img src="image_11.png" /><figcaption>FIGURE 7
Extraction of the crack centerline in the image coordinate system. (A) Crack centerline; (B) reconstruction of crack and centerline in the image coordinate system.</figcaption></figure>

annotated crack image dataset. When selecting the images to annotate, different environmental conditions are considered, such as shadows and the complexity of the concrete background, to improve the performance of the model.

The annotated dataset contains RGB images of two different sizes (256 × 256 pixels and 227 × 227 pixels); each image contains between 0 and 4 cracks. With a computer vision annotation tool (CVAT), we have annotated 300 images containing 493 crack instances and have split them into train (260 images) and validation (40 images) sets. For each image, a binary mask is created, in which the value 0 is assigned to background pixels and the value i is assigned to the pixels of the i-th crack. To enhance the diversity of the dataset, data augmentation methods are applied randomly, such as image flipping, blur, and shadowing.

The transfer learning methodology is used in the training of the model. The model is developed based on a pre-trained instance segmentation model Mask R-CNN, which is trained on a large-scale dataset, namely, Microsoft COCO (Lin et al., 2014). As mentioned previously, our annotated dataset is not a large one, while MS COCO contains over 200,000 annotated images. The pre-trained Mask R-CNN model is able to detect 80 different categories of objects, but cracks are not included. Thus, by applying transfer learning, we can train a model on our annotated crack dataset for crack detection, without losing the robustness of the model by fine-tuning the pre-trained one. The model is constructed upon ResNet50 with feature pyramid networks (FPN) as the backbone. The model is flexible and has the ability to be generalized to other instance-level recognition tasks (Zhao et al., 2019), which allows us to fine-tune for crack detection. The multi-task loss function 1 is optimized during the training. The architecture of the adopted model is shown in Figure 4. The crack segmentation task is achieved in two stages: first, the input image is fed to the backbone networks for feature extraction and then region proposal networks (ROI) generate a number of Region of Interest (ROI) candidates in which an optimal one will be obtained through regression. Then the feature map and the chosen ROI are aligned and a fixed-size feature map is obtained; it is sent to two branches, one for the classification and bounding box, while the other generates a mask for pixel-level segmentation.

$$L = L_{\text{class}} + L_{\text{box}} + L_{\text{mask}} \quad (1)$$

The training stage was carried out with an Intel(R)i7-9850H CPU, 32 GB RAM, and NVIDIA Quadro RTX3000 GPU. The model was built and trained upon the open-source Tensorflow-Keras frame. In the real-time implementation, the model takes an image frame captured by the camera and the target output contains a score, a bounding box, and a binary mask for the detected crack if it exists in the input image.

As shown in Figure 5, The trained model is tested on some sample images with different sizes that are not contained in the training or validation dataset. Figure 6 shows the bounding boxes and binary masks generated for detected crack instances. The shape of the crack can be reconstructed with its contour in the image coordinate system. The real-time implementation of the model is also tested with the same graphic card on the video stream captured by a USB camera with a resolution of 640 × 480 pixels; it achieves 2 ~ 4 frames per second (FPS).
<figure><img src="image_12.png" /><figcaption>**FIGURE 8**
Crack centerline extraction and localization. (A) Crack centerline in the raw image; (B) reconstruction of crack and centerline in the image coordinate system (pixels); (C) reconstruction of crack and centerline in the real dimension (mm).</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>**FIGURE 9**
Schematic of an inter-vertebra modeled as a parallel robot with 3 UPS-1 UP.</figcaption></figure>

### 2.2.2 Crack centerline extraction and localization

To allow the soft arm to access and follow the shape of the detected crack, the localization consists of two steps depicted as follows: centerline extraction and frame transformation.

First, we extract the centerline of the crack in the image frame. The trajectory to be followed by the robot in the reparation system should represent the structural shape of the detected crack. In the image coordinate system, given S is the set of points of the crack region bounded by C which is the

contour of the crack shape, the centerline of the crack is the set of points $p \in S$ having more than one closest point on the contour C. As the output of the detection model contains a binary mask representing the crack, by applying the morphological medial axis transformation algorithm (also referred as skeletonization) on the binary mask, we can obtain the coordinates of the points on the centerline in the image frame. Figure 7 shows an example of crack centerline extraction in the image frame. The medial axis transformation also gives the local widths of the crack at every point on the centerline, expressed in pixels units.
<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>

A frame transformation is then applied to the points on the centerline in the image frame obtained from the previous step. For the soft arm to reach the detected crack, it is necessary to locate the centerline in the frame of the soft arm and in the world frame. With a depth camera embedded in the UAV, the distance between the camera and the scanned surface can be obtained.

For the operation of predictive maintenance, the tool path for the end effector of the soft arm robot is identical to the crack centerline. The centerline is expressed as $[u \ v]^T$ in the pixel

frame. In the frame of the scanned surface, the tool path of the end effector of the robot $X_{ee}$ can be given by

$$X_{ee} = R^{-1}(M^{-1}(sU) - t) \quad (2)$$

where.

* R is the rotation matrix of the camera fixed to the UAV.

* M is the intrinsic matrix of the camera.

* $t = [t_1, t_2, t_3]$ is the translation vector of the camera with respect to the scanned surface frame.
<table><caption>TABLE 1 Results achieved by each neural network model on the test samples.</caption><thead><tr><td>Neural network topologies</td><td>Neurons</td><td>MSE</td></tr></thead><tbody><tr><td>MLP, pressures section 1 (2 layers)</td><td>11</td><td>4.3242.10<sup>-5</sup></td></tr><tr><td>MLP, pressures section 2 (2 layers)</td><td>11</td><td>3.7519.10<sup>-5</sup></td></tr></tbody></table>

* $s = t_3$ is the depth value, as the UAV stabilizes and is perpendicular to the scanned surface, and it is equal to the distance.

* $U = [u \ v \ 0]$ is the vector of the centerline in the image frame.

Figure 8 shows an example of localization of the extracted centerline in real dimension. The unit of the obtained $X_{ee}$ is in mm and will be sent to the controller of the soft arm robot to execute the maintenance procedure; the robot guides the tip of the nozzle to deposit material for repairing the localized crack in an injection way. The unit of the local widths of the crack is also transformed to mm after the localization step; this allows to classify the crack into microscopic, mesoscopic, and macroscopic according to its size.

The overall processing time from the capture of an image to the calculation of the tool path takes around 0.3 s, allowing a real-time implementation of this visual-based approach.

## 2.3 Model-based inverse kinematic control of the soft arm

### 2.3.1 Assumptions

The inverse kinematic model (IKM) of the soft arm is developed under the following assumptions:

* The continuum manipulator is considered as a series of N = 8 vertebrae.

* An inter-vertebra is a flexible and non-deformable structure with 3-DoF mobility. It is modeled with 3 UPS-1 UP (3 universal-prismatic-spheric-1 universal-prismatic) joints.

* The manipulator yaw motion is not applicable to the existing mechanical links between the tubes.

* The UAV is considered stable and static in a reference position with contact with the wall, allowing the soft arm to operate autonomously without external disturbances.

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /><figcaption>Tube 2</figcaption></figure>

<caption>FIGURE 12
Comparison between the desired pressures and the predicted pressures of the three bending tubes. (A) Tube 1. (B) Tube 2. (C) Tube 3.</caption>
<figure><img src="image_19.png" /><figcaption>FIGURE 13
Stereo-vision system for trajectory tracking.</figcaption></figure>

Based on these assumptions, our soft arm is a series of eight parallel robots composed of 8 vertebrae with 24 DoF in total (Figure 9).

It consists of a lower and upper vertebra connected by three limbs with the same kinematic configuration, and a central leg. The limbs are modeled by a kinematic configuration of type UPS, in which only the prismatic joints are active, allowing to control the position and orientation of the upper vertebra relative to the lower vertebra. $q_{ij}$ represents the variation of the length of the prismatic joint, where $j = 1, .., 3$ is the index of the active joint and $i$ is the index of the frame. The central leg is modeled by a kinematic configuration of type UP located in the center of an inter-vertebra. It is considered a passive joint.

The inverse kinematic equations (IKEs) of an inter-vertebra of the soft continuum manipulator (Figure 9) are obtained by calculating the joint variables $q_i, j=1:3$, corresponding to the pose (position and orientation) of the upper vertebra's center, relative to the lower vertebra frame. In the case of the considered soft arm, the inter-vertebra is considered as a 3-DoF parallel robot, because of movement constraints related to the passive kinematic chain universal-prismatic (UP). In fact, the rotation with respect to the z axis, denoted by Rot ($z, \phi_i$), and the translations relative to the x and y axes denoted by Trans ($x, X_i$) and Trans ($y, Y_i$), respectively, are not considered, because it does not exist a movement on these axes. Only the translation along the z axis is possible which is denoted by Trans ($z, Z_i$). Hence, the IKEs can be formulated as follows:

$$q_{i,j} = f(Z_i, \psi_i, \theta_i) \qquad (3)$$

where the angles $\theta_i$ and $\psi_i$ indicate pitch and roll angles, respectively.

$A_{ij}$ represents the connection point between the extensible driving leg $j = 1, .., 3$ and the vertebra $i$, as shown in Figure 10. For each vertebra, the points $A_{i,1}, A_{i,2}, A_{i,3}$ form an equilateral triangle. The frame $R_{i-1}(O_{i-1}, x_{i-1}, y_{i-1}, z_{i-1})$ is attached to the lower vertebra of origin $O_{i-1}$, center of the triangle $A_{i-1,1}, A_{i-1,2}, A_{i-1,3}$ and the frame $R_i(O_i, x_i, y_i, z_i)$ is attached to the upper vertebra of origin $O_i$ located at the center of the equilateral triangle $A_{i,1}, A_{i,2}, A_{i,3}$. Knowing that the entire shape of the soft arm is conical, it is necessary to find the circumcircle radius $r_n$ of the considered vertebra, where $n = 1, .., N$ is the number of vertebrae. Let $r_{max}$ and $r_{min}$, respectively, be the radius of the base and the apex of the backbone, then the radius of each vertebra $r_n$ can be calculated by

$$r_n = N_i(r_{min} - r_{max}) + r_{max} \qquad (4)$$

where $N = 8$ is the number of vertebrae. Therefore, the IKE for the $i^{th}$ vertebra can be expressed as follows (Lakhal et al., 2015):

$$\begin{aligned} q_{i,1}^2 &= Z_i^2 + 2r_i Z_i S \theta_i - 2r_{i-1} r_i C \theta_k + r_{i-1}^2 + r_i^2 \\ q_{i,2}^2 &= Z_i^2 + Z_i r_i (\sqrt{3} C \theta_i S \psi_i - S \theta_i) - r_i r_{i-1} \left(\frac{\sqrt{3}}{2} S \theta_i S \psi_i \right. \\ & \qquad \left. + \frac{3}{2} C \psi_i + \frac{1}{2} C \theta_i \right) + r_{i-1}^2 + r_i^2 \\ q_{i,3}^2 &= Z_i^2 - Z_k r_k (\sqrt{3} C \theta_k S \psi_k + S \theta_k) \\ & \qquad + r_i r_{i-1} \left(\frac{\sqrt{3}}{2} S \theta_i S \psi_i - \frac{3}{2} C \psi_i - \frac{1}{2} C \theta_i \right) + r_{i-1}^2 + r_i^2 \end{aligned} \qquad (5)$$
<figure><img src="image_20.png" /><figcaption>FIGURE 14
Trajectory of the tip of the arm during the reparation process of crack 1. (A) 3D position tracking of crack 1. (B) X-axis error. (C) Y-axis error. (D) Z-axis error.</figcaption></figure>

The notations C and S stand for the cosine and sine functions, respectively.

### 2.3.2 Kinematic tracking control of the soft arm

The inverse kinematics tracking control proceeds as shown in Figure 11. The desired posture of the soft arm is generated by applying a set of the desired pressures to the three bending tubes. The desired positions of the soft arm tip is applied to IKM, and the predicted lengths generated by the IKM are used as input to the length-pressure converter, approximated by a multi-layer perceptron neural network (MLPNN) (Melingui et al., 2017) as shown in Table 1.

The pressures generated by the length-pressure converter are applied to the soft arm by means of the internal proportional-integral-derivative (PID) controllers of the Festo VEMA piezo-electric valves. The position predicted by the Optitrack system is compared with the desired positions shown in Figure 12.

## 3 Results

This section presents the experiment results of the applied predictive maintenance. It focuses on the validation of the developed kinematic models. Different cracks are considered on a tile plate sample to validate the algorithms of detection, localization, and repair.

### 3.1 Real-time implementation

The tracking process is shown in Figure 13. The system is based on the Optitrack motion capture system and consists of a set of 10 infrared cameras, as described in the previous paragraph. After calibration, a precision of 0.3 mm for measuring the 3-D displacements of the UAV and the tip of the arm is obtained. The reflection markers placed at the base of the arm are used to obtain the coordinates of the base relative to the UAV.
<figure><img src="image_21.png" /><figcaption>FIGURE 15
Trajectory of the tip of the arm during the reparation process of crack 2. (A) 3D position tracking of crack 2. (B) X-axis error. (C) Y-axis error. (D) Z-axis error.</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>FIGURE 16
Repair steps for crack 1.</figcaption></figure>
## 3.2 Experimental results

Figure 14A, Figure15A show the trajectories of the tip of the arm (nozzle) during the deposit of putty materials on crack 1 (vertical shape) and crack 2 (horizontal shape), respectively. The desired trajectories represent the center-lines of the reconstructed shapes of the cracks after a crack detection and classification technique. The measured trajectory is issued from the Optitrack system and the estimated trajectory is generated from the inverse kinematic model of the soft arm. It is noticed from these experimental tests that the estimated kinematic model reconstructs globally the desired trajectory, while the measured trajectory based on the Optitrack vision capture is perhaps sensitive to the micro-motion of the UAV in its stationary positioning. Figures 14B, C represent the Euclidean errors along x, y, and z axes for crack 1, whereas, Figures 15B, 15c, and 15d represent the Euclidean errors along x, y, and z axes, respectively, for crack 2. It is noticed that the errors are less than 5 mm. However, the error in z can be relatively higher than the others and can be explained by the gravity effect on the soft arm.

Euclidean errors in Figure 14 and Figure 15 conclude that the proposed kinematics model is able to predict the position of the tip of the nozzle with position errors less than 5 mm, where the nozzle diameter is about 5 mm.

In Figure 16, a sequential representation of the deposit material during the deposit of a putty material is shown. The accuracy of tracking tasks by the system is shown in the video (Lakhal et al., 2021), for online target tracking.

Finally, based on the results shown in Figure 16 and the Euclidean errors in Figure 14, the performance of the microscopic crack detection and localization using the deep learning technique is demonstrated.

which has faster and more accurate stability than the millimeter-scale drone. The stability of a UAV in flight is slightly more complex, especially for millimeter-scale movements. In our case study, the UAV is in a stationary position throughout the repair, which means that the sum of the opposing forces is zero, and therefore, the displacement of the system's center of gravity is small due to the ground effect. In addition, when the soft arm comes into contact with the wall, it unloads to maintain a constant force when depositing the material, but also to avoid large disturbances to the movement of the drone that could involve a crash. Indeed, the combination of the soft arm capabilities and the flying mobile platform offers great cooperation to perform an operational task. The soft arm can reach complex positions of the crack, while the UAV can guide and load the arm with different postures. In view of the results obtained, the proposed concept achieves promising performances in terms of precision and robustness for automatic detection and repairs in predictive construction maintenance. The accuracy is in the millimeter range. The trajectory tracking is well-respected despite the total instability of the drone. However, as we can see from the curves, the intervention time is about 60 s for a crack of about 80 mm in size. In fact, for precise control and optimal material deposition quality, it is preferable to work in quasi-static kinematics. If one wishes to increase the speed of loading and unloading of the pressure of the flexible arm, it is necessary to consider using other types of sensors than piezos. In Figure 16, the results are relatively satisfactory in terms of the quality of the material deposit. Indeed, a more or less identical contact force is obtained along the whole trajectory. Nevertheless, we have determined the gain by assuming that the forces are exerted only on the z axis. However, curved trajectories require larger deformations of the flexible arm, which implies that the contact force is not only on the z-axis but on all three axes.

## 4 Discussion

An artificial intelligence-based method using deep learning techniques has been developed to detect and repair cracks automatically. After applying the automatic detection algorithm to the wall surface, the classification of the cracks is obtained by dividing them according to their sizes into microscopic (less than 5 mm width), mesoscopic (in between 5 and 10 mm width), and macroscopic (more than 10 mm width). In this work, we focused on the microscopic cracks, where the preparation of the surface is not required and only the repair is needed. Then, the localization of the centerline of the crack shape is determined. Using the current position of the UAV, the ultrasonic sensor and the camera embedded in the robot give the measures for planning the trajectory to be followed by the tip of the soft continuum arm. The model-based control of the position of the arm tip is applied along the crack with various disturbances such as not-zero stability of the UAV but also the contact of the soft arm during the deposit of the material. This is because errors are corrected by the soft arm,

## 5 Conclusion

This study deals with an integrated robotic concept design for automatic crack detection and repair. This concept is composed of a stationary UAV and a soft continuum arm. The latter is used to guide the putty material but also to reach complex shapes of microscopic cracks in flat and oblique surfaces of wall surfaces. The concept integrates the tasks of detection and localization of the crack shapes using deep learning techniques. The repair task is summarized by a continuous deposit of the material to fill the cracks. The stationary UAV interacts with the wall surface using wheeled contact, while the soft arm is not directly in contact with the external surface. The methodology of the inverse kinematic control of the soft arm has been deployed. A neural network-based predictive model allows for estimating the relationship between the bending tube's length and the input pressure. The experimental results show the performance of the kinematic control as a function of material deposition accuracy. In future work, the objective is to implement a joint control on the robot position and the material deposition flow
rate, as a function of the width and the depth of the crack, to improve
the quality of the deposit.

Data availability statement

The raw data supporting the conclusion of this article will be
made available by the authors, without undue reservation.

Author contributions

All authors discussed the results and contributed to the final
manuscript. RM conceived the presented idea. MK and OL
carried out the experiment. All authors contributed to the
design of the system and implementation of the research. XY
developed the methodology of crack detection and localization
algorithm. MK and OL developed the kinematics models of the
soft arm. MK and OL established the test bench and studied the
behavior of the soft arm.

Funding

This research work has been conducted with the support of
EU funding through SPEED project (www.smartportsecosystem.

References

Aggelis, D., Kordatos, E., Strantza, M., Soulioti, D., and Matikas, T. (2011). Ndt approach for characterization of subsurface cracks in concrete. *Constr. Build. Mater.* 25, 3089–3097. doi:10.1016/j.conbuildmat.2010.12.045

Albishi, A. M., Boybay, M. S., and Ramahi, O. M. (2012). Complementary splitting resonator for crack detection in metallic surfaces. *IEEE Microw. Wirel. Compon. Lett.* 22, 330–332. doi:10.1109/LMWC.2012.2197384

Anantharaman, R., Velazquez, M., and Lee, Y. (2018). “Utilizing mask r-cnn for detection and segmentation of oral diseases,” in *2018 IEEE international conference on bioinformatics and biomedicine (BIBM)* (IEEE).

Chen, I.-M., Asadi, E., Nie, J., Yan, R.-J., Law, W. C., Kayacan, E., et al. (2016). “Innovations in infrastructure service robots,” in *ROMANSY 21-robot design, dynamics and control* (Springer), 3–16.

Daachi, B., Madani, T., and Benallegue, A. (2012). Adaptive neural controller for redundant robot manipulators and collision avoidance with mobile obstacles. *Neurocomputing* 79, 50–60. doi:10.1016/j.neucom.2011.10.001

De Paz, J. P. Z., Castañeda, E. C., Castro, X. Y. S., and Jiménez, S. M. R. (2013). “Crack detection by a climbing robot using image analysis,” in *CONIELECOMP 2013, 23rd international conference on electronics, communications and computing*, 87–91. doi:10.1109/CONIELECOMP.2013.6525765

Dorafshan, S., and Maguire, M. (2018). Bridge inspection: Human performance, unmanned aerial systems and automation. *J. Civ. Struct. Health Monit.* 8, 443–476. doi:10.1007/s13349-018-0285-4

Dorafshan, S., Maguire, M., Hoffer, N. V., and Coopmans, C. (2017). “Challenges in bridge inspection using small unmanned aerial systems: Results and lessons learned,” in *2017 international conference on unmanned aircraft systems (ICUAS)* (IEEE), 1722–1730.

Dorafshan, S., Thomas, R. J., Coopmans, C., and Maguire, M. (2018a). “Deep learning neural networks for suas-assisted structural inspections: Feasibility and application,” in *2018 international conference on unmanned aircraft systems (ICUAS)* (IEEE), 874–882.

com) from the Interreg 2 Seas Program and Interreg NWE
CIRMAP project (www.nweurope.eu/cirmap).

Conflict of interest

The authors declare that the research was conducted in the
absence of any commercial or financial relationships that could
be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the
authors and do not necessarily represent those of their affiliated
organizations, or those of the publisher, the editors, and the
reviewers. Any product that may be evaluated in this article, or
claim that may be made by its manufacturer, is not guaranteed or
endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found
online at: https://www.frontiersin.org/articles/10.3389/frobt.2022.
980800/full#supplementary-material

Dorafshan, S., Thomas, R. J., and Maguire, M. (2018b). Comparison of deep convolutional neural networks and edge detectors for image-based crack detection in concrete. *Constr. Build. Mater.* 186, 1031–1045. doi:10.1016/j.conbuildmat.2018.08.011

Dorafshan, S., Thomas, R. J., and Maguire, M. (2018c). Sdnet2018: An annotated image dataset for non-contact concrete crack detection using deep convolutional neural networks. *Data brief* 21, 1664–1668. doi:10.1016/j.dib.2018.11.015

Escande, C., Chettibi, T., Merzouki, R., Coelen, V., and Pathak, P. M. (2015). Kinematic calibration of a multisection bionic manipulator. *Ieee. ASME. Trans. Mechatron.* 20, 663–674. doi:10.1109/TMECH.2014.2313741

Feng, W., Hu, Z., Wu, W., Yan, J., and Ouyang, W. (2019). Multi-object tracking with multiple cues and switcher-aware classification. arXiv preprint arXiv: 1901.06129.

Flah, M., Suleiman, A. R., and Nehdi, M. L. (2020). Classification and quantification of cracks in concrete structures using deep learning image-based techniques. *Cem. Concr. Compos.* 114, 103781. doi:10.1016/j.cemconcomp.2020.103781

Hameed, K., Chai, D., and Rassau, A. (2022). Score-based mask edge improvement of mask-rcnn for segmentation of fruit and vegetables. *Expert Syst. Appl.* 190, 116205. doi:10.1016/j.eswa.2021.116205

He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017). “Mask r-cnn,” in *Proceedings of the IEEE international conference on computer vision*, 2961–2969.

Jia, W., Tian, Y., Luo, R., Zhang, Z., Lian, J., and Zheng, Y. (2020). Detection and segmentation of overlapped fruits based on optimized mask r-cnn application in apple harvesting robot. *Comput. Electron. Agric.* 172, 105380. doi:10.1016/j.compag.2020.105380

Jiang, H., Wang, Z., Liu, X., Chen, X., Jin, Y., You, X., et al. (2017). “A two-level approach for solving the inverse kinematics of an extensible soft arm considering viscoelastic behavior,” in *2017 IEEE international conference on robotics and automation (ICRA)*, 6127–6133. doi:10.1109/ICRA.2017.7989727
Karlik, B., and Aydin, S. (2000). An improved approach to the solution of inverse kinematics problems for robot manipulators. *Eng. Appl. Artif. Intell.* 13, 159–164. doi:10.1016/S0952-1976(99)00050-0

Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2017). Imagenet classification with deep convolutional neural networks. *Commun. ACM* 60, 84–90. doi:10.1145/3065386

Kucuksubasi, F., and Sorguc, A. (2018). *Transfer learning-based crack detection by autonomous uavs*. CoRR abs/1807.11785. doi:10.48550/arXiv.1807.11785

Lakhal, O., Kahouadji, M., and Yang, X. (2021). *Detection and crack reparation with uav*. Available at: https://drive.google.com/file/d/1AFDw5UIIs0tjcsSDvQApTC1wLDpGQSre/view?usp=sharing.

Lakhal, O., Melingui, A., and Merzouki, R. (2015). Hybrid approach for modeling and solving of kinematics of a compact bionic handling assistant manipulator. *Ieee. ASME. Trans. Mechatron*. 21, 1326–1335. doi:10.1109/tmech.2015.2490180

Lau, K.-T. (2003). Fibre-optic sensors and smart composites for concrete applications. *Mag. Concr. Res.* 55, 19–34. doi:10.1680/macr.2003.55.1.19

Li, Y., Chen, X., Zhu, Z., Xie, L., Huang, G., Du, D., et al. (2019). “Attention-guided unified network for panoptic segmentation,” in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 7026–7035.

Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., et al. (2014). “Microsoft coco: Common objects in context,” in *European conference on computer vision* (Springer), 740–755.

Maguire, M., Dorafshan, S., and Thomas, R. J. (2018). *Sdnet2018: A concrete crack image dataset for machine learning applications*.

Melingui, A., Ahanda, J. J.-B. M., Lakhal, O., Mbede, J. B., and Merzouki, R. (2017). Adaptive algorithms for performance improvement of a class of continuum manipulators. *IEEE Trans. Syst. Man. Cybern. Syst.* 48, 1531–1541. doi:10.1109/ tsmc.2017.2678605

Melingui, A., Escande, C., Benoudjit, N., Merzouki, R., and Mbede, J. (2014). *19th IFAC World Congress*, 47, 9353–9358. doi:10.3182/20140824-6-ZA-1003. 01758Qualitative approach for forward kinematic modeling of a compact bionic handling assistant trunkIFAC Proc. Vol.

Melingui, A., Lakhal, O., Daachi, B., Mbede, J. B., and Merzouki, R. (2015). Adaptive neural network control of a compact bionic handling arm. *Ieee. ASME. Trans. Mechatron*. 20, 2862–2875. doi:10.1109/TMECH.2015.2396114

Mohan, A., and Poobal, S. (2018). Crack detection using image processing: A critical review and analysis. *Alexandria Eng. J.* 57, 787–798. doi:10.1016/j.aej.2017.01.020

Oliveira, H., and Correia, P. L. (2013). Automatic road crack detection and characterization. *IEEE Trans. Intell. Transp. Syst.* 14, 155–168. doi:10.1109/ITITS.2012.2208630

Özgenel, Ç. F., and Gönenç Sorguç, A. “Performance Comparison of Pretrained Convolutional Neural Networks on Crack Detection in Buildings,” ISARC 2018 Berlin.

Palermo, F., Konstantinova, J., Althoefer, K., Poslad, S., and Farkhatdinov, I. (2020). “Implementing tactile and proximity sensing for crack detection,” in

2020 IEEE international conference on robotics and automation (ICRA) (IEEE), 632–637.

Park, S. E., Eem, S.-H., and Jeon, H. (2020). Concrete crack detection and quantification using deep learning and structured light. *Constr. Build. Mater.* 252, 119096. doi:10.1016/j.conbuildmat.2020.119096

Perez, H., Tah, J. H., and Mosavi, A. (2019). Deep learning for detecting building defects using convolutional neural networks. *Sensors* 19, 3556. doi:10.3390/s19163556

Phung, M. D., Hoang, V. T., Dinh, T. H., and Ha, Q. (2017). *Automatic crack detection in built infrastructure using unmanned aerial vehicles*. arXiv preprint arXiv:1707.09715.

Prasanna, P., Dana, K. J., Gucunski, N., Basily, B. B., La, H. M., Lim, R. S., et al. (2016). Automated crack detection on concrete bridges. *IEEE Trans. Autom. Sci. Eng.* 13, 591–599. doi:10.1109/tase.2014.2354314

Reinhart, R. F., Shareef, Z., and Steil, J. J. (2017). Hybrid analytical and data-driven modeling for feed-forward robot control †. *Sensors* 17, 311. doi:10.3390/s17020311

Shi, Y., Cui, L., Qi, Z., Meng, F., and Chen, Z. (2016). Automatic road crack detection using random structured forests. *IEEE Trans. Intell. Transp. Syst.* 17, 3434–3445. doi:10.1109/tits.2016.2552248

Silva, W. R. L. d., and Lucena, D. S. d. (2018). Concrete cracks detection based on deep learning image classification. *Multidiscip. Digit. Publ. Inst. Proc.* 2, 489.

Tsai, Y. J., Kaul, V., and Yezzi, A. (2013). Automating the crack map detection process for machine operated crack sealer. *Automation Constr.* 31, 10–18. doi:10.1016/j.autcon.2012.11.033

Yan, R.-J., Kayacan, E., Chen, I.-M., and Tiong, L. K. (2017). “A novel building post-construction quality assessment robot: Design and prototyping,” in *Intelligent robots and systems (IROS), 2017 IEEE/RSJ international conference on* (IEEE), 6020–6023.

Yao, G., Wei, F., Yang, Y., and Sun, Y. (2019). Deep-learning-based bughole detection for concrete surface image. *Adv. Civ. Eng.*, 1–12. doi:10.1155/2019/8582963

Zhang, W., Zhang, Z., Qi, D., and Liu, Y. (2014). Automatic crack detection and classification method for subway tunnel safety monitoring. *Sensors* 14, 19307–19328. doi:10.3390/s141019307

Zhao, Z.-Q., Zheng, P., Xu, S.-t., and Wu, X. (2019). Object detection with deep learning: A review. *IEEE Trans. Neural Netw. Learn. Syst.* 30, 3212–3232. doi:10.1109/tnnls.2018.2876865

Zhu, G., Fan, Z., Chen, W., You, Y., Huang, S., Liang, W., et al. (2019). “Design and implementation of a manipulator system for roadway crack sealing,” in *2019 IEEE 9th annual international conference on CYBER technology in automation, control, and intelligent systems (CYBER)*, 1327–1331. doi:10.1109/CYBER46603.2019.9066587