# Semantic Recognition and Location of Cracks by Fusing Cracks Segmentation and Deep Learning

Authors: Qing An; Xijiang Chen; Xiaoyan Du; Jiewen Yang; Shusen Wu; Ya Ban
Journal: Complexity
 (2021)

---

## Abstract

For a long time, cracks can appear on the surface of concrete, resulting in a number of safety problems. Traditional manual detection methods not only cost money and time but also cannot guarantee high accuracy. Therefore, a recognition method based on the combination of convolutional neural network and cluster segmentation is proposed. The proposed method realizes the accurate identification of concrete surface crack image under complex background and improves the efficiency of concrete surface crack identification. The research results show that the proposed method not only classifies crack and noncrack efficiently but also identifies cracks in complex backgrounds. The proposed method has high accuracy in crack recognition, which is at least 97.3% and even up to 98.6%.


---

## Body

## 1. Introduction

At present, the use of Chinese concrete structure is fairly wide, involving many engineering industries. But with various cracks generated in the process of construction and using, potential safety hazard has been brought to engineering construction and maintenance. Now, the most common detection method is still manual detection, which is not only inefficient, but also brings safety problems. And the more advanced nondestructive detection methods are costly and cannot realize absolute noncontact detection. In recent years, with the continuous development of computer vision technology, the realization of high-precision crack detection based on computer technology has become a hot spot. The effects of detection methods are mainly focused on the sample-data cluster formation protocol to tackle the leaderless cluster formation problems [1]. They are mainly divided into traditional image processing methods (including edge detection [2], threshold segmentation [3], and region segmentation [4]) and machine learning methods (including deep learning [5] and clustering segmentation [6]).For the traditional image processing methods, Xu et al. [7] used phase angle and gray distribution to improve the Canny operator, which can identify fine surface cracks, but generate more noise. Rivera et al. [8] tried to detect the cracks on the concrete surface by morphological methods and crack splicing, which can remove noise well, but the splicing effect of the crack’s overall skeleton is not very good. Su and Yang [9] proposed an algorithm of image segmentation enhancement, named morphological segmentation based on edge detection-II (MSED-II), to concrete crack segmentation. Lu et al. [10] used a new double-threshold algorithm combined with morphological denoising to realize the identification of strain hardening cementitious composites crack images, but this method cannot identify cracks with complex backgrounds. Based on the edge detection algorithm, Nikravesh and Nezamivand Chegini [11] proposed a detection method using wavelet transform, which can identify crack images with insignificant grayscale differences better and can identify cracks under a certain complex background, but the effect is not significant enough.Compared with traditional image processing methods, Rao et al. [12] proposed a faster and simpler single-level convolutional neural network based on real-time target detection technology, which can not only reduce the impact of background on crack identification to a certain extent, but also identify various surface cracks of concrete bridges, but the accuracy is only 66%. Based on deep fully convolutional neural network, Ren et al. [13] proposed a multiscale crack feature extraction network structure for concrete tunnels named CrackSegNet, which has a strong ability to split the entire crack, but it still has trouble in crack extraction with complex background. Dung and Anh [14] introduced “Full Convolutional Neural Network” to realize effective detection of crack targets in a complex background and reduce false marks, but the processing efficiency is still not high enough. Based on the improved and optimized convolutional neural network “GoogLeNet,” Ye et al. [15] have realized the high-precision identification of multiple cracks with complex background, but the identification accuracy of the crack data set is not very high.For the accurate identification of concrete surface cracks, we proposed a method of concrete surface crack identification in this paper. It is based on the combination of convolutional neural network and clustering segmentation algorithm and can achieve accurate identification and size calculation of concrete surface crack images.


## 2. Preliminary Identification of Cracks

### 2.1. Crack Identification Network (CIN)

According to the diversity of crack shapes and the uncertainty of crack magnitude, this paper proposes a convolution neural network structure called crack identification network (CIN) based on the convolutional neural network, as shown in Figure1. Set serial number as i, and layer i as Li; convolution operation i as Ci; and pooling operation i as Pi. L1, L3, L4, L6, L8, and L9 are convolutional layers; L2, L5, L7, and L10 are pooling layers; L11 is fully connected layer; L12 is ReLu layer; L13 is dropout layer; and L14 is softmax layer.Figure 1
Schematic of crack identification network (CIN).The basic structure of the model consists of six convolutional layers, four pooling layers, a fully connected layer, an input layer, and an output layer. All convolutional layers and fully connected layers are batch normalized to improve the generalization ability of the model. In order to ensure the images which may be too large or too small not to affect the recognition effect, set the input images size as256×256×3. The three dimensions are length, width, and RGB components (red, green, and blue components). In order to perform feature compression better and simplify network complexity, set pooled window size of L2, L5, L7, and L10 as 2 × 2, 4 × 4, 4 × 4, and 6 × 6 and step size as 2 and apply the maximum pooling rule. In order to obtain deeper image features and ensure that the complexity of the network would not increase, set convolution kernel size of convolutional layers L1, L3, L6, and L8 as 3 × 3 and convolution kernel size of convolutional layers L4 and L9 as 5 × 5; step size of all convolutional layers is 1. After convolution and pooling, through the Flatten operation, a vector containing 512 elements is output to the ReLu layer. Then, the partial neuron connection is randomly cut off by the dropout layer, and the dependence between network model parameters is reduced, the robustness of the model is improved, and overfitting is effectively prevented. Finally, the softmax layer is used to judge whether the data after the above processing is a crack image.According to the constructed crack identification network, the cracks can be preliminarily identified through the following steps. First, we established a large number of crack data sets, and the data set was divided into a training set and a verification set. Next, we used the training set and validation set to train and validate the crack identification network model, and last, we used test set to test the model to predict the effect of the model prediction. The overall identification process is shown in Figure2.Figure 2
Flowchart of identification.


### 2.2. Data Set Making

The crack acquisition mechanism shown in Figure3 was constructed to obtain the crack image to make a data set. We used the CCD industrial camera (Basler acA1300–30 gm) mounted on the drone to collect 1000 original images, most of which were crack pictures with complex backgrounds (such as crack pictures with rough surfaces, moss, bumps, depressions, and stains), and each image was different due to different photographing conditions. The size of the collected original images was 4896 × 3672 pixels. They are too large to input directly, so they are divided into 512 × 512 subimages in steps of 248, and 1000 original images are divided into 46000 subimages. In order to detect cracks and eliminate noise more effectively, some subimages were rotated. The final data set contained 58,000 subimages, 48,000 of which were used for training and 10,000 for testing and verifying the effectiveness of the network.Figure 3
Crack collection mechanism.In order to train out a crack detection model with high robustness and high accuracy, the generated data set contains a wide range of crack image types. For example, the crack images contain cross cracks, thick and thin cracks, and cracks with debris. The noncrack images contain concrete surfaces with shadows, bumps, and textures, as shown in Figure4.Figure 4
Example of crack and noncrack images. (a) Crack images. (b) Noncrack images.
(a)(b)It can be seen from Figure4 that the blue dotted box in the first image of Figure 4(a) marks the intersection of the cross crack, the box in the second image marks the fallen leaves in the crack image, and the last two images are examples of thick and fine cracks. In Figure 4(b), the blue dotted elliptical frame in the first noncracked image marks the shaded part of the concrete surface. The elliptical frame in the second image marks the depression on the concrete surface. The elliptical frame in the third image marks the bumps on the concrete surface, and the oval frame in the last image marks the texture of the concrete surface.In order to verify the recognition accuracy of the recognition model effectively and ensure the simplicity of the detection method, we used the 5-fold cross-validation [16] to verify and train 5 sets of models to obtain the training results of the 5 sets of models. Then, we selected a training result randomly to show the accuracy and loss rate curves of the training set and the verification set, which is shown in Figure 5.Figure 5
Training and validation results. (a) Training results. (b) Validation results.
(a)(b)It can be seen from Figure5(a) that the model convergence speed is very fast. In the case with a small amount of sample data, the loss curve drops fast. The curve tends to be stable at 20 epochs, and it indicates that the sample training is almost completed. It can be seen from Figure 5(b) that the accuracy of the verification set continues to rise, and the loss value continues to decrease. The accuracy of the verification set tends to 100% at 20 epochs; the loss value tends to 0%, and it indicates that training has finished at 20 epochs.Figure6 shows the results of the 5-fold cross-validation. The accuracy rates of the five sets of model training sets are all above 99%, and the average accuracy rate of the validation set is 99.1%. The accuracy of the validation set of the five groups of models has a little change, but the range of change is small, which shows that the trained crack network model has good robustness and stability.Figure 6
Accuracy rate of training and validation for 5 different groups.Some images were selected from the data set randomly to test the effect of crack identification of the crack identification network, as shown in Figure7.Figure 7
Example of identification results. (a) Original image. (b) Crack classification and identification results.
(a)(b)From Figure7(b), it is clearly visible that the crack identification network based on deep learning can classify cracks and noncracks accurately. On the one hand, as shown in Figure 7(b), in which the first and second layers belong to the classified cracks Images, while the third and fourth layers belong to the classified noncrack images. On the other hand, the cracks can be identified and located cracks accurately, as shown in the blue box in Figure 7(b). It can be seen from Figure 7(a) that the resolution of the original image is different. For example, the resolution of the first image is significantly greater than that of the second. However, the first and second images in Figure 7(b) can identify the cracks accurately, indicating that the method is not affected by the resolution. At the same time, there are other miscellanies around the cracks, such as the leaves, as shown in Figure 7(a). The blue boxes in the second and seventh pictures of Figure 7(b) show the cracks around the leaves clearly. It indicates that the method is not affected by the surrounding miscellanies. As shown in the first and seventh pictures of Figure 7(b), the cracks in the first picture of Figure 7(a) are significantly larger than those in the seventh picture of Figure 7(a), but the identification effect is basically the same. The fourth picture in Figure 7(a) shows cross cracks, but the fourth picture in Figure 7(b) gives three cross blue box identification results, indicating that the method can automatically decompose the cross crack identification result into straight crack identification result. The noncrack image in the blue dotted oval frame in Figure 7(a) contains large convex and concave concrete surfaces, and the solid blue line in the oval frame shows the concrete surface with large texture. The identification result shown in Figure 7(b) is not affected by those, indicating that the method is not affected by the surface characteristics of the crack. From the above analysis, it can be seen that the crack identification network we proposed is not affected by the image resolution, the surrounding miscellanies, nor the surface characteristics of the crack.Although the crack identification network accurately positions the determined crack images, the positioning results lack the identification of complete cracks and continuity and cannot eliminate the influence of background miscellanies on the cracks. If we only clip and quantitative calculate according to above results, a large error will occur, so it is necessary to divide the cracks more accurately to achieve quantitative identification of cracks.


## 2.1. Crack Identification Network (CIN)

According to the diversity of crack shapes and the uncertainty of crack magnitude, this paper proposes a convolution neural network structure called crack identification network (CIN) based on the convolutional neural network, as shown in Figure1. Set serial number as i, and layer i as Li; convolution operation i as Ci; and pooling operation i as Pi. L1, L3, L4, L6, L8, and L9 are convolutional layers; L2, L5, L7, and L10 are pooling layers; L11 is fully connected layer; L12 is ReLu layer; L13 is dropout layer; and L14 is softmax layer.Figure 1
Schematic of crack identification network (CIN).The basic structure of the model consists of six convolutional layers, four pooling layers, a fully connected layer, an input layer, and an output layer. All convolutional layers and fully connected layers are batch normalized to improve the generalization ability of the model. In order to ensure the images which may be too large or too small not to affect the recognition effect, set the input images size as256×256×3. The three dimensions are length, width, and RGB components (red, green, and blue components). In order to perform feature compression better and simplify network complexity, set pooled window size of L2, L5, L7, and L10 as 2 × 2, 4 × 4, 4 × 4, and 6 × 6 and step size as 2 and apply the maximum pooling rule. In order to obtain deeper image features and ensure that the complexity of the network would not increase, set convolution kernel size of convolutional layers L1, L3, L6, and L8 as 3 × 3 and convolution kernel size of convolutional layers L4 and L9 as 5 × 5; step size of all convolutional layers is 1. After convolution and pooling, through the Flatten operation, a vector containing 512 elements is output to the ReLu layer. Then, the partial neuron connection is randomly cut off by the dropout layer, and the dependence between network model parameters is reduced, the robustness of the model is improved, and overfitting is effectively prevented. Finally, the softmax layer is used to judge whether the data after the above processing is a crack image.According to the constructed crack identification network, the cracks can be preliminarily identified through the following steps. First, we established a large number of crack data sets, and the data set was divided into a training set and a verification set. Next, we used the training set and validation set to train and validate the crack identification network model, and last, we used test set to test the model to predict the effect of the model prediction. The overall identification process is shown in Figure2.Figure 2
Flowchart of identification.


## 2.2. Data Set Making

The crack acquisition mechanism shown in Figure3 was constructed to obtain the crack image to make a data set. We used the CCD industrial camera (Basler acA1300–30 gm) mounted on the drone to collect 1000 original images, most of which were crack pictures with complex backgrounds (such as crack pictures with rough surfaces, moss, bumps, depressions, and stains), and each image was different due to different photographing conditions. The size of the collected original images was 4896 × 3672 pixels. They are too large to input directly, so they are divided into 512 × 512 subimages in steps of 248, and 1000 original images are divided into 46000 subimages. In order to detect cracks and eliminate noise more effectively, some subimages were rotated. The final data set contained 58,000 subimages, 48,000 of which were used for training and 10,000 for testing and verifying the effectiveness of the network.Figure 3
Crack collection mechanism.In order to train out a crack detection model with high robustness and high accuracy, the generated data set contains a wide range of crack image types. For example, the crack images contain cross cracks, thick and thin cracks, and cracks with debris. The noncrack images contain concrete surfaces with shadows, bumps, and textures, as shown in Figure4.Figure 4
Example of crack and noncrack images. (a) Crack images. (b) Noncrack images.
(a)(b)It can be seen from Figure4 that the blue dotted box in the first image of Figure 4(a) marks the intersection of the cross crack, the box in the second image marks the fallen leaves in the crack image, and the last two images are examples of thick and fine cracks. In Figure 4(b), the blue dotted elliptical frame in the first noncracked image marks the shaded part of the concrete surface. The elliptical frame in the second image marks the depression on the concrete surface. The elliptical frame in the third image marks the bumps on the concrete surface, and the oval frame in the last image marks the texture of the concrete surface.In order to verify the recognition accuracy of the recognition model effectively and ensure the simplicity of the detection method, we used the 5-fold cross-validation [16] to verify and train 5 sets of models to obtain the training results of the 5 sets of models. Then, we selected a training result randomly to show the accuracy and loss rate curves of the training set and the verification set, which is shown in Figure 5.Figure 5
Training and validation results. (a) Training results. (b) Validation results.
(a)(b)It can be seen from Figure5(a) that the model convergence speed is very fast. In the case with a small amount of sample data, the loss curve drops fast. The curve tends to be stable at 20 epochs, and it indicates that the sample training is almost completed. It can be seen from Figure 5(b) that the accuracy of the verification set continues to rise, and the loss value continues to decrease. The accuracy of the verification set tends to 100% at 20 epochs; the loss value tends to 0%, and it indicates that training has finished at 20 epochs.Figure6 shows the results of the 5-fold cross-validation. The accuracy rates of the five sets of model training sets are all above 99%, and the average accuracy rate of the validation set is 99.1%. The accuracy of the validation set of the five groups of models has a little change, but the range of change is small, which shows that the trained crack network model has good robustness and stability.Figure 6
Accuracy rate of training and validation for 5 different groups.Some images were selected from the data set randomly to test the effect of crack identification of the crack identification network, as shown in Figure7.Figure 7
Example of identification results. (a) Original image. (b) Crack classification and identification results.
(a)(b)From Figure7(b), it is clearly visible that the crack identification network based on deep learning can classify cracks and noncracks accurately. On the one hand, as shown in Figure 7(b), in which the first and second layers belong to the classified cracks Images, while the third and fourth layers belong to the classified noncrack images. On the other hand, the cracks can be identified and located cracks accurately, as shown in the blue box in Figure 7(b). It can be seen from Figure 7(a) that the resolution of the original image is different. For example, the resolution of the first image is significantly greater than that of the second. However, the first and second images in Figure 7(b) can identify the cracks accurately, indicating that the method is not affected by the resolution. At the same time, there are other miscellanies around the cracks, such as the leaves, as shown in Figure 7(a). The blue boxes in the second and seventh pictures of Figure 7(b) show the cracks around the leaves clearly. It indicates that the method is not affected by the surrounding miscellanies. As shown in the first and seventh pictures of Figure 7(b), the cracks in the first picture of Figure 7(a) are significantly larger than those in the seventh picture of Figure 7(a), but the identification effect is basically the same. The fourth picture in Figure 7(a) shows cross cracks, but the fourth picture in Figure 7(b) gives three cross blue box identification results, indicating that the method can automatically decompose the cross crack identification result into straight crack identification result. The noncrack image in the blue dotted oval frame in Figure 7(a) contains large convex and concave concrete surfaces, and the solid blue line in the oval frame shows the concrete surface with large texture. The identification result shown in Figure 7(b) is not affected by those, indicating that the method is not affected by the surface characteristics of the crack. From the above analysis, it can be seen that the crack identification network we proposed is not affected by the image resolution, the surrounding miscellanies, nor the surface characteristics of the crack.Although the crack identification network accurately positions the determined crack images, the positioning results lack the identification of complete cracks and continuity and cannot eliminate the influence of background miscellanies on the cracks. If we only clip and quantitative calculate according to above results, a large error will occur, so it is necessary to divide the cracks more accurately to achieve quantitative identification of cracks.


## 3. Accurate Crack Segmentation

After the crack identification in the previous section, the initial crack image was obtained, but the crack in the crack image was not segmented accurately. For this, we achieve accurate segmentation of cracks based on improvedK-means clustering algorithm.


### 3.1. ImprovedK-Means Clustering Segmentation Algorithm

Although theK-means clustering algorithm [17] has fast and efficient segmentation characteristics, it has high requirement for high clustering center selection and is very easy to converge to the local optimal solution, thereby missing the global optimal solution. In view of this, we used dynamic particle swarm optimization (DPSO) to improve the K-means clustering algorithm. The specific steps of improved K-means are as follows:Step 1. Convert the color image to grayscale image to obtain the initial grayscale image of the crack, and then set the initial cluster numberk.Step 2. Determine the clustering centerm according to the K-means algorithm.Step 3. After the clustering center is determined, calculate the fitness valuefi of the particles i [18], which is shown in the following equation:(1)fi=∑i=1n∑j=1kxi−mj2.Here,xi is the i th data point in the data point set X and mj is the j th cluster center.Step 4. Determine the particle inertia coefficientw and learning factor s according to equations (2) and (3) [18]:(2)w=wmin−wmax−wmin×fi−fminfavg−fmin,fi≤fave,wmax,fi>fave.Here,wmax is the maximum inertia coefficient; wmin is the minimum inertia coefficient; fi is the current adaptation value of particles; fave is the current average adaptation value of all particles; and fmin is the minimum adaptation value of all particles.(3)s1=s1,int+s1,fin−s1,inttmax×t,s2=s2,int+s2,fin−s2,inttmax×t.Here,s1 and s2 represent the particle’s self-learning ability and the learning ability to the excellent collective; s1,int and s2,int represent the particle’s initial learning ability; s1,fin and s1,fin represent the particle’s final learning ability; s1,int>s2,int; s1,fin>s2,fin; and tmax represents the maximum time for algorithm operation.Step 5. Use the inertia coefficientw and learning factor s obtained in step 4 to update the particle velocity Vi [18] and position Xi [18], which are shown in the following equation:(4)xijt+1=xijt+vijt+1,j=1,2,3,…,d,vijt+1=wvijt+s1r1pij−xijt+s2r2pgj−xijt.Here,r1 and r2 are two random numbers which are evenly distributed on the interval 0,1; t is time; pij is the position of particle i during the update process; and pgi is the best position that particle i experienced during the update process.Step 6. When the maximum number of iterations is reached or the particle swarm fitness variance a converges toδ2 fixed value, the global optimal solution is output to obtain the particle swarm fitness variance δ2 [18], which is shown in the following equation; otherwise, repeat step two to step four.(5)δ2=1n∑i=1nfi−fave2.Step 7. Take the optimal solution of step 5 as the optimal clustering center, and then obtaink clusters.Step 8. Perform criterion function judgment. The index function of the Davidson Burger Index (DBI) [19] is adopted as the criterion function of the final clustering result, and its calculation formula is shown as follows:(6)DBI=1k∑i=1k−1∑j=i+1kmaxSi+SjDisi,j.Here,Disi,j represents the distance between the i th cluster and the j th cluster; Si represents the sum of the standard error of the Euclidean distance between each data point in the i th cluster and the center mi of the cluster; and k is the total number of clusters in the data set.Step 9. IfDBI converges, the clustering result at this time is output as the final result; if it does not converge, repeat steps 2 to 8 until it converges, and then output the result. The overall flowchart of the improved K-means algorithm is shown in Figure 8.


### 3.2. Algorithm Performance Analysis

We used the evaluation index of precision, recall, andF-measure to evaluate the effectiveness of the algorithm. The specific evaluation index [20] is defined as shown in Figure 9.Figure 8
Overall flowchart of the improvedK-means algorithm.Figure 9
Evaluation index.In Figure9, P stands for precision rate; R stands for recall rate; F stands for F-measure value; A stands for the number of images that are crack images and correctly identified as crack; B stands for the number of images that are background images but identified as crack; C represents the number of images that are crack images but identified as background; and D represents the number of images that are background images and are correctly identified as background.


### 3.3. Comparison with Traditional Algorithms

Based on the crack data set in Section 2, taking a few simple crack images as an example, compare the segmentation effects of the improvedK-means algorithm with the traditional improved Otsu [21], improved Canny [22] and improved median filtering [23]. Taking the presence and absence of rough surfaces and protrusions as examples, the comparison of segmentation effects is performed as shown in Figures 10 and 11, respectively.Figure 10
Comparison of rough surface and smooth surface. (a) Original image; (b) segmentation of improved Otsu algorithm; (c) segmentation of improved Canny algorithm; (d) segmentation of improved median filter; and (e) segmentation of this paper’s algorithm.
(a)(b)(c)(d)(e)Figure 11
Comparison of salient surface and flat surface. (a) Original image; (b) segmentation of improved Otsu algorithm; (c) segmentation of improved Canny algorithm; (d) segmentation of improved median filter; and (e) segmentation of this paper’s algorithm.
(a)(b)(c)(d)(e)It can be seen from Figure10 that, for the first crack image with rough surface, although the improved Otsu algorithm, Canny algorithm, and median filter algorithm can all segment the crack, but there is still some noise, but the improved K-means algorithm has almost no noise, and the contour of the crack is also clearer. For the second crack image, because the surface is smooth, all algorithms have few noise, but the crack skeleton integrity and continuity of the improved K-means algorithm are higher than the traditional algorithms. It can be seen from the above comparison that the precise segmentation of the improved K-means algorithm has good noise resistance and accuracy.As can be seen from Figure11, for the first image with raised cracks, although the improved Otsu algorithm, Canny algorithm, and median filter algorithm can all segment the cracks, the cracked skeletons are all thick, and the bulge is also segmented out. In addition to almost no noise, the improved K-means algorithm can also restore the outline of the thin original crack skeleton more accurately. For the second crack image without bumps, because the surface has no bumps, the traditional algorithm only has a small amount of noise, but the crack continuity is poor. But the improved K-means algorithm does not have noise, while ensuring integrity and continuity of the crack skeleton well. It can be seen from the above comparison that the precise segmentation of the improved K-means algorithm has good noise resistance and accuracy.Then, combine the algorithms with the crack identification network, and compare and analyze the overall identification effect. The result of evaluation indicators is shown in Figure12.Figure 12
Comparison of evaluation indicators of each algorithm.As can be seen from Figure12, based on the data set of this article, the indicators of the improved Otsu algorithm and the improved Canny algorithm are less than 80%, while the improved median filter algorithm only has a precision rate slightly higher than 80%, and the other two indicators are also no more than 80%; and the improved K-means algorithm has a recall rate and F-measure value of more than 90%, and the accuracy rate has reached 97%. It can be seen from the above that the processing effect of the improved K-means algorithm is much higher than the above traditional algorithms.


### 3.4. Comparison with Clustering Algorithms

Based on the crack data set, the improvedK-means segmentation algorithm is used to identify crack pictures on the surface of the concrete with moss, fallen leaves, or water stains, and the identification results are combined with the K-means algorithm [24], means shift algorithm [25], and fuzzy C-means algorithm for comparative analysis, as shown in Figure 13.Figure 13
Comparison with traditional algorithms. (a) Original image; (b) segmentation of improved Otsu algorithm; (c) segmentation of improved canny algorithm; (d) segmentation of improved median filter; and (e) segmentation of this paper’s algorithm.
(a)(b)(c)(d)(e)It can be seen from Figure13 that when identifying the first and second cracks with moss in Figure 13(a), the improved K-means algorithm removes almost all noise and guarantees the integrity of the crack skeleton well. For the third cracked image with fallen leaves in Figure 13(a) (where the fallen leaves are marked with a blue dotted box), the other three algorithms have failed to eliminate the noise from the fallen leaves. The two cracks connected to the second small segment were clearly identified. For the identification of the fourth crack with fallen leaves and shadows in Figure 13(a) (where the fallen leaves are marked with a blue dotted box and the shadow is marked with a blue solid line), the K-means algorithm, mean shift algorithm, and fuzzy C-means algorithm can hardly identify cracks because the cracks are too fine, and neither can remove the noise caused by the fallen leaves and shadows. However, the improved K-means algorithm identified the entire fine cracks well when the fallen leaves and shadow noises are completely removed. For the last picture with water stains in Figure 13(a) (where water stains are marked with solid blue boxes), the other three types of algorithms all have a lot of noise and the upper left part of the crack is missing, and the lower rightmost part of the crack is unable to be identified due to the influence of water stains and the grayscale difference of the concrete surface, but the improved K-means algorithm completes the clear identification of the overall crack without noise, and the small arc in the middle of the crack is also clearly identified. The above analysis shows that the crack identification method is not only not affected by moss, shadow, water stains, and miscellanies, but also not affected by the size of the crack and the surface characteristics. Even for such cracks under complex backgrounds, this paper’s identification method can still identify and segment the crack skeleton with high accuracy, and it also guarantees the great advantage of no noise and the integrity and continuity of the crack skeleton. And the comparison of the evaluation indexes of the algorithms is shown in Figure 14.Figure 14
Comparison of evaluation indicators of each algorithm.It can be seen from Figure14 that based on the data set, the recall and F-measure values of the improved K-means algorithm are both above 90%, and the accuracy rate is 97%, which is still much higher than other three clustering algorithms. It can be inferred that the performance of the improved K-means algorithm is better than the other clustering algorithms.


## 3.1. ImprovedK-Means Clustering Segmentation Algorithm

Although theK-means clustering algorithm [17] has fast and efficient segmentation characteristics, it has high requirement for high clustering center selection and is very easy to converge to the local optimal solution, thereby missing the global optimal solution. In view of this, we used dynamic particle swarm optimization (DPSO) to improve the K-means clustering algorithm. The specific steps of improved K-means are as follows:Step 1. Convert the color image to grayscale image to obtain the initial grayscale image of the crack, and then set the initial cluster numberk.Step 2. Determine the clustering centerm according to the K-means algorithm.Step 3. After the clustering center is determined, calculate the fitness valuefi of the particles i [18], which is shown in the following equation:(1)fi=∑i=1n∑j=1kxi−mj2.Here,xi is the i th data point in the data point set X and mj is the j th cluster center.Step 4. Determine the particle inertia coefficientw and learning factor s according to equations (2) and (3) [18]:(2)w=wmin−wmax−wmin×fi−fminfavg−fmin,fi≤fave,wmax,fi>fave.Here,wmax is the maximum inertia coefficient; wmin is the minimum inertia coefficient; fi is the current adaptation value of particles; fave is the current average adaptation value of all particles; and fmin is the minimum adaptation value of all particles.(3)s1=s1,int+s1,fin−s1,inttmax×t,s2=s2,int+s2,fin−s2,inttmax×t.Here,s1 and s2 represent the particle’s self-learning ability and the learning ability to the excellent collective; s1,int and s2,int represent the particle’s initial learning ability; s1,fin and s1,fin represent the particle’s final learning ability; s1,int>s2,int; s1,fin>s2,fin; and tmax represents the maximum time for algorithm operation.Step 5. Use the inertia coefficientw and learning factor s obtained in step 4 to update the particle velocity Vi [18] and position Xi [18], which are shown in the following equation:(4)xijt+1=xijt+vijt+1,j=1,2,3,…,d,vijt+1=wvijt+s1r1pij−xijt+s2r2pgj−xijt.Here,r1 and r2 are two random numbers which are evenly distributed on the interval 0,1; t is time; pij is the position of particle i during the update process; and pgi is the best position that particle i experienced during the update process.Step 6. When the maximum number of iterations is reached or the particle swarm fitness variance a converges toδ2 fixed value, the global optimal solution is output to obtain the particle swarm fitness variance δ2 [18], which is shown in the following equation; otherwise, repeat step two to step four.(5)δ2=1n∑i=1nfi−fave2.Step 7. Take the optimal solution of step 5 as the optimal clustering center, and then obtaink clusters.Step 8. Perform criterion function judgment. The index function of the Davidson Burger Index (DBI) [19] is adopted as the criterion function of the final clustering result, and its calculation formula is shown as follows:(6)DBI=1k∑i=1k−1∑j=i+1kmaxSi+SjDisi,j.Here,Disi,j represents the distance between the i th cluster and the j th cluster; Si represents the sum of the standard error of the Euclidean distance between each data point in the i th cluster and the center mi of the cluster; and k is the total number of clusters in the data set.Step 9. IfDBI converges, the clustering result at this time is output as the final result; if it does not converge, repeat steps 2 to 8 until it converges, and then output the result. The overall flowchart of the improved K-means algorithm is shown in Figure 8.


## 3.2. Algorithm Performance Analysis

We used the evaluation index of precision, recall, andF-measure to evaluate the effectiveness of the algorithm. The specific evaluation index [20] is defined as shown in Figure 9.Figure 8
Overall flowchart of the improvedK-means algorithm.Figure 9
Evaluation index.In Figure9, P stands for precision rate; R stands for recall rate; F stands for F-measure value; A stands for the number of images that are crack images and correctly identified as crack; B stands for the number of images that are background images but identified as crack; C represents the number of images that are crack images but identified as background; and D represents the number of images that are background images and are correctly identified as background.


## 3.3. Comparison with Traditional Algorithms

Based on the crack data set in Section 2, taking a few simple crack images as an example, compare the segmentation effects of the improvedK-means algorithm with the traditional improved Otsu [21], improved Canny [22] and improved median filtering [23]. Taking the presence and absence of rough surfaces and protrusions as examples, the comparison of segmentation effects is performed as shown in Figures 10 and 11, respectively.Figure 10
Comparison of rough surface and smooth surface. (a) Original image; (b) segmentation of improved Otsu algorithm; (c) segmentation of improved Canny algorithm; (d) segmentation of improved median filter; and (e) segmentation of this paper’s algorithm.
(a)(b)(c)(d)(e)Figure 11
Comparison of salient surface and flat surface. (a) Original image; (b) segmentation of improved Otsu algorithm; (c) segmentation of improved Canny algorithm; (d) segmentation of improved median filter; and (e) segmentation of this paper’s algorithm.
(a)(b)(c)(d)(e)It can be seen from Figure10 that, for the first crack image with rough surface, although the improved Otsu algorithm, Canny algorithm, and median filter algorithm can all segment the crack, but there is still some noise, but the improved K-means algorithm has almost no noise, and the contour of the crack is also clearer. For the second crack image, because the surface is smooth, all algorithms have few noise, but the crack skeleton integrity and continuity of the improved K-means algorithm are higher than the traditional algorithms. It can be seen from the above comparison that the precise segmentation of the improved K-means algorithm has good noise resistance and accuracy.As can be seen from Figure11, for the first image with raised cracks, although the improved Otsu algorithm, Canny algorithm, and median filter algorithm can all segment the cracks, the cracked skeletons are all thick, and the bulge is also segmented out. In addition to almost no noise, the improved K-means algorithm can also restore the outline of the thin original crack skeleton more accurately. For the second crack image without bumps, because the surface has no bumps, the traditional algorithm only has a small amount of noise, but the crack continuity is poor. But the improved K-means algorithm does not have noise, while ensuring integrity and continuity of the crack skeleton well. It can be seen from the above comparison that the precise segmentation of the improved K-means algorithm has good noise resistance and accuracy.Then, combine the algorithms with the crack identification network, and compare and analyze the overall identification effect. The result of evaluation indicators is shown in Figure12.Figure 12
Comparison of evaluation indicators of each algorithm.As can be seen from Figure12, based on the data set of this article, the indicators of the improved Otsu algorithm and the improved Canny algorithm are less than 80%, while the improved median filter algorithm only has a precision rate slightly higher than 80%, and the other two indicators are also no more than 80%; and the improved K-means algorithm has a recall rate and F-measure value of more than 90%, and the accuracy rate has reached 97%. It can be seen from the above that the processing effect of the improved K-means algorithm is much higher than the above traditional algorithms.


## 3.4. Comparison with Clustering Algorithms

Based on the crack data set, the improvedK-means segmentation algorithm is used to identify crack pictures on the surface of the concrete with moss, fallen leaves, or water stains, and the identification results are combined with the K-means algorithm [24], means shift algorithm [25], and fuzzy C-means algorithm for comparative analysis, as shown in Figure 13.Figure 13
Comparison with traditional algorithms. (a) Original image; (b) segmentation of improved Otsu algorithm; (c) segmentation of improved canny algorithm; (d) segmentation of improved median filter; and (e) segmentation of this paper’s algorithm.
(a)(b)(c)(d)(e)It can be seen from Figure13 that when identifying the first and second cracks with moss in Figure 13(a), the improved K-means algorithm removes almost all noise and guarantees the integrity of the crack skeleton well. For the third cracked image with fallen leaves in Figure 13(a) (where the fallen leaves are marked with a blue dotted box), the other three algorithms have failed to eliminate the noise from the fallen leaves. The two cracks connected to the second small segment were clearly identified. For the identification of the fourth crack with fallen leaves and shadows in Figure 13(a) (where the fallen leaves are marked with a blue dotted box and the shadow is marked with a blue solid line), the K-means algorithm, mean shift algorithm, and fuzzy C-means algorithm can hardly identify cracks because the cracks are too fine, and neither can remove the noise caused by the fallen leaves and shadows. However, the improved K-means algorithm identified the entire fine cracks well when the fallen leaves and shadow noises are completely removed. For the last picture with water stains in Figure 13(a) (where water stains are marked with solid blue boxes), the other three types of algorithms all have a lot of noise and the upper left part of the crack is missing, and the lower rightmost part of the crack is unable to be identified due to the influence of water stains and the grayscale difference of the concrete surface, but the improved K-means algorithm completes the clear identification of the overall crack without noise, and the small arc in the middle of the crack is also clearly identified. The above analysis shows that the crack identification method is not only not affected by moss, shadow, water stains, and miscellanies, but also not affected by the size of the crack and the surface characteristics. Even for such cracks under complex backgrounds, this paper’s identification method can still identify and segment the crack skeleton with high accuracy, and it also guarantees the great advantage of no noise and the integrity and continuity of the crack skeleton. And the comparison of the evaluation indexes of the algorithms is shown in Figure 14.Figure 14
Comparison of evaluation indicators of each algorithm.It can be seen from Figure14 that based on the data set, the recall and F-measure values of the improved K-means algorithm are both above 90%, and the accuracy rate is 97%, which is still much higher than other three clustering algorithms. It can be inferred that the performance of the improved K-means algorithm is better than the other clustering algorithms.


## 4. Quantitative Identification of Cracks

Use deep learning and improvedK-means algorithm to accurately extract the crack but cannot determine the size of the crack. For this, this section calculates the physical size of the crack accurately according to the premise determination of the crack pixel size.


### 4.1. Crack Physical Size

The physical size of the cracks, namely, the area, length, average width, and occupation ratio, needs to be calculated. The calculation method is as follows:(1)
Crack Area. Calculate the number of pixels included in each connected domain, according to the actual physical size corresponding to each pixel, as shown in the following equation. Thus, the area of each connected domain is obtained, and the areas of all connected domains are summed.(7)Pa=α⋅Pi.Here,Pa is the actual physical size of the crack, the unit is mm2; Pi is the pixel size of the crack, the unit is pixel2; and α is the zoom ratio of the crack, the unit is mm2/pixel2.(2)
Crack Length. Thin the image of the crack skeleton. The crack width after thinning is a single pixel; then calculate half of the perimeter to get the number of pixels of the crack at this time, and get the crack physical length according to formula (7).(3)
Average Crack Width. The average crack width is the ratio of the crack area to the length. The calculation formula of the average crack width can be obtained by combining the calculation method of the crack length, which is shown as follows:(8)di=2SiCi,1≤i≤n.Here,Si is the area of the i th crack; Ci is the perimeter of the i th crack; di is the average width of the i th crack; and n is the total number of cracks.(4)
Crack Occupancy Ratio. It is the ratio of the area of the crack to the total area of the image.


### 4.2. Crack Marking

To facilitate the calculation of the size of the crack, the acquired crack image needs to be marked. Based on the data set, combined with the output of the CIN network model, the crack pictures were tested for marking through morphological method. The software used is MATLAB 2016b, the processor is Intel® Core™ i5-8300H CPU @2.30 GHz, the memory is 8.00 GB, the graphics card is NVIDIA GeForce GTX1050Ti, and the operating system is Windows 10.In order to verify the identification performance better, the identification test was carried out by taking the crack shown in Figure15 as an example. And the crack was marked in sections to facilitate the size calculation; at the same time, the grayscale image was reversed to improve the effect of marking. And, Figure 15(a) is the original image of the crack; the identification result of CIN is shown in Figure 15(b); the segmentation of the identified crack is shown in Figure 15(c); and the marking of the split cracks is shown in Figure 15(d).Figure 15
Crack marking. (a) Original image; (b) identification of CIN; (c) segmentation; (d) marking.
(a)(b)(c)(d)It can be seen from Figure15 that the marking of the crack skeleton is very clear and unambiguous due to the precise identification and segmentation, and almost every large turning point is segmented to facilitate the subsequent size calculation.


### 4.3. Crack Size Calculation

The method of this paper was used to identify and segment cracks and mark them. Then, we chose two images as an example to perform the size calculation, which is shown in Figure16. The crack size calculation method can achieve quantitative identification of cracks. The shooting equipment used in this article was CCD industrial camera (Basler acA 1300–30 gm), and the zoom ratio of the camera in image shooting is α 0.21 mm/pixel.Figure 16
Example of size calculation. (a) Crack 1. (b) Crack 2.
(a)(b)It can be seen from Figure16 that segmented marking is adopted due to the different widths of each segment. Crack 1 in Figure 16(a) contains 8 segments, and crack 2 in Figure 16(b) contains 4 segments. Table 1 shows the pixel size and occupancy ratio of each segment of 1 and crack 2 as well as the overall pixel size. Combining Table 1 and the zoom ratio, the actual sizes of the cracks 1 and 2 were obtained and compared with the measurement results of the crack measuring instrument. The results are shown in Table 2.Table 1
Pixel size of cracks 1 and 2.
Crack numberSegmentation numberWidth (pixel)Length (pixel)Average width (pixel)Overall length (pixel)Area (pixel2)Occupation ration (%)Crack 1141074.63119359940.7624633497451525616366286752588367Crack 2159134.75138462131.47261063424044125Table 2
Actual size comparison results of cracks 1 and 2.
Crack numberAverage width of quantitatively calculation (mm)Overall length of quantitatively calculation (mm)Area of quantitatively calculation (mm2)Average width of crack gauge measurement (mm)Overall length of crack gauge measurement (mm)Area of crack gauge measurement (mm2)Crack 10.97250.53264.331.00251.20261.52Crack 21.00290.64273.990.98288.42270.26It can be seen from Tables1 and 2 that the error between the size obtained by the identification method and the size obtained by the crack measuring instrument is not very large. In order to verify the accuracy more effectively, a part of the image was randomly extracted from the data set. This part of the image was divided into 10 categories, and each category selected a representative image, as shown in Figure 17. The results of these 10 crack images were identified and quantified and compared with the average width, overall length, and area measured by the crack measuring instrument. The statistics and comparison results are shown in Table 3, and the accuracy chart is shown in Figure 18.Figure 17
Original crack image for quantitative calculation.Table 3
Comparison of statistical results.
Group numberAverage width of quantitatively calculation (mm)Average width of crack gauge measurement (mm)Error (mm)Overall length of quantitatively calculation (mm)Overall length of crack gauge measurement (mm)Error (mm)Area of quantitatively calculation (mm2)Area of crack gauge measurement (mm2)Error (mm2)10.810.800.01202.66205.863.20206.35203.462.8920.760.740.02185.13189.304.17169.14172.553.4130.870.860.01221.37224.753.38249.90254.734.8340.800.820.02216.17212.353.82218.45222.193.7450.780.800.02200.65205.024.37206.40202.623.7860.810.800.01210.60207.682.92212.11208.193.9270.760.780.02204.19200.343.85201.96197.934.0380.800.780.02197.57201.223.65195.80199.023.2290.800.820.02207.03210.753.72217.81213.784.03100.910.930.02247.73243.684.05279.35284.485.13Figure 18
Comparison of accuracy of statistical results.It can be seen from Table3 and Figure 16 that for 10 representative crack images, the accuracy of the average width calculated by the quantitative identification method is not less than 97.3% compared with the average width measured by the crack measuring instrument, and the highest accuracy rate is up to 98.84%; the accuracy of the overall length is no less than 97.8%, and the highest accuracy is even more than 98.6%; the accuracy of the area is no less than 97.9%, and the highest accuracy is up to 98.5%. From the above comparative analysis, we can see that the quantitative identification method has good stability, and the accuracy is extremely high. Compared with the instrument measurement, the error obtained is basically within the controllable range.


## 4.1. Crack Physical Size

The physical size of the cracks, namely, the area, length, average width, and occupation ratio, needs to be calculated. The calculation method is as follows:(1)
Crack Area. Calculate the number of pixels included in each connected domain, according to the actual physical size corresponding to each pixel, as shown in the following equation. Thus, the area of each connected domain is obtained, and the areas of all connected domains are summed.(7)Pa=α⋅Pi.Here,Pa is the actual physical size of the crack, the unit is mm2; Pi is the pixel size of the crack, the unit is pixel2; and α is the zoom ratio of the crack, the unit is mm2/pixel2.(2)
Crack Length. Thin the image of the crack skeleton. The crack width after thinning is a single pixel; then calculate half of the perimeter to get the number of pixels of the crack at this time, and get the crack physical length according to formula (7).(3)
Average Crack Width. The average crack width is the ratio of the crack area to the length. The calculation formula of the average crack width can be obtained by combining the calculation method of the crack length, which is shown as follows:(8)di=2SiCi,1≤i≤n.Here,Si is the area of the i th crack; Ci is the perimeter of the i th crack; di is the average width of the i th crack; and n is the total number of cracks.(4)
Crack Occupancy Ratio. It is the ratio of the area of the crack to the total area of the image.


## 4.2. Crack Marking

To facilitate the calculation of the size of the crack, the acquired crack image needs to be marked. Based on the data set, combined with the output of the CIN network model, the crack pictures were tested for marking through morphological method. The software used is MATLAB 2016b, the processor is Intel® Core™ i5-8300H CPU @2.30 GHz, the memory is 8.00 GB, the graphics card is NVIDIA GeForce GTX1050Ti, and the operating system is Windows 10.In order to verify the identification performance better, the identification test was carried out by taking the crack shown in Figure15 as an example. And the crack was marked in sections to facilitate the size calculation; at the same time, the grayscale image was reversed to improve the effect of marking. And, Figure 15(a) is the original image of the crack; the identification result of CIN is shown in Figure 15(b); the segmentation of the identified crack is shown in Figure 15(c); and the marking of the split cracks is shown in Figure 15(d).Figure 15
Crack marking. (a) Original image; (b) identification of CIN; (c) segmentation; (d) marking.
(a)(b)(c)(d)It can be seen from Figure15 that the marking of the crack skeleton is very clear and unambiguous due to the precise identification and segmentation, and almost every large turning point is segmented to facilitate the subsequent size calculation.


## 4.3. Crack Size Calculation

The method of this paper was used to identify and segment cracks and mark them. Then, we chose two images as an example to perform the size calculation, which is shown in Figure16. The crack size calculation method can achieve quantitative identification of cracks. The shooting equipment used in this article was CCD industrial camera (Basler acA 1300–30 gm), and the zoom ratio of the camera in image shooting is α 0.21 mm/pixel.Figure 16
Example of size calculation. (a) Crack 1. (b) Crack 2.
(a)(b)It can be seen from Figure16 that segmented marking is adopted due to the different widths of each segment. Crack 1 in Figure 16(a) contains 8 segments, and crack 2 in Figure 16(b) contains 4 segments. Table 1 shows the pixel size and occupancy ratio of each segment of 1 and crack 2 as well as the overall pixel size. Combining Table 1 and the zoom ratio, the actual sizes of the cracks 1 and 2 were obtained and compared with the measurement results of the crack measuring instrument. The results are shown in Table 2.Table 1
Pixel size of cracks 1 and 2.
Crack numberSegmentation numberWidth (pixel)Length (pixel)Average width (pixel)Overall length (pixel)Area (pixel2)Occupation ration (%)Crack 1141074.63119359940.7624633497451525616366286752588367Crack 2159134.75138462131.47261063424044125Table 2
Actual size comparison results of cracks 1 and 2.
Crack numberAverage width of quantitatively calculation (mm)Overall length of quantitatively calculation (mm)Area of quantitatively calculation (mm2)Average width of crack gauge measurement (mm)Overall length of crack gauge measurement (mm)Area of crack gauge measurement (mm2)Crack 10.97250.53264.331.00251.20261.52Crack 21.00290.64273.990.98288.42270.26It can be seen from Tables1 and 2 that the error between the size obtained by the identification method and the size obtained by the crack measuring instrument is not very large. In order to verify the accuracy more effectively, a part of the image was randomly extracted from the data set. This part of the image was divided into 10 categories, and each category selected a representative image, as shown in Figure 17. The results of these 10 crack images were identified and quantified and compared with the average width, overall length, and area measured by the crack measuring instrument. The statistics and comparison results are shown in Table 3, and the accuracy chart is shown in Figure 18.Figure 17
Original crack image for quantitative calculation.Table 3
Comparison of statistical results.
Group numberAverage width of quantitatively calculation (mm)Average width of crack gauge measurement (mm)Error (mm)Overall length of quantitatively calculation (mm)Overall length of crack gauge measurement (mm)Error (mm)Area of quantitatively calculation (mm2)Area of crack gauge measurement (mm2)Error (mm2)10.810.800.01202.66205.863.20206.35203.462.8920.760.740.02185.13189.304.17169.14172.553.4130.870.860.01221.37224.753.38249.90254.734.8340.800.820.02216.17212.353.82218.45222.193.7450.780.800.02200.65205.024.37206.40202.623.7860.810.800.01210.60207.682.92212.11208.193.9270.760.780.02204.19200.343.85201.96197.934.0380.800.780.02197.57201.223.65195.80199.023.2290.800.820.02207.03210.753.72217.81213.784.03100.910.930.02247.73243.684.05279.35284.485.13Figure 18
Comparison of accuracy of statistical results.It can be seen from Table3 and Figure 16 that for 10 representative crack images, the accuracy of the average width calculated by the quantitative identification method is not less than 97.3% compared with the average width measured by the crack measuring instrument, and the highest accuracy rate is up to 98.84%; the accuracy of the overall length is no less than 97.8%, and the highest accuracy is even more than 98.6%; the accuracy of the area is no less than 97.9%, and the highest accuracy is up to 98.5%. From the above comparative analysis, we can see that the quantitative identification method has good stability, and the accuracy is extremely high. Compared with the instrument measurement, the error obtained is basically within the controllable range.


## 5. Conclusion

This paper proposes a concrete surface crack identification and size calculation method combining deep learning convolutional neural network, clustering segmentation, and morphological methods, which can effectively classify concrete cracks and noncrack images and accurately segment and quantitatively identify cracks. The main conclusions are as follows:(1)
Using deep learning convolutional neural network for training, a preliminary crack recognition model is obtained. After verification and testing, the identification accuracy rate reached more than 99%. At the same time, the crack identification model of deep convolutional neural network is combined with clustering segmentation and morphological methods to propose a hybrid multialgorithm crack identification extraction method called CIN.(2)
Compared with traditional segmentation algorithms and similar clustering segmentation algorithms, the improvedK-means algorithm can identify a variety of concrete surface cracks in complex backgrounds while ensuring extremely high segmentation accuracy.(3)
Based on the improvedK-means algorithm, combined with morphological methods, the identification of cracks, quantitative identification, and calculation of physical size are realized. Experiments show that the accuracy of quantitative identification is at least 97.3%, and the highest can reach 98.6%.(4)
It provides a certain theoretical basis for ensuring the safety of crack detection personnel on the concrete surface, reducing workload, and maintaining and safety testing of concrete structures. It also provides some research bases for the identification of similar cracks under higher precision and more complex conditions.



---

*Source: 3159968-2021-08-09.xml*