# Detection and Segmentation of Cement Concrete Pavement Pothole Based on Image Processing Technology

Authors: Mingxing Gao; Xu Wang; Shoulin Zhu; Peng Guan
Journal: Mathematical Problems in Engineering
 (2020)

---

## Abstract

Potholes are the most common form of distress on cement concrete pavements, which can compromise pavement safety and ridability. Thus, timely and accurate pothole detection is an important task in developing proper maintenance strategies and ensuring driving safety. This paper proposes a method of integrating the processing of grayscale and texture features. This method mainly combines industrial camera to realize rapid and accurate detection of pothole. Image processing techniques including texture filters, image grayscale, morphology, and extraction of the maximum connected domain are used synergistically to extract useful features from digital images. A machine learning model based on the library for support vector machine (LIBSVM) is constructed to distinguish potholes from longitudinal cracks, transverse cracks, and complex cracks. The method is validated using data collected from agricultural and pastoral areas of Inner Mongolia, China. The comprehensive experiments for recognition of potholes show that the recall, precision, and F1-Score achieved are 100%, 97.4%, and 98.7%, respectively. In addition, the overlap rate between the extracted pothole region and the original image is estimated. Images with an overlap rate greater than 90% accounted for 76.8% of the total image, and images with an overlap rate greater than 80% accounted for 94% of the total image. A comparison discloses that the proposed approach is superior to the existing method not only from the perspective of the accuracy of pothole detection but also from the perspective of the segmentation effect and processing efficiency.


---

## Body

## 1. Introduction

Cement concrete pavement, as one of the important parts of a road network, has higher strength, better stability, longer service life, and lower maintenance cost. However, in reality, because cement concrete pavement must bear various loads and natural factors [1], cement concrete pavement defects arise gradually, severely affecting the functionality of pavement. Pothole is the most common form of distress on cement concrete pavement [2, 3] with a minimum dimension of 150 mm [4]. Potholes have significant influences on the running quality of vehicles [5, 6], and they can compromise pavement ridability and safety and can even be the cause of major accidents [6–9]. Therefore, the focus of road maintenance is to handle potholes [10]. Rapidly and accurately detecting potholes in cement pavement is an important prerequisite for the road management department in formulating scientific and effective maintenance strategies and implementing distress treatment [6, 11, 12].Current pothole detection methods can be divided into two groups: manual and automatic. The traditional manual approaches can help detect and evaluate potholes, but it is time-consuming, labor-intensive, and expensive [13, 14]. Moreover, while well-defined criteria for assessment exist, subjectivity and experience of the raters affect the ratings [15]. To overcome these limitations, digital image-based automated pothole evaluation has been gradually replacing the manual evaluation because of its improved efficiency and operational safety [16]. Most existing research in pothole detection uses 3D reconstruction-based methods [17–20], ground penetrating radar (GPR) technology [8], unmanned aerial vehicle- (UAV-) based detection systems [5, 21], video-based detection [22, 23], and laser scanning technology [20, 24–26]. However, these automated techniques often use sophisticated equipment and algorithms that require high costs, cumbersome operation, and maintenance costs; consequently, it is not easy to promote the application.Recently, with the development of image processing technology and machine learning algorithms, research on pavement pothole detection has developed in the direction of simplicity, efficiency, and low cost. The combination of image gray threshold and image texture is an effective method to realize pothole recognition and segmentation [6, 27, 28]. In an additional study by Koch et al. [22], they enhanced the texture properties of defect regions and deployed the scheme over pavement surface videos to realize the automation of asphalt pothole detection. Jog et al. [29] presented an approach that is used to improve recognition results by using visual and spatial characteristics of potholes. The 2D method relies on the texture extraction process to divide the image into segments of pothole and nonpothole areas. The 3D recognition method detects the depth and width of potholes and assesses their severity. Finally, the combination of the outputs allows to correctly identify and assess the potholes in the analyzed video. Based on multiscale texture-based image filtering, Ouma and Hahn [30] used the wavelet transform and fuzzy c-means to detect potholes on asphalt road pavement, and this method is suitable for detecting and extracting incipient potholes. During the detection of pavement distress, the extraction and identification of defects are performed together. Support vector machine (SVM) is a good method for identifying potholes in the case of a small number of samples and complex problems [2, 5, 31]. Lin and Liu [32] employed texture-based features to detect potholes using SVM classifiers. Chacra and Zelek [33] built upon the describable texture dataset methods and then classified features using an SVM classifier into good quality and damaged road. Gavilán et al. [34] proposed a linear SVM classifier based on texture features. The classifier was trained to distinguish up to 10 different types of pavement that appear in Spanish roads.Current research on pothole detection is mainly based on asphalt pavement, while the existing detection methods based on cement concrete pavement defects are limited to the extraction of cracks [35] and faulting [36]. Compared with asphalt pavement, cement pavement is essentially different in texture and color [34] because of the differences between the materials [37, 38]. For image recognition, the pavement detection algorithm changes the accuracy because of the difference in pavement color [39]. The pothole area of cement concrete pavement is very close to the gray level, color, and texture of the pavement background. Consequently, the pothole cannot be completely extracted only by an image gray level. The main aim of this study is to focus on cement concrete pavement, using image grayscale and texture features, combining with the LIBSVM classifier, and relying on simple acquisition equipment to realize fast, accurate, and low-cost detection and segmentation of potholes, providing a means and method for rapid road detection.


## 2. Pothole Detection Method

The pothole detection method in this paper mainly includes two processes. First, the pothole is segmented through texture filtering, image gray level, and morphological operation. Second, potholes are detected by quantifying image features and using a LIBSVM classifier to train and test the feature values.


### 2.1. Pothole Segmentation Principle and Process

Image segmentation is the crucial first step of pavement distress detection [40]. Its accuracy and reliability are critical for subsequent pothole identification. The texture is one of the most important defining characteristics of an image, and it is characterized based on gray tone spatial dependencies [41]. In the road data, the texture inside the potholes is coarser than the nearby pavement [22, 42]. Compared with other features, the texture can consider both the macroscopic properties and the fine structure of the pothole. Therefore, the texture feature can be used to extract the pothole area. Texture information belongs to image information, which is used to represent the roughness or irregular information. Entropy, as a statistical measure of randomness, is a measure of the information content of an image [43]. Therefore, an entropy filter is chosen to calculate the texture information and represents the nonuniformity or complexity of the texture in the pothole. When calculating texture entropy, a gray level co-occurrence matrix (GLCM) can be established [44].Suppose there is a pixel pointx1,y1 with a gray value of i on the image, a pixel point at a distance of d, direction of α, and a gray value of j corresponds to x2,y2; these two points form a gray pair i,j. When the pixels are constantly moving, they correspond to different gray levels. The GLCM is a square matrix formed by the frequency of each pair of gray levels, and its mathematical definition is [45, 46](1)Pi,j,d,α=x1,y1,x2,y2fx1,y1=i,fx2,y2=j,x=0,1,…,Nx−1;y=0,1,…,Ny−1.The GLCM is normalized:(2)pi,j,d,α=Pi,j,d,αR,where i,j=0,1,…,G−1; G is the gray level; Nx and Ny are the rows and columns of the image matrix; and R is the normalized constant.Statistics on GLCM were obtained to obtain texture entropy, and the findings provide the following results [45]:(3)ENT=−∑i∑jpi,jlogpi,j.When the complexity of the image is high, the entropy value is large; otherwise, it is small. In the pavement image, the texture of the pothole is more complicated than the background, and the entropy value changes more violently. The results show that texture entropy can distinguish the pothole and the background. After filtering, the texture image is converted into a grayscale image and a binary image, and then a morphological operation is performed. The following proposed method is implemented in MATLAB. Figure1 shows the pothole extraction process.(1)
Input RGB image.(2)
Texture Filtering. The entropy value is used to detect the boundary information of the pothole, and the texture image is obtained to distinguish the pothole and the background effectively.(3)
Grayscale. The texture image matrix is normalized to further generate a grayscale image.(4)
Binary. Compared with the pavement background, the interior of the pothole is relatively dark. The binary image can be used to describe the pothole profile to achieve initial extraction of the pothole. Here, the threshold method is used to obtain the binary image.(5)
Morphological. “Remove the small object” function removes the impurities in the background of the image. The minimum size of the pothole is 150 mm, which is converted to the image size of 1537.27 pixels. Here, objects smaller than 1500 pixels are removed in this work. The closing operation makes the pothole contour smoother, closes small holes in the image, and joins the broken contour. The filling operation closes the void area in the pothole image and then obtains the closed area.(6)
Image Mask. The pothole area from the preprocessing is overlaid on the original image by a mask operation to clarify the pothole extraction effect.(7)
Actual Area. The pothole in the binary image is white as the pixel value is 1, and the background area is black as the pixel value is 0. The pothole area is the white area range of the binary map. The actual area is obtained by collecting the pothole image in which the physical scale is placed as a reference. The scale value e is obtained according to the pixel value covered by the ruler, and e is the pixel value represented per millimeter. e is computed according to the following equation:(4)e=LN,where e = the scale value, L = the actual size of the scale in the image and the unit is cm, and N = the pixel value occupied by the ruler.Figure 1
Pothole extraction process: (a) original image, (b) texture image, (c) grayscale, (d) binary image, (e) remove small object, (f) morphological, (g) extraction results, and (h) mask image.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)The actual area is defined as follows:(5)S=A0×e2,where S is the actual area of the pothole and the unit is cm2 and A0 is the pixel value of the pothole area.


### 2.2. Feature Selection and Calculation

Pothole recognition needs to be achieved by selecting eigenvalues to quantify the damaged region of the image [47]. Image features reflect the difference between images. The representative features are critical for the rapid and accurate recognition of potholes in the pavement. At present, geometric features are commonly used to measure pavement damage image features [5, 47–50]. In this paper, five representative geometric characteristics are selected.(1)
AreaA0 is used to determine the number of pixels of the white area in the binary image of the pothole. In general, the area of the pothole is the largest.(6)A0=∑Pi,wherePi is the pixel of the pothole.(2)
PerimeterC is used to determine the number of pixels in the boundary of the pothole.(7)C=∑ci,whereci is the pixel at the pothole boundary.(3)
CircularityM is used to evaluate the similarity between an image object and a circle. The amplitude reflects the complexity of the measured boundary.(8)M=4πA0C2.(4)
Rectangle degreePr is used to evaluate the fullness of the pothole to the minimum external rectangle.(9)Pr=A0AMER,whereAMER is the minimum external rectangular area of the target region.(5)
Aspect ratioPwl is used to distinguish between linear and nonlinear distress:(10)Pwl=LMERHMER,where LMERandHMER are the length and width of the minimum external rectangle in the target area.The images collected in this paper mainly include potholes, longitudinal cracks, transverse cracks, and complex cracks (net-shaped crack and block crack). To improve the identification accuracy of potholes, the feature details regarding the connected domain, projection feature, and fractal feature are added according to the extracted distress region and shape feature.(1)
Connected domain feature: the number of connected domains of potholes, transverse cracks, and longitudinal cracks is 1 while the largest connected domain is extracted, and the number of connected domains of complex cracks is more than 1. Therefore, the number of connected domains can be used to distinguish the types of pavement defects.(2)
Projection feature [14]: projection is highly useful for characterizing the type of defects in cement concrete pavement because different types of defects tend to have distinctive projection properties. Project a binary image to the X and Y axes and count the number of pixels in the horizontal and vertical directions. By defining the projection ratio, a longitudinal crack often results in maximizing the projection ratio. This situation is reversed in the case of an image with a transverse crack. Images with potholes and complex cracks have more stable values of projection ratios.(11)T=XmaxYmax,Xmax=maxxi−minxi,Ymax=maxyj−minyj,whereXmax is the maximum difference value of the image on the X-axis, Ymax is the maximum difference value of the image on the Y-axis, and xi and yj are the pixel values of the image projected on the X-axis and Y-axis, respectively.(3)
Fractal feature: fractal features can describe different forms of damage and are used to characterize the complexity of the damaged area [51]. Different types of defect regions have different space occupancies in 2D images. In general, the fractal dimension of the pothole is the largest. The fractal dimension can judge the dimension of the target area, which can be calculated using the box-dimension method.(12)D=−limn⟶0logNnlogn,where n is the size of the box and Nn is the number of boxes covering the pothole.


### 2.3. Support Vector Machine

Different features are extracted from each image to distinguish potholes from other distress types. To this end, a machine learning algorithm is selected, and the extracted features are used as training data to construct a detection model. The new data are assigned as the test data to the corresponding categories in the detection model to achieve the mapping between classes. An SVM is a classification system derived from statistical learning theory [5]. The traditional statistical-based pattern recognition method (such as convolutional neural network) requires a significant amount of training samples to ensure the performance of its classification and recognition [31, 52]. Therefore, preparing training data requires amount of time and effort [52]. SVM can solve practical problems (such as small sample size, complex problem, optimal solution, and high-dimensional space) [49] with fast computing speed, simple structure, and low cost [33]. Therefore, this paper adopts the method based on the SVM classifier. The core idea is to construct an optimal classification surface in the sample space to maximize the distance between the plane and different samples, thereby effectively improving the generalization ability of the algorithm [49].Given a training set of instance-label pairsxi,yi, introduce slack variables ζi for each sample point; C-support vector machine (C-SVM)) training phase boils down to solving the following optimization problems [53]:(13)minw,b,ζ12ω2+C∑i=1Nζisubject  toyiωxi+b≥1−ζi,ζi≥0for anyi=1,…,N,where ω, b, and ζ are normal vector to the hyperplane, offset of the hyperplane, and slack variables, respectively, yi is label of the examples and yi∈1,−1, xi is the feature vector, N is the number of pixels, and C > 0 is a penalty parameter.Using the method of Lagrange multipliers one can obtain the dual formulation [54, 55]:(14)minα12∑i,j=1Nαiαjyiyjxi⋅xj−∑iNαi,(15)subject to∑i=1Nαiyi=0,0≤αi≤C,i=1,2,…,N,where αi is the Lagrange multipliers.After problems (14) and (15) are solved, find the optimal solution αi∗; the optimal ω and b satisfy(16)ω∗=∑i=1Nαi∗yixi,b∗=yi−∑i=1Nαi∗yixi⋅xj.So, the optimal hyperplane and the decision function are, respectively:(17)ω∗⋅x+b∗=0,fx=sgnω∗⋅x+b∗=sgn∑i=1Nαi∗yixi⋅xj+b∗.When there are data points closest to the hyperplane, these are the so-called support vectors. The support vectors are the critical elements of the training set because they contain all the information necessary to reconstruct the hyperplane [55]. An SVM is one of the nonprobabilistic binary classifiers to assign new examples to one category or the other. It means that one SVM can only solve the two-class problems [31] and cannot directly classify multiple types. To ensure recognition accuracy when detecting pavement potholes, it is often necessary to distinguish other types of distress, so this paper faces the problem of multiclassification. To meet such classification requirements, LIBSVM [56] is chosen as the SVM tool. LIBSVM is a library for support vector machines that can solve multiclassification problems. Its goal is to promote SVM as a convenient tool, and its operation is simple and fast. It integrates C-support vector classification (C-SVC), ν-support vector classification (ν-SVC), distribution estimation (one-class SVM), ε-support vector regression (ε-SVR), and ν-support vector regression (ν-SVR) [56]. Different SVM formulations and kernels are supported by this package [57], and it also provides an automatic model selection tool for C-SVM classification [58]. Therefore, using the LIBSVM can produce good classification results without selection parameters, which is hard to decide when using SVM [58]. Evaluation of the prediction effect of LIBSVM can be calculated by the following measures [56]:(18)accuracy=correctly predicted datatotal testing data×100%.


## 2.1. Pothole Segmentation Principle and Process

Image segmentation is the crucial first step of pavement distress detection [40]. Its accuracy and reliability are critical for subsequent pothole identification. The texture is one of the most important defining characteristics of an image, and it is characterized based on gray tone spatial dependencies [41]. In the road data, the texture inside the potholes is coarser than the nearby pavement [22, 42]. Compared with other features, the texture can consider both the macroscopic properties and the fine structure of the pothole. Therefore, the texture feature can be used to extract the pothole area. Texture information belongs to image information, which is used to represent the roughness or irregular information. Entropy, as a statistical measure of randomness, is a measure of the information content of an image [43]. Therefore, an entropy filter is chosen to calculate the texture information and represents the nonuniformity or complexity of the texture in the pothole. When calculating texture entropy, a gray level co-occurrence matrix (GLCM) can be established [44].Suppose there is a pixel pointx1,y1 with a gray value of i on the image, a pixel point at a distance of d, direction of α, and a gray value of j corresponds to x2,y2; these two points form a gray pair i,j. When the pixels are constantly moving, they correspond to different gray levels. The GLCM is a square matrix formed by the frequency of each pair of gray levels, and its mathematical definition is [45, 46](1)Pi,j,d,α=x1,y1,x2,y2fx1,y1=i,fx2,y2=j,x=0,1,…,Nx−1;y=0,1,…,Ny−1.The GLCM is normalized:(2)pi,j,d,α=Pi,j,d,αR,where i,j=0,1,…,G−1; G is the gray level; Nx and Ny are the rows and columns of the image matrix; and R is the normalized constant.Statistics on GLCM were obtained to obtain texture entropy, and the findings provide the following results [45]:(3)ENT=−∑i∑jpi,jlogpi,j.When the complexity of the image is high, the entropy value is large; otherwise, it is small. In the pavement image, the texture of the pothole is more complicated than the background, and the entropy value changes more violently. The results show that texture entropy can distinguish the pothole and the background. After filtering, the texture image is converted into a grayscale image and a binary image, and then a morphological operation is performed. The following proposed method is implemented in MATLAB. Figure1 shows the pothole extraction process.(1)
Input RGB image.(2)
Texture Filtering. The entropy value is used to detect the boundary information of the pothole, and the texture image is obtained to distinguish the pothole and the background effectively.(3)
Grayscale. The texture image matrix is normalized to further generate a grayscale image.(4)
Binary. Compared with the pavement background, the interior of the pothole is relatively dark. The binary image can be used to describe the pothole profile to achieve initial extraction of the pothole. Here, the threshold method is used to obtain the binary image.(5)
Morphological. “Remove the small object” function removes the impurities in the background of the image. The minimum size of the pothole is 150 mm, which is converted to the image size of 1537.27 pixels. Here, objects smaller than 1500 pixels are removed in this work. The closing operation makes the pothole contour smoother, closes small holes in the image, and joins the broken contour. The filling operation closes the void area in the pothole image and then obtains the closed area.(6)
Image Mask. The pothole area from the preprocessing is overlaid on the original image by a mask operation to clarify the pothole extraction effect.(7)
Actual Area. The pothole in the binary image is white as the pixel value is 1, and the background area is black as the pixel value is 0. The pothole area is the white area range of the binary map. The actual area is obtained by collecting the pothole image in which the physical scale is placed as a reference. The scale value e is obtained according to the pixel value covered by the ruler, and e is the pixel value represented per millimeter. e is computed according to the following equation:(4)e=LN,where e = the scale value, L = the actual size of the scale in the image and the unit is cm, and N = the pixel value occupied by the ruler.Figure 1
Pothole extraction process: (a) original image, (b) texture image, (c) grayscale, (d) binary image, (e) remove small object, (f) morphological, (g) extraction results, and (h) mask image.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)The actual area is defined as follows:(5)S=A0×e2,where S is the actual area of the pothole and the unit is cm2 and A0 is the pixel value of the pothole area.


## 2.2. Feature Selection and Calculation

Pothole recognition needs to be achieved by selecting eigenvalues to quantify the damaged region of the image [47]. Image features reflect the difference between images. The representative features are critical for the rapid and accurate recognition of potholes in the pavement. At present, geometric features are commonly used to measure pavement damage image features [5, 47–50]. In this paper, five representative geometric characteristics are selected.(1)
AreaA0 is used to determine the number of pixels of the white area in the binary image of the pothole. In general, the area of the pothole is the largest.(6)A0=∑Pi,wherePi is the pixel of the pothole.(2)
PerimeterC is used to determine the number of pixels in the boundary of the pothole.(7)C=∑ci,whereci is the pixel at the pothole boundary.(3)
CircularityM is used to evaluate the similarity between an image object and a circle. The amplitude reflects the complexity of the measured boundary.(8)M=4πA0C2.(4)
Rectangle degreePr is used to evaluate the fullness of the pothole to the minimum external rectangle.(9)Pr=A0AMER,whereAMER is the minimum external rectangular area of the target region.(5)
Aspect ratioPwl is used to distinguish between linear and nonlinear distress:(10)Pwl=LMERHMER,where LMERandHMER are the length and width of the minimum external rectangle in the target area.The images collected in this paper mainly include potholes, longitudinal cracks, transverse cracks, and complex cracks (net-shaped crack and block crack). To improve the identification accuracy of potholes, the feature details regarding the connected domain, projection feature, and fractal feature are added according to the extracted distress region and shape feature.(1)
Connected domain feature: the number of connected domains of potholes, transverse cracks, and longitudinal cracks is 1 while the largest connected domain is extracted, and the number of connected domains of complex cracks is more than 1. Therefore, the number of connected domains can be used to distinguish the types of pavement defects.(2)
Projection feature [14]: projection is highly useful for characterizing the type of defects in cement concrete pavement because different types of defects tend to have distinctive projection properties. Project a binary image to the X and Y axes and count the number of pixels in the horizontal and vertical directions. By defining the projection ratio, a longitudinal crack often results in maximizing the projection ratio. This situation is reversed in the case of an image with a transverse crack. Images with potholes and complex cracks have more stable values of projection ratios.(11)T=XmaxYmax,Xmax=maxxi−minxi,Ymax=maxyj−minyj,whereXmax is the maximum difference value of the image on the X-axis, Ymax is the maximum difference value of the image on the Y-axis, and xi and yj are the pixel values of the image projected on the X-axis and Y-axis, respectively.(3)
Fractal feature: fractal features can describe different forms of damage and are used to characterize the complexity of the damaged area [51]. Different types of defect regions have different space occupancies in 2D images. In general, the fractal dimension of the pothole is the largest. The fractal dimension can judge the dimension of the target area, which can be calculated using the box-dimension method.(12)D=−limn⟶0logNnlogn,where n is the size of the box and Nn is the number of boxes covering the pothole.


## 2.3. Support Vector Machine

Different features are extracted from each image to distinguish potholes from other distress types. To this end, a machine learning algorithm is selected, and the extracted features are used as training data to construct a detection model. The new data are assigned as the test data to the corresponding categories in the detection model to achieve the mapping between classes. An SVM is a classification system derived from statistical learning theory [5]. The traditional statistical-based pattern recognition method (such as convolutional neural network) requires a significant amount of training samples to ensure the performance of its classification and recognition [31, 52]. Therefore, preparing training data requires amount of time and effort [52]. SVM can solve practical problems (such as small sample size, complex problem, optimal solution, and high-dimensional space) [49] with fast computing speed, simple structure, and low cost [33]. Therefore, this paper adopts the method based on the SVM classifier. The core idea is to construct an optimal classification surface in the sample space to maximize the distance between the plane and different samples, thereby effectively improving the generalization ability of the algorithm [49].Given a training set of instance-label pairsxi,yi, introduce slack variables ζi for each sample point; C-support vector machine (C-SVM)) training phase boils down to solving the following optimization problems [53]:(13)minw,b,ζ12ω2+C∑i=1Nζisubject  toyiωxi+b≥1−ζi,ζi≥0for anyi=1,…,N,where ω, b, and ζ are normal vector to the hyperplane, offset of the hyperplane, and slack variables, respectively, yi is label of the examples and yi∈1,−1, xi is the feature vector, N is the number of pixels, and C > 0 is a penalty parameter.Using the method of Lagrange multipliers one can obtain the dual formulation [54, 55]:(14)minα12∑i,j=1Nαiαjyiyjxi⋅xj−∑iNαi,(15)subject to∑i=1Nαiyi=0,0≤αi≤C,i=1,2,…,N,where αi is the Lagrange multipliers.After problems (14) and (15) are solved, find the optimal solution αi∗; the optimal ω and b satisfy(16)ω∗=∑i=1Nαi∗yixi,b∗=yi−∑i=1Nαi∗yixi⋅xj.So, the optimal hyperplane and the decision function are, respectively:(17)ω∗⋅x+b∗=0,fx=sgnω∗⋅x+b∗=sgn∑i=1Nαi∗yixi⋅xj+b∗.When there are data points closest to the hyperplane, these are the so-called support vectors. The support vectors are the critical elements of the training set because they contain all the information necessary to reconstruct the hyperplane [55]. An SVM is one of the nonprobabilistic binary classifiers to assign new examples to one category or the other. It means that one SVM can only solve the two-class problems [31] and cannot directly classify multiple types. To ensure recognition accuracy when detecting pavement potholes, it is often necessary to distinguish other types of distress, so this paper faces the problem of multiclassification. To meet such classification requirements, LIBSVM [56] is chosen as the SVM tool. LIBSVM is a library for support vector machines that can solve multiclassification problems. Its goal is to promote SVM as a convenient tool, and its operation is simple and fast. It integrates C-support vector classification (C-SVC), ν-support vector classification (ν-SVC), distribution estimation (one-class SVM), ε-support vector regression (ε-SVR), and ν-support vector regression (ν-SVR) [56]. Different SVM formulations and kernels are supported by this package [57], and it also provides an automatic model selection tool for C-SVM classification [58]. Therefore, using the LIBSVM can produce good classification results without selection parameters, which is hard to decide when using SVM [58]. Evaluation of the prediction effect of LIBSVM can be calculated by the following measures [56]:(18)accuracy=correctly predicted datatotal testing data×100%.


## 3. Results and Discussion

In this section, the detection and segmentation results of the collected images are mainly analyzed and the effectiveness of the proposed method is verified. In this paper, four sections of roads with typical characteristics of cement concrete pavement defects in Hinggan League, Inner Mongolia, are selected to collect images. The four sections are from the Chaxin Line, Chagong Line, Eti Line, and Qixing Line. The test vehicle is a microbus, and the collection equipment is a Work Power WP-UC200 industrial camera. During the acquisition process, the camera is placed vertically on the rear side of the vehicle and kept at the same height. The image acquisition system and the image processing system are integrated into the same equipment to monitor and process images in real time. Figure2 shows the process of collection for road condition.Figure 2
Collection road condition: (a) acquisition equipment, (b) industrial camera, and (c) image processing interface.
(a)
(b)
(c)


### 3.1. Detection of Pavement Potholes

#### 3.1.1. Removal of Nondefect Images

Before pothole recognition, the collected pavement images should be screened to remove nondefect images to improve computing speed. The differences in defect images and nondefect images are based on texture. After the preprocessing stage, the defect region is extracted, while a nondefect image has no connected domain or small connected domain area. Therefore, this paper selects the ratio of the connected domain areaAsum to the image area Atotal as the qualification condition to screen the image. Under the same threshold conditions, 50 nondefect images and 50 defect images were selected for the trial calculation to determine the proportional relationship between the connected domain area and the total image area. The trial results are shown in Figure 3. In the case of ignoring the abnormal point, the maximum value of the nondefective image on the y-axis was selected as the filtering condition. In other words, when Asum<0.03Atotal, the image is considered as nondefect, and the pavement image is roughly screened.Figure 3
Comparison of trial results with defect and nondefect images.


#### 3.1.2. Pothole Detection Results

According to the collected 610 defect images, the sample types were labeled. The damage of the pavement aggregate exposed in the image is determined as a pothole, and there were 250 potholes examined in this section. The crack that is nearly perpendicular to the driving direction is determined as a transverse crack, and there were 150 images examined in this section. A crack parallel to the driving direction is determined as a longitudinal crack, and there were 110 images examined in this section. Cracks composed of two or more small polygons are judged to be complex cracks, and there were 100 images examined in this section.The sample label is defined by four types: class one is pothole, class two is transverse crack, class three is longitudinal crack, and class four is complex crack, respectively. The four types of samples have eight attributes, namely, the number of connected domains, projection ratio, fractal dimension, area, perimeter, circularity, rectangle degree, and aspect ratio. Because the sample size is small, to evaluate the appropriateness of feature selection cases, the dataset is divided into two separated sets: training set (70%) and validating set (30%). Based on recognition results, the recall, precision, accuracy, and F1-Score are calculated to underline the ability of the LIBSVM classifier to recognize the pavement pothole. For this purpose, we manually counted the number of true positives (TP, correctly detected pothole), false positives (FP, wrongly detected pothole), true negatives (TN, correctly detected as nonpothole), and false negatives (FN, wrongly detected as nonpothole). While recall is a measure for detection completeness, precision describes the detection exactness or fidelity. Accuracy is used to describe the average correctness of a recognition process. F1-Score is used to measure the accuracy of the classification model. Based on the TP, TN, FP, and FN, recall, precision, accuracy, and F1-Score can be defined as follows [22, 53]:(19)recall=TPTP+FN,precision=TPTP+FP,accuracy=TP+TNTP+TN+FP+FN,F1−Score=2×precision×recallprecision+recall.The recognition results are shown in Figure4 and Table 1. It can be seen that most of the pothole images and nonpothole images can be detected correctly. However, false detection still exists in distress images. Two samples of complex cracks were misdetected as longitudinal cracks. The image was determined to be a complex crack based on an obvious crack and multiple small cracks, but during preprocessing, the small cracks were treated as background impurities to remove, resulting in segmentation error. A longitudinal crack and a complex crack were mistakenly detected as potholes mainly because of the presence of sand soil inside and around the crack, causing excessive image segmentation. In a word, the recognition of cement concrete pavement potholes can achieve a recall of 100%, precision of 97.4%, and F1-Score of 98.7%. The results validate that most potholes in cement concrete pavement images can be correctly detected by LIBSVM and that the selected feature value is reliable. In a study by Yousaf et al. [2], they computed the scale-invariant feature transform features and trained and tested features with SVMs. The results show that they achieved a recall of 94.1%, precision of 97.0%, accuracy of 95.7%, and F1-Score of 95.5%. Hoang [14] used the Gaussian filter, steerable filter, and integral projection to extract features from the pothole images. A dataset was collected to train and validate the predictive performance of the least squares SVM (LS-SVM). Hoang obtained a pothole detection accuracy of 89%. Experimental results demonstrate that our algorithm achieves better performance in terms of precision, recall, and F1-Score than the existing SVM-based pavement pothole detection algorithms.Figure 4
Recognition results of defects.Table 1
Recognition result of defects.
Category
Potholes
Transverse cracks
Longitudinal cracks
Complex cracks
Total
TP
75
—
—
—
75
FP
—
0
1
1
2
TN
—
45
32
27
104
FN
0
—
—
2
2
Recall (%)
100
Precision (%)
97.4
Accuracy (%)
97.8
F1-Score (%)
98.7


### 3.2. Pothole Segmentation Results

#### 3.2.1. Analysis of Pothole Segmentation

To evaluate the performance of the algorithm, this section selects 250 pothole images detected in Section3.1 to analyze the effect of the detection algorithm. The pothole images are divided into six types to compare the detection results under different conditions. The pothole images are classified as follows: (1) pothole with clear boundary, (2) pothole with clear boundary with pebble, (3) pothole with clear boundary with sandy soil inside, (4) pothole with fuzzy boundary, (5) pothole with fuzzy boundary with pebble, and (6) pothole with fuzzy boundary with sandy soil inside. Figure 5 presents the extraction results of several potholes; (a) is the original image and (b) is the extracted image.Figure 5
The extraction results of several potholes: (a) original image and (b) extraction result.As shown in Figure5, the texture and grayscale features are obvious for the first and the second types of potholes. They show that potholes can be completely separated from the pavement background. Pebbles in potholes have little influence on the algorithm, and the processing result is ideal. However, the third pothole type is due to the existence of sandy soil on the pavement, and the segmentation of the pothole is excessive and the result of segmentation is not ideal.Class4, Class 5, and Class 6 present the effects of pebbles and sandy soil on the extraction algorithm when the boundary of the pothole is blurred. The algorithm can still segment the pothole well when the boundary is blurred. While the pebble is contained in the pothole, the texture information of the pothole is missing, so the segmentation has errors. Similarly, potholes containing sandy soil lead to the absence of segmentation, and the effect of segmentation is not good.Table2 shows the area of different types of potholes. The actual area of the extracted pothole is obtained, and the unit is cm2. To verify the accuracy of the detection algorithm for potholes, the pothole area obtained by actual measured in the case of neglecting minor errors is used as a standard reference. The area extraction rate, which is the ratio of total extracted area Se (cm2) to standard total area Ss (cm2), is calculated to further verify the extraction effect of different types of potholes. In addition, potholes with clear boundaries and fuzzy boundaries have high extraction rates. Pebbles have an effect on the extraction of the pothole, but the influence is small, and the algorithm has a better effect on potholes with severe or slight damage. However, when the pothole is covered by sandy soil, the texture information is missing and the processing accuracy is decreased, indicating that the algorithm has certain errors in detecting images with insufficient texture features.Table 2
Comparison of pothole area among different types.
Pothole type
Number
Total areaSe (cm2)
Standard total areaSs (cm2)
Area extraction rate (%)
1
70
100696.35
102203.92
98.5
2
40
54570.95
57737.90
95.4
3
30
39918.45
43271.03
92.3
4
39
71771.82
74145.52
96.8
5
38
101340.41
107686.51
94.1
6
33
93415.14
103519.76
90.2The processing results of 250 pothole images are shown in Table3, and the overlap rate between the extracted pothole region and the original ones is estimated. In other words, the ratio of the overlap portion to the original image pothole area is estimated. The image ratio is defined to evaluate the percentage of the number of pothole images in a certain overlap range to the total image. The overlap rate of more than 90% accounts for 76.8% of the pavement pothole images, the overlap rate of more than 85% accounts for 92%, the overlap rate of more than 80% accounts for 94%, and the overlap rate of less than 80% accounts for 6%. By observing the images with the overlap rate of less than 80%, most of these images are of potholes with sandy soil. Hence, the processing algorithm still needs further improvement when analyzing potholes with sandy soil.Table 3
Statistics of segmentation results of 159 potholes.
Overlap rate
Number of potholes
Image ratio (%)
95%∼100%
106
42.4
90%∼95%
86
34.4
85%∼90%
38
15.2
80%∼85%
5
2.0
60%∼80%
15
6.0


#### 3.2.2. Comparison of Pothole Segmentation Methods

To further verify the extraction effect of cement concrete pavement images, the proposed method, Otsu [59], edge detection [60], K-means [61], and watershed algorithm [62] are used to detect cement concrete pavement potholes and asphalt pavement potholes. While Otsu, edge detection, and watershed algorithm are based on grayscale images, K-means is based on RGB images, as shown in Figures 6 and 7. The figures show that the proposed method is better for extracting cement concrete pavement potholes, while the other four algorithms are better for extracting asphalt pavement potholes. The results show that the algorithm based on texture and grayscale features is suitable for detecting cement concrete pavement potholes.Figure 6
Extraction results of cement pavement pothole by different algorithms: (a) original image, (b) proposed method, (c) Otsu, (d) edge detection, (e)K-means, and (f) watershed algorithm.
(a)
(b)
(c)
(d)
(e)
(f)Figure 7
Extraction results of asphalt pavement pothole by different algorithms: (a) original image, (b) proposed method, (c) Otsu, (d) edge detection, (e)K-means, and (f) watershed algorithm.
(a)
(b)
(c)
(d)
(e)
(f)Table4 reports the processing time taken (in seconds) for the proposed approach and the existing method. For the five presented cement concrete pavement pothole images, the mean time of the proposed method is 2.243 s, while Otsu, edge detection, K-means, and watershed algorithms achieved average times of 2.622 s, 3.087 s, 4.750 s, and 3.628 s, respectively. Overall, the proposed method has higher efficiency in pothole segmentation.Table 4
Comparison of segmentation time.
Images
Proposed method (s)
Otsu (s)
Edge detection (s)
K-means (s)
Watershed algorithm (s)
Figure6 (1)
2.607
1.854
2.642
3.993
4.197
Figure6 (2)
2.555
2.582
3.438
4.772
3.977
Figure6 (3)
2.525
3.016
4.071
6.294
3.253
Figure6 (4)
2.624
2.861
2.585
5.658
3.574
Figure6 (5)
0.905
2.795
2.698
3.033
3.138
Average
2.243
2.622
3.087
4.750
3.628


## 3.1. Detection of Pavement Potholes

### 3.1.1. Removal of Nondefect Images

Before pothole recognition, the collected pavement images should be screened to remove nondefect images to improve computing speed. The differences in defect images and nondefect images are based on texture. After the preprocessing stage, the defect region is extracted, while a nondefect image has no connected domain or small connected domain area. Therefore, this paper selects the ratio of the connected domain areaAsum to the image area Atotal as the qualification condition to screen the image. Under the same threshold conditions, 50 nondefect images and 50 defect images were selected for the trial calculation to determine the proportional relationship between the connected domain area and the total image area. The trial results are shown in Figure 3. In the case of ignoring the abnormal point, the maximum value of the nondefective image on the y-axis was selected as the filtering condition. In other words, when Asum<0.03Atotal, the image is considered as nondefect, and the pavement image is roughly screened.Figure 3
Comparison of trial results with defect and nondefect images.


### 3.1.2. Pothole Detection Results

According to the collected 610 defect images, the sample types were labeled. The damage of the pavement aggregate exposed in the image is determined as a pothole, and there were 250 potholes examined in this section. The crack that is nearly perpendicular to the driving direction is determined as a transverse crack, and there were 150 images examined in this section. A crack parallel to the driving direction is determined as a longitudinal crack, and there were 110 images examined in this section. Cracks composed of two or more small polygons are judged to be complex cracks, and there were 100 images examined in this section.The sample label is defined by four types: class one is pothole, class two is transverse crack, class three is longitudinal crack, and class four is complex crack, respectively. The four types of samples have eight attributes, namely, the number of connected domains, projection ratio, fractal dimension, area, perimeter, circularity, rectangle degree, and aspect ratio. Because the sample size is small, to evaluate the appropriateness of feature selection cases, the dataset is divided into two separated sets: training set (70%) and validating set (30%). Based on recognition results, the recall, precision, accuracy, and F1-Score are calculated to underline the ability of the LIBSVM classifier to recognize the pavement pothole. For this purpose, we manually counted the number of true positives (TP, correctly detected pothole), false positives (FP, wrongly detected pothole), true negatives (TN, correctly detected as nonpothole), and false negatives (FN, wrongly detected as nonpothole). While recall is a measure for detection completeness, precision describes the detection exactness or fidelity. Accuracy is used to describe the average correctness of a recognition process. F1-Score is used to measure the accuracy of the classification model. Based on the TP, TN, FP, and FN, recall, precision, accuracy, and F1-Score can be defined as follows [22, 53]:(19)recall=TPTP+FN,precision=TPTP+FP,accuracy=TP+TNTP+TN+FP+FN,F1−Score=2×precision×recallprecision+recall.The recognition results are shown in Figure4 and Table 1. It can be seen that most of the pothole images and nonpothole images can be detected correctly. However, false detection still exists in distress images. Two samples of complex cracks were misdetected as longitudinal cracks. The image was determined to be a complex crack based on an obvious crack and multiple small cracks, but during preprocessing, the small cracks were treated as background impurities to remove, resulting in segmentation error. A longitudinal crack and a complex crack were mistakenly detected as potholes mainly because of the presence of sand soil inside and around the crack, causing excessive image segmentation. In a word, the recognition of cement concrete pavement potholes can achieve a recall of 100%, precision of 97.4%, and F1-Score of 98.7%. The results validate that most potholes in cement concrete pavement images can be correctly detected by LIBSVM and that the selected feature value is reliable. In a study by Yousaf et al. [2], they computed the scale-invariant feature transform features and trained and tested features with SVMs. The results show that they achieved a recall of 94.1%, precision of 97.0%, accuracy of 95.7%, and F1-Score of 95.5%. Hoang [14] used the Gaussian filter, steerable filter, and integral projection to extract features from the pothole images. A dataset was collected to train and validate the predictive performance of the least squares SVM (LS-SVM). Hoang obtained a pothole detection accuracy of 89%. Experimental results demonstrate that our algorithm achieves better performance in terms of precision, recall, and F1-Score than the existing SVM-based pavement pothole detection algorithms.Figure 4
Recognition results of defects.Table 1
Recognition result of defects.
Category
Potholes
Transverse cracks
Longitudinal cracks
Complex cracks
Total
TP
75
—
—
—
75
FP
—
0
1
1
2
TN
—
45
32
27
104
FN
0
—
—
2
2
Recall (%)
100
Precision (%)
97.4
Accuracy (%)
97.8
F1-Score (%)
98.7


## 3.1.1. Removal of Nondefect Images

Before pothole recognition, the collected pavement images should be screened to remove nondefect images to improve computing speed. The differences in defect images and nondefect images are based on texture. After the preprocessing stage, the defect region is extracted, while a nondefect image has no connected domain or small connected domain area. Therefore, this paper selects the ratio of the connected domain areaAsum to the image area Atotal as the qualification condition to screen the image. Under the same threshold conditions, 50 nondefect images and 50 defect images were selected for the trial calculation to determine the proportional relationship between the connected domain area and the total image area. The trial results are shown in Figure 3. In the case of ignoring the abnormal point, the maximum value of the nondefective image on the y-axis was selected as the filtering condition. In other words, when Asum<0.03Atotal, the image is considered as nondefect, and the pavement image is roughly screened.Figure 3
Comparison of trial results with defect and nondefect images.


## 3.1.2. Pothole Detection Results

According to the collected 610 defect images, the sample types were labeled. The damage of the pavement aggregate exposed in the image is determined as a pothole, and there were 250 potholes examined in this section. The crack that is nearly perpendicular to the driving direction is determined as a transverse crack, and there were 150 images examined in this section. A crack parallel to the driving direction is determined as a longitudinal crack, and there were 110 images examined in this section. Cracks composed of two or more small polygons are judged to be complex cracks, and there were 100 images examined in this section.The sample label is defined by four types: class one is pothole, class two is transverse crack, class three is longitudinal crack, and class four is complex crack, respectively. The four types of samples have eight attributes, namely, the number of connected domains, projection ratio, fractal dimension, area, perimeter, circularity, rectangle degree, and aspect ratio. Because the sample size is small, to evaluate the appropriateness of feature selection cases, the dataset is divided into two separated sets: training set (70%) and validating set (30%). Based on recognition results, the recall, precision, accuracy, and F1-Score are calculated to underline the ability of the LIBSVM classifier to recognize the pavement pothole. For this purpose, we manually counted the number of true positives (TP, correctly detected pothole), false positives (FP, wrongly detected pothole), true negatives (TN, correctly detected as nonpothole), and false negatives (FN, wrongly detected as nonpothole). While recall is a measure for detection completeness, precision describes the detection exactness or fidelity. Accuracy is used to describe the average correctness of a recognition process. F1-Score is used to measure the accuracy of the classification model. Based on the TP, TN, FP, and FN, recall, precision, accuracy, and F1-Score can be defined as follows [22, 53]:(19)recall=TPTP+FN,precision=TPTP+FP,accuracy=TP+TNTP+TN+FP+FN,F1−Score=2×precision×recallprecision+recall.The recognition results are shown in Figure4 and Table 1. It can be seen that most of the pothole images and nonpothole images can be detected correctly. However, false detection still exists in distress images. Two samples of complex cracks were misdetected as longitudinal cracks. The image was determined to be a complex crack based on an obvious crack and multiple small cracks, but during preprocessing, the small cracks were treated as background impurities to remove, resulting in segmentation error. A longitudinal crack and a complex crack were mistakenly detected as potholes mainly because of the presence of sand soil inside and around the crack, causing excessive image segmentation. In a word, the recognition of cement concrete pavement potholes can achieve a recall of 100%, precision of 97.4%, and F1-Score of 98.7%. The results validate that most potholes in cement concrete pavement images can be correctly detected by LIBSVM and that the selected feature value is reliable. In a study by Yousaf et al. [2], they computed the scale-invariant feature transform features and trained and tested features with SVMs. The results show that they achieved a recall of 94.1%, precision of 97.0%, accuracy of 95.7%, and F1-Score of 95.5%. Hoang [14] used the Gaussian filter, steerable filter, and integral projection to extract features from the pothole images. A dataset was collected to train and validate the predictive performance of the least squares SVM (LS-SVM). Hoang obtained a pothole detection accuracy of 89%. Experimental results demonstrate that our algorithm achieves better performance in terms of precision, recall, and F1-Score than the existing SVM-based pavement pothole detection algorithms.Figure 4
Recognition results of defects.Table 1
Recognition result of defects.
Category
Potholes
Transverse cracks
Longitudinal cracks
Complex cracks
Total
TP
75
—
—
—
75
FP
—
0
1
1
2
TN
—
45
32
27
104
FN
0
—
—
2
2
Recall (%)
100
Precision (%)
97.4
Accuracy (%)
97.8
F1-Score (%)
98.7


## 3.2. Pothole Segmentation Results

### 3.2.1. Analysis of Pothole Segmentation

To evaluate the performance of the algorithm, this section selects 250 pothole images detected in Section3.1 to analyze the effect of the detection algorithm. The pothole images are divided into six types to compare the detection results under different conditions. The pothole images are classified as follows: (1) pothole with clear boundary, (2) pothole with clear boundary with pebble, (3) pothole with clear boundary with sandy soil inside, (4) pothole with fuzzy boundary, (5) pothole with fuzzy boundary with pebble, and (6) pothole with fuzzy boundary with sandy soil inside. Figure 5 presents the extraction results of several potholes; (a) is the original image and (b) is the extracted image.Figure 5
The extraction results of several potholes: (a) original image and (b) extraction result.As shown in Figure5, the texture and grayscale features are obvious for the first and the second types of potholes. They show that potholes can be completely separated from the pavement background. Pebbles in potholes have little influence on the algorithm, and the processing result is ideal. However, the third pothole type is due to the existence of sandy soil on the pavement, and the segmentation of the pothole is excessive and the result of segmentation is not ideal.Class4, Class 5, and Class 6 present the effects of pebbles and sandy soil on the extraction algorithm when the boundary of the pothole is blurred. The algorithm can still segment the pothole well when the boundary is blurred. While the pebble is contained in the pothole, the texture information of the pothole is missing, so the segmentation has errors. Similarly, potholes containing sandy soil lead to the absence of segmentation, and the effect of segmentation is not good.Table2 shows the area of different types of potholes. The actual area of the extracted pothole is obtained, and the unit is cm2. To verify the accuracy of the detection algorithm for potholes, the pothole area obtained by actual measured in the case of neglecting minor errors is used as a standard reference. The area extraction rate, which is the ratio of total extracted area Se (cm2) to standard total area Ss (cm2), is calculated to further verify the extraction effect of different types of potholes. In addition, potholes with clear boundaries and fuzzy boundaries have high extraction rates. Pebbles have an effect on the extraction of the pothole, but the influence is small, and the algorithm has a better effect on potholes with severe or slight damage. However, when the pothole is covered by sandy soil, the texture information is missing and the processing accuracy is decreased, indicating that the algorithm has certain errors in detecting images with insufficient texture features.Table 2
Comparison of pothole area among different types.
Pothole type
Number
Total areaSe (cm2)
Standard total areaSs (cm2)
Area extraction rate (%)
1
70
100696.35
102203.92
98.5
2
40
54570.95
57737.90
95.4
3
30
39918.45
43271.03
92.3
4
39
71771.82
74145.52
96.8
5
38
101340.41
107686.51
94.1
6
33
93415.14
103519.76
90.2The processing results of 250 pothole images are shown in Table3, and the overlap rate between the extracted pothole region and the original ones is estimated. In other words, the ratio of the overlap portion to the original image pothole area is estimated. The image ratio is defined to evaluate the percentage of the number of pothole images in a certain overlap range to the total image. The overlap rate of more than 90% accounts for 76.8% of the pavement pothole images, the overlap rate of more than 85% accounts for 92%, the overlap rate of more than 80% accounts for 94%, and the overlap rate of less than 80% accounts for 6%. By observing the images with the overlap rate of less than 80%, most of these images are of potholes with sandy soil. Hence, the processing algorithm still needs further improvement when analyzing potholes with sandy soil.Table 3
Statistics of segmentation results of 159 potholes.
Overlap rate
Number of potholes
Image ratio (%)
95%∼100%
106
42.4
90%∼95%
86
34.4
85%∼90%
38
15.2
80%∼85%
5
2.0
60%∼80%
15
6.0


### 3.2.2. Comparison of Pothole Segmentation Methods

To further verify the extraction effect of cement concrete pavement images, the proposed method, Otsu [59], edge detection [60], K-means [61], and watershed algorithm [62] are used to detect cement concrete pavement potholes and asphalt pavement potholes. While Otsu, edge detection, and watershed algorithm are based on grayscale images, K-means is based on RGB images, as shown in Figures 6 and 7. The figures show that the proposed method is better for extracting cement concrete pavement potholes, while the other four algorithms are better for extracting asphalt pavement potholes. The results show that the algorithm based on texture and grayscale features is suitable for detecting cement concrete pavement potholes.Figure 6
Extraction results of cement pavement pothole by different algorithms: (a) original image, (b) proposed method, (c) Otsu, (d) edge detection, (e)K-means, and (f) watershed algorithm.
(a)
(b)
(c)
(d)
(e)
(f)Figure 7
Extraction results of asphalt pavement pothole by different algorithms: (a) original image, (b) proposed method, (c) Otsu, (d) edge detection, (e)K-means, and (f) watershed algorithm.
(a)
(b)
(c)
(d)
(e)
(f)Table4 reports the processing time taken (in seconds) for the proposed approach and the existing method. For the five presented cement concrete pavement pothole images, the mean time of the proposed method is 2.243 s, while Otsu, edge detection, K-means, and watershed algorithms achieved average times of 2.622 s, 3.087 s, 4.750 s, and 3.628 s, respectively. Overall, the proposed method has higher efficiency in pothole segmentation.Table 4
Comparison of segmentation time.
Images
Proposed method (s)
Otsu (s)
Edge detection (s)
K-means (s)
Watershed algorithm (s)
Figure6 (1)
2.607
1.854
2.642
3.993
4.197
Figure6 (2)
2.555
2.582
3.438
4.772
3.977
Figure6 (3)
2.525
3.016
4.071
6.294
3.253
Figure6 (4)
2.624
2.861
2.585
5.658
3.574
Figure6 (5)
0.905
2.795
2.698
3.033
3.138
Average
2.243
2.622
3.087
4.750
3.628


## 3.2.1. Analysis of Pothole Segmentation

To evaluate the performance of the algorithm, this section selects 250 pothole images detected in Section3.1 to analyze the effect of the detection algorithm. The pothole images are divided into six types to compare the detection results under different conditions. The pothole images are classified as follows: (1) pothole with clear boundary, (2) pothole with clear boundary with pebble, (3) pothole with clear boundary with sandy soil inside, (4) pothole with fuzzy boundary, (5) pothole with fuzzy boundary with pebble, and (6) pothole with fuzzy boundary with sandy soil inside. Figure 5 presents the extraction results of several potholes; (a) is the original image and (b) is the extracted image.Figure 5
The extraction results of several potholes: (a) original image and (b) extraction result.As shown in Figure5, the texture and grayscale features are obvious for the first and the second types of potholes. They show that potholes can be completely separated from the pavement background. Pebbles in potholes have little influence on the algorithm, and the processing result is ideal. However, the third pothole type is due to the existence of sandy soil on the pavement, and the segmentation of the pothole is excessive and the result of segmentation is not ideal.Class4, Class 5, and Class 6 present the effects of pebbles and sandy soil on the extraction algorithm when the boundary of the pothole is blurred. The algorithm can still segment the pothole well when the boundary is blurred. While the pebble is contained in the pothole, the texture information of the pothole is missing, so the segmentation has errors. Similarly, potholes containing sandy soil lead to the absence of segmentation, and the effect of segmentation is not good.Table2 shows the area of different types of potholes. The actual area of the extracted pothole is obtained, and the unit is cm2. To verify the accuracy of the detection algorithm for potholes, the pothole area obtained by actual measured in the case of neglecting minor errors is used as a standard reference. The area extraction rate, which is the ratio of total extracted area Se (cm2) to standard total area Ss (cm2), is calculated to further verify the extraction effect of different types of potholes. In addition, potholes with clear boundaries and fuzzy boundaries have high extraction rates. Pebbles have an effect on the extraction of the pothole, but the influence is small, and the algorithm has a better effect on potholes with severe or slight damage. However, when the pothole is covered by sandy soil, the texture information is missing and the processing accuracy is decreased, indicating that the algorithm has certain errors in detecting images with insufficient texture features.Table 2
Comparison of pothole area among different types.
Pothole type
Number
Total areaSe (cm2)
Standard total areaSs (cm2)
Area extraction rate (%)
1
70
100696.35
102203.92
98.5
2
40
54570.95
57737.90
95.4
3
30
39918.45
43271.03
92.3
4
39
71771.82
74145.52
96.8
5
38
101340.41
107686.51
94.1
6
33
93415.14
103519.76
90.2The processing results of 250 pothole images are shown in Table3, and the overlap rate between the extracted pothole region and the original ones is estimated. In other words, the ratio of the overlap portion to the original image pothole area is estimated. The image ratio is defined to evaluate the percentage of the number of pothole images in a certain overlap range to the total image. The overlap rate of more than 90% accounts for 76.8% of the pavement pothole images, the overlap rate of more than 85% accounts for 92%, the overlap rate of more than 80% accounts for 94%, and the overlap rate of less than 80% accounts for 6%. By observing the images with the overlap rate of less than 80%, most of these images are of potholes with sandy soil. Hence, the processing algorithm still needs further improvement when analyzing potholes with sandy soil.Table 3
Statistics of segmentation results of 159 potholes.
Overlap rate
Number of potholes
Image ratio (%)
95%∼100%
106
42.4
90%∼95%
86
34.4
85%∼90%
38
15.2
80%∼85%
5
2.0
60%∼80%
15
6.0


## 3.2.2. Comparison of Pothole Segmentation Methods

To further verify the extraction effect of cement concrete pavement images, the proposed method, Otsu [59], edge detection [60], K-means [61], and watershed algorithm [62] are used to detect cement concrete pavement potholes and asphalt pavement potholes. While Otsu, edge detection, and watershed algorithm are based on grayscale images, K-means is based on RGB images, as shown in Figures 6 and 7. The figures show that the proposed method is better for extracting cement concrete pavement potholes, while the other four algorithms are better for extracting asphalt pavement potholes. The results show that the algorithm based on texture and grayscale features is suitable for detecting cement concrete pavement potholes.Figure 6
Extraction results of cement pavement pothole by different algorithms: (a) original image, (b) proposed method, (c) Otsu, (d) edge detection, (e)K-means, and (f) watershed algorithm.
(a)
(b)
(c)
(d)
(e)
(f)Figure 7
Extraction results of asphalt pavement pothole by different algorithms: (a) original image, (b) proposed method, (c) Otsu, (d) edge detection, (e)K-means, and (f) watershed algorithm.
(a)
(b)
(c)
(d)
(e)
(f)Table4 reports the processing time taken (in seconds) for the proposed approach and the existing method. For the five presented cement concrete pavement pothole images, the mean time of the proposed method is 2.243 s, while Otsu, edge detection, K-means, and watershed algorithms achieved average times of 2.622 s, 3.087 s, 4.750 s, and 3.628 s, respectively. Overall, the proposed method has higher efficiency in pothole segmentation.Table 4
Comparison of segmentation time.
Images
Proposed method (s)
Otsu (s)
Edge detection (s)
K-means (s)
Watershed algorithm (s)
Figure6 (1)
2.607
1.854
2.642
3.993
4.197
Figure6 (2)
2.555
2.582
3.438
4.772
3.977
Figure6 (3)
2.525
3.016
4.071
6.294
3.253
Figure6 (4)
2.624
2.861
2.585
5.658
3.574
Figure6 (5)
0.905
2.795
2.698
3.033
3.138
Average
2.243
2.622
3.087
4.750
3.628


## 4. Conclusion

In this study, a method is proposed for pothole detection and segmentation based on cement concrete pavement that integrates the grayscale and texture features, and the following conclusion can be drawn:(1)
This paper presents a LIBSVM method to distinguish potholes from longitudinal cracks, transverse cracks, and complex cracks. In regard to the experiment results, the proposed method for recognition of potholes reaches an overall F1-Score of 98.7% and accuracy of 97.8% with 97.4% precision and 100% recall, demonstrating much better performance than that of the SVM-based detection algorithms.(2)
A total of 250 pothole images were collected on four roads, and the overlap ratio between the extracted pothole and the original image was calculated. The overlap rate of more than 90% accounts for 76.8% of the pavement pothole images, and the overlap rate of more than 80% accounts for 94%. Regarding the experiment results, the proposed method is effective in segmenting potholes in different pavement images.(3)
Comparing with the Otsu, edge detection,K-means, and watershed methods, the proposed method showed a better segmentation effect and processing efficiency of cement concrete pavement potholes. This was because the proposed method integrates the grayscale and texture features to recognize and detect pavement potholes, rather than using grayscale or texture alone.(4)
Pebbles showed little significant influence on the detection of potholes. However, the sandy soil has a certain influence on the detection of potholes because the texture information is missing for potholes covered by sandy soil. Therefore, further work for improving the performance processing of potholes covered by sandy soil is necessary for the pothole detection system.



---

*Source: 1360832-2020-01-28.xml*