# Computer Vision-Based Patched and Unpatched Pothole Classification Using Machine Learning Approach Optimized by Forensic-Based Investigation Metaheuristic

Authors: Nhat-Duc Hoang; Thanh-Canh Huynh; Van-Duc Tran
Journal: Complexity
 (2021)

---

## Abstract

During the phase of periodic asphalt pavement survey, patched and unpatched potholes need to be accurately detected. This study proposes and verifies a computer vision-based approach for automatically distinguishing patched and unpatched potholes. Using two-dimensional images, patched and unpatched potholes may have similar shapes. Therefore, this study relies on image texture descriptors to delineate these two objects of interest. The texture descriptors of statistical measurement of color channels, the gray-level cooccurrence matrix, and the local ternary pattern are used to extract texture information from image samples of asphalt pavement roads. To construct a classification model based on the extracted texture-based dataset, this study proposes and validates an integration of the Support Vector Machine Classification (SVC) and the Forensic-Based Investigation (FBI) metaheuristic. The SVC is used to generalize a classification boundary that separates the input data into two class labels of patched and unpatched potholes. To optimize the SVC performance, the FBI algorithm is utilized to fine-tune the SVC hyperparameters. To establish the hybrid FBI-SVC framework, an image dataset consisting of 600 samples has been collected. The experiment supported by the Wilcoxon signed-rank test demonstrates that the proposed computer vision is highly suitable for the task of interest with a classification accuracy rate = 94.833%.


---

## Body

## 1. Introduction

The network of asphalt pavement roads is a crucial element of infrastructure in modern societies [1–4]. As pointed out by [5], asphalt pavement roads significantly support social interaction as well as economic development. In many regions in the world, economic growth is correlated with the extension of asphalt pavement networks. Nevertheless, since these networks are constantly expanded in recent decades, maintaining them becomes a costly and arduous task especially for developing countries like Vietnam. It is because the financial resources of developing countries are often restricted and central governments or provincial authorities are struggling to find a balance between the funding used to construct new road networks and the funding needed to recover deteriorated existing ones.Poorly maintained asphalt pavements lead to a vast number of traffic accidents. According to the record of WHO [6], the number of people lives taken by traffic crashes is roughly 1.35 million per year. Moreover, traffic accidents cost most nations 3% of their gross domestic product. Particularly in Vietnam, there were 14.510 traffic accident cases in 2020 alone; these accidents caused the death of 6700 people and injured 10840 ones [7]. Thus, the proper survey on the road status and timely maintenance is crucial for identifying and restoring pavement defects. These can help to reduce the number of traffic accidents.Among various types of asphalt pavement distress (e.g., cracks, potholes, patches, raveling, bleeding, depression, etc.), potholes are easily encountered and have been recognized as a dangerous type of defect. Potholes are typically observed as bowl-shaped holes in the pavement surface caused by the removal of surfacing materials (refer to Figure1). This type of defect causes a sudden change in road elevation and creates hazardous situations for drivers especially in the cases of inclement weather conditions (e.g., heavy rainfall). Therefore, pavement roads damaged by potholes must be quickly identified and maintained to recover smooth-running surfaces [5].Figure 1
Appearances of potholes in the asphalt pavement surface.Patching is the common technique employed to deal with pavement potholes (refer to Figure2). Pavement patching involves the processes of filling potholes with surfacing materials such as hot mix asphalt or asphalt emulsion mixes. To enhance the efficiency of the pavement recovering task, patching should be performed timely when potholes are early formed. If left unpatched, potholes can be the cause of serious traffic crashes and human casualties especially for motorcycle drivers. Moreover, it is beneficial to prevent water intrusion into the pavement structure as soon as possible. Water intrusion has been seen as the main factor that deepens potholes or triggers other severe failures in the asphalt pavement surface. These facts emphasize the importance of periodic pavement surveys and timely as well as the correct identification of potholes.Figure 2
Appearances of patched potholes in the asphalt pavement surface.In many developing countries, pavement surveys based on human technicians and the manual process of pavements’ visual data processing are still required and performed by local authorities or traffic management agencies. Although these processes can attain high accuracy of pavement evaluation, the manual process is painstakingly low in productivity. Moreover, a large and expanding network of asphalt pavements slows down the pavement survey process and makes timely detection of pavement distresses an impossible mission. Another issue of the manual process is that the survey outcome is considerably affected by subjective judgments of human inspectors. Therefore, scholars and practitioners are increasingly relied on automated approaches to improving the productivity and objectivity of periodic road pavement surveys.In recent years, due to the availability of low-cost digital cameras and rapid advancement of 2-dimensional (2D) digital image processing techniques, computer vision has gained popularity and has been proven to be a feasible tool for asphalt pavement surveys. Nevertheless, computer vision-based automatic pothole detection still presents a challenging task due to the complex nature of the asphalt pavement background. Pavement distresses often coexist with noisy signals caused by stains, irregular lighting conditions, traffic marks, etc.Accordingly, various computer vision-based methods have been proposed in the literature to deal with the problem of pavement pothole recognition. Zhou et al. [8] propose a replacement of a low-speed human-based approach using an integrated image processing system; this system mainly relies on discrete wavelet transform for asphalt pavement distress classification. A comparative study performed in [9] assesses several multiresolution texture analysis methods based on a wavelet, ridgelet, and curvelet-based texture extractors used for pothole detection. Koch and Brilakis [10] rely on histogram shape-based thresholding coupled with morphological operators to isolate pothole shape; the authors employ 70 image samples for testing and obtain a classification accuracy of 85%.A pothole recognition and evaluation scheme based on 2D shape analysis, image coarseness comparison, and image thresholding techniques has been proposed in [11]. Buza et al. [12] put forward an unsupervised learning method based on image processing techniques and spectral clustering. Sundra Murthy and Varaprasad [13] employ histogram analysis, edge detection, and contour following to deal with the task of interest. Ryu et al. [14] attempt to improve an existing intelligent transportation system service by the utilization of a 2D image-based pothole detection with image processing techniques of segmentation, region extraction, and morphological filtering. A novel approach based on semantic texton forests used with 2D video frames has been proposed in [15]. Kamaliardakani et al. [16] employ a heuristic thresholding method for detecting sealed crack damages. Fuzzy c-means clustering algorithm and morphological reconstruction have been utilized by [17] for recognizing potholes on asphalt pavement based on 2D-color images.Encouraged by the successes of image processing techniques applied in pothole detection, more recent works have explored the feasibility of advanced supervised machine learning models in dealing with the task at hand. Yousaf et al. [5] construct a support vector machine (SVM) trained by a set of scale-invariant feature transform (SIFT) features for recognizing potholes in labeled images. A machine learning-based approach using least squares support vector machine and neural network with steerable filter-based feature extraction has been proposed in [18]. Maeda et al. [19] and Cao et al. [20] recently put forward deep neural network-based approaches for recognizing asphalt pavement defects including potholes.Overall, there is an increasing trend of applying image processing and machine learning methods for pothole detection using 2D digital images. This trend in the academic community has been recognized by previous reviewing works of [21–23]. In line with this trend of study and motivated by the fact that asphalt pavement roads in different regions may exhibit different surfacing features due to various factors including the use of surfacing materials, construction methods, traffic loads, and other weather/local conditions, it is necessary to investigate other advanced image processing and machine learning solutions for dealing with the task of automatic pothole detection. The reason is that because of discrepancies in characteristics of asphalt pavement roads, a data-driven method can obtain good detection accuracy for a certain study area but it may not perform well on image data collected from other study regions.More importantly, it is necessary for road surveying systems to be able to distinguish between unpatched potholes and patched potholes. The reason is that if patched objects are correctly identified, the false positive rate of the pothole detection process can be reduced. In addition, as pointed out in previous studies [24–26], patched areas are considered pavement defects and they should be detected with high accuracy. As observed from Figures 1 and 2, it can be argued that using 2D image samples, potholes and patched potholes can have similar shapes. Therefore, image texture analysis used for extracting the coarseness of image regions is helpful to recognize them. Texture descriptors [27–42] have been proved to be highly useful for image classification in various fields. In this study, the highly discriminative local ternary pattern is employed.In addition, based on the current literature, it can be seen that pothole detection methods have mainly relied on individual machine learning methods. Integrations of machine learning and metaheuristic approaches have rarely been investigated for the task at hand. In various fields, the successful utilization of metaheuristics in optimizing machine learning models has been demonstrated [43–46]. Nevertheless, the applications of such hybrid scheme for pothole recognition are still limited.Thus, the current study is an attempt to fill this gap in the literature by proposing a machine learning-metaheuristic integration for coping with the problem of interest. The employed machine learning model is SVM [47] because this machine learning method has been proven to be a highly capable tool for pattern recognition especially for asphalt pavement image data [5, 48, 49]. To further optimize the performance of the SVM model used for the task of the pothole and patched pothole detection, this study relies on a newly proposed metaheuristic of Forensic-Based Investigation (FBI) [50]. FBI is a novel metaheuristic motivated by the suspect investigation-location-pursuit process that is used by police officers and its performance has been confirmed by various optimization tasks. Accordingly, the current work proposes combining the two methods of SVM and FBI to establish an integrated data-driven model utilized for computer vision-based pothole-patched pothole recognition.The subsequent parts of the study are organized as follows. The second section reviews the research methodology including the techniques of image texture analysis, the computational intelligence approaches, and a set of collected image samples. The next section describes the structure of the proposed integration of FBI optimized SVM used for pothole-patched pothole detection. The fourth section reports experimental results. Concluding remarks of this study are stated in the final section.


## 2. Research Methodology

This section of the article reviews the employed research method including image texture used for feature extraction, the machine learning approach of SVM used for pattern recognition, the FBI metaheuristic used for model optimization, and the collected pavement image dataset.


### 2.1. Image Texture Analysis

In the image processing field, the texture is a crucial tool used for visual perception and has been the core of many computer vision systems. Texture analysis is used to represent the degree of coarseness or fineness of objects within digital image samples. This type of analysis has been widely used in various fields of study including civil engineering [51], remote sensing [52], biomedical imaging [53], and industrial engineering [54, 55]. Based on the collected image samples, meaningful features that represent the texture properties of image regions can be computed and used for object classification [34]. In this study, statistical measurement of image pixel intensity, gray-level cooccurrence matrix’s properties, and local ternary pattern are the utilized texture descriptors used for distinguishing between potholes and patched potholes.


#### 2.1.1. Statistical Properties of Color Texture

Given an image sampleI within which there are 3 color channels of red (R), green (G), and blue (B), a first-order histogram P(I) can be calculated for each channel to represent its statistical distribution [56]. Based on the computed first-order histogram of the three color channels (R, G, and B), the indices of mean μc, standard deviation σc, the skewness δc, kurtosis ηc, entropy ρc, and range Δc are computed separately for each channel. Since each channel yields 6 color texture-based indices, the total number of extracted features representing the statistical properties of color texture is 6 × 3 = 18. The equations used for computing the 6 color texture-based indices for each color channel c are presented as follows [51]:(1)μc=∑i=0NL−1Ii,c×PcI,(2)σc=∑i=0NL−1Ii,c−μc2×PcI,(3)δc=∑i=0NL−1Ii,c−μc3×PcIσc3,(4)ηc=∑i=0NL−1Ii,c−μc4×PcIσc4,(5)ρc=−∑i=0NL−1PcI×log2PcI,(6)Δc=MaxIc−MinIc,where NL = 256 denotes the number of discrete intensity values with an 8-bit image sample.


#### 2.1.2. Gray-Level Cooccurrence Matrix (GLCM)

GLCM [57, 58] is a useful image processing tool used for representing properties regarding the repeated occurrence of certain gray-level patterns. Herein, the first is the analysis to convert an RGB image to a grayscale one [59]. Subsequently, the relationship between 2 pixels within this grayscale image sample is modeled using two parameters: r-the distance relationship and θ-the rotation relationship. Accordingly, a matrix denoting Pδ is used to represent a probability of the two gray levels of i and j with respect to the relationship dictated by r and θ [60]. Based on the normalized Pδ denoted as PδN, the four indices of the angular second moment (AM), contrast (CO), correlation (CR), and entropy (ET) can be calculated as follows [58]:(7)AM=∑i=1Ng∑j=1NgPδNi,j2,(8)CO=∑k=0Ng−1k2∑i=1|i−j|=kNg∑j=1NgPδNi,j,(9)CR=∑i=1Ng∑j=1Ngi×j×PδNi,j−μXμYσXσY,(10)ET=−∑i=1Ng∑j=1NgPδNi,jlogPδNi,j,where Ng is the number of gray-level values. μX,μY,σX, and σY are the means and standard deviations of the marginal distribution associated with PδNi,j [51, 57].


#### 2.1.3. Local Ternary Pattern (LTP)

Putting forward in [61], LTP is an extension of the well-defined Local Binary Pattern (LBP). LBP [62] is an effective tool for characterizing local gray intensity patterns. This method considers a local neighborhood around a certain pixel and thresholds the neighboring pixels based on the value of the central pixels. Accordingly, a local texture descriptor consisting of a 3 × 3 binary matrix is used to describe the local gray intensity structure of a certain region within an image sample.Tan and Triggs [61] propose using an additional thresholding value to construct a ternary structure which can be more discriminative and less sensitive to noise in uniform regions. Therefore, the LTP texture descriptor is highly suitable for modeling gray intensity patterns of asphalt pavement roads. Using the LTP method, the value of each thresholded pixel within a local region is either −1, 0, or 1. Therefore, the computation of LTP is split into two separated matrices called a lower pattern and an upper pattern (refer to Figure 3). Similar to the standard LBP, histograms of the two aforementioned matrices can be calculated and concatenated to establish an integrated texture discriminator.Figure 3
Demonstration of the LTP texture descriptor.Mathematically speaking, the LTP texture descriptor can be described as follows:(11)s′u,ic,t=1ifu≥ic+t0ifu−ic<t−1ifu≤ic−t,where ic is the gray intensity of the central pixel. t = 5 denotes an additional threshold. u is the grayscale value of a neighboring pixel.


### 2.2. Support Vector Machine Classification (SVC)

SVC, introduced by [47], is a powerful supervised learning method used for multivariate data analysis especially for pattern classification. The model structure of this machine learning method is trained via the framework of a structural risk minimization scheme [63]. This feature helps an SVC model to be resilient to noisy data samples and to better guard against overfitting. Given a training dataset set D=x,y|x∈Sandy=fx and a target function f: X⟶−1,+1 (−1 = “patched pothole” and +1 = “pothole”), the SVM can be used to learn f(x) by constructing an approximated function f^x:X⟶−1,+1.The overall learning phase of SVC is illustrated in Figure4. The first step of the learning process is to construct a function to map the data in the original input space into a high-dimensional feature space. It is noted that the original input space consists of numerical data generated from the aforementioned texture descriptors. In such high-dimensional feature space, a hyperplane can be learnt to act as a decision boundary to separate the input data into two regions of “pothole” and “patched pothole”. To identify this hyperplane, it is required to solve the following constrained optimization problem:(12)MinimizeJpw,e=12wTw+C12∑k=1Nek2,subjected to ykwTφxk+b≥1−ek, k=1,…,N, ek≥0. w∈Rn and b∈R denote the parameters of the hyperplane. e represents the vector of slack variables. C is the penalty coefficient. φx is the aforementioned nonlinear data mapping function.Figure 4
Demonstration of the SVC learning phase.A notable advantage of the SVC machine learning method is that it does not require the explicit form of the mapping functionφx. The quantity of interest is the dot product of φx denoting a kernel function K(xk, xl). For nonlinear and multivariate data classification, the commonly used kernel function is the radial basis kernel function (RBKF) shown as follows:(13)Kxk,xl=exp−xk−xl22σ2,where σ is a tuning parameter of the kernel function.


### 2.3. Forensic-Based Investigation (FBI)

As described in the previous section, the training phase of an SVC model necessitates a proper setting of the penalty coefficientC and the RBKF’s tuning parameter σ. These are two crucial hyperparameters of SVC. The former expresses the amount of penalty imposing on misclassified data samples. The latter specifies the locality of the employed kernel function which affects the generalization of a trained SVC model. Since the task of hyperparameter setting can be formulated as an optimization problem, this study relies on the FBI metaheuristic to assist the SVC model’s training phase.FBI [50] is a stochastic search engine motivated by the forensic investigation process. The searching process of this stochastic search engine mimics the real operation of forensic investigation which contains five steps: open a case, interpretation of findings, the direction of inquiry, action, and prosecution [64]. The searching process of the FBI can be broken down into two phases: the investigation phase (called phase A) and the pursuit phase (called phase B). The former is carried out by a team of investigators. The latter is performed by a team of police agents. These two phases are repeated until a stopping condition (i.e., a maximum number of iterations) is met.In phase A, based on the trace recorded at the current locationXij, a new investigation location Xij,new is computed as follows:(14)Xij,new=X+ijr×Xij−Xki+Xhj2,where i, k, h∈1,2,…,NS denote three investigation locations. NS is the total number of search agents. j = 1,2,…, D where D is the number of decision variables. r represents a random number ∈−1,1.To determine where we should receive more investigation, the FBI relies on a probability featuring the quality of traces collected at a location. This probability value is computed via(15)PXij=pmax−pXipmax−pmin,where pXi denotes the objective function value of the location Xi. pmax and pmin are the largest and smallest value of the objective function in the current population, respectively.Based on the computed probability associated with each available location, the updated suspected location is given by(16)XA2,i=XBest+∑b=1a2αb×XA,bj,(17)XA2,ij=XBest+XA,dj+rn×XA,ej−XA,fi,where XBest denotes the best-found location. a2 is the number of locations that influence XA2,i. αb denotes the effectiveness coefficient ranging from −1 to 1. rn is a random number generated uniformly within [0, 1]. d, e, f, and i are the suspected locations and the first three indices are chosen randomly.In phase B, after receiving the report from the investigation team, each agentBi moves close to the location that is associated with the highest possibility. This movement is presented as follows:(18)XB1,ij=rn0XB,ij+XA,dj+rn1×XBest−XB,ij,where rn0 and rn1 are random numbers generated uniformly within [0, 1].Subsequently, each agentBi exchanges information with other agents to enhance the searching reliability. Thus, the newly updated suspected location is given by(19)XB2,ij=XB,rj+rn2×XB,rj−XB,ij+rn3×XBest−XB,rj,(20)XB2,ij=XB,ij+rn4×XB,ij−XB,rj+rn5×XBest−XB,ij,where rn2, rn3, rn4, and rn5 are random numbers generated uniformly within [0, 1].


### 2.4. The Collected Image Dataset

As mentioned earlier, the main objective of this study is to process digital image samples of asphalt pavements for detecting patched potholes and unpatched potholes. The employed pattern recognition method is SVC. Since SVC is a supervised machine learning method, a set of labeled image data must be prepared to train its model structure. This study has collected 600 asphalt pavement image samples within which the number of data samples in each class (“potholes” and “patched potholes”) is 300. This selection of sample size is to guarantee a balanced classification. The label of each image sample has been assigned by human inspectors. The collected image samples are demonstrated in Figure5.Figure 5
Demonstration of the collected image samples: (a) Class label −1 (patched pothole) and (b) Class label +1 (unpatched pothole).
(a)(b)The image dataset has been collected during field trips of the asphalt pavement survey in Da Nang city (Vietnam). The employed digital cameras are the 18-megapixel resolution Canon EOS M10 and the Gopro Hero 9 with 23.6-megapixel resolution. The images were manually taken by human inspectors. The camera is positioned at a distance of approximately 1 meter above the road surface. The image samples have obtained in cloudy weather conditions to ensure relatively consistent lighting conditions.


## 2.1. Image Texture Analysis

In the image processing field, the texture is a crucial tool used for visual perception and has been the core of many computer vision systems. Texture analysis is used to represent the degree of coarseness or fineness of objects within digital image samples. This type of analysis has been widely used in various fields of study including civil engineering [51], remote sensing [52], biomedical imaging [53], and industrial engineering [54, 55]. Based on the collected image samples, meaningful features that represent the texture properties of image regions can be computed and used for object classification [34]. In this study, statistical measurement of image pixel intensity, gray-level cooccurrence matrix’s properties, and local ternary pattern are the utilized texture descriptors used for distinguishing between potholes and patched potholes.


### 2.1.1. Statistical Properties of Color Texture

Given an image sampleI within which there are 3 color channels of red (R), green (G), and blue (B), a first-order histogram P(I) can be calculated for each channel to represent its statistical distribution [56]. Based on the computed first-order histogram of the three color channels (R, G, and B), the indices of mean μc, standard deviation σc, the skewness δc, kurtosis ηc, entropy ρc, and range Δc are computed separately for each channel. Since each channel yields 6 color texture-based indices, the total number of extracted features representing the statistical properties of color texture is 6 × 3 = 18. The equations used for computing the 6 color texture-based indices for each color channel c are presented as follows [51]:(1)μc=∑i=0NL−1Ii,c×PcI,(2)σc=∑i=0NL−1Ii,c−μc2×PcI,(3)δc=∑i=0NL−1Ii,c−μc3×PcIσc3,(4)ηc=∑i=0NL−1Ii,c−μc4×PcIσc4,(5)ρc=−∑i=0NL−1PcI×log2PcI,(6)Δc=MaxIc−MinIc,where NL = 256 denotes the number of discrete intensity values with an 8-bit image sample.


### 2.1.2. Gray-Level Cooccurrence Matrix (GLCM)

GLCM [57, 58] is a useful image processing tool used for representing properties regarding the repeated occurrence of certain gray-level patterns. Herein, the first is the analysis to convert an RGB image to a grayscale one [59]. Subsequently, the relationship between 2 pixels within this grayscale image sample is modeled using two parameters: r-the distance relationship and θ-the rotation relationship. Accordingly, a matrix denoting Pδ is used to represent a probability of the two gray levels of i and j with respect to the relationship dictated by r and θ [60]. Based on the normalized Pδ denoted as PδN, the four indices of the angular second moment (AM), contrast (CO), correlation (CR), and entropy (ET) can be calculated as follows [58]:(7)AM=∑i=1Ng∑j=1NgPδNi,j2,(8)CO=∑k=0Ng−1k2∑i=1|i−j|=kNg∑j=1NgPδNi,j,(9)CR=∑i=1Ng∑j=1Ngi×j×PδNi,j−μXμYσXσY,(10)ET=−∑i=1Ng∑j=1NgPδNi,jlogPδNi,j,where Ng is the number of gray-level values. μX,μY,σX, and σY are the means and standard deviations of the marginal distribution associated with PδNi,j [51, 57].


### 2.1.3. Local Ternary Pattern (LTP)

Putting forward in [61], LTP is an extension of the well-defined Local Binary Pattern (LBP). LBP [62] is an effective tool for characterizing local gray intensity patterns. This method considers a local neighborhood around a certain pixel and thresholds the neighboring pixels based on the value of the central pixels. Accordingly, a local texture descriptor consisting of a 3 × 3 binary matrix is used to describe the local gray intensity structure of a certain region within an image sample.Tan and Triggs [61] propose using an additional thresholding value to construct a ternary structure which can be more discriminative and less sensitive to noise in uniform regions. Therefore, the LTP texture descriptor is highly suitable for modeling gray intensity patterns of asphalt pavement roads. Using the LTP method, the value of each thresholded pixel within a local region is either −1, 0, or 1. Therefore, the computation of LTP is split into two separated matrices called a lower pattern and an upper pattern (refer to Figure 3). Similar to the standard LBP, histograms of the two aforementioned matrices can be calculated and concatenated to establish an integrated texture discriminator.Figure 3
Demonstration of the LTP texture descriptor.Mathematically speaking, the LTP texture descriptor can be described as follows:(11)s′u,ic,t=1ifu≥ic+t0ifu−ic<t−1ifu≤ic−t,where ic is the gray intensity of the central pixel. t = 5 denotes an additional threshold. u is the grayscale value of a neighboring pixel.


## 2.1.1. Statistical Properties of Color Texture

Given an image sampleI within which there are 3 color channels of red (R), green (G), and blue (B), a first-order histogram P(I) can be calculated for each channel to represent its statistical distribution [56]. Based on the computed first-order histogram of the three color channels (R, G, and B), the indices of mean μc, standard deviation σc, the skewness δc, kurtosis ηc, entropy ρc, and range Δc are computed separately for each channel. Since each channel yields 6 color texture-based indices, the total number of extracted features representing the statistical properties of color texture is 6 × 3 = 18. The equations used for computing the 6 color texture-based indices for each color channel c are presented as follows [51]:(1)μc=∑i=0NL−1Ii,c×PcI,(2)σc=∑i=0NL−1Ii,c−μc2×PcI,(3)δc=∑i=0NL−1Ii,c−μc3×PcIσc3,(4)ηc=∑i=0NL−1Ii,c−μc4×PcIσc4,(5)ρc=−∑i=0NL−1PcI×log2PcI,(6)Δc=MaxIc−MinIc,where NL = 256 denotes the number of discrete intensity values with an 8-bit image sample.


## 2.1.2. Gray-Level Cooccurrence Matrix (GLCM)

GLCM [57, 58] is a useful image processing tool used for representing properties regarding the repeated occurrence of certain gray-level patterns. Herein, the first is the analysis to convert an RGB image to a grayscale one [59]. Subsequently, the relationship between 2 pixels within this grayscale image sample is modeled using two parameters: r-the distance relationship and θ-the rotation relationship. Accordingly, a matrix denoting Pδ is used to represent a probability of the two gray levels of i and j with respect to the relationship dictated by r and θ [60]. Based on the normalized Pδ denoted as PδN, the four indices of the angular second moment (AM), contrast (CO), correlation (CR), and entropy (ET) can be calculated as follows [58]:(7)AM=∑i=1Ng∑j=1NgPδNi,j2,(8)CO=∑k=0Ng−1k2∑i=1|i−j|=kNg∑j=1NgPδNi,j,(9)CR=∑i=1Ng∑j=1Ngi×j×PδNi,j−μXμYσXσY,(10)ET=−∑i=1Ng∑j=1NgPδNi,jlogPδNi,j,where Ng is the number of gray-level values. μX,μY,σX, and σY are the means and standard deviations of the marginal distribution associated with PδNi,j [51, 57].


## 2.1.3. Local Ternary Pattern (LTP)

Putting forward in [61], LTP is an extension of the well-defined Local Binary Pattern (LBP). LBP [62] is an effective tool for characterizing local gray intensity patterns. This method considers a local neighborhood around a certain pixel and thresholds the neighboring pixels based on the value of the central pixels. Accordingly, a local texture descriptor consisting of a 3 × 3 binary matrix is used to describe the local gray intensity structure of a certain region within an image sample.Tan and Triggs [61] propose using an additional thresholding value to construct a ternary structure which can be more discriminative and less sensitive to noise in uniform regions. Therefore, the LTP texture descriptor is highly suitable for modeling gray intensity patterns of asphalt pavement roads. Using the LTP method, the value of each thresholded pixel within a local region is either −1, 0, or 1. Therefore, the computation of LTP is split into two separated matrices called a lower pattern and an upper pattern (refer to Figure 3). Similar to the standard LBP, histograms of the two aforementioned matrices can be calculated and concatenated to establish an integrated texture discriminator.Figure 3
Demonstration of the LTP texture descriptor.Mathematically speaking, the LTP texture descriptor can be described as follows:(11)s′u,ic,t=1ifu≥ic+t0ifu−ic<t−1ifu≤ic−t,where ic is the gray intensity of the central pixel. t = 5 denotes an additional threshold. u is the grayscale value of a neighboring pixel.


## 2.2. Support Vector Machine Classification (SVC)

SVC, introduced by [47], is a powerful supervised learning method used for multivariate data analysis especially for pattern classification. The model structure of this machine learning method is trained via the framework of a structural risk minimization scheme [63]. This feature helps an SVC model to be resilient to noisy data samples and to better guard against overfitting. Given a training dataset set D=x,y|x∈Sandy=fx and a target function f: X⟶−1,+1 (−1 = “patched pothole” and +1 = “pothole”), the SVM can be used to learn f(x) by constructing an approximated function f^x:X⟶−1,+1.The overall learning phase of SVC is illustrated in Figure4. The first step of the learning process is to construct a function to map the data in the original input space into a high-dimensional feature space. It is noted that the original input space consists of numerical data generated from the aforementioned texture descriptors. In such high-dimensional feature space, a hyperplane can be learnt to act as a decision boundary to separate the input data into two regions of “pothole” and “patched pothole”. To identify this hyperplane, it is required to solve the following constrained optimization problem:(12)MinimizeJpw,e=12wTw+C12∑k=1Nek2,subjected to ykwTφxk+b≥1−ek, k=1,…,N, ek≥0. w∈Rn and b∈R denote the parameters of the hyperplane. e represents the vector of slack variables. C is the penalty coefficient. φx is the aforementioned nonlinear data mapping function.Figure 4
Demonstration of the SVC learning phase.A notable advantage of the SVC machine learning method is that it does not require the explicit form of the mapping functionφx. The quantity of interest is the dot product of φx denoting a kernel function K(xk, xl). For nonlinear and multivariate data classification, the commonly used kernel function is the radial basis kernel function (RBKF) shown as follows:(13)Kxk,xl=exp−xk−xl22σ2,where σ is a tuning parameter of the kernel function.


## 2.3. Forensic-Based Investigation (FBI)

As described in the previous section, the training phase of an SVC model necessitates a proper setting of the penalty coefficientC and the RBKF’s tuning parameter σ. These are two crucial hyperparameters of SVC. The former expresses the amount of penalty imposing on misclassified data samples. The latter specifies the locality of the employed kernel function which affects the generalization of a trained SVC model. Since the task of hyperparameter setting can be formulated as an optimization problem, this study relies on the FBI metaheuristic to assist the SVC model’s training phase.FBI [50] is a stochastic search engine motivated by the forensic investigation process. The searching process of this stochastic search engine mimics the real operation of forensic investigation which contains five steps: open a case, interpretation of findings, the direction of inquiry, action, and prosecution [64]. The searching process of the FBI can be broken down into two phases: the investigation phase (called phase A) and the pursuit phase (called phase B). The former is carried out by a team of investigators. The latter is performed by a team of police agents. These two phases are repeated until a stopping condition (i.e., a maximum number of iterations) is met.In phase A, based on the trace recorded at the current locationXij, a new investigation location Xij,new is computed as follows:(14)Xij,new=X+ijr×Xij−Xki+Xhj2,where i, k, h∈1,2,…,NS denote three investigation locations. NS is the total number of search agents. j = 1,2,…, D where D is the number of decision variables. r represents a random number ∈−1,1.To determine where we should receive more investigation, the FBI relies on a probability featuring the quality of traces collected at a location. This probability value is computed via(15)PXij=pmax−pXipmax−pmin,where pXi denotes the objective function value of the location Xi. pmax and pmin are the largest and smallest value of the objective function in the current population, respectively.Based on the computed probability associated with each available location, the updated suspected location is given by(16)XA2,i=XBest+∑b=1a2αb×XA,bj,(17)XA2,ij=XBest+XA,dj+rn×XA,ej−XA,fi,where XBest denotes the best-found location. a2 is the number of locations that influence XA2,i. αb denotes the effectiveness coefficient ranging from −1 to 1. rn is a random number generated uniformly within [0, 1]. d, e, f, and i are the suspected locations and the first three indices are chosen randomly.In phase B, after receiving the report from the investigation team, each agentBi moves close to the location that is associated with the highest possibility. This movement is presented as follows:(18)XB1,ij=rn0XB,ij+XA,dj+rn1×XBest−XB,ij,where rn0 and rn1 are random numbers generated uniformly within [0, 1].Subsequently, each agentBi exchanges information with other agents to enhance the searching reliability. Thus, the newly updated suspected location is given by(19)XB2,ij=XB,rj+rn2×XB,rj−XB,ij+rn3×XBest−XB,rj,(20)XB2,ij=XB,ij+rn4×XB,ij−XB,rj+rn5×XBest−XB,ij,where rn2, rn3, rn4, and rn5 are random numbers generated uniformly within [0, 1].


## 2.4. The Collected Image Dataset

As mentioned earlier, the main objective of this study is to process digital image samples of asphalt pavements for detecting patched potholes and unpatched potholes. The employed pattern recognition method is SVC. Since SVC is a supervised machine learning method, a set of labeled image data must be prepared to train its model structure. This study has collected 600 asphalt pavement image samples within which the number of data samples in each class (“potholes” and “patched potholes”) is 300. This selection of sample size is to guarantee a balanced classification. The label of each image sample has been assigned by human inspectors. The collected image samples are demonstrated in Figure5.Figure 5
Demonstration of the collected image samples: (a) Class label −1 (patched pothole) and (b) Class label +1 (unpatched pothole).
(a)(b)The image dataset has been collected during field trips of the asphalt pavement survey in Da Nang city (Vietnam). The employed digital cameras are the 18-megapixel resolution Canon EOS M10 and the Gopro Hero 9 with 23.6-megapixel resolution. The images were manually taken by human inspectors. The camera is positioned at a distance of approximately 1 meter above the road surface. The image samples have obtained in cloudy weather conditions to ensure relatively consistent lighting conditions.


## 3. The Proposed Computer Vision-Based Forensic-Based Investigation Metaheuristic Optimized Machine Learning for Patched and Unpatched Pothole Detection

This section describes the overall structure of the proposed model used for automatic detection of the pothole and patched pothole. The model is an integration of image processing techniques, supervised machine learning-based data classification, and metaheuristic optimization. In detail, the image texture descriptors including statistical measurements of color channels, GLCM, and LTP are used to compute numerical features of image samples. Based on such extracted features, SVC incorporated with the FBI metaheuristic is used to generalize a decision boundary that separates the collected dataset into two distinctive classes of “patched pothole” and “pothole”. The FBI is utilized to assist the SVC training phase by optimizing its tuning parameters (the penalty coefficient and the RBKF parameter). The overall structure of the proposed hybrid model is presented in Figure6.Figure 6
The proposed image processing-based Forensic-Based Investigation optimized machine learning for patched and unpatched pothole detection.The operational flow of the model can be divided into three main steps:(i)
Feature extraction(ii)
Model optimization(iii)
Model prediction and evaluationThe feature computation module has been programmed by the authors in Visual C# .NET. Additionally, the FBI optimized SVC module is constructed in a MATLAB environment with the built-in functions provided in the Statistics and Machine Learning Toolbox [65]. The source code of the FBI is provided by [50,66]. The final prediction model based on image texture computation and the optimized SVC model is constructed and compiled in Visual C# .NET with the assistance of the Accord.NET Framework [67].


### 3.1. Feature Extraction

In this step, texture descriptors including statistical measurements of color channels, GLCM, and LTP are employed to calculate features of the collected image samples. The statistical properties of image color, including three channels (R, G, and B), yield 18 features which are the mean, standard deviation, skewness, kurtosis, entropy, and range. To compute the GLCM-based texture, four GLCM withr = 1 and θ = 0o, 45o, 90o, and 135o are computed as suggested by [57]. Since each matrix yields four properties of AM, CO, CR, and ET, the number of features generated by the GLCM texture descriptor is 4 × 4 = 16. In addition, the LBP feature descriptor produces 118 features that represent the local structure of grayscale image samples [61]. According to the suggestion of previous works, the threshold t of the LBP is set to be 5 [61, 68]. Accordingly, there are 152 texture-based features employed for categorizing patched potholes and potholes. Demonstrations of the feature extraction step are provided in Figure 7.Figure 7
Demonstration of the feature extraction process: (a) a sample of the “patched pothole” class and (b) a sample of the “pothole” class.
(a)(b)Subsequently, the extracted dataset has been randomly divided into a training set (70%) and a testing set (30%). The first set is employed for the model construction phase, and the second set is reserved for inspecting the model generalization capability. Moreover, to standardize the input data range, theZ-score equation is employed:(21)XZ=XD−MXSTDX,where XZ and XD are the normalized and the original texture features, respectively. MX and STDX are to the mean value and the standard deviation of the texture features, respectively.


### 3.2. Model Optimization

The FBI metaheuristic with the number of searching agents of 20 and the maximum number of searching iterations of 100 is used to optimize the SVM model. FBI aims at finding an appropriate set of the SVC hyperparameters including the penalty coefficient and the RBKF parameter. These hyperparameters strongly affect the learning and the predictive capability of the SVC model. The inappropriate setting of the penalty coefficient and the RBKF parameter may lead to either an overfitted or underfitted model. Hence, the objective of the FBI optimization process is to identify a set of hyperparameters that demonstrates a good training accuracy as well as generalization property. Moreover, the lower and the upper bounds of the parameters searched by the FBI metaheuristic are set to be 0.001 and 1000, respectively.In order to optimize the SVC model, this study employs a 5-fold cross-validation-based objective function. This objective function is mathematically described as follows [69]:(22)OF=∑k=15FNRk+FPRk5,where FNRk and FPRk are false negative rate (FNR) and false positive rate (FPR) calculated in the kth run, respectively.The FNR and FPR indices are calculated according to the following equations [69]:(23)FNR=FNFN+TP,(24)FPR=FPFP+TN,where FN, FP, TP, and TN denote the false negative, false positive, true positive, and true negative data samples, respectively.


### 3.3. Model Prediction and Evaluation

In this step, the SVM model with the optimized hyperparameters (the penalty coefficient and the RBFK parameter) is constructed in Visual C# .NET framework 4.7.2 classify data instances in the testing set into two categories of “patched pothole” and “pothole. Moreover, it is noted that the developed computer program has been implemented on the ASUS FX705GE-EW165T (Core i7 8750H and 8 GB Ram) platform.To evaluate the model prediction capability, a set of performance measurement metrics is used. The employed performance measurement metric consists of classification accuracy rate (CAR), precision, recall, negative predictive value (NPV), F1 score, and area under the receiver operating characteristic curve (AUC) [70, 71]. The construction of the receiver operating characteristic curve is provided in [71]. The indices of CAR, precision, recall, NPV, and F1 score are calculated according to the following formulas:(25)CAR=NCNA100%,(26)precision=TPTP+FP,(27)recall=TPTP+FN,(28)NPV=TNTN+FN,(29)F1score=2TP2TP+FP+FN,where NC and NA are the numbers of correctly predicted data and the total number of data, respectively. As mentioned earlier, FN, FP, TP, and TN are the false negative, false positive, true positive, and true negative data samples, respectively.


## 3.1. Feature Extraction

In this step, texture descriptors including statistical measurements of color channels, GLCM, and LTP are employed to calculate features of the collected image samples. The statistical properties of image color, including three channels (R, G, and B), yield 18 features which are the mean, standard deviation, skewness, kurtosis, entropy, and range. To compute the GLCM-based texture, four GLCM withr = 1 and θ = 0o, 45o, 90o, and 135o are computed as suggested by [57]. Since each matrix yields four properties of AM, CO, CR, and ET, the number of features generated by the GLCM texture descriptor is 4 × 4 = 16. In addition, the LBP feature descriptor produces 118 features that represent the local structure of grayscale image samples [61]. According to the suggestion of previous works, the threshold t of the LBP is set to be 5 [61, 68]. Accordingly, there are 152 texture-based features employed for categorizing patched potholes and potholes. Demonstrations of the feature extraction step are provided in Figure 7.Figure 7
Demonstration of the feature extraction process: (a) a sample of the “patched pothole” class and (b) a sample of the “pothole” class.
(a)(b)Subsequently, the extracted dataset has been randomly divided into a training set (70%) and a testing set (30%). The first set is employed for the model construction phase, and the second set is reserved for inspecting the model generalization capability. Moreover, to standardize the input data range, theZ-score equation is employed:(21)XZ=XD−MXSTDX,where XZ and XD are the normalized and the original texture features, respectively. MX and STDX are to the mean value and the standard deviation of the texture features, respectively.


## 3.2. Model Optimization

The FBI metaheuristic with the number of searching agents of 20 and the maximum number of searching iterations of 100 is used to optimize the SVM model. FBI aims at finding an appropriate set of the SVC hyperparameters including the penalty coefficient and the RBKF parameter. These hyperparameters strongly affect the learning and the predictive capability of the SVC model. The inappropriate setting of the penalty coefficient and the RBKF parameter may lead to either an overfitted or underfitted model. Hence, the objective of the FBI optimization process is to identify a set of hyperparameters that demonstrates a good training accuracy as well as generalization property. Moreover, the lower and the upper bounds of the parameters searched by the FBI metaheuristic are set to be 0.001 and 1000, respectively.In order to optimize the SVC model, this study employs a 5-fold cross-validation-based objective function. This objective function is mathematically described as follows [69]:(22)OF=∑k=15FNRk+FPRk5,where FNRk and FPRk are false negative rate (FNR) and false positive rate (FPR) calculated in the kth run, respectively.The FNR and FPR indices are calculated according to the following equations [69]:(23)FNR=FNFN+TP,(24)FPR=FPFP+TN,where FN, FP, TP, and TN denote the false negative, false positive, true positive, and true negative data samples, respectively.


## 3.3. Model Prediction and Evaluation

In this step, the SVM model with the optimized hyperparameters (the penalty coefficient and the RBFK parameter) is constructed in Visual C# .NET framework 4.7.2 classify data instances in the testing set into two categories of “patched pothole” and “pothole. Moreover, it is noted that the developed computer program has been implemented on the ASUS FX705GE-EW165T (Core i7 8750H and 8 GB Ram) platform.To evaluate the model prediction capability, a set of performance measurement metrics is used. The employed performance measurement metric consists of classification accuracy rate (CAR), precision, recall, negative predictive value (NPV), F1 score, and area under the receiver operating characteristic curve (AUC) [70, 71]. The construction of the receiver operating characteristic curve is provided in [71]. The indices of CAR, precision, recall, NPV, and F1 score are calculated according to the following formulas:(25)CAR=NCNA100%,(26)precision=TPTP+FP,(27)recall=TPTP+FN,(28)NPV=TNTN+FN,(29)F1score=2TP2TP+FP+FN,where NC and NA are the numbers of correctly predicted data and the total number of data, respectively. As mentioned earlier, FN, FP, TP, and TN are the false negative, false positive, true positive, and true negative data samples, respectively.


## 4. Experimental Results and Discussions

As aforementioned, the FBI is employed to optimize the SVM-based pothole-patched pothole detection model. Using 20 searching agents and after 100 iterations, the metaheuristic algorithm has found the best penalty coefficient = 118.941 and the RBKF parameter = 20.300. The best-found cost function value which is described in equation (22) is 1.047. The FBI algorithm is capable of locating the most desired values of the searched variables after 41 iterations. The searching progress of the FIB metaheuristic is illustrated in Figure 8. It is noted that, in Figure 8, the model performance is computed according to equation (22) which is stated in the previous section.Figure 8
The FBI-based optimization progress.As stated earlier, the collected dataset which includes 600 instances has been randomly divided into a training set (70%) and a testing set (30%). The former is employed for model construction and the latter is served as novel data instances for verifying the model’s predictive capability. Additionally, to reliably assess the predictive performance, this study has repeated the model training and verification processes 20 independent times. The statistical measurements attained from this multiple model training and verification phases are utilized for model evaluation. This process can be helpful in diminishing the variation caused by the randomness in data sampling.In addition, to demonstrate the superiority of the newly constructed model, its performance is compared to those of other capable benchmark models including Relevance Vector Machine (RVM) [72, 73], Random Forest Model (RFM) [74], and Classification Tree Model (CTM) [75, 76]. RVM is constructed with the source code provided in [73]. The RFM and CTM approaches are trained with MATLAB’s Statistics and Machine Learning Toolbox [65]. It is noted that grid search procedures with the fivefold cross-validation process [77] are employed to set the tuning parameters of the benchmark models. The RVM’s parameter of basis width is selected to be 0.01. In addition, the number of individual decision trees used in the RFM is 50 and the parameter of minimum leaf size is 1.The model prediction results are reported in Table1 that shows the mean and standard deviation (Std) of each performance measurement index obtained from the testing process. As seen from this table, the proposed integration of FBI and SVC (denoted as FBI-SVC) has achieved the most desired predictive performance in terms of the CAR = 94.833%, the precision = 0.935, the recall = 0.964, the NPV = 0.963, and the F1 score = 0.949. The RVM is the second-best model with the CAR = 90.833%, the precision = 0.904, the recall = 0.914, the NPV = 0.914, and the F1 score = and 0.909, followed by the RFM (with the CAR = 89.889%, the precision = 0.881, the recall = 0.923, the NPV = 0.920, and the F1 score = 0.901) and CTM (with the CAR = 85.889%, the precision = 0.869, the recall = 0.847, the NPV = 0.851, and the F1 score = 0.857). The model comparison in terms of the CAR is illustrated in Figure 9. The results of the proposed model and the benchmark approach with respect to the precision, recall, NPV, and F1 score are provided in Figures 10 and 11 to ease the visual comparison.Table 1
Experimental result comparison.
MetricsFBI-SVCRVMRFMCTMMeanStdMeanStdMeanStdMeanStdCAR (%)94.8331.91790.8332.15289.8892.02085.8892.196Precision0.9350.0270.9040.0260.8810.0230.8690.033Recall0.9640.0190.9140.0280.9230.0330.8470.027NPV0.9630.0170.9140.0260.9200.0310.8510.022F1 score0.9490.0190.9090.0220.9010.0210.8570.021AUC0.9890.0060.9620.0150.9620.0100.8810.037Figure 9
Performance comparison with respect to the CAR.Figure 10
Performance comparison with respect to precision, recall, and NPV.Figure 11
Performance comparison with respect to the F1 score and AUC.Particularly for the AUC index, the FBI-SVC is the best model (AUC = 0.989), followed by RVM (AUC = 0.962), RFM (AUC = 0.962), and CTM (AUC = 0.881). The receiver operating characteristic curves of the proposed model and the benchmark model employed for patched pothole–pothole classification are demonstrated in Figure12. A receiver operating characteristic curve (ROC) is a graphical plot widely used for diagnosing the predictive capability of classification models. ROC and AUC are crucial metrics because they include important predictive outcomes in terms of FPR and TPR in one plot. As can be observed from Figure 12, the AUC of FBI-SVC is very close to 1 meaning that it has achieved a highly accurate predictive result. Furthermore, the box plots of CAR, F1 score, and AUC are provided in Figures 13–15.Figure 12
ROCs of the prediction models: (a) FBI-SVC, (b) RVM, (c) RFM, and (d) CTM.
(a)(b)(c)(d)Figure 13
Box plot of CAR values obtained from the employed machine learning models.Figure 14
Box plot of F1 score values obtained from the employed machine learning models.Figure 15
Box plot of AUC values obtained from the employed machine learning models.Particularly, Figure14 displays the models’ performance in terms of the F1-score. This index is an important and representative measurement of predictive accuracy. The F1-score is a combination of the precision and recall indices. In other words, this performance measurement metric is the harmonic mean of precision and recall. The possible value of the F1-score ranges from 0 and 1; 1 indicates a perfect classification model. The higher the value of this index is, the better the predictive accuracy is. The value of the F1-score = 0.989 obtained from the FBI-SVC indicates a highly capable model used for patched/nonpatched pothole detection. Moreover, based on the boxplots shown in Figure 14, it can be observed that the proposed model’s performance in terms of F1-score is significantly better than those of the benchmark methods.Moreover, to confirm the superiority of the proposed FBI-SVC, this study has relied on the nonparametric Wilcoxon signed-rank test [78] with the significant level (p value) = 0.05. The crucial indices of CAR, F1 score, and AUC are the subjects of the Wilcoxon signed-rank test. The test outcomes of pairwise model comparison are reported in Tables 2–4, respectively. Evidently, with p values <0.05, it is able to reject the null hypotheses of equal model performances and confirm the superiority of the proposed FBI-SVC.Table 2
p values obtained from the Wilcoxon signed-rank test results with CAR index.
ModelsFBI-SVCRVMRFMCTMFBI-SVCX0.000110.000090.00009RVM0.00011X0.211560.00014RFM0.000090.21156X0.00023CTM0.000090.000140.00023XTable 3
p values obtained from Wilcoxon signed-rank test results with F1 score index.
ModelsFBI-SVCRVMRFMCTMFBI-SVCX0.000100.000090.00009RVM0.00010X0.295880.00014RFM0.000090.29588X0.00019CTM0.000090.000140.00019XTable 4
p values obtained from Wilcoxon signed-rank test results with AUC index.
ModelsFBI-SVCRVMRFMCTMFBI-SVCX0.000090.000100.00009RVM0.00009X0.765200.00009RFM0.000100.76520X0.00013CTM0.000090.000090.00013XAlthough FBI-SVC has achieved the most desired predictive performance in distinguishing between patched potholes and potholes. This model has also committed several misclassifications as demonstrated in Figure16. Inspecting the misclassified cases, it is revealed that a patched area surrounded by raveling and partly covered by traffic marks can lead to falsely predicted potholes (refer to Figures 16(a) and 16(b)). Moreover, the cases of irregular lighting conditions and a pothole cover by a layer of dirt have caused false predictions of patched potholes as shown in Figures 16(c) and 16(d). This indicates that more efforts should be needed to deal with such irregular scenarios and improve the model prediction accuracy.Figure 16
Illustrations of misclassified cases.


## 5. Concluding Remarks

Asphalt pavements play a very important role in the national and local transportation networks. Timely information regarding the distresses appearing on pavement surface is crucial for pavement management agencies to make a cost-effective decision on maintenance method and schedule. To obtain accurate and up-to-date information on the serviceability of asphalt pavement road sections, periodic surveys need to be performed regularly and the visual information attained by such surveys needs to be processed in a timely manner. To expedite this visual information processing, this study proposes an intelligent model for automatically identifying patched potholes and unpatched potholes. Potholes are widely encountered pavement distress and may cause serious damages to vehicles as well as human casualties. However, based on 2D digital images, potholes and patched potholes can have similar shapes. Therefore, this study has proposed employing image texture information to delineate these two objects.With such research motivations, this study has relied on image texture descriptors of statistical information of color channels, GLCM, and LTP. These three texture descriptors are employed to extract useful information regarding the coarseness/fineness of image regions containing patched and unpatched potholes. An image set consisting of 600 asphalt pavement image samples has been collected and used to construct a texture-based dataset. Within this dataset, the number of data samples in each category is 300. Using the established texture-based dataset, a novel machine learning approach that hybridizes the SVC pattern classifier and the FBI metaheuristic has been proposed to construct a decision boundary that can deliver prediction of the class labels.The experimental outcomes supported by the Wilcoxon signed-rank test show that the FBI-SVC model is superior to those of the benchmark approaches. Good classification results with CAR = 94.833%, precision = 0.935, recall = 0.964, NPV = 0.963, F1 score = 0.949, and AUC = 0.989 show that the proposed FBI-SVC is well suited for the task of patched-unpatched pothole classification. Further extensions of the current research work may include the following:(i)
The application of the FBI-SVC to detect other forms of pavement distresses (e.g., alligator crack, raveling, blurred traffic marks, etc.)(ii)
The investigation of other advanced texture descriptors for characterizing local structure or pattern of image regions containing pavement distresses(iii)
Collecting more image samples to enhance the generalization and applicability of the current computer vision method



---

*Source: 3511375-2021-09-06.xml*