# Automatic Road Pavement Assessment with Image Processing: Review and Comparison

Authors: Sylvie Chambon; Jean-Marc Moliard
Journal: International Journal of Geophysics
 (2011)

---

## Abstract

In the field of noninvasive sensing techniques for civil infrastructures monitoring, this paper addresses the problem of crack detection, in the surface of the French national roads, by automatic analysis of optical images. The first contribution is a state of the art of the image-processing tools applied to civil engineering. The second contribution is about fine-defect detection in pavement surface. The approach is based on a multi-scale extraction and a Markovian segmentation. Third, an evaluation and comparison protocol which has been designed for evaluating this difficult task—the road pavement crack detection—is introduced. Finally, the proposed method is validated, analysed, and compared to a detection approach based on morphological tools.


---

## Body

## 1. Introduction

The evaluation of road quality is an important task in many countries, like in France, where the national roads are inspected each three years in order to estimate the needed reparations. To estimate the quality, these aspects can be taken into account: the adherence, the microtexture, the macrotexture, and the surface degradations. Before 1980, all these inspections were accomplished manually. Since 1980, this task can be automated with noninvasive techniques to be more comfortable, less dangerous for employees and users of the road but also more efficient and less expensive than manual methods. Many systems have been proposed, based on ground-penetrating radar [1] or laser system [2]. However, for noninvasive evaluation of surface degradations, the recent research results seem more promising with optical image-processing approaches for these reasons [3]:(1)
The acquisition systems based on optical devices are easier to design and to use than other kinds of systems (they are less sensitive to movement and to vibrations than other systems).(2)
They also allow a dense acquisition (each millimeter), that is, the acquisition can be realized for the whole road surface, whereas for the other systems, like laser, the measurements are available every 4 millimeters at normal speed (90 km/h).1(3)
The measurement of the defects is more precise than with other systems because, as explained in (2), enough information is available.(4)
Even if the images are not always well contrasted, they are more contrasted than the images/signals that can be given by other devices, that is, the ratio between noise and signal is greater with optical sensor than with other kind of sensors.Nowadays, many acquisition systems are available [3, 4], see Table 1 (interested readers can find details about the evaluation of such systems in [5, 6]). Moreover, to the best of our knowledge, many methods for semiautomatic detection of road defects can be found in the literature but only one is commercialized (by INO2). Among all the existing approaches, it is difficult to know which one is the most adapted to the task and what is the actual method that is the most used. This is why the first goal of this paper is to present a state of the art of assessment methods in noninvasive control based on image processing.Table 1
The quality evaluation systems of the road pavement based on optical devices L, CLS, LC, and IPT mean, respectively, that the described system involves a laser, a controlled lighting system, a linear camera and/or some image processing tools. The first set corresponds to systems without image processing tools. While the second group of systems are able to provide nondense measurement, the third group of systems can give a dense evaluation of the quality of the road.
NameYearCountryLCLSLCIPTADDA, automated distress data acquisition1991USA—ACM, automated crack monitor1991USA—SIRANO1991France—HARRIS, Highways Agency Road Research Information Systema1999UK——ADA, automated distress analyserb2007USA——AIGLE RNc2008France—AMACd2004France————Profilograph and lasere2007Danemark———REAL, road excellent automatic loggingf1992Japan————ARAN, automatic road analyser and then FuGro ADVantageg1993/2003Canada———PAVUEh1999Sweden——RoadCracki2008Australia———ahttp://www.trl.co.uk/facilities/mobile_test_equipment/highways_agency_road_research_information_system.htm.bhttp://www.waylink.com/.chttp://www.cete-normandie-centre.equipement.gouv.fr/IMG/pdf/15-AigleRNpress_cle243947.pdf.dhttp://www.vectra.fr/sites/fr/materiels-et-solutions/auscultations-des-chaussees/amac.html.ehttp://www.greenwood.dk/road.php.fhttp://www.pasco.co.jp/eng/solutions/geospatial/ground/.ghttp://www.roadware.com.hhttp://rst.ramboll.se/en/produkter/rst%20pavue.aspx.ihttp://www.csiro.au/solutions/psaa.html.The detection of crack is difficult in the context of road surface evaluation because the signal to detect is weakly represented (1.5% of the whole image) and weakly contrasted (the road possesses a texture that hides the crack). Recent methods have shown their limits; the detection contains a lot of false detections (induced by the particular texture of the road), and the detection is not enough precise (the given result is a region of detection and not the skeleton with the width of the crack). The main default of the existing methods is the fact that the specific geometry of the crack—it is a thin and linear object—is not taken into account. In consequence, the second aim of this work is to introduce a new method that takes into account some geometric properties of the cracks.Even if this problem is hard and very important in the field of civil engineering, as far as we are concerned, there is no protocol for evaluating and for comparing existing methods, and it is difficult to know what kind of methods has to be chosen for this task. In consequence, with the multiple methods proposed in the literature, it seems important to evaluate and to compare the various methods in order to validate previous work and to identify the approaches that can be employed and/or the methods that need improvements. So, the third aspect discussed in this paper is the introduction of such a protocol.In consequence, the objectives are as follows: first, to give a state of the art of the existing methods in noninvasive control based on image processing for estimating the quality of the road surface, second, to present our method, and, third, to introduce a protocol of evaluation and comparison that allows to highlight the advantages and drawbacks of each method.


## 2. Automatic Road Crack Detection

In the literature, many methods have been introduced to detect thin objects in textured images, like that in medical imagery, for the detection of blood vessels [27], and satellite imagery, for road network detection [28]. Since 1990, algorithms have been proposed for the semiautomatic detection of road cracks (interested readers can see [29] for details about road imaging systems and their limits). For the detection of cracks, three components have to be taken into account      (1)
acquisition (see Table 2 for details),(2)
storage, and(3)
image processing.Table 2
The imaging systems. For each kind of configuration, the given details are the orientation of the sensor (O.) that can be perpendicular or not to the road plane, the number of sensors, the presence (—) or not of a laser, the presence (—) or not of overlapping images (R.) between the different acquisitions. For the sensors, we distinguish 2D sensors (CCD, charged couple device) from 1D sensor (linear camera coupled with laser).
O.SensorLaserR.Citations⊥1 (2D sensor)[7–17]⊥1 (2D sensor)—[18, 19]⊥More than one (2D sensors)[20, 21]⊥More than one (1D sensors)[22, 23]⊥More than one (2D sensors)—[24, 25]⊥More than one (2D sensors)—[26]In this paper, only the last step is studied, but the choices for the two first steps are important for the success of the image treatment. Moreover, most of the references are given in the field of road quality assessment, but some of them come from different applications, like cracks and defects in concrete (for bridges or pipelines), on ceramics or on metallic surfaces (for industrial applications). For road cracks, most of the time, these hypotheses can be exploited.(1)
Photometric hypotheses
(Hp1)
The crack pixels are darker than the road pixels.(Hp2)
The gray-level distributions of road crack and road surface are independent.(2)
Geometric hypotheses
(Hg1)
A crack is a thin continuous object.(Hg2)
A crack is a set of connected segments with different orientations.(Hg3)
A crack does not have a constant width on the whole length.(3)
Photometric and geometric hypotheses
(Hpg1)
The points inside a crack can be considered as points of interest, from a photometric and/or a geometric point of view.These different hypotheses can be complementary, like (Hp1) and (Hp2) or (Hg1) and (Hg3), but some of them are opposite, like (Hg1) and (Hg2). The hypothesis (Hpg1) combines two kinds of constraint because the definition of a point of interest (POI), that is, a significant point in a scene, can be expressed both with photometric constraints (some hypotheses about the distribution of gray levels near POI can be made) and geometric constraints (a point of interest can be a corner, an edge, or any kind of geometric structure).In the field of image processing, the semiautomatic methods and the automatic detection approaches are considered, and these five families can be distinguished, see Table3.      (1)
Based onhistogram analysis (hypotheses Hp1and Hp2), these methods are the most ancient and the most popular. They use a thresholding based on an histogram analysis [7, 30, 31], with Gaussian hypotheses [9] and/or an adaptive or a local thresholding [32, 33]. These approaches are simple and not time consuming, but they also give many false detections. In fact, these methods assume that the two gray level distributions (the road pavement distribution and the crack distribution) can be separated based on a global level statistics (histogram3). In Figure 1, we can see that most of the time, this hypothesis is not valid.(2)
Based onmathematical morphological tools [15, 33–38] (hypotheses Hp1 and Hg1), an initial thresholding is needed, and the results contain less false detections than methods based on histogram analysis. However, the major drawback of this kind of techniques is that the quality of the results is highly dependent on the parameter choices.(3)
Based on alearning phase in order to alleviate the problems of the two first groups of methods [39, 40] (hypotheses Hp1 and Hp2), most of the approaches are based on neural networks [8, 41, 42]. The drawback is the learning step that cannot allow a fast and fully automatic analysis.(4)
Based onfiltering, the most recent ones (hypotheses Hp1, Hg1, and Hg3). Edge extraction by filtering with fixed scale is not adapted to the task of the detection of road cracks because the width of the crack is not constant, and this is why many methods are based on wavelet detections [17, 25, 43, 44] with adaptive filtering [27, 45, 46] (these approaches will be detailed in the Section 4), contourlets [47], Gabor’s filters [48], finite impulse response filter (FIR) [26], and methods using models based on partial differential equations (PDE) [49, 50]. Some techniques also use autocorrelation filtering [51, 52] (a similarity score is estimated between some targets that simulate cracks and all the targets of the original image). An other kind of algorithms is based on texture analysis [53, 54] (the crack is considered as a noise inside a texture).(5)
Based on an analysis of amodel [55, 56] (hypotheses Hp1, Hg1–g3, and Hpg1). Most of these approaches are based on a local analysis versus a global analysis in order to take into account the local properties and the global properties of a crack, by multiscale analysis of texture combined with an algorithm of minimal path [55] or by local detection of points of interest combined with geodesic contours [56].Table 3
The classification of existing methods for crack detection (or detection of thin objects in textured images) into five different families. For each family, the hypotheses used by the methods of the family are specified.
Histogram (Hp1, Hp2)Learning (Hp1, Hp2)Morphology (Hp1, Hg1)Filtering (Hp1, Hg1–g3)Model-based (Hp1, Hg1–g3, Hpg1)[7, 9, 11, 12, 14, 20, 22, 30–32, 35, 57–62][8, 10, 13, 21, 39–42, 63–65][15, 23, 33, 34, 36, 37, 66–69][17, 25–28, 38, 43, 44, 46–54, 70, 71][16, 24, 45, 55, 56, 72, 73]Examples of histograms (second row) of images (first row) with cracks. These histograms contain only one mode, and it is impossible to separate the gray-level distribution of cracks from the gray-level distribution of the road pavement.In conclusion, we can notice the following.(i)
Many methods have been proposed, but the problem is not still solved. The results contain many false positives, and the detections are incomplete. Moreover, most of the existing techniques can give interesting results for only one given class of road pavement, that is, the performance of the method is dependent on the road texture.(ii)
Methods based on histogram study, even those that are local, do not express correctly the problem, that is, they do not take into account geometric characteristics of the cracks and photometric characteristics of the road pavement.(iii)
Learning methods are efficient,s but the learning step is expensive (the time and the investments from the users that are not expert in image processing).For all these reasons, even if learning methods have been used in our previous work, this paper focuses on the presentation of two methods proposed to alleviate the limits of the old ones by obtaining a dense detection with a low rate of false detections.


## 3. Proposed and Compared Methods

Before introducing the proposed method, we briefly present the preliminary works that motivate and justify our proposition. First of all, a neuron-based method has been tested [13], on the real images of size 768×512 presented in Section 4.2. Results are interesting, but learning methods are not easy to use for a nonspecialist in image processing, and the users have to spend a lot of time for setting the parameters and for building the database for the learning step before using the method. The main goal is to propose a system that facilitates the work of users and not a system that induces a lost time by including a learning phase and a maintenance each year in order to maintain the performances of the system.4 In consequence, we have now focused our work on methods that allow automatic processing, and, in particular, we present two approaches as follows.      (1)
The first,Morph, belongs to the families (1) and (2) because it combines thresholding and refinement by morphological analysis.(2)
The second,GaMM, of families (4) and (5), is based on the advantage of multiscale analysis and local modelling of the crack.Morph has been proposed before GaMM and is quite equivalent to the method presented in [15]. The contributions of this section are about GaMM; we propose a new model for the sites and the potentials used in the Markovian model. The advantages of this new method will be illustrated with qualitative and quantitative results in Section 5.


### 3.1. Morphological Method (Morph)

The chosen approach is based on hypothesesHp1, Hg1, and Hg3, and it follows these steps.      (1)
Preprocessing of the images: to reduce the influence of the texture and to increase the contrast between the road pavement and the crack.(2)
Binarization by thresholding (the threshold is different in the various variant, and a local threshold can be used).(3)
Refinement by closing.(4)
Segmentation with shape analysis.(5)
Extraction of the characteristics of the cracks.For step (1), three variants are developed, based on the combination of these local tools: an erosion in gray levels, a conditional median filtering, a histogram equalization, a mean filtering (these preprocessings are detailed in Section5.1). The step (4) is realised in two stages. First, a labeling by analysis of the connected components is realized. Second, the size and the shape of each component is determined in order to remove components which have a shape that is not similar to a crack; the shape of a crack has to be a thin object. It induces constraints on the width w and the height h of the component. More precisely, from an expert point of view, a crack is not significant if h<50 cm, but we can suppose that we manage to detect only a small part of the crack, and this constraint becomes h<7.5 mm. Moreover, the mean width and the maximal width of the crack have to respect these rules: wmin<3 mm and wmax<6.5 mm. All these thresholds are empirically set. In Figure 2, we illustrate the kind of results obtained at each step for the 3 variants. The next step of the method Morph merges the 3 results (with a weighted sum, and the weights are chosen with a learning phase). The final stage refines the result by computing the closing in gray levels of the fusion result.Figure 2
The different steps of the methodMorph. The conditional filtering is applied when the gray level is higher than 40 (to prevent the removal of the crack). The last step is proposed in order to reduce false detections and to complete the detection.


### 3.2. Adaptive Filtering and Markovian Modelling (GaMM)

More recently, our work focused on the field of wavelet decomposition. As it is difficult to chose the mother wavelet5 well adapted to the detection of road cracks, the adaptive filter theory seems convenient, and, in particular, it allows to build a mother wavelet adapted to our task. We present the first step of the algorithm based on adaptive filtering (hypotheses Hp1 and Hg3) and the second stage on Markovian segmentation that can take into account the particular geometry of the crack (Hg2 and Hg3).


#### 3.2.1. Algorithm

The goal of this algorithm, presented in Algorithm1, is to obtain, step (1), a binarization (black pixels for the background and white pixels for the cracks) and a refinement of this detection by using a Markovian segmentation, step (2). Using adaptive filtering is important in order to allow the detection of the crack with nonconstant width (hypothesis Hg3).6 The number of scales for the adaptive filtering has to be chosen and depends on the resolution of the image. By supposing a resolution of 1 mm per pixel, by choosing 5 scales, a crack with a width from 2 mm to 1 cm can be detected. Moreover, the number of directions (for the filtering) also has to be chosen, and it seems natural to take these four directions: [0,π/4,π/2,3π/4] that correspond to the four usual directions used for crack classification. The adaptive filtering is applied in each scale, each direction, and then all the results are merged on each scale (mean of the coefficients). The results of this filtering is the initialization of the Markovian segmentation step.Algorithm 1: The studied algorithm for the method based on adaptive filtering and segmentation by Markovian modelling. The step (1) leads to a binary image using adaptive filtering, while step (2) refines this result with a Markovian modelling.
InputRoad imagesInitializationNumber of scales and anglesSteps(1)For each scale do(1a)For each direction doEstimate Adaptive Filter (AF)(1b) Merge AF in all the directions(2)For each scale do(2a) Initialization of the sites (Markov)(2b)While not (stop condition) doUpdating of the sites(3) Fusion of the results on each scale


#### 3.2.2. Adaptive Filtering

Some details are provided in order to realize step (1a) and (1b) in Algorithm1. The ψ∈ℒ2(ℝ2)7 function is a wavelet if(1)∫R2|Ψ(x)|2‖x‖2dx<∞,withx=(i,j),
where Ψ is the Fourier transform of ψ. Equation (1) induces that ∫ℝ2ψ(x)dx=0. The wavelet family is defined for each scale s and for each position u, by(2)ψ(x)=12ψ(Rθ((x-u)s)),
where Rθ is a rotation of angle θ.One of the main difficulties for applying a wavelet decomposition is the choice of the mother waveletψ. Numerous functions are used in the literature: the Haar wavelet, the Gaussian derivatives, the Mexican hat filter, the Morlet wavelet. It is very hard to determine which one is the best for a given application. In the case of crack detection, two elements are present: the crack (if there is a crack) and the background (the road surface can be viewed as a repetitive texture). The goal of the crack detection is to recognize a signal (its shape is known up to a factor) mixed with a noise whose characteristics are known. Consequently, adaptive filtering is well designed for the problem: extracting singularities in coefficients estimated by a wavelet transform. If s is a discrete and deterministic signal with s=(s1⋯sN), N the number of samples, and z=(z1⋯zN) is a noisy observation of s, b is supposed to be an additive noise: z=s+b. The main hypothesis is that this second-order noise is centered and stationary, with the autocorrelation function ϕbb of terms ϕbb(i,j)=ϕbb|i-j|, independent of the signal s. The adaptive filter h of s is defined by(3)h=ϕbb-1s.
The crack signal depends on the definition of the crack. In this paper, like in most of the papers of this domain, crack pixels correspond to black pixels surrounded by background pixels (road pixels). This is why, in [46], a crack is a piecewise constant function f, defined for each position x∈ℝ by:(4)f(x)={-aIfx∈[-T2,T2],0,elsewhere,
where the factor a and the threshold T have to be determined. It does not correspond to a realistic representation of the crack. Because of subsampling, lights, and orientation of the camera, the signal is more like a Gaussian function with zero mean(5)f(x)=-ae(-1/2)(x/σ)2,
where a is the size of the crack and depends on σ, the deviation of the Gaussian law, that is, a=1/σ(2π). Consequently, the term σ allows to fix the width of the crack (like threshold T in (4)). Finally, for step (1), h is estimated for each of the 5 scales, as explained in the beginning of Section 3.2.1, and ϕbb is interpolated in order to have the same size. Then the filter is rotated in order to cover the 4 orientations.


#### 3.2.3. Segmentation

The goal of this part is to extract shapes, that is, cracks, using the detection maps estimated at the first stage of the algorithm (step (2a) of Algorithm1). For the first step of segmentation (initialization), the sites are of size 3×3, consequently, a regular grid is considered in the image. In [46], four configurations are possible and represented in Figure 3 (the part inside the rectangle with low gray levels). The initialization of the sites is based on the configuration that maximizes the coefficients obtained with the adaptive filtering. More formally, if we denoted γ2,0, γ2,π/4, γ2,π/2, and γ2,3π/4, the four configurations, the best configuration γbest is:(6)γbest=argmax⁡α∈[0,…,3π/4]m2,α,
where m2,α is the mean of the coefficients on the considered configuration γ2,α. These four configurations do not represent all the possibilities and are not realistic configurations. In fact, all these four configurations are centered, whereas it is possible to have some noncentered configurations. Consequently, we use the set of sixteen configurations illustrated in Figure 3 (all the presented sites). By modifying the number of configurations, we need to adapt the initialization of sites, and (6) becomes(7)γbest=argmax⁡i∈[0;4],α∈[0,…,3π/4]mi,α,
where mi,α is the mean of the coefficients on the considered configuration γi,α.Figure 3
The sixteen configurations in order to improve the modeling of sites. The four initial configurations proposed in [46] are in the bold rectangle, the sites are represented by the clearer gray levels, and, for the proposed configurations, the sites are represented by the darker gray levels.The image is considered as a finite set ofsites denoted 𝒮={s1,…,sN}. For each site, the neighborhood is defined by: 𝒱s={s′∣s∉𝒱s′&s′∈𝒱s⇒s∈𝒱s′}. A cliquec is defined as a subset of sites in 𝒮 whose every pair of distinct sites are neighbors.These random fields are considered as follows.(1)
The observation fieldY={ys} with s∈𝒮. Here, ys is the mean of the coefficients on the site.(2)
The descriptor fieldL={ls} with s∈𝒮. If there is a crack ls=1 elsewhere ls=0.At each iteration, a global cost, or a sum of potentials that depends on the values of the sites and the links between neighborhoods, is updated. This global cost takes into account the coefficients of the sites (computed from the coefficients estimated during the first part of the algorithm: adaptive filtering) and the configurations of each site and its neighbor sites (the 8 neighbors). More formally, the global cost is the sum of all the potential functions of the sites. This potential function contains two terms as follows.(8)us=α1u1(s)+(1-α1)∑s′∈Vsu2(s,s′).
The first term, u1, corresponds to the data term, and it evaluates how a site is similar to a crack from a photometric point of view (hypotheses Hp1 and Hp2). This term is based on the results given by the adaptive filtering. The second term, u2, represents the constraintsinduced by the neighbors of the site. More precisely, it estimates the consistency between a site and each neighbor site, and it takes into account the geometric hypotheses Hg2 and Hg3. The choice of the value α1 depends on the importance of each part of (8), and it will be discussed in Section 5.1.1.The functionu1 is given by, (9)u1(ys,ls=1)={eξ1(k-ys)2Ifys≥k,1elsewhere,u1(ys,ls=0)={eξ2(ys-k)2Ifys<k,1elsewhere.
The parameters ξ1, ξ2, and k have to be fixed.8 For the definition of u2, we have to determine the number of cliques. In [46], 4 cliques are possible and the 8-connexity is considered. The potential function proposed in the precedent work only considers the difference of orientations between two neighborhoods and not the position between the two sites of the clique, see Table 4. Some cases are not penalized with the old configuration. For example, these two unfavorable cases are not penalized as follows:      (i)
two sites with the same orientation but with no connection between them,(ii)
two sites with the same orientation, but their position makes them parallel.Table 4
The functionu2 used in [46]. This table presents the values u2(s′,s) for the sites in low gray levels in Figure 3. In our experiments, like the authors, we have chosen β1=-2, β2=-1, and β3=2.
γ2,0γ2,π/4γ2,π/2γ2,3π/4γ2,0β1β2β3β2γ2,π/4β2β1β2β3γ2,π/2β3β2β1β2γ2,3π/4β2β3β2β1This is why, with the sixteen configurations that are presented in Figure3, the potential has to take into account the differences of orientations between two sites (there are 16×16 possibilities) and the position of the two sites (there is 8 possibilities because we consider the 8 neighbors). Consequently, the new potential function u2 follows these two important rules.  (R1)
The lower the difference of orientations between two sites, the lower the potential.(R2)
The lower the distance between two sites, the lower the potential (in this case, the distance means the minimal distance between the extremities of the two segments).More formally, if(i)
d denotes the Euclidean distance between the two closest extremities of the sites, with d∈[0,dmax],9(ii)
θ1 and θ2 are the orientations of, respectively, s={pi}i=1⋯Ns, and s′={pi′}i=1⋯Ns′ where, pi, respectively pi′, is the pixel i of the Ns, respectively Ns′, pixels that composes the site s, respectively s′,(iii)
θe is the angle between the two sites; the u2 function is defined by
(10)u2(s′,s)=α2(|2θe-θ1-θ2|2π)+(1-α2)(J(NbC)min⁡i,j(d(si,sj′))dmax⁡-NbC3),
where NbC indicates the number of connected pixels between the two sites s and s′, and J(x) equals 1 if x=0 and 0 elsewhere. The first term is induced by the rule about the orientations, (R1). This term equals zero when the sites have the same orientation, and this orientation is the same as the orientation between the sites, that is, θe=θ2=θ1. This first term penalizes the configurations where the sites do not have the same orientation but also the particular case where they are parallel, see example (a) in Figure 4. The second term and the third term express the rule (R2) about the distances. Two aspects have to be distinguished: the number of connected pixels, when the sites are connected, and, on the contrary, that is, when the sites are not connected, the distance between the sites. It allows to give low influence at disconnected sites and also to increase the cost of sites that are parallel but connected, see example (b) in Figure 4. To study the influence of all these terms, the equation has been normalized, and the different terms have been weighted (using α2, the choices for α2 will be discussed in Section 5.1.1).Examples of the functionu2. These two examples of sites with their respective neighbors show the behavior of the potential u2 with the two considered aspects: orientation and distance. In example (a), with the help of the orientation term, the configuration s3 is penalized and, s2 is less penalized than s3. In example (b), with the help of the two terms about the distance, the site s3 is penalized, compared to s1. On the contrary, the particular case of s2 is favorable, and it compensates the penalty given by the orientations.
(a)(b)


## 3.1. Morphological Method (Morph)

The chosen approach is based on hypothesesHp1, Hg1, and Hg3, and it follows these steps.      (1)
Preprocessing of the images: to reduce the influence of the texture and to increase the contrast between the road pavement and the crack.(2)
Binarization by thresholding (the threshold is different in the various variant, and a local threshold can be used).(3)
Refinement by closing.(4)
Segmentation with shape analysis.(5)
Extraction of the characteristics of the cracks.For step (1), three variants are developed, based on the combination of these local tools: an erosion in gray levels, a conditional median filtering, a histogram equalization, a mean filtering (these preprocessings are detailed in Section5.1). The step (4) is realised in two stages. First, a labeling by analysis of the connected components is realized. Second, the size and the shape of each component is determined in order to remove components which have a shape that is not similar to a crack; the shape of a crack has to be a thin object. It induces constraints on the width w and the height h of the component. More precisely, from an expert point of view, a crack is not significant if h<50 cm, but we can suppose that we manage to detect only a small part of the crack, and this constraint becomes h<7.5 mm. Moreover, the mean width and the maximal width of the crack have to respect these rules: wmin<3 mm and wmax<6.5 mm. All these thresholds are empirically set. In Figure 2, we illustrate the kind of results obtained at each step for the 3 variants. The next step of the method Morph merges the 3 results (with a weighted sum, and the weights are chosen with a learning phase). The final stage refines the result by computing the closing in gray levels of the fusion result.Figure 2
The different steps of the methodMorph. The conditional filtering is applied when the gray level is higher than 40 (to prevent the removal of the crack). The last step is proposed in order to reduce false detections and to complete the detection.


## 3.2. Adaptive Filtering and Markovian Modelling (GaMM)

More recently, our work focused on the field of wavelet decomposition. As it is difficult to chose the mother wavelet5 well adapted to the detection of road cracks, the adaptive filter theory seems convenient, and, in particular, it allows to build a mother wavelet adapted to our task. We present the first step of the algorithm based on adaptive filtering (hypotheses Hp1 and Hg3) and the second stage on Markovian segmentation that can take into account the particular geometry of the crack (Hg2 and Hg3).


### 3.2.1. Algorithm

The goal of this algorithm, presented in Algorithm1, is to obtain, step (1), a binarization (black pixels for the background and white pixels for the cracks) and a refinement of this detection by using a Markovian segmentation, step (2). Using adaptive filtering is important in order to allow the detection of the crack with nonconstant width (hypothesis Hg3).6 The number of scales for the adaptive filtering has to be chosen and depends on the resolution of the image. By supposing a resolution of 1 mm per pixel, by choosing 5 scales, a crack with a width from 2 mm to 1 cm can be detected. Moreover, the number of directions (for the filtering) also has to be chosen, and it seems natural to take these four directions: [0,π/4,π/2,3π/4] that correspond to the four usual directions used for crack classification. The adaptive filtering is applied in each scale, each direction, and then all the results are merged on each scale (mean of the coefficients). The results of this filtering is the initialization of the Markovian segmentation step.Algorithm 1: The studied algorithm for the method based on adaptive filtering and segmentation by Markovian modelling. The step (1) leads to a binary image using adaptive filtering, while step (2) refines this result with a Markovian modelling.
InputRoad imagesInitializationNumber of scales and anglesSteps(1)For each scale do(1a)For each direction doEstimate Adaptive Filter (AF)(1b) Merge AF in all the directions(2)For each scale do(2a) Initialization of the sites (Markov)(2b)While not (stop condition) doUpdating of the sites(3) Fusion of the results on each scale


### 3.2.2. Adaptive Filtering

Some details are provided in order to realize step (1a) and (1b) in Algorithm1. The ψ∈ℒ2(ℝ2)7 function is a wavelet if(1)∫R2|Ψ(x)|2‖x‖2dx<∞,withx=(i,j),
where Ψ is the Fourier transform of ψ. Equation (1) induces that ∫ℝ2ψ(x)dx=0. The wavelet family is defined for each scale s and for each position u, by(2)ψ(x)=12ψ(Rθ((x-u)s)),
where Rθ is a rotation of angle θ.One of the main difficulties for applying a wavelet decomposition is the choice of the mother waveletψ. Numerous functions are used in the literature: the Haar wavelet, the Gaussian derivatives, the Mexican hat filter, the Morlet wavelet. It is very hard to determine which one is the best for a given application. In the case of crack detection, two elements are present: the crack (if there is a crack) and the background (the road surface can be viewed as a repetitive texture). The goal of the crack detection is to recognize a signal (its shape is known up to a factor) mixed with a noise whose characteristics are known. Consequently, adaptive filtering is well designed for the problem: extracting singularities in coefficients estimated by a wavelet transform. If s is a discrete and deterministic signal with s=(s1⋯sN), N the number of samples, and z=(z1⋯zN) is a noisy observation of s, b is supposed to be an additive noise: z=s+b. The main hypothesis is that this second-order noise is centered and stationary, with the autocorrelation function ϕbb of terms ϕbb(i,j)=ϕbb|i-j|, independent of the signal s. The adaptive filter h of s is defined by(3)h=ϕbb-1s.
The crack signal depends on the definition of the crack. In this paper, like in most of the papers of this domain, crack pixels correspond to black pixels surrounded by background pixels (road pixels). This is why, in [46], a crack is a piecewise constant function f, defined for each position x∈ℝ by:(4)f(x)={-aIfx∈[-T2,T2],0,elsewhere,
where the factor a and the threshold T have to be determined. It does not correspond to a realistic representation of the crack. Because of subsampling, lights, and orientation of the camera, the signal is more like a Gaussian function with zero mean(5)f(x)=-ae(-1/2)(x/σ)2,
where a is the size of the crack and depends on σ, the deviation of the Gaussian law, that is, a=1/σ(2π). Consequently, the term σ allows to fix the width of the crack (like threshold T in (4)). Finally, for step (1), h is estimated for each of the 5 scales, as explained in the beginning of Section 3.2.1, and ϕbb is interpolated in order to have the same size. Then the filter is rotated in order to cover the 4 orientations.


### 3.2.3. Segmentation

The goal of this part is to extract shapes, that is, cracks, using the detection maps estimated at the first stage of the algorithm (step (2a) of Algorithm1). For the first step of segmentation (initialization), the sites are of size 3×3, consequently, a regular grid is considered in the image. In [46], four configurations are possible and represented in Figure 3 (the part inside the rectangle with low gray levels). The initialization of the sites is based on the configuration that maximizes the coefficients obtained with the adaptive filtering. More formally, if we denoted γ2,0, γ2,π/4, γ2,π/2, and γ2,3π/4, the four configurations, the best configuration γbest is:(6)γbest=argmax⁡α∈[0,…,3π/4]m2,α,
where m2,α is the mean of the coefficients on the considered configuration γ2,α. These four configurations do not represent all the possibilities and are not realistic configurations. In fact, all these four configurations are centered, whereas it is possible to have some noncentered configurations. Consequently, we use the set of sixteen configurations illustrated in Figure 3 (all the presented sites). By modifying the number of configurations, we need to adapt the initialization of sites, and (6) becomes(7)γbest=argmax⁡i∈[0;4],α∈[0,…,3π/4]mi,α,
where mi,α is the mean of the coefficients on the considered configuration γi,α.Figure 3
The sixteen configurations in order to improve the modeling of sites. The four initial configurations proposed in [46] are in the bold rectangle, the sites are represented by the clearer gray levels, and, for the proposed configurations, the sites are represented by the darker gray levels.The image is considered as a finite set ofsites denoted 𝒮={s1,…,sN}. For each site, the neighborhood is defined by: 𝒱s={s′∣s∉𝒱s′&s′∈𝒱s⇒s∈𝒱s′}. A cliquec is defined as a subset of sites in 𝒮 whose every pair of distinct sites are neighbors.These random fields are considered as follows.(1)
The observation fieldY={ys} with s∈𝒮. Here, ys is the mean of the coefficients on the site.(2)
The descriptor fieldL={ls} with s∈𝒮. If there is a crack ls=1 elsewhere ls=0.At each iteration, a global cost, or a sum of potentials that depends on the values of the sites and the links between neighborhoods, is updated. This global cost takes into account the coefficients of the sites (computed from the coefficients estimated during the first part of the algorithm: adaptive filtering) and the configurations of each site and its neighbor sites (the 8 neighbors). More formally, the global cost is the sum of all the potential functions of the sites. This potential function contains two terms as follows.(8)us=α1u1(s)+(1-α1)∑s′∈Vsu2(s,s′).
The first term, u1, corresponds to the data term, and it evaluates how a site is similar to a crack from a photometric point of view (hypotheses Hp1 and Hp2). This term is based on the results given by the adaptive filtering. The second term, u2, represents the constraintsinduced by the neighbors of the site. More precisely, it estimates the consistency between a site and each neighbor site, and it takes into account the geometric hypotheses Hg2 and Hg3. The choice of the value α1 depends on the importance of each part of (8), and it will be discussed in Section 5.1.1.The functionu1 is given by, (9)u1(ys,ls=1)={eξ1(k-ys)2Ifys≥k,1elsewhere,u1(ys,ls=0)={eξ2(ys-k)2Ifys<k,1elsewhere.
The parameters ξ1, ξ2, and k have to be fixed.8 For the definition of u2, we have to determine the number of cliques. In [46], 4 cliques are possible and the 8-connexity is considered. The potential function proposed in the precedent work only considers the difference of orientations between two neighborhoods and not the position between the two sites of the clique, see Table 4. Some cases are not penalized with the old configuration. For example, these two unfavorable cases are not penalized as follows:      (i)
two sites with the same orientation but with no connection between them,(ii)
two sites with the same orientation, but their position makes them parallel.Table 4
The functionu2 used in [46]. This table presents the values u2(s′,s) for the sites in low gray levels in Figure 3. In our experiments, like the authors, we have chosen β1=-2, β2=-1, and β3=2.
γ2,0γ2,π/4γ2,π/2γ2,3π/4γ2,0β1β2β3β2γ2,π/4β2β1β2β3γ2,π/2β3β2β1β2γ2,3π/4β2β3β2β1This is why, with the sixteen configurations that are presented in Figure3, the potential has to take into account the differences of orientations between two sites (there are 16×16 possibilities) and the position of the two sites (there is 8 possibilities because we consider the 8 neighbors). Consequently, the new potential function u2 follows these two important rules.  (R1)
The lower the difference of orientations between two sites, the lower the potential.(R2)
The lower the distance between two sites, the lower the potential (in this case, the distance means the minimal distance between the extremities of the two segments).More formally, if(i)
d denotes the Euclidean distance between the two closest extremities of the sites, with d∈[0,dmax],9(ii)
θ1 and θ2 are the orientations of, respectively, s={pi}i=1⋯Ns, and s′={pi′}i=1⋯Ns′ where, pi, respectively pi′, is the pixel i of the Ns, respectively Ns′, pixels that composes the site s, respectively s′,(iii)
θe is the angle between the two sites; the u2 function is defined by
(10)u2(s′,s)=α2(|2θe-θ1-θ2|2π)+(1-α2)(J(NbC)min⁡i,j(d(si,sj′))dmax⁡-NbC3),
where NbC indicates the number of connected pixels between the two sites s and s′, and J(x) equals 1 if x=0 and 0 elsewhere. The first term is induced by the rule about the orientations, (R1). This term equals zero when the sites have the same orientation, and this orientation is the same as the orientation between the sites, that is, θe=θ2=θ1. This first term penalizes the configurations where the sites do not have the same orientation but also the particular case where they are parallel, see example (a) in Figure 4. The second term and the third term express the rule (R2) about the distances. Two aspects have to be distinguished: the number of connected pixels, when the sites are connected, and, on the contrary, that is, when the sites are not connected, the distance between the sites. It allows to give low influence at disconnected sites and also to increase the cost of sites that are parallel but connected, see example (b) in Figure 4. To study the influence of all these terms, the equation has been normalized, and the different terms have been weighted (using α2, the choices for α2 will be discussed in Section 5.1.1).Examples of the functionu2. These two examples of sites with their respective neighbors show the behavior of the potential u2 with the two considered aspects: orientation and distance. In example (a), with the help of the orientation term, the configuration s3 is penalized and, s2 is less penalized than s3. In example (b), with the help of the two terms about the distance, the site s3 is penalized, compared to s1. On the contrary, the particular case of s2 is favorable, and it compensates the penalty given by the orientations.
(a)(b)


## 3.2.1. Algorithm

The goal of this algorithm, presented in Algorithm1, is to obtain, step (1), a binarization (black pixels for the background and white pixels for the cracks) and a refinement of this detection by using a Markovian segmentation, step (2). Using adaptive filtering is important in order to allow the detection of the crack with nonconstant width (hypothesis Hg3).6 The number of scales for the adaptive filtering has to be chosen and depends on the resolution of the image. By supposing a resolution of 1 mm per pixel, by choosing 5 scales, a crack with a width from 2 mm to 1 cm can be detected. Moreover, the number of directions (for the filtering) also has to be chosen, and it seems natural to take these four directions: [0,π/4,π/2,3π/4] that correspond to the four usual directions used for crack classification. The adaptive filtering is applied in each scale, each direction, and then all the results are merged on each scale (mean of the coefficients). The results of this filtering is the initialization of the Markovian segmentation step.Algorithm 1: The studied algorithm for the method based on adaptive filtering and segmentation by Markovian modelling. The step (1) leads to a binary image using adaptive filtering, while step (2) refines this result with a Markovian modelling.
InputRoad imagesInitializationNumber of scales and anglesSteps(1)For each scale do(1a)For each direction doEstimate Adaptive Filter (AF)(1b) Merge AF in all the directions(2)For each scale do(2a) Initialization of the sites (Markov)(2b)While not (stop condition) doUpdating of the sites(3) Fusion of the results on each scale


## 3.2.2. Adaptive Filtering

Some details are provided in order to realize step (1a) and (1b) in Algorithm1. The ψ∈ℒ2(ℝ2)7 function is a wavelet if(1)∫R2|Ψ(x)|2‖x‖2dx<∞,withx=(i,j),
where Ψ is the Fourier transform of ψ. Equation (1) induces that ∫ℝ2ψ(x)dx=0. The wavelet family is defined for each scale s and for each position u, by(2)ψ(x)=12ψ(Rθ((x-u)s)),
where Rθ is a rotation of angle θ.One of the main difficulties for applying a wavelet decomposition is the choice of the mother waveletψ. Numerous functions are used in the literature: the Haar wavelet, the Gaussian derivatives, the Mexican hat filter, the Morlet wavelet. It is very hard to determine which one is the best for a given application. In the case of crack detection, two elements are present: the crack (if there is a crack) and the background (the road surface can be viewed as a repetitive texture). The goal of the crack detection is to recognize a signal (its shape is known up to a factor) mixed with a noise whose characteristics are known. Consequently, adaptive filtering is well designed for the problem: extracting singularities in coefficients estimated by a wavelet transform. If s is a discrete and deterministic signal with s=(s1⋯sN), N the number of samples, and z=(z1⋯zN) is a noisy observation of s, b is supposed to be an additive noise: z=s+b. The main hypothesis is that this second-order noise is centered and stationary, with the autocorrelation function ϕbb of terms ϕbb(i,j)=ϕbb|i-j|, independent of the signal s. The adaptive filter h of s is defined by(3)h=ϕbb-1s.
The crack signal depends on the definition of the crack. In this paper, like in most of the papers of this domain, crack pixels correspond to black pixels surrounded by background pixels (road pixels). This is why, in [46], a crack is a piecewise constant function f, defined for each position x∈ℝ by:(4)f(x)={-aIfx∈[-T2,T2],0,elsewhere,
where the factor a and the threshold T have to be determined. It does not correspond to a realistic representation of the crack. Because of subsampling, lights, and orientation of the camera, the signal is more like a Gaussian function with zero mean(5)f(x)=-ae(-1/2)(x/σ)2,
where a is the size of the crack and depends on σ, the deviation of the Gaussian law, that is, a=1/σ(2π). Consequently, the term σ allows to fix the width of the crack (like threshold T in (4)). Finally, for step (1), h is estimated for each of the 5 scales, as explained in the beginning of Section 3.2.1, and ϕbb is interpolated in order to have the same size. Then the filter is rotated in order to cover the 4 orientations.


## 3.2.3. Segmentation

The goal of this part is to extract shapes, that is, cracks, using the detection maps estimated at the first stage of the algorithm (step (2a) of Algorithm1). For the first step of segmentation (initialization), the sites are of size 3×3, consequently, a regular grid is considered in the image. In [46], four configurations are possible and represented in Figure 3 (the part inside the rectangle with low gray levels). The initialization of the sites is based on the configuration that maximizes the coefficients obtained with the adaptive filtering. More formally, if we denoted γ2,0, γ2,π/4, γ2,π/2, and γ2,3π/4, the four configurations, the best configuration γbest is:(6)γbest=argmax⁡α∈[0,…,3π/4]m2,α,
where m2,α is the mean of the coefficients on the considered configuration γ2,α. These four configurations do not represent all the possibilities and are not realistic configurations. In fact, all these four configurations are centered, whereas it is possible to have some noncentered configurations. Consequently, we use the set of sixteen configurations illustrated in Figure 3 (all the presented sites). By modifying the number of configurations, we need to adapt the initialization of sites, and (6) becomes(7)γbest=argmax⁡i∈[0;4],α∈[0,…,3π/4]mi,α,
where mi,α is the mean of the coefficients on the considered configuration γi,α.Figure 3
The sixteen configurations in order to improve the modeling of sites. The four initial configurations proposed in [46] are in the bold rectangle, the sites are represented by the clearer gray levels, and, for the proposed configurations, the sites are represented by the darker gray levels.The image is considered as a finite set ofsites denoted 𝒮={s1,…,sN}. For each site, the neighborhood is defined by: 𝒱s={s′∣s∉𝒱s′&s′∈𝒱s⇒s∈𝒱s′}. A cliquec is defined as a subset of sites in 𝒮 whose every pair of distinct sites are neighbors.These random fields are considered as follows.(1)
The observation fieldY={ys} with s∈𝒮. Here, ys is the mean of the coefficients on the site.(2)
The descriptor fieldL={ls} with s∈𝒮. If there is a crack ls=1 elsewhere ls=0.At each iteration, a global cost, or a sum of potentials that depends on the values of the sites and the links between neighborhoods, is updated. This global cost takes into account the coefficients of the sites (computed from the coefficients estimated during the first part of the algorithm: adaptive filtering) and the configurations of each site and its neighbor sites (the 8 neighbors). More formally, the global cost is the sum of all the potential functions of the sites. This potential function contains two terms as follows.(8)us=α1u1(s)+(1-α1)∑s′∈Vsu2(s,s′).
The first term, u1, corresponds to the data term, and it evaluates how a site is similar to a crack from a photometric point of view (hypotheses Hp1 and Hp2). This term is based on the results given by the adaptive filtering. The second term, u2, represents the constraintsinduced by the neighbors of the site. More precisely, it estimates the consistency between a site and each neighbor site, and it takes into account the geometric hypotheses Hg2 and Hg3. The choice of the value α1 depends on the importance of each part of (8), and it will be discussed in Section 5.1.1.The functionu1 is given by, (9)u1(ys,ls=1)={eξ1(k-ys)2Ifys≥k,1elsewhere,u1(ys,ls=0)={eξ2(ys-k)2Ifys<k,1elsewhere.
The parameters ξ1, ξ2, and k have to be fixed.8 For the definition of u2, we have to determine the number of cliques. In [46], 4 cliques are possible and the 8-connexity is considered. The potential function proposed in the precedent work only considers the difference of orientations between two neighborhoods and not the position between the two sites of the clique, see Table 4. Some cases are not penalized with the old configuration. For example, these two unfavorable cases are not penalized as follows:      (i)
two sites with the same orientation but with no connection between them,(ii)
two sites with the same orientation, but their position makes them parallel.Table 4
The functionu2 used in [46]. This table presents the values u2(s′,s) for the sites in low gray levels in Figure 3. In our experiments, like the authors, we have chosen β1=-2, β2=-1, and β3=2.
γ2,0γ2,π/4γ2,π/2γ2,3π/4γ2,0β1β2β3β2γ2,π/4β2β1β2β3γ2,π/2β3β2β1β2γ2,3π/4β2β3β2β1This is why, with the sixteen configurations that are presented in Figure3, the potential has to take into account the differences of orientations between two sites (there are 16×16 possibilities) and the position of the two sites (there is 8 possibilities because we consider the 8 neighbors). Consequently, the new potential function u2 follows these two important rules.  (R1)
The lower the difference of orientations between two sites, the lower the potential.(R2)
The lower the distance between two sites, the lower the potential (in this case, the distance means the minimal distance between the extremities of the two segments).More formally, if(i)
d denotes the Euclidean distance between the two closest extremities of the sites, with d∈[0,dmax],9(ii)
θ1 and θ2 are the orientations of, respectively, s={pi}i=1⋯Ns, and s′={pi′}i=1⋯Ns′ where, pi, respectively pi′, is the pixel i of the Ns, respectively Ns′, pixels that composes the site s, respectively s′,(iii)
θe is the angle between the two sites; the u2 function is defined by
(10)u2(s′,s)=α2(|2θe-θ1-θ2|2π)+(1-α2)(J(NbC)min⁡i,j(d(si,sj′))dmax⁡-NbC3),
where NbC indicates the number of connected pixels between the two sites s and s′, and J(x) equals 1 if x=0 and 0 elsewhere. The first term is induced by the rule about the orientations, (R1). This term equals zero when the sites have the same orientation, and this orientation is the same as the orientation between the sites, that is, θe=θ2=θ1. This first term penalizes the configurations where the sites do not have the same orientation but also the particular case where they are parallel, see example (a) in Figure 4. The second term and the third term express the rule (R2) about the distances. Two aspects have to be distinguished: the number of connected pixels, when the sites are connected, and, on the contrary, that is, when the sites are not connected, the distance between the sites. It allows to give low influence at disconnected sites and also to increase the cost of sites that are parallel but connected, see example (b) in Figure 4. To study the influence of all these terms, the equation has been normalized, and the different terms have been weighted (using α2, the choices for α2 will be discussed in Section 5.1.1).Examples of the functionu2. These two examples of sites with their respective neighbors show the behavior of the potential u2 with the two considered aspects: orientation and distance. In example (a), with the help of the orientation term, the configuration s3 is penalized and, s2 is less penalized than s3. In example (b), with the help of the two terms about the distance, the site s3 is penalized, compared to s1. On the contrary, the particular case of s2 is favorable, and it compensates the penalty given by the orientations.
(a)(b)


## 4. Evaluation Protocol

For the evaluation of automatic crack detection by image processing methods, to the best of our knowledge, no evaluation and comparison protocol has been proposed in the community. However, in all the countries, for estimating the quality of the road surface, it is important to know exactly the size and the width of defects, that is, to detect precisely the defect. This is why it seems important to characterize quantitatively the performances of the methods. For building this kind of protocol, it is necessary, first, to choose the tested images, second, to choose how to build reference segmentations, and, third, to determine the criteria used for the quantitative analysis. For estimating the reference segmentations, two approaches can be used(1)
To compute synthetic images with synthetic defects. The exact position of the defects is known, and the reference segmentations can be considered as ground truth.(2)
To propose reliable segmentations of real images. It supposes that we are able to provide a segmentation that is reliable enough to be employed as a reference. For evaluation, these segmentations can be called “pseudoground truth.”The two solutions are studied, and, we explain how the manual segmentations (that are our references) are computed. Before, we briefly describe the acquisition system.


### 4.1. Acquisition

The acquisition system used for the dataset of our experiments is described in Figure5. It contains 4 video cameras with 3 sensors in gray levels in the backside of the car and 1 color in front of the car. The first camera is needed to determine the environment conditions (weather, location, traffic) whereas the three other ones are used for the crack detection. The resolution of this one is smaller than the 3 others, and; moreover, the optical axis is not perpendicular to the road surfaces, on the contrary of the 3 others. The 3 cameras have been physically synchronized directly during the acquisition. To be independent of the illumination problems, nine stroboscopic lights have been added. The position of the lights is perpendicular to the road plane, and they are distant from the surface of 1 meter. The light power has been chosen in order to not deteriorate the visualisation of the road pavement and the defects.The acquisition system used for the evaluation. In (a), the acquisition system is illustrated whereas in (b) it shows the user interface. In (c, d), an example of the final images is given. In (d), we can see the road that is visible in (c). The processing is done 1 meter by 1 meter, that is, independently on each image presented in (d). The surface contains two reparations of vertical cracks. In some cases, the 3 sensors do not have the same settings, and the global illuminations are different, so it can generate some “false cracks”. This aspect has been easily taken into account in a preprocessing step by eliminating the junction area in the region of interest.
(a)
System(b)
Human machine interface(c)
Environment images (notused)(d)
Processed images


### 4.2. Reference Images

The most difficult is to propose images with a reference segmentation. On the first hand, we introduce synthetic images with a simulated crack (the size of these images is256×256).10 As shown in Figure 6, the result is not realistic enough. It does not seem realistic because the contrast is too important between the road and the crack. Moreover, the interruptions of the crack, the changes in the direction, the presence of many paths, and so forth, in the default, are not simulated. In order to be more realistic, it seems that we have to design and to implement a complex heuristic to simulate the crack, and it represents too much effort for having only a synthetic default. This is why, on the other hand, we have simulated different defects on real images that previously contain no defect (the size of these images are 768×512 and 1920×480). The result is more realistic, but the shape and the photometric aspect of the cracks (which are randomly chosen) does not seem realistic enough. This is why it appears important to propose a set of real images (size 768×512 and 1920×480) with manual segmentations that are reliable enough to be considered as reference segmentations. To summarize, the two first kinds of images allow to propose an exact evaluation and to illustrate theoretically the behavior of the method whereas the last kind of images allows to validate the work on real images with a pseudoground truth.Figure 6
Examples of the images tested.


### 4.3. Reference Segmentations

For real images, we briefly explain how the manual segmentations are validated. Four experts have manually segmented the images with the same tools11 and in the same conditions. Then, the four segmentations are merged, following these rules.      (1)
A pixel marked as a crack by more than two experts is considered as a crack pixel.(2)
Every pixel marked as a crack and next to a pixel kept by step (1) or (2) is also considered as a crack.The second rule is iterative and stops when no pixel is added. Then, the result is dilated with a squared structuring element of size3×3. To evaluate the reliability of the reference segmentations, we estimate, first, the percentage of covering between each operator and, second, the mean distance, D, between each pixel (detected by only one expert and not kept in the reference image) and the reference segmentation.Table5 shows some results for 5 of the 42 images manually segmented. We have distinguished 5 families: the first one contains images acquired in static whereas the four other ones are acquired in dynamic. Moreover, we have 4 different kinds of road pavement acquired in dynamic. The 5 images have been taken in order to show results on each of these families. We can notice that the first 2 images are the most reliable because the mean error is less than 2 pixels. The precision of these results is satisfactory. On the contrary, the last 3 images show the important variabilities between operators and how it is difficult to extract a segmentation for these images and, in particular, in the image 936, where the error is due to a bad interpretation of one of the four operators who finds a defect that does not exist.Table 5
The comparison of the 4 manual segmentations for estimating the final reference segmentations. For each image, we specify: the percentage of pixels in the whole image that are preserved as crack pixels in the final reference segmentation (F), the percentage of recovering between 2, 3 and 4 manual segmentations and the sum of these 3 percentages (S). For all the crack pixels that are not preserved in the final reference segmentation, the mean distance to this segmentation is given (D).
ImagesF (%)2 (%)3 (%)4 (%)S (%)D (pix)420.426.6914.594.245.481.454630.1723.465.950.3929.81.49360.4123.527.410.931.837.05410.3322.647.311.3331.283.56881.4422.748.231.2332.22.76By analyzing the results for the criterionD, presented in Table 5, we can classify the 42 tested images in 3 categories, that is, images with the following.      (1)
Areliable segmentation: the criterion D<Tr. It means that all the operators have built segmentations that are quite near to each other.(2)
A segmentation that ismoderately reliable: the criterion Tr≤D<Ta. It means that some parts of the crack are not easy to segment, and there are local errors.(3)
Anambiguous segmentation: the criterion D≥Tr. It clearly shows that the images are difficult to segment, and, in most of the cases, it means that some parts are detected as a crack whereas they are not and reversely.The threshold have been empirically chosen:Tr=2 and Ta=4. In Figure 7, first, we present the mean distance, Di, i∈{1,2,3,4}, between the final reference segmentation, Sr, and each manual segmentation, Si (obtained with each operator), and, second, the criterion D for each real image of our protocol. The first graph illustrates how it is important to combine the four manual segmentations instead of using only one manual segmentation. Indeed, we can notice that each operator, alternately, gives an interpretation that is different from the three others. The second graph explains how the thresholds are chosen for determining the detections that are “accepted” for the evaluation, see Section 4.4 for explanations about accepted detections.The variations between each manual segmentations that are used for building pseudoground truth. The first graph represents for each operator (one curve for one operator) and each image (x-coordinates), cf. Figure 16 for the corresponding images, the mean distance, Di, i∈{1,2,3,4}. The second graph presents D, cf. Section 4.3 and Table 5. This graph allows us to distinguish the different categories of images (red axes): reliable (D<Tr), moderately reliable (Tr≤D<Ta), ambiguous (Ta≤D). In the two graphs, the purple axes represent the five different samples of images (each sample corresponds to a kind of road pavement). The first ones were acquired with a static system whereas the four others were acquired with a dynamic system.
(a)(b)The three categories of reference segmentations are illustrated in Figure8. Overall, the four segmentations are near to each other, and, if the segmentations are combined, it permits to detect the width of the crack. However, these examples also present some difficulties of the crack segmentation: areas where the cracks are less visible and regions where the texture elements that have the same size and/or the same gray levels as the crack pixels. Thus, in some cases, one operator extends the crack or gives a different shape. In some extreme cases, the operator can even confound a crack and another object of the scene (a piece of wood, e.g.). In another way, these examples highlight the interest in combining different segmentations in order to obtain reference segmentations as reliable as possible.The levels of difficulties of the tested images. On these images, we present the four manual segmentations. The codes are red (light and dark), blue, green, for each of the four operators. The parts in yellow correspond to the parts of the cracks detected by more than one operator. There are two examples per category of segmentation. To better visualize, only a part of the images is shown. In the ambiguous images, we clearly see the mistake of the operator in dark red.


### 4.4. Criteria of Efficiency

In this section, we introduce how the reference segmentation and the estimated segmentation are compared. In Figure9, we present common evaluation criteria that are used for segmentation evaluation:      (1)
the percentage of correct detections (true positives) (TP),(2)
the percentage of false positives (FP),(3)
the percentage of false negatives (FN), and(4)
the similarity coefficient (DICE).Figure 9
The evaluation criteria. In this figure, two segmentations of the same crack are represented the black line corresponds to a manual (or reference) segmentation, and the red line is an estimated segmentation. The goal is to evaluate the quality of the estimated segmentation that corresponds to the positives (P). All the nonselected pixels that do not correspond to the crack are called the true negatives (TN). The piece of line with the two colors (red and black) are the correct detections or true positives (TP). In this table, different criteria are introduced, but, in this work, we have used the DICE because this coefficient well represents what we want to measure: the quality of the detection against the percentage of the crack that is detected, in order to determine how to reduce false detections whilst increasing the density.This last criterion seems to be the most significant because it evaluates the ratio between the FP and the FN, and it resumes the results of all the criteria. Moreover, it directly expresses what is important to evaluate: how the method can reduce errors of detection whilst increasing the density of good detections.For real images, the detections that are “accepted” have been added in order to tolerate a small error on the localization of crack pixels. This criterion is needed because perfect detection seems, for the moment, difficult to reach, see the results in Table5. In consequence, these accepted pixels have been included in the estimation of the similarity coefficient or DICE. The threshold for accepted detections equals 0 for synthetic images whereas it depends on the mean distances, see D in Table 5, for the real images.


## 4.1. Acquisition

The acquisition system used for the dataset of our experiments is described in Figure5. It contains 4 video cameras with 3 sensors in gray levels in the backside of the car and 1 color in front of the car. The first camera is needed to determine the environment conditions (weather, location, traffic) whereas the three other ones are used for the crack detection. The resolution of this one is smaller than the 3 others, and; moreover, the optical axis is not perpendicular to the road surfaces, on the contrary of the 3 others. The 3 cameras have been physically synchronized directly during the acquisition. To be independent of the illumination problems, nine stroboscopic lights have been added. The position of the lights is perpendicular to the road plane, and they are distant from the surface of 1 meter. The light power has been chosen in order to not deteriorate the visualisation of the road pavement and the defects.The acquisition system used for the evaluation. In (a), the acquisition system is illustrated whereas in (b) it shows the user interface. In (c, d), an example of the final images is given. In (d), we can see the road that is visible in (c). The processing is done 1 meter by 1 meter, that is, independently on each image presented in (d). The surface contains two reparations of vertical cracks. In some cases, the 3 sensors do not have the same settings, and the global illuminations are different, so it can generate some “false cracks”. This aspect has been easily taken into account in a preprocessing step by eliminating the junction area in the region of interest.
(a)
System(b)
Human machine interface(c)
Environment images (notused)(d)
Processed images


## 4.2. Reference Images

The most difficult is to propose images with a reference segmentation. On the first hand, we introduce synthetic images with a simulated crack (the size of these images is256×256).10 As shown in Figure 6, the result is not realistic enough. It does not seem realistic because the contrast is too important between the road and the crack. Moreover, the interruptions of the crack, the changes in the direction, the presence of many paths, and so forth, in the default, are not simulated. In order to be more realistic, it seems that we have to design and to implement a complex heuristic to simulate the crack, and it represents too much effort for having only a synthetic default. This is why, on the other hand, we have simulated different defects on real images that previously contain no defect (the size of these images are 768×512 and 1920×480). The result is more realistic, but the shape and the photometric aspect of the cracks (which are randomly chosen) does not seem realistic enough. This is why it appears important to propose a set of real images (size 768×512 and 1920×480) with manual segmentations that are reliable enough to be considered as reference segmentations. To summarize, the two first kinds of images allow to propose an exact evaluation and to illustrate theoretically the behavior of the method whereas the last kind of images allows to validate the work on real images with a pseudoground truth.Figure 6
Examples of the images tested.


## 4.3. Reference Segmentations

For real images, we briefly explain how the manual segmentations are validated. Four experts have manually segmented the images with the same tools11 and in the same conditions. Then, the four segmentations are merged, following these rules.      (1)
A pixel marked as a crack by more than two experts is considered as a crack pixel.(2)
Every pixel marked as a crack and next to a pixel kept by step (1) or (2) is also considered as a crack.The second rule is iterative and stops when no pixel is added. Then, the result is dilated with a squared structuring element of size3×3. To evaluate the reliability of the reference segmentations, we estimate, first, the percentage of covering between each operator and, second, the mean distance, D, between each pixel (detected by only one expert and not kept in the reference image) and the reference segmentation.Table5 shows some results for 5 of the 42 images manually segmented. We have distinguished 5 families: the first one contains images acquired in static whereas the four other ones are acquired in dynamic. Moreover, we have 4 different kinds of road pavement acquired in dynamic. The 5 images have been taken in order to show results on each of these families. We can notice that the first 2 images are the most reliable because the mean error is less than 2 pixels. The precision of these results is satisfactory. On the contrary, the last 3 images show the important variabilities between operators and how it is difficult to extract a segmentation for these images and, in particular, in the image 936, where the error is due to a bad interpretation of one of the four operators who finds a defect that does not exist.Table 5
The comparison of the 4 manual segmentations for estimating the final reference segmentations. For each image, we specify: the percentage of pixels in the whole image that are preserved as crack pixels in the final reference segmentation (F), the percentage of recovering between 2, 3 and 4 manual segmentations and the sum of these 3 percentages (S). For all the crack pixels that are not preserved in the final reference segmentation, the mean distance to this segmentation is given (D).
ImagesF (%)2 (%)3 (%)4 (%)S (%)D (pix)420.426.6914.594.245.481.454630.1723.465.950.3929.81.49360.4123.527.410.931.837.05410.3322.647.311.3331.283.56881.4422.748.231.2332.22.76By analyzing the results for the criterionD, presented in Table 5, we can classify the 42 tested images in 3 categories, that is, images with the following.      (1)
Areliable segmentation: the criterion D<Tr. It means that all the operators have built segmentations that are quite near to each other.(2)
A segmentation that ismoderately reliable: the criterion Tr≤D<Ta. It means that some parts of the crack are not easy to segment, and there are local errors.(3)
Anambiguous segmentation: the criterion D≥Tr. It clearly shows that the images are difficult to segment, and, in most of the cases, it means that some parts are detected as a crack whereas they are not and reversely.The threshold have been empirically chosen:Tr=2 and Ta=4. In Figure 7, first, we present the mean distance, Di, i∈{1,2,3,4}, between the final reference segmentation, Sr, and each manual segmentation, Si (obtained with each operator), and, second, the criterion D for each real image of our protocol. The first graph illustrates how it is important to combine the four manual segmentations instead of using only one manual segmentation. Indeed, we can notice that each operator, alternately, gives an interpretation that is different from the three others. The second graph explains how the thresholds are chosen for determining the detections that are “accepted” for the evaluation, see Section 4.4 for explanations about accepted detections.The variations between each manual segmentations that are used for building pseudoground truth. The first graph represents for each operator (one curve for one operator) and each image (x-coordinates), cf. Figure 16 for the corresponding images, the mean distance, Di, i∈{1,2,3,4}. The second graph presents D, cf. Section 4.3 and Table 5. This graph allows us to distinguish the different categories of images (red axes): reliable (D<Tr), moderately reliable (Tr≤D<Ta), ambiguous (Ta≤D). In the two graphs, the purple axes represent the five different samples of images (each sample corresponds to a kind of road pavement). The first ones were acquired with a static system whereas the four others were acquired with a dynamic system.
(a)(b)The three categories of reference segmentations are illustrated in Figure8. Overall, the four segmentations are near to each other, and, if the segmentations are combined, it permits to detect the width of the crack. However, these examples also present some difficulties of the crack segmentation: areas where the cracks are less visible and regions where the texture elements that have the same size and/or the same gray levels as the crack pixels. Thus, in some cases, one operator extends the crack or gives a different shape. In some extreme cases, the operator can even confound a crack and another object of the scene (a piece of wood, e.g.). In another way, these examples highlight the interest in combining different segmentations in order to obtain reference segmentations as reliable as possible.The levels of difficulties of the tested images. On these images, we present the four manual segmentations. The codes are red (light and dark), blue, green, for each of the four operators. The parts in yellow correspond to the parts of the cracks detected by more than one operator. There are two examples per category of segmentation. To better visualize, only a part of the images is shown. In the ambiguous images, we clearly see the mistake of the operator in dark red.


## 4.4. Criteria of Efficiency

In this section, we introduce how the reference segmentation and the estimated segmentation are compared. In Figure9, we present common evaluation criteria that are used for segmentation evaluation:      (1)
the percentage of correct detections (true positives) (TP),(2)
the percentage of false positives (FP),(3)
the percentage of false negatives (FN), and(4)
the similarity coefficient (DICE).Figure 9
The evaluation criteria. In this figure, two segmentations of the same crack are represented the black line corresponds to a manual (or reference) segmentation, and the red line is an estimated segmentation. The goal is to evaluate the quality of the estimated segmentation that corresponds to the positives (P). All the nonselected pixels that do not correspond to the crack are called the true negatives (TN). The piece of line with the two colors (red and black) are the correct detections or true positives (TP). In this table, different criteria are introduced, but, in this work, we have used the DICE because this coefficient well represents what we want to measure: the quality of the detection against the percentage of the crack that is detected, in order to determine how to reduce false detections whilst increasing the density.This last criterion seems to be the most significant because it evaluates the ratio between the FP and the FN, and it resumes the results of all the criteria. Moreover, it directly expresses what is important to evaluate: how the method can reduce errors of detection whilst increasing the density of good detections.For real images, the detections that are “accepted” have been added in order to tolerate a small error on the localization of crack pixels. This criterion is needed because perfect detection seems, for the moment, difficult to reach, see the results in Table5. In consequence, these accepted pixels have been included in the estimation of the similarity coefficient or DICE. The threshold for accepted detections equals 0 for synthetic images whereas it depends on the mean distances, see D in Table 5, for the real images.


## 5. Experimental Results

In this section, two aspects are studied and presented:(1)
the evaluation of the method based on an adaptive filtering and a Markovian modelling in order to characterize its behavior, to estimate the best parameters and to determine the best variant;(2)
the comparison to theMorph method.


### 5.1. Adaptive Filtering and Markovian Modelling

We want to determine, first, how to fix the different parameters, second, the preprocessing steps that are necessary, and, finally, which variant is the most efficient. In consequence, these points have been studied.(i)
Parameter values. The weights α1, (8), and α2, (10), are tested from 0 to 1 with a step of 0.1.(ii)
Preprocessings. These preprocessings have been experimented to reduce noises induced by texture, to increase the contrast of the defect and to reduce the light halo in some images as follows.
(1)
Threshold. This preprocessing has been proposed in order to reduce the light halo in the last 4 images presented in Figure 6 and in all the images of the last four categories in Figure 7. Each pixel lower than a given threshold is replaced by the local average of the gray levels.(2)
Smoothing. A mean filter of size 3×3 is applied to reduce the granularity of the texture.(3)
Erosion. An erosion (in gray level) with a square structuring element of size 3×3 is also applied to reduce the granularity of the texture.(4)
Restoration. It combines the advantages of all the previous methods in three steps: a histogram equalization, a thresholding (like Threshold), and an erosion (like Erosion).In order to preserve the crack signal, each pixel under a given threshold is not filtered.12(iii)
Algorithm variants. Four variants are compared as follows.
(1)
Init. This is the initial method proposed in [46].(2)
Gaus. This variant supposes that the distribution of the gray levels inside a crack follows a Gaussian function, see Section 3.2.2.(3)
InMM. This is the initial version with an improvement of the Markovian modelling (new definition of the sites and of the potential function), see Section 3.2.3.(4)
GaMM. This is the method Gaus with the new Markovian modelling.(iv)
Comparison. We have compared this method with the method based on morphological tools and that is quite similar to [15], Morph.


#### 5.1.1. Influence of Parameters

Among all the results, two conclusions can be done.(1)
For each variant and each preprocessing, the weights between the term for adaptive filtering and the term for the Markovian modelling should be the same, see (8), that is, α1=0.5. However, when the weight given to adaptive filtering is the largest, the quality of the results is lower than that when it is the reverse. It means that in this kind of application, the geometric information is more reliable than the photometric information. It seems coherent with the difficulties induced by the acquisition.(2)
For the Markovian modelling, we have noticed that the results are the best when the weights are the same between the orientation term and the distance term, see (10), that is, α2=0.5. However, better results are obtained when the weight of the orientation is greater than the weight of the distance one instead of the reverse. It means that the orientation characteristics are more reliable than the distance ones, and this remark is coherent with the fact that cracks present strong spatial constraints. Moreover, it is also linked with the difficulties induced by the acquisition (the lighting system makes the photometric information less reliable).


#### 5.1.2. Preprocessing

These tests have been done with real images, because the synthetic images do not need preprocessings. These conclusions can be made for the needed preprocessing per method:(i)
Init: restoration;(ii)
Gaus: restoration;(iii)
InMM: threshold;(iv)
GaMM: erosion.However, for the first dataset (acquired with lighting conditions more comfortable than the lighting conditions of the next 4 ones), the preprocessing is not significant for increasing the quality of the results. Moreover, with the new Markovian modelling, the preprocessing step does not significantly increase the quality of the results.


#### 5.1.3. Variants

The results are presented for(1)
synthetic images,(2)
real images.For the first category, the ground truth is available whereas, for the second category, a pseudoground truth is used and the detections which are accepted are taken into account in the evaluation, that is, a threshold is applied to the distance between the segmentation estimated by the evaluated method and the pseudoground truth segmentation. The thresholds applied on the distance for the accepted detections are determined with the results given in Table5, column D.In Figure10, the evolution of the similarity coefficients, or DICE, is presented for the 11 synthetic images, in Figure 10(a), and 10 of the real images, in Figure 10(b). With synthetic images, the method GaMM is clearly the best for most of the images. However, for one image (the fifth), the results are worse than the results of the method Gaus, but they are still correct (DICE = 0.72). On the contrary, for the most difficult images (the 3 first ones that contain a real road background), the method GaMM obtains acceptable results (DICE > 0.5) whereas the other methods are not efficient at all. Illustrations are given in Figures 11 and 12; they show how the method GaMM can reduce false detections.The variations of the similarity coefficients (given in Figure9) for the 4 variants. The first graph shows the results for synthetic images (the first 3 ones are obtained from real images with simulated defect), and the second graph presents the results with real images. The good performances of the methods InMM and GaMM can be noticed.
(a)(b)Figure 11
The segmentation results on some synthetic images presented in Figure6. These are the results obtained with the four variants, and they show how the method GaMM gives the clearest result. We can also notice the good results of the method InMM.Figure 12
The segmentation results on some real images. These are the results obtained with the real images presented in Figure6. The method InMM obtains the clearest detection (i.e., with less false detection), but we can also notice the good quality of the detection map with the method GaMM.


### 5.2. Results and Comparison with Morph

Finally, we have compared the results ofGaMM on each of the complementary dataset (32 images) with Morph. The mean DICE is 0.6 with GaMM whereas it is 0.49 with Morph, see Figure 13. It shows how GaMM can outperform Morph. However, if we compare image per image, the results show that in 50% of the cases GaMM is the best, see illustrations of these results in Figures 14 and 15. More precisely, GaMM seems more efficient with ambiguous images, whereas Morph is the best with reliable images. Finally, we can also precise the execution time for the two methods: about 1 minute for GaMM and 5 seconds for Morph with a processor Intel core 2 duo of 2 GHz. These execution times give only some indications because the implementation, in particular for GaMM, has not been optimized.Figure 13
The comparison of the similarity coefficients betweenGaMM and Morph. The purple axes correspond to the five sets of tested images. For the first set that corresponds to real images with no illumination problems, the results are mixed whereas, for the four other sets, GaMM is the best. The mean of this criterion is 0.6 (variance = 0.0257) for GaMM whereas it is 0.49 (variance = 0.0750) for Morph. However, this method has one step of characterization of the cracks (not introduced in GaMM), and this step can remove cracks that do not respect the characteristics of a cracks (in length, size, and shape). This step contributes to reduce errors, but, in some difficult cases, it decreases the performances of the detection, compared to GaMM.Figure 14
The differences betweenMorph and GaMM. Examples with synthetic images and real ones. Morph is more efficient than GaMM with simple synthetic images whereas GaMM has a better behavior with real images.Figure 15
The differences betweenMorph and GaMM. Examples with real images acquired on a vehicle. The detection with GaMM is more complete than the detection with Morph.Figure 16
The reference segmentations (or pseudoground truth) used in our evaluation protocol for real images. In each original image, the reference segmentation is highlighted in red. Visually, these manual segmentations seem correct and useful for being the reference segmentations in our comparison protocol.


## 5.1. Adaptive Filtering and Markovian Modelling

We want to determine, first, how to fix the different parameters, second, the preprocessing steps that are necessary, and, finally, which variant is the most efficient. In consequence, these points have been studied.(i)
Parameter values. The weights α1, (8), and α2, (10), are tested from 0 to 1 with a step of 0.1.(ii)
Preprocessings. These preprocessings have been experimented to reduce noises induced by texture, to increase the contrast of the defect and to reduce the light halo in some images as follows.
(1)
Threshold. This preprocessing has been proposed in order to reduce the light halo in the last 4 images presented in Figure 6 and in all the images of the last four categories in Figure 7. Each pixel lower than a given threshold is replaced by the local average of the gray levels.(2)
Smoothing. A mean filter of size 3×3 is applied to reduce the granularity of the texture.(3)
Erosion. An erosion (in gray level) with a square structuring element of size 3×3 is also applied to reduce the granularity of the texture.(4)
Restoration. It combines the advantages of all the previous methods in three steps: a histogram equalization, a thresholding (like Threshold), and an erosion (like Erosion).In order to preserve the crack signal, each pixel under a given threshold is not filtered.12(iii)
Algorithm variants. Four variants are compared as follows.
(1)
Init. This is the initial method proposed in [46].(2)
Gaus. This variant supposes that the distribution of the gray levels inside a crack follows a Gaussian function, see Section 3.2.2.(3)
InMM. This is the initial version with an improvement of the Markovian modelling (new definition of the sites and of the potential function), see Section 3.2.3.(4)
GaMM. This is the method Gaus with the new Markovian modelling.(iv)
Comparison. We have compared this method with the method based on morphological tools and that is quite similar to [15], Morph.


### 5.1.1. Influence of Parameters

Among all the results, two conclusions can be done.(1)
For each variant and each preprocessing, the weights between the term for adaptive filtering and the term for the Markovian modelling should be the same, see (8), that is, α1=0.5. However, when the weight given to adaptive filtering is the largest, the quality of the results is lower than that when it is the reverse. It means that in this kind of application, the geometric information is more reliable than the photometric information. It seems coherent with the difficulties induced by the acquisition.(2)
For the Markovian modelling, we have noticed that the results are the best when the weights are the same between the orientation term and the distance term, see (10), that is, α2=0.5. However, better results are obtained when the weight of the orientation is greater than the weight of the distance one instead of the reverse. It means that the orientation characteristics are more reliable than the distance ones, and this remark is coherent with the fact that cracks present strong spatial constraints. Moreover, it is also linked with the difficulties induced by the acquisition (the lighting system makes the photometric information less reliable).


### 5.1.2. Preprocessing

These tests have been done with real images, because the synthetic images do not need preprocessings. These conclusions can be made for the needed preprocessing per method:(i)
Init: restoration;(ii)
Gaus: restoration;(iii)
InMM: threshold;(iv)
GaMM: erosion.However, for the first dataset (acquired with lighting conditions more comfortable than the lighting conditions of the next 4 ones), the preprocessing is not significant for increasing the quality of the results. Moreover, with the new Markovian modelling, the preprocessing step does not significantly increase the quality of the results.


### 5.1.3. Variants

The results are presented for(1)
synthetic images,(2)
real images.For the first category, the ground truth is available whereas, for the second category, a pseudoground truth is used and the detections which are accepted are taken into account in the evaluation, that is, a threshold is applied to the distance between the segmentation estimated by the evaluated method and the pseudoground truth segmentation. The thresholds applied on the distance for the accepted detections are determined with the results given in Table5, column D.In Figure10, the evolution of the similarity coefficients, or DICE, is presented for the 11 synthetic images, in Figure 10(a), and 10 of the real images, in Figure 10(b). With synthetic images, the method GaMM is clearly the best for most of the images. However, for one image (the fifth), the results are worse than the results of the method Gaus, but they are still correct (DICE = 0.72). On the contrary, for the most difficult images (the 3 first ones that contain a real road background), the method GaMM obtains acceptable results (DICE > 0.5) whereas the other methods are not efficient at all. Illustrations are given in Figures 11 and 12; they show how the method GaMM can reduce false detections.The variations of the similarity coefficients (given in Figure9) for the 4 variants. The first graph shows the results for synthetic images (the first 3 ones are obtained from real images with simulated defect), and the second graph presents the results with real images. The good performances of the methods InMM and GaMM can be noticed.
(a)(b)Figure 11
The segmentation results on some synthetic images presented in Figure6. These are the results obtained with the four variants, and they show how the method GaMM gives the clearest result. We can also notice the good results of the method InMM.Figure 12
The segmentation results on some real images. These are the results obtained with the real images presented in Figure6. The method InMM obtains the clearest detection (i.e., with less false detection), but we can also notice the good quality of the detection map with the method GaMM.


## 5.1.1. Influence of Parameters

Among all the results, two conclusions can be done.(1)
For each variant and each preprocessing, the weights between the term for adaptive filtering and the term for the Markovian modelling should be the same, see (8), that is, α1=0.5. However, when the weight given to adaptive filtering is the largest, the quality of the results is lower than that when it is the reverse. It means that in this kind of application, the geometric information is more reliable than the photometric information. It seems coherent with the difficulties induced by the acquisition.(2)
For the Markovian modelling, we have noticed that the results are the best when the weights are the same between the orientation term and the distance term, see (10), that is, α2=0.5. However, better results are obtained when the weight of the orientation is greater than the weight of the distance one instead of the reverse. It means that the orientation characteristics are more reliable than the distance ones, and this remark is coherent with the fact that cracks present strong spatial constraints. Moreover, it is also linked with the difficulties induced by the acquisition (the lighting system makes the photometric information less reliable).


## 5.1.2. Preprocessing

These tests have been done with real images, because the synthetic images do not need preprocessings. These conclusions can be made for the needed preprocessing per method:(i)
Init: restoration;(ii)
Gaus: restoration;(iii)
InMM: threshold;(iv)
GaMM: erosion.However, for the first dataset (acquired with lighting conditions more comfortable than the lighting conditions of the next 4 ones), the preprocessing is not significant for increasing the quality of the results. Moreover, with the new Markovian modelling, the preprocessing step does not significantly increase the quality of the results.


## 5.1.3. Variants

The results are presented for(1)
synthetic images,(2)
real images.For the first category, the ground truth is available whereas, for the second category, a pseudoground truth is used and the detections which are accepted are taken into account in the evaluation, that is, a threshold is applied to the distance between the segmentation estimated by the evaluated method and the pseudoground truth segmentation. The thresholds applied on the distance for the accepted detections are determined with the results given in Table5, column D.In Figure10, the evolution of the similarity coefficients, or DICE, is presented for the 11 synthetic images, in Figure 10(a), and 10 of the real images, in Figure 10(b). With synthetic images, the method GaMM is clearly the best for most of the images. However, for one image (the fifth), the results are worse than the results of the method Gaus, but they are still correct (DICE = 0.72). On the contrary, for the most difficult images (the 3 first ones that contain a real road background), the method GaMM obtains acceptable results (DICE > 0.5) whereas the other methods are not efficient at all. Illustrations are given in Figures 11 and 12; they show how the method GaMM can reduce false detections.The variations of the similarity coefficients (given in Figure9) for the 4 variants. The first graph shows the results for synthetic images (the first 3 ones are obtained from real images with simulated defect), and the second graph presents the results with real images. The good performances of the methods InMM and GaMM can be noticed.
(a)(b)Figure 11
The segmentation results on some synthetic images presented in Figure6. These are the results obtained with the four variants, and they show how the method GaMM gives the clearest result. We can also notice the good results of the method InMM.Figure 12
The segmentation results on some real images. These are the results obtained with the real images presented in Figure6. The method InMM obtains the clearest detection (i.e., with less false detection), but we can also notice the good quality of the detection map with the method GaMM.


## 5.2. Results and Comparison with Morph

Finally, we have compared the results ofGaMM on each of the complementary dataset (32 images) with Morph. The mean DICE is 0.6 with GaMM whereas it is 0.49 with Morph, see Figure 13. It shows how GaMM can outperform Morph. However, if we compare image per image, the results show that in 50% of the cases GaMM is the best, see illustrations of these results in Figures 14 and 15. More precisely, GaMM seems more efficient with ambiguous images, whereas Morph is the best with reliable images. Finally, we can also precise the execution time for the two methods: about 1 minute for GaMM and 5 seconds for Morph with a processor Intel core 2 duo of 2 GHz. These execution times give only some indications because the implementation, in particular for GaMM, has not been optimized.Figure 13
The comparison of the similarity coefficients betweenGaMM and Morph. The purple axes correspond to the five sets of tested images. For the first set that corresponds to real images with no illumination problems, the results are mixed whereas, for the four other sets, GaMM is the best. The mean of this criterion is 0.6 (variance = 0.0257) for GaMM whereas it is 0.49 (variance = 0.0750) for Morph. However, this method has one step of characterization of the cracks (not introduced in GaMM), and this step can remove cracks that do not respect the characteristics of a cracks (in length, size, and shape). This step contributes to reduce errors, but, in some difficult cases, it decreases the performances of the detection, compared to GaMM.Figure 14
The differences betweenMorph and GaMM. Examples with synthetic images and real ones. Morph is more efficient than GaMM with simple synthetic images whereas GaMM has a better behavior with real images.Figure 15
The differences betweenMorph and GaMM. Examples with real images acquired on a vehicle. The detection with GaMM is more complete than the detection with Morph.Figure 16
The reference segmentations (or pseudoground truth) used in our evaluation protocol for real images. In each original image, the reference segmentation is highlighted in red. Visually, these manual segmentations seem correct and useful for being the reference segmentations in our comparison protocol.


## 6. Conclusions

In conclusion, this paper gives a review about image-processing methods for the crack detection of road pavement. It can help the researchers who want to choose and to adapt an auscultation method to the constraints of the transport structure that is studied (it depends on the quality of the surface, the needs of the auscultation). Moreover, a new method for the detection of road cracks has been introduced, and we have presented a new evaluation and comparison protocol for automatic detection of road cracks. As far as we are concerned, we proposed real images with ground truth for the first time in the community. Actually, this dataset is available on this website:http://perso.lcpc.fr/sylvie.chambon/FISSURES/. The new method, GaMM, has been validated by the proposed protocol and compared to a previous one, Morph. This evaluation shows the complementarity of the two methods: the Morph method obtains more true positives than the GaMM method whereas this one reduces the percentage of false positives.Our first improvements of this work will focus on the evaluation and comparison protocol. We want to increase our data set by taking into account the different qualities of road surface or road texture (because for the moment, each proposed method seems very dependent on the quality of the road texture). In a second step, our future work will include new experiments about the acquisition system. Indeed, the acquisitions, and the results obtained with the acquisition system presented have shown its limits for example, in Figure6, some parts of the crack are not “visible.” It comes from the fact that, to highlight the crack, it depends on the orientation of the lights and of the sensors. Using one single sensor and one light always in the same position/orientation, we can sometimes miss some defaults in the acquisition. So, it seems important to study other kinds of systems to improve the quality of the automatic treatments. Finally, we want to improve the GaMM method by adding the extraction of the crack characteristics, like that in Morph.



---

*Source: 989354-2011-10-26.xml*