Bridge Maintenance, Safety, Management, Health Monitoring and Informatics – Koh & Frangopol (eds)© 2008 Taylor & Francis Group, London, ISBN 978-0-415-46844-2

Development of an Advanced Inspection System
for Weathering Steel Bridges based on Digital Image Recognition
S. GotoUbe Machinery Corporation, Ube, Yamaguchi, Japan

T. Aso & A. Miyamoto
Graduate School of Science & Engineering, Yamaguchi University, Ube, Japan

ABSTRACT: The monitoring and condition assessment for weathering steel bridges are becoming important aspects on appropriate material selection in planning and design stages, corrosion
prediction and further maintenance planning during their service life. But the visual inspection
is inferior to quantitative and objective characteristics, and the dispersion of evaluation by theinspection person is wide. Therefore, the development of the new evaluation method that has
high reliable, quantitative, and efficient characteristics is necessary. This paper presents the visual inspection method of rust based on image processing and pattern recognition. Three feature
extraction methods; Gray level co-occurrence matrix, Fourier Transform and Multi-resolution
analysis with two-dimensional wavelet decomposition characterize a representative set of 558
images. Then, SVM is used to classify a set of texture features for various corrosion levels. The
classifier trained by Wavelet feature outperformed the others in terms of the identification accuracy.

1 INTRODUCTION
Non-painted bridge with weathering steel (weathering steel bridge) is a representative steel
bridge that will be used in minimum-maintenance. While dryness and moisture repeating appropriately in the atmosphere, weathering steel can form adherent rust that has the quality for corrosion speed becomes slow. Therefore, if appropriate use methods and maintenance are administered, it will be the superior steel which enable bridges to be used semi-permanently in nonpainted. As displayed in Figure 1, the constructive quantity of weathering steel bridges has increased considerably in Japan (Japan Association of Steel Bridge Construction 2007).
Maintenance work in order to verify the evaluation of rust conditions and the corrosion-proof
performance could not be missed. The popular NDT methods such as ion transfer resistancemethod, electrochemical potential method, X-ray diffraction method, spectroscopic identification techniques are employed to measure the corrosion behaviors from the point of view of corrosion loss, and chemic, electric and magnetic properties of the rust patina. At present, inspection and evaluation methods of rust cannot ignore the influence of the sediment besides rust, andthe system that evaluates a lot of inspection results comprehensively is not established. Therefore, there are many cases where the exterior inspection that designates the standard photographas basis is attached importance, but the visual inspection is inferior to quantitative and objective
characteristics, and the dispersion of evaluation by the inspection person is wide. The evaluationof rust conditions is an important process for the maintenance of weathering steel bridges andinfluences the maintenance strategy in the future. The development of the new evaluation
method that has high reliable, quantitative, and efficient characteristics is necessary.
Recently, digital image recognition has been used experimentally in the automated system forbridge inspection, pavement management, sewer pipeline inspection, material engineering. The
critical issues of this approach contain imaging-based feature extraction and pattern classifica-

1630

(%)

0

Weathering steel bridge
all steel bridges
Proportion of weathering steel bridges(year)

0
2005

5
2002

20
1999

10

1996

40

1993

15

1990

60

1987

20

1984

80

1981

25

1978

(㬍10 4ton)

100

Figure 1. Constructive quantity of weathering steel bridges in Japan.

tion algorithm. The approaches for extracting textural features are very diverse, traditional approaches such as Gray Level Co-occurrence Matrix (GLCM) methods, transform-based approaches based on Fast Fourier Transform (FFT) and Wavelet Transform (WT). The WT analysis has shown better performance in many cases due to its outstanding capabilities in spacefrequency decomposition at different scale. Another crucial issue is the pattern classification,
numerous classifiers using the extracted features have been employed, including nearest
neighbor classifier, Bayes classifier, probabilistic neural network, and learning vector quantization. Recently, the support vector machine (SVM) is becoming a successful method for patternrecognition.
This paper presents the visual inspection method of rust based on image processing and pattern recognition. Three feature extraction methods; GLCM, FFT and WT characterize a representative set of 558 images. Then, SVM is used to classify a set of texture features for various
corrosion levels. The classifier trained by Wavelet feature outperformed the others in terms ofclassification rate.
2 VISUAL INSPECTION BASED CONDITION ASSESSMENT
Japan Association of Steel Bridge Construction and Japan Iron and Steel Federation proposed
the visual inspection based criterion for condition assessment of weathering steel. As shown in
Table 1, the condition states are categorized into 5 levels ranging from 5 to 1. The rust patina
with an average granule size of smaller than 5mm diameter is rated as acceptable, while the rust
flakes with granule sizes in the range of 5mm ~ 25mm diameter indicate the requirement of detail inspection. The very poor condition state, i.e. level 1 implies the area observed having thickand loose laminar sheets is failed completely and appears very severe corrosion. Besides the
texture, the color of the rust patina is also highlighted in rating. As displayed in Figure 2, 6 representative sample images of weathering steel specimen extracted from 9 years’ exposure tests
on 41 weathering steel bridges around in Japan are used as references for practical rating. The
variations of color and texture of images with respect to condition state can be clearly observed.
Note that the pro-posed criterion requires expert or trained staff to use subjective judgment onvariations of color and texture properties for decision-making. Moreover, to promote the capabilities of the method for quantitative analysis, the thickness measurement on the rust layer issuggested as an indispensable supplement. As shown in Table 1, when the rust thickness issmaller than 400ȝm, its condition state can be viewed as acceptable. Generally, the electromagnetic coating thickness tester is applied to quantitative thickness measurement.
It is evidence that the visual inspection based condition assessment approaches are still timeconsuming and subjective. The current study is to develop image processing and pattern classification techniques to conduct a quantitative assessment on the condition state of corroded
weathering steel bridges.

1631

Table 1. Rating criterion based on visual inspection and rust thickness measurement.
Condition state
5
4
3

Attribute
Acceptable

2

Require detailinspection

1

FailedDescription of rust patina
Very thin protective oxide film
Average granule size: <1mm diameter
Average granule size: 1~5mm diameter
Granule size: 5~25mm diameter
Having rust flakes
Thick and loose laminar sheets,very severe corrosion

Rust thickness< 200µm

Rating: 5

Rating: 4

Rating: 3

Rating: 2

Rating: 1

Rating: 1< 400µm> 400µm< 800µm> 800µm

Figure 2. Appearance of 9 years’ exposure test on weathering steel specimen and the corresponding rating results via visual inspection (specimen size: 150mm u 100mm ).

3 FEATURE EXTRACTION BASED ON IMAGE PROCESSING
3.1 Gray Level Co-occurrence Matrix
GLCM is representative of the statistical approaches to texture analysis. The GLCM of an image is an estimate of the second-order joint probability, PG (i, j ) of the intensity values of two
pixels (i and j), a distance G apart along a given direction T , i.e., the probability that i and jhave the same intensity. This joint probability takes the form of a square array PG , with row andcolumn dimensions equal to the number of discrete gray levels (intensities) in the image being
examined. If an intensity image were entirely flat (i.e. contained no texture), the resulting
GLCM would be completely diagonal. As the image texture increases (i.e. as the local pixel intensity variations increase), the off-diagonal values in the GLCM become larger.
A quantitative analysis of the GLCM has been proposed through 14 textural descriptors calculated by PG . In this study, four of the most commonly used descriptors (angular second moment, contrast, correlation, and entropy) are used to extract textural features from GLCMs.
n 1 n 1

ASM¦¦ ^PG (i, j )`2(1)

i 0 j 0n 1 n 1CON¦¦ (i  j ) 2 PG (i, j )(2)­° n1 n1½°®¦¦ ijPG (i, j )  P x P y ¾ V xV y
°̄ i 0 j 0°¿(3)i 0i 0COR

1632

n 1 n 1

ENT¦¦ PG (i, j ) log^PG (i, j )`(4)

i 0 j 0Here the means and variances in the x and y direction are given byPx

n 1 n 1¦ i¦ PG (i, j ), P y
i 0 j 0

Vx

n 1

n 1

i 0

j 0

n 1

n 1

j 0

i 0

¦ j ¦ PG (i, j )¦ (i  P x ) 2 ¦ PG (i, j ), V y

(5)

n 1

n 1

j 0

i 0

¦ ( j  P y ) 2 ¦ PG (i, j )(6)

3.2 Fourier transform
The two-dimensional Fast Fourier Transform (2-D FFT) of an image can be thought of as a twodimensional representation of the spatial power spectrum of the image. Theoretical details regarding the 2-D FFT can be obtained from the image-processing literature and many researchers
have proposed the use of FFT spectra as texture feature descriptors. In this study, the commonlyused descriptor is used to extract textural features from FFT spectra.
d (r ' )2S³0 F (r ',T ) dT
2

(7)

Here F (r ' ,T ) is Fourier spectrum in polar coordinates system, r ' is the distance from thecenter of FFT spectra map, and T is the central angle. The center of FFT spectrum map is the
direct current, and as r ' is far from the center, the frequency component is higher. Then,d (r ' ) is an estimate of the frequency energy at each level. In this study, the ratio of the frequency energy is proposed as follow:
Rr 'd (r ' ); r ' 100,200,...,500¦ d (r ' )(8)

3.3 Wavelet transform
Whereas the 2-D FFT performs a frequency decomposition of an image, the 2-D wavelet transform performs a space-frequency decomposition, which is more suitable for texture analysis.
The wavelet energy signatures reflecting the distribution of energy along the frequency domain
over scale and orientation are often employed as the texture features. To eliminate the effect ofillumination intensity, the normalized energy vector is used to describe the energy distribution
at each level (Shin & Hryciew 2004).

Ein

ELH i  EHLi  EHH i
k

¦ (E

LH m

(9)

 EHLm  EHH m )m 1Here Ein is the normalized energy distribution at the decomposition level i , k is the largest
decomposition level, E LH m , E HLm , E HH m are respectively the energy of the detail subimages
LH m , HLm , HH m at level m . Three sub-images (LH, HL, and HH) respectively related to
low-high (vertical), high-low (horizontal), high-high (diagonal) frequency sub-bands denote thefinest scale wavelet coefficients, namely, detail images. Note that for each subimage with
M u N pixels at that level, the energy can be, for instance, in the case of low-high (LH) orientation, expressed as
E LH m¦ C LH ( x, y)

2

(10)

m

M,N

1633

For the rust patina of weathering steel under acceptable condition states of 3, 4 and 5shown in
Figure 2, its surface image appears appropriate to be described by the normalized wavelet energy distribution due to its having relative small and uniform compact granule. While in thestates of 1 and 2, the corroded surface image shows severe irregularity. Its texture content ineach scale showing strong local characteristics should be captured both in global and local distribution of the energy. In this study, a simple method partially considering the local texture
content is proposed as follow:

Eil

E HHi

(11)

E LH i  E HLi

Here Eil represents the ratio of the energy in the diagonal direction to the addition of that invertical and horizontal directions. Hence, a texture feature vector can be constructed in the formE

( E1n , E2n , ... , Ekn , E1l , E2l , ... , Ekl )(12)

4 PATTERN CLASSIFICATION BASED ON SUPPORT VECTOR MACHINE
Recently, an emerging pattern classification method termed support vector machine (SVM) has
received a great deal of attention and demonstrated excellent performance in terms of patternclassification. The basic concept of SVM is to transform the data set to a high dimensional kernel-induced feature space and determine the separating hyperplane that has maximum distanceto the closest points of the training set. The success of SVM is attributed to its employing thestructural risk minimization principle rather than the commonly used empirical risk minimization in artificial neural network. Thus the idea of SVM is to determine a classifier which minimizes a bound on the generalization error instead of simply minimizing the training error. In addition, the SVM training will be equivalent to solving a quadratic programming problem.
Consequently, the solution is always unique and globally optimal. A brief introduction of thebroadly applied models of SVM namely maximal margin classifier and soft margin classifier ispresented here.
Consider a linearly separable training set S be {( x1 , y1 ),...( x n , yn )} where x i  R D is a
D  dimensional vector of input attributes and R is the corresponding output label. The SVM
requires to solve nonlinear classification problem in the formnminw, b , [subject tow  w  C ¦ [i ,i 1

yi w  ) ( x i )  b t 1  [ i ,(13)[ i t 0.
Here [ i , i 1,..., n are the slack variables and C is a penalty parameter used in the margin
constraints to promote the robust measure of the margin distribution. For nonlinear case, a kernel function K ( x i , x j ) ) (x i ), ) (x j ) . is introduced to map the low-dimension input space
R d to high-dimension feature space F using the nonlinear function ) , i.e. zi ) ( x i ) .
There are many possible choices of the kernel functions such as linear, polynomial, radial basis
function (RBF), and sigmoid function. In the present study, the RBF kernel (Gaussian kernel)
with kernel parameter J controlling its width given by the follow is used.
K ( x , y)2exp(J x - y )(14)Hence, there are two parameters C and J having effect on the performance of SVM.

1634

5 EXPERIMENTS
5.1 Flow chart
A schematic of the advanced inspection system for weathering steel using image processing and
SVM is shown in Figure 3. The surface image data of weathering steel acquired by digital camera were preprocessed to create standard rescaled grayscale image for image processing. Then,the SVMs were implemented to classify a set of texture features into various corrosion levels related to condition state of the surface images of weathering steel. The quality assessment was finally reported. In this study the LIBSVM toolbox is employed for classification (Hsu et al.,2003).
5.2 Image acquisition
As earlier stated that the following classification and comparison of the image texture arestrongly scale dependent, consequently, to determine the magnification and resolution of theoriginal image, the image acquisition starts by placing two square sheets with grid points
marked and a distance of 30cm on the imaged surface. Two types of images with
2048 u 1536 and 2560 u 1920 pixels were acquired. Once the image resolution was known,
the image rescaling was implemented to ensure each image having the same resolutionof 1650 / 30 55 pixel/cm . Then, a set of 558 images each with a size of 1024 u 1024 pixels
were split from the rescaled images and transformed to 256 grayscale images for further texture
analysis. To compare the performance of the proposed system with that of the traditional visualinspection method for classification, the condition state of each image was rated according to
the criteria shown in Table 1 and Figure 2 by experienced inspector. The electromagnetic coating thickness tester is utilized to measure the thickness of the loose rust patina as a complementary for rating.

5.3 Training and testing samples
With the created grayscale images, texture features are extracted using the image processing.
Table 2 shows texture features from each processing method of an image data set with their
number of dimensions. GLCMs is calculated from four displacement vectors T 0 $ , 45$ , 90 $ ,and 135$ , and used their average. The scale of the displacement vector is chosen to be 10, 20, or
50 pixels. Wavelet texture features are calculated using the coif1 wavelet basis, and the rescaled
image is decomposed with level equals to 6. To check the robustness of the SVM in dealing

Original image

Database
Image feature
Condition rank

Preprocessing
Analysis image

i =10ං50%

Support Vector Machine(30 times repetition)(100- i ) %Training sample
Training

Image processing
RGB˧Grayscale˧Rescale

Feature vector
Classifier

Testing sample
Classification rank

Figure 3. Imaging-based condition rating of weathering steel using SVM.

1635

with problems with limited training samples, a faction of the total texture feature vectors of theimages ranging from 10% to 50% are employed to train the SVM classifier, while the rest areused for testing. Another critical issue is to consider the randomness of the training and testingprocedures. In this study, for each case with different texture features and partition percentagesof training and test sets, 30 times of classification procedure are randomly conducted using
cross-validation method and all the identification accuracies are averaged to evaluate the SVM
classifiers.
5.4 Experimental results
Figure 4 shows the averaged identification accuracy in each case. Comparison of each accuracyshows the classifier trained by Wavelet feature outperformed the others. On the other hand, theclassifier trained by GLCM and FFT features of case3 has much the same accuracy as trained by
Wavelet feature in training images (50%), but it is not sufficient in comparison with the numberof dimensions. Although considering both of GLCM which gives local statistical features and
FFT which gives global frequency features, it can not be trained effectively.
In wavelet analysis, note that the decomposition level is an important issue and has unavoidable effect on the further classification. In order to investigate the performance of the wavelet
texture features in different decomposition levels, the rescaled image is decomposed with level
equals to 5, 6, 7, and 8. Figure 5 shows the averaged identification accuracy in the case of eachdecomposition level. As can be seen, with the increase of the decomposition level, the averaged
identification accuracy has the trend of increase. When the half samples are used for training inlevel 7, the best classification results (about 87%) can be obtained. However, in the case of levels 6, 7 and 8, the variation appears to be very small. The wavelet feature in low decompositionlevel analysis is low dimension, and enhances the confidence level on condition that classifying
by same samples. Moreover, the calculation time for wavelet analysis and training by SVM canbe reduced, and it can be thought that the optimum decomposition level is 6 for identification.
Table 2. Image feature and dimension.
Case

Image
processing

FeatureDimension

1

GLCM

ASM,CON,COR,ENT(r=10,20,50)

12

2

FFT

Rr’
(r’ = 100ං500)

5

3
4

Case1 and Case2

17

E

12

WT

Identification accuracy (%)

90

1:GLCM
2:FFT

80

3:GLCM+FFT
4:WT

70
60
50
10
20
30
40
50
Proportion of samples used for training (%)

Figure 4. Averaged identification accuracy.

1636

Identification accuracy (%)

90

Level 5
Level 6

85

Level 780

Level 8

75
70
65

10
20
30
40
50
Proportion of samples used for training (%)
Figure 5. Averaged identification accuracy in the case of each decomposition level.
Table 3. Classification result of 50% training images in the case of level 6.
Rating12
3
4
5

Identification result
1
2
3
4
5
1,213
14
63
(94)
(1)
(5)
44 1,774 191
1645(2) (86)
(9)
(1)
(2)
84 312 1,395 152
7(4) (16) (72)
(8)
28
10
97 1,575(1)(1)(6) (92)
470
4
24 1,218(5)(2) (93)

Sum

Number
(%)

1,290
(100)
2,070
(100)
1,950
(100)
1,710
(100)
1,320
(100)

Table 3 shows the total of classification results of 50% training images and 30 classificationsin the case of level 6. The Rating-1 image has the best accuracy 94%, and the lowest accuracy is
72% in the Rating-3 image. They are high accurate classifications, and the proposed inspection
system for weathering steel using image processing and SVM can be realized efficiently.
6 CONCLUTIONSIn this paper, we proposed the advanced inspection system based on image processing and SVM
for weathering steel bridges. Three feature extraction; GLCM, FFT and WT characterize a representative set of 558 images, and SVM is used to classify a set of texture features for various
corrosion levels.
Comparison of each accuracy shows the classifier trained by Wavelet feature outperformed
the others. Investigating the performance of the wavelet texture features in different decomposition levels, the variation appears to be very small in the case of levels 6, 7 and 8. Hence, it canbe thought that the optimum decomposition level is 6 for identification. They are high accurateclassifications, and the proposed inspection system can be realized efficiently.
REFERENCES
Hsu, C. W., Chang, C. C. & Lin, C. J. 2003. A practical guide to support vector classification. Availableat www.csie.ntu.edu.tw/~cjlin/libsvm.
Shin, S. & Hryciew, R. D. 2004. Wavelet analysis of soil mass images for partical size determination. J.
Comput. Civ. Eng. 18(1): 19-27.
The Kozai Club (Iron and Steel Mill Products Association) and Japan Association of Steel Bridge Construction. 2001. Application of Weathering Steel to Highway Bridges (Explanation).

1637