# Damage Recognition of Road Auxiliary Facilities Based on Deep Convolution Network for Segmentation and Image Region Correction

Authors: Yuanshuai Dong; Yanhong Zhang; Yun Hou; Xinlong Tong; Qingquan Wu; Zuofeng Zhou; Yuxuan Cao
Journal: Advances in Civil Engineering
 (2022)

---

## Abstract

The damage of road auxiliary facilities poses a major hidden danger to driving safety. It is urgent to study a method that can automatically detect the damage of the road auxiliary facilities and provide help for the maintenance of traffic safety auxiliary facilities. In the method for identifying the absence of road auxiliary facilities based on deep convolutional network for image segmentation and image region correction, the PointRend model based on the deep convolutional networks (CNN) is first used to achieve the pixel-level fine segmentation of the auxiliary facilities area, and then, the multiple images in the same image are segmented. In anti-glare panel area, on the largest outer polygon estimated by the convex hull algorithm, the optimal outer quadrilateral is determined according to the distance between the vertices, and then, the anti-glare panel area correction is completed by affine transformation and finally through the image one-dimensional projection mapping and adjacent shading. The distance correlation between the boards realizes the identification and positioning of the missing light-shielding board. The highway anti-glare panel missing recognition method based on deep convolution image segmentation and correction uses the vertex distance to quickly determine the external quadrilateral, which is suitable for estimating the shape of the area in a dynamic scene. After actual testing and verification, it can accurately and efficiently identify the disease of the anti-glare plate. Compared with traditional image segmentation methods, the method using the PointRend target segmentation model has better segmentation quality for target details, and it is more robust when dealing with background interference.


---

## Body

## 1. Introduction

The highway anti-glare panels are located in the middle of the two roads in the opposite direction of the highway, which can block the direct light of the lamps driving in the reverse lane diagonally opposite, prevent dazzle, and ensure that the driver’s eyes are not exposed to strong light. At the same time, it can also help the wind flow guidance to avoid the crosswind (the wind flow perpendicular to the direction of the highway) blowing down and damaging the anti-glare panel. The installation and deployment of the anti-glare panel and defect condition monitoring play a great role in ensuring road driving safety. In practice, due to force majeure such as traffic accidents and wind damage, the missing of highway anti-glare panel is inevitable, which brings great hidden dangers to road safety protection. Timely detection and location of such abnormal problems are very important for the rapid recovery of the road safety environment.In this study, the high-resolution camera collects the image data of the anti-glare panel on the highway. As shown in Figure1, computer vision and depth learning methods are used to identify and detect the missing anti-glare panel on the highway. The main processing steps of this subject include anti-glare panel target detection, morphological correction, and missing recognition.Figure 1
The highway anti-glare panel: (a) anti-glare panel; (b) anti-glare panel; (c) inclined anti-glare panel.
(a)(b)(c)The anti-glare panel target detection realizes the accurate segmentation of the anti-glare panel area. Due to the complex background on both sides of the highway, the traditional image target segmentation based on threshold processing has a poor effect [1, 2]. Local visual features can accurately describe the local content of an image and are widely used in target detection. Zhang [3] extracted the maximum stable extreme regions’ (maximally stable extremal regions, MSER) feature of the target to detect the road sign target. However, the computational complexity of local features is high, which is difficult to meet the needs of highway real-time video data processing, and the robustness of local features to complex backgrounds is still poor. A deep convolution neural network (CNN) uses a multilayer convolution layer to extract advanced image features and has achieved better performance than classical algorithms in image classification, target detection, and recognition. Du et al. [4] used the Darknet algorithm as the basic model, fused the RESNET model convolution layer to extract image depth features, and finally realized the parking space detection in a highway service area. The early target detection and segmentation model based on depth school can realize anti-glare panel detection; for example, Mask RCNN has high accuracy in target detection [5], but the result of target boundary segmentation is vague and unclear, relatively. The accurate positioning and fine segmentation of the anti-glare panel target directly affect the estimation accuracy of the later anti-glare panel shape. Therefore, this study uses the pixel-level depth semantic segmentation model PointRend [6] for anti-glare panel detection. The model can realize the semantic level target segmentation results in a complex background and is suitable for the detection and segmentation of anti-glare panel areas in a highway environment.In the detection results of an anti-glare panel, morphological correction is a necessary processing technology to assist in judging the attitude of anti-glare panel. Since the plane of the mask is almost perpendicular to the direction of the road, in order to collect the rich target content of the mask, the angle between the camera and the direction of the road is less than 90° when the mask image is collected so that multiple mask target images can be collected in one image. However, due to the transmission transformation effect of the lens, the size and shape of the anti-glare panel at different distances from the lens are inconsistent. Therefore, it is necessary to carry out transmission correction on the detected anti-glare panel area to make the vertical dimensions of all anti-glare panels basically consistent. Yang and Yang [7] modeled fisheye imaging and realized fisheye image correction through inverse transformation. Du et al. [8] extracted SURF (Speeded Up Robust Feature) feature sets of two images from different perspectives, matched the feature point sets to estimate the affine change matrix, and finally realized image correction and registration. The image correction method based on local visual feature matching can realize arbitrary viewing angle transformation, but the process of image feature set extraction and matching is very time-consuming, which makes it difficult to meet the real-time processing requirements of the highway. The distribution consistency of multiple anti-glare panels in the image has a strong correlation with the affine transformation matrix. Therefore, this study makes full use of the region information of multiple anti-glare panels, estimates the external polygon of the image anti-glare panel region by two-dimensional convex hull technology, and then determines the approximate optimal external quadrilateral according to the distance between the vertices of the polygon. Finally, the affine transformation matrix is estimated according to the external rectangle information to realize the accurate correction of the anti-glare panel area in the image.The ultimate purpose of this study is to recognize the missing state of the corrected anti-glare panel image. Wang et al. [9] used the support vector machine (SVM) to train the classifier to identify the missing well cover image by analyzing the color difference between the missing target area and the normal area. Min et al. [10] used the Canny detector [11] to extract the target contour and calculate the contour curvature and realized the rapid detection of missing track fasteners through target curvature matching. These methods only aim at the recognition and analysis of individual targets, while the lack of an anti-glare panel in this study needs to be combined with other adjacent anti-glare panels.Therefore, this study proposes to judge whether the image is missing based on the one-dimensional projection mapping of the image and the consistency of the distance between adjacent anti-glare panels. Firstly, the corrected anti-glare panel template image is projected to the vertical direction to accurately locate the position of each anti-glare panel. Secondly, the distance between adjacent anti-glare panels is calculated. According to the characteristics of consistent spacing of normal anti-glare panels and obvious large spacing of the missing anti-glare panels, the position of missing anti-glare panels is determined to realize the accurate identification of missing anti-glare panels. This study purchased the image acquisition device, designed and manufactured the adsorption bracket and camera balance device, and developed the integrated central control device based on the requirements of an engineering application, which formed a set of road auxiliary facility damage identification equipment. The engineering application results show that the research in this study can identify the missing and damage of road auxiliary facilities effectively.


## 2. Semantic Segmentation of Anti-Glare Panel Based on Deep Learning

The accuracy of detection and location of the anti-glare panel area is vital to subsequent damage identification. In a typical semantic segmentation network, the original image will be downsampled, labeled as predicted within the CNN network, and then upsampled to the original resolution. For example, the final label prediction mask generated by the instance segmentation network Mask RCNN is only 28 × 28, which will lead to too smooth label prediction near the edge of the final object. Therefore, this study uses the PointRend (Point rendering) model based on the deep convolution network to achieve an efficient image target detection and segmentation method. By simulating the classical computer graphics methods as effective rendering, image segmentation is regarded as a rendering problem to solve the oversampling and undersampling problems faced by pixel labeling tasks. PointRend neural network module uses an iterative subdivision algorithm to adaptively select key boundary pixels and perform point-based segmentation prediction to achieve the purpose of fine semantic segmentation.


### 2.1. Steps of Image Semantic Segmentation with PointRend

The remarkable feature of this model is that PointRend can be built on the basis of existing depth target segmentation models, such as Mask RCNN [6], FCN [12], and DeepLabv3 [13], to achieve efficient target detection and accurate segmentation. In contrast, the PointRend model can make the excessively smooth area generated in other methods and the boundary of the target area clearer.The PointRend method realizes image semantic segmentation through iterative upsampling and multilayer perceptron classification prediction as follows:(i)
In step 1, process the original image through the existing classical target segmentation model (such as Mask RCNN) and take the model output feature map as the initialization output map of the PointRend model.(ii)
In step 2, the output result is sampled up by double bilinear interpolation to obtain the rough prediction mapIi.(iii)
In step 3, the “hard points” ofN are selected, mainly the points with the highest uncertainty, that is, the pixels with a binary prediction probability close to 0.5.(iv)
In step 4, for each difficulty, the representation vector is obtained. The representation vector is composed of two parts: (1) fine-grained features, which can be obtained by bilinear interpolation on the low-level feature map and (2) the high-level feature (course prediction) is obtained in step 2.(v)
In step 5, a multilayer perceptron (MLP) is used to calculate the representation vector to obtain a new prediction, and an updated rough prediction figure ofIi+1 is obtained.(vi)
In step 6, if the output image resolution is less than the original image resolution, repeat the following steps 2–5. Otherwise, the output prediction figureIi+1 is the final fine segmentation result.


### 2.2. PointRend Module Content

The algorithm not only realizes the accurate segmentation of image details but also greatly improves the model efficiency. The computational power demand is only 2.6% of the Mask RCNN algorithm. The PointRend module consists of three main modules:Module 1 (point selection): select a few “difficulties” for prediction to avoid overcalculating all pixels in high-resolution output. The core idea of PointRend is to adaptively select the points to predict the segmentation label in the image plane. Theoretically, these points should be widely distributed in the high-frequency area of the image (such as the edge of the image).Module 2 (pointwise feature representation): extract the characterization features of “difficulties,” which are obtained by bilinear interpolation of 4 pixels near “difficulties” on the grid. This method can use the subpixel information of the channel dimension to predict the high-resolution tags and realize semantic segmentation. PointRend constructs point-by-point features on selected points by combining fine-grained features and coarse prediction. In terms of fine-grained features, feature vectors are extracted from each sampling point in the CNN feature map to show fine segmentation details. Although fine-grained features can analyze details, they also have two shortcomings. (a) They do not contain specific region information. For segmentation tasks, different labels may be predicted at the same point. (b) Feature mapping for fine-grained features may contain only relatively low-level information. This requires coarse prediction to provide more global background. Taking the example segmentation as an example, the rough segmentation prediction graph can be 7 × 7 in the Mask RCNN output diagram of a lightweight mask head.Module 3 (Point Head): a small neural network is used to represent prediction labels based on point-by-point features. For the feature representation of each selected point, PointRend uses a simple multilayer perceptron (MLP) for point-by-point segmentation prediction. Multilayer perceptrons share weights at all points (all regions). Moreover, because the multilayer perceptron will predict the segmentation label for each point, it can be trained by the segmentation loss of a specific task.In this project, the PointRend model is used to realize accurate detection, positioning, segmentation, and extraction of anti-glare panel targets on the highway. Figure2 shows the results of anti-glare panel segmentation using the PointRend model. It can be seen that anti-glare panels of different sizes have been accurately detected and segmented, and the boundary of the anti-glare panel in the segmentation results is very clear without fracture. Therefore, the detection and segmentation of the anti-glare panel in the actual image can be completed by using the PointRend depth model.Figure 2
Segmentation result of anti-glare panel based on PointRend.To show the comparison of different segmentation results, we compared the PointRend method to Mask RCNN [6] and DeeplabV3 [13] on two test samples, see Figure 3. Because its subdivision strategy efficiently computes high-resolution segmentation maps, the PointRend outputs crisp object boundaries in regions that are over-smoothed by Mask RCNN. The DeeplabV3 sometimes leads to redundant or missing segmentation areas near corners, and the PointRend produces visually reasonable segmentation boundaries.Figure 3
Segmentation comparison of (a) samples, (b) Mask RCNN [6], (c) DeepLabv33 [13], and (d) PointRend.
(a)(b)(c)(d)We evaluate PointRend on segmentation tasks using a collected dataset containing 1000 anti-glare panels from 5 different scenes. The segmentation labels are manually produced as same as the COCO. The 80% are for training the PointRend model and the rest for testing. We also observe quantitative improvements even though the standard intersection-over-union-based metrics-mIoU. The PointRend improves strong Mask RCNN and DeepLabV3 models by a significant margin, as shown in Table1.Table 1
The mIoU on our collected dataset.
MethodMask RCNNDeeplabV3PointRendmIoU88.1890.0593.34


## 2.1. Steps of Image Semantic Segmentation with PointRend

The remarkable feature of this model is that PointRend can be built on the basis of existing depth target segmentation models, such as Mask RCNN [6], FCN [12], and DeepLabv3 [13], to achieve efficient target detection and accurate segmentation. In contrast, the PointRend model can make the excessively smooth area generated in other methods and the boundary of the target area clearer.The PointRend method realizes image semantic segmentation through iterative upsampling and multilayer perceptron classification prediction as follows:(i)
In step 1, process the original image through the existing classical target segmentation model (such as Mask RCNN) and take the model output feature map as the initialization output map of the PointRend model.(ii)
In step 2, the output result is sampled up by double bilinear interpolation to obtain the rough prediction mapIi.(iii)
In step 3, the “hard points” ofN are selected, mainly the points with the highest uncertainty, that is, the pixels with a binary prediction probability close to 0.5.(iv)
In step 4, for each difficulty, the representation vector is obtained. The representation vector is composed of two parts: (1) fine-grained features, which can be obtained by bilinear interpolation on the low-level feature map and (2) the high-level feature (course prediction) is obtained in step 2.(v)
In step 5, a multilayer perceptron (MLP) is used to calculate the representation vector to obtain a new prediction, and an updated rough prediction figure ofIi+1 is obtained.(vi)
In step 6, if the output image resolution is less than the original image resolution, repeat the following steps 2–5. Otherwise, the output prediction figureIi+1 is the final fine segmentation result.


## 2.2. PointRend Module Content

The algorithm not only realizes the accurate segmentation of image details but also greatly improves the model efficiency. The computational power demand is only 2.6% of the Mask RCNN algorithm. The PointRend module consists of three main modules:Module 1 (point selection): select a few “difficulties” for prediction to avoid overcalculating all pixels in high-resolution output. The core idea of PointRend is to adaptively select the points to predict the segmentation label in the image plane. Theoretically, these points should be widely distributed in the high-frequency area of the image (such as the edge of the image).Module 2 (pointwise feature representation): extract the characterization features of “difficulties,” which are obtained by bilinear interpolation of 4 pixels near “difficulties” on the grid. This method can use the subpixel information of the channel dimension to predict the high-resolution tags and realize semantic segmentation. PointRend constructs point-by-point features on selected points by combining fine-grained features and coarse prediction. In terms of fine-grained features, feature vectors are extracted from each sampling point in the CNN feature map to show fine segmentation details. Although fine-grained features can analyze details, they also have two shortcomings. (a) They do not contain specific region information. For segmentation tasks, different labels may be predicted at the same point. (b) Feature mapping for fine-grained features may contain only relatively low-level information. This requires coarse prediction to provide more global background. Taking the example segmentation as an example, the rough segmentation prediction graph can be 7 × 7 in the Mask RCNN output diagram of a lightweight mask head.Module 3 (Point Head): a small neural network is used to represent prediction labels based on point-by-point features. For the feature representation of each selected point, PointRend uses a simple multilayer perceptron (MLP) for point-by-point segmentation prediction. Multilayer perceptrons share weights at all points (all regions). Moreover, because the multilayer perceptron will predict the segmentation label for each point, it can be trained by the segmentation loss of a specific task.In this project, the PointRend model is used to realize accurate detection, positioning, segmentation, and extraction of anti-glare panel targets on the highway. Figure2 shows the results of anti-glare panel segmentation using the PointRend model. It can be seen that anti-glare panels of different sizes have been accurately detected and segmented, and the boundary of the anti-glare panel in the segmentation results is very clear without fracture. Therefore, the detection and segmentation of the anti-glare panel in the actual image can be completed by using the PointRend depth model.Figure 2
Segmentation result of anti-glare panel based on PointRend.To show the comparison of different segmentation results, we compared the PointRend method to Mask RCNN [6] and DeeplabV3 [13] on two test samples, see Figure 3. Because its subdivision strategy efficiently computes high-resolution segmentation maps, the PointRend outputs crisp object boundaries in regions that are over-smoothed by Mask RCNN. The DeeplabV3 sometimes leads to redundant or missing segmentation areas near corners, and the PointRend produces visually reasonable segmentation boundaries.Figure 3
Segmentation comparison of (a) samples, (b) Mask RCNN [6], (c) DeepLabv33 [13], and (d) PointRend.
(a)(b)(c)(d)We evaluate PointRend on segmentation tasks using a collected dataset containing 1000 anti-glare panels from 5 different scenes. The segmentation labels are manually produced as same as the COCO. The 80% are for training the PointRend model and the rest for testing. We also observe quantitative improvements even though the standard intersection-over-union-based metrics-mIoU. The PointRend improves strong Mask RCNN and DeepLabV3 models by a significant margin, as shown in Table1.Table 1
The mIoU on our collected dataset.
MethodMask RCNNDeeplabV3PointRendmIoU88.1890.0593.34


## 3. Fast Region Correction Based on Two-Dimensional Convex Hull

Due to the tilt of the camera angle, the anti-glare panel in the image presents an irregular trapezoidal effect of large near and small far, which brings great difficulties to the consistency determination between anti-glare panels in the later stage. Generally, the trapezoidal area of the anti-glare panel needs to be corrected to a rectangular shape according to the lens transmission parameters. However, due to the inconsistent transmission parameters of the image sequence caused by the vehicle driving the camera, the fixed parameters cannot be used to correct the image, and the parameters need to be estimated adaptively according to the anti-glare panel area in each frame of the image. Although the traditional method based on local feature (such as SIFT) matching can effectively estimate the correction parameters, the algorithm has high complexity and is difficult to meet the needs of real-time processing in practical engineering. Therefore, a fast region correction method based on a two-dimensional convex hull is proposed in this study. The basic idea is that first the Canny edge detection algorithm is used to extract the region contour of the detected anti-glare panel region, then the two-dimensional convex hull technology is used to determine the region’s circumscribed polygon, and then the appropriate circumscribed quadrilateral is selected according to the distance between the convex hull vertex and the image vertex. Finally, the transmission transformation parameters are estimated through the quadrilateral vertices. Thus, the correction of the anti-glare panel area is realized.Firstly, the anti-glare panel area is detected by the PointRend model. In order to remove redundant data, this study detects the edge of the detected anti-glare panel and extracts the area contour. As one of the image features with sharp local intensity change, the edge is usually the pixel with the largest image derivative. In this study, the classical Canny edge detection algorithm [11] is used to extract the edge structure of the anti-glare panel. The Canny detector is widely used to extract a single-pixel edge map of an image. For the simple shape of the anti-glare panel area, the Canny detector is proper for edge extraction and subsequent convex hull algorithm in this study.Based on the local edge pixel set of the image, the convex hull model of the anti-glare panel area is obtained by a two-dimensional convex hull. A convex hull is defined as the intersections of all convex sets containing x for a given set X in a real vector space v. In two-dimensional Euclidean space, a convex hull can be imagined as a convex polygon containing just all points. Common two-dimensional convex hull algorithms include the Graham scanning method, the stepping method, and the center method. In this study, the Graham scanning method is used to calculate the convex hull of the anti-glare panel area. The specific steps are as follows:(1)
Assuming that the edge pixel set of the anti-glare panel edge area isE=P0,P1,…,Pn, first take the lower left pixel in E as the starting point. Then, connect all other edge pixels with P0 in a line, calculate the included angle between the line and the horizontal direction, and sort according to the included angle.(2)
Suppose the sorted pixel sequence is, and then, traverse the sequence in order. IfP˜kP˜k−1 turns clockwise P˜kP˜k+1, it indicates that P˜k is a convex hull vertex, and the pixels P˜k are retained. Otherwise, the pixels P˜k are removed from the collection P˜. Until all pixels in the sequence P˜ are traversed and judged, the vertices of the two-dimensional convex hull are finally retained.(3)
For all pixels in the pixel setP˜, the two-dimensional convex hull of the anti-glare panel area is obtained.The sparse descriptionP˜=P˜0,P˜1,…,P˜n of the anti-glare panel area can be obtained by using the two-dimensional convex hull algorithm. The white area is the detection result of the anti-glare panel, as shown in Figure 4, the black area is the background, and the green closed curve is the two-dimensional convex hull of the anti-glare panel area.Figure 4
Convex hulls of the anti-glare panel in two dimensions.The anti-glare panel area is an arbitrary convex polygon, and only four-pixel positions are required for image correction to estimate the affine transformation matrix. In order to quickly find the optimal vertices of irregular polygons, the distances from the four vertices of the image to all the vertices of the two-dimensional convex hull are calculated in this study. For each pixel of the edge pixel of the anti-glare panel, the distance to the four vertices of the image is calculated, respectively, and the four pixels with the smallest distance are selected as the four verticesS′=S0′,S1′,S2′,S4′ of the anti-glare panel area before the transformation. It is assumed that the corrected anti-glare panel area is rectangular and the vertex coordinates of the rectangle are S=S0,S1,S2,S4:(1)S′=Sa11a12a13a21a22a23a31a31a33=SφS,S′,(2)I′x,y=φS,S′Ix,y,where a11∼a33 is the parameter of the transformation matrix.Then, the image transmission transformation matrix must satisfy equation (1), and then, the transmission transformation matrix φS,S′ can be obtained according to the least square estimation method. Finally, the deformed image I′ is transformed by equation (2) to obtain the corrected image, as shown in Figure 5.Figure 5
Correction results of the anti-glare panel area.


## 4. Localized of Missing Shading Plate Based on Image Projection and Distance Consistency

The distribution of the anti-glare panel in the horizontal direction tends to be consistent after image correction, and the anti-glare panel faces vertically upward. According to the shape and distribution characteristics of anti-glare panels, this study proposes to use image projection to accurately locate the position of each anti-glare panel and then judge the missing situation and locate the missing position of anti-glare panels according to the distance consistency between adjacent anti-glare panels.Since the shape of the anti-glare panel after correction is uniformly distributed in the horizontal direction and the difference is obvious, the corrected imageI′x,y is projected in the horizontal direction to obtain a one-dimensional projection curve, as shown in Figure 6(a). The anti-glare panel area has a large value on the projection curve, while the non-anti-glare panel area has a small value. However, because the anti-glare panel has a certain width, it is difficult to locate the center of the anti-glare panel through the local maximum. In this study, the center of the anti-glare panel area is enhanced by using the local Gaussian filter to smooth the curve, as shown in Figure 6(b).Figure 6
The projection curve and smoothing result of anti-glare panel: (a) one-dimensional projection of the image; (b) one-dimensional curve smoothing.
(a)(b)After projection transformation, the result is(3)Hx=∑yI′x,y.After Gaussian filtering the projection curve, the result is(4)Hsx=Hx∗gx,σ.Furthermore, this study uses one-dimensional local nonmaximum suppression technology to search for each local maximum position of a one-dimensional projection curve, which corresponds to the accurate positioning of the center position of each anti-glare panel. The core idea of nonmaximum suppression technology is to compare the size of the central data with other data in the local one-dimensional or two-dimensional neighborhood of the central data. If the central data are larger than the other data, the central data are a local maximum and are retained; otherwise, the data center is not retained. The exact position of each anti-glare panel can be obtained by suppressing the local maximum on the one-dimensional projection curve, as shown in Figure7:(5)Loc=argxx|NMSHsx,where NMS represents a nonmaximum suppression operation. Generally, normal anti-glare panels are evenly distributed in the horizontal direction, and the spacing distance is basically the same, so the distance between adjacent anti-glare panels is almost equal. However, when the anti-glare panel is missing, the distance between adjacent anti-glare panels at the missing position becomes significantly larger. Therefore, firstly, this study calculates the distance between adjacent anti-glare panels according to the anti-glare panel position obtained by the following equation:(6)li=xi+1−xi,where xi,xi+1∈Loc.Figure 7
Local nonmaximum suppression.Then, allli are sorted, and then, the median distance lMed is taken as the consistency distance standard of the anti-glare panel. Then, compare the size of all li and the median distance lMed. If li is greater than 1.3 × lMed, it is determined that there is a missing between the corresponding two anti-glare panels of li. Otherwise, the corresponding anti-glare panel is normal, as shown in Figure 8. Finally, the one-dimensional position of the missing anti-glare panel is inversely mapped to the two-dimensional original image to realize the accurate recognition of the missing anti-glare panel in the image.Figure 8
The location of the missing anti-glare panel is based on distance consistency.


## 5. Equipment integration and Engineering Application

Based on the requirements of the engineering application, a high-definition industrial camera and supporting lens were purchased, a triangle adsorption bracket and integrated central control device were designed and manufactured, and data processing software was developed, which formed a set of road ancillary facility damage detection and recognition devices. The image acquisition device is composed of a camera and an adsorption bracket. The camera and the adsorption bracket are installed together to form an image acquisition device, and a balance device is designed between the bracket and the camera to ensure the balance of camera shooting, as shown in Figures9(a) and 9(b). The image acquisition device is installed on the hood of the vehicle or the side of the vehicle by adsorption, and the road ancillary facilities are photographed laterally, as shown in Figures 10(a) and 10(b). The power supply system is powered by the vehicle inverter. The integrated central control device is a mechanical shell designed independently in this study. The GPS data processing module, power module, embedded core processing platform, and large-capacity solid-state hard disk are integrated into the integrated central control device. The multidimensional connection holes are designed on the shell of the device so that the central control device can be connected with the image acquisition device, GPS positioning device, and vehicle power supply device efficiently.Figure 9
(a) The integrated central control device. (b) The triangle adsorption bracket.
(a)(b)Figure 10
(a) The image acquisition device that is installed on the side of the vehicle. (b) The image acquisition device that is installed on the hood of the vehicle.
(a)(b)The road ancillary facility damage detection and recognition device based on a deep learning algorithm has been tested in Shanxi, Hainan Province, and other places in China where a large number of road ancillary facilities images have been collected and have been classified, identified, and located by the depth supercomputing workstation, intelligent image processing, and deep learning technology. From the results of several engineering tests, it is proved that the deep semantic segmentation and image correction algorithms developed in this study can identify the deformation and missing of the anti-glare plate accurately, as shown in Figure11.Figure 11
Identification of the anti-glare plate missing and deformation.


## 6. Conclusions

(1)
Pointing at the recognition problem of road ancillary facilities damage, a road auxiliary facility recognition method based on deep semantic segmentation and image correction is proposed in this study. Firstly, the pixel-level target is accurately detected and segmented by using the depth semantic target segmentation model PointRend, and then, the deformed anti-glare plate area is accurately corrected by using the two-dimensional convex hull method and transmission transformation. Finally, the missing position is determined according to the consistency of the distance between adjacent anti-glare panels on the image projection curve. This method has high accuracy in target detection and has certain adaptability to image affine transformation. On the measured sample dataset, the recognition accuracy of the road auxiliary facilities damage was 96%.(2)
The road ancillary facilities damage detection device developed in this study is a portable and lightweight device that can be installed on any vehicle to carry out detection business, which reduces the difficulty of traditional device detection operation. A low-cost road ancillary facility rapid detection method combines hardware and software has been established; it greatly reduces the cost of detection, reduces personnel investment, improves detection efficiency, and improves detection accuracy compared with traditional detection technology. Road ancillary facility detection based on lightweight devices and deep learning algorithms can identify ancillary facility damage early, establish an active maintenance mode, save maintenance costs, and ensure the road operation safety.



---

*Source: 5995999-2022-04-13.xml*