# Automatic Detection and Classification of Road, Car, and Pedestrian Using Binocular Cameras in Traffic Scenes with a Common Framework

Authors: Yongchao Song; Jieru Yao; Yongfeng Ju; Yahong Jiang; Kai Du
Journal: Complexity
 (2020)

---

## Abstract

In order to solve the problems of traffic object detection, fuzzification, and simplification in real traffic environment, an automatic detection and classification algorithm for roads, vehicles, and pedestrians with multiple traffic objects under the same framework is proposed. We construct the final V view through a considerate U-V view method, which determines the location of the horizon and the initial contour of the road. Road detection results are obtained through error label reclassification, omitting point reassignment, and so an. We propose a peripheral envelope algorithm to determine sources of vehicles and pedestrians on the road. The initial segmentation results are determined by the regional growth of the source point through the minimum neighbor similarity algorithm. Vehicle detection results on the road are confirmed by combining disparity and color energy minimum algorithms with the object window aspect ratio threshold method. A method of multifeature fusion is presented to obtain the pedestrian target area, and the pedestrian detection results on the road are accurately segmented by combining the disparity neighbor similarity and the minimum energy algorithm. The algorithm is tested in three datasets of Enpeda, KITTI, and Daimler; then, the corresponding results prove the efficiency and accuracy of the proposed approach. Meanwhile, the real-time analysis of the algorithm is performed, and the average time efficiency is 13 pfs, which can realize the real-time performance of the detection process.


---

## Body

## 1. Introduction

With the rapid development of driverless and assisted driving technologies, autonomous vehicles should have safety functions such as obstacle collision warning, road departure warning, and speed maintenance [1]; then, they can analyze and understand the environment around them and discriminate roads, cars, pedestrians, buildings, and so on in a traffic scene [2]. Road detection is a basic task for many computer vision applications, such as road network extraction, robot autonomous navigation, global navigation satellite system (GNSS), and unmanned aerial vehicle images [3]. Vehicle and pedestrian detection and classification on the road are one of the challenges of advanced driver assistance systems (ADAS) and are essential for traffic safety applications. Based on this, road detection [4], vehicle detection [5, 6], and pedestrian detection [7] are the key steps to realize vehicle autonomous driving technology.As we all know, compared to monocular vision, stereo vision can provide richer information such as depth. Labayrade et al. [8] introduced stereo vision information into the traffic scene and used V view to realize obstacle detection in road environment. In recent years, stereo vision technology has been widely used in road detection, obstacle detection, and other fields [9, 10].Stereo vision-based road detection can obtain accurate road contour estimates and provide clear path driving information, which has been studied for many years. Oh et al. [11] proposed a road detection method combining illumination invariant algorithm and stereo vision. The disparity image and road direction with illumination invariant were estimated; the road probability map was calculated, whether each pixel belongs to the road was estimated, and then the joint bilateral filter was used to optimize the road detection result. Vitor et al. [12] proposed a road detection method that combines 2D image segmentation and 3D image processing. The features obtained by the two-dimensional segmentation technique based on the watershed transform were combined with v-disparity classification elements to form the feature descriptor of the artificial neural network, and the road area, obstacles, and nonclassified area information were estimated. Wu et al. [13] proposed a nonparametric technique road surface detection algorithm that only uses depth cues (abbreviated as the NT-RD method). Relying on the four inherent properties of the disparity image of the road scene, combined with the U view and V view, the road pavement extraction was realized. Wang et al. [14] proposed a method of combining the initial road surface obtained from road surface axis calibration in logarithmic color space and the road surface extracted from stereo vision V view to filter out false detection pixels and obtain accurate road detection results. Guo et al. [15] proposed an automatic estimation method of stereo vision homography matrix based on feature correspondence and region optimization and constructed a road boundary detection algorithm based on stereo vision homography and HMM. An observation probability function based on the state sequence was proposed to obtain the optimal boundary between road and nonroad areas. Li et al. [16] proposed a stereo vision road boundary detection method based on multicues fusion and established three Bayesian models based on the boundary region normal vector, height, and color cues. The points with the highest confidence level were fitted to the road boundary curve using the support vector regression (SVR) method. Xie et al. [17] proposed a binocular vision drivable region detection algorithm based on the cascading framework. The corrected binocular image pair disparity image was given, and the U-V view was calculated in a probabilistic manner. The road region was obtained by the RANSAC plane fitting method. Cheng et al. [18] proposed a stereo vision road edge detection method that integrates 16-dimensional descriptors such as appearance, geometry, and parallax information. The Dijkstra road boundary model with vanishing point constraints was used to search the two minimum cost paths according to the constructed cost map to obtain the curb detection results. Su et al. [19] proposed a texture voting strategy based on stereo vision V view and vision ranging technology to achieve fast vanishing point detection. Formulated lane detection problem was a graph search process. A Dijkstra minimum path lane model with vanishing point limitation was constructed to realize lane detection. Zhang et al. [20] proposed a Dijkstra road model based on vanishing point constraints to implement stereo vision road detection. A weighted sampling RANSAC sample line fitting strategy was used to detect the horizontal line. A vanishing point estimation method for horizontal line and pavement area constraints was proposed. A Dijkstra minimum cost map with vanishing point limitation was constructed to implement road boundary segmentation.Obstacle detection in stereo vision is an effective method for safe driving of vehicles and avoiding collisions. Reliable obstacle detection methods have been the focus of research in the industry. Wang et al. [21] proposed using the convex hull method to extract the region of interest of the obstacle, performing U-view operation on the region of interest and using the connected region extraction method to detect multiple obstacles. Fakhfakh et al. [22] proposed an improved V view that fuses confidence values, used U and V views to estimate potential obstacle boundaries, and used a weighted V-view method to detect obstacles. Kang et al. [23] proposed a probabilistic polar coordinate grid map to analyze the structural characteristics and ergodicity of the volume of the grid map, generated the nearest obstacle and a larger range of potential obstacles in each search direction, and obtained obstacle detection results. Yoo et al. [24] extracted the three features of disparity, superpixel, and pixel gradient, calculated the disparity reliability from superpixel segment and pixel gradient, and proposed the use of reliability voting, CIELAB color similarity, and distance similarity between superpixels to achieve obstacle detection. Burlacu et al. [25] proposed a multicharacterized stereo image obstacle detection framework and used multiple representations of the disparity image: V view, U view, and θ view to achieve obstacle detection.However, the abovementioned method only performs single traffic object detection or general obstacle detection, and the obstacle detection research is oriented to the entire scene, not directly to traffic participants, and does not accurately classify multiple traffic objects. In recent years, with the rapid development of deep learning technology, many methods of applying deep learning have appeared to realize the detection of traffic objects. Han et al. [26] proposed road detection methods based on GANs semisupervised and CGANs weak supervised learning. The method uses a large number of labeled and unlabeled images for training. Dairi et al. [27] proposed the problem of urban scene monitoring and obstacle tracking based on unsupervised deep learning methods. A hybrid encoder integrating a deep Boltzmann machine (DBM) and an automatic encoder (AE) is designed. It combines the greedy learning characteristics of DBM and the dimensionality reduction capability of AE and can accurately and reliably detect the existence of obstacles. Unfortunately, most deep learning-based approaches have high execution time and hardware requirements and are expensive and costly, and only a few of them are suitable for real-time applications [28, 29].In contrast, our approach can deal with the above problems. Based on the above, we propose to implement the classification detection of roads, vehicles, and pedestrians under the same framework. Bicubic interpolation is used to obtain the corrected stereo disparity image. The considerate U-V view method is used to determine the initial contour of the road and obtain the road detection results. We propose a peripheral envelope algorithm to obtain the area of interest on the road and determine the source points of vehicles and pedestrians. Based on the vehicle’s good color similarity and disparity similarity, we construct a minimum energy algorithm to complete the vehicle information and extract the vehicle using the aspect ratio threshold method. A method of multifeature fusion such as aspect ratio, perspective ratio, and area ratio is proposed to obtain the pedestrian target area, and the neighborhood target similarity and energy minimization algorithm is used to accurately extract and segment the pedestrian.


### 1.1. Contributions

Briefly, this paper makes the following some main contributions:(i)
For the complex problem of traffic multiobject classification detection in traffic environment perception, we propose to realize the automatic detection and classification of roads, vehicles, and pedestrians under a common framework, which avoids the detection of a single traffic object or the detection of general obstacles and improves the pertinence of detection objects.(ii)
A new disparity image correction method is proposed to provide conditions for accurate classification and detection of subsequent traffic objects such as roads, vehicles, and pedestrians.(iii)
A considerate U-V view method is proposed to obtain the final V view and the initial contour of the road, which avoids the incorrect estimation of the road contour caused by the traditional straight-line fitting method for the V view of uphill and downhill sections.(iv)
A new method for detecting and classifying vehicles and pedestrians on the road is proposed. A peripheral envelope algorithm is used to obtain the source points of vehicles and pedestrians on the road. Multifeature fusion and threshold segmentation methods are used in combination with the minimum energy and similarity algorithm to achieve classification detection.


### 1.2. Organization

The rest of the paper is organized as follows. Section2 describes the stereo vision road scene model applied by the method in this paper. Section 3 introduces the process of obtaining the initial contour of the road by the proposed considerate U-V view method. Section 4 describes the detailed steps of multitraffic object road, vehicle, and pedestrian classification detection proposed in this paper. Section 5 mainly presents the experiments, including datasets, standards, compared methods, results, and necessary discussions. Finally, Section 6 concludes this paper and describes some future research directions.


## 1.1. Contributions

Briefly, this paper makes the following some main contributions:(i)
For the complex problem of traffic multiobject classification detection in traffic environment perception, we propose to realize the automatic detection and classification of roads, vehicles, and pedestrians under a common framework, which avoids the detection of a single traffic object or the detection of general obstacles and improves the pertinence of detection objects.(ii)
A new disparity image correction method is proposed to provide conditions for accurate classification and detection of subsequent traffic objects such as roads, vehicles, and pedestrians.(iii)
A considerate U-V view method is proposed to obtain the final V view and the initial contour of the road, which avoids the incorrect estimation of the road contour caused by the traditional straight-line fitting method for the V view of uphill and downhill sections.(iv)
A new method for detecting and classifying vehicles and pedestrians on the road is proposed. A peripheral envelope algorithm is used to obtain the source points of vehicles and pedestrians on the road. Multifeature fusion and threshold segmentation methods are used in combination with the minimum energy and similarity algorithm to achieve classification detection.


## 1.2. Organization

The rest of the paper is organized as follows. Section2 describes the stereo vision road scene model applied by the method in this paper. Section 3 introduces the process of obtaining the initial contour of the road by the proposed considerate U-V view method. Section 4 describes the detailed steps of multitraffic object road, vehicle, and pedestrian classification detection proposed in this paper. Section 5 mainly presents the experiments, including datasets, standards, compared methods, results, and necessary discussions. Finally, Section 6 concludes this paper and describes some future research directions.


## 2. Stereo Vision Road Scene Model

The stereo cameras are installed on the same plane, and the horizontal axes of the two cameras are on the same line and have the same parameters [30].Assume that the tilt angle of the camera relative to the vertical plane is θ, the distance between the optical centers of the stereo camera is b, the focal length of the left and right cameras is f, the height of the camera from the horizontal plane is h, and a point in the road scene is px,y,z. The points projected to the left and right image planes are plxl,yl and prxr,yr, respectively, as shown in Figure 1.Figure 1
Stereo vision road scene model.The center of the two cameras is the origin of the world coordinate system. Because the left and right cameras have relative displacement only in the horizontal direction, then(1)b=Ol−Or,(2)xlf=X+b/2Zxrf=X−b/2Z.Disparity in 3D coordinates:(3)d=xl−xr=fbZ.u0,v0 is the projection coordinate of the optical center of the camera. According to the imaging principle, the formula for projecting a point in the world coordinate system onto the left and right camera imaging planes is(4)ul,r=u0+fX±b/2Y⋅sinθ+Z⋅cosθ,v=v0+fY⋅cosθ−Z⋅sinθY⋅sinθ+Z⋅cosθ.The disparity value corresponding to each pixel constitutes a disparity image. Given the valueΔ=ul−ur, we can obtain disparity:(5)Δ=fbY⋅sinθ+Z⋅cosθ.


## 3. Road Initial Contour Detection by Considerate U-V View

Stereo vision-based traffic object detection has significant advantages. It does not require prior knowledge and model building of traffic objects and is not sensitive to background changes caused by weather changes such as shadows, lighting, and reflections. At the same time, U and V views are the basic method of stereo vision processing. How to process the initial U and V views to meet specific traffic objects: road, vehicle, and pedestrian detection is one of the research focuses in this paper.


### 3.1. Disparity Image Acquisition and Correction

We use the “SemiGlobal” [31] disparity estimation algorithm to obtain disparity images, as shown in Figure 2(a). In stereo vision technology, due to uncontrollable factors such as occlusion, complex textures, and reflective lights, the pixels of the reference image cannot match the corresponding points in the target image, which causes invalid matching points in the disparity image that cannot obtain depth information and affect subsequent detection algorithm. An improved bicubic interpolation algorithm is proposed to reassign invalid matching points. Combined with the actual traffic scene, the bicubic interpolation kernel function is redefined, the nuclear response range is expanded, and a 5×7 convolution kernel is used. At the same time, the disparity values of all invalid matching points are set to 0, and the influence of the invalid matching points on the correction result of the disparity image is removed. The interpolation basis function is expressed as follows:(6)Sx=1−λ+3⋅x2+λ+2⋅x3,x<2,−4⋅λ+8⋅λ⋅x−5⋅λ⋅x2+λ⋅x3,2≤x<4,0,x≥4.Figure 2
(a) The original disparity image and (b) the disparity image of interpolation revised.
(a)(b)The target pixel valuefi+u,j+v is obtained by the following bicubic interpolation formula:(7)fi+u,j+v=A∗B∗C.Among them,(8)A=Su+2Su+1SuSu−1Su−2,(9)B=fi−2,j−3⋯fi−2,j+3⋮⋱⋮fi+2,j−3⋯fi+2,j+3,(10)C=Sv+3Sv+2⋯Sv−2Sv−3T.The disparity imagep after interpolation correction obtained by equation (7) is shown in Figure 2(b).


### 3.2. Considerate U-V View

We propose a considerate U-V view method, including obstacle removal on the initial U and V views, inverse view transformation, approximate classification, minimum distance calculation, and noise removal, which constructs the final V view, obtains the initial contour of the road and the horizon position, and provides a basis for the detection of specific traffic objects (Figure3).Figure 3
Considerate operation details of U and V views. (a) Original image. (b) Modified diaparity image. (c) Initial U view. (d) U view after removing obstacles (enlarged). (e) Diaparity image after removing obstacles. (f) Initial V view. (g) V view after removing obstacles. (h) Final V view.


#### 3.2.1. Stereo Vision Road Scene Characteristics

In a stereoscopic road scene, the road surface is on a horizontal plane. Pedestrians and vehicles on the road may be in the vertical and oblique vertical planes directly in front of them or on the vertical planes on the sides of the road. The buildings and trees are on the vertical sides of the sides of the road. The specific scene is shown in Figure1.The stereo vision road scene features are as follows:(a)
The road is from near to far; the farther away from the camera, the smaller the corresponding road scene disparity value and the approximately uniform change.(b)
Obstacle has approximately equal disparity value on the vertical plane. U view mainly reflects the characteristics of obstacle, including the size and range of obstacles. The road has approximately equal disparity value on the horizontal plane. The V view mainly reflects the characteristics of the road, including road extension and road shape.(c)
After the larger obstacles (buildings and trees) are removed by U view, the road is the main part of the current scene. After the road is determined, the corresponding area of interest is obtained, and the traffic object is on it.(d)
In the captured image, when the obstacle and the road point are projected on the same line, the distance from the obstacle to the camera is smaller than the distance from the road point to the camera.


#### 3.2.2. Final V View

The road surface can be divided into straight, uphill, and downhill sections according to the shape of the terrain profile. Therefore, the road surface can be projected as a straight line or a curve in the V view. Obviously, how to obtain the final V view is a necessary step to obtain the initial contour of the road and implement road detection.A more detailed description of the final V disparity image acquisition algorithm is presented in Algorithm1:Algorithm 1: Road peripheral enveloping algorithm.
Require: DI: Disparity image, Δmax, Δmin: Disparity values corresponding to the maximum and minimum intensity points for each line, Deltav: Disparity value of the vth row in the V view, td: Disparity threshold, λd: Euclidean distance, S: Represents the cumulative sum of the Euclidean distances of the disparity values of adjacent rows in the V view;Ensure: Final V view;(1)for e doach frame(2) OriginalDI performs horizontal and vertical projection to obtain initial U and V views;(3) Remove large obstacles in the initial U view;(4) The new U view is inversely transformed into a new disparity image and a new V view is obtained. The disparity value range of    each row of the new V view isΔm−td,Δm+td;(5)WhileΔv≠0do(6)  The new V view is scanned progressively from bottom to top, removing disparity value points that do not meet theminΔv+1≤maxΔv≤maxΔv+1 condition;(7)if n thnew V view still has noise(8)   Find the line where the noise is located, make the point where the disparity point of this line meetsS=∑∀Δv,Δv−1λdΔv,Δv−1 as the reference point of the road, and update the range of road deviation where the noise is      located to Δmin±td;(9)end if(10)  Obtain the Final V view using the considerate U-V view method.(11)end while(12)end for(13) return “Final V view”.


#### 3.2.3. Horizon

The horizon is the dividing line between the road and background. It points to the end of the road, and the accuracy of its position will directly affect the segmentation effect of the road. Pixels on the horizon are infinitely far from the camera, so disparity value is zero. The horizon position can be obtained from the terminal of the point in the V view and the intersection of the obstacle lines in the U view. This article uses the final V view to stop extending to the left; the final number of rows, that is, the number of rows with disparity value of 0 is determined as the horizon.The specific position of the horizon can be formulated as follows:(11)HL=xvΔv=0,where HL represents the horizon of the original image and x represents the number of lines in the V view.


## 3.1. Disparity Image Acquisition and Correction

We use the “SemiGlobal” [31] disparity estimation algorithm to obtain disparity images, as shown in Figure 2(a). In stereo vision technology, due to uncontrollable factors such as occlusion, complex textures, and reflective lights, the pixels of the reference image cannot match the corresponding points in the target image, which causes invalid matching points in the disparity image that cannot obtain depth information and affect subsequent detection algorithm. An improved bicubic interpolation algorithm is proposed to reassign invalid matching points. Combined with the actual traffic scene, the bicubic interpolation kernel function is redefined, the nuclear response range is expanded, and a 5×7 convolution kernel is used. At the same time, the disparity values of all invalid matching points are set to 0, and the influence of the invalid matching points on the correction result of the disparity image is removed. The interpolation basis function is expressed as follows:(6)Sx=1−λ+3⋅x2+λ+2⋅x3,x<2,−4⋅λ+8⋅λ⋅x−5⋅λ⋅x2+λ⋅x3,2≤x<4,0,x≥4.Figure 2
(a) The original disparity image and (b) the disparity image of interpolation revised.
(a)(b)The target pixel valuefi+u,j+v is obtained by the following bicubic interpolation formula:(7)fi+u,j+v=A∗B∗C.Among them,(8)A=Su+2Su+1SuSu−1Su−2,(9)B=fi−2,j−3⋯fi−2,j+3⋮⋱⋮fi+2,j−3⋯fi+2,j+3,(10)C=Sv+3Sv+2⋯Sv−2Sv−3T.The disparity imagep after interpolation correction obtained by equation (7) is shown in Figure 2(b).


## 3.2. Considerate U-V View

We propose a considerate U-V view method, including obstacle removal on the initial U and V views, inverse view transformation, approximate classification, minimum distance calculation, and noise removal, which constructs the final V view, obtains the initial contour of the road and the horizon position, and provides a basis for the detection of specific traffic objects (Figure3).Figure 3
Considerate operation details of U and V views. (a) Original image. (b) Modified diaparity image. (c) Initial U view. (d) U view after removing obstacles (enlarged). (e) Diaparity image after removing obstacles. (f) Initial V view. (g) V view after removing obstacles. (h) Final V view.


### 3.2.1. Stereo Vision Road Scene Characteristics

In a stereoscopic road scene, the road surface is on a horizontal plane. Pedestrians and vehicles on the road may be in the vertical and oblique vertical planes directly in front of them or on the vertical planes on the sides of the road. The buildings and trees are on the vertical sides of the sides of the road. The specific scene is shown in Figure1.The stereo vision road scene features are as follows:(a)
The road is from near to far; the farther away from the camera, the smaller the corresponding road scene disparity value and the approximately uniform change.(b)
Obstacle has approximately equal disparity value on the vertical plane. U view mainly reflects the characteristics of obstacle, including the size and range of obstacles. The road has approximately equal disparity value on the horizontal plane. The V view mainly reflects the characteristics of the road, including road extension and road shape.(c)
After the larger obstacles (buildings and trees) are removed by U view, the road is the main part of the current scene. After the road is determined, the corresponding area of interest is obtained, and the traffic object is on it.(d)
In the captured image, when the obstacle and the road point are projected on the same line, the distance from the obstacle to the camera is smaller than the distance from the road point to the camera.


### 3.2.2. Final V View

The road surface can be divided into straight, uphill, and downhill sections according to the shape of the terrain profile. Therefore, the road surface can be projected as a straight line or a curve in the V view. Obviously, how to obtain the final V view is a necessary step to obtain the initial contour of the road and implement road detection.A more detailed description of the final V disparity image acquisition algorithm is presented in Algorithm1:Algorithm 1: Road peripheral enveloping algorithm.
Require: DI: Disparity image, Δmax, Δmin: Disparity values corresponding to the maximum and minimum intensity points for each line, Deltav: Disparity value of the vth row in the V view, td: Disparity threshold, λd: Euclidean distance, S: Represents the cumulative sum of the Euclidean distances of the disparity values of adjacent rows in the V view;Ensure: Final V view;(1)for e doach frame(2) OriginalDI performs horizontal and vertical projection to obtain initial U and V views;(3) Remove large obstacles in the initial U view;(4) The new U view is inversely transformed into a new disparity image and a new V view is obtained. The disparity value range of    each row of the new V view isΔm−td,Δm+td;(5)WhileΔv≠0do(6)  The new V view is scanned progressively from bottom to top, removing disparity value points that do not meet theminΔv+1≤maxΔv≤maxΔv+1 condition;(7)if n thnew V view still has noise(8)   Find the line where the noise is located, make the point where the disparity point of this line meetsS=∑∀Δv,Δv−1λdΔv,Δv−1 as the reference point of the road, and update the range of road deviation where the noise is      located to Δmin±td;(9)end if(10)  Obtain the Final V view using the considerate U-V view method.(11)end while(12)end for(13) return “Final V view”.


### 3.2.3. Horizon

The horizon is the dividing line between the road and background. It points to the end of the road, and the accuracy of its position will directly affect the segmentation effect of the road. Pixels on the horizon are infinitely far from the camera, so disparity value is zero. The horizon position can be obtained from the terminal of the point in the V view and the intersection of the obstacle lines in the U view. This article uses the final V view to stop extending to the left; the final number of rows, that is, the number of rows with disparity value of 0 is determined as the horizon.The specific position of the horizon can be formulated as follows:(11)HL=xvΔv=0,where HL represents the horizon of the original image and x represents the number of lines in the V view.


## 3.2.1. Stereo Vision Road Scene Characteristics

In a stereoscopic road scene, the road surface is on a horizontal plane. Pedestrians and vehicles on the road may be in the vertical and oblique vertical planes directly in front of them or on the vertical planes on the sides of the road. The buildings and trees are on the vertical sides of the sides of the road. The specific scene is shown in Figure1.The stereo vision road scene features are as follows:(a)
The road is from near to far; the farther away from the camera, the smaller the corresponding road scene disparity value and the approximately uniform change.(b)
Obstacle has approximately equal disparity value on the vertical plane. U view mainly reflects the characteristics of obstacle, including the size and range of obstacles. The road has approximately equal disparity value on the horizontal plane. The V view mainly reflects the characteristics of the road, including road extension and road shape.(c)
After the larger obstacles (buildings and trees) are removed by U view, the road is the main part of the current scene. After the road is determined, the corresponding area of interest is obtained, and the traffic object is on it.(d)
In the captured image, when the obstacle and the road point are projected on the same line, the distance from the obstacle to the camera is smaller than the distance from the road point to the camera.


## 3.2.2. Final V View

The road surface can be divided into straight, uphill, and downhill sections according to the shape of the terrain profile. Therefore, the road surface can be projected as a straight line or a curve in the V view. Obviously, how to obtain the final V view is a necessary step to obtain the initial contour of the road and implement road detection.A more detailed description of the final V disparity image acquisition algorithm is presented in Algorithm1:Algorithm 1: Road peripheral enveloping algorithm.
Require: DI: Disparity image, Δmax, Δmin: Disparity values corresponding to the maximum and minimum intensity points for each line, Deltav: Disparity value of the vth row in the V view, td: Disparity threshold, λd: Euclidean distance, S: Represents the cumulative sum of the Euclidean distances of the disparity values of adjacent rows in the V view;Ensure: Final V view;(1)for e doach frame(2) OriginalDI performs horizontal and vertical projection to obtain initial U and V views;(3) Remove large obstacles in the initial U view;(4) The new U view is inversely transformed into a new disparity image and a new V view is obtained. The disparity value range of    each row of the new V view isΔm−td,Δm+td;(5)WhileΔv≠0do(6)  The new V view is scanned progressively from bottom to top, removing disparity value points that do not meet theminΔv+1≤maxΔv≤maxΔv+1 condition;(7)if n thnew V view still has noise(8)   Find the line where the noise is located, make the point where the disparity point of this line meetsS=∑∀Δv,Δv−1λdΔv,Δv−1 as the reference point of the road, and update the range of road deviation where the noise is      located to Δmin±td;(9)end if(10)  Obtain the Final V view using the considerate U-V view method.(11)end while(12)end for(13) return “Final V view”.


## 3.2.3. Horizon

The horizon is the dividing line between the road and background. It points to the end of the road, and the accuracy of its position will directly affect the segmentation effect of the road. Pixels on the horizon are infinitely far from the camera, so disparity value is zero. The horizon position can be obtained from the terminal of the point in the V view and the intersection of the obstacle lines in the U view. This article uses the final V view to stop extending to the left; the final number of rows, that is, the number of rows with disparity value of 0 is determined as the horizon.The specific position of the horizon can be formulated as follows:(11)HL=xvΔv=0,where HL represents the horizon of the original image and x represents the number of lines in the V view.


## 4. Multiple Traffic Object Classification Detection

Roads, vehicles, and pedestrians constitute the three most basic elements of road traffic scenarios. Research on them can provide basic guarantees for the management and control of urban traffic. It is of great significance for alleviating traffic congestion, reducing traffic accidents, ensuring travel safety and efficiency, and improving the intelligent construction of urban traffic. We propose to realize the classification detection of roads, vehicles, and pedestrians under the same framework in the stereoscopic traffic scene. The overall flow chart of the algorithm is shown in Figure4.Figure 4
The overall flow chart of the algorithm.


### 4.1. Road Detection

The three road shapes of straight line section, uphill section, and downhill section present simple straight shapes in the V view. Therefore, the traditional Hough transform [32] and RANSAC [33] straight line fitting can no longer accurately reflect the road information. We combine the final V view and horizon position obtained by the considerate U-V view method, as shown in Figure 5(b). Rough detection of the road surface is achieved through inverse transformation to obtain the initial contour of the road, as shown in Figure 5(c), but there are still spot spots that are misclassified or missing. Therefore, we propose to reclassify road spots that may be misjudged or missed based on the initial contour of the road by reclassifying the wrong class and reassigning missing points to obtain road detection results. The specific method is as follows:Step 1: the initial contour of the road should be flat and continuous. However, due to the slight fluctuations of the road surface, there is a small area of nonroad category on the road surface. Use the following equation (12) to reclassify the nonroad category of small area:(12)cx,y,l=125∑i=−22∑j=−22lx,y−lx+i,y+j.Among them,l is the class label at coordinates x,y in the binary image obtained in (1) (as shown in Figure 5(d)). When the point is off-road, l=0; when the point is the road, l=1. The class label value on each x,y should ensure that the value of c is minimal.Step 2: after the reclassification of the error class, the small area is completed, and there may also be the error classification point of the larger area. We calculate the initial position and length of the continuous class of each row to achieve reassignment of larger misclassification points:(13)ifln−1=ln+1≠lnandDn<te,thenln−1=ln+1=ln.Figure 5
Road detection algorithm. (a) Original image. (b) Horizon position. (c) Road initial contour. (d) Road point completion. (e) Detection result.
(a)(b)(c)(d)(e)Among them,ln represents the nth region with continuous class labels in a row and te represents the error classification point threshold. When the nth consecutive label area in a row is inconsistent with the left and right areas, and the length Dn of the nth consecutive label area is less than te, the label of this area is reassigned to the same label as the left and right area labels. The road detection result obtained after reclassifying the misjudgment points by the abovementioned method is shown in Figure 5(e).


### 4.2. Vehicle Detection on the Road

The vehicle travels on the road and occupies the inherent range of the road. The acquisition of road contour is the basis of vehicle detection on the road. The algorithm process of vehicle detection on the road proposed in this paper isStep 1: determine the road detection result as the region of interest, which contains the source pointsIp required for vehicle detection on the road. In order to obtain the peripheral contour of the road area, a road peripheral contour estimation algorithm is proposed and named as the peripheral envelope algorithm. The detailed implementation process is described in Algorithm 2. The estimation result is shown in Figure 6(b), and the vehicle source point Ip is shown in Figure 6(c).Step 2: according to the disparity range ofIp and the similarity algorithm between neighbors, Ip performs region growth. First, select the points in Ip as the initial seed points and search its 8 neighborhoods. For the pixels in the range of Δs−tv,Δs+tv, consider it as new seed points (where tv is the vehicle threshold and Deltas is the disparity value of the seed point), iteratively search on this principle until no disparity value is found that meets the rules.Step 3: after the initial segmentation of the vehicle according to Step 2, it is then accurately segmented by the minimum energy algorithm:(14)E=Ed+Ec.Among them,(15)Edx,y=∑∀Δx,Δy∈OλdΔx,Δy.Edx,y represents the disparity similarity between two adjacent pixels, the disparity on the same object should not be much different, O represents an obstacle, Δx,Δy represents the set of all adjacent pixels, and λdΔx,Δy represents the disparity distance between adjacent pixels.(16)Ecx,y=∑∀Cx,Cy∈O1−Cx‖Cx‖⋅Cy‖Cy‖.Ecx,y represents the color similarity of two adjacent pixels, the colors of the same object should have similarity, and Cx and Cy are color vectors of adjacent pixels.Step 4: for the object aggregation region obtained in Step 3, determine the final vehicle detection result on the road by setting the object aspect ratio threshold, as shown in Figure6(d).Algorithm 2: Road peripheral enveloping algorithm.
Require: ROI: Road area detection results, t: number of valid lines for road detection;Ensure: Road peripheral contour;(1)Whilet∉φdo(2) Obtain the coordinate point setspli=xli,yli and pri=xri,yri of the left and right edges of the ROI, and count the    number of left and right coordinate points t1 and t2;(3)Whilei<t1,t2do(4)  Find the maximum valuesmaxxli and maxxri in pli and pri, and use their corresponding coordinate points as the initial     coordinate points pl0 and pr0;(5)  According to the vector outer product formula:c=a∗b∗sinθ, with pl0 and pr0 as the starting points, scan other     coordinate points pli and pri, respectively, and sort them in ascending order from the longitudinal distance from their     respective starting points;(6)  The obtained vertices are sequentially connected to obtain the road outline.(7)end while(8)end while(9) return “Road peripheral contour”.Figure 6
The detection algorithm of vehicle on the road. (a) Original image. (b) Road periphery envelope. (c) Vehicle source point. (d) Detection result.
(a)(b)(c)(d)


### 4.3. Pedestrian Detection on the Road

Similar to the initial steps of the vehicle detection method on the road, that is, the human source point on the road is determined by the contour of the road area, as shown in Figure7(b). For pedestrian detection on the road, we propose to use a multifeature fusion method to obtain the pedestrian target area and combine the disparity neighbor similarity algorithm and the energy minimum algorithm to accurately extract and segment the pedestrian target area. Multifeature fusion target extraction methods mainly use features such as aspect ratio, perspective ratio, and area ratio.Figure 7
The detection algorithm of pedestrian on the road. (a) Original image. (b) Road periphery envelope. (c) Pedestrian source point. (d) Detection result.
(a)(b)(c)(d)


#### 4.3.1. Aspect RatioRlw

Different objects on the road have their corresponding fixed size aspect ratios. Because pedestrians are flexible and show different postures, there will be a small range of fluctuations in the aspect ratio of pedestrians on the road. The aspect ratio can be defined as follows:(17)Rlw=PlPw,where Pl represents the height of pedestrian on the road circumscribed rectangle and Pw represents the width of pedestrian circumscribed rectangle.


#### 4.3.2. Perspective RatioRp

The width of the road surface gradually narrows from near to far and intersects the horizon. Traffic objects on the road are distributed in different parts of the image, and the width of pedestrians gradually decreases with the depth of field. The pedestrian perspectiveRp can be defined as follows:(18)Rp=PlRw,where Rw represents the width of the road surface corresponding to the row where the center coordinates of the pedestrian circumscribed rectangle are.


#### 4.3.3. Area RatioRa

Pedestrians swing their arms or wave their arms when they walk, and they cannot be accurately detected as pedestrians on the road using only aspect ratio and perspective ratio features. We fuse the area ratio feature and define the area ratio as the ratio of the actual number of pedestrian pixels detected to the total number of pixels wrapped around the pedestrian rectangle. The specific formula is(19)Ra=∫Acx,ydsPl⋅Pw,where Acx,y represents the contour curve of the pedestrian on the road.After using the multifeature fusion method to obtain the pedestrian target area, due to the existence of defects and incomplete areas, it is proposed to use the neighborhood similarity and energy minimum algorithm to accurately segment the pedestrian target to determine the final pedestrian detection result, as shown in Figure7(d).


## 4.1. Road Detection

The three road shapes of straight line section, uphill section, and downhill section present simple straight shapes in the V view. Therefore, the traditional Hough transform [32] and RANSAC [33] straight line fitting can no longer accurately reflect the road information. We combine the final V view and horizon position obtained by the considerate U-V view method, as shown in Figure 5(b). Rough detection of the road surface is achieved through inverse transformation to obtain the initial contour of the road, as shown in Figure 5(c), but there are still spot spots that are misclassified or missing. Therefore, we propose to reclassify road spots that may be misjudged or missed based on the initial contour of the road by reclassifying the wrong class and reassigning missing points to obtain road detection results. The specific method is as follows:Step 1: the initial contour of the road should be flat and continuous. However, due to the slight fluctuations of the road surface, there is a small area of nonroad category on the road surface. Use the following equation (12) to reclassify the nonroad category of small area:(12)cx,y,l=125∑i=−22∑j=−22lx,y−lx+i,y+j.Among them,l is the class label at coordinates x,y in the binary image obtained in (1) (as shown in Figure 5(d)). When the point is off-road, l=0; when the point is the road, l=1. The class label value on each x,y should ensure that the value of c is minimal.Step 2: after the reclassification of the error class, the small area is completed, and there may also be the error classification point of the larger area. We calculate the initial position and length of the continuous class of each row to achieve reassignment of larger misclassification points:(13)ifln−1=ln+1≠lnandDn<te,thenln−1=ln+1=ln.Figure 5
Road detection algorithm. (a) Original image. (b) Horizon position. (c) Road initial contour. (d) Road point completion. (e) Detection result.
(a)(b)(c)(d)(e)Among them,ln represents the nth region with continuous class labels in a row and te represents the error classification point threshold. When the nth consecutive label area in a row is inconsistent with the left and right areas, and the length Dn of the nth consecutive label area is less than te, the label of this area is reassigned to the same label as the left and right area labels. The road detection result obtained after reclassifying the misjudgment points by the abovementioned method is shown in Figure 5(e).


## 4.2. Vehicle Detection on the Road

The vehicle travels on the road and occupies the inherent range of the road. The acquisition of road contour is the basis of vehicle detection on the road. The algorithm process of vehicle detection on the road proposed in this paper isStep 1: determine the road detection result as the region of interest, which contains the source pointsIp required for vehicle detection on the road. In order to obtain the peripheral contour of the road area, a road peripheral contour estimation algorithm is proposed and named as the peripheral envelope algorithm. The detailed implementation process is described in Algorithm 2. The estimation result is shown in Figure 6(b), and the vehicle source point Ip is shown in Figure 6(c).Step 2: according to the disparity range ofIp and the similarity algorithm between neighbors, Ip performs region growth. First, select the points in Ip as the initial seed points and search its 8 neighborhoods. For the pixels in the range of Δs−tv,Δs+tv, consider it as new seed points (where tv is the vehicle threshold and Deltas is the disparity value of the seed point), iteratively search on this principle until no disparity value is found that meets the rules.Step 3: after the initial segmentation of the vehicle according to Step 2, it is then accurately segmented by the minimum energy algorithm:(14)E=Ed+Ec.Among them,(15)Edx,y=∑∀Δx,Δy∈OλdΔx,Δy.Edx,y represents the disparity similarity between two adjacent pixels, the disparity on the same object should not be much different, O represents an obstacle, Δx,Δy represents the set of all adjacent pixels, and λdΔx,Δy represents the disparity distance between adjacent pixels.(16)Ecx,y=∑∀Cx,Cy∈O1−Cx‖Cx‖⋅Cy‖Cy‖.Ecx,y represents the color similarity of two adjacent pixels, the colors of the same object should have similarity, and Cx and Cy are color vectors of adjacent pixels.Step 4: for the object aggregation region obtained in Step 3, determine the final vehicle detection result on the road by setting the object aspect ratio threshold, as shown in Figure6(d).Algorithm 2: Road peripheral enveloping algorithm.
Require: ROI: Road area detection results, t: number of valid lines for road detection;Ensure: Road peripheral contour;(1)Whilet∉φdo(2) Obtain the coordinate point setspli=xli,yli and pri=xri,yri of the left and right edges of the ROI, and count the    number of left and right coordinate points t1 and t2;(3)Whilei<t1,t2do(4)  Find the maximum valuesmaxxli and maxxri in pli and pri, and use their corresponding coordinate points as the initial     coordinate points pl0 and pr0;(5)  According to the vector outer product formula:c=a∗b∗sinθ, with pl0 and pr0 as the starting points, scan other     coordinate points pli and pri, respectively, and sort them in ascending order from the longitudinal distance from their     respective starting points;(6)  The obtained vertices are sequentially connected to obtain the road outline.(7)end while(8)end while(9) return “Road peripheral contour”.Figure 6
The detection algorithm of vehicle on the road. (a) Original image. (b) Road periphery envelope. (c) Vehicle source point. (d) Detection result.
(a)(b)(c)(d)


## 4.3. Pedestrian Detection on the Road

Similar to the initial steps of the vehicle detection method on the road, that is, the human source point on the road is determined by the contour of the road area, as shown in Figure7(b). For pedestrian detection on the road, we propose to use a multifeature fusion method to obtain the pedestrian target area and combine the disparity neighbor similarity algorithm and the energy minimum algorithm to accurately extract and segment the pedestrian target area. Multifeature fusion target extraction methods mainly use features such as aspect ratio, perspective ratio, and area ratio.Figure 7
The detection algorithm of pedestrian on the road. (a) Original image. (b) Road periphery envelope. (c) Pedestrian source point. (d) Detection result.
(a)(b)(c)(d)


### 4.3.1. Aspect RatioRlw

Different objects on the road have their corresponding fixed size aspect ratios. Because pedestrians are flexible and show different postures, there will be a small range of fluctuations in the aspect ratio of pedestrians on the road. The aspect ratio can be defined as follows:(17)Rlw=PlPw,where Pl represents the height of pedestrian on the road circumscribed rectangle and Pw represents the width of pedestrian circumscribed rectangle.


### 4.3.2. Perspective RatioRp

The width of the road surface gradually narrows from near to far and intersects the horizon. Traffic objects on the road are distributed in different parts of the image, and the width of pedestrians gradually decreases with the depth of field. The pedestrian perspectiveRp can be defined as follows:(18)Rp=PlRw,where Rw represents the width of the road surface corresponding to the row where the center coordinates of the pedestrian circumscribed rectangle are.


### 4.3.3. Area RatioRa

Pedestrians swing their arms or wave their arms when they walk, and they cannot be accurately detected as pedestrians on the road using only aspect ratio and perspective ratio features. We fuse the area ratio feature and define the area ratio as the ratio of the actual number of pedestrian pixels detected to the total number of pixels wrapped around the pedestrian rectangle. The specific formula is(19)Ra=∫Acx,ydsPl⋅Pw,where Acx,y represents the contour curve of the pedestrian on the road.After using the multifeature fusion method to obtain the pedestrian target area, due to the existence of defects and incomplete areas, it is proposed to use the neighborhood similarity and energy minimum algorithm to accurately segment the pedestrian target to determine the final pedestrian detection result, as shown in Figure7(d).


## 4.3.1. Aspect RatioRlw

Different objects on the road have their corresponding fixed size aspect ratios. Because pedestrians are flexible and show different postures, there will be a small range of fluctuations in the aspect ratio of pedestrians on the road. The aspect ratio can be defined as follows:(17)Rlw=PlPw,where Pl represents the height of pedestrian on the road circumscribed rectangle and Pw represents the width of pedestrian circumscribed rectangle.


## 4.3.2. Perspective RatioRp

The width of the road surface gradually narrows from near to far and intersects the horizon. Traffic objects on the road are distributed in different parts of the image, and the width of pedestrians gradually decreases with the depth of field. The pedestrian perspectiveRp can be defined as follows:(18)Rp=PlRw,where Rw represents the width of the road surface corresponding to the row where the center coordinates of the pedestrian circumscribed rectangle are.


## 4.3.3. Area RatioRa

Pedestrians swing their arms or wave their arms when they walk, and they cannot be accurately detected as pedestrians on the road using only aspect ratio and perspective ratio features. We fuse the area ratio feature and define the area ratio as the ratio of the actual number of pedestrian pixels detected to the total number of pixels wrapped around the pedestrian rectangle. The specific formula is(19)Ra=∫Acx,ydsPl⋅Pw,where Acx,y represents the contour curve of the pedestrian on the road.After using the multifeature fusion method to obtain the pedestrian target area, due to the existence of defects and incomplete areas, it is proposed to use the neighborhood similarity and energy minimum algorithm to accurately segment the pedestrian target to determine the final pedestrian detection result, as shown in Figure7(d).


## 5. Experiment

In this section, we use multiple experiments to evaluate the performance of the traffic object classification detection system proposed in this paper. We selected and tested multiple image sequences including different road scenes, different weather conditions, different city streets, and suburban roads:(1)
Enpeda dataset [34] is a synthetic stereo vision dataset, which contains 496 pairs of traffic scene images with a resolution of 640×480. The traffic scene in the dataset is relatively simple. Roads are divided into two types: planar and nonplanar, and there are more vehicles.(2)
KITTI dataset [35] is a real-life stereo image dataset that contains 194 pairs of discontinuous traffic scene images with different resolutions. The dataset has complex traffic scenes and contains occluded areas.(3)
Daimler dataset [36] is a real-life stereo image dataset with a large number of images, over 21,790 pairs, collecting 27-minute continuous urban road traffic images with a resolution of 640×480. The dataset contains a wealth of traffic objects such as roads, vehicles, and pedestrians, including continuous and complex road conditions.The original images, disparity images, and interpolation-corrected disparity images of each standard dataset are shown in Figure8, where Figure 8(a) is the original image, Figure 8(b) is the original disparity image, and Figure 8(c) is the corrected disparity image.Figure 8
Comparison of disparity figure before and after correction, from top to bottom is Enpeda, KITTI, and Daimler dataset, respectively. (a) Original image. (b) Original disparity image. (c) Corrected disparity image.
(a)(b)(c)


### 5.1. Road Detection Results

In this paper, the road surface is obtained by the proposed algorithm. It is experimentally determined that the thresholdtd of Enpeda dataset is 3 and KITTI and Daimler datasets is 5. For the threshold te for removing misclassification points, the experimental setting is 3. According to our proposed algorithm and NT-RD algorithm, experiments are performed on three standard datasets, and the road detection results of some typical sections are shown in Figure 9.Figure 9
Typical road pavement detection results. (1) Original image (left). (2) Initial road contour for NT-RD method. (3) Initial road contour for ours. (4) Road detection result for NT-RD method. (5) Road detection result for ours.
(a)(b)(c)(d)(e)(f)Figures9(a) and 9(b) are the road detection results of the selected Enpeda dataset, Figure 9(a) is the uphill section, Figure 9(b) is the downhill section, and Figure 9(b) the road section contains vehicles on the road. Figures 9(c) and 9(d) are the road detection results of the selected KITTI dataset, and they are all horizontal sections. The road conditions in Figure 9(c) are complicated, including multiple vehicles, and the road is relatively congested. Figures 9(e) and 9(f) are road detection results of the selected Daimler dataset, Figure 9(e) is a downhill section with pedestrians on the road, and Figure 9(f) is an uphill section with vehicles on the road and a large pavement area. The roads in Figures 9(b)–9(d) contain a lot of shadows, and the roads in Figures 9(e) and 9(f) have traffic markings.It can be seen from Figure9 that, in the Enpeda dataset, the detection results of our method and the NT-RD method are basically the same. In the KITTI and Daimler datasets, the detection results of our method are significantly better than the NT-RD method. For different road environments, our algorithms are robust and can successfully detect road surfaces.Considering the large number of images in the standard dataset, for quantitative analysis, we use a random sampling method to randomly extract 100 pairs of images from each of the three standard datasets to mark, use the proposed method for road surface detection and analysis, and randomly repeat sampling five times. The following performance indexes are defined for quantitative analysis of the road detection results: (1) accuracy rateP, (2) recall rate R, and (3) comprehensive evaluation index F. Let ζ be the real area of the road and ξ be the actual detection result of the road; then, P=∑ζ⋅ξ/∑ξ, R=∑ζ⋅ξ/∑ζ, and F=2PR/P+R, PRF∈01. P and R are complementary. F is the weighted harmonic average of P and R and can reconcile and comprehensively reflect the accuracy rate P and the recall rate R. The closer F is to 1, the better the road detection is. Due to the random sampling method in this article, there are slight fluctuations in the P, R, and F values, and the standard deviation of the fluctuations is σ. The comparison of the road surface performance indexes between ours and NT-RD method is shown in Table 1. The comparison of the comprehensive evaluation index F of the detection results is shown in Figure 10.Table 1
Road detection algorithm performance comparison of our and NT-RD method.
DatasetMethodP±σR±σF±σEnpedaNT-RD†0.9603±0.07410.9132±0.03900.9362±0.0511Ours‡0.9847±0.02170.9662±0.01730.9754±0.0193KITTINT-RD0.8816±0.12640.9323±0.09680.9062±0.1096Ours0.9372±0.03890.9637±0.02550.9503±0.0308DaimlerNT-RD0.9106±0.08320.9331±0.10460.9217±0.0927Ours0.9335±0.05680.9418±0.08740.9376±0.0689† is algorithm used in Reference [10]. ‡ is algorithm proposed in this paper.Figure 10
Comprehensive evaluation indexF curve comparison of three datasets. (a) F value comparison of Enpeda dataset. (b) F value comparison of KITTI dataset. (c) F value comparison of Daimler dataset.
(a)(b)(c)It can be seen from Table1 that, in the Enpeda, KITTI, and Daimler datasets, the performance indexes P, R, and F of our method are significantly higher than the NT-RD method. It can be seen from Figure 10 that the average value of the comprehensive evaluation index F of the proposed method is large and the fluctuation is small. The experimental results show that the results of the proposed road detection algorithm can achieve the expected results and are significantly better than the NT-RD method in terms of P, R, and F.


### 5.2. Vehicle Detection Results on Road

It is proposed to perform vehicle detection on the road based on the results of road detection. In the experiment, the vehicle thresholdtv is set to 5, and the vehicle aspect ratio threshold range is 0.5 to 2.Considering that binocular vision has not been used for accurate vehicle detection in related studies, this paper will separately explain the accuracy of the proposed vehicle detection algorithm from qualitative and quantitative aspects. The experiments are performed on the image pairs containing valid vehicles in three standard datasets. The Enpeda dataset contains 477 valid vehicles, the KITTI dataset contains 173 valid vehicles, and the Daimler dataset contains 462 valid vehicles in a time with more vehicles. Some typical vehicle detection results on the road are shown in Figure11.Figure 11
Typical vehicle detection result on the road. (1) Original image (left). (2) Region of interest. (3) Vehicle source point. (4) Initial detection result. (5) Detection result.
(a)(b)(c)(d)(e)(f)Among them, Figures11(a) and 11(b) are vehicle detection results of the Enpeda dataset, Figures 11(c) and 11(d) are vehicle detection results of the KITTI dataset, and Figures 11(e) and 11(f) are vehicle detection results of the Daimler dataset. From the qualitative detection result in the above figures, it can be seen that the proposed algorithm can accurately detect whether it is a single-car or multi-car scene. Vehicles failed to be detected in Figures 11(d) and 11(e) because the vehicles in Figure 11(e) are not included in the road detection results and the vehicles are far away in Figure 11(d); the body and background are integrated.Similarly, the results of the proposed vehicle detection algorithm are evaluated using Precision, Recall, and comprehensive evaluation indexF. The accuracy rate is for the prediction result, which indicates how many of the results detected as vehicles are real vehicles calibrated in the dataset; the recall rate is for the original sample, which indicates how many real vehicles calibrated in the dataset are accurately detected. The dark gray area indicates the type of dataset, and the light gray area indicates the number of real and effective vehicles in the three datasets and the actual number of vehicles detected by the algorithm. From these data, the accuracy rate, recall rate, and comprehensive evaluation index can be calculated. The calculation formulas are shown in (20)–(22) [37]:(20)P=TPTP+FP,(21)R=TPTP+FN,(22)F=2P⋅RP+R.Among them,TP is the true rate, that is, it is detected as a vehicle and it is indeed a vehicle; FP is a false positive rate, that is, it is detected as a vehicle, which is actually a nonvehicle object; TN is a true negative rate, that is, the vehicle is not detected but is actually a vehicle; and FN is a false negative rate, that is, the vehicle is not detected and is indeed a nonvehicle object. The detection results are shown in Table 2.Table 2
The vehicle detection result mixing matrix of three standard datasets.
Algorithm AssessmentEnpedaKITTIDaimlerRecall469182451Ground TruthEnpeda4774180087.63KITTI1730163094.21Daimler4620040287.01Precision89.1389.5689.1489.44From the above experimental results, it can be known that the vehicle detection on the road algorithm proposed in this paper can accurately measure the area where the vehicle is located and has better robustness under complex background. The algorithm only detects traffic objects in theROI region, reducing the range of possible vehicles, thereby reducing the calculation time, increasing the calculation speed, and meeting the real-time requirements. However, the algorithm still has missing detection and false detection. The main reasons are as follows: (1) there is a lot of noise and invalid areas in the disparity image. If such areas are large, incomplete correction will affect the detection results. (2) There are errors in road detection results, resulting in inaccurate ROI regions and missed detection of road vehicles. (3) There is not much difference in disparity and color similarity between the road near the vehicle and the vehicle, which affects the accurate division of the vehicle.


### 5.3. Pedestrian Detection Results on the Road

We select road scenes from typical datasets to verify the proposed pedestrian detection algorithm on the road. Experiments have determined that the range of the pedestrian’s aspect ratio thresholdRlw is set to 1.5–5, the perspective ratio Rp threshold is set to 0.12–0.45, and the area ratio Ra is less than 0.75. Some typical pedestrian detection results on the road are shown in Figures 12; Figures 12(a) and 12(c) are single pedestrian detection results, and Figures 12(b) and 12(d) are multiple pedestrian detection results. At the same time, the road scenes in Figures 12(b)–12(d) are more complicated, and there are more types of nonpedestrian obstacles.Figure 12
Typical pedestrian detection result on the road. (1) Original image (left). (2) Region of interest. (3) Pedestrian source point. (4) Pedestrian detection result. (5) Comparison.
(a)(b)(c)(d)From the above qualitative experimental results, it can be known that the pedestrian detection algorithm proposed in this paper can accurately measure the area where pedestrians are located and has better robustness under complex backgrounds.


### 5.4. Real-Time Analysis

The experiment in this paper uses a Windows 10 64 bit operating system, a CPU with Intel Core i5, 3.2 GHz, and a 32G memory PC as the experimental platform. The experimental environment is MATLAB 2017b.For real-time analysis, this section uses a random sampling method to randomly extract 100 pairs of images from the three standard datasets for labeling. The method for automatic detection and classification of traffic objects proposed in this paper can be summarized into five stages: (1) acquisition and correction of disparity images, (2) considerate U-V view, (3) road surface detection, (4) vehicle detection on the road, and (5) pedestrian detection on the road. Figure13 shows the average processing time for each stage of each frame in each dataset. The processing time for road detection, road vehicle detection, and pedestrian detection is calculated separately. It can be seen from Figure 13 that the total processing time of the KITTI dataset is the longest, which is about 99.8 ms (10pfs). It is mainly a road in the city and there are many objects in the scene. It takes a relatively long time in road detection, vehicle detection, and pedestrian detection on the road, and its image resolution is larger than that of the Enpeda and Daimler datasets. The least time consuming is the Enpeda dataset, with a time of about 59.4 ms (17pfs). This dataset is artificially synthesized, the target scene is relatively simple, and the pedestrian scene is lacking. The total processing time of the Daimler dataset is about 79.2 ms (13pfs). The scenes of roads, vehicles, and pedestrians on the road are rich, and the time is between the two datasets.Figure 13
The real-time analysis result of three standard datasets.The real-time analysis results show that, in the actual application process, a certain number of interval frames can be selected for processing, and the real-time requirements can be met without affecting the final results of the experiment.


## 5.1. Road Detection Results

In this paper, the road surface is obtained by the proposed algorithm. It is experimentally determined that the thresholdtd of Enpeda dataset is 3 and KITTI and Daimler datasets is 5. For the threshold te for removing misclassification points, the experimental setting is 3. According to our proposed algorithm and NT-RD algorithm, experiments are performed on three standard datasets, and the road detection results of some typical sections are shown in Figure 9.Figure 9
Typical road pavement detection results. (1) Original image (left). (2) Initial road contour for NT-RD method. (3) Initial road contour for ours. (4) Road detection result for NT-RD method. (5) Road detection result for ours.
(a)(b)(c)(d)(e)(f)Figures9(a) and 9(b) are the road detection results of the selected Enpeda dataset, Figure 9(a) is the uphill section, Figure 9(b) is the downhill section, and Figure 9(b) the road section contains vehicles on the road. Figures 9(c) and 9(d) are the road detection results of the selected KITTI dataset, and they are all horizontal sections. The road conditions in Figure 9(c) are complicated, including multiple vehicles, and the road is relatively congested. Figures 9(e) and 9(f) are road detection results of the selected Daimler dataset, Figure 9(e) is a downhill section with pedestrians on the road, and Figure 9(f) is an uphill section with vehicles on the road and a large pavement area. The roads in Figures 9(b)–9(d) contain a lot of shadows, and the roads in Figures 9(e) and 9(f) have traffic markings.It can be seen from Figure9 that, in the Enpeda dataset, the detection results of our method and the NT-RD method are basically the same. In the KITTI and Daimler datasets, the detection results of our method are significantly better than the NT-RD method. For different road environments, our algorithms are robust and can successfully detect road surfaces.Considering the large number of images in the standard dataset, for quantitative analysis, we use a random sampling method to randomly extract 100 pairs of images from each of the three standard datasets to mark, use the proposed method for road surface detection and analysis, and randomly repeat sampling five times. The following performance indexes are defined for quantitative analysis of the road detection results: (1) accuracy rateP, (2) recall rate R, and (3) comprehensive evaluation index F. Let ζ be the real area of the road and ξ be the actual detection result of the road; then, P=∑ζ⋅ξ/∑ξ, R=∑ζ⋅ξ/∑ζ, and F=2PR/P+R, PRF∈01. P and R are complementary. F is the weighted harmonic average of P and R and can reconcile and comprehensively reflect the accuracy rate P and the recall rate R. The closer F is to 1, the better the road detection is. Due to the random sampling method in this article, there are slight fluctuations in the P, R, and F values, and the standard deviation of the fluctuations is σ. The comparison of the road surface performance indexes between ours and NT-RD method is shown in Table 1. The comparison of the comprehensive evaluation index F of the detection results is shown in Figure 10.Table 1
Road detection algorithm performance comparison of our and NT-RD method.
DatasetMethodP±σR±σF±σEnpedaNT-RD†0.9603±0.07410.9132±0.03900.9362±0.0511Ours‡0.9847±0.02170.9662±0.01730.9754±0.0193KITTINT-RD0.8816±0.12640.9323±0.09680.9062±0.1096Ours0.9372±0.03890.9637±0.02550.9503±0.0308DaimlerNT-RD0.9106±0.08320.9331±0.10460.9217±0.0927Ours0.9335±0.05680.9418±0.08740.9376±0.0689† is algorithm used in Reference [10]. ‡ is algorithm proposed in this paper.Figure 10
Comprehensive evaluation indexF curve comparison of three datasets. (a) F value comparison of Enpeda dataset. (b) F value comparison of KITTI dataset. (c) F value comparison of Daimler dataset.
(a)(b)(c)It can be seen from Table1 that, in the Enpeda, KITTI, and Daimler datasets, the performance indexes P, R, and F of our method are significantly higher than the NT-RD method. It can be seen from Figure 10 that the average value of the comprehensive evaluation index F of the proposed method is large and the fluctuation is small. The experimental results show that the results of the proposed road detection algorithm can achieve the expected results and are significantly better than the NT-RD method in terms of P, R, and F.


## 5.2. Vehicle Detection Results on Road

It is proposed to perform vehicle detection on the road based on the results of road detection. In the experiment, the vehicle thresholdtv is set to 5, and the vehicle aspect ratio threshold range is 0.5 to 2.Considering that binocular vision has not been used for accurate vehicle detection in related studies, this paper will separately explain the accuracy of the proposed vehicle detection algorithm from qualitative and quantitative aspects. The experiments are performed on the image pairs containing valid vehicles in three standard datasets. The Enpeda dataset contains 477 valid vehicles, the KITTI dataset contains 173 valid vehicles, and the Daimler dataset contains 462 valid vehicles in a time with more vehicles. Some typical vehicle detection results on the road are shown in Figure11.Figure 11
Typical vehicle detection result on the road. (1) Original image (left). (2) Region of interest. (3) Vehicle source point. (4) Initial detection result. (5) Detection result.
(a)(b)(c)(d)(e)(f)Among them, Figures11(a) and 11(b) are vehicle detection results of the Enpeda dataset, Figures 11(c) and 11(d) are vehicle detection results of the KITTI dataset, and Figures 11(e) and 11(f) are vehicle detection results of the Daimler dataset. From the qualitative detection result in the above figures, it can be seen that the proposed algorithm can accurately detect whether it is a single-car or multi-car scene. Vehicles failed to be detected in Figures 11(d) and 11(e) because the vehicles in Figure 11(e) are not included in the road detection results and the vehicles are far away in Figure 11(d); the body and background are integrated.Similarly, the results of the proposed vehicle detection algorithm are evaluated using Precision, Recall, and comprehensive evaluation indexF. The accuracy rate is for the prediction result, which indicates how many of the results detected as vehicles are real vehicles calibrated in the dataset; the recall rate is for the original sample, which indicates how many real vehicles calibrated in the dataset are accurately detected. The dark gray area indicates the type of dataset, and the light gray area indicates the number of real and effective vehicles in the three datasets and the actual number of vehicles detected by the algorithm. From these data, the accuracy rate, recall rate, and comprehensive evaluation index can be calculated. The calculation formulas are shown in (20)–(22) [37]:(20)P=TPTP+FP,(21)R=TPTP+FN,(22)F=2P⋅RP+R.Among them,TP is the true rate, that is, it is detected as a vehicle and it is indeed a vehicle; FP is a false positive rate, that is, it is detected as a vehicle, which is actually a nonvehicle object; TN is a true negative rate, that is, the vehicle is not detected but is actually a vehicle; and FN is a false negative rate, that is, the vehicle is not detected and is indeed a nonvehicle object. The detection results are shown in Table 2.Table 2
The vehicle detection result mixing matrix of three standard datasets.
Algorithm AssessmentEnpedaKITTIDaimlerRecall469182451Ground TruthEnpeda4774180087.63KITTI1730163094.21Daimler4620040287.01Precision89.1389.5689.1489.44From the above experimental results, it can be known that the vehicle detection on the road algorithm proposed in this paper can accurately measure the area where the vehicle is located and has better robustness under complex background. The algorithm only detects traffic objects in theROI region, reducing the range of possible vehicles, thereby reducing the calculation time, increasing the calculation speed, and meeting the real-time requirements. However, the algorithm still has missing detection and false detection. The main reasons are as follows: (1) there is a lot of noise and invalid areas in the disparity image. If such areas are large, incomplete correction will affect the detection results. (2) There are errors in road detection results, resulting in inaccurate ROI regions and missed detection of road vehicles. (3) There is not much difference in disparity and color similarity between the road near the vehicle and the vehicle, which affects the accurate division of the vehicle.


## 5.3. Pedestrian Detection Results on the Road

We select road scenes from typical datasets to verify the proposed pedestrian detection algorithm on the road. Experiments have determined that the range of the pedestrian’s aspect ratio thresholdRlw is set to 1.5–5, the perspective ratio Rp threshold is set to 0.12–0.45, and the area ratio Ra is less than 0.75. Some typical pedestrian detection results on the road are shown in Figures 12; Figures 12(a) and 12(c) are single pedestrian detection results, and Figures 12(b) and 12(d) are multiple pedestrian detection results. At the same time, the road scenes in Figures 12(b)–12(d) are more complicated, and there are more types of nonpedestrian obstacles.Figure 12
Typical pedestrian detection result on the road. (1) Original image (left). (2) Region of interest. (3) Pedestrian source point. (4) Pedestrian detection result. (5) Comparison.
(a)(b)(c)(d)From the above qualitative experimental results, it can be known that the pedestrian detection algorithm proposed in this paper can accurately measure the area where pedestrians are located and has better robustness under complex backgrounds.


## 5.4. Real-Time Analysis

The experiment in this paper uses a Windows 10 64 bit operating system, a CPU with Intel Core i5, 3.2 GHz, and a 32G memory PC as the experimental platform. The experimental environment is MATLAB 2017b.For real-time analysis, this section uses a random sampling method to randomly extract 100 pairs of images from the three standard datasets for labeling. The method for automatic detection and classification of traffic objects proposed in this paper can be summarized into five stages: (1) acquisition and correction of disparity images, (2) considerate U-V view, (3) road surface detection, (4) vehicle detection on the road, and (5) pedestrian detection on the road. Figure13 shows the average processing time for each stage of each frame in each dataset. The processing time for road detection, road vehicle detection, and pedestrian detection is calculated separately. It can be seen from Figure 13 that the total processing time of the KITTI dataset is the longest, which is about 99.8 ms (10pfs). It is mainly a road in the city and there are many objects in the scene. It takes a relatively long time in road detection, vehicle detection, and pedestrian detection on the road, and its image resolution is larger than that of the Enpeda and Daimler datasets. The least time consuming is the Enpeda dataset, with a time of about 59.4 ms (17pfs). This dataset is artificially synthesized, the target scene is relatively simple, and the pedestrian scene is lacking. The total processing time of the Daimler dataset is about 79.2 ms (13pfs). The scenes of roads, vehicles, and pedestrians on the road are rich, and the time is between the two datasets.Figure 13
The real-time analysis result of three standard datasets.The real-time analysis results show that, in the actual application process, a certain number of interval frames can be selected for processing, and the real-time requirements can be met without affecting the final results of the experiment.


## 6. Conclusions

(1)
The classification detection of roads, vehicles, and pedestrians with multiple traffic objects under the same detection framework is proposed, which makes up for the ambiguity and unity of traditional detection methods. At the same time, it is the first time to classify detection of vehicles and pedestrians on the road.(2)
The proposed road detection link is only based on a considerate U-V view method, using geometric characteristics without the need to build a complex road model, and the final V view can obtain accurate uphill and downhill road detection results without straight and curve fitting.(3)
The proposed method for detecting vehicles and pedestrians on the road carries the results of road detection, uses the proposed peripheral envelope algorithm to determine the contour of the road periphery, defines the source points of vehicles and pedestrians, combines the characteristics of vehicles and pedestrians, proposes multifeature and multithreshold fusion, and constructs disparity similarity and energy minimization algorithm to obtain the detection results of vehicles and pedestrians on the road.(4)
The effectiveness of the proposed method is tested on three standard datasets of Enpeda, KITTI, and Daimler, including different traffic environments, different road alignments, different vehicles, pedestrian distribution, and different road scenes. The comprehensive evaluation index F values for road detection are 97.54%, 95.03%, and 93.76%; the overall comprehensive evaluation index F value for vehicle detection on the road is 89.44%; the pedestrian detection on the road also obtains better detection results.(5)
The random sampling method is adopted to reflect the time efficiency of the entire dataset by the detection time of some frames of the dataset. The overall detection process is broken down into five stages to gradually determine the corresponding time, and the average time efficiency is 13 pfs, which can achieve real-time detection.



---

*Source: 2435793-2020-05-14.xml*