# An Obstacle Detection and Distance Measurement Method for Sloped Roads Based on VIDAR

Authors: Guoxin Jiang; Yi Xu; Xiaotong Gong; Shanshang Gao; Xiaoqing Sang; Ruoyu Zhu; Liming Wang; Yuqiong Wang
Journal: Journal of Robotics
 (2022)

---

## Abstract

Environmental perception systems can provide information on the environment around a vehicle, which is key to active vehicle safety systems. However, these systems underperform in cases of sloped roads. Real-time obstacle detection using monocular vision is a challenging problem in this situation. In this study, an obstacle detection and distance measurement method for sloped roads based on Vision-IMU based detection and range method (VIDAR) is proposed. First, the road images are collected and processed. Then, the road distance and slope information provided by a digital map is input into the VIDAR to detect and eliminate false obstacles (i.e., those for which no height can be calculated). The movement state of the obstacle is determined by tracking its lowest point. Finally, experimental analysis is carried out through simulation and real-vehicle experiments. The results show that the proposed method has higher detection accuracy than YOLO v5s in a sloped road environment and is not susceptible to interference from false obstacles. The most prominent contribution of this research work is to describe a sloped road obstacle detection method, which is capable of detecting all types of obstacles without prior knowledge to meet the needs of real-time and accurate detection of slope road obstacles.


---

## Body

## 1. Introduction

With increasing public attention to the field of traffic safety, the automobile industry is developing in the direction of intelligence, with many studies on autonomous driving by engineers and scientific researchers. Autonomous driving does not refer to a single technological field, but it is a product of the development and integration of automotive electronics, intelligent control, and breakthroughs related to the Internet of Things [1, 2]. The principle is that autonomous driving systems obtain information on the vehicle and the surrounding environment through an environmental perception system. Then, the information is analyzed and processed by the processor, and the obstacle information in front of the vehicle is detected. Combining with the vehicle dynamics model, the obstacle avoidance path planning and lateral control of the vehicle are realized [3–7].Environmental perception systems, which need to perform functions such as object classification, detection, segmentation, and distance estimation, have become a key component of autonomous vehicles. These systems can not only provide important traffic parameters for autonomous driving but also perceive surrounding obstacles, such as stationary or moving objects, including roadblocks, pedestrians, and other elements [8]. During the vehicle’s movement, radar (laser, millimeter wave), infrared and vision sensors are used to collect environmental information to determine whether a target is in a safe area [9–11]. However, the price of infrared sensors and radars is relatively high, and most of them are limited to advanced vehicles [12]. Compared with other sensor systems, monocular vision requires only one camera to capture images and analyze scenes, thereby reducing the cost of detection solutions. Moreover, the camera can work at a high frame rate and provide rich information from long distances under good lighting and favorable weather conditions [13]; therefore, detection methods based on machine vision are being more and more widely adopted.Machine learning can be used to achieve object classification for vision-based obstacle detection [14, 15]. However, traditional machine learning methods can only detect known types of obstacles (see Figure 1). If the vehicle cannot detect an unknown type of obstacles accurately, it is very likely that a traffic accident will occur. This situation is not conducive to the safe driving of the vehicle; therefore, in this study, we propose an unsupervised learning-based obstacle detection method, which allows the detection of both known- and unknown-type obstacles in complex environments.Figure 1
Fast R-CNN. Normal cars are detected, but the overturned car and the box are not detected.Traditional obstacle detection methods, such as motion compensation [16–18] and optical flow methods [19–22], allow the detection of obstacles of different shapes and at various speeds. However, these methods require the extraction and matching of a large number of object points, which increases the computational load. Therefore, in this study, we adopt a Vision-IMU (inertial measurement unit)-based detection and ranging method, abbreviated as VIDAR, which can realize fast matching and feature point processing of the detection area and improve the obstacle detection speed and detection effectiveness.VIDAR is an obstacle detection method developed for horizontal roads. When obstacles and test vehicles are located on different slopes, there will be imaging parallax, which will lead to the detection of false obstacles as real ones, resulting in a large measurement error, thereby affecting the detection accuracy. To cope with the impact of slope changes, in this study, we take the slope of road into account during the model establishment, and analyze the specific situation according to the position relationship between the detected vehicle and the obstacle. We thus propose an obstacle detection and distance measurement method for sloped roads based on VIDAR. In the proposed method, slope and distance information are provided by digital maps [23–26].The rest of this study is structured as follows: in Section2, we review the research on obstacle detection and visual ranging. In Section 3, the conversion process from world coordinates to camera coordinates and the ranging principle of VIDAR are introduced. In Section 4 the detection process of real obstacles on sloped roads is outlined and the ranging and speed measurement models are established. Simulated and real experiments are presented in Section 5 and the experimental results are compared with the detection results of YOLO v5s to demonstrate the detection accuracy of the proposed method. In Section 6, the proposed method and our findings are summarized, and the study is concluded.


## 2. Related Work

Obstacle detection still forms one of the most significant research foci in the development of intelligent vehicles. With the improvement and optimization of monocular vision, obstacle detection based on monocular vision has attracted the attention of researchers. Most of the research on the detection of obstacles using monocular vision is based on the optimization of machine vision and digital image processing to improve the accuracy and speed of detection. S. Wang proposed a novel image classification framework that integrates a convolutional neural network (CNN) and a kernel extreme learning machine to distinguish the categories of extracted features, thus improving the performance of image classification [27]. Nguyen proposed an improved framework based on fast response neural network (Fast R-CNN). The basic convolution layer of Fast R-CNN was formed using the MobileNet architecture, and the classifier was formed using the deep separable convolution structure of the MobileNet architecture, which improved the accuracy of vehicle detection [28]. Yi proposed the improved YOLO v3 neural network model, which introduced the concept of Faster R-CNN’s anchor box, and used a multiscale strategy, thus greatly improving the robustness of the network in small object detection [29]. Wang K.W. proposed an efficient fully convolutional neural network, which could predict the occluded part of the road by analyzing foreground objects and the existing road layout, thereby improving the performance of the neural network [30]. Although the above methods improved the accuracy of obstacle detection, they require a large number of sample data for network training and the range of samples must cover all obstacle types; otherwise, the obstacles cannot be detected.Monocular ranging pertains to the use of a single camera to capture images and perform distance calculations. Zhang et al. used a stereo camera system to compute a disparity map and use it for obstacle detection. They applied different computer vision methods to filter the disparity map and remove noise in detected obstacles, and a monocular camera in combination with the histogram of oriented gradients and support vector machine algorithms to detect pedestrians and vehicles [31]. Tkocz studied the ranging and positioning of a robot in motion, considering the scale ambiguity of monocular cameras. However, only experimental research has been done on the speed and accuracy of measurement [32]. Meng C designed a distance measurement system based on a fitting method, where a linear relationship between the pixel value and the real distance is established according to the pixel position of the vehicle in the imaging plane coordinate, thus realizing adaptive vehicle distance measurement under monocular vision [33]. Zhe proposed a method for detecting vehicles ahead, which combined machine learning and prior knowledge to detect vehicles based on the horizontal edge of the candidate area [34]. These methods were only used for the measurement of distance to other vehicles and are not applicable to other types of obstacles.Rosero proposed a method for sensor calibration and obstacle detection in an urban environment. The data from a radar, 3D LIDAR, and stereo camera sensors were fused together to detect obstacles and determine their shape [35]. Garnett used a radar to determine the approximate location of obstacles, and then used bounding box regression to achieve accurate positioning and identification [36]. Caltagirone proposed a novel LIDAR-camera fusion fully convolutional network and achieved the most advanced performance on the KITTI road benchmark [37]. Although sensor fusion methods reduce the processing load and achieve improved detection accuracy, these methods are based on flat roads and are not suitable for complex slope road environments.To solve the above problems, we propose an obstacle detection and distance measurement method for sloped roads based on VIDAR. This method does not require a priori knowledge of the scene and uses the road slope information provided by a digital map and the vehicle driving state provided by an IMU to construct distance measurement and speed measurement models, which allow the detection of obstacles in real time, as well as the distance and movement state of the obstacles.


## 3. Methodology

The obstacle detection model of VIDAR is based on pinhole camera model, which can accurately calculate the distance between vehicles and obstacles.


### 3.1. Coordinate Transformation

The camera can map the coordinate points of the three-dimensional world to the two-dimensional imaging plane. This imaging principle is consistent with the pinhole model principle, so camera imaging can be described by pinhole model.If we want to determine the correspondence between the object point and the image point, we must establish the coordinate system needed by vision system, including world coordinate system, camera coordinate system, imaging plane coordinate system, and pixel coordinate system. The transformation process from the world coordinate system to the pixel coordinate system is shown in Figure2.Figure 2
Transformation between coordinate systems.Pixel coordinateu,v and image plane coordinate x,y are on the same plane, and the X and Y axes are parallel. The corresponding position of the original point in the image plane coordinate system is u0,v0. Both the world and the camera coordinate systems are 3D coordinates, which are associated through the camera. According to the principle of keyhole imaging, the camera coordinate system can be obtained through a transformation of the coordinate axes of the world coordinate system, so the conversion relation between the two coordinate systems must be deduced. The conversion equation from the world to the pixel coordinate system is shown in(1)ZC=μv1=1/dx0μ001/dyv0001f000f0001RXwYwZw+T=ax0μ00ayv0001XwRYwZw+T,where R and T are the external parameters. The internal and external parameters can be obtained through camera calibration.


### 3.2. Obstacle Ranging Method

The obstacle ranging principle is also based on the pinhole model principle. For the convenience of expression, we installed the camera on a test vehicle and a vehicle on a sloped road was regarded as the obstacle. The feature points of the obstacle were detected, and the lowest point was taken as the intersection point between the obstacle and the road surface (see Figure3). In the case of normal detection by the system, the camera collects image information, and by processing the image information, feature points in the image can be extracted. By measuring the distance of the feature point, it can be determined whether the obstacle where the feature point is located has a height. For real obstacles, tracking the feature point at the lowest position can calculate the moving speed of the obstacle, judge the motion state of the obstacle, and provide data support for the safe driving of the vehicle. As long as the camera can capture images normally, all obstacles in the captured scene can be detected. The number of detected obstacles is related to the number of extracted feature points.Figure 3
Schematic diagram of obstacle ranging model (in order to visualize the detection principle, the nonreal proportional relationship is shown in the figure).Letf be the effective focal length of the camera, ∂ be the pitch angle, μ be the pixel size, h be the mounting height of the camera and the camera center be the optical center of the lens. Let x0,y0 be the coordinate origin of imaging plane coordinate system, and x,y be the intersection coordinate of the obstacle and the road plane in the image plane coordinate system. The horizontal distance between the camera and the obstacle can be obtained using(2)d=htanϑ+arctany0−yμ/f.


## 3.1. Coordinate Transformation

The camera can map the coordinate points of the three-dimensional world to the two-dimensional imaging plane. This imaging principle is consistent with the pinhole model principle, so camera imaging can be described by pinhole model.If we want to determine the correspondence between the object point and the image point, we must establish the coordinate system needed by vision system, including world coordinate system, camera coordinate system, imaging plane coordinate system, and pixel coordinate system. The transformation process from the world coordinate system to the pixel coordinate system is shown in Figure2.Figure 2
Transformation between coordinate systems.Pixel coordinateu,v and image plane coordinate x,y are on the same plane, and the X and Y axes are parallel. The corresponding position of the original point in the image plane coordinate system is u0,v0. Both the world and the camera coordinate systems are 3D coordinates, which are associated through the camera. According to the principle of keyhole imaging, the camera coordinate system can be obtained through a transformation of the coordinate axes of the world coordinate system, so the conversion relation between the two coordinate systems must be deduced. The conversion equation from the world to the pixel coordinate system is shown in(1)ZC=μv1=1/dx0μ001/dyv0001f000f0001RXwYwZw+T=ax0μ00ayv0001XwRYwZw+T,where R and T are the external parameters. The internal and external parameters can be obtained through camera calibration.


## 3.2. Obstacle Ranging Method

The obstacle ranging principle is also based on the pinhole model principle. For the convenience of expression, we installed the camera on a test vehicle and a vehicle on a sloped road was regarded as the obstacle. The feature points of the obstacle were detected, and the lowest point was taken as the intersection point between the obstacle and the road surface (see Figure3). In the case of normal detection by the system, the camera collects image information, and by processing the image information, feature points in the image can be extracted. By measuring the distance of the feature point, it can be determined whether the obstacle where the feature point is located has a height. For real obstacles, tracking the feature point at the lowest position can calculate the moving speed of the obstacle, judge the motion state of the obstacle, and provide data support for the safe driving of the vehicle. As long as the camera can capture images normally, all obstacles in the captured scene can be detected. The number of detected obstacles is related to the number of extracted feature points.Figure 3
Schematic diagram of obstacle ranging model (in order to visualize the detection principle, the nonreal proportional relationship is shown in the figure).Letf be the effective focal length of the camera, ∂ be the pitch angle, μ be the pixel size, h be the mounting height of the camera and the camera center be the optical center of the lens. Let x0,y0 be the coordinate origin of imaging plane coordinate system, and x,y be the intersection coordinate of the obstacle and the road plane in the image plane coordinate system. The horizontal distance between the camera and the obstacle can be obtained using(2)d=htanϑ+arctany0−yμ/f.


## 4. Research Approach

In the traditional VIDAR model, it is assumed that the test vehicle and obstacles are on the same plane. However, when the test vehicle and the obstacles are on roads with different slopes, this will cause a deviation of the distance measurement. In order to enhance the visual detection accuracy and expand the visual ranging application scenarios, in this study, we take the slope into account and establish an obstacle detection model for the sloped road.


### 4.1. Establishment of the Distance Measurement Model

The sloped road mentioned in this study refers to a road where the test vehicle and the obstacles are not on the same slope. When measuring distance, the above situation can be simplified into two models.The distance model between the camera and obstacles on a sloped road with obstacles in front of the test vehicle are shown in Figure4. Let the light blue line be the auxiliary line, and the red dot on the obstacle be any detected object point.Figure 4
Diagram of distance models. (a) Situation 1: the test vehicle on a flat road and obstacles on an uphill road. (b) Situation 2: the test vehicle on a flat road and obstacles on a downhill road.
(a)(b)LetC′ be a point on the road’s surface, C be the image point of C′ on the sloped road’s surface, C″ be the intersection point where CC′ extends to the imaginary horizontal plane, and S′ be the distance from the camera to the beginning of road slope change. Let di be the horizontal distance between the camera and C″, and dii be the horizontal distance between the camera and C′.Using triangle similarity, equation (3) can be obtained through the geometric relationships shown in Figure 4:(3)h1dii−S′=tanα,h+kh1dii=tanθi,θi=∂+arctany0−yiμf.The expression ofdii is further derived by(4)dii=S′∗tanα+khtanα+ktanθi.When the slope of the road where the obstacle is located is larger than that of the test vehicle,k=−1. In the opposite case, k=1.


### 4.2. Determination of the Real Obstacle on Sloped Roads

In the process of the test vehicle’s movement, road images were collected twice. The imaging diagram of the stationary obstacles is shown in Figure5. Let A′ and B′ be points on the road surface, and A and B be the corresponding image points. The first point of the obstacle on the image plane is A. As the camera moves with the test vehicle, and the y axis on the image plane moves from axis y1 to the axis y2, we obtain the point B of the obstacle on the image plane. A″ is the intersection, where AA′ extends to the imaginary horizontal plane, and accordingly for B″. Δd is the movement distance of the camera (i.e., the test vehicle), d1 is the horizontal distance from the camera to A″, d2 is the same, d11 is the horizontal distance from the camera to A′, and accordingly for d22.Figure 5
Schematic diagram of stationary obstacle imaging. (a) Situation 1: the test vehicle moving on a flat road and stationary obstacle on an uphill road. (b) Situation 2: the test vehicle moving on a flat road and stationary obstacle on a downhill road.
(a)(b)d11 and d22 can be calculated using equation (4). The relationship between d11 and d22 can be approximated as d11=d22+Δd, but the real relationship is d11=d22+Δd+Δl. If d11≠d22+Δd, the object points are not on the road surface. Using this method, it can be determined whether the obstacle has a height (i.e., it is a real obstacle).


### 4.3. Special Case of Obstacle Detection

A special case should be excluded during obstacle detection. When the test vehicle and the obstacles are moving at the same time, the imaging point of the camera light on the road surface through an object point of the obstacle coincides with each other. VIDAR is unable to detect obstacles in this case.The diagrams of obstacle detection in complex environments are shown in Figure6. Let LA be the distance (along the road where the obstacle is located) between the highest point of the obstacle and the object point of the road surface when the test vehicle is moving for the first time. Similarly, LB is the distance when the vehicle moves for the second time. The letters in Figure 6 have the same meaning as the letters above.Figure 6
Schematic diagram of obstacles and camera imaging in complex environments, (a) Situation 1: Test vehicle driving on a flat road and dynamic obstacles on an uphill road, (b) Situation 2: Test vehicle driving on a flat road and dynamic obstacles on a downhill road.
(a)(b)Let the speeds of the detecting vehicles and obstacles bev and v′, respectively. When the imaging point of the road’s intersection point and the obstacle’s object point passes through the camera, the relationships between hv, v, v′, LA and LB are as follows:(5)hvtanα+bθ1=LA,hvtanα+bθ2=LB,LA−LB=v′t,Δd=vt.When the slope of the road where the obstacle is located is larger than that of the test vehicle,b=1, while b=−1 in the opposite case.Therefore, VIDAR can be used in all cases except whenLA−LB=v′t, and v′=v/Δdhv/tanα+bθ1−hv/tanα+bθ2. Therefore, the proposed method using a monocular camera to detect obstacles on sloped roads is convenient and feasible. The detection process only includes tracking and calculating the position of the object point, which can shorten the detection time and reduce computational resource consumption.


### 4.4. Speed Measuring Model of the Sloped Road Obstacle

Obstacles can be imaged in the camera photosensitive element. By extracting and calculating the feature points of the collected obstacle images, we can calculate the feature points that are not on the road surface, that is, the feature points whose height is not zero. The object points with nonzero height are morphologically processed to obtain the obstacles’ areas. The movement state of the obstacles can be determined through tracking and calculating the speed of the lowest point.When the test vehicle is moving, the obstacles, camera, and the lowest point of the road will form images (see Figure7). At this time, the horizontal distance between the lowest point of the obstacle and the camera can be expressed as dii.Figure 7
Schematic diagram of the camera and the lowest point on the road. (a) Situation 1: the test vehicle moving on a flat road and dynamic obstacles on an uphill road. (b) Situation 2: the test vehicle moving on a flat road and dynamic obstacles on a downhill road.
(a)(b)LetA be the image plane point corresponding to the lowest point of the obstacle at time t and B corresponding point at t+Δt. The relationship between v′d11, d22 and Δd is as follows:(6)d11−d22+Δdcosα=v′⋅Δt,where Δd=v∗Δt, with v being the speed of the test vehicle. When d11=d22+v∗Δt, the obstacle is stationary; otherwise, it is moving with a speed of(7)v′=d11−d22−v⋅ΔtΔt∗cosα.


### 4.5. Obstacle Detection on Sloped Roads Using VIDAR

In this study, an obstacle detection and distance measurement method for sloped roads based on VIDAR is proposed, which can quickly judge and eliminate false obstacles that without height, and at the same time identify real obstacles and judge their movement state. The detection process is as follows (see Figure8).Figure 8
Flow chart of slope road obstacle detection method based on VIDAR (see the following for specific steps).Step 1.
Update camera parameters using the IMU:(1)
Calibration of the camera’s initial internal and external parameters: the camera’s parameters, such as the focal lengthf, mounting height h, pixel μ, and pitch angle ∂ are obtained through calibration.(2)
Data acquisition: the camera is used to collect images and the IMU is used to collect inertial data. The acquisition frequency of the IMU is larger than that of the camera.(3)
Update of camera parameters: the frequency relationship between the IMU and camera is established and the camera parameters at timet are calculated periodically according to the inertial data.Step 2.
Obtain the road information.
Acquire the road slopeα and the distance S from the test vehicle to the sloped road using the digital map.Step 3.
Regional background extraction:(1)
Two consecutive images are taken as the total obstacle detection areaBi during the running of the test vehicle (see Figures 9(a) and 9(b)).(2)
The lane line is detected and the image within the lane line is extracted asGi.(3)
Machine learning is used to process the images, detect and classify specific types of obstacles. The area setFi of known types of obstacles is obtained, where Fi=f1,f2,…,fk, and k is the number of known obstacles.(4)
The known obstacle areaFi in the total obstacle detection area Gi is eliminated and the background area Ni (Ni=Gi−Fi) is extracted as the VIDAR data to be detected.Figure 9
Obstacle detection based on VIDAR and digital map. (a) Two images collected during the movement of the vehicle. (b) Lane line detection on the image and extraction of the detection range. (c) Feature point detection and matching on the images within the range. (d) Extraction of feature points. (e) Determination of whether the extracted feature points have height. (f) Removal of feature points without height. (g) Morphological processing on feature points. (h) Tracking of the lowest point of obstacles.Step 4.
Image processing and obstacle detection:(1)
Object points are extracted from the background areasNi and Ni+1 of two consecutive images. With Ni as the background region template map and Ni+1 as the background region real-time map, the matching regions Mi and Mi+1 are obtained using a fast image region matching method based on region feature extraction, as shown in Figure 9(c).(2)
The object points setPi of matching area Mi+1 is extracted, as shown in Figure 9(d).(3)
The distance between the test vehicle and the object point is calculated. The horizontal distance between the camera and the imaged object point on the imaginary road isdi=h/tanθi. The horizontal distance between the camera and the imaged object point on the real road is dii=S′⋅tanα+kh/tanα+ktanθi. The calculation process of dii is shown in Figure 10. First, the pixel coordinates of object points are obtained through the transformation of the coordinate axes. Then the slope information is obtained through Step 2, and finally the distance is obtained through the ranging model.(4)
The object points with height in setPi are extracted (see Figure 9(e)). Calculate dii and di+1i+1 as the vehicle is moving continuously. If dii=di+1i+1+Δd, then the object points are on the road surface (without height), so the object points pij are eliminated. If dii≠di+1i+1+Δd, then the object points are not on the road surface (i.e., they have a nonzero height). The object points are extracted to obtain the object point set pi′.(5)
Morphological processing is applied to the image of the object point setpi′ (Figure 9(f)). The target image is Ei, and the structural element is B, which is used to apply a closing operation on Ei and obtain C connected regions. The real obstacle region Oi is thus obtained, where Oi=Ei⋅Bi=Ei⊕Bi⊖Bi, and Oi=oi,1,oi,2,…,oi,c.(6)
Edge detection of real obstacles, shown in Figure9(g).(7)
According to the detection result of (6), the lowest object point of each obstacle area is extracted, as shown in Figure 9(h). The lowest object point set Pi″ constitutes the obstacle area.(8)
Each object point inPi″ is tracked during the movement of the test vehicle.(9)
Get the movement state of the obstacles is obtained. The movement speed of the obstacle where the object point is located can be obtained by tracking each object point inPi″. If di+1i+1−di+2i+2=vi+1⋅Δt, the obstacle on which these object points are located is static. If di+1i+1−di+2i+2≠vi+1⋅Δt, the obstacle is moving with an instantaneous speed vi+1′=di+1i+1−di+2i+2−vi+1⋅Δt/Δt∗cosαi.
The proposed obstacle detection method can be used to detect real obstacles in complex environments and determine their movement state, which is beneficial for vehicles to take timely measures and avoid accidents.Figure 10
Flow chart of horizontal distance calculation.


## 4.1. Establishment of the Distance Measurement Model

The sloped road mentioned in this study refers to a road where the test vehicle and the obstacles are not on the same slope. When measuring distance, the above situation can be simplified into two models.The distance model between the camera and obstacles on a sloped road with obstacles in front of the test vehicle are shown in Figure4. Let the light blue line be the auxiliary line, and the red dot on the obstacle be any detected object point.Figure 4
Diagram of distance models. (a) Situation 1: the test vehicle on a flat road and obstacles on an uphill road. (b) Situation 2: the test vehicle on a flat road and obstacles on a downhill road.
(a)(b)LetC′ be a point on the road’s surface, C be the image point of C′ on the sloped road’s surface, C″ be the intersection point where CC′ extends to the imaginary horizontal plane, and S′ be the distance from the camera to the beginning of road slope change. Let di be the horizontal distance between the camera and C″, and dii be the horizontal distance between the camera and C′.Using triangle similarity, equation (3) can be obtained through the geometric relationships shown in Figure 4:(3)h1dii−S′=tanα,h+kh1dii=tanθi,θi=∂+arctany0−yiμf.The expression ofdii is further derived by(4)dii=S′∗tanα+khtanα+ktanθi.When the slope of the road where the obstacle is located is larger than that of the test vehicle,k=−1. In the opposite case, k=1.


## 4.2. Determination of the Real Obstacle on Sloped Roads

In the process of the test vehicle’s movement, road images were collected twice. The imaging diagram of the stationary obstacles is shown in Figure5. Let A′ and B′ be points on the road surface, and A and B be the corresponding image points. The first point of the obstacle on the image plane is A. As the camera moves with the test vehicle, and the y axis on the image plane moves from axis y1 to the axis y2, we obtain the point B of the obstacle on the image plane. A″ is the intersection, where AA′ extends to the imaginary horizontal plane, and accordingly for B″. Δd is the movement distance of the camera (i.e., the test vehicle), d1 is the horizontal distance from the camera to A″, d2 is the same, d11 is the horizontal distance from the camera to A′, and accordingly for d22.Figure 5
Schematic diagram of stationary obstacle imaging. (a) Situation 1: the test vehicle moving on a flat road and stationary obstacle on an uphill road. (b) Situation 2: the test vehicle moving on a flat road and stationary obstacle on a downhill road.
(a)(b)d11 and d22 can be calculated using equation (4). The relationship between d11 and d22 can be approximated as d11=d22+Δd, but the real relationship is d11=d22+Δd+Δl. If d11≠d22+Δd, the object points are not on the road surface. Using this method, it can be determined whether the obstacle has a height (i.e., it is a real obstacle).


## 4.3. Special Case of Obstacle Detection

A special case should be excluded during obstacle detection. When the test vehicle and the obstacles are moving at the same time, the imaging point of the camera light on the road surface through an object point of the obstacle coincides with each other. VIDAR is unable to detect obstacles in this case.The diagrams of obstacle detection in complex environments are shown in Figure6. Let LA be the distance (along the road where the obstacle is located) between the highest point of the obstacle and the object point of the road surface when the test vehicle is moving for the first time. Similarly, LB is the distance when the vehicle moves for the second time. The letters in Figure 6 have the same meaning as the letters above.Figure 6
Schematic diagram of obstacles and camera imaging in complex environments, (a) Situation 1: Test vehicle driving on a flat road and dynamic obstacles on an uphill road, (b) Situation 2: Test vehicle driving on a flat road and dynamic obstacles on a downhill road.
(a)(b)Let the speeds of the detecting vehicles and obstacles bev and v′, respectively. When the imaging point of the road’s intersection point and the obstacle’s object point passes through the camera, the relationships between hv, v, v′, LA and LB are as follows:(5)hvtanα+bθ1=LA,hvtanα+bθ2=LB,LA−LB=v′t,Δd=vt.When the slope of the road where the obstacle is located is larger than that of the test vehicle,b=1, while b=−1 in the opposite case.Therefore, VIDAR can be used in all cases except whenLA−LB=v′t, and v′=v/Δdhv/tanα+bθ1−hv/tanα+bθ2. Therefore, the proposed method using a monocular camera to detect obstacles on sloped roads is convenient and feasible. The detection process only includes tracking and calculating the position of the object point, which can shorten the detection time and reduce computational resource consumption.


## 4.4. Speed Measuring Model of the Sloped Road Obstacle

Obstacles can be imaged in the camera photosensitive element. By extracting and calculating the feature points of the collected obstacle images, we can calculate the feature points that are not on the road surface, that is, the feature points whose height is not zero. The object points with nonzero height are morphologically processed to obtain the obstacles’ areas. The movement state of the obstacles can be determined through tracking and calculating the speed of the lowest point.When the test vehicle is moving, the obstacles, camera, and the lowest point of the road will form images (see Figure7). At this time, the horizontal distance between the lowest point of the obstacle and the camera can be expressed as dii.Figure 7
Schematic diagram of the camera and the lowest point on the road. (a) Situation 1: the test vehicle moving on a flat road and dynamic obstacles on an uphill road. (b) Situation 2: the test vehicle moving on a flat road and dynamic obstacles on a downhill road.
(a)(b)LetA be the image plane point corresponding to the lowest point of the obstacle at time t and B corresponding point at t+Δt. The relationship between v′d11, d22 and Δd is as follows:(6)d11−d22+Δdcosα=v′⋅Δt,where Δd=v∗Δt, with v being the speed of the test vehicle. When d11=d22+v∗Δt, the obstacle is stationary; otherwise, it is moving with a speed of(7)v′=d11−d22−v⋅ΔtΔt∗cosα.


## 4.5. Obstacle Detection on Sloped Roads Using VIDAR

In this study, an obstacle detection and distance measurement method for sloped roads based on VIDAR is proposed, which can quickly judge and eliminate false obstacles that without height, and at the same time identify real obstacles and judge their movement state. The detection process is as follows (see Figure8).Figure 8
Flow chart of slope road obstacle detection method based on VIDAR (see the following for specific steps).Step 1.
Update camera parameters using the IMU:(1)
Calibration of the camera’s initial internal and external parameters: the camera’s parameters, such as the focal lengthf, mounting height h, pixel μ, and pitch angle ∂ are obtained through calibration.(2)
Data acquisition: the camera is used to collect images and the IMU is used to collect inertial data. The acquisition frequency of the IMU is larger than that of the camera.(3)
Update of camera parameters: the frequency relationship between the IMU and camera is established and the camera parameters at timet are calculated periodically according to the inertial data.Step 2.
Obtain the road information.
Acquire the road slopeα and the distance S from the test vehicle to the sloped road using the digital map.Step 3.
Regional background extraction:(1)
Two consecutive images are taken as the total obstacle detection areaBi during the running of the test vehicle (see Figures 9(a) and 9(b)).(2)
The lane line is detected and the image within the lane line is extracted asGi.(3)
Machine learning is used to process the images, detect and classify specific types of obstacles. The area setFi of known types of obstacles is obtained, where Fi=f1,f2,…,fk, and k is the number of known obstacles.(4)
The known obstacle areaFi in the total obstacle detection area Gi is eliminated and the background area Ni (Ni=Gi−Fi) is extracted as the VIDAR data to be detected.Figure 9
Obstacle detection based on VIDAR and digital map. (a) Two images collected during the movement of the vehicle. (b) Lane line detection on the image and extraction of the detection range. (c) Feature point detection and matching on the images within the range. (d) Extraction of feature points. (e) Determination of whether the extracted feature points have height. (f) Removal of feature points without height. (g) Morphological processing on feature points. (h) Tracking of the lowest point of obstacles.Step 4.
Image processing and obstacle detection:(1)
Object points are extracted from the background areasNi and Ni+1 of two consecutive images. With Ni as the background region template map and Ni+1 as the background region real-time map, the matching regions Mi and Mi+1 are obtained using a fast image region matching method based on region feature extraction, as shown in Figure 9(c).(2)
The object points setPi of matching area Mi+1 is extracted, as shown in Figure 9(d).(3)
The distance between the test vehicle and the object point is calculated. The horizontal distance between the camera and the imaged object point on the imaginary road isdi=h/tanθi. The horizontal distance between the camera and the imaged object point on the real road is dii=S′⋅tanα+kh/tanα+ktanθi. The calculation process of dii is shown in Figure 10. First, the pixel coordinates of object points are obtained through the transformation of the coordinate axes. Then the slope information is obtained through Step 2, and finally the distance is obtained through the ranging model.(4)
The object points with height in setPi are extracted (see Figure 9(e)). Calculate dii and di+1i+1 as the vehicle is moving continuously. If dii=di+1i+1+Δd, then the object points are on the road surface (without height), so the object points pij are eliminated. If dii≠di+1i+1+Δd, then the object points are not on the road surface (i.e., they have a nonzero height). The object points are extracted to obtain the object point set pi′.(5)
Morphological processing is applied to the image of the object point setpi′ (Figure 9(f)). The target image is Ei, and the structural element is B, which is used to apply a closing operation on Ei and obtain C connected regions. The real obstacle region Oi is thus obtained, where Oi=Ei⋅Bi=Ei⊕Bi⊖Bi, and Oi=oi,1,oi,2,…,oi,c.(6)
Edge detection of real obstacles, shown in Figure9(g).(7)
According to the detection result of (6), the lowest object point of each obstacle area is extracted, as shown in Figure 9(h). The lowest object point set Pi″ constitutes the obstacle area.(8)
Each object point inPi″ is tracked during the movement of the test vehicle.(9)
Get the movement state of the obstacles is obtained. The movement speed of the obstacle where the object point is located can be obtained by tracking each object point inPi″. If di+1i+1−di+2i+2=vi+1⋅Δt, the obstacle on which these object points are located is static. If di+1i+1−di+2i+2≠vi+1⋅Δt, the obstacle is moving with an instantaneous speed vi+1′=di+1i+1−di+2i+2−vi+1⋅Δt/Δt∗cosαi.
The proposed obstacle detection method can be used to detect real obstacles in complex environments and determine their movement state, which is beneficial for vehicles to take timely measures and avoid accidents.Figure 10
Flow chart of horizontal distance calculation.


## 5. Experiment and Evaluation

The proposed method can be used for obstacle detection in complex environments with improved accuracy, as well as distance and speed measurement of obstacles. Obstacle detection and distance measurement were realized in Matlab, whereas all experiments were performed on a desktop PC with the Intel(R) Xeon(R) Silver 4210 CPU.


### 5.1. Simulation Experiment

In this study, experimental equipment was used to simulate a detection environment so as to verify the detection effect of obstacles on sloped roads based on VIDAR. The experimental equipment included: a test vehicle equipped with an OV5640 camera unit and a JY61p IMU (Figure11(a)), vehicle scale models (Figure 11(b)), bottle caps and paper (Figure 11(c)), and simulated sloped road (Figure 11(d)). Among them, the test vehicle was used to analyze the road environment and detect its own driving state, scaled vehicle models were used to simulate known obstacles, while the bottle caps and paper were used to simulate unknown obstacles. The road slope was set to 13°.Figure 11
Equipment for simulating experiment. (a) Test vehicle. (b) Vehicle scale models. (c) Bottle caps and paper. (d) Simulated sloping road.
(a)(b)(c)(d)The bottle cap was taken as the real obstacle of unknown type, and the paper pasted on the simulated road was taken as the pseudoobstacle of unknown type. The angular acceleration and acceleration data of the vehicle were obtained by the IMU installed by the vehicle. Quaternion method is used to solve the camera attitude, and the pitching angle of the camera is updated. The velocity data are used to calculate the horizontal distance between the vehicle and the obstacle. The height of the obstacle is calculated by the change of the distance before and after the movement, so as to determine whether the detected obstacle is a real obstacle. The video collected using the OV5640 camera comprised an image sequence at 12FPS, which was used for obstacle detection. The results obtained using the original VIDAR and VIDAR on sloped roads are shown in Figure12, while the test results of the simulation experiment are summarized in Table 1.Figure 12
Comparison of obstacle detection effect in simulated environment. (a) Original VIDAR. (b) VIDAR on sloped roads.
(a)(b)Table 1
Test results of the simulation experiment.
ObstacleThei th movement of the test vehicleDetection distance (cm)The test vehicle moving distanceWhether it has heightWhether to exclude1116.53NoYes219.52118.53YesNo221.23124.53YesNo227.0The test results of the simulation experiment is shown in Table1.It can be seen from Figure12 that the original VIDAR can detect unknown types of obstacles such as bottle caps, but it will detect false obstacles as real obstacles, resulting in low accuracy of obstacle detection. However, the obstacle detection method for sloping roads based on VIDAR can eliminate false obstacles, which makes up for the wrong detection of unknown types of obstacles on sloping roads; therefore, compared with the original VIDAR, our proposed method can detect obstacles more accurately.


### 5.2. Real Environment Experiment

In the real environment, purely electric vehicles were used as test vehicles (see Figure13). As a sensor, the camera can adapt to complex environments and collect environmental information in real time (we only used the left camera). The camera was installed at a height of 1.60 m. The IMU used for locating the test vehicle and reading of its movement state in real time was installed at the bottom of the test vehicle. GPS was used for accurate location positioning. Through the combination of GPS and IMU, the real-time position information of the test vehicles and obstacles can be obtained, and then the trajectory information of vehicles and obstacles can be obtained. A digital map was used to obtain accurate road information such as distance and slope. A calculation unit was used to process the data in real time.Figure 13
Schematic diagram of the test vehicle.Accurate calibration of camera parameters was a prerequisite for the whole experiment and is a very important task for obstacle detection methods. In this paper, Zhang Zhengyou’s camera calibration method was adopted to calibrate the DaYing camera. First, the camera was fixed to capture images of a checkerboard at different positions and angles. Then, the key points of the checkerboard were selected and used to establish a relationship equation. Finally, the internal parameter calibration was realized. The camera calibration result is shown in Figure14.Figure 14
Schematic diagram of camera calibration results.Camera distortion includes radial distortion, thin lens distortion, and centrifugal distortion. The superposition of the three kinds of distortion results in a nonlinear distortion, the model of which can be expressed in the image coordinate system as follows:(8)δxx,y=s1xx2+y2+2p1xy+p2y3+k1xx2+y2δyx,y=s2yx2+y2+2p2xy+p1y3+k1xx2+y2,where s1 and s2 are centrifugal distortion coefficients; k1 and k2 are radial distortion coefficients, and p1 and p2 are the distortion coefficients of thin lenses.Because the centrifugal distortion of the camera is not considered in this study, the internal reference matrix of the camera can be expressed as shown in(9)M=5.9774e+030949.884305.9880e+03357.0539001.The calibration of the camera’s external parameters can be calculated by taking the edge object points of lane lines. The calibration results are shown in Table2.Table 2
Calibration results of camera external parameters.
External parameter typeParameter sizePitch angle1.20Yaw angle3.85Rotation angle2.37Since the images in the public data set were all data collected by other cameras, different camera parameters will affect the accuracy of ranging, so we used the VIDAR-Slope database (Figure15), the images in which where collected using a DaYing camera. The collection frequency was 77 frames/min, and there are 2270 images in total. The experiment and image collection took place in Shandong University of Technology’s driving school and experimental building. We selected the downhill section of the parking lot for the experiment. In the process of obstacle detection, the test vehicle moves at a constant speed of 25 km/h.Figure 15
The VIDAR-Slope database (part of data).The detection results of YOLO v5s and the method proposed in this study are shown in Figure16. The accuracy of obstacle detection was measured through the number of true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). Let a be an obstacle that is correctly classified as a positive example, b be an obstacle that is wrongly classified as a positive example, c be an obstacle that is correctly classified as a negative example, and d be an obstacle that is incorrectly identified as a negative example. Then, TP=∑i=1nai, FP=∑i=1nbi, TN=∑i=1nci, FN=∑i=1ndi.Figure 16
Comparison of partial test results of YOLO v5s and proposed method. (a) YOLO v5s can only detect known type obstacles. (b) The obstacle method proposed in this article can detect pedestrians (known types) and boxes (unknown types).
(a)(b)The YOLO series is a representative target detection framework based on deep learning. There are four versions of the target detection network: namely YOLO v5s, YOLO v5m, YOLO v5l, and YOLO v5x. Among them, YOLO v5s is the smallest and has the fastest speed, so we choose it for comparative experiments.Comparing the two methods, it can be seen that the stability of the proposed method is higher than that of YOLO v5s. YOLO v5s lacks training in unknown types of obstacles, and will consequently offer reduced safety when used in realistic vehicle situations. However, the proposed obstacle detection method does not require training and can detect all types of obstacles, thus ensuring its effectiveness of obstacle detection results on sloped roads. The total number of obstacles in the target area in the VIDAR-Slope database was 9526. The results of YOLO v5s and proposed method are shown in Table3.Table 3
Obstacle detection results in sloped roads of YOLO v5s and the proposed method.
Detection methodInput valueTPFPTNFNYOLO v5s9526671628089151214Proposed method95269124402530485In the results’ analysis, Accuracy (A), Recall (R), and Precision (P) were used as evaluation indices for the two obstacle detection methods, calculated through the following equations:(10)A=TP+TNTP+TN+FP+FN,(11)R=TPTP+FN,(12)P=TPTP+FP.The Accuracy, Recall, and Precision of YOLO v5s and the method proposed in this study are shown in Table4.Table 4
Evaluation indices of YOLO v5s and the proposed method.
Detection methodA (%)R (%)P (%)YOLO v5s81.7384.6870.50Proposed method90.3694.9495.77The experimental results in Tables3 and 4 show that due to vehicle fluctuations and other factors, misjudgment or misdetection may occur during vehicle movement. Compared with YOLO v5s, the accuracy of the obstacle detection method proposed in this study is increased by 8% and its precision is increased by 26.4%, which demonstrates its improved obstacle detection capability on sloped roads.In terms of detection accuracy, we also compared our method with other commonly used target detection methods. The detection results are shown in Table5. It is evident that the proposed obstacle detection method achieves an accuracy higher than state of the art methods.Table 5
Evaluation indices of YOLO v5s and the proposed method.
Detection methodA (%)Fast R-CNN76.53SSD71.28Fast YOLO78.95SSD 30073.96YOLO v5s81.73Proposed method90.36The real-time nature of obstacle detection refers to the ability to process every image frame collected in time. In terms of detection speed, YOLO v5s and the proposed method were used to process 2270 images and the respective average obstacle detection times were calculated. The results are shown in Table6.Table 6
Evaluation indices of detections of YOLO v5s and the proposed method.
Detection methodDetection time (s)YOLO v5s0.164Proposed method0.201Since the average detection time of the method proposed by us is 0.201s, in order to ensure the detection of obstacles under normal driving conditions, the speed of the detected vehicle must be less than or equal to the ratio of detection distance to the average detection time.Compared with YOLO v5s, the method proposed in this study saves the training step of data set. The modified method firstly uses machine learning to detect obstacles of known types, but it needs to process feature points of obstacles of unknown types, so the final detection time is longer than that of YOLO v5s. But it can still meet the demand of real-time detection.In order to verify the reliability of the distance measurement method proposed in this study and the feasibility of practical application, we have done a set of obstacle detection experiments. We first use a fixed camera to take pictures of the real road environment ahead and record the process. The result of IMU data processing is shown in Figure17. Then we select a few frames of images during the progress of the obstacle for processing. Finally, the distance between the camera and the obstacle in front is calculated, and the detection result is shown in Figure 18.Figure 17
IMU data processing result (speed and distance of the test vehicle).Figure 18
Obstacle distance detection. (a) 2021 8 16 10 0 32.825. (b) 2021 8 16 10 0 34.327. (c) 2021 8 16 10 0 35.766. (d) 2021 8 16 10 0 38.630.
(a)(b)(c)(d)The comparison results are shown in Table7.Table 7
Distance measurement results based on monocular vision.
TimeSlope (°)ObstacleActual distance (m)Measuring distance (m)Distance difference (m)Error (%)2021 8 16 10 0 32.8257.35113.79313.9260.1330.964213.60913.8000.1911.403312.65112.6370.0140.111414.19814.0520.1461.0282021 8 16 10 0 34.3277.21113.71913.8260.1070.780213.49913.5940.0950.704312.26512.3500.0850.693414.27014.0910.1791.2542021 8 16 10 0 35.7667.10113.53213.4220.1100.813213.35813.1960.1621.213312.30912.1310.1781.446413.99813.8750.1230.8792021 8 16 10 0 38.6306.88113.32113.2280.0930.698213.10213.0890.0130.099312.03711.9310.1060.881413.71313.655−0.0580.423Analyzing the difference between the actual and measured distance results, it was found that the difference lied mostly between 0.013 and 0.191. This phenomenon is caused by the slight change in the posture of the vehicle.This study is based on the obstacle detection method of VIDAR and the use of a digital map for distance measurement experiments. The experimental results show that the error of this method is less than 2% at short distances (<20 m), and the distance measurement effect is better than that reported by Guo Lei’s. Moreover, existing vision-based ranging requirements call for a measurement error of less than 5% [36]. Therefore, from the distance measurement results, the vision-based ranging algorithm proposed in this article meets the requirements in measurement accuracy and can achieve accurate distance measurement to obstacles.


## 5.1. Simulation Experiment

In this study, experimental equipment was used to simulate a detection environment so as to verify the detection effect of obstacles on sloped roads based on VIDAR. The experimental equipment included: a test vehicle equipped with an OV5640 camera unit and a JY61p IMU (Figure11(a)), vehicle scale models (Figure 11(b)), bottle caps and paper (Figure 11(c)), and simulated sloped road (Figure 11(d)). Among them, the test vehicle was used to analyze the road environment and detect its own driving state, scaled vehicle models were used to simulate known obstacles, while the bottle caps and paper were used to simulate unknown obstacles. The road slope was set to 13°.Figure 11
Equipment for simulating experiment. (a) Test vehicle. (b) Vehicle scale models. (c) Bottle caps and paper. (d) Simulated sloping road.
(a)(b)(c)(d)The bottle cap was taken as the real obstacle of unknown type, and the paper pasted on the simulated road was taken as the pseudoobstacle of unknown type. The angular acceleration and acceleration data of the vehicle were obtained by the IMU installed by the vehicle. Quaternion method is used to solve the camera attitude, and the pitching angle of the camera is updated. The velocity data are used to calculate the horizontal distance between the vehicle and the obstacle. The height of the obstacle is calculated by the change of the distance before and after the movement, so as to determine whether the detected obstacle is a real obstacle. The video collected using the OV5640 camera comprised an image sequence at 12FPS, which was used for obstacle detection. The results obtained using the original VIDAR and VIDAR on sloped roads are shown in Figure12, while the test results of the simulation experiment are summarized in Table 1.Figure 12
Comparison of obstacle detection effect in simulated environment. (a) Original VIDAR. (b) VIDAR on sloped roads.
(a)(b)Table 1
Test results of the simulation experiment.
ObstacleThei th movement of the test vehicleDetection distance (cm)The test vehicle moving distanceWhether it has heightWhether to exclude1116.53NoYes219.52118.53YesNo221.23124.53YesNo227.0The test results of the simulation experiment is shown in Table1.It can be seen from Figure12 that the original VIDAR can detect unknown types of obstacles such as bottle caps, but it will detect false obstacles as real obstacles, resulting in low accuracy of obstacle detection. However, the obstacle detection method for sloping roads based on VIDAR can eliminate false obstacles, which makes up for the wrong detection of unknown types of obstacles on sloping roads; therefore, compared with the original VIDAR, our proposed method can detect obstacles more accurately.


## 5.2. Real Environment Experiment

In the real environment, purely electric vehicles were used as test vehicles (see Figure13). As a sensor, the camera can adapt to complex environments and collect environmental information in real time (we only used the left camera). The camera was installed at a height of 1.60 m. The IMU used for locating the test vehicle and reading of its movement state in real time was installed at the bottom of the test vehicle. GPS was used for accurate location positioning. Through the combination of GPS and IMU, the real-time position information of the test vehicles and obstacles can be obtained, and then the trajectory information of vehicles and obstacles can be obtained. A digital map was used to obtain accurate road information such as distance and slope. A calculation unit was used to process the data in real time.Figure 13
Schematic diagram of the test vehicle.Accurate calibration of camera parameters was a prerequisite for the whole experiment and is a very important task for obstacle detection methods. In this paper, Zhang Zhengyou’s camera calibration method was adopted to calibrate the DaYing camera. First, the camera was fixed to capture images of a checkerboard at different positions and angles. Then, the key points of the checkerboard were selected and used to establish a relationship equation. Finally, the internal parameter calibration was realized. The camera calibration result is shown in Figure14.Figure 14
Schematic diagram of camera calibration results.Camera distortion includes radial distortion, thin lens distortion, and centrifugal distortion. The superposition of the three kinds of distortion results in a nonlinear distortion, the model of which can be expressed in the image coordinate system as follows:(8)δxx,y=s1xx2+y2+2p1xy+p2y3+k1xx2+y2δyx,y=s2yx2+y2+2p2xy+p1y3+k1xx2+y2,where s1 and s2 are centrifugal distortion coefficients; k1 and k2 are radial distortion coefficients, and p1 and p2 are the distortion coefficients of thin lenses.Because the centrifugal distortion of the camera is not considered in this study, the internal reference matrix of the camera can be expressed as shown in(9)M=5.9774e+030949.884305.9880e+03357.0539001.The calibration of the camera’s external parameters can be calculated by taking the edge object points of lane lines. The calibration results are shown in Table2.Table 2
Calibration results of camera external parameters.
External parameter typeParameter sizePitch angle1.20Yaw angle3.85Rotation angle2.37Since the images in the public data set were all data collected by other cameras, different camera parameters will affect the accuracy of ranging, so we used the VIDAR-Slope database (Figure15), the images in which where collected using a DaYing camera. The collection frequency was 77 frames/min, and there are 2270 images in total. The experiment and image collection took place in Shandong University of Technology’s driving school and experimental building. We selected the downhill section of the parking lot for the experiment. In the process of obstacle detection, the test vehicle moves at a constant speed of 25 km/h.Figure 15
The VIDAR-Slope database (part of data).The detection results of YOLO v5s and the method proposed in this study are shown in Figure16. The accuracy of obstacle detection was measured through the number of true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). Let a be an obstacle that is correctly classified as a positive example, b be an obstacle that is wrongly classified as a positive example, c be an obstacle that is correctly classified as a negative example, and d be an obstacle that is incorrectly identified as a negative example. Then, TP=∑i=1nai, FP=∑i=1nbi, TN=∑i=1nci, FN=∑i=1ndi.Figure 16
Comparison of partial test results of YOLO v5s and proposed method. (a) YOLO v5s can only detect known type obstacles. (b) The obstacle method proposed in this article can detect pedestrians (known types) and boxes (unknown types).
(a)(b)The YOLO series is a representative target detection framework based on deep learning. There are four versions of the target detection network: namely YOLO v5s, YOLO v5m, YOLO v5l, and YOLO v5x. Among them, YOLO v5s is the smallest and has the fastest speed, so we choose it for comparative experiments.Comparing the two methods, it can be seen that the stability of the proposed method is higher than that of YOLO v5s. YOLO v5s lacks training in unknown types of obstacles, and will consequently offer reduced safety when used in realistic vehicle situations. However, the proposed obstacle detection method does not require training and can detect all types of obstacles, thus ensuring its effectiveness of obstacle detection results on sloped roads. The total number of obstacles in the target area in the VIDAR-Slope database was 9526. The results of YOLO v5s and proposed method are shown in Table3.Table 3
Obstacle detection results in sloped roads of YOLO v5s and the proposed method.
Detection methodInput valueTPFPTNFNYOLO v5s9526671628089151214Proposed method95269124402530485In the results’ analysis, Accuracy (A), Recall (R), and Precision (P) were used as evaluation indices for the two obstacle detection methods, calculated through the following equations:(10)A=TP+TNTP+TN+FP+FN,(11)R=TPTP+FN,(12)P=TPTP+FP.The Accuracy, Recall, and Precision of YOLO v5s and the method proposed in this study are shown in Table4.Table 4
Evaluation indices of YOLO v5s and the proposed method.
Detection methodA (%)R (%)P (%)YOLO v5s81.7384.6870.50Proposed method90.3694.9495.77The experimental results in Tables3 and 4 show that due to vehicle fluctuations and other factors, misjudgment or misdetection may occur during vehicle movement. Compared with YOLO v5s, the accuracy of the obstacle detection method proposed in this study is increased by 8% and its precision is increased by 26.4%, which demonstrates its improved obstacle detection capability on sloped roads.In terms of detection accuracy, we also compared our method with other commonly used target detection methods. The detection results are shown in Table5. It is evident that the proposed obstacle detection method achieves an accuracy higher than state of the art methods.Table 5
Evaluation indices of YOLO v5s and the proposed method.
Detection methodA (%)Fast R-CNN76.53SSD71.28Fast YOLO78.95SSD 30073.96YOLO v5s81.73Proposed method90.36The real-time nature of obstacle detection refers to the ability to process every image frame collected in time. In terms of detection speed, YOLO v5s and the proposed method were used to process 2270 images and the respective average obstacle detection times were calculated. The results are shown in Table6.Table 6
Evaluation indices of detections of YOLO v5s and the proposed method.
Detection methodDetection time (s)YOLO v5s0.164Proposed method0.201Since the average detection time of the method proposed by us is 0.201s, in order to ensure the detection of obstacles under normal driving conditions, the speed of the detected vehicle must be less than or equal to the ratio of detection distance to the average detection time.Compared with YOLO v5s, the method proposed in this study saves the training step of data set. The modified method firstly uses machine learning to detect obstacles of known types, but it needs to process feature points of obstacles of unknown types, so the final detection time is longer than that of YOLO v5s. But it can still meet the demand of real-time detection.In order to verify the reliability of the distance measurement method proposed in this study and the feasibility of practical application, we have done a set of obstacle detection experiments. We first use a fixed camera to take pictures of the real road environment ahead and record the process. The result of IMU data processing is shown in Figure17. Then we select a few frames of images during the progress of the obstacle for processing. Finally, the distance between the camera and the obstacle in front is calculated, and the detection result is shown in Figure 18.Figure 17
IMU data processing result (speed and distance of the test vehicle).Figure 18
Obstacle distance detection. (a) 2021 8 16 10 0 32.825. (b) 2021 8 16 10 0 34.327. (c) 2021 8 16 10 0 35.766. (d) 2021 8 16 10 0 38.630.
(a)(b)(c)(d)The comparison results are shown in Table7.Table 7
Distance measurement results based on monocular vision.
TimeSlope (°)ObstacleActual distance (m)Measuring distance (m)Distance difference (m)Error (%)2021 8 16 10 0 32.8257.35113.79313.9260.1330.964213.60913.8000.1911.403312.65112.6370.0140.111414.19814.0520.1461.0282021 8 16 10 0 34.3277.21113.71913.8260.1070.780213.49913.5940.0950.704312.26512.3500.0850.693414.27014.0910.1791.2542021 8 16 10 0 35.7667.10113.53213.4220.1100.813213.35813.1960.1621.213312.30912.1310.1781.446413.99813.8750.1230.8792021 8 16 10 0 38.6306.88113.32113.2280.0930.698213.10213.0890.0130.099312.03711.9310.1060.881413.71313.655−0.0580.423Analyzing the difference between the actual and measured distance results, it was found that the difference lied mostly between 0.013 and 0.191. This phenomenon is caused by the slight change in the posture of the vehicle.This study is based on the obstacle detection method of VIDAR and the use of a digital map for distance measurement experiments. The experimental results show that the error of this method is less than 2% at short distances (<20 m), and the distance measurement effect is better than that reported by Guo Lei’s. Moreover, existing vision-based ranging requirements call for a measurement error of less than 5% [36]. Therefore, from the distance measurement results, the vision-based ranging algorithm proposed in this article meets the requirements in measurement accuracy and can achieve accurate distance measurement to obstacles.


## 6. Conclusion

In this study, an obstacle detection method based on VIDAR is applied to complex environments, avoiding the drawbacks of machine learning methods that can only detect known obstacles. Moreover, by integrating slope information into the VIDAR detection method, real obstacles can be detected on sloped roads, and distance and speed measurement of obstacles can be realized, which has important research value for autonomous vehicles and active safety systems. It can be seen from the results that the proposed method is effective in improving the accuracy and speed of obstacles detection and can meet the requirements of obstacle detection in complex environments. Obstacle detection in complex road environment is the basis for safe driving of vehicles. Therefore, obstacle avoidance path planning and speed control based on obstacle detection are our future research directions.



---

*Source: 5264347-2022-04-15.xml*