# Detection and Tracking of Road Barrier Based on Radar and Vision Sensor Fusion

Authors: Taeryun Kim; Bongsob Song
Journal: Journal of Sensors
 (2016)

---

## Abstract

The detection and tracking algorithms of road barrier including tunnel and guardrail are proposed to enhance performance and reliability for driver assistance systems. Although the road barrier is one of the key features to determine a safe drivable area, it may be recognized incorrectly due to performance degradation of commercial sensors such as radar and monocular camera. Two frequent cases among many challenging problems are considered with the commercial sensors. The first case is that few tracks of radar to road barrier are detected due to material type of road barrier. The second one is inaccuracy of relative lateral position by radar, thus resulting in large variance of distance between a vehicle and road barrier. To overcome the problems, the detection and estimation algorithms of tracks corresponding to road barrier are proposed. Then, the tracking algorithm based on a probabilistic data association filter (PDAF) is used to reduce variation of lateral distance between vehicle and road barrier. Finally, the proposed algorithms are validated via field test data and their performance is compared with that of road barrier measured by lidar.


---

## Body

## 1. Introduction

The driver assistance systems (DAS) such as adaptive cruise control (ACC), forward collision warning, and lane departure warning systems have been commercialized on the market [1]. They have evolved to more intelligent DAS such as automatic emergency braking (AEB), lane change assistance (LCA), and lane keeping assistance (LKA) systems [2]. As prototypes of a highly automated vehicle have been introduced on the media recently, reliability of the performance becomes more important. That is, once false decision is made by a computer or vehicle, it makes the driver have low reliability of the system and may thus result in no use of the system. The reliability of the decision mainly counts on accurate detection and recognition of multiple obstacles and vehicles. For instance, Honda Motor Company had to recall certain model year 2014-2015 Acura vehicles with AEB in the United States. The reason was that a collision mitigation braking system (CMBS) may inappropriately interpret certain roadside infrastructure such as iron fences or metal guardrails as obstacles and unexpectedly apply the brakes [3]. Furthermore, NHTSA in the United States investigated complaints alleging unexpected braking incidents of the autonomous braking system in Jeep Grand Cherokee vehicles with no visible objects on the road [3].The detection and tracking algorithms of road barrier, which may be called either road border or boundary in the literature, depend on sensor configuration and their models for the road barrier. Most of the sensor configurations are single or a combination of radar [1, 2, 4, 5], camera [6], and lidar (or laser scanners) [7, 8] to recognize the drivable area via reflections from guardrail and curb. Next, extended objects such as road and road barrier are described as clothoid, circle, and elliptical model and their tracking algorithm is based on Kalman filter, probabilistic data association filter (PDAF), and interacting multiple model (IMM) PDAF [1, 2, 4, 9]. In this study, it is assumed that a front radar and a monocular camera are only used for detection and tracking of road barrier. Although additional sensors can be implemented for better performance or the lidar may be used as in the literature, the sensor configuration is limited in the viewpoint of commercialization in the near future. For instance, cost of sensors, robustness to weather, installation inside bumper, and popularity in automotive market are considered to choose the sensor configuration.The contribution of this paper is to enhance reliability of road border detection when only few tracks are generated by radar with respect to road border and to improve lateral position accuracy of road border. Since the performance of a commercial radar relies on material and geometry of tunnel and guardrail, different number of tracks are made depending on driving environments. Thus, the estimation for stationary tracks out of detection range is proposed for better performance of road barrier detection. Furthermore, the tracking algorithm of road barrier based on a probabilistic data association filter (PDAF) is proposed to reduce variation of lateral offset, which is a lateral distance between an ego vehicle and road barrier.


## 2. Problem Statement

When commercial radars for driver assistance systems such as ACC and AEB are used, two challenging problems will be considered in this paper. A normal detection scenario is shown in Figure1(a) and the corresponding tracks are shown in Figure 1(b). Two tracks to front vehicles are marked as a square and the others marked as × correspond to left guardrail in Figure 1(a). However, as shown in Figures 1(c) and 1(d), few tracks of radar are generated for the road barrier. Its detection performance may rely on the material type and/or shape of road barrier. This problem may lead to difficulty in determining whether there is road barrier in either left or right side.Figure 1
Detection characteristics to road barrier by radar.
(a)
Snapshot of driving scene
(b)
Radar tracks corresponding to road barrier and vehicles
(c)
Snapshot of driving scene
(d)
Radar tracks corresponding to road barrier and vehiclesNext, radar tracks of road barrier are compared with cloud points with magenta color measured by a front lidar sensor in Figures2(b) and 2(d). As shown in Figures 2(a) and 2(c), inaccuracy of lateral position measured by radar with respect to lidar measurements can be larger in the same driving scenario. This is expected to have large variance of lateral position of road barrier when only radar is used to recognize it.Figure 2
Road barrier detection by radar and lidar.
(a)
Snapshot att
1
(b)
Comparison between radar tracks and lidar points
(c)
Snapshot att
2
(d)
Comparison between radar tracks and lidar points


## 3. Road Barrier Detection

The proposed road barrier detection based on sensor fusion of radar and monocular camera consists in four steps: selection of region of interest (ROI), estimation, clustering, and representation. First, the selection of ROI is roughly described in Figure3. That is, based on the assumption that road barrier is placed on either left or right side, zone ② is defined with respect to a body fixed coordinate. If the road in Figure 3 is modeled as [4](1)
y
=
1
2
κ
x
2
+
φ
x
,where x and y are longitudinal and lateral position, respectively, in a body fixed coordinate in Figure 4, κ is curvature, and φ is the angle between the longitudinal axis of the vehicle and the road lane from a monocular camera as shown Figure 4. Then, zone ② is written as(2)
f
barrier
x
=
1
2
κ
x
2
+
φ
x
+
ε
1
,where ε
1 is a lateral offset which determines a width of zone ②; that is, l
m
i
n
≤
ε
1
≤
l
m
a
x and -
l
m
a
x
≤
ε
1
≤
-
l
m
i
n.Figure 3
Region of interest (ROI) for road barrier detection.Figure 4
Definitions of lateral offset (l
i) and projection point.Next, based on detection range of radar, zone ③ in Figure3 is defined for estimation of stationary tracks. Before the estimation, a motion attribute to determine whether tracks in zone ② are either stationary or dynamic is decided by (3)
p
i
=
dynamic
if
v
+
R
˙
i
+
y
i
ψ
˙
>
ε
2
stationary
otherwise
,where v is the vehicle velocity, ψ
˙ is the yaw rate, R
˙ is the range rate, and subscript i stands for the ith radar track. It is remarked that there is uncertainty of detection range of radar. So, it is essential to divide regions either to track or to estimate. Once a stationary track in zone ② enters zone ③, it is estimated based on discrete Kalman filters as follows [10]:(4)
x
^
k
+
1
∣
k
=
A
x
^
k
∣
k
,
P
k
+
1
∣
k
=
A
P
k
-
1
∣
k
-
1
A
T
+
Q
k
,where x
=
y
v
y
x
v
x
T and x and y denote the relative longitudinal and lateral position, respectively, and v
x and v
y are the relative longitudinal and lateral velocity.The measurement update equations are given by(5)
x
^
k
+
1
∣
k
+
1
=
x
^
k
+
1
∣
k
+
K
k
+
1
z
k
+
1
-
H
x
^
k
+
1
∣
k
,
P
k
+
1
∣
k
+
1
=
P
k
+
1
∣
k
-
K
k
+
1
H
P
k
+
1
∣
k
,where K
k
+
1
=
P
k
+
1
∣
k
H
T
H
P
k
+
1
∣
k
H
T
+
R
k
+
1
-
1.Since a constant velocity (CV) is considered, the system matrixA and measurement matrix H are written as [10, 11](6)
A
=
1
T
0
0
0
1
0
0
0
0
1
T
0
0
0
1
,
H
=
1
0
0
0
0
0
1
0
0
0
0
1
,where T is sampling time.In the third step or clustering, in order to group tracks corresponding to road barrier among stationary tracks including estimated track, projection points are calculated as follows (see circles in Figure4):(7)
l
i
=
y
i
-
1
2
κ
x
i
2
+
φ
x
i
,where the subscript i stands for the ith track of radar. After that, the projection points are classified by left or right if the projection points are positioned in zone ②. If the distance between jth and (
j
-
1
)th projection points is less than ε
2, jth k is increased and (
j
-
1
)th k is defined as a breakpoint. If the distance between (
m
-
1
)th and mth projection points is greater than ε
2, the breakpoint is generated at (
m
-
1
)th k as follows: (8)
k
j
=
k
j
-
1
+
1
if
l
j
-
l
j
-
1
<
ε
3
0
otherwise
for
j
=
2
,
…
,
m
.Finally, if there are two projection points in two breakpoints ork
m
-
1
≥
1, all ith tracks satisfying the following inequality are regarded as road barrier [2]:(9)
x
p
≤
x
i
≤
x
p
+
1
,
f
barrier
x
p
-
ε
4
≤
y
i
≤
f
barrier
x
p
+
ε
5
for
p
=
1
,
…
q
,
i
=
1
,
…
,
n
.Considering the y-coordinates as erroneous in comparison with x-coordinates, the clothoid model which can be approximated by a two-order polynomial is calculated in(10)
f
r
b
x
0
=
1
2
κ
x
2
+
φ
x
+
l
0
,where l
0
=
l
1
+
l
m
/
2. It is noted that the calculation of clothoid model corresponds to creation of road barrier.The procedure to detect road barrier is shown in Figure5. First, six tracks among seven ones coming from radar are classified as stationary objects. Then, they are projected to y-axis along the road model and the corresponding projection points are shown as circles in Figure 5(a). Next, based on distance between two projection points in (7), two breakpoints are determined as shown in Figure 5(b) and six tracks thus are classified as road barrier based on (9). Then the clothoid model in (10) is determined and shown as a solid line in Figure 5(c). After a few seconds, the closest stationary track in zone ② goes into zone ③ (also refer to Figure 3) and it becomes out of detection range and estimated based on discrete Kalman filter (see also a diamond mark in Figure 5(d)).Figure 5
Procedure of road barrier detection.
(a)
Projection
(b)
Clustering
(c)
Road barrier detection
(d)
Estimation


## 4. Tracking Road Barrier

The tracking road barrier is composed of creation, maintenance, and deletion. Creation step uses the result of road barrier detection in (10). Maintenance step based on PDAF tracks lateral offset of road barrier. So, it is rewritten as(11)
f
r
b
x
=
1
2
κ
x
2
+
φ
x
+
l
^
,where l
^ is a tracked value via PDAF and will be derived later.While the lane information detected by a monocular camera sensor is useful to model roads, its performance depends on light and road conditions. For example, if the lane mark is worn out or covered by soil or snow, it may result in false detection of lane mark. Thus, considering the condition of lane marks, whether measurements by a monocular camera or estimate value is used is decided as follows [12]:(12)
κ
=
κ
v
if
σ
L
=
HIGH
&
σ
R
=
HIGH
ψ
˙
v
otherwise
φ
=
φ
v
if
σ
L
=
HIGH
&
σ
R
=
HIGH
0
otherwise
,where σ is confidence of right or left lanes, ψ
˙ is yaw rate, and v is vehicle speed.The PDAF is based on discrete Kalman filter and a state variable is defined as(13)
x
=
y
v
y
T
,where y and v
y are the relative lateral position and velocity, respectively. For time update of the state variable and error covariance matrix,(14)
x
^
k
∣
k
-
1
=
A
d
x
^
k
-
1
∣
k
-
1
,
P
k
∣
k
-
1
=
A
d
P
k
-
1
∣
k
-
1
A
d
T
+
Q
k
.System and measurement metrics are as follows:(15)
A
d
=
1
T
0
1
,
H
=
1
0
.If the number of projection points in two breakpoints is m, the measurement is defined as(16)
z
k
=
l
j
k
j
=
1
m
,where j means the number of projection points.In the gating region it has to be decided which measurements (i.e., clustered projection points) are associated with which existing projection points. The corresponding residual and residual covariance matrixS are calculated as(17)
v
i
k
=
z
i
k
-
H
x
^
k
∣
k
-
1
,
S
k
∣
k
-
1
=
H
P
k
∣
k
-
1
H
T
+
R
k
.All measurements are checked whether the normalized residual satisfies the following thresholding condition.(18)
v
i
T
k
S
-
1
k
∣
k
-
1
v
i
k
≤
ε
5
2
.After that, valid measurements which are in gating region are combined to single residual. The weighting is according to the likelihood values of the corresponding measurements. Finally, the measurement update of the state variable calculates estimated track using the combined residual [13]:(19)
x
^
k
∣
k
=
x
^
k
∣
k
-
1
+
K
k
v
k
,
l
^
k
∣
k
=
H
x
^
k
∣
k
,where K
k
=
P
k
∣
k
-
1
H
T
H
P
k
∣
k
-
1
H
T
+
R
-
1, v
k
=
∑
i
=
1
m
k
β
i
v
i
k, and (20)
β
i
=
exp
⁡
-
1
/
2
v
i
T
k
S
-
1
k
∣
k
-
1
v
i
k
∑
j
=
1
m
k
exp
⁡
-
1
/
2
v
i
T
k
S
-
1
k
∣
k
-
1
v
i
k
.


## 5. Experimental Validation

As listed in Table1, both radar and monocular camera, which are available commercially, are installed on a test vehicle and a front lidar is in addition used for performance comparison. Although a large amount of driving data has to be used for validation, the primitive evaluation of the performance is conducted with driving data of 13 minutes including the driving scenarios in Figures 1 and 2. Most of driving data have been tested on a highway. It is also noted that the driving data corresponding to tollgate, exit zone, and area without any tunnel or guardrail is not considered for performance evaluation.Table 1
Specification of environment sensors.
Sensor
Detection range
Field of view
Radar
Long mode
174 m
±10°
Short mode
60 m
±45°
Monocular camera
90 m
±40°
Lidar
200 m
±42.5°Based on detection characteristics of radar, five environment scenarios on highway are considered. The first and second scenarios are shown in Figure1 and called concrete+steel and concrete guardrail, respectively, in the paper. It is interesting to remark that different detection characteristics for the concrete guardrail are shown in Figures 1(c) and 2. In addition, steel guardrail, curb, and tunnel are included for validation as shown in Figure 6. It is thought that most of road barrier on highway in Korea can be described by one of five environment scenarios.Figure 6
Additional environment scenarios for detection and tracking of road barrier.
(a)
Steel guardrail
(b)
Curb
(c)
TunnelThe first case shown in Figures1(c) and 1(d) is revisited. That is, few tracks for road barrier are generated by radar. As shown in Figure 7, only two tracks with respect to left guardrail are generated at t
1 = 123.7 (sec). After 0.8 sec, the nearest stationary track stays out of detection range of radar as shown in Figure 7(d). The solid lines represent trajectory of tracks from t = 123.7 to 124.5 (sec). Since the track corresponding to road barrier is located in zone ③ in Figure 3, it becomes a road barrier candidate (see also a diamond mark in the figure). Then there are still two breakpoints for road barrier and the detected road barrier is represented as shown in Figure 7(d).Figure 7
Road barrier detection via estimation.
(a)
Snapshot att
1 = 123.7 (sec)
(b)
Creation of road barrier att
1
(c)
Snapshot att
2 = 124.5 (sec)
(d)
Trajectory of tracks fromt
1 to t
2Two driving scenarios in which lateral position of tracks by radar may be inaccurate are considered when an ego vehicle drives along tunnel or guardrail as shown in Figures8(a) and 8(c). The proposed tracking algorithm of road barrier is compared with lidar and Kalman filter based approach in [4]. It is shown in Figures 8(b) and 8(d) that the performance of the proposed detection and tracking of road barrier is closer to that of lidar. Furthermore, it is shown in Figure 9 that four different approaches are compared with respect to lateral offset. Then, their relative quantitative performances with respect to lidar are evaluated in terms of root mean square error (RMSE) of lateral offset and recognition accuracy. They are summarized with respect to environment scenarios in Table 2.Table 2
Performance comparison of tracking of road barrier.
Scenario
Detection only
Kalman filter
PDA filter
Perception of RB (%)
Concrete
65.78
67.53
81.42
Tunnel
62.21
63.49
81.30
Curb
70.96
76.39
89.52
Concrete + iron
98.53
98.87
100.00
Iron
98.76
99.53
100.00
Total
67.75
69.22
84.32
RMSE (m)
Concrete
0.5396
0.4148
0.3992
Tunnel
0.8010
0.8265
0.7692
Curb
2.6398
2.8001
2.6089
Concrete + iron
0.7354
0.7183
0.6481
Iron
0.7583
0.7438
0.7068
Total
1.1497
1.2231
1.0992Figure 8
Performance comparison of road barrier tracking.
(a)
Snapshot att = 79.8 (sec)
(b)
Tracking performance in tunnel
(c)
Snapshot att = 21.9 (sec)
(d)
Tracking performance at curved roadFigure 9
Time response of lateral offset of road barrier.Two performance measures are used for validation depending on environment scenarios. The first one is perception of road barrier (RB) and defined as ratio of detection period of RB by the proposed algorithm to detection by image manually. To describe the tracking performance, RMSE of lateral offset between sensor fusion of radar and vision and that of lidar and vision is used for the second performance measure. Finally, the performance comparison is summarized in Table2. It is validated that the proposed algorithm improves perception of road barrier more when few tracks are often generated by radar in cases of concrete guardrail, tunnel, and curb on highway. Furthermore, it is shown that tracking performance of the proposed algorithm is more robust than others in different environment scenarios.The detection and tracking algorithms are proposed to overcome two problems: one is when few tracks to road barrier are generated by radar and the other is when inaccuracy of lateral position becomes worse in a short time and it happens frequently. Both estimation and clustering methods are combined to handle the first problem and the tracking algorithm of lateral offset of road barrier based on PDAF is proposed to deal with the second problem. Its performance is evaluated with comparison of that of lidar and other approaches in the literature. Although it is shown via field test data that the proposed algorithm is good enough to recognize road barrier without lidar, it is quite necessary to validate it with massive field test data in the near future in order to consider various driving scenarios.



---

*Source: 1963450-2016-09-26.xml*