# 3D Road Scene Monitoring Based on Real-Time Panorama

Authors: Yuezhou Wu; Changjiang Liu; Shiyong Lan; Menglong Yang
Journal: Journal of Applied Mathematics
 (2014)

---

## Abstract

Road monitoring helps to control the regional traffic situation so as to adjust the traffic flow. Real-time panorama is conducive to timely treat traffic accidents and to greatly improve traffic capacity. This paper designs a 3D road scene monitoring framework based on real-time panorama. The system is the combination of large scale panorama, satellite map textures, and 3D scene model, in which users can ramble freely. This paper has the following contributions. Firstly, land-points were extracted followed by motion detection, then comotion algorithm was applied to land-points from adjacent cameras, and homography matrix was constructed. Secondly, reference camera was chosen and transformed to overhead viewpoint; subsequently multiviews were morphed to the same viewpoint and stitched to panorama. Finally, the registration based on high-precision GPS information between 2D road panorama and 3D scene model was also proposed. The proposed framework has been successfully applied to a large road intersection monitoring. Experimental results are furnished at the end of the paper.


---

## Body

## 1. Introduction

Independent camera only provides local information. As a result, it is difficult to make a reasonable judgment and immediate response to the global situation. Image mosaic can combine a set of images into a larger image with a wider field of view of the scene. It helps to improve observers’ spatial awareness.Over the years, numerous algorithms for image stitching have been developed. Image stitching is typically solved by finding global parametric warps to bring images into alignment [1]. 2D projective warps are parameterized by 3
×
3 homography, if the scene is planar or if the views differ purely by rotation [2, 3]. The warps of video images from multicameras were described in [4, 5]; however, they only focused on the images (infrared and visible images) of similar viewpoint. Lee et al. [6] introduced an automatic method to warp multiview images to a unified coordinate plane. They proposed time-space registration based on trajectory extraction and on the basis the homography matrices from images to the reference plane were estimated. However the trajectory accuracy might influence fit precision. Reference [7] presented a method for ground plane estimation from image pairs and land-points, instead of static features such as color, shape, and contours, for image matching. Szlávik et al. [8] proposed a method for matching partially overlapping image pairs where the object of interest was in motion, namely, comotion algorithm. It was valid even if the motion is discontinuous and in an unstructured environment. Szlávik et al. [9] also applied comotion model to project a couple of camera images to the reference road plane.Recently significant progress in the synthetic vision system (SVS) [10, 11], which is a computer-generated reality system, has been achieved. And that computer-generated image, namely, virtual image, is complementary to optical sensor-based vision. SVS supplies users with good immerse visualization. For example, virtual and reality technology has been studied in landmarks (horizon, runway) detection from an aircraft in low visibility conditions [12, 13].In this paper, we propose a 3D road monitoring method based on real-time panorama. We have synthesized mosaicked images and static 3D scene model. Firstly, moving targets were detected and land-points were extracted. Secondly, binary sequences in continuous timeT were generated according to whether the pixel is land-point set or not. Subsequently land-points from adjacent cameras were matched according to similarity measure, resulting in homography transformation of adjacent cameras. Thirdly, choose a camera as a reference one and wrap the view to overhead view according to the world coordinates of ground control points (GCPs) and corresponding image coordinates. Thus with the transformation relationships between adjacent cameras, all the views were unified to overhead view. Finally, panorama was generated based on stitching lines. In the end of the paper, we proposed a novel panorama presentation of 3D road scene with real-time traffic situation. Specific process is shown in Figure 1.Figure 1
Flowchart of 3D road scene monitoring based on real-time panorama.


## 2. Transformation of Multiview Based on Land-Points Comotion Method

The multiple views would be unified to the same viewpoint. In this paper 2D projective transformation brings all the images aligned to a common ground plane with overhead view. There are three main steps.(1)
Homography relations are implemented via land-points comotion statistical maps between adjacent cameras.
(2)
Referenced image is projected to the ground plane on the basis of the world coordinate of GCPs and the corresponding image coordinates.
(3)
From steps (1) and (2), all the views are converted to the overhead view.


### 2.1. Calculation of Land-Points

Assuming the moving blobs has been detected before the calculation of land-points, land-point [7] is the point of central axis projected to the ground. The calculation of land-point is as follows.(1)
Binarization of video images using hybrid Gauss background algorithm: moving blobs are denoted as(1)
O
=
{
(
x
,
y
)
∣
f
(
x
,
y
)
=
255
}
.
(2)
Trace the contour ofO, name the contour C, and only the blobs whose area of external polygon is greater than T
a (≥10) pixe
l
2 are counted.
(3)
The external rectangle ofC is denoted as
(2)
R
=
{
(
x
,
y
)
∣
x
min
⁡
≤
x
≤
x
max
⁡
,
y
min
⁡
≤
y
≤
y
max
⁡
}
.
(4)
Linel
j perpendicular to the x-axis x
=
x
j
,
x
min
⁡
≤
x
j
≤
x
max
⁡; the distance from point P
i
(
x
i
,
y
i
)
∈
O to line l
j is
(3)
D
(
P
i
,
l
j
)
=
|
x
i
-
x
j
|
.
(5)
Calculate the abscissa of land-point(4)
x
mid
=
arg
min
⁡
j
{
median
P
i
∈
O
(
D
(
P
i
,
l
j
)
)
}
.
If there are more than onex
j meeting
(5)
x
j
=
min
⁡
j
{
median
P
i
∈
O
(
D
(
P
i
,
l
j
)
)
}
,
thenx
mid is the average of these quantities.
(6)
Get land-point(
x
mid
,
y
max
⁡
).The land-point of moving blobs is illustrated in Figure2.Figure 2
Land-point of moving blob.


### 2.2.  Matching of Land-Points Based on Comotion Method

In our approach no prior knowledge is needed, and the method also works well in images of randomly scrambled motion. It is difficult to realize the registration among images with different position and angle of view. In this paper, the road surface served the reference plane so as to solve the image registration with comotion statistics maps [4, 5]. The hybrid Gauss background model with a fine performance in shadow removal was constructed to detect the foreground of camera video. For any pixel at tth frame, a binary variable seq
Λ
(
i
,
j
,
t
) indicates whether the pixel belongs to foreground. Continuous seq
Λ
(
i
,
j
,
t
)
(
1
≤
t
≤
T
) construct sequence vector se
q
Λ
(
i
,
j
) [6],
(6)
seq
Λ
(
i
,
j
)
=
[
seq
Λ
(
i
,
j
,
1
)
,
…
,
seq
Λ
(
i
,
j
,
T
)
]
,
where
(7)
seq
Λ
(
i
,
j
,
t
)
=
{
0
,
if
(
i
,
j
)
is
background
at
t
th
frame
1
,
if
(
i
,
j
)
is
foreground
at
t
th
frame
,
1
≤
i
≤
W
Λ
,
1
≤
j
≤
H
Λ
,
Λ
∈
{
A
,
B
}
.The aforementioned is illustrated in (6). Among them, subscript B is a reference image and A is a matched one. Let W and H denote the width and height of image, respectively. And T is the length of the video to participate in the registration.Hence coded sequenceseq can be divided into two categories: valid motion point and noise point with almost no or frequent movement. Let t
s be static point threshold and let t
n be the noise point threshold. Subsequently characteristic function α
Λ
(
i
,
j
) is defined as follows:
(8)
α
Λ
(
i
,
j
)
=
{
0
,
if
∥
se
q
Λ
(
i
,
j
)
∥
1
<
t
s
or
m
∥
diff
(
seq
Λ
(
i
,
j
)
)
∥
1
>
t
n
1
,
else
,
where ∥
·
∥
1 is the L
1-norm and diff
(
·
) is the difference of vector, for a vector v is [
v
(
2
)
-
v
(
1
)
,
…
,
v
(
n
)
-
v
(
n
-
1
)
].Point(
i
,
j
) will be removed from candidate matching pairs if α
Λ
(
i
,
j
)
=
0. Here similarity measure of the binary sequence is depicted as L
1-norm. Thus from (8) perfect matching objective function can be written as follows:
(9)
map
(
A
(
i
,
j
)
)
=
arg
min
⁡
(
p
,
q
)
∈
U
B
∥
seq
A
(
i
,
j
)
-
seq
B
(
p
,
q
)
∥
1
,
(
i
,
j
)
∈
U
A
,
(
p
,
q
)
∈
U
B
,
with
(10)
U
Λ
=
{
(
i
,
j
)
∣
α
Λ
(
i
,
j
)
=
1
,
1
≤
i
≤
W
Λ
,
1
≤
j
≤
H
Λ
}
,
Λ
∈
{
A
,
B
}
.In general, pixel(
i
,
j
) which is valid moving point in image A based on (9) matches the moving point (
p
*
,
q
*
) in image B if and only if the L
1-norm of seq
A
(
i
,
j
)-seq
B
(
p
*
,
q
*
) is minimum.In (9) the nearest neighbor searching is implemented with ANN (approximate nearest neighbor) library [14] developed by Mount and Arya. Define the dimension of searching space as T and Minkowski norm to be L
1-norm. ANN supports kd-trees and box-decomposition data structures, which greatly improves the efficiency of high dimensional feature matching.


### 2.3. Homography between Adjacent Cameras

According to the camera model, the relationship between world coordinate and image pixel coordinate is(11)
Z
c
[
u
v
1
]
=
[
α
x
0
u
0
0
α
y
v
0
0
0
1
]
[
R
∣
t
]
[
X
w
Y
w
Z
w
1
]
=
[
m
11
m
12
m
13
m
14
m
21
m
22
m
23
m
24
m
31
m
32
m
33
m
34
]
[
X
w
Y
w
Z
w
1
]
,
where α
x, α
y, u
0, and v
0 are called the internal camera parameters, R and t are called the external parameters, and Z
c is depth coefficient.When the world point falls in thez
=
0 plane, (11) is cast as
(12)
Z
c
[
u
v
1
]
=
[
m
11
m
12
m
13
m
21
m
22
m
23
m
31
m
32
m
33
]
[
x
w
y
w
1
]
.Customarily (12) is rewritten as
(13)
λ
i
[
x
i
′
y
i
′
1
]
=
H
[
x
i
y
i
1
]
,
where H
=
(
h
i
j
)
3
×
3 is called a homography.Based on (9), we get the corresponding land-point pairs between images A and B to estimate 8 parameters of the projective transformation model; pixel (
x
i
′
,
y
i
′
) is the destination pixel in image B corresponding to the source pixel (
x
i
,
y
i
) in image A. Substitute {
x
i
,
y
i
}
,
{
x
i
′
,
y
i
′
} in (13) for the land-point pairs; then the homography H is computed such that
(14)
∑
i
‍
[
(
x
i
′
-
h
11
x
i
+
h
12
y
i
+
h
13
h
31
x
i
+
h
32
y
i
+
h
33
)
2
+
(
y
i
′
-
h
21
x
i
+
h
22
y
i
+
h
23
h
31
x
i
+
h
32
y
i
+
h
33
)
2
]
.In many practical situations this assumption is not valid because land-points with noise are mismatched [15]. To improve the robustness of the transformation model, random sample consensus (RANSAC) procedure is applied to the estimation of homography matrix [16].


### 2.4. Projection of the Reference Camera Based on GCPs

Given the center of visual range of camerai
C
i
(
i
=
1,2
,
…
) and the center of the panorama C
0, then the reference camera i
* is chosen via
(15)
i
*
=
arg
min
⁡
i
|
C
i
-
C
0
|
.As shown in Figure3, camera 3 is the reference camera satisfied with (15).Figure 3
Visual field of each camera.The view of reference camera will be projected to the ground plane based on GCPs by means of measurement. That is to say, the view has been transformed to overhead view till now. For any quadrangleA
B
C
D on the ground, the intersection of edges A
C and B
D is the origin of the world coordinate and x-axis parallels A
B pointing right. The coordinate system and length of each edge are as shown in Figure 4.Figure 4
World coordinate system.Given|
A
B
|
=
a, |
B
C
|
=
b, |
C
D
|
=
c, |
D
A
|
=
d, |
B
D
|
=
e, and |
A
C
|
=
f, according to simple geometric relations, the vertices of quadrilateral A
B
C
D are as below [17]:
(16)
x
A
=
-
α
tan
⁡
β
tan
⁡
α
+
tan
⁡
β
,
y
A
=
-
x
A
tan
⁡
α
,
x
B
=
a
-
x
A
,
y
B
=
y
A
,
x
C
=
x
B
-
b
cos
⁡
⁡
∠
B
,
y
C
=
y
B
-
b
sin
⁡
∠
B
,
x
D
=
x
A
+
d
cos
⁡
⁡
∠
A
,
y
D
=
y
A
-
d
sin
⁡
∠
A
.In engineering application, so as to get better visual effect, similarity transformation is necessary:(17)
x
i
′
=
s
x
x
i
+
Δ
x
,
y
i
′
=
s
y
y
i
+
Δ
y
,
i
∈
{
A
,
B
,
C
,
D
}
,
where proper translation Δ
x and Δ
y makes the coordinate nonnegative, and scaling coefficient determines the height from viewpoint to the ground.Suppose the coordinates of 4 points from (16), (17), and the corresponding image points substitute {
x
i
′
,
y
i
′
} and {
x
i
,
y
i
} in (13); then the view of reference camera has been transformed to overhead view.


### 2.5. Viewpoints Unifying

Now suppose there are three cameras. LetG
i be the homography transforming the current viewpoint to overhead one. Suppose camera 2 is the reference camera; then
(18)
G
1
≅
G
2
H
12
,
H
12
≅
H
21
-
1
,
G
3
≅
G
2
H
32
,
H
32
≅
H
23
-
1
,
where H
21 and H
23 are obtained in Section 2.3 and G
2 in Section 2.4.


## 2.1. Calculation of Land-Points

Assuming the moving blobs has been detected before the calculation of land-points, land-point [7] is the point of central axis projected to the ground. The calculation of land-point is as follows.(1)
Binarization of video images using hybrid Gauss background algorithm: moving blobs are denoted as(1)
O
=
{
(
x
,
y
)
∣
f
(
x
,
y
)
=
255
}
.
(2)
Trace the contour ofO, name the contour C, and only the blobs whose area of external polygon is greater than T
a (≥10) pixe
l
2 are counted.
(3)
The external rectangle ofC is denoted as
(2)
R
=
{
(
x
,
y
)
∣
x
min
⁡
≤
x
≤
x
max
⁡
,
y
min
⁡
≤
y
≤
y
max
⁡
}
.
(4)
Linel
j perpendicular to the x-axis x
=
x
j
,
x
min
⁡
≤
x
j
≤
x
max
⁡; the distance from point P
i
(
x
i
,
y
i
)
∈
O to line l
j is
(3)
D
(
P
i
,
l
j
)
=
|
x
i
-
x
j
|
.
(5)
Calculate the abscissa of land-point(4)
x
mid
=
arg
min
⁡
j
{
median
P
i
∈
O
(
D
(
P
i
,
l
j
)
)
}
.
If there are more than onex
j meeting
(5)
x
j
=
min
⁡
j
{
median
P
i
∈
O
(
D
(
P
i
,
l
j
)
)
}
,
thenx
mid is the average of these quantities.
(6)
Get land-point(
x
mid
,
y
max
⁡
).The land-point of moving blobs is illustrated in Figure2.Figure 2
Land-point of moving blob.


## 2.2.  Matching of Land-Points Based on Comotion Method

In our approach no prior knowledge is needed, and the method also works well in images of randomly scrambled motion. It is difficult to realize the registration among images with different position and angle of view. In this paper, the road surface served the reference plane so as to solve the image registration with comotion statistics maps [4, 5]. The hybrid Gauss background model with a fine performance in shadow removal was constructed to detect the foreground of camera video. For any pixel at tth frame, a binary variable seq
Λ
(
i
,
j
,
t
) indicates whether the pixel belongs to foreground. Continuous seq
Λ
(
i
,
j
,
t
)
(
1
≤
t
≤
T
) construct sequence vector se
q
Λ
(
i
,
j
) [6],
(6)
seq
Λ
(
i
,
j
)
=
[
seq
Λ
(
i
,
j
,
1
)
,
…
,
seq
Λ
(
i
,
j
,
T
)
]
,
where
(7)
seq
Λ
(
i
,
j
,
t
)
=
{
0
,
if
(
i
,
j
)
is
background
at
t
th
frame
1
,
if
(
i
,
j
)
is
foreground
at
t
th
frame
,
1
≤
i
≤
W
Λ
,
1
≤
j
≤
H
Λ
,
Λ
∈
{
A
,
B
}
.The aforementioned is illustrated in (6). Among them, subscript B is a reference image and A is a matched one. Let W and H denote the width and height of image, respectively. And T is the length of the video to participate in the registration.Hence coded sequenceseq can be divided into two categories: valid motion point and noise point with almost no or frequent movement. Let t
s be static point threshold and let t
n be the noise point threshold. Subsequently characteristic function α
Λ
(
i
,
j
) is defined as follows:
(8)
α
Λ
(
i
,
j
)
=
{
0
,
if
∥
se
q
Λ
(
i
,
j
)
∥
1
<
t
s
or
m
∥
diff
(
seq
Λ
(
i
,
j
)
)
∥
1
>
t
n
1
,
else
,
where ∥
·
∥
1 is the L
1-norm and diff
(
·
) is the difference of vector, for a vector v is [
v
(
2
)
-
v
(
1
)
,
…
,
v
(
n
)
-
v
(
n
-
1
)
].Point(
i
,
j
) will be removed from candidate matching pairs if α
Λ
(
i
,
j
)
=
0. Here similarity measure of the binary sequence is depicted as L
1-norm. Thus from (8) perfect matching objective function can be written as follows:
(9)
map
(
A
(
i
,
j
)
)
=
arg
min
⁡
(
p
,
q
)
∈
U
B
∥
seq
A
(
i
,
j
)
-
seq
B
(
p
,
q
)
∥
1
,
(
i
,
j
)
∈
U
A
,
(
p
,
q
)
∈
U
B
,
with
(10)
U
Λ
=
{
(
i
,
j
)
∣
α
Λ
(
i
,
j
)
=
1
,
1
≤
i
≤
W
Λ
,
1
≤
j
≤
H
Λ
}
,
Λ
∈
{
A
,
B
}
.In general, pixel(
i
,
j
) which is valid moving point in image A based on (9) matches the moving point (
p
*
,
q
*
) in image B if and only if the L
1-norm of seq
A
(
i
,
j
)-seq
B
(
p
*
,
q
*
) is minimum.In (9) the nearest neighbor searching is implemented with ANN (approximate nearest neighbor) library [14] developed by Mount and Arya. Define the dimension of searching space as T and Minkowski norm to be L
1-norm. ANN supports kd-trees and box-decomposition data structures, which greatly improves the efficiency of high dimensional feature matching.


## 2.3. Homography between Adjacent Cameras

According to the camera model, the relationship between world coordinate and image pixel coordinate is(11)
Z
c
[
u
v
1
]
=
[
α
x
0
u
0
0
α
y
v
0
0
0
1
]
[
R
∣
t
]
[
X
w
Y
w
Z
w
1
]
=
[
m
11
m
12
m
13
m
14
m
21
m
22
m
23
m
24
m
31
m
32
m
33
m
34
]
[
X
w
Y
w
Z
w
1
]
,
where α
x, α
y, u
0, and v
0 are called the internal camera parameters, R and t are called the external parameters, and Z
c is depth coefficient.When the world point falls in thez
=
0 plane, (11) is cast as
(12)
Z
c
[
u
v
1
]
=
[
m
11
m
12
m
13
m
21
m
22
m
23
m
31
m
32
m
33
]
[
x
w
y
w
1
]
.Customarily (12) is rewritten as
(13)
λ
i
[
x
i
′
y
i
′
1
]
=
H
[
x
i
y
i
1
]
,
where H
=
(
h
i
j
)
3
×
3 is called a homography.Based on (9), we get the corresponding land-point pairs between images A and B to estimate 8 parameters of the projective transformation model; pixel (
x
i
′
,
y
i
′
) is the destination pixel in image B corresponding to the source pixel (
x
i
,
y
i
) in image A. Substitute {
x
i
,
y
i
}
,
{
x
i
′
,
y
i
′
} in (13) for the land-point pairs; then the homography H is computed such that
(14)
∑
i
‍
[
(
x
i
′
-
h
11
x
i
+
h
12
y
i
+
h
13
h
31
x
i
+
h
32
y
i
+
h
33
)
2
+
(
y
i
′
-
h
21
x
i
+
h
22
y
i
+
h
23
h
31
x
i
+
h
32
y
i
+
h
33
)
2
]
.In many practical situations this assumption is not valid because land-points with noise are mismatched [15]. To improve the robustness of the transformation model, random sample consensus (RANSAC) procedure is applied to the estimation of homography matrix [16].


## 2.4. Projection of the Reference Camera Based on GCPs

Given the center of visual range of camerai
C
i
(
i
=
1,2
,
…
) and the center of the panorama C
0, then the reference camera i
* is chosen via
(15)
i
*
=
arg
min
⁡
i
|
C
i
-
C
0
|
.As shown in Figure3, camera 3 is the reference camera satisfied with (15).Figure 3
Visual field of each camera.The view of reference camera will be projected to the ground plane based on GCPs by means of measurement. That is to say, the view has been transformed to overhead view till now. For any quadrangleA
B
C
D on the ground, the intersection of edges A
C and B
D is the origin of the world coordinate and x-axis parallels A
B pointing right. The coordinate system and length of each edge are as shown in Figure 4.Figure 4
World coordinate system.Given|
A
B
|
=
a, |
B
C
|
=
b, |
C
D
|
=
c, |
D
A
|
=
d, |
B
D
|
=
e, and |
A
C
|
=
f, according to simple geometric relations, the vertices of quadrilateral A
B
C
D are as below [17]:
(16)
x
A
=
-
α
tan
⁡
β
tan
⁡
α
+
tan
⁡
β
,
y
A
=
-
x
A
tan
⁡
α
,
x
B
=
a
-
x
A
,
y
B
=
y
A
,
x
C
=
x
B
-
b
cos
⁡
⁡
∠
B
,
y
C
=
y
B
-
b
sin
⁡
∠
B
,
x
D
=
x
A
+
d
cos
⁡
⁡
∠
A
,
y
D
=
y
A
-
d
sin
⁡
∠
A
.In engineering application, so as to get better visual effect, similarity transformation is necessary:(17)
x
i
′
=
s
x
x
i
+
Δ
x
,
y
i
′
=
s
y
y
i
+
Δ
y
,
i
∈
{
A
,
B
,
C
,
D
}
,
where proper translation Δ
x and Δ
y makes the coordinate nonnegative, and scaling coefficient determines the height from viewpoint to the ground.Suppose the coordinates of 4 points from (16), (17), and the corresponding image points substitute {
x
i
′
,
y
i
′
} and {
x
i
,
y
i
} in (13); then the view of reference camera has been transformed to overhead view.


## 2.5. Viewpoints Unifying

Now suppose there are three cameras. LetG
i be the homography transforming the current viewpoint to overhead one. Suppose camera 2 is the reference camera; then
(18)
G
1
≅
G
2
H
12
,
H
12
≅
H
21
-
1
,
G
3
≅
G
2
H
32
,
H
32
≅
H
23
-
1
,
where H
21 and H
23 are obtained in Section 2.3 and G
2 in Section 2.4.


## 3. 3D Road Scene Based on Multiview Panorama

2D multiview panoramas from multiple cameras installed top near the road were generated in real time. 2D panoramic road texture, satellite map texture, and 3D scene model were fused. The process is referred to as 2D-3D image registration. The whole data architecture has been shown in Figure5.Figure 5
Data architecture of 2D and 3D image registration.


### 3.1. Registration of Panorama and Satellite Map Texture

GPS geodetic coordinate transformation was introduced in [18, 19]. Suppose point of panorama corresponds to GPS geodetic coordinate and the position precision (Table 1) meets the demand of measurement.Table 1
Parameters of GPS.
Item
Property value
Heading
0.05 degrees (4 m baseline)
Pitch
0.10 degrees (4 m baseline)
Position accuracy
0.3 m
Speed accuracy
0.02 m/sThe integrated navigation system XW-ADU5630 was used to obtain the GPS information of panorama’s key-points, and the pixels of satellite map texture corresponding to the key-points also have GPS information. Therefore it is easy to match the panorama with satellite map texture based on GPS information.Applying the set of correspondences (GCPs) to (13), satellite map texture is registered to panorama. Therefore fusion of panorama and satellite map texture can be easily achieved. Panorama with satellite map texture is called extended panorama.


### 3.2. 2D-3D Image Registration

In our system, all the views are projected to the common road plane, which is the region of interest (ROI) of extended panorama. As a result there is much perspective deformation in top buildings above the ground. Therefore valid road area needs to be clipped out and registered to 3D model (Figure12). Therefore observers can toggle 3D rendering engine to view the scene freely.When we align the points on ground in 3D model and corresponding points in extended panorama, 2D-3D image registration is completed. A similarity transformation (or more simply similarity) is conducted so as to align the 3D model and 2D panorama. The similarity can be written more concisely in block form as(19)
[
x
′
y
′
z
′
1
]
≅
[
s
R
t
0
T
1
]
[
x
y
z
1
]
.A similarity in (19) has seven degrees of freedom and can be computed from four point pairs, that is, four corresponding GCPs. The GCPs are manually picked from ground plane of 3D scene model and 2D extended panorama.


## 3.1. Registration of Panorama and Satellite Map Texture

GPS geodetic coordinate transformation was introduced in [18, 19]. Suppose point of panorama corresponds to GPS geodetic coordinate and the position precision (Table 1) meets the demand of measurement.Table 1
Parameters of GPS.
Item
Property value
Heading
0.05 degrees (4 m baseline)
Pitch
0.10 degrees (4 m baseline)
Position accuracy
0.3 m
Speed accuracy
0.02 m/sThe integrated navigation system XW-ADU5630 was used to obtain the GPS information of panorama’s key-points, and the pixels of satellite map texture corresponding to the key-points also have GPS information. Therefore it is easy to match the panorama with satellite map texture based on GPS information.Applying the set of correspondences (GCPs) to (13), satellite map texture is registered to panorama. Therefore fusion of panorama and satellite map texture can be easily achieved. Panorama with satellite map texture is called extended panorama.


## 3.2. 2D-3D Image Registration

In our system, all the views are projected to the common road plane, which is the region of interest (ROI) of extended panorama. As a result there is much perspective deformation in top buildings above the ground. Therefore valid road area needs to be clipped out and registered to 3D model (Figure12). Therefore observers can toggle 3D rendering engine to view the scene freely.When we align the points on ground in 3D model and corresponding points in extended panorama, 2D-3D image registration is completed. A similarity transformation (or more simply similarity) is conducted so as to align the 3D model and 2D panorama. The similarity can be written more concisely in block form as(19)
[
x
′
y
′
z
′
1
]
≅
[
s
R
t
0
T
1
]
[
x
y
z
1
]
.A similarity in (19) has seven degrees of freedom and can be computed from four point pairs, that is, four corresponding GCPs. The GCPs are manually picked from ground plane of 3D scene model and 2D extended panorama.


## 4. Results and Discussion

Our experiments are conducted on the workstation with an Intel Xeon E5-2650 CPU and two graphic cards (NVIDIA GTX Titan Black). Six Samsung web cameras with 2.0-Megapixel are set up at the top of a building next to the road, monitoring a large crossroad. In the experiment, six cameras with three types of focal length (4 mm, 6 mm, and 8 mm) were used.


### 4.1. Abridged General View of Generation of Extended Panorama

(
1) Motion Detection. The hybrid Gauss background model was applied to detect moving blobs (foreground). The results are demonstrated in Figure 6. The moving blobs, especially cars, have high accuracy to generate nice matching pairs.Moving blobs detection: (a) visible image from camera 2; (b) the moving blobs of Figure (a); (c) visible image from camera 3; (d) the moving blobs of Figure (c).
(a)
(b)
(c)
(d)After comotion process, land-points comotion statistics map [9] is shown in Figure 7. The higher the statistics value at a given position the brighter the corresponding pixel in the image.Comotion statistics maps for land-points: (a) visible image from camera 1; (b) comotion statistics map of camera 1; (c) visible image from camera 2; (d) comotion statistics map of camera 2.
(a)
(b)
(c)
(d)(
2) Overhead View. Four points easily to be measured on the ground were chosen, and the corresponding image coordinates were picked manually as shown in Figure 8.Overhead view: (a) visible image from camera 3; (b) overhead view of Figure (a).
(a)
(b)(
3) Panorama and Satellite Map Texture Registration. Figure 9 is the result of merging ROI of real-time traffic panorama and 2D satellite map texture. It is shown in Figure 9 on the left. On the right there are 6-channel real HD video images.Figure 9
Fusion of multiview panorama and satellite map texture.(
4) Viewpoint Roam. The 3D model is driven based on OSG. Users can observe the road at any viewpoint. In Figure 10, we give two screenshots from two viewpoints. It helps observers to inspect the traffic situation in any direction and react promptly in emergency.Effect of augmented reality 3D road scene: (a) from viewpoint 1, (b) from viewpoint 2.
(a)
(b)


### 4.2. Experiments on Other Roads

Figure11 shows the experiments on other two roads based on the method proposed. One is the panorama of civic center from 6-channel videos. The other is the panorama of a plaza from 2-channel videos. The polygon region in red dotted lines is mosaicked image.Mosaicked images of two arterial roads.
(a)
(b)Figure 12
Mosaicked images in [6].


### 4.3. Comparative Experiments

A multisource images (visible image, satellite map texture, and 3D scene model) fusion method was proposed. By applying the method in [6], we have found the regions in the neighborhood of tall buildings have much projective deformation, as shown in Figure 12. The polygon region in red dotted lines is mosaicked image from real-time video images. And the polygon region in sky blue dotted lines is ROI of panorama, which is clipped out in our method as shown in Figure 12.


## 4.1. Abridged General View of Generation of Extended Panorama

(
1) Motion Detection. The hybrid Gauss background model was applied to detect moving blobs (foreground). The results are demonstrated in Figure 6. The moving blobs, especially cars, have high accuracy to generate nice matching pairs.Moving blobs detection: (a) visible image from camera 2; (b) the moving blobs of Figure (a); (c) visible image from camera 3; (d) the moving blobs of Figure (c).
(a)
(b)
(c)
(d)After comotion process, land-points comotion statistics map [9] is shown in Figure 7. The higher the statistics value at a given position the brighter the corresponding pixel in the image.Comotion statistics maps for land-points: (a) visible image from camera 1; (b) comotion statistics map of camera 1; (c) visible image from camera 2; (d) comotion statistics map of camera 2.
(a)
(b)
(c)
(d)(
2) Overhead View. Four points easily to be measured on the ground were chosen, and the corresponding image coordinates were picked manually as shown in Figure 8.Overhead view: (a) visible image from camera 3; (b) overhead view of Figure (a).
(a)
(b)(
3) Panorama and Satellite Map Texture Registration. Figure 9 is the result of merging ROI of real-time traffic panorama and 2D satellite map texture. It is shown in Figure 9 on the left. On the right there are 6-channel real HD video images.Figure 9
Fusion of multiview panorama and satellite map texture.(
4) Viewpoint Roam. The 3D model is driven based on OSG. Users can observe the road at any viewpoint. In Figure 10, we give two screenshots from two viewpoints. It helps observers to inspect the traffic situation in any direction and react promptly in emergency.Effect of augmented reality 3D road scene: (a) from viewpoint 1, (b) from viewpoint 2.
(a)
(b)


## 4.2. Experiments on Other Roads

Figure11 shows the experiments on other two roads based on the method proposed. One is the panorama of civic center from 6-channel videos. The other is the panorama of a plaza from 2-channel videos. The polygon region in red dotted lines is mosaicked image.Mosaicked images of two arterial roads.
(a)
(b)Figure 12
Mosaicked images in [6].


## 4.3. Comparative Experiments

A multisource images (visible image, satellite map texture, and 3D scene model) fusion method was proposed. By applying the method in [6], we have found the regions in the neighborhood of tall buildings have much projective deformation, as shown in Figure 12. The polygon region in red dotted lines is mosaicked image from real-time video images. And the polygon region in sky blue dotted lines is ROI of panorama, which is clipped out in our method as shown in Figure 12.


## 5. Conclusion

This paper implements a reasonable integration of 2D multiview overhead panorama, 2D satellite map texture, and 3D model to synthesize 3D road scene. The 3D road scene monitoring system has been successfully applied to traffic supervision. The fusion method is innovative and enables the observers to have the sense of immersion. Besides, 3D road scene OSG driven can be translated, rotated, and scaled as one wishes.In this paper, we focused mainly on the generation of 2D panorama and the presentation of 3D road scene. In future work, we will research on intelligent video analysis and traffic situation analysis.



---

*Source: 403126-2014-08-21.xml*