Evaluation of Image Processing Algorithms on Vehicle Safety System
Based on Free-viewpoint Image Rendering
Akitaka Oko1,∗ , Tomokazu Sato1 , Hideyuki Kume1 , Takashi Machida2 and Naokazu Yokoya1
Abstract— Development of algorithms for vehicle safety systems, which support safety driving, takes a long period oftime and a huge cost because it requires an evaluation stage
where huge combinations of possible driving situations shouldbe evaluated by using videos which are captured beforehandin real environments. In this paper, we address this problem
by using free viewpoint images instead of the real images.
More concretely, we generate free-viewpoint images from a
combination of a 3D point cloud measured by laser scanners
and an omni-directional image sequence acquired in a realenvironment. We basically rely on the 3D point cloud forgeometrically correct virtual viewpoint images. In order toremove the holes caused by the unmeasured region of the
3D point cloud and to remove false silhouettes in surface
reconstruction, we have developed a technique of free-viewpoint
image generation that uses both a 3D point cloud and depth
information extracted from images. In the experiments, we haveevaluated our framework with a white line detection algorithm
and experimental results have shown the applicability of freeviewpoint images for evaluation of algorithms.

I. Introduction
An image-based vehicle safety system supports safety
driving by detecting and alerting dangerous situations to thedriver, and it reduces the risk of traﬃc accidents. Thus, thedemand for good image processing algorithms for a vehicle
safety system is rapidly growing. One problem here is thecost for evaluating developed algorithms. In order to evaluate
the algorithms, a vast amount of images should be captured
under huge combinations of possible driving situations inreal environments. Although there exist studies [1], [2],[3] that use computer graphics (CG) images for evaluation,
experimental results showed that the reality of CG images
are not suﬃcient for evaluating these algorithms [4]. On theother hand, a free-viewpoint image rendering method basedon real images can be used for generating images of roadenvironments for evaluation. However, actually this kind oftechniques has not been used for evaluation of algorithms
on vehicle safety systems because the quality of generated
images does not reach the suﬃcient level.
In this paper, we propose a free-viewpoint image rendering
framework for generating images for a virtual camera pathas shown in Figure 1 that can be used for evaluating image
processing algorithms on vehicle safety systems. When we
focus on the algorithms such as white line detection, roaddetection, and obstacles detection, (i) errors of 3D geometry
1 Graduate School of Information Science, Nara Institute of Scienceand Technology (NAIST), Nara, Japan. {akitaka-o, tomoka-s,

hideyuki-k, yokoya}@is.naist.jp
2 Toyota

Central

R&D

Laboratory,

machida@mosk.tytlabs.co.jp

Incorporated,

Aichi,∗ He is currently working at Honda Motor Company, Limited.

Japan.

and (ii) lack of information in generated images will makethe evaluation results inaccurate. For reconstructing accurate
geometry in the generated images for virtual viewpoints, wefirst measure a 3D point cloud from surrounding environments of driving roads by LiDAR mounted on a vehicle
and apply a surface reconstruction technique [5]. In order
to compensate missing regions in the generated depth maps
due to occlusion or lack of sampling points, we also extract
3D information from omni-directional images whose camera
positions are known. By using both the 3D point cloud
and the depth information extracted from images, we cangenerate free-viewpoint images based on compensated 3D
geometry.
The main contributions of our work are: (1) developmentof a high-quality free-viewpoint image generation framework
which focuses on generating accurate and hole-less images,(2) world-first applicability verification of the use of thesynthesized free-viewpoint images in place of the real images
in the evaluation stage for actual image processing algorithms
on vehicle safety systems.
This paper is outlined as follows. In Section 2, evaluation
methods for safety systems and free-viewpoint image rendering techniques are reviewed. In Section 3, the proposed
framework and its implementation are detailed. Evaluation
results are presented in Section 4. Finally Section 5 summarizes the present study.
II. Related Work
A. Evaluation of image processing algorithms on vehicle
safety systems
One of the straight-forward ways to reduce the cost foralgorithm evaluation is utilization of virtual environments
constructed by CG [1], [2], [3]. In a virtual environment, any
combinations of situations can be virtually generated. TASS
[1] developed such virtual environments, and algorithms
including white line detection can be evaluated using thegenerated environment. Weinberg et al. [2] proposed a CGbased driving simulator. Although such a CG-based environment can be used to evaluate image processing algorithms ofvehicle safety systems, the experimental results showed thatthe reality of generated images is not suﬃcient to use CG
images for evaluating these algorithms [4]. In order to generate photo-realistic CG images for this purpose, a huge cost isrequired. For increasing the reality of generated images, Satoet al. [3] employed a free-viewpoint image rendering method
based on real images for generating background images ofthe scene. However, important parts of the scene such as
road surfaces or curbs for evaluating algorithms were stillFig. 1. Example of generated images for virtual camera path that simulates lane change. These images were generated by data captured from the vehicle
running on the left lane.

generated based on CG models. Although the use of a virtualenvironment or a virtualized real environment is expected to
drastically reduce the time and cost for developing image
processing algorithms for vehicle safety systems, existing
techniques for generating images do not reach a suﬃcientquality of images for evaluation.
B. Virtualization of real world by surface reconstruction
One alternative to the CG-based systems is to use avirtualized 3D world based on real world data. With thedevelopment of 3D scanning systems for outdoor environments, e.g., Topcon IP-S2 [6] and Velodyne HDL-64E [7],it becomes much easier to acquire a 3D point cloud andtextures for large outdoor environments.
In order to generate textured 3D models of road environments, 3D surfaces should be first generated from the 3D
point cloud. Schnabel et al. [8] proposed a method that fits
3D primitives to the 3D point cloud. It successfully works
when the geometry of a target 3D point cloud is suﬃciently
simple to be approximated by the primitives. Yang et al.
[9] proposed a method called surface splatting that allocatesa local plane for each point to reconstruct 3D surfaces. In
order to generate smoother surfaces than a combination ofplanes or primitives, Liang et al. [5] proposed a surface
reconstruction method that estimates surfaces as the boundaries of interior and exterior regions in a 3D point cloud.
Although this surface reconstruction technique can fill small
gaps in the 3D point cloud by finding local minima of anenergy function that is based on the mixture of Gaussian, theproblem of the artifacts called fattened-silhouettes, where an
object looks fatter than the actual geometry, often appears ingenerated 3D surfaces. In this paper, we reduce the fattenedsilhouettes using a view-dependent depth test technique withmultiple depth maps estimated from images.
One remaining problem of surface reconstruction is missing surfaces for the region where no 3D point data is
acquired. These unrecovered surfaces will be a severe problem in evaluation of an image-based vehicle safety systembecause the objects that should exist will disappear from theimage. In this paper, we call such artifacts cut-silhouettes
and we tackle with this problem by using a free-viewpoint
image rendering technique with view-dependent geometry
and view-dependent texture mapping.
C. Free-viewpoint image rendering
Free-viewpoint image rendering is a technique that generates an image whose viewpoint is freely determined. Modelbased rendering (MBR) directly renders the image froma virtual viewpoint using view-independent 3D geometry(3D model) and view-independent (fixed) textures. Althoughreconstructed 3D surfaces by the methods introduced in theprevious section can be rendered by MBR, errors of thereconstructed 3D model appear as distortion in the rendered
textures in this simple renderer. In order to reduce such distortion, view-dependent texture is known as an eﬀective solution [10]. In addition to the use of view-dependent texture
mapping, a view-dependent geometry generation technique[11] is also proposed to remove the problem of the missing
surfaces in generated images. In this paper, although we
employ the combination of view-dependent texture mapping
and view-dependent geometry, we extend the existing freeviewpoint image rendering techniques for reducing fattened/cut-silhouettes in order to generate photo-realistic images
for evaluation of vehicle safety systems.
III. Free-viewpoint Image Rendering from 3D Point Cloudand Images
A. Overview
Figure 2 shows the pipeline of free-viewpoint image
rendering in the proposed framework. The pipeline is consistsof two stages: (a) the pre-processing stage and (b) the image
rendering stage. In the pre-processing stage, a 3D point
cloud and omni-directional images of a road environment are
acquired using a scanning vehicle such as IP-S2 [6]. Fromthese data, an omni-directional dense depth map is generated
for each frame of the omni-directional image consideringboth the 3D point cloud and the image data. In the rendering
stage (b), after setting a virtual camera position, we estimate
a view-dependent depth map for the virtual camera position.
The estimated depth map is then refined to reduce fattened/cut-silhouettes using view-dependent depth test (VDDT)
with multiple dense depth maps prepared in the stage (a).
Using the refined depth map for the virtual camera position,a free-viewpoint image is rendered using a view-dependent
texture mapping technique. Each stage in our pipeline is
described in the following sections.
B. Pre-processing stage(a-1) Acquisition of 3D point cloud and omni-directional
imagesIn the proposed framework, a 3D point could and omnidirectional images that are taken along a road are requiredwith the accurate positions of a scanning vehicle. In order
to collect these data, we employed Topcon IP-S2 [6]. In thissystem, three SICK LiDARs, Point Grey Ladybug 3, Topcon

(a) Pre-processing stage(a-1) acquisition of 3D point cloud andomni-directional images(a-2) generation of omni-directional depth maps(a) original image(b) re-projected 3D point cloud(c) segmented image(d) dense omni-directional
depth map(b) Image rendering stage(b-1) initial depth map estimation by viewdependent surface reconstruction(b-2) refinement of depth map by VDDT(b-3) view dependent texture mapping
Fig. 2.

Fig. 3.

Dense depth map estimation using both 3D point cloud and image.
A pixel colored in pink indicates an infinity depth value.

Pipeline of proposed free-viewpoint image rendering framework.

fattened-silhouette

RTK-GPS, and an odometry sensor are mounted on a vehicle,
and the 3D point cloud of a surrounding environment withabsolute position and omni-directional images can be easilycollected by driving the vehicle.
(a-2) Generation of omni-directional depth maps
In order to remove fattened-/cut-silhouettes in generated
images, we estimate an omni-directional dense depth map
for each omni-directional image using both the 3D point
cloud and the omni-directional image (Figure 3). Here, foreach viewpoint, we first select the visible 3D points out of
the 3D point cloud by comparing the distances from theviewpoint to the 3D point and estimate the surface by using
the surface reconstruction technique by Liang et al. [5]. Afterremoving the hidden points that exist on the far side of
the reconstructed surface, sparse depth data generated byprojecting selected 3D points (Fig. 3 (b)) are converted toa dense depth map (Figure 3 (d)) using the image data. For
compensating infinite depth region including the sky region,
we first divide the omni-directional image into the sky region
and the ground region based on the intensity of the pixels.
After that, as shown in Figure 3 (c), the ground region of
the omni-directional image is segmented by the watershedsegmentation algorithm [12] using the projected point as aseed of each segment, and the depth value for every pixel isfilled based on the segments with the depth of the seed.
C. Image rendering stage(b-1) Initial depth map estimation by view-dependent
surface reconstruction
In the image rendering stage, we first estimate an initial
depth map by applying Liang’s surface reconstruction technique [5] with a view-dependent manner. More concretely,for each pixel on an image of a virtual camera, 3D points
that exist around the ray from a viewpoint ĉv of the virtual
camera to the direction of the pixel on the image plane are
selected first. Using selected 3D points, the closest depth,
for which the score function defined in the paper [5] gives a
local maximum, is selected. Figure 4 shows an example ofthe estimated initial depth map and the generated image. As
shown in this figure, generated image with the initial depth

cut-silhouette(a) depth map
Fig. 4.

(b) free-viewpoint image

Depth map and free-viewpoint image with fattened/cut-silhouette.

map has fattened-/cut-silhouettes.
(b-2) Refinement of depth map by VDDTIn order to reduce the fattened-/cut-silhouettes, we employ
view-dependent depth test (VDDT). The idea is inspired by
Yang’s method [9], but we extend the test scheme to handle
both two problems with a noisy input. VDDT consists offattened-silhouette removal and cut-silhouette compensation
as follows.
(i) Fattened-silhouette removal: Fattened-silhouette is detected for each pixel by checking the consistency of thedepths prepared in the pre-processing stage. As shown in
Figure 5, 3D position p̂ x for target pixel p x is recovered using
the initial depth. This 3D position p̂ x is then projected onto
the original images of the top N-nearest camera positionsci (i = 1 . . . N) and 3D position pi which is recovered using
the projection result for p̂ x to ci is acquired. Using theserecovered 3D points, the consistency of estimated depths forthe 3D position p x is evaluated as the sum of weighted errors
as follows:∑
wi ||pi − p̂ x ||Err(p x ) = i ∑
,
i wi

(1)
1


 θ+λ||p̂x −ci || ; ||p̂ x − ci || ≤ ||pi − ci ||wi = 
,
0; otherwise
where θ in [0, π] is defined by θ = ∠ĉv p̂ x ci as in Figure 5
and λ is a pre-defined parameter. Using this error function, an
initial depth that is inconsistent with the depths on original
images is detected. For the pixels for which Err exceeds a

3000>
2500
2000
1500
1000
500
0

map
(a)
considering fattened-silhouette

Fig. 6.

(b) depth map without
fattened-silhouette

Fattened-silhouette removal result.

Virtual camera
Original camera
3000>
2500

Fig. 5.

Parameters in VDDT.
2000

given threshold, we find another depth value that greater than||p̂ x − ĉv || using the method [5]. Figure 6 shows an example
of function Err and a depth map after refinement. From thisfigure, we can see that fattened-silhouettes of objects are
successfully removed.
(ii) Cut-silhouette compensation: After removing fattenedsilhouettes, missing surfaces in the estimated depth map arecompensated. For detection of cut-silhouettes, we use thesame function Err defined in Eq. (1) with diﬀerent weight
ŵi :

1


 θ+λ||p̂x −ci || ; ||p̂ x − ci || > ||pi − ci ||ŵi = .
(2)0; otherwise
We judge the pixels, where Err with weight ŵi exceeds agiven threshold, as cut-silhouettes and for these pixels, wefind an alternative depth value using the image-based freeviewpoint rendering technique proposed by Sato et al. [11].
Figure 7 shows an example of the function Err with ŵi anda compensated depth map where cut-silhouettes of objects
are compensated.
(b-3) View-dependent texture mapping
For each pixel in the image for a virtual camera, its color isdetermined from omni-directional images which are acquiredin the pre-processing stage (a-1). Here we employ the viewdependent texture mapping technique by Sato et al. [11] todetermine the pixel color.
IV. Experiments
In this section, we verify the applicability of the proposed
framework for evaluation of image processing algorithms on
vehicle safety systems. For this purpose, we have compared
the outputs of the white line detection algorithm [13] forreal images and virtual images generated by the proposed
framework. Figure 8 shows a 3D point cloud and an example
frame of the omni-directional image sequence for the roadand the surroundings acquired by Topcon IP-S2 [6] forthis experiment. While capturing these data, real images forthe comparison were also taken by two real camera units(Point Grey Research: Flea3) mounted on the vehicle at thepositions shown in Figure 9. Table 1 shows parameters of
the real cameras used in this experiment.

1500
1000
500
0

map
(a)
considering cut-silhouette

Fig. 7.

(b) depth map with compensatedregion of cut-silhouette

Result for compensating region of cut-silhouette.

From acquired data, we have generated two image sequences (31 frames for each sequence) for two virtual
cameras that simulate a 60 m drive with the speed of 36
km/h. The camera parameters and camera paths of two
virtual cameras were manually set to be the same as thoseof the external cameras. Table 2 shows the translation androtation of the virtual cameras from the omni-directional
camera. As shown in Figure 10 (a), a generated virtual image
had some diﬀerences from a corresponding real image in thetone of color and the bottom part of the image where thevehicle was framed-in in the real image, and disappeared in
the generated image. For reducing undesired eﬀects due tothese diﬀerences, we manually adjusted the tone of color ofgenerated images and cropped the bottom parts (8% for thecenter camera and 10% for the right camera) of both realand generated images (Figure 10 (b)).
Figure 11 shows comparison between real images andvirtual images generated by the proposed framework. As we
can see in this figure, very similar images to the real images
were successfully generated except for tiny artifacts.
Figure 12 shows white lines detected by the existing
method [13] in the images shown in Figure 11. Oﬀset
positions of white lines detected from the center and theright camera positions are shown in Figure 13. For the center
camera, the average absolute diﬀerences between pairs ofoﬀset positions for the real and virtual cameras were 37 mm
and 24 mm for the right and left white lines, respectively.
For the right camera, they were 35 mm and 25 mm. The
diﬀerence is reasonable considering the stability and theaccuracy of the white line detection algorithm.
This evaluation was done for a very short video sequenceas the first step of the development. We recognize the neces-Table 1. Specifications of the camera units.

resolution
field of view

640 × 512 px.
67 × 56 deg.

Table 2. Translation and rotation of virtual cameras from omni-directional
camera.

vertical translation
horizontal translation
yaw rotation

center camera-70.8 cm
1.4 cm
0.0 deg.

right camera-72.8 cm
40.4 cm
3.5 deg.

(a) original real image (left) and generated image (right)(b) edited real image (left) and generated image (right)
Fig. 10.

Adjustment of a tone of color and cropping of bottom part ofimages.

Fig. 8. Acquired 3D point cloud (left) and omni-directional image (right).

omni-directional
cameraAcknowledgement

external
right camera
external
center camera
Fig. 9.

Positions of the camera units.

sity for doing a large scale evaluation with various algorithms
using the proposed framework. One remaining problem ofsuch a large scale evaluation is computational cost for
generating images. Currently, using Intel(R) Core(TM) i73970X (and nVidia GTX680 in the processes (a-2), (b1)), generating one omni-directional depth map in the preprocessing stage took 58 minutes and generating one freeviewpoint image in the rendering stage took 72 minutes, on
average.
V. Conclusion
In this paper, we have proposed a free-viewpoint image
rendering technique and outputs of the white line detection
algorithm have been compared using real images and virtual
images generated by the proposed framework. From theresults, the errors of detected positions of white lines werevery small. This indicates the applicability of our freeviewpoint image rendering framework as an alternative tothe real images to evaluate image processing algorithms on
vehicle safety systems. In the future, we will improve theeﬃciency of the proposed framework and its implementation,
and we will conduct a large scale evaluation with various
image processing algorithms on vehicle safety systems andvarious driving situations.

This research was supported in part by Toyota Central
R&D Lab., Inc., the Japan Society for the Promotion of
Science (JSPS) Grants-in-Aid for Scientific Research, Nos.
23240024 and 25540086.
References[1] TASS prescan https://www.tassinternational.com/prescan-overview[2] G. Weinberg, and B. Harsham, “Developing a Low-Cost Driving Simulator for the Evaluation of In-Vehicle Technologies,” Proc. Int. Conf.
Automotive User Interfaces and Interactive Vehicular Applications,pp.51-54, 2009.
[3] R. Sato, S. Ono, H. Kawasaki, and K. Ikeuchi, “Real-time Image
Based Rendering Technique and Eﬃcient Data Compression Method
for Virtual City Modeling,” Proc. Meeting on Image Recognition andUnderstanding, IS-3-28, pp.1087-1092, 2007. (in Japanese)[4] H. Yasuda, N. Yamada and E. Teramoto, “An Evaluation Measure ofVirtual Camera Images for Developing Pedestrian Detection Systems,”
Proc. 2011 IEICE General Conference, D-11-80, p.80, 2011. (inJapanese)[5] J. Liang, F. Park, and H. Zhao, “Robust and Eﬃcient Implicit Surface
Reconstruction for Point Clouds Based on Convexified Image Segmentation,” Journal of Scientific Computing, Vol.54, No.2-3, pp.577-602,
2013.
[6] IP-S2 http://www.topcon.co.jp/en/positioning/products/product/3dscan
ner/ip s2.html[7] HDL-64E http://velodynelidar.com/lidar/hdlproducts/hdl64e.aspx[8] R. Schnabel, P. Degener, and K. Reinhard, “Completion and Reconstruction with Primitive Shapes,” Proc. Eurographics, Vol.28, No.2,
pp.503-512, 2009.
[9] R. Yang, D. Guinnip, and L. Wang, “View-dependent Textured Splatting,” The Visual Computer, Vol.22, No.7, pp.456-467, 2006.
[10] P. Debevec, Y. Yu, and G. Borshkov, “Eﬃcient View-dependent Imagebased Rendering with Projective Texture-mapping,” Proc. Eurographics Rendering Workshop, pp.105-116, 1998.
[11] T. Sato, H. Koshizawa, and N. Yokoya, “Omnidirectional Freeviewpoint Rendering Using a Deformable 3-D Mesh Model,” Int.
Journal of Virtual Reality, Vol.9, No.1, pp.37-44, 2010.
[12] S. Beucher, and F. Meyer, “The Morphological Approach to Segmentation: The Watershed Transformation,” Mathematical Morphology in
Image Processing, Marcel Dekker Press, pp.433-481, 1992.
[13] A. Watanabe, and M. Nishida, “Lane Detection for a Steering Assistance System,” Proc. IEEE Intelligent Vehicles Symp., pp.159-164,
2005.

Fig. 11.

frame #1

frame #1

frame #11(a) center camera

frame #11(b) right camera

Comparison of real images (left column) and virtual images (right column) for corresponding positions.

frame #1

frame #1

frame #11(a) center camera

frame #11(b) right camera

Fig. 12.

Result of white line detection for real images (right column) and virtual images (left column).

31

31

25

25

frame number

frame number

20
15
10
5

20
15
10
5

1

1

-1.5

-1.0

-0.5

0.0 0.5 1.0
offset distance

1.5

2.0 [m]

-2.0

-1.5

-1.0 -0.5 0.0 0.5
offset distance

1.0

1.5

distance to left white linefrom center real images

distance to right white linefrom center real images

distance to left white linefrom right real images

distance to right white linefrom right real images

distance to left white linefrom center virtual images

distance to right white linefrom center virtual images

distance to left white linefrom right virtual images

distance to right white linefrom right virtual images(a) center camera
Fig. 13.

(b) right camera

Oﬀset positions of detected white lines for real and virtual images.

[m]