IEICE TRANS. INF. & SYST., VOL.E91–D, NO.7 JULY 2008

1893PAPERSpecial Section on Machine Vision and its Applications

Precise Top View Image Generation without Global MetricInformation
Hiroshi KANO†a) , Member, Keisuke ASARI† , Nonmember, Yohei ISHII† ,and Hitoshi HONGO† , Members

SUMMARY We describe a practical and precise calibration method forgenerating a top view image that is transformed so that a planar object suchas the road can be observed from a direction perpendicular to its surface.
The geometric relation between the input and output images is describedby a 3 × 3 homography matrix. Conventional methods use large planar calibration patterns to achieve precise transformations. The proposed method
uses much smaller element patterns that are placed in arbitrary positions
within the view of the camera. One of the patterns is used to obtain an
initial homography. Then, the information from all of the patterns is usedby a non-linear optimization scheme to reach a global optimum homography. The experiment done to evaluate the method showed that the precisionof the proposed method is comparable to that of the conventional method
where a large calibration pattern is used, making it more practical for automotive applications.
key words: top view image, homography, calibration

1. Introduction
The number of rear view backup cameras being installedin automobiles is increasing. The main use of such cameras is to show the blind spots of vehicles, however, it isdiﬃcult to correctly perceive distances by simply viewing
the direct input image because of the perspective eﬀect of
the camera. A top view image transformation, shown inFig. 1, was proposed to eliminate the perspective eﬀect tohelp drivers park [1]. It is also used to detect lane markers
on the road [2].
To generate a top view image, the camera must be calibrated. This is usually a process of obtaining the camera’s
parameters such as its focal length, pixel size, mounting location, and mounting angles. When a top view image isgenerated, a homography matrix [3] is computed from theseparameters. The homography matrix is a 3 × 3 planar projective transformation matrix, transforming the input image
into a top view image. In this paper, calibration means theprocess of computing the 3 × 3 matrix.
To compute the matrix, there must be at least four setsof corresponding points shared by the input and the transformed images. However, in general, to transform the image precisely, many corresponding points should be laid out
evenly across the entire image. This gives rise to a practicalproblem when conventional calibration methods are applied
Manuscript received October 26, 2007.
Manuscript revised January 25, 2008.
†The authors are with the Digital Technology Research Center, R&D H.Q., SANYO Electoric Co., Ltd., Daito-shi, 574–8534
Japan.
a) E-mail: Hiroshi.Kano@sanyo.com
DOI: 10.1093/ietisy/e91–d.7.1893

Fig. 1Top view image generation.

Fig. 2

Image of rear view camera.

to an automobile. Figure 2 shows a typical image obtainedby a rear view camera. The road surface that the camera
covers is large because the field of view is wide and deep.
The calibration pattern used in this view must have reference points associated with 2D absolute coordinates. As aresult, the calibration pattern becomes larger, for example
10 m × 10 m, and must contain precise reference points.
We propose a practical calibration method to solve theproblem that does not require a global 2D coordinate in thecalibration area. Instead, it uses only the shape informationof several smaller square patterns to compute the homogra-c 2008 The Institute of Electronics, Information and Communication Engineers
Copyright IEICE TRANS. INF. & SYST., VOL.E91–D, NO.7 JULY 2008

1894

phy matrix. It is enough to place them roughly so that they
cover a large area of a camera view as a whole. The proposed method is useful especially when a rear view camera
is installed in a location such as an automotive body shopwhere it is diﬃcult to keep a wide space for camera calibration.
The rest of the paper is organized as follows: in Sect. 2,
we describe conventional calibration methods, Sect. 3 describes the basic idea of the proposed method, Sect. 4
presents results of both a simulation and an experiment using actual images, and in Sect. 5, we conclude the paper.
2. Camera Model, Calibration, and Homography
2.1 Camera Model and Homography
From the point of view of geometrical characteristics, a pinhole camera is a device that transforms a 3D world coordinate into a 2D image coordinate [3]. This relation is described by Eq. (1). Here, matrix P3×4 is a 3 × 4 matrix calleda projective transformation matrix. [X, Y, Z, 1]t is a homogeneous 3D world coordinate point in the scene, and [u, v, 1]t is
the corresponding homogeneous 2D image coordinate point.
There is equality in the homogeneous coordinate in (1).
[u, v, 1]t  P3×4 · [X, Y, Z, 1]t(1)Matrix P3×4 is decomposed into Eq. (2). Here, K3×3 isa 3 × 3 matrix, the elements of which are computed from internal camera parameters such as the focal length and pixel
size. R3×3 is a 3 × 3 rotation matrix, t3×1 is a 3 × 1 translation vector, and [R3×3 |t3×1 ] is a 3 × 4 matrix concatenating
R3×3 and t3×1 ; The camera position and angles relative to the3D world coordinate frame. These are the external camera
parameters.
P3×4 = K3×3 · [R3×3 |t3×1 ](2)

When there is a plane in the scene, a 3D coordinate ofa point on the plane is described by x · i + y · j + d, where(x, y) is the 2D coordinate on the plane, i and j are the directional unit vectors along the axes, and d is the point oforigin. Therefore, the homogeneous 3D coordinate of thepoint on the plane is described by Eq. (3), where Q is a 4 × 3matrix. By substituting (3) for (1), we obtain Eq. (4). Equation (4) indicates that the relation between the 2D coordinates of a plane in the scene and the image plane is describedby a 3 × 3 matrix H in the homogeneous sense. The 3 × 3
matrix is called the homography matrix.

x·i+y·j+dt[X, Y, Z, 1] =1(3)= Q4×3 · [x, y, 1]t[u, v, 1]t  P3×4 · Q4×3 · [x, y, 1]t= H3×3 · [x, y, 1]t

is to follow the steps, described in 2.1, one by one. For
example, by using Zhang’s multiple-plane calibration [4], it
is possible to obtain Eqs. (1) and (2), and then by using theequation of a plane corresponding to the road, we can obtain
homography that transforms the image plane into the roadplane. However, this method is impractical for automotive
applications because the calibration pattern is large and has
to be observed from several diﬀerent directions.
Another similar method is to obtain the internal camera
parameters beforehand and then use the design values of theexternal camera parameters to determine Eqs. (1), (3) and(4). However, this method does not work well, because thedesign values of the external parameters are not suﬃciently
accurate. As a result, the top view images generated by thismethod frequently become distorted, as illustrated in Fig. 3.
The two methods described above are indirect in thesense that the camera is calibrated as a 3D-to-2D projective
device first, and then the 2D-to-2D projective homography
is computed afterwards. As a matter fact, no 3D-to-2D projective characteristics are needed to determine the homography.
In a direct method, a large calibration pattern is placedon the road. On the surface, there are some reference points
with absolute 2D coordinate values. Four sets of corresponding points between the reference points of the input
image and the calibration pattern are suﬃcient to computethe matrix. However, in general, to obtain a precise matrix,
many corresponding points should be laid out evenly acrossthe entire image. The precision of this method is illustrated
in Fig. 4. Though this method is stable, it requires a large
calibration pattern. This direct method is hereafter referred
to as the conventional method.

Fig. 3 Left: input image, right: top view image using design values ofexternal parameters.

(4)

2.2 Calibration of Homography Matrix
One elaborate method of obtaining the homography matrix

Fig. 4 Left: input image, right: top view image by direct method using alarge pattern (circles are reference points used for calibration).

KANO et al.: PRECISE TOP VIEW IMAGE GENERATION

1895

3. Proposed Method without Global Metric Information
3.1 Basic Idea
It is thought that the calibration pattern must cover a large
area of the input image for the calibration to be precise. The
conventional method uses a large planar pattern, such as acheckerboard. The proposed method uses smaller elemental
patterns and places them at arbitrary positions on the road’s
surface, as illustrated in Fig. 5. It is the same as the conventional method in that the patterns cover a major part of theinput image as a whole. However, it is diﬀerent in that thepatterns are not precisely aligned with each other.
Not all element patterns are necessarily the same, but
the dimensions of all of them must be known in advance.
One of them, which is selected in order to estimate an ini-tial homography, must have at least four reference points. In
this paper, all of the element patterns are the same and aresquare in shape. They are placed evenly, but not necessarily aligned precisely. Starting from an initial homography
computed using one of the element patterns, the tentative
homography matrix is optimized by iteratively minimizing
the error function, taking into account all patterns.
3.2 Optimization Procedure
The procedure for optimizing the homography is shown inFig. 6. In the first step, one of the element patterns is selected. In the second step, a homography matrix that transforms the selected pattern into a fixed size square is computed. Then, this matrix is used as the initial value of an
iterative process in the third step, where the matrix is modified iteratively using an evaluation function obtained fromall patterns.
The evaluation function is defined by Eq. (5). Here dn
is the distance between two corresponding corners of an element pattern and a pattern transformed by a tentative homography, as illustrated in Fig. 7. The two quadrangles arelaid out as shown. There are three dn’s for each pattern andall dn’s over all patterns are summed up to obtain one evaluation value. Powell’s quadratically convergent method isused as the algorithm to minimize the evaluation function.
⎛⎞ ⎜⎜⎜ ⎟⎟2⎟⎜⎜⎜⎟⎟⎟
|dn|
D=(5)
⎝⎠
all patterns each patternFig. 5

Conventional method and proposed method.

4. Experimental Results
4.1 Accuracy Evaluation by Simulation

Fig. 6

Fig. 7

Homography optimization procedure.

Error between element pattern and transformed quadrangle.

We conducted a simulation experiment as follows. It wasassumed that there were 100 points on a plane surface. They
were aligned on an equally-spaced 10 × 10, 100 mm squaregrid, as shown in Fig. 8 (a). These reference points wereobserved using a camera. It was assumed that a VGA image
such as the one shown in Fig. 8 (b) was obtained. Random
noise of average 0 and standard deviation σ was added tothe points in the camera image to obtain the observed points.
The small dots in Fig. 8 (b) are the points corresponding to
the reference points, and the small crosses are the observed
points that fluctuate around the dots. (The small crosses in
the figure do not represent the actual computed positions, theactual positions are actually much closer to the dots. They
are shown only for ease of understanding).
In the experimental evaluation of the proposed
method’s accuracy, the vertices of several squares wereused as reference points to compute the homography matrix. Point correspondences were used for the conventional
method. Figure 9 shows the reference points and the squares
used for these evaluations. Sixteen evenly distributed points
on the plane were used for the conventional method (A). Weevaluated four cases for the proposed method: two squares

IEICE TRANS. INF. & SYST., VOL.E91–D, NO.7 JULY 2008

1896

Fig. 8Fig. 9

Accuracy evaluation by simulation: five cases.

Table 1Results of accuracy evaluation by simulation.

Simulation data.

(B1), four inner squares (B2), four outer squares (B3) andnine inner squares (B4). We considered that the precision ofhomography obtained by A was the standard that should beachieved by the proposed method.
Table 1 summarizes the results. In this table, σ isthe standard deviation of random noise added to the observed points, Ave. is the average of the distances between
the reference points mapped onto the image coordinate using the computed homography and the observed points, andSD is their standard deviation. Table 1 shows that B1 (twosquares) is the worst and B2 (four inner squares) is not as
good as A. B3 is comparable to A and B4 is a little bit worse
than A. We also found that the more squares we used for theproposed method, the more precise the obtained homography became. From this experiment, we found that placement of four square patterns in the outer area is comparable
to the conventional method using 16 reference points evenly
spaced within the calibration area.

4.2 Accuracy Evaluation Using Actual Images
We conducted an experiment to test the eﬀectiveness of theproposed method. We used the CCA-BC200 manufactured
by SANYO Electric Co., Ltd., as the rear view camera. The
horizontal view angle of the camera is 135 degrees. The
lens distortion was corrected in advance as shown in Fig. 10.
We prepared two patterns for the accuracy evaluation. OneKANO et al.: PRECISE TOP VIEW IMAGE GENERATION

1897

Fig. 10

Distortion correction of wide angle camera.

was an all-checkerboard pattern. The other was a roughlyplaced-four-square pattern. The four-square pattern also hada checkerboad on it for the accuracy evaluation.
As shown in (AA) of Fig. 11, for the accuracy evaluation of the conventional method, ten intersecting points wereselected from the all-checkerboard pattern. The ten pairs ofinput and transformed image coordinates were carefully extracted. For each evaluation of the proposed method using
the all-checkerboard pattern, a diﬀerent number of squareson the checkerboard was used, as shown in (BB1) to (BB3)in Fig. 11. The number used varied between one, two andfour. We also conducted the accuracy evaluation of the proposed method using the roughly-placed-four-square pattern.
For this pattern, the four squares surrounding the checkerboard were used for the calibration and the checkerboard
was used only for the accuracy evaluation.
The calibration results are shown in Table 2. An error is the distance between a coordinate value computed by
the estimated homograpy and the ideal coordinate value ofeach intersecting point, as shown by the grid lines in Fig. 11.
Max. is the maximum value of the errors of the all intersecting points, Min. is the minimum value of the errors, Ave. isthe average, and SD is the standard deviation. Area in Table 2 is the area covered by the calibration pattern for eachcase. The unit value of Area is the area of one square in thecheckerboard.
Table 2 shows that, in terms of SD, AA is 2.92, BB3 is
3.10 and CC is 2.89, and in terms of Ave., AA is 3.94, BB3is 4.73 and CC is 3.18. Therefore, it can be seen that theaccuracy of BB3 and CC, both of them using the proposed
method where four squares placed at similar positions, iscomparable to AA. In terms of Area, AA is 54. BB3 andFig. 11

Accuracy evaluation in real environment: four cases.

Table 2

Experimental accuracy results.

CC are only 4. This means that when these methods areused for the rear view camera of an automobile, the cost of
installing the calibration pattern on the road surface is muchsmaller for BB3 and CC than for AA.
4.3 Application to Actual Rear View Camera
The proposed method using two square patterns was applied
to a rear view camera installed onto a vehicle. The pattern is
1.5 m × 1.5 m. The generated image is shown in Fig. 12. In
the top view image, the lower portion around the two squarepatterns are precise but a little distortion is found in the upper portion of the image. This is because we used only two

IEICE TRANS. INF. & SYST., VOL.E91–D, NO.7 JULY 2008

1898

“Image processing technology for rear view camera (1): Developmentof lane detection system,” R&D Review of Toyota CRDL, vol.38,no.2, pp.31–36, 2003.
[3] R. Hartley and A. Zisserman, Multiple view geometry in computervision, Second ed., p.87–93, Cambridge Univ. Press, 2003.
[4] Z. Zhang, “A flexible new technique for camera calibration,” IEEE
Trans. Pattern Anal. Mach. Intell., vol.22, no.11, pp.1330–1334, 2000.
[5] W. Press, S. Teukolsky, W. Vettering, and B. Flannery, Numerical
Recipes in C, Second ed., Cambridge University Press, 1992.

Hiroshi Kano
received his B.E., M.E. andD.E. degrees in information engineering from
Kyoto University in 1982, 1984 and 1998 respectiviley. He is currently a senior manager
at the Digital Technology Research Center inSANYO Electric Co., Ltd. His research interest includes image processing, computer vision,and their applications to camera equipment suchas digital cameras and on-vehicle cameras.

Keisuke Asarireceived his B.E. andM.E. degrees in information engineering from
Okayama University in 2004 and 2006 respectiviley. He is currently a researcher at the Digital
Technology Research Center in SANYO Electric Co., Ltd. His research interests include image processing and its application to automobileuse.
Fig. 12Top view generation using camera mounted on a vehicle.

square patterns placed close to the camera. However, thecalibration process itself was easy and the result shows thata practical top view image was obtained.
5. Conclusion
A practical method for generating a top view image basedon homograpy was proposed. In this method, several smallpatterns are spaced evenly so that they cover the entire viewof a camera. However, they are not necessarily aligned precisely with each other. Therefore global metric information
is not necessary for this proposed method. The experimental results showed that the procedure was more practical thanthe conventional method but had the same precison.
In future work, we will attempt to reduce the computation cost of the proposed method for the embedded implementation into an automobile.
References[1] H. Shimizu, “Overhead view parking support systems,” Proc. 11th
World Congress on ITE, 2004.
[2] A. Takahashi, Y. Ninomiya, M. Ohta, M. Nishida, and N. Yoshikawa,Yohei Ishii
received his B.Ag. and M.Ag.
degrees in agricultural engineering from Kyoto
University in 1996. He is currently a seniorresearcher at the Digital Technology ResearchCenter in SANYO Electric Co., Ltd. His research interests include image processing and its
application to automobile use.

Hitoshi Hongo
received his D.E. degree
in information science from Gifu University in
1999. He is currently a manager at the Digital
Technology Research Center in SANYO Electric Co., Ltd. His research interests include image processing, pattern recognition and their applications to automotive use.