3D Modeling of Outdoor Environments
by Integrating Omnidirectional Range and Color Images
Toshihiro ASAI, Masayuki KANBARA, Naokazu YOKOYANara Institute of Science and Technology
8916-5 Takayama Ikoma Nara, Japan 630-0192{toshih-a, kanbara, yokoya}@is.naist.jp
Abstract

This paper describes a 3D reconstruction method forwide area outdoor environments. In this paper, in order
to measure wide outdoor environments, we use an omnidirectional laser rangefinder which can measure the distancesfrom the sensor to objects with high accuracy and an omnidirectional multi-camera system (OMS) which can capture
a wide-angle high-resolution image. Using the two omnidirectional sensors, wide outdoor environments can be digitized efficiently. Note that omnidirectional range and color
images are acquired approximately at the same position toregister both of the images geometrically.

This paper describes a 3D modeling method for wide area
outdoor environments which is based on integrating omnidirectional range and color images. In the proposed method,
outdoor scenes can be efficiently digitized by an omnidirectional laser rangefinder which can obtain a 3D shape withhigh-accuracy and by an omnidirectional multi-camera system (OMS) which can capture a high-resolution color image. Multiple range images are registered by minimizing
the distances between corresponding points in the different
range images. In order to register multiple range images
stably, points on plane portions detected from the range
data are used in registration process. The position and orientation acquired by RTK-GPS and gyroscope are used as
initial values of simultaneous registration. The 3D modelobtained by registration of range data is mapped by textures selected from omnidirectional images in considerationof the resolution of texture and occlusions of the model. In
experiments, we have carried out 3D modeling of our campus with the proposed method.

One of the most important problems in modeling using a
rangefinder is registration of multiple range data. In mostconventional methods for registration of multiple range
data, the ICP algorithm [15] is often used. In the method,
the distance between corresponding points in paired range
data is defined as registration error, and a transformation
matrix is calculated so that the error is minimized. Therefore, to apply the ICP algorithm, it is necessary for a pair ofrange data to include overlapped regions. Since outdoor environments are complex, many occlusions occur and there
are objects which cannot be measured by the rangefinder
stably. In order to solve this problem, Gunadi et al. [11] register local range data acquired on the ground into a global
range data acquired from the sky. Zhao et al. [8] and Frühet al. [12] acquire the range data and estimate the positionand orientation of the rangefinder while it is moving. In
these methods, the precision of generated model is determined by the precision of sensor position and orientation.
These methods need to measure the sensor position and orientation under a motion, accurately.

1. Introduction
3D models of outdoor environments can be used in a
number of fields such as site simulation, human navigation and virtual walk-through. However, such 3D models
are often made manually with high costs, so recently automatic 3D modeling methods have been widely investigated;for example, 3D shape estimation from an image sequence[1]-[5] and 3D measurement of outdoor environments by alaser rangefinder [6]-[10]. The merit of the former is thatthe method can reconstruct a 3D model with only a camera.
However, since it is difficult for vision-based methods to reconstruct a 3D model with high accuracy, the approaches
are not suitable for 3D modeling of wide area outdoor environments. On the other hand, the latter can widely measure
the shape of object with high accuracy. Thus it is useful forwide area outdoor environments.

In this paper, in order to register multiple range dataof wide area stably, only planar surfaces that are extracted
from the range data are used in registration process because
typical outdoor environments contain many plane regions
such as walls and roads. Using the detected plane portions, the range data are simultaneously registered by the
improved ICP algorithm [16], [17] . In this case, the position and orientation of sensor system, which are measured
1Antenna of RTK-GPS(Nikon-Trimble, LogPakII)
Omnidirectional
multi-camera system(Point Grey Research, Ladybug)
Omnidirectionallaser rangefinder(Riegl, LMS-Z360)

Gyroscope(Tokimec, TISS-5-40)

Figure 1. Sensor system mounted on a vehicle.

by RTK-GPS and gyroscope, are used as initial values in
optimization by the ICP algorithm. The surface model isgenerated by polygonizing the registered range data. The
surface model is then texture-mapped by omnidirectional
images selected in consideration of resolution and occlusion.
This paper is structured as follow. Section 2 briefly describes the sensor system used in this study. Section 3 explains the registration of range data acquired at multiple positions. Section 4 describes the method for generating a textured model mapped by selected omnidirectional images. In
Section 5, experimental results of wide area 3D modelingare described. Finally, Section 6 gives summary and future
work.

Table 1. Specification of LMS-Z360

measurable angle
measurable range
measurement accuracy
minimum step angle
maximum resolution

horizontal:
360◦
vertical:-50◦ ∼40◦
1m∼200m±12mm
0.01◦
horizontal: 0.0025◦
vertical:
0.002◦

Omnidirectional Multi-camera System (OMS)
Fig. 3 shows an example of omnidirectional color image
generated from images acquired by the OMS Ladybug. The
OMS has radially located six camera units in a camera blockand their positions and orientations are fixed. Since eachcamera can acquire a 768×1024 pixels image, the OMS canacquire high-resolution omnidirectional color images which
covers more than 75% of full spherical view. The Ladybug
is calibrated geometrically and photometrically in advance[14].

2. Sensor System
This section describes the sensor system. Fig. 1 illustrates the sensor system mounted on a vehicle. In the proposed method, the position and orientation of sensor system
are fixed during acquisition of data. The system equips theomnidirectional laser rangefinder (Riegl, LMS-Z360), omnidirectional multi-camera system (Point Grey Research,
Ladybug), RTK-GPS (Nikon-Trimble, LogPakII), and gyroscope (Tokimec, TISS-5-40).

RTK-GPS and gyroscope
RTK-GPS and gyroscope are used to measure the position and orientation of sensor system, respectively. The
specification of RTK-GPS is shown in Table 2 (a). In general, a yaw value measured by the gyroscope includes an accumulative error. The gyroscope is interlocked with RTKGPS in order to correct the accumulative error by measuring the direction of movement calculated by GPS data undermotion. The specification of gyroscope is shown in Table 2(b).

2.1. Specifications of Sensors
Omnidirectional Rangefinder
The specification of the rangefinder is shown in Table
1. Angle and resolution of measured range data can be
determined by user. Fig. 2 shows an example of acquired omnidirectional range image in which the distancefrom the rangefinder is coded in intensity. This rangefinder
measures environments omnidirectionally by illuminating alaser beam radially. Therefore, the density of measuring
points becomes sparse in a far place and dense in a nearplace from the sensor. The rangefinder takes at least 45 seconds for omnidirectional measurement.

2.2. Alignment of Sensor Coordinate Systems
There are four coordinate systems in the sensor system;
the rangefinder, OMS, RTK-GPS, and gyroscope coordinate
systems. Geometrical relationship among the coordinate
systems is fixed, and these coordinate systems are regis2

Figure 2. Omnidirectional range image.

Figure 3. Omnidirectional color image.

sitions in the gyroscope coordinate system are known. The
markers are placed at positions which can be measured by
the rangefinder as shown in Fig. 5(a). The transformation(b) in Fig. 4 is estimated by measuring the position of markers with respect to the rangefinder as shown in Fig. 5(b).
(c) Transformation between gyroscope and global coordinate systems : The coordinate systems of RTK-GPS(global coordinate system) and gyroscope are aligned automatically, because these sensors are interlocked with eachother.

Table 2. Specifications of RTK-GPS and gyroscope(a) Accuracy of RTK-GPS.
latitude±3.0cmlongitude ±3.0cm
altitude±4.0cm(b) Accuracy of gyroscope.
yaw
±2◦
roll
±0.5◦
pitch ±0.5◦

tered to RTK-GPS (global) coordinate system as shown in
Fig. 4. Figs. 4(a), (b) and (c) illustrate transformation matrices among sensor coordinate systems. The methods for
estimating the transformation matrices are described below.
(a) Transformation between OMS and rangefinder coordinate systems : By giving corresponding points in range
and color images, the transformation matrix can be estimated by the method in [14].
(b) Transformation between rangefinder and gyroscope
coordinate systems : The transformation matrix can be estimated by measuring more than three markers whose po-

3. Generation of 3D Surface Model
A surface model of environment is generated by integrating multiple range data. Therefore, all range data are firstregistered to the GPS coordinate system. Since the positionand orientation acquired by the sensors include some errors,
they are used as initial values in registration and should be
optimized. The ICP algorithm [15] is usually used for optimal registration of multiple range images. In the conventional ICP algorithm, the distance between corresponding3

zL

nar regions are calculated, and then the plane in one data
is searched which corresponds to each point in the otherdata. Multiple range data are simultaneously registered for
optimization of transformation matrices. The position andorientation acquired by RTK-GPS and gyroscope are used
as an initial value in the optimization. the flowchart of theregistration is shown in Fig. 6 and the detail of each process
is described in the following.

yL
xL
zr

OMS

(a)
Alt.

yr

RTK-GPSxr(b)

rangefinder

xg

Lon.

gyro sensor

(c)

Lat.
global coordinate system

start

yg
zg

Plane detection from range data (3.1.1)
Transformation of range data to global coordinate system

Figure 4. Relationship among the coordinate systems of the sensors.

Search of the plane which corresponds
to each point of the plane portion (3.1.2)
Estimation of transformation matrix fromcorresponding points and planes (3.1.3)

markerno

Is solution converged?
yes
end

Figure 6. Procedure of registring range data(a) Layout of markers(b) Range image of markers

3.1.1. Plane Detection from Range Data. Planar regions
are detected from range data by local plane fitting. We
employ the renormalization method [13] for planar region
detection and the quadtree segmentation recursively. The
whole range image is taken as an initial region. The distances between estimated plane and points in the region arecalculated and when at least one distance is bigger than athreshold, the region is split. On the other hand, when all
the distances are smaller than the threshold, the region isdefined as a plane portion. The points which are not defined
as a plane portion are not used for registration process.

Figure 5. Alignment of rangefinder and gyroscope
coordinate systems.

points in paired range data is defined as a registration error, and the transformation matrix is calculated so that theerror is minimized. The present rangefinder measures the
distance by rotating the laser scan, thus the spatial densityof data points depends on the distance; that is, close objects
are measured densely and far objects are measured sparsely.
In registering range data obtained at different positions, thiscauses the problem that multiple points may correspond toa single point. The solution tends to fall into a local minimum, because the correspondences between points are discrete and do not include the surface information about anobject [16]. The proposed method first detects plane regions
from all range data, and then determines point-to-plane correspondences. Finally, the transformation matrix of range
data is calculated by overlapping the corresponding planes.

3.1.2. Search of Corresponding Plane. The plane portion of range data is searched from different range data.
The plane correspondence is described below. Let RDn
be range data n (n = 1, · · · , N ), Pni be a planar regionin RDn (i = 1, · · · , I) and Qnij be a point in the plane
Pni (j = 1, · · · , J). The normal vector of Pni is denoted by Nni . A plane corresponding to the point Qnij
is searched from range data other than the range data n. A
plane Pkl corresponding to the point Qnij is selected so that
|Qnij Qx |, which means the distance between Qnij and Qx ,is minimized. Note that Pkl and Qnij satisfy two conditions shown in Fig. 7: the inner product of Nkl and Nni
is below a threshold (Fig. 7(a)) and a point Qx where thevector Nkl passing through point Qnij intersects the plane
Pkl exists (Fig. 7(b)). Pkl1 is chosen as the plane which

3.1. Simultaneous Registration of Multiple Range
Data
Multiple range data are registered by overlapping thecorresponding planes among range data. For this purpose
as a pre-processing, planar regions in range data are detected and the normal vectors at the measured points in pla4

where Qu and Pu are Qu and Pu after transformationby (Rn1 , Tn1 ) and (Rn2 , Tn2 ), and the orientations estimated in step 1 are fixed.

Qx1

Pkl1

Nkl1
Pni
Qnij

Pkl2
Nkl2

Qx2

The corresponding plane is searched again after the step 2
and the process is iterated until the solution is converged.
Multiple dimension downhill simplex method [18], which
does not need the derivatives, is used for the optimization.

Nni(a) Selection by threshold of an inner product.
Qx1
Nkl1 Qx2
Qnij

3.2. Polygonization of Range Data
Pkl1

A polygonal representation is generated from each range
data by connecting four corners of each region defined asa plane portion in the plane detection process in order toreduce the number of polygons. In a non-plane portion, apolygon is generated by connecting adjoining pixels which
are its neighbors and one of its diagonal neighbors. A range
data partially overlaps other range data. The quantity ofdata is reduced by removing redundant polygons at overlapping regions. Polygons are generated from range data inorder of input. The generated polygons are searched whichcorrespond to the vertices of the generating polygon withthe method described in Section 3.1.2. When distances between vertices and intersection Qx are less than a threshold,the polygon is deleted as a redundant one as shown in Fig.
8. Note that only the polygon defined as a plane portion isdeleted to maintain the quality of model and to reduce theamount of data.

Pkl2

Nkl2

(b) Selection by existence of an intersection.
Figure 7. Selection of corresponding plane.

corresponding to Qnij in both Fig. 7(a) and (b), .
3.1.3. Estimation of position and orientation of range
data. The sensor position of range data is estimated fromthe distance between a point and the corresponding plane
and the sensor orientation of range data is estimated fromthe inner product of normal vectors of corresponding point
and plane. Let Rn and Tn be sensor orientation and positionof range data n (n = 1, · · · , N ), respectively.
step 1. The orientations Rn are estimated by maximizing
the correlation CN defined as the sum of inner products of normal vectors of a point Qu and a plane Pu(u = 0, · · · , U ), where Qu is the point in RDn1 and
Pu is the plane in RDn2 , Pu is the plane which corresponds to the point Qu , and U represents the numberof points which have corresponding planes.

Generated polygon
Vertex of the generating polygon

CN =

U


Intersection Qx(Rn1 Nnu ) · (Rn2 Nmu ) → max,

(1)

Figure 8. Deletion of overlapping areas.

u=0

where Nnu and Nmu are the normal vectors of Qu andPu respectively.

4. Generation of Textured Model

step 2. The positions Tn are estimated by minimizing theerror ET which is defined as the sum of distances between corresponding point and plane as follows.

ET =

U

u

distance(Qu , Pu ) → min,

The 3D shape obtained in the previous section is colored by textures selected from omnidirectional color images. The geometrical relationship between the rangefinder
and the OMS is known by the alignment described in Section 2. Therefore, the camera position and orientation ofomnidirectional images are also estimated by registering therange data, and the correspondence between a 2-D point in(2)

5

omnidirectional images and a 3-D point in range data canbe easily determined.

:acquisition point
50m

4.1. Selection of Omnidirectional Image
Each polygon on the 3D shape is colored by the texture
selected from the image which gives the highest resolution.
A polygon is projected onto each omnidirectional image
and then the image with the biggest area of the polygon ischosen. However, when occlusions occur this strategy fails.
For example, as shown in Fig. 9, the resolution of texture
generated from camera 2 is higher than the camera 1. However, the correct texture can not be taken by the camera 2.
The occlusion is detected when the generated surface modelintersects with a triangular pyramid determined by triangular patch vertices and the projection center of camera. In
such a case, the next highest resolution image is selected,
and occlusion is re-detected.

A

Triangular polygon

Figure 10. Range data acquisition points.

Camera 2

selected as a reference. Fig. 11 illustrates a 2D CAD dataof our campus overlaid with the generated model. Examplesof rendering the generated model are shown in Fig. 12.

Camera 1

Figure 9. Selection of omnidirectional image inconsideration of occlusion.

6. Conclusion
This paper has proposed a 3D modeling method which
is based on integrating omnidirectional range and color
images for wide area outdoor environments. In experiments, a 3D model is actually generated from omnidirectional range and color images acquired at 68 points in our
campus. We can observe many small data lacking portions
mainly caused by occlusions in the generated model. In order to measure them efficiently, we are planning to carry out
continuous measurement during the movement of sensors.
Moreover, the sense of incongruity is observed in the generated model, when different images are selected in neighboring polygons. Such an effect is mainly caused by thevarying illumination conditions during the measurement of
the whole area. This problem in generating a textured 3Dmodel should further be investigated.

5. Experiments
We have carried out experiments of reconstructing our
campus. In experiments, the range and color images are acquired at 68 points in our campus (about 250m × 300m).
Fig. 10 illustrates the data acquisition points in our campus.
Since the proposed method requires overlapping portions
among range data for registration, we have acquired thedata paying that attention. The sensor coordinate systems
are aligned in advance by the method described in Section
2.2. The resolution of each omnidirectional range image is1024×512. A cluster consisting of 24 PCs (CPU: Pentium4
1.7Ghz, Memory: 1024MB) is used for finding corresponding plane, and single PC (CPU: Pentium4 1.8Ghz, Memory:
2048MB) is used for registration of multiple range data.
The time required for registration is about 7 days. In registration, the range data acquired at point A in Fig. 10 isReferences[1] T. Sato, M. Kanbara, N. Yokoya and H. Takemura: “Dense
3-D Reconstruction of an Outdoor Scene by Hundredsbaseline Stereo Using a Hand-held Video Camera,” Interna-

6Model of an Urban Environment Using Vehicle-Borne Laser
Range Scanners and Line Cameras,” Machine Vision andApplications, Vol. 14, No. 1, pp. 35-41, 2003.
[9] P. K. Allen, A. Troccoli, B. Smith, S, Murray, I. Stamos,
and M. Leordeanu: “New Methods for Digital Modeling ofHistoric Sites,” IEEE Computer Graphics and Applications,Vol. 23, pp. 32–41, 2003.
[10] Y. Sun, J. K. Paik, A. Koschan and M. A. Abidi: “3D Reconstruction of Indoor and Outdoor Scenes Using a Mobile
Range Scanner,” Proc. Int. Conf. on Pattern Recognition,
Vol. 3, pp. 653–656, 2002.
[11] C. R. Gunadi, H. Shimizu, K. Kodama, and K. Aizawa:“Construction of Large-scale Virtual Eenvironment by Fusing Range Data, Texture Images and Airbone Altimetry
Data,” Proc. Int. Symposium on 3D Data Processing, Visualization and Transmission, pp. 772-775, 2002.
[12] C. Früh and A. Zakhor: “Constructing 3D City Models by
Merging Aerial and Ground Views,” IEEE Computer Graphics and Applications, Vol. 23, pp. 52–61, 2003.
[13] Y. Kanazawa and K. Kanatani: “Reliability of Fitting a Plane
to Range Data,” IEICE Trans. on Information and Systems,
Vol. E78-D, No. 12, pp. 1630–1635, 1995.
[14] S. Ikeda, T. Sato, and N. Yokoya: “Panoramic Movie Generation Using an Omnidirectional Mmulti-camera System
for Telepresence”, Proc. 13th Scandinavian Conf. on Image
Analysis, pp. 1074–1081, 2003.

Figure 11. 2D CAD data overlaid on generated 3D
model[15] P. J. Besl and N. D. McKay: “A Method for Registration of
3-D Shapes,” IEEE Trans. on Pattern Analysis and MachineIntelligence, Vol. 14 No. 2, pp. 239–256, 1992.
[16] K. Pulli: “Multiview Registration for Large Data Sets,” Proc.
Int. Conf. on 3D Digital Imaging and Modelling, pp. 160–
168, 1999.

tional Jour. of Computer Vision, Vol. 47, No. 1-3, pp. 119–
129, 2002.
[2] S. Ozawa, M. Notomi, and H. Zen: “A Wide Scope Modeling to Reconstruct Urban Scene”, Proc. ISPRS Commission V Int. Symposium on Real-Time Imaging and Dynamic
Analysis, pp. 370–376, 1998.

[17] T. Oishi, R. Sagawa, A. Nakazawa, R. Kurazume, and K.
Ikeuchi: “Parallel Alignment of a Large Number of Range
Images,” Proc. Int. Conf. on 3D Digital Imaging and Modeling, pp. 195–202, 2003.

[3] C. Tomasi and T. Kanade: “Shape and Motion from Image
Streams under Orthography A Factorization Method,” International Jour. of Computer Vision, Vol. 9, No. 2, pp. 137–
154, 1992.

[18] W. H. Press, B. P. Flannery, S. A. Teukolsky, and W. T. Vetterling: Numerical Recipes in C, Cambridg University Press
1988.

[4] M. Pollefeys, R. Koch, M. Vergauwen, A. A. Deknuydt and
L. J. V. Gool: “Three-demintional Scene Reconstruction
from Images,” Proc. SPIE, Vol. 3958, pp. 215–226, 2000.
[5] M. Okutomi and T. Kanade: “A Multiple-baseline Stereo,”
IEEE Trans. on Pattern Analysis and Machine Intelligence,Vol. 15, No. 4, pp. 353–363, 1993.
[6] S. F. El-Hakim, C. Brenner, and G. Roth: “A Multi-sensor
Approach to Creating Accurate Virtual Environments,” Jour.
of Photogrammetry & Remote Sensing, Vol. 53, pp. 379–
391, 1998.
[7] H. Zhao and R. Shibasaki: “Reconstruction of Textured Urban 3D Model by Fusing Ground-Based Laser Range and
CCD Images,” IEICE Trans. Inf. & Syst., Vol. E-83-D,
No. 7, pp. 1429–1440, 2000.
[8] H. Zhao and R. Shibasaki: “Reconstructing a Textured CAD

7

Figure 12. Generated 3D model with texture

8