Monocular Road Mosaicing for Urban Environments
Andreas Geiger
Institut für Mess- und Regelungstechnik
Universität Karlsruhe (TH), Germany
geiger@mrt.uka.de

Abstract—Marking-based lane recognition requires an unobstructed view onto the road. In practice however, heavy traffic
often constrains the visual field, especially in urban scenarios
such as urban crossroads.
In this paper we present a novel approach to road mosaicing
for dynamic environments. Our method is based on a multistage
registration procedure and uses blending techniques. We showthat under modest assumptions accurate registration is possible
from monocular image sequences. We further demonstrate thatfusing visual information from previous frames into the current
view can greatly extend the camera’s field of view.

I. I NTRODUCTION
Intelligent vehicles require a good view onto the road when
visual positioning according to lane markers is required [1],[2]. Often however, the field of view is strongly limited. Thisis because traffic signs, other traffic participants or buildings
frequently occlude large parts of the road, as illustrated infigure 1 and 2. Treating occlusions as noise can lead tounsatisfying results in such situations.
We tackle this problem by making use of the parallax
effect – the apparent displacement of an object viewed along
two different lines of sight. Knowing the ego-position and orientation of the vehicle and identifying a light ray as comingfrom a particular point on the road allows for incrementally
generating a virtual map of the road.1 This map can be used
to reconstruct parts of the road which become occluded infuture image frames. Figure 1 shows this idea. While at framet + 1 the view of the ego-vehicle is partly barred by theother car, the occluded part of the road is visible at framet due to the relative displacement of object and camera. In
order to remove artificial edge artifacts from the mosaic we
use blending techniques which are a well-studied tool in thecomputer graphics community.
In recent years ego-positioning and mapping experienced
a lot of attention [3]–[7]. Recent advances have shown thateven long distances can be robustly mapped by mobile robots[8]. Many of the techniques, however, use stereo-information,
landmark points, additional sensor types and/or assume static
scenes. Our method also differs from traditional stitching
techniques [9], [10] since we have to deal with low resolution
in far field due to the tilted and low camera position withrespect to the road.
The method described in this paper solely uses informationfrom a monocular grayscale camera, thus supporting a cheap
1 We call this map virtual, since we are only interested in the correct poseparameters. The target surface to project on can be either a bird’s eye view
(figure 2(b)) or a single camera view (figure 2(c)).

Fig. 1. The parallax effect. Combining information from previous frameswith the current view increases the visual field with respect to the road.

and easy application in automotive systems. Since we workwith monocular images only, ego-positioning is challenging
and we have to make assumptions about the environment
which are summarized as follows:An estimate of the camera intrinsics (focal length, principal point, distortion) and extrinsics (camera height,
inclination) is given.
• The road surface to be considered can be approximatedby a plane and offers enough texture (i.e. road markings)
for the registration process.
•Note that the first assumption is valid for all calibrated
camera setups and the second assumption holds for a large
number of interesting inner-city scenarios.
The remainder of this paper is structured as follows: First
we show how representative frames can be found in the image
sequence. Then our four-stage registration process is describedfollowed by a description of the image-blending techniques weuse. Finally the paper is concluded with an outlook on futurework.
II. PARAMETER INITIALIZATION
A. Geometrical scene description
Approximating the road surface by a plane enables us todescribe the perspective mapping from one image plane toanother image plane and from image planes to the road via
homographies [11]pj ≃ Hpi(a) First frame (bird’s eye view)(b) Mosaicing result(c) Overlay (best viewed in color)

Fig. 2. Comparison of single image road-views with road mosaics. Figure 2(a) shows a bird’s eye perspective of the road when projecting the current
camera image only. Figure 2(b) shows the mosaicing result of our algorithm with the visually estimated ego-position of the camera for 10 keyframes. The
images are combined in a way that removes spurious obstacles from the road and leads to a better overlook. Figure 2(c) shows an artificial overlay (red) ofa single frame with the mosaicing result computed only from previous frames (gray values).

where pi is a homogeneous point lying on plane i and H is a
3 × 3-matrix describing the projection from plane i to plane
j. Here ≃ denotes that the equation is defined up to a scale
factor.
In theory the 8 homography parameters could be estimatedfrom frame to frame using standard techniques like DLT2 .
However we do not know which of the correspondences
between two subsequent frames actually lie on the road. Furthermore small registration errors would accumulate quickly[3], [7], leading to bad results when projecting points alongthe ” homography chain”.
Thus we decided to use a direct parameterization scheme
which further features interpretable parameters in contrast toelementwise homography parameterization. This also allows
for putting priors on the parameters in order to perform temporal integration and improve the optimization result. Figure
3 shows the transformation matrices Trr which project fromroad to road, Trc which project from road to camera, K which
project from camera to the image plane and Tii which projectfrom image to image. Due to the planarity assumption all
projections can be expressed as 3 × 3 homography matrices.
Transforming a point from one coordinate system into theother is easily expressed via concatenations and inversions ofthese matrices. The individual mappings are parameterized as

K(f, c)Trc (h, θ, φ)



fx

0

cx





=  0 fy cy 
0 0 1
cos φ − sin φ cos θ=  sin φ cos φ cos θ0sin θ

2 Direct Linear Transform [11].

− sin φ sin θhcos φ sin θh 
− cos θhFig. 3.
Geometrical description of the scene. This figure shows thetransformations we use for projecting between different coordinate systems.
The colored boxes denote associate sections in this paper.

Trr (α, t)



cos α


=  sin α0− sin α
cos α
0

t1




t2 
1

where f,c are the camera intrinsics, h is the camera height,t represents the camera translation over ground and φ,θ,αstands for roll, pitch and yaw. While the calibration matrix
K is assumed constant over time, Trc and Trr vary. The
goal in registration is now to estimate the parameter set Θ =(f, c, [h1 , θ1 , φ1 ]...[hN , θN , φN ], [α1 , t1 ], ..., [αN −1 , tN −1 ])which relates N keyframes to each other.
B. Keyframes and keyfeatures
Since we can not make use of landmark points, accumulation of errors is a severe problem averting the use of all
frames for registration. By picking only a subset as keyframes
we also save computational power because optimizing for theparameters is done in a lower-dimensional space. Thus wej+1
distance |kpjr1 − pjr2 k2 − kpj+1
r1 − pr2 k2 | from each other.
Figure 4(a) illustrates this process for a single pair.

III. R EGISTRATION(a) Harris corner tracking(b) Snakes over time

Fig. 4. Tracking Harris corners. Figure 4(a) depicts two Harris corner tracks
between two consecutive keyframes in the image (top) and road (bottom)
coordinate system. Sustaining tracks are shown over time in figure 4(b).
The green tracks are selected as keyfeatures for parameter initialization as
described in section III-A. Keyframes are indicated by blue vertical bars.

search for the smallest set of keyframes which still allows for
accurate registration.
To do so we first track Harris corners [12], [13] over thesequence using template matching via cross-correlation. New”tracks” are initialized at points distant from existing active
tracks. Tracks end when the cross-correlation value falls below
a threshold τ or simple smoothness or geometrical constraints
are violated. Figure 4 depicts this process.
As mentioned earlier, we are interested in a small set ofrepresentative keyframes with a large number of tracks whichconnect two consecutive keyframes. Furthermore we want the
keyframes to be uniformly distributed. In other words we wish
to minimize
Ekey = aN − b

N
−1
X
i=1

ti+1
ψ(Cii+1 ) + c
i

N
−1
X
i=1

|ti+1
−
i

tN
1
| (1)
N

with respect to the number of keyframes N . Here tji denotes
the time between keyframe i and keyframe j, Cii+1 is thenumber of tracks connecting keyframe i and keyframe i +1 and ψ stands for the logistic function. In our experiments
we set a = b = c = 1 and choose the logistic parameterssuch that 4-5 connections between consecutive keyframes are
considered sufficient. Because minimization of equation 1 withrespect to N leads to exponential complexity we approximate
the solution by a simple greedy algorithm: We initialize theset of keyframes to the set of all frames and remove the framewith the highest gain in Ekey until no further improvement
is possible. A typical solution of this algorithm is shown infigure 4(b).
Reliable Ego-pose initialization further requires selecting
a subset of tracks connecting pairs of keyframes which lie
on or close to the road. To achieve this we project feature−1 jcandidates onto the road3 via pjr = T−1
pi for each pair of
rc K
consecutive keyframes j,j + 1. Since distances on the road areinvariant under global rotation and translation we select thosefeatures as keyfeatures which exhibit the smallest change in

Accurate image alignment is of utmost importance since
it has a direct impact on the mosaicing result. As it turns
out, even small registration errors in the image coordinate
system can cause large errors in the road coordinate system.
This is because the camera is highly tilted with respect to therelevant road plane leading to decreasing image resolution andregistration accuracy in the far field. Note that this is not thecase for traditional stitching tasks.
Because of this we employ a four-stage registration algorithm which first estimates the ego-pose, secondly segments
keyframes into foreground and background, afterwards finds
additional road correspondences via a RANSAC4 -based search
and finally performs bundle adjustment in a probabilistic setup
with Gaussian Process priors.
A. Ego-pose initialization
We first initialize the calibration parameters f, c and theroad-to-camera transformation parameters (h1 , θ1 , φ1 ) = ... =(hN , θN , φN ) according to our prior knowledge about thecamera setup. The only missing parameters to complete thevector Θ0 are the road-to-road transformation parameters.
As shown in [14] the least squares problem to fitting a 3D
point set to another can be solved in closed form by computing
the singular value decomposition of a 3 × 3 matrix. We makeuse of this algorithm for our 2D point fitting problem in orderto minimize
X j2j
Epoint=kpri − pj+1
(2)
ri k
i

where pjri denotes road point i at keyframe j. Solving equation
2 results in the initial estimate for the missing translation androtation parameters (αj , tj ) for j = {1, ..., N − 1}.
B. Road segmentation
Given our initial estimate Θ0 we intend to refine the parameters in a global optimization scheme. This requires accurate
road-to-road correspondences between keyframes which aresought in the next section. A first step consists of segmenting
foreground objects from the background (road). Since we areusing a monocular setup, stereo-information is not accessible
for this task. We tackle this problem by observing that points
on the road usually transform in a way distinct from points
lying on moving or static objects (e.g. cars).
Applying the sparse iterative version of Lucas-Kanade optical flow estimation (as described in [15]) on Harris corners
below the horizon line5 , results in 2-dimensional optical flow
vectors vreal for salient pixels pi , where the index i denotesthat it lies in the image plane. The virtual optical flow vvirt tocompare against is calculated by projecting the ”road flow”
into the image, using the estimed parameters from section
4 RANdom SAmple Consensus.

3 Here the a-prior knowledge about h,θ and φ is used.

5 The horizon line can be easily determined by h,θ and φ.

III-A. The final error value verr is computed by weighting
the norm of the optical flow differences with the angularagreement of both vectors, thereby penalizing contradicting
vector directions.

vTreal vvirt
verr = kvreal − vvirt k2 2 −kvreal k2 kvvirt k2∂pi ∂pi T
vreal = (,)∂u ∂vvvirt = KTrc (pr + pv ) − pi
pv
pr

=

−1
Tjrr pr − Tj−1
pr
rr

0
50
100
150
200
250
300
350

(tj+1 − tj−1 )−1 −1= Trc K pi

400
450

Here pr denotes road coordinates and pv is the road flow
vector for this point, obtained by an approximation to the
trajectory tangent. After calculating verr for all points which
exhibit enough texture [13] we apply nearest-neighbor clustering and remove clusters with less than 10 points. We extend theconvex hull of all remaining point sets to approximately cover
the whole object. Points inside the polygons, above the horizon
line or on the engine hood are masked while the remaining
regions are assumed to stem from the road. An example of thesegmentation result is depicted in figure 5 for a single frame.

100

200

300

400

500

600

Fig. 5. Road segmentation using virtual optical flow. Here we depict the
optical flow (yellow) and the virtual optical flow (green) for keyframe 7 of
the test sequence from figure 4(b). To give robust results, we do not use dense
optical flow, but instead calculate the optical flow only at locations with highsaliency. Green circles indicate deviations in optical flow and the red polygon
shows the refined convex hull which is used for masking objects.

C. RANSAC-based correspondence refinement
Having segmented the keyframes into road and non-road
regions enables us to find correspondences more precisely.
We employ an iterative correspondence search making use of
the planarity assumption. To do so we warp the image andmask from keyframe j + 1 to the image coordinate system of
keyframe j using bilinear interpolation and the homographyjTjii = K Tj+1
rc Trr Trc

−1

K−1

as depicted in figure 6. We then calculate Harris corners for all
non-masked pixels in the warped image j + 1 and search forthe best match in image j. Template matching and RANSAC
are employed for finding inlier correspondences and estimating
the homography Tjii at the same time, making use of the DLT
algorithm for calculating each random sample. This procedure
is repeated for a smaller search area, a bigger template size andthe new Tjii until convergence (typically after 2-3 iterations).
The remaining inlier correspondences are then used in thebundle adjustment stage described in the next section.

This section describes how we find the optimal parameters
using all information available to us. We perform temporal integration via bundle adjustment with priors on the parameters.
The regularization term helps when dealing with ambiguities
or small registration errors as in our case.
We seek to maximize the posterior probability of the parameters given the observations with respect to the parameters(3)(b) Iteration 2

Fig. 6.
RANSAC correspondence refinement. This figure illustrates
iteration 1 and 2 of the RANSAC correspondence search. While the red channelshows keyframe j, the green channel displays keyframe j + 1 warped to theimage coordinate system of keyframe j. Inlier correspondences are depicted
in red and green while outliers are marked black. For the sake of clarity thisfigure does not show the object masks which were used.

where Θ are the parameters and Pj is the match probability which depends on the correspondences found between
keyframe j and j + 1 and the geometric parameters.
Since we do not consider correspondences between nonconsecutive keyframes the pairwise likelihood is conditionallyindependent
P (P1 , ..., PN −1 |Θ) =D. Global optimization with priors

P (Θ|P1 , ..., PN −1 ) ∝ P (P1 , ..., PN −1 |Θ)P (Θ)(a) Iteration 1

N
−1
Y
j=1

P (Pj |Θ)(4)

and we assume white noise on the correspondences
1
P (Pj |Θ) ∝ exp(− dTj Σ−1 dj )2with Σ = diag(σd2 , ..., σd2 )T and the reprojection error distancei+dij = kTjii pi−
j − pj k2 .

Here pi−j denotes the i’th correspondence of Pj in the image
coordinate system of keyframe j, Tjii = Tjii (Θ) represents the(a) Initialization(b) Global optimization result(c) Gain compensation result(d) Gain compensation & blending

Fig. 7. Road mosaicing. While figure 7(a) shows the initialization configuration, figure 7(b) depicts the combination of base images after optimizing theparameters with equation 3. Artificial edge artifacts emerge due to different gains, small registration errors and object shadows. We reduce this effect bycompensating for the gain (7(c)) and blending the base images at multiple bands as depicted in figure 7(d). However, as can be seen from the marker at thebottom left of 7(d), small registration errors persist due to the high sensitivity of the extrinsic parameters and the low resolution of the camera.

homography for transforming pi−
j into the image coordinate
system of keyframe j + 1 and pi+
j is the corresponding point
in keyframe j + 1.
To circumvent ambiguities and compensate for accumulating registration errors we do not optimize the loglikelihood log P (P1 , ..., PN −1 |Θ) directly, but rather add
priors on the parameters and optimize the log-posterior
log P (Θ|P1 , ..., PN −1 ).
Assuming independence of the parameters we have
P (Θ) = P (f, c)P (h)P (θ)P (φ)P (v)P (ω)(5)

where v and ω are the velocity and the angular rate of thecamera over ground. Since we assume the camera intrinsics
to be fixed we model the probability(f, c) ∼ N (µc |Σc )(6)

according to our knowledge from the calibration process.
The remaining parameters are assumed to vary over the sequence, i.e. are functions over time. We encourage smoothness
by putting Gaussian Process priors [16] on the function spaceand model the mean function of the individual processes via
our prior knowledge about the camera setup (e.g. height of thecamera). Thus we havef (t) ∼ GP(µf (t), σf (t, t′ ))(7)with f ∈ {h, θ, φ, vx , vy , ω}. For all f we set the meanfunction identical to the initial estimate of the parameterµf (t) ≡ f0 and we model the covariance function σf (t, t′ )using the squared exponential kernel2

σf (t, t′ ) = σf2l exp(−(t − t′ )) + δt=t′ σf2η2σf2wwith kernel height σf2l , width σf2w and noise σfη for eachfunction f . The hyperparameters σf2l , σf2w are set according toour belief about the variance and the smoothness of f . A small
noise term is added to increase stability when calculating theinverse of the covariance matrix for P (f ).

Integrating the priors (5) and the likelihood (4) into(3) and taking the logarithm yields the log-posterior
log P (Θ|P1 , ..., PN −1 ) which can be maximized by standard
gradient descent techniques like Scaled Conjugate Gradients[17].
IV. M OSAICING
Simple merging of road images leads to unsatisfactory
results due to differences in gain, vignetting, object shadowsand registration errors as depicted in figure 7(b). Thus, in orderto create the final road mosaics we perform two additional
steps. After generating ”base images”, we first compensate
for the camera gain which might have changed during the ride.
Then we combine the road images using multi-band blending[9], [10] to generate a visually pleasing result and remove
artificial edges.
A. Creating base images
Having estimated the parameters Θ, image points can betransformed from each keyframe to any other keyframe or a
global road coordinate system. Thus occluded pixels in one
image can be replaced by fusing intensity information fromother images. After selecting a target coordinate system (e.g.
the road coordinate system for generating top-down views)
we generate one base image per keyframe which contains
all visible pixels warped into the target coordinate systemusing bilinear interpolation (this is illustrated in figure 8(a)).
Combining the base images to the final mosaic is done bytaking pixels from the closest base image. Here closest refers
to the Euclidian distance from the camera center of the baseimage to the pixel’s global road position.
B. Gain compensation
In this section, we show how we can solve for the overall
gain (a photometric parameter) in closed-form. The gain isdenoted as the vector g = (g1 , ..., gN )T where gi represents
the gain of base image Ii . Our goal is to minimize the error
function
X
Egain =
(gi µij − gj µji )2
(8)
i6=j

results in the final road mosaic
Imosaic = IΣKσ +

X

kσ
BΣ

k

with K the total number of bands. The blending result is
depicted in figure 7(d).

(a) Base image(b) Blending weights

Fig. 8. Base image and blending weight. Figure 8(a) shows the base image
I0i for a keyframe of the test sequence. The corresponding blending weightWkσ
i is depicted in figure 8(b) for σ = 1.5 and k = 0. The small images on
the right side show the red patch for bands k = {1, 3, 5}.

where µij stands for the mean of pixels in image Ii which
overlap with image Ij . We exclude the trivial solution g ≡ 0
via the constraint gT g = N using Lagrange multipliers. Thisavoids the choice of parameters as in [10]. Equation 8 isquadratic in g, thus the problem can be rewritten as minimizing
Egain = gT Ag subject to gT g = N . This leads to theeigenvalue problem
Ag̃ = λg̃
where

( P

for i = j−µij µji
otherwise√
N g̃ with g̃ being the
The solution is given by g =eigenvector corresponding to the smallest eigenvalue of A. The
result of compensating the gain is depicted in figure 7(c).
Aij =k6=i µik

C. Multi-band blending
Following [9], [10] we use multi-band blending to confine
the effect of artificial edges due to the reasons discussed above.
The idea is to blend low frequencies over a large spatial
range, and high frequencies over a short range. Therefore
the blending weights Wi and base images Ii need to besmoothed for the different bands and the bandpass images Bkσiare calculated by differencing the smoothed base images. For
k ≥ 1 we have(k−1)σ

Wkσ
i

= Wi

Ikσ
i
kσ
Bi

=
=

⊗ gσ̃(k)(k−1)σIi⊗ gσ̃(k)(k−1)σ
Ii
− Ikσ
i .

where the standard√ deviation of the Gaussian blurring kernel
is set to σ̃(k) = 2k + 1σ such that the range of wavelengths
does not change for subsequent bands. Here W0i and I0i denotethe base weights and base images respectively and ⊗ is the
convolution operator. Blending weights for a single keyframe
are depicted in figure 8(b).
Combining overlapping images for each band linearly
P
kσ
kσ
kσ
i Wi (u, v)Ii (u, v)
PIΣ (u, v) =kσi Wi (u, v)
P
kσ
kσ
kσ
i Wi (u, v)Bi (u, v)
PBΣ(u, v) =kσi Wi (u, v)

V. C ONCLUSION AND FUTURE WORKIn this paper we have shown that accurate road mosaicing
is possible from monocular image sequences only, even inheavy traffic situations. This leads to an important gain in
visual information for subsequent processing steps. Further
work will focus on an iterative version of the algorithm which
is able to add keyframes to an existing map in real-time. Sinceroad segmentation (described in section III-B) is key to finding
accurate correspondences we also intend to consider additionalfeatures like appearance or stereo information for this step.
ACKNOWLEDGMENT
The author would like to thank the Karlsruhe School of
Optics and Photonics and the Deutsche Forschungsgemeinschaft (Collaborative Research Center Cognitive Automobiles)
for supporting this work.
R EFERENCES[1] C. Duchow, “A novel, signal model based approach to lane detection
for use in intersection assistance,” Intelligent Transportation Systems
Conference, 2006. ITSC ’06. IEEE, pp. 1162–1167, 2006.
[2] M. Aly, “Real time detection of lane markers in urban streets,” Intelligent
Vehicles Symposium, 2008 IEEE, pp. 7–12, June 2008.
[3] A. Milella and R. Siegwart, “Stereo-based ego-motion estimation using
pixel tracking and iterative closest point,” Computer Vision Systems,
2006 ICVS ’06. IEEE International Conference on, pp. 21–21, Jan. 2006.
[4] J. Horn, A. Bachmann, and T. Dang, “Stereo vision based ego-motion
estimation with sensor supported subset validation,” Intelligent Vehicles
Symposium, 2007 IEEE, pp. 741–748, June 2007.
[5] A. J. Davison, I. D. Reid, N. D. Molton, and O. Stasse, “Monoslam:
Real-time single camera slam,” Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 29, no. 6, pp. 1052–1067, 2007.
[6] T. Y. Tian, C. Tomasi, and D. J. Heeger, “Comparison of approaches toegomotion computation,” CVPR, p. 315, 1996.
[7] R. Goecke, A. Asthana, N. Pettersson, and L. Petersson, “Visual vehicle
egomotion estimation using the fourier-mellin transform,” Intelligent
Vehicles Symposium, 2007 IEEE, pp. 450–455, June 2007.
[8] M. Pollefeys, “Detailed real-time urban 3d reconstruction from video,”
IJCV, vol. 78, no. 2-3, pp. 143–167, July 2008.
[9] P. J. Burt and E. H. Adelson, “A multiresolution spline with application
to image mosaics,” ACM Transactions on Graphics, vol. 2, pp. 217–236,
1983.
[10] Brown, Matthew, Lowe, and David, “Automatic panoramic image stitching using invariant features,” International Journal of Computer Vision,
vol. 74, no. 1, pp. 59–73, August 2007.
[11] R. I. Hartley and A. Zisserman, Multiple View Geometry in Computer
Vision, 2nd ed. Cambridge University Press, 2004.
[12] C. Harris and M. Stephens, “A combined corner and edge detection,” inProceedings of The Fourth Alvey Vision Conference, 1988, pp. 147–151.
[13] J. Shi and C. Tomasi, “Good features to track,” CVPR, pp. 593–600,
Jun 1994.
[14] K. S. Arun, T. S. Huang, and S. D. Blostein, “Least-squares fitting of
two 3-d point sets,” PAMI, vol. 9, no. 5, pp. 698–700, 1987.
[15] J. Y. Bouguet, “Pyramidal implementation of the lucas kanade feature
tracker: Description of the algorithm,” 2002. [Online]. Available:
http://robots.stanford.edu/cs223b04/algo tracking.pdf[16] C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine
Learning (Adaptive Computation and Machine Learning). The MIT
Press, 2005.
[17] M. F. Moeller, “A scaled conjugate gradient algorithm for fast supervised
learning,” Neural Networks, vol. 6, pp. 525–533, 1993.