High-resolution Video Mosaicing for Documents and Photos
by Estimating Camera Motion
Tomokazu Satoab , Sei Ikeda a, Masayuki Kanbaraab ,Akihiko Iketani b, Noboru Nakajimab , Naokazu Yokoyaab , Keiji Yamadaba Nara Institute of Science and Technology, Ikoma, Nara, Japanb NEC, Ikoma, Nara, Japan

ABSTRACT
Recently, document and photograph digitization from a paper is very important for digital archiving and personal data transmission through the internet. To realize easy and high quality digitization of documents andphotographs, we propose a novel digitization method that uses a movie captured by a hand-held camera. In our
method, ﬁrst, 6-DOF(Degree Of Freedom) position and posture parameters of the mobile camera are estimatedin each frame by tracking image features automatically. Next, re-appearing feature points in the image sequence
are detected and stitched for minimizing accumulated estimation errors. Finally, all the images are merged asa high-resolution mosaic image using the optimized parameters. Experiments have successfully demonstrated
the feasibility of the proposed method. Our prototype system can acquire initial estimates of extrinsic camera
parameters in real-time with capturing images.
Keywords: Video mosaicing, Extrinsic camera parameter recovery, Image feature tracking

1. INTRODUCTION
In recent years, document and photograph digitization from a printed or drafted paper is very important for
digital archiving and personal data transmission through the internet. Though many people wish to digitize
documents on a paper easily, now heavy and large image scanners are required to obtain high quality digitization.
To realize easy and high quality digitization of documents and photographs, a hand-held camera that can be
freely moved is competent. A high-resolution image of a document is expected to be synthesized by applying avideo mosaicing technique to a movie taken by the mobile camera.
In the literature of video mosaicing, a number of methods have been explored and proposed. Szeliski1
developed an image based video mosaicing method using 8-DOF projective image transformation parameters
between pairs of input images. His method can be used when a target is a plane (planar image mosaicing) or
optical centers of images are approximately ﬁxed throughout the video capturing (panorama image mosaicing).
After his work, several methods are proposed for extending this mosaicing technique.2–7 One of the major
extensions is the use of image features instead of all the pixels in images in order to reduce calculation cost.2–4
Although the calculation cost is drastically decreased by such an extension, a synthesized image from a long
image sequence is usually distorted because they only stitch features between successive frames and ignore there-appearing features. There exists an error accumulation problem in estimating projective image transformation
parameters between images.
In order to synthesize a high-resolution mosaic image with minimum distortion from a plane object tracing
movie, we propose a novel video mosaicing method that estimates 6-DOF extrinsic camera parameters instead ofusing general 8-DOF projective image transformation parameters. First, our method estimates 6-DOF extrinsic
camera parameters for each frame tracking image features in the captured images. After the initial estimate,re-appearing features are stitched throughout the input images and accumulated errors are minimized. Our
method is based on the assumption that the documents are given on a plane paper and the image plane of
Further author information: (Send correspondence to T.S)
T.S, S.I, M.K, N.Y: E-mail: {tomoka-s,sei-i,kanbara,yokoya}@is.aist-nara.ac.jp
A.I, N.N, K.Y: E-mail: {iketani,noboru,yamada}@ccm.cl.nec.co.jp

(A) Extrinsic camera parameter estimation(a) Initial estimation by tracking features
Iterate from first frame to last frame(b) Detection of re-appearing features(c) Refinement of estimated camera parameters(B) Generation of mosaic image

Figure 1. Flow diagram of image mosaicing.

camera in the ﬁrst frame is approximately parallel to the plane object. Additionally, for reducing the degreeof freedom of camera parameters, intrinsic camera parameters must be estimated in advance and they must be
ﬁxed throughout the image capturing.
This paper is structured as follows. Section 2 describes a method for generating a high-resolution mosaic
image. In Section 3, experimental results with documents and photos show the feasibility and the accuracy ofthe proposed method. Finally, Section 4 describes conclusion and future work.

2. VIDEO MOSAICING BY ESTIMATING EXTRINSIC CAMERA PARAMETERS
This section describes a method for generating a mosaic image with minimum distortion by estimating extrinsic
camera parameters in capturing images. As shown in Figure 1, the proposed method ﬁrst estimates extrinsic
camera parameters of a freely moving mobile camera (A), and a mosaic image is then generated by using theestimated parameters (B).
In the following sections, ﬁrst, extrinsic camera parameters and error functions are deﬁned. The stages (A)and (B) are then described in some detail.

2.1. Definition of extrinsic camera parameters
In this section, extrinsic camera parameters and an error function for estimating them are deﬁned. In generalmosaicing methods for planar mosaicing, the 8-DOF projective image transformation parameter H f is used forimage registration and is deﬁned for each successive image pair as follows:(au(f +1)p , av(f +1)p , a)T = Hf (uf p , vf p , 1)T ,⎛⎞m1f m2f m3f
Hf = ⎝ m4f m5f m6f ⎠ ,m7f m8f1(1)(2)

where a is a parameter, and (uf p , vf p ) denotes the 2-D position of the feature p in the f-th frame image. By
using this equation, images can be registered even if intrinsic parameters of the camera are unknown. However,this expression easily accumulates estimation errors because transformation parameters are deﬁned only between
two successive frames.
To reduce accumulative errors, we deﬁne the 6-DOF transformation matrix Mf instead of Hf in Eq. (2)using known intrinsic parameters. In this paper, as shown in Figure 2, the transformation matrix Mf of thef-th frame is deﬁned between the mosaic image plane and the f-th frame image plane using camera position(t1f , t2f , t3f ) and camera posture (r1f , r2f , r3f ) parameters of the f-th frame.
(auˆf p , avˆf p , a)T = Mf (xp , yp , 1)T ,⎛c1 c3 + s1 s2 s3
Mf = ⎝ −s1 c3 + c1 s2 s3
c2 s3

s1 c2
c1 c2
−s2

⎞
t1f
t2f ⎠ ,
t3f

(3)

(4)

Mosaic image plane(xp, yp)M1(u1p, v1p)
First frame

Mf

M2
H1

(u2p, v2p) H2
2nd frame

Hf-1 (ufp, vfp)
f-th frame

Figure 2. Mosaic image plane and camera.

si = sin (rif ), ci = cos (rif )(i = 1, 2, 3),(5)where (xp , yp ) is the position of the feature p in the mosaic image plane, (û f p , v̂f p ) is the projected position of(xp , yp ) to the f-th frame image with the ideal camera model. The position (û f p , v̂f p ) in the ideal camera image
is transferred to the position (uf p , vf p ) in the real camera image by known intrinsic camera parameters including
focus, aspect, optical center and distortion parameters. The transformation matrix Mf deﬁned in Eq. (4) isessentially the same as a usual extrinsic camera matrix except the omission of z-axis parameters, because atarget object is always on the z=0 plane.
On the other hand, the position (uf p , vf p ) computed by Eq. (3) and known intrinsic camera parameters is
not always consistent with an actually detected position (uf p , vf p ) in the real image due to quantization anddetecting errors. In this paper, the squared errors Ef p is deﬁned as an error function for the feature p in thef-th frame as follows.

Ef p = (uf p − uf p )2 + (vf p − vf p )2 .
(6)
The sum of Ef p is employed for estimating Mf and (xp , yp ) in the following section.

2.2. Extrinsic camera parameter estimation
As shown in Figure 1, the proposed extrinsic camera parameter estimation method is constructed of the three
processes. First, extrinsic camera parameters Mf from the ﬁrst frame to the last frame are estimated by tracking
features with the iterating process for each frame (a). Re-appearing features in the whole acquired images arethen detected for bounding features even if they do not appear in successive frames (b). Finally, estimatedextrinsic camera parameters are reﬁned globally in the whole input image sequence (c). The following brieﬂy
describes each process.
2.2.1. Initial estimation by tracking features
In this process for each frame, an initial estimate of extrinsic camera parameter Mf is computed by tracking
image features. This process is basically an extension of our previous work.8 In the ﬁrst frame, Mf is set as
an identity matrix in accordance with the assumption that the image plane in the ﬁrst frame is approximately
parallel to the target object. The positions (xp , yp ) in the mosaic image plane of image features in the ﬁrst frame
are also computed based on this assumption. Note that even if the target object and the image plane of the ﬁrst
frame are not accurately parallel to each other, they are corrected at the reﬁnement process.
In the succeeding frames (f > 1), Mf is determined by iterating the following steps until the last frame.
Tracking of image features: All of image features are automatically tracked from the previous frame to thecurrent frame by using a standard template matching with Harris corner detector.9 The RANSAC approach10 is also employed for eliminating outliers.

Projection to mosaic
image plane

(a)

(b)

(c)

Normalized crosscorrelation test(d)Figure 3. Detection of re-appearing features. (a) camera path, posture and feature positions on mosaic image plane, (b)sampled frames of input images, (c) templates of a feature on diﬀerent images, (d) templates projected to a mosaic image
plane.

Extrinsic camera parameter estimation: The tracked position (u f p , vf p ) in the f-th frame and the positionin the mosaic image plane (x p , yp ) are used for estimating the extrinsic camera parameter Mf . In thisstep, the error function p Ef p is minimized by the non-linear squared minimization.
Estimation of feature position on mosaic plane: The position (x p , yp ) in the mosaic
 image plane of eachfeature p is computed and reﬁned in every frame by minimizing the error function f Ef p .
Addition and deletion of features: In order to obtain accurate estimates of camera parameters, good features should be selected. The set of natural features is automatically updated by checking conditions offeatures using multiple criteria.8
By iterating the above steps, initial estimates of extrinsic parameters Mf and feature positions (xp , yp ) in themosaic image plane are automatically computed from the ﬁrst frame to the last frame.
2.2.2. Detection of re-appearing features
In an input movie, some of image features usually come in to and come out from the sight of the camera inseveral times due to the camera motion. In this research, these re-appearing features are detected for bounding
features to reduce accumulative estimation errors.
As shown in Figure 3, the same image feature in a non-successive image pair often exhibits diﬀerent looks as
an eﬀect of camera motion. To remove this eﬀect, ﬁrst, templates of all the features are projected to the mosaic
image plane. Next, feature pairs whose distance is less than a given threshold are selected and tested with thenormalized cross correlation function using multi-resolution templates. By using this approach, re-appearing
features can be detected even if the templates of the features are rotated and distorted by camera motion.
2.2.3. Refinement of estimated camera parameters
In this process, the accumulation of estimation errors is minimized over the whole input. The accumulated
estimation error E is given by the sum of re-projection errors as in Eq. (7) and is minimized with respect to thecamera parameters Mf and the feature positions (xp , yp ) in the mosaic image plane over the whole input.

Ef p .
(7)E=
fp

By minimizing this error function, extrinsic camera parameters are reﬁned and accumulated errors of the initial
estimates are minimized.

2.3. Generation of mosaic image
At the ﬁnal stage, a mosaic image is generated by projecting pixels in the mosaic image plane to all the frame
images using Eq. (3) with reﬁned 6-DOF extrinsic camera matrix Mf . In this paper, a blending method is
employed to generate a smooth mosaic image.
A color of pixel on the mosaic plane C(x, y) is computed by colors of input images I f (u, v) and blending
function W (u, v) as follows.
f W (uf , vf )If (uf , vf )C(x, y) =,(8)f W (uf , vf )W (u, v) =(u2 + v2 )−1 ;0;(u, v) ⊆ image region,
otherwise(9)

where (uf , vf ) is a projected position of (x, y) to the f-th frame image plane obtained by applying Eq. (3) andknown intrinsic camera parameters.

3. EXPERIMENTS
To show the feasibility of the proposed method, a prototype image mosaicing system is constructed with adesktop PC (Pentium-4 3.2GHz, Memory 2GB) and a calibrated IEEE1394 CCD camera (Aplux C104T). The
camera is hand-held. In experiments, two kinds of plane papers are used for the target objects. One is a printed
A4 size document (sequence 1). The other is a printed photograph on an A4 size paper (sequence 2). In both
papers, plus marks (+) are printed on 40mm grid points for quantitative evaluation.

3.1. Mosaicing for a document
As shown in Figure 4, a target paper of document is captured as 640 × 480 images of 150 frames at 15 fps. First,
initial estimates of extrinsic camera parameters are acquired by automatically tracking image features as shownin Figure 5. In this experiment, 95 points on average of the image features are tracked per frame. After the initial
estimate, re-appearing features are detected and accumulated errors are minimized. The number of detected
re-appearing features is 129 points in this experiment, and the average re-projection error of the features after
reﬁnement is 0.75 pixel. Figure 6 illustrates the acquired extrinsic camera parameters and the feature positions
on the mosaic image plane after the reﬁnement process. The curved line shows the estimated camera path andpyramids show the camera postures in every 10 frames.
Finally, all the input images are uniﬁed into a mosaic image by using the estimated camera parameters asshown in Figure 7. The generated mosaic image size is 1600 × 1916. We can conﬁrm that the distortion ofsynthesized images is very little. However, some part of the mosaic image is unclear due to the shortage of the
input image resolution. Super-resolution techniques should be attempted to solve it. The performance of oursystem for this sequence is as follows: 15 fps for image acquisition and initial parameter estimation, 1 second for
detecting re-appearing features in 150 frames, 25 seconds for camera parameter reﬁnement, and 16 seconds formosaic image generation.

3.2. Mosaicing for a photographIn this experiment, the target is a printed photograph. The target photograph is captured as 640 × 480 images of
150 frames at 15 fps as shown in Figure 8. Simultaneously, image features are automatically tracked for acquiring
initial estimates as shown in Figure 9. The average number of tracked features is 90 points per frame. Afterthe detection of 210 re-appearing features, the extrinsic parameters are reﬁned. The average re-projection errorof the features is 0.98 pixel. Figure 10 shows the extrinsic camera parameters and the feature positions on themosaic image plane after reﬁnement. It can be observed that the recovered camera path is smooth and there isno discontinuity.
Finally, the mosaic image is generated as a 1600 × 1997 size image as shown in Figure 11. The performance of
our system for this sequence is as follows: 15 fps for image acquisition and initial parameter estimation, 1 secondfor detecting re-appearing features in 150 frames, 42 seconds for camera parameter reﬁnement, and 21 seconds
for mosaic image generation. Although we can conﬁrm that the distortion of synthesized images is very little,
the re-projection error of the features is bigger and the computational time for the reﬁnement is longer than the
results for the document because there are more similar patterns in the photograph than in the document.

ﬁrst frame

30-th frame

60-th frame

90-th frame

120-th frame

150-th frame

Figure 4. Sampled frames of an input movie (sequence 1).

ﬁrst frame

30-th frame

60-th frame

90-th frame

120-th frame

150-th frame

Figure 5. Result of feature tracking (sequence 1).

(a) top view(b) side view
Figure 6. Estimated extrinsic camera
parameters and feature positions (sequence 1).

Figure 7. Generated mosaic image (sequence 1).

ﬁrst frame

30-th frame

60-th frame

90-th frame

120-th frame

150-th frame

Figure 8. Sampled frames of an input movie (sequence 2).

ﬁrst frame

30-th frame

60-th frame

90-th frame

120-th frame

150-th frame

Figure 9. Result of feature tracking (sequence 2).

(a) top view(b) side view
Figure 10. Estimated extrinsic camera parameters and feature positions(sequence 2).

Figure 11. Generated mosaic image (sequence 2).

3.3. Quantitative evaluation
In this section, the distortions of the generated mosaic images are quantitatively evaluated by measuring the
distances between adjacent grid positions. In the above two experiments, plus marks (+) have been printed
on the target paper at every 40mm grid positions for this evaluation. First, the positions of the plus marks
are acquired manually in the generated mosaic image. The distances between adjacent plus marks are thencomputed in the unit of pixel. The average, maximum, minimum and standard deviation of the distances areshown in Table 1. The percentage of each value from the average distance is also shown in parenthesis. In
this table, the standard deviation is considered as the average distortion of the generated image. Although theaverage distortion of the photograph is a little worse than that of the document, both the average distortions
are suﬃciently little for the purpose of digital archiving and personal data transmission.
Table 1. Distances of adjacent grid points in generated mosaic images [pixels(percentage from average)]targetdocument
photographaverage
255.5(100.0)
245.6(100.0)

maximum
258.0(101.0)
249.0(101.4)

minimum
253.0(99.0)
243.0(98.9)

standard deviation
1.20(0.47)
1.27(0.52)

4. CONCLUSION
This paper has proposed a method for generating a mosaic image from a freely moving camera with minimum
distortion. In experiments, two kinds of mosaic images are successfully generated in short time. In future
work, super-resolution techniques should be attempted using estimated camera parameters for acquiring higher
resolution images. The mosaicing method for non-plane targets will also be explored to realize a more usefuland practical mosaicing system.

REFERENCES
1. R. Szeliski: “Image Mosaicing for Tele-Reality Applications,” Proc. IEEE Workshop on Applications ofComputer Vision, pp. 230–236, 1994.
2. N. Chiba, H. Kano, M. Higashihara, M. Yasuda and M. Osumi: “Feature-based Image Mosaicing,” Proc.
IAPR Workshop on Machine Vision Applications, pp. 5–10, 1998.
3. S. Takeuchi, D. Shibuichi, N. Terashima and H. Tominage: “Adaptive Resolution Image Acquisition Using
Image Mosaicing Technique from Video Sequence,” Proc. IEEE Int. Conf. on Image Processing, vol. I,
pp. 220–223, 2000.
4. C.T. Hsu, T.H. Cheng, R.A. Beuker and J.K. Hong: “Feature-based Video Mosaicing,” Proc. IEEE Int.
Conf. on Image Processing, vol. II, pp. 887–890, 2000.
5. M. Lhuillier, L. Quan, H. Shum and H. T. Tsui: “Relief Mosaicing by Joint View Triangulation,” Proc.
IEEE Int. Conf. on Computer Vision and Pattern Recognition, vol. I, pp. 785–790, 2001.
6. W. Du and H. Li: “Construction of Image Mosaics with Video Texture,” Proc. Asian Conf. on Computer
Vision, vol. II, pp. 871–876, 2002.
7. U. Bhosle, S. Chaudhuri and S.D. Roy: “A Fast Method for Image Mosaicing Using Geometric Hashing,”
IETE Journal of Research, Special Issue on Visual Media Processing, vol. 48, no. 3-4, pp. 317–324, 2002.
8. T. Sato, M. Kanbara, N. Yokoya and H. Takemura: “Dense 3-D Reconstruction of an Outdoor Scene by
Hundreds-baseline Stereo Using a Hand-held Video Camera,” Int. Jour. of Computer Vision, vol. 47, no.
1-3, pp. 119–129, 2002.
9. C. Harris and M. Stephens: “A Combined Corner and Edge Detector,” Proc. Alvey Vision Conf., pp. 147–
151, 1988.
10. M.A. Fischler and R.C. Bolles: “Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography,” Communications of the ACM, vol. 24, no. 6,
pp. 381–395, 1981.