Augmented Reality Based on Estimation of
Defocusing and Motion Blurring from Captured Images
Bunyo OKUMURA ∗Masayuki KANBARA ∗Naokazu YOKOYA ∗Graduate School of Information Science
Nara Institute of Science and Technology
8916–5 Takayama, Ikoma, Nara, 630–0192, JapanA BSTRACT
Photometric registration is as important as geometric registration to
generate a seamless augmented reality scene. Especially the difference in image quality between a real image and virtual objects
caused by defocusing and motion blurring in capturing a real scene
image easily exhibits the seam between real and virtual worlds.
To avoid this problem in video see-through augmented reality, it
is necessary to simulate the optical system of camera when virtual
objects are rendered. This paper proposes an image composition
method for video see-through augmented reality, which is based on
defocusing and motion blurring estimation from the captured real
image and rendering of virtual objects with blur effects. In experiments, the effectiveness of the proposed method is confirmed bycomparing a real image with virtual objects rendered by the proposed method.
CR Categories: H.5.1 [Information interfaces and presentation]:
Multimedia Information Systems—Artificial, augmented, and virtual realities
Keywords: augmented reality, photometric registration, blur estimation, point spread function, image quality

1

I NTRODUCTION

Augmented reality is a technology which provides users withlocation-based information by overlaying virtual objects on the realworld[1, 2]. In augmented reality, it is important to resolve someregistration problems including geometric and photometric registration problems. The geometric registration means that the position and the orientation of virtual objects are consistent with thereal world under varying viewpoints. The photometric registrationmeans that shadow and shading of virtual objects and their image
quality are consistent with the real world.
Recently, there are a number of researches that are intended toincrease the photometric consistency[3, 4, 5]. However, they haveconcentrated mainly on shadow and shading of virtual objects andthere are few attempts focused on the quality of virtual objects inthe composed image. Especially, in video see-through augmented
reality, the seamlessness of the real and virtual worlds is affectedby the differences in image quality between superimposed virtual
objects and the real image. This problem is caused by the difference between real and virtual camera models. In general, an
ideal camera model which means that the image quality is not degraded by the lens is used when rendering virtual objects. However, there is deterioration in an image captured by a real camera.
Some attempts have been done to reduce the difference between
real and virtual worlds, which change the image quality of real
image and rendered virtual objects with “cartoon-like” or “sketchlike” representation[6, 7].
∗ e-mail: {bunyo-o, kanbara, yokoya}@is.naist.jp

In order to resolve the real and virtual camera model inconsistency, it is necessary to estimate the deterioration of captured image
and to represent the blur effect on virtual objects. In image processing and computer vision, image restoration methods based on
estimation of image degradation described by the point spread function (PSF) have been proposed[8]. However, these techniques needmuch calculation cost because most of them are based on iterative
computation. Thus such techniques are not suitable for augmented
reality which requires real time operation. On the other hand, incomputer graphics, there are some researches which achieve realistic rendering of virtual objects simulating the real camera[9, 10].
However, it is also difficult for these techniques to apply to augmented reality because lens parameters, which are changed by lens
operations, always need to be known and intrinsic camera parameters should be calibrated in advance in the related work.
To realize consistent image quality in augmented reality, this paper copes with both defocusing and motion blurring caused by thecamera, that are typical factors of the deterioration of image. We
propose a technique which equalizes the quality of the real image
captured by the real camera and the rendered virtual objects. The
proposed method merges a captured image and rendered virtual objects with blur effects estimated from a real scene image containing a marker. We also propose an improved geometric registration
method which can detect the image marker accurately by eliminating the blur effect from the captured image by using the estimatedblur parameters.
This paper is structured as follows. Section 2 briefly reviews
related work. Section 3 describes a blur model of image, a blurestimation method, and a method for estimating the camera position and orientation from an image marker. A method of rendering
virtual objects based on reproduction of estimated blur is explained
as well. In Section 4 experimental results are described. Finally,
Section 5 gives conclusion and future work.
2

R ELATED WORK

Estimation of image degradation has been studied in computer vision. In computer graphics, some techniques which can render virtual objects with blur effects have been proposed.
Estimations of blur from images: In computer vision, in order torestore the degraded image, most methods estimate the PSF whichmodels the degradation of image and recover the image by removing the effects of the PSF. In general, these methods can be divided into two approaches according to assumptions. One estimates the blur effect based on analyzing patterns on a target object
as knowledge. The other does not have any knowledge about thetarget objects. One of the former approaches is an estimation of
the PSF of a scanner[11]. This technique can estimate the scanner property described by the PSF scanning known patterns. The
latter approach is called blind deconvolution, which attempts to recover target objects from a set of blurred images. There are a number of techniques which treat the same problem. The depth fromdefocusing[12, 13, 14] and the super-resolution[15, 16, 17] are typ-ical techniques, which also estimate the PSF simultaneously.
Blur representation on virtual objects: In computer graphics,
there are a number of techniques to reproduce various effects
caused by a camera. These techniques contribute to render virtual
objects realistically. For example, Nakamae et al.[18] have proposed a photo montage technique for landscape environment assessment. This technique can improve the quality of final montage image with consideration of the seamlessness of the border
between a photograph and superimposed virtual objects. Furthermore, there are a number of researches to generate more realistic
effects caused by a real camera assuming lens model of the camera. For example, Kolb et al.[9] have proposed a rendering technique simulating an optical system of real camera. This method
can reproduce the depth of focus effect which is characterized by
the design of lens. Asada and Baba[10] have proposed a rendering
technique with a new camera model which has zoom, focus and irisparameters. Their method can treat the relation among lens parameters systematically and achieve rendering of virtual objects withthe depth of field effect in real time.
The proposed method in this paper combines blur estimationin computer vision with blur representation in computer graphics.
Note that most of PSF estimation methods in computer vision are
aimed at arbitrary PSFs. These methods take much calculation cost
and thus it is difficult to apply such methods for augmented reality which requires real time processing. In the proposed method,in order to achieve the real time processing, the PSF which represents both defocusing and motion blurring is modeled approximately and the parameter of PSF is estimated by using an image
marker whose color and shape are known. On the other hand, incomputer graphics, most techniques for rendering blur effects assume off-line processing. In addition, many parameters changed bylens operations of the camera are required to simulate blur effects.
In the proposed method, the blur effect is represented by applying
a simple filter determined by the PSF to rendered virtual objects.
Moreover, the calculation cost of this processing is reduced by using graphics hardware acceleration.
3

AUGMENTED REALITY BASED ON BLUR ESTIMATION

The proposed method reduces the difference in image quality between the real and virtual objects in augmented reality by estimating the PSF from an observed image marker in a real scene, andreproducing the PSF to virtual objects. We use pixel intensities
around the edge of image marker to estimate the PSF. When theimage is blurred, the intensity changes gradually along the direction orthogonal to the edge. In this paper, the change in intensity
profile on the edge is described as the width of blur. The shape of
PSF is estimated by integrating the width of blur in edges in various
directions.
In some augmented reality systems, virtual objects are composed
around image markers in the real world[19]. In this case, we canassume that the blur effect on virtual objects is represented by the
degree of blur at the marker because the PSF is related to the depthof the target object and the change of depth is not large around themarker in most case.
Fig. 1 illustrates a flow diagram of the proposed method. First,
the real scene is captured by a camera, and image markers whosecolor and shape are known are detected (Fig. 1 (A)). In the second
step, the width of blur is estimated by fitting a function defined by amodel of PSF to intensities in an edge region of the marker, and theparameters of PSF are acquired by integrating the estimated widthof blur in various directions (Fig. 1 (B)). Next, the camera position and orientation in marker coordinate system are estimated byrecovering the shape of the marker using the estimated blur parameters (Fig. 1 (C)). Finally, virtual objects are rendered with the blureffect and are composed with the real image (Fig. 1 (D)). The blur(A) Detection of image marker(B) Estimation of PSF parameters from captured image(C) Estimation of camera position and orientationin consideration of blur effect(D) Rendering of virtual objects with blur effectFigure 1: Flow diagram of the proposed method.
Focal plane
Image plane

Lens

ObjectOptical Axisfwv

Optical disk
caused by the defocusing(radius of optical disk: r)

f : focal length of the lens v : distance from the lens to the image planew : depth from the lens to the object
Figure 2: Defocusing caused by the lens.

model of a camera and the image marker as well as processes (A)to (D) are explained in the following sections.
3.1 Model of defocusing and motion blurring
In general, the blur effect caused by the camera includes two major factors: defocusing and motion blurring. The blurred image is
generated by convolution of an ideal image with the PSF which expresses effects of defocusing and motion blurring. The blur caused
by defocusing depends on depth from the center of lens to an object, the aperture of lens, and the resolution of camera. The blur ofimage is caused by defocusing of lens as shown in Fig. 2. Note thatthe radius r of an optical disk on the image plane is defined by Eq.
(1), and r depends on the position in the image[20].
¯¯¯1 1 1¯¯
r = ργ v ¯ − − ¯¯ ,
(1)
f v w
where f , w and v mean the focal length of lens, the depth from lensto an object and the distance from lens to the image plane, respectively. γ and ρ mean the aperture of lens and a constant which
depends on the resolution of CCD, respectively.
In general, two dimensional circular spread of light is used asa kernel of PSF (pill-box) of defocusing. In addition, the radius rof optical disk depends on the position in the image. In this paper,
the PSF is assumed to be a function which does not depend on theposition in the image[21], since virtual objects are placed around amarker in the captured image; that is, we assume that the radius of
optical disk r is constant around the marker. The PSF of blur caused
by defocusing of camera is defined as follows:½ 1; where x2 + y2 ≤ r2
π r2
R(x, y; r) =,(2)0
; otherwise
where x and y represent the position of target pixel, and r shows
the radius of the PSF which is equivalent to the radius of opticaldisk. In this paper, r is considered as the parameter of blur caused
by defocusing.

The motion blurring depends on the motion of camera, movement of target object and the exposure of camera. The motion ofcamera and the movement of target object are generally very complicated, and so the formulation of the PSF is very complex. In
this paper, we assume that the motion of camera and the movementof target object can be represented by one-dimensional translationin the image plane of camera. The PSF of motion blurring is now
again as follows:
L(x, y; l, θ )
where

µ 0¶
x
y0

=

¢
1 ¡ 0
u(x + l) − u(y0 − l) δ (y0 ),2lµ=Figure 3: An example of marker.

(3)

(4)

Note that x and y represent the position of target pixel. The function u(x) shows the unit step function and δ (t) denotes the Dirac’sdelta function. l and θ mean the length of uniform motion and the
direction of motion, respectively. Note that l and θ are consideredas parameters of blur effect by motion blurring.
This paper treats the blur effect caused by both defocus and camera motion. The model of PSF is obtained by convoluting Eq. (2)with Eq. (3). However, the calculation cost of convolution requiredin estimation of the PSF parameters is not small, thus it is not acceptable for augmented reality. In order to achieve the real timeprocessing, we simplify the PSF. In the proposed method, the following approximated PSF is employed.
P(x, y; r, l, θ )(=1
π ((r+l)2 +r2 )

0
where

µ 0¶
x
y0

µ
=

³

x0
r+l

´2

;
;

+
otherwisecos θ− sin θ

sin θ
cos θ

³ 0 ´2
y
r

¶µ ¶
x
.
y

(b) Circular edge for estimating
the parameters of PSF(c) Corners for estimating
the position and orientationof camera

3.3 Detection of marker from captured image: Process (A)

¶µ ¶
sin θ
x
.
cos θ
y

cos θ
− sin θ(a) ID area

≤1 ,

(5)

(6)

Here, x and y represent the position of target pixel in the image. r,l and θ show the radius of defocusing blur, the length of uniform
motion and the direction of motion, respectively. In this paper, r, land θ are considered as parameters of the PSF of blur. The above
equation means an elliptical spread of light on an image plane. The
direction of major axis of the ellipse shows the motion direction ofcamera. From this approximation of PSF, the profile of blurred edge
can be described by a formula, so that calculation cost of estimating
the PSF parameters can be reduced.

Marker of known shape and color is detected by the following standard processes.
1. Extraction of marker region: An image captured by camera is binarized and is labeled. Regions whose size is large
enough are detected as candidates of marker.
2. Detection of marker corners: Each edge of the marker isdetected by fitting a straight line. Corners of marker are thendetermined by calculating intersections of detected lines.
Note that the position of corner of marker may include some errors
caused by the blur effect. When a threshold is not suitable in the binarization process, the marker will be shrunk or expanded because
the intensity of blurred edge changes gradually. Since the camera
position and orientation are estimated by solving the PnP problem(n = 4)[19], the accuracy of position and orientation of camera isaffected by the accuracy of detected corner positions in the image.
In this paper, this problem is resolved by eliminating the blur effectin process (C) described in Section 3.6.
3.4 Estimation of width of blur: Process (B)The width of blur is estimated in various directions at multiple
points on the circular edge of marker before estimating parametersof PSF. This section explains estimation of the width of blur fromchange of intensity along the direction orthogonal to the edge.
Provided that an ideal edge profile of marker is expressed by thestep function shown in Fig. 4(a), the intensity of edge affected by
the blur effect expressed by the PSF[13] is given as follows:f (~p; d,~p0 , imin , imax ) imin; t < −1g(t)(imax − imin ) + imin ; −1 ≤ t ≤ 1 ,=i; t >1max

3.2 Image marker
A square marker with known shape and color is used to estimate thecamera position and orientation[19]. Fig. 3 illustrates the pattern ofthe marker used in this study. The edge of inner circle in the marker
is used to estimate the parameters of PSF. The role of each part ofthe marker is as follows.
(a) ID area: This area is used to identify the marker by recognizing the pattern of black circle.
(b) Circular edge: This edge is used to estimate parameters of
PSF. The proposed method acquires the PSF from the parameters estimated at multiple points on the edge in various directions.
(c) Corners of square marker: These corners are used to determine the camera position and orientation in the marker coordinate system for geometric registration.

(7)

where
t

=

g(t)

=

(~p −~p0 ) ·~o
,
d
´
³
p
1 1
+
t 1 − t 2 + arcsint ,
2 π

(8)
(9)

and ~o is the normalized direction orthogonal to the edge. ~p and~p0 are the position of pixel and the position of edge which is notaffected by the blur effect, respectively. The parameter d means
the width of blur which is equivalent to the width of slope of theintensity along the direction orthogonal to the edge. imin and imax
show the minimum and maximum values of intensities of the edge.
Parameters of blur are estimated by fitting a function shown in Eq.
(7) to the edge profile in the image. Fig. 4(a) illustrates the profiles of ideal and blurred edges. Parameters d,~p0 , imin and imax arewhich represent edges of the marker. The camera position and orientation are estimated from positions of corners of the marker in
the image. The detail of processing is as follows.

Intensity ofpixel

i m ax
Ideal edge

Width of blur dIntensity of pixelr

1. The process described in Section 3.4 is applied to outer edgesof the marker since this process can estimate the accurate edge
position ~p0 under blur effect.

f ( p)i m inr

Edge position p0r
Position of pixel ( p i)(a) Edge profile and its function fitting.

(b) Sampled pixels
used in blurestimation.

2. The lines of outer edge of the marker are estimated by fittinglines to edge positions.
3. Four corners of the square marker are determined from intersections of lines, and the camera position and orientation in
the marker coordinate system are then computed by solving
the PnP problem (n = 4).

Figure 4: Function fitting to blurred edge and sampled pixels usedin blur estimation.

estimated by minimizing the error defined by Eq. (10) using thequasi-Newton method.
Eedge (d,~p0 , imin , imax )N=∑ {I(~pi ) − f (~pi ; d,~p0 , imin , imax )}2 .

(10)

i=1

3.7

In Eq. (10), ~pi , I(~pi ) and N mean the position of the i-th pixel, theintensity of the pixel along the direction orthogonal to the edge andthe number of sampled pixels, respectively. Lines in Fig. 4(b) indicate pixels used for blur estimation. The position of circular edgein the image marker is determined from a marker region detected inprocess (A) described in Section 3.3.
3.5 Estimation of PSF parameters: Process (B)The blur sizes estimated at multiple points on the edge of circle areintegrated to determine parameters of the PSF. Note that the kernelof PSF has an elliptical shape and the eccentricity of the ellipse is
the same as the estimated width of blur. The parameters r, l and θof the PSF are estimated by minimizing the error function definedby Eq. (11) using the quasi-Newton method.
EPSF (r, l, θ )

M

=

∑

j=1

©

ª2
d j − h(θ j ; r, l, θ ),Eedge, j

(11)

where
h(θ j ; r, l, θ )

=

r³

1
´2 ³
´ ,
cos(θ j −θ )sin(θ j −θ ) 2+
r+lr

Note that the camera position and orientation can be more accurate
than those estimated by standard techniques such as [19] because
the positions of corners estimated by this process are not influenced
by the blur effects.

(12)

and j represents the index of edge. r, l and θ mean parameters of
the PSF described in Section 3.1. h(θ j ; r, l, θ ) represents the eccentricity of the ellipse about the direction θ . d j and Eedge, j denote thewidth of blur estimated from j-th point on edge and the error in theestimation mentioned in Section 3.4, respectively.
3.6 Estimation of camera position and orientation in consideration of blur effects: Process (C)In order to estimate the camera position and orientation accurately,
it is necessary to recover the shape of marker by removing the influence of blur. There are two approaches to recover the marker
shape. One is an image processing method based on inverse filtering. The marker is restored by applying the inverse filter to theimage degraded by blur effects. The other is a restoration method
using geometrical features used to estimate the edge of marker. In
this paper, to achieve a real time processing required by the augmented reality, the latter based on the estimation of edge position isapplied. Corners of marker are detected from intersections of lines

Rendering of virtual objects with blur effects: Process (D)In order to achieve real time computation, the processes of rendering and simulation of blur effects are computed using graphics
hardware. First, virtual objects are rendered to a texture buffer withthe camera position and orientation estimated in Section 3.6. Then,a smoothing filter with the estimated PSF is applied to the rendering result. The kernel of PSF is the same as Eq. (5). Here, only aneighborhood of 4(r +l)2 pixels is calculated to accelerate the computation. Furthermore, the branching function of OpenGL shading
language is used so that pixels outside of the PSF are excluded.
4

E XPERIMENTS

In experiments, in order to confirm that the proposed method candecrease the difference in image quality between the real image
and virtual objects, we have compared the real image with a virtual object rendered by the proposed method. Experiments are carried out using a desktop PC (CPU : PentiumD 3.0GHz, Memory :3.25GByte, Graphics card : Radeon X1900XTX) and a USB camera (ARGO Lu-135c, Resolution : 1024 × 768 pixels, Capturing
frame rate: 15fps). Square markers are placed in the real world anda virtual object is composed by recognizing the markers. Moreover, the graphics hardware is used when the filtering process of
blur effects is applied to virtual objects as mentioned in the previous section. First, we confirm the accuracy of parameters of PSF
estimated by the proposed method described in Sections 3.4 and
3.5.
4.1 Evaluation of camera position estimation using blur parameters
In this section, in order to verify the accuracy of camera position
estimation using blur parameters, we have carried out a quantitativeevaluation. The estimated distance from camera to marker is investigated under varying the focus of camera and fixing the positionsof camera and marker. Here, the distance from camera to marker is
set to 900mm.
Fig. 5 shows the relationship between the estimated width ofblur and the estimated camera position when the focus of camerachanges. Figs. 5 (a) and (b) are cases when the focus is changed tonear and far, respectively. They also illustrate selected 250 samples
under various focus values. From these results, we can confirm thatthe distance from camera to marker by the conventional method[19]increases when the size of defocusing is large. This is become theregion of marker is detected by a simple binarization method with

Proposed Method

Mean error in estimatedparameter of defocusing[pixel]Estimated distance fromcamera to marker [mm]

960
950
940
930
920
910
900
890

Conventional method

2

5

l=0
l = 0.5l=1
l = 1.5l=2
l = 2.5l=3
l = 3.5
l=4
l = 4.55

r = 0.5
r=1
r = 1.5
r=2
r = 2.5
r=3
r = 3.5
r=4
r = 4.5

1.5
1
0.5
0
-0.5
0

2

3

4
5
6
7
8
9 10 11 12
Estimated size of defocusing (r ) [pixel]

13

0.5

1

14

1.5
2
2.5
3
3.5
The size of defocusing (r ) [pixel]

44.5(a) The difference in the size of defocusing (r).

(a) Shortsighted focusing.
960
950
940
930
920
910
900
890

Mean error in estimatedParameter of motion
blurring [pixel]

Estimated distance fromcamera to marker [mm]1.5Proposed Method
Conventional method

1
0.5
0
-0.5
-1
0

2

3

4
5
6
7
8
9 10 11 12
Estimated size of defocusing (r ) [pixel]

1314(b) Longsighted focusing.

0.5

1

1.5
2
2.5
3
3.5
4
The size of motion blurring (l ) [pixel]

4.5(b) The difference in the size of motion blurring (l).
Figure 6: Errors between estimated parameter and parameter usedin simulation.

Figure 5: Estimated distance from camera to marker under change
of focus of camera.

a threshold and thus extracted region is smaller than the actual sizeof the marker. On the other hand, it is clearly shown in Fig. 5 thatthe distance is estimated accurately by the proposed method even if
the width of blur is large.
4.2

Evaluation of estimated parameters of PSF in simulation

To verify the accuracy of parameters of PSF, we have carried out aquantitative evaluation of estimated parameters in simulation. Images used in this experiment are generated with various blur values.
The parameter of defocusing r is changed by 0.5 from 0.5 to 4.5.
Parameters of motion blurring l and θ are changed by 0.5 from 0
to 4.5 and by π /8 from 0 to π , respectively. In this evaluation, we
have computed the difference between ground truth and the estimated parameters. The difference larger than zero means that theestimated parameter of blur is larger than the simulated parameter.
Fig. 6 illustrates the relationship between the simulation andestimated parameters. In Fig. 6(a), each sequence shows the variation of the parameter r for different values of parameter l. Except
for the case where parameter r is 0.5, the difference of parameter r
between the simulation and estimated parameter is about 0.5. Fig.
6(b) shows the difference of the parameter l. Each sequence shows
the variation of the parameter l for different values of parameterr. The difference of parameter l between the simulation and estimated parameter is about −0.5 excepting the case where parameterr is 0.5. In addition, the difference of the parameter θ between thesimulation and estimated parameter is from −0.24 to 0.24 radians.
Here, estimated parameters of PSF include errors because the estimation of width of blur becomes unstable when the parameter r issmaller than 1. However, we consider that the estimated parametersof PSF are almost accurate in order to reproduce both defocusing
and motion blurring.
4.3 Comparison between real and virtual objects in augmented scene image
To confirm the effectiveness of the proposed method, rendered virtual objects are compared with a real image fixing and changing the

focus of camera. We compare real objects captured by the camera
with composed virtual objects. Composed results of virtual objects
by a conventional method which does not represent the blur of real
image are also shown.
Static scene with motion blurring: Fig. 7 shows real and augmented scene images under motion blurring. Figs. 7(a), (b) and (c)show a captured image, an augmented image without considerationof motion blurring and an augmented image with consideration of
motion blurring, respectively. We can see the difference between
the real object and the virtual object in Fig. 7(b). For example, thepattern in the real cube circumscribed by (A) in Fig. 7(a) is affectedby the blur effect and horizontal edges in the image are disappeared.
However, edges of the virtual object circumscribed by (B) in Fig.
7(b), which is rendered without motion blurring estimation, exist
distinctly. On the other hand, edges of the virtual object circumscribed by (C) in Fig. 7(c), which is rendered with motion blurring
estimation, are disappeared and the appearance is similar to that of
the real object.
Static scene with different focus of camera: Fig. 8 shows theexperimental scene. A cylindrical real object is located at 150mm
from the camera. The marker and the real object for evaluation are
at 300mm from the camera. A checker board is placed at 900mm
from the camera. The real object to be compared is a cube whose
surfaces have grid patterns, and the same object is merged into areal image as a virtual object. When the virtual object is rendered,
the light environment is manually tuned and the shadow was notrendered.
Fig. 9 shows augmented scene images with different focuses.
Fig. 9(i) is focused on a near object, (ii) is focused on a marker, and(iii) is focused on a far object. In augmented scenes without blurestimation in column (a) of Fig.9, though a blur effect on the real
object (right cube) is caused by defocusing, the virtual object (leftcube) is composed without such an effect. Therefore, there can beobserved the inconsistency of image quality between the real image
and the virtual object and the seamlessness is not achieved. On theother hand, in column (b) of Fig. 9, the incompatibility caused by
the difference in image quality has been reduced because the image quality of virtual object is matched with the real image. In the
proposed method, the reproduced blur effect is uniform around themarker, although the difference in image quality is reduced. The

(A)

(B)

(C)

(a) Captured image(b) Rendered image without motion blur(Camera motion is upward in the image.)estimation.

(c) Rendered image with motion blurestimation.

Figure 7: Results of merging a virtual object into a real scene image with motion blurring

The former makes it possible to represent blur effects more accurately, and the latter makes it possible to apply blur effects on virtual
objects not only around the marker but also at arbitrary positions.
R EFERENCES

150mm 300mm

900mm

Figure 8: Experimental environment

frame rate is approximately 15fps for rendering without blur estimation and 10fps for rendering with blur estimation. The frame rateis downed because the proposed method requires additional computational cost for estimating parameters of PSF and for applying blureffects on virtual objects.
5

C ONCLUSION AND FUTURE WORKThis paper has proposed an image composition method which isbased on estimation of blur from a real image for video see-throughaugmented reality. The proposed method can estimate parametersof approximated PSF by fitting a function to edges in a marker andby fitting an ellipse to the width of blur in various directions of themarker edge. We have also proposed a new method for estimating
the camera position and orientation, which can eliminate the blureffect from the captured image. In experiments, have confirmed thatthe proposed method correctly estimates the parameters of PSF in
simulation. Consequently, the method can seamlessly merge virtual
objects into the real image which is affected by blur effects caused
by defocusing and by motion blurring of camera. This has been
proven through experiments.
In the present study, the approximated PSF has an elliptical shapeand does not depend on the depth in the image. There are some limitations caused by this assumption. The virtual objects can be placed
only around the marker, and PSFs of real cameras may not have an
elliptical form. The following topics should further be investigatedin future work.
• Consideration of arbitrary PSFs
• Calibration of intrinsic parameters of camera ( f , v, ρ and γ
shown in Section 3.1)

[1] R. T. Azuma: “A Survey of Augmented Reality,” Presence: Teleoperators and Virtual Environments, Vol. 6, No. 4, pp. 355–385, 1997.
[2] H. Regenbrecht, G. Baratoff and W. Wilke: “Augmented Reality
Projects in the Automotive and Aerospace Industries,” IEEE Computer Graphics and Applications, Vol. 25, No. 6, pp. 48–56, 2005.
[3] P. Debevec: “Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-based Graphics with Global Illuminationand High Dynamic Range Photography,” Proc. of SIGGRAPH ’98,
pp. 189–198, 1998.
[4] J. Unger, A. Wenger, T. Hawkins, A. Gardner and P. Debevec: “Capturing and Rendering with Incident Light Fields,” Proc. of 14th Eurographics workshop on Rendering, pp. 141–149, 2003.
[5] M. Kanbara and N. Yokoya: “Real-time Estimation of Light Source
Environment for Photorealistic Augmented Reality.,” Proc. of 17th
IAPR Int. Conf. on Pattern Recognition (ICPR2004), pp. 911–914,
2004.
[6] J. Fischer, D. Bartz and W. Strasser: “Stylized Augmented Reality for
Improved Immersion,” Proc. of IEEE Virtual Reality 2005 (VR’05),
pp. 195–202, 2005.
[7] M. Haller, F. Landerl and M. Billinghurst: “A Loose and Sketchy Approach in a Mediated Reality Environment,” GRAPHITE ’05: Proc.
of 3rd Int. Conf. on Computer Graphics and Interactive Techniques inAustralasia and South East Asia, pp. 371–379, 2005.
[8] J. Flusser and T. Suk: “Degraded Image Analysis: An Invariant Approach,” IEEE Trans. Pattern Anal. Mach. Intell., Vol. 20, No. 6,
pp. 590–603, 1998.
[9] C. Kolb, D. Mitchell and P. Hanrahan: “A Realistic Camera Model for
Computer Graphics,” Proc. of SIGGRAPH ’95, pp. 317–324, 1995.
[10] N. Asada and M. Baba: “A Unified Camera Model of Zoom, Focus
and Iris Parameters for Camera-Calibrated Computer Graphics,” Proc.
of 6th Int. Conf. on Computer Graphics and Imaging, pp. 101–106,
2000.
[11] E. H. B. Smith: “Scanner Parameter Estimation Using Bilevel Scansof Star Charts,” Proc. of 6th Int. Conf. on Document Analysis andRecognition (ICDAR 2001), pp. 1164–1168, 2001.
[12] A. N. Rajagopalan and S. Chaudhuri: “An MRF Model-Based Approach to Simultaneous Recovery of Depth and Restoration from Defocused Images,” IEEE Trans. Pattern Anal. Mach. Intell., Vol. 21,
No. 7, pp. 577–589, 1999.
[13] N. Asada and M. Baba: “A Thin Lens Based Camera Model for Depth
Estimation from Defocus and Translation by Zooming,” Proc. of 15th
Int. Conf. on Vision Interface, pp. 274–281, 2002.

(i) Focused on a near object.

(ii) Focused on a marker.

(iii) Focused on a far object.
(a) Rendered without blur estimation.

(b) Rendered with blur estimation.

Figure 9: Results of merging a virtual object into a real scene image with defocusing. (Left column: rendering without blur estimation, rightcolumn: rendering with blur estimation. top row: focused on a near object, center row: focused on a marker, bottom row: focused on a far
object.)[14] P. Favaro and S. Soatto: “A Geometric Approach to Shape from Defocus,” IEEE Trans. Pattern Anal. Mach. Intell., Vol. 27, No. 3, pp. 406–
417, 2005.
[15] M. Irani and S. Peleg: “Improving Resolution by Image Registration,”
CVGIP: Graphical Models and Image Processing, Vol. 53, No. 3,
pp. 231–239, 1991.
[16] S. C. Park, M. K. Park and M. G. Kang: “Super-Resolution Image Reconstruction: A Technical Overview,” IEEE Signal Processing Magazine, Vol. 20, No. 3, pp. 21–36, 2003.
[17] I. Begin and F. P. Ferrie: “Blind Super-Resolution Using a LearningBased Approach,” Proc. of 17th IAPR Int. Conf. on Pattern Recognition (ICPR2004), pp. 85–89, 2004.

[18] E. Nakamae, K. Harada and T. Ishizaki: “A Montage Method: theOverlaying of the Computer Generated Images onto a BackgroundPhotograph,” Computer Graphics, pp. 207–214, 1986.
[19] H. Kato, M. Billinghurst, I. Poupyrev, K. Imamoto and K. Tachibana:“Virtual Object Manipulation on a Table-top AR Environment,” In
Proc. of IEEE/ACM Int. Symp. on Augmented Reality, pp. 111–119,
2000.
[20] A. P. Pentland: “A New Sense for Depth of Field,” IEEE Trans. PatternAnal. Mach. Intell., Vol. 9, No. 4, pp. 523–531, 1987.
[21] B. K. P. Horn: Robot Vision, MIT Press, 1986.