JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007

1

Diminished Reality Based on Image Inpainting
Considering Background Geometry
Norihiko Kawai, Member, IEEE, Tomokazu Sato, Member, IEEE, and Naokazu Yokoya, Member, IEEE
Abstract—Diminished reality aims to remove real objects from video images and fill in the missing regions with plausible
background textures in real time. Most conventional methods based on image inpainting achieve diminished reality by assumingthat the background around a target object is almost planar. This paper proposes a new diminished reality method that considers
background geometries with less constraints than the conventional ones. In this study, we approximate the background geometryby combining local planes, and improve the quality of image inpainting by correcting the perspective distortion of texture andlimiting the search area for finding similar textures as exemplars. The temporal coherence of texture is preserved using thegeometries and camera pose estimated by visual-SLAM (Simultaneous Localization and Mapping). The mask region that
includes a target object is robustly set in each frame by projecting a 3D region, rather than tracking the object in 2D image
space. The effectiveness of the proposed method is successfully demonstrated using several experimental environments.
Index Terms—Diminished reality, Object removal, Image inpainting, Video inpainting, Real-time✦

1

I NTRODUCTIONDIMINISHED reality, which visually removes real
objects from video images by filling in the missing regions with background textures in real time,
can be used for various applications. For example,
some pieces of furniture may be removed to simulatedifferent arrangements (Fig. 1(a)), signboards can beremoved for landscape simulations (Fig. 1(b)), andaugmented reality (AR) markers can be hidden toachieve seamless fusion between virtual objects andthe real world [1], [2], [3]. Diminished reality methods can be classified into two categories: One uses
actual background images, and the other generates aplausible background image from information aroundthe target object.
The former methods [4], [5], [6], [7], [8], [9], [10] remove real objects by using actual background scenes.
In these methods, there are several ways to capture
the actual background. Cosco et al. [4] captured thebackground scene beforehand, Li et al. [8] collected
photos from the internet, and Lepetit et al. [7] used
past frames in which the background was captured.
Other methods [5], [6], [9], [10] have employed multiple cameras. These approaches can be used for applications that need to show actual background images.
For scenes in which the actual background of a target object cannot be observed, or for cases where it isburdensome for users to capture the background, it isnecessary to generate plausible background textures.

• N. Kawai is with the Graduate School of Information Science, Nara
Institute of Science and Technology, Ikoma, Nara, Japan, 630-0192.
E-mail: norihi-k@is.naist.jp• T. Sato and N. Yokoya are with Nara Institute of Science andTechnology.

(a) Furniture removal(b) Signboard removal
Fig. 1. Example applications of diminished reality.
Images on the left are inputs, and those on the rightare our results.

For this task, image inpainting techniques, which
generate plausible textures using information fromaround the target objects in a static image, haveoften been used [1], [2], [3], [11], [12]. In the image

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007

inpainting-based methods, most use the assumptionthat the background geometry is planar, and the
motion of a camera is constrained so that the wholetarget object exists in a frame. The inpainted results
are not always comparable to those by the-state-of-art
image inpainting methods.
For these problems, we propose an image
inpainting-based diminished reality method, which isan extension of [13], assuming that target objects are
fixed in the 3D environment and no dynamic objects
exist in front of or behind those. The proposed
method uses camera poses and geometry in thescene estimated by Visual-SLAM (Simultaneous
Localization and Mapping), which enables to move acamera with comparatively free motion and generate
high-quality inpainted results. Especially, we pay
attention to three issues to achieve high-quality
diminished reality: (1) the temporal coherenceof textures, (2) the quality of image inpainting,and (3) the determination of mask regions inwhich foreground textures are to be replaced withbackground ones. For these issues, we employ thefollowing approaches: (1) The temporal coherence for
inpainted textures is preserved using visual-SLAMin conjunction with a background geometry thatis approximated from multiple local planes. (2) We
employ a semi-dynamic approach which can adopt
a high-quality image inpainting method such as one
that considers geometric and photometric extensionsof texture patterns but usually takes several seconds.
The quality of image inpainting is also improved
based on the generation of multiple rectified images
and a truncated search region. (3) Mask regions arerobustly determined in each frame based on 3D
tracking of the region of interest.
The rest of this paper is organized as follows: The
next section introduces related work and our contribution. Sections 3 and 4 describe the proposed method.
In Section 5, we show and discuss experimental results. We conclude this paper in Section 6.

2

R ELATED WORKIn this section, we review some related work in consideration of the three issues described above anddescribe our contribution.
2.1

Temporal coherence of textures

The method in [11] attempts to reduce texture flickering between frames by propagating patch correspondences in image inpainting from frame to frame.
However, it is insufficient to achieve geometric consistency between frames taken with large camera
motion. To overcome this problem, Herling et al. [12]improved their original method [11] by employing
a homography, and thus determined the search areas in the next frame by assuming the backgroundaround the target object to be almost planar. Since the2methods in [1], [3] assume that a target object is an
AR marker and placed on a plane, they also used ahomography calculated from the marker to generatea background image or synthesize an inpainted result
on an AR marker. These methods successfully preserve the temporal coherence of planar scenes.
However, if the background of the target scene
is not planar, and the scene is observed from quite
different viewpoints, as shown in Fig. 1(a), changes
in the appearance of textures on different geometries
cannot be compensated using a single homography.
This results in the unnatural motion of inpainted
textures. In the proposed method, we approximate thebackground geometry of the target objects by combining the local planes. For this, the scene around thetarget object is divided into multiple planes, whose
number is automatically determined, and inpainted
textures are successfully overlaid on the target object
by considering the estimated planes and camera-pose
given by visual-SLAM.
2.2

Quality of image inpainting

Siltanen [1] mixed several specific pixel values in amarker-hiding application. Although this method canrapidly generate textures, it is difficult to generate
natural and complex textures using such a simple approach. To synthesize more natural textures for diminished reality, Herling et al. [11] applied an examplebased image inpainting method [14] and a high-speed
approximate nearest neighbor search [15] with use ofgrayscale and reduction of resolution. Moreover, Herling et al. [12] have improved their energy function
by considering spatial costs, enhancing the qualityof inpainting while preserving real-time processing.
However, geometric and photometric expansions of
exemplar texture patterns, which are considered in
the recent image inpainting methods [16], [17] todeal with flipping and rotating patterns with different
brightness, are difficult to be employed due to thecomputational cost. Therefore, results inpainted by
the methods in [11], [12] do not always comparable
to those by such recent image inpainting methods.
In addition, although the methods in [11], [12] apply
image inpainting to the original appearance of the
input image, they often produce unnatural results
especially when an image’s regular patterns have a
distorted perspective and the amount of perspective
distortion is large.
To overcome the problem of computational costwhen using recent image inpainting methods, weemploy a semi-dynamic approach, which conducts
two processes concurrently, as in methods [2], [3]:image inpainting for a key frame, and the overlay of
the inpainted texture for each frame. In this approach,though target objects are hidden with incomplete
textures until the image inpainting finishes, advanced
image inpainting methods can be applied.

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007

To solve the problem of the image quality influenced by perspective distortion, some researchers including us have tried to remove the influence. Our
previous marker-hiding method [3] corrected the perspective distortion using an AR marker, meaning thatthe size of regular texture patterns could be unified
based on the assumption that an AR marker exists on
a plane. In the proposed method, we extend this idea
for more general scenes. In the field of image inpainting for a static image, some methods are proposed for
perspective correction [18], [19]. These methods successfully correct perspective distortions by manuallygiving some interactions. However, such interactions
give the user burdens and take some time before starting diminished reality in our situation. Therefore, we
use a 3D geometry to deal with perspective correction.
Specifically, we generate multiple rectified images,
one for each of the estimated planes. In addition tothis, we add a constraint to automatically limit the
search region using geometries around a target object,thus increasing the quality of inpainted textures, while
the whole input image is searched for similar texture
patterns in the conventional methods.
2.3 Determination of mask regions
Mask regions (those that include target objects) haveto be found in each frame to ensure that the objects are removed from the image. Objects such as
AR markers [1], [2], [3] can easily be tracked using
software libraries (e.g., ARToolkit [20]), allowing themask regions to be determined in real time. In othercases, various approaches are used to track the target
objects and find the mask regions. For example, anactive contour algorithm has been applied to detectand track objects [11], but this method is not robust for
textured backgrounds. For this problem, Herling et al.
[12] use their original feature points that store the appearance of the image, and distribute them around thetarget objects. Tracking the feature points segments
the image into the mask and other regions in eachframe. Although this method works well for scenes
with textured backgrounds, it has the limitation thatthe entire object must always be in the video frame. In
the previous version of our research [13], we did not
update mask regions through all frames. Therefore,target objects have to exist almost on the backgroundobject so that the target objects can be in the mask
regions.
In the proposed method, we robustly determine themask regions in all frames by tracking the 3D region
including target objects in 3D space that are generated
in a key frame. Projecting the 3D region from thecamera position onto the estimated planes sets mask
regions in each frame of the rectified images, andensures that the target objects do not exit the mask
regions. In addition, the target objects do not always
have to be in the video frame, unlike in conventional
methods [11], [12].

3

However, since the initial 3D region is generated
only from the key frame, the projected mask regions
often become large when the target objects are distantfrom the background object and the camera largely
moves from the key frame position, which reduces theframe rate. For such a case, our method also provides
a user an option to crop the 3D region, which works
with re-inpainting the image with a smaller missing
region.
2.4

Contribution

In summary, our primary contributions to the field ofdiminished reality are: (1) the preservation of temporal coherence for inpainted textures in more generalscenes than conventional methods can handle, (2)improved quality of image inpainting, and (3) robust
determination of mask regions in each frame. The
first point uses visual-SLAM in conjunction with abackground geometry that is approximated from multiple local planes. The second relies on the generationof multiple rectified images and a truncated searchregion, the employment of a semi-dynamic approachwith a high-quality image inpainting method, and the
update of inpainted results when the actual background can be observed. The third contribution isbased on 3D tracking of the region of interest.

3 D IMINISHED R EALITYBACKGROUND G EOMETRYC ONSIDERING

Fig. 2 shows the pipeline of the proposed diminishedreality technique. Our method first analyzes the target
scene and carries out the pre-processing of diminished reality (I). Diminished reality is then achievedby a semi-dynamic approach that conducts two processes concurrently, as in our previous marker-hiding
method [3]: example-based image inpainting for a key
frame (II), and the overlay of the inpainted texture
for each frame (III). Although process (II) is not
performed in real-time, users can start applications
immediately by performing processes (II) and (III)concurrently. Within several seconds of starting process (II), users can experience diminished reality witha completely inpainted result.
In the proposed framework, the background geometry is required to have a multiple-plane representation, and the background must have some textures
to enable the extraction of planes and estimation ofcamera poses by visual-SLAM. Therefore, the target
scene is a little restricted. However, the proposed
method is applicable in many real environments, because artificial and natural geometries, such as artificial tiles, garden trees, and grass ground, often consistof multiple local planes and have some texture. In thefollowing, we describe processes (I), (II), and (III) in
detail.

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007

4(I) Scene analysis(I-1) Initialize camera pose and feature points(I-2) Select target region(I-3) Estimate multiple planes(I-4) Segment image and determine 3D target region(a) Enclosing(I-5) Set rectification parameter and rectify image(b) Selected region(I-6) Limit image region
Start Diminished Reality(II) Image inpainting(III) Background overlay
for a key frame(II-3) Update
pixel values
Iterate until convergence(III-2) Estimate camera pose
(III-3) Rectify image(III-4) Set mask region

Next frame(II-2) Search forsimilar texture(III-1) Input Image

Inpainted results(II-1) Giveinitial values(III-5) Adjust luminance of
texture in key frame(III-6) Overlay(III-7) Combination of
overlaid rectified images

Fig. 2. Pipeline of the proposed diminished reality
technique.

3.1(c) Delaunay triangula- (d) Normal vectors oftion
features

Scene Analysis

During pre-processing for the proposed diminishedreality method, the target scene is analyzed and theimage is divided into multiple images to improve thequality of image inpainting. Specifically, the camera
pose and 3D coordinates of feature points are first estimated by initializing visual-SLAM (I-1). A user thenmanually selects a region that includes target objects
by enclosing the region, as shown in Figs. 3(a) and (b)(I-2). In this step, pixels on the line are periodicallyand sequentially stored while a user drags the line,
and the pixels and lines connecting them are stored as
the boundary of the target region. The frame when theuser finishes enclosing the region is set as a key frame
and is used for image inpainting in process (II). The
selected 2D region is converted to a 3D target region,and this is used to set the mask regions in each framein process (III). The conversion is performed in thelater step of process (I), described later in this section.
Next, feature points that are outside the target region and whose chessboard distance from the boundary of the target region is less than a threshold,which is empirically determined, are picked up, andneighboring feature points are connected by Delaunay
triangulation [21] from their 2D image coordinates, as
shown in Fig. 3(c). Normal vectors of the generated
triangles are calculated using the 3D coordinates of(e) Labeledpointsfeature(f) Segmented image

Fig. 3. Scene analysis.
feature points, and normal vectors of feature points
are determined by averaging the normal vectors of
all triangles including the feature points, as shownin Fig. 3(d). Next, each feature point is classified
into multiple groups based on the difference between
its normal vector and the mean normal vector of
all feature points in a group, which is iteratively
updated using mean-shift [22]. It should be noted thatthe number of groups is automatically determined.
Specifically, a feature point is first selected at random
as xi (i is the index of feature points), and the normal
vector m is calculated in an iterative manner as:
ΣN
j=1 w(nj )nj,
mt (ni ) =M
1 (nj · mt−1 (ni ) > C)w(nj ) =0 (otherwise) ,

(1)
(2)

where ni and nj indicate the unit normal vectors offeature points xi and xj . mt (ni ) denotes the unit mean
vector in the t-th iteration, and the process starts withm0 (ni ) = ni . M is some normalization factor suchthat mt (ni ) becomes a unit vector. N is the number offeature points picked up in the above process, and C isa constant threshold. After several iterations, featurepoints nj satisfying nj · mt (ni ) > C are labeled as
belonging to the same group, and these points are
removed. Among those that remain, a new featurepoint is randomly selected, and the above process is

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007

5

Plane2

pixel(group2)
CameraVisibleplane

Invisible
plane

pixel(group1)
Visibleplane

Rectification

Plane1

Invisible
plane

Limiting searching region

Fig. 4. Pixel grouping.
Estimated plane
Enclosed line
Nearest
Optical centerfeature pointof camera
3D target
Image
regionplane
Estimated plane
Target object

3D target
regionInpainting

Target object

Fig. 6. Rectification and region limitation for inpainting.
Fig. 5. 3D target region.
repeated until all feature points are labeled, as shownin Fig. 3(e). Next, a plane is fitted to the feature pointsof each group using LMeds (Least Median of Squares)[23]. The number of planes K is basically the same asthat of labeled groups, but if the number of featurepoints in a particular group is much smaller than inothers, the group is removed.
All the fitted planes are projected onto the image
plane, and each pixel is grouped according to whichplane is visible at the pixel of the camera image asshown in Fig. 4. According to the pixel grouping,
the whole image, including the missing region, issegmented as shown in Fig. 3(f). In addition, the3D target region is generated from the 2D selected
region, so the 3D region must include the target object.
Specifically, as shown in Fig. 5, the feature point on
the target object that is nearest the camera positionis first found. The 3D mesh model generated by thefollowing planes is then set as the 3D target region.
• Estimated plane for background region.
• Plane that includes the nearest feature point andis parallel to the image plane of the camera.
• Planes between any pair of two rays that passthrough the optical center of the camera andadjacent two points on the line drawn by the user
around the object to remove (Fig. 3(a)).
Next, the rectified images are generated from the
key frame, and the information for rectifying thesubsequent frames are stored at the same time (I5). As shown in Fig. 6, the perspective distortion of
input images is corrected by calculating homographymatrices for K planes as if each plane was captured
by a camera in front of it. The number of rectified
images is the same as the number of planes. We now

describe how to calculate a homography matrix Hkfor plane k.
Here, we associate a 3D length, two directions, anda position in the scene with a pixel length, horizontal and vertical axes, and a center of the rectified
image. Specifically, we calculate the 3D length lk
corresponding to a single pixel at the center of gravity
gk of the feature points on plane k by projecting apixel corresponding to the gravity center and one of
its adjacent pixels onto plane k. We then use two
arbitrary perpendicular vectors of length λlk on planek as basis vectors (uk , vk ) for the x and y axes of therectified image, where λ is a constant that influences
the resolution of the rectified image. The larger thevalue of λ, the lower the resolution of the rectified
image. The center of gravity is set as the center of therectified image.
Finally, Hk is calculated from the four pairs of
pixels determined by projecting the four 3D points(gk , gk + uk , gk + vk , gk + uk + vk ) on plane k to therectified image and the key frame’s input image plane.
Homography matrices are calculated for all the planesin the same way. The 3D coordinates of the four 3D
points on each plane k and the 2D coordinates of thecorresponding four pixels in each rectified image arestored and later used for rectifying the subsequent
frames in process (III).
In the final step (I-6), we limit the search region inwhich textures can be used as exemplars for inpainting in process (II) based on the segmented image.
Thus, textures from other planes cannot be used as
exemplars.
3.2

Image Inpainting for Multiple Rectified Images

We apply an example-based image inpainting method
to each rectified image in which a region is limited.

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007

Camera ofm-th frame

Estimated
plane

Input

Camera ofn-th frame
3D target
regionMask region ofm-th frame

Mask region ofn-th frame

Estimated plane

Fig. 7. Determination of mask region on each rectified
image by projecting the 3D target region to each plane.
The rectified images of the key frame in which missing
regions are inpainted are retained for the backgroundoverlay process (III). For image inpainting, the proposed framework can adopt arbitrary example-based
methods that use global optimization, such as thosein [16], [17], [24] and the inpainting part in [12]. In
Figs. 1 and 6, we employ an existing image inpainting
method [17], which allows for brightness compensation and the symmetric transformation of texture
patterns, and runs at relatively high speed.
Specifically in process (II), after initializing parameters of the missing regions, e.g., the average valueof boundary pixels, the inpainting method iterates
two processes in order to minimize an energy function based on the similarity between missing regions
and the remainder of the image. The first processsearches for similar patterns (II-2), and the second
updates pixel values in the missing regions (II-3) ina similar way to [17]. In each iteration, process (II)stores the tentative inpainted result in the memory
shared with process (III). After the energy converges,
the completely inpainted result is stored and used inprocess (III).
3.3

6

Real-time Overlay of Inpainted Textures

In process (III), after capturing an image (III-1) andcalculating a camera pose using visual-SLAM (III-2),
a rectified image is generated for each plane using thecurrent camera pose and information for rectification,
which is stored in step (I-5), (III-3). On each rectified
image, a mask region is then determined by projecting
the 3D target region from the optical center of thecurrent frame’s camera onto each plane, as shownin Fig. 7 (III-4). The right-hand images of Fig. 8
show an example of the 3D target region and the
projected mask regions. It should be noted that thetarget objects always exist within the projected mask
regions, regardless of the camera pose and projectedmask shapes, because the 3D target region includes
the target objects.
Next, the mask regions of rectified images are filled
in using the texture in the rectified images of the

Input withtarget region(1st and 3rdcolumn)Rectifiedimage 1
Rectifiedimage 2

Outsideof image

Rectifiedimage 3
Key frame

Key frame Current frame
withinpainted texture

Fig. 8.
Divided grids for calculating luminancechanges.
key frame in which the object regions are inpainted.
Because there is usually some difference in the luminance of the key frame and the current frame, we
adjust the luminance of the key frame’s texture (III5) before overlaying this texture on the mask regions(III-6). In this study, we use a similar approach as inour previous marker-hiding method [3], whereby weestimate luminance changes in the mask region fromthe changes in the surrounding region using rectified
images between the key frame and the current frame.
We do not directly use the amplitude of luminancechanges in the target object, but instead estimate thisfrom the surrounding region, because the photometricproperties of the object and the surrounding materials
are different.
The difference from our previous marker-hiding
method [3] is that the luminance change of each
rectified image is calculated separately, because thedegree of luminance change on each plane is thought
to differ according to the plane’s normal direction. In
addition, the number of pixels in the mask regions
varies in each frame because of the projection of the
3D target region. In the following, we describe themethod for calculating the luminance change coefficients by which pixel values in the rectified images of
the key frame are multiplied when they are overlaid
on the mask regions in the rectified images of thecurrent frame.
As shown in Fig. 8, we first divide the rectified
images of the key frame with inpainted texture and acurrent frame into grids of s × s pixels. For efficient
processing, the grid-wise luminance change coefficients are calculated before the pixel-wise coefficientsare determined. We then calculate luminance change
coefficients of the grid according to the following two

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007

conditions.
• The grid does not include the mask region or
areas outside the image in the current frame.
• The adjacent grid includes the mask region in thecurrent frame.
The luminance change coefficient is calculated by
dividing the sum of pixel values in the grid for thecurrent frame by that for the key frame. Next, thecoefficients of grids that include the mask region in
the current frame are estimated using the coefficients
calculated for surrounding grids. In this study, we
assume that adjacent grids have small luminancechanges because of ambient illumination changes andsoft shadows caused by users, and therefore the luminance change coefficient of each grid is calculated so
as to minimize the following cost function E:(αˆp − αˆq )2 ,(3)E=(p,q)∈R

where α̂p and α̂q are luminance change coefficientsof grids p and q, and R is the set of pairs of gridindexes from the eight neighboring grids. It should
be noted that some surrounding grids extend outsideof the image because of camera motion, as shown inrectified image 2 of the current frame in Fig. 8. In
such cases, the coefficients of the grids including thisoutside area are not included in Eq. (3). The luminancechange coefficients that minimize E are analyticallycalculated by differentiating E with respect to eachluminance ratio and solving the system of equations.
After calculating the luminance change coefficientfor each grid, the luminance change coefficient α̂pof grid p is copied to pixel-wise luminance change
coefficient αj of the central pixel j in grid p, andthe pixel-wise luminance change coefficients for theremaining pixels, excluding the central pixels in thegrids, in the mask region are calculated by bilinear
interpolation. In this process, the coefficients for RGB
are independently calculated and multiplied in orderto deal with chromatic changes.
Finally, the texture of each rectified image of the
key frame is overlaid on the mask region of each
rectified image of the current frame. The rectified
images are transformed to the original appearance ofthe current frame using homographies, and these arethen combined to produce the final output (III-6).

4

O PTIONAL U PDATE OF 3D TARGET RE GION AND INPAINTED IMAGES
When a target object is distant from the backgroundobjects, the projected mask region is larger, as thecamera position is farther from the key frame position
as shown in Fig. 9. Such a situation reduces the frame
rate because the number of parameters in Eq. (3)increases.
For such a case, our method provides users an
option to crop a part of the 3D target region with

7

Key frame

A subsequent frameFig. 9. Inputs and mask regions varying according tocamera poses.

a user interaction based on the idea of Shape-fromSilhouette to reduce the projected mask regions. This
process is also accompanied by re-inpainting the key
frame with the additional exemplar textures, which
are newly visible background, and a smaller missing
region, which may yield better results than the firstinpainting.
Specifically, a user first switches the display mode
from showing the target object-less image to showing
a translucent mask, and crop the 3D target region by
drawing a curve as shown in Fig. 10(a). For cropping
the 3D region, our method calculates the intersection
positions of the drawn curve with lines on the image
plane projected from 3D ones connecting the front
polygon and the background plane, and moves thepoints on the background plane to 3D positions corresponding to the intersections as shown in Fig. 10(b).
Next, the image regions corresponding to the removed
3D region are added to the missing regions in the key
frame images with Poisson blending [25] as shownin Figs. 10(c) and (d), because the intensities of thecurrent and key frames may differ. After that, image
inpainting is again applied to the key frame images
that still have missing regions.

5

E XPERIMENTS

We carried out experiments in several environments
using a PC with Windows 7, Core i7-3820QM 2.7 GHz
CPU, 8 GB of memory, and a GeForce GT 650M GPU
for input images of resolution 640 × 480 captured bya USB camera (Logicool Qcam Pro 9000). We used
the GPU-based function provided by OpenCV 2.4.5
with CUDA support for image rectification in process(III). We used PTAM [26] for the visual-SLAM andinpainting methods [17]. We set λ = 1.5 for the rectified image resolution, and set s = 20 for luminanceadjustment by considering the trade-off of quality andprocessing time. We set the segmentation threshold as
C = cos(π/6).

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007(a) Switching display mode and drawing a curve(b) Updating 3D target region(c) Rectified images with missing regions
for first inpainting(d) Rectified images with missing regions forsecond inpainting after adding visible backgroundFig. 10. Cropping of 3D target region and updating ofrectified key frame images.

5.1 Comparison of Proposed and Conventional
Methods
We compare the results given by the proposed methodwith those from a conventional approach using asingle homography with respect to the quality ofimage inpainting and temporal coherence of textures
for two scenes.
First, we demonstrate the effectiveness of image
rectification and search region limitation in terms of
the quality of image inpainting. Fig. 11 compares the
inpainted results. Figs. 11(f) show the result obtainedby applying the inpainting method [17] to the original
input image in Figs. 11(b). For scene 1, as we can see,unnatural textures were generated in the target region
due to the perspective distortion and the complexityof textures. For scene 2, plausible textures were generated for the garden plants, but slightly blurry texture
were generated on the ground due to the perspective
distortion. On the other hand, as shown in Figs. 11(c)and (d), the proposed method successfully segmented
the images by finding three and two planes for scenes
1 and 2, respectively, and gave natural textures to the8target regions in the rectified images, which have various regular and random textures. By combining the
inpainted rectified images, natural textures could be
generated in the target region, as shown in Figs. 11(e).
Next, we consider the temporal coherence of
textures by comparing results using the proposed
method with those from a conventional approach
using a single homography [12], [3] using the same
sequences as Fig. 11. In the conventional approach,
we fitted one plane to the feature points near amask region using the least-squares method, and the
inpainted result generated by the proposed method(Fig. 11(e)) was geometrically transformed using theplane for each frame. To focus on the temporal coherence, we used the same inpainted result. Fig. 12
compares the output. As shown in the bottom imagesof Fig. 12, camera motion causes edges to become
disconnected on the boundaries of the target regions
because of different geometries between the target andsurrounding regions. In contrast, as shown in the middle images of Fig. 12, the temporal coherence of the
inpainted textures is preserved under various camera
movements. Conventional methods [12], [3] have thelimitation that the whole of a target object must
exist in an image for each frame to continue visually
removing the object. However, the proposed method
can successfully overlay natural textures on the mask
region using the extracted planes and camera pose
estimated by visual-SLAM, even if the camera motion
causes the target object to leave the image, as in thethird and sixth column images of Fig. 12.
5.2

Results for Various Scenes

In this section, we report the results of experiments
for additional two scenes under different conditions.
In Figs. 13 and 14, Fig. (a) shows the key frame, withthe top row showing the input image, the middle row
showing the segmented mask region, and the bottom
row showing the inpainted results of rectified images.
Figs. (b) to (e) show subsequent frames captured fromvarious viewpoints; the top row shows input images,
the middle row shows the mask regions, and thebottom row shows output images.
First, we discuss the experiment for the scene in
Fig. 13, in which the optical parameter of the camera
automatically changes with camera motion becauseof the large difference in luminance between sunnyand shady areas and the low dynamic range of thecamera. In this scene, the mask region of the key frame
is inpainted when the camera’s optical parameter
adjusts to a shady area, as shown in Fig. (a). The
optical parameter is adjusted according to this shadyarea in Figs. (b) and (d), and to sunny areas in Figs. (c)and (e). From the results, we can confirm that the
inpainted textures in the target region are successfully
adjusted by the luminance adjustment of the proposed
method.

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007

(a)

(b)

9

(a)

(b)

(c)

(c)

(d)

(d)

(e)

(f)
Scene 1(e)(f)
Scene 2

Fig. 11. Comparison of inpainted results for two scenes: (a) input image, (b) segmented mask region, (c) threerectified and region-limited images, (d) inpainted results for rectified images, (e) result from the combination ofrectified images, and (f) result of applying the inpainting method to original appearance of input image (b).

Scene 1

Scene 2

Fig. 12. Comparison of results from different viewpoints by the proposed method and the conventional approach
using one homography for two scenes. Top: input images. Middle: results given by the proposed method. Bottom:
results given by the conventional approach.

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007

OutputInpainted
rectified images

Segmented
mask region

Mask region

Input

Input

10

(a) Key frame

(b)

(c)

(d)

(e)

OutputInpainted
rectified images

Segmented
mask region

Mask region

Input

Input

Fig. 13. Experiment for a scene with the camera’s optical parameter changed: (a) key frame, (b) to (e)subsequent frames.

(a) Key frame

(b)

(c)

(d)

(e)

Fig. 14. Experiment for a scene in which the target object is distant from the background objects: (a) key frame,(b) to (e) subsequent frames.

Second, we consider the experiment for the scene
in Fig. 14, in which the target object is distant fromthe background objects. In this scene, we cropped
the 3D target region because the mask region became
large when the camera position moved from the key
frame position, as shown in Fig. (c). As a result, theprojected mask region reduced as shown in Fig. (d).
Comparing the result before cropping in Fig. (b) withthat in Fig. (e) in which the camera moved backnear to the key frame position after cropping, we canconfirm that the inpainted texture was replaced with
the newly visible background in the large area of themask region.
5.3

Parameters and Computational TimeWe now study the influence of the number of planes
and parameters on computational time and quality.
First, we discuss the computational time with different number of planes using three scenes in which
one, two, and three planes are estimated, as shown in
Fig. 15. Table 1 shows the computational time of eachprocess, as well as the frame rate for 2,000 frames withand without GPU after commencing the diminishedreality method. The proposed method implements

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007

11

TABLE 1
Computational time.

Scene(a) 1 plane(b) 2 planes(c) 3 planes
Scene(a) 1 plane(b) 2 planes(c) 3 planes

Scene
Analysis(a-3) to (a-6)
30 msec.
78 msec.
95 msec.
Frame ratewith GPU
32.39 fps
24.80 fps
21.79 fps

Inpainting(Max. No. of missing
pixels in rectified images)
3.30 sec. (8091 px)
4.52 sec. (12108 px)
4.42 sec. (9466 px)
Frame rate
without GPU
27.48 fps
20.48 fps
15.48 fps

TABLE 2Computational time with different parameters.
(a)

(b)

(c)

Fig. 15. Scenes for measuring computational time.
From top to bottom, inputs, rectified images, resultsof scenes with (a) one plane, (b) two planes, (c) threeplanes.

parallel processing to inpaint rectified images andadjust the luminance of inpainted textures in thecase of multiple planes. The time taken for image
inpainting is recorded as the time when the inpaintingof all rectified images is complete. For scene analysis,
the computational time for the three cases is smallenough for a user to wait for diminished reality tostart. For inpainting, the computational time mainly
depends on the size of the missing region, rather
than the number of planes. In terms of frame rate,
the GPU effectively speeds up the processing, but
the processing speed decreases as more planes are
estimated because of sequential rectification for eachplane. Since our implementation employed OpenCV
to use the GPU, the frame rate may be improved withmore sophisticated management of multi-core CPU in
conjunction with GPU.
Next, we discuss the influence of the parameters
λ for the resolution of the rectified image and gridsize s × s for luminance adjustment on the qualityof overlaid texture and computational time. In thisexperiment, we use the environment in which one
plane is estimated, like in Fig. 15(a). First, we show thecomputational time with different λ and s in Table 2.
Because larger λ lowers the resolution of a rectified
image, the inpainting speeds up and the frame rateis increased with larger λ. Because larger s decreases
the number of parameters in Eq. (3), the frame rate isincreased with larger s. Next, we compare the qualityof overlaid textures with different parameters in termsof resolution and luminance. First, we discuss the resolution with different values of λ in Fig. 16. When thecamera comes close to the target object, we can realizethat the resolution of overlaid texture is insufficientParameters
λ = 1.5 s = 5
λ = 3.0 s = 5
λ = 3.0 s = 20

Inpainting(No. of missing pixels)
2.99 sec (7654 px)
1.22 sec. (1905 px)
1.87 sec. (1884 px)

Frame ratewith GPU
29.94 fps
31.46 fps
32.49 fpswith λ = 3.0, as shown in Fig. 16(b), meaning theresolution of the rectified image is too low. Second,Fig. 17 compares the quality of luminance adjustment. In indoor environments, there are often local
luminance changes caused by object occlusion andnear light sources. Especially in such an environment,
the adjustment of luminance of inpainted texture isinsufficient with the large grid size, as shown in theboundary of the missing region in Fig. 17(b). Fromthese results, we have to carefully set the parametersconsidering the supposed environment in conjunction
with considering the trade-off between the computational time and the quality.
5.4

Discussion and Limitations

As described in sections 5.1 and 5.2, the proposed
method works well for the various real environments
as shown in Figs. 12, 13 and 14. This means thatthe various real scenes can be represented by a small
number of planes and image inpainting is effective
for such scenes. However, the proposed method has
theoretical limitations, which were not demonstrated
in the experiments. This section summarizes and discusses the limitations of the proposed method.

5.4.1 SLAM
Since we used PTAM [26] as a visual-SLAM method,
the target scenes must have some textures to enable
the extraction of feature points to calculate camera
poses. For plane detection, each plane must have some
textures for extracting a sufficient number of featurepoints to fit an appropriate plane. If, for example, ascene consists of two planes and one of the planes has
few textures, the proposed method may detect only
one plane and not achieve the temporal coherence.
In addition, SLAM methods such as PTAM usually

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007

(a)

12

Input

Input

Segmented mask region

Mask region

Inpainted
Diminished result
rectified images(a) Key frame

Diminished result(b)Fig. 16. Resolution of overlaid texture with different
values of λ: (a) λ = 1.5 (b) λ = 3.0. Top: output. Bottom:
input with a mask.

(b) Subsequent frameFig. 18. Example of limitations: for a scene witha curved background: (a) key frame, (b) subsequent
frame.

(a)(b)

Fig. 17. Adjustment of luminance with different valuesof s: (a) s = 5 (b) s = 20. Top: output. Bottom: input
with a mask.
require a user a specific start to initialize the camera
pose, which may give some burden for the user.

5.4.2 Background geometry
Since the proposed method assumes that the background geometry is approximated by multiple planes,
it cannot handle the scenes that do not satisfy theassumption such as a curved background geometry
and no background objects with which we can see
a horizontal line. For example, for a scene with acurved background geometry as shown in Fig. 18,
while the background is divided into multiple planes,
despite its curved shape, and the missing region ofeach rectified image is plausibly inpainted by the
proposed method, disconnected texture appeared on
the boundary of the mask region when the camera

moves in a subsequent frame. In addition, it cannot
handle individual but parallel planes because ourclustering of feature points uses only normal vectors
and feature points on such planes are grouped into
the same cluster. In such a case, only a single plane
is fitted to the feature points despite the existence ofmultiple planes, and the inappropriate plane fittingcases texture disconnection according to camera motion.

5.4.3 Quality of overlaid textures
The quality of overlaid texture in each frame largely
depends on the inpainted result for a key frame.
Therefore, the key frame should include sufficiently
large amount of sample textures with high resolution
and good focus to obtain good results by inpainting.
In addition, the geometry should be correctly estimated to correct perspective distortion and preserve
temporal coherence of textures. However, even if therequirement is satisfied, the quality of the output
frame is often limited because of the following reasons:• A part of the mask region is not covered by the
background texture because the partial area iscorresponding to a region outside the rectified
image of the key frame when the camera largely
moves.
• The overlaid image blurs when the camera moves
close to the background object because the resolution of the inpainted result is fixed.
• The overlaid image cannot represent the strong
cast shadows and spotlights when they come on

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007

the mask region because our method forces theluminance change coefficients for adjacent grids
to be similar.
• The overlaid textures with high quality may
cause visual inconsistency when the quality of
the captured frame is deteriorated by some noisesuch as motion blurs due to camera rapid motion.

5.4.4 Selection of target objects
Drawing a contour around a target object as precisely
as possible is desirable because a small target region
tends to enable the better inpainted result. Supposethat the proposed method runs on smartphones andtablet PC, it may be difficult for a user to draw
it precisely. Therefore, we should consider such an
approach as [12] in which the contour around thetarget object shrinks down to the boundary of it afteruser’s rough selection.

6

C ONCLUSION

We have proposed a new diminished reality method
for more general scenes than conventional methods.
By approximating the background geometry using a
combination of local planes, the perspective distortionof texture is corrected and the search area is limited,thus improving the quality of image inpainting. The
temporal coherence of texture is also preserved using
the geometries and camera pose estimated by visualSLAM. Mask regions are robustly set in each frame
using a 3D region, instead of tracking target objects
in 2D image space, and the 3D region is interactively
cropped and simultaneously inpainted results are updated with the additional visible background.
In the experiments, we compared the results givenby the proposed method with those from a conventional approach using a single homography. This confirmed that the proposed method gives more natural
results with comparatively unrestricted camera motion. In addition, we demonstrated the effectiveness ofthe proposed method for various scenes. However, theproposed method has the limitations that the camera
movement is still restricted by various backgroundgeometries and the management of a key frame, andthe quality of diminished reality largely depends on
the robustness of SLAM.
To address the former problem, we must deal withadditional geometries and extend the key frame using
subsequent frames. As for the robustness of SLAM,
we can employ a different method according to thescene. For example, in indoor scenes, we could apply an RGBD camera-based SLAM method, such as
KinectFusion [27], as these robustly estimate background geometries even for non-textured regions.

ACKNOWLEDGMENTS
This work was partially supported by Grants-in-Aid
for Scientific Research Nos. 23240024, 24700118 and13

15K16039 from the Japan Society for the Promotion of
Science (JSPS).

R EFERENCES
S. Siltanen, “Texture generation over the marker area,” in Proc.
Int. Symp. on Mixed and Augmented Reality, 2006, pp. 253–254.
[2] O. Korkalo, M. Aittala, and S. Siltanen, “Light-weight marker
hiding for augmented reality,” in Proc. Int. Symp. on Mixed andAugmented Reality, 2010, pp. 247–248.
[3] N. Kawai, M. Yamasaki, T. Sato, and N. Yokoya, “Diminishedreality for AR marker hiding based on image inpainting
with reflection of luminance changes,” ITE Trans. on Media
Technology and Applications, vol. 1, no. 4, pp. 343–353, 2013.
[4] F. I. Cosco, C. Garre, F. Bruno, M. Muzzupappa, and M. A.
Otaduy, “Augmented touch without visual obtrusion,” in Proc.
Int. Symp. on Mixed and Augmented Reality, 2009, pp. 99–102.
[5] A. Enomoto and H. Saito, “Diminished reality using multiple
handheld cameras,” in Proc. ACCV’07 Workshop on Multidimensional and Multi-view Image Processing, 2007, pp. 130–135.
[6] S. Jarusirisawad, T. Hosokawa, and H. Saito, “Diminishedreality using plane-sweep algorithm with weakly-calibrated
cameras,” Progress in Informatics, no. 7, pp. 11–20, 2010.
[7] V. Lepetit and M.-O. Berger, “An intuitive tool for outlining
objects in video sequences: Applications to augmented anddiminished reality,” in Proc. Int. Symp. on Mixed and Augmented
Reality, 2001, pp. 159–160.
[8] Z. Li, Y. Wang, J. Guo, L.-F. Cheong, and S. Z. Zhou, “Diminished reality using appearance and 3d geometry of internet
photo collections,” in Proc. Int. Symp. on Mixed and Augmented
Reality, 2013, pp. 11–19.
[9] T. Tsuda, H. Yamamoto, Y. Kameda, and Y. Ohta, “Visualization methods for outdoor see-through vision,” in Proc. Int.
Conf. on Artificial Reality and Telexistence, 2005, pp. 62–69.
[10] S. Zokai, J. Esteve, Y. Genc, and N. Navab, “Multiview paraperspective projection model for diminished reality,” in Proc.
Int. Symp. on Mixed and Augmented Reality, 2003, pp. 217–226.
[11] J. Herling and W. Broll, “Advanced self-contained object removal for realizing real-time diminished reality in unconstrained environments,” in Proc. Int. Symp. on Mixed andAugmented Reality, 2010, pp. 207–212.
[12] J. Herling and W. Broll, “High-quality real-time video inpainting with pixmix,” IEEE Trans. Visualization and Computer
Graphics, vol. 20, no. 6, pp. 866–879, 2014.
[13] N. Kawai, T. Sato, and N. Yokoya, “Diminished reality considering background structures,” in Proc. Int. Symp. on Mixed
and Augmented Reality, 2013, pp. 259–260.
[14] D. Simakov, Y. Caspi, E. Shechtman, and M. Irani, “Summarizing visual data using bidirectional similarity,” in Proc. IEEE
Conf. on Computer Vision and Pattern Recognition, 2008, pp. 1–8.
[15] C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman,“PatchMatch: A randomized correspondence algorithm forstructural image editing,” ACM Trans. on Graphics, vol. 28,
no. 3, pp. 1–11, 2009.
[16] S. Darabi, E. Shechtman, C. Barnes, D. B. Goldman, and P. Sen,
“Image melding: Combining inconsistent images using patchbased synthesis,” ACM Trans. on Graphics, vol. 31, no. 4, pp.
82:1–82:10, 2012.
[17] N. Kawai and N. Yokoya, “Image inpainting considering
symmetric patterns,” in Proc. Int. Conf. on Pattern Recognition,
2012, pp. 2744–2747.
[18] D. Pavić, V. Schönefeld, and L. Kobbelt, “Interactive image
completion with perspective correction,” The Visual Computer,vol. 22, no. 9, pp. 671–681, 2006.
[19] J.-B. Huang, J. Kopf, N. Ahuja, and S. B. Kang, “Transformation guided image completion,” in Proc. Int. Conf. on
Computational Photography, 2013, pp. 1–9.
[20] H. Kato and M. Billinghurst, “Marker tracking and hmd
calibration for a video-based augmented reality conferencing
system,” in Proc. Int. Workshop on Augmented Reality, 1999, pp.
85–94.
[21] D. T. Lee and B. J. Schachter, “Two algorithms for constructing
a delaunay triangulation,” Int. Journal of Computer and Information Sciences, vol. 9, no. 3, pp. 219–242, 1980.
[1]

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007[22] K. Fukunaga and L. Hostetler, “The estimation of the gradientof a density function, with applications in pattern recognition,” IEEE Trans. on Information Theory, vol. 21, no. 1, pp. 32–
40, 1975.
[23] D. L. Massart, L. Kaufman, P. Rousseeuw, and A. Leroy,“Least median of squares: A robust method fo outlier andmodel error detection in regression and calibration,” Analytica
Chimica Acta, vol. 187, pp. 171–179, 1986.
[24] Y. Wexler, E. Shechtman, and M. Irani, “Space-time completionof video,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 29, no. 3, pp. 463–476, 2007.
[25] P. Pérez, M. Gangnet, and A. Blake, “Poisson image editing,”
ACM Trans. on Graphics, vol. 22, no. 3, pp. 313–318, 2003.
[26] G. Klein and D. Murray, “Parallel tracking and mapping
for small AR workspaces,” in Proc. Int. Symp. on Mixed andAugmented Reality, 2007, pp. 225–234.
[27] S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe,
P. Kohli, J. Shatton, S. Hodges, D. Freeman, A. Davison, and
A. Fitzgibbon, “Kinectfusion: Real-time dense surface mapping and tracking,” in Proc. Int. Symp. on Mixed and Augmented
Reality, 2011, pp. 127–136.

Norihiko Kawai received his B.E. degree in
informatics and mathematical science from
Kyoto University in 2005. He received his
M.E, and Ph.D. degrees in information science from Nara Institute of Science andTechnology (NAIST) in 2007 and 2010, respectively. He was a research fellow of theJapan Society for the Promotion of Scienceand postdoctoral fellow at the University ofCalifornia at Berkeley in 2010-2011. He has
been an assistant professor at NAIST since
2011.

Tomokazu Sato received his B.E. degree incomputer and system science from Osaka
Prefecture University in 1999. He received
his M.E. and Ph.D. degrees in information
sciences from Nara Institute of Science andTechnology (NAIST) in 2001 and 2003. He
was an assistant professor at NAIST in 20032011. He was a visiting resercher in Czech
Technical University in Prague in 2010-2011.
He has been an associate professor at
NAIST since 2011.

Naokazu Tokoya received his B.E., M.E.,
and Ph.D. degrees in information and computer sciences from Osaka University in
1974, 1976, and 1979, respectively. He
joined Electrotechnical Laboratory (ETL) of
the Ministry of International Trade and Industry in 1979. He was a visiting professor at
McGill University in 1986-1987 and has beena professor at Nara Institute of Science andTechnology (NAIST) since 1992. He has also
been a vice president at NAIST since April
2013.

14