Paper
Diminished Reality for AR Marker Hiding Basedon Image Inpainting with Reflection of Luminance
Changes
Norihiko Kawai † , Masayoshi Yamasaki † , Tomokazu Sato (member)† ,Naokazu Yokoya (member)†
Abstract

This paper proposes a new diminished reality technique which removes AR markers from a user’s view image.

In order to achieve natural marker hiding, three factors should be considered; (1) naturalness of texture generated on amarker area. (2) geometric consistency between consecutive frames, (3) photometric consistency between a marker area andits surrounding. In this study, assuming that an area around a marker is locally planar, the marker area in the first frame
image is inpainted using the rectified image to achieve high-quality inpainting. The unique inpainted texture is overlaid on themarker region in subsequent frames according to camera pose for temporal geometric consistency. Global and local luminancechanges around the marker are reflected to the inpainted texture for photometric consistency.

Key words: Augmented Reality, Diminished Reality, Marker Hiding, Image Inpainting

1. Introduction
With the spread of tablet computers and portable
game devices with cameras, it is becoming popular toexperience applications of video see-through augmented
reality (AR) which overlays virtual objects on video
capturing the real world. In a number of AR applications, marker-based geometric registration is employedbecause of its robustness and easiness 1) . However, suchmarker-based AR applications have the limitation thatseamless fusion between virtual objects and the realworld is not achieved due to the existence of markers. To overcome this problem, diminished reality techniques, which remove real objects from video images byoverlaying background texture onto the object regions,have been proposed. Diminished reality techniques can
be classiﬁed into two categories: One uses actual background images and the other generates plausible background image from information around target objects.
The former methods 2)∼7) generate background texture to be overlaid on markers using actual backgroundscenes. In these methods, there are several ways to capture actual background scenes. Cosco et al. 2) capture
the background scene beforehand. Lepetit et al. 3) use
past frames that captures the background in video seReceived February 25, 2013; Revised May 7, 2013; Accepted June 11,2013† Nara Institute of Science and Technology

quence. Other methods 4)∼7) use multiple cameras. For
the purpose of removing markers in AR applications,the method which captures the background scene beforehand is applicable. However, from a standpoint ofa user, it is burdensome to capture background images
before placing markers and starting an application.
On the other hand, in the latter methods 8) 9) 10)
which generate plausible background image from information around target objects, a user can eﬃciently start
an application without troublesome operations. The
proposed method also pays attention on this approach.
These methods visually remove real objects by applying image inpainting techniques which have been developed to remove undesired objects in a static image.
In this approach, three factors should be considered toachieve natural marker hiding; (1) naturalness of texture generated on a marker area, (2) geometric consistency between consecutive frames, (3) photometric consistency between the marker area and its surroundings.
We review conventional methods while paying attention to these factors and highlight our contribution in
the following.
(1) Naturalness of generated texture
Siltanen 8) generates a texture on a marker area bydetermining each pixel value in the marker area fromseveral pixel values of speciﬁc positions around themarker. This method can generate textures very fast.
However, it is diﬃcult to generate natural and complex(Nara, Japan)

ITE Transactions on Media Technology and Applications Vol. 1, No. xx, pp. 1–11 (2013)(1) 1

(a) Input Image
Fig. 1(b) Rectiﬁed Image

Correction of perspective distortion.

textures by such a simple approach. In order to synthesize more natural textures, Herling et al. 9) have applied exemplar-based image inpainting method 11) withhigh-speed approximate nearest neighbor search 12) todiminished reality. In this method, the whole input
image except a marker area is searched for texture patterns similar to the pattern around the target marker
area, and pixel values in the target area are determinedusing the similar patterns. Although exemplar-based
inpainting methods yield good results in many cases, it
is known that they are weak for perspective distortion
such as shown in Fig. 1(a). To solve this problem, theproposed method inpaints a marker region using theimage in which the perspective distortion is corrected
as shown in Fig. 1(b) based on the assumption that anarea around the marker is locally planar.
(2) Geometric consistency between consecutive
framesSiltanen 8) and Herling et al. 9) generate textures inevery frame. Therefore, they tend to cause unnaturalgeometric changes between frames. Although Herling
et al. 9) attempt to reduce texture ﬂickering between
frames by propagating patch correspondences in image
inpainting, it is insuﬃcient to achieve geometric consistency between frames taken with large camera motion.
On the other hand, Korkalo et al. 10) overlay the unique
texture inpainted in the ﬁrst frame according to camera
pose instead of inpainting every frame. Our proposed
method also employs the same approach to achieve temporal geometric consistency.
(3) Photometric consistency between marker
area and its surroundings
Siltanen 8) and Herling et al. 9) implicitly adjust thecolors of texture according to global luminance changes,
which are caused by changes of ambient illuminationand photometric parameters of camera, as shown in
Fig. 2(a), by using exemplar textures from the current frame image. On the other hand, Korkalo etal. 10) have employed a method that explicitly treats
global luminance changes. This method generates a
2 (2)(a) Global changes
Fig. 2(b) Local changes

Types of luminance changes.

low-resolution texture for the current frame and combine the texture with high-resolution texture generatedusing the ﬁrst frame. As far as we know, there has
been no method that explicitly solves the problems ofboth global and local luminance changes, which arecaused by cast shadow, like in Figs. 2(a) and (b). In
the proposed method, both global and local luminancechanges around the marker are explicitly detected andthe changes are reﬂected to the inpainted texture toachieve photometric consistency.
In summary, our solutions for above problems are; (1)correction of perspective distortion in image inpainting
for natural texture generation, (2) use of the unique
inpainted texture in all the frames for temporal geometric consistency, and (3) explicitly considering bothglobal and local luminance changes for photometric consistency. In the proposed method, extrinsic camera parameters are calculated using AR markers, and the positions of markers need to be ﬁxed while hiding markers.

2. Overview of Proposed Method
Fig. 3 illustrates the proposed pipeline. The method
consists of two processes: (A) image inpainting for generating texture for marker areas, and (B) marker hiding
by reﬂecting luminance changes and overlaying the inpainted textures on marker areas for every frame. Because image inpainting process (A) requires several seconds in order to generate a high-quality backgroundtexture, we perform processes (A) and (B) concurrently
not to keep a user waiting. If multiple markers exist ina scene, a set of processes (A) and (B) is independently
and concurrently performed for each marker in order
not to decrease the frame rate.
In this section, we describe processes (A) and (B)brieﬂy. We then describe process (B) in detail in Sec-ITE Transactions on Media Technology and Applications Vol. 1, No. xx (2013)

(A) Image inpainting(a-1) Specifying marker region(a-3) Update of pixel values(a-4) Output of tentative
inpainted result(b-1) Image Input
only (b-2) Correction ofperspective distortionfor 1stframe(b-3) Detection of
luminance changes(b-4) Reflection of
luminance changes

next frame(a-2) Searching forsimilar texture

Frame(B) Marker hiding(b-5) Overlay
Fig. 3

Flow of the proposed method.

Input image

Fig. 4tion 3.
2. 1 Image Inpainting
For process (A), an exemplar-based inpainting
method is applied to the ﬁrst frame image whose perspective distortion is corrected by process (B). The
correction of perspective distortion is eﬀective for
exemplar-based inpainting methods to give better results as shown later in Fig. 13 because that uniﬁes thesize and shape of texture patterns used as exemplars.
As an inpainting method, an arbitrary exemplarbased inpainting method using global optimization suchas Wexler’s method 13) and our previous method 14) isapplicable in the proposed framework. Generally, insuch exepmlar-based image inpainting methods, an energy function is deﬁned based on the similarity between
a missing region and the remainder of image. The missing region is then iteratively restored so as to minimize
the energy function. As the concrete steps of process(A), the marker area in the rectiﬁed ﬁrst frame is speciﬁed as a missing region (a-1). The region is inpainted
by alternately searching for similar patterns (a-2) andupdating pixel values (a-3) until the energy converges.
In this process, the tentative inpainted result is passed
to process (B) at each iteration (a-4).
2. 2 Marker Hiding
In process (B), a scene containing a marker is ﬁrst
captured (b-1). The marker is detected and the input
image is rectiﬁed by calculating homography matrix H
as if the scene is observed with the viewing direction
perpendicular to the marker (b-2). Luminance changes
around the marker are then detected (b-3), and theluminance of the texture generated by process (A) isadjusted according to the changes around the marker(b-4). Finally, the adjusted texture is overlaid onto themarker area in the original input image using inverse
homography matrix H −1 (b-5). In the following section, we describe steps (b-2) to (b-5) in detail.
Paper

Rectified image

Correction of perspective distortion using homography matrix.

3. Marker Hiding with Reﬂection of Luminance Changes
3. 1 Correction of Perspective Distortion
As shown in Fig. 4, perspective distortion of an input image is corrected by converting the image as if thescene is captured by a camera in front of the marker instep (b-2). Concretely, four corner points of a marker
are extracted from an input image, and homographymatrix H is calculated from four pixel pairs between
the extracted corners and four corner points of the ﬁxed
size square through all frames on a rectiﬁed image 15) .
The whole input image is then converted using the homography matrix H.
By correcting the perspective distortion, the size andshape of a texture pattern are uniﬁed. That not only
results in generating high-quality texture by image inpainting in process (A), but also easily determines pixel
correspondences between diﬀerent frames of input images for detecting luminance changes of textures insteps (b-3) and (b-4).
3. 2 Detection and Reﬂection of Luminance
Changes
The luminance of the inpainted marker region generated by process (A) is adjusted by detecting both
local luminance changes (e.g. by cast shadows) andglobal luminance changes (e.g. by ambient illumination changes and camera parameter changes) in orderto preserve photometric consistency with high-quality.
In these steps, a region including a marker is divided
into regular grids as shown in Fig. 5. Here, we do notdirectly use the amplitude of luminance changes in themarker region but estimate it from the marker’s surrounding region Ψ (grid with a diagonal line in Fig. 5)
because photometric properties of markers and the sur(3) 3

Non cast shadow region

Texture generated byinpainting process (A)Fig. 5

Divided grids.
Fig. 7(a-3-1) Detection of shadows* LCR: luminancechange ratio

Do shadows exist around the marker?
YesNo(a-3-2) Interpolation of shadow regions in the marker(a-3-3) Calculation of local LCR(a-3-4) Calculation of global LCR

Fig. 6

Detailed steps in step (b-3) in Figure 3.

rounding materials are diﬀerent.
Fig. 6 illustrates the detailed steps of detection andreﬂection of luminance changes. Cast shadow regions
around a marker are ﬁrst detected (b-3-1). The existence of cast shadows is then checked. If cast shadows exist around the marker, shadow regions in themarker region are interpolated (b-3-2) and local luminance change ratio β between the ﬁrst frame and thecurrent frame is calculated (b-3-3). Then global luminance change ratio αi for each pixel xi (i is a pixel
index) in the remainder of the shadow region is calculated while excluding the inﬂuence of cast shadows (b3-4). If no cast shadows exist around the marker, only
the global luminance change ratio is calculated (b-34). Here, the local and global luminance change ratios
of RGB are separately calculated. Finally, as shownin Fig. 7, the luminance of the texture generated byprocess (A) is adjusted by multiplying local luminancechange ratio β to the interpolated shadow region andglobal luminance change ratio αi to the remainder ofthe marker region (b-4) as follows:⎧⎨βJ(x ) (cast shadow regions)i
In (xi ) =⎩α J(x ) (otherwise) ,
i

i

(1)
where In (xi ) is a pixel value of xi in the marker regionof the rectiﬁed n-th frame image and J(xi ) is a pixel
value of xi in the image generated by process (A).
In the following, we describe the methods to detect
cast shadows around a marker and calculate local and
4 (4)Cast shadow region

Adjustment of luminance of inpainted texture.

global luminance change ratios in detail.
3.2.1 Detection of Shadow Regions aroundMarkerIn shadow detection step (b-3-1), assuming that castshadows do not exist in the ﬁrst frame and the intensities of cast shadow regions are much smaller than otherregions, we use divided grids in surrounding region Ψ asshown in Fig. 8. Provisional global luminance change
ratio δ for the regions in which cast shadows do not exist in Ψ is then calculated in an iterative manner, andcast shadow regions are detected using the calculatedchange ratio δ.
Concretely, images of the ﬁrst frame and the n-thframe are ﬁrst smoothed with Gaussian ﬁlter to reducethe inﬂuence of noise. Provisional global luminancechange ratio δt in the t-th iteration is then calculated
as follows:¯u ∈Ψ ω(up )In (up ),(2)δt =  p¯up ∈Ψ ω(up )I1 (up )where I¯n (up ) is the average of pixel values in grid up (pis a grid index) in the rectiﬁed n-th frame image. Here,because weight ω(up ) should be small in non-shadow
regions, ω(up ) is deﬁned using luminance change ratio
δt−1 in the (t − 1)-th iteration as follows:ω(up ) =1,
dif(up ) + 1

dif (up ) = max(δt−1 I¯1 (up ) − I¯n (up ), 0),

(3)
(4)

where we set δ0 = 1 in the ﬁrst iteration. After iterating
the above processes several times, the grid whose difvalue computed by Eq. (4) is larger than a threshold isdetermined as the cast shadow region.
After classifying each grid into shadow region andnon-shadow region, the above processes using Eqs. (2)to (4) are performed pixel-wise by replacing grids in
Eqs. (2) to (4) with pixels in two adjacent grids between
which a shadow boundary exists. As a result, each pixel
is classiﬁed into shadow region and non-shadow region.

ITE Transactions on Media Technology and Applications Vol. 1, No. xx (2013)

Rectified imageof the 1st frame
x δt

xi

Rectified imageof the nth frame

feedback
Calculatexjxj(a) Shadow around marker(b) Interpolated shadow region

dif (u ) and δ tFig. 8Detection of cast shadows.

3.2.2 Interpolation of Shadow Regions
Assuming that a boundary of a cast shadow in amarker is straight, the shadow region is interpolated in
the marker in step (b-3-2). From an image representing shadow regions as shown in Fig. 9(a), the boundary pixels of the cast shadow on the boundary line between region Ψ and the marker are detected. Then pairsof shadow boundary pixels are decided and connected.
Concretely, to decide pairs of the shadow boundaries,
the following cost function is used.
C = |θi − θij | + |θj − θij |,Fig. 9Interpolation of shadow (White is shadow region,
gray is marker area, black is other region).

RegionΨ:Boundary pixel ofcast shadow

Marker Region:3 x 3 window
for calculating
gradient direction(5)

where θij indicates the gradient direction of shadowboundary in connecting pixel xi and xj (j is a pixel
index and i =| j). θi and θj indicate the gradient direction of shadow at pixel xi and xj , respectively. These
directions are calculated by the following equation.
⎧ ⎨tan−1 Ly(Lx > 0)
 Lx 
(6)θ=⎩tan−1 Ly + π (otherwise) ,Lx
where Lx and Ly represent the magnitude of horizontal and vertical edges and calculated by convolving a
3 × 3 region including a boundary pixel in the image
whose pixel values are dif with horizontal and vertical
Prewitt ﬁlters, respectively, as shown in Fig. 10. The
optimal pair which gives the smallest cost C is decided,
and the next pair is decided after removing the optimal
pair. This process is iterated until all pairs are decided.
Next, as shown in Fig. 9(b), all the pairs are connectedwith straight lines and the area surrounded by shadowboundaries is set as shadow region.
3.2.3 Calculation of Local Luminance Change
Ratio
Local luminance change ratio β is calculated using
pixels in the shadow region in region Ψ as follows:In (xi ),(7)
β = xi ∈S
xi ∈S I1 (xi )where S is a set of pixels in the shadow region in Ψ.
3.2.4 Calculation of Global Luminance Change
Ratio
In step (b-3-4), a luminance change ratio of each grid
Paper

xi

Fig. 10Calculation of gradient direction of shadowboundary.

in marker’s surrounding region Ψ is calculated, and luminance change ratios in the marker are then estimated.
Calculation of global luminance change ratioaround marker
Luminance change ratio αˆp in each grid up in region
Ψ is ordinarily calculated with the ratio of sum of pixel
values in a grid between the ﬁrst frame and the current
n-th frame as follows:
x ∈u In (xi ),(8)αˆp =  i p
xi ∈up I1 (xi )Here, it should be noted that, in case that cast shadows
exist in some grids, it is diﬃcult to properly estimate
global luminance change ratios in a marker because of
the inﬂuence of cast shadows. To exclude the inﬂuenceof cast shadows, as shown in Fig. 11, luminance change
ratio αˆp of the grid up which includes cast shadows inregion Ψ is replaced using luminance change ratios αˆqand αˆr of both ends of the cast shadow grids uq and ur(q and r are grid indexes and p =| q=| r) as follows:αˆp =(D − d)αˆq + dαˆr
,
D

(9)

where D represents the Manhattan distance between
grids uq and ur . d is the Manhattan distance between
grids up and uq .

(5) 5

uq

d

up

Shadow

Color-adjusted
textureD× H −1ur

Fig. 11

Replacement of global luminance change ratios
for excluding inﬂuence of cast shadows.
Input image

Estimation of global luminance change ratio in
markerGlobal luminance change ratios in a marker are interpolated using the calculated change ratios in region Ψ.
On the assumption that adjacent grids have close luminance changes under global luminance changes such as
ambient illumination changes, global luminance change
ratio of each grid is calculated so as to minimize thefollowing cost function E.
E=(αˆp − αˆq )2 ,(10)(p,q)∈R

R is a set of all index pairs of neighboring grids. The
luminance change ratio which minimizes E satisﬁes thefollowing equation.
∂E
= 0,
∂ αˆp

(11)

where grid up is in the marker. Each global luminancechange ratio for each grid is uniquely calculated with asystem of equations because the number of the above
equations is equal to the number of parameters. After calculating a change ratio for each grid, the change
ratio is set to the central pixel of the grid and global luminance change ratio αi for each pixel xi in the marker
is calculated by bilinear interpolation.
3. 3 Overlay of color-adjusted texture
The color-adjusted texture in step (b-4) is overlaid
on the marker region using inverse matrix of homography H, which is calculated in step (b-2). Concretely,as shown in Fig. 12, the rectiﬁed image in which thecolor of inpainted texture is adjusted is converted using
inverse homography matrix H −1 . The pixel values inthe target region of the converted image are then copied
to the pixels in the original input image.
6 (6)Fig. 12Output image

Overlay of color-adjusted texture.

4. Experiments
To demonstrate the eﬀectiveness of the proposed
method, we have carried out experiments by placing
markers on several environments. In the ﬁrst experiment, we verify the eﬀectiveness of correcting perspective distortion for image inpainting by comparing inpainted results using original and rectiﬁed images as
inputs. Next, we verify the eﬀectiveness of the proposed
method comparing results with several approaches. Finally we show two examples of augmented reality applications using the proposed method.
In experiments, we used a PC with Core i7 3.47 GHz
CPU and 12 GB of memory and a USB camera with
automatic white balance and gain. As an inpainting
method, we used our inpainting method 14) with highspeed searching method 12) . The resolution of input images is 640 × 480. We set parameters as shown in Table
1 balancing the quality of image inpainting and processing cost.
4. 1 Veriﬁcation of Eﬀectiveness of Correcting Perspective Distortion
Fig. 13 shows the comparison of inpainted results for
two kinds of images (a) and (e) using original and rectiﬁed images as input for image inpainting. From theresults in ﬁgures (c) and (g), it can be seen that unnatural textures are generated from original images because
appropriate samples for inpainting do not exist due toTable 1 Parameters in experiments.
Marker size in rectiﬁed image
80 × 80 pixels
Target region for inpainting
140 × 140 pixels
Grid size
14 × 14 pixels
Number of iterations in shadow detection
5

ITE Transactions on Media Technology and Applications Vol. 1, No. xx (2013)

(a) Input image(b) Rectiﬁed Image(c) Inpainted result using (a) as in- (d) Inpainted result using (b) as inputput(e) Input image(f) Rectiﬁed Image(g) Inpainted result using (e) as in- (h) Inpainted result using (f) as input
put

Fig. 13Comparison of inpainting using original input and rectiﬁed images.

the perspective distortion. On the other hand, as shownin ﬁgures (d) and (h), natural textures are successfully
generated by the proposed method using rectiﬁed images (b) and (f). From these results, we conﬁrm thatperspective distortion correction is eﬀective to inpaintmarker regions.
The processing time for image inpainting was eight
seconds on average. Therefore, in the prototype system,
we can experience AR applications with completely inpainted results in about eight seconds after starting applications.
4. 2 Comparison of Marker Hiding Results
We compared results by four approaches using two sequences. Sequence 1 consists of 360 frames and includes
ambient illumination changes as shown in Fig. 14. Sequence 2 consists of 340 frames and includes cast shadows as shown in Figs. 15 and 16. Figure (a) indicates input images, (b) indicates results by inpainting
for every frame, which is conventional approach 9), (c)indicates results by the proposed method without reﬂection of luminance changes, (d) indicates results by
the proposed method with reﬂecting only global luminance changes, and (e) indicates results by the proposed
method with reﬂecting both local and global luminancechanges.
For sequence 1 including ambient illumination
changes, in the results by the inpainting for every frame
shown in Fig. 14(b), it can be seen that some frames give
good results but some frames give unnatural textureson the marker region due to the perspective distortion,
resulting in not preserving temporal geometric consisPaper

tency. On the other hand, temporal geometric consistency is preserved by the proposed method as shownin Figs. 14(c), (d) and (e). Although photometric consistency is not achieved without reﬂection of luminancechanges as shown in Fig. 14(c), natural textures are generated with reﬂecting only global luminance changes asshown in Fig. 14(d). Combining adjustment methods
for local and global luminance changes also gives great
results without adverse eﬀects as shown in Fig. 14(e).
For sequence 2 including cast shadows, the geometry of results by inpainting for every frame is diﬀerent
from each other as shown in Figs. 15 and 16(b). Unlike this result, the temporal geometric consistency ispreserved by the proposed method. On the other hand,
we can also conﬁrm that photometric consistency is notachieved without the reﬂection of luminance changes asshown in Figs. 15 and 16(c). In the scene including castshadows, reﬂecting only global luminance changes gives
blurry shadow edges as shown in Fig. 15(d). However,combining the methods for local and global changes
gives clear cast shadows in the marker region as canbe seen in Fig. 15(e). From the results with various
viewpoints shown in Fig. 16, we can see that the proposed method which uses the unique inpainted result
for all the frames can preserve temporal geometric consistency in case of a largely moving camera as well as
photometric consistency. In the 4th frame of sequence
2 in Fig. 16, we can recognize that inpainting has notﬁnished yet, but in the 20th frame, the whole texture
has mostly been completed although the inpainting has
not ﬁnished yet at the time. This is because the in(7) 7

Frame
No.

78

79

80

81

(b) Output image(Inpainting
every frame)(a) Input image

Fig. 14(c) Output image(No reflection of
luminance changes)(d) Output image(Reflecting global
luminance changes)(e) Output image(Reflecting local and global
luminance changes)

Sequence 1 including ambient illumination changes (continuous frames)

Frame
No.
136

137

138

139

(a) Input image(b) Output image(Inpainting
every frame)

Fig. 158 (8)(c) Output Image(No reflection of
luminance changes)(d) Output image(Reflecting global
luminance changes)(e) Output image(Reflecting local and global
luminance changes)

Sequence 2 including cast shadows (continuous frames)

ITE Transactions on Media Technology and Applications Vol. 1, No. xx (2013)

Frame
No.
4

20

116

240

(a) Input image(b) Output image(Inpainting
every frame)

Fig. 16(c) Output Image(No reflection of
luminance changes)(e) Output image(Reflecting local and global
luminance changes)

Sequence 2 including cast shadows from various viewpoints

painting method 14) with searching method 12) employsa multi-scale approach, resulting in mostly generatinggeometry of texture for the whole target region in a few
seconds.
The frame rate was 19.7 fps for sequence 1 and 18.8
fps for sequence 2, respectively. The reason why theframe rate for sequence 2 is a little smaller is that it
costs an additional time to detect cast shadows and interpolate shadow regions in the marker.
4. 3 Examples of AR Applications by Hiding
Markers
We have constructed prototype AR systems using the
proposed marker hiding technique considering scenarios of hiding markers in playing games and simulating
furniture arrangement with portable devices. In theAR systems, virtual objects are overlaid after markers are hidden every frame. Top and bottom images
in Figs. 17(a) and (b) show the input images withmarkers and the output images in which markers are
removed and virtual characters and furniture are overlaid, respectively. As can be seen from the result in
Fig. 17(a), we can be absorbed in playing games without noticing the existence of markers. As can be seenfrom the result in Fig. 17(b), the markers do not ruin
the atmosphere of the room when we conﬁrm whether
Paper(d) Output image(Reflecting global
luminance changes)

the furniture matches the room or not.

5. Conclusion
We have proposed a new diminished reality techniquefor removing AR markers. Correcting the perspective
distortion of images enables high-quality image inpainting. Geometric and photometric consistencies of generated textures for marker regions are achieved by reﬂecting both local and global luminance changes separately
and overlaying the texture on the marker according tocamera motion. In experiments, we have demonstrated
the eﬀectiveness of the proposed method using several
sequences. In future work, we will develop a marker
hiding method for a non-planer background and consider more complex luminance changes such as curved
cast shadows. We will also aim at developing a diminished reality method for removing various real objects
from video images in conjunction with a localizationand mapping method.

Acknowledgment
This work was partially supported by Grant-in-Aid
for Scientiﬁc Research Nos. 23240024 and 24700118
from the Ministry of Education, Culture, Sports, Science and Technology.
(9) 9

(a) Game using virtual characters(b) Furniture arrangement simulation

Fig. 17

Examples of AR applications using the proposed marker hiding (Top: input images with markers,
bottom: output AR images).

References
1) H. Kato and M. Billinghurst: “Marker Tracking and HMD Calibration for a Video-based Augmented Reality Conferencing System”, Proc. Int. Workshop on Augmented Reality, pp.85–94(1999)
2) F. I. Cosco, C. Garre, F. Bruno, M. Muzzupappa and M. A.
Otaduy: “Augmented Touch Without Visual Obtrusion”, Proc.
Int. Symp. on Mixed and Augmented Reality, pp.99–102 (2009)3) V. Lepetit and M.-O. Berger: “An Intuitive Tool For Outlining
Objects in Video Sequences: Applications To Augmented And
Diminished Reality”, Proc. Int. Symp. on Mixed and Augmented
Reality, pp.159–160 (2001)
4) A. Enomoto and H. Saito: “Diminished Reality Using Multiple Handheld Cameras”, Proc. ACCV’07 Workshop on Multidimensional and Multi-view Image Processing, pp.130–135 (2007)5) S. Jarusirisawad, T. Hosokawa and H. Saito: “Diminished Reality
Using Plane-sweep Algorithm with Weakly-calibrated Cameras”,
Progress in Informatics, 7, pp.11–20 (2010)6) T. Tsuda, H. Yamamoto, Y. Kameda and Y. Ohta: “Visualization Methods for Outdoor See-Through Vision”, Proc. Int. Conf.
on Artiﬁcial Reality and Telexistence, pp.62–69 (2005)
7) S. Zokai, J. Esteve, Y. Genc and N. Navab: “Multiview Paraperspective Projection Model for Diminished Reality”, Proc. Int.
Symp. on Mixed and Augmented Reality, pp.217–226 (2003)
8) S. Siltanen: “Texture Generation over the Marker Area”, Proc.

10 (10)Int. Symp. on Mixed and Augmented Reality, pp.253–254 (2006)
9) J. Herling and W. Broll: “Advanced Self-contained Object Removal for Realizing Real-time Diminished Reality in Unconstrained Environments”, Proc. Int. Symp. on Mixed and Augmented Reality, pp.207–212 (2010)10) O. Korkalo, M. Aittala and S. Siltanen: “Light-weight Marker
Hiding for Augmented Reality”, Proc. Int. Symp. on Mixed andAugmented Reality, pp.247–248 (2010)11) D. Simakov, Y. Caspi, E. Shechtman and M. Irani: “Summarizing
Visual Data Using Bidirectional Similarity”, Proc. IEEE Conf. on
Computer Vision and Pattern Recognition, pp.1–8 (2008)
12) C. Barnes, E. Shechtman, A. Finkelstein and D. B. Goldman:“PatchMatch: A Randomized Correspondence Algorithm for
Structural Image Editing”, ACM Trans. on Graphics, 28, 3, pp.1–
11 (2009)
13) Y. Wexler, E. Shechtman and M. Irani: “Space-Time Completion of Video”, IEEE Trans. on Pattern Analysis and MachineIntelligence, 29, 3 (2007)
14) N. Kawai, T. Sato and N. Yokoya: “Image Inpainting Considering Brightness Change and Spatial Locality of Textures and
Its Evaluation”, Proc. Paciﬁc-Rim Symp. on Image and Video
Technology, pp.271–282 (2009)15) R. Hartley and A. Zisserman: Multiple View Geometry in Computer Vision, Cambridge University Press, ISBN: 0521540518(2004)ITE Transactions on Media Technology and Applications Vol. 1, No. xx (2013)

Norihiko Kawai received his B.E. degree
in informatics and mathematical science from Kyoto University in 2005. He received his M.E. and
Ph.D. degrees in information science from the Nara
Institute of Science and Technology in 2007 and2010, respectively. He was a research fellow of theJapan Society for the Promotion of Science and a
postdoctoral fellow at the University of California
at Berkeley in 2010-2011. He has been an assistantprofessor at the Nara Institute of Science and Technology since 2011. He is a member of IEEE, IEICE,IPSJ and VRSJ

Masayoshi Yamasaki received his B.E.
degree from Sasebo National College of Technology
in 2010. He received his M.E. degree in information science from the Nara Institute of Science andTechnology in 2012. He has been working at Canon
Inc. since 2012.

Tomokazu Sato received his B.E. degree
in computer and system science from the Osaka
Prefecture University in 1999. He received his M.E.
and Ph.D. degrees in information science from the
Nara Institute of Science and Technology in 2001
and 2003, respectively. He was an assistant professor at the Nara Institute of Science and Technology
in 2003-2011. He was a visiting researcher at Czech
Technical University in Prague in 2010-2011. He
has been an associate professor at the Nara Institute of Science and Technology since 2011. He is amember of IEEE, IEICE, IPSJ, VRSJ and ITE

Naokazu Yokoya received his B.E., M.E.,
and Ph.D. degrees in information and computer sciences from Osaka University in 1974, 1976, and1979, respectively. He joined Electrotechnical Laboratory (ETL) of the Ministry of International
Trade and Industry in 1979. He was a visiting professor at McGill University in Montreal in 1986-87
and has been a professor at the Nara Institute of
Science and Technology since 1992. He is a fellow
of IPSJ, IEICE and VRSJ and a member of IEEE,
ACM SIGGRAPH, JSAI, JCSS and ITE.

Paper(11) 11