AR Marker Hiding Based on
Image Inpainting and Reflection of Illumination Changes
Norihiko Kawai

Masayoshi Yamasaki

Naokazu Yokoya∗

Tomokazu Sato

Graduate School of Information Science, Nara Institute of Science and Technology, JapanA BSTRACT

Index Terms: H.5.1 [Information Interfaces and Presentation]:
Multimedia Information Systems—Artificial, augmented, and virtual realities
1 I NTRODUCTION
In augmented reality (AR) applications, marker-based geometricregistration is often used because of its robustness and easiness.
However, such marker-based AR has the limitation that seamlessfusion between virtual objects and the real world is not achieved
due to the existence of markers. To solve this problem, diminishedreality methods, which remove real objects by overlaying background texture onto the object regions, have been proposed [2, 3, 5].
Among the methods, we focus on an approach which generates textures on objects using image inpainting instead of capturing the real
background. In the inpainting-based approach, three factors should
be considered to achieve natural marker hiding; (1) naturalness oftexture generated on a marker area, (2) geometric consistency between frames, (3) photometric consistency between the marker area
and its surrounding. We describe conventional methods and the proposed method while focusing on the three factors.
Herling et al. [3] have applied an exemplar-based inpainting
method to each frame of an input video. Just applying an exemplarbased inpainting method to an original input image often generates
unnatural textures because of its weakness for perspective distortion. There exists another problem caused by inpainting for every frame; although the method tries to reduce texture flickering bytemporally propagating patch correspondences for inpainting, unnatural geometric changes between frames are still observable. To
overcome these problems, in the proposed method, perspective distortion is corrected for input of inpainting for natural texture generation (1), and the unique inpainted texture is used in all the frames
without trying to re-inpaint for temporal geometric consistency (2),On the other hand, Korkalo et al. [5] have focused on photometric consistency in diminished reality. The method generates a
low-resolution texture for a current frame and combine it with highresolution texture inpainted using the first frame. While this method
can treat only global luminance changes, the proposed method considers both global and local ones for photometric consistency (3).
∗ e-mail:{norihi-k, tomoka-s, yokoya}@is.naist.jp

next frameThis paper proposes a new method of diminished reality whichremoves AR markers from a user’s view image in real time. To
achieve natural marker hiding, assuming that an area around amarker is locally planar, the marker area in the first frame is inpainted using the rectified image to achieve high-quality inpainting.
The unique inpainted texture is overlaid on the marker region ineach frame according to camera motion for geometric consistency.
Both global and local luminance changes around the marker are
separately detected and reflected to the inpainted texture for photometric consistency.

(A) Marker hiding(B) Image inpainting(a-1) Image Input(b-1) Specifying marker region(a-2) Correction ofperspective distortion(a-3) Detection ofluminance changes

onlyfor 1stframe(a-4) Reflection ofluminance changes(b-2) Searching forsimilar texture(b-3) Update of pixel values(b-4) Output of tentative
inpainted result(a-5) Overlay

Figure 1: Flow of the proposed method.
(a-3-1) Detection of shadows* LCR: luminancechange ratio

Do shadows exist around the marker?

YesNo(a-3-2) Interpolation of shadow regions in the marker(a-3-3) Calculation of local LCR(a-3-4) Calculation of global LCR

Figure 2: Detailed steps in step (a-3).

Figure 3:grids.

Divided

2 R EAL - TIME AR M ARKER HIDING
Figure 1 shows the proposed pipeline. The method consists of twoprocesses: (A) marker hiding by reflecting luminance changes andoverlaying the inpainted textures on markers for every frame, and(B) image inpainting for marker areas. Although process (B) is
not performed in real-time for generating high-quality texture, users
can use the AR system without waiting time by performing the twoprocesses concurrently. In process (B), we employ exemplar-based
image inpainting method [4] with searching method [1]. The inpainting method is applied to the first frame image of which theperspective distortion is corrected. In process (A), after inputting
an image (a-1), the image is rectified in step (a-2). By correcting
the perspective distortion, the size and shape of a texture pattern are
unified. That results in generating high-quality texture by image inpainting in process (B). In addition, pixel correspondences between
different frames of input images can be easily determined.
2.1 Detection and reflection of luminance changes
The luminance of the texture generated by process (B) is adjusted
by considering local luminance changes (e.g. by cast shadows) andglobal luminance changes (e.g. by ambient illumination changes
and camera parameter changes). It should be noted that we do notdirectly use the amplitude of luminance changes in the marker but
estimate them from the marker’s surrounding because photometricproperties of markers and the surrounding materials are different.
Step (a-3) calculates local luminance change ratio β for shadow
region and global one αi for each pixel xi in the remainder of themarker. The ratios of RGB are separately calculated. Then step(a-4) adjusts the luminance of the texture generated by process (B)by multiplying β to the interpolated shadow region and αi to theremainder. Figure 2 shows the detailed steps of step (a-3). In thesesteps, a marker and the surrounding region Ψ (grid with a diagonalline in Figure 3) are divided into grids for efficient processing.

Rectified imageof the 1st frame

Rectified imageof the nth frame

feedback×δ t

Calculatedif (u ) and δ t(a) Input Image

Figure 4: Detection of cast shadows.

2.1.1 Calculation of local luminance change ratio
In step (a-3-1), we assume that shadows do not exist in the firstframe and the intensities of shadow regions are much smaller thanthe other regions. As shown in Figure 4, provisional global luminance change ratio δt for the regions in which cast shadows do notexist in Ψ is first calculated in iterative manner as follows:δt = ∑ ω (u)I¯n (u)/ ∑ ω (u)I¯1 (u),u∈Ψ(1)(b) No rectification(d) Reflecting only global(c) No reflection(e) Reflecting local and global

Figure 5: Comparison of results of continuous frames in a sequence
including cast shadows.

u∈Ψ

where I¯n (u) is the average of pixel values in grid u in the n-thframe rectified image. t is number of iterations. We define ω (u)as 1/(dif(u) + 1) wheredif(u) = max(δt−1 I¯1 (u) − I¯n (u), 0),(2)

because weight ω (u) should be small in non-shadow regions. Afterseveral iterations, the grid whose dif value by Eq. (2) is larger than athreshold is determined as cast shadow region. The above processes
are then performed pixel-wise for only two grids which include theshadow boundary, and each pixel is classified into shadow and nonshadow regions.
In step (a-3-2), the shadow boundaries in region Ψ are first detected. To decide pairs of the boundaries, cost |θi − θi j | + |θ j − θi j |is used. Here θi j is the gradient direction of shadow boundary inconnecting pixel xi and x j . θi and θ j are the gradient directionsof shadow at pixel xi and x j . The pair which gives the smallest
cost is decided, and after removing the pair, a next pair is decided.
This process is iterated until all pairs are decided, and the areasurrounded by shadow is set as shadow region. Local luminancechange ratio β is determined by the ratio of sum of pixel values inthe shadow region in region Ψ between the first and current frames.
2.1.2 Calculation of global luminance change ratio
In step (a-3-4), luminance change ratio αˆu in each grid u in region
Ψ is first calculated with the ratio of sum of pixel values in a grid of
the first and current frames. To exclude the influence of cast shadows, ratio αˆu of the grid u which includes cast shadows in region Ψis reset by linear interpolation using change ratios of non-shadowgrids. α̂ of each grid in the marker is then calculated so as to minimize cost E.
E=∑ (αˆu − αˆv )2 ,(3)(u,v)∈R

where u and v are indexes of grids. R is a set of pairs of grids
indexes which are in eight neighbor. Change ratio αi for each pixel
xi in the marker is then calculated by bilinear interpolation.
3 E XPERIMENTS
We performed experiments by putting markers on several environments. In experiments, we used a PC with Core i7 3.47 GHz CPU
and 12 GB of memory for input images with resolution 640 × 480
captured by a USB camera with automatic white balance and gain.
We compared results by four approaches using a sequence including cast shadows. In the results by inpainting without rectification in Figure 5(b), unnatural textures are generated due tothe perspective distortion. The results in Figures 5(c), (d) and (e)are generated with rectification and different reflections methods.
Without reflection of luminance changes, photometric consistency
is not achieved as shown in Figure 5(c). Reflecting only global

Figure 6: Examples of AR applications using the proposed marker
hiding (Top: input images with markers, bottom: output AR images).

luminance changes gives blurry shadow edges as shown in Figure5(d). Reflecting both local and global changes, clear cast shadowsare successfully given as shown in Figure 5(e). The frame rate was
18.8 fps and the processing time for image inpainting in process(B) was eight seconds. Therefore, we can experience AR applications with completely inpainted results in about eight seconds afterstarting.
Figure 6 shows prototype AR systems using the proposed
method considering scenarios of hiding markers in playing games
and simulating furniture arrangement. We can enjoy AR applications without being concerned about the existence of markers.
4 C ONCLUSION
We have proposed a new diminished reality method for removing
AR markers in real time. Correction of the perspective distortionof images enables high-quality image inpainting. Geometric andphotometric consistencies are achieved by reflecting both local andglobal luminance changes separately and overlaying the texture on
the marker according to camera motion. Future work involves developing a marker hiding method for a non-planer background.
ACKNOWLEDGEMENTS
This work was partially supported by JSPS KAKENHI Nos.
23240024 and 24700118.
R EFERENCES[1] C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman. PatchMatch: A randomized correspondence algorithm for structural image
editing. ACM Transactions on Graphics, 28(3):1–11, 2009.
[2] A. Enomoto and H. Saito. Diminished reality using multiple handheldcameras. In Proc. ACCV’07 Workshop on Multi-dimensional and Multiview Image Processing, pages 130–135, 2007.
[3] J. Herling and W. Broll. Advanced self-contained object removal for
realizing real-time diminished reality in unconstrained environments.
In Proc. ISMAR, pages 207–212, 2010.
[4] N. Kawai, T. Sato, and N. Yokoya. Image inpainting considering brightness change and spatial locality of textures and its evaluation. In Proc.
PSIVT, pages 271–282, 2009.
[5] O. Korkalo, M. Aittala, and S. Siltanen. Light-weight marker hidingfor augmented reality. In Proc. ISMAR, pages 247–248, 2010.