Augmented Reality Image Generation with
Virtualized Real Objects Using
View-Dependent Texture and Geometry
Abstract
Yuta Nakashima
NAIST8916-5 Takayamacho, Ikoma
Nara, 630-0192 Japann-yuta@is.naist.jp

Tomokazu Sato
NAIST
8916-5 Takayamacho, Ikoma
Nara, 630-0192 Japan
tomoka-s@is.naist.jp

Yusuke Uno
NAIST
8916-5 Takayamacho, Ikoma
Nara, 630-0192 Japan

Naokazu Yokoya
NAIST8916-5 Takayamacho, Ikoma
Nara, 630-0192 Japan
yokoya@is.naist.jp

Norihiko Kawai
NAIST
8916-5 Takayamacho, Ikoma
Nara, 630-0192 Japannorihi-k@is.naist.jp

Augmented reality (AR) images with virtualized real objects
can be used for various applications. However, such AR image generation requires hand-crafted 3D models of that objects, which are usually not available. This paper proposes aview-dependent texture (VDT)- and view-dependent geometry (VDG)-based method for generating high quality AR
images, which uses 3D models automatically reconstructedfrom multiple images. Since the quality of reconstructed 3Dmodels is usually insuﬃcient, the proposed method inflates
the objects in the depth map as VDG to repair chipped object boundaries and assigns a color to each pixel based on
VDT to reproduce the detail of the objects. Backgroundpixel exposure due to inflation is suppressed by the use of
the foreground region extracted from the input images. Our
experimental results have demonstrated that the proposed
method can successfully reduce above visual artifacts.

Author Keywords
AR with virtualized real objects, view-dependent texture,view-dependent geometry

ACM Classification Keywords
H.5.1 [Information Interfaces and Presentation]: Multimedia Information SystemsArtificial, augmented, and virtualrealities

Introduction
Virtually arranging real objects in a real environment using augmented reality (AR) technology potentially has awide range of applications such as virtual room planning[1]. Such applications usually require virtualizing the realobjects, i.e., to create their 3D models so as to generate
their arbitrary viewpoint images. One of the most straightforward approaches for virtualization is to hand-craft 3Dmodels of the real objects. However, 3D modeling generally is a laborious task, requiring special skills. This may
prevent ordinary developers and users from involving suchapplications. Therefore, automatic techniques for virtualizing real objects are strongly desired.

Figure 1: Examples of generated AR images. From top tobottom: CMPMVS [6], CMPMVS with VDT [3], and proposed
method.

Recently, dense 3D reconstruction [4, 6] and image-based
rendering [5, 8] have been studied, which can virtualize
real objects from multiple images containing them. Amongthese techniques, dense 3D reconstruction is preferable toimage-based rendering as it requires less input images thanimage-based rendering. However, even the state-of-the-art
dense 3D reconstruction techniques usually suﬀer from insuﬃcient accuracy, resulting in visual artifacts such as (i)loss of detailed shapes and texture and (ii) rough boundaries, as in the top image of Figure 1.
One approach to solve the above problems while using inaccurate 3D models is to use view-dependent texture (VDT)
proposed by Debevec et al. [3]. VDT adaptively selects theimage that best suits each triangle patch of the 3D model
under a certain criterion and apply the corresponding image patch to the 3D model. As in Figure 1(middle), VDT
recovers the detailed shape and texture. However, as canbe seen in the figure, the generated image still suﬀer fromproblem (ii) because VDT does not change the shape ofthe 3D model at all. Also, it requires fine adjustment of
the input image to the 3D model.

Figure 2: Overview of proposed method.

This paper proposes a method for AR image generationfrom automatically reconstructed 3D models as in Figure1(bottom). The proposed method incorporates VDT withview-dependent geometry (VDG) and foreground extraction, where VDG, in this paper, is defined as to modify
a depth map of 3D models given a viewpoint so that thedepth map suits the viewpoint for generating an image ofthe virtualized real objects. Our main idea is to inflate
the object regions in the depth map by VDG to alleviate
the chipped boundaries. Background pixel exposure due toobject region inflation is suppressed by ignoring the nonforeground region of the input images when VDT determines each pixel’s color. The proposed method is suitable for objects whose images can be taken from anywherearound it. To verify its advantage, we visually demonstrate
the quality of generated AR images. We also verify that theproposed method can work in real-time, which is essential
property toward AR applications.

AR Image Generation with VDT and VDG

Figure 3: Original (top) andinflated (bottom) depth maps,
where inflated one is generated by20 iterations.

The proposed method consists of oﬄine and online stages(Figure 2). Given set S = {I1 , . . . , IK } of input images
that contain real objects (referred to as the target), the offline stage reconstructs a 3D model of the target, e.g., using
a multi-view stereo (MVS) method [4, 6, 10], and extracts
its regions in the input images as foreground. The target
in the reconstructed 3D model may be then extracted if it
contains extra portion. For foreground extraction, we canuse GrabCut [9] to reduce the burden to extract the foreground manually. Although both dense 3D reconstructionand foreground extraction may require manual operations,they are less laborious than hand-crafting 3D models. The
online stage firstly estimates the pose of real camera CE
from the captured image IE as the viewpoint for the virtualized target, e.g., using ARToolkit [7], and generates thedepth map given the 3D model and the pose of CE . Then,since the errors are inevitable even for the state-of-the-art
3D reconstruction techniques, the proposed method inflates
the target regions in the depth map to alleviates visual artifacts around the boundaries of the target. We then selecta suitable image Ik from S, based on which colors of thecorresponding pixels are determined. Finally, we superimpose the generated image of the virtualized target to thecaptured image. The details of object region inflation byVDG, VDT-based color assignment, and generated image
superimposition are given below.
Object region inflation by VDG : To alleviate the problemof chipped boundary, the proposed method inflates the target region in the depth map so that the boundary in thegenerated image can be formed by the boundaries in theinput images. Given the viewpoint, the online stage generates the depth map of the 3D model. Let Ω denote the setof pixels in target region, i.e., the image region onto which
the 3D model of the target is projected, and thus the depthvalues are calculated over Ω. The depth value for pixel i
is denoted by di . The target region is inflated by applying
the following smoothing filter with the 3 × 3 uniform kernel
to the depth map:∑1dj for i ∈ Ω̄|Ω ∩ Φi |d′i =,(1)j∈Ω∩Φidiotherwisewhere Φi is the set of pixels that the kernel centered at
pixel i covers. |Ω ∩ Φi | is the number of pixels in set Ω ∩Φi and Ω̄ is the complementary set of Ω. This keep theinside of target region Ω unchanged so as not to smooththe boundary within the targets. To get suﬃcient inflation,
the filter is applied iteratively. Figure 3 shows examples ofan original depth map and an inflated one.
VDT-based color assignment: To reproduce the detail ofthe target, we assign a color to each pixel in the inflated
target region based on the input images considering occlusion.
First, the proposed method finds the image in S that suits
for picking the color of pixel i in the inflated target region.
Given the intrinsic and extrinsic camera parameters of CE ,we can regain 3D position pi of pixel i whose depth value
in the depth map after inflation is d′i , where the world coordinate can be set arbitrarily. Let tE and tk denote thetranslation of CE and Ck , respectively, where tk is thecamera that shot k-th image Ik ∈ S. The suitable image
is selected so that it can fulfill the following requirements:(i) To reproduce the detail of the target, as described in[3], the angle formed by tE − pi and tk − pi must be as
small as possible among all k’s as long as it fulfills the requirement (ii). (ii) Since pi is a point on the 3D modeland thus can be occluded in some images in S, pi must
be visible in the selected image. We can easily find Ik that

fulfills the requirement (i) because this is equivalent to findk that gives the largest value of sk,i defined bysk,i = (tE − pi ) · (tk − pi )/(|tE − pi | |tk − pi |).

Figure 4: Examples of (a) generated image and (b) AR image.
Red pixels in (a) represent background label.

Figure 5: Image datasets for DS1(left) and DS2 (right).

(2)

To verify the visibility of 3D position pi , we generate thedepth map of the 3D model using the camera parametersof Ck . This depth map generation can be done at theoﬄine stage because the camera parameters and the 3Dmodel are fixed. At the online stage, we project pi onto Ik
again using the camera parameters of Ck . The depth value
corresponding to pi is denoted by dk,i and the projected
position on Ik is denoted by (xk,i , yk,i ). Position (xk,i , yk,i )
identifies the corresponding depth value ek,i in the depth
map generated for Ck . If pi is occluded, dk,i is further from
Ck than ek,i . Therefore, we judge that pi is visible whendk,i = ek,i and pi is in the view frustum of Ck . The lattercondition can be verified using Ck ’s camera parameters.
In order to achieve real-time VDT-based color assignment,
we reduce the computational burden by avoiding one-byone visibility verification as follows: Firstly, we calculate
sk,i and verify visibility if sk,i exceeds the current greatest
value obtained previously. If pi is visible, current greatest
value sk̂,i is updated to sk,i This process is repeated fromk = 1 to K to find k ∗ that gives the greatest value of sk̂,i .
Finally, we assign a color to pixel i using the foregroundregions of Ik∗ . If the pixel at (xk∗ ,i , yk∗ ,i ) in Ik∗ is in theforeground regions, its color is assigned to pixel i. Otherwise, a label representing background is assigned to it.
An example of a generated image is shown in Figure 4(a),where the pixels to which background labels are assigned
are represented by red. This color assignment applied to theinflated depth map alleviates chipped boundaries because
the boundaries of the generated image is formed by actual
boundaries in the foreground regions of the input images.

Generated view superimposition: Here we generate an AR
image by superimposing the generated image on the image
IE captured by CE . The pixels in the inflated target region
are superimposed on IE , but the pixels with backgroundlabels are ignored. An example of the generated image isshown in Figure 4(b).

Experimental results
We experimentally verified the visual quality of the proposed method. Two image datasets, i.e., DS1 and DS2,
were used in this experiment as input images (Figure 5),
whose specification is summarized in Table 1. To reconstruct the 3D models from these datasets, we used CMPMVS [6], where the intrinsic and extrinsic camera parameters for input images were estimated by Zhang’s method[11] and VisualSFM [2], respectively. Since the 3D models
originally output from CMPMVS usually contain non-object
portions, we manually extracted the target (Figure 6). For
foreground extraction, we used GrabCut [9]. The seed region for initializing GrabCut was given manually, and weinteractively refined the target regions. For camera pose
estimation at the online stage, we adopted ARToolkit [7].
For DS2, we manually selected 59 images from the dataset
to reduce the computational cost in VDT-based color assignment. The depth maps were generated from the 3Dmodels using OpenGL.
To demonstrate the superiority of the proposed method, weshow generated images by two baseline methods, i.e., (A) amodel-based method and (B) a VDT-based method, as wellas (C) the proposed method. In the model-based method,the 3D model of the target was rendered with the color
assigned to each vertex by the dense 3D reconstruction
technique. The VDT-based method applies the original
VDT technique to the 3D model, where suitable texture
was selected for each triangle of the 3D model.

Table 1: Specification of datasets. Numbers of (i) images and(ii) triangle patches.

(i)
(ii)

DS1
24
36,051DS2571
19,408Figure 7 shows AR images by the baseline and proposed
methods. The model-based method gave a blur-like visual
artifacts. This is because it did not apply texture to themodel and the colors assigned to the vertices were not sufficient to reproduce the detail. Also, the result indicatedthat the 3D models reconstructed by CMPMVS were inaccurate, producing chipped boundaries.
In the VDT-based method, the blur-like visual artifacts werealleviated owed to the VDT technique. However, the problem of chipped boundaries was still significant. The AR images by the VDT-based method also exposed backgroundpixels due to the inaccurate 3D model.

Figure 6: Obtained 3D models forDS1 (left) and DS2 (right).

PC used in this experiment was equipped with Intel Core i7
2600K 3.40 GHz, 8 GBytes RAM, and NVIDIA GeFore GTX
560 Ti, whose OS and Graphic API were Windows 7 64 bit
and OpenGL, respectively. Although the frame rates of theVDT-based and the proposed methods were comparable,
they were currently not suﬃcient for the smooth motion.
Current implementations of the VDT-based and proposed
methods use only CPU for most part of its computation.
Using the GPU may improve the frame rate, allowing us toimplement smooth texture transition as well.

The proposed method overcame the above problems. Roughboundaries were smoothed by VDG by inflating the target regions, while foreground extraction alleviated excessive
background pixel exposure.
However, the proposed method introduced another causeof rough boundaries, i.e., errors in alignment of the 3Dmodels and the input images incorrectly assigned colors inbackground regions to the target regions, which is noticeable in the red rectangle in Figure 8. This problem is potentially solved by establishing a criterion to find images
with large alignment errors and excluding them. Anotherproblem of the proposed method was texture transition.
The original proposal of the VDT technique [3] reduces thediscontinuity in texture by determining the color based onweighted average of multiple texture images. However, in
this work, we omitted this because of its computationalcost, and clear discontinuity in texture was observable in
the proposed method as in the blue rectangle and its closeview in Figure 8.
The frame rates of the model-based, VDT-based, and the
proposed methods were summarized in Table 2, where theFigure 7: Examples of generated AR images. From top tobottom rows: Model-based method, VDT-based method, andproposed method. From left to right columns: generated ARimages for DS1 and their close views, and generated AR images
for DS2 and their close views.

SummaryThis paper has presented a method for generating AR images with real objects whose 3D models are obtained bydense 3D reconstruction techniques. Usually, 3D models based on such techniques are not very accurate, andthus they produce significant visual artifacts. The pro-Table 2: Frame rates. (A) modelbased, (B) VDT-based, and (C)proposed methods.

(A)
(B)
(C)

DS1
94.48 fps
11.18 fps
8.32 fpsDS2
95.34 fps
6.40 fps
4.32 fps

posed method reduces visual artifacts by using the VDG
and VDT techniques, where VDT covers the lack of thedetailed shape, and VDG smoothes rough boundaries of
the 3D models by leveraging extracted foreground. Our experimental results successfully demonstrated the superiorityof the proposed method compared with two baseline methods. The proposed method drastically reduces the burden
to hand-craft detailed 3D models of real objects and thus
potentially encourages AR applications with virtualized realobjects. The applications of the proposed method include
virtually arranging pieces of furniture and AR sightseeing,in which historical heritages that have been currently lostand preserved only as miniatures are presented as in Figure 9. Our future work is implementing smooth texture
transition as well as the use of GPU.

Figure 9: AR sightseeing in a historic site: Example applicationof proposed method.
Examples of roughFigure 8:
boundaries introduced due to foreground extraction indicated by red
rectangles and their close views(top), and discontinuous texture
transition indicated by blur rectangle (bottom).

AcknowledgementsThis work is supported by the Ministry of Education, Culture, Sports, Science and Technology, Grant-in-Aid for Scientific Research (A) No. 23240024. Some images in thiswork are by the courtesy of Todaiji Temple, Japan.

References[1] Augmented Reality Solutions. VizAR.
http://youreality3d.com/.
[2] Changchang, W. VisualSFM: A visual structure frommotion system, 2011.
http://www.cs.washington.edu/homes/ccwu/vsfm/.
[3] Debevec, P., Yu, Y., and Borshukov, G. Eﬃcientview-dependent image-based rendering with projective
texture-mapping. In Proc. 9th Eurographics Rendering
Workshop (1998), 13 pages.
[4] Furukawa, Y., Curless, B., Seitz, S. M., and Szeliski,
R. Towards internet-scale multi-view stereo. In Proc.
2010 IEEE Conf. Computer Vision and Pattern Recognition (2010), 1434–1441.
[5] Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen,
M. F. The lumigraph. In Proc. ACM SIGGRAPH ’96(1996), 43–54.
[6] Jancosek, M., and Pajdla, T. Multi-view reconstruction preserving weakly-supported surfaces. In Proc.
2011 IEEE Conf. Computer Vision and Pattern Recognition (2011), 3121–3128.
[7] Kato, H., and Billinghurst, M. Marker tracking andHMD calibration for a video-based augmented reality
conferencing system. In Proc. 2nd IEEE/ACM Int.
Workshop on Augmented Reality (1999), 85–94.
[8] Matusik, W., Buehler, C., Raskar, R., Gortler, S. J.,
and McMillan, L. Image-based visual hulls. In Proc.
ACM SIGGRAPH ’00 (2000), 369–374.
[9] Rother, C., Kolmogorov, V., and Blake, A. Grabcut:
Interactive foreground extraction using iterated graphcuts. In Proc. ACM SIGGRAPH ’04 (2004), 309–314.
[10] Seitz, S. M., Curless, B., Diebel, J., Scharstein, D.,
and Szeliski, R. A comparison and evaluation of multiview stereo reconstruction algorithms. In Proc. 2006
IEEE Conf. Computer Vision and Pattern Recognition(2006), 519–528.
[11] Zhang, Z. A flexible new technique for camera calibration. IEEE Trans. Pattern Analysis and MachineIntelligence 22, 11 (2000), 1330–1334.