HOLE-FREE TEXTURE MAPPING BASED ON LASER REFLECTIVITY
Shuji Oishi

Ryo Kurazume, Yumi Iwashita, Tsutomu Hasegawa

Graduate School of Information Scienceand Electrical Engineering
Kyushu University, Fukuoka, Japan

Graduate Faculty of Information Scienceand Electrical Engineering
Kyushu University, Fukuoka, Japan

ABSTRACTFor creating a three-dimensional (3D) model of a real object
using a laser scanner and a camera, texture mapping is aneffective technique to enhance the reality. However, in casethat the positions of the camera and the laser scanner differ
from each other, some texture-less regions (holes) may exist on the object surface where the appearance information is
missing due to the occlusion or out-of-sight of the camera.
In this paper, we propose a new texture completion techniqueutilizing laser reflectivity for hole-free texture mapping. The
laser reflectivity, which denotes the power of a reflected laser
light/pulse, is obtained as by-product of the range information
at laser scanning. Since the laser reflectivity captures the appearance property of the target as a camera image, it is reasonable that the regions with similar reflectance properties have
similar color textures. Based on this idea, texture informationin these holes is copied and pasted from the other texture regions according to the similarity and the order determined by
the texture and laser reflectivity. To verify the performanceof the proposed technique, we carried out texture completion
experiments in real scenes.
Index Terms— Texture completion, Reflectance image,Texture mapping, Laser scanner
1. INTRODUCTION
In recent years, 3D geometric modeling of real-world objects
with laser scanners has become very popular and widely usedin many areas, such as robotics, landscape surveying, digital archive of cultural heritages[1], and so on. Once a 3D
geometric model of a target object is obtained, a more realistic virtual model can be created by adding color information to the surface of the geometric model. For photo-realisticmodeling, texture mapping is an effective technique and still
remains one of important research topics in Virtual Reality(VR). In texture mapping, appearance information is added
to a 3D geometric model by mapping texture images on thesurface. However, in case that a 3D geometric model and atexture image are acquired from different viewpoints throughtwo independent sensors such as a laser scanner and a digital camera, some texture-less regions (holes) may exist on the978-1-4799-2341-0/13/$31.00 ©2013 IEEEFig. 1. In case that the positions of the camera and the laser scanner
differ from each other, some texture-less regions (holes) may exist
on the object surface where the appearance information is missing
due to the occlusion or out-of-sight of the camera.
object surface where the appearance information is missing
due to the occlusion by obstacles or out-of-sight of the camera (Fig.1).
For hole-free texture mapping, it is necessary to copy orsynthesize a proper texture image in the holes from the otherregions or other camera images. Boehm[2] proposed a simple
texturing method by capturing multiple texture images fromdifferent viewpoints to suppress occluded regions. Ortin etal.[3] proposed a similar technique employing the information of adjacent images. These methods, however, reconstruct
a virtual texture from multiple images taken from different
viewpoints by assuming a planar target object and calculatinga homography matrix between images. Moreover, the qualityof the synthesized texture depends on how proper images aretaken from different points of view.
On the other hand, Kawai et al.[4] proposed a completion method that restores the shape and the color of the object with geometrical and photometrical holes simultaneously.
They selected proper regions with geometrical or photometrical information from other parts of the object by minimizing
energy functions based on the similarities of geometries andtextures. However, if the resolution of the shape and the colordiffer, blurry artifacts are appeared since this technique is apoint-based approach.

1381

ICIP 2013

Meanwhile, texture (or shape) inpainting are quite effective techniques to solve this problem. Komodakis et al.[5]proposed a patch-exemplar based method that solves texture
inpainting as a discrete global optimization problem utilizing a belief propagation to avoid undesirable artifacts. For
proper inpainting, the order of the completion is also quite
important. Criminisi et al.[6], Sun et al.[7], and Li et al.[8]developed image completion techniques that determined theorder of the completion by extending some curves or lines
from a known region to an unknown region manually or automatically. Although these methods offered some excellent
results, unfavorable discontinuities are still remained due tothe geometrical and photometrical complexity of the target.
In this paper, we propose a novel hole-free texturing technique utilizing laser reflectivity. The laser reflectivity, which
denotes the power of a reflected laser light/pulse, is obtained
as by-product of the range information at laser scanning. Byaligning the reflectivity according to the order of scanning,a reflectance image, which contains rich appearance information of the target surface, is obtained. This image is quite similar to a camera image since each pixel indicates the intensity
on the surface of the targets under a single-frequency lightsource. Owing to this similarity, we first determine a completion order of a texture image in an occluded region according
to the reflectance value, and then synthesize the partial texture
by copying and pasting a proper texture patch from the otherregions with similar color and laser reflectivity.
The rest of this paper is organized as follows. In Section
2, we describe the texture completion technique, including
the details of reflectance images, the generation of databasepatches, the definition of completion priority based on thelaser reflectivity, and the completion algorithm. In section 3,
we show some experimental results to verify the effectivenessof the proposed method. In section 4, we conclude this paperand mention the future work.

2. PROPOSED METHODOLOGY
2.1. Reflectance image(a) Range image(b) Reflectance image

Fig. 2. Range and reflectance images(a) Scene 1(b) Scene 2

Fig. 3. Input textures
2.2. Texture completion utilizing laser reflectivity
In this section, we propose a new texture completion technique for a partially-occluded texture image, which extends patch-based completion techniques[5][6][7][8] to areflectance image. In this paper, we assume that the texture image is aligned to the geometrical model precisely
beforehand[9]. Our first step is generating a database ofsmall image patches from non-occluded regions in the texture
image. We define the region of occluding objects Ω manually
so that the rest of the texture image is specified as a source
region Φ. Next, we determine the completion order (priority)of occluded region Ω taking into account structures in a reflectance image. Finally, we calculate the similarity between
example patches in the database and the query patch centeredat the top-prioritized pixel in Ω, and complete the texture
image copying the most similar database patch to the missing
region.
2.2.1. Generation of database patches

A laser scanner (A time-of-flight range sensor) obtains arange value by measuring the round-trip time of a laser pulse
reflected by a surface of a target object. Figure 2(a) shows
an example of a range image acquired by a 3D laser scanner.
On the other hand, most laser scanners provide the strengthof the reflected light (reflectivity). Figure 2(b) shows a reflectance image that depicts reflectance values as a grayscale
image. As mentioned above, an unique reflectance value isdetermined for each pixel in the range image. In other words,
the range image and the reflectance image are precisely andfundamentally aligned.

At the beginning of the proposed technique, we divide a input
texture image I into a missing region Ω and a source regionΦ(=Ω̄) manually(Fig.3(a)(b)), where the former is supposed
to have only reflectivity and the latter is supposed to have reflectivity and texture. Next, we create a example patch consisting of k × k pixels centered at a pixel in Φ, and construct
a database D for all pixels in Φ. Note that we discard patches
including pixels in the missing region. Moreover, each patch
holds not only partial texture image but also the corresponding reflectance image, and both information describe the characteristic of each example patch.

1382

(a) Reflectance image(b) Priority map(c) Completed texture
without reflectance image
Experimental result (Scene 1)(d) Completed texture
with reflectance image(e) Reflectance image(f) Priority map(h) Completed texture
with reflectance image(g) Completed texture
without reflectance image
Experimental result (Scene 2)

Fig. 4. Texture completion results
2.2.2. Completion priority based on laser reflectivity
A noteworthy fact of image inpainting is that the completion order significantly affects the quality of the restored
image[5][6][7][8]. Inspired by these researches, we define”Priority map”, shown in Fig.4(b)(f) utilizing reflectivity.
Since reflectance images have appearance information of thetarget objects, the structures of the scene can be extracted bydetecting edges in the reflectance image. By applying Cannyfilter to the reflectance image, the priority is calculated asfollows:

P riority(m) =⎧⎨1
⎩ k2

M (p) =

p∈N (m)

M (p)

⎫⎬⎭

+ C(m)(1)

1 (p ∈ Φ)
0 (otherwise)

Where, m denotes a pixel in the missing region Ω, N (m)denotes a patch with k × k pixels centered at pixel m, andC(m) is a value of pixel m in the canny-filtered reflectanceimage. According to M (p), the pixel m on the boundary between the source region Φ and the missing region Ω has highpriority. Furthermore, when the pixel m is located on a structural edge in the canny-filtered reflectance image, the priority
becomes higher.
2.2.3. Completion algorithm
We complete the deteriorated texture image by copying themost similar example patch to the missing region. The simi-larity between a query patch q centered at the top-prioritized
pixel in Ω and one of database patches p is calculated basedon texture and reflectivity as follows:

Similarity(p, q) =

{SSD(Tp , Tq ) + αSSD(Rp , Rq )}−1p∈D(2)SSD(I, J) =kk {I(u, v) − J(u, v)}

2(3)u=0 v=0

Where, Tp and Rp denote the texture and reflectance image belonging to patch p, and α is a gain parameter. Notethat we synthesize a reflectance image taken from the viewpoint of the texture image by interpolating the reflectivity on
3D points (Fig.4(a)(e)). The best patch b for the query patch
q is determined by finding the patch in D that maximizes thesimilarity, and complete the missing region in q by copying
the texture image of b.
3. EXPERIMENTS
This section introduces the results of texture completion experiments. Range and reflectance images are captured by alaser scanner (SICK, LMS151) and a turn table. The size ofthese images is 760 × 1135 pixels. Texture images are taken
by a digital camera (Nikon, D300) and registered on the corresponding 3D geometric models[9]. We set parameters as
k = 15, α = 1.0 in the first experiment, and k = 13, α = 0.7
in the second experiment.

1383

(a) Original texture(b) Completed texture
without reflectance image(c) Completed texture
with reflectance image

Texture mapping (Scene 1)(d) Original texture(e) Completed texture
without reflectance image(f) Completed texture
with reflectance image

Texture mapping (Scene 2)
Fig. 5. Texture mapping results
At first, we carried out the preliminary experiment in a
simple situation shown in Fig.3(a). In this experiment, we cut
out a part of the input texture image and attempted to restore
the missing region. Fig.4(a)(b) show a reflectance image anda priority map. Fig.4(c)(d) show the completion results without and with the reflectance image. By taking into accounta property of the reflectance image, our proposed techniquesuccessfully filled in the missing region compared with theconventional technique. Textured 3D models of Scene 1 areshown in Fig.5(a)-(c).
Next, we addressed the experiment in a more complex
scene shown in Fig.3(b) where an obstacle occludes the walland the texture behind it cannot be seen. Fig.4(e) shows thereflectance image from the same viewpoint of the texture image, and the completion order is determined by considering
the structures of the reflectance image as shown in Fig.4(f).
The completion results after applying the conventional technique and the proposed technique are shown in Fig.4(g)(h)respectively. 3D geometric models of Scene 2 after applying texture mapping are shown in Fig.5(d)-(f). As shownthese figures, our proposed technique outperformed the conventional technique which caused some artifacts. However,the proposed technique also produced partial discontinuous
textures, especially in a vertical direction. The reason why
these artifacts occurred was that sufficient appearance infor-mation in the reflectance image could not be acquired due tothe low resolution of the laser scanner.

4. CONCLUSION
In this paper, we proposed a new texture completion technique for texture mapping utilizing laser reflectivity. The proposed technique is able to generate the texture image in an
occluded region of the 3D geometric model accurately by prioritizing the completion order.
In the future, we will improve the proposed technique sothat it works well even with a low resolution reflectance image. Furthermore, we will extend our proposed method andgenerate the entire texture of a 3D geometric model from asingle picture.

Acknowledgement
This work was supported in part by JSPS KAKENHI Grant
Number 246404 and a Grant-in-Aid for Scientific Research(B) (23360115).

1384

5. REFERENCES[1] Ryo Kurazume, Yusuke Noda, Yukihiro Tobata, KaiLingemann, Yumi Iwashita, and Tsutomu Hasegawa,“Laser-based geometric modeling using cooperative multiple mobile robots,” in Proc. IEEE International Conference on Robotics and Automation, 2009, pp. 3200–3205.
[2] J. Boehm, “Multi-image fusion for occlusion-free facadetexturing,” in IAPRS Vol. XXXV, part 5, 2004, pp. 867–872.
[3] D. Ortin and F. Remondino, “Generation of occlusionfree images for realistic texture mapping,” in International Archives of Photogrammetry, Remote Sensing andSpatial Information Sciences, 2005, p. 36(5/W17).
[4] N. Kawai, A. Zakhor, T. Sato, and N. Yokoya, “Surface
completion of shape and texture based on energy minimization,” in Proc. International Conference on Image
Processing, 2011, pp. 913–916.
[5] G. Tziritas N. Komodakis, “Image completion using efficient belief propagation via priority scheduling and dynamic pruning,” in IEEE Trans. Image Process. 16 (11),2007, pp. 2649–2661.
[6] A. Criminisi, P. Perez, and K. Toyama, “Region fillingand object removal by exemplar-based inpainting,” inIEEE Trans. Image Process. 13 (9), 2004, pp. 1200–1212.
[7] J. Jia J. Sun, L. Yuan, “Image completion with structure
propagation,” in ACM Trans. Graphics 24 (3), 2005, pp.
861–868.
[8] Ming Zhao Shutao Li, “Image inpainting with salient
structure completion and texture propagation,” in Pattern
Recognition Letters 32 (9), 2011, pp. 1256–1266.
[9] Ryo Inomata, Kenji Terabayashi, Kazunori Umeda, andGuy Godin, “Registration of 3d geometric model andcolor images using sift and range intensity images,” inProceedings of 7th International Symposium on VisualComputing(ISVC2011), 2011, pp. 325–336.

1385