WISS2007

3D Reconstruction from Multiple Videos in Real Environments
H. Kim

R. Sakamoto

I. Kitahara

T. Toriyama

K. Kogure ∗Summary.
We propose an advanced visual hull technique to compensate for outliers using thereliabilities of the silhouettes. The proposed method consists of a background subtraction techniqueusing multiple thresholds and a reliability-based shape-from-silhouette algorithm. They are connectedby the intra-/inter-silhouette reliabilities to compensate for carving errors from defective segmentation orpartial occlusion which may occur in real environments. In ﬁnal view rendering, we use a cinematographic
camera control system and an ARToolkit to control virtual cameras.

1Introduction

We have introduced Cinematized Reality,
which aimed to record unexpected moments andcreate movie-like footage by a virtual camera
generated from multiple cameras [2]. However,this system still has problems for applications toreal environments because it is based on a visual hull technique that carves 3D space withsilhouette information. In real scenes, the silhouette includes outliers from defective segmentation and occlusions, and these outliers directly
aﬀect the visual hull as shown in Fig. 1. Therefore we propose an advanced visual hull technique to compensate for outliers using reliabilities of the silhouettes.

23D Reconstruction

We extract silhouette information from video
sequences using the background subtraction technique, which is one of the most common approaches. In the proposed approach, we include
the reliability of the segmentation results in thesilhouette masks. We classify all regions into
four categories using multiple thresholds basedon background subtraction BD, as in Eq. (1).
LI and LB indicate the luminance components ofpixel p in the current frame and the backgroundmodel, respectively, and σ is standard deviationof the background model. The ﬁnal silhouetteis extracted from the categorized regions using∗

Copyright is held by the author(s).
Hansung Kim, Ryuuki Sakamoto, Tomoji Toriyama
and Kiyoshi Kogure, Knowledge Science Lab, ATR,Itaru Kitahara, Dept. of Intelligent Interaction Technologies, Univ. of Tsukuba

morphological processes [3]:BD(p) = |LI (p) − lB (p)|BD(p) < K1 σ(p) ⇒ (a)ReliableBackgroundK σ(p) ≤ BD(p) ≤ k2 σ(p) 1
⇒ (b)SuspiciousBackgroundK
2 σ(p) ≤ BD(p) ≤ k3 σ(p)⇒(c)SuspiciousF oreground




Kσ(p)
≤ BD(p) 3⇒ (d)ReliableF oreground(1)

Then the visual hull is constructed by projecting silhouettes from each view into the space using projection matrixes obtained by camera calibration. To avoid damages from outliers, weconsider both the intra- and inter-reliabilities ofthe silhouettes that represent the reliability ofthe segmentation itself and the reliability fromthe relationship to the other silhouettes, respectively. We establish a set of rules to distinguishoutliers in silhouette masks:
Segmentation error: if the projected point ofa certain view is in a suspicious background region and all other projected points are in theforeground regions of each image plane, then thezG

v

Figure 1. Visual hull with outliers

WISS 2007

Figure 2. 3D reconstruction with occlusions

point is an outlier due to segmentation error.
Partial occlusion: if the projected point of acertain view is in a reliable background region
and both projected points in the images of neighbor cameras are in the reliable foreground regions, then the point is an outlier due to partialocclusion.
If the projected point of a voxel in the silhouette is judged to be segmentation error or a partially occluded point, the point is excluded incomputing the visual hull.
In Fig.2, a foreground model is partially occluded by a table in certain views so that the legsof the model were cut oﬀ in the results generated
from a conventional method. In the test set, theresult was aﬀected even by segmentation errors.
However, the proposed method constructed morenatural models by compensating for the outliers.

3

Virtual View Rendering

In the proposed 3D video system, we provide
two methods to generate a ﬁnal video according
to the purpose of rendering. ARToolkit provides
very easy real-time interface for free-view rendering [1]. Viewers can control a ﬁducial marker
and a small USB camera as if they are a miniature set of the 3D space and a real camera toshoot the scene, respectively. In order to changea voxel-based 3D model to mesh model, we usea marching cube algorithm [4]. Figure 3 showsscenes that a user is controlling a USB camera and the generated video from the designatedcamera position. Although we can generate freeviewpoint video by controlling a real camera, thesway of the camera leads to viewers becoming
confused and the ﬁnal video will shows unstable
and naive scene. The second method is oﬀ-line
rendering using a cinematographic virtual camera control system that helps the user to generate ﬁnal videos from the 3D model by referringto the database of experts camera works [2].
Many kinds of camera shots and principal camera actions are stored in the system as expertise.
Therefore, even non-experts can easily convert

Figure 3. Real-time rendering

Figure 4. Rendering with camera works

the 3D video to attractive movies with the helpof the system s expertise. Figure 4 shows anexample of a generated 3D video with camera
controls in the Cinematized Reality system.

4Conclusion

We proposed a 3D construction technique using multiple videos in a real environment wheresegmentation errors and occlusion exist. Then,
we provided two methods for generating ﬁnal 3D
videos by user s interaction. Future work will
include improving reconstruction speed. We are
now implementing the algorithm on GPU for acceleration.

Acknowledgement
This research was supported by the National
Institute of Information and Communications Technology.

References[1] http://www.hitl.washington.edu/artoolkit/.
[2] H. Kim, R. Sakamoto, I. Kitahara, andK. Kogure. Cinematized Reality: Cinematographic 3D Video System for Daily Life Using Multiple Outer/Inner Cameras. In Proc.
3DCINE workshop in CVPR 2006, p. 168, 2006.
[3] H. Kim, R. Sakamoto, I. Kitahara, T. Toriyama,and K. Kogure. Robust Silhouette Extraction
Technique Using Background Subtraction. InProc. MIRU 2007, pp. 552–557, 2007.
[4] W. E. Lorensen and H. E. Cline. Marching
Cubes: A high resolution 3D surface construction algorithm. Computer Graphics, 21(4):163–169, 1987.