NAIST-IS-MT1251124

Master’s Thesis

Free-viewpoint Image Generation-based Human
Motion Reenactment from a Single RGB-D
Video Stream

Fabian Lorenzo Dayrit

March 13, 2014

Department of Information Science
Graduate School of Information Science
Nara Institute of Science and Technology

A Master’s Thesis
submitted to Graduate School of Information Science,
Nara Institute of Science and Technology
in partial fulfillment of the requirements for the degree of
MASTER of ENGINEERING
Fabian Lorenzo Dayrit
Thesis Committee:
Professor Naokazu Yokoya
Professor Hirokazu Kato
Associate Professor Tomokazu Sato
Assistant Professor Yuta Nakashima(Supervisor)(Co-supervisor)(Co-supervisor)(Co-supervisor)

Free-viewpoint Image Generation-based Human
Motion Reenactment from a Single RGB-D
Video Stream∗
Fabian Lorenzo Dayrit

Abstract
Videos are invaluable for trainers demonstrating complex actions. They enable
learners to study the motion of these actions at any time. However, conventional
video is only 2-dimensional. Generally, viewers are able to compensate for theloss of the third dimension by making good inferences about the motion of theaction. However, if some ambiguity is present in the action, the motion may
become harder to comprehend.
Free-viewpoint image generation techniques attempt to solve this problem byallowing viewers to watch the action from any viewpoint they choose. However,most existing free-viewpoint image generation systems also require the action tobe simultaneously captured from multiple viewpoints. Such systems require large
camera arrays and are diﬃcult to set up.
This thesis proposes a system to generate free-viewpoint images of humanmotion using a single RGB-D sensor by synthesizing viewpoints other than the onethat was originally captured. Our system is composed of two stages: capturing the
motion of a human subject’s action, and displaying a reenactment of the action,
which is a free-viewpoint rendering of the action using augmented reality. To
generate an image from a novel viewpoint, we find the most appropriate texture
among our captured frames based on the subject’s pose and viewpoint similarity
and apply this texture to a rough three-dimensional model.
∗

Master’s Thesis, Department of Information Science, Graduate School of Information
Science, Nara Institute of Science and Technology, NAIST-IS-MT1251124, March 13, 2014.

i

We have implemented a prototype which runs on a mobile computer. We haveconducted a user study to determine if i) the system increases comprehension of
3-dimensional motion, ii) the system allows users to compare the reenactment’s
motion to a real human’s motion, and iii) the system has other real-world applications.

Keywords:
Augmented reality, free-viewpoint image generation, human motion capture/reenactmentii

単一 RGB-D ストリームに基づく
自由視点画像生成による人物動作の再現∗
Fabian Lorenzo Dayrit

内容梗概
舞踊などのように特定の動作を学習する場合, 指導者の動作見本の映像記録は,学習者がいつでもその動きを学習することが可能となることから, 極めて有用である. しかし, 通常の映像は三次元の動きを二次元に投影して記録するため, 学習者が映像のみから複雑な三次元の動きを推測することが困難な場合がある.
この問題に対して, 自由視点画像生成技術を利用することで, 学習者が任意の視点からその動きを観測することを可能にするシステムが提案されている. これらのシステムは, 通常の映像では失われていた奥行方向の動きについても, 視点を変更することにより観測可能となる. しかし, このようなシステムは多くの場合,対象となる人物のすべての方向からの見えを記録するために複数の視点から撮影された映像や対応する奥行マップが必要となることから, 一般の指導者や学習者が手軽に利用することはできない.
そこで本研究では, 単一の RGB-D センサーのみを利用して動きを撮影することにより, 任意の視点からその動きを観測可能にするシステムを提案する. 単一
視点から撮影された映像には, 例えば対象の人物の背面からの視点の画像を生成する場合に, テクスチャが存在しないという問題が生じる. 提案システムでは, 撮影されたすべてのフレームから最も適切なフレームをテクスチャとして利用することでこの問題を解決する.
実験では, 事前に撮影された人物動作に基づいて, モバイルデバイス上でリア
ルタイムに任意の視点の画像を生成するプロトタイプを作成し, ユーザスタディ
より提案システムの有用性を検証する.
∗

奈良先端科学技術大学院大学 情報科学研究科 情報科学専攻 修士論文, NAIST-IS-MT1251124,
2014 年 3 月 13 日.

iii

キーワード拡張現実, 自由視点画像生成, 人物動作の取得・再現

iv

Contents
1. Introduction

1

2. Related Literature
2.1 Learning motion with AR . . . . . . . . . . . . . . . . . . . . . .
2.2 Free-viewpoint image generation . . . . . . . . . . . . . . . . . . .
2.2.1 Using RGB frames . . . . . . . . . . . . . . . . . . . . . .
2.2.2 Using RGB-D sensors . . . . . . . . . . . . . . . . . . . . .
2.2.3 Using human pose estimation . . . . . . . . . . . . . . . .
2.3 Summary and relation to our work . . . . . . . . . . . . . . . . .

4
4
6
6
7
8
9

3. The AR Reenactment System
3.1 Overview of AR reenactment system . . . . . . . . . . . . . . . .
3.2 Capturing stage . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2.1 RGB-D sensor pose estimation . . . . . . . . . . . . . . . .
3.2.2 Motion capture . . . . . . . . . . . . . . . . . . . . . . . .
3.2.3 Rough 3D model building . . . . . . . . . . . . . . . . . .
3.3 Reenactment stage . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3.1 Viewer camera pose estimation . . . . . . . . . . . . . . .
3.3.2 Appropriate texture selection . . . . . . . . . . . . . . . .
3.3.3 Applying the 3D model . . . . . . . . . . . . . . . . . . . .
3.3.4 Applying texture . . . . . . . . . . . . . . . . . . . . . . .

12
12
15
15
15
16
18
18
18
20
20

4. Implementation

25

5. Experimental Results
5.1 Reenactment results . . . . . . . . . . . . . . . . . . . . . . . . .
5.2 User study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

28
28
31
31
36

6. Conclusion

38

Acknowledgements

40v

References

41vi

List of Figures
1

2

3
4

5
6
7
8
9
10

11
12
13

Free-viewpoint image generation for human action [1]. A person isrendered from arbitrary viewpoints for the purpose of generating
character animation. . . . . . . . . . . . . . . . . . . . . . . . . .
1
Examples of capturing setups for free-viewpoint image generation
systems for moving scenes. Most capture from multiple cameras,such as [2] (top row) and [3] (bottom row). . . . . . . . . . . . . .
2
Examples of an RGB frame captured by a user (left) and a reenactment by the proposed system (right). . . . . . . . . . . . . . .
3
The post-stroke rehabilitation system proposed by Hondori et al [4].
Users are directed by the system to perform specific hand motions.
In the first column, users reach for boxes to play sounds; in thesecond column, users pour virtual water on the blue spot; in thethird column, users grasp randomly positioned circles. . . . . . . .
4Dancing with a robot [5]. The robot carries a screen that simulates
the appearance of a missing dancer. . . . . . . . . . . . . . . . . .
5
The AR instruction system proposed by Henderson and Feiner [6].
The AR arrows instruct the user on how to move in 3D space. . .
6AR “magic mirror” [7]. The ideal motions are overlayed on top ofa mirror image of the user. . . . . . . . . . . . . . . . . . . . . . .
7
Synthesizing a new viewpoint of a dance [2]. Left and right frames
are interpolated to produce the image in the center. . . . . . . . .
8Detailed model and motion of a subject captured by De Aguiar etal. [8] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9Human pose estimation and free-viewpoint image generation system by Carranza et al. [9] First column: general 3D model. Second
column: two frames of input. Third and fourth column: 3D model
fit to the input. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Capturing two users with three handheld Kinects simultaneously [10]. 11
Overview of the proposed system. . . . . . . . . . . . . . . . . . . 13
Coordinate systems and the transformations relating them. . . . . 14

vii

14

15
16

17

18
19
20
21
22
23
24
25
26
27

(a) The skeleton representation of the subject. Circles are joints,
and segments are body parts. (b) Corresponding depth frame withdefinitions of some angles. (c) Rectangles fitted to each body part.
Vector transformations. . . . . . . . . . . . . . . . . . . . . . . . .
First column: cylinder model. Second column: Texturing using
only the head body part’s projection. Third column: Texturingwith individual body part projection. . . . . . . . . . . . . . . . .
Left: cylinder model in the viewer’s camera. Right: RGB frame
for coloring with overlaid cylinders. Ω is the transformation that
relates them. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Rotating a cylinder so that it faces front. . . . . . . . . . . . . . .
Our representation of the skeleton. Circles: joints, rounded rectangles: body parts. . . . . . . . . . . . . . . . . . . . . . . . . . .
Left: frame from an RGB video. Right: equivalent reenactment
with double-limb error. . . . . . . . . . . . . . . . . . . . . . . . .
Comparison of the same frame of a reenactment in our system from
diﬀerent angles. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Left column: sample frames from an RGB video. Right column:
equivalent reenactments. . . . . . . . . . . . . . . . . . . . . . . .
Left: misregistered pose applied to the cylinder model. Right:
texture applied to the misregistered pose. . . . . . . . . . . . . . .
Left column: sample frames from the RGB video that we used in
the user study. Right column: equivalent reenactments. . . . . . .
Left column: sample frames from the RGB video that we used in
the user study. Right column: equivalent reenactments. . . . . . .
Left column: sample frames from the RGB video that we used in
the user study. Right column: equivalent reenactments. . . . . . .
Evaluation results for questions our user study. 1 means strong
disagreement and 5 means strong agreement. . . . . . . . . . . . .

16
19

21

22
23
26
29
29
30
31
32
33
34
37

List of Tables
1

Questions asked in our user study (originally in Japanese). . . . .
viii

35

1. Introduction
In recent years, watching and sharing videos over the Internet has become more
popular, thanks to video sharing services such as YouTube. Anyone can use
these services to share their own videos as well as explore what others haveshared. Among the types of videos shared are instructional videos, which arevideos which aim to educate the viewer. A subset of these are videos of people
demonstrating some physical action: perhaps a martial arts kick, or a diﬃcult
dance move.
These videos are convenient because learners and instructors do not have to
be in the same room together. Anyone can watch them at any time. However,they do have a drawback. A video holds a two-dimensional (2D) projection of the
motion that was originally three-dimensional (3D). Normally, this is fine, as wehumans can usually infer motion, even from a 2D video. If there is any ambiguityin the video, however, it may become diﬃcult to comprehend the motion.
Perhaps augmented reality (AR) can oﬀer a solution. AR is the integrationof virtual elements with the real world [11]. AR applications augment a realenvironment in some way in order to provide users with a more entertaining or
educational experience (e.g., [6, 7]).
Free-viewpoint image generation techniques also tackle this problem by rendering the video, or a part of the video such as its subject, from an arbitrary
viewpoint (e.g., Fig. 1). However, in order to render moving scenes, most of thesesystems require simultaneous capture from multiple viewpoints (Fig. 2). Thismeans that users must set up multiple cameras, which prevents the average userfrom being able to capture scenes with the system.

Figure 1: Free-viewpoint image generation for human action [1]. A person
is rendered from arbitrary viewpoints for the purpose of generating characteranimation.
1

Figure 2: Examples of capturing setups for free-viewpoint image generation systems for moving scenes. Most capture from multiple cameras, such as [2] (toprow) and [3] (bottom row).
This thesis will discuss a special case of free-viewpoint image generation, focusing solely on the human subject of the video. From now on, we will referto such a synthesis of a moving human subject as a reenactment. As a special
case of free-viewpoint image generation, a reenactment may also be viewed froman arbitrary viewpoint. If this viewpoint is not one of the original capturing
viewpoints, it must synthesize the most likely appearance of the subject fromthe viewpoint. Finally, a reenactment must follow the recorded motions of thesubject in the video sequence.
We propose a system that uses a single RGB-D sensor to capture and record2Figure 3: Examples of an RGB frame captured by a user (left) and a reenactmentby the proposed system (right).
a moving subject, then generates an AR reenactment to be overlaid on top of an
RGB stream (Fig. 3). In our system, a user captures and records the subject withan RGB-D sensor, and the data is stored in a database. Upon a viewer’s request,
the motion is reenacted based on the stored data. Finally, the reenactment isoverlaid on top of the viewer’s camera.
Requiring only a single RGB-D sensor makes our proposed system easy to use.
The challenge is synthesizing a good reenactment from our single-sensor stream,
because multiple viewpoints of the subject are necessary. Our observations in thisregard are that (i) the subject’s rough 3D model can be generated based on thedepth frame and the pose of the subject in that frame, and that (ii) the subject
may, over the course of the entire recording, expose diﬀerent angles of himself or
herself to the camera, which we may then use to color the rough 3D model.
This thesis is organized as follows. In section 2 we will discuss systems that are
related to our proposed system: AR learning systems, and free-viewpoint image
generation systems, with a focus on those that make use of RGB-D sensors andhuman pose estimation. In section 3 we will outline and describe each section of
our proposed system in detail. We will cover capturing the motion of a subject
and rendering the reenactment. In section 4 we will go into the specifics of how
we implemented the system, including the hardware and software we used. In
section 5 we will describe our user study and discuss its results. We conclude ourthesis in section 6.
3

Figure 4: The post-stroke rehabilitation system proposed by Hondori et al [4].
Users are directed by the system to perform specific hand motions. In the firstcolumn, users reach for boxes to play sounds; in the second column, users pourvirtual water on the blue spot; in the third column, users grasp randomly positioned circles.

2. Related LiteratureThis section discusses related research work. We discuss AR systems with asimilar focus on human motion. We will then discuss the diﬀerence between our
proposed system and existing free-viewpoint image generation systems. We also
take a special look at free-viewpoint image generation systems that utilize RGB
cameras, RGB-D sensors, and human pose estimation.

2.1 Learning motion with AR
AR is the combination of a scene from the real world with a virtual object.
Our system aims to display a virtual subject with AR. There exist similar AR
applications that were developed to provide viewers with an augmented view ofmotion in order to aid users’ learning.
Hondori et al. [4] propose a system for users who have suﬀered a stroke. The
rehabilitation process usually requires repetitive motion on the part of the user.
The system makes the user perform motions that are commonly used in daily
life, such as reaching, grasping, and pointing. The AR portion of the systemgenerates virtual targets for the user to interact with (Fig. 4).
Tsuchida, Terada, and Tsukamoto [5] propose a learning support system
specifically for dancers in a formation. In the place of a missing dancer, they4

Figure 5: Dancing with a robot [5]. The robot carries a screen that simulates theappearance of a missing dancer.
used a self-propelled robot with a screen displaying the appearance of the dancer(Fig. 5). The robot moves in space according to how the dancer would have
moved. Users who tried the system danced more accurately, i.e. closer to theactual trajectory, with the robot than without.
The system proposed by Henderson and Feiner [6] shows the user instructions
on how to do a specific procedure, by way of arrows and labels in 3D space
attached to key objects (Fig. 6). Users wore an optical see-through head-mounted
display, and the instructions were overlayed on top of their view. Users who weresurveyed preferred to use the proposed system over an instructional video withsimilar content displayed on an LCD monitor.
The YouMove system [7] first records and tracks the motion of the subject
using an RGB-D sensor. Afterwards, viewers stand in front of a “magic mirror”(Fig. 7) and the system overlays the subject’s motions on their reflection. In thisway, viewers can more easily copy diﬃcult motions. The system also provides acomparison between the subject and a viewer using 3D stick figures, which theviewer can rotate in order to view the motions from diﬀerent directions. However,5Figure 6: The AR instruction system proposed by Henderson and Feiner [6]. The
AR arrows instruct the user on how to move in 3D space.
as useful as they are, stick figures lack realism and naturalness.

2.2 Free-viewpoint image generation
Considering this, free-viewpoint image generation techniques can be used to provide complete renderings of video or parts of video. Free-viewpoint image generation systems use various methods in order to generate images of a scene fromarbitrary viewpoints, including using RGB cameras, using RGB-D sensors, andusing human pose estimation. This section introduces some of the research workthat has been done on these techniques.
2.2.1 Using RGB frames
Several free-viewpoint image genration methods generate a 3D scene from multiple RGB frame captures of the same scene. Some of the first developed of thesesystems used the image-based visual hull (IBVH) [12]. IBVH-based techniques
build a 3D model of an object by capturing it from multiple viewpoints and integrating the sillhouettes in each image. The original IBVH system was used
for static objects. However, Würmlin et al. [13] use this technique in order togenerate free-viewpoint image sequences of a moving subject. Since the subject
was in motion, his sillhouette changed every frame. To resolve this problem, they
made use of multiple cameras, all capturing the subject simultaneously.
Similar to these are the systems based on voxels and marching cubes [14, 15]:
Matsuyama and Takai [16] and Starck, Miller, and Hilton [1]. These systems
first generate, using multiple cameras, a voxel representation of the subject, and6

Figure 7: AR “magic mirror” [7]. The ideal motions are overlayed on top of amirror image of the user.
then convert it into a 3D mesh using the marching cubes algorithm. The mesh iscolored using view dependent textures in [16], while [1] blends the RGB frames
from each camera into one integrated texture.
Another class of these systems interpolates captured views in order to generate
novel ones. Zitnick et al. [2] and Karsten et al. [17] segment frames into layers
and then blend the layers captured from two cameras in order to render an image
from a virtual viewpoint that is somewhere in between the two cameras.
2.2.2 Using RGB-D sensors
Other systems use a combination of depth and RGB data in order to generate
free-viewpoint images. Dai and Yang [18] capture and render a subject in realtime, from an arbitrary viewpoint, using multiple RGB-D sensors. Each sensor’s
7

Figure 8: Synthesizing a new viewpoint of a dance [2]. Left and right frames areinterpolated to produce the image in the center.
foregound layer is merged to produce the final result. Alexiadis, Zarpalas, andDaras [19] also capture a dynamic scene using multiple RGB-D sensors. Eachsensor’s output is converted into 3D meshes and merged, taking care to remove
redundant polygons.
Depth data also can be converted into 3D point clouds. Point clouds frommultiple viewpoints can be integrated to form a representation of the entire scene.
Waschbüsch et al. [3] capture an RGB-D stream from projector-camera combinations around the scene (Fig. 2 (bottom)), and then convert the color and depth
data into 3D points, which they then integrate. Kainz et al. [20] also make use ofmultiple RGB-D streams. They use a combination of point cloud integration and
IBVH. The point cloud method usually produces noisy edges and IBVH does notdetect concavities, but an intersection of the two produces a model with cleanedges and proper concavities.
Other interesting systems include De Aguiar et al. [8], who in addition tousing RGB frames, also use a laser scanner to construct a 3D mesh model of thesubject in advance, which is similar to using a depth sensor. They then capture
the subject’s motion and use keypoints in each frame to transform the model.
In order to locate the 3D positions of the keypoints, they capture from multiple
cameras simultaneously. Using this method, they are able to capture a detailed
mesh with motion (Fig. 9).
2.2.3 Using human pose estimation
Some free-viewpoint image generation techniques dedicated to synthesizing novel
images of human actions also use human pose estimation at some point.
8

Figure 9: Detailed model and motion of a subject captured by De Aguiar et al. [8]Carranza et al. [9] first initialize a general 3D model to the body shape of a
subject (Fig. 10). Then, by capturing that subject using multiple cameras, they
are able to obtain sillhouettes from multiple viewpoints over multiple frames. For
each frame, they then find the pose of the 3D model that fits to each sillhouette.
Shotton et al. [21] developed an algorithm that estimates human motion inreal time for the Microsoft Kinect. This was used for the systems proposed by
Ye et al. [10] and Malleson et al. [22]. In [10], three Kinects are used in order togenerate free-viewpoint images of human motion (Fig. 11). Each Kinect capturesa point cloud of the scene, similar to the systems above. In order to correctly
integrate the point cloud, they use a number of constraints, such as the extrinsic
parameters of each Kinect, and the pose of the subjects. Using this method,they are able to generate free-viewpoint image sequences of up to two subjects.
In [22], on the other hand, only a single Kinect is used. They build a 3D modelof a subject using voxels and apply the subject’s motion to the model in order togenerate a free-viewpoint image sequence. To accomplish this, they capture thesubject’s pose in each frame, and then assign voxels to defined body parts.

2.3 Summary and relation to our work
In this section we discussed ways to possibly increase users’ comprehension of
motion, for the purpose of learning. The two methods that we focused on wereAR and free-viewpoint image generation. AR systems are useful for comparisons
between virtual and real world objects. Free-viewpoint image generation systems
are useful for viewing diﬀerent perspectives of the same object.

9

Figure 10: Human pose estimation and free-viewpoint image generation system
by Carranza et al. [9] First column: general 3D model. Second column: twoframes of input. Third and fourth column: 3D model fit to the input.
Our proposed system renders reenactments, which are free-viewpoint imagesof a human subject, e.g. the output of [1, 8, 9]. Additionally, so that we are notforced to use multiple cameras, we also propose a method to capture reenactmentswith a single RGB-D sensor, using human pose estimation. The advantage of
our system compared to existing AR-based learning support systems is the factthat we synthesize the complete appearance of the subject that the viewer is
attempting to copy. The existing systems are more abstract; this means that
their output is not so realistic or natural. In the next part we will go into thedetails of our AR reenactment system.

10

Figure 11: Capturing two users with three handheld Kinects simultaneously [10].

113. The AR Reenactment System
In this section, we will discuss the details of our proposed AR reenactment system.
We will go into how a user can capture the motion of a subject, and how a viewer
can view the synthesized reenactment.

3.1 Overview of AR reenactment system
As shown in Fig. 12, the proposed AR reenactment system consists of two stages:
capturing stage and reenactment stage.
During the capturing stage, a user of the proposed system captures the motionof a subject using a single RGB-D sensor, resulting in a stored video stream
consisting of NV video frames. The RGB-D sensor’s pose is estimated using avisual SLAM technique [23] in an arbitrarily set world coordinate system (Fig. 13).
Each video frame consists of an RGB frame, a depth frame, and the subject’s
estimated pose, which is represented by the positions of a predefined set of joints,
called a skeleton, in the world coordinate system. We then generate a rough3D model of the subject based on the skeletons and the depth frames. The 3Dmodel consists of one cylinder per body part and only defines their shapes; wewill assign their position and rotation during the reenactment stage. The video
stream and the 3D model are stored in a database for later use in the reenactment
stage. Additionally, the SLAM technique generates a set of map points in order totrack the camera’s pose. We store those so we can use the same world coordinate
system in the reenactment stage.
During the reenactment stage, we synthesize an image of the subject fromthe stored video stream from the viewpoint of a viewer’s camera. The pose of
the viewer’s camera is estimated using the visual SLAM technique and the 3D
map stored in the database. We next synthesize the k-th frame of the subject’s
reenactment by applying the rough 3D model which we generated during thecapturing stage to the k-th skeleton in the stored video stream, and afterward
texturing it with the RGB frame from an appropriate stored video frame. Finally,
the proposed system presents the reenactment superimposed on the real-time
RGB frame captured by the viewer’s camera.
Since the proposed system involves three coordinate systems, i.e., the world,
12Capturing stage

Reenactment stage

RGB-D sensor(input)Viewer camera(input)RGB-D sensor
pose estimation

Viewer camera
pose estimation

Motion capture

Appropriate
texture selection

Rough 3D modelbuilding

Applying
the 3D model

Database

Applying
textureReenactment(output)

Figure 12: Overview of the proposed system.

13

RGB-D sensor
for capturing stage

Transformation
matrix Mn
Worldcoordinate system
Viewer's camera
for reenactment stage

Transformation
matrix M*

Figure 13: Coordinate systems and the transformations relating them.

14the RGB-D sensor’s, and the viewer’s, we need to transform between them(Fig. 13). The subject’s skeleton is initially in the RGB-D sensor’s coordinate
system and is transformed into the world coordinate system before it is stored
in the database. This transformation is done using transformation matrix Mn ,which is obtained through camera pose estimation. For synthesizing the subject’s
reenactment, the stored skeleton is converted from the world coordinate system
to the viewer camera’s coordinate system. The transformation matrix used forthis conversion is M∗ , which is again obtained using camera pose estimation.

3.2 Capturing stage
We first estimate the RGB-D sensor’s pose in the world coordinate system. Next,
we track and capture the subject’s skeleton based on the depth frame obtained
from the RGB-D sensor. Lastly, we generate a rough 3D model of the subject’s
body using the stored video stream.
3.2.1 RGB-D sensor pose estimation
As we mentioned, our RGB-D sensor may move while capturing. Therefore, we
set an arbitrary world coordinate system and track the sensor’s pose within it. For
this purpose, we employ a visual SLAM system (e.g. PTAMM [23]) to constantly
track our RGB-D sensor’s translation and rotation. For the n-th video frameof the video stream, we estimate the RGB-D sensor’s pose as extrinsic camera
parameters Mn .
3.2.2 Motion capture
Figure 14 (a) shows the NJ joints and NBP body parts that compose a skeletonof the subject, where a body part is the segment formed by a specific pair ofjoints. The skeleton of the subject’s body in the n-th frame can be extracted andtracked using a human pose estimation technique, e.g., [21]. Assuming a single
subject in the scene, we denote the skeleton in the n-th frame by
Sn = {sn,i |i = 1, . . . , NJ },

15

(1)

θRn θLn

ϕRn ϕL
n

(a)

(b)

(c)

Figure 14: (a) The skeleton representation of the subject. Circles are joints, andsegments are body parts. (b) Corresponding depth frame with definitions of someangles. (c) Rectangles fitted to each body part.
where sn,i is the 3D position of the i-th joint of the skeleton. Since human pose
estimation gives the joints positions in the RGB-D sensor’s coordinate system,
shown in Fig. 13, using the inverse of Mn , which maps a 3D coordinate from theworld coordinate system to the RGB-D sensor’s coordinate system, we transformthe 3D joint positions in Sn by s0n,i = M−1
n sn,i for all i in Sn and define theskeleton in the world coordinate system as S0n = {s0n,i |i = 1, . . . , NJ }. We store
the n-th video frame, i.e., skeleton S0n , the RGB frame In , and depth frame Dn
in the database.
3.2.3 Rough 3D model building
To render the reenactment of the subject, we use a 3D model of the subject.
In this step, we build the shape of the 3D model. In this work, we chose torepresent each body part with cylinders. Since the heights of the cylinders are
trivially determined from the length of the body part in the skeleton of a stored
video frame, we only need to determine the radii of the cylinders. For this, we firstfind the index of a single representative frame n̂ from the stored video stream andthen fit rectangles to the subject’s region in the depth frame of the representative
16

frame Dn̂ , which can viewed as a projection of the cylinders onto the image planeof the RGB-D sensor.
To easily obtain radii and heights of the cylinders based on the rectangles
that can be considered as the cylinders’ projection, the directions of their heights
should be perpendicular to the optical axis of the RGB-D sensor and body parts
must not overlap. This means that the representative frame should contain thesubject’s appearance that meet the following requirements: (i) both arms should
be away from the body, (ii) the line segments formed by the joints corresponding
to both hands should be parallel to the image plane as possible, and (iii) the legsshould be uncrossed. The camera should, as much as possible, not be rotated for
capturing the representative frame.
These requirements ensure that the representative frame has body parts thatare separate from each other as shown in Fig. 14 (a), which makes it easier tobuild an accurate 3D model of the subject’s body. Such a pose may be requestedby the user who captures the action, but it may also be captured during thenormal course of recording. Using notations as in Fig. 14 (b), these requirements
can be empirically encoded as
LF (n) = θnL lnL + θnR lnR + λg(φRn , φn ).

(2)

The first and second terms, θnL lnL and θnR lnR , reward poses where the subject holds
his/her left and right arms out to the side and parallel to the image plane, as θnL
and θnR are the angles between the torso and the left and right arms in S0n , and lnL
and lnR are the x-components of the normalized left and right arm vectors. The
L
third term g(φRn , φn ), which rewards poses with legs uncrossed, is defined as1 : if φR > φLnn
RLg(φn , φn ) =.
(3)
0 : otherwise
λ is an empirically-defined constant.
We obtain the index of the most appropriate frame in the sense of the above
criterion by maximizing F , i.e.,n̂ = arg max F (n).

(4)

n

We then find the rectangles that fit each body part in Dn̂ , as in Fig. 14 (c).
Radius ra,b of the cylinder for the body part with joints a and b is then given17

as the length of the line segments perpendicular to the body part segment. For
compensating the diﬀerences in the body part segment length from frame toframe, we store in the database the radius ratio given by wa,b /ksn̂,a − sn̂,b k foreach body part, where wa,b is the width of the fitting rectangle.

3.3 Reenactment stage
In the reenactment stage, a viewer with a mobile device captures an RGB-only
video stream from a camera installed in the device. The proposed system synthesizes the reenactment sequentially for this real-time stream. For each frame,
we estimate extrinsic camera parameters M∗ of the viewer’s camera again by the
SLAM technique based on the 3D map stored in the database. The reenactment
stage then transforms the skeleton to the viewer’s camera coordinate system using M∗ , applies the rough 3D model stored in the database, and colors it basedon the appropriate RGB frame in the database. Finally, the reenactment is superimposed on the frame of real-time RGB video stream to be displayed on theviewer’s mobile device.
3.3.1 Viewer camera pose estimation
In the reenactment stage, we track the viewer camera’s pose in order to convert
the subject’s skeleton from the world coordinate system to the viewer camera
coordinate system. In order to use the same world coordinates as in our capturing
stage, we use the 3D map generated during the capturing stage.
3.3.2 Appropriate texture selection
Since our 3D model of the subject is very rough and no color is assigned to it,
we apply textures to our 3D model. For a static scene, view-dependent texture
mapping proposed by Debevec et al. [24] works well for this purpose by assigningas textures those images which were captured from the viewpoint close to thatof the novel image to be synthesized. But we cannot adopt it naively because
the proposed system captures a moving subject and uses only a single RGB-D
sensor and thus there are no RGB frames that show the same scene at the sametime from diﬀerent viewpoints. Our idea for solving this problem is based on our18RGB-D sensor
coordinate system(frame k)Worldcoordinate system

sk
s'n_

s'k

sn_

s*k

RGB-D sensor
coordinate system_(frame n)Viewer camera
coordinate system

Figure 15: Vector transformations.
observation that there may be RGB frames that show the subject in a similarpose, which means that we can select a frame such that the joint positions in theselected frame are close to those in the novel image to be synthesized.
When reenacting the subject’s appearance from the k-th skeleton, S0k , wefirst transform joint s0k,i in the world coordinate system into the viewer camera’s
coordinate system using M∗ , giving us S∗k . We also transform S0n for all n into
its original RGB-D sensor’s coordinate system using Mn , giving us Sn . Figure 15
shows the three coordinate systems and the Since the position of the subject in theworld coordinate system diﬀers frame by frame, to make the selection translation
invariant, the position of a specific joint is subtracted from each joint’s position in19

order to make the specific joint coincide with the origin. In this work, we choose
the neck joint shown in Fig. 14 (a) as the origin. We select the appropriate stored
video frame, where the associated skeleton Sn (in the original RGB-D sensor’s
coordinate system) is closest to the S∗k (in the viewer camera’s coordinate system).
To summarize, we find appropriate frame index n̄ byn̄ = arg minnNJ∑k(s∗k,i − s∗k,neck ) − (sn,i − sn,neck )k,

(5)

i=1

where s∗k,neck and sn,neck are the neck joint position of S0k and S0n , respectively.
This method of texture selection does not preserve the subject’s facial expression. For our purpose, however, it is suﬃcient to make the subject’s motion
comprehensible.
3.3.3 Applying the 3D model
We now render the cylinder model based on the radius ratios stored in the
database, the frame in the sequence that we are reenacting, and the joints inthe viewer camera coordinate system. For the body part associated with joint
indices a and b, the cylinder axis endpoints will be at s∗k,a and s∗k,b . The radius
will be the length of the cylinder axis multiplied by the radius ratio computed
earlier: rk,a,b = ks∗k,a − s∗k,b kra,b , where ra,b is the radius ratio associated with the
body part.
3.3.4 Applying texture
Now that we have selected an appropriate RGB frame, we can apply it to thecyinder model. However we cannot just copy the texture naively. Since the poses
represented by S∗k and Sn̄ are not exactly the same, naively copying the cylinder tothe selected RGB frame can lead to inconsistency between the cylinders and theframe (Fig. 16). We thus find transformation Ωk,n̄ individually for each cylinderthat compensates for the diﬀerence between the subject’s poses in S∗k and Sn̄ . We
apply this transformation to each point on the cylinder to find its corresponding
pixel location on the RGB frame, and determine its color from that pixel.
We want to transform the cylinder with axis endpoints s∗k,a and s∗k,b and radius
rk,a,b , which the cylinder in viewer camera coordinates, into the cylinder with axis
20Figure 16: First column: cylinder model. Second column: Texturing using only
the head body part’s projection. Third column: Texturing with individual bodypart projection.
endpoints sn̄,a and sn̄,b and radius rn̄,a,b , which is the corresponding cylinder inthe RGB frame we are extracting from (Fig. 17).
All equations from here will refer to the body part formed by joints a andb. We omit the subscript unless we need to refer to one of the two joints inparticular.
First we find the axis vectors vk∗ for the viewer camera cylinder and vn̄ for the
RGB frame cylinder:vk∗ = s∗k,a − s∗k,b(6)vn̄ = sn̄,a − sn̄,b .
The rotation is then calculated by:
(
( ∗T
))
vk vn̄
∗−1
Rk,n̄ = Θ vk × vn̄ , cos.
kvk∗ kkvn̄ k

(7)

(8)

Θ(w, θ) is the Rodrigues rotation algorithm, i.e. the algorithm that finds therotation matrix given an axis and an angle. The axis is the cross product between
the axis vectors, and the angle is the angle between them. The scale is the ratio
between the lengths of the two vectors:
sk,n̄ =

kvk∗ k,kvn̄ k

21

(9)

Ωk,n
Figure 17: Left: cylinder model in the viewer’s camera. Right: RGB frame forcoloring with overlaid cylinders. Ω is the transformation that relates them.
the translation is the diﬀerence between the joints with index a:tk,n̄ = sn̄,a − s∗k,a ,and the resulting transformation is (Fig. 17):][Ωk,n̄ = sk,n̄ Rk,n̄ tk,n̄

(10)

(11)

Finally, we need to compensate for facing, which means ensuring that theresulting cylinder faces front, i.e. the side of the cylinder that the viewer cansee. This means that we need to find the vector that faces front, which is theunit vector orthogonal to the cylinder axis that is closest to the ray from the axis
endpoint to the viewing point (Fig. 18). We calculate the front vector of the firstcylinder and apply Ωk,n̄ to it to see where it ends up. Then, we calculate theideal facing vector of the second cylinder and rotate it along its axis.
We take the viewing ray from an axis endpoint (e.g., −s∗k,a ). The front vectorof a cylinder is calculated by taking the vector rejection of that viewing ray fromthe cylinder axis:(12)
fk∗ = v̂k∗ × (−s∗k,a × v̂k∗ )Applying Ωk,n̄ to it, we obtain the intermediate facing:0= Ωk,n̄ (s∗k,a + fk∗ )
fk,n̄

22

(13)

vn

Ωk,n

v*k

Q*k,n

fk*

fn

*
sk,a

sn,a

f'k,n

-s*k,a-sn,aViewer's camera

RGB frame n

Figure 18: Rotating a cylinder so that it faces front.
We now calculate the ideal facing for the cylinder on frame n̄, again using thevector rejection:fn̄∗ = v̂n̄∗ × (−s∗n̄,a × v̂n̄∗ )(14)Finally, we rotate the intermediate facing into the ideal facing by using the cylinder axis on frame n̄ and the angle between the two facings:
(
(
))
0 T
f
f
k,n̄ n̄Q∗k,n̄ = Θ vn̄ , cos−1(15)0kfk,n̄ kkfn̄ k
The final transformation is thus, for viewer camera coordinate point p∗k and RGB
frame coordinate point pn̄ :pn̄ = Q∗k,n̄ Ωk,n̄ p∗k(16)
We then apply the intrinsic camera matrix to pn̄ in order to get the pixel
value at (x, y) on the RGB frame. We use our human pose estimation techniquein order to determine if it belongs to the subject, or to the background. In thelatter case, we ignore the pixel.

23

Finally, we superimpose the reenactment on the real-time RGB frame captured from the viewer’s camera.

24

4. Implementation
We implemented the capturing stage of the proposed system on an Intel i7 with a
3.20GHz processor and 32GB of RAM. We used a Microsoft Kinect as our RGBD sensor. We implemented the reenactment stage on a Microsoft Surface Pro 2
with a 1.60GHz processor and 4GB of RAM. We used its embedded camera as
the viewer’s camera. The intrinsic parameters for Kinect and Surface’s cameras
were preliminarily calibrated. We used OpenGL [25] to render the reenactment.
We set constant λ for 3D model building to be 2000.
For camera pose estimation in both capturing and reenactment stages, we employed Parallel Tracking and Multiple Mapping (PTAMM) [26], which is capableof storing the 3D map for later uses, in order that we can use the shared world
coordinate system in these stages. We set the world coordinate system according
to PTAMM.
In order to continuously track the subject’s joints, we made use of OpenNI [27]and NiTE [28] skeleton tracking. Our representation of the skeleton contains 16
joints and 11 body parts (Fig. 19). Unfortunately, the NiTE skeleton tracker
returns joints in the depth sensor’s coordinate system, a diﬀerent coordinate
system from PTAMM. Thus we converted the skeleton joints captured by theskeleton tracker into the coordinate system used by PTAMM. To do this, we
calculate a transformation matrix (i.e., rotation, translation, and scale) between
the two coordinate systems.
Happily, PTAMM tracks a number of map points, i.e. feature points withwhich it estimates the camera pose, as well as their 3D coordinate in the camera
coordinate system. We now query the 3D camera coordinate representation ofeach map point. Next, we take a depth frame captured by our RGB-D sensor,
and for each map point, find the corresponding depth pixel and convert it into
a 3D point in the RGB-D sensor’s coordinate system, according to our sensor’s
driver. We now need to find the transformation between the RGB-D sensor points
and the PTAMM map points. Given that pi is the i-th map point and qi is thecorresponding point projected to the RGB-D sensor’s coordinate system, we findthe transformation, i.e. rotation R, translation t, and scale s, in the sense of

25

1
1
2
6
9

4
3
6

3
2
5

5
8

4

7
11 16 10
9

8

13

12

7

11 10
15

14

Figure 19: Our representation of the skeleton. Circles: joints, rounded rectangles:
body parts.

26

least squares, as:(R̄, t̄, s̄) = arg min(R,t,s)N∑kpi − (sRqi + t)k2 .

(17)

i=1

According to [29], we can obtain R̄ by


1 0
0

 T
R̄ = U  0 1
0V ,T
0 0 det(UV )(18)where U and VT are obtained from the singular value decomposition of thecovariance matrix of the two sets of points:
UΣVT =

N∑

p0i q0 i ,
T

(19)

i=1
0
0
where pi and qi are the points adjusted by their centroid:p0i = pi −

1 ∑
pj ,
N j=1

(20)

q0i = qi −

1 ∑
qj .
N j=1

(21)

N

N

We can use the rotation and the centroid-adjusted points to form another leastsquares equation to find the scale:s̄ = arg minsN∑

kp0i − sR̄q0i k2 ,

(22)

i=1

which we solve by finding the ordinary least squares estimator:N∑
p0i T R̄q0i.
s̄ = i=1N∑
0 2
kR̄qi k(23)

i=1

Lastly, the translation is the diﬀerence between the properly rotated and scaled
centroids.
NN∑∑t̄ = −s̄R̄
qj +
pj
(24)
j=1
∗

j=1

For any point q that we obtain from the RGB-D sensor, we can thus convert it∗to the correct coordinate system by p∗ = s̄R̄q + t̄.
27

5. Experimental Results
We evaluated the system using our prototype, and we also conducted a user study
to find out how users compare the reenactments to conventional video. In thissection we describe the reenactment results, as well as the results of the user
study.

5.1 Reenactment results
Figures 20–26 show the reenactments generated by the proposed system. We cansee that the reenactments generated by the system are coherent, i.e. they are
consistent with regards to the recorded motion, the subject’s appearance, andthe change in viewpoint.
However, there are some major errors that would possibly impede comprehension. The most noticeable error is the doubling of body parts, which can be seenin Fig. 20. This is the result of self-occlusion by the subject. The 3D model coloring process divides the RGB frame into body parts for texturing each cylinder inthe rough 3D model. But the texture extraction does not account for occlusion,
which sometimes results in unwanted objects being included in the texture. At
large changes in viewing angle, we observe a diﬀerent type of error. In the image
on the first column and first row of Fig. 21, the subject’s legs are incomprehensible. Also, large artifacts have appeared in the arm regions. This is the resultof choosing inappropriate RGB frame to color the cylinders, or simply that more
appropriate RGB frames do not exist. This type of error makes it harder to comprehend the output, as the body parts appear to face the wrong way and adjacentbody parts do not combine well. Another type of error can be seen in Fig. 22.
If the environment has changed since the time of capture, the reenactment may
be rendered in the wrong place. Lastly, the skeleton tracker sometimes fails toregister the correct joint positions (Fig. 23). This appears to happen often when
the subject moves quickly or faces away from the RGB-D sensor. The eﬀects are
two-fold. For one, it introduces a skeleton into the sequence that is inconsistent
with the other skeletons. If this skeleton is textured, it produces bad results, asin the figure. For another, the skeleton is misregistered with the associated RGB
frame. If we use that frame as a texture, we risk extracting textures from the28

Figure 20: Left: frame from an RGB video. Right: equivalent reenactment withdouble-limb error.

Figure 21: Comparison of the same frame of a reenactment in our system from
diﬀerent angles.
29

Figure 22: Left column: sample frames from an RGB video. Right column:
equivalent reenactments.

30

Figure 23: Left: misregistered pose applied to the cylinder model. Right: texture
applied to the misregistered pose.
wrong parts of the RGB frame.
On average, the system runs at 9fps on the Surface during reenactment.

5.2 User study
5.2.1 Setup
We asked a martial artist to serve as our subject and captured him performing a
Taekwondo form. We captured the motion with a fixed-position Kinect, considering that users who capture an RGB-D video stream generally do not move while
capturing. Before capturing, we made a 3D map with the PTAMM system. The
resulting RGB-D stream was 23 seconds long and consisted of 580 frames. For
comparison between the proposed system and conventional video, we also compiled the RGB component of our captured stream into a separate video. Figs. 24,
25, and 26 show some example frames from the RGB images, compared with the
AR reenactment generated by our proposed system.

31

Figure 24: Left column: sample frames from the RGB video that we used in theuser study. Right column: equivalent reenactments.

32

Figure 25: Left column: sample frames from the RGB video that we used in theuser study. Right column: equivalent reenactments.

33

Figure 26: Left column: sample frames from the RGB video that we used in theuser study. Right column: equivalent reenactments.

34

Table 1: Questions asked in our user study (originally in Japanese).
#

Questions

Translation

Q1

普通のカメラで記録した人物が環境

Were you able to comprehend the recorded
motion on the conventional video?

中のどこにいるか把握できていると思いますか？

Q2

提案システムで記録した人物が環境中のどこにいるか把握できているとWere you able to comprehend the recorded
motion on the proposed system?

思いますか？

Q3

提案システムでは、視聴位置によって人物の位置の把握しやすさが変化
しますか？

Q4

普通のカメラの映像と比べて、提案
システムで視聴した場合、人物の動作は把握しやすいと思いますか？

Q5

普通のカメラの映像と比べて、提案
システムで生成した画像の質は満足できますか？

Q6

実際の人物の動作と普通のビデオで記録された人物の動作は比較しやすいと思いましたか？

Q7

実際の人物の動作と提案システムで記録された人物の動作は比較しやすいと思いましたか？

Q8

普通のカメラで記録された映像に比べて、提案システムでのパフォーマ

Did it become easier to comprehend therecorded motion on the proposed system
as you moved the perspective?
Was it easier to comprehend the recorded
motion on the proposed system, than on
the conventional video?
Were you satisfied by the quality of theimage generated by the proposed system,compared to the conventional video?
Were you easily able to compare the motions of the real person with the recorded
motion on the conventional video?
Were you easily able to compare the motions of the real person with the recorded
motion on the proposed system?
Is the proposed system more fun to use
than conventional video?

ンスの視聴は楽しいと思いますか？

Q9

提案システムはどのようなアプリ
ケーションに有用だと思いますか、特Do you think that this system would be
useful for learning specific motions?

定の動作の習得？

Q10

提案システムはどのようなアプリ
ケーションに有用だと思いますか、パフォーマンスの視聴？

35

Do you think that this system would be
useful for watching performances?

The user study was in three parts. During Part 1, participants were asked
to watch the 2D conventional video first, and then the reenactment generated
by the proposed system. Both were presented on the Microsoft Surface’s display. The participant was allowed to move around while watching the proposedsystem’s reenactment. This part investigates whether the proposed systems aids
user comprehension of 3D motion. During Part 2, a real person attempted tocopy the motion of the recorded subject. The participants first simultaneously
watched the conventional video and the motions of the real person; afterward,
they simultaneously watched the reenactment and the motions of the real person.
Participants were again allowed to move freely while watching the reenactment.
Part 2 is dedicated to show the eﬀectiveness of the proposed system over theconventional video for learning and training of a specific motion, considering that
if the participants were easily able to see the diﬀerences between an expert anda learner, this would imply an easier learning process. The questions in Part 3
were designed to show the applicability of the proposed system. After watching
the video, they were asked to answer the questions in Table 1. The answers were
multiple choice and free entry. The multiple choice answers were on a scale of 1,
I strongly think not, to 5, I strongly think so.
5.2.2 Results
Figure 27 shows the participants’ answers. For part 1, the answers show that the
proposed system did not improve much from the conventional video. In terms ofpure comprehension, the results of Q2 over Q1 shows that the system improved
slightly. Almost all of the participants agreed that viewing the reenactment froma diﬀerent viewpoint improved comprehension (Q3). Q4 was split almost in half,
meaning that half of the participants rated the proposed system as more comprehensible than conventional video. And for Q5, the participants unanimously
answered in the negative, meaning that nobody was satisfied with the quality of
the reenactment. Some comments from the participants were that the output was
jittery, that the sillhouette wasn’t smooth, and that if they moved drastically, theimage would become corrupted.
In comparisons with a real person for part 2, the proposed system improved
slightly over conventional video (Q6 and Q7). A common comment was that it36

Part 1
Part 2
Part 3

Question #Number of participants
Figure 27: Evaluation results for questions our user study. 1 means strong disagreement and 5 means strong agreement.
was easier to compare the motions of the real human with the reenactment.
Finally, in part 3, most of the participants thought that the system had good
applicability, from Q8–Q10 as well as the comments.

37

6. Conclusion
In this work, we introduce the concept of reenactments for the purpose of learning.
To realize the reenactments, we developed a system that can synthesize a freeviewpoint image of a moving subject using a single RGB-D sensor.
We conducted a user study to gauge the eﬀectiveness of the system for the
purpose of learning a motion. Our results showed that the system was comprehensible when viewed from viewpoints close to the capturing viewpoint. In thesecases, the reenactments were able to synthesize the subject’s appearance and motion close to what was expected from viewing the RGB video. Another strengthof the system was making comparisons in real time. The reenactment and thereal world appear on the same screen, making it trivial for viewers to see thediﬀerence. This is a direct benefit over conventional video, where viewers mustswitch their attention between the screen and the real world.
On the other hand, the system has some limitations. The reenacted motion islimited to what the system can see, meaning that any motions that the subject
does that the sensor does not capture can not be reenacted. Adding to this, theappearance of the subject is drawn from the actual frames captured. This meansthat if, for example, the captured frames do not include a particular viewpoint,
the output may suﬀer when that viewpoint is requested.
Also, the reenactments also contained some rendering errors. The further
away from the original viewpoint, the more errors appeared. Since the motivationof the system was to increase comprehensibility at viewpoints other than theoriginal, this is an undesirable result. We identified some possible causes. First,
the texture selection may be choosing inappropriate textures. We must redefine
the criteria for an appropriate texture. In our current implementation, the twoskeletons being compared must simply be similar distance-wise. However, there
may exist a definition that gives better results. Next, the original RGB frames arebadly aﬀected by self-occlusion. Currently, we use each RGB frame as-is, meaning
we do not perform any processing before using them as textures. This means thatthe cylinder projection for texture extraction simply extracts the whole cylinder
shape. We need to isolate each body part from its neighbors and fill in the gaps
in order to solve this problem. Lastly, the skeleton tracker sometimes fails to
track the correct position of the joints. This happens when the motion is too
38

fast, and sometimes when the subject’s back is to the camera. However, filteringor correcting for its output may improve results.
The system also has the potential to grow in other ways. One way is toreplace the RGB-D sensor used for capturing with an RGB camera, and to doskeleton tracking on the RGB frames. This would increase the accessibility ofthe system by allowing mobile users to capture reenactments. Another way is
to introduce a method to relocate reenactments. Currently, the reenactment
must be rendered over the originally captured environment. Naively relocating it
introduces rotation and scaling errors, so the new environment must be carefully
considered.
During the user study, most of those surveyed answered that the system could
also be applicable to many things, e.g. performance, and this is certainly anotherpossible avenue for future development. However, our ultimate goal is to implement the system as a social reenactment-sharing network. Users will be able tocapture reenactments wherever they want, and potential viewers will be able tosee if a previous user captured a reenactment close to their location.

39

Acknowledgements
First of all, I would like to sincerely thank Professor Naokazu Yokoya for taking
me in and welcoming me to the Media and Vision Computing laboratory. His
generosity gave me the chance to complete my Master’s course in Japan. I am
grateful for all the help and support I received from his laboratory.
I would also like to thank Professor Hirokazu Kato, Professor Tomokazu Sato,and Professor Yuta Nakashima. Thanks to them, I was able to produce better
work. Although I can safely say that I deserved most of the headaches that camemy way, a combination of their good ideas, direction, and some tough love, went
into my research.
Professor Hiroyuki Seki was my introduction to NAIST, and he was very
accomodating towards me. He made my transition here as painless as possible.
He was also head of the international program that I studied under.
The members of the Media and Vision Computing laboratory welcomed me
like a family. Thank you for the fruitful conversations and the good advice, andthe companionship during late nights. Additional thanks go to Professor Norihiko
Kawai and Takahito Aoto for meeting me at at the station when I came to NAIST
and helping me take care of my aﬀairs.
I’d also like to throw “shout-outs” to all my NAIST friends. I met manycolorful characters during my stay here. You’re all super cool. I wish everyone
the best of luck in research (and life). Many thanks also to Max Krichenbauer
and Satoshi Nakao for performing for my data set! You guys got some smoothmoves.
For my Master’s course I was supported by the Docomo scholarship. Thankyou for this wonderful opportunity! I hope to pay it forward to the next generationof dreamers.
Maraming salamat to my family, who seem to have gone above and beyond
the call of duty to support me. It feels like you were right there with me every
step of the way. Thanks always for your love!

40

References[1] J. Starck, G. Miller, and A. Hilton, “Video-based character animation,” inProc. ACM SIGGRAPH Eurographics Symposium on Computer Animation,
pp. 49–58, 2005.
[2] C. Zitnick, S. Kang, M. Uyttendaele, S. Winder, and R. Szeliski, “Highquality video view interpolation using a layered representation,” ACM Transactions on Graphics, vol. 23, no. 3, pp. 600–608, 2004.
[3] M. Waschbüsch, S. Würmlin, D. Cotting, F. Sadlo, and M. Gross, “Scalable 3D video of dynamic scenes,” The Visual Computer, vol. 21, no. 8-10,
pp. 629–638, 2005.
[4] H. Hondori, M. Khademi, L. Dodakian, S. Cramer, and C. V. Lopes, “A
spatial augmented reality rehab system for post-stroke hand rehabilitation,”in Proc. Conference on Medicine Meets Virtual Reality, 2013.
[5] S. Tsuchida, T. Terada, and M. Tsukamoto, “A system for practicing formations in dance performance supported by self-propelled screen,” in Proc.
Augmented Human International Conference, pp. 178–185, 2013.
[6] S. Henderson and S. Feiner, “Augmented reality in the psychomotor phaseof a procedural task,” in Proc. IEEE International Symposium on Mixed andAugmented Reality, pp. 191–200, 2011.
[7] F. Anderson, T. Grossman, J. Matejka, and G. Fitzmaurice, “YouMove:
Enhancing movement training with an augmented reality mirror,” in Proc.
ACM Symposium on User Interface Software and Technology, pp. 311–320,
2013.
[8] E. De Aguiar, C. Stoll, C. Theobalt, N. Ahmed, H.-P. Seidel, and S. Thrun,“Performance capture from sparse multi-view video,” in ACM Transactions
on Graphics, vol. 27, 10 pages, 2008.
[9] J. Carranza, C. Theobalt, M. Magnor, and H. Seidel, “Free-viewpoint videoof human actors,” ACM Transactions on Graphics, vol. 22, no. 3, pp. 569–
577, 2003.
41

[10] G. Ye, Y. Liu, Y. Deng, N. Hasler, X. Ji, Q. Dai, and C. Theobalt, “Freeviewpoint video of human actors using multiple handheld kinects,” IEEE
Transactions on Cybernetics, vol. 43, no. 5, pp. 1370–1382, 2013.
[11] O. Bimber, R. Raskar, and M. Inami, Spatial augmented reality. AK Peters
Wellesley, 2005.
[12] W. Matusik, C. Buehler, R. Raskar, S. Gortler, and L. McMillan, “Imagebased visual hulls,” in Proc. Conference on Computer Graphics and Interactive Techniques, pp. 369–374, 2000.
[13] S. Würmlin, E. Lamboray, O. Staadt, and M. Gross, “3D video recorder,” inProc. Pacific Conference on Computer Graphics and Applications, pp. 325–
334, 2002.
[14] W. B. Culbertson, T. Malzbender, and G. Slabaugh, “Generalized voxel
coloring,” in Proc. International Workshop on Vision Algorithms: Theory
and Practice, pp. 100–115, 2000.
[15] W. Lorensen and H. Cline, “Marching cubes: A high resolution 3D surface
construction algorithm,” in ACM SIGGRAPH Computer Graphics, vol. 21,
pp. 163–169, 1987.
[16] T. Matsuyama and T. Takai, “Generation, visualization, and editing of 3Dvideo,” in Proc. International Symposium on 3D Data Processing Visualization and Transmission, pp. 234–245, 2002.
[17] K. Müller, A. Smolic, K. Dix, P. Merkle, P. Kauﬀ, and T. Wiegand, “Viewsynthesis for advanced 3D video systems,” EURASIP Journal on Image andVideo Processing, 11 pages, 2009.
[18] B. Dai and X. Yang, “A low-latency 3D teleconferencing system with image
based approach,” in Proc. ACM SIGGRAPH International Conference on
Virtual-Reality Continuum and Its Applications in Industry, pp. 243–248,
2013.

42

[19] D. Alexiadis, D. Zarpalas, and P. Daras, “Real-time, full 3-D reconstructionof moving foreground objects from multiple consumer depth cameras,” IEEE
Transactions on Multimedia, vol. 15, no. 2, pp. 339–358, 2013.
[20] B. Kainz, S. Hauswiesner, G. Reitmayr, M. Steinberger, R. Grasset, L. Gruber, E. Veas, D. Kalkofen, H. Seichter, and D. Schmalstieg, “OmniKinect:Real-time dense volumetric data acquisition and applications,” in Proc. ACM
Symposium on Virtual Reality Software and Technology, pp. 25–32, 2012.
[21] J. Shotton, T. Sharp, A. Kipman, A. Fitzgibbon, M. Finocchio, A. Blake,
M. Cook, and R. Moore, “Real-time human pose recognition in parts fromsingle depth images,” Communications of the ACM, vol. 56, no. 1, pp. 116–
124, 2013.
[22] C. Malleson, M. Klaudiny, A. Hilton, and J. Guillemaut, “Single-view
RGBD-based reconstruction of dynamic human geometry,” in Proc. IEEE
Workshop on Dynamic Shape Capture and Analysis, pp. 307–314, 2013.
[23] R. Castle, G. Klein, and D. Murray, “Video-rate localization in multiple maps
for wearable augmented reality,” in Proc. IEEE International Symposium on
Wearable Computers, pp. 15–22, 2008.
[24] P. Debevec, C. Taylor, and J. Malik, “Modeling and rendering architecturefrom photographs: a hybrid geometry- and image-based approach,” in Proc.
Conference on Computer Graphics and Interactive Techniques, pp. 11–20,
1996.
[25] “OpenGL.” http://opengl.org/, 2014.
2014].

[Online; accessed January 28,[26] R. Castle, G. Klein, and D. Murray, “Parallel Tracking and Multiple Mapping source code.” http://www.robots.ox.ac.uk/~bob/research/
research_ptamm.html, 2014. [Online; accessed January 28, 2014].
[27] PrimeSense, “OpenNI.” http://openni.org/, 2014. [Online; accessed January 28, 2014].

43

[28] PrimeSense, “NiTE.” http://www.openni.org/files/nite/, 2014. [Online; accessed January 28, 2014].
[29] W. Kabsch, “A discussion of the solution for the best rotation to relate two
sets of vectors,” Acta Crystallographica Section A, vol. 34, no. 5, pp. 827–828,1978.

44