Paper
Free-viewpoint Mobile Robot Teleoperation Interface
Using View-dependent Geometry and Texture
Fumio Okura (student member)†∗ , Yuko Ueda † , Tomokazu Sato (member)† ,Naokazu Yokoya (member)†
Abstract

This paper proposes a free-viewpoint interface for mobile-robot teleoperation, which provides viewpoints thatare freely configurable by the human operator head pose. The viewpoints are acquired by a head tracker equipped on a head
mounted display. A real-time free-viewpoint image generation method based on view-dependent geometry and texture isemployed by the interface to synthesize the scene presented to the operator. In addition, a computer graphics model of therobot is superimposed on the free-viewpoint images using an augmented reality technique. We developed a prototype system
based on the proposed interface using an omnidirectional camera and depth cameras for experiments. The experiments underboth virtual and physical environments demonstrated that the proposed interface can improve the accuracy of the robot
operation compared with first- and third-person view interfaces, while the quality of the free-viewpoint images generated by
the prototype system was satisfactory for expressing the potential advantages on operational accuracy.

Key words: mobile robot, teleoperation, free-viewpoint image generation, view-dependent geometry and texture

1. Introduction
Historically, many types of mobile robots have beendeveloped and employed to operate on behalf of humans
for various situations2) . The importance of the teleoperation interface has increased signiﬁcantly, particularly
for working in unknown and/or extreme environments,
such as disaster areas with narrow pathways and unforeseen obstacles. Although there has been considerable research devoted to the automatic control of mobile
robots3)4) , most practical robots are still controlled byhuman operators using video images captured by robotmounted cameras. These include PackBot5) , which was
deployed for surveillance of the Fukushima Daiichi nuclear power plant in Japan after the 2011 earthquake.
A human operator should have suﬃcient skills for controlling a robot to prevent it from colliding with its surroundings so it can safely and eﬀectively complete its
assigned tasks. To achieve successful operations, it isimportant to determine the most eﬀective way to represent to its human operators the ﬁeld of view surrounding the robot. This objective is even further essential
This paper is an extended version of the one presented at the 2013
IEEE/RSJ International Conference on Intelligent Robots andSystems1) .
Received August 3, 2013; Revised October 18, 2013; Accepted
November 20, 2013
† Nara Institute of Science and Technology (NAIST)(Nara, Japan)∗ JSPS Research Fellow

because human vision is the most important sense usedby humans to grasp the surrounding environment inteleoperation tasks. To this end, there have been numerous studies on image presentation approaches forteleoperation interfaces of mobile robots6)–17) .
In terms of image presentation for human operators,
existing remote-control interfaces are classiﬁed in two
categories: those that provide a ﬁrst-person view fromthe robot (that is, from the position of the camera on
the robot), and those that provide a third-person view(a bird’s-eye view from above the robot). While theseinterfaces are currently used in practical applications,problems with them still remain that decrease robot
operation safety. One key problem is the diﬃculty ingrasping distances from the robot to surrounding objects. To address this problem, we propose a mobilerobot teleoperation interface that uses a free-viewpoint
image generation technique from computer-vision and-graphics ﬁelds. The proposed interface provides thehuman operator with a novel way to grasp the environment surrounding a robot in a remote site. As demonstrated by our experiments, the interface realizes an intuitive robot operation from viewpoints that are freely
conﬁgurable by a head-mounted display (HMD) and ahead tracker, which provides photorealistic textures ofreal-world environments, as shown in Fig. 1. Our experiments also demonstrate that the proposed interface
improves the safety of mobile-robot operations.

ITE Transactions on Media Technology and Applications Vol. xx, No. xx, pp. 1–12 (20xx)(1) 1

Fig. 1

Left column: Images from the prototype systempresented for the human operator (right column)
based on the free-viewpoint teleoperation interface.

2. Related work
2. 1 Mobile-robot teleoperation interfaces
The most common image presentation approach formobile-robot teleoperation is based on the ﬁrst-person
view6) , which is the scene directly captured by robotmounted cameras. Most studies and robotic products
have employed monocular cameras to provide a ﬁrstperson view for surveillance in environments such as
mineﬁelds7) and sewers8) . Omnidirectional cameras18)
are often used for ﬁrst-person-view interfaces that enable operators to conﬁgure their view direction to scan
the scene. Compared to monocular cameras, omnidirectional cameras have a shorter delay when changing
view direction. However, there are two problems withomnidirectional cameras with regard to the operator’s
understanding of the robot’s surroundings. For one, in
omnidirectional images, scenes in a downward angle are
not visible because the robot occludes them. Second,
it is diﬃcult to grasp distances between the robot andsurrounding objects.
Interfaces that provide third-person views, which areviews from directly above or from above and diagonallybehind a robot, have been developed to overcome theabove problems. Packbot5) and Quince9) , mobile-robots
employed for the surveillance of the Fukushima Daiichi
nuclear power plant, were operated through a thirdperson-view interface using a pair of robots: one moved
forward for surveillance, while the other captured images of the ﬁrst robot from behind. As another exam2 (2)ple, a study by Shiroma et al.10) obtained images of arobot from above by physically mounting a camera on
a long arm. Their investigation demonstrated that, in
terms of speed and safety, the third-person view is more
eﬀective than the ﬁrst-person view. Although these approaches are expected to facilitate grasping distances
to surrounding objects, they do not completely resolve
the occlusion problem that occurs in downward scenes.
In most situations, however, it is diﬃcult to directly
capture a third-person view from physically mounted
cameras; therefore, image processing and/or multisensor integration approaches, which combine information captured from the robot, are often employed
to generate third-person views. Time Follower’s Vision11) , for example, a remote-control vehicle visualpresentation system, provides a viewpoint from behindby displaying images captured several seconds previously as the robot moves forward. To generate morecustomized viewpoints, three-dimensional (3D) shapesof objects in the surrounding environment acquired
from depth sensors mounted on a robot are often usedwith the images captured by a camera. The interfaces proposed by Saitoh et al.12) , Nielsen et al.13) ,and Ferland et al.14) provide operators with both theﬁrst-person view and 3D models reconstructed basedon simultaneous localization and mapping (SLAM) approaches19) . Kelly et al.15) have realized a photorealistic
third-person-view interface by appropriately mapping
images to 3D shapes in an outdoor environment. Imageprocessing-based third-person view interfaces basicallycombine multiple textures and/or 3D shapes captured
from multiple locations to generate the scene at the bottom of the robot, and then artiﬁcially superimpose theappearance of the robot. Therefore, these interfaces canimprove the visibility of scenes of the bottom or far sideof the robot by using a transparent robot appearance,such as the technique used in Time Follower’s Vision11) .
In ordinary third-person-view interfaces describedabove, the operator’s viewpoint is ﬁxed or selectablefrom a few viewpoints that are conﬁgured in advance.
Although these types of third-person-view interfaces
improve the speed and safety of teleoperations, theproblems mentioned earlier still remain with the difﬁculty in grasping the distance to surrounding objects
on the far sides of the robot. Because of the limitationof selectable viewpoint variation, the problems are especially apparent in complex environments, such as thosecontaining narrow pathways and obstacles. From thisITE Transactions on Media Technology and Applications Vol. xx, No. xx (20xx)perspective, one of the ultimate forms of third-personview interfaces is a Virtual Environment Vehicle Interface (VEVI)16)17) , which provides a freely conﬁgurable
view; the viewpoint and direction can be freely changedby the operator using an HMD and a head tracker. Although an interface with a freely conﬁgurable view is expected to provide intuitive and safe operations, the existing VEVIs16)17) have been developed only as virtualreality interfaces without using real-world textures.
This study realizes an intuitive and freely conﬁgurable third-person-view interface using an HMD anda head tracker for providing photorealistic textures ofreal-world environments. The proposed interface employs state-of-the-art image processing and computer
vision methods; therefore, the robot’s far side is visible by superimposing a transparent robot appearance.
That is, our approach resolves the occlusion problem,
while facilitating grasping distances to surrounding objects on the far sides of the robot. We investigate theeﬀectiveness of a prototype free-viewpoint-operation interface for a real environment through evaluations. The
approaches used to realize freely conﬁgurable views
with the textures of a real environment are describedin the following section.
2. 2 Free-viewpoint image generation
In the ﬁelds of computer graphics and computer vision, techniques for generating freely conﬁgurable views
from multiple images are referred to as free-viewpoint
image generation (or arbitrary- or novel-viewpoint image generation). One free-viewpoint image generation
approach is known as model-based rendering (MBR).
This approach is a traditional computer graphics/vision
pipeline that reconstructs 3D shapes of real environments ﬁrst, and then maps images of the environment
over them as textures. At present, 3D shapes can beacquired in real-time from environments ranging froma small desktop space20) to a large outdoor environment21) . In this approach, the quality of the freeviewpoint images generated by MBR is directly aﬀectedby the accuracy of the 3D shapes; that is, unnatural distortions or missing areas in the views are easily exposed.
On the other hand, image-based rendering (IBR) generates free-viewpoint images without using explicit 3D
shapes. There have been numerous studies on IBR
techniques, such as view morphing22) , light-ﬁeld rendering23)24) , and lumigraph rendering25)26) . Although IBR
reduces missing areas in resulting images, the approachrequires images captured at a large number of placesand directions. Otherwise, large distortions may apPaper

pear27) .
In recent years, the main approach of free-viewpoint
image generation is to use hybrid techniques that combine MBR and IBR with the goal of resolving theirrespective problems. The primary approach of hybrid free-viewpoint generation is view-dependent texture mapping (VDTM), which was originally proposed
by Debevec et al.28) . This technique selects and blends
multiple textures acquired from multiple cameras, andthen maps the blended textures onto the 3D models. Although the textures acquired at proper positions are selected in VDTM approaches, large errors
in the 3D shapes still cause signiﬁcant distortion on
the resultant free-viewpoint images. State-of-the-art
methods of hybrid rendering appropriately transformthe 3D shapes depending on the position of the virtual
viewpoint, while also selecting and blending the textures29)30) . These hybrid-rendering approaches, which
are sometimes referred to as view-dependent geometry
and texture (VDGT), generate reasonably clear freeviewpoint images, even though the 3D shapes include
large errors (e.g., when there are missing regions in 3D
shapes).
The proposed interface employs a VDGT hybridrendering method similar to the one proposed by Satoet al.30) . Because this VDGT method does not achieve
real-time processing, we simpliﬁed and improved it torealize real-time image generation. In addition, we superimposed a 3D model of the mobile robot on thefree-viewpoint images using an augmented reality (AR)technique31) . The combined technique of free-viewpoint
image generation and AR is referred to herein as augmented free-viewpoint image generation. In the following sections, details of the interface are describedwith a prototype system example using an omnidirectional camera and four depth cameras. We also discuss
the eﬀectiveness of the free-viewpoint interface throughevaluations in both virtual and physical environments.

3. Augmented free-viewpoint interface
3. 1 Overview
Fig. 2 provides an overview of the proposed freeviewpoint interface. The robot is equipped with a conventional camera and depth cameras for acquiring environmental information such as textures and 3D shapesof the surroundings. The environmental informationand odometry from the wheels of the robot are transmitted to the site of the human operator. A server(3) 3

head pose

Omnidirectional & depth cameras
textures
3D shapes

free-viewpoint
imagesServerPC

Transmitter
odometry

Robot

HMD

textures
3D shapes
odometry
Transmitter

Robot

Head
tracker

control signal(a) Images acquired by the omnidirectional multi-camera system.

Joystick

Human operator(b) Depth images from four depth cameras.

Red arrows: control signals of the robot.
Black arrows: information used for free-viewpoint generation.

Fig. 3(a) Data ﬂow diagram of the prototype with the proposed freeviewpoint interface.

Robot

Human operator(b) Images of the prototype system.
Fig. 2

Prototype system based on the proposed interface.

receives the information from the robot and generates
in real-time augmented free-viewpoint images that are
displayed on an HMD. The viewpoint of these generated images synchronously changes with the operator’shead pose (position and direction), which is acquiredby a head tracker mounted on the HMD. The details of
the augmented free-viewpoint image generation method
used in the proposed interface are described in Section 3. 4. Although the experiment employed a simple,
wheeled robot operating with joystick control signals,
including forward/backward movement and rotation,
the proposed interface is compatible with other typesof mobile robots and control methods.
3. 2 Prototype system
As mentioned earlier, the prototype system speciﬁcations are based on the proposed free-viewpoint interface, as shown in Fig. 2(b).
( 1 ) Robot conﬁguration
The prototype robot has an omnidirectional camera
and four depth cameras. Examples of the environmental information, including textures and 3D shapes as
depth images, captured by the prototype system areshown in Fig. 3. The robot (Reference Hardware,
Mayekawa Manufacturing Co.) is equipped with four
wheels that enable forward/backward movement androtation. An omnidirectional multi-camera system (Ladybug2, Point Grey Research, Inc.) mounted on top of
4 (4)Environmental information acquired from theprototype robot.

the robot captures textures of the omnidirectional ﬁrstperson view. Four depth cameras (Kinect, Microsoft,Inc.) are mounted on four sides of the robot to acquire depth images of the surroundings. The horizontalﬁeld-of-view of each depth camera is 57◦ , which indicates that the four depth cameras cannot simultaneously cover all of the robot’s surroundings.
The relative position and orientation between the omnidirectional and depth cameras are calibrated in advance. We prepare a special calibration pattern consisting of a plane and circular constructions, as shownin Fig. 4(a). The pattern is captured from numerous
positions and directions by omnidirectional and depth
cameras mounted on the robot (see Figs. 4(b) and 4(c)).
The center point of each marker is manually designatedon omnidirectional images. The corresponding marker
captured by the depth camera is then selected from thedepth image. The 3D centroid of the selected marker
is calculated using the depth; it is used as the 3D point
corresponding to the 2D point designated on the omnidirectional image. The relative position and orientation
are calculated by solving the perspective-n-point (PnP)
problem from the 2D-3D correspondences.
Environmental information is newly acquired andtransmitted to the server in the prototype system when
one of three conditions is met: the moving distanceof the robot exceeds a threshold from the latest image
acquisition; the rotation angle of the robot exceeds athreshold; or, a certain period of time has elapsed. The
omnidirectional and depth images are acquired withrough software-based synchronization. The maximum
lag time of the synchronization depends on the maximum frame rate of the sensors (e.g., less than 33 ms in
our implementation using 30 fps sensors). In our experiments, we did not ﬁnd negative eﬀects due to the delay; however, more accurate synchronization might be

ITE Transactions on Media Technology and Applications Vol. xx, No. xx (20xx)

(a) Calibration pattern.

(b) Calibration patterncaptured by a camera
in an omnidirectional
multi-camera system.
Fig. 4(c) Calibration pattern captured by adepth camera.

Calibration between omnidirectional and depth
cameras.

required under operations involving rapid robot movement. The acquired images are combined into one large
image and transmitted by a wireless HDMI extenderwith small delays (< 1 ms) for the experimental environment.
( 2 ) Conﬁguration for human operator
The human operator wears an HMD (HMZ-T1, Sony)that displays augmented free-viewpoint images. This
HMD is equipped with a receiver for an electromagnetic sensor (Fastrak, Polhemus, Inc.) that operates
with a transmitter to measure the six degrees of freedom
pose of the operator’s head. Augmented free-viewpoint
images are generated from environmental information(textures and depth images from the robot), odometry, operator head pose, and the 3D robot model in theserver. When only one shot of the depth image set isused to generate the free-viewpoint images, large missing areas appear due to occlusions. In addition, thereis lack of depth information from the four depth cameras because they do not cover the entire view from therobot. The proposed interface uniﬁes time-series 3D
point clouds to reduce the missing areas. The most recent environmental information received from the robot
is combined with older information by 3D point cloudalignment using the odometry information as the initial estimate for each transmission of environment information. The augmented free-viewpoint images are
Paper

generated in real-time from 3D point clouds and omnidirectional images. The point clouds are combined by
the L0-norm-based alignment process, which is a simpliﬁed implementation of the method by Hieida et al.32) .
It should be noted that the uniﬁcation process is implemented as a thread separated from the augmented
free-viewpoint image generation process. That is, thecalculation cost of the uniﬁcation process does not aﬀect
the frame rate of resultant augmented free-viewpoint
image sequences presented to the human operator. The
depth-image uniﬁcation and augmented free-viewpoint
image generation processes are described in more detailin the following sections.
3. 3 Unification of multiple depth images
The proposed interface accurately estimates the position and orientation of the robot for each imageacquisition operation by aligning the time-series depth
images while minimizing the norms among the multiple 3D point clouds. The norms are minimized byusing odometry data as the initial estimation. The
prototype system employs L0-norm minimization witha two-dimensional exhaustive search algorithm. Using
L0-norm is a robust solution for point cloud alignment
with large outliers; however, it is diﬃcult to minimize
L0-norm using gradient-based minimization algorithms.
To achieve a real-time process, the 3D points that exist
in a certain interval of height are ﬁrst projected on a 2D
horizontal plane. The minimum value of cost function
based on L0-norm is then searched for in a 2D solution space by changing the rotation and translation intiny intervals around the acquired odometry. Although
our implementation searches for a minimum in a direct manner, it would be possible to use pairs of techniques for more eﬃcient searching, such as a simultaneous localization and mapping based on L0-norm minimization32) . Because the proposed interface does not
specify the alignment methods for point clouds, otheralignment techniques can be employed, such as modern iterative closest-point algorithms33) , feature-basedmatching approaches34) , and registration using Gaussian mixture models35) .
When aligning point cloud p to another point cloudq, L0-norm |pi , qj |0 is deﬁned as{
0 (∃j, |pi − qj |2 <= ϵ) ,|pi , qj |0 =(1)
1 (otherwise)
where ϵ denotes a tiny distance that can be regarded as
an identical point. The system minimizes the E(R, t)deﬁned as the sum of L0-norms with changing rotation(5) 5

(a) Alignment by odometry only.
Fig. 5(b) Alignment using L0-norm.

Point clouds aligned before and after reﬁnement
between the two positions of acquired images,
denoted in blue and pink, respectively.

Viewpoint
Fig. 6

View-dependent geometry generation.

Meshmatrix R and translation vector t from p to q as∑
E(R, t) =|Rpi + t, qj |0 .
(2)i

The transformation from the robot to the real-world coordinates that is speciﬁed by R and t is also used toaugmented free-viewpoint image generation.
Examples of point clouds with and without the L0norm-based alignment process are shown in Fig. 5.
Misalignments between two point clouds are reducedby the L0-norm-based alignment process.
3. 4 Augmented free-viewpoint image generation
Augmented free-viewpoint images are generated fromthe uniﬁed point clouds, operator’s viewpoint, and 3D
robot model in three steps: 1) view-dependent geometry generation; 2) view-dependent texture mapping;and 3) superimposition of 3D robot model.
A free-viewpoint image generation method with viewdependent geometry and texture is employed for thisstudy30) . The existing method30) requires the preliminary reconstruction of 3D shapes using multi-view
stereo approaches, and it does not achieve real-timeprocessing. Notably, a nonlinear optimization process
in the method30) that uses a whole sequence to achieve
high-quality image generation, requires a considerable
calculation cost. We used 3D point clouds acquiredwith depth cameras and realized real-time processing
by not employing the optimization processes in the geometry generation and pixel-wise texture selection. In
addition, the 3D model of the robot was superimposedon the free-viewpoint images using a standard AR technique.
( 1 ) View-dependent geometry generation
The view-dependent geometry generation process
produces a depth image of the operator’s view using
point clouds to reduce any missing areas and unnatural
distortions in the free-viewpoint images. The depth isestimated from the combined point clouds in the fol6 (6)Camera 1
Viewpoint
Fig. 7

Camera 2

View-dependent texture selection. Camera 2 isselected as the mesh texture in this case.

lowing steps.
(i) Divide the view plane of the operator into triangular meshes, as shown in Fig. 6, and employ
pi as the vertices of the meshes.
(ii) Project the 3D points onto the view plane. Some
3D points may be far from the appropriate
depth, such as those existing over walls.
(iii) Estimate the depth of each vertex pi , and compare the depths of the projected points neighboring pi . The depth value di of the point p̂ithat has the smallest depth is employed as thedepth of the vertex pi .
It is possible that there are no p̂i corresponding topi because of lack of depth information. This lack is
caused by the occurrence of occlusions in a complex
environment and by the limited ﬁeld-of-view of depth
cameras. If there are neighboring vertices whose depth
values have been successfully estimated, di is determined using linear interpolation of the valid depth values. Otherwise, di is set as the largest value that canbe measured by the depth cameras (4000 mm in our
prototype system).
( 2 ) View-dependent texture mapping
For each mesh produced in view-dependent geometry generation, the appropriate texture is selected fromtime-series omnidirectional textures captured for the
given mesh. The generated geometry may include someerrors in the 3D shapes. As shown in Fig. 7, we deﬁne
α as the angle between two vectors from the center ofthe mesh: one is to the camera capturing the texture of

ITE Transactions on Media Technology and Applications Vol. xx, No. xx (20xx)

(a) Free-viewpoint image.

Fig. 8(b) Augmented free-viewpoint image.

Goal

Superimposition of 3D robot model.
Fig. 9the mesh, and the other is to the viewpoint to be generated. The pose of the robot estimated in Section 3. 3 isused for the pose of the camera. The proposed method
selects the texture that has the smallest α because thedistortion of the appearance caused by 3D shape errors is smaller when α is smaller. Finally, the selected
texture is projected and mapped onto the mesh. This
texture selection strategy is common in some IBR andhybrid rendering approaches28) .
( 3 ) Superimposition of 3D robot model
In the proposed interface, the hand-made 3D robot
model is superimposed using the robot pose information
estimated by aligning the depth images, as described in
Section 3. 3, as well as the odometry information. The
generated free-viewpoint images do not include the appearance of the robot itself, as shown in Fig. 8(a). The
transmission of the depth images, which requires a suﬃciently large bandwidth, may cause large delays. Therefore, the robot may not be superimposed in the appropriate position in a free-viewpoint image when only
the robot pose estimated in Section 3. 3 is used. To
generate augmented free-viewpoint images while considering such delays, changes in the robot pose since
the most recent acquisition of depth images are calculated from odometry information; these are used withthe depth alignment-based pose information. The prototype system 3D model is transparently superimposed
to improve the visibility of the scene on the robot’s far
side, along with a virtual arrow indicating the robot’s
travel direction, as shown in Fig. 8(b).

4. Virtual environment experiments
When operating a remote robot in a real environment, there are many factors aﬀecting a human operator’s experience, such as the quality of the freeviewpoint images generated by the proposed approach.
We therefore conducted an evaluation to investigate the
Paper

Start
Experimental virtual environment.

interface characteristics using freely conﬁgurable viewsin an ideal (i.e., virtual) environment.
The human operator conﬁguration was the same as
the one discussed in Section 3. 2. Participants at a remote site operated a virtual robot in a simulation environment using CG models. The participants performed
the following two tasks:
Task 1: Run through a path as quickly and safely as
possible without colliding with the wall (the path
included narrow passages obstacles, as shown in
Fig. 9).
Task 2: Approach the wall as closely as possible,
without colliding with it (such behavior is sometimes required in practical situations to accurately
operate a robot with arms beside a wall).
For each task, we compared three interfaces: theﬁrst-person view, third-person view, and proposed augmented free-viewpoint interface. The third-person view
was ﬁxed at a diagonal location behind the robot at approximately 45◦ above the horizontal location. The images presented to the operators were produced without
the proposed free-viewpoint image generation methodby rendering the virtual environment from only the conﬁgured viewpoint using a traditional graphics library, as
shown in Fig. 10. The participants were ten people intheir twenties or thirties. Most of them were not familiar with the robot operation tasks. Therefore, we ﬁrst
conducted training on joystick robot operation in a virtual environment diﬀerent from the one used for the experiment. The operations were successfully completed
without a wall collision, except for the ﬁrst-person viewinterface in Task 1 above (average 1.1 times of collision).
In addition to the investigation using objective factors, we conducted subjective evaluations based on thefollowing three questions to ascertain the operator’s impression of each interface:
Q1: Were the obstacles on the ground easily recog(7) 7

70

120

60
Distance to wall [cm](b) Virtual third-person view.

Operating time [sec.](a) Virtual ﬁrst-person view.

140

100
80

*

60

*

40

*

*
*

50
40
30
20

20

10
0

Task 1
first

third

0

Task 2
free

first

third

free

(a) Operating time for each task. (b) Distance to the wall in thesecond task.

5

Best

4.5

*
*

*
*

4
3.5

first

3

third

2.5

free

2
1.5

Worst 1(c) Virtual free-viewpoint.
Fig. 10

Q1

Examples of each interface view in a virtual environment.

Q3(c) Questionnaire results.
Fig. 11

nized?
Was it possible to grasp the distance between the
robot and the wall?
Q3: Could you operate the robot without feeling delays?
The participants answered these questions after eachstage of the experiments using a scale from one (worst)
to ﬁve (best).
Fig. 11 shows the results of the experiments in thevirtual environment, as well as the pairs that had signiﬁcant diﬀerence p < 0.05. We employed a Friedman
multiple comparison test with a Bonferroni-adjusted
post-hoc Wilcoxon signed-rank test. This comparison scheme is a nonparametric alternative of one-way
repeated-measures analysis of variance (ANOVA), witha post-hoc paired t-test with Bonferroni correction. The
correction determines the signiﬁcant level to prevent
increasing type-one errors in the multiple comparison
test. In the results for Task 1, shown in Fig. 11(a), theoperating times to complete the task were not signiﬁcantly diﬀerent among the three interfaces. In Task 2,
the free-viewpoint interface was signiﬁcantly more accurate than the other interfaces, as shown in Fig. 11(b);instead a longer operating time was taken with that interface to approach very close to the wall. This implies
that the free-viewpoint interface could generate viewpoints that enable the operators to eﬀectively grasp the

Q2

Results of the virtual environment experiments(‘*’ indicates a signiﬁcant diﬀerence by a paired
multiple comparison test, p < 0.05).

Q2:

8 (8)

distance between the robot and the wall, which is shownin the bottom-right of Fig. 10(c).
The results of the questionnaires, presented inFig. 11(c), were that the free-viewpoint interface hadsigniﬁcantly higher ratings for both the ﬁrst and second questions, which concerned the ability to recognize the surrounding environment. This means thatthe proposed interface reduces ambiguity in the operator’s recognition of the virtual environment surroundings. For the third question, there were no signiﬁcantdiﬀerences among three interfaces. They had exactly
the same ratings from all participants. The diﬀerencesamong interfaces had fundamentally little impact onperception of the delay, at least in the given environment, which had a tiny delay.
The results of these experiments indicate that the
proposed interface oﬀers the advantage of accurate andsafe operation over short operating time. The results therefore imply that the proposed interface is potentially valuable for mobile-robot remote navigation
tasks, such as surveillance.

5. Physical environment experiments
We performed experiments in a physical environment
using the prototype system described in Section 3. 2.

ITE Transactions on Media Technology and Applications Vol. xx, No. xx (20xx)

(a) First-person view.

(b) Third-person view.
Fig. 12

ObstaclesStage 2

Examples of each interface view in a physical environment.

Stage 3
Object toapproach

Stage 1

Start
Fig. 13

Map of experimental physical environment.

The experiment participants again consisted of ten people in their twenties or thirties, who operated the robot
using the same three interfaces discussed in the previous
section. In this experiment, the ﬁrst-person view was
generated from the most current omnidirectional image, and the participants could freely change their view
direction. The third-person viewpoint was ﬁxed at a
diagonal position behind the robot, whose images weregenerated using the same technique used in the proposed free-viewpoint interface. The third-person viewalso enabled the participants to change their view direction. Examples of the view of each interface are shownin Fig. 12. Free-viewpoint images presented for thehuman operator from a couple of viewpoints are shownin Fig. 1 of Section 1. The runway for the robot usedin the experiment was constructed in three stages, as
shown in Fig. 13. The participants were directed tooperate the robot with respect to each stage without
colliding with the wall and obstacles. A task was set
for each stage as follows:Stage 1:Run through a straight narrow passage as
quickly as possible.
Stage 2:
Run through a curved passage with obstacles as quickly as possible.
Stage 3:
Approach the wall as closely as possible.
We evaluated the operating time for each stage, alongwith the distance to the wall in Stage 3. In Stage 3, a
Paper(c) Free-viewpoint.

participant collided with the wall with the third-personview and the free-viewpoint interface, respectively. In
Stage 2, another participant also collided with the wallwith the third-person view interface. The participants
were allowed to retry the stage when a collision occurred. In addition to objective factors, we investigatedoperator subjective issues by posing to participants thesame three questions described in Section 4.
In our experiment, the environmental information
was newly acquired and uniﬁed when one of the following conditions was met: 1) 60 cm movement of therobot, 2) 20◦ rotation of the robot, or 3) 200 seconds
elapsed from the last image capture. Using these conditions, the images were acquired at an average of 13.9
locations through all stages. The uniﬁcation processof the environmental information was performed in lessthan 33 ms in the given small environment used in thisexperiment, where the process does not aﬀect the frame
rate of the augmented free-viewpoint video presented tothe examinees. The augmented free-viewpoint image
was successfully presented on an HMD at 30 fps.
Fig. 14(a) shows the operating time for each stage
and interface. The ﬁgure also shows the pairs that
had signiﬁcant diﬀerence p < 0.05 calculated by thesame manner as outlined in Section 4. In Stage 1, thefree-viewpoint interface, together with the third-personview interface, enabled the operators to quickly complete the task. The participants most often ﬁxed their
viewpoint when using the proposed interface for running through the straight pathways. In those cases,
the free-viewpoint interface exhibited behavior similar
to that of the third-person view. When using the proposed interface, the distance to the wall in Stage 3 wassigniﬁcantly smaller than with the others, as shown in
Fig. 14(b), but the operating time was longer than in
the ﬁrst-person view. This implies that the operator
using the proposed interface takes additional time toclosely approach the wall compared to the other interfaces. These results demonstrate the same trend as do
the virtual environment experiments: The proposed in(9) 9

90Operating time [sec.]

70

60

*

*
*

*

*

*
*

50
Distance to wall [cm]

80

60
50
40
30
20
10

40
30
20
10

0

Stage 1
first

Stage 2
third

Stage 3free(a) Operating time of each stage.

Best

5
4.5
4

*
*

*

*

0
first

third

free

(b) Distance to the wall inStage 3.

Fig. 15

Artifact in an augmented free-viewpoint image.

*
*

first
3

Operator’s
viewpoint

Acquiredtextures

3.5
third

2.5

free

2
1.5

Robot

Worst 1
Q1

Q2

Q3

Occludedtextures(c) Questionnaire results.
Fig. 14

Wall

Physical environment experiment results (‘*’
indicates a signiﬁcant diﬀerence by a paired
multiple comparison test, p < 0.05).

terface improves operation accuracy. In addition, quick
operation was achieved in some cases of the proposedinterface used with the third-person view, particularly
in simple environments.
The results of the questionnaires are shown in
Fig. 14(c). Similar to the evaluation result in the virtualenvironment, the proposed interface received higher ratings for both the ﬁrst and second questions, but not thethird question. We can ﬁnd the same trend in ideal (virtual) and physical environments from both objective
and subjective aspects. Therefore, the free-viewpoint
image generation process in our prototype system successfully expresses the potential advantage of the proposed interface.

6. Discussions
6. 1 Quality of augmented free-viewpoint image
From our experiments in the physical environment,
we have conﬁrmed that the quality of images generated
by the prototype system is satisfactory for expressing
the potential advantage demonstrated in the virtualenvironment experiments. Nevertheless, to realize amore eﬀective interface, the image quality should befurther improved. In the bottom of Fig. 15, for example, artifacts appear in the augmented free-viewpoint
10 (10)Fig. 16

Occlusion problem in free-viewpoint image generation. Shaded area denotes where textures
are occluded by the wall but required to complete the operator’s view.

image. They could cause false recognition by operatorsand therefore lack of safe operation. Such artifacts are
generated due to the occlusion problem, as illustrated
in Fig. 16. Although environmental information (i.e.,
3D shape and textures of the environment) cannot bephysically acquired in the region occluded by surrounding objects (e.g., walls), the generation process for theoperator’s view sometimes requires them. A promisingalternative for increasing safety is to display these textures in the free-viewpoint image using another form,
such as by ﬁlling them with a prominent color to highlight them as occluded textures.
6. 2 Limitations
Although the proposed free-viewpoint interface does
not essentially restrict speciﬁc system conﬁguration andimplementation, our prototype system based on the
proposed interface has some limitations. In the implementation of the depth uniﬁcation process, we employed
the flat floor assumption; i.e., our prototype does notaccommodate the slope of the ﬂoor or tilt of the robot.
To overcome this problem, 3D point cloud registration
approaches33)34) can be applied. In addition, the prototype system is specialized for indoor use because of
the hardware limitation (e.g., depth cameras cannot be
used outdoors). Other robots, sensors, or devices can

ITE Transactions on Media Technology and Applications Vol. xx, No. xx (20xx)

be employed for our proposed interface by modifying
some implementations.

7. Conclusions and future work
In this paper, we have proposed a teleoperation interface for mobile robots with a freely conﬁgurable
viewpoint using photorealistic textures of the physicalworld. This free-viewpoint enables human operators
to intuitively change their viewpoints using an HMD
and a head tracker. A free-viewpoint image generation method with view-dependent geometry and texture
has been simpliﬁed and improved to achieve real-timeprocessing for the proposed interface. In addition, we
achieved augmented free-viewpoint image generation inwhich a 3D model of the robot was superimposed on
the free-viewpoint image using AR techniques. Our experiments conducted in both virtual and physical environments have conﬁrmed that the proposed interface
has advantages in terms of operational accuracy overtime required to complete tasks, while the quality of
the generated free-viewpoint images is satisfactory fordemonstrating the advantage of our proposed interface.
In future work, we will investigate the eﬀects of delayin the proposed interface in a setting with large delays
in environment information transmission, improve theprototype system for more practical situations, and enhance the quality of the free-viewpoint image generation
process.

Acknowledgement
This research was supported by the Japan Society for
the Promotion of Science (JSPS) Grants-in-Aid for Scientiﬁc Research, Nos. 24700208 and 23240024, Grantin-Aid for JSPS Fellows No. 25–7448, and the “AmbientIntelligence” project granted by Ministry of Education,
Culture, Sports, Science and Technology (MEXT). Finally, we would like to thank Dr. Kentaro Takemura,
former Assistant Professor of Robotics Lab., Nara Institute of Science and Technology, Nara, Japan, for his
support in our prototype system implementation.
References
1) F. Okura, Y. Ueda, T. Sato and N. Yokoya: “Teleoperation of
Mobile Robots by Generating Augmented Free-Viewpoint Images”, Proc. 2013 IEEE/RSJ Int. Conf. Intelligent Robots & Syst.
(IROS’13), pp.665–671 (2013)
2) G. N. DeSouza and A. C. Kak: “Vision for Mobile Robot Navigation: A Survey”, IEEE Trans. Pattern Analysis & MachineIntelligence, 24, 2, pp.237–267 (2002)
3) R. Siegwart, I. R. Nourbakhsh and D. Scaramuzza: Introduction
to Autonomous Mobile Robots, MIT Press, Cambridge (2011)4) H. Choset: “Coverage for Robotics–A Survey of Recent Results”,
Annals of Mathematics & Artiﬁcial Intelligence, 31, 1, pp.113–
Paper

126 (2001)5) B. M. Yamauchi: “PackBot: A Versatile Platform for Military
Robotics”, Proc. SPIE, 5422, Unmanned Ground Vehicle Technology VI, pp.228–237 (2004)
6) T. Fong and C. Thorpe: “Vehicle Teleoperation Interfaces”, Autonomous Robots, 11, 1, pp.9–18 (2001)
7) D. W. Hainsworth: “Teleoperation User Interfaces for Mining
Robotics”, Autonomous Robots, 11, 1, pp.19–28 (2001)
8) R. T. Laird, M. H. Bruch, M. B. West, D. A. Ciccimaro and H. R.
Everett: “Issues in Vehicle Teleoperation for Tunnel and Sewer
Reconnaissance”, Proc. 2000 IEEE Workshop Vehicle Teleoperations Interfaces (2000)
9) K. Nagatani, S. Kiribayashi, Y. Okada, S. Tadokoro,
T. Nishimura, T. Yoshida, E. Koyanagi and Y. Hada: “Redesignof Rescue Mobile Robot Quince”, Proc. 2011 IEEE Int. Sympo.
Safety, Security, & Rescue Robotics (SSRR’11), pp.13–18 (2011)10) N. Shiroma, N. Sato, Y. Chiu and F. Matsuno: “Study on Effective Camera Images for Mobile Robot Teleoperation”, Proc.
13th IEEE Int. Workshop Robot & Human Interactive Commun.
(ROMAN’04), pp.107–112 (2004)
11) M. Sugimoto, G. Kagotani, H. Nii, N. Shiroma, F. Matsuno and
M. Inami: “Time Follower’s Vision: A Teleoperation Interface
with Past Images”, IEEE Comput. Graphics & Appl. Mag., 25,
1, pp.54–63 (2005)
12) K. Saitoh, T. Machida, K. Kiyokawa and H. Takemura: “A 2D3D Integrated Interface for Mobile Robot Control Using Omnidirectional Images and 3D Geometric Models”, Proc. Fifth
IEEE/ACM Int. Sympo. Mixed & Augmented Reality (ISMAR’06), pp.173–176 (2006)
13) C. W. Nielsen, M. A. Goodrich and R. W. Ricks: “Ecological Interfaces for Improving Mobile Robot Teleoperation”, IEEE
Trans. Robotics, 23, 5, pp.927–941 (2007)
14) F. Ferland, F. Pomerleau, C. T. Le Dinh and F. Michaud:“Egocentric and Exocentric Teleoperation Interface Using RealTime, 3D Video Projection”, Proc. Fourth ACM/IEEE Int. Conf.
Human-Robot Interaction (HRI’09), pp.37–44 (2009)15) A. Kelly, N. Chan, H. Herman, D. Huber, R. Meyers, P. Rander,
R. Warner, J. Ziglar and E. Capstick: “Real-Time Photorealistic
Virtualized Reality Interface for Remote Mobile Robot Control”,
Int. J. Robotics Research, 30, 3, pp.384–404 (2011)16) B. Hine, P. Hontalas, T. Fong, L. Piguet, E. Nygren and
A. Kline: “VEVI: A Virtual Environment Teleoperations Interface for Planetary Exploration”, Proc. 25th SAE Int. Conf. Environmental Syst. (1995)
17) L. A. Nguyen, M. Bualat, L. J. Edwards, L. Flueckiger, C. Neveu,
K. Schwehr, M. D. Wagner and E. Zbinden: “Virtual Reality Interfaces for Visualization and Control of Remote Vehicles”, Autonomous Robots, 11, 1, pp.59–68 (2001)18) K. Yamazawa, Y. Yagi and M. Yachida: “Omnidirectional Imaging with Hyperboloidal Projection”, Proc. 1993 IEEE/RSJ Int.
Conf. Intelligent Robots & Syst. (IROS’93), 2, pp.1029–1034(1993)
19) M. G. Dissanayake, P. Newman, S. Clark, H. F. Durrant-Whyte
and M. Csorba: “A Solution to the Simultaneous Localizationand Map Building (SLAM) Problem”, IEEE Trans. Robotic &Aut., 17, 3, pp.229–241 (2001)20) S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe,
P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison and
A. Fitzgibbon: “KinectFusion: Real-time 3D Reconstruction andInteraction Using a Moving Depth Camera”, Proc. 24th ACM
Sympo. User Interface Software & Techn. (UIST’11), pp.559–568(2011)
21) P. Merrell, A. Akbarzadeh, L. Wang, P. Mordohai, J. M. Frahm,
R. Yang, D. Nistér and M. Pollefeys: “Real-Time Visibility-Based
Fusion of Depth Maps”, Proc. 11th IEEE Int. Conf. Comput. Vision (ICCV’07), pp.1–8 (2007)22) S. M. Seitz and C. R. Dyer: “View Morphing”, Proc. ACM SIGGRAPH’96, pp.21–30 (1996)23) M. Levoy and P. Hanrahan: “Light Field Rendering”, Proc. ACM
SIGGRAPH’96, pp.31–42 (1996)24) T. Naemura, T. Takano, M. Kaneko and H. Harashima: “RayBased Creation of Photo-Realistic Virtual World”, Proc. Third
Int. Conf. Virtual Syst. & Multimedia (VSMM’97), pp.59–68(1997)
25) S. J. Gortler, R. Grzeszczuk, R. Szeliski and M. F. Cohen: “TheLumigraph”, Proc. ACM SIGGRAPH’96, pp.43–54 (1996)
26) C. Buehler, M. Bosse, L. McMillan, S. Gortler and M. Cohen: “Unstructured Lumigraph Rendering”, Proc. ACM SIG-

(11) 11

GRAPH’01, pp.425–432 (2001)27) S. B. Kang, R. Szeliski and P. Anandan: “The Geometry-Image
Representation Tradeoﬀ for Rendering”, Proc. 2000 IEEE Int.
Conf. Image Processing (ICIP’00), 2, pp.13–16 (2000)
28) P. E. Debevec, C. J. Taylor and J. Malik: “Modeling and Rendering Architecture from Photographs: A Hybrid Geometry- andImage-Based Approach”, Proc. ACM SIGGRAPH’96, pp.11–20(1996)
29) M. Irani, T. Hassner and P. Anandan: “What Does the Scene
Look Like from a Scene Point?”, Proc. Seventh European Conf.
Comput. Vision (ECCV’02), pp.883–897 (2002)30) T. Sato, H. Koshizawa and N. Yokoya: “Omnidirectional FreeViewpoint Rendering Using a Deformable 3-D Mesh Model”, Int.
J. Virtual Reality, 9, 1, pp.37–44 (2010)31) R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier and
B. MacIntyre: “Recent Advances in Augmented Reality”, IEEE
Comput. Graphics & Appl. Mag., 21, 6, pp.34–47 (2001)
32) Y. Hieida, T. Suenaga, K. Takemura, J. Takamatsu and T. Ogasawara: “Real-Time Scan-Matching Using L0-Norm Minimization Under Dynamic Crowded Environments”, Proc. FourthWorkshop Planning, Perception & Navigation for Intelligent Vehicles, pp.257–262 (2012)33) A. Segal, D. Haehnel and S. Thrun: “Generalized-ICP”, Proc.
2009 Robotics: Science & Syst. (RSS’09), 25, pp.26–27 (2009)34) R. B. Rusu, N. Blodow and M. Beetz: “Fast Point Feature Histograms (FPFH) for 3D Registration”, Proc. 2009 IEEE Int.
Conf. Robotics & Aut. (ICRA’09), pp.3212–3217 (2009)35) B. Jian and B. C. Vemuri: “Robust Point Set Registration Using Gaussian Mixture Models”, IEEE Trans. Pattern Analysis &Machine Intelligence, 33, 8, pp.1633–1645 (2011)

Fumio Okura received his M.E. degree in
information science from Nara Institute of Scienceand Technology in 2011. Since 2011 he has been
pursuing his Ph.D. at Nara Institute of Science andTechnology. He has been a research fellow of theJapan Society for the Promotion of Science since
2013.

Yuko Ueda received her B.S. degree in information and computer science from Nara Women’s
University in 2011. She received her M.E. degree in
information science from Nara Institute of Scienceand Technology in 2013. She has been working at
Sony Corporation since 2013.
Tomokazu Sato received his B.E. degree
in computer and system science from Osaka Prefecture University in 1999. He received his M.E. and
Ph.D. degrees in information science from Nara Institute of Science and Technology in 2001 and 2003,
respectively. He was an assistant professor at Nara
Institute of Science and Technology from 2003 to
2011, when he became an associate professor.

Naokazu Yokoya Naokazu Yokoya received
his B.E., M.E., and Ph.D. degrees in informationand computer sciences from Osaka University in
1974, 1976, and 1979, respectively. He joined Electrotechnical Laboratory (ETL) in 1979. He was avisiting professor at McGill University in 1986-87
and has been a professor at Nara Institute of Science and Technology since 1992. He has also beena vice president at Nara Institute of Science andTechnology since April 2013.

12 (12)

ITE Transactions on Media Technology and Applications Vol. xx, No. xx (20xx)