Augmented Telepresence
Using Autopilot Airship and Omni-directional Camera
Fumio Okura∗

Masayuki Kanbara∗

Naokazu Yokoya∗

Nara Institute of Science and Technology (NAIST)A BSTRACT

User’s inputThis study is concerned with a large-scale telepresence system
based on remote control of mobile robot or aerial vehicle. The proposed system provides a user with not only view of remote site but
also related information by AR technique. Such systems are referred to as augmented telepresence in this paper. Aerial imagery
can capture a wider area at once than image capturing from theground. However, it is difficult for a user to change position anddirection of viewpoint freely because of the difficulty in remotecontrol and limitation of hardware. To overcome these problems,
the proposed system uses an autopilot airship to support changing user’s viewpoint and employs an omni-directional camera forchanging viewing direction easily. This paper describes hardware
configuration for aerial imagery, an approach for overlaying virtual
objects, and automatic control of the airship, as well as experimental results using a prototype system.
Index Terms: H.5.1 [Information Interfaces and Presentation]:
Multimedia Information Systems—Artificial, augmented, and virtual realities I.2.9 [Artificial Intelligence]: Robotics—Autonomousvehicles

Desired position anddirection of view

1 I NTRODUCTION
This paper proposes a telepresence system which can superimpose
virtual objects on a scene of remote site with AR technique. The
telepresence system with augmented reality is referred to as augmented telepresence in this paper. In related work of the augmented
telepresence, additional information used for teleoperation is overlaid on images captured by a mobile robot on the ground [1]. Another work of augmented telepresence uses unmanned aerial vehicle(UAV). Aerial images captured from radio controlled helicopter areused for augmented telepresence system [2]. Augmented telepresence systems using aerial images can superimpose much information on the image than systems using ground images because aerialimagery can observe a wider area. Such systems are expected to
be used for not only teleoperation but also overlaying annotation tobuildings or geographical information from the sky.
However, it is difficult to change position of user’s viewpoint
in augmented telepresence from the sky because remote controlof UAV should be concerned with a number of factors: changingspeed, turning left or right, changing altitude, and some environmental factors such as status of wind. In addition, if a standard
camera is employed for observing the real scene, rotation of the vehicle is required to change user’s viewing direction. It is usuallydifficult to get an arbitrary viewing direction in real-time.
In the proposed system, an autopilot airship supports a user tosee an augmented scene which the user want to look around. Theautopilot airship limits degrees of freedom (DOF) of remote control
so that the user can control the vehicle simply. In addition, the proposed system equips an omni-directional camera which can capture
a nearly full spherical view to enable the user to look around thescene without rotation of the vehicle.
∗ e-mail: {fumio-o, kanbara, yokoya}@is.naist.jp

ProposedSystem
Position
Direction
ImagesData Flow
Data Acquisition

Airship

Position of airship
Direction of airship
Aerial omni-directional images

GPSGyroscopeOMSPositionDirection

Overlaying
virtual objects

Auto control of airship

Control signal

OutputAugmented view from desired
position and direction
Figure 1: Data flow of proposed system

Figure 2:airship

Unmanned Figure 3: Aerial panoramic image capturedby omni-directional camera

2 S YSTEM OVERVIEW
Figure 1 illustrates data flow of the proposed system. User should
input position and direction of a desired view. The system displays
an augmented view from the position and direction. For automatic
control and overlaying virtual object, a GPS and a gyroscope are
equipped with the airship.
The proposed system consists of the following three processes.
1. Data acquisition
2. Overlaying virtual objects
3. Automatic control of airship
Detail of each process is described below.
2.1 Data Acquisition
Omni-directional aerial image is captured by an omni-directional
multi camera system (OMS) equipped on an unmanned airshipshown in Figure 2. Position and direction of the airship are measured by a GPS and a gyroscope. Figure 3 shows a panoramic image generated by using intrinsic and extrinsic parameters of eachcamera of the OMS [3].
2.2 Overlaying virtual objects
To generate an augmented image, the panoramic image captured by
the OMS is projected onto a virtual spherical surface which has asufficiently large radius centered at the camera, and is converted to
a common planar-perspective image in the desired viewing direction. Virtual objects are overlaid on the perspective image by using
position and direction of the OMS.

200▲： Origin of auto control●： End of auto control
34.734 ■： Waypoints]
ge34.7335[deditut 34.733
aL34.7325
135.7325]190[mCdeu185
itltA180BD135.7335

A135.7345

Longitude [deg]B

195

135.7355

CA

D

50

100 150 200

175
170

0Time [s]Figure 4: Trajectory of autopilot airship (Left: ground projection,Right: altitude): ‘∗’ in the figure indicates positions of airship every
50 seconds.

Position and direction of the OMS are determined by the extrinsic camera parameter estimation method [4] based on structurefrom-motion and GPS data. At present, real-time camera parameter
estimation is not implemented for on-line telepresence. In extrinsic
camera parameter estimation using GPS data in large-scale environment, feature tracking often fails mainly due to error of the GPS. To
overcome the problem, a long video sequence is divided into shortsequences with hundreds of frames. Neighboring sequences areoverlapped, and estimated camera parameters on overlapped frames
are integrated by weighted averaging. Direction information fromgyroscope is used as initial direction of the OMS in each sequence.
2.3 Automatic control of airshipTo support changing user’s viewpoint efficiently, the proposed system provides user with the following two different approaches tocontrol airship.
1. Following waypoints designated by user
2. Reducing DOF of remote control
Each approach is detailed below.
2.3.1 Following waypoints designated by user
In this approach, the airship is controlled to follow the waypoints
using a proportional control scheme [5]. The points can be designated in advance or on the fly. This approach is suitable when thedesired viewpoint is clearly defined.
2.3.2 Reducing DOF of remote control
If desired position of viewpoint is not clear, it is required to control the airship to desired direction from current position. In usualremote control, the user should consider three DOF: speed, rudder,and elevation. This approach reduces DOF so as to control only
speed and rudder, changing elevation automatically by using theproportional method [5].
3 E XPERIMENTS
3.1 Prototype system
To evaluate the proposed augmented telepresence system, we havecarried out two experiments using a prototype system:• Automatic control of airship by designating points in advance,• Overlaying a virtual object on stored images captured fromthe airship.
Our unmanned airship shown in Figure 2 was used for a prototypesystem. The airship equips an OMS (Point Grey Research, Ladybug3), a differential-GPS (Hitachi Zosen Corporation, P4-GPS),and a fiber-optic gyroscope (Tokyo Keiki, TISS-5-40). All sensors
were connected to a laptop PC on the airship to store and transmit
captured data. Another PC placed on the ground was used to generate control signal for the airship. A transmitter was connected tothe PC on the ground to transmit control signal to the airship.

Figure 5: Examples of user’s view from different viewpoints

3.2 Automatic control of airship
In the experiment about automatic control, the airship was controlled around two points (230 m distance, 70 m altitude from theground). The airship was controlled manually during takeoff andlanding. Position and direction of the airship and control signal
were generated in 4 Hz. Figure 4 illustrates a trajectory of the autopilot airship. For the most part, the airship was controlled aroundthe points. However, sometimes the airship drifted from the points
mainly due to measurement error of the gyroscope.
3.3 Overlaying a virtual object
Virtual objects were overlaid on omni-directional images in offline processing because of the difficulty of omni-directional imagetransmission in the present system configration. The airship wascontrolled manually during capturing aerial images. In the experiment, a 3D model of an old palace in Heijo-capital was overlaid
on a defunct base of the palace captured by the prototype system.
The 3D model was successfully overlaid at appropriate positions invarying user’s views as shown in Figure 5.
4 C ONCLUSION AND FUTURE WORKThis paper has proposed a large-scale augmented telepresence system using aerial images captured from unmanned airship. The autopilot airship supports to change position of user’s viewpoint, andan omni-directional camera realizes to change user’s viewing direction without rotation of the airship. A GPS and a gyroscope are
equipped on the airship to measure position and direction of the airship and the OMS. The experiments have shown that the airship canbe controlled appropriately for most part, and a 3D model of an oldpalace is overlaid at appropriate position on a defunct base of thepalace. In future work, real-time transmission of omni-directional
images is required as well as real-time extrinsic camera parameter
estimation to realize a real-time augmented telepresence system.
ACKNOWLEDGEMENTS
This work was supported in part by the “Ambient Intelligence”
project granted by the Ministry of Education, Culture, Sports, Science and Technology.
R EFERENCES[1] S. Lawson, J. Pretlove, A. Weeler, and G. Parker. Augmented reality asa tool to aid the telerobotic exploration and characterization of remoteenvironments. Presence, Vol. 2, No. 4, pp. 352–567, 2002.
[2] H. Kim, J. Kim, and S. Park. A bird’s-eye view system using augmentedreality. In Proc. of the Thirty-Second Annual Simulation Symposium,pp. 126–132, 1999.
[3] S. Ikeda, T. Sato, and N. Yokoya. High-resolution panoramic movie
generation from video streams acquired by an omnidirectional multicamera system. In Proc. of IEEE Int. Conf. on Multisensor Fusion andIntegration for Intelligent System (MFI2003), pp. 155–160, 2003.
[4] Y. Yokochi, S. Ikeda, T. Sato, and N. Yokoya. Extrinsic camera parameter estimation based-on feature tracking and GPS data. In Proc.
of Asian Conf. on Computer Vision (ACCV2006), Vol. 1, pp. 369–378,
2006.
[5] E. Paiva, J. Azinheira, J. Ramos, A. Moutinho, and S. Bueno. Project
AURORA: Infrastructure and flight control experiments for a roboticairship. J. of Field Robotics, Vol. 23, No. 2–3, pp. 201–222, 2006.