KJMR 2010

Augmented Telepresence from the Sky:
AR using Autopilot Airship andOmni-directional Camera
Fumio Okura, Masayuki Kanbara, and Naokazu Yokoya
Nara Institute of Science and Technology (NAIST)

Augmented Telepresence
Augmented Reality

Telepresence

Augmented Telepresence2Augmented Telepresence
using Remote Controlled Vehicle or Robots
Features• A camera equipped with vehicle captures view of remote site.
• User can move around in remote site by controlling vehicle.
Expected information
overlaid on images• Information for remotecontrol of vehicle• Annotation of buildings,
landscape, and so on• Old buildings used in cultural
heritage application

VehicleControl

User

Augmentedview3

Related Work
Augmented Telepresence using ground images[S. Lawson, et al., 02]• Information for remote control is overlaid on
images taken by ground robot
Augmented Telepresence using aerial images[H. Kim, et al., 99]• Uses aerial images taken by a helicopter.
• Controls the helicopter manually.

Camera View

Benefits of Aerial Augmented Telepresence
Much information can be overlaid on the images
because aerial vehicle can capture wide area.
Augmented View4Problem

Problem and Solutions• A standard camera observes the real scene, hence rotation ofvehicle is needed to change posture of user’s view.
• Manual control of vehicle affected by many factors.

It’s difficult for users to change position and posture ofuser’s view to desired viewpoint.
Solution

1. To change position of viewpoint
Use an autopilot vehicle forsupport to change user’s viewpoint

2. To change posture of view
Use an omni-directional camera

5

What is the Best Vehicle
for Aerial Augmented Telepresence?
Altitude

Satellite

10km
1km
100m

10mAirplane
Helicopter
Airship

Images for Augmented Telepresence• Image acquisition with high resolution• Capturing side face of buildings

Capturing from low altitude？

Unmanned Helicopter
Airship
Crane1mCar, etc.
6

Benefits to Use Airship
Altitude

Flight speed
Flight time
Easy to control
SafetyAirplaneHigh

Helicopter
MidAirshipLowHigh

Mid××××××Low√√√

Unmanned airship is suitable for
Aerial Augmented Telepresence.
7

Goal• GoalDevelopment of Aerial Augmented Telepresence
system which enables the users to change positionand posture of the viewpoint easily• Approach– Acquisition of information• Omni-directional aerial images• Position and posture of the airship– Overlaying virtual objects– Automatic control of the airship
8System Overview
User
Input ofthe proposed system

Desired behavior ofairship
Desired posture of
viewpoint
Output of
the proposed system
Augmented view fromdesired position andposture.

Proposed system
Acquisition of information
Position of the airship
Posture of the airship
Omni-directional
aerial images
Position
PostureImagesOverlaying
virtual objects
Virtual objects

GPSGyroscope
Omni-directional
cameraPosition
PostureControlsignalAutomatic controlof airship9Acquisition of Information• Position of airship: GPS• Posture of airship: Gyroscope• Omni-directional images: Omni-directional camera
Camera position and posture information can be calculated
from position and posture of airship.

Omni-directional
cameraGyroscope

Aerial view taken by omni-directional camera

10Overlaying Virtual Objects
1. A perspective image is produced
from an omni-directional image byusing desired posture of user’s view.

2. Position and posture information from GPS and gyroscopeis used to overlay virtual objects to the images.

11Automatic Control of Airship
Control around Designated Points• The airship is controlled around designated points.
• The user can designate the points online.
Designated point
Designated by the user

Designated point

This method is good for going to desired viewpoint.
12Automatic Control of Airship
Reducing Degree of Freedom
Manual Control• Control factors– Throttle– Rudder (right, left)– Elevator– Tilt, and so on

Control supported by autopilot• Control factors
Manual– Throttle– Rudder (right, left)– Elevator
Automatic– Tilt, and so on• Environmental factors• Environmental factors– Wind– Buildings, and so on– Wind– Buildings, and so on

This method is good for “walk in the sky”.

13Experiments• Automatic control of airship– Controlled around two points(230m distance, 70m altitude
from the ground)– Simple proportional control
based on related work
[Paiva, et al., 06] is applied.

• Overlaying a virtual object to aerial images
captured in advance– Off-line implementation– The airship is controlled manually to take images.
14

System Constitution
Unmanned Airship(12m length)

DifferentialGPS

Fiber-optic
GyroscopeOmni-directionalCameraP4-GPS

TISS-5-40

Ladybug3(Nippon GPS Solutions)(Tokimec)(Point Grey Research)

SkyImages
Position
PostureControl signal

PC

Transmitter
Control Signal

GroundD/A converter

15

Experimental Result:

Automatic Control of Airship

UpperSide16

Experimental Result:

Overlaying a Virtual Object“大極殿(Daigokuden)”,an old building of Nara capital wasoverlaid on old base of Daigokuden.

17

Discussion• Airship was controlled automatically.
– Some errors occurred by micro compass in thegyroscope.

• Augmented aerial images were produced.
– Registration errors occurred in some frames.
Drift18

Conclusion
We have proposed an Augmented Telepresence
system which enables the user to change
position and direction of view.
Problem
It’s difficult for users to change position and posture of user’s view
to desired viewpoint.
Solution•Autopilot airship is used to support users changing position ofviewpoint easily.
•Omni-directional camera is used to change posture of view.
19Future Work• Improvement of automatic control• Improvement of registration and image quality– Vision based registration methods for correcting
errors– Adjustment of image quality• Development of real-time Augmented
Telepresence– Realizing real-time processing• Real-time transfer of omni-directional images– Implementation of user interface

20