Telexistence Drone: Design of a Flight Telexistence Systemfor Immersive Aerial Sports Experience
Hirohiko Hayakawa
Charith Lasantha Fernando
MHD Yamen Saraiji
h.hayakawa@kmd.keio.ac.jp
charith@kmd.keio.ac.jp
yamen@kmd.keio.ac.jp
Kouta Minamizawa
Susumu Tachi
kouta@kmd.keio.ac.jp
tachi@tachilab.org
ABSTRACTIn this paper, a new sports genre, “Aerial Sports” is introduced
where the humans and robots collaborate to enjoy space as awhole new field. By integrating a flight unit with the user’s
voluntary motion, everyone can enjoy the crossing physicallimitations such as height and physique. The user can diveinto the drone by wearing a HMD and experience the provided binocular stereoscopic visuals and sensation of flightusing his limbs effectively. In this paper, the requirements
and design steps for a Synchronization of visual informationand physical motion in a flight system is explained mainly
for aerial sports experience. The requirements explained inthis paper can be also adapted to the purpose such as searchand rescue or entertainment purposes where the coupled body
motion has advantages.

Figure 1. Telexistence Drone System Overview

As the applications, we can think of a race through a course
created in the space, where humans and robots can cooperate.
In this paper, we call this kind of sports “Aerial Sports”. In
previous research about telexistence [4] the authors mention
that to provide a sense of presence in a remote environment,
there should be visual, auditory and kinesthetic sensation coupled with the body. TELESAR V [1], shows the quality ofawareness that a person can experience through synchronousvisual, auditory, haptic and kinesthetic sensations. The significance and diversity of sport using the air [2] [?] explains
new kinds of sports in air. However, flight telexistence or
immersive aerial sports are not realized. Therefore, in thisresearch, we use visual, auditory and kinesthetic sensationto achieve being in a flight experience and further to find out
the requirements of a flight telexistence system for immersive
aerial sports experience.

Author Keywords

Aerial Sports, Drone Telexistence, Flight Experience,
Augmented Sports, Virtual Reality
ACM Classification Keywords

H.5.2. Information Interfaces and Presentation (e.g. HCI):
User Interfaces:Interaction styles
INTRODUCTIONHumans enjoy sports everyday in their daily life. But these
sports are rather limited in space and the players only have
limited freedom. Sports like Skydiving, Sea Diving gives
much immersive experience to the player by merging the 3D
space in to their sports boundaries. However, these extreme
sports are rather difficult and dangerous to try by many people
at once. If we can provide an unlimited space for the playerswith safety, there could be many interesting new sports andeven humans could see the potential of getting supernatural
powers during sports. If there was a robot that could fly in thespace and we could experience being that robot in the sky,
what would we do?

Virtual reality contents such as Birdly [3], allows the user
to experience the feeling of flight with a HMD and the control unit. But it has no effect on the real world and therefore
could not be used for creating a real-time sports experience.
As shown in Figure 1, by integrating the flight unit with theuser and thus crossing physical limitations such as height andphysique, anyone can become a player and enjoy space as a
new field of sports while staying on the ground.

Graduate School of Media Design, Keio University.
Address: 4-1-1 Hiyoshi, Kohoku-ku,Yokohama-shi, 223-8526 JapanSYSTEM DESCRIPTION

Unlike FPV Drone (First Person View) systems commerciallyavailable where the latency is not a critical issue, to experience of flying inside the drone, captured 3D stereoscopic
visuals has to be streamed back to the ground user in realtime. Secondly, the ground user’s head position and orientation has to be exactly matched with the drone’s motion sothat the user could feel the sensation of controlling the dronePermission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the Owner/Author.
Copyright is held by the owner/author(s).
AH ’15, Mar 09-11, 2015, Singapore, Singapore
ACM 978-1-4503-3349-8/15/03.
http://dx.doi.org/10.1145/2735711.2735816

171of Aerial Sports “Earthlon” where human and drone collaborate to run and fly in different stages in the game. In thefirst part the user run like in a normal race and then he uses
his telexistence drone to fly to the goal in the second part. It
suggests that, Aerial Sports not only could expand the possibilities of play but also the playing tactics that were normally
impossible so far.
The system is in early stages. However, as shown in Figure 4
full flight tests were conducted to evaluate the system performance as well the drone flight experience. During the test
flights, users were able to control the drone with their natural
movements and drive the drone in a desired path.
Figure 2. System Block Diagramwith his voluntary motion. The aim of the visual information transmission (streaming) system is to provide equivalentvisual sensation to the ordinary human vision such as active
viewpoint, motion parallax, binocular stereoscopy.
Figure 2 shows the overall system structure of the flight telexistence system. A 1 DoF (Tilt) Robotic head is attached toa Hexacoptor Drone system (Model No: DJI F550 FlameWheel ARF). Two USB 2 Cameras (Model No: LogitechC615) with a Wide Conversion (x0.5) lens fixture is used tocapture a wide field of view (HV: 128 ×72) from the sky.
The visual frames were grabbed at a rate of 30FPS by theOn Board PC (Intel NuC D53427RKE), encoded with H.264
and sent back to the ground player with GStreamer pipeline.
The end to end video latency is measured to be 150ms when
the distance is within the range of 100m. Players motion
is captured from the Oculus DK2 (position, orientation) andsent to the drone as a PPS (Pulse Per Second) signal to control the altitude, roll, pitch, pan and tilt (1 DoF Robot). A
commercial high power Wireless LAN router (Model No:WZR-1750DHP, IEEE802.11ac ch110, BW: 40MHz, Burst:600Mbps) is used to receive the transmitted data from the twocollinear 5GHz antennas mounted at the Drone.

Figure 4. Telexistence Drone Mobile System Installed on a Park

CONCLUSION

In this paper, a new sports genre, “Aerial Sports” is introduced
where the humans and robots collaborate to enjoy space as awhole new field. The effectiveness of the system has been realized by implementing “Telexistence Drone” a Hexacoptor
with onboard robotic camera head with stereovision. Several
test flights has been conducted to evaluate the system performance and enjoyment of crossing physical limitations such asheight and physique through voluntary motion. In the future,
the validity of flight type Aerial Sports has to be conducted.

To carry the onboard PC, battery for 25min of flight andPower supplies, the system was modified with special motors, carbon fiber propellers etc... The system is capable oflifting a payload of 4.0kg and the current total payload of thesystem was measured to be 3.3kg.

ACKNOWLEDGMENTS

This work was supported by JSPS KAKENHI Grant#26240030.
REFERENCES

1. Charith Lasantha Fernando, Masahiro Furukawa, T. K. S.
K. K. S. K. M., and Tachi, S. Design of telesar v fortransferring bodily consciousness in telexistence.
IEEE/RSJ International Conference on Intelligent Robotsand Systems (October 2012), 5112–5118.

AERIAL SPORTS EXPERIENCE

2. Kei Nitta, K. H., and Rekimoto, J. Hoverball: augmented
sports with a flying ball. In AH’14 Proceedings of the 5thAugmented Human International Conference (2014).
3. Rheiner, M. Birdly an attempt to fly.
Figure 3. Aerial Sports Concept, Earthlon constituted by run and flight

4. S.tachi. Telexistence. World Scientific, 2010.
http://www.worldscibooks.com/compsci/7079.html.

As explained in the system description, the flight unit can follow the user according to his motion. For example, similar tothe concept of triathlon, as shown in Figure 3, we can think

172