Interactive Instant Replay: Sharing Sports Experience
using 360-degrees Spherical Images and Haptic Sensation
based on the Coupled Body Motion
Yusuke Mizushina
Keio University
4-1-1, Hiyoshi, Kohoku-ku,
Yokohama-shi, Kanagawa, Japanmizushina@kmd.keio.ac.jp

Wataru Fujiwara
Xvi Inc.
3-15-12, Nishi-Shimbashi,
Minato-ku, Tokyo, Japanfujiwara@xvi.co.jp

Charith Lasantha Fernando
Kouta Minamizawa
Keio University
Keio University
4-1-1, Hiyoshi, Kohoku-ku,
4-1-1, Hiyoshi, Kohoku-ku,
Yokohama-shi, Kanagawa, Japan Yokohama-shi, Kanagawa, Japancharith@tachilab.org
kouta@tachilab.org
ABSTRACT

Tomoaki Sudou
DCAJ
23-3, Chiyoda-ku,Tokyo, Japansudou@dcaj.or.jp
Susumu Tachi
Keio University
4-1-1, Hiyoshi, Kohoku-ku,
Yokohama-shi, Kanagawa, Japantachi@tachilab.org

movement. In contrast, Saga et al. [2] have proposed“HapticVideo” to record the work of a skilled person, toplay it dynamically with kinesthetic sense presentation
devices. In this case, the system converts the environmental
information to the impedance information, visual
information is synchronized with the user's work also canbe presented, thereby realizing the record and replay ofskilled work. From this research, it is believed that therecorded specific experience can be relive by presenting therecorded haptic and visual information is coupled to themovement of the user. So, in this research we propose asystem named “Interactive Instant Replay” that the user canshare sports experience using 360-degrees spherical images
and haptic sensation based on the coupled body motion.

We propose “Interactive Instant Replay” system that theuser can experience previously recorded sports play with
360-degrees spherical images and haptic sensation. The
user wears a HMD, holds a Haptic Racket and experience
the first person sports play scene with his own coupledbody motion. The system proposed in this paper could be
integrated with existing television broadcasting data thatcan be used in large sports events such as 2020 Olympic, toexperience the same sports play experience at home.
Author Keywords

Augmented Sports; Instant Replay; Haptic Interaction.
ACM Classification Keywords

H.5.1. Information interfaces and presentation (e.g., HCI):
Multimedia Information Systems
INTRODUCTION

There have been many technologies that transmit theexperience of haptic sensation up to date. Yamauchi et al.
[1] allow transmitting haptic information (roughness,friction, and softness) over 300 km distance using Internet.
These systems are possible to transmit tactile information inaccordance with the motion of the user in real time by themaster-slave system. However, it is not yet realize to relive
recorded experience with haptic information and body

Figure1. Interactive Instant Replay
INTERACTIVE INSTANT REPLAY

In television sports broadcast, instant replay video is used toshow the fine details of the playing scene. The moment ofmotion of the players that cannot be seen in normal instantreplay can be seen during slow-motion. Thus it is more
appealing to find out the fine details about the game play. In
the Interactive Instant Replay, the regenerated combined
video playing of a sports athlete to the movement of theuser, it is possible to vary the playback speed interactively
based on the user tracked motion. Furthermore haptic
information is also presented in a slow motion manner.
Hashimoto et al. [3] has proposed haptic slow motion
presentation method that generating a slow motion tactile

Permission to make digital or hard copies of part or all of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. Copyrights for thirdparty components of this work must be honored. For all other uses, contact
the Owner/Author.
Copyright is held by the owner/author(s).
AH '15, Mar 09-11, 2015, Singapore, Singapore
ACM 978-1-4503-3349-8/15/03.
http://dx.doi.org/10.1145/2735711.2735778

227

by stretching in the tactile information acquired by the time
axis direction. In this system also we use a similar method
to achieve the slow motion haptic. In our previous study,“Haptic Broadcasting [4]”, use player’s third-person view
video as presenting visual information. In this study, haveused player’s first-person 360-degrees view video more sothat the user can feel as its own experience.

video when recorded, was calculated the rate of tactile
presentation.

Figure 3. 360-degrees Spherical VideoDEMOSTRATIONIn Augmented Human 2015, we demonstrate our system in
2 configurations. The Audience is able to 1) experience thegame play as an audience standing in the middle of thecourt; 2) experience interactive haptic slow motion (Figure.
1) as the players first-person view with 360-degrees
spherical images. Furthermore, the audience can not only
feel the interactivity, but also the impact of the shuttle as
they were playing at a slow motion manner by swinging theracket slowly at their own speed.

Figure 2. Interactive Instant Replay system
IMPLEMENTATIONIn this research we constructed a system to target thebadminton game. The user is able to relive the experience
to hit the badminton shuttle in player's point of view. In“TECHTILE toolkit [5]”, it is proposed to deal with haptics
easily than conventional by treating haptic information asaudio signals. As shown in Figure 1, A tactile microphone(Primo: MX-E4758) is used as the haptic sensor that cancapture low frequencies up to 100 Hz or less vibration
generated during shuttle impact. A wearable wireless audio
transmitter (SHURE: SVX14/PGA31) is attached to theplayer to transmit haptic content between the input racket
and a computer. A Haptic actuator (Tactile Labs: HaptuatorMark II) and an amplifier is used as the display. In addition,to achieve a haptic slow motion effect proposed by
Hashimoto and Kajimoto [4] into interactive gameplay,racket is tracked using an motion tracking system “OptiTrack”. The motion data from the racket is used to change
the playback speed of the recorded haptic content. Thus,
audience can feel entire game play experience as theywould be playing at a slow motion manner by swinging theracket slowly. Thus, by swinging the racket several times itis also at possible to feel the difference of hitting the sweetspot. We used eight GoPro cameras in a helmet
configuration to take first-person view 360-degrees
spherical images of badminton player. To capture the video
in high frame rate, the video was captured at maximum of240fps. The 8 GoPro cameras footage were stitched and360-degrees spherical images were processed in Unity
Game Engine and presented using Oculus DK2.

ACKNOWLEDGMENTS

This work was supported by JST-CREST Haptic Media
Project and Kanto Bureau of Economy, Trade and Industry.
The 360 degrees camera used in this work was developedby NHK Enterprises, Inc. in corporation with Dverse Inc.,and 360 degrees camera design, video shooting and editing
was conducted by HOME360 Inc.
REFERENCES

1. Yamauchi, T., Okamoto, S., Konyo, M., Hidaka, Y.,
Maeno, T., and Tadokoro, S. Real-time remotetransmission of multiple tactile properties throughmaster-slave robot system. In Proceeding of IEEE ICRA
2010, pp. 1753-1760 (2010).
2. Saga, S., Vlack, K., Kajimoto, H. and Tachi, S. Haptic
Video, ACM SIGGRAPH 2005, Emerging
Technologies, Los Angeles, USA (2005.8)
3. Mizushina, Y., Fernando, C., Minamizawa, K. andTachi, S. Haptic Broadcasting - System of Transmitting
the Experience in Badminton, EuroHaptics 2014,Versailles, France (2014.6)
4. Hashimoto, Y., and Kajimoto, H. Slow Motion Replayof Tactile Sensation, In Proceeding of IEEE ICAT 2010,
pp. 51-56 (2010).

By making one-to-one correspondence to the frame numberof the video and the racket of the position, and allowed tovideo presentation, body movement and the coupling. The
moment of racket speed and tactile collision obtained fromthe motion data from the ratio of the racket speed in the

5. Minamizawa, K., Kakehi, Y.,Nakatani, M., Mihara, S.
and Tachi, S. TECHTILE toolkit - A prototyping toolfor design and education of haptic media, Laval Virtual
VRIC 2012, Laval, France(2013)

228