atmoSphere: Designing Cross-Modal Music Experiences Using
Spatial Audio with Haptic Feedback
Haruna Fushimi

Keio University Graduate school ofMedia Design
Yokohama, Kanagawa 223-8526
haruna@kmd.keio.ac.jp

Kazuya Yanagihara

Keio University Graduate School ofMedia Design
Yokohama, Kanagawa 223-8526
yanagihara@kmd.keio.ac.jp

Daiya Kato

Keio University Graduate School ofMedia Design
Yokohama, Kanagawa 223-8526
i.mas.trunk@kmd.keio.ac.jp

Kouta Minamizawa

Keio University Graduate School ofMedia Design
Yokohama, Kanagawa 223-8526
kouta@kmd.keio.ac.jp

Youichi Kamiyama

Keio University Graduate School ofMedia Design
Yokohama, Kanagawa 223-8526

Kai Kunze

Keio University Graduate School ofMedia Design
Yokohama, Kanagawa 223-8526
kai@kmd.keio.ac.jp

Figure 1: (a) atmoSphere / (b) Motion Synchronization / (c) User Experience

ABSTRACTKEYWORDS

We use cross-Modal correspondence -the interaction between twoor more sensory modalities- to create an engaging user experience.
We present atmoSphere, a system that provides users immersive
music experiences using spatial audio and haptic feedback. We
focused on cross-modality of auditory and haptic sensations toaugment the sound environment. �e atmoSphere consists of a
spatialized music and a sphere shaped device which provides haptic
feedback. It provides users imagination of large sound environment
although they feel haptic sensation in their hands. First user feedback is very encouraging. According to participants, atmoSpherecreates an engaging experience.

Haptics, Spatial Sound, Cross-Modality, atmoSphere

CCS CONCEPTS•Human-centered computing →Haptic devices; •Hardware→Haptic devices;

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor pro�t or commercial advantage and that copies bear this notice and the full citationon the �rst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
SIGGRAPH ’17 Emerging Technologies, Los Angeles, CA, USA© 2017 Copyright held by the owner/author(s). 978-1-4503-5012-9/17/07. . . $15.00
DOI: h�p://dx.doi.org/10.1145/3084822.3084845

ACM Reference format:
Haruna Fushimi, Daiya Kato, Youichi Kamiyama, Kazuya Yanagihara, Kouta
Minamizawa, and Kai Kunze. 2017. atmoSphere: Designing Cross-Modal
Music Experiences Using Spatial Audio with Haptic Feedback. In Proceedingsof SIGGRAPH ’17 Emerging Technologies, Los Angeles, CA, USA, July 30 August 03, 2017, 2 pages.
DOI: h�p://dx.doi.org/10.1145/3084822.3084845

1INTRODUCTION

Cross-modal e�ects in human perception are de�ned as interactions
between two or more di�erent human senses. E�ects concerning
audio and vision are very well understood and o�en used in Human Computer Interaction as well as other disciplines. As haptic
feedback technologies are more broadly available, we also see more
and more works that tackle cross-modal e�ects with touch andvision as well as audio [Fujisaki et al. 2015]. Still only very few
researchers use these principles for interaction or user experience
designs [Gu et al. 2015; Israr et al. 2012; Makino et al. 2016].
In our research we explore how to use cross-modal e�ects andcorrespondences to accelerate learning, increase immersion, andrelaxation. �is paper focuses on the later two: immerssion andrelaxation.

SIGGRAPH ’17 Emerging Technologies, July 30 - August 03, 2017, Los Angeles, CA, USA
�e contributions are as follows: (1) we present atmoSphere a
haptic device mapping directional audio to corresponding haptic
sensations on a round, hand-held device, (2) we show two usage
scenarios for atmoSphere, one to augment the spatial perceptionof a music perfromance and one to augment the relaxation during
mindfulness training.

2

SYSTEM COMPONENTS

�e atmoSphere (Figure 1(a)) provides immersive music experience using cross-modal e�ects. Surround haptics[Israr et al. 2012]
proposed that a grid of vibrating actuators can generate moving
tactile strokes on the skin. Po2[Israr et al. 2015] suggested illusionof tactile sensation by providing vibrations on the hand. Reality
Jockey[Fan et al. 2013] proposed that spatial motion of sound withhaptic feedback gave users the illusion that the vibration source
was also moving. It proposed combination of haptic and auditory
sensation augments your spatial perception.

2.1

Sphere Device

�e atmoSphere device consists of a USB audio interface(Roland UA1010), a Preampli�er and a power ampli�er, 8 tactile sound transducers (Acouve Laboratory Vp2), a USB-audio transducer (PLANEXPL-US35AP), a noise-cancelling headphone (Boss �ietComfort35), a 3D printed sphere made of white ABS resin. AtmoSphereconsists of 10 seperate parts (with spacing in between). Each of the
8 side parts has a tactile sound transducer (Figure 1(b)). �ere is acushioning material between a transducer and an ABS resin part toreduce the noise of the transducer vibrations.
A sphere shaped device is linked the image of sound environment.
It is divided into 8 parts and each of them has a tactile soundtransducer. Users can touch 2 or 3 parts with one hand so that they
can feel movement of vibration. It provides users spatial motion oftactile feeling corresponding to spatial audio (Figure 1(b)).

2.2

Spatial Sound Music�e music for atmoSphere was composed with binaural recordedaudio. We conducted binaural recording of various kinds of sounds
like hand claps, whisper voice, and water sound. �ese sounds
were recorded from 8 directions. �e localization of these sounds
are correspond to haptic feedback so that users can feel movementof sound.

3USER EXPERIENCES

We implement two user demonstrations, augmenting a music experience and a mindfulness meditation. Both are designed to give theuser a relaxing feeling.

3.1

Augmented Music

Users wear headphones and hold a sphere so that they hear and feelsound. For example, they can feel someone walking around (Figure1(c)). �e sphere provides vibrations in conjunction with suchstereoscopic localization of sound. It enables users to imagine large
sound environment although they feel vibration in their hands.

3.2

H. Fushimi et. al.

Augmented Mindfulness

We use J!NS MEME, smart glasses that measure relative eye andhead movements, to infer the relaxation state of the user[Ishimaruet al. 2015]. We provide synchronized haptic and audio feedback
with the relaxation state. �e user hears the sound of a circulatingbrush stroke (as in calligraphy) moving around him with circularhaptic feedback in a rhythmic pa�ern (stimulating him to breath
accordingly). �e more relaxed the user gets the lower the soundand the haptic feedback.

4USER FEEDBACK

We showcased atmoSphere in several public events from our graduate school (forums and open labs). Participants’ reactions andcomments indicate that atmoSphere can provided entertaining experiences. However, cross-modal correspondence seems to depend
highly on the type of sound and haptic feedback. Participants reportthat footsteps and rain drops seem to work very well, for example auser commented: ” It really seems that someone is walking aroundme.”

5CONCLUSION AND FUTURE WORK

Combining haptic feedback and spatial audio, we created an engaging user experience considering user comments. We will evaluate
immersion and engagement quantitatively in user studies.
In a next step we will test further spatial audio and haptic feedback design to explore other cross-modal correspondences. Also wewill look more into application areas related to immersion, learning,
localization and relaxation.

ACKNOWLEDGEMENT
�is work is partly supported by JSPS KAKENHI Grant Number
26700018 and JST PRESTO Grant Number JPMJPR16D4.

REFERENCES

Kevin Fan, Hideyuki Izumi, Yuta Sugiura, Kouta Minamizawa, Sohei Wakisaka,Masahiko Inami, Naotaka Fujii, and Susumu Tachi. 2013. Reality jockey: li�ing the barrier between alternate realities through audio and haptic feedback. In
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM,2557–2566.
Waka Fujisaki, Midori Tokita, and Kenji Kariya. 2015. Perception of the material
properties of wood based on vision, audition, and touch. Vision research 109 (2015),185–200.
Heng Gu, Susana Sanchez, Kai Kunze, and Masahiko Inami. 2015. An augmented ereader for multimodal literacy. In Adjunct Proceedings of the 2015 ACM International
Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015
ACM International Symposium on Wearable Computers. ACM, 353–356.
Shoya Ishimaru, Kai Kunze, Katsuma Tanaka, Yuji Uema, Koichi Kise, and MasahikoInami. 2015. Smart eyewear for interaction and activity recognition. In Proceedings ofthe 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing
Systems. ACM, 307–310.
Ali Israr, Seung-Chan Kim, Jan Stec, and Ivan Poupyrev. 2012. Surround haptics:
tactile feedback for immersive gaming experiences. In CHI’12 Extended Abstracts
on Human Factors in Computing Systems. ACM, 1087–1090.
Ali Israr, Siyan Zhao, Kyna McIntosh, JaeKyun Kang, Zachary Schwemler, Eric Brockmeyer, Mark Baskinger, and Moshe Mahler. 2015. Po2: augmented haptics forinteractive gameplay. In ACM SIGGRAPH 2015 Emerging Technologies. ACM, 21.
Yasutoshi Makino, Yoshikazu Furuyama, Seki Inoue, and Hiroyuki Shinoda. 2016.
HaptoClone (haptic-optical clone) for mutual tele-environment by real-time 3D
image transfer with midair force feedback. In Proceedings of the 2016 CHI Conferenceon Human Factors in Computing Systems. ACM, 1980–1990.