# Accelerating the Physical Experience of Immersive and Penetrating Music Using Vibration-Motor Array in a Wearable Belt Set

Tomoko Yonezawa, Shota Yanagi, Naoto Yoshida, Yuki Ishikawa

## ► To cite this version:

Tomoko Yonezawa, Shota Yanagi, Naoto Yoshida, Yuki Ishikawa. Accelerating the Physical Experience of Immersive and Penetrating Music Using Vibration-Motor Array in a Wearable Belt Set. 15th International Conference on Entertainment Computing (ICEC), Sep 2016, Wien, Austria. pp.173-186, 10.1007/978-3-319-46100-7_15. hal-01640270

HAL Id: hal-01640270

https://inria.hal.science/hal-01640270

Submitted on 20 Nov 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Accelerating the Physical Experience of Immersive and Penetrating Music Using Vibration-motor Array in a Wearable Belt Set

Tomoko Yonezawa, Shota Yanagi, Naoto Yoshida, and Yuki Ishikawa

Kansai university, Takatsuki-shi, Osaka 5691095, Japan
{yone,k463362,k030913}@kansai-u.ac.jp

**Abstract.** In this research, we aim to create a heightened and physical musical experience by combining electronic sound and time-lagged multiple vibrations that surround the user's neck, chest, and back. The purpose of the research is to elevate an immersive and extended experiences of music as though the sound source had a physical presence. We developed a wearable interface of a vibration-motor array with separately controlled multiple vibration motors to simulate both strong bass sounds and movement of the physical presence of the sound source. The control of intensity and the time differences among the motors produce not only the illusion of spatial presence but also the physical penetration of the ongoing sound. The results of the evaluations showed the effects of 1) the combination of vibration and sound in the musical experience and 2) time differences of the starting timings between the front and back vibrations when creating the illusion of physical penetration as though the sound had a physical presence.

**Keywords:** Sound and vibration, Vibration-motor array, Realistic sensation's expressions, Physical penetration and localization

## 1 Introduction

Since the 1980s when portable music players became popular, we have been to enjoy pre-recorded music with sophisticated and lightweight players in an easy and familiar way. These players have small speakers or need external headphones that are not advantageous for creating a high-powered deep bass sound, although musical experiences in live performances or movie theaters provide a strong presence of sound as a physical phenomenon with sensible vibrations. Such experiences are realized by a powerful output of volume using large-scale devices, such as large speakers or woofers, that are not suitable for personal use.

Humans perceive various stimuli as being multimodally combined. Sub-channel stimuli can change the perception of the main stimulus, as can be seen in McGurk effect [5]. Consequently, we considered that the sound and music experience of live performance or theater with real mega volume is not perceived by the sense of hearing alone. We especially focused on the sense of touch on sound vibration; as Merchel [6] discussed in relation to the influence of the listener's experience
of whole-body vibrations, the natural feeling of sound and music is expected to
provide vibrations to a greater or lesser extent.

To achieve the experience of realistic sound and music of massive volume but
without expensive audio equipment, we have developed a wearable interface with
a vibration-motor array that produces vibrotactile expressions in synchroniza-
tion with the sound of music. Multiple vibrators provide a localized illusion of
both sound and vibration. Furthermore, we focused on an illusion of the physical
penetration of a sound source through the user's body based on fictive physical
presence by the differences in starting times among the vibration motors. Not
just massive volume at live concerts, localized vibration would aid in creating
a realistic experience of immersive opera and parades with actors' movements.
The physical penetration is also expected to emphasize the presence of virtual
objects passing through the user's body in 3D theaters. As described above, the
surrounding vibrations reproduce these physical experiences.

Thus, the localized vibration and illusion of physical penetration would also
provide a novel music experience. Moreover, the sound localization would become
easier not only based on normal differences of level and time (ILD: interaural
level difference and ITD: interaural time difference) among multiple vibrations
but also based on the illusion of the physical penetration from the ongoing sound.

As an application of this configuration, the use of the vibration-motor array
as a monitor combined with sound feedback would help a player in a live perfor-
mance, especially for directional and expressive assignments while the wearable
device does not restrict the user's actions. As another application, the proposed
method is also expected to give an impact on people with hearing difficulties.
The vibration can individually express the rhythm of music even without any
sound.

## 2 RELATED WORKS

First, we refer to studies that have suggested the importance of vibration to
the experience of music. Merchel and Altinsoy [5] discussed the effect of whole-
body vibrations on the listener's experience of listening to concert recordings
with a box-type chair with a large vibrating system in it. Their results showed
that music with vibration, —though only from the buttocks,— was judged better
than simple sound alone. Chang [1] et al. suggested that audio-based vibrotac-
tile feedback on a mobile phone is preferred to simple audio and is recognized as
better sound. Thus, it is presumed that vibration improves the musical experi-
ence, especially with a small device and low volume. Yoshida et al. [11] provided
vibration stimuli in the user's palm using an electromagnetic coil device driven
by low-frequency components of the sound. Their evaluation showed that the
different types of vibrations elevated the listener's sensitivity.

There are researches on educational systems in which vibration is used as
a tool for learning music. Miura and Sugimoto [7] aimed to give each person a
vibration of a particular rhythm of the assigned part to learn on an instrument
in ensemble. They showed that a vibration device could become an effective tool
for rhythm learning. Similar to their application, our proposed system is also
expected to be adopted not only for rhythm learning but also for understanding
sound localization to develop musical experiences, though our main purpose is
not education itself.

Second, there are discussions on various fixed/wearable systems of haptic
expressions for realistic sensation of sound and musical experience. Lindeman
et al. designed VR immersive devices with multiple haptic feedbacks covering
the user's upper body [8, 9]. You et al. [12] developed a sound-specific vibration-
array interface according to a sound stream using 16 oscillators. They found the
shoulder was the most effective place to fix the attachment among the wrist,
forearm, upper arm, shoulder or neck to increase immersiveness. There is also
a palm-sized mobile haptic device with dual-band stimuli to produce low- and
high-frequency bands for musical experiences [2]. The dual-band stimuli showed
predominance in making a good impression; however, the user's action is limited
while holding the device in the hand. In contrast, our wearable system does not
restrict the action of the user. According to their dual-channel feedbacks, we
applied different frequency bands to express various types of music. Sakuragi
et al. [10] reduced vibration motors to a minimum setup with actuators only
at the clavicle to efficiently provide wide vibrations with as far-reaching stimuli
as possible. For more extended experiences, we focused on the vibration-motor
array on not only the clavicle but also other places on the user's body.

The above three music devices [12, 2, 10] cover a narrow surface of the body
from only one direction except Lindeman's VR garment [8], which was not for a
musical experience. Even the 4x4 table vibration array has difficulty in generat-
ing distinct localizations because it covers only a narrow surface. The palm-sized
device and the simple placement on the clavicle also cannot provide vibrations
on a wide or surrounding surface. For an immersive experience and localization
of the sound from various directions, we considered that a wider surface and
arrayed multiple vibrations should be prepared. Accordingly, we adopted mul-
tiple vibration motors placed in various directions from the center of the body.
Also, the three above studies used small vibrators that made it difficult to cre-
ate strong and slow vibrations like real loudspeakers. In contrast, we adopted
two types of vibration motors: low-frequency and strong vibrators for the low
frequency band and a small brushless motor for the midrange frequency band.

Chair-type systems provide vibration stimuli with sound for a similar pur-
pose. Karam et al. [3] suggested a chair-type haptic system fusing eight chan-
nels. The system expresses various vibrations of multiple musical instruments
in music using haptic devices fixed on various parts of the body. There is also
a chair-type vibration machine, Bodysonic [4], to extend the sense of hearing
music. The chair-type device is considered to cover the user's body; however,
the strong sound experience in a live performance provides the music experience
with sound from the front to the back.

Differently to Karam et al., we focus on haptic vibrations from the front, back,
left, and right to elevate sound localization and realistic sensation as though the
user were surrounded with musical sound. The novel point of our proposed sys-
<figure><img src="image_2.png" /><figcaption>Fig. 1. System structure</figcaption></figure>

tem is the differences in starting times of the vibrations among multiple vibration motors to create the illusion of physical penetration as though the sound had a physical presence.

# 3 Physical experience system of sound and music

## 3.1 Summary of system structure

In order to expand the sound and experience of using portable headphones/earphones, we implemented a synchronized vibration-motor array system as follows. The system consists of 1) a sound analysis unit and 2) a control unit of a vibration-motor array (see Fig. 1). First, the sound-analysis unit analyzes the signal of on-going music. Second, the control unit of the vibration-motor array 1) distributes the level of each vibration motor and 2) calculates the time delay for each motor based on the analyzed data and parameter settings. Finally, the original hardware device of the vibration-motor array produces vibrations in synchronization with the sounds of the music.

## 3.2 Hardware structure and vibration-motor-array device

The hardware of the system consists of a laptop PC, a set of headphones, an original wearable set of the vibration-motor array, an AVR microcomputer (Arduino UNO), and an Nch-independent MOSFET.

The proposed vibration-motor array device is configured as a wearable belt set for the chest with four vibration motors and a choker with four vibration motors (Figs. 2 and 3). Two dual-axis vibration motors of 6000 RPM (revolutions per minute), which are assigned to the low-frequency power of the sound, are also placed on the upper rubber band of the chest belt in the vicinity of the third and fourth ribs. Two of the same 6000 RPM motors are placed on the upper rubber band of the belt on the left and right sides of the back 15 cm from the center of the spine. On the choker, two micro-brushless vibration motors of 13000 RPM, which are assigned to the middle-frequency power of sound, are placed on the left and right of the front of neck near the clavicles. Two of the same motors are also placed on the left and right at the back of the neck. The left and right pairs of micro-brushless vibration motors are placed at 8cm intervals.
<figure><img src="image_3.png" /><figcaption>Fig. 2. Vibration motor array device</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 3. Fixed wearable suit with vibration motor array device</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 4. Sound-analysis unit</figcaption></figure>

The AVR microcomputer performs pulse width modulation (PWM) controls for each of eight channels through the Nch-independent MOSFET in synchronization with the volume levels of each frequency band based on the data sent by the sound-analysis unit. The left motors are operated by reflecting the level of sound source from the left channel. The right motors reflect the right channel. Accordingly, the difference in the intensity of vibration between the left and right channels is automatically reproduced, including several-milliscond differences, such as the interaural time difference between the left and right ears. Therefore, localized sound can be felt emphatically by the differences in the intensity of vibrations between the left and right sides.

3.3 Sound-analysis unit

The sampling rate of sound processing is 44100 Hz. The analysis window size
is 1024 samples. The FFT analysis of the sound at every 43 Hz derived the
levels for following frequency bands: the low-frequency band is from 43 to 86
Hz, and the midrange frequency band is from 258 to 301 Hz. The FFT analysis
is processed for each channel (left and right). The four values of the levels,
the left/right and low/midrange frequency bands, are converted into 0 to 255
levels to control each vibration motor. The music sound is output by headphones
connected to the PC at the same time as the vibration stimuli (see Fig. 4). The
system is basically using a sound file, but it can also operate real-time processes
of external input with the audio interface system. Accordingly, the system is
expected to be applied to improvements of realistic sensation in both interactive
sounds during game play and real-time audio in watching TV.
<figure><img src="image_6.png" /><figcaption>Fig. 5. Vibration-motor controls with delay for physical penetration</figcaption></figure>

## 3.4 Vibration-motor control unit

There are two methods of vibration-motor controls. The first is the direct conversion of the frequency power to each motor without special delay except the interaural time difference. The second is the unique method of time-lag among the motors that was adopted.

**Vibration-motor control without additional delay:** First, the sound-analysis unit sends the power levels of each frequency band of both the left and the right channels and converts them into levels from 0 to 255. Second, the vibration-motor control unit operates the PWM controls for each vibration motor. Thus, the eight motors are controlled to correspond to the sound-analysis result.

**Vibration-motor controls with designed delay for generating illusion of physical penetration by sound:** The vibration-motor control unit produces an illusion of physical penetration of sound by time lags in starting the vibrations among the motors. The processing frame rate is set to 120 fps. The delay time is set from 0 to 1000 ms every 100 ms. That is, the second vibration motor starts its operation after 0–1000 ms when the first vibration motor begins its vibrations (Fig. 5). When there is physical penetration by a real object through the human body, it is assumed that the first shock will be perceived on the front surface and the second shock will be perceived on the back surface after the passage through the flesh. Although sound and music do not have any physical presence, the assumed stimuli with the differences in starting times of vibrations are expected to create a physical penetration of sound.

To make the user perceive vibrations by an eccentric rotor, it takes $x - t$ ms ($x$ is the time of the length of the sound, and $t$ is the starting time among the vibration motors), which is a longer period than short-time pulse signals. Accordingly, there is an overlap between the timing of the front and back vibrations. Moreover, the overlapped timing is expected to create the feeling of passage through the body, as we aimed physical penetration.

# 4 EXPERIMENT

## 4.1 Evaluation of the effect of creating realistic sensations with physical vibrations in synchronization with the ongoing music

**Purpose of experiment:** We tried to verify the effectiveness of vibration stimuli for creating realistic sensations of the ongoing sound of the music.
<table><caption>**Table 1.** Conditions in the experiment of synchronized vibrations with music</caption><thead><tr><th></th><th colspan="2">Sound(A) Vibration(B)</th><th></th><th colspan="2">Sound(A) Vibration(B)</th></tr></thead><tbody><tr><td>pt.01</td><td>none</td><td>none</td><td>pt.09</td><td>middle</td><td>none</td></tr><tr><td>pt.02</td><td>none</td><td>weak</td><td>pt.10</td><td>middle</td><td>weak</td></tr><tr><td>pt.03</td><td>none</td><td>middle</td><td>pt.11</td><td>middle</td><td>middle</td></tr><tr><td>pt.04</td><td>none</td><td>strong</td><td>pt.12</td><td>middle</td><td>strong</td></tr><tr><td>pt.05</td><td>low</td><td>none</td><td>pt.13</td><td>high</td><td>none</td></tr><tr><td>pt.06</td><td>low</td><td>weak</td><td>pt.14</td><td>high</td><td>weak</td></tr><tr><td>pt.07</td><td>low</td><td>middle</td><td>pt.15</td><td>high</td><td>middle</td></tr><tr><td>pt.08</td><td>low</td><td>strong</td><td>pt.16</td><td>high</td><td>strong</td></tr></tbody></table>

**Hypothesis:** The vibration stimuli covering the user's upper body will elevate the realistic sense of the sound's presence and power.

**Participants:** Twenty-five university students aged from 19 to 22 years old.

**Conditions:** We prepared 16 conditions according to two factors: A) four different volumes of sound (s none, low, middle, high) and B) four different strengths of vibration (v. none, weak, medium, strong). The volume of sound for each condition was set as follows. The "s. none" means without any sound. The "low" is the lowest sound that the audio interface can output. The "middle" is 15 dB higher than low. The "high" is 5 dB higher than middle. The strengths of vibration for each condition were as follows. The "v. none" means without any vibration. The "medium" is the linear-mapped strength of the vibration synchronized with the volume levels of the sound converted from 0 to 255 levels. The "weak" is 0.5 times the power of medium. The "strong" is 1.5 times the power of medium. A branching table of the conditions is shown in Table 1. The 16 conditions with two factors were randomly arranged for counterbalance.

**Procedures and Instructions:** The participants were instructed to listen to the music for 40 seconds (rock music played for 20 seconds and then changed into ballad music for 20 seconds) through the headphones and our proposed vibration-motor array device for each condition. The time difference between the motors was not generated in this experiment. The combined conditions of sound volume and vibration using each level in each factor were performed in the repeated measurement. When the 40 seconds of music finished, the participants were instructed to evaluate the statements that followed. The other 15 conditions were repeatedly conducted with both listening and evaluation.

**Evaluation Items:** The participants made an evaluation using a five-point rating scale of the relevance (5: very relevant, 4: somewhat relevant, 3: even, 2: somewhat irrelevant, 1: irrelevant) of the following statements.

Q1 You felt you were in a live performance.

Q2 You felt the sound source was close to you.

Q3 You felt the sound surround you.

Q4 You felt the whole space vibrating from the sound.

Q5 You felt the sound had spread.

Q6 You felt the sound had moved.

Q7 You felt the volume was high.

Q8 It was easy to feel changes in the strength of the sound.

Q9 You didn't feel anything unnatural about the vibration.

Q10 The vibration was consistent with the sound.

Q11 It was easy to listen to.

Q12 The sound was hollow.

Q13 You felt a change at exciting points in the music.

Q14 You felt power in the music.

Q15 You felt high.

Q16 Your body moved unconsciously.
<figure><img src="image_7.png" /><figcaption>Fig. 6. MOS results in the experiment of synchronized physical vibrations with music</figcaption></figure>

Q17 You felt a sense of overdose on music.

Q18 You received a warm impression.

Q19 You felt comfortable with the sound.

Q20 You felt comfortable with the vibration.

Q21 You felt comfortable with the harmony of sound and vibration.

**Results:** The results of the mean opinion scores (MOS) are shown in Fig. 6. The results of the analyses of variance (ANOVA) with repeated measurement among the conditions ($p<.05$) are shown in Table 2.

The subjective evaluations showed significance for both factors except questions 11, 12, and 16. No significant result by vibration factor was found without sound. There were no significant differences in the conditions without vibration in the result of the evaluation questions 9, 10, 12, 16, and 21. Almost all of the evaluation items, except those above, showed significant differences between the with-sound and without-sound conditions at every vibration level. Besides the conditions with or without sound/vibration, there were significant differences among different sound intensities. In addition, there were interactions and simple effects between conditions with and without vibration at each volume. There were also significant differences in **high-medium** and **high-weak** strengths of the vibration. However, there was no significant difference in the vibration strength between **medium** and **weak** when the volume was in the **middle** for Q11 and Q19.

## 4.2 Evaluation of the time difference among vibration motors for the illusion of physical penetration of sound

**Purpose of experiment:** We tried to verify 1) whether the new experience of physical penetration of sound could be created by the emphatic time lags of the starting times between multiple vibration motors, such as forward to backward, and 2) whether the proposed time improves the realistic sensation and the sense of sound image localization.

**Hypotheses:** The emphatic time lags of starting times between forward and backward vibration motors 1) generate the feeling of physical penetration of the sounds, 2) improve the realistic sensation and 3) make clear the sense of sound image localization.
<table><caption>**Table 2.** Two-factor ANOVAs in the experiment of synchronized vibrations</caption><thead><tr><th rowspan="2"></th><th colspan="2">loudness(A)</th><th colspan="2">strength of vibration(B)</th><th colspan="2">Interaction</th><th colspan="5">Multiple comparison based on simple main effects</th></tr><tr><th>F(24)</th><th>p</th><th>F(24)</th><th>p</th><th>multi.comp.</th><th>multi.comp.</th><th>A(b2)</th><th>A(b3)</th><th>A(b4)</th><th>B(a1)</th><th>B(a2)</th><th>B(a3)</th><th>B(a4)</th></tr></thead><tbody><tr><td>Q1</td><td>132.0 &lt; 0.01*</td><td>all</td><td>26.0 &lt; 0.01*</td><td>exp<sub>3</sub>-3</td><td>5.50 &lt; 0.01*</td><td>4[1.2,3.1.2]</td><td>exp<sub>4</sub>-3</td><td>exp<sub>4</sub>-3</td><td>exp<sub>3</sub>-2</td><td>—</td><td>[2.3.4]-1</td><td>[2.3.4]-1</td><td>all</td></tr><tr><td>Q2</td><td>147.0 &lt; 0.01*</td><td>all</td><td>17.4 &lt; 0.01*</td><td>[2.3.4]-1</td><td>5.24 &lt; 0.01*</td><td>exp<sub>2</sub>-1</td><td>[2.3.4]-1</td><td>exp<sub>3</sub>-4</td><td>all</td><td>—</td><td>[2.3.4]-1</td><td>[2.3.4]-1.3.2</td><td>4[1.2]</td></tr><tr><td>Q3</td><td>143.0 &lt; 0.01*</td><td>all</td><td>55.6 &lt; 0.01*</td><td>[2.3.4]-1</td><td>5.97 &lt; 0.01*</td><td>4[1.2,3.1.2]</td><td>[2.3.4]-1.4.3</td><td>exp<sub>3</sub>-4</td><td>exp<sub>4</sub>-3</td><td>—</td><td>[2.3.4]-1</td><td>[2.3.4]-1.3.2</td><td>[2.3.4]-1</td></tr><tr><td>Q4</td><td>77.8 &lt; 0.01*</td><td>exp<sub>4</sub>-3</td><td>14.5 &lt; 0.01*</td><td>[2.3.4]-1</td><td>8.93 &lt; 0.01*</td><td>4[1.2]</td><td>[2.3.4]-1</td><td>[2.3.4]-1.3.2</td><td>exp<sub>4</sub>-3</td><td>—</td><td>[2.3.4]-1.2.4</td><td>[2.3.4]-1.3.2</td><td>[2.3.4]-1.4.2</td></tr><tr><td>Q5</td><td>83.4 &lt; 0.01*</td><td>exp<sub>4</sub>-3</td><td>10.5 &lt; 0.01*</td><td>[2.3.4]-1</td><td>2.54 &lt; 0.01*</td><td>[2.3.4]-1.4.2</td><td>exp<sub>4</sub>-3</td><td>[2.3.4]-1</td><td>exp<sub>4</sub>-3</td><td>—</td><td>[2.3.4]-1</td><td>[2.3.4]-1</td><td>[2.3.4]-1</td></tr><tr><td>Q6</td><td>101.0 &lt; 0.01*</td><td>exp<sub>4</sub>-3</td><td>31.0 &lt; 0.01*</td><td>[2.3.4]-1</td><td>6.71 &lt; 0.01*</td><td>[4[1.2],3.1]</td><td>[2.3.4]-1</td><td>exp<sub>3</sub>-4</td><td>exp<sub>4</sub>-3</td><td>—</td><td>[2.3.4]-1</td><td>[2.3.4]-1</td><td>exp<sub>4</sub>-3</td></tr><tr><td>Q7</td><td>162.0 &lt; 0.01*</td><td>all</td><td>11.5 &lt; 0.01*</td><td>4[2.3]</td><td>8.38 &lt; 0.01*</td><td>exp<sub>2</sub>-1</td><td>exp<sub>3</sub>-2</td><td>all</td><td>all</td><td>—</td><td>[2.3.4]-1</td><td>exp<sub>2</sub>-1</td><td>4[1.2,3]</td></tr><tr><td>Q8</td><td>96.1 &lt; 0.01*</td><td>exp<sub>4</sub>-3</td><td>13.5 &lt; 0.01*</td><td>[2.3.4]-1</td><td>4.29 &lt; 0.01*</td><td>4[1.2,3.1.2]</td><td>[2.3.4]-1</td><td>exp<sub>3</sub>-4</td><td>exp<sub>4</sub>-3</td><td>—</td><td>exp<sub>3</sub>-4</td><td>[2.3.4]-1.3.2</td><td>4-1</td></tr><tr><td>Q9</td><td>29.4 &lt; 0.01*</td><td>exp<sub>4</sub>-3</td><td>7.98 &lt; 0.01*</td><td>[2.3.4]-1</td><td>3.83 &lt; 0.01*</td><td>—</td><td>[3[1.2],4.1]</td><td>[2.3.4]-1</td><td>exp<sub>3</sub>-2</td><td>—</td><td>3-1</td><td>[2.3]-1</td><td>[2.3.4]-1</td></tr><tr><td>Q10</td><td>67.1 &lt; 0.01*</td><td>exp<sub>4</sub>-3</td><td>25.8 &lt; 0.01*</td><td>[2.3.4]-1</td><td>8.84 &lt; 0.01*</td><td>—</td><td>[2.3.4]-1</td><td>[2.3.4]-1</td><td>all</td><td>—</td><td>exp<sub>2</sub>-3</td><td>[2.3.4]-1.3.4</td><td>[2.3.4]-1.4.2</td></tr><tr><td>Q11</td><td>127.7 &lt; 0.01*</td><td>exp<sub>4</sub>-3</td><td>2.398 &lt; 0.075 +</td><td>—</td><td>1.96 &lt; 0.046*</td><td>[2.3.4]-1</td><td>exp<sub>4</sub>-3</td><td>[2.3.4]-1</td><td>all</td><td>—</td><td>3-4</td><td>—</td><td>[4.3]-1</td></tr><tr><td>Q12</td><td>18.9 &lt; 0.01*</td><td>exp<sub>4</sub>-3</td><td>1.610 &lt; 0.195</td><td>—</td><td>1.66 &lt; 0.100+</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>Q13</td><td>98.2 &lt; 0.01*</td><td>exp<sub>4</sub>-3</td><td>20.4 &lt; 0.01*</td><td>[2.3.4]-1</td><td>5.48 &lt; 0.01*</td><td>exp<sub>4</sub>-3</td><td>[2.3.4]-1</td><td>[2.3.4]-1</td><td>exp<sub>4</sub>-3</td><td>—</td><td>exp<sub>2</sub>-3</td><td>[2.3.4]-1</td><td>[2.3.4]-1</td></tr><tr><td>Q14</td><td>107.0 &lt; 0.01*</td><td>exp<sub>4</sub>-3</td><td>34.0 &lt; 0.01*</td><td>[2.3.4]-1.2-3</td><td>13.35 &lt; 0.01*</td><td>4[1.2,3.1.2]</td><td>[2.3.4]-1.2-3</td><td>exp<sub>3</sub>-4</td><td>exp<sub>4</sub>-3</td><td>—</td><td>exp<sub>3</sub>-4</td><td>all</td><td>exp<sub>4</sub>-3</td></tr><tr><td>Q15</td><td>94.5 &lt; 0.01*</td><td>exp<sub>4</sub>-3</td><td>10.7 &lt; 0.01*</td><td>[2.3.4]-1</td><td>7.41 &lt; 0.01*</td><td>4[1.2,3.1.2]</td><td>[2.3.4]-1</td><td>exp<sub>3</sub>-4</td><td>all</td><td>—</td><td>exp<sub>3</sub>-4</td><td>4-1.3[1.2]</td><td>[2.3.4]-1.4.2</td></tr><tr><td>Q16</td><td>49.1 &lt; 0.01*</td><td>exp<sub>4</sub>-3</td><td>7.61 &lt; 0.01*</td><td>[2.3.4]-1</td><td>1.84 &lt; 0.063 +</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>Q17</td><td>110.0 &lt; 0.01*</td><td>exp<sub>4</sub>-3</td><td>14.4 &lt; 0.01*</td><td>[2.3.4]-1</td><td>3.20 &lt; 0.01*</td><td>exp<sub>4</sub>-3</td><td>[2.3.4]-1</td><td>exp<sub>4</sub>-3</td><td>exp<sub>4</sub>-3</td><td>—</td><td>[2.3.4]-1</td><td>[2.3.4]-1.3.2</td><td>[3.4]-1</td></tr><tr><td>Q18</td><td>70.4 &lt; 0.01*</td><td>[2.3.4]-1</td><td>2.851 &lt; 0.043*</td><td>3-1</td><td>3.28 &lt; 0.01*</td><td>[2.3.4]-1</td><td>[2.3.4]-1</td><td>[2.3.4]-1</td><td>exp<sub>4</sub>-3</td><td>—</td><td>—</td><td>[2.3.4]-1</td><td>[3.4]-1</td></tr><tr><td>Q19</td><td>106.0 &lt; 0.01*</td><td>exp<sub>3</sub>-4.2</td><td>3.78 &lt; 0.014*</td><td>3-1</td><td>2.36 &lt; 0.0146*</td><td>[2.3.4]-1</td><td>[2.3.4]-1</td><td>[2.3.4]-1</td><td>exp<sub>3</sub>-2</td><td>—</td><td>3[1.4]</td><td>—</td><td>[4.3]-1</td></tr><tr><td>Q20</td><td>51.4 &lt; 0.01*</td><td>exp<sub>4</sub>-3</td><td>21.6 &lt; 0.01*</td><td>[2.3.4]-1</td><td>5.64 &lt; 0.01*</td><td>—</td><td>[2.3.4]-1</td><td>[2.3.4]-1</td><td>all</td><td>—</td><td>exp<sub>3</sub>-2</td><td>[2.3.4]-1</td><td>[2.3.4]-1</td></tr><tr><td>Q21</td><td>93.9 &lt; 0.01*</td><td>exp<sub>4</sub>-3</td><td>21.236 &lt; 0.01*</td><td>[2.3.4]-1</td><td>9.78 &lt; 0.01*</td><td>—</td><td>[2.3.4]-1</td><td>[2.3.4]-1</td><td>all</td><td>—</td><td>3[1.4][2.1.4]</td><td>[2.3.4]-1</td><td>[2.3.4]-1.4.2</td></tr></tbody></table>

¶ expX-X means all combinations except X-X

**Participants:** Twenty-five university students aged from 19 to 22 years old.

**Conditions:** We prepared 11 conditions of within-subject design according to a factor of time lag between forward and backward vibration motors: from 0 ms to 1000 ms every 100 ms. The conditions are shown in Table 3. The 11 conditions were randomly arranged for counterbalance.

**Procedures and Instructions:** The participants were instructed to listen to the sound of a bass drum for 30 seconds with headphones and the vibration-motor array device. The forward vibration motor was synchronized with the real-time sound, and the backward vibration motor was operated after the specified time lags for each condition. Accordingly, the backward vibration motors produced stimuli with one of the 11 different time lags. After each 30-second experience, the participants evaluated the impression for the statements of evaluation items that follow.

**Evaluation Items:** The participants made an evaluation using a five-point rating scale of the relevance (5: very relevant, 4: somewhat relevant, 3: even, 2: somewhat irrelevant, 1: irrelevant) of the following statements.

Q1 The backward vibration was later than the forward vibration.

Q2 The forward vibration was later than the sound.

Q3 The backward vibration was later than the sound.

Q4 You felt as though the sound passed through your body from forward to backward.

Q5 You felt the sound was just like the original sound of a bass drum.

Q6 You felt the bass drum was played in front of you.

Q7 The sound generated from the bass drum hit the front of your body.

Q8 The sound generated from the bass drum passed through your body.

Q9 You felt that the bass drum was close to you.

Q10 You felt a punch of the sound.

Q11 The vibration was natural in connection with the sound.

Q12 You felt the sound cover you.

Q13 The vibration was natural.

Q14 The sound passed through the flesh of your body.

Q15 The sound passed through the bones of your body.

Q16 You felt comfortable with the sound.

Q17 You felt comfortable with the vibration.

Q18 You felt comfortable with the harmony of the sound and the vibration.
<table><caption>**Table 3.** Conditions in the experiment of physical penetration</caption><thead><tr><th></th><th>pt.01</th><th>pt.02</th><th>pt.03</th><th>pt.04</th><th>pt.05</th><th>pt.06</th><th>pt.07</th><th>pt.08</th><th>pt.09</th><th>pt.10</th><th>pt.11</th></tr></thead><tbody><tr><th>Time difference (ms)</th><td>0</td><td>100</td><td>200</td><td>300</td><td>400</td><td>500</td><td>600</td><td>700</td><td>800</td><td>900</td><td>1000</td></tr></tbody></table>

<figure><img src="image_8.png" /><figcaption>**Fig. 7.** MOS results in the experiment of physical penetration</figcaption></figure>

**Q19** You felt comfortable with the feeling of the sound passing through your body.

**Results:** The results of the MOSs and ANOVA with repeated measurements among the conditions (p<.05) are shown in Fig. 7 and Table 4. The evaluations except Q2 and Q7 showed significance. In particular, there were significant results in the 100–200 ms and 400–1000 ms lags. The most important results are Q13 and Q14, which showed the highest values in 200ms lags as evaluations for hypothesis 1).

# 5 DISCUSSION

## 5.1 The effects of vibration on the realistic sensation of sound

From the results of two factorial analyses of variance, the sound with vibration provided a stronger feeling of immersion and realistic sensation than simple sound by itself. Therefore it is suggested that our proposed method of providing vibration from various directions can elevate the illusion of the physical presence of the sound. Moreover, we conjecture that the realistic sensation of the sound is improved and that the loudness of the sound is strongly perceived, especially when the volume of the sound is low.

The harmony between the vibration and the sound may stimulate comfortable feelings; however, if the vibration was too strong, this reduced the level of comfort. Moreover, there was no negative influence on the perception of the sound quality from the vibration. These results confirmed the hypothesis in Section 4.1. For further application for people with hearing difficulties, we should evaluate the musical perception using vibration without sound.

## 5.2 Illusion of the physical penetration of sound

At first, from the results for Q2, there were no perceptual time lags from the musical sound stimuli and the vibration, while the vibration motors required
<table><caption>**Table 4.** One-factor ANOVAs in the experiment of physical penetration</caption><thead><tr><th rowspan="2"></th><th colspan="2">Time difference</th><th rowspan="2">Multiple comparison (from pt.1 to pt.11)</th></tr><tr><th>F(19)</th><th>p</th></tr></thead><tbody><tr><td>Q1</td><td>26.049</td><td>&lt;0.01*</td><td>[3,4,5,6,7,8,9,10,11]-[1,2],2-1</td></tr><tr><td>Q2</td><td>1.566</td><td>0.1174</td><td>-</td></tr><tr><td>Q3</td><td>19.402</td><td>&lt;0.01*</td><td>[3,4,5,6,7,8,9,10,11]-[1,2],2-1</td></tr><tr><td>Q4</td><td>6.845</td><td>&lt;0.01*</td><td>6-11,4-[5,11],3-[5,7,8,9,10,11],2-[5,7,8,9,10,11],</td></tr><tr><td>Q5</td><td>5.959</td><td>&lt;0.01*</td><td>4-11, 3-[5,6,7,8,9,10,11],2-[7,8,10,11],1-11</td></tr><tr><td>Q6</td><td>7.448</td><td>&lt;0.01*</td><td>4-11,3-[5,7,8,9,10,11],2-[5,7,8,11],1-[5,7,11]</td></tr><tr><td>Q7</td><td>1.464</td><td>0.1538</td><td>-</td></tr><tr><td>Q8</td><td>6.021</td><td>&lt;0.01*</td><td>3-[5,8,10,11],2-[8,10,11],1-[10,11],4-11</td></tr><tr><td>Q9</td><td>5.115</td><td>&lt;0.01*</td><td>3-[7,11],2-[7,8,9,10,11],1-[7,11]</td></tr><tr><td>Q10</td><td>5.760</td><td>&lt;0.01*</td><td>3-[10,11],2-[5,8,10,11],1-[5,8,10,11]</td></tr><tr><td>Q11</td><td>9.516</td><td>&lt;0.01*</td><td>4-11,3-[5,6,7,8,9,10,11],2-[5,6,7,8,9,10,11],1-[8,11]</td></tr><tr><td>Q12</td><td>6.809</td><td>&lt;0.01*</td><td>3-[4,5,6,7,8,9,10,11],2-[5,7,8,11],</td></tr><tr><td>Q13</td><td>7.590</td><td>&lt;0.01*</td><td>3-[5,6,7,8,9,10,11],2-[7,8,10,11],1-[8,11],4-11</td></tr><tr><td>Q14</td><td>8.050</td><td>&lt;0.01*</td><td>6-1,4-[8,11], 3-[5,7,8,9,10,11],2-[5,7,8,9,10,11],1-11</td></tr><tr><td>Q15</td><td>8.643</td><td>&lt;0.01*</td><td>6-11,4-[5,7,8,11],3-[5,7,8,9,10,11],2-[5,7,8,9,10,11],1-11</td></tr><tr><td>Q16</td><td>7.343</td><td>&lt;0.01*</td><td>3-[5,6,7,8,10,11],2-[5,7,8,9,10,11],1-[5,11]</td></tr><tr><td>Q17</td><td>7.600</td><td>&lt;0.01*</td><td>4-11,3-[5,7,8,9,10,11],2-[5,7,8,10,11],1-[5,8,10,11],</td></tr><tr><td>Q18</td><td>10.037</td><td>&lt;0.01*</td><td>3-[4,5,6,7,8,9,10,11],2-[5,6,7,8,9,10,11],1-[5,8,11]</td></tr><tr><td>Q19</td><td>10.270</td><td>&lt;0.01*</td><td>4-[5,8,11],3-[5,6,7,8,9,10,11],2-[5,6,7,8,9,10,11],1-[5,7,8,11]</td></tr></tbody></table>

a few milliseconds for perceivable stimuli from the start of the rotation of the motor.

Next, from the results of the factorial analyses of variance, the time lag around 100-200 ms from front to back could provide a feeling as though 1) the sound passed through the user's body without any unnatural or incongruent feeling and 2) a bass drum were being played in front of the user.

Moreover, the participants could perceive the difference in the front and back vibrations even with only 100 ms of time lag. The significant difference in the feeling of physical penetration was confirmed in the statements regarding the penetration of both flesh and bone. Based on the results that the MOS for Q14 was higher than that of Q15, it is possible that the penetration of flesh is stronger than that of bone. This indicates that humans can feel an illusion of the physical penetration of sound.

From these results, the hypotheses in section 4.2 were confirmed. We consider that the illusion of physical penetration of sound elicits both a realistic sensation and localization of the sound with an impression of physical presence. The advantage of multiple sound sources with each different direction of the physical penetration should be discussed in future work.

## 6 CONCLUSIONS

In this paper, we proposed a wearable system with a vibration-motor array in synchronization with sound to create an emphasized experience of music that included both a strong bass sound and the illusion of the physical presence of the sound source. The multiple vibration motors expressed not only strength and localization but also fictive penetration through the user's body. The vibration motors were placed surrounding the user's neck, chest, and back.

We evaluated the effects of 1) the combination of vibration with sound in the musical experience and 2) the differences in starting time between the front and back vibrations on creating the illusion of physical penetration. From the results, it is confirmed that 1) the surrounding vibration in synchronization
with the sound of music produced both a realistic sensation and a punch of the
musical experience and that 2) the appropriate time lags between the front and
back vibration motors created an illusion of physical penetration of the sound;
at the same time, sound localization and realistic sensation were also elevated.

In future work, we should consider the possibility of the proposed device for
musical experience of people with hearing difficulties. It is also suggested that the
proposed vibration-motor array could be combined with the visual presentation
of sound through head-mounted displays.

Acknowledgments

This research was supported in part by JSPS KAKENHI 25700021 and JSPS
KAKENHI 15H01698. The authors would like to thank the participants in the
experiment.

References

1. A. Chang and C. O'Sullivan. Audio-haptic feedback in mobile phones. In *CHI '05 Extended Abstracts on Human Factors in Computing Systems*, CHI EA '05, pages 1264–1267, New York, NY, USA, 2005. ACM.

2. I. Hwang, H. Lee, and S. Choi. Real-time dual-band haptic music player for mobile devices. *Haptics, IEEE Transactions on*, 6(3):340–351, July 2013.

3. M. Karam, C. Branje, G. Nespoli, N. Thompson, F. A. Russo, and D. I. Fels. The emoti-chair: an interactive tactile music exhibit. pages 3069–3074, 2010.

4. A. Komatsu. Body-felt sound unit and vibration transmitting method therefor, Aug. 15 1995. US Patent 5,442,710.

5. H. McGurk and J. MacDonald. Hearing lips and seeing voices. 1976.

6. S. Merchel and M. E. Altinsoy. Auditory-tactile music perception. In *Proceedings of Meetings on Acoustics*, volume 19, page 015030. Acoustical Society of America, 2013.

7. S. Miura and M. Sugimoto. Supporting children's rhythm learning using vibration devices. In *CHI '06 Extended Abstracts on Human Factors in Computing Systems*, CHI EA '06, pages 1127-1132, New York, NY, USA, 2006. ACM.

8. R. W. Lindeman, R. Page, Y. Yanagida and J. L. Sibert. Towards full-body haptic feedback: the design and deployment of a spatialized vibrotactile feedback system. In *Proc. of VRST 2004*, pp. 146–149. 2004.

9. R. W. Lindeman, J. L. Sibert, E. Mendez-Mendez, S. Patil and D. Phifer. Effectiveness of Directional Vibrotactile Cuing on a Building-Clearing Task, in *Proc. of ACM CHI 2005*, pp. 271–280, 2005.

10. R. Sakuragi, S. Ikeno, R. Okazaki, and H. Kajimoto. CollarBeat: Whole Body Vibrotactile Presentation via the Collarbone to Enrich Music Listening Experience. In M. Imura, P. Figueroa, and B. Mohler, editors, *ICAT-EGVE 2015 - International Conference on Artificial Reality and Telexistence and Eurographics Symposium on Virtual Environments*. The Eurographics Association, 2015.

11. R. Yoshida, T. Ideguchi, and K. Ooshima. An examination of a music appreciation method incorporating tactile sensations from artificial vibrations. In *Fourth International Conference on Innovative Computing, Information and Control (ICICIC)*, pages 417–420, Dec 2009.

12. Y. You, H. Lee, M. Y. Sung, K. Jun, and J.-S. Kang. Sound-specific vibration interface: Its performance of tactile effects and applications. In *The 9th International Conference for Young Computer Scientists*, pages 1266–1271, Nov 2008.