2021 IEEE 10th Global Conference on Consumer Electronics (GCCE)

How important is the timing of vibrotactile stimuli
for the excitation of an emotional response foraudio-visual content?
Ibuki Tara, Shogo Okamoto, Yasuhiro Akiyama, and Yoji Yamada
Abstract—In this study, we investigated the effect of the timingof vibrotactile stimuli in the context of audio-visual content, on
feeling of viewers. We used videos of ﬁgure skating performances,and vibrotactile stimuli were supplied to the upper body of theviewer using voicecoil motors, according to the jump scenes inthe videos. The increase in skin conductance of the palm inresponse to the experience was measured, and the experimental
participants scored their level of excitement with a questionnaire.
We performed experiments under conditions with and without
the vibrotactile stimuli. For the vibrotactile conditions, the timingof the vibrotactile stimuli was varied by 200 ms. The increases
in the skin conductance and the questionnaire scores in thevibrotactile conditions were signiﬁcantly high compared to thosein the non-vibrotactile condition. However, the timing of thevibrotactile stimuli had no marked effect on these values.

Fig. 1. Vibrotactile vest with eight high-ﬁdelity voicecoil motors [8].

I. I NTRODUCTION
Vibrotactile stimuli can induce various emotional responses[1]–[4] and have been used to enhance the quality of experience using audio-visual content, such as video games
and movies [5]–[7]. In these studies, physiological indices,such as heart rate and skin conductance, were measured,and questionnaires were used to record subject impressions.
Previous studies investigated the effect of vibration frequencyof the vibrotactile stimuli and the spatiotemporal patterns ofvibrators on the quality of experience. However, the effectof the timing of vibrotactile stimuli has not been thoroughly
investigated. Thus, in this study, we investigated the effectof the timing of vibrotactile stimuli on the quality of viewer
experience of audio-visual content.
The vibrotactile stimuli were presented to the upper bodies of participants at ﬁve intervals according to the various
sections of the audio-visual content. We measured the skinconductance of the palm and employed a questionnaire forevaluating subjective impressions. We veriﬁed the effect ofthe timing by testing the increase in skin conductance andevaluating the scores obtained from the questionnaire.
II. E XPERIMENT
A. Audio-visual and vibrotactile stimuli
We selected six videos of ﬁgure skating performances forthe audio-visual content. All six videos were of different
This study was in part supported by MEXT Kakenhi (20H04263).
All the authors are with the Department of Mechanical Systems Engineering, Nagoya University, Japan. Shogo Okamoto is also with the Departmentof Computer Sciences, Tokyo Metropolitan University.

978-1-6654-3675-5/21/$31.00 ©2021 IEEE

Fig. 2. Five levels of the timing of vibrotactile stimuli.

skaters, each of which started at the entry of the performer andended after the ﬁrst jump and lasted approximately a minute.
We used voicecoil motors (Vp408, Acouve Lab., Japan)
for the vibrotactile stimuli. A vest was used to attach thevoicecoil motors to the left and right sides of the shoulders,
chest, abdomen, and waist of the participants. The vibration
frequency was maintained at 70 Hz, and the amplitude wasmodulated along a triangle wave. The amplitude reached thepeak over 640 ms after the onset and decreased linearly for
another 640 ms. All the voicecoil motors were driven at thesame time. The stimulation was presented once according to
the jump scene in each video.
We experimented under six conditions of vibrotactile stimuli. In one condition, the vibrotactile stimuli were not presented, and in the other conditions, the timing of the vibrotactile stimuli was varied across ﬁve levels. We considered thereference stimulus to be the moment when the peak vibration
amplitude coincided with the moment when the feet of theskater took off from the ground. We also selected four timingsof the vibrotactile stimuli that shifted by 200 ms from thereference level, as shown in Fig. 2. The order of the videos andcombinations of the video and vibrotactile stimuli conditions
were randomized for each participant.

183

2021 IEEE 10th Global Conference on Consumer Electronics (GCCE)

Fig. 3. Mean and standard error of skin conductance and questionnaire scoresfor conditions with and without vibrotactile stimuli. * and ** indicate asigniﬁcant difference at p < 0.05 and 0.01, respectively.

B. Record of the level of excitement
We measured the skin conductance on the palms of theparticipants and used it as an indicator of emotions. The
skin conductance varies according to sweating from arousal
emotions such as excitement and anger. The skin conductancegenerally starts rising within 1 to 2 s after the onset of thestimulus, reaches the peak within 1 to 2 s, and then gradually
decreases. Participants scored their level of excitement at thejump scene on a nine-grade scale questionnaire.
C. Participants
Seven male university students who were unaware of thestudy objectives participated in the experiments.
D. Experimental tasks
The participants experienced the audio-visual content combined with the vibrotactile stimuli at various timings, as well
as at one instance without the vibrotactile stimuli. After each
video, they recorded their subjective impressions on the ninegrade scale.
E. Analysis
We used the increases in the skin conductance from its onsetof the rising to the peak and the questionnaire scores for theanalysis. These values were standardized (z-valued) for eachparticipant. The increase in the skin conductance and questionnaire scores were compared between the aforementionedconditions using t-tests. In this test, we compared the meanvalues of the ﬁve vibrotactile conditions and those of the nonvibrotactile condition. In addition, we veriﬁed the effect of thetiming of the vibrotactile stimuli by comparing the changes
in skin conductance and questionnaire scores under the ﬁvetiming conditions using t-tests with Bonferroni correction.
III. R ESULTS
There was a moderate correlation between the increases inthe skin conductance and questionnaire scores (r = 0.41,
p = 0.0066). Fig. 3 compares the mean of the skin conductance and questionnaire scores between the conditions
with and without vibrotactile stimuli. The vibrotactile stimuli
signiﬁcantly increased the skin conductance and resulted inhigher questionnaire scores. Fig. 4 shows the changes in theFig. 4. Mean and standard error of skin conductance and questionnaire scoresfor different timings of the vibrotactile stimuli. No signiﬁcant differences wereobserved between any two timings.

skin conductance and questionnaire scores for the ﬁve timing
conditions, and no signiﬁcant differences were observed forboth values. Almost all participants reported that the level ofexcitement was subject to change based on different timingsof the vibrotactile stimuli; however, the most effective timing
for the videos varied between individuals.
IV. C ONCLUSION
The vibrotactile stimuli increased skin conductance and alsobore higher questionnaire scores when the participants weremade to watch the ﬁgure skating videos. Therefore the qualityof audio-visual contents can be enhanced by vibrotactile stimuli. However, there were no signiﬁcant differences between
these values for the ﬁve timing conditions of the vibrotactile
stimuli. It suggests that the timing of vibrotactile stimuli might
not be related to the quality of audio-visual contents so much.
Furthermore, we intend to experiment with different types ofvideos to derive more general conclusions about the effect ofthe timing.
R EFERENCES[1] S. Okamoto, A. Tamada, T. Sumiya, and S. Yokoi, “Design of pleasantvibrotactile stimulation through a seat,” in Proceedings of IEEE Global
Conference on Consumer Electronics. IEEE, 2017, pp. 54–55.
[2] H. Hasegawa, S. Okamoto, K. Ito, and Y. Yamada, “Affective vibrotactile
stimuli: relation between vibrotactile parameters and affective responses,”
International Journal of Affective Engineering, pp. 171–180, 2019.
[3] S. A. Macdonald, S. Brewster, and F. Pollick, “Eliciting emotion withvibrotactile stimuli evocative of real-world sensations,” in Proceedings ofInternational Conference on Multimodal Interaction, 2020, pp. 125–133.
[4] C. E. Seim, B. Ritter, K. E. Flavin, M. G. Lansberg, and A. M. Okamura,“Affective ratings of vibrotactile signals in older adults with and without
history of stroke,” in Proceedings of IEEE World Haptics Conference.

1842021 IEEE 10th Global Conference on Consumer Electronics (GCCE)[5] P. Lemmens, F. Crompvoets, D. Brokken, J. Van Den Eerenbeemd,and G.-J. de Vries, “A body-conforming tactile jacket to enrich movie
viewing,” in IEEE World Haptics 2009-Third Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment andTeleoperator Systems, 2009, pp. 7–12.
[6] M. Kim, S. Lee, and S. Choi, “Saliency-driven real-time video-to-tactile
translation,” IEEE Transactions on Haptics, vol. 7, no. 3, pp. 394–404,2013.
[7] G. Karafotias, A. Teranishi, G. Korres, F. Eyssel, S. Copti, and M. Eid,
“Intensifying emotional reactions via tactile gestures in immersive ﬁlms,”
ACM Transactions on Multimedia Computing, Communications, andApplications, vol. 13, no. 3, pp. 1–17, 2017.
[8] I. Tara, S. Okamoto, Y. Akiyama, and Y. Yamada, “Temporal characteristics of skin conductance and subjective intensity in response to unpleasantaudio-visual and vibrotactile stimuli,” in IEEE International Conferenceon Intelligence and Safety for Robotics, 2021, pp. 24–27.

185