Journal of Robotics, Networking and Artificial Life
Vol. 7(3); December (2020), pp. 156–159
DOI: https://doi.org/10.2991/jrnal.k.200909.003; ISSN 2405-9021; eISSN 2352-6386
https://www.atlantis-press.com/journals/jrnal

Research Article

A Tactile Sense Centered Virtual Reality game by using
Biometric feedback
Yasuhiro Suzuki*
Department of Complex Systems Science, Graduate School of Informatics, Furocho Chikusa, Nagoya City 464-860, Japan

ARTICLE INFOABSTRACTArticle History

We construct a Virtual Reality, VR system centered on the tactile sensation. Tactile sensation has a more significant effect on
sensibility than audiovisual. Therefore, if tactile sensation and audiovisual sense are combined, we can create “virtual presence”
in the VR space. In this system, a user’s biological information is linked to the audiovisual data presented. A player plays with
virtual objects through physical body movements. And the system feeds-back biometric information and deliver the change of
biometric data through the sensory presentation to the user.

Received 11 November 2019
Accepted 25 May 2020
Keywords
Virtual reality
gametactile sensebio-feedback© 2020 The Authors. Published by Atlantis Press B.V.
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION
In conventional Virtual Reality (VR) technology and applications,visual information is mainly used [1], while auditory and tactile
information have been used side by side. In this research, we construct a VR system that focuses on tactile sense and secondary toaudiovisual perception. The tactile sensation has a strong influence on sensitivity as compared with the audiovisual sense [2]. For
example, it has shown that vibrotactile plays a significant role in
the perception of music, and by amplifying certain vibrations in
a concert venue or music reproduction system can improve themusic experience [3]. And it has reported that higher visual realism can also give more robust fear responses which may be indicative of stronger presence [4]. Therefore, by combining vibrotactile
and visual stimuli, we aim to provide the virtual presence to thevirtual object in the VR space. To do that we measure the user’s
biometric information [5] and make a game controller tangible andcreate a virtual object in cooperation with audiovisual information.
Through feedback, the sensory presentation via biometric information, virtual items to generate a sense that users are cooperatingwith virtual objects.

2. HEARTOID
The purpose of this research is to give a sense of presence to a virtual object in the VR space by feeding back the baptized biologicalinformation based on the idea of the “heart picnic [6].” Therefore,
we created a virtual object in the VR space that made the user’s
biometric information tangible and constructed a tactile VR gamesystem, the “Heartoid [7]”, using the virtual object.
Email: ysuzuki@nagoya-u.jp*This system is a VR game that touches a virtual beating object(sphere) that makes the player’s heart sound tangible. This spherecorresponds to a vibratory tactile sensation presentation device inthe heart picnic, and by touching this sphere existing in the VR space
via the controller, the player can touch the tunable heart sound. The
tactile presentation device used was a tactile presentation device
built into the controller of HINE’s VINE VR. The sphere regularly
vibrates finely based on the results of real-time measurement andanalysis of the heartbeat sound and the movement (acceleration)of the player, and the amplitude of the vibration synchronized withthe magnitude of the heart sound — the subtle vibration selectedaccording to the player’s biological condition and movement.
The player’s body and the sphere are connected by a thin thread(Figure 1). The thread visually complements the tunable heartsound, suggesting that the player’s heart sound sent to the sphere,
and suggesting a relationship between the player and the area withthe thread. While the string is connected, the area floats in the air
against gravity and is alive.

2.1. Flow of the Game
2.1.1. Introduction
The player touches the sphere and feels the heartbeat of participant’s heart through the vibration of the sphere.

2.1.2. Target game (about 3 min)The goal is to hit the sphere and break 50 targets in time (Figure 2).
The heart rate fluctuates during the game. Also, the larger thesphere, the easier it is to hit multiple targets at the same time.

Y. Suzuki / Journal of Robotics, Networking and Artificial Life 7(3) 156–159

Figure 1 | From the game screen. The user and the heart (sphere) areconnected by wires. The user feels the heartbeat from the tactile feedbackof heart sounds.

157

Figure 3 | Ending of the game. When the target application game ends,
the controller becomes scissors. The user cuts the sphere connected tohimself with the scissors using the scissors. When the thread reduced, thesphere falls by gravity. The sphere does not beat and does not return to me
when thrown.

microphone attached to the experimental heart sound meterHBS-NA) and connected to a PC with a stereo mini-plug. A 500 Hz
low-pass filter was used to reduce microphone noise and improveheart sound detection performance.

3.1. Realtime Bio-feedback
3.1.1. Analyzing body movements
Figure 2 | Snapshot of the game. The black circle in the center is thesphere corresponding to the heart. The ring around it is the target. The
user throws a sphere to break the target. Fifty targets generated.

Therefore, it is necessary to make physical movements and makeutterances. Therefore, vocalization and respiratory heart rate fluctuate. The change of the mental and physical state is presented bytactile sensation and made tangible.

2.1.3. Ending
The player cuts the thread connecting the sphere to himself. At theend of the game, the controller becomes scissors. Use the scissors
to cut the thread. When the thread is reduced, the sphere drops
according to gravity and stops vibrating. The ball can be grabbed
and thrown but does not respond in response to the user’s heartrate. Also, throwing a sphere will not return to the player’s hand(Figure 3).

3. SYSTEM COMPONENTS
We used the HTC Vive for the VR head-mounted display, controller,and tracking system. For the tactile presentation, we used the tactile
presentation device built into the controller unit attached to Vive.
The player’s heart sound was acquired by using a stethoscope withbuilt-in microphone (Yagami Co., Ltd., Nagoya, Japan, heart soundTo implement the whole system, we used the Unity, which is aplatform to create video game by Unity Technology Inc.
In Unity, the velocity and angular velocity of the controller are saved
as a three-dimensional vector and dynamically updated within afixed frame of physics calculation every 0.01 s. Hence, to obtain
the acceleration, the difference between the velocity and angularvelocity vector in the previous frame and the current velocity andangular velocity vector is taken and divided by the time (0.01)
between the frames to obtain the acceleration and angular acceleration vector.
By taking the magnitude of these acceleration and angular acceleration vectors, a three-dimensional vector is collected into a scalar.
After that, one scaled data is obtained by multiplying the volumeof acceleration and the magnitude of the angular acceleration ofthe same frame by applying appropriate correction terms. The general expression of the nth frame (1), where Vn denotes Controller’svelocity vector at the nth frame, angVn: Controller’s angularvelocity vector at ∆t denotes time between frames k and l, that arearbitrary 32-bit floating-point number, while V0 = (0, 0.0).	   An = kVn − Vn −1
angVn − angVn −1
(1)+l∆t∆t

The system obtains time-series data of acceleration, by adding thisdata to the end of the array over 20–30 physics calculation frames
until the tactile analysis is completed. Since the input array (one-­dimensional time series data) is cleared once for each analysis, theacceleration data at the previous examination does not affect thecurrent analysis result.

158

Y. Suzuki / Journal of Robotics, Networking and Artificial Life 7(3) 156–159

3.1.2. Analyzing biometric data
A developed application server for biometric information analysis
is set up, and POST, which is a request method supported by Hyper
Text Transfer Protocol (HTTP); HTTP loads hypertext link in theWorld Wide Web (WWW). POST delivers binary time series data;The server analyzes biometric information based on the request
and returns the result to the client-side as an HTTP response.
HTTP communication of this system is implemented by WWW classwhich is Unity HTTP communication wrapper class. Since HTTP
communication by WWW class is executed by the coroutine function, which is an asynchronous process in Unity, the game simulation
execution loop does not stop during biometric information analysis.
However, there is a risk that the execution frame rate will be
reduced due to the overhead of converting a vast numerical array
into a byte string for a request. For this, the transformation wasdivided into several frames and divided. To confirm its effectiveness, we requested data to a local application server and measured
the average time taken for a response to be about 0.3 s. Therefore,
it is possible to analyze the tactile sensation biometric information
about every 0.3 s while the game is running.
Note that this Web application analysis method enables the development of applications with real-time biometric information analysis functions in various development environments, as long as it isan environment that supports HTTP communication.
In other words, not only Unity, but also Unreal Engine, which hasa proven track record for higher-end applications, and Cocos-2dx,
which has a proven track record in 2D games, HTTP communication is supported as standard, so this real-time tactile analysis ispossible. There is high versatility.
Also, when building an application development environment suchas a game engine for in-house use, this real-time tactile analysis
can be easily performed by supporting HTTP communication andcreating a wrapper.

3.1.3. Tactileization
This system uses vibrotactiles to perform tactileization. For tactileization of biometric information, it is necessary to select the vibrotactile sensation that is output according to the analysis result ofbiometric data. In this system, the biological information was classified, and the vibrotactile sense assigned to each cluster in advancewas presented.
For the selected vibrotactile, the output voice block was analyzed
for each execution frame, the value of vibration intensity was determined for each block, and it was output to the vibration of a device
such as a controller. The reason why one value is obtained at the timeof vibration output is that the vibration Application Programming
Interface (API) in devices such as game controllers can often output
only one value of vibration strength (driving motor driving force).

4. CONCLUSION
In this research, we demonstrated the possibility of developinga haptic content game that affects sensibility by measuring andanalyzing biological information and presenting haptics in realtime (making haptic). A user evaluation test will be performedin the future. Still, a user who tried for the preliminary survey
said, “The sphere (corresponding to the heart) is alive because
the changes due to physical movement are fed back by tactile
feedback. Some comments have been foreseen that it affects sensibility, such as “I get a feeling.” And “I hesitate to cut the wireat the end.”
As future tasks, in addition to user evaluation, improvement of theexecution speed of biological information analysis in this system
will be significant for broader application. At present, it is not possible to support game designs that require review at every gameframe (less than 0.0032 s for 30 FPS). Therefore, by speeding up
the processing, it is desired to improve the method so that thetechnique can be applied to a game application requiring highspeed processing such as a shooting game.

CONFLICTS OF INTEREST
The author declares no conflicts of interest.

ACKNOWLEDGMENT
The author would like to thank Mr. Yoshihito Ushida, for his
sophisticated skills and outstanding efforts. This research was supported by the Grant-in-aid for Scientific Research No. 15K00268
and 15K2105.

REFERENCES
[1]

[2]
[3]

[4]

[5]

[6]
[7]

J. Radianti, T.A. Majchrzak, J. Fromm, I. Wohlgenannt, A systematic review of immersive virtual reality applications for higher
education: design elements, lessons learned, and research agenda,
Comput. Educ. 147 (2020) 103778.
C. Basdogan, F. Giraud, V. Levesque, S. Choi, A review of surface haptics: enabling tactile effects on touch surfaces, IEEE
Transactions on Haptics, IEEE, 13 (2020), 450–470.
S. Merchel, M.E. Altinsoy, Auditory-tactile experience ofmusic, in: S. Papetti, C. Saitis (Eds.), Musical haptics. Springer
series on touch and haptic systems, Springer, Cham, 2018,
pp. 123–148.
J. Hvass, O. Larsen, K. Vendelbo, N. Nilsson, R. Nordahl, S. Serafin,
Visual realism and presence in a virtual reality game, 2017
3DTV Conference: The True Vision - Capture, Transmissionand Display of 3D Video (3DTV-CON), IEEE, Copenhagen,Denmark, 2017, pp. 1–4.
M. Inazawa, X. Hu, Y. Ban, Biofeedback interactive VR systemusing biological information measurement HMD, SIGGRAPH
Asia 2019 Emerging Technologies (SA ’19), Association for
Computing Machinery, New York, NY, USA, 2019, pp. 5–6.
J. Watanabe, Y. Kawaguchi, A. Sakakura, H. Ando, Heart picnic:
workshop on touching the beat, Trans. Virtual Reality Soc. Japan
16 (2011), 303–306.
Y. Ushida, Y. Suzuki, Heartoid, touch and feel the heart andthrow it, Tactile Virtual Reality Game, IPSJ, Interaction 3B55
(2018), 1086–1088.

Y. Suzuki / Journal of Robotics, Networking and Artificial Life 7(3) 156–159

AUTHOR INTRODUCTION
Dr. Yasuhiro Suzuki
He is an Associate Professor of Graduate
School of Informatics, Nagoya University
Japan. graduated from Japan Advanced
Institute of Science and Technology in
1995, received Doctor degree of Infor­
matics from Kyoto University in 2001.

159