様式２

平成２９年度 生体医歯工学共同研究実施報告書

受理年月日
受理番号

2023

平成 30 年 3 月 15 日
生体医歯工学共同研究拠点 研究所長会議 議長 殿

共同研究代表者
所属機関 University of Ontario Instituteof Technology (UOIT)
職
名 Associate Professor
氏
名 Bill Kapralos
印
勤務先所在地 〒L1H 7K42000 Simcoe Street North, Oshawa,
Ontario L1H 7K4, Canada
電話番号 +1-905-721-8668 x2882
ＦＡＸ番号 +1-905-721-3167
E-mailアドレス : Bill.Kapralos@uoit.ca
下 記 に よ り 、 共 同 研 究 の 実 施 報 告 を 致 し ま す。
記
（和）
研

究

題

目
（英）Examining the effect of sound on haptic fidelity perception in virtual environments

研

究

領

域

研

究

期

間

１．生体材料に関する基礎・応用研究
２．生体工学に関する基礎・応用研究
３．生体機能分子に関する基礎・応用研究
４．化学・電気・機械・材料工学の生体応用研究

平成 29 年 4 月 17 日 ～ 平成 30 年 3 月

31 日

研究組織
氏

名

Bill Kapralos

Michael Jenkin

Robert Allison

Julita Vassileva

Hirokazu Taki

所属機関・部局等
Faculty of Business and Information Technology,
University of Ontario Institute of Technology,
CanadaDepartment of Electrical Engineering andComputer Science, Lassonde School ofEngineering, York University, Canada
Department of Electrical Engineering andComputer Science, Lassonde School ofEngineering, York University, Canada
Computer Science Department, University of
Saskatchewan, Canada
Faculty of Systems Engineering,

職

名

役割分担

Associate
Professor

LeaderProfessor

Participant

Associate
Professor

Participant

Professor

Participant

Professor

ParticipantWakayama University, Japan
Hidetoshi Nonaka
Karen Collins

Adam Dubrowski

Noriyuki Matsuda

Magdalina Todorova

Michael Vynnycky

Patrick Hung

Miguel Vargas Martin

Andrew Hogue

Sanshiro Sakai
Hiroshi Inokawa
Kamen Kanev
Alvaro Joffre Uribe

Robert Shewaga

RobertCodd-Downey

Graduate School of Information Science andTechnology, Hokkaido University, Japan
Canadian Centre of Arts and Technology at the
University of Waterloo, Canada
Divisions of Emergency Medicine and Pediatrics,
Faculty of Medicine, Memorial University,
CanadaFaculty of Systems Engineering,
Wakayama University, Japan
Department of Computer Science, Faculty ofMathematics and Informatics, Sofia University,
Bulgaria
Division of Metals Casting, Department ofMaterial Science Engineering, Royal Institute ofTechnology, Sweden
Faculty of Business and Information Technology,
University of Ontario Institute of Technology,
CanadaFaculty of Business and Information Technology,
University of Ontario Institute of Technology,
CanadaFaculty of Business and Information Technology,
University of Ontario Institute of Technology,
CanadaFaculty of Informatics, Shizuoka University,
Japan
Research Institute of Electronics, Shizuoka
University, Japan
Faculty of Informatics, Shizuoka University,
Japan
Multimedia and Industrial Engineering, Nueva
Garanada Mil. University
Faculty of Business and Information Technology,
University of Ontario Institute of Technology,
CanadaDepartment of Electrical Engineering andComputer Science, Lassonde School ofEngineering, York University, Canada

Professor

Participant

Associate
Professor

Participant

Associate
Professor

Participant

Associate
Professor

Participant

Associate
Professor

Participant

Head of Division

Participant

Associate
Professor

Participant

Associate
Professor

Participant

Associate
Professor

Participant

Professor

Participant

Professor

Participant

Professor

Participant

Assistant
Professor

Participant

Graduate
Student

Participant

Graduate
Student

Participant（共同研究をした教員名を記載）

生体医歯工学共同研究拠点内対応教員

Hiroshi Inokawa, Professor
Research Institute of Electronics,
Shizuoka University, Japan

研究成果
Introduction
The simulation of the sense of touch falls under the field of haptics which collectively refers to machine touch andhuman-machine touch interactions and includes all aspects of information acquisition and object manipulation
through touch by humans, machines, or a combination of the two within real, virtual, or teleoperated scenarios [1].
Haptic feedback plays an important role in our daily interactions [2]. The sense of touch allows us to interact with
objects, and in conjunction with other senses (e.g., sight and hearing), the sense of touch provides multimodal

feedback to better accomplish a variety of tasks [3]. Since the birth of virtual reality [4], research has focused on
stimulating the senses and increasing our interactions with computer-generated worlds, through the use of avariety of input/output technologies including head mounted displays [5], 3D sound [6], smell sprays [7], andelectro-mechanical devices to provide haptic feedback [8]. However, early virtual reality (VR) input/output
devices were bulky and cost-prohibitive thus limiting their use and application to research institutions andindustry [3]. In the real world, senses interact with one another and alter each other’s processing and ultimately
perception. Our prior work that has examined the effect of sound on visual fidelity perception has shown a strong
influence of sound on the visual scene (e.g., see [2]). However, very little prior work has considered multi-modal
interactions with the other senses, particularly the sense of touch. Here, we build upon our prior work byconsidering the effect of sound on haptic fidelity perception. More specifically, what effect does sound have on
the perception of touch, and more importantly, can sound lead to an increase in the perception of haptic (touch)fidelity in a virtual environment? Our goal is to determine whether sound can be used to increase the perception of
haptic fidelity inherent in low-end consumer-level haptic devices to allow the use of such devices (coupled withthe appropriate auditory cues) in applications that require higher fidelity at a fraction of the associated cost.
Within the scope of this Collaborative Research Project, we have conducted an experiment that has examined theeffect of sound on haptic fidelity perception within the context of a virtual drilling task (virtual drilling through a
block of wood). This scenario was selected as haptic feedback plays an important role during drilling, providing
the user with information regarding the drilling process (e.g., how far the drill bit has traveled, whether the drill
bit is moving through the material or is stuck, and whether something is wrong with the device or the drill bit).
Furthermore, drilling is a fundamental component of various medical procedures including dental bone surgery[9], endoscopic endonasal drilling [10], dental implant surgery [11], orthopedic surgery (e.g., temporal bone
surgery, knee replacement and to create a hole for screw insertion to fix bone fractures) [14] [12], and needle
insertion [13] amongst many others. Although preliminary, our results do indicate that sound can influence haptic
fidelity perception albeit, greater work remains.
Experimental Procedure
Overview: The Unity3D game engine was used to develop the virtual drilling scenario (graphical user interface
and the virtual environment). Haptic feedback was generated with the Novint Falcon, a consumer level, a
low-fidelity haptic device that provides three degrees-of-freedom (3DOF) and force feedback over the threecoordinate axes upon contact with virtual objects. The device’s actuators allow touch feedback and it can be
programmed relative to the configured mechanical properties of the virtual object. The Novint Falcon has aworkspace of 10.6 cm3 with a maximum force of 8.9 N, a resolution of 400 dpi, and an approximate cost of $250
USD. A graphical overview of the virtual drilling scenario architecture is provided in Fig. 1.

Fig. 1. Virtual drilling scenario system architecture.
The system is defined with respect to the drilling procedure and the interactions required to accomplish it. With

respect to the drilling procedure, we identify the following parameters required by the system: i) drill speed, ii)
type of drill bit, iii) drilling sounds, and iv) material (wood) mechanical properties. The system receives the user’s
input in the form of thrust movements towards/away from the (virtual) material being drilled andactivation/deactivation of the (virtual) drill. The movement and button pressing status are sent to the haptic
software which will respond accordingly based on the mechanical system (e.g., drill and material properties). A
graphical user interface (GUI) provides visual feedback that includes visuals, sound, and haptics providing theuser with cues regarding the drilling process and when the drilling task has been completed. Additionally,
information (e.g., drilling depth, and user movements) regarding the drilling process is collected and stored.
The auditory stimuli consisted of five auditory sound condition that were either non-contextual (i.e., not related tothe task) or contextual (i.e., related to the task). The four non-contextual conditions consisted of: i) no sound atall, ii) white noise, iii) classical music (“Sarabande” by Bach), and iv) heavy metal music (“Holy Wars” by
Megadeth). The one contextual auditory condition consisted of i) drill sound while drilling through wood. The
drilling sound comprised a recording of a drill drilling through wood. The recording was made in an Eckel
audiometric room to limit any external noise (air condition “hums”, etc.) and reverberation of the generated
sounds within the environment, at a sampling rate of 44.1 kHz. The white noise sound was sampled at a rate of
44.1 kHz and band-pass filtered using a 256-point Hamming windowed finite impulse response (FIR) filter withlow and high-frequency cut-off frequencies of 200 Hz and 10 kHz respectively. All auditory stimuli were
presented to the participants with a pair of headphones.

The virtual drilling scenario was comprised of various trials that required the participants to (virtually) drillthrough a piece of wood. Participants manipulated the Novint Falcon using its gripper and main button to drill.
They were also provided with a visual target indicating the drilling point (Fig. 2). Once the task was completed,participants were presented with a dialogue box that asked them to rate the haptic fidelity (Fig 3). Each trial was
repeated four times for a total of 20 trials whose order was randomized. The experiment took approximately 15
minutes to complete. Prior to starting the experiment participants were provided with a description of theexperiment and asked to complete a brief demographic questionnaire that asked them their age range, gender,prior drilling experience, and whether they had any issues with their hearing). Finally, upon completion of alltrials, participants were asked to rate the following two questions on a Likert scale ranging from 1 (not at all) to 7
(very much): i) whether they believed that sound influenced their perception of haptic fidelity, and ii) whether thesound played during the drilling process was important to them in completing the drilling task.
Results
Data collected from each participant included the haptic fidelity rating. As previously described, each trial was
repeated four times; the average of these four repetitions was then analyzed. A summary of the results for haptic
fidelity perception across each of the five auditory conditions is provided in Fig. 4. As shown, the responses
varied only slightly. The maximum rating (5.6) corresponds to the “Drill Sound” (contextual) auditory condition
whereby the sound corresponded to the task at hand. This of course is not surprising and is in-line with prior workthat considered the effect of sound on visual fidelity perception and saw higher visual fidelity perception ratings
when the sound corresponded (was contextual) to the visual scene (see [2]). The minimum (4.7) ratingcorresponds to the “Classic Music” auditory condition albeit the difference between the “Classic Music” auditory
condition and the “Metal Music” (4.8), “No Sound” (4.9) and “White Noise” (4.9) conditions are negligible. It
should also be noted that the differences are not statistically significant. Fig. 5 illustrates several examples ofparticipants virtually drilling through wood during their participation in the experiment. Finally, responses to thetwo “exit questions” were as follows. With respect to the question asking whether sound influenced theirperception of haptic fidelity, six (54.5%) chose “7”, three (27.3%) chose “6” and one (9.1%) chose “5” and “4”.
With respect to the question asking whether they believed that the sound played during the drilling process wasimportant three (27.3%) chose “7”, five (45.5%) chose “6”, two (18.2%) chose “5”, and one (9.1%) chose “4”.
Overall, the majority of the participants believe that sound is an important part of the drilling process and that it
can influence their haptic fidelity. Additionally, participants expressed feeling different haptic feedback across thefive audio scenarios, which can be seen from Fig. 4 box plot.

Conclusions
We have conducted an experiment that was motivated by prior work that saw a strong influence of sound on visual
fidelity perception and the potential to, through the appropriate use of sound, allow lower-end (and less expensive)
haptic devices to be used in place of higher fidelity yet cost prohibitive haptic devices. Although the results are
preliminary and greater work remains, the results presented here do indicate that sound can potentially influence
haptic fidelity perception and more specifically, a contextual sound that matches/corresponds to the task being
performed should be included. Furthermore, participants generally strongly believed that sound is an importantpart of the drilling process and that it can influence haptic fidelity perception; all participants felt different haptic
feedback while the mechanical model remained the same for the duration of the experiment. In addition, theexperiment presented here has provided tremendous insight regarding the difficulties and potential problems to be
encountered when conducting multimodal experiments that include haptic devices. There are many factors thatmust be considered including the measurement and modeling of the physical forces involved (here drillingthrough wood), and the potential issues associated with haptic devices, particularly lower-end devices (e.g., thepotential need to re-calibrate the device during experimentation).
References[1] V. Hayward, O. Astley, and M. Cruz-Hernandez, “Haptic interfaces and devices,” Sensor, 2004.
[2] N. Jafari, K. D. Adams, and M. Tavakoli, “Haptics to improve task performance in people with disabilities:
A review of previous studies and a guide to future research with children with disabilities,” J. Rehabil.
Assist. Technol. Eng., vol. 3, no. 0, pp. 1–13, Oct. 2016.
[3] G. Burdea and P. Coiffet, Virtual reality technology. Wiley & Sons, 2003.
[4] B. C. Srivsan M. A., “Haptics in virtual enviroments: taxonomy, research status and challenges,” Comput.

Grpahics, vol. 21, p. 12, 1997.
[5] M. Friedman, K. Friedrich, M. Queisner, and C. Stein, “Conceptualizing Screen Practices: How
Head-Mounted Displays Transform Action and Perception,” MediaTropes, vol. 6, no. 1, pp. i–v, 2016.
[6] T. Chatzidimitris, D. Gavalas, and D. Michael, “SoundPacman: Audio augmented reality in location-based
games,” in 2016 18th Mediterranean Electrotechnical Conference (MELECON), 2016, pp. 1–6.
[7] B. Serrano, R. M. Baños, and C. Botella, “Virtual reality and stimulation of touch and smell for inducing
relaxation: A randomized controlled trial,” Comput. Human Behav., vol. 55, pp. 1–8, 2016.
[8] M. A. B. Husman, H. F. Maqbool, M. I. Awad, A. Abouhossein, and A. A. Dehghani-Sanij, “A wearable
skin stretch haptic feedback device: Towards improving balance control in lower limb amputees,” in 2016
38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society(EMBC), 2016, pp. 2120–2123.
[9] J. Wu, G. Yu, D. Wang, Y. Zhang, and C. C. L. Wang, “Voxel-Based Interactive Haptic Simulation of Dental
Drilling,” in Volume 2: 29th Computers and Information in Engineering Conference, Parts A and B, 2009,
pp. 39–48.
[10] B. L. Tai et al., “A physical simulator for endoscopic endonasal drilling techniques: technical note,” J.
Neurosurg., vol. 124, no. 3, pp. 811–816, Mar. 2016.
[11] H. Kinoshita et al., “Development of a Drilling Simulator for Dental Implant Surgery.,” J. Dent. Educ.,
vol. 80, no. 1, pp. 83–90, Jan. 2016.
[12] Q. Wang et al., “Haptic rendering of drilling process in orthopedic surgical simulation based on thevolumetric object,” in 2015 IEEE International Conference on Digital Signal Processing (DSP), 2015, pp.
1098–1101.
[13] C. G. Corrêa, D. M. Tokunaga, E. Ranzini, F. L. S. Nunes, and R. Tori, “Haptic interaction objective
evaluation in needle insertion task simulation,” in Proceedings of the 31st Annual ACM Symposium on
Applied Computing - SAC ’16, 2016, pp. 149–154. 2017.
Meetings/Presentations
1.
2.

Research Advancements in HCI: Environments and Perception (M.Jenkin, 2018.02.21, RIE 204)

Research Advancements in Fidelity Enhanced Virtual Environments (A.Quevedo, 2018.02.23, RIE 201)
The work was carried out at multiple partner institutions. The development of
the virtual drilling framework primarily took place at UOIT within Dr. Bill
Kapralos’ Games and Media Entertainment Research Lab (GAMER Lab), a
state-of-the-art, interdisciplinary research laboratory where faculty, visiting
professors, and students (graduate and undergraduate alike) conduct research
related to serious games, human factors, simulation, games for fitness andhealth, applied game design, augmented reality, stereo-vision gaming, andaffective computing. The GAMER Lab is equipped with a variety ofstate-of-the-art equipment including stereoscopic displays, physiologicalmonitoring equipment, tabletop computer, high-fidelity audio equipment,
various input devices (including a haptic device), amongst others. The
使用した設備・資料・試料等
hardware and software required to complete the development of the
auscultation app accessible within the GAMER Lab was made available to theproject members and used to support this project. Once the virtual drilling
framework has been developed, we have incorporated the haptic device(Novint Falcon). This was completed in the laboratory of Dr. Bill Kapralos
who already had a Novint Falcon haptic device. The experiments that will
examine the effect of sound on haptic fidelity perception were conducted in Dr.
Kapralos’ GAMER Lab. However, by replicating the hardware at multiple
institutions, it will allow, if needed, continuing experiments to be conducted at
the Research Institute of Electronics and in Dr. Jenkin’s lab, too.

本研究成果に関連する論文発表状況
1. B. Kapralos, A. Uribe-Quevedo, and A. Dubrowski. Immersive technologies for medical education. In N.
Lee (Ed.) Encyclopedia of Computer Graphics and Games, Springer International Publishing (to appear).
2. B. Kapralos, F. Moussa, K. Collins, and A. Dubrowski. Levels of fidelity and multimodal interactions. In
P. Wouters and H. van Oostendorp (Eds.) Techniques to Improve the Effectiveness of Serious Games,
Springer Advances in Game-based Learning Book Series, Ch. 5, pp. 79-101, 2017.
3. M. Melaisi, M. Nguyen, A. Uribe-Quevedo, and B. Kapralos. The effect of sound on haptic fidelity
perception. In Proceedings of the IEEE Global Engineering Education Conference (EDUCON) 2017,
April 25-28, 2017, Athens, Greece, pp. 709-712.

次年度の共同研究継続の有無

有

・

無

拠点内対応教員とご相談の上ご記入ください。
継続の場合には次年度の研究計画をご記入願います。

次年度の研究計画（継続の場合）
In an effort to develop a greater understanding of the effect of sound on haptic fidelity perception, we will expand
upon our prior work that examined the effect of sound on haptic fidelity perception with the addition ofelectroencephalogram (EEG) monitoring with the aim of correlating brain signals to haptic-sound interactions.
More specifically, we will repeat the experiment described here whereby participants will drill through a block ofwood (to two drilling depths, 3 cm, and 6 cm), in the presence of contextual sounds (those associated with thedrilling process itself), and non-contextual sounds (those not associated with the drilling process). The contextual
sounds will consist of: i) drilling in the air, and ii) drilling through wood. The non-contextual sounds will consistof i) classical music, ii) heavy metal music, iii) white noise, and iv) silence (no sound). Interactions will be limited
to drilling through the block of wood to a pre-defined depth with the drill bit positioned at the appropriate starting
position (e.g., no need to move and position the drill to the appropriate starting position). This will ensure
participants focus solely on the drilling depth and thus limiting any potential confusion and/or frustration
associated with aiming and moving the drill such that it aligned with the correct drilling position.
We will use the Emotiv Epoc EEG wireless neuroheadset as it is an affordable BCI that allows capturing brain
data from 14 channels (AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF42). Wireless EEG technology
is a viable alternative for measuring subjectivity in evaluation scenarios. The Emotiv data acquisition frequency
occurs within a 1-100 Hz threshold. There are five bands of normal EEG frequencies, and more specifically: i)delta, associated to deep sleep, dreamless state, immune system functions, and collective consciousness, ii) theta,associated with light sleep, meditation memory, vivid imagery), iii) alpha, associated with relaxation with eyes
closed, creativity, visualization, super learning, iv) beta, associated to normal working state of consciousness,
focus, alertness, learning, concentration, five physical senses, and v) gamma, associated to hyper-learning activity.
The activity meaning associated with each sensor placement is presented as follows: AF3 - attention, AF4 judgment, F7 - verbal expression, F8 - emotional expressions, F3 – motor planning, F4 - motor planning for left
upper body, FC5 - right body controller, FC6 - left body controller, T7 - Verbal memory, T6 - emotional memory,
P7 – verbal understanding, P8 - Emotional understanding and motivation, and finally O1 and O2 - visual
processing. The Emotiv Epoc wireless feature allows the user to conduct activities without the discomfort andlimitations of wired systems that could affect its mobility and natural movements. To process the Emotiv data, theEmotiv Testbench will be used in conjunction with the fast Fourier transform implemented in Matlab and the
Matlab EEG Toolbox.
To better understand the participant’s perception of haptic fidelity, we analyzed the BCI data during the threestages (before, during, and after drilling). For each trial, we will analyze the BCI signals in the frequency domain
for each of the 14 channels. We will then filter all the data using a band-pass filter and perform an Interdependent
Component Analysis on the filtered data (to eliminate the segments of the signals that correspond to blinking andinvoluntary muscular movements) and then determine if there is a relation between the signals and the fidelity
rating.