A10:
PHYSIOLOGICAL ACOUSTICS

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0611

Multiscale photonic imaging of the cochlea:
From implants to neurons
Daniel KEPPELER1,2; Christoph A. KAMPSHOFF1,2,3; Anupriya THIRUMALAI1,2, Jannis J.
SCHAEPER4, Tabea QUILITZ1,2, Tim SALDITT4,5,6, Tobias MOSER1,2,3,5,6
1

Institute for Auditory Neuroscience, University Medical Center Göttingen, Germany
2
3

Department of Otolaryngology, University Medical Center Göttingen, Germany
4

5

InnerEarLab, University Medical Center Göttingen, Germany

Institute for X-ray Physics, University of Göttingen, Germany

Cluster of Excellence “Multiscale Bioimaging: From Molecular Machines to Networks of Excitable Cells,"
University of Göttingen, Germany
6

Auditory Neuroscience and Optogenetics Laboratory, German Primate Center, Germany

ABSTRACTThe development of novel therapies to restore hearing requires detailed information on spatial dimensions,tissue morphology and the sensorineural status of the cochlea. However, the cochlea is deeply embedded in
the temporal bone and thereby hard to access for imaging techniques. Here, we employed three-dimensional
X-ray phase-contrast tomography and light-sheet fluorescence microscopy and their combination in species
that serve as established animal models for auditory research. While light-sheet fluorescence microscopy
allowed for specific immunolabelling of auditory nerve cells, X-ray phase-contrast tomography enabled us
to get structural information in uniform voxel size and utilization of subcellular features like the cell nucleus
without specific sample preparation. Multiscale and multimodal imaging of cochlear morphology will
facilitate preclinical studies of innovative approaches to deafness such as gene therapy and cochlear implants.
Keywords: cochlea, X-ray phase-contrast tomography, light-sheet fluorescent microscopy

OVERVIEW
Profound hearing loss can be treated with electrical cochlear implants which are used to stimulate
the auditory nerve through electric current and thereby circumvent the dysfunctional or lost hair cells.
However, the snail-shaped hearing organ called cochlea is filled with saline solution leading to widecurrent spread thus causing broad neural activation (1). Instead of precise stimulation of small
populations of auditory neurons, large areas are activated resulting in reduced spatial resolution andpoor speech understanding especially in challenging acoustic environments (2).
Optical stimulation allows for a direct activation of the auditory nerve using focused light leading
to improved spectral resolution (3–5). This is achieved through a combination of micro-scale lightemitter technology (6–8) and optogenetics – a genetic approach enabling precise neural control upon
the expression of light-gated ion channels in the auditory nerve, called channelrhodopsins (9–11).
For the development of the (optogenetic) gene therapy and optical cochlear implants (oCI), detailed
knowledge of the cochlear morphology is crucial. Therefore, we employed multiscale X-ray andfluorescence imaging for quantitative assessment of scala tympani, vestibuli, and media, as well as
the spiral ganglion in Rosenthal’s canal for various species commonly used in auditory research. The
combination of light-sheet fluorescent microscopy and X-ray phase-contrast tomography allowed
specific immunolabelling of auditory nerve cells and retrieval of structural information in uniform
voxel size.

ACKNOWLEDGEMENTS
This work was funded by the European Research Council through the Advanced Grant “OptoHear”
to T.M. under the European Union’s Horizon 2020 Research and Innovation program (Grant Agreement No. 670759) and the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation)
funding under Germany’s Excellence Strategy (EXC 2067/1-390729940) to T.S. and T.M., as well as
by the Leibniz Program of the DFG, Fraunhofer, and Max-Planck Cooperation Program (NeurOpto
grant) to T.M. This research is supported by the award of the Fondation Pour l’Audition (FPA RD 2020-10).

REFERENCES
1.

A. Kral, R. Hartmann, D. Mortazavi, R. Klinke, Spatial resolution of cochlear implants: the electricalfield and excitation of auditory afferents. Hear. Res. 121, 11–28 (1998).

2.

R. Kang, et al., Development and Validation of the University of Washington Clinical Assessment ofMusic Perception Test. Ear Hear. 30, 411–418 (2009).

3.

V. H. Hernandez, et al., Optogenetic stimulation of the auditory pathway. J. Clin. Invest. 124, 1114–1129(2014).

4.

A. Dieter, et al., μLED‐based optical cochlear implants for spectrally selective activation of the auditory
nerve. EMBO Mol. Med. 12 (2020).

5.

A. Dieter, C. J. Duque-Afonso, V. Rankovic, M. Jeschke, T. Moser, Near physiological spectral
selectivity of cochlear optogenetics. Nat. Commun. 10 (2019).

6.

C. Goßler, et al., GaN-based micro-LED arrays on flexible substrates for optical cochlear implants. J.
Phys. Appl. Phys. 47, 205401 (2014).

7.

E. Klein, C. Gossler, O. Paul, P. Ruther, High-density µLED-based Optical Cochlear Implant with
Improved Thermomechanical Behavior. Front. Neurosci. 12 (2018).

8.

D. Keppeler, et al., Multichannel optogenetic stimulation of the auditory pathway using microfabricatedLED cochlear implants in rodents. Sci. Transl. Med. 12, eabb8086 (2020).

9.

D. Keppeler, et al., Ultrafast optogenetic stimulation of the auditory pathway by targeting‐optimized
Chronos. EMBO J., e99649 (2018).

10. T. Mager, et al., High frequency neural spiking and auditory signaling by ultrafast red-shifted
optogenetics. Nat. Commun. 9 (2018).
11. B. Bali, et al., Utility of red‐light ultrafast optogenetic stimulation of the auditory pathway. EMBO Mol.
Med. 13 (2021).

ABS-0735

Transmission of sound through a gel-ﬁlled ear canal: A case study
Janani Fernandez(1) , Ville Pulkki(1) , Petteri Hyvärinen(1) , Antti Aarnisalo(2)(1) Acoustics Lab, Department of Signal Processing and Acoustics, Aalto University, Espoo, Finland, janani.fernandez@aalto.ﬁ(2) Head and Neck Center, Department of OtorhinolaryngologyHead and Neck Surgery, Helsinki University Hospital andUniversity of Helsinki, Finland

ABSTRACTDuring function magnetic resonance imaging (fMRI), it would be beneﬁcial to image the brain while sounds
are played to the subject. Therefore, there is a need to both block the fMRI machine noise and deliver soundwaves to the subject within the machine enclosure. Additionally, the system should also be free of magnetic
components, as their presence may disrupt the function of the fMRI machine. In this paper, a method is proposed
to address these issues in the form of ﬁlling the ear canal with a gel, to block the machine noise, then deliversound through the gel via a ceramic piezoelectric stack. The piezo actuator is placed against the boundaryof the air and the gel. The sound waves traverse through the gel to the tympanic membrane and then further
along the auditory chain. There is also some degree of transmission achieved through bone conduction. A casestudy of this proposed system was performed. It was found that the gel within the ear canal provides adequate
hearing protection, and the piezo actuator was capable of transmitting sound to the listener, and sound level was
perceived to be higher with the piezo against the gel than against the mastoid.

Keywords: Sound, Insulation, Transmission

1

INTRODUCTION

During imaging with functional magnetic resonance imaging (fMRI) machines, the noise levels are very large
in magnitude, reaching values of around 100 dBA or even exceeding it [1, 2]. These sound levels can cause
damage to hearing systems of the patients within the machines [3, 4]. It is also problematic to play sounds/speech to the person(s) within the conﬁnes of the machine due to the high level of background noise causedby the machine. It is possible to play audio when the machine is switched off during sparse imaging; however,for certain studies, particularly acoustical studies, it may be beneﬁcial to playback audio to the subject within
the machine and image the brain simultaneously. Additionally, it may beneﬁt the operator of the machine to beable to communicate with the subject, for example to give instructions or keep them calm during the procedure.
A further constraint is that metal components should be avoided within the machine due to the high magnetic
eddy currents, therein causing the components to malfunction. The current solution to this problem is for thepatient to wear fMRI compatible earphones over which appropriate sound protection muffs are worn [5, 6, 7].
One possible alternative solution for achieving sound protection may involve ﬁlling the patient’s ear canalwith a ﬂuid such as gel, cream, or saline solution. With the added mass to the tympanic membrane, it has been
observed that some measure of hearing protection is achieved [8]. Covering the entire tympanic membrane withsuch ﬂuid would change the sound transmission path from a couple between air and the tympanic membrane, toa coupling between air and the ﬂuid, and then the ﬂuid to the tympanic membrane. This may potentially be themost effective sound insulation obtainable in this setting. An added beneﬁt is that this may also reduce fMRI
susceptibility artifacts caused by the air in the ear canals [9]. However, to the best of the authors’ knowledge,
such a solution is not currently employed in practice.
While using such a ﬂuid-based hearing protection solution, however, the question then becomes how toplay sounds to the patients; since this method would not permit the current solution of employing air tubes
to transmit sound. Therefore, any solution to this problem must rely on producing the necessary displacement

Figure 1. Piezo actuator with mass afﬁxed.
required to propagate sound through the ﬂuid. In [10], where ﬂuid-based hearing protection was explored, it
was found that the larger the distance between the transducer and the receiver, the more force was required totransmit sound. This was due to sound energy losses arising as a result of the high viscosity of the medium
and also acoustical and dispersion off and from the walls of the tube. Additionally, distortions may arise due toreﬂections off the walls of the tube. As a further constraint, given the present application of interest, any sounddelivery method must also meet the hardware requirements for fMRI machines; i.e. magnetic materials shouldbe placed inside the machine.
One may postulate that bone-conduction-based sound reproduction could be one possible solution to these
aforementioned problems. However, their use in spatial audio studies is limited due to the crosstalk between
the two ears [11, 12]. Additionally, the pressure on skin may lead to discomfort in some patients. However,
one other possible solution, which is the topic of the present study, is to employ the use of a ceramic piezo
actuator. Such a device can be placed close to the ear, i.e. at the entrance to the ear canal, at the boundaryof the gel/ﬂuid and air, and also circumvents the hardware restrictions. Such a solution would also have thebeneﬁt of reducing fMRI susceptibility artifacts.
This paper reports a case study in which the sound transmission of a piezo actuator through ultrasound gel
is investigated. The proposed system is described in Section 2, the evaluation is detailed in Section 3, theresults and discussion are in Section 4, and the conclusions in the ﬁnal Section, 5.

2

Proposed Solution

The proposed solution involves the use of a piezo ampliﬁer or controller and a piezo actuator placed at theouter boundary of a gel ﬁlled ear canal. The piezo selected for this study was a ceramic AE0505D16F fromThor labs ([13]), which is a high voltage piezoelectric actuator capable of upto 11.6 (micro)mm displacement
when run at the recommended driver voltage of 100 V. It was assumed that this would provide sufﬁciently large
force and displacement to enable sufﬁcient coupling of the acoustic signal with the gel and, subsequently, themass loaded tympanic membrane. Additionally, to overcome losses due to the acoustical transmission througha viscous ﬂuid. As per the recommendation of the manufacturers of the piezo actuator, a mass is attached tothe end of the piezo to increase the force coupled to the gel. Importantly, it is highlighted that such a device
is also suitable for use in fMRI machines. Figure 1 presents a photograph of the piezo actuator with the mass
afﬁxed to the end.

The piezo was used in conjunction with the recommended single channel MDT693B piezo controller. It is
important to note that this device will essentially operate as a half-wave rectiﬁer. This is due to the the natureof such piezo actuators, since a reverse bias would damage these types of actuators. It is therefore advisable
to apply a DC gain to the signal before it is delivered to the piezo controller. A voltage divider circuit wastherefore utilised in the signal chain to create this necessary DC bias.

3

Evaluation

An ultrasound gel, Aquasonic 100 from Parker, was selected for this preliminary case study. An early observation of the authors was that the gel provided approximately 30 dB sound reduction at low frequencies up to
1 kHz and approximately 48 dB sound reduction for frequencies between 1 kHz and 8 kHz. This was found by
performing an informal audiometry test for one of the authors. The audiometry test was performed with an open
ear canal and then repeated with the same ear canal completely ﬁlled with the gel to test its sound insulation
capabilities.
The ﬁlling of the ear canal was conducted by a licensed medical professional. The ear canal was ﬁrst
cleaned thoroughly, then the gel was piped into it carefully with the aid of a syringe topped with a dull needle.
Afterwards, the gel was removed with the aid of saline solution and a vacuum device. Figure 2 (a) and (b) show
the medical professional ﬁlling and then removing the gel from the ear canal, and Figure 2 (c) is a photographof the ear canal ﬁlled with gel. As a licensed medical professional was required for this procedure, only one
subject completed the evaluation in its entirety.
A Digilent Analog Discovery II device was used to transmit data to the piezo ampliﬁer. The output voltage
range of the device was -5 to +5 V. Additionally, due to the aforementioned half-wave rectiﬁer nature of theampliﬁer, a DC bias of 2.5 V was added to the signal.
The evaluation of the prototype piezo actuator system was performed via an AB comparison listening test.
The test stimuli utilised were sine tones of the following frequencies: 125 Hz, 250 Hz, 500 Hz, 750 Hz, 1 kHz,
2 kHz, 3 kHz, 4 kHz, 6 kHz, 8 kHz. These frequencies were chosen as they were previously used for the informal
audiometry test to gauge the sound insulation of the ultrasound gel. During the listening test, the perceived
sound level produced by the prototype was compared to the sound level produced by the driver of a pair of
Vic Firth SIH2 Stereo Isolation headphones. The right driver of the headphones was removed while keeping theheadphone structure intact to allow access to the right ear of the test participant.
The test procedure was as follows. A sine tone of 0.5 s length was played through the headphone driver at
a speciﬁed sound level. The same stimulus was then played via the piezo driver in the subject’s other ear after
a 0.2 s pause. The test participant was asked to change the voltage level across the piezo until the perceived
sound level produced by the piezo actuator was considered to be equal the level produced by the headphone
driver by pressing the left or right mouse buttons to indicate whether the piezo actuator or the headphone driver
was perceived as louder. The two sound levels tested were 65 dBA and 75 dBA. The initial starting peak voltage
was set at 0.1 V output from the Digilent device, upon which the 2.5 V DC bias voltage was added and thesignal then ampliﬁed with a gain factor of 7.5. The step size of the voltage increase or decrease was initially
set at 6 dB, which was then changed to 3 dB after the ﬁrst reversal. This change to the output voltage wasapplied to the Digilent device output voltage. Once the test participant perceived the levels to be equal, they
indicated this by pressing the central mouse button. The test stimuli was then changed to the next since tone,
and the procedure repeated until all sine tones were thus compared.
The AB comparison test was repeated several times. One instance was with no gel in the ear canal andthe piezo actuator merely placed at the entrance to the ear canal. This was done to establish a baseline forcomparison. In the next repetition, the piezo actuator was placed against the test subject’s mastoid bone todetermine the effect via bone conduction. For the next repetition of the test an ear plug placed inside the ear
canal, close to the tympanic membrane but with a small gap between them. The ear canal was then ﬁlled with
gel until its entrance. The piezo was placed against the gel, but in such a manner that it did not touch thesubject’s skin and only made contact with the gel. The ﬁnal test was performed with the ear plug removed
and the ear canal completely ﬁlled with gel from the entrance to the tympanic membrane. The piezo actuator
was then placed against the gel in a similar manner to the previous test. It was hoped that the comparison of

(a) A subject having their ear canal ﬁlled.

(b) A subject having their ear canal cleaned.

(c) A subject’s ﬁlled ear canal.

(d) A subject taking the listening test.

Figure 2. Photographs taken during testing.

these last two test cases would help determine if the sound signal was indeed travelling to the cochlea via the
tympanic membrane or if the signal was taking a different path.
These test cases and piezo actuator positions are summarised in Table 1. A photograph of a subject taking
the listening test can be seen in Figure 2 (d). Additionally, two pieces of music were played to the subject for
an informal assessment of the sound quality of the proposed system.
Table 1. Listening test cases.

4

Test case No.

Piezo actuator position

Ear canalI

At entrance of ear canalNot ﬁlledII

Against mastoid bone

Not ﬁlledIII

At entrance of ear canal

Partially ﬁlled, plug placed between tympanic membrane and gelIVAt entrance of ear canalCompletely ﬁlled with gel

Results and Discussion

The results of the listening test are presented in Figure 3. It can be seen in Figure 3 (a) that when the test was
conducted with the sound level set at 65 dBA, for frequencies below 2 kHz, the piezo actuator consumed lesspower to match the headphone driver sound levels when placed at the entrance of a ﬁlled or partially ﬁlled ear
canal (test cases III and IV), in comparison to an empty ear canal (test case I). Additionally, the sound levels
perceived with test case II required more power to generate in comparison to the previous test cases. The test
case that required the most power for this frequency range was test case I. At stimuli frequencies above 3 kHz,the piezo actuator required more power to transmit sound through the completely ﬁlled ear canal, while theother three cases did not have signiﬁcantly large differences in performance. It may be noted that for the test
case II, the subject perceived the 125 Hz and 250 Hz stimuli to be located at the contra-lateral ear, rather than
the side at which the piezo actuator was positioned. The subject also expressed that in order to better perceive
the sound through the piezo actuator they placed their head close to the device, which caused some discomfort
over the course of the test.
During the informal testing using music tracks, the subject was able to identify them when they were played
through the piezo actuator in all the test cases. The quality of the music was stated to be distorted in all
instances, with the music appearing to be bandpassed. The test cases I and II required approximately 10 times
the peak voltage of the other test cases. The music was perceived as being distorted, and the lyrics wereintelligible but also distorted for test case I. In test case II, the same amplitude utilised for test case I wasperceived to be softer and the intelligibility of the lyrics was not signiﬁcantly different to the previous test case.
In this case, it was also noted that the bass was better perceived, but the subject reported no other differences
regarding the sound quality. In the test case II, the peak voltage across the piezo needed to be reduced bya factor of 10 to 20 mV output from the Digilent device. The quality of the music was described to be theworst of the four test cases, with a narrow frequency range of the perceived music. However, the lyrics of themusic were still intelligible. In comparison, the completely ﬁlled ear canal had better quality for the same peakvoltage, with less distortion and a more broadband range perceived, i.e. more bass and higher frequencies were
heard. The lyrics were also reported to be easier to understand in this test case.
From the test results, it can be inferred that the piezo actuator is capable of transmitting speech through theselected gel. The perceived distortion appears to be a function of the piezo, as this was present even in the test
case I. It was also noted that the distortion caused increased with an increase in peak voltage, and therefore thebest perceived sound quality was in test case IV, which required the least power for transmission of the soundsignal. Comparing test cases III and IV indicates that the method required less power than bone conduction
utilising the same piezo actuator. At frequencies below 2 kHz, there is over a 10 dB difference in the required
peak voltage, which implies that the signal path in test case IV was through the tympanic membrane and(a) Test conducted with sound level set at 65 dBA.

(b) Test conducted with sound level set at 75 dBA.

Figure 3. Results of the listening test. The solid line represents the results when the ear canal is completely
ﬁlled with gel and the piezo actuator placed at the entrance of the ear canal. The dashed line represents theresults when the ear canal is partially ﬁlled with gel and a plug placed between the tympanic membrane andthe gel. The piezo actuator was placed at the entrance of the ear canal. The line marked with the "*" symbol
represents the results when the ear canal was not ﬁlled with any gel and the piezo actuator was placed at theentrance of the ear canal. The ﬁnal line, marker with the "+" symbol represents the results when the ear canal
was not ﬁlled with any gel and the piezo actuator was placed against the mastoid bone.

ossicles, rather than through bone conduction. Additionally, it may be noted that the informal testing indicatedthat test case IV had better channel separation at low frequencies than test case II, which motivates the use of
the technique in binaural hearing research.
While this method shows promise and addressed much of the outlined goals, it is noted that this is a limitedcase study with only one subject and it is, therefore, difﬁcult to draw conclusions. In addition, a medical
professional is required for the ﬁlling of the ear; due to the ﬁlling and cleaning of the ear canal. This may
therefore limit its application in practice and also render future studies difﬁcult to conduct.

5

CONCLUSIONSIn this case study, a method for simultaneously attenuating background noise while still facilitating communication with a patient within the conﬁnes of an fMRI machine was presented. It was found that ﬁlling the ear
canal with Aquasonic 100 ultrasound gel, then placing a piezo actuator at the entrance of the ear canal suchthat it is making contact with the gel provides sound insulation while delivering signals to the subject who
participated in the case study.
The results of this limited case study indicate that the proposed apparatus may present a feasible solution to
the problem of communication with patients within the fMRI machine while providing hearing protection against
the noise of the machine. The results from single subject suggest an advantage of > 15dB at frequencies below
2 kHz. The interaural crosstalk was also reported to be not existing.
Future study would involve a larger study with more subjects and potentially a piezo actuator with reduced
distortion.

REFERENCES[1] Cho ZH, Park SH, Kim JH, Chung SC, Chung ST, Chung JY, Moon CW, Yi JH, Sin
CH, Wong EK. Analysis of acoustic noise in MRI. Magnetic resonance imaging. 1997 Jan
1;15(7):815-22.
[2] Price DL, De Wilde JP, Papadaki AM, Curran JS, Kitney RI. Investigation of acoustic noiseon 15 MRI scanners from 0.2 T to 3 T. Journal of Magnetic Resonance Imaging: An Ofﬁcial
Journal of the International Society for Magnetic Resonance in Medicine. 2001 Feb;13(2):28893.
[3] Bongers S, Slottje P, Kromhout H. Hearing loss associated with repeated MRI acquisition
procedure-related acoustic noise exposure: an occupational cohort study. Occupational andEnvironmental Medicine. 2017 Nov 1;74(11):776-84.
[4] Brummett RE, Talbot JM, Charuhas P. Potential hearing loss resulting from MR imaging.
Radiology. 1988 Nov;169(2):539-40.
[5] McJury MJ. Acoustic noise and magnetic resonance imaging: a narrative/descriptive review.
Journal of Magnetic Resonance Imaging. 2022 Feb;55(2):337-46.
[6] Riecker A, Mathiak K, Wildgruber D, Erb M, Hertrich I, Grodd W, Ackermann H. fMRI
reveals two distinct cerebral networks subserving speech motor control. Neurology. 2005 Feb
22;64(4):700-6.
[7] Gurd JM, Amunts K, Weiss PH, Zaﬁris O, Zilles K, Marshall JC, Fink GR. Posterior parietal
cortex is implicated in continuous switching between verbal ﬂuency tasks: an fMRI study
with clinical implications. Brain. 2002 May 1;125(5):1024-38.
[8] Abdalla MK, Bassiony MA, Aziz MT, Shewel YG. Effect of loading of the central part of
the tympanic membrane on pure tone audiometry. Acta Otorhinolaryngologica Italica. 2018
Jun;38(3):229.

[9] Ojemann JG, Akbudak E, Snyder AZ, McKinstry RC, Raichle ME, Conturo TE. Anatomic
localization and quantitative analysis of gradient refocused echo-planar fMRI susceptibility
artifacts. Neuroimage. 1997 Oct 1;6(3):156-67.
[10] Öyry A. Transmission of piezoelectrically induced sound in a liquid ﬁlled tube. 2019.
[11] MacDonald JA, Henry PP, Letowski TR. Spatial audio through a bone conduction interface:
Audición espacial a través de una interfase de conducción ósea. International journal of audiology. 2006 Jan 1;45(10):595-9.
[12] Farrell NF, Hartl RM, Benichoux V, Brown AD, Cass SP, Tollin DJ. Intracochlear measurements of interaural time and level differences conveyed by bilateral bone conduction
systems. Otology neurotology: ofﬁcial publication of the American Otological Society,
American Neurotology Society [and] European Academy of Otology and Neurotology. 2017
Dec;38(10):1476.
[13] Thorlabs - AE0505D16F Piezoelectric Actuator, Max Displacement 17.4 ţm, 6.5 mm x 6.5mm x 20.0 mm [Internet]. Thorlabs.com. 2022 [cited 10 August 2022]. Available from:https://www.thorlabs.com/thorproduct.cfm?partnumber=AE0505D16F

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0987

Multi-modal microstructure imaging of the normal and noise
exposed cochlea ex vivo
James B. NORRIS1; Jens HJORTKJÆR3; Jannis SCHAEPER5; Susana AZNAR4; Tim SALDITT5;
Torsten DAU3; Hartwig SIEBNER1,6,7; M. Charles LIBERMAN8; Tim B. DYRBY1,2
1

Danish Research Centre for Magnetic Resonance, Centre for Functional and Diagnostic Imaging and Research,
Copenhagen University Hospital Amager and Hvidovre, Copenhagen, Denmark2

Department of Applied Mathematics and Computer Science, Technical University of Denmark, Kgs. Lyngby,
Denmark

3

Hearing Systems Section, Department of Health Technology, Technical University of Denmark, Kgs. Lyngby,
Denmark
4

Research Laboratory for Stereology and Neuroscience, Copenhagen University Hospital Bispebjerg andFrederiksberg, Copenhagen, Denmark5

Institut für Röntgenphysik, Georg-August-Universität Göttingen, Friedrich-Hund-Platz, Göttingen, Germany
678Institute for Clinical Medicine, University of Copenhagen, Copenhagen, Denmark

Department of Neurology, Copenhagen University Hospital Bispebjerg and Fredriksberg, Denmark.

Eaton-Peabody Laboratories and Department of Otolaryngology, Head and Neck Surgery, Massachusetts Eyeand Ear, Boston, USA

ABSTRACT
Historically, cochlear pathology has been studied by serial sectioning of the temporal bone or microdissectionof the sensory epithelium. Sectioned material is challenging because of the spiraling nature of the cochlear
duct, and organ of Corti microdissection is limiting because other key structures (e.g., the stria) must be
removed for high-resolution imaging. In contrast, both CT and diffusion MRI of intact cochleae yieldisotropic 3D image stacks that can be cropped and virtually re-sectioned in any plane without loss ofresolution, thereby combining the strengths of both classic approaches. Here, we apply diffusion MRI, CT
and optical microscopy of osmicated cochleas of normal and noise exposed guinea pigs. We demonstrate avariety of MRI techniques, including diffusion tensor imaging (DTI), neurite orientation dispersion anddensity imaging (NODDI) and micro diffusion tensor (DT) imaging, as well as CT imaging and x-ray
phase contrast tomography. We show how this multimodal imaging of intact specimens can be used tocharacterize and quantify cochlear pathology in 3D space.
Keywords: Cochlea, Imaging, MRI, CT

ACKNOWLEDGEMENTS
This work was supported by the Novo Nordisk Foundation Synergy Grant NNF17OC0027872

1

gf@aaa.com

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0988

Electrode montage and stimulation protocols to dissociate auditory
nerve function in non-invasive electrocochleography
Jonatan MÄRCHER-RØRSTED1; Torsten DAU1; Miguel TEMBOURY-GUTIÉRREZ1; Gerard
ENCINA-LLAMAS1; Michael BILLE2; Jesper YDE2; Mike HEINZ3; Jens HJORTKJÆR1,4
1

Hearing Systems Section, Department of Health Technology, Technical University of Denmark, Kgs. Lyngby,
Denmark2

Copenhagen Hearing and Balance Center; Ear, Nose and Throat (ENT) & Audiology Clinic, Rigshospitalet,
Copenhagen University Hospital, Denmark3

Department of Speech, Language and Hearing Sciences, Purdue University, West
Lafayette, Indiana, USA

4

Danish Research Centre for Magnetic Resonance, Centre for Functional and Diagnostic Imaging and Research,
Copenhagen University Hospital Amager and Hvidovre, Copenhagen, Denmark

ABSTRACT
Non-invasive measures of the functional status of the cochlear nerve in vivo are critical for hearing
diagnostics. Early components of the click-evoked auditory brainstem response (ABR) potentially dissociate
hair cell activity from action potentials in the auditory nerve, but standard scalp electrode measurements arenoisy. Clinical ABRs consequently focus typically on higher-amplitude brainstem response components.
Here, we recorded responses to tones and clicks in humans and chinchillas with electrodes in the ear canalor placed on the tympanic membrane. Reference electrodes were either placed at the scalp vertex as inclassical ABR montage, or in electrocochleographic montage close to the ear. As expected, classical ABRmontage with vertex reference emphasized brainstem components of the responses to clicks at high soundpressure levels. Electrocochleographic montage yielded well-localized summating and action potential
components of the click response in both humans and chinchillas. Low-frequency tones at high levels elicitedstrong steady-state frequency-following responses (FFRs) in the electrocochleographic montage. In mastoidto-vertex montage, on the other hand, FFRs had ripples in the response waveform suggesting an interference
between nerve and brainstem sources. This indicates that low-frequency FFRs recorded
electrocochleographically is sensitive to cochlear nerve function and its pathologies.
Keywords: Electrocochleography, ABR, FFR

ACKNOWLEDGEMENTS
This work was supported by the Novo Nordisk Foundation Synergy Grant NNF17OC0027872

1

gf@aaa.com

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0602

Towards a model of auditory signal processing and perception in
individual listeners with sensorineural hearing loss
Lily Cassandra Paulick1; Helia Relaño-Iborra1; Torsten Dau1
1

Technical University of Denmark, Denmark

ABSTRACT
Computational models provide a powerful tool for simulating and testing hypotheses related to the processingof sound in the healthy and impaired auditory system. The inclusion of realistic implementations of theauditory processing stages can help to determine how a deficit in one of these stages can affect the overallsystem’s performance. In the present study, a realistic model of the inner-hair cell transduction was integrated
in an auditory signal processing and perception model (Jepsen et al., 2008; Relaño-Iborra 2019). Inner haircells transform basilar membrane motion into receptor potentials. This stage has in previous modellingstudies been simplified as an envelope extraction stage consisting of a half-wave rectification followed by
low-pass filtering. Here, the more realistic simulations are validated from both a physiological andpsychoacoustic point of view.
Keywords: Auditory modelling, Inner hair cell transduction

1. INTRODUCTION
Various computational auditory models have been developed over the last decades in parallel with
the increasing knowledge of auditory processing and perception. The models largely vary in theirapproach and their applicability in different configurations (see Osses Vecchi et al., 2022, for a review
of monaural auditory models). While recent modelling studies are successful in predicting
physiological and behavioural data in normal-hearing listeners, they still face difficulties when it
comes to predicting the large differences across individual listeners in the hearing-impaired
population. Cochlear hearing loss is largely associated with damage to both inner hair cells (IHC) andouter hair cells (OHC) as well as synaptic loss in the auditory nerve, thus the implementations of thesestages are crucial for a better understanding and analysis of the impaired system.
In the present study, a physiologically inspired model of the IHC transduction was implemented
with the specific objective of integrating it into the computational audit ory signal processing andperception model (CASP; Jepsen et al., 2008; Jepsen and Dau, 2011). In CASP, the IHC stage wasoriginally implemented as a half-wave rectification and a low-pass filter stage which does not reflect
the compressive behaviour of the IHC input-output function as observed in mammalian physiologicalrecordings (Dallos, 1985). Recent studies emphasised the importance of this saturating behaviour athigh sound pressure levels for the encoding of supra-threshold sounds (e.g., Carney, 2018). Here, theIHC stage of the original CASP model was replaced by a more realistic IHC transduction and theconsequences of this modification for the performance of the model in psychophysical signal detection
conditions were evaluated. The rationale was that a more realistic simulation of the IHC transduction
processes allows for more accurate predictions of the data, particularly in conditions that are strongly
affected by hearing impairment, where recent models still face difficulties.

2. The IHC Model
The non-linear IHC model, to be integrated in CASP, is inspired by previously developed
biophysical models, such as those proposed by Shamma et al. (1986), Sumner et al. (2002), LopezPoveda and Eustaquio-Martín (2006) and Verhulst et al. (2018).
The IHC membrane potential is modelled with a passive electrical circuit analogy as in Altoè et al.
1

lpau@dtu.dk

(2018). For simplicity, the fast- and slow-acting potassium (𝐾 + ) currents 𝑔𝐾,𝑓 and 𝑔𝐾,𝑠 , respectively,
are assumed to be voltage-independent, constant values. Solving this circuit analogy results in thefollowing differential equation for the receptor potential V:𝑑𝑉

𝐶𝑚 𝑑𝑡 + 𝑔𝑀𝐸𝑇 (𝑉 − 𝐸𝑃) + 𝑔𝐾,𝑓 (𝑉 − 𝐸𝐾,𝑓 ) + 𝑔𝐾,𝑠 (𝑉 − 𝐸𝐾,𝑠 ) = 0,
where 𝐶𝑚 represents the cell capacitance, 𝐸𝑃 the endochlear potential, 𝑔𝐾,𝑓/𝑠 the fast and slow
potassium conductances, and 𝐸𝐾,𝑓/𝑠 the fast and slow reversal potentials, respectively. 𝑔𝑀𝐸𝑇
represents the mechano-electric transducer current, that, in turn, can be calculated from the cilia
displacement 𝑥 using a second-order Boltzmann function fit:𝑔𝑀𝐸𝑇 =
1+ 𝑒

−

𝐺𝑚𝑎𝑥
𝑥−𝑢0
𝑥−𝑢0
𝑠1 (1+𝑒 − 𝑠0 ).

The overall shape of this Boltzmann fit is determined by the parameters 𝑢0 , 𝑠0 and 𝑠1. The values ofthe parameters are provided in Table 1 and conform to those considered in the IHC model
implementation of Verhulst et al. (2018) which was based on animal recordings from Jia et al.
(2007) and Johnson et al. (2011).
𝑪𝒎 – cell capacitance
𝑬𝑷 – endochlear potential
𝒈𝑲,𝒇 – potassium conductance, fast𝒈𝑲,𝒔 – potassium conductance, slow
𝑬𝑲,𝒇 – reversal potential, fast
𝑬𝑲,𝒔 – reversal potential, slow
𝑮𝒎𝒂𝒙 – maximum conductance
𝒙𝟎 , 𝒔𝟎 , 𝒔𝟏 – Boltzmann function parameters12.5pF90mV
19.8nS19.8nS
71mV
78mV
30nS
20nm, 16nm, 35nm

Table 1: Parameters of the proposed non-linear IHC model. Parameters are chosen in agreement with the IHC model
implementation in Verhulst et al., 2018 based on animal recordings from Jia et al., 2007 and Johnson et al., 2011. For
simplicity, however, the fast and slow potassium conductance are assumed voltage independent.

3. Physiological Response
The behaviour of the non-linear IHC model was evaluated in comparison to available animal
recordings. The inputs to the IHC model are the outputs of the preceding preprocessing stages in the
CASP model, including outer- and middle-ear filtering of the acoustic signal and the basilar-membraneprocessing using the dual resonance non-linear cochlear filterbank (Lopez-Poveda & Meddis, 2001).
The parameters of the CASP model remained unmodified and a detailed description of the processing
stages can be found in Jepsen et al. (2008).
The main two aspects of the IHC transduction are reproduced by the model: the compressive
behaviour of the input-output level function, and the frequency dependent phase-locking represented
here by the AC/DC ratio of the IHC receptor potential.
Figure 1 shows the input-output function of the IHC model both for the original IHC model (bluefunction) consisting of half-wave rectification and low-pass filtering, and for the proposed non-linear
IHC model (red function), shown here for the ‘on-frequency’ channel at 600 Hz in response to a 600Hz tone. The peak potential was normalized relative to the maximum peak potential of all levels. The
compressive growth is visible in the input-output function of the non-linear model whereas the originalmodel shows a linear growth with increasing sound pressure level. For a comparison to animal data,
see Russell et al., (1986).

Figure 1: The input-output characteristics are shown for the original model (blue), consisting of half-wave
rectification and low-pass filtering, and the proposed non-linear model (red) on a linear (left) and logarithmic(right) scale. In both cases, the on-frequency channel output is shown in response to a sinusoidal signal at 600 Hz.
The response is normalized relative to the maximum peak potential. over all levels. For comparison to animal data,
see Russell et al. (1986).

inpu 2 shows the AC/DC ratio of the IHC receptor potential. Phase-locking is known to decrease
Figurewith stimulus frequency, due to the decrease of the AC component with increasing stimulus frequency
compared to the steady DC component (Palmer & Russell, 1986). For each frequency, the ratio isshown at the output of the on-frequency channel in response to a tone at the corresponding frequency
with an input sound pressure level (SPL) of 80 dB. The results obtained for the two IHC models
resemble each other, with a less steep decline observed above 1 kHz for the non-linear IHC model. In
the original IHC model, a 2nd order low-pass filter with a cut-off frequency of 1 kHz was applied after
half-wave rectification. Both models show a clear decrease of the AC/DC ratio towards higher
frequencies, reflecting the limitations of phase-locking at these frequencies. For a comparison withanimal data, see Palmer & Russell (1986).

Figure 2: The AC/DC ratio is shown for the original IHC model (blue) and the proposed non-linear IHC model(red) as a function of frequency. At each frequency the on-channel output is shown in response to sinusoidal
input signals at 80dB. A clear decrease as a function of frequency can be observed in the AC/DC ratio,reflecting the envelope extraction behaviour of the IHCs.

4. Psychoacoustic Response
CASP has been shown to successfully model human performance in various psychoacoustic
conditions, including forward masking, spectral masking and modulation detection (Jepsen et al.,2008) as well as speech intelligibility (Relaño-Iborra et al., 2019). Here, we compared the simulation
results obtained with the original CASP model with those obtained with the CASP model integrating
the non-linear IHC model. Figure 3 shows the simulations of auditory thresholds for both models.
Note that the auditory thresholds were calibrated within the model and therefore do not display its
predictive power.

Figure 3: Standard auditory thresholds (ISO 389-7, 2019), along with the model simulations
are shown for the CASP implementation with the original IHC model (red) and the proposed
non-linear IHC model (yellow). Both simulations show reasonable agreement with thestandard auditory thresholds.

As can be seen from the figure, both models show similar behaviour to the standard thresholds (ISO389-7, 2019). In order to simulate the auditory thresholds, internal noise was added after the non linearity at the stage of the basilar membrane, i.e., after the dual-resonance non-linear filterbank
implementation. For both models this internal noise was adjusted to account for the auditory
thresholds. Nonetheless, as shown here, the inclusion of a non-linear IHC stage still allows for the
implementation within the model of audibility limitations.

5. Summary and Outlook
A non-linear IHC transduction model was integrated in the computational auditory processing andperception model and evaluated independently from a physiological perspective, showing the
expected phase-locking characteristics and compressive behaviour at higher sound pressure levels.
Subsequently, the model was validated when integrated in the CASP modelling framework regarding
the implementation of auditory thresholds. Following these initialisation steps, the performance of themodel can now be evaluated in large set of psychoacoustic measurements, including simultaneous andforward masking, spectral masking and modulation detection. This will first be done for normalhearing listeners, and then for hearing-impaired listeners. An investigation of the effects of the more
realistic implementation of IHC transduction on the prediction of data from hearing -impaired listeners
will be of particular interest.

REFERENCES
Altoè, A., Pulkki, V., & Verhulst, S. (2018). The effects of the activation of the inner-hair-cellbasolateral K+ channels on auditory nerve responses. Hearing Research, 364, 68–80.
https://doi.org/10.1016/J.HEARES.2018.03.029
Carney, L. H. (2018). Supra-Threshold Hearing and Fluctuation Profiles: Implications for Sensorineuraland Hidden Hearing Loss. In JARO - Journal of the Association for Research in Otolaryngology(Vol. 19, Issue 4, pp. 331–352). Springer New York LLC. https://doi.org/10.1007/s10162-018-06695
Dallos, P. (1985). Response Characteristics of Mammalian Cochlear Hair Cells. In The Journal of
Neuroscience (Vol. 5, Issue 6).
International Organization for Standardization. (2019). ISO 389-7: 2019 Acoustics - Reference zero forthe calibration of audiometric equipment. In International ISO Standard 389-7 (Third).
Jepsen, M. L., & Dau, T. (2011). Characterizing auditory processing and perception in individual
listeners with sensorineural hearing loss. The Journal of the Acoustical Society of America, 129(1),262–281. https://doi.org/10.1121/1.3518768
Jepsen, M. L., Ewert, S. D., & Dau, T. (2008). A computational model of human auditory signal
processing and perception. The Journal of the Acoustical Society of America, 124(1), 422–438.
https://doi.org/10.1121/1.2924135
Jia, S., Dallos, P., & He, D. Z. Z. (2007). Mechanoelectric transduction of adult inner hair cells. Journalof Neuroscience, 27(5), 1006–1014. https://doi.org/10.1523/JNEUROSCI.5452-06.2007
Johnson, S. L., Beurg, M., Marcotti, W., & Fettiplace, R. (2011). Prestin-Driven Cochlear Amplification
Is Not Limited by the Outer Hair Cell Membrane Time Constant. Neuron, 70(6), 1143–1154.
https://doi.org/10.1016/J.NEURON.2011.04.024
Lopez-Poveda, E. A., & Eustaquio-Martín, A. (2006). A biophysical model of the inner hair cell: The
contribution of potassium currents to peripheral auditory compression. JARO - Journal of the
Association for Research in Otolaryngology, 7(3), 218–235. https://doi.org/10.1007/s10162-0060037-8
Lopez-Poveda, E. A., & Meddis, R. (2001). A human nonlinear cochlear filterbank. The Journal of the
Acoustical Society of America, 110(6), 3107–3118. https://doi.org/10.1121/1.1416197
Osses Vecchi, A., Varnet, L., Carney, L. H., Dau, T., Bruce, I. C., Verhulst, S., & Majdak, P. (2022). A
comparative study of eight human auditory models of monaural processing. Acta Acustica, 6, 17.
https://doi.org/10.1051/aacus/2022008
Palmer, A. R., & Russell, I. J. (1986). Phase-locking in the cochlear nerve of the guinea-pig and its
relation to the receptor potential of inner hair-cells. In Hearrng Research (Vol. 24).
Relaño-Iborra, H., Zaar, J., & Dau, T. (2019). A speech-based computational auditory signal processing
and perception model. The Journal of the Acoustical Society of America, 146(5), 3306–3317.
https://doi.org/10.1121/1.5129114
Russell, I. J., Cody, A. R., & Richardson, G. P. (1986). The responses of inner and outer hair cells inthe basal turn of the guinea-pig cochlea and in the mouse cochlea grown in vitro. In Hearing
Research (Vol. 22). Elsevier.
Shamma, S. A., Chadwick, R. S., Wilbur, J., Morris, K. A., & Rinzel, J. (1986). A Biophysical Modelof Cochlear Processing: Intensity Dependence of Pure Tone Responses. The Journal of the
Acoustical Society of America, 80.
Sumner, C. J., Lopez-Poveda, E. A., O’Mard, L. P., & Meddis, R. (2002). A revised model of the innerhair cell and auditory-nerve complex. The Journal of the Acoustical Society of America, 111(5),2178. https://doi.org/10.1121/1.1453451
Verhulst, S., Altoè, A., & Vasilkov, V. (2018). Computational modeling of the human auditory
periphery: Auditory-nerve responses, evoked potentials and hearing loss. In Hearing Research (Vol.
360, pp. 55–75). Elsevier B.V. https://doi.org/10.1016/j.heares.2017.12.018

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0678

Measurement of auditory characteristics of cartilage conduction
under reproducible transducer position, application force, andvibration acceleration
Etsushi NOZAKI1; Ken TAKAKI2; Yoshihiro KAWAHARA3
1

Graduate School of Information Science and Technology, The University of Tokyo, Japan
2

Graduate School of Engineering, The University of Tokyo, Japan

3

Graduate School of Engineering, The University of Tokyo, Japan

ABSTRACT
Cartilage conduction (CC) is a method of presenting sound by applying vibrations to the auricular cartilage.
CC transmits sound efficiently under various transducer positions and weak application force below 1 N,
which realizes comfortable devices. Although the auditory characteristics of CC are sensitive to these
conditions, existing measurement methods have difficulty quantifying the applied force and vibration withvarious transducer positions. The challenges were (i) the precise control of the transducer position andapplication force and (ii) isolation of the characteristics of the transducer from the actual auditory
characteristics. In this paper, we proposed a measurement system to simultaneously enable transducer
position adjustment, 0.1 N step control over application force under 1 N, and direct monitoring of the applied
vibration acceleration. Our method could cancel out transducer resonances from the observed ear canal soundpressure (ECSP) by using the applied vibration acceleration as a reference. The measurement confirmed
ECSP’s dependence on transducer position and a positive correlation between application force and ECSP
under application force below 0.6 N. Our method is the first to realize CC characteristics measurement under
reproducible transducer position, application force, and vibration acceleration, which is essential for the
standardization of the CC measurement process.
Keywords: Cartilage conduction, Ear canal sound pressure, Sensor

1. INTRODUCTION
Cartilage conduction (CC) is a technology that allows sound perception without blocking the ear
by applying vibrations that contain acoustic signals to the auricular cartilage. Both headphones andhearing aids using CC exist, possessing advantages and disadvantages over devices that use air
conduction.
Most headphones are inserted into the ear canal or cover the auricle, which blocks sound from thesurrounding environment. Consequently, it obstructs communication with people around the user andmay cause danger as the user may not notice nearby vehicles. Furthermore, earplugs inserted into theear canal can cause discomfort and otitis externa when worn for long periods of time. Therefore, open ear headphones that can transmit sound without blocking the ear have become popular in recent years.
In particular, wireless headsets using CC, such as OpenRun from Shokz (1), have found wide
application in recent years because of their comfort. However, the amplitude and frequency
characteristics of sound are unstable because the auditory characteristics of cartilage bone conduction
vary greatly depending on the contact conditions of the transducer with the ear cartilage, causing
differences in timbre between individuals and between the left and right ears. There fore, the current
acoustic reproduction performance is lower than that of air-conducted headphones.
In contrast, hearing aids using CC have been developed (2, 3). These hearing aids are used forpatients who cannot use air conduction hearing aids because of ear canal obstruction. Compared tobone conduction hearing aids, which require strong pressure on the skin and discomfort, or bone 1
2
3

nozaki@akg.t.u-tokyo.ac.jp
takaki@akg.t.u-tokyo.ac.jp

kawahara@akg.t.u-tokyo.ac.jp

anchored hearing aids (BAHA), which require surgical implantation of a metal fitting in the skull, CC
hearing aids require only soft contact between the skin and the transducer. Thus, CC hearing aids canbe applied to patients with external auditory canal atresia and patients not comfortable wearing
conventional hearing aids. Hearing aids require a fitting process in which the frequency response anddynamics of the presented sound are precisely adjusted according to the patient’s hearing. However,
fitting is difficult in the case of CC because the auditory characteristics vary significantly dependingon the contact conditions of the transducer with the cartilage.
Therefore, it is necessary to evaluate the changes in auditory characteristics depending on thecontact conditions of the transducer of the device with the auricle when designing a CC device.
However, to make such an evaluation, it is necessary to measure the auditory characteristics of the
CC under various conditions and determine the quantitative relationship between the vibration input
from the device and the perceived sound.
Some studies have quantified changes in the auditory characteristics of the CC depending on theconditions of contact of the transducer with the auricle (4, 5, 6). However, these studies have two
problems: (i) the transducer position and application force cannot be precisely controlled, and ( ii) the
frequency characteristics of the transducer cannot be isolated from the measurement results.
For (i), the transducer position and application force, previous studies put the transducer onto theentrance of the ear canal (2, 7, 8), pressed it down by hand (4), attached it to tape (5, 9), and fixed itwith a rubber headband (7). However, it is difficult to accurately fix the presentation position andapplication force using these methods; even when they are fixed, it is challenging to change theapplication force freely. Considering the auditory characteristics of cartilage conduction are
significantly affected by these uncontrolled parameters, the reproducibility and comprehensiveness ofthe measurement are insufficient. In addition, the application force on the auricle that is comfortablefor humans to wear is 0.3–1.5 N (10). However, no previous studies have measured forces below 1.0
N, except for Shimokura’s study (4), where the transducer was held down by hand. The force appliedby the hand is unstable, resulting in insufficient reliability.
As for (ii), the frequency characteristics of the transducer, previous studies have used custom
fabricated ring-shaped piezoelectric resonators (4, 5, 7, 8, 9, 11) and flat-plate resonators (6, 12).
However, each transducer has a frequency response, and even if the voltage applied to the transduceris specified, it is unclear how much force or acceleration is produced. Moreover, the frequency
response depends on the contact conditions between the transducer and auricle. Therefore, to clarify
the auditory characteristics of cartilage conduction, it is necessary to quantify the acceleration of thetransducer when it is in contact with the auricle. However, no standard calibrators can simulate themechanical properties and quantify the vibrations of the ear cartilage. Furthermore, small-sizedtransducers used for cartilage conduction generally exhibit prominent resonance peaks in the output
force or acceleration when driven at a constant voltage. Therefore, the frequency characteristics ofthe transducer have a greater influence on the measured auditory characteristics than the cartilage
conduction itself.
In this paper, we propose a measurement system that solves the above two bottlenecks whilemeasuring the auditory characteristics of CC. Instead of simply fixing the transducer using an off -theshelf apparatus, as in previous studies, we designed the original measurement system to meet therequirement for the measurement specific to CC measurements, such a s a narrow space for thetransducer and a weak applied force.
The main contributions of this paper are as follows:1.

A measurement system was designed from scratch that incorporates an adjustable fitting
mechanism that can hold the transducer at multiple positions, a force sensor to monitor theforce applied to the skin, and an accelerometer to monitor the vibration applied to the skin.
This paper explains the requirements and the implementation of the system, as well as theevaluation of the sensors.

2.

The measurement process of the frequency response of ear canal sound pressure (ECSP)
ensures a sufficient signal-to-noise ratio (SNR) and validates the result using magnitude
squared coherence (MSC).

3.

Measurement of auditory characteristics of CC with three transducer positions and weak force
applied to the skin (up to 0.6 N) compared to previous studies. By measuring the ECSP per
vibration acceleration level instead of the ECSP per voltage level, we isolated the frequency
characteristics of the transducer from the measurement results.

2. RELATED WORK
2.1 Cartilage conduction (CC)
CC is a method of presenting sound by applying vibrations containing acoustic signals to the
cartilage of the auricle rather than to the bone. Hosoi first presented cartilage conduction in 2004 (13,14) and found that perceived speech intelligibility was not inferior to that of air or bone conduction
and that auditory characteristics were significantly affected by the force applied to the transducer.
The most important factor of CC distinguishing it from bone conduction (BC) is the mechanism ofsound transmission to the inner ear. Shimokura et al. (4) investigated three pathways of the CC sound:direct air pathway (air-conducted sound from the transducer reaches the eardrum through the ear
canal); cartilage-air pathway (vibrations of the aural cartilage and tissue surrounding the external
auditory canal generate sound in the ear canal, which reaches the eardrum by air conduction);
cartilage-bone pathway (vibration of the aural cartilage induces BC sound). They found that the
cartilage-air pathway is dominant in CC hearing in normal-hearing people, which differs from BC,
where the cartilage-air path is not dominant in an unoccluded ear canal. This condition allows the
loudness of the perceived CC sound to be approximated by ECSP measurements because it reflects
the sum of the direct and cartilage-air pathways.
2.2 Transducer position and auditory characteristics of BC and CC
Mastoids and foreheads have been the major research objectives for studying the auditory
characteristics of BC. The mastoid is physically close to the inner ear, making it suitable for BC;
however, it is small and requires precise transducer positioning. McBride et al. (15) investigatedauditory thresholds at various locations on the skull and found a difference of approximately 15 dB
between the condyle, which is closer to the inner ear, and the jaw, which is farther away. Dobrev etal. (16) measured auditory thresholds by pressing transducers at seven locations around the auricle
and found that the condyle, which is closer to the inner ear, had the lowest threshold. These results
suggest that the auditory threshold is the lowest when the bone near the inner ear vibrates in the BC.
Moreover, to investigate the relationship between the position of the transducer and the auditory
threshold in CC, Shimokura et al. (5) measured the ECSP with the transducer fixed with double -sided
tape at four positions: entrance to the ear canal, behind the concha, on the ear lobe, and overtop of thetemporal bone. The results showed that the ECSP at the entrance to the ear canal and behind the concha
was the highest; the ECSP at these locations was > 30 dB higher than that a t the top of the temporalbone.
2.3 Transducer application force and auditory characteristics of BC and CC
In BC, the relationship between application force and auditory characteristics has long been studied
extensively, with values ranging from 2.0 N (17) to 7.4 N (18) proposed as application forces that canpresent sound with good reproducibility. A typical international standard specifies an application forceof 5.4 ±0.5 N when a B71 transducer is used. However, this application force is strong and causes
discomfort. Therefore, there are proposals to present sounds with a weaker application force. Toll etal. (19) measured auditory thresholds when the mastoid protrusions were presented with a headband
at a force of 2.4 N to 5.4 N, concluding that no significant difference was observed.
In contrast, Shimokura et al. (4) studied the relationship between applied force and auditory
characteristics. In the experiment, the participants held and pressed the transducer against the auricleby hand and adjusted the force by monitoring the force value from the film-like force sensor placed
between the transducer and the auricle. However, the disadvantage of this method is that it is difficult
to maintain a stable and precise application force owing to the lack of precision of the sensor andinstability of the force generated by the hand. Although a low application force of 0.77 N was achieved
with the hand-holding method (4), there are no examples of measurements of CC hearing with
controlled application forces lower than it.
2.4 Measurement of the transducer output
For air conduction and BC, standardized methods exist for calibrating the transducer output. For
air conduction, calibration was performed by converting the sound pressure level output by thetransducer to a voltage using a mechanical coupler, as specified in IEC 60318-3 or IEC 60318-5. For
the BC, a mechanical coupler (called an artificial mastoid) specified in IEC 60318 -6 is used forcalibration by converting the force level output by the transducer into a voltage . The artificial mastoidis a mechanical coupler that simulates the mechanical impedance of the human mastoid and forehead.
When the transducer is fixed at 5.4 N, the force output by the transducer is converted to a voltage.
However, the actual force output of the transducer may vary in real measurement conditions where
the mechanical coupling to the bone is different for each person or transducer position.
However, for CC, there is no standardized output calibration method for the transducer. Calibration
using artificial mastoids is also not possible because the cartilage and the mastoid have different
mechanical impedances. Previous studies used custom-made ring-shaped piezoelectric transducers (4,5, 7, 8, 9, 11) and flat-plate transducers (6, 12). However, the output of the transducers in this study
is defined by the voltage applied to the transducers because the output characteristic of theacceleration per voltage of these transducers is unknown. Consequently, even if the frequency
characteristics of the auditory threshold or ECSP are obtained, they cannot be evaluated as thecharacteristics of the CC itself because they include the characteristics of the transducer.
When vibration is directly measured while it is being presented to a human without cal ibrating thetransducers with a coupler, such as an artificial mastoid, an impedance head is used. In some BC
measurements, the transducer output was directly measured using an impedance head (20, 21, 22, 23).
However, it is difficult to apply it to cartilage conduction because the size of the entire mechanism,
including the transducer and impedance head, is large (a few centimeters in size), making it difficult
to contact narrow parts of the auricle.
To solve this problem, Hodgetts et al. (24) attached a small microelectromechanical system(MEMS) accelerometer to the back of the transducer to measure the acceleration. However, it seems
unlikely that the acceleration on the back of the transducer and the acceleration of the surface incontact with the skin coincide at high frequencies. Other calibration methods are also being
investigated. Shimokura et al. (11) reproduced the low-frequency portion of the CC characteristics
below 1.5 kHz by attaching a polyurethane tube that mimics the ear canal to a human sk ull model andsuccessfully reproduced the high-frequency portion above 2 kHz using the head and torso simulator(HATS) auricular model. However, because this method is not based on the theoretical modeling of
mechanical behavior, it cannot be said that this calibration method can be used for other transducers.

3. DESIGN AND EVALUATION OF THE MEASUREMENT SYSTEM
This section describes the design and evaluation of a measurement system that presents soundgenerated by CC. As discussed in Section 1, the measurement system must satisfy the following two
requirements: (i) the transducer position and application force can be precisely controlled and (ii) the
frequency characteristics of the transducer can be isolated from the measurement results . We first
introduce an overview of the system and then explain the detailed design to achieve the requirements.,
In this study, we have clearly defined the requirements to be met by the measurement system as awhole, considering skin contact conditions to be achieved and the mechanical properties of themeasurement fixture, unlike previous studies which focused on simply attaching existing components.
Subsequently, we implemented and evaluated the system based on thes e requirements.
3.1 System overview
Figure 1(a) shows the head-worn measurement system. This system consists mainly of a headband(3M™ speedglass™ headband 533000), three metal arms, a force sensor unit, a transducer unit, andan interface printed circuit board (PCB) fixed to the headband.
The headband and metal arms form the adjustment mechanism of the measurement system. The
headband has three adjustable straps at the top, forehead, and back of the head, allowing the headband
to fit a variety of head shapes. The headband secures the metal arms to the subject’s head at one end,
and the three metal arms with joints can fix the transducer to the desired position and orientation.
Figure 1(b) shows a more detailed image of the transducer. A linear rail was attached to the metalarm at the other end, which enabled fine adjustment of the transducer position and application force.
The transducer unit, comprising the transducer, accelerometer, and metal contact part with the skin,is held by the force sensor unit attached to the linear rail.
The interface PCB was connected to a controller board that controlled the electronics of the systemand transmitted the sound signals to the transducer. The controller board has three functions: (i) readtransducer vibration acceleration and application force from the sensors, (ii) control linear rail by

Figure 1 – (a) Proposed head-worn measurement system. (b-c) Enlarged images around the transducer.

Figure 2 – (a) Transducer used in the system. (b) Disassembled view of the transducer unit.
(c) Transducer unit held by the force sensor unit.
signal from the controller board, and (iii) transmit sound signals from controller board to transducer.
3.2 Design and evaluation for controlling transducer position
The requirements for the system for controlling transducer position are as follows:1.

The adjuster mechanism should accommodate individual differences in head shape.

2.

The size of the resonator part, including the resonator, accelerometer, and skin contact par t,
should be maintained at approximately 15 mm square to fit a narrow gap between the auricle
and head.

3.

The transducer can be fixed to a wide range of areas around the auricle, including the front
and back.

All participants could wear the headband owing to the adjustable headband of the system, thus
fulfilling the first requirement.
To satisfy the second requirement, we used a miniature transducer shown in Figure 2(a) (12 × 5 ×3 mm) and a miniature accelerometer (IIS3DWB, STMicro, 2.5 × 3.0 × 0.83 mm). By putting thesesmall devices into one unit using CNC-milled aluminum parts and 3D printed parts, as shown in
Figure 2(b, c), the whole transducer unit without the supporting iron rod and PCB was fitted within
15 mm square.
The third requirement was met by using three metal arms with joints. The measurement confirmedthat the system could position the transducer at three locations: mastoid, behind the concha, andtragus of all participants.
3.3 Design and evaluation for controlling application force
The requirements for the system for controlling application force are below:
1. The measurement system was attached to the head of the person to be measured to minimize
the effect of unintended head motion of the measurement participant.

2. The resonator has to be pressed against the skin with a force between 0.1–1.5 N (or until large
deformation of the skin is noted) as sound presented by CC can be perceived well under theapplied force as weak as 0.1 N. The resonator position must not be moved by the reaction to
the applied force.
3. The force sensor should monitor the applied force on the skin with a resolution of 0.1 N or
higher in the above force range.
Although it is not possible to completely eliminate the effect of head motion, we could conduct allmeasurements within ±0.05 N of the target application force by attaching the system to the head, as
noted in the first requirement. In addition, adjusting the rough transducer position using the metalarms and then pressing the transducer against the skin using a linear rail confirmed that the second
requirement was met for all participants.
In addition, a miniature force sensor (HSFPAR004A, Alps Alpine) and a 24-bit analog-to-digitalconverter (ADC) (LTC2485, Analog Devices) enabled item 3. To calibrate the sensor, the outputs fromthe sensor loaded with a known weight (up to 500 g) were compared with the actual force. The root
mean squared error (RMSE) was 0.01384 N, sufficiently smaller than 0.1 N. Another cause of error
in the measured force is the weight of the transducer unit. To cancel th is effect, the angle of thetransducer unit against gravitational acceleration was measured using an accelerometer, and thegravitational force calculated from the angle and the weight was subtracted from the force sensor’s
raw data.
3.4 Design and evaluation for monitoring transducer output
The requirements for the system for monitoring transducer output are as follows:
1. A small accelerometer with a bandwidth of at least 4 kHz was placed in the vicinity of the skincontact area.
2. The parts of the contact zone between the accelerometer and skin should have a suf ficiently
higher mechanical impedance than the skin and should be mechanically coupled tightly to theaccelerometer.
To satisfy the first requirement, we used a wide-bandwidth accelerometer (IIS3DWB, STMicro)with a flat frequency response of up to 6 kHz and a sampling frequency of 26.667 kHz.
The bandwidth of the sensor and the mechanical properties of the entire system are important for
precisely measuring high-frequency acceleration on the skin surface. However, the mechanicalproperties of a large part of the system can be ignored by placing the accelerometer in the immediate
vicinity of the skin contact area, as shown in Figure 2(b). If the mechanical impedance of theaccelerometer itself and the parts between the accelerometer and the skin are sufficiently higher thanthat of the skin, the accelerations of the accelerometer and the surface of the skin are almost the same.
Therefore, we used CNC-milled A7075 aluminum parts for the transducer unit and assembled themwith M1.4 screws and an adhesive (Aron Alpha® Pro, Toagosei) for rigidity.
To evaluate the accuracy of the vibration acceleration measurement, we compared the acceleration
output from the sensor and the vibration acceleration at the surface of the skin contact part measured
using a laser Doppler vibrometer (LDV) (MSA-500, Polytec). Skin contact was simulated using aphantom (cylindrical, outer diameter of 12 mm, inner diameter of 4.5 mm, thickness of 3 mm) madeof a special urethane resin (human skin gel C15, Exseal Corporation). The measurement confirmedthat the difference between the vibration acceleration level (VAL) from the sensor and the LDV waswithin ±2 dB for a frequency range of 200 Hz to 8 kHz when the transducer unit was i n contact withthe skin phantom.

4. MEASUREMENT METHOD
4.1 Measurement of ECSPThe measurement participants (three males and one female, aged 21 to 25 years) sat in a chair andwere fitted with the measurement system described in the previous section. A probe mic rophone (ER7C, Etymotic Research) was inserted into the ear canal near the eardrum by CC. First, an exponential
swept sine (ESS) signal from 100 Hz to 12 kHz rising in 1 s was presented twice with an interval of
0.5 seconds, and the transducer acceleration was measured during the presentation. From theFigure 3 – (a) The relationship between ECSP per vibration acceleration and the transducer position (seeFigure 3-(b) for the correspondence between the symbols and the positions; 0.4 N application force;mean values for all participants; shadows are standard deviations; points with MSC < 0.35 are
excluded). (b) The transducer position (dotted circle means the transducer was pressed behind theauricle). (A) behind the concha, (B) tragus, (C) mastoid.
frequency characteristics of the vibration acceleration measured above, a compensation filter was
calculated to make the vibration acceleration as flat as possible at a VAL of 132±5 dB. The frequency
characteristics of the vibration acceleration and ECSP of the transducer were measured simultaneously,
while the filtered ESS signal was presented ten times at an interval of 0.5 seconds, and the average ofthese measurements was used to calculate the frequency characteristics of the vibration accelerationand ECSP. The number of FFT points was 8192, and the Blackman–Harris window was used to
calculate the frequency responses. The frequency response of the ECSP per unit vibration acceleration
was calculated by dividing the ECSP by the measured VAL.
In addition, we used the MSC to ensure that the resulting frequency response was not dominatedby noise. MSC was calculated from the presented signal and the measured waveform ; data points,
where the calculated MSC values were below 0.35, were excluded from the plot. We chose a thresholdof 0.35 because it corresponds to a significance level of 95%, which means that the possibility of thedata point containing only noise is less than 5%.
4.2 Transducer position and application force
Measurements were performed by pressing the transducer against the auricle using the proposed
measurement system. The effect of subtle deviations in the position and angle of the transducer was
also investigated by reapplying the transducer and repeating the measurement thrice for eachtransducer position and application force. The application force was varied in 0.1 N in steps of 0.0 N(non-contact state). The maximum force was set at a value that would not distress the participants ateach position (0.4 N at tragus, 0.6 N behind the concha, and 0.5 N at mastoid. The force sensor values
were constantly monitored during the measurements to confirm that the error did not exceed ±0.05 N.
The skin contact part was hemispherical with a diameter of 8 mm, dispersing pressure andpreventing skin damage. This part was prepared by CNC machining A7075 aluminum to ensure highstiffness.

5. RESULTS AND DISCUSSION
5.1 Transducer position and ECSP
Figure 3(a) shows the relationship between the ECSP per vibration acceleration and the transducer
position. Although the overall frequency response is not entirely flat, it does not exhibit a downward
or upward trend, suggesting that constant acceleration is suitable for a presentation with flat frequency
response. This result contrasts with the previous measurement by Shimokura et al. (4), in which theresult had a frequency response with a prominent peak with a difference in magnitude greater than 40

Figure 4 –Relationship between ECSP per vibration acceleration and the transducer application
force (mean values for all participants; points with MSC < 0.35 are excluded). (a) behind theconcha, (b) tragus, (c) mastoid.
dB across the frequency. However, the previous measurement measured the ECSP per vibration andfailed to cancel out the frequency characteristics of the transducer. Therefore, the 40 dB peak in it was
likely a result of transducer resonation.
Overall, the presentation to the mastoid had the smallest ECSP per vibration acceleration , while
the presentation to the tragus had the largest ECSP per vibration acceleration over almost the entire
frequency range. This result does not contradict the previous measurement of Shimokura et al. (5).
The presentation to the tragus had the largest ECSP per vibration acceleration at frequencies below 3
kHz, while a large ECSP per vibration acceleration characterized the presentation to the back of theconcha at frequencies above 3 kHz and a flat frequency response overall.
5.2 Transducer application force and ECSP
Figure 4 shows the relationship between the mean ECSP per vibration acceleration and transducer
application force. At an application force of 0.0 N, there was no contact between the transducer andthe skin, indicating that sound propagation was only by the direct air pathway from the transducer. As
shown in Figure 4(a, b), because of the contribution of the cartilage-air pathway, the ECSP per
vibration acceleration was larger when the skin behind the concha or the tragus was in contact withthe transducer. In addition, the stronger the application force, the larger the ECSP per vibration
acceleration. However, it was confirmed that the effect of the applied force was negligible above 4
kHz. As the position of the transducer does not change significantly in the contact state, the larger
ECSP per vibration acceleration can be attributed to the cartilage-air pathway.
By contrast, as shown in Figure 4(c), the ECSP per vibration acceleration was almost unchanged
when the mastoid was in contact with the transducer. Therefore, for the presentation of the boneconducted sound to the mastoid at forces below 0.5 N, the effect of ECSP generated by vibrating theskull is small, indicating sound transmission through other pathways.
In Figure 4(b), the ECSP per vibration acceleration in the non-contact and contact states are almostTable 1 –Standard deviation of the ear canal sound pressure per vibration acceleration for eachtransducer position and application force.
Application force

Behind the concha

[N]

[dB]

0.0

Tragus [dB]

Mastoid [dB]

6.3

6.9

6.8

0.1

6.5

6.3

6.7

0.2

6.6

6.3

6.7

0.3

6.4

6.2

6.7

0.4

6.5

6.3

7.0

0.5

6.7

—

6.9

0.6

6.8

—

—

identical in the frequency band above 3 kHz, showing that the contribution of the direct air pathway
to the ECSP was dominant, thus indicating that the sound radiated by the transducer into the air was
too loud. In all the cases in Figure 4, the frequencies of the peaks and notches in the contact stateroughly correspond to those in the non-contact state.
However, in (a) and (b), some notches showed a slight change in the center frequency dependingon the application force, whereas others did not (e.g., notches near 250 Hz and 2.5 kHz). This isbecause the change in the application force or the change in the coupling with the transducer causechanges in the resonant frequency. Therefore, the cartilage-air pathway was dominant in thisfrequency range. In contrast, notches with unchanged center frequencies were likely to be caused by
the direct air pathway. The depth of such notches decreases as the applied force increases because theinfluence of the direct air pathway becomes weaker.
Table 1 presents the standard deviations of ear canal sound pressure per vibration acceleration foreach transducer position and application force. For each condition, the standard deviations acrossparticipants and trials were calculated. Then, the median value was selected along the frequency axis.
The standard deviation of the ECSP per vibration acceleration for all transducer positions does notvary significantly from 6 dB to 7 dB depending on the contact and non-contact states but on thestrength of the application force in the contact state.
5.3 Limitations
As described in Section 5.2, an excessive amount of air-conducted sound is observed in the results.
This observation is supported by the fact that the ECSP in the non-contact state and contact state isroughly the same in specific frequency ranges, and that peaks and notches of ECSP frequency
responses in the non-contact state are present in the contact state, indicating interference of the direct
air pathway with the cartilage-air pathway. As the direct air pathway relies only on the sound leakage
from the transducer and the transmission characteristics of the air-conducted sound between thetransducer and eardrum, this situation is not ideal for measuring the characteristics of the cartilageair pathway of the CC sound.
To determine whether the variation in measurement results is primarily due to the variation in the
direct air pathway or whether the cartilage-air pathway also has a large enough effect of causing
individual measurements to vary, it is desirable to reduce the sound radiated into the air by optimizing
the shape of the transducer and the vibration isolation with the entire measurement fixture.
The main reason for the high noise emitted by the jig into the air is that the transducer unit, force
sensor unit, linear rail, and metal arms are all rigidly fixed, and no component absorbs the vibration.
Therefore, vibrations generated by the transducer can be transmitted to the metal arms without
attenuation and radiated from the large surface of the metal arms. A vibration-absorbing rubber
adhesive is used in the transducer unit to fix the transducer to the adapter components. However,advanced anti-vibration measures are required to prevent sound radiation. In particular, incorporat ing
a component with high vibration isolation performance between the two units would effectively
prevent vibration transmission between the transducer unit and force sensor unit .

6. CONCLUSIONSIn this paper, we proposed a measurement system to solve two problems in the evaluation ofpractical devices for CC: (i) the inability to precisely control the transducer position and application
force, and (ii) the inability to isolate the frequency characteristics of the transducer from themeasurement results. Using this system, we clarified the auditory characteristics of the CC at various
presentation positions and with a weak force considering the skin contact conditions of the transducer
in a practical device.
The adjuster mechanism of the proposed measurement system is composed of a headband and a
metal arm, enabling the system to be worn on heads of various shapes and fix the transducer at any
position around the auricle. In addition, a small unit incorporating a force sensor and an acceleration
sensor was created to monitor the applied force and vibration acceleration.
Finally, using the proposed measurement system, we measured the ECSP of four experimental
participants at various presentation positions and with a weak force, controlled the applied force, andmonitored the applied vibration acceleration. Auditory characteristics were obtained by isolating theinfluence of the frequency characteristics of the transducer itself by calculating ECSP per vibration
acceleration. Overall, the results revealed a generally flat frequency response shape with respect tothe vibration acceleration masked by the frequency characteristics of the transducer itself in previous
studies. More specifically, the transducer position at the tragus and the stronger application force
resulted in a large ECSP per vibration acceleration. However, it was found that the vibration isola tionmeasures to prevent vibration propagation from the transducer to the metal arm are necessary to reducethe influence of the direct air pathway on the measurement results.
Future research goals include standardizing the auditory characteristics of CC and the measurement
method for the characteristics of CC audio devices.

ACKNOWLEDGEMENTS
This work was supported by JST ERATO, Japan, under Grant No. JPMJER1501. The authors would
also like to thank Yoshio Mita (The University of Tokyo) for technical assis tance in using the laser
Doppler vibrometer and Seiji Nakagawa (Chiba University) for advice on CC research.

REFERENCES
1. Shenzhen Voxtech Co.,Ltd. Shokz OpenRun; 2022. (Accessed on 02/08/2022). https://shokz.com/
products/openrun.
2. Shimokura R, Hosoi H, Iwakura T, Nishimura T, Matsui T. Development of monaural and binaural behindthe-ear cartilage conduction hearing aids. Applied Acoustics. 2013;74(11):1234-40.
3. Nishimura T, Hosoi H, Saito O, Shimokura R, Yamanaka T, Kitahara T. Cartilage Conduction Hearing
Aids for Severe Conduction Hearing Loss. Otology & Neurotology. 2018;39(1):65-72.
4. Shimokura R, Hosoi H, Nishimura T, Yamanaka T, Levitt H. Cartilage conduction hearing. The Journal of
the Acoustical Society of America. 2014;135(4):1959-66.
5. Shimokura R, Hosoi H, Nishimura T, Yamanaka T. Aural cartilage vibration and sound measured in theexternal auditory canal for several transducer positions. Journal of Temporal Design in Architecture andthe Environment. 2013 Jan;12:137-43.
6. Jitsukawa T, Otsuka S, Nakagawa S. Effects of cartilage conduction vibrator placement in the pinna on thedetection threshold and the ear canal sound pressure. In: Proceedings of the 23rd International Congresson Acoustics: integrating 4th EAA Euroregio 2019: 9-13 September 2019 in Aachen, Germany. Aachen,
Germany; 2019. p. 6595-7.
7. Nishimura T, Hosoi H, Saito O, Miyamae R, Shimokura R, Matsui T, et al. Is cartilage conduction classified
into air or bone conduction? The Laryngoscope. 2014;124(5):1214-9.
8. Nishimura T, Hosoi H, Saito O, Miyamae R, Shimokura R, Yamanaka T, et al. Cartilage Conduction Is
Characterized by Vibrations of the Cartilaginous Portion of the Ear Canal. PLOS ONE. 2015 Mar;10(3):111.
9. Nishimura T, Hosoi H, Saito O, Miyamae R, Shimokura R, Matsui T, et al. Cartilage conduction efficiently
generates airborne sound in the ear canal. Auris Nasus Larynx. 2015;42(1):15-9.
10. Liao F, Zheng J, Chen Q, Chen H, Qi X, inventors; Shenzhen Voxtech Co., Ltd, assignee. Systems forbone conduction speaker. 10609496; 2015.
11. Shimokura R, Hosoi H, Nishimura T, Iwakura T, Yamanaka T. Simulating cartilage conduction sound to
estimate the sound pressure level in the external auditory canal. Journal of Sound and Vibration.

2015;335:261-8.
12. Yap GS, Otsuka S, Yumoto M, Nakagawa S. Evaluation of Frequency Resolution Characteristic ofCartilage conduction Hearing using Physiological and Psychological Measurement. In: Proceedings ofthe 23rd International Congress on Acoustics: integrating 4th EAA Euroregio 2019: 9-13 September 2019in Aachen, Germany. Aachen, Germany; 2019. p. 6668-9.
13. Tamura M, Abe Y, Hosoi H, inventors; NEC TOKIN Corporation, Hosoi H, assignees. Receiver. Japanese
Patent Application Number 2004-166644; 2004.
14. Tamura M, Abe Y, Hosoi H, inventors; NEC TOKIN Corporation, Hosoi H, assignees. Approach in theuse of cartilage conduction speaker. Japanese Patent Number 4541111; 2004.
15. McBride M, Letowski T, Tran P. Bone conduction reception: Head sensitivity mapping. Ergonomics.
2008;51(5):702-18.
16. Dobrev I, Stenfelt S, Röösli C, Bolt L, Pfiffner F, Gerig R, et al. Influence of stimulation position on thesensitivity for bone conduction hearing aids without skin penetration. International Journal of Audiology.
2016;55(8):439-46.
17. Harris JD, Haines HL, Myers CK. A helmet-held bone conduction vibrator. The Laryngoscope.
1953;63(10):998-1007.
18. Lau CC. The effect of coupling force on bone conduction audiometry. British Journal of Audiology.
1986;20(4):261-8.
19. Toll LE, Emanuel DC, Letowski T. Effect of static force on bone conduction hearing thresholds andcomfort. International Journal of Audiology. 2011;50(9):632-5.
20. Corliss ELR, Koidan W. Mechanical Impedance of the Forehead and Mastoid. The Journal of the
Acoustical Society of America. 1955;27(6):1164-72.
21. Flottorp G, Solberg S. Mechanical impedance of human head bones (forehead and mastoid portion of thetemporal bone) measured under ISO/IEC conditions. The Journal of the Acoustical Society of America.
1976;59(4):899-906.
22. Lundgren H. Bone Conduction Transducers and Output Variability [Master’s thesis]. Göteborg, Sweden:
Chalmers University of Technology; 2010.
23. Queller JE, Khanna SM. Changes in bone conduction thresholds with vibrator contact area. The Journalof the Acoustical Society of America. 1982;71(6):1519-26.
24. Hodgetts W, Scott D, Maas P, Westover L. Development of a Novel Bone Conduction Verification Tool
Using a Surface Microphone: Validation With Percutaneous Bone Conduction Users. Ear and Hearing.
2018;39(6):1157-64.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0984

Electrocochleographic frequency following responses as apotential marker of age-related peripheral degeneration
Miguel TEMBOURY-GUTIÉRREZ11, Jonatan MÄRCHER-RØRSTED1, Gerard ENCINALLAMAS1, Jens HJORTKJÆR1,2, Michael BILLE3, Jesper Borchort YDE3 and Torsten DAU1,3
1
2

Hearing Systems, Health Technology Department, Technical University of Denmark (DTU), Denmark

Danish Research Center for Magnetic Resonance, Center for Functional Diagnostic Imaging and Research,
Copenhagen University Hospital, Denmark3

Copenhagen Hearing and Balance Center; Ear, Nose and Throat (ENT) & Audiology Clinic, Rigshospitalet,
Copenhagen University Hospital, Denmark

ABSTRACT
Wave-I of the transient-evoked auditory brainstem response has been proposed as a marker of peripheral
neural degeneration, since it represents auditory nerve (AN) activity and can be separated in time from presynaptic hair-cell potentials and subsequent brainstem potentials. However, evidence suggests that steadystate potentials, like frequency following responses (FFRs), might be more representative measures of neuraldegeneration. In steady-state potentials, activity from different generators overlap in time and may be difficult
to disentangle. Here, we measure peripheral responses with electrocochleography to quantify age-related ANdegeneration. We disentangle AN responses from ‘central’ FFRs associated with brainstem generators andcorroborate the absence of such brainstem activity with transient stimuli. We estimate the magnitude of thepresynaptic cochlear microphonic that can obscure the AN contribution to the overall peripheral responsewith a high-frequency tone where the AN is minimally responsive. Recordings were obtained from 15 young
and 14 older human participants with near-normal audiograms in response to high-level 10-ms tones at 516,
1032 and 3096 Hz and 100-µs clicks. Our results show a reduced peripheral response to the tones and clicks
in the older participants, consistent with the presence of age-related degeneration across AN fibers in older
listeners with near-normal hearing thresholds.
Keywords: Diagnosis, Hearing, Electrophysiology

1. INTRODUCTION
Results from animal studies suggest that the synapses between inner hair cells (IHCs) and a uditory
nerve fibers (ANFs) disconnect after exposure to loud noises or due to aging [1,2]. This has beentermed cochlear synaptopathy (CS), which leads to the loss of the disconnected ANFs. These effects
can precede IHC and outer hair cell (OHC) dysfunction and thus can go undetected in hearing
threshold tests [3]. ANF loss could be associated with deficits in the processing and perception ofcomplex sounds, such as speech or music, in the presence of background noise [4]. However, a direct
link between such limitations at supra-threshold levels and sensorineural hearing loss (SNHL) remains
unclear in humans [5].
Several electrophysiological markers have been proposed to quantify AN dysfunction. Wave-I of
the auditory brainstem response (ABR), also called the compound action potential (CAP), is a farfield potential that is generated by the AN and reflects peripheral neural degeneration [6]. The
advantage of this marker is that, since it is evoked by transient stimuli, the contributions of eachgenerator are separated in time and can be isolated. However, hair cell damage can affect suprathreshold ABR amplitudes and AN fiber loss can therefore not be isolated with these measures [7].
The frequency following response (FFR) reflects synchronized phase-locking to the fine-structureof sustained acoustic stimuli. Modeling results from Märcher-Rørsted et al. (2022) show that, at high1

mtegu@dtu.dk

levels and low stimulation frequencies, FFR amplitudes are unaffected by medium -to-high-frequency
OHC dysfunction. Additionally, computer AN simulations suggest that ANF loss could, at least partly,
explain the age-related reduction in FFR amplitudes that has been reported in the literature [8-10].
However, with sustained stimuli, potentials originating from different subcortical generators are
difficult to disentangle in the far-field. Here we used electrocochleography to isolate the response
from the auditory periphery and measured FFRs in young and older participants with near-normal
hearing thresholds.

2. METHODS
2.1 Participants
Twenty-nine subjects with clinical hearing thresholds below 25 dB hearing level (HL) up to 6 kHz
were recruited for the study. They were separated into a group of young (n = 15, 20 -31 years, 5 male)and older (n = 14, 58-79 years, 4 male) participants. The ear canal status was checked via otoscopeand extended high-frequency thresholds (10 to 16 kHz) were recorded with IP30 insert phones(RadioEar).
2.2 Stimuli
Transient evoked potentials (ABRs and CAPs) were obtained by stimulating with 100-µs clicks,
presented 6000 times with alternating polarity at 115.5 dB ppSPL. FFRs were recorded at two
stimulation levels. Short, 10-ms pure tone pips were presented at 100 dB ppSPL at 516, 1032 and3096 Hz (1-ms up and down ramps), and longer 250-ms pure-tone bursts at 80 dB rmsSPL at 516 and1086 Hz. The clicks and tone pips were presented at 12 repetitions per second in b oth condensation(C) and rarefaction (R) phases of 3000 presentations each. Longer tone-bursts were presented with
two repetitions per second, 150 times for each polarity (1-ms up and down ramps). All stimuli were
presented monaurally to the ear with the lowest average audiometric thresholds.
2.3 Data acquisition and analysis
The BioSemi ActiveTwo system was used to record the electrophysiological data from electrodeson the vertex (FPz) and the ipsilateral mastoid (P9 or P10), with a sampling frequency of 16384 Hz.
Electrocochleographic potentials were recorded simultaneous using a tympanic membrane (TM)electrode (Sanibel). An Ear Nose and Throat (ENT) medical doctor inspected the ear canal, appliedliquid local anesthesia (xylocaine (lidocaine), and 10 minutes later, removed the anesthetic and placed
the TM electrode on the ear drum of the ipsilateral ear.
Peripheral components where obtained with the TM-to-ipsilateral mastoid configuration, where
the CAP and the summating potential (SP) peak-to-trough amplitude and latency were extracted in
response to the clicks. With this configuration, we obtained ‘peripheral FFRs’, or auditory nerve
neurophonics (ANNs) to pure tones, where we obtained the spectral amplitude, signal -to-noise ratio(SNR), and latency – calculated as the maximum of the cross-correlation between the response andthe stimulus. Brainstem responses were measured between the ipsilateral mastoid and vertex
electrodes. ABR wave-V, amplitudes and latencies were derived for the transient stimuli and FFR
spectral amplitudes, SNRs and latencies were recorded in response to the pure tones.

3. RESULTS
Figure 1 shows across group averaged responses to 100-µs clicks and 10-ms pure tone pips (516
and 1032 Hz) recorded simultaneously from three different electrode configurations : A) Mastoid-tovertex, B) TM-to-vertex and C) TM-to-mastoid. The top row shows polarity-independent brainstemcomponents in panels A and B (ABR waves III and V) and the enhanced peripheral component (CAP)in panels B and C – configurations that use the TM electrode. The middle and bottom rows show thepolarity-dependent part of the response to the tones (i.e. (C-R)/2). For the middle row, panel A showsa response composed of a peripheral contribution and a brainstem contribution that are also present
in panel B. Yet, for configurations using the TM electrode, the peripheral component is enhanced, as
expected. In panel C, the brainstem component is absent such that the peripheral component can beisolated. The bottom row only shows the peripheral contributions for the different electrodeconfigurations, since there is minimal phase-locking in the brainstem to frequencies above 1 kHz [11].
To study the peripheral response we thus used the configuration shown in panel C, TM-to-mastoid.

Figure 1. Averaged responses recorded simultaneously with different electrode configurations. Top row: polarity-independent
ABR/CAP responses to 100-µs clicks, (C+R)/2. Middle and bottom rows: 516 and 1032 Hz polarity-independent FFR/ANN
responses to 10-ms pure tones, (C-R)/2. A) Mastoid-to-vertex (P10-FPz) configuration: shows reduced-peripheral andbrainstem contributions, which are entangled. B) TM-to-vertex (FPz) configuration: shows enhanced peripheral responses but
brainstem components are still present. C) TM-to-mastoid (P10) configuration: enhanced peripheral contributions are isolated
since the brainstem components are absent. Shaded areas represent standard error.

Figure 2 shows the group average response, divided into young (orange) and older (purple)
participants. Traces in the time domain were corrected for individual latencies to aid visualization.
The upper panels A-E show potentials obtained with the TM-to-mastoid configuration representing
peripheral components. The bottom panels F-J show the potentials obtained with the mastoid-to-vertex
configuration representing brainstem contributions. The CAP amplitude (panel A) was reduced by
60% in the older group, while w-V amplitude (panel F) did not show a statistically significant
reduction. Panels B-E show the ANN to 516 Hz and 1032 Hz in the time and frequency domain,
respectively, and panels G-J show the corresponding FFR results. The ANN spectral amplitudes werereduced by 72% and 63% for the older participants at 516 and 1032 Hz, respectively. The FFR spectral
amplitudes were reduced by 25% and 61% for the older participants at 516 and 1032 Hz, respectively.

4. DISCUSSION
We observed reductions in the pure-tone FFR spectral amplitudes of older participants with nearnormal audiometric thresholds, which are consistent with previous results. Additionally, a similar
reduction was observed in the FFRs originating in the AN, the ANN, confirming the modeling resultspresented in Märcher-Rørsted et al. (2022). Although significant differences for thresholds in theextended high-frequency region, revealed the presence of basal OHC dysfunction in the older
participants, the AN simulations from Märcher-Rørsted et al. (2022) indicated that these differences
do not affect the phase-locking in the AN to low-frequency high-level pure-tones. These results
suggest that neural deficits already at the level of the AN in older listeners affect the amplitude of thephase-locked potentials measured at the scalp.

ACKNOWLEDGEMENTS
This work was financed by the Oticon Centre of Excellence for Hearing and Speech Science

(CHeSS) supported by the Oticon Foundation and took place in the Copenhagen Hearing and BalanceCenter at Rigshospitalet, Copenhagen, Denmark.

Figure 2. Delay-corrected average responses for young (orange) and older (purple) participants. Top panels, A-E wererecorded with the TM-mastoid configuration, isolating the peripheral response, while bottom panels, F-J were recorded withthe mastoid-vertex configuration, with an enhanced brainstem response. Panels A and F show the CAPs and ABRs, respectively.
Panels B-E and G-J show ANNs and FFRs in the time and frequency domain, respectively.

REFERENCES
1. Kujawa SG, Liberman MC. Adding insult to injury: cochlear nerve degeneration after “temporary” noiseinduced hearing loss. Journal of Neuroscience. 2009 Nov 11;29(45):14077-85.
2. Sergeyenko Y, Lall K, Liberman MC, Kujawa SG. Age-related cochlear synaptopathy: an early-onset
contributor to auditory functional decline. Journal of Neuroscience. 2013 Aug 21;33(34):13686-94.
3. Chambers AR, Resnik J, Yuan Y, Whitton JP, Edge AS, Liberman MC, Polley DB. Central gain restores
auditory processing following near-complete cochlear denervation. Neuron. 2016 Feb 17;89(4):867-79.
4. Bharadwaj HM, Verhulst S, Shaheen L, Liberman MC, Shinn-Cunningham BG. Cochlear neuropathy andthe coding of supra-threshold sound. Frontiers in systems neuroscience. 2014 Feb 21;8:26.
5. Bramhall N, Beach EF, Epp B, Le Prell CG, Lopez-Poveda EA, Plack CJ, Schaette R, Verhulst S, Canlon
B. The search for noise-induced cochlear synaptopathy in humans: Mission impossible?. Hearing research.
2019 Jun 1;377:88-103.
6. Liberman MC, Epstein MJ, Cleveland SS, Wang H, Maison SF. Toward a differential diagnosis of hidden
hearing loss in humans. PloS one. 2016 Sep 12;11(9):e0162726.
7. Suthakar K, Liberman MC. Noise Masking in Cochlear Synaptopathy: Auditory Brainstem Response vs.
Auditory Nerve Response in Mouse. Journal of Neurophysiology. 2022 May 18.
8. Clinard CG, Tremblay KL, Krishnan AR. Aging alters the perception and physiological representation of
frequency: evidence from human frequency-following response recordings. Hearing research. 2010 Jun
1;264(1-2):48-55.

9. Presacco A, Simon JZ, Anderson S. Speech-in-noise representation in the aging midbrain and cortex:
Effects of hearing loss. PloS one. 2019 Mar 13;14(3):e0213899.
10. Märcher-Rørsted J, Encina-Llamas G, Dau T, Liberman MC, Wu PZ, Hjortkjær J. Age-related reduction
in frequency-following responses as a potential marker of cochlear neural degeneration. Hearing Research.
2022 Feb 1;414:108411.
11. Liu LF, Palmer AR, Wallace MN. Phase-locked responses to pure tones in the inferior colliculus. Journalof neurophysiology. 2006 Mar;95(3):1926-35.

ABS-0124

Research platforms for interactive hearing devices in the field and in the lab
Hendrik KAYSER, Angelika KOTHE, Volker HOHMANN, Giso GRIMM
Auditory Signal Processing & Hearing Devices and Cluster of Excellence “Hearing4all”
Carl von Ossietzky Universität, 26111 Oldenburg, Germany{hendrik.kayser, angelika.kothe, volker.hohmann, g.grimm}@uol.de

ABSTRACT
To maximize the benefit from hearing device algorithms, the user’s movement behavior is increasingly being
incorporated into signal processing. The development and evaluation of such behavior-driven algorithms imposes increased demands on the computational performance and sensor interfaces of hearing devices, but also
on the ecological validity of test environments as well as experimental paradigms. A portable research platform
allows the integration of signal processing and sensor data on small hardware, but is limited by hardware performance constraints such as computational power, battery life, and sensor interfaces. Network-based distributionof sensors and audio allows parts of the processing to be outsourced to a laboratory environment. This not only
enables performance improvement, but also new types of interactive experiments. As an example, low-delay
interactive audio-visual virtual reality serves as a tool that enables the use of ground truth information and provides reproducibility and scalable complexity of communication tasks. This paper presents multiple researchtools, including a portable research platform that integrates audio signal processing and EEG data acquisition,
and an apparatus to capture interactive communication behavior in a distributed "Wizard-of-Oz" experiment in
virtual environments with network-based distribution of high-quality spatial audio and head motion data.

Keywords: Hearing devices, User behavior, Interactive communication, Virtual reality

1

INTRODUCTION

Maximum benefit from hearing devices is usually achieved through directional filtering [10]. However, as selectivity increases, the interaction between head movement and the user benefit of the devices plays an increasingly
important role [3, 14, 12] (see also [7] for an overview of self-motion in a hearing device context). Also algorithms for estimating the direction of arrival [16] and blind source separation [2, 22] interact with head motion.
The user benefit of hearing devices can be improved by either compensating self-motion [1], but also by estimating the auditory attention based on analysis of movement behavior [8] or EEG [20].
To develop and evaluate hearing devices controlled by human behavior, ‘subject-in-the-loop’ test environments are required, i.e., environments, which take into account the device user and the user’s natural behavior.
For an ecologically valid evaluation of such behavior-controlled devices, the user’s behavior in everyday-lifeneeds to be similar during measurements, at least to the extend that the behavior-controlled hearing device
function reflects function in real-life [18].
This means that experiments may be performed in the field, which comes at the cost of limited reproducibility and test sensitivity, and requires portable hardware. Alternatively, if experiments are performed in alaboratory, much care needs to be taken that the behavior is natural.
In the remainder of this paper, methods and key components to perform experiments with behavior-controlled
algorithms are described, and their potential as well as limitations are discussed.

2

METHODS

2.1 Portable Hearing Laboratory
The Portable Hearing Laboratory [21] is a lightweight ARM Cortex-A8 based processor board with a battery,
professional hearing aid headset with cable connection to a multi-channel audio interface and various connectors

Figure 1. The Portable Hearing Laboratory (left panel) and a realization of an advanced lab, the Gesture Lab
at University of Oldenburg (right panel).
for expansions like additional line-in and line-out connectors for microphones and headsets and wireless network
connection for remote control and data exchange (see Figure 1). The Portable Hearing Laboratory runs an
optimized Linux operating system “Mahalia” and the open Master Hearing Aid (openMHA, [17]) software.
This device is aimed at a large group of researchers, from algorithm developers to audiological researchers.
Because of its flexibility, it can facilitate the development of novel signal processing schemes. As a portabledevice with a battery runtime of several hours, it also enables the evaluation of new algorithms in the field.
When combined with additional sensors such as motion sensors or EEG [5], it can serve as a platform forbehavioral analysis in the field.
2.2 Advanced lab for ecologically valid analysis of interaction between movement behavior and hearing device
benefit
In hearing device research, the findings often differ between experiments performed in the field from experiments performed in the laboratory [4]. This discrepancy can be attributed in part to an oversimplification of
acoustic conditions in the laboratory. Another factor is the lack of ecological validity of human behavior inlaboratory experiments. The goal of an advanced laboratory is to bridge the gap between predictions from field
experiments and laboratory experiments [15] (see Figure 1 for an example). This can be achieved by simulating
the complexity and variability of acoustic conditions of real life also in the laboratory, e.g., by adding reverberation, complex noise fields, adding moving sources, and allowing for self-motion of the participants. The“Toolbox for Acoustic Scene Creation and Rendering” (TASCAR, [9]) is a tool developed for the purpose ofrealizing such an advanced lab. Time-domain low-delay interactive simulation of virtual acoustic environments
can be rendered to loudspeakers or headphones.
In previous studies, it has been shown that when animation accuracy is sufficiently high, subjects exhibit
movement behavior that is close to natural behavior. Hendrikse et al. [13] showed that the simulation ofcommunication-related head movements and speech-controlled lip movements of the virtual characters is necessary to achieve the closest possible natural movement behavior of the test subjects. Hartwig et al. [11] compared
behavior in triadic conversations in real settings and with telepresence by virtual characters. They showed thatwith the animation accuracy found to be sufficient in [13], the effect of virtualization is negligible, while theeffect of interactivity on movement behavior is large.
2.3 Remote distribution of audio and behavioral data
For telepresence reproduction of conversations by virtual characters and simultaneous recording of motion behavior, conventional videoconferencing systems are inadequate due to a large audio signal delay and poor audio
quality, often caused by noise reduction and echo cancellation algorithms. The “ovbox” [6] is a tool that
overcomes these problems. Originally developed during the pandemic for low-delay transmission of audio signals over the Internet for musicians, it supports head tracking and distribution of head movement data to all
connected devices. In this way, the TASCAR-based system enables interactive dynamic 3D audio playback. Extensions allow the distribution of any UDP network-based sensor data, such as head movement, gaze direction,
EEG or data from other bio-physical sensors.

audio data
motion data
subject input

Room 1
headphones

Advanced Lab
motionsensor

motionsensor

remote audio client(ovbox)remote audio client(ovbox)loudspeakers

remote audio
sessionRoom 2visual scene(blender)avataravatarmic

virtual acoustic scene(TASCAR)data
loggingratinginterfacemic

motionsensor

headphones

remote audio client(ovbox)mic

Figure 2. Signal flow of example study on the influence of simulated head movement on immersion, which
combines an advanced lab (left) with remote transmission of audio and behavior data (‘ovbox’).
2.4 Distributed interactive conversations in virtual reality
An example study which uses this technology investigates the effect of simulated head movement behavior on
immersion. This experiment is a kind of Wizard-of-Oz experiment [19], where virtual characters are controlled
by remote experimenters (see Figure 2 for a signal flow). In the experiment, different levels of accuracy ofhead movement reproduction were applied: The most accurate level is the transmission of videos of the remoteexperimenter’s faces. For the next level, measured head movement of the remote experimenters can be applied
to the virtual characters in low-delay real-time. The other levels use the automated head movement as used in[13, 11], or no head movement at all.

3

CONCLUSIONSIn this paper, we present research platforms that enable the development and evaluation of behaviorally controlled hearing devices. The “Portable Hearing Laboratory” is a portable and flexible hardware that runs thehearing aid software openMHA and which provides interfaces for connecting sensors, such as EEG, to be
used for behavioral control of algorithms. For the development and evaluation of such behavior-controlled
algorithms, a test environment is needed that provides sufficient reproducibility and test sensitivity, but also

ecologically valid human behavior, to ensure reliable control. This is achieved with the “Toolbox for Acoustic Scene Creation and Rendering” (TASCAR), which can simulate interactive virtual acoustic environments inlow-delay real-time and send control data to game engines for visualization. Distributed low-delay interactive
communication, including low-delay distribution of biophysical sensor data over the Internet, e.g., for low-delayhead motion animation, is realized based on “ovbox”. This tool allows the laboratory environment to be extended to the Internet, enabling hybrid communication experiments that combine natural communication withthe advantages of a traditional laboratory.
The software of all these tools is open source and is developed and deployed using modern continuous
integration workflows to ensure maximum reliability and reproducibility. Thus, they provide key components forreproducible science based on novel experimental paradigms including natural human communication behavior,
and are striving to increase the level of ecological validity of human behavior in hearing device research.

ACKNOWLEDGEMENTS
Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 352015383– SFB 1330 B1 and under Germany’s Excellence Strategy – EXC 2177/1 – Project ID 390895286.

REFERENCES[1] R. Abdipour, A. Akbari, M. Rahmani, and B. Nasersharif. Binaural source separation based on spatial
cues and maximum likelihood model adaptation. Digital Signal Processing, 36(C):174–183, jan 2015.
[2] J. Anemüller and B. Kollmeier. Adaptive separation of acoustic sources for anechoic conditions: A constrained frequency domain approach. Speech Communication, 39(1-2):79–95, 2003.
[3] W. O. Brimijoin and M. A. Akeroyd. The effects of hearing impairment, age, and hearing aids on theuse of self-motion for determining front/back location. Journal of the American Academy of Audiology,
27(7):588–600, 2016.
[4] M. T. Cord, R. K. Surr, B. E. Walden, and O. Dyrlund. Relationship between laboratory measures of directional advantage and everyday success with directional microphone hearing aids. Journal of the American
Academy of Audiology, 15(5):353–364, 2004.
[5] S. Dasenbrock, S. Blum, S. Debener, V. Hohmann, and H. Kayser. A step towards neuro-steered hearing
aids: Integrated portable setup for time- synchronized acoustic stimuli presentation and eeg recording.
Current Directions in Biomedical Engineering, 7(2):855–858, 2021.
[6] G. Grimm. The ORLANDOviols consort box (ovbox). https://github.com/gisogrimm/ov-client,
2020.
[7] G. Grimm, M. M. Hendrikse, and V. Hohmann. Review of Self-Motion in the Context of Hearing andHearing Device Research. Ear and hearing, 41(Supplement 1):48S–55S, nov 2020.
[8] G. Grimm, H. Kayser, M. M. E. Hendrikse, and V. Hohmann. A gaze-based attention model for spatiallyaware hearing aids. Speech Communication; 13. ITG Symposium, pages 231–235, 2018.
[9] G. Grimm, J. Luberadzka, and V. Hohmann. A Toolbox for Rendering Virtual Acoustic Environments in
the Context of Audiology. Acta Acustica united with Acustica, 105(3):566–578, 2019.
[10] V. Hamacher, J. Chalupper, J. Eggers, E. Fischer, U. Kornagel, H. Puder, and U. Rass. Signal Processingin High-End Hearing Aids: State of the Art, Challenges, and Future Trends. EURASIP Journal on Advancesin Signal Processing, 2005(18):152674, dec 2005.
[11] M. Hartwig, V. Hohmann, and G. Grimm. Speaking with avatars - influence of social interaction on
movement behavior in interactive hearing experiments. In IEEE VR 2021 Workshop: Sonic interactions in
Virtual Environments (SIVE), pages 94–98, 2021.

[12] M. M. E. Hendrikse, G. Grimm, and V. Hohmann. Evaluation of the influence of head movement on
hearing aid algorithm performance using acoustic simulations. Trends in Hearing, 24:1–20, jan 2020.
[13] M. M. E. Hendrikse, G. Llorach, G. Grimm, and V. Hohmann. Influence of visual cues on head andeye movements during listening tasks in multi-talker audiovisual environments with animated characters.
Speech Communication, 101(June 2017):70–84, 2018.
[14] M. M. E. Hendrikse, G. Llorach, G. Grimm, and V. Hohmann. Realistic audiovisual listening environmentsin the lab: analysis of movement behavior and consequences for hearing aids. In Proceedings of the 23rd
International Congress on Acoustics, pages 7616–7622, Aachen, 2019.
[15] V. Hohmann, R. Paluch, M. Krueger, M. Meis, and G. Grimm. The Virtual Reality Lab: Realization andApplication of Virtual Sound Environments. Ear & Hearing, 41(Supplement 1):31S–38S, nov 2020.
[16] H. Kayser and J. Anemüller. A discriminative learning approach to probabilistic acoustic source localization. In 14th International Workshop on Acoustic Signal Enhancement (IWAENC), pages 99–103. IEEE,
2014.
[17] H. Kayser, T. Herzke, P. Maanen, M. Zimmermann, G. Grimm, and V. Hohmann. Open community
platform for hearing aid algorithm research: open Master Hearing Aid (openMHA). SoftwareX, 17:100953,
jan 2022.
[18] G. Keidser, G. Naylor, D. S. Brungart, A. Caduff, J. Campos, S. Carlile, M. G. Carpenter, G. Grimm,
V. Hohmann, I. Holube, S. Launer, T. Lunner, R. Mehra, F. Rapport, M. Slaney, and K. Smeds. The
Quest for Ecological Validity in Hearing Science: What It Is, Why It Matters, and How to Advance It.
Ear & Hearing, 41(Supplement 1):5S–19S, nov 2020.
[19] J. F. Kelley. An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems (TOIS), 2(1):26–41, 1984.
[20] J. A. O’Sullivan, A. J. Power, N. Mesgarani, S. Rajaram, J. J. Foxe, B. G. Shinn-Cunningham, M. Slaney,
S. A. Shamma, and E. C. Lalor. Attentional Selection in a Cocktail Party Environment Can Be Decodedfrom Single-Trial EEG. Cerebral Cortex, 25(7):1697–1706, 2015.
[21] C. Pavlovic, H. Kayser, P. Maanen, T. Herzke, V. Hohmann, S. Prakash, and R. Kasayan. Open portable
platform for hearing aid research. In American AuditorySociety Meeting (AAS), Scottsdale, USA, 2019.
[22] K. Reindl, Y. Zheng, and W. Kellermann. Speech enhancement for binaural hearing aids based on blindsource separation. In Communications, Control and Signal Processing (ISCCSP), 2010 4th International Symposium on, pages 1–6. IEEE, 2010.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0144

Evaluating hearing devices in real life using Ecological Momentary
Assessment
Inga HOLUBE1; Petra VON GABLENZ1; Joerg BITZER1
1

Institute of Hearing Technology and Audiology, Jade University of Applied Sciences, Oldenburg, Germany

ABSTRACT
It is not enough to determine the quality of hearing devices in technical measurements with artificial stimuli.
The subjective evaluation of the devices in real life is a crucial factor in the context of quality testing. A
method to capture the individual’s perspective is Ecological Momentary Assessment (EMA) which allows
for repeated assessments in natural environments as well as data collection of objective environmental
characteristics. We developed an EMA system that includes two near-ear microphones attached to glasses
and wireless data transmission to a smartphone. The smartphone continuously analyzes objective data andmanages surveys which are repeatedly presented to the user of the EMA system. In a field study, first-timeand follow-up hearing aid wearers used the EMA system for 3-4 days before hearing aid fitting and after
hearing aid adaptation. The collected data show considerable inter-individual variability both in the type oflistening events assessed and hearing aid benefit. However, comparisons of subjective assessment before andafter hearing aid uptake are subject to intra-individual changes in the characteristics of attended environments.
Thus, EMA studies provide new insights into user, device, and environment interaction, but at the cost of
careful study design and high participant persistence.
Keywords: Ecological Momentary Assessment, hearing aid, benefit

1. INTRODUCTION
Hearing aids are typically verified with technical measurements in the test box or with probemicrophone measurements and validated with clinical speech tests, sometimes supplemented byadditional laboratory measurements with higher ecological validity (1). To assess the everyday life
hearing aid benefit, anecdotic descriptions or questionnaires are commonly used. Questionnaires often
describe listening situations that need to be assessed retrospectively for the last few weeks. However,
the responses are subject to various bias effects such as memory bias, recency effect, limited context
sensitivity (e.g., 2, 3).
One method to avoid the disadvantages of conventional questionnaire surveying is EMA. The term
is synonymous with experience sampling or ambulatory assessment (4). EMA is defined as “real-timecollection of data about momentary states, collected in natural environments, with multiple repeatedassessments over time” (5). The method may involve subjective assessments as well as objective
parameters about the environment or physical status of the part icipants. Due to the availability andcapabilities with smartphones, EMA is becoming increasingly popular in hearing research (6). This
contribution gives an example of a real-life evaluation of hearing aids using EMA. More detailed
information about the study can be found in (7) and (8).

2. Methods
2.1 EMA system olMEGA
The used EMA system olMEGA consists of a smartphone with Android operating system, twoomnidirectional microphones and a transmission box (9, 10). The two microphones were attached with
heat-shrinkable sleeves to eyeglass temples on each side of the head. They were connected by cables
to a transmission box which could be attached to clothing. The transmission box included a wirelessBluetooth stereo connection to the smartphone. The system is able to process and store acoustic
1

inga.holube@jade-hs.defeatures for a duration of at least 8 hours in stereo without charging. The stored features are time
averaged and sub-sampled to protect privacy, i.e., no audio is stored (11). Software and hardware
specification are available under open source license at https://github.com/ol-MEGA.
In addition to the collection of acoustic features, the smartphone app also included an adaptive
digital questionnaire that allowed single and multiple choices. The study participants were prompted
every 30 min and asked to take a survey. In this contribution, only the survey results are shown. At
the start, the participants entered whether the assessments refer to the momentary situation or asituation within the last 30 min. Then, they selected the current location and the current activity frompredefined categories and indicated whether familiar or unfamiliar speech sources were present. 7point categorical scales were used to assess speech understanding, listening effort, localization ofsounds, loudness, pleasantness, disability, involvement in group communication, difficulty by
conversation partners, importance of hearing well, and mood. The questionnaire is given in detail in
Gablenz et al. (7)2.2 Interventional study
The study was conducted in cooperation with local hearing care professionals. They recruited 24participants who met the German indication criteria for hearing aid uptake. All but three participants
had not previously been fitted with hearing aids. Participants used olMEGA for 3 -4 days to assess
their daily life hearing (preEMA). After that, the hearing care professionals fitted and optimized thehearing aids individually, followed by an adaptation phase. The duration of the adaption phase wasindividually determined and lasted at least three weeks. 16 participants (8 women, 3 follow-up fittings,
median age 67 years) performed another evaluation with olMEGA for 3-4 days after completion of
the adaptation phase (postEMA).

3. Results
Data set
A total of 1705 surveys were completed by the participants, of which 885 were completed before
hearing aid fitting (preEMA) and 820 after the adaptation phase with hearing aids (postEMA). Thus,on average 106 surveys were completed by each participant, but with high variability between
participants in the number and type of situations assessed. At the individual level, moderate agreement
was found between target sound types (natural speech in quiet, natural speech in presence of othersounds, electro-acoustically presented speech, i.e., radio, TV, phone, loudspeaker, non-speech
listening targets) in preEMA and postEMA (8).
3.1

3.2 Hearing aid benefit
With EMA, the individual hearing aid benefit can be analyzed in everyday life. Figure 1 shows as
an example the difference in the assessment of speech understanding (How well or poorly do you
understand?) in the first EMA phase (preEMA) compared to the second EMA phase (postEMA). For
each participant marked with a number in Figure 1, the ratings for speech understanding in therespective EMA phase were averaged. Almost all participants show an improvement in the rating,which, however, varies greatly from individual to individual.
Ratings for listening effort and disability show a comparable hearing aid benefit as those for speech
understanding. Figure 2 shows the mean improvement of these items for each participant in scale units.
The hearing aid benefit ranges from no change (participant 11) to improvements of over 4 (out of 7)
scale units (participants 9 and 13). The three follow-up participants (14-16) show little benefit with
the new hearing aids over the previous hearing aids.

Figure 1 – Assessment of speech understanding on a 7-point categorical scale before hearing aid fitting(preEMA) and after the hearing aid adaptation phase, i.e., with hearing aids (postEMA). Solid lines: newhearing-aid fittings, dashed lines: follow-up fittings, numbers: individual participant.

Figure 2 – Mean subjective improvement by hearing aids (difference between pre- and postEMA) of speech
understanding, listening effort and hearing-related disability for the 16 participants. Participants 14-16
compared the new hearing aids with their previous ones (follow-up fittings).

4. Discussion
This contribution exemplifies the use of EMA in the evaluation of hearing aids in everyday life.
The results reveal large individual differences in hearing aid benefit for different items. No correlation
with pure-tone hearing loss or the duration of the adaptation phase could be demonstrated.
In the context of this study, two aspects should be briefly discussed: Analysis methods for EMA
and the burden on participants. Regarding the analysis, Figure 1 and 2 show mean scores forparticipants, although the ratings were given on a categorical scale, which actually excludes
parametric analysis methods such as mean values. In (7), the non-parametric method nonoverlap of
all pairs (NAP) was proposed to indicate individual hearing aid benefit, and a cumulative link mixed
model (CLMM) was used to analyze the impact of hearing aid uptake on group level. Both methods
are appropriate for data of this type and distribution. H owever, the use of parametric analysis methods
is more widespread and might be feasible under certain cir cumstances (12).
As for the second aspect, EMA places high demands on the participants compared to filling out apaper questionnaire once or twice. This must be considered in the design of the study, e.g., whendeciding upon the duration of the data collection and the frequency of prompts. Participants are asked
to carry extra equipment at all times and to complete repeated assessments. Not all participants arewilling to take on this burden. They have to be specially trained and intensively motivated to maximize
the validity of everyday data.

5. CONCLUSIONSThis contribution presents a brief look at EMA methodology for evaluating hearing aids. EMA
enables an individual analysis of subjective assessments that goes far beyond an interview during avisit to the hearing care professional. This tool can be used to investigate new devices or algori thms
in the research and development phase in everyday life as well as to develop new counseling andfitting procedures for clinical practice.

ACKNOWLEDGEMENTS
The authors thank Alejandro Garavito Arango for preparing the data export routines, Maximilian
Hehl, Florian Schmitt, Kristin Sprenger, and Annäus Wiltfang for performing the laboratory
measurements, the local HA acousticians for their kind cooperation. This work was supported by the
Hearing Industry Research Consortium and the Research Fund of Jade University of Applied Sciences.

REFERENCES
1. Keidser G, Naylor G, Brungart D, Caduff A, Campos J, Carlile S, Carpenter M, Grimm G, Hohmann V,
Holube I, Launer S, Lunner T, Mehra R, Rapport F, Slaney M, Smeds K. The quest for ecological validity
in hearing science: What it is, why it matters, and how to advance it. Ear and Hearing. 2020; 41(Suppl.
1): 5S-19S.
2. Bradburn NM, Rips LJ, Shevell SK. Answering autobiographical questions: The impact of memory andinference on surveys. Science. 1987; 236:157-161.
3. Kahneman D. Objective happiness. In: Kahneman D, Diener E, Schwarz N, editors. Well-Being: The
foundation of hedonic psychology. Russel Sage Foundation; 1999. p. 85-105.
4. Trull TJ, Ebner-Priemer U. The role of ambulatory assessment in psychological science. Curr Dir Psychol
Sci. 2014; 23: 466-470.
5. Stone AA, Shiffman S, Atienza AA. The science of real-time data capture – self-reports in health research.
2007. Oxford University Press.
6. Holube I, von Gablenz P, Bitzer J. Ecological momentary assessment in hearing research: Current state,challenges, and future directions. Ear and Hearing. 2020; 41(Suppl. 1): 79S-90S.
7. von Gablenz P, Kowalk U, Bitzer J, Meis M, Holube I. Individual hearing aid benefit in real life evaluatedusing ecological momentary assessment. Trends in Hearing. 2021; 25:1-18.
8. von Gablenz P, Kowalk U, Bitzer J, Meis M, Holube I. Individual hearing aid benefit: Ecologicalmomentary assessment of hearing abilities. In: Kressner A, Regev J, C.-Dalsgaard J, Tranebjaerg L,
Santurette S, Dau T, editors. Proceedings of the International Symposium on Auditory and AudiologicalResearch. Vol. 7: Auditory Learning in Biological and Artificial Systems. The Danavox Jubilee
Foundation; 2020. p. 213-220.
9. Kowalk U, Kissner S, von Gablenz P, Holube I, Bitzer J. An improved privacy-aware system for objective
and subjective ecological momentary assessment. In: Santurette S, Dau T, C.-Dalsgaard J, Tranebjaerg L,
Andersen T, Poulsen T, editors. Proceedings of the International Symposium on Auditory andAudiological Research. Vol. 6: Adaptive Processes in Hearing. The Danavox Jubilee Foundation; 2017.
p. 25-40B.
10. Kowalk U, Franz S, Groenewold H, Holube I, von Gablenz P, Bitzer J. olMEGA: An open source android
solution for ecological momentary assessment. GMS Z Audiol (Audiol Acoust). 2020; 2:Doc08: 1-9.
11. Bitzer J, Kissner S, Holube I. Privacy-aware acoustic assessments of everyday life. J Audio Eng Soc.
2016; 64:395-404.
12. Oleson JJ, Jones MA, Jorgensen EJ, Wu Y-H. Statistical considerations for analyzing ecologicalmomentary assessment data. Journal of Speech, Language, and Hearing Research. 2022; 65: 344-360.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0168

A platform for calibrated home-based audiometry: from personal
hearing self-assessment to population-based audiology.
Amaury Hazan1; Jonatan Rivilla1; Kamil Budzińsky1; Num Mendez1; Ad Snik1, 2; Jacques Kinsbergen1
1
2

Jacoti BV, Belgium

Department of Biophysics, Donders Institute for Brain Cognition and Behaviour, Radboud University,
Netherlands

ABSTRACT
Purpose: We have developed a platform for home-based hearing assessment and study whether the datacollected can provide audiological insights at the population level. Method: Our platform comprises threecomponents: An ear-level device - such as a smartphone headset or ear bud - which transducers are calibrated,a control logic that is executed on a smartphone and performs the hearing test and a cloud database that stores
anonymized hearing assessment and fitting data for every user. Our platform has been integrated with
consumer technology, e.g., iOS smartphones or True Wireless ear buds. The calibration of the ear-level
hearing device allows hearing thresholds to be expressed in dB HL and represent a valid audiogram. The
control logic comprises a calibrated noise monitoring component to make sure the measured thresholds are
not masked by noise. We aggregate the resulting hearing thresholds and perform an analysis of the dataobtained for thousands of audiograms. Results: Data analysis shows that the resulting thresholds distribution
can be compared with the data of an otologically normal population and follows similar trends with respect
to tested frequency and user demographics. Conclusions: Our home-based assessment platform may provide
a useful way of collecting audiologically relevant data outside of the clinic.
Keywords: Self-assessment, home-based, audiometry

1. INTRODUCTION
Jacoti Hearing Center (referred to as JHC) is an Apple iOS–based application that provides a fully
automated and software-controlled hearing test. JHC is intended as a self-assessment by lay persons
at home, without supervision of a hearing professional. JHC hearing thresholds are stored on the Jacoti
Earcloud (EC) database and anonymized for further study.
The JHC app has been calibrated for use with wired Apple EarPods using the thresholddetermination method [1, 2]. Using a group of normal-hearing subjects, the Reference EquivalentThreshold Sound Pressure Levels (RETSPL) for Apple EarPods have been determined.
In [3], we have shown auditory thresholds determined by JHC self-assessment and clinicalaudiometry did not differ significantly for each tested frequency. Furthermore, test–retest reliability
for 1641 app users was found to be comparable to results published in the literature regarding test–
retest reliability. The work presented did not consider EC data from users who did not repeat a hearing
test. Likewise, we did not analyze the hearing thresholds in terms of demographic data such as age
group or gender.
The JHC sign-up process comprises a questionnaire in which every user self-reports demographic
information such as age, gender, and perceived hearing loss. In the present work, we aim at analyzing
users hearing thresholds stored in EC for an extended number of users and study the distribution ofthese thresholds depending on self-reported age group and gender information.

2. METHOD
We identify self-assessment sessions stored in EC where 11 frequencies were tested for each tested

ear. The tested frequencies are: 125, 250, 500, 750, 1000, 1500, 2000, 3000, 4000, 6000, 8000 Hz.
While the JHC users’ measurements are anonymized prior to further analysis, we keep a coarsegrained summary of each user demographics: each session is labeled with a user decade range (e.g.,
20-30, 30-40 years) and a gender. For each decade range and gender, we compute the average andstandard deviation of hearing thresholds at each tested frequency.
The baseline for comparing the extracted thresholds is given in [4], where a statistical model forcomputing hearing thresholds related to age and gender of an otologically normal population ispresented. For each gender and decade range, the median thresholds corresponding to this gender andan age equal to the decade range middle value are computed. Therefore, for each gender and decade
range, we can compare the distribution of hearing thresholds obtained with JHC with the thresholds
of a normal hearing individual for the same gender and comparable age.

3. RESULTS
We collected and analyzed hearing thresholds from 5496 full audiograms (3935 from male users,
1561 from female users). In Figure 1., we show the hearing thresholds distributions computed fromEC session (left) and compare them with values predicted by ISO 7029 for the same gender andcomparable age (right).

Figure 1. Comparison of median hearing thresholds among gender and age: EC versus ISO-7029:2017.
Earcloud data originates from 5496 audiograms in which 11 frequencies (125, 250, 500, 750, 1000, 1500,
2000, 3000, 4000, 6000, 8000) Hz were tested.

3.1 DISCUSSION
We observe some level of agreement between EC thresholds, and the data predicted by [4]. In
particular, we observe a general trend of higher thresholds with increasing frequency and a similar
distribution of thresholds based on age groups. This observation suggests that both users self-reported
age data and hearing thresholds obtained with JHC self-test tend to be accurate. The comparison
between EC and ISO data also shows discrepancies. The hearing threshold distribution appears to be
shifted to higher thresholds compared to the threshold distribution given in [4]. For young subjects,
ISO data predicts median thresholds around 0 dB HL (by definition). In contrast, because JHC only
tests hearing thresholds comprised between 10 and 85 dB HL, even normal hearing subjects would be
assigned hearing thresholds of 10 dB HL. Additionally, users that decided to perform a self-assessmentof hearing thresholds using JHC may, as a population, have higher hearing thresholds than an
otologically normal population. Further statistical analysis will attempt at identifying possible bias
and confounding factors.

4. CONCLUSIONS
Using gender and age range information, we observe that the distribution of hearing thresholds
extracted from Jacoti home-based hearing assessment platform share similarities with the distributionof hearing threshold in an otologically normal hearing population. Those similarities suggest ourplatform offers potential for conducting population-based hearing studies.

REFERENCES
1. Acoustical Society of America. (2018). Specification for audiometers. American National Standard (ANSI
S3.6 2018).
2. Coninx, F., Zarowski, A., Wack, N., & Offeciers, E (2015). Apple Headset (AHS) calibration.
https://jacoti.com/reports/2015-10-31_antwerp_study_AHS_calibration.pdf
3. Hazan, A., Luberadzka, J., Rivilla, J., Snik, A., Albers, B., Méndez, N., Wack, N., Paytuvi, O., Zarowski,
A., Offeciers, E., & Kinsbergen, J. (2022). Home-Based Audiometry With a Smartphone App: Reliable
Results?.
Americanjournalofaudiology,
1–9.
Advanceonlinepublication.
https://doi.org/10.1044/2022_AJA-21-00191
4. International Organization for Standardization. (2017). Acoustics — Statistical distribution of hearing
thresholds related to age and gender. International Organization Standard (ISO 7029:2017)

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0182

Using Ecological Momentary Assessments (EMA) to understandhearing aid experiences in everyday life in order to facilitate hearing
aid fine-tuning
Karolina SMEDS1; Dina LELIC2; Daniel PARKER31
2
3

ORCA Europe, WS Audiology, Sweden

Scientific Audiology, WS Audiology, Denmark

Applied Audiological Research, WS Audiology, Denmark

ABSTRACT
We have for a long time focused on ensuring successful real-life hearing for hearing aid users. In previous
research, we have learned about everyday signal-to-noise ratios and classified listening tasks that people
perform in real life. During Ecological Momentary Assessments (EMA), this classification has been used to
study people’s auditory reality. This knowledge about the average hearing aid user has been valuable inhearing aid design. However, hearing aid users can have individual and specific needs. New interactive
technology has given hearing aid users a chance to create their own hearing aid programs while in a specificlistening situation. Here we report on a different interactive approach, where a 2-week EMA trial was used
to learn about hearing aid users’ good and difficult listening experiences. While reporting on theseexperiences, hearing aid data were logged. Contrasts in hearing aid data between good and difficult situations
could be identified. For a subset of participants, hearing aids were fine-tuned based on the acquired data. A
week-long follow-up EMA period showed that the data-driven fine tuning was successful for all but one test
participant. In conclusion, moments of hearing aid data combined with subjective reports of real-life listening
experiences offer valuable insights for a data-driven hearing aid fine-tuning process.
Keywords: EMA, Hearing Aids, Data-Driven Fine-Tuning

1

INTRODUCTION

Ecological Momentary Assessment (EMA) has become an increasingly popular research tool inhearing research. EMA involves repeated sampling of test participants’ experiences in their naturalenvironment. The method aims to minimize recall bias and maximize ecological validity (1). By using
EMA in hearing research, we have learned about people’s auditory reality, perceived hearing difficulties, and hearing device use and benefit (2).
1.1 Auditory reality and hearing aid preference
Our own EMA work has focused on studying people’s auditory reality and understanding hearing
aid preference in everyday listening situations. In a literature study, Wolters et al. (3) investigatedcommon listening situations that people encounter, and in the Common Sound Scenarios (CoSS)
framework these situations were categorized into seven “task categories”. In subsequent EMA studies,
we have learned about how often these situations occur in peoples’ everyday life, how important they
are to hear well in, and how difficult it is to hear in the situations (4).
We have also used EMA to evaluate people’s preferred hearing aid settings in everyday life using
direct paired comparisons (4). The advantage of using EMA was that evaluations were made in theparticipants’ everyday listening situations. Two hearing aid settings were implemented in two hearing
aid programs, and the participants indicated their preferred program in the moment when they were
prompted to report. The EMA questionnaire was implemented on a smartphone using Google forms.
Switching between hearing aid programs was done using a traditional remote control, and there wasno connection between the smartphone and the hearing aids.
Later, we developed a proprietary smartphone app that could administer the EMA protocol. With
the new app, the smartphone and the hearing aids were connected. In a subsequent study, the app wasevaluated (5) using similar research questions as in our previous work. For the auditory reality
investigations, the new app provided information about the environment, such as sound levels(registered by the hearing aid microphones) and sound classes (as detected by the hearing aid’s soundclassification system). For the hearing aid program preference, the new app could provide randomized
and blinded paired comparisons. From this and similar studies, we have acquired knowledge about
the average hearing aid user’s auditory reality and hearing aid setting preference. However, hearing
aid users can have individual and specific needs. Below, we describe the traditional hearing aid fitting
process and how individual needs are usually dealt with.
1.2 Traditional hearing aid fitting
When fitting hearing aids, the end goal is that each client will get hearing aids with the settings
that are the most appropriate. Traditionally, a combination of methods is used to reach this goal. Earlyin the process, a theoretical, prescriptive approach is used. A prescription makes it possible to calculate
initial hearing aid settings based on characteristics related to the client, most commonly the audiogram.
The audiogram data are combined with psychoacoustical theories and large data sets to calculate theinitial hearing aid settings, which should be appropriate for an average hearing aid user with the statedaudiogram. Subsequently, the fitting process enters an empirical phase, where the client uses thehearing aids in everyday life. During this period, the client likely experiences situations when the
hearing aids function well and others where they do not. When the client comes back to the hearing
care professional (HCP), there is a focus on the situations where the hearing aids did not work well,and the hearing aids are fine-tuned to alleviate the problems experienced in these situations.
There are several difficulties associated with this fine-tuning process. First, clients need toremember the situations in which there were difficulties during the home-trial period. Next, they need
to be able to describe both the situation and the perceived problem. Then, the HCP needs to understandboth the description of the listening environment and the description of the problem and transform thedescribed difficulty into actionable fine-tuning adjustments. However, it is not always easy for theclient to accurately describe real-life moments of hearing difficulties or for the HCP to transfer theseverbal descriptions into meaningful hearing aid adjustments (6).
From modern hearing aids, we can retrieve both data that describe the listening environment anddata that show the hearing aid settings in that environment. Ideally, these data could help the HCP in
the fine-tuning process. However, to be helpful, these hearing aid data need to be tied to subjective
evaluations of the situations.
1.3 Good and difficult listening experiences
Lelic et al. (7) found a way of using EMA to tie hearing aid data to subjective experiences. They
investigated if contrasts between good and difficult listening experiences could be identified in
objective hearing aid data. Sixteen participants were instructed to report, using the provided EMA app,when they experienced a listening situation that was either difficult or exceptionally good. When theparticipants started an EMA survey, they first indicated if it was a difficult or good situation bypressing on a sad or happy emoji. Then they described the listening environment and activity in theirown words. If a difficult situation was indicated, the participants were also asked to describe why it
was difficult.
While the participants answered the questionnaire, hearing aid data were logged at a time
resolution of two seconds. The collected data included information about the acoustic environment,such as the sound level, the hearing aid detected sound classes, as well as presence of speech andtransient sounds. Further, the app logged information about the hearing aid settings, such as the gainin each compression channel, hearing aid program, and whether certain adaptive features were active
or not.
On a group level, the results showed that difficult listening situations were characterized by higher
sound levels and more frequent activation of high-level noise reduction and directional microphones.
The hearing aids also classified the situation as Party more frequently for the difficult situations.
However, for 14 out of the 16 participants, Lelic et al. found individual contrasts that deviated fromthe group data for at least one hearing aid parameter. The authors suggested that this type of hearing
aid data, together with the subjective evaluations, could be used when fine-tuning hearing aids. The
rest of this paper will report on a pilot study, where a subset of the participants in the Lelic et al. study
received hearing aid fine-tuning based on their momentary reports.

2

METHODS

Eight participants from the original study had their research hearing aids fine-tuned based on theirreports and the collected hearing aid data. The fine-tuning was done immediately after the initial twoweek trial period. After the fine-tuning session, the participants used the hearing aids for another week,where they again reported difficult and good situations using the EMA app. After the one-week period,
an exit interview was conducted to understand experiences with the fine-tuned hearing aids.
2.1 Participants
From the original study, all participants who had at least as many difficult reports as good ones
were included (4 people). In addition, participants who had a larger number of good than difficult
reports were included if they had more than 5 reports of difficult situations (4 people). There were 5
females and 3 males who completed the current study, and their average age was 65 years (ranging
from 55 to74 years).
2.2 Hearing aids
The participants were fitted with Widex EVOKE 440 receiver-in-canal hearing aids. For details
related to the baseline fitting, see reference (7). Central to the study, a MATLAB script was used tomodify the hearing aids by enabling an extended logging functionality, called the Eventlog.
2.3 EMA app setup
The EMA app setup is described previously (7). The logged hearing aid parameters included:• Sound level in 15 frequency bands with center frequencies 125, 250, 350, 500, 630, 800, 1000,
1250, 1600, 2000, 2500, 3200, 4000, 6000, and 8000 Hz• Speech detection in the input signal (binary variable – yes or no)• Transient activity in the input signal (binary variable – yes or no)• Detected sound class corresponding to those available in the test HAs (8)• Insertion gains in 15 frequency bands (listed above)• High-level noise reduction activity (binary variable – yes or no)• Low-level noise reduction activity (binary variable – yes or no)• Wind noise reduction activity (binary variable – yes or no)Once the participant started the study app, the Eventlog was simultaneously initiated and recorded
one sample of hearing aid data every two seconds for a maximum of 70 samples. Additionally, volume
control use during the entire hearing aid wear time was recorded, although these data were not published in the original work of Lelic et al.
2.4 Fine-tuning strategy
An experienced audiologist (the third author) performed the fine-tunings. Before each fine-tuning
session, the participant’s field data were analyzed, and potential fine-tuning adjustments wereconsidered. When the participant arrived, the audiologist showed the summary graphs of the collectedhearing aid data together with subjective reports of difficult situations. The participant could verbally
supplement the descriptions retrieved through the app. The audiologist then performed the fine-tuning
based on the data-driven dialogue. In most cases, the fine-tuned hearing aid settings were implemented
in a new hearing aid program to not disrupt the good listening experiences. This was particularly
important for participants who had submitted a substantial number of good reports.

3

RESULTSThe following fine-tuning actions were the most common: adding pre-defined hearing aid programs
available in the fitting software, creating a personalized hearing aid program , or adjusting the overall
gain or the gain in a particular frequency range.
3.1 Group results
Seven of the participants thought the fine-tuning adjustments were helpful. One person (participant
8, Table 1) was not content with the fine-tuned hearing aids. The number of positive and negative
reports as well as participant comments from the exit interview are presented in Table 1. During thesecond field-trial period, some participants focused mainly on the difficult situations, which had been
the focus of the fine-tuning session.

Table 1 – Summary of the number of submitted reports by each participant during the first field trial period, the fine-tuning actions, the number of reports during the second field-trial period andcomments given after the second field-trial period.
Partic- Before
ipant Fine-Tuning
6

Good
reports25

Difficultreports
7

Fine-tuning actions

After Fine-Tuning

Party program (P2)Good
reports8

Difficultreports5

Social program (P3)

13

61

10

Microphone set to omnidirectional mode (P2)

52

2

15

25

10

Comfort program (P2)

16

14

16

14

12

-5 dB (250-2500 Hz) (P2)

12

1

0

2

1

5

5

2

3

8

Comfort program (P3)

4

2

17

+5 dB overall gain
Transport program (P2)Social program (P3)

10

3

4

+5 dB for soft sounds(1000-2500 Hz) and -3
dB (350-500Hz)
Urban program (P2)

14

4

5

Party program (P3)-5 dB overall (P2)
-3 dB for loud sounds
(P3)

8

7

10

-3 dB for loud andnormal sounds (8002000 Hz) (P4)
Comfort program (P2)Party program (P3)

Comments after fine-tuning
Thinks it works better than before. It was fun trying
the 2 programs. With one program he could hearmore of what was going on around him and withother the person in front of him. Selected program
based on intention. In many cases he could use theUniversal program, but in other occasions he chose to
use the special programs. There were no situations
where he had a lot of difficulty after he tried one of
the special programs.
Has been testing mostly the omni program, which hasreally improved her speech understanding,
localization, and separation of sounds. Even in danceclasses, with loud music, she could hear the instructor
with the omni program, which she could not do with
the Universal program.
Her hearing experience has improved. She really likes
the Comfort program, uses it 1-2 hours a day, mainly
in situations with many people. It helps with both
comfort and speech understanding. She has beenturning down the volume less often and is reporting
good situations that were previously uncomfortably
loud. She still has some speech understanding
difficulties, mostly where there is poor acoustics.
Switched to Comfort program in the kitchen to reducenoise. Switched back to program 2 or Universal so she
could understand her husband. It has gone muchbetter this week since the adjustments. She has used
all 3 programs, depending on the situation. She used
Comfort in the store and on the golf course, which
made it much more comfortable. Has notexperienced the same speech understanding
difficulties as during the first 2 weeks.
He was overall happy with the fine-tuning. Overall
gain was perfect. Liked the Transport and Social
programs when they were appropriate. He tried theTransport program in a train. He tried the Social
program in shopping mall, supermarket and at aparty/ wedding event where it helped him.
She confirmed that the adjustments in gain improved
her speech understanding in meeting situations, andon Skype. Substantially better but not optimal. Also,
the reduction of gain in the low frequencies reduced
the “reverberant” sound quality that she wasexperiencing a lot in the beginning. The special
programs did not help much in difficult situations.
She reported good situations when watching TV andunderstanding speech in groups – situations thatwere difficult before. She reported two difficult
situations, both because the sound level was toohigh. As observed in objective data, she had not tried
to decrease the volume in these two situations.

The fine-tuning was not successful. The Party
program only helped when the sound level was notvery high. The Comfort program dampened all sounds
too much. This participant had many difficulties invery loud situations because it was too loud.

3.2 Fine-tuning examples
Below, two fine-tuning examples are given. A large amount of hearing aid data was collected during
each EMA report. Here, examples of data that informed the fine-tuning process are presented. The
participants’ identifying numbers are the same as in reference (7). Hence, it is possible to look at Table
2 in that publication to see individual hearing aid data.
3.2.1 Participant 13
Participant 13 (female, 61 years) submitted 71 reports during the first field-trial period. Ten werein difficult situations. She described the difficult situations in the following way: 1. The sound is
muffled, 2. Surrounding sounds from different directions are a “mish-mashed”, 3. When there are
more inputs from different directions at one time, it is difficult to separate words.
When looking at her hearing aid data, we could see that she had reported difficult situations when
the directional microphone was particularly active (Figure 1). Based on this and her subjective reports,
the suggested solution was to provide the participant with an extra program where the microphone
was locked in omni-directional mode.
In the week following the fine-tuning, the participant reported 54 situations. Only two of thesewere difficult. One was a phone conversation at the office, which was perceived as very loud, and one
was in the back seat of a car, were it was difficult to hear the conversation from the front of the car.
After the last field-trial week, she reported that she had mainly used the second program where themicrophone was in omni-directional mode. This was confirmed by the objective hearing aid data,
which showed that program 2 was used 70% of the time. This program had improved her speech
understanding, localization, and separation of sounds.

Figure 1 – Hearing aid data for participant 13 showing the directional microphone activity ingood and difficult situations.
3.2.2 Participant 4
Participant 4 (male, 55 year) submitted 19 reports during the first field-trial period. Only two wereof exceptionally good situations and 17 situations were reported to be difficult. The subjective reportsof hearing difficulties included: 1. When there is traffic noise, 2. When a soft-spoken person talks,
and 3. When many people are around, with multiple conversations outside and in living rooms.
His hearing aid data revealed that he particularly struggled when the detected hearing aid soundclass was Urban or Traffic (Figure 2). The Urban sound class is flagged in situations with varied anddiverse sounds and the Traffic sound class is flagged in situations with low-frequency noise. This is
in line with the difficult environments that the participant described. Further, he had increased thevolume control in some situations and for the most part reported difficult situations when the volume
control was at the default (Figure 3), which indicated that he might need more amplification. The
suggested solution was to increase the overall gain by 5 dB. In addition, two new programs were
provided, Transport and Social. The Transport program is optimized for situations with low-frequency
sounds, such as car noise. The Social program is optimized for situations characterized by multiple
conversations at the same time.

Figure 2 – Hearing aid data from participant 4 showing the sound classes which were flagged
while the participant was reporting. For all samples during one reported situation (judged to beeither good or difficult), the percentage of samples (vertical axis) in the different sound classes(horizontal axis) was calculated. For the difficult situations, the bars represent the mean of thesepercentages, and the error bars the standard error of the mean. For the good reports, the individual
percentages are depicted by the diamonds. The numbers next to the diamonds indicate which of thetwo good reports the data belong to. The Quiet sound class was mainly detected during the firstreport, and the Social sound class was mainly detected during the second report.

Figure 3 – Hearing aid data for participant 4 showing the participant’s use of the volume control
during the two-week trial. In the top row, the registered sound levels are shown and below is theself-selected volume control setting. The time stamps of his 17 reported difficult situations aremarked on top in red, and the time stamps of the two reported good situations are marked below thelevel graph in green.
In the week following the fine-tuning, the participant reported only two difficult situations: conversation over a mobile phone with wind noise present, and a one-to-one conversation with wind noisepresent. Hence, both situations related to difficulties in wind noise and were different from the difficulties he expressed during the first two weeks of the trial.
After the last field-trial week, he commented that he had used the Transport program when he was
outdoors. Here the program had helped a lot. Speech was heard more clearly, and the program did not
muffle the sounds. He visited a big shopping center with lots of people and found that the Social
program gave a clear improvement in speech understanding.

4

DISCUSSION

The work of Lelic et al. (6) showed that by using EMA and an Eventlog, which logged hearing aid
parameters, subjective evaluations of good and difficult listening situations could be meaningfully
linked to data collected from the hearing aid at the time of reporting. It was suggested that thisinformation could be useful for hearing aid fine-tuning. In the current paper, we have presented a pilot
study where a subset of the participants from the first study had the research hearing aids fine-tuned
based on the momentary data they submitted.
Generally, the data-driven fine-tuning process worked well. Seven out of the eight participants
were happy with the fine-tuned hearing aid settings. The one participant who was not satisfied would
likely have benefitted from at least one additional follow-up session to improve his hearing aid fit.
However, iterative fine-tuning until reaching the most appropriate hearing aid settings was beyond
the scope of this study. The goal was to investigate whether hearing aid data coupled with subjective
reports had the potential to be used for fine-tuning.
Most fine-tuning adjustments were implemented in an extra program. The use of several programs
seemed to work well. It is probably beneficial not to introduce several programs to first-time hearing
aid users at the first fitting visit since it can be difficult to know when to use the various programs.
However, when a certain situation is described as difficult after a home-trial period and a special
program is created for that situation, it becomes easier to understand when to use the program.
The now suggested fine-tuning strategy is thought to be used by HCPs when fine-tuning theirclients’ hearing aids. In modern hearing aids, there is often a possib ility for clients to fine-tune their
hearing aids themselves by, for instance, creating own programs for certain situations using machinelearning algorithms and paired comparisons (9). The two strategies are complementary and might bepreferred by different client groups.
In addition to the difficult situations, the participants in this study were asked to report thesituations they experienced as exceptionally good. The purpose of this design was to better understandhow the hearing aids could be personalized to improve the difficult listening situations while notcompromising on the good ones. However, it was inspiring that many participants reported such ahigh number of exceptionally good listening experiences. Encouraging hearing aid users to focus also
on good listening experiences may strengthen the realization that hearing aids positively impact
everyday life. This, in turn, can lead to increased satisfaction with and appreciation of the hearing
aids (10, 11).

5

CONCLUSIONSIn conclusion, EMA data constituting self-reported subjective experiences and objective hearing
aid data offered valuable insights that could be used for data-driven fine-tuning.

ACKNOWLEDGEMENTS
We want to thank Jakob Nielsen who enabled this study by setting up the logging functionality. Jakob
Nielsen, Ole Hau and Jesper Theill contributed to the interpretation of the logged hearing aid data.
Christina Johansson Folgeroe helped define the fine-tuning guidelines. Finally, we want to thank thetest participants who contributed to this research.

REFERENCES
1. Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annu Rev Clin Psychol.
2008;4:1-32.
2. Holube I, von Gablenz P, Bitzer J. Ecological Momentary Assessment in Hearing Research: Current State,
Challenges, and Future Directions. Ear Hear. 2020;41 Suppl 1:79S-90S.
3. Wolters F, Smeds K, Schmidt E, Christensen EK, Norup C. Common Sound Scenarios: A context-driven
categorization of everyday sound environments for application in hearing-device research. J Am Acad
Audiol. 2016;27(7):527-40.
4. Smeds K, Gotowiec S, Wolters F, Herrlin P, Larsson J, Dahlquist M. Selecting scenarios for hearingrelated laboratory testing. Ear Hear. 2020;41 Suppl 1:20S-30S.
5. Jensen NS, Hau O, Lelic D, Herrlin P, Wolters F, Smeds K. Evaluation of auditory reality and hearing aids
using an Ecological Momentary Assessment (EMA) approach. In: Ochmann M, Vorländer M, Fels J,
editors. 23rd International Congress on Acoustics (ICA); September 9-13, 2019; Aachen, Germany:

Berlin, Germany : Deutsche Gesellschaft für Akustik; 2019.
6. Jenstad LM, Van Tasell DJ, Ewert C. Hearing aid troubleshooting based on patients' descriptions. Journalof the American Academy of Audiology. 2003;14(07):347-60.
7. Lelic D, Nielsen J, Parker D, Marchman Ronne F. Critical hearing experiences manifest differently acrossindividuals: insights from hearing aid data captured in real-life moments. Int J Audiol. 2021:1-9.
8. Kuk F, Schmidt E, Jessen AH, Sonne M. New technology for effortless hearing: A “Unique” perspective.
Hearing Review. 2015;22(11):32-6.
9. Balling LW, Mølgaard LL, Townend O, Nielsen JBB, editors. The Collaboration between Hearing Aid
Users and Artificial Intelligence to Optimize Sound. Seminars in Hearing; 2021: Thieme Medical
Publishers, Inc.
10. Saunders GH, Dillard LK, Frederick MT, Silverman SC. Examining the Utility of Photovoice as an
Audiological Counseling Tool. J Am Acad Audiol. 2019;30(5):406-16.
11. Lelic D, Parker D, Herrlin P, Wolters F, Smeds K. Focusing on positive listening experiences improves
hearing aid satisfaction in experienced hearing aid users. Journal of Hearing Science. 2022;12(1).

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0442

Relationships between hearing loss and hearing aid usage in realworld
Niels H. PONTOPPIDAN1; Jeppe H. CHRISTENSEN1; Louisa MURDIN2; Athanasios BIBAS3;
Dimitris KIDIKIS3; Giorgos DRITSAKIS4; Doris-Eva BAMIOU4
1
2

Eriksholm Research Centre, Oticon A/S, Denmark

Ear Noise and Throat Services, Guy’s and St. Thomas’ NHS Foundation Trust, London, United Kingdom
3

Dept. of Otolaryngology, National & Kapodistrian University of Athens, Greece
4

Ear Institute, University College London, United Kingdom

ABSTRACT
Many hearing aids are fitted with multiple programs that the user can choose between in different situations.
In the H2020 EVOTION project hundreds of hearing aid users were fitted according to their audiogram with
Oticon VAC rationale with four adaptive programs that differed in the noise management profile. The
programs differed on how much the noise was attenuated and the threshold at which the device started toremove noise. The hearing aids also transmitted continuous data about the sound environment and theoperation of the hearing aid to a dedicated app on a smartphone. The participants had a wide range of hearing
losses and was recruited from amongst new and experienced hearing aid users in six different clinics in UK,
Greece, and Denmark.
The data from the hearing aid users was collected for up to a year to investigate relationships between clinicalfactors and usage patterns. The data enables a detailed investigation of the complexity of sound environments
throughout the day, relation between complexity as function of time to the clinical factors including andbeyond the audiogram.
Keywords: Real world hearing aids usage, clinical data, hearing aid logging data

1. INTRODUCTION
This paper covers the early exploration and modeling of a joint analysis of continuous hearing aid
data, fitting data, and clinical data for a subset of the participants of the EVOTION project. The
EVOTION project had the objective to develop tools and demonstrations of big data supporting public
hearing health policies. Among key questions for public hearing health policies is the ability to predict
the effect of hearing health care with respect to both the standard intervention and variations of thestandard intervention. In this paper daily use of the hearing aids acts as the proxy for quantifying theeffect of the standard intervention, and it is analyzed as function of both contextual data characterizing
every hearing day life of the participants and clinical data describing the participants.

2. PARTICIPANTS
The participants in came from hearing clinics in United Kingdom, Greece, and Denmark that all
followed the same clinical protocol(1). However, this paper only deals with the data from the Danish
participants. This imposes some limitations on the data analysis, as the number of participants is only
26 out of the 400, and moreover, the participants at the Danish site also have, narrower range ofaudiograms, more experience in using hearing aids, more experience from participating in hearing
research. Thus, the present results must be interpreted with these limitations in mind.
Upon enrolling in the study participants in the project was 2 initial visits for collecting the clinicaldata, fitting the hearing aids, and instructing the participants in the use of the hearing aids and the1

npon@eriksholm.com

companion mobile phone app. Participants was fitted according to their individual audiogram
following Oticon’s proprietary fitting rational with the Oticon EVOTION hearing aid prototype
specially developed for the EVOTION project. Real ear measures were performed to verify the fit.
All participants were fitted with 4 programs that provided increasing levels of help from the spatial
noise reduction system in the devices. The default program was the default setting of the fitting
software, and 2 programs provided more help than the default program, and fourth program provided
less help than the default. Upon completing the two initial visits, the participants used the hearing
aids and companion mobile app for 9-12 months in their daily lives.

3. Data
One type of data available for this analysis comes from clinical records describing the participants
and their hearing. This includes age, audiogram, previous use of hearing aids, Glasgow Hearing Aid
Benefits Profile (GHABP)(2), Montreal Cognitive Assessment (MoCA)(3), Hospital Anxiety andDepression (HADS)(4), etc. The other type of the data comes from continuous logging data collectedfrom the participants’ hearing aids during use. This data contains characteristics of the encounteredsound environments, e.g. sound pressure level, noise floor, signal to noise ratio as estimated inside
the hearing aid, the data also provides a four class classification of the sound environment and selected
program. Sample hearing aid data is available for download (5) and described in the companion datareport(6). When the participant had their hearing aids connected to their smartphone running thecompanion app the detailed HA logging data was collected once every minute from both hearing aids.
All clinical and hearing aid logging data was associated with a randomized identifier such that clinicaldata and hearing logging data can be jointly analyzed.
The present data set which is a subset of the full data consists of data from 26 participants providing
6 MIO logging points from the hearing aids. The counting the unique combinations of participants
and dates shows that more than 3000 unique user-days was logged.

4. PREPROCESSING
When combining the responses from several questionnaires across a number of participants alongwith other descriptors like age, correlation among the variables must be accounted for before the data
can be analyzed with linear mixed model methods. With the limited number of participants in thesample, GHABP and MoCA, is represented by the aggregated total score, and HADS represented by
one total score for the anxiety and one for depression. The second step in the preprocessing aims toaccount for correlation between Age, GHABP, MoCA, and HADS and reduce the dimensionality with
Principal Components Analysis (named USER_DATA_PCA).
The data from the hearing aids is first aggregated to compute the daily duration of use for everyone
for each day with sufficient data along with daily averages of the contextual parameters.
Correlation is also an issue with the aggregated logging data from the hearing aids where thecorrelation can originate from the design of the estimators and from the inherent correlations between
the estimators caused by the sound environments, and thus in the modeling this space is reduced(named dailyuse_PCA).
4.1 USER DATA
Five parameters: Age, GHABP, MoCA, HADS (Anxiety), and HADS (Depression) was
individually scaled to remove mean and normalize the variation. The 5 eigenvalues out of the PCA
followed a linear trend ranging from 6.8 to 2.7 thus the dimensionality was not reduced however, thesubsequent modeling can happen without correlation between the variables. The clinical records also
store information about participants hearing status, in the modeling here, we include the presence oftinnitus, previous experience with hearing aids, and the audiogram summarized by the WHO
classification(7).
4.2 DAILY USE DATA
The hearing aids store 20 contextual parameters on the smartphone every minute. For estimating
the duration of daily use, we assume that each datapoint accounts for one minute of usage whilst also
accounting for overlaps between data from left and right hearing aids. Thus, the estimate of the daily
usage is almost a count of datapoints stored for everyone each day. The contextual parameters are:Sound Pressure Level, Signal-to-noise ratio, Noise floor, Modulation envelope (supplement to Sound

Pressure Level), and Modulation Index (supplement to Signal-to-noise ratio) in the full frequency
range 0-10 kHz, and three frequency ranges from 0-1.4 kHz, 1.4-4.1 kHz, and 4.1-10 kHz. Since thisis just a preliminary analysis the number of components is fixed to 5 for the subsequent modeling,
and inspection of the eigenvalues shows that this accounts for 97% of the variability in the input space

5. LINEAR MIXED MODELSThe linear mixed model analysis is performed with R in Rstudio using the LME4 package.
Model: scale(dailyuse) ~ factor(TINNITUS_BOTH) + factor(PREVIOUS_AID) +USER_DATA_PCA1 + USER_DATA_PCA2 + USER_DATA_PCA3 +USER_DATA_PCA4 + USER_DATA_PCA5 + dailyuse_PCA1 + dailyuse_PCA2 +dailyuse_PCA3 + dailyuse_PCA4 + dailyuse_PCA5 + (1 | pta4_who)
The results of the mixed linear modeling is summarized in Table 1 and Table 2 that indicate that
previous experience with hearing aids, 3 out of 5 USER_DATA principal components, 2 of 5 dailyuse
principal components, and the degree of hearing loss a have a significant impact on the daily use withsignificance lower than 0.01. The model also outputs the intercepts for the categories of the hearing
loss according to WHO(7), estimated as 0.11 for Normal, -0.19 for Slight, 0.03 for Mild, and 0.04 forModerate.
Table 1 - Fixed effects
Estimate

Std. Errordft value

Pr(>|t|)(Intercept)-7.518e-01

1.424e-01

3.742e+01-5.2795.77e-06 ***TINNITUS BOTH Ears (Yes)-1.980e-02

5.468e-02

1.082e+03-0.3620.717376

PREVIOUS AID EITHER (Yes)

8.375e-01

1.258e-01

3.513e+03

6.657

3.22e-11 ***USER_DATA_PCA1-5.591e-02

8.732e-02

3.310e+03-0.6400.521994

USER_DATA_PCA2-3.192e-01

1.057e-01

6.502e+02-3.0200.002627 **USER_DATA_PCA3

1.956e-03

9.019e-02

3.527e+03

0.022

0.982701

USER_DATA_PCA4-5.896e-01

9.860e-02

1.812e+03-5.9802.68e-09 ***USER_DATA_PCA5

8.751e-01

1.019e-01

1.054e+03

8.590< 2e-16 ***dailyuse_PCA1-1.634e+01

1.430e+00

3.488e+03

11.427< 2e-16 ***dailyuse_PCA2

1.269e+00

1.397e+00

3.522e+03

0.908

0.364063

dailyuse_PCA3-1.466e+00

1.444e+00

3.529e+03-1.0150.310073

dailyuse_PCA4-4.510e+00

1.903e+00

3.520e+03-2.3690.017873 *dailyuse_PCA5

5.029e+00

1.411e+00

3.465e+03

3.564

0.000371 ***--Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Table 2 - Random effects
Df

Pr(>Chisq)(1 | pta4_who)
14-4900.4
9828.833.97
1--Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

5.598e-09 ***<none>

Npar

logLikAIC

15-4883.4

9796.8LRT

Since the analysis is not done on the full data set, and since there might be some biases related tothe participants in the current sample further interpretation of the data is not performed. However, thepreliminary analysis suggests that the daily use is a function of the contextual parameters logged fromthe hearing aids, data from hearing aid fitting process (the WHO hearing loss type), previous
experience with hearing aids, and age, and finally clinical data which describes the individual
according to GHABP, MoCA, and HADS questionnaires.

6. FINAL REMARKSAs this analysis only covers a subset of our joint data and with the assumption that it is not covering
the full data set, we have not drawn firm conclusions from the analysis. Instead, this paper serves asa tangible example of joint analysis of continuous hearing aid logging data and clinical data
The preliminary analysis indicates that the daily use of hearing aids can be predicted based onclinical user characteristics measured with standard questionnaires like GHABP, MoCA, and HADS
combined with continuous contextual data logged from hearing aids, and fitting data. Further work
will combine the data from more clinical sites to provide a full demonstration of the opportunities
arising from combing continuous, fitting, and clinical data.

ACKNOWLEDGEMENTS
The authors would like to acknowledge contributions from all partners of the EVOTION project.
The work was partially funded by European Commission through Grant 727521 EVOTION – Big
Data Supporting Public Hearing Health Policies.

REFERENCES
1. Dritsakis G, Kikidis D, Koloutsou N, Murdin L, Bibas A, Ploumidou K, et al. Clinical validation of apublic health policy-making platform for hearing loss (EVOTION): Protocol for a big data study. BMJ
Open. 2018;8(2):e020978.
2. Gatehouse S. Glasgow Hearing Aid Benefit Profile: Derivation and validation of a client-centeredoutcome measure for hearing aid services. J Am Acad Audiol. 1999;10(02):80–103.
3. Nasreddine ZS, Phillips NA, Bédirian V, Charbonneau S, Whitehead V, Collin I, et al. The Montreal
Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment. J Am Geriatr Soc.
2005;53(4):695–9.
4. Zigmond AS, Snaith RP. The hospital anxiety and depression scale. Acta Psychiatr Scand.
1983;67(6):361–70.
5. Christensen JH, Pontoppidan N, Rossing R, Anisetti M, Bamiou DE, Spanoudakis G, et al. Fully synthetic
longitudinal real-world data from hearing aid wearers for public health policy modeling [Internet].
Zenodo; 2019 [cited 2019 May 6]. Available from: https://zenodo.org/record/2668210
6. Christensen JH, Pontoppidan NH, Rossing R, Anisetti M, Bamiou DE, Spanoudakis G, et al. Fully
synthetic longitudinal real-world data from hearing aid wearers for public health policy modeling. Front
Neurosci. 2019;13:850.
7. Olusanya BO, Davis AC, Hoffman HJ. Hearing loss grades and the International classification offunctioning, disability and health. Bull World Health Organ. 2019;97(10):725.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0514

Spatial characteristics of auditory spatial attention
in a horizontal plane
Shuichi SAKAMOTO1; Satoshi TOMIMATSU2; Tetsuaki KAWASE3
1,2

Research Institute of Electrical Communication andGraduate School of Information Sciences, Tohoku University, Japan
3

Graduate School of Biomedical Engineering, Tohoku University, Japan

ABSTRACT
Humans can extract specific sounds from spatially distributed distractors. However, hearing impaired
listeners experience challenges in this regard. Thus, it is important to apply the underlying mechanism of thiseffect to assistive hearing devices. One of the most important factors in this mechanism is auditory selective
attention. This study investigated the characteristics of auditory selective attention in a spatial domain. During
the experiment, four-mora word sequences uttered by a male speaker were presented by 12 circularly
arranged loudspeakers. Furthermore, other four-mora words were uttered by a female speaker, replacing someof the four-mora words presented by the 12 loudspeakers randomly. The listeners were asked to direct theirattention to one of the loudspeakers and respond by identifying whether the word uttered by the female
speaker was the target word. The results revealed that the correct identification response rate was higher when
the word uttered by the female speaker was presented from the direction of increased auditory attention.
Moreover, the shape of the spatial window of auditory selective attention was broader in the sideways thanin the frontal direction. These results can be used effectively to determine the shape of the spatial window inthe speech enhancement algorithm.
Keywords: Auditory spatial attention, cocktail party effect, auditory scene analysis

1. Introduction
People are exposed to sounds from their surroundings daily. Furthermore, people can extract a
specific sound from the spatially distributed distractor sounds despite the complexity of the situation.
This phenomenon is known as the cocktail party effect [1]. Several studies on the cocktail party effecthave investigated how people can perceptually extract a desired sound from other sounds [2-5].
However, hearing impaired listeners experience immense challenges extracting target sounds fromother distractors. Thus, it is important to understand the mechanism underlying this phenomenon todevelop advanced hearing assistive devices. One of the most important factors associated with thisphenomenon is auditory selective attention which refers to the enhanced extraction of a specific soundcharacteristic based on increased attention. Frequency [6], time [7] and several other factors havebeen reported as examples of these characteristics.
This study highlights the effects of auditory selective attention in the spatial domain. Sound fromthe direction where listeners’ attention is directed can be heard easily in noisy environments. A
previous study reported that the effect of auditory spatial attention depends on the listening task [8].
The word intelligibility of the four-mora target words increased with increased attention to thedirection in which the words were presented to the listeners from spatially distributed distractors.
However, the detection threshold of the narrow-band noise was not significantly influenced byattention. Furthermore, this study found a spatial spread of attention, represented by a U-shaped
function with a maximum value in the direction of attention.
The spatial pattern of auditory selective attention would likely vary depending on where listeners
1

saka@ais.riec.tohoku.ac.jp
satoshi.tomimatsu.s2@dc.tohoku.ac.jp
3
kawase@orl.med.tohoku.ac.jp
2direct their attention. Studies have researched several auditory phenomena with directional
dependency [9-11]. It is crucial to accurately estimate the directional dependency of the spatial patternassociated with auditory selective attention to ensure that binaural hearing devices can appropriately
generate the optimum width of the beamforming.
Thus, this study investigated the characteristics of auditory spatial attention and how attention tovarious azimuthal angles affects the performance of target sound extraction, especially when the
directions of the attention and sound source are different.

2. ExperimentThis experiment comprised 30 participating listeners (19males and 11 females, average age: 21.7 years) with normal
hearing acuity. During the experiment, four-mora word
sequences uttered by a male speaker were presented by 12
circularly arranged loudspeakers (Fig. 1). The inter-stimulusinterval (ISI) of the words in the word sequence presented byeach loudspeaker ranged from 0 to 400 ms. These words wereregarded as distractors. Furthermore, four-mora words wereuttered by a female speaker, replacing some of the four-mora
words presented by the 12 loudspeakers. The timing anddirection of the four-mora words expressed by the female
Fig. 1 Experimental setup
speaker were randomly decided. Before starting each session,
the target word was indicated by using a display set in front of
the listeners. In the random condition, the listeners were asked merely to judge whether the worduttered by the female speaker was the target word as fast as possible. Similarly, the listeners wereasked to direct their attention to one of the loudspeakers and identify the target word in the attention
condition. The percentage of correct identification of the target word was calculated and converted
using arcsine square root transformation [12]. The difference in the converted percent correct between
the random and attention conditions reflected the magnitude of the effect of auditory spatial attention.

3. Results and Discussion
Figure 2 shows the calculated difference in percent
correct between the random and the attention conditionsin the frontal region (−90°~90° ). The difference wasplotted as a function of the sound source direction
relative to the direction of the attended loudspeaker. As
shown in Fig. 2, the difference in the correct response
rate was higher when the word was presented from theattended direction of the listeners. Furthermore, theshapes of spatial windows formed by attention varied
based on the direction of attention. The shape of thespatial window of auditory selective attention is broader
Fig. 2 Difference of the percent correctwhen their attention is directed to the sides than to thebetween the random and thefront. This result suggests that listening on the sides is
attention conditions
largely affected by spatial attention while attending to
Note:
Data
are presented as a function of
the front significantly suppresses the ability to listen to
soundsource direction relative tothe target sound presented from the sides. A similartheattended
direction. The
directional dependency has been observed for various
percentage
isconvertedusing
auditory characteristics such as the minimum audible
arcsinesquareroot
transformation.
angle (MAA) [9]. The MAA in the frontal region was
smaller than that in the lateral region. Based on thisdirectional dependency, the width of the spatial window of speech enhancement or noise suppression
can be appropriately determined.

4. Conclusion
In this study, the spatial characteristics of auditory spatial attention were investigated. The resultsof the experiment revealed that the width of the spatial window of auditory spatial attention depends

on the direction in which listeners direct their attention. The width of the attentional window isnarrower when attention is directed to the frontal region in contrast to the side region. Analyzing brain
waves [13], gaze direction, and head orientation [14], among others, enhances the ability to sense thedesired direction of the listener. By combining this knowledge with the findings of this study, theappropriate width of the beam forming created by binaural hearing devices can be optimized.

ACKNOWLEDGEMENTS
This study was partially supported by JSPS KAKENHI, Grant Numbers 19H04145 and 22H00523.

REFERENCES
1. Cherry E.C. Some experiments on the recognition of speech, J. Acoust. Soc. Am. 1953; 25, pp.975-979.
2. Bregman A.S. Auditory scene analysis: The perceptual organization of sound, Cambridge, Massachusetts,
USA: MIT Press; 1990.
3. Bronkhorst A.W. The cocktail party phenomenon: A review of research on speech intelligibility in multitalker conditions, Acta Acust. United Ac. 2000; 86, pp.117-128.
4. Bronkhorst A.W. The cocktail-party problem revisited: Early processing and selection of multi-talker
speech, Atten. Percept. Psychophys. 2015; 77, pp.1465-1487.
5. Ebata M. Spatial unmasking and attention related to the cocktail party problem, Acoust. Sci. & Tech.
2003; 24, pp.208-219.
6. Greenberg G.Z. and Larkin W.D. Frequency-response characteristic of auditory observers detecting
signals of a single frequency in noise; The probe-signal method, J. Acoust. Soc. Am. 1968; 44, pp.15131523.
7. Sanders L.D. and Astheimer L.B. Temporally selective attention modulates early perceptual processing:
Event-related potential evidence, Percept. & Psychophys. 2008; 70, pp.732-742.
8. Teraoka R., Saamoto S., Cui Z., Suzuki Y., and Shioiri S. Effects of listening task characteristics on
auditory spatial attention in multisource environment, Acoust. Sci. & Tech. 2021; 42, pp.12-21.
9. Mills A.W. On the minimum audible angle, J. Acoust. Soc. Am. 1958; 30, pp.237-246.
10. Saberi K., Dostal L., Sadralodabai T., Bull V., and Perrott D.R. Free-field release from masking, J. Acoust.
Soc. Am. 1991; 90, pp.1355-1370.
11. Grange J.A. and Culling J.F., The benefit of head orientation to speech indelibility in noise, J. Acoust.
Soc. Am. 2016; 139, pp.703-712.
12. Sokal R.R. and Rohlf F.J. Biometry; the principles and practice of statistics in biological research, Third
edition, New York, New York, USA: W.H. Freeman; 1995.
13. Worden M.S., Foxe J.J., Wang N., and Simpson G.V. Anticipatory biasing of visuospatial attention
indexed by retinotopically specific α-bank electroencephalography increases over occipital cortex, J.
Neurosci. 2000; 20, pp.1-6.
14. Doshi A. and Trivedi M.M. Head and eye gaze dynamics during visual attention shifts in complex
environments, J. Vision. 2012; 12, pp.1-16.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0634

Trends in user interactions with hearing aids
Matin F. MCKINNEY1; Lue (Derek) DU; Kenneth JENSEN; Jumana HARIANAWALA
Starkey, USA

ABSTRACT
Modern hearing aids offer a great deal of automatic parameter adjustment in response to dynamic and wideranging acoustic environments. Nevertheless, hearing aid wearers have the ability to override current
parameter settings through the utilization of on-device controls to adjust volume levels, switch between preprogrammed memories of environment-specific parameters, and invoke specific processing features of theirdevices. In addition, modern hearing devices contain wireless communication capabilities that allow them tobe connected to and controlled by devices such as smartphones. This capability greatly enhances the types
and range of control hearing aid wearers can have on their hearing aids. The current study uses data collected
through hearing device data logs and connected smartphone apps to analyze user behavior and interactivity
with their hearing aids. Individual user control behavior is examined, and on-device control patterns arecompared to app-based control patterns. We also consider demographic and audiologic factors as potential
inputs in an attempt to model user control behavior patterns.
Keywords: Hearing Aids, User Interaction

1. METHOD
We use hearing aid data logging records to examine the amount of time hearing aid wearers spend
in their default memory relative to time spent in specialized memories and use this measure as a proxyfor hearing aid memory interaction, i.e., the more time spent outside the default memory correlateswith more interaction with hearing aid memory controls. Additionally, we can examine how often
hearing aid wearers activate specific features in the hearing aid and how much they utilize thecompanion mobile app.

2. RESULTS
Figure 1 shows data on hearing aid memory usage as a function of the utilization of Starkey’s Edge
Mode hearing aid feature. Edge Mode provides users with instantaneous context-dependent parameter
adaptation. The data indicate that users who make use of their specialized memories are also more
likely to utilize hearing aid features Edge Mode to customize hearing aid settings. The data also show
many hearing aid wearers do not make use of specialized memories or hearing aid features at all.

Figure 1. Bars show the percentage of devices in categories of default memory usage. The left bar shows devices whose users
have not used Starkey Edge Mode; the right bar shows devices whose users have used Edge Mode.

1martin_mckinney@starkey.com

Figure 2 (left panel) shows how often hearing aid wearers activate Edge Mode from their default
memory as a function of time spent in their default memory. The diagonal band of users going frombottom left to upper right activate Edge Mode from their default memory in a manner proportional tothe default memory use. The users in the upper left part of the plot activate Edge Mode from theirdefault memory at a greater proportion than the time they spend in their default memory. Finally, users
in the lower right part of the plot activate Edge Mode from their default memory at a smaller
proportion than the time they spend in their default memory. The relatively large number of users in
the lower right part of the plot is consistent with the idea that users tend to utilize their specialized
hearing aid memories before activating Edge Mode and then activate it from the specialized memory.
Figure 2 (right panel) shows data for two groups of hearing aid wearers: 1) those with mild tomoderate hearing loss; 2) those with severe to profound hearing loss. The data show the median
number of daily Edge Mode activations as a function of median daily companion app events (e.g.
screen view, button press, etc.). Bootstrapping[1] our data shows significant differences across thegroups, where those with mild-moderate hearing loss activate Edge More often and utilize thecompanion app slightly less than the severe-profound hearing loss group. These differences ininteractivity types across hearing-loss groups may stem from the specific benefits each group receives
from the specific types of interactions. Further investigation is required to specifically identify these
possible causes.

Figure 2. Left panel shows the number of Edge Mode activations as a function of time spent in default memory, per user (eachpoint represents a single hearing aid wearer). Right panel shows the median daily Edge Mode activations vs the median daily
companion app events for two groups of users: 1) mild to moderate hearing loss; 2) severe to profound hearing loss.
Bootstrapping the median values revealed significant differences in the group medians for daily Edge Mode activations(p<0.01) and median daily companion app events (p<0.05).

3. CONCLUSION
Our data show that a large portion of hearing aid users do not interact with their devices to takeadvantage of specialized memory settings or adaptive hearing aid features. Those that do interact with
their device tend to utilize multiple customization methods. Finally, hearing loss degree may play arole in the specific type of interaction and customization that hearing aid wearers utilize most.

REFERENCES[1]

B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap. CRC Press, 1994.

ABS-0785

System for portable audio-based EEG feature extraction and adaptive
playback
Wannes VAN RANSBEECK(1) , Arthur VAN DEN BROUCKE(2) , Sarah VERHULST(3)(1) Hearing Technology Lab @ WAVES Ghent University, Belgium, wannes.vanransbeeck@ugent.be(2) Hearing Technology Lab @ WAVES Ghent University, Belgium, arthur.vandenbroucke@ugent.be(3) Hearing Technology Lab @ WAVES Ghent University, Belgium, sarah.verhulst@ugent.be

ABSTRACTThe interactions between humans and music are often studied through an impulse-response paradigm where thehuman response is considered purely a reaction to the present signal, hence neglecting more subtle relations andunderlying patterns. In this paper a portable biomonitoring and music playback system will be presented, studying the correlation between humans and music, with a more in-depth focus on analysing and characterising thisinteraction based on embodiment and bio-feedback. A Raspberry Pi environment is extended with an 8 channel
16kHz ADS-chip, capable of mobile high sample rate capture of EEG data from cEEGrid-electrodes, whichholds future extension possibilities to heart rate, saturation and movement. Furthermore, a high amplification
is present to allow subcortical signal capture along with adaptive music playback to alter and monitor playback
synchronised with data capture. This permits on-board analysis of the music interaction and response-based
playback adaptation. The system provides EEG data capture with a 3400Hz 3dB-bandwidth and RMS inputreferred noise of 0.503µV. A comparison was made with state-of-the-art cortical capturing devices through an
ABR measurement with stationary Biosemi EEG equipment. Additionally, an overview will be presented of the
aimed interaction and performance with regard to synchronisation of audio and EEG measurements and theiranalysis.

Keywords: Bio-feedback, interaction, EEG

1

INTRODUCTION

Stemming from the shown positive sociological, medical and health oriented influence of sound and music on ahuman being (1, 2, 3, 4), there currently is an omnipresent interest in the characterisation of these interactions(5, 6, 7, 8). For example, their influence has been found to be beneficial for both rehabilitation in general (26)as specifically in treatment of the Parkinson’s disease (27, 28), mental diseases (autism spectrum disorder) (29,30) as well as general well-being (31). Said interactions are evaluated during presentation through bio-markers(EEG, heart rate, saturation, movement and etc.) which then provide a more in-depth view of how the sound isperceived and is responded to (9, 10, 11). However, the response is often considered a pure stimulus response,
i.e. a pure one way result of the input. Neglecting then any interaction or more intricate relations which may
be present in human-machine or human-human music interactions, as have been reported by mathematician,philosophers, musicologist and other scientists (12, 13, 14). Additionally, motivation in test subjects may belimited due to the lack of interaction potential.
In line with the above reasoning a more human-like adaptive music playback system, based on bio-markers,
is hence preferred. Although the idea of altering playback based on human response in general is not innovating, as multiple implementations have been made based on brain potentials, heart rate and others (32, 33, 34,35), their implementations do often not consider the resulting interaction which is realised between both listenerand sound or do this to a limited extend. It should however be stated that work by Dasenbrock et al. (38)provides a similar feedback aimed implementation as the presented work. However, here a more standalonesystem is aimed for playback and capture, suited as well for sub-cortical EEG capture, for example the Auditory Brainstem Response (ABR). The latter presenting higher demands on playback together with a more clear
emphasis on interaction.
In this paper a proof of concept system (POC) is presented, which is aimed at overcoming these limitations
by offering a portable and light-weight system for in-depth real-time analysis of the interaction as it manifests
itself between bio-markers (internally recorded EEG or acquired through I2C or SPI communication) and sound.
The analysis here will be based on spectral and temporal properties of both. The system is furthermore
capable of adapting the music playback based on mentioned analysis, thus creating an adaptive loop. In turn,a more human-like interaction between man and machine will be aimed for, better suited for human-like music
responses and realising an adaptive engaging environment which can be evaluated in real time.

2

SYSTEM DESIGN

2.1 Hardware layout
The system layout is presented below in Figure 1 and consists of two main components 1) An EEG capture
module containing a preamplifier, low pass filters, an ADC and optional capabilities of microphone integrationand 2) A central computing Raspberry Pi 3b+ module, for processing, external device communication over I2C
and SPI, storage and an audio playback module.

Figure 1. Overview of hardware layout (upper) and data flows (lower) inside the proof of concept system,
based on illustrations provided by and used with permission of Van Den Broucke (15). Process 1a) capture ofbiomarkers, Process 1b) storage of biomarkers, Process 2) interaction analysis, Process 3) adaptive playback.
2.1.1 EEG capture
The EEG capture and its subcomponents (Process 1a, Process 1b), providing signal processing and storage, areshortly discussed below. Additional documentation and evaluation of the EEG capture can be found in (15).
The auditory evoked potentials (EEG) are recorded by cEEGrid electrodes consisting of 8 electrodes placedaround the ear, developed by the group of Bleichner and Debener (17, 18). Which, due to their shape andplacement, are dedicated to hearing diagnostics and in particular the evaluation of them through the capturing ofEnvelope Following Response (EFR) and ABR signals. Second, the captured activity is amplified through lownoise (7nV/Hz, 3400Hz 3dB bandwidth) preamlifiers with a gain factor of 100, which ensures being capable ofcapturing subcortical EEG data. The preamplifiers furthermore possess a common mode rejection ratio (CMRR)
of -110dB and an offset voltage of 0.5µV and were equipped with a driven-right leg feedback loop to ensure aproper range for the preamplifiers. Third, the amplified signals are passed onto the motherboard through a lowpass anti-aliasing filter before being processed by a 16kHz analog-to-digital 24bit ADS1299 chip designed for
medical instrumentation studies. This also includes portable EEG measurements due to its low cost and powerusage (15). An external trigger input can also be provided to the ADS-chip to allow full ABR measurementson the system. A 7.4V, 6000mAh lithium battery was furthermore connected to the motherboard serving as themain power supply during operation for both the motherboard and processing unit.
An overview of the specification of the EEG-system may be found below in Table 1 taken from the indepth performance analysis and discussion in Van Den Broucke (15). It should however be stated that the mainprocessing in the latter paper for EEG-measurement is evaluated and controlled through an ESP32-WROOM-32(19) chip from the Espressif Company. Alternatively, communication and storage will here be handled througha Raspberry Pi. Nevertheless, this should not influence the actual functioning of the EEG capture.
Table 1. Specification of EEG-system as taken from Van Den Broucke (15)Specification

System performance

Maximum sample rate

16000Hz

Sample rate accuracy< 200ps

Sampling skew

25ps

Bandwidth (-3dB)

3400HzTotal input ref. noise

0.5µVrms

Harmonic distortion< 0.001%

Resolution

24bit

CMRR(at 50Hz)> 110dB

2.2 General interaction system and human
The main processing unit consists of a Raspberry Pi 3B+ from Raspberry Pi (21) running Debian 10. The
system contains a 64-bit quad-core processor running at 1.4GHz, with 1GB of ram and a 128GB micro-SD
card. Furthermore, besides the included 2.4GHz and 5GHz Wi-Fi 802.11n and Bluetooth 4.1 Low Energy(BLE) (20), a multitude of 40 GPIO pins is present for external communication, providing future extension toother hardware for bio-marker input. Likewise, the Serial Peripheral Interface (SPI) communication to the ADCchip is handled over said GPIO pins. Additionally a 3.5 mm stereo jack is available with PWM-regulated audio
playback which currently serves as the main playback module (21). The Raspberry Pi, the main processing hub,controls the different processes of the system. These being then Process 1a) capture and Process 1b) storage ofboth internal (EEG) and external bio-markers, Process 2) providing analysis of the recorded responses in relation
to presented sound and lastly Process 3) playback and altering presented sound in a real-time manner to create
an engaging and adaptive environment as illustrated in figure 1. Care was expressed in the implementation ofsaid processes here towards ensuring real-time synced accessibility of data and responsiveness of the system
in its audio playback. All processes are written and implemented in C, C++ or Python. Furthermore, themeasurement data recorded in Process 1a is shared between processes and sub-processes through a circularbuffer. Here read and write communication is managed by semaphore and read/write indexes onto the circularbuffer with implemented overflow and underflow signaling should a process or sub-process lag.

3

Evaluation of the current system

To evaluate performance of the system as a whole, the processes are individually assessed. This being, for
Process 1, an EEG measurement with comparison to state-of-the-art. Process 2 and Process 3 will be evaluatedthrough data-flow observations and overflows in the circular buffer during processing and synchronised playback.

3.1 Process 1 evaluation
The comparison for a proper EEG measurement is made to an ActiveTwo Mk2 BioSemi system (37) in alaboratory setup through an ABR measurement. The reference system captured evoked potentials around thehead through active electrodes (sintered Ag-AgCl) operating at 24bit resolution with a sampling rate of 16384Hz
and was secured onto the head via a 64 channel Biosemi headcap.
The experiment followed the measurement procedure as outlined in Bleichner et al. (18). Test subjects (5participants) were positioned inside of a double-walled and electrically shielded listening booth. Each participant
took place in a reclining share after which electrodes were attached to the participant’s head. For the Biosemi
measurement the central-midline electrode Cz (10-20 system) and two earlobe electrodes (EXT1 and EXT2), oneplaced on each earlobe, are recorded. The common CMS and driven-right-leg (DRL) electrodes were connected
to forehead and nose respectively. The Cz measurements were afterwards referenced to the average of EXT1
and EXT2. Alternatively the cEEGrid electrodes were placed around the right ear with a double-sided sticker
using the same DRL and CMS positioning serving as main reference. The cEEGrid channels were referenced
as well, but to other cEEGrid channels in line with (16). For both electrode setups conductive gel was applied.
The stimuli consisted of 4000 clicks of alternating polarity and presented to the right ear via an in-ear
speaker. These clicks occurred at an average rate of 10Hz with a uniformly distributed jitter up to 10% in period. The click itself consisted of an 80µs pulse followed by silence and calibrated to a 100dB peak-equivalentsound pressure level presented inside the outer ear. The clicks and triggers were generated in MATLAB at asampling rate of 48kHz and stored together with the EEG data for evaluation.
The processing of the ABR results was done offline in Python for both Cz Biosemi measurements and thePOC. A low-pass and high-pass 4th order Butterworth filter were applied to the signals with respective cutoff
frequencies of 2000Hz and 100Hz. Afterwards 20ms epochs were taken, with the onset corresponding to thetrigger signal and epoch removal of the 15% with the highest amplitudes. The latter then being attributed toartifacts of head movement and others. The mean of the remaining epochs along with bootstrapped statisticalnoise floor and confidence interval of the mean (16), were calculated and are depicted in Figure 2.

Figure 2. Recorded and processed mean ABR response of 4000 clicks together with its calculated statisticalnoise floor and standard error of 1 standard deviation for a single subject recorded via the POC and reference
Biosemi system.
3.2 Process 2/3 evaluation
To ensure a proper time synced system, both processing and audio playback will be run, managed and maintained in Python to control both time-accurate data processing and audio playback. Here, playback is realisedusing the Psychtoolbox-3 library inside PsychoPy (36) for Python.
Data processing was evaluated under combined load of both Process 1a and Process 1b to serve as an initial
verification of correct real-time communication under limited load. A time-stamped 0.5s circular buffer was
implemented for data-communication and was allocated in the shared memory directory (dev/shm). Furthermore,access to the buffer was restricted through semaphores and checked for discrepancies in data between bothcapture and processing. However, an initial verification of data integrity was first performed through comparisonof measurement data as stored directly via Process 1a as well as though the circular buffer Process 1b. This
double storage only served as an initial verification of data and the added storage in Process 1a was deactivatedfor all other measurements. A timestamp was furthermore provided every 0.5s for syncing inside the sharedmemory structure of Process 1. The measurement window contained a 400s interval to ensure a full ABR
measurement could be performed with a 10Hz stimulation frequency and a maximum of 4000 repetitions, whichshould moreover be sufficient to capture a full musical interaction with songs of lengths less than 6.5 minutes.
Once content was verified as correctly transferred, overflow in Python is examined under load. Here thereadout of the shared circular buffer in Process 2 into an array was executed by Python while a simple peakdetection algorithm ran on the last 1024 recorded values per channel (8 electrode channels from the cEEGrid),
serving as initial load. Maximal offset between total written samples to the buffer by Process 1a and evaluatedsamples by Process 2 was measured over a 400s interval as well.

Figure 3. Audio offset and response time illustration, left and right respectively, of the POC. With the originalplanned audio "Scheduled" and the measured potentials by the ADS-chip "Recorded" and "Alteration".
Evaluation of audio playback is performed to assess both audio consistency of playback as well as responsiveness, i.e. the observed delay to signalled audio changes. As for audio consistency to the internal clock,offset in time between expected presentation time (output through audio jack), based on sample rate and buffer
length as predicted by Psychtoolbox, is compared to the arrival time as recorded by the ADS chip. Audio
output is directly looped back and connected to the preamplifier input of the system to evaluate the latter offset.
Sound production itself was setup with a 4096 sample buffer together with a sample rate of 48kHz. An excerpt
was taken from Johnny Nash’s "I Can See Clearly Now" as presented audio. The resulting delay for acoustic
playback between command execution and sound presentation, based on buffer length and sample rate, should
hence not exceed 85.3ms.
Additionally, response time is evaluated between recorded potentials and an associated introduced levelchange of the presented audio (setting a non-zero amplitude). For the latter a threshold detection was implemented on the recorded potential in Python, which upon activation altered the presented playback level. Figure
3 illustrates both the later recorded audio offset (left) as well as the response time (right). Both figures present
scaled versions of presented and recorded signals to increase visibility of the main delays.

4

DISCUSSION AND FUTURE CONSIDERATIONS

4.1 Process 1 EEG acquisition
Comparing the measured EEG responses in Figure 2 to a reference ABR measurement performed as well withcEEGrid (16) similar trends can be observed. Overlaying with the Cz measurement, performed after the cEEGrid
measurement to avoid interference, Wave-I, Wave-III and Wave-V can be identified. The latter being characteristic deflections within the first 10ms after stimulus presentation stemming from ascending relay stations of theauditory pathway (22). A clear identifiable Wave-I, Wave-III and Wave-V is present at 2.5ms, 5ms and 7ms

respectively in both measurements. However for the Wave-I, this peak is less visible and seems to be shiftedcompared to the expected Cz timing, potentially being a consequence of the measurements not being performed
simultaneously but in concession. Furthermore, delays as observed are in line with expectations for a 100dB
SPL knowing a 1 millisecond audio presentation delay is present in the system (23, 24). Additionally, amplitudes, as observed here for the cEEGrid recording, also agree in magnitude ( 0.1µV) to the findings in Garrett
et al. (16). This also applies to the noise floor measurements (+/-0.05µV), which agree with both recorded
measurements via the Biosemi system as well as with findings by Garrett, hence our proof of concept performs
equally well to the latter setup and the Biosemi system. This proof is furthermore backed up by similar resultspresented in (15).
4.2 Process 2 data processing and storage
Observing then data-flow as a whole within the system, an overall loss-free communication is established between processes. An initial assessment confirmed that a 400s measurement can be recorded between ADS chipand the main computer without a single lost sample for all 8 channels. Real-time data transfer of this collected
data was successfully accomplished during a 100s measurement (16kHz, 8 channels) over the cyclic buffer between processes Process 1a and Process 1b, with a character match for each individual line written to both .txt
files before and after the cyclic buffer. The latter confirms a perfect data transmission over the circular buffer.
Extension however to larger measurement intervals caused cyclic buffer overflows and associated loss of data
between processes. This is likely a consequence of the limited memory availability under load as the current
implementation buffered the full measurement in Process 1a for verification here. Alternatively, the continuous
write function in Process 1b may require too much processing time during recording, resulting in a data overflow
in the data collection. Further optimisation to this process as well as a planned hardware upgrade to a Raspberry
Pi 4 will likely solve this limitation. Alternatively a binary format storage of the data is considered to avoid
redundant bit writing and reduce writing load.
Evaluation under a limited load (only Process 1a and no storage Process 1b) of Python Process 2 furthermore
illustrated that the system was capable of keeping up with the real-time provided data, acquire and process itfrom the circular buffer with no loss of data (400s, 8 channels at 16kHz). The amount of read-in unprocessed
samples (for each of the 8 read in channels) never exceeded 450 which, translated to time, results in a timedelay of 28.1ms when sampled at 16kHz. As the delay is considerable and comparable in magnitude to theunavoidable delay introduced by the audio-buffer (∼80ms) the prior value was found lacking and currently
unacceptable. The combined delay (∼108ms) of both exceeds tolerable values audio music interactions (25) ofa maximal 50ms delay and is thus insufficient. A smaller audio-buffer will also be essential here to allow toachieve this 50ms goal which is currently impossible without introducing audible playback glitches.
It should additionally be noted that under the presented limited load, with no storage yet introduced, thecommunication with the ADS chip suffered from loss of a samples 32 in 6.400.000. The EEG capture can suffer
undesired frequency shifts and hence corrupt the measured brain activity data. A 0-loss ADS communication
will be aimed for. Further optimisation of resource management along with improved hardware is expected tosolve the above mentioned issues and allow perfect communication with the ADS under load.
4.3 Process 3 audio playback
Lastly, regarding audio playback, the offset between planned playback time and actual playtime is illustrated
in Figure 3. The offset between both seems to be more or less consistent over time and marginal (5ms) incomparison to the delays introduced by both processing and audio buffering. Nevertheless a compensation of
the average offset for said delay will be included in the final audio playback.
In addition the response time, as illustrated in Figure 3, seems to be consistent with the observations made insection 4.2. A clear time-delay is present between the detection of a high signal by the Python script (Alteration= 0.3), which is situated at 3.54s, and the following change of playback amplitude at 3.71s. Resulting in a
total response time of 170ms. Additionally the audio offset seems to have been altered and thus also requires
further investigation. Made observations confirm the present delay inside the system between measured response(recorded by the ADS chip) and audio adaptation is currently to high to allow the system to be operated in realtime in an adaptive manner compared to the desired 50ms delay (25).

The main limitations of the current proof of concept, being response times above 50ms, sample loss and data
buffer overflow, seem to be mainly a consequence of sub-optimal resource management, too large audio buffer
and code optimisation. A more efficient file storage (binaries) is aimed to be implemented along with reduced
audio-buffer sizes to create more headroom for actual processing. Furthermore, a dedicated core assignment,
optimised processing and a hardware improvement into a Raspberry Pi 4 will be implemented. Optionally adedicated sound card is considered.

5

CONCLUSIONSIn this paper a proof of concept system is presented and evaluated aimed at realising adaptive and interactive
music interaction based on recorded EEG data or other biomarkers. The system is currently capable of real-time
capture and storage of filtered and digitised EEG data at 16kHz with 8 channels via cEEGrid electrodes up to
400 seconds. A real-time analysis and capture of the EEG data time-synced to presented audio is implemented
allowing peak detection and further processing. Moreover, a reliable audio stream at 48kHz is produced by
the system capable of adapting playback to real-time captured brain activity. Hardware improvements and code
optimisation will be implemented to ensure loss free data capture and lower response time to real time (50ms).

ACKNOWLEDGEMENTS
Work supported by UGENT BOF IOP EarDiMon (WVR), European Research Council ERC-POC 899858 (SV)and FWO-SBO 1SB4421N (AVDB).

REFERENCES[1] Provasi J, Blanc L, Carchon I. The Importance of Rhythmic Stimulation for Preterm Infants in the NICU.
Children (Basel).8(8):660.,2021.
[2] Ashoori A, Eagleman DM, Jankovic J. Effects of Auditory Rhythm and Music on Gait Disturbances in
Parkinson’s Disease. Front Neurol. 2015;6:234.
[3] Cherko M, Hickson L, Bhutta M. Auditory deprivation and health in the elderly. Maturitas. 2016
Jun;88:52–7.
[4] Münzel T, Sørensen M, Schmidt F, Schmidt E, Steven S, Kröller-Schön S, et al. The Adverse Effects ofEnvironmental Noise Exposure on Oxidative Stress and Cardiovascular Risk. Antioxid Redox Signal. 2018
Mar 20;28(9):873–908.
[5] Haslbeck FB, Jakab A, Held U, Bassler D, Bucher HU, Hagmann C. Creative music therapy to promote
brain function and brain structure in preterm infants: A randomized controlled pilot study. Neuroimage Clin.
2020;25:102171.
[6] Fedotchev A, Parin S, Polevaya S, Zemlianaia A. EEG-based musical neurointerfaces in the correction ofstress-induced states. Brain-Computer Interfaces. 2022;9(1):1–6.
[7] Leow LA, Watson S, Prete D, Waclawik K, Grahn JA. How groove in music affects gait. Exp Brain Res.
2021 Aug;239(8):2419–33.
[8] Caruso GL, Nijs LL , Leman ML, Hepworth-Sawyer R, Paterson J, Toulson R. ’My avatar and me’:technology-enhanced mirror in monitoring music performance practice. In Focal Press - Routledge 2021;
2021. Available from: http://lib.ugent.be/catalog/pug01:8692364.
[9] Collett R, Salisbury I, Loeb RG, Sanderson PM. Smooth or Stepped? Laboratory Comparison of Enhanced
Sonifications for Monitoring Patient Oxygen Saturation. Hum Factors. 2020 Feb 1;62(1):124–37.
[10] Filippa M, Panza C, Ferrari F, Frassoldati R, Kuhn P, Balduzzi S, et al. Systematic review ofmaternal voice interventions demonstrates increased stability in preterm infants. Acta Paediatr. 2017
Aug;106(8):1220–9.
[11] Varni G, Mancini M, Volpe G, Camurri A. A System for Mobile Active Music Listening Based on Social
Interaction and Embodiment. Mobile Networks and Applications. 2011;16:375–84.
[12] Rodriguez-Fornells A, Rojo N, Amengual JL, Ripollés P, Altenmüller E, Münte TF. The involvement ofaudio-motor coupling in the music-supported therapy applied to stroke patients. Ann N Y Acad Sci. 2012
Apr;1252:282–93.
[13] Leman ML, The expressive moment: how interaction (with music) shapes human empowerment [Internet].
Cambridge; 2016. Available from: http://lib.ugent.be/catalog/pug01:8554951.

[14] Martin CP, Glette K, Nygaard TF, Torresen J. Understanding Musical Predictions With an Embodied Interface for Musical Machine Learning. Frontiers in Artificial Intelligence [Internet]. 2020;3. Available from:https://www.frontiersin.org/articles/10.3389/frai.2020.00006.
[15] Van Den Broucke A. Wireless and Wearable EEG Acquisition Hardware Using Around-The-Ear cEEGrid
Electrodes. To be published in Proc BIOCAS2022; Taipie, Taiwan. 2022 Oct 13-15.
[16] Garrett M, Debener S, Verhulst S. Acquisition of Subcortical Auditory Potentials With Around-the-Ear
cEEGrid Technology in Normal and Hearing Impaired Listeners. Frontiers in Neuroscience [Internet].
2019;13. Available from: https://www.frontiersin.org/articles/10.3389/fnins.2019.00730.
[17] Bleichner MG, Mirkovic B, Debener S. Identifying auditory attention with ear-EEG: cEEGrid versus highdensity cap-EEG comparison. J Neural Eng. 2016 Dec;13(6):066004.
[18] Bleichner MG, Debener S. Concealed, Unobtrusive Ear-Centered EEG Acquisition:cEEGrids
for Transparent EEG. Frontiers in Human Neuroscience [Internet]. 2017;11. Available from:https://www.frontiersin.org/articles/10.3389/fnhum.2017.00163.
[19] Espressif Systems. espressif.com. [Online]. Available from:https://espressif.com/sites/default/
files/documentation/esp32-datasheet-en.pdf.
[20] Raspberry Pi. raspberrypi.com. [Online]. Available from: https://www.raspberrypi.com/products/raspberrypi-3-model-b/.
[21] Raspberry Pi. raspberrypi.org. [Online]. Available from:https://static.raspberrypi.org/files/productbriefs/Raspberry-Pi-Model-Bplus-Product-Brief.pdf.
[22] Melcher JR, Kiang NY. Generators of the brainstem auditory evoked potential in cat. III: Identified cell
populations. Hear Res. 1996 Apr;93(1–2):52–71.
[23] Lewis JD, Kopun J, Neely ST, Schmid KK, Gorga MP. Tone-burst auditory brainstem response wave V
latencies in normal-hearing and hearing-impaired ears. The Journal of the Acoustical Society of America.
2015 Nov;138(5):3210—3219.
[24] Verhulst S, Jagadeesh A, Mauermann M, Ernst F. Individual Differences in Auditory Brainstem Response
Wave Characteristics: Relations to Different Aspects of Peripheral Hearing Loss. Trends Hear. 2016 Nov
11;20.
[25] Chew E, SAWCHUK A, Kyriakakis C, Papadopoulos C, François A, Volk A, et al. DISTRIBUTED IMMERSIVE PERFORMANCE. 2022 Jul 15.
[26] Sihvonen AJ, Särkämö T, Leo V, Tervaniemi M, Altenmüller E, Soinila S. Music-based interventions inneurological rehabilitation. Lancet Neurol. 2017 Aug;16(8):648–60.
[27] Benoit CE, Dalla Bella S, Farrugia N, Obrig H, Mainka S, Kotz SA. Musically Cued Gait-Training Improves Both Perceptual and Motor Timing in Parkinson’s Disease. Frontiers in Human Neuroscience [Internet]. 2014;8. Available from: https://www.frontiersin.org/articles/10.3389/fnhum.2014.00494.
[28] Kadivar Z, Corcos DM, Foto J, Hondzinski JM. Effect of step training and rhythmic auditory stimulation
on functional performance in Parkinson patients. Neurorehabil Neural Repair. 2011 Sep;25(7):626–35.
[29] Dieringer ST, Porretta DL, Sainato D. Music and On-task Behaviors in Preschool Children With Autism
Spectrum Disorder. Adapt Phys Activ Q. 2017 Jul;34(3):217–34.
[30] Finnigan E, Starr E. Increasing social responsiveness in a child with autism. A comparison of music andnon-music interventions. Autism. 2010 Jul;14(4):321–48.
[31] Croom AM. Music practice and participation for psychological well-being: A review of how music influences positive emotion, engagement, relationships, meaning, and accomplishment. Musicae Scientiae. 2015
Mar 1;19(1):44–64.
[32] Buhmann J, Desmet F, Moens B, Van Dyck E, Leman M. Spontaneous Velocity Effect of Musical Expression on Self-Paced Walking. PLOS ONE. 2016 May 11;11(5):e0154414.
[33] Moens B, Muller C, van Noorden L, Franěk M, Celie B, Boone J, et al. Encouraging Spontaneous Synchronisation with D-Jogger, an Adaptive Music Player That Aligns Movement and Music. PLOS ONE. 2014
Dec 9;9(12):e114234.
[34] Sourina O, Liu Y, Nguyen MK. Real-time EEG-based emotion recognition for music therapy. Journal on
Multimodal User Interfaces. 2012 Mar 1;5(1):27–35.
[35] Yuksel, Beste Filiz, Afergan, Daniel, Peck, Evan, Griffin, Garth, Harrison, Lane, Chen, Nick, et al.
BRAAHMS: A Novel Adaptive Musical Interface Based on UsersĆognitive State. In: Proceedings of theInternational Conference on New Interfaces for Musical Expression. Zenodo; 2015. p. 136–9[36] PsychoPy. [Online]. Available from: https://www.psychopy.org/.
[37] Biosemi. [Online]. Available from: https://www.biosemi.com/activetwo_full_specs.htm.
[38] Dasenbrock S, Blum S, Debener S, Hohmann V, Kayser H. A Step towards Neuro-Steered Hearing Aids:
Integrated Portable Setup for Time- Synchronized Acoustic Stimuli Presentation and EEG Recording. Current Directions in Biomedical Engineering. 2021;7(2): 855-858.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0799

User behavior modifies the effect of noise reduction scheme on
speech intelligibility
Kang SUN1,2,╪; Dorothea WENDT2,3; Lars BRAMSLØW1; Niels H. PONTOPPIDAN1
1
2

Eriksholm Research Centre, Oticon A/S, Denmark

Department of Applied Mathematics and Computer Science, Technical University of Denmark, Denmark3

Department of Health Technology, Technical University of Denmark, Denmark

ABSTRACT
Noise reduction schemes (NR) in hearing aids (HA) can improve speech understanding for people withhearing impairment (HI). Until now, the audio-only presentation has dominated laboratory tests in hearing
science. The visual component, however, is known to affect sound perception and human behavior. In thisstudy, two speech-in-noise tests, one using audio-only presentation (Test 1) and the other using VirtualReality (VR)-based audiovisual presentation (Test 2), were administrated on twenty-seven HI participants to
explore the effect of NR via a range of HA settings. The participants were instructed to fix their gaze (andwere not instructed on head movements) in Test 1, but could freely move their heads in Test 2. Moreover, in
Test 2, a binary head movement classification (HMC) was recorded, and a simulator sickness questionnaire(SSQ) was used to assess the cybersickness. The result indicated that NR was a significant factor in both tests,
which agreed with its appreciation found in earlier studies. However, the effect size in Test 2 was smaller.
When involving HMC, a significant interaction between HMC and NR was found. Namely that for theparticipants classified as with head movement, NR was found not significant. Moreover, HMC was foundsignificant in explaining all cybersickness symptoms. That is the participants classified as with head
movement experienced greater cybersickness.
Keywords: Noise reduction scheme, Virtual reality, Behavior

1. INTRODUCTION
Noise reduction schemes (NR, hereafter NR stands for both beamforming and postfilter) in hearing
aids (HA) can improve speech understanding for people with hearing impairment (HI). On one hand,the size of the effect is more pronounced in laboratory tests over field tests (e.g., [1]). The reasoning
for such differences remains unclear. For example, the audio-only presentation has dominatedlaboratory tests in hearing science until now. The visual component, however, is known to affect soundperception and human behavior. To understand the underlying mechanism of communication successin multi-dimensional real-life cases, it is vital to learn the effect of potential dimensions and how theyaffect speech understanding. On the other hand, the effect of NR was often evaluated in a manner ofcomparing on versus off states (e.g., [2]). Despite the different approaches to NR’s implementation[3], it is commonly believed that NR is beneficial for HI [4]. Therefore, NR is often applied (i.e., onstate) in clinical practice. Modern HA provides multiple levels of NR (e.g., [5]) to help HA users todifferent degrees. The exploration of the effect of NR at different levels remains nevertheless limited.
With these in mind, we designed two speech-in-noise tests in this study to investigate the effect ofmultiple HA features including NR on speech intelligibility (SI). Test 1 used an audio-only
presentation while Test 2 used audiovisual stimuli. In total, 27 HI participants with mild -to-moderatehearing loss were included in both tests. It is hypothesized that the benefit of increasing level of NR
differs in the two tests due to stimulus format differences.

╪ knsu@eriksholm.com

2. METHOD
2.1 Programs
Three HA features in device Oticon Opn S were included and configured from Oticon fitting
software Genie. Next to NR, feature B concerns frequency-specific amplification in fine-tuning. While
feature C is related to compensation in loudness perception for soft sounds. Two variables (min andmax, which stands for the lowest and highest possibility when this feature is turned on ) were taken
for each feature. The combination of the features and their variables formed the 8 programs in thisstudy (Table 1). As a reference, a medium setting for all three features, often referred to as the“common seen” setting, was also taken (program medium). Other HA settings in the device and theapparatus (speaker unit, dome, etc) were kept identical to the participants’ own individual HA settings(default program). All programs were applied in both tests.
Table 1 – HA programs and corresponding feature settings
Features | Program

1

2

3

4

5

6

7

8

mediumNoise Reduction

min

min

min

min

max

max

max

max

mediumFeature B

min

min

max

max

min

min

max

max

mediumFeature C

min

max

min

max

min

max

min

max

medium2.2

Setup

Test 1
Fig. 1 illustrates the experimental setup in Test 1. All loudspeakers formed a circle of 1.2m radius.
Each masker loudspeaker played a 4-talker babble noise. The masker noise level was fixed at 65 dB
SPL. The target loudspeaker presented the Danish DAT corpus [6]. Participants first obtained theirindividual SRT50 (Speech Recognition Threshold) with the HA program medium as reference. The
signal-to-noise ratio (SNR) was then fixed for the rest of the test (see Procedure). For each program,the speech intelligibility (SI) data was then collected relative to the medium baseline (50%).
Calibration was made at the participant’s seat position at the same height as the loudspeakers, using
a B&K 2250 sound level meter (Brüel & Kjær, Denmark). During the test, the participants sat in aheight-adjustable chair where their ears are set at the loudspeaker level. Participants were instructed
to fixate their gaze during the task, but no instruction on head movement was given. Test 1 was
conducted in a sound-proofed booth.

Target (0°)

1.2m

Maskers (±90°, ±150°)
65 dB
Figure 1 – Experimental setup (top view) in Test 1.

Test 2
Fig. 2 [down] illustrates the experimental setup in Test 2. A loudspeaker array (formed by 24
loudspeakers) played a pre-recorded cafeteria scene (Fig. 2 [up]) which contained 4 actors surrounding
the recording position. Note that the experimental setup in Test 2 was different from in Test 1 since
the intention was to visually only use the front half plane in Test 2. During the playback, the scenesound level was fixed at 65 dB SPL. The front loudspeaker presented the Danish HINT corpus [7],which was then augmented with the recorded scene via convolution with prerecorded room impulse
responses. During the test, the participants wore an Oculus VR headset, which visually presented therecorded scene and synchronized the audio playback. Details of the pre-recording, laboratory
implementation, and augmentation could be found in [8].
Like Test 1, the calibration was conducted at the participant’s seat and a height-adjustable chairaligned the participant’s ear level at the loudspeaker level in Test 2. Test 2 was conducted in an
anechoic chamber. The procedure and data collection were also identical to Test 1. However, no
instruction on gaze or head fixation was given in Test 2. Participants were free to rotate and check thescene in the VR headset at their wish during the whole test.
Moreover, the cybersickness of the test paradigm in Test 2 was investigated via a simulator sickness
questionnaire (SSQ). Four principal symptoms (nausea, oculomotor, disorientation, and total score)of cybersickness were measured. Details of the cybersickness measures could be found in [8].

T

1.4m

Figure 2 – [down] Experimental setup in Test 2 (Left: top view sketch of the setup; Right: illustration of aparticipant during the test);[up] Pre-recorded scene presented in the VR headset (Left: top view of the actors and recording position;
Right: the recorded view from the point of the A/V recorder, i.e., the participants’ view in the test.).
2.3 Ethics and Procedure
An exemption letter from the Regional Ethical Committee of the Capital Region (Hovedstaden),Denmark was obtained prior to the investigation (H-20068237). The participants signed a consent

form after receiving oral and written information, including the co ntinuous option of retracting theirconsent. All personal data were treated confidentially according to the Danish Data Protection Law.
In both tests, participants were instructed to perform a speech-in-noise task to evaluate HA settings.
For each program, a list of 20 sentences was presented. The exact settings were single -blinded, andthe order of the programs was randomized and balanced (in both tests). The two tests were performedon different days. Participants were not informed that the programs set in the two tests were identical.
Two lists for the adaptive procedure with the medium program were conducted. Only the SRT fromthe second list was taken as the SNR for the rest of the test. The two adaptive lists also served astraining lists for participants to familiarize themselves with the task.

3. RESULTS AND ANALYSIS
3.1 Speech intelligibility
Fig. 3 shows the SI result for all the programs (Table 1) in Tests 1 and 2. In Test 1, 3 programs (P5,P7, P8) obtained higher mean SI than the medium program, up to 9.1% (P7). A clear deviation (up to
20.8% between P3 and P7 in mean score, the same hereafter) of SI between all programs could befound. In Test 2, 2 programs (P6, P7) obtained a higher mean SI than the medium program, up to 1.7%
(P6). In comparison with Test 1, a smaller deviation of (up to 8% between P1 and P6) of SI between
all programs could be observed. The SI result suggested that a bigger difference between programs
could be found in Test 1. Meanwhile, all programs in Test 2 showed a limited difference from themedium program (grey line).

Speech intelligibility relative toreference (program medium)

15%

Test 1
Test 2
medium

10%
5%
0%
-5%
-10%
-15%
-20%
1

2

3

4

5
Program6

7

8

Figure 3 – SI results (mean and 95% CI) for all programs in both tests. Note that results were taken as
relative to SI in the reference (medium program, marked as the grey line in this figure).
To investigate the effect of NR in SI, an analysis of variance (ANOVA) test was applied. NR was
found a significant factor in both Test 1 (𝐹1,214=51.644, 𝑝<0.001) and Test 2 (𝐹 1,214=4.497, 𝑝=0.035),
as shown in Fig. 4. Clearly, the SI increased with the increasing level of NR (from min to max).
However, the SI increase (14.3% in Test 1 and 4.9% in Test 2) was smaller in Test 2. Moreover, theother two HA features (feature B and C, Table 1) individually were not found significant in both Tests
1 and 2 (not shown). Agreed with recent findings (e.g., [9]), NR was perceived as useful and bringing
the most contrast among other explored HA features. The benefit of NR, h owever, was found lesspronouncing in field tests than in laboratory tests. Interestingly, with the increase of stimulus
dimensions (audiovisual in Test 2 over audio-only in Test 1) in this study, this trend was found again.

Speech intelligibility relative toreference (program medium)10%

Test 1
Test 2
medium

5%
0%
-5%

-10%
-15%
min
0

max
1
Noise Reduction

Figure 4 – Effect of NR on SI results (mean and 95% CI) in both tests (relative to initial medium program).
3.2 Test 2: Head movement
Since participants could freely move their heads in Test 2 and browse the recorded scene in the VR
headset, their head movement behaviors were captured during the test. Two distinct groups – one who
never turned their head during the whole test (12p, 45%) and those who did so regardless of thefrequency and time (15p, 55%) – were identified. The binary head movement classification (HMC)
was then labeled as one participant variable. Using HMC and NR as factors, an ANOVA test wasapplied to the SI results in Test 2 (Table 2).
Similar to Fig. 4, NR was a significant factor, while HMC was not. Interestingly, a significant
interaction between HMC and NR (𝑝=0.029, Fig. 5) was found. Specifically, for participants with nohead movement in Test 2, a clear SI increase (10.7% in mean score) with increasing NR level could
be observed. The other group, who had head movement, did not show a clear difference in SI (0.4%)
when NR was set at the min or max level. Since the results were calculated relative to the program
medium individually, it only enabled the comparison between the medium, max, and min within thesame group. In Fig. 5, readers are recommended to compare the effect of NR in each group internally,i.e., not across the two groups at the same NR level.
The stimulus format in Test 2 evoked uninstructed head movement behavior, which was closer toreal-life situations instead of constantly facing toward the target talker in Test 1. Though only checked
by a binary classification (HMC), this human behavior might provide a possible perspective to explain
the limited effect of NR found in field studies.
Table 2 – ANOVA test result on SI in Test 2.
Cases

Sum of Squares

df

Mean SquareFp

Noise Reduction

0.162

1

0.162

5.636

0.018

Head movement

0.046

1

0.046

1.582

0.21

NR*HMC

0.139

1

0.139

4.847

0.029

Residuals

6.097

212

0.029

Note: Type III Sum of Squares.

Speech intelligibility relative toreference (program medium) - Test2

10%
5%

Noise Reduction min
Noise Reduction max
medium

0%
-5%

-10%
-15%
no
do movement
0 movement
1
Head movement classification

Figure 5 – Interaction between HMC and NR in SI (mean and 95% CI) in Test 2.
3.3 Cybersickness
Via the cybersickness questionnaire (SSQ), a score for all principal symptoms of cybersickness
was collected and computed for each participant in Test 2. Among all principal symptoms, Nausea
concerns general discomfort, increased salivation, sweating, and stomach awareness; Oculomotor isrelated to fatigue, headache, and eyestrain; Disorientation directs to fullness of head, dizziness, andvertigo; and Total score is the sum of the above three. Though participants showed high appraisals
and acceptance levels of the test paradigm, and limited cybersickness symptoms on average (see [8]),there was still an individual difference in cybersickness among the participants. To further explore thetwo distinct groups in HMC, an ANOVA test was applied to all principal symptoms of cybersickness
using HMC as a factor. HMC was found significant in explaining all symptoms (Fig. 6) including
Nausea (𝐹 1,214=56.781, 𝑝<0.001), Oculomotor (𝐹 1,214=79.151, 𝑝<0.001), Disorientation(𝐹1,214=36.995, 𝑝<0.001), and Total score (𝐹1,214=72.276, 𝑝<0.001). Namely, the participants withhead movement experienced greater cybersickness (Fig. 6).
Cybersickness is distinct from motion sickness in that the user is often stationary but has acompelling sense of self-motion through moving visual imagery. Earlier research (e.g., [10]) indicatedthe Postural Instability Theory [11] and the relative position of the participants might contribute tocybersickness. Though VR technology has been rapidly advanced in recent decades, technical aspects
such as visual lag due to visual transitions (even though the recording in Test 2 was stationary itself)might also contribute to cybersickness. Therefore, the effect of HMC on cybersickness agreed withprevious knowledge.

25%

Symptom degree

20%

NauseaOculumotor
Disorientation

15%

Total score

10%
5%
0%
no
do movement
0 movement
1
Head movement classification

Figure 6 – Effect of HMC on principal symptoms in cybersickness(y axis: 100% indicates the highest degree).

4. DISCUSSION AND CONCLUSION
This study presented two Tests. Tests 1 and 2 were not only different in stimulus format (audioonly versus audiovisual) and fixation instruction (yes versus no), but also in experimental setup andacoustical setting (Fig. 1 and 2). The study reported that NR was a significant factor in affecting the
SI regardless of the test difference. Specifically, the SI increased with the increasing level of NR. This
increasing benefit was however more pronounced in Test 1, which used an audio-only presentationand contained gaze fixation instruction. In Test 2, which used audiovisual stimuli and contained no
instruction in gaze or head movement, the SI increase with the increasing level of NR was smaller.
Such results could be explained by a binary head movement classification (HMC), particularly byparticipants who were classified as the head movement group. Moreover, HMC was found significant
in all principal symptoms of cybersickness. Specifically, participants who had head movement
experienced greater cybersickness in all components.
Though Test 2 was still far from field tests in terms of stimulus matrix, it nevertheless demonstrated
the decreasing SI benefit of NR while the head movement was not instructed. Such user behavior
captured in the study illustrates the possible real-life scenarios of how HA devices are used, which
increases the ecological validity of Test 2 [12]. User behavior is important to consider when testinge.g., HA signal processing and VR experience.
The difference between Tests 1 and 2 in the effect of NR could potentially be driven by fixation
instruction, stimulus format, stimulus material, and setup. It remained discussion which one (ones)
caused the behavior difference fundamentally. Further investigation on the effect of these items could
be of benefit. Furthermore, despite the HMC being addressed by the cybersickness, a continuous
measure of head movement is desired and would further examine the correlation between SI, head
movement, and cybersickness. In general, motion tracking of human behavior in laboratory tests isgetting popular in recent years. It comes as a good tool to reveal the underlying mechanism of, forexample, the human perception and benefit of HA features.

ACKNOWLEDGEMENTS
Part of the results was presented at International Hearing Aid Research Conference (IHCON) 2022
in Tahoe City, CA. This study is supported by the Danish Innovation Foundation and Demant
Foundation. The foundations are gratefully acknowledged. The authors would like to thank the clinicalaudiologists Soffi Skovlund Jensen and Lena Havtorn, and all test participants in this study.

REFERENCES
1. Wu, Y.H., Stangl, E., Chipara, O., Hasan, S.S., DeVries, S. and Oleson, J., 2019. Efficacy andeffectiveness of advanced hearing aid directional and noise reduction technologies for older adults withmild to moderate hearing loss. Ear and hearing, 40(4), p.805.
2. Ohlenforst, B., Wendt, D., Kramer, S.E., Naylor, G., Zekveld, A.A. and Lunner, T., 2018. Impact of SNR,
masker type and noise reduction processing on sentence recognition performance and listening effort as
indicated by the pupil dilation response. Hearing research, 365, pp.90-99.
3. Launer, S., Zakis, J.A. and Moore, B.C., 2016. Hearing aid signal processing. Hearing aids, pp.93-130.
4. Levitt, H., 2001. Noise reduction in hearing aids: A review. Journal of rehabilitation research anddevelopment, 38(1), pp.111-122.
5. Le Goff, N., Jensen, J., Pedersen, M.S. and Callaway, S.L., 2016. An introduction to OpenSound
Navigator™. Oticon A/S.
6. Nielsen, J.B., Dau, T. and Neher, T., 2014. A Danish open-set speech corpus for competing-speech studies.
The Journal of the Acoustical Society of America, 135(1), pp.407-420.
7. Nielsen, J.B. and Dau, T., 2011. The Danish hearing in noise test. International journal of audiology,
50(3), pp.202-208.
8. Sun, K., Pontoppidan, N.H., Wendt, D. and Bramsløw, L., 2022. Perception of Virtual Reality Based
Audiovisual Paradigm for People with Hearing Impairment. Euroregio/Baltic Nordic-Acoustic Meetings(BNAM2022) Joint Acoustics Conference.
9. Pasta, A., Petersen, M.K., Jensen, K.J. and Larsen, J.E., 2019, September. Rethinking hearing aids asrecommender systems. In HealthRecSys@ RecSys (pp. 11-17).
10. LaViola Jr, J.J., 2000. A discussion of cybersickness in virtual environments. ACM Sigchi Bulletin, 32(1),
pp.47-56.
11. Riccio, G.E. and Stoffregen, T.A., 1991. An ecological theory of motion sickness and postural instability.
Ecological psychology, 3(3), pp.195-240.
12. Keidser, G., Naylor, G., Brungart, D.S., Caduff, A., Campos, J., Carlile, S., Carpenter, M.G., Grimm, G.,
Hohmann, V., Holube, I. and Launer, S., 2020. The quest for ecological validity in hearing science: What
it is, why it matters, and how to advance it. Ear and hearing, 41(Suppl 1), p.5S.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0934

Why You Should Listen to Cyborgs: Autoethnographic Methods andEcological Validity in Interactive Acoustics Research
Krista KENNEDY1 and Abigail LONG1
1

Syracuse University, U.S.A.

ABSTRACTThis paper offers a humanities-based perspective on autoethnographic data collection from long-term d/Deaf
wearers of hearing aids. Smart hearing aids afford the d/Deaf wearer enhanced sociality as well as closeinteraction with smart acoustic agents that are shaped by the wearer’s habits and needs. The wearer is in turnshaped by these technologies down to the neurological level, keeping speech processing centers active toenable social and mental health. Such long-term, successful adopters are contemporary cyborgs, humans
whose bodies integrate compulsory technological components. These experienced users often constructlistening ecologies that incorporate a variety of analog and digital acoustic components that include speakers,
directional mics, smart phones, mylar balloons, and simply sensing vibrations in addition to their hearing
aids or cochlear implants. Autoethnographic methods can capture the nuanced, subjective experiences ofcyborgs who work closely with acoustic devices across multiple decades and in highly variable environments.
Among other things, autoethnography offers data on the ways highly functional wearers use aids to manage
disability disclosure and inequity and create highly specific listening ecologies.
Keywords: Hearing Aids, Qualitative Research Methods, Ecological Validity

1. INTRODUCTION
This paper offers a Humanities perspective on research methodology and ethics in interactive
acoustics research. The authors’ home discipline is rhetorical studies, a field that is broadly concernedwith the ways that effective persuasion functions within cultural contexts. Primary investigator
Kennedy’s research focuses on the ways that humans use technology persuasively and are persuaded
to use technologies, particularly in disability contexts. A deaf scholar and hearing aid wearer of morethan four decades, they are particularly interested in the ways that hearing aid wearers use interactive
acoustics technologies to persuasively navigate social contexts, negotiate disability disclosure, andmaintain neurological auditory processing health in ways that facilitate rhetoricability [1], or theability to be considered persuasive in social exchanges. Only extended abstract written in the English
language will be accepted for inclusion in the Proceedings. Technical Program Committee reserves
the right to reject any extended abstract considered inappropriate for the Proceedings, even if theabstract was previously accepted.
In this paper, we discuss autoethnographic methods as a way of working with long-term hearing
aid wearers 2 to develop useful, ethically harvested data. As Rapport and Hughes have argued,
qualitative methods have vital implications for understanding nuanced experiences of disability anddisease, including “people’s emotions, relationships, decisions, and support needs, and can report ona range of difficult subjects, such as anxiety, stigma, and pain” [3]. Autoethnography is an important
addition to the qualitative toolbox that they describe, offering a way of listening closely to long-termd/Deaf people. Given the decades of lived experience that these hearing aid wearers have withnegotiating a wide variety of dynamic, real-world acoustic environments, autoethnography has muchto offer as a means of gathering qualitative data that has applications for ecological validity as
described by Keidser, et al [4].

1 krista01@syr.edu. Pronouns: they/them.
2 Following Liza Potts' assertion that "while referring to people as users is easy, doing so undermines the notion of how centrally important participation has
become in systems," (8) [3], we use the term "wearer" rather than "patient" or "user" to emphasize bodily and active engagement in socially-focused choices.

2. EXPERIENCED WEARERS: STUDYING CYBORGS
Much of the audiological and, to some extent, related acoustic literature focuses on two central
populations: pediatric and geriatric. This is appropriate, given that language acquisition and language
retention, respectively, are paramount concerns for these populations in order to develop and maintainneurological and social health. However, these hearing aid wearers tend to have little to moderate
experience with successfully using amplified sound, given the fact that young users simply have not
had much time to experience negotiating the use of an aid while older users are typically late-deafened
and thus become new users of hearing aids later in life. Neither population typically encounters the
widely varied acoustic environments that mid-life professionals may encounter. As study participants,
members of pediatric and geriatric populations can offer rich data but within relatively narrowcontexts.
However, another population that remains understudied has deeply contextual data to offer: longterm hearing aid wearers who have decades of experience with using assistive listening devices tonegotiate rich, varied social contexts in their personal and professional lives. Such long-term,
successful adopters are contemporary cyborgs, humans whose ‘bodies’ integrate compulsory
technological components in order to maintain basic, everyday functionality [5]. While hearing aids
are less physically integrated than cochlear implants, both types of assistive devices directly impact
the listener’s neurological health, providing a stream of social data that strengthens the auditory cortex
and, consequently, auditory processing. This in turn shapes sociality, access to education andeconomic security, and access to safety. Without this deep entanglement and collaboration with theirlistening devices, these wearers simply would not be the people they are or live the lives they areliving.
These experienced wearers are, by necessity, agile users of assistive devices, often constructing
listening ecologies that incorporate a variety of internal and external acoustic components that include
speakers, directional mics, mylar balloons [6], and smart phones alongside hearing aids or cochlear
implants. Well aware of their cumulative strengths and limitations, experienced wearers are also adept
at attentive listening in a wide range of acoustic environments and often (but not always) cognizantof the cognitive and physiological fatigue that results from listening strain. Their long-term experiencehas given them extensive knowledge about navigating social aspects of aid visibility, use, and failures.
In short, this population has a tremendous amount to contribute to research on ecological validity.

3. ETHICAL CONSIDERATIONSHuman subjects research requires the careful negotiation of risk to the well-being of study subjects.
While Institutional Review Boards may approve research as meeting baseline ethical standards andresearchers may indeed fully intend to respect their patients, patient dignity is always at risk when
patients are under study—particularly when subjects are members of marginalized communities [7].
One example of this risk is algorithmic data collection from smart hearing aids, which offers
significant benefits to wearers, researchers, and manufacturers but also imposes dignitary harm bycompulsorily harvesting data (including geolocation data) for biocapitalistic purposes thatcompromise wearer dignity, privacy, and security [8-10]. Current U.S. Health Insurance Portability
and Accountability Act (HIPAA) and E.U. General Data Protection Regulation (GDPR) doctrines do
not adequately address these consent issues [11]. As a result, wearers of smart hearing aids can often
be treated like data rather than humans, continuing a trajectory of what disability scholar Mike Oliver
terms “research as alienation” for disabled people. “Disability research should not be seen as a set oftechnical objective procedures carried out by experts but as part of the struggle by disabled people tochallenge the oppression they currently experience in their daily lives,” writes Oliver [12]. Acoustic
scientists, as researchers in a vital field that impacts the daily lives of d/Deaf people, can be importantallies through their work and their decisions about research methods.
Acoustics researchers who investigate issues of ecological validity have deployed a wide range ofother research methods that offer varying degrees of direct involvement to their study subjects,
including participant diaries. These methods exist on a spectrum of not only data types, but of respectthat invites a deeper investment from disabled study participants while simultaneously recognizing
their dignity as humans who co-create data and research. Including autoethnographic methods
repositions hearing aid wearers as “agents of study” [13], recognizing that the usefulness of acoustic
technologies emerges through the negotiated, active experiences of users. Such methods offer waysof ethically listening to disabled patient populations and creating partnerships that further thedevelopment of interactive acoustic devices. In particular, autoethnography can offer nuanced insight
into the embodied experiences of marginalized groups [13], including disabled or non-normate
bodyminds. 3 By bringing critical attention to these embodied experiences, this method gives
researchers and readers alike the opportunity to interrogate their interpretive assumptions and broaden
their conceptions of what is needed and what is possible. In the case of experienced wearers withdecades of negotiating listening situations under their belts, more is often possible than one might
suppose.

4. AUTOETHNOGRAPHY AS EMPIRICAL METHOD
While other patient-centered methodological approaches such as diaries produce valuable data, wesuggest that autoethnographic methods have the potential to produce data that even more directly
involves the wearer and their insights in complex aspects of validity. Autoethnography also offers an
ethical approach that both values and centers disabled experiences of impairment and of using
listening aids to negotiate social situations, including managing disability disclosure, individual ethos,and the complexities of social interaction in variable contexts.
Autoethnography has long been deployed as a valid empirical method by researchers in the fields
of sociology and anthropology [15-16]. More recently, it has been taken up in educational research[17], health environment design [18], and architectural soundscape design [19]. This method issometimes misunderstood as being extended personal narrative or simply relying on the interpretationof diaries. However, the practice of autoethnography moves beyond those elements to incorporate
rigorously gathered, critically analyzed subjective data that connect personal experiences to broadersocial contexts and present sociological insights [20]—some of the central elements that compriseinvestigations into ecological validity. As Leon Anderson notes, an autoethnographer’s involvementin the experience being studied grants an “added vantage point for accessing certain kinds of data,”
eliciting questions that may only emerge from personal, “insider meanings” [21]. O’Hara characterizes
autoethnographic writing as a “scientific method” for contextualizing and analyzing the experiences
which underlie an individual clinician’s assumptions [18]. Ellis and Bochner propose “systematic
sociological introspection and emotional recall” as methods for understanding a subjective experience[16]. While experience and memory are central data collection methods for this approach,autoethnographers also ground their experiences in other data gathered through artifacts, interviews,
and written self-observation, reflection, and analysis [20].
Other forms of quantitative data are often incorporated into successful autoethnographic studies.
In a study directly tied to acoustic considerations, Gunnar Cerwén’s hybrid autoethnographic study ofsoundscapes in landscape design [19] employs an analytical autoethnographic approach [21] todiscover ways the Soundscape Actions design tool can improve soundscape design. Cerwén argues
for this hybrid approach in investigating soundscape design in Japanese gardens because it allows forconsideration of the subjective “interplay between the senses” perceived within a specific environment,
noting how subjective, autoethnographic representations can be compared to gain further insights withthe potential for generalizability [19]. The study integrates autoethnographic sensory accounts ofJapanese gardens with field recordings, sound pressure level readings, photographs, and other research
artifacts to study the effects of layered auditory and visual masking strategies (water features, birds,
vegetation, moss, walls, etc.). The researcher’s autoethnographic vignettes capture complex embodied
experiences (such as the disturbing feeling of a bus rattling past, breaking through the garden’s visualand auditory masking features) that might otherwise be considered outliers, yet impact a visitor’s
experience of a garden’s tranquility. Such grounded, data-based narratives of this sort of embodied
experience uncover much about the real-world acoustic impact of garden design features and havesolid potential to do the same in studies of ecological validity for interactive acoustic devices. Throughjuxtaposition and comparative analysis of autoethnographic narratives from long-term hearing aid
wearers, what pertinent yet unaccounted-for factors might acoustic researchers uncover?

5. CYBORG AUTOETHNOGRAPHIC DATA
The cyborg listeners—long-term hearing aid and cochlear implant wearers—described in section
two are well positioned to offer insight into subjective aspects of interactive acoustic device use. For
3 Price developed “the term bodymind to emphasize that although 'body' and 'mind' usually occupy separate conceptual and linguistic territories, they are deeplyintertwined” [14].

example, commonplace pandemic elements of social space include masks that obscure facial
expressions and lips, plexiglass barriers that both impede and bounce sound, and social distancingthat necessarily separates the listener from sound sources. Wearers with sufficient pre-pandemic
experience of navigating social situations know not only the practical differences that these changes
imposed, but also the ways that these necessary changes impacted their d/Deaf anxiety, listening
fatigue, sociality, and mental health. They likely have visceral examples of the ways that masks havechanged their health care experiences and the emotional energy needed to navigate those contexts andretain their ethos as persuasive self-advocates. They have stories about how friends have been willingor not willing to work with their requests for repetition, to speak up, or to meet for coffee or meals in
outdoor environments that impose a different set of challenges than the challenges of a restaurant
environment. They know what app settings they choose for their smart aids in order to better handle
a visit to the bank or whether they’ve simply opted to handle their monetary transactions entirely
online now. This is only one small example of a problem set: others might include professors
navigating lecture halls, auditoriums, meeting rooms, offices, outdoor events, and dining settings;
parents acting in a range of social contexts on behalf of their children; or deaf audiophiles (they exist;
Kennedy is one) who create listening environments in their houses, attend concerts, or listen to musicin cars and throughout their houses. In each case, autoethnography can be deployed as a sole method
or in hybrid approaches that include interviews, acoustic data, and other research methods.

6. CONCLUSION
In this paper, we examined the potential of autoethnographic research methods as an ethicalapproach for studying the subjective experiences of long-term hearing aid and cochlear implant
wearers. This method, whether used alone or in a hybrid design, offers a uniquely ethical, empiricalapproach for listening closely to the lived experiences of disabled research subjects while respecting
their dignity and involving them in interpretation of the research. It also yields rich data concerning
the nuanced, contextualized experiences of research participants who are accustomed to navigating
practical and emotional aspects of varied, high-stakes listening environments.

ACKNOWLEDGEMENTS
The authors gratefully acknowledge support from Profs. Collin Gifford Brooke and Rebecca Moore
Howard in the Department of Writing Studies, Rhetoric, and Composition at Syracuse University.

REFERENCES
1. Prendergast, C. “On the rhetorics of mental disability,” in Embodied rhetorics: disability in language andculture, eds. J. Wilson and C. Lewiecki-Wilson, Carbondale: Southern Illinois UP, pp. 45-60, 2001.
2. Rapport, F. and Hughes, S.E. “Frameworks for change in hearing research: valuing qualitative methods
in the real world,” Ear and Hearing, 41, pp. 91S-98S, 2020.
3. Potts, L. Social media in disaster response: how experience architects can build for participation,
Routledge, 2014.
4. Keidser, G., Naylor, G., Brungart, D.S., Caduff, A., Campos, J., Carlile, S., Carpenter, M.G., Grimm, G.,
Hohmann, V., Holube, I., Launer, S., Lunner, T., Mehra, R., Rapport, F., Slaney, M., and Smeds, K. “Thequest for ecological validity in hearing science: what it is, why it matters, and how to advance it,” Ear
and Hearing, vol. 41, pp. 5S-19S, 2020.
5. Weise, J. “The dawn of the ‘tryborg’,” The New York Times, 30 Nov. 2016.
6. Serinus, J. V. “‘Listening’: a video tribute to Art Dudley,” Stereophile, 2021.
7. Pearson, H. and Dickens, B. “(Re)framing qualitative research as a prickly artichoke: peeling back thelayers of structural ableism within the institutional review process,” in Centering diverse bodyminds incritical qualitative inquiry, eds. J. N. Lester and E. A. Nusbaum, Routledge, pp. 81-94, 2021.
8. Kennedy, K., Wilson, N., and Tschider, C. “Balancing the halo: data surveillance and algorithmic opacity
in smart hearing aids,” Rhetoric of Health and Medicine, vol. 4, no. 2, pp. 33-74, 2021.
9. Kennedy, K. and Wilson, N. “The banality of digital aggression: algorithmic data surveillance in medical
wearables,” in Digital ethics: rhetoric and responsibility in online aggression, hate speech, andharassment, eds. J. Reyman and E. Sparby, Routledge, pp. 214-230, 2019.
10. Tschider, C. “Prescribing exploitation,” Maryland Law Review, forthcoming 2023.
11. Tschider, C. A. and Kennedy, K. “Data discrimination: the international regulatory impasse of AI-enabled
devices,” in Legal, social, and ethical perspectives on health and technology, eds. M. Fathisalout-Bollon

and A. Berti Suman, Presses Universitaires Savoie Mont Blanc, pp. 147-168, 2020.
12. Oliver, M. “Changing the social relations of research production?,” Disability, Handicap & Society, vol.
7, no. 2, pp. 101-114, 1991.
13. Richards, R. “Writing the othered self: autoethnography and the problem of objectification in writing
about illness and disability,” Qualitative Health Research, vol. 18, no. 12, pp. 1717-1728, 2008.
14. Price, M. Mad at school: Rhetorics of mental disability and academic life. Ann Arbor, MI: University ofMichigan Press, 2011.
15. Hayano, D. “Auto-ethnography: paradigms, problems, and prospects,” Human Organization, vol. 38, pp.
99–104, 1979.
16. Ellis, C. and Bochner, A. “Autoethnography, personal narrative, reflexivity: researcher as subject,” in The
handbook of qualitative research (2nd edition), eds. N. Denzin and Y. Lincoln, Thousand Oaks, CA: Sage,
pp. 733–768, 2000.
17. Hughes, S., Pennington, J. L., and Makris, S. “Translating autoethnography across the AERA standards:
toward understanding autoethnographic scholarship as empirical research,” Educational Researcher, vol.
41, pp. 209-219, 2012
18. O’Hara, S. “Autoethnography: the science of writing your lived experience.” HERD: Health
Environments Research & Design Journal, vol. 11, no. 4, pp. 14-17, 2018.
19. Cerwén, G. “Listening to Japanese gardens: an autoethnographic study on the Soundscape Action design
tool,” International Journal of Environmental Research and Public Health, vol. 16, p. 4648, 2019.
20. Chang, H. “Autoethnography in health research: growing pains?,” Qualitative Health Research, vol. 26,
no. 4, pp. 443-451, 2016.
21. Anderson, L. “Analytic autoethnography,” Journal of Contemporary Ethnography, vol. 35, no. 4, pp. 37395, 2006.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0165

Machine-learning-based audio algorithms for cochlear
synaptopathy compensation: which speech features are enhanced?
Marjoleen WOUTERS1; Fotios DRAKOPOULOS2; Sarah VERHULST3
1,2,3

Hearing Technology @ WAVES, Dept. Information Technology, Ghent University, Belgium

ABSTRACT
Auditory models have been used for decades to develop audio signal processing algorithms in hearing aids.
Here, we used a biophysically-inspired auditory model, in a differentiable convolutional neural-network(CNN) description (CoNNear), to train different end-to-end machine-learning- (ML) based audio signal
processing algorithms that maximally restore cochlear synaptopathy (CS) affected auditory-nerve (AN)responses. Based on the reference normal-hearing (NH) model and a hearing-impaired (HI) model, we usedbackpropagation to design several ML-based algorithms, using the same CNN encoder-decoder architecture
but different loss functions focusing on different aspects of the AN responses. Processing of pure tones andwords by the ML-algorithms showed enhanced AN responses to both low- and high-frequency pure tones,
and to vowels and consonants in quiet, but responses were usually not restored to the NH-level. The
algorithms generally sharpened the onset response to speech and improved the stimulus dynamic range. In
an unconstrained operation, the ML-algorithms added more energy to the higher frequencies, degrading
speech quality and intelligibility. We will objectively assess the effect of these compensation algorithms onsound quality and speech intelligibility in future clinical experiments.
Keywords: Cochlear Synaptopathy, Machine-Learning, Hearing-Aid Processing

1. INTRODUCTION
Exposure to noise or ototoxic drugs and ageing are common causes of sensorineural hearing loss
(SNHL) in humans, and often result in irreversible damage of the outer hair cells (OHCs) or synapses
to the auditory nerve (AN), i.e. cochlear synaptopathy (CS) (1, 2, 3). Several studies have suggested
that CS results in a loss of the low- (LSR), medium- (MSR) and high-spontaneous rate (HSR) AN
fibers (ANFs), in which the LSR and MSR are the first to be lost (2). CS degrades encoding of thetemporal envelope in sound, which may contribute to a variety of perceptual abnormalities such as
speech-in-noise difficulties and decreased speech intelligibility (2, 4, 5).
However, pure tone audiometric thresholds, related to OHC loss, are not affected in CS, therefore
CS is referred to as “hidden hearing loss” (6). Studies on animal models have shown that the loss of
ANFs and synapses to the AN, related to cochlear neuropathy and synaptopathy, are the first signs ofpermanent hearing damage and occur earlier in time than OHC loss (1, 7). Since the audiogram is aninsensitive marker for damage to the AN and loss of synapses, patients suffering from CS will
experience difficulties understanding speech in challenging situations while their hearing thresholds
remain normal. Thus, it is expected that a large group of the noise-exposed or ageing population
suffers from CS, which still remains undiagnosed based on their audiogram and will therefore not betreated properly. Non-invasive diagnostic techniques of CS have been recently proposed based on
auditory-evoked potentials (AEPs) (8).
The current hearing-aid (HA) algorithms focus on compensating for the elevated audiometricthresholds, but do not specifically compensate for the hearing difficulties related to CS, and therefore
offer no treatment to patients who suffer from CS. The non-linear dynamic-range compression strategy
of current hearing aids even reduces the amplitude fluctuations of the temporal envelope, what mighteven worsen the hearing ability in case of CS (9, 10, 11). HA algorithms aiming to compensate formarjoleen.wouters@ugent.be
fotios.drakopoulos@ugent.be3
sarah.verhulst@ugent.be1
2

OHC loss and CS-related hearing impairment hence need to be computationally more complex thanstandard HA algorithms, in order to be able to grasp the complex non -linear working mechanism of
the auditory system.
Auditory models have been used for decades to develop audio signal processing algorithms in HAs.
Typically, the difference signal between a normal-hearing (NH) and hearing-impaired (HI) model isused to design such algorithms, but only recently machine-learning (ML) methods have made theirentry in this field. Specifically, when adopting differentiable descriptions of biophysical models of
hearing-impairment, it is possible to fully backpropagate through the models and design a new typeof ML-based audio signal processing that compensates for different aspects of SNHL (12, 13). The
objective of this work is to investigate several ML-based HA algorithms able to restore CS, based ona convolutional neural network (CNN) description of a NH and HI auditory model. We will also
investigate which sound and speech features are modified when letting several ML algorithms decide
the most-optimal solution to compensate for CS.

2. METHODS
CoNNear Auditory Model
We used a convolutional neural network model of the auditory periphery, CoNNear (14, 15, 16),that was developed starting from a biophysically inspired computational model of the human auditory
periphery (17). The CoNNear model provides a fast and differentiable description of the auditory
stages (basilar membrane (BM) vibrations, auditory nerve firing, inner-hair-cell (IHC) potential) of
the computational model across 201 simulated tonotopic cochlear locations , with center frequencies(CFs) spaced according to the Greenwood place-frequency map of the cochlea (18).
The NH CoNNear model is shown in Figure 1, and simulates the AN response 𝑟F to an auditory
input 𝑥, sampled at a rate of 20kHz since the CoNNear model operates at a sampling frequency of
20kHz. The CoNNear model consists of three distinct modules: the cochlear stage (CoNNearcochlea ),IHC stage (CoNNear IHC ), and ANF stage. The ANF stage is subdivided in the three different types of
ANFs: CoNNear ANfH, CoNNear ANfM and CoNNear ANfL for the HSR, MSR and LSR ANFs, respectively.
The responses of the three ANF types are combined together to yield the final summed AN response
𝑟F, by using weights HNH, MNH and LNH that correspond to the number of HSR, MSR and LSR fibers
in a NH periphery (HNH = 13, MNH = 3 and LNH = 3 as reported in Verhulst et al. (17)). The
CoNNearcochlea , CoNNear IHC and CoNNear ANf modules comprise encoder-decoder CNN architecturesthat have the advantage to backpropagate across them, facilitating the development of individualized
audio-enhancement methods.
2.1

Figure 1 – CoNNear model of the NH auditory periphery (12)2.2 DNN-Based CS-Compensating HA Algorithms
From the NH CoNNear model, we can obtain a HI CoNNear model by retraining the CoNNear cochlea
stage via transfer learning to simulate OHC loss (19), and by changing the weights of the different
types of ANFs in the CoNNear ANf stage to model AN fiber loss, related to CS. The CoNNear HI
periphery model can be individualized based on frequency-dependent degrees of OHC loss and CS.
The individualized degree of CS and OHC loss of a listener can be obtained from diagnostic
measurements using the rectangular amplitude-modulated envelope-following responses (RAMEFRs) and distortion-product oto-acoustic emissions (DPOAEs), respectively (20, 21).
Consequently, based on the reference NH model and a HI CoNNear model, we can use
backpropagation to design ML-based audio signal processing algorithms that optimally compensate
for CS (12, 13). The training procedure for a deep-neural-network- (DNN) based HA algorithm hasbeen explained by Drakopoulos et al. (12, 13) and goes as follows: the DNN-HA model processes the
input speech 𝑥 into 𝑥̂ such that the difference between the NH CoNNear model response 𝑟𝐹 and the
HI CoNNear model response 𝑟̂𝐹 is minimized. Different CS-compensating HA algorithms can be
designed in this way by defining a different loss function (12, 13). These functions can focus on
minimizing different aspects of the AN responses (e.g. free training, using more or less cochlea rchannels, limiting the frequency range, time-domain and frequency representations, also summed
across CFs). Thanks to the modular nature of CoNNear, the loss functions can be fine -tuned for eachof its distinct modules to optimally compensate for hearing -impairment in each of the modules.
In this work, we trained three DNN-HA models (CS_vow, CS_vow_cons and CS_freq) that tocompensate for a hearing-impairment with a CS profile of H HI = 7, M HI = 0 and L HI = 0 ANFs, and no
OHC loss. A CNN encoder-decoder architecture was used that comprised 16 layers (8 in the encoderand 8 in the decoder) as described in (12). The three HA models were trained using different loss
functions, which are listed in Table 1. A more detailed explanation of the different components of theloss functions that can be used during training is given by Drakopoulos et al. (13). In the CS_vow
model, the time-domain AN responses were squared (loss function 𝑙𝑟2 in (13)) and only AN responses
above a certain threshold were minimized (threshold 𝑇𝑟 in (13)). The squaring of the time-domain
responses was done to emphasize the temporal contrast of the speech envelope modulation to focus
the optimization on the enhancement of the most excited regions, since temporal envelope coding isessential for robust speech intelligibility (22, 23). The loss function that focused on only minimizing
the AN responses above a specific threshold was applied to further focus on the temporal peaks of the
responses. In the CS_vow_cons model, the AN response threshold 𝑇𝑟 was applied as well, and an
additional loss function was included between the root-mean-square (RMS) difference of theunprocessed and processed signals to ensure that restoration can be achieved without amplification of
the stimulus. In the CS_freq model, the loss function 𝑙𝑟2 was used and a frequency-weighting wasapplied to emphasize the processing of the low-frequency CFs (𝑓𝑟𝑒𝑞. 𝑒𝑚𝑝ℎ𝑎𝑠𝑖𝑠 in (13)), so that thehigh frequencies were processed less than the low. Emphasizing the low CFs in the optimization mightachieve better benefits in speech intelligibility since the speech corpus mostly contains energy at low
frequencies.
Table 1 – Loss functions used per DNN-HA model
Loss function

CS_vow

CS_vow_cons

Low-frequency CFs emphasizedx

Squared time-domain AN responsesx

AN response thresholdx

RMS difference - processed and unprocessed signals

CS_freqxxx

2.3 HA Model Evaluation
The three trained HA models were evaluated on their ability to compensate for the considered CS
profile of H HI = 7, M HI = 0 and LHI = 0 ANFs. Post-mortem data from recent temporal-bone studies
have shown that NH people have lost more than half of their AN innervations after the age of 50,
therefore we chose this CS profile of severe AN fiber loss (24, 25).
Pure tone stimuli and a battery of words in quiet were processed with the three trained DNN -HA
models, to evaluate the auditory feature restoration capabilities of the ML algorithms using transfer
functions and auditory model simulations. The DNN-HA processed stimuli were given to the HI
CoNNear model with the considered CS profile, in order to compare the simulated AN responses of
the NH CoNNear model to the responses of the HI CoNNear model, with and without applying the
HA processing. This way, we could investigate the difference in responses between the NH and HI
models, and see how the HA processing affects the output for the HI case, aiming to restore the AN
responses to the NH level. In this work, we present the processing outcomes for several pure tones
and the word ‘David’, extracted from the Flemish Matrix corpus (26). The three pure tones used as
input to the DNN-HA models respectively had a frequency of 500Hz, 2kHz and 8kHz with a durationof 404.6ms and an initial silence of 5ms, and were presented at a level of 70 dB SPL relative to thereference pressure p 0 = 2 ∙ 10−5Pa, with a sampling frequency of 20kHz. The word ‘David’ was alsopresented at 70 dB SPL, with a sampling frequency of 20kHz, and had a duration of 516ms. The DNNHA models require an input that is a multiple of 256 samples, hence zero-padding was applied at theend of the ‘David’ stimulus.
In order to investigate the transfer-function characteristics of the three HA models, the magnitude
spectra of the unprocessed and processed pure tones were visualized. For the word ‘David’ as input
stimulus, we analyzed several outputs obtained from the simulated CoNNear AN responses . The firstof the three presented outputs is the excitation pattern at the level of the basilar membrane, reflecting
the root-mean-square (RMS) over time of the vibration of the BM per CF. The BM excitation patterns
shows the vibration amplitude of the BM in function of the CFs along its length, in response to thefull input stimulus ‘David’. The second presented output is the excitation pattern at the level of theAN, reflecting the RMS over time of the summed AN response, which is the sum of the simulated
firing rates of the HSR, MSR and LSR ANFs, each weighted by their respective number of fibers
present (for the HI CoNNear model: H HI = 7, M HI = 0 and LHI = 0; for the NH CoNNear model: H NH =13, MNH = 3 and LNH = 3), per CF. The third presented output is the wave-1 (W-1) response, which isthe sum of the summed AN response across the different frequency channels (CFs), in time, calibrated
by a factor in order to match experimentally recorded wave-1 amplitudes.

3. RESULTS
3.1 HA Processing Analysis
We evaluated the auditory feature restoration capabilities of the three trained ML-based HA
algorithms, using transfer functions and auditory model simulations as described in the Methods. We
present the processing outcomes for several pure tones and a speech stimulus and investigate which
auditory features the different HA processing algorithms focused on to compensate for CS.
3.2 Processing of Pure Tones
The magnitude spectra in Figure 2 show the difference in processing between the three HA models
CS_vow, CS_vow_cons and CS_freq for pure tones of 500Hz, 2kHz and 8kHz, compared to theunprocessed pure tones that were given as input to the HA models. It can be seen from the plot of the
500Hz pure tone that the processing by the CS_vow model largely enhances the low frequencies andadds energy to the high frequencies in response to a low-frequency stimulus. In the plots of the 2kHz
and 8kHz pure tones, the blue curve corresponding to the processing by the CS_vow model is almost
zero dB SPL or lower for all frequencies, this HA model hence suppresses high frequencies above thephase locking limit. In the plots of all three pure tones, it can be seen that the CS_vow_cons model
adds energy to the high frequencies in the processing to minimize the AN response difference. For thepure tone inputs of both low and high frequencies, this HA model tries to restore the AN response by
exciting the CF regions that were not excited by the stimulus before processing. Hence, in thisunconstrained operation, the CS_vow_cons model adds more energy to the higher frequencies, which
created audible high-frequency tonal components. This effect could be reduced by applying afrequency weighting to the loss functions, which was done in the model CS_freq. It can be seen fromthe plots that the CS_freq model enhances the low-frequency pure tone of 500Hz, compared to theunprocessed case, and also still enhances the high-frequency input tones of 2kHz but not of 8kHz (dueto the frequency weighting). The CS_freq model does this without the addition of high-frequency
components. This CS_freq model clearly excites the CF regions closer to the input stimulus
frequencies, compared to the CS_vow and CS_vow_cons models.
From these magnitude spectra, we can learn that the combination of loss functions used in the
CS_vow model only enhanced the low-frequency pure tones, and failed to enhance the high-frequency
tones. The CS_vow_cons and CS_freq models enhanced both low- and high-frequency content, but
the lower frequencies in a lesser extent than in the CS_vow model. An additional frequency weighting
applied to the loss functions in the CS_freq model avoided the HA model to add too much energy tothe high frequencies in the aim to compensate for CS, by focusing the optimization more on the lowfrequency CFs. The CS_freq model shows a better energy distribution after processing, this HA model
focuses on enhancing the frequencies more close to the input frequency in comparison with CS_vow
and CS_vow_cons.
3.3 Processing of a Speech Stimulus
Figure 3 shows the excitation patterns to the input stimulus ‘David’ for the basilar membrane
vibration, and Figure 4 shows the excitation pattern of the summed AN response. The different curves
on the plots respectively show the simulated response of the NH CoNNear model, the impairedresponse of the HI model without processing the input stimulus, and the response of the HI model
after processing the input stimulus by the three HA models, i.e. the HA -processed responses.

Figure 2 – Magnitude spectra of unprocessed pure tones of 500Hz, 2kHz and 8kHz and their processedversions for the CS_vow, CS_vow_cons and CS_freq HA models
As can be seen from the excitation patterns of the BM vibration in Figure 3, the curves of the NH
and unprocessed HI excitation patterns overlap, since the used HI CoNNear model only included CS
and no OHC loss. The BM vibration excitation patterns show that the CS_vow model processing
amplified all frequencies compared to the unprocessed condition, while this amplification is lesspronounced in the CS_vow_cons and CS_freq models. Similar to what we observed in the pure tone
magnitude spectra, the CS_vow_cons model added more energy to the high frequencies (above 2kHz),
while this effect was reduced in the CS_freq model.

Figure 3 – Excitation patterns showing basilar membrane vibration to the input stimulus ‘David’ for theCS_vow, CS_vow_cons and CS_freq HA models

The excitation patterns of the summed AN responses in Figure 4 already show a clear difference
between the NH and HI responses to the unprocessed stimulus. The severe loss of ANFs in our chosen
HI CoNNear model significantly decreased the AN responses compared to NH. The summed AN
response shows that the CS_vow_cons and CS_freq models slightly increased the firing rate of the
ANFs across all frequencies, compared to the unprocessed HI case. We observe again the addition ofhigh-frequency firing in the CS_vow_cons model, creating audible high-frequency tonal componentsthat degraded speech quality and intelligibility. In the CS_vow model, the firing rate is in generalattenuated for frequencies below around 2kHz, but the higher frequencies are enhanced, compared tothe unprocessed case.

Figure 4 – Excitation patterns showing summed AN response to the input stimulus ‘David’ for the CS_vow,
CS_vow_cons and CS_freq HA models
From the plots in Figure 3, it can be observed that all HA models increased the BM vibration over
all frequencies, compared to the NH case, but the excitation patterns at the level of the AN in Figure
4 show that the enhancement obtained by the HA processing does not restore the summed AN responseto the level of the NH condition at all. We can learn from these Figures that the combination of theloss functions used in the CS_vow model was not able to enhance the firing rate at the level of the
AN for low frequencies (below around 2kHz). The CS_vow_cons and CS_freq models better enhanced
the excitation at the level of the AN across all CFs, probably due to the restriction of the RMS of theprocessed output (CS_vow_cons) and the emphasis on the low-frequency content (CS_freq). The
frequency weighting in the CS_freq model emphasizing the low frequencies had not much impact on
the suppression of the enhancement of the AN firing at high frequencies, compared to CS_vow_cons.
Figure 5 shows the time-domain waveform of the input stimulus ‘David’ and the wave-1 responses
to this input stimulus for the three different HA models, compared to the NH and HI unprocessed
responses. The occurrence of vowels and consonants is indicated in the plots. From the plots in Figure
5, it can be observed that CS_vow, CS_vow_cons and CS_freq all enhance the wave -1 response to thevowels ‘a’ and ‘i’, compared to the unprocessed HI response. The CS_vow model shows the largest
enhancement of vowels, for the vowel ‘i' even to the level of NH, but this HA model shows no responseto the consonants ‘d’ and ‘v’, the consonant response is even reduced compared to the HI unprocessed
response. The CS_vow_cons model and CS_freq model both show vowel and consonant enhancement,
but the wave-1 responses were not restored to the level of NH.
From the plots in Figure 5, we can learn that simultaneous vowel and consonant enhancement could
only be obtained when using the combinations of loss functions of the CS_vow_cons and CS_freq
models. The algorithms generally sharpened the onset response to speech and improved the stimulus
dynamic range. The onset response to both vowels and consonants was especially enhanced in the
CS_freq model, by using a loss function of the squared time-domain response that emphasized thetemporal contrast of the speech envelope modulation.

4. CONCLUSION
In line with findings of two other studies (12, 13) on ML-based end-to-end CS-compensating
algorithms, of which this work is a further application, we showed that such HA algorithms can be
designed through backpropagation in a fully automized way, without the need for prior assumptions
on the signal processing. The constraints in the loss functions of the trained algorithms cause
differences in restored auditory features to compensate for CS. Specifically, the used loss functions
influence whether the HA algorithms focus on enhancing low and/or high frequencies, and vowels
and/or consonants. Different combinations of loss functions were used to investigate which spe echfeatures were enhanced after processing. The CS_vow HA model, trained with a loss function of thesquared time-domain responses and that minimized the AN response only above a specific threshold,
was only able to enhance the low-frequency pure tones, and the AN response to vowels, not toconsonants. The CS_vow_cons model, trained with a loss function on the RMS difference between
the processed and unprocessed signal, and that minimized the AN response only above a specificthreshold, was able to enhance both low- and high-frequency pure tones and AN responses to bothvowels and consonants, but with addition of unwanted high-frequency tones that degraded soundquality. Enhancement of both low- and high-frequency pure tones as well as vowels and consonants
was also obtained for the CS_freq model, trained with a loss function that emphasized the lowfrequency CFs and that had a loss function of the squared time-domain responses. The latter CS_freq
model was the best performing HA model, its loss functions focused most on the low-frequency
content, which is most present in the speech corpus, sharpened the onset of the AN response, andemphasized the temporal contrast of the speech envelope modulation, which is beneficial for speech
intelligibility. Moreover, the emphasis of the lower frequencies in the CS_freq model reduced the
addition of unwanted high frequency components, such that the sound quality was less degraded thanin the CS_vow_cons model. However, none of the designed HA models was able to restore the AN
response to speech close to the level of NH for the used HI CoNNear model with a severe loss of
ANFs. The outcomes of this work suggest that an optimal compensation of such a severe loss of AN Fs
might not be possible using HA strategies, so the restoration of perceptual-relevant aspects can be abetter solution.
In future work, we will objectively assess the effect of these compensation algorithms on soundquality and speech intelligibility in clinical experiments (e.g. Flemish Matrix SRT task). FromDPOAEs and RAM-EFRs measurements, we will first assess the patient’s CS profile and degree of
OHC loss in order to create their individualized HI CoNNear model to use in the backpropagation
loop for the design of their individualized DNN-based HA model. Before we can start creating these
individualized HA models, we should first optimize the design of the loss functions of the DNN -HA
models such that they can optimally compensate for different types of hearing loss. A complete
restoration of the severe loss of ANFs that was presented in this work might not be possible, instead
we can focus on restoring relevant auditory aspects as much as possible. The CoNNear model can also
be expanded with a brainstem processing module in order to more precisely individual ize the HI modeland restore hearing loss.

ACKNOWLEDGEMENTS
This work was supported by the European Research Council (ERC) under the Horizon 2020
Research and Innovation Programme (grant agreement No 678120 RobSpear).

I

A
D

V

D

D
D

A
I
D
D

V

D

A
I
D
D

V

D

A
I
D
D

V

D

Figure 5 – Time-domain waveform of input stimulus ‘David’ and the wave-1 responses before and after
processing with the CS_vow, CS_vow_cons and CS_freq HA models

REFERENCES
1. Kujawa SG, Liberman MC. Adding insult to injury: Cochlear nerve degeneration after “temporary” noiseinduced hearing loss. J Neurosci. 2009;29(45):14077–85.
2. Furman AC, Kujawa SG, Liberman MC. Noise-induced cochlear neuropathy is selective for fibers with

low spontaneous rates. J Neurophysiol. 2013;110(3):577–86.
3. Lobarinas E, Spankovich C, Le Prell CG. Evidence of “hidden hearing loss” following noise exposuresthat produce robust TTS and ABR wave-I amplitude reductions. Hear Res. 2017;349:155–63.
4. Bharadwaj HM, Verhulst S, Shaheen L, Liberman MC, Shinn-Cunningham BG. Cochlear neuropathy andthe coding of supra-threshold sound. Front Syst Neurosci. 2014;8(26).
5. Mepani AM, Verhulst S, Hancock KE, Garrett M, Vasilkov V, Bennett K, et al. Envelope following
responses predict speech-in-noise performance in normal-hearing listeners. J Neurophysiol.
2021;125(4):1213–22.
6. Plack CJ, Barker D, Prendergast G. Perceptual consequences of “hidden” hearing loss. Trends Hear.
2014;18:1–11.
7. Sergeyenko Y, Lall K, Liberman MC, Kujawa SG. Age-related cochlear synaptopathy: An early-onset
contributor to auditory functional decline. J Neurosci. 2013;33(34):13686–94.
8. Keshishzadeh S, Garrett M, Vasilkov V, Verhulst S. The derived-band envelope following response and its
sensitivity to sensorineural hearing deficits. Hear Res. 2020;392:107979.
9. Drullman R, Festen J, Plomp R. Effect of reducing slow temporal modulations on speech reception. J
Acoust Soc Am. 1994;95:2670–80.
10. Rance G, McKay C, Grayden D. Perceptual Characterization of Children with Auditory Neuropathy. Ear
Hear. 2004;25(1):34–46.
11. Drakopoulos F, Vasilkov V, Osses Vecchi A, Wartenberg T, Verhulst S. Model-based hearing-restoration
strategies for cochlear synaptopathy pathologies. bioRxiv. 2022.
12. Drakopoulos F, Verhulst S. A differentiable optimisation framework for the design of individualized
DNN-based hearing-aid strategies. In: ICASSP 2022 IEEE International Conference on Acoustics, Speechand Signal Processing (ICASSP). 2022. p. 351–5.
13. Drakopoulos F, Verhulst S. A Neural-Network Framework for the Design of Individualised Hearing-LossCompensation. Preprint arXiv:2207.07091. 2022.
14. Baby D, Van Den Broucke A, Verhulst S. A convolutional neural-network model of human cochlearmechanics and filter tuning for real-time applications. Nat Mach Intell. 2021;3(2):134–43.
15. Drakopoulos F, Baby D, Verhulst S. A convolutional neural-network framework for modelling auditory
sensory cells and synapses. Commun Biol. 2021;4(1).
16. Verhulst S, Baby D, Drakopoulos F, Van Den Broucke A. A neural network model for cochlear mechanics
and processing. WO2020249532, 2020.
17. Verhulst S, Altoè A, Vasilkov V. Computational modeling of the human auditory periphery: Auditorynerve responses, evoked potentials and hearing loss. Hear Res. 2018;360:55–75.
18. Greenwood DD. A cochlear frequency‐position function for several species—29 years later. J Acoust
Soc Am. 1990;87:2592-2605.
19. Van Den Broucke A, Baby D, Verhulst S. Hearing-impaired bio-inspired cochlear models for real-time
auditory applications. In: Proceedings of the Annual Conference of the International SpeechCommunication Association, INTERSPEECH. 2020. p. 2842–6.
20. Keshishzadeh S, Verhulst S. Individualized Cochlear Models Based on Distortion Product Otoacoustic
Emissions. In: Proceedings of the Annual International Conference of the IEEE Engineering in Medicine
and Biology Society, EMBS. 2021. p. 403–7.
21. Keshishzadeh S, Garrett M, Verhulst S. Towards Personalized Auditory Models: Predicting Individual
Sensorineural Hearing-Loss Profiles From Recorded Human Auditory Physiology. Trends Hear.
2021;25:1–22.
22. Parthasarathy A, Bartlett EL, Kujawa SG. Age-related Changes in Neural Coding of Envelope Cues:
Peripheral Declines and Central Compensation. Neuroscience. 2019;407:21–31.
23. Vasilkov V, Garrett M, Mauermann M, Verhulst S. Enhancing the sensitivity of the envelope-following
response for cochlear synaptopathy screening in humans: The role of stimulus envelope. Hear Res.
2021;400:108132.
24. Viana L, O’Malley J, Burgess B, Jones D, Oliveira C, Santos F, et al. Cochlear neuropathy in humanpresbycusis: confocal analysis of hidden hearing loss in post-mortem tissue. Hear Res. 2015;327:78–88.
25. Wu PZ, Liberman LD, Bennett K, de Gruttola V, O’Malley JT, Liberman MC. Primary Neural
Degeneration in the Human Cochlea: Evidence for Hidden Hearing Loss in the Aging Ear. Neuroscience.
2019;407:8–20.
26. Houben R, Koopman J, Luts H, Wagener, KC, van Wieringen A, Verschuure H, Dreschler WA.
Development of a Dutch matrix sentence test to assess speech intelligibility in noise. Int J Audiol.
2014;53(10):760–3.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0363

Basic characteristics of distantly-presented bone-conduction
hearing in the audible-frequency range: Detection threshold andfrequency- and temporal resolutions
Seiji NAKAGAWA1, 2, 3; Hiromu ISHKAWA2; Sho OTSUKA1, 2, 3
1
2

Ctr. for Frontier Medical Eng., Chiba Univ., Japan

Dept. of Medical Eng., Grad. School of Sci. & Eng., Chiba Univ., Japan
3

Med-Tech Link Ctr., Chiba Univ. Hospital, Japan

ABSTRACTAs a vibrator needs to be pressed onto the osseous parts of the head, bone conduction is often accompanied
by pain and esthetic problems. To solve these problems, a “distant presentation” that involves presenting
vibrators to the neck, trunk, or upper limb was proposed. Our previous studies focused on the perceptual andpropagation characteristics of distantly presented bone-conducted sounds in the ultrasonic range. However,
only a few studies have been conducted in the audible-frequency range. In this study, to examine the basic
properties of distantly presented bone-conduction perception in the audible-frequency range, hearing
thresholds, difference limens for frequency, and temporal modulation transfer functions were measured withinsulated air-conducted sounds. The results indicate that the distance attenuation is much larger than that in
the ultrasonic range, and the degradation of frequency and temporal information occurring in the propagation
process of bone-conducted sounds is sufficiently small for the transmission of sound information.
Keywords: Bone conduction, Distant presentation, Audible-frequency range, Hearing thresholds, Frequency
and temporal resolution

1. INTRODUCTION
Several studies have reported that high-frequency sound above 20 kHz can be perceived clearly
via bone-conduction (BC). This audible ultrasound through BC is referred to as bone-conducted
ultrasound (BCU) (1–3). Generally, BC sounds are presented to the osseous parts of the head. However,BC sounds can also be perceived even when presented to body parts distant from the head, such as
the neck, trunk and upper limbs4). This “distant presentation” is expected to be applied to novel BC
devices.
Our previous studies focused on the perception and propagation characteristics of distantlypresented BCU (4-6). On the other hand, although it is indispensable to investigate also in the audiblefrequency range to elucidate the essential mechanisms of distant presentation of BC, a limited numberof studies have been conducted (7).
In this study, to examine the basic perceptual characteristics of distantly-presented BC sound in
the audible-frequency range, hearing thresholds, difference limens for frequency (DLFs), whichreflect the frequency resolution, and temporal modulation transfer functions (TMTFs), which reflect
the temporal resolution, were measured.
All experiments were approved by the Institutional Review Board of the Life Science Research of
Chiba University. Necessary information about the experiments was provided to the participants andinformed consent was obtained from each participant before the experiments.

2. METHODS
Seven males (21-26 years) with normal hearing participated. BC stimuli were presented to thefollowing body parts by a BC vibrator (Radioear B-81): (a) Mastoid, (b) Sternocleidomastoid, (c)Clavicle, (d) Acromion (only for the hearing threshold measurement) (Fig. 1). In all experiments, toevaluate only BC sounds, air-conducted (AC) sounds radiated from the vibrator were insulatedVibrator(a) Mastoid(b) Sternocleido
mastoid(c) Clavicle(d) Acromion

Figure 1 – Stimulus placements used in the experiments.
strongly by wearing urethan earplugs, silicone earplugs and an earmuff (except for the mastoid)
together. All experiments were conducted in an anechoic room.
Experiment Ⅰ: Measurement of hearing thresholds
As the stimuli, 250, 500, 750, 1000, 2000, 3000, 4000 Hz tone bursts with a duration of 800 ms
including 75 ms rising/falling ramps were used. Hearing thresholds were measured using a 1 up-2
down three-alternative forced-choice (3AFC) adaptive procedure with a decision rule that estimatedthe 70.7% correct point on the psychometric function (8).
2.1

Experiment Ⅱ: Measurement of DLFs
As the stimuli, tone bursts with the center frequencies (CFs) of 250, 500, 750, 1000, 2000, 4000
Hz including 30-ms rising/falling ramps were used (duration: 400 ms). DLFs were measured using a
1 up-2 down 2AFC adaptive procedure. Stimulus output levels were set to 10-15 dB SL. In each trial,
two tone bursts that were equally spaced in linear frequency on either side of the CF were presented,
and participants were requested to respond to the stimulus with higher frequency. The deviation fromthe CF (Δf) was varied adaptively.

2.2

Experiment Ⅲ: Measurement of TMTFs
TMTF shows the threshold of sinusoidal amplitude-modulation (SAM) detection as a function ofmodulation frequency. The SAM detection threshold is determined systematically by measuring thedetection of modulation depth. A double sideband amplitude modulation was applied to 1000-Hz and
4000-Hz sinusoidal carriers. SAM stimuli are expressed as follows:

2.3

𝑓 𝑡

1

𝑚

sin 2𝜋𝑓 𝑡

sin 2𝜋𝑓 𝑡1Here, m is the modulation depth, and f m and f c are the modulation and carrier frequency,
respectively.
TMTFs were measured using a 1 up-2 down 3AFC adaptive procedure. Stimulus output levels were
set to a level at which each participant could detect modulation sufficiently, based on 25 dB SL. In
each trial, three BC stimuli (duration: 800 ms including 75 ms rising/falling ramps) were presented
sequentially. One of the three stimuli was modulated, and the others were unmodulated. Participants
were requested to respond to the modulation interval. The modulation depth in the target intervals wasvaried adaptively.

3. RESULTS
Experiment Ⅰ: Measurement of hearing thresholds
Figure 2 shows relative thresholds when the threshold for the mastoid served as the reference. The
hearing thresholds increased as the stimulus placements got further from the head. Additionally, inthe distal parts, the relative thresholds increased as the stimulus frequency increased.

3.1

Experiment Ⅱ: Measurement of DLFs
Figure 3 shows DLFs in each stimulus placement. In the mastoid, the DLF was about 0.3% at all
CFs, and no significant differences were observed between the mastoid and distal parts at 500-3000
Hz. However, the DLF was increased at 250 and 4000 Hz in the clavicle.

3.2

Figure 2 - Relative threshold of the distantly-presented bone-conducted sound.
The threshold for the mastoid in each participant served as the reference (0 dB).

250

500

1000

2000

4000

Figure 3 - Frequency difference limens (DLF) for each stimulus placement.
Experiment Ⅲ: Measurement of TMTFs
Figure 4 (a) and (b) show modulation thresholds in each stimulus placement using the 1000-Hz
and 4000-Hz sinusoidal carriers, respectively. In the case of the 1000-Hz carrier, no significant
variations depending on increasing modulation frequency were observed in all stimulus placements.
Additionally, no significant increases as the stimulus placements got further from the head wereobserved. In the case of 4000-Hz carrier, the modulation thresholds increased at a modulation
frequency of 100 Hz or higher, however, no significant differences were observed among the stimulus
placements.

3.3

4. DISCUSSIONSIn our previous study on the hearing threshold of distantly-presented BCU4), the increase in the
hearing threshold from the mastoid was only about 5 dB in the sternocleidomastoid, 20 dB in theupper arm. On the other hand, in this study, the increase in the hearing threshold was about 20-40 dB
in the sternocleidomastoid, 35-65 dB in the clavicle and acromion. These results indicated that
distance attenuation is much larger in the audible-frequency range compared to the ultrasonic range.
At the mastoid and sternocleidomastoid, the DLF was the same as that of AC sound at all CFs9).
In addition, no significant differences in DLF were observed among stimulus placements at almost all
CFs. These results indicated that degradation of frequency information occurred in the propagation
process of BC sounds is sufficiently small.
No significant differences of TMTF were observed among stimulus placements at all modulation
frequencies regardless of the carrier frequencies. This result indicated that no degradation of temporal
information of BC sounds occurred in the propagation process in the human body.

ab

Figure 4 – TMTFs for at carrier frequencies of (a) 1000 Hz and (b) 4000 Hz.

ACKNOWLEDGEMENTS
A part of this work was supported by JSPS KAKENHI Grant Numbers JP19K22950 and
JP20H04497, and a Research Grant for the Next Generation Research Incubator from the Institute for
Global Prominent Research, Chiba University for SN.

REFERENCES
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.

V. Gavreau, Comptes Rendu Acad. Sci. 226, 2053 (1948) [in French].
R. J. Pumphrey, Nature 166, 571 (1950).
R. J. Bellucci and D. E. Schneider, Ann. Otol. Rhinol. Laryngol. 71, 719 (1962).
S. Nakagawa, R. Ogino, and S. Otsuka, Jpn. J. Appl. Phys. 57, 07LD22 (2018).
R. Ogino, S. Otsuka, and S. Nakagawa, Jpn. J. Appl. Phys. 58, SGGE12 (2019).
S. Nakagawa, K. Doi, R. Ogino, and S. Otsuka, Jpn. J. Appl. Phys. 58, SGGE18 (2019).
K. Doi, R. Ogino, S. Otsuka, and S. Nakagawa, Jpn. J. Appl. Phys. 59, SKKE26 (2020).
R. Ogino, K. Doi, S. Otsuka and S. Nakagawa: ARO Abstracts. 43 (2020) 368.
H. Levitt, J. Acoust. Soc. Am. 49, 467 (1971).
G. S. Yap, S. Otsuka and S. Nakagawa: Proc. of the 23rd Int’l Congress on Acoustics. (2019) 6668-6669.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0818

Effects of tempo of acoustic stimuli during sleep
Shota MAKI1†; Sho OTSUKA1-3; Seiji NAKAGAWA1-3‡
1
2

Department of Medical Engineering, Chiba University, Japan

Center of Frontier Medical Engineering, Chiba University, Japan
3

Med-Tech Link Center, Chiba University Hospital, Japan

ABSTRACTInsomnia becomes a common disorder nowadays. Since pharmacological treatments for insomnia have therisk of an addiction, developments of nonpharmacological tools to improve sleep disturbance are needed.
According to the brainwave-entrainment hypotheses, it is thought that acoustic stimuli with rhythms imitating
brainwaves during sleep can improve sleep quality. However, former studies showed that the effects ofacoustic stimuli with δ-rhythm were insufficient. In this study, we proposed a novel acoustic stimulus thatstepwisely changes its tempo after sleep onset, from θ-tempo (4 Hz) to δ-tempo (0.8 Hz), and examined
whether it shortens sleep latency and increases slow-wave sleep (SWS) duration. Sleep states of participants
were examined by EEG in three stimulus conditions: (i) δ-tempo; (ii) changing tempos from θ to δ at 60 min
after lights-out; (iii) no stimulus (control condition). Experiments were carried out in a soundproof room andstarted at 15:00. The stimulus began to be presented 1 min after lights-out and continued for 120 min.
Apparent changes of brainwave oscillation were observed after onset or change of each stimuli, however, nosignificant improvements were observed in the sleep latency and the SWS duration even in the conditions i
and ii. Time of the experiment may affect the results, however, further investigation is needed.
Keywords: Acoustic stimulation, Sleep, EEG

1. INTRODUCTION
Sleep disorders, including insomnia, cause a reduction in quality of life, which is thought to be a
problem around the world. People with sleep disorders have trouble falling asleep or getting restorable
and comfortable sleep. Pharmacological treatment is the most effective way, however, it has the riskof a decrease in effects caused by long-term use and an addiction, so developments ofnonpharmacological tools to improve sleep disturbance are needed.
Listening to sedative music is thought to be one of the useful way to improve sleep. Actually, thepositive effects of listening to music on subjective ratings of sleep have been reported by someresearchers (1, 2, 3). On the other hand, in contrast to the subjective ratings of sleep, consistent effects
on objective sleep quality measured by polysomnography have not been reported so far. Lazic andOgilvie (4) did not observe significant effects of music listening in physiological indices. Similarly,
Kuula et al. (5) did not find significantly positive effects of sedative music on sleep-quality indices;
sleep onset latency (the time needed to fall asleep after lights-off) and duration of deep sleep. Unlike
them, however, Chen et al. (6) reported that they observed positive effects of music on objective sleep
parameters: listening to music after falling asleep enhanced the duration of deep sleep only in a groupof participants with long sleep onset latencies. In Addition, Cordi et al. (7) showed that relaxing music
during the nap shortened the first stage of sleep and enhanced the duration of deep sleep and delta
band power of EEG in participants with low suggestibility.
According to a literature reviewing compiling hypotheses about mechanisms of how acoustic
stimulation such as music aids sleep, it is possible that the rhythmic structure or tempo of an acoustic
stimulus entrain the listener’s brainwaves (8). It is well known that the rhythm of brainwaves of†

shota813@chiba-u.jp, ‡ s-nakagawa@chiba-u.jp

animals and humans during sleep changes stepwisely: as stages of sleep become deeper, the rhythmslows down like α (8-13 Hz, awake) – θ (4-8 Hz, stage 1 and early stage 2) – δ (0.5-4 Hz, late stage 2
and stage 3 called slow-wave sleep: SWS). Hence, acoustic stimuli with tempos imitating brainwaves
during sleep may affect the sleep state and induce deep sleep when stimuli have slow tempos.
Although some researchers (4,8,9) examined effects of music with a slow tempo (< 4 Hz), none ofthem showed results which supported brainwave-entrainment hypotheses. However, music has a lotof acoustic features, for example, loudness, pitch, timbre, and so on. Besides, listener’s musicalpreference is also related to whether the listener can relax and sleep well with the music [10]. These
factors make it difficult to identify the mechanism of the way music help people slee p, therefore, it isneeded to exclude these factors except tempo from acoustic stimuli to examine the brainwaveentrainment hypotheses.
Ngo et al. (11) examined whether brainwaves can be induced by rhythmic pink noise bursts
fluctuating at 0.8 Hz. They showed that the stimulation significantly increased and entrained
endogenous slow oscillation of brainwaves but it delayed sleep onset latency. Schade et al. (12)assessed the ability of pink noise to elicit delta power and SWS. They reported that a pink noise
fluctuating at 0.8 Hz enhanced low-frequency power, however this effects appeared even when sounds
designed for disrupting sleep played. Considering these reports, a pink noise fluctuating at 0.8 Hz
seems to be an effective way to improve sleep, but its effects are not sufficient and the limited numberof the former studies have shown similar results.
In this study, we proposed a novel acoustic stimulus that stepwisely changes its tempo after sleep
onset as changes of endogenous brainwaves, from θ-tempo (4 Hz) to δ-tempo (0.8 Hz), and examined
whether it shortened sleep onset latency and enhanced SWS.

2. METHODS
2.1 Participants, experimental design and procedures
Participants were assigned and examined in three experimental conditions: (i) δ-tempo (4 males;mean age = 22.3 ± 0.415 years); (ii) θ to δ at 60 min after lights-out (at 15:00) (4 males and 2 females;mean age = 22.2 ± 0.344 years); (iii) no stimulation (control condition) (4 males and 1 female; mean
age = 22.4 ± 0.245 years). In each condition, participants were asked to sleep in a soundproo f room(see Figure 1). Previously, all participants were required to go to bed at 11:00 PM-0:00 AM and get
up at 7:00 AM-8:00 AM since two days before the experiment. Additionally, participants were notallowed to ingest alcohol and caffeine and take a vigorous exercise on the experiment day.
In the condition (i) and (ii), a pink noise burst was presented to the participants with a constanttempo of 0.8 Hz, stimulus onset asynchrony (SOA) of 1.25 s, which was equivalent to a frequency ofbrainwaves during SWS. In the condition (ii), a stimulus with a constant tempo of 4.0 Hz, which was
equivalent to a frequency of brainwaves during an early stage of sleep, was also presented. Acoustic
stimulus started 1 min after lights-out and continued for 120 min.

Figure 1 – Sleeping in the soundproof room.

2.2 Acoustic stimulation
The acoustic stimuli were bursts of pink noise of 50-ms duration, with a 5-ms rising- and fallingramps. Sound intensity was adjusted not to disturb each participant’s sleep before the sleep session
started using an attenuator (System 3, Tucker Davis Technologies). Stimuli were presented binaurally
via insert earphones (E-A-RTONE 3A, 3M).
2.3 Questionnaire
After the sleep experiments, participants were asked to answer a simple questionnaire to confirm
how they subjectively evaluated their sleep. The questionnaire consisted of following questions; [1]“Did you have a good sleep?”, [2] “Did you fall asleep early?”, [3] “Was there any factor to preventyou from sleeping?”. In the questions of [1] and [2], participants were required to answer in five-point
scale; (a) Agree (score: +5), (b) Slightly agree (score: +4), (c) Neutral (score: +3), (d) Slightly disagree(score: +2), or (e) Disagree (score: +1). The questions of [3] was open-ended.
2.4 Sleep recordings, scoring and EEG data analysis
EEG during sleep were recorded from single channel (C4 in the int’l 10-20 system) referenced to
the right ear-lobe (A2) using Ag-Cl electrodes and an EEG system (MEB-2312, Nihon Kohden).
Impedance of each channel was kept less than 5 kΩ. Signals were first analog-filtered between 0.1
and 50 Hz. All recordings were sampled at 400 Hz and stored in order to analyze offline. A horizontal
EOG was measured for detecting rapid-eye-movement (REM) sleep and artefact during experiments.
Offline scoring of sleep was done for 30-s epochs according to the standard criteria (13) by one ofexperimenter. Following parameters were used as indices for sleep quality; total sleep time (TST),
sleep latency (SL), and the percentage of the duration of SWS in TST (%SWS). TST was defined asa sum of the duration of each sleep stage. SL was defined as the interval in which stage 1 occurred at
first.
EEG power analyses based on fast Fourier transformations (FFT) were performed with MATLAB.
We applied a band-pass filter to EEG signal in range of 0.5-30 Hz. The data was segmented into 1600
data point wide epochs (4 seconds) with an overlap of 160 data points and applied Hamming-windowof 10%, then the FFT was calculated. We obtained the average power in each frequency band; delta(0.5-4 Hz), theta (4-8 Hz), and the total power (0.5-30 Hz).
2.5 Statistical analysis
Consecutive spectrum was segmented into epochs of 10 min and a n effect of the stimulus conditionon each index of sleep quality was analyzed by the one-way ANOVA in each segment using R software(14). A Bartlett-Test was conducted to check the homoscedasticity prior to the one-way ANOVA. Afterthe one-way ANOVA, a Bonferroni’s multiple comparison test was applied. The level of significancewas set to p = 0.05.

3. RESULTS
3.1 Sleep quality
No significant differences were found in either TST (F[2, 12] = 0.719, p = 0.507) or %SWS (F[2,12] = 2.579, p = 0.117) among three conditions. We found a significant difference only in SL (F[2,12]= 7.493, p = 0.00773, see Table 1) between condition (i) (δ-tempo only) and condition (ii) (θ→δtempo) (p = 0.00659) and between condition (i) and condition (iii) (control) (p = 0.0401, see Figure2). Although there were no significant differences, %SWS in condition (ii) showed the highest
percentage among three conditions. In contrast to the former report (11), stimuli with only δ tempo
did not increase the amount of SWS.

Table 1 – Sleep indices in three stimulation conditions.
δ-tempoθ→δ-tempo

Control(n = 6)(n = 6)(n = 5)

TST [min]

102 ± 3.09

110 ± 2.67

SL [min]15.4 ± 1.50

%SWS

9.80 ± 3.21

Sum SqMean Sq

F-value

P value

103 ± 8.23

201.2

100.6

0.719

0.507

5.58 ± 1.04

7.90 ± 2.53

238.6

119.32

7.493

0.00773

23.8 ± 5.29

17.1 ± 3.70

476.2

238.12

2.579

0.117

Figure 2 – Sleep indices in three stimulation conditions.
3.2 Spectral analyses
In order to verify effects of the tempo of acoustic stimuli on brainwaves during sleep, we compared
power spectrum of brainwaves related to sleep between three conditions. The total power (0.5-30 Hz)
did not differ significantly among conditions through 120 min sleep (see Figure 3). Also, the theta
power (4-8 Hz) did not show significant differences except the last 10 min (δ versus control; p < 2e16, θ→δ versus control; p < 2e-16). The theta power in condition (i) was slightly higher than the
power in condition (ii) in the first 60 min after lights-out although the difference was not significant(see Figure 4).

Figure 3 –Time courses of the total power in each condition (Mean ± SD).

Figure 4 – Time courses of the theta power in each condition (Mean ± SD).

Figure 5 – Time courses of the delta power in each condition (Mean ± SD).
The delta Power (0.5-4 Hz) significantly differ in some time segments, 20-30 min (δ versus control;p < 2e-16, θ→δ versus control; p < 2e-16) and 90-100 min (δ versus θ→δ; p < 2e-16, control versus
θ→δ; p < 2e-16). Through the experiment time, the power of the delta wave in condition (ii) was
higher than that in other conditions (see Figure 5).
3.3 Questionnaire
The Average scores of each question are shown in Figure 6. All participants filled out thequestionnaire about their sleep. In the question [1], “Did you have a good sleep?”, most of theparticipants in each condition evaluated their sleep affirmatively. There was no difference of theaverage score among the conditions, however, the average sc ore in condition (i) was the highest. In
the question [2], “Did you fall asleep early?”, the majority of the participants in each condition agreed.
Similar to the result of the question [1], the average score in condition (i) was the highest although
there was no difference. In addition, in the question [3], “Were there any factors to prevent you fromsleeping? Some answers saying “The mattress was not soft enough.”, “I got thirsty.”, or “The cablesof electrodes were annoying.” were found.

Figure 6 – The average scores for the questionnaire in each condition

4. Discussion
We proposed a novel acoustic stimulus that stepwisely changes its tempo after sleep onset as changes ofendogenous brainwaves, and examined its effects on the sleep onset latency and the SWS. Our results showed
there were some effects of the tempo of stimuli, however, no significant improvements of the sleep quality
were observed.
The stimuli with the only δ tempo (0.8 Hz) which former researchers used in their experiment (11)delayed SL compared to other conditions. This corresponds to the former report (11), however, thestimuli with only the only δ tempo did not increased the amount of SWS compared to the condition
with no stimuli. It is difficult to conclude the effects of the stimulus with θ tempo, this inconsistency
between the former report and ours may arise from the difference of the experiment time. Ngo et al.
examined the effects of the pink noise with 0.8-Hz tempo on sleep at night (11:00 PM), whereas our
experiments were conducted at daytime (3:00 PM). Melatonin, the one of sleep-promoting substances,increases at night based on circadian rhythm, so humans generally get sleepy at night. Therefore, it ispossible that enough amounts of SWS did not appear in the daytime sleep among the group ofparticipants in our study. Besides, it is said that the amount of SWS depends on the hours of advance
wakefulness (15); the more one spends time staying awake before going to bed, the deeper one cansleep. Considering this, it is difficult to deny the possibility that the experiment time affected theresults.
The stimulus we proposed in this study, the tempo changes from θ (4 Hz) to δ (0.8 Hz), shortened
SL and increased the amount of SWS compared to the control condition although the differences were
not significant. In our expectation, SL would be shortened when brainwaves during sleep entrain thetempo of the stimulus, however, there was no significant increase in the θ power caused by thestimulus with the θ tempo. It is difficult to give an explanation to this result, it might be that the θ
tempo is inappropriate for brainwaves to entrain. There has been no study examining the effects when
the stimulus with the θ tempo were presented during sleep, therefore further studies are needed.
By the open-ended questionnaire, some participants pointed that there were several defects in theexperiment environment. Since most of the participants experienced the sleep in the soundproof roomwith EEG recordings for the first time, they became nervous during the experiment. We may need tofamiliarize the participants with environments of the experiment.

5. CONCLUSIONS
Based on the brainwave-entrainment hypotheses, we proposed a novel acoustic stimulus withchanging the tempo from θ (4 Hz) to δ (0.8 Hz) and examined its effects on the daytime sleep.
Although the stimulus shortened SL and increased the amount of SWS compared to the control
condition, the θ power was not significantly increased by the stimulus with the θ tempo. In this study,
it is difficult to conclude the effects of the stimulus with the θ tempo, therefore, it may be necessarythat we conduct experiments using stimuli with the only θ tempo to examine its effects on sleep. In
addition, review of the experiment conditions and further investigation are needed.

ACKNOWLEDGEMENTS
This work was supported by JSPS KAKENHI Grant Numbers JP19K22950 and JP20H04497, anda Research Grant for the Next Generation Research Incubator from the Institute for Global Prominent
Research, Chiba University for SN.

REFERENCES
1. Jespersen KV et al. Music for insomnia in adults. Cochrane. Database. Syst. Rev. 2015;2015(8):CD010459.
2. De Niet G et al. Music-assisted relaxation to improve sleep quality: Meta-analysis. J. Adv. Nurs.
2009;65(7):1356-64.
3. Feng F et al. Can music improve sleep quality in adults with primary insomnia?. Int. J. Nurs. Stud.
2018;77:189-196.
4. Lazic SE and Ogilvie RD. Lack of efficacy of music to improve sleep: A polysomnographic andquantitative EEG analysis. Int. J. Psychophysiol. 2007;63(3):232-9.
5. Kuula L et al. The effects of presleep slow breathing and music listening on polysomnographic sleep
measures - a pilot trial. Sci. Rep. 2020;10(1):1-9.
6. Chen C-K et al. Sedative music facilitates deep sleep in young adults. J. Altern. Complement. Med.
2014;20(4):312-7.
7. Cordi MJ et al. Effects of relaxing music on healthy sleep. Sci. Rep. 2019;9:9079.
8. Dickson GT and Schubert E. How does music aid sleep ? Literature review. Sleep. Med. 2019;63:142150.
9. Picard LM et al. Music as a sleep aid in fibromyalgia. Pain. Res. Manag. 2014;19(2):97-101.
10. Chang ET et al. The effects of music on the sleep quality of adults with chronic insomnia using evidence
from polysomnographic and self-reported analysis: A randomized control trial. Int. J. Nurs Stud.
2012;49(8):921-30.
11. Ngo HVV et al. Induction of slow oscillations by rhythmic acoustic stimulation. J. Sleep. Res.
2013;22:22-31.
12. Schade MM et al. Enhancing slow oscillations and increasing N3 sleep proportion with supervised, nonphased-locked pink noise and other non-standard auditory stimulation during NREM sleep. Nat. Sci.
Sleep. 2020;12:411-29.
13. Rechtschaffen A and Kales A. A Manual of Standardized Terminology, Techniques and Scoring System
for Sleep Stages of Human Subjects. Gov. Printing Office, Washington DC;1968.
14. R Core Team (2020). R: A language and environment for statistical computing. R Foundation for
Statistical Computing, Vienna, Austria.
15. Knowles JB et al. Slow-wave sleep in daytime and nocturnal sleep: An estimate of the time course of“Process S”. J. Biol. Rhythms. 1986;1(4):303-8.

PROCEEDINGS of the24th International Congress on
Acoustics
October 24 to 28, 2022 in Gyeongju,
Korea

ABS-1012

A comparison of object-based and scene-based compression in virtual
reality
Joanna LUBERADZKA1,2, Christi MILLER1, Andy MUEHLHAUSEN1, Jeff CRUKLEY3,4, Melinda
ANDERSON1, Shaikat HOSSAIN1, Thomas LUNNER1
1
2

Conducted research during her internship at Reality Labs
3

4

Reality Labs, Meta, USA

Data Science and Statistics, Toronto, ON, Canada

Department of Psychology, Neuroscience, and Behavior, McMaster University, Canada

ABSTRACT

Despite technological advances, the main function of hearing aids remains soundamplification. To avoid overamplification, hearing aids use wide dynamic range compression(WDRC), which allows for controlling the amount of gain depending on the input soundlevel. As a side effect of these non-linear modifications, WDRC introduces undesired
distortions, especially when applied to sound mixtures. In this study, we introduce analternative approach, in which individual sound objects are separated prior to compression.
Although the potential benefit of such processing has been discussed previously, perceptual
evidence has not been investigated to date. We created a virtual reality (VR)-based listening
experiment in which conventional, scene-based WDRC is compared with the proposed objectbased WDRC, via measures of speech intelligibility, listening effort, comfort, and preference.
Acoustic scene analysis is designed to capture the known detrimental effects of conventional
WDRC: level fluctuations of the target signal interrupted by a competing masker; lack ofadequate target gain; when the target signal is buried in noise; and decreased long-term signalto-noise ratio (SNR) due to amplification of low-level background noise. Preference ratingsand objective intelligibility scores indicated benefit and motivate further development ofobject-based approaches.
Keywords: Hearing Correction, Wide Dynamic-Range Compression, Virtual Reality
1

INTRODUCTION

Hearing loss is one of the most prevalent chronic health conditions in older adults. The World
Health Organization estimates that by 2050 at least 700 million people will require hearing
rehabilitation (7). Apart from the obvious perceptual consequences, unaddressed hearing loss may
negatively impact cognitive, social, and economical aspects of life at the individual level. Althougha universal cure for hearing impairment does not yet exist, assistive listening devices havedemonstrated reduction of negative consequences (2, 26). Hearing aids help by capturing sound inlistener’s surroundings and delivering the amplified acoustic signal to their ears.

1

joa.luberadzka@gmail.com

Hearing loss leads to reduced audibility of soft sounds, but it does not lower the sensitivity to loudsounds. On the contrary, reduced dynamic range of audible sounds increases susceptibility to changes insound levels for people with hearing impairment (23). As a result, there are limits to the amount of
amplification a hearing aid can provide while avoiding excessive amplification.
Use of input-level-dependent amplification (gain) rather than a fixed gain for all sounds is one of thefundamental principles of modern hearing correction. Current hearing aids are typically equipped withdynamic range compression (WDRC), which automatically adjusts gain based on input signal level. Moregain is applied to low intensity sounds to improve audibility, and no or less gain is applied to high sounds
to prevent uncomfortable and painful presentation levels.
When low or high intensity sounds appear in isolation, the appropriate amount of amplification can beprovided with this approach. However, in many common acoustic environments, sound sources withdifferent intensities are present at the same time. In such listening scenarios, WDRC acts on the
superposed mixture signal captured by the microphone. This means that, most of the time, the function of
the compressor is triggered by the loudest sound source present in the scene and the gain determined forthis signal is applied to the whole mixture.
There are several detrimental effects related to applying dynamic range compression to the soundmixtures, which have been previously discussed in the literature ( 4, 5, 14, 21, 25). The first effect, here
referred to as the insufficient gain effect occurs when a signal of interest has a lower intensity than amasking signal. In this case, the masker triggers WDRC and the resulting gain is lower than the gain thatwould be triggered by the target signal in isolation (See Fig. 2.A.). As a result, the target signal is notprovided with a sufficient gain. The second effect, here called the aggravated long-term signal-to-noise
ratio (SNR) effect occurs in conditions with low-level background noise. When higher-level target signals
are not active, amplification is adjusted based on soft noise signals (See Fig. 2.B.). Even though the signalof interest is not influenced directly, the level of irrelevant information in the pauses is magnified, whichin the long term can make the listening situation more challenging. The third effect, here termed thefluctuating level effect occurs when a signal of interest is occasionally interrupted with a higher intensity
masker. The loud masker triggers the compressor which decreases the gain for the time during and shortly
after the masker’s activity. This causes fluctuations in the level of the target signal, which are especiallyperceivable for compressors with a long release-times as gain lags behind dynamic change of the envelopeand cuts the onsets of lower-level target signals (See Fig. 2.C.).
The above discussed effects are caused because in conventional scene-based amplification the gain isdetermined jointly for all the sound sources in the mixture. Recent technological advances in the field of
binaural source separation and spatial filtering (3, 8, 17) permit possibilities of alternative approaches inwhich the signals of individual sound objects are separated from the mixture prior to amplification (SeeFig.1). The potential of these approaches in limiting detrimental effects of compression has been
investigated previously. Corey and Singer (2017) showed that a multi-source compressor is lesssusceptible to noise and co-modulation effects than a conventional compressor. Despite encouraging
results of the objective evaluations, the perceptual consequences of the object-based compression scheme
have not been studied extensively so far.

Figure 1: Illustration of the difference between the conventional scene-based compression and the investigatedobject-based compression. Figure inspired by (6).
The validity of perceptual studies related to hearing aid benefit has been revisited in several recent
studies (12, 15, 22). The literature suggests consensus regarding the importance of addressing the aspect of
ecological validity when designing listening experiments in the lab. Emerging virtual reality technologies
are often mentioned as an approach to achieve a higher degree of realism in the lab (9, 11, 18). There are a
few examples of studies adopting virtual reality in listening tests with hearing aids (10, 19, 20, 24).
This study contributes to understanding the potential real-life benefits and limitations of object-based
amplification schemes in hearing devices. The aim of our listening experiments was to quantify the benefit

of object-based compression with respect to scene-based compression in a simulated virtual reality
environment. We assumed a perfect source separation and compared compression strategies in twolistening experiments performed by participants with hearing impairment. Both experiments were
performed in virtual reality using headphones, a head-mounted display, and contained stimuli for which
the levels were chosen to reflect the three detrimental effects of the scene-based method detailed above.
We measured objective intelligibility scores and subjective ratings of intelligibility, listening effort,
willingness to listen and preference.

Figure 2: Schematic representation of three detrimental effects thatmight occur when applying WDRC to sound mixtures.

2
2.1

METHODS
Experiment A

Procedure:
The first experiment was designed to measure objective intelligibility scores in an acousticallycontrolled audio-visual scene using a balanced speech corpus. The procedure consisted of a training block
followed by the experimental blocks. Each block consisted of 5 sentences from the Connected Speech Test
(CST) corpus. The CST test is used to measure intelligibility of everyday speech and is intended primarily forquantifying hearing aid benefit (27). The test consists of 48 lists of connected sentences, each passage concerninga familiar topic. Sentences in each list become gradually specific.
The task of the participant is to repeat each sentence. At the end of the block, there was a questionnaire
comprised of 3 questions: 1. How many words do you feel you recognized correctly?, 2. Approximately how long
would you be willing to sustain listening in this scenario? 3. How much effort did you have to use to understandthe sentences? Each question had 5 possible ordered responses (See Fig. 3.A.).
Intelligibility scores and self-reported ratings were collected in the following combinations of conditions: 2
WDRC strategies: object-based, scene-based, 2 interferer types: diffuse cafeteria background noise, competingtalker, 2 different SNRs: -5dB SNR, +10 dB SNR, and 2 visual representations of the target talker: with lipreading cues (video of a person), and without lip-reading cues (oval) (see Fig.3.B.) . In total there were 16
experimental conditions.
Conditions were randomized for each participant. We chose CST passages with average difficulty as
determined in a preliminary experiment. Sentences were randomized across experimental conditions. Total test
time was 15-20 minutes per participant.
Hypotheses:
The combinations of SNR and interferers were designed to reflect the 3 detrimental effects of the WDRC
applied to the mixture (See Fig. 3.C.). Competing talker at poor SNR represented the fluctuating level effect, inwhich the interferer triggers compression an causes distortion in the lower-level target signal. In thiscondition we expected the object-based compression to provide benefit. Competing talker at good SNR
represents the fluctuating level effect, however here the situation is flipped: the target signal triggers thecompression and causes distortions in the interferer. In this condition we expected the object-based WDRC
to lead to the same or worse results as the scene-based compression. Diffuse interferer at poor SNR
represents the insufficient gain effect. We expected object-based WDRC to provide benefit as it applied
more gain to the target signal than the scene-based WDRC. Diffuse interferer at good SNR represented theaggravated long-term SNR effect. We expected object-based compression to provide benefit as it did notamplify background noise in speech pauses. We additionally expected that the visual condition without lipreading cues would demonstrate benefit of object-based compression.

2.2

Figure 3: Experiment A: Speech intelligibility and subjective ratings with CST speech corpus.

Figure 4: Experiment B: Preference rating with natural conversations.
Experiment B
Procedure:
The second experiment was designed to capture the preference of the participants in a less acousticallycontrolled audio-visual scene with recorded natural conversations. The procedure consisted of a training
scene, followed by 3 test scenes with simultaneous conversations. In each scene, participants were
instructed to follow one conversation. Their task was to switch between two signal processing options (Aand B) while following the conversation and when ready, submit their preference using a slider (See Fig.
4.A.). The slider offered discrete responses on a Likert scale. There were 3 test scenes, which differed in
the arrangement of the target versus masker conversations: Target Right Soft, Target Front Soft, Target
Front Loud (See Fig. 4.B.). After each scene there was a control multiple-choice question about the topic
of the followed conversation. The level ratio between target and masker conversation was varied byplacing them in different spatial locations of the scene. Each scene contained looped speech material of
approximately 1-2 minutes in duration. A low-level cafeteria background noise was present in all scenes.
Hypotheses:
The scenes were designed to reflect the 3 detrimental effects of the WDRC applied to the mixture (See Fig.
4.C.). However, in contrast to Experiment A, in Experiment B we did not create a separate condition to reflect
each effect. We were rather interested in measuring a cumulative influence of all potential effects on participants’
preferences. In general, the Target Right Front scene and the Target Front Soft scene reflected a condition
with a competing interferer at poor SNR, in which the fluctuating level effect is likely to occur, thus we
expected preference for object-based WDRC. At times when the louder interfering conversation was verybusy, we could have observed the insufficient gain effect in the scene-based WDRC, which could shift
participants’ preferences towards object-based WDRC. The Target Front Loud scenes reflects the condition
with a competing interferer at good SNR, in which we expected the preference for the scene-based WDRC.
All scenes contained low-level background noise responsible for the aggravated long-term SNR effect,
which could shift preferences in favor of object-based WDRC.
2.3

Participants and WDRC presets

The amount of amplification in the WDRC had to be matched with the degree of hearing loss. To achievethis, we created 4 gain presets: N0, N1, N2, and N3. They were based on 4 standard audiograms (1) andthe gain characteristics in each category were computed with the NAL-N2 procedure (16). A total of 27
participants with varying degrees of symmetrical hearing impairment were invited to take part in the study.
participants were recruited using an online questionnaire. An audiogram was measured for each participant
the day before the experiment. Based on the average hearing threshold from 1, 2, and 4 kHz, eachparticipant was assigned to one gain preset group (<18=N0,18-28=N1,29-40=N2,41-60=N3). There were 2
participants in the N0 group, 15 participants in the N1 group, 3 participants in the N2 group, and 7
participants in the N3 group. In all groups, attack time was set to 10 ms and release time to 200 ms.
Participants who used hearing aids were asked to remove them prior to the experiment.
2.4

Setup and equipment

Both experiments were performed using the Oculus Quest Headset connected with the Bayerdynamic
DT 770 Pro Headphones. The audiovisual restaurant scene was created using Unity game engine. We used
the Unity Experiment Framework open source package for data logging and control of the experimental
procedure. For acoustic simulations we used the Oculus Spatializer Plugin for Unity, which implements
head-movement dependent distance attenuation and HRTF-based spatialization. Room acoustics modelingwas not included in the simulations. The recorded cafeteria noise was added in a first order ambisonics Bformat. The two WDRC methods were implemented by importing the same C++-based DSP plugin atdifferent sound processing stages of the Unity scene (See Fig. 5). For object-based WDRC, the plugin wasapplied at the level of the source, before spatialization; for scene-based WDRC, the plugin was applied at
the level of the listener, after the mixing of the spatialized signals. Compression parameters were identicalfor both ears.

Figure 5: Setup (Oculus Quest 2 + headphones) and signal
processing flow in Unity for object- and scene-based WDRC.
In Experiment A the playback system was calibrated using the KEMAR head and torso simulator. We
determined the digital level of speech-shaped noise played back with the Quest 2 playback system at
maximum volume which corresponds to 70 dB SPL headphone output. Speech-shaped noise was generatedusing the CST speech corpus and the physical level was computed by comparing the measured output withthe 1 kHz Bruel&Kjar calibrator. The target level was kept constant and the level of the competing talker
and diffuse noise maskers were varied to achieve the required SNR at the virtual listener.
In Experiment B the digital levels were adjusted manually at the maximum system volume to achieve
naturally sounding scenes.

3

RESULTS

3.1

Experiment A
Scoring was conducted by counting the number of correctly identified words out of the total possible
target words per sentence. Since words can only be correct or incorrect and the scoring is count-based, abinomial model was appropriate for analyzing the resulting data. The results of experiment A wereanalyzed using a hierarchical binomial model, which estimates the probability of correct word
identification, under a Bayesian framework. Self-report ratings were collected with a 5-point ordinal
response scale, so outcome data were best considered and modeled as ordered categorical (ordinal)
responses. For these data, we constructed a hierarchical ordinal multiple regression model under aBayesian framework.
Figure 6 presents the results for two interferer types at two SNRs for the WDRC conditions. We
observed the expected tendency demonstrating benefit of object-based WDRC for most gain preset groups
under competing talker and poor SNR conditions. However, significant performance differences were only
observed for the N2 group without lip-reading cues. Object-based WDRC provided more benefit when no
lip-reading cues available. There were no significant performance differences between object- and scenebased WDRC under competing talker and positive SNR conditions; most of the predicted responses wereclose to 100 %, which suggests participants encountered ceiling effects caused by insufficient difficulty inthe speech stimuli at +10 dB SNR. Object-based WDRC tended toward to higher intelligibility for the N3
group, which conflicts with our initial hypothesis for this condition.
Our expectations were confirmed in the diffuse noise condition at poor SNR. We observed a significant
benefit of object-based WDRC in 3 participant groups (N0, N2, N3). Although the tendency is visible forall groups, the results for the largest group of listeners (N1) were not considered significant.
Results for the diffuse interferer at high SNR were influenced by ceiling effects. The task in thiscondition may have been too easy to measure differences between WDRC strategies. Additionally,
tendencies confirming the influence of SNR, and greater difficulty for the competing talker, and greaterdifficulty without lip-reading cues was observed. In contrast to the objective intelligibility scores, no clear
tendencies confirming the advantage of object-based WDRC was observed in self-reported ratings.

Figure 6: Results of intelligibility scoring in Experiment A. Points represent
median probability of correct word recognition and error bars represent the
89% highest density credible interval surrounding the median values.
Statistically significant differences between Scene and Object are marked withasterisks. Symbol of lips represents conditions with video of a person in which
lip reading was possible.
3.2

Experiment B
Preference ratings in Experiment B were collected with a 5-point Likert response scale. A hierarchicalordinal multiple regression model under a Bayesian framework was used. The upper panel of Figure 7
plots raw counts of responses together with the predicted response counts from the model. The lower panel

of Figure 7 plots aggregated responses, computed as a weighted sum of individual response probabilities,
which can be interpreted as a measure of central tendency (similar to mean scores).
Object-based WDRC was slightly preferred by participants in the N1 and N2 groups. Participants inthese groups preferred object-based WDRC in all scenes, but particularly in scenes where the interfering
conversation was louder than the target conversation.

Figure 7: Results of preference rating in Experiment B. Upper
panel plots counts of responses together with the predicted
response counts from the model. Bars represent the observed
quantity of each response category as a function of Scene. Points
depict the median posterior predicted count and error bars depict
the 89% highest density credible interval surrounding the medianpredicted counts. Lower panel plots aggregate ratings for eachscene, which can be interpreted as mean scores. Points represent
the median rating, error bars depict the 89% highest density
credible interval.

DISCUSSION
The goal of the experiments presented in this study was to quantify the potential benefit of object-based
WDRC. Objective scores in Experiment A showed promising results in favor of object-based WDRC.
Results indicated that object-based WDRC improved speech intelligibility, especially in poor SNR, diffuse
noise, and when visual cues were not present. Subjective ratings in Experiment B confirmed thisadvantage. In the Experiment B paradigm with a more realistic sound material and without lip-readingcues N1 and N2 participants tended to prefer object-based WDRC, especially in poor SNR. Conversely,
subjective ratings in Experiment A did not align intelligibility scores. Participants reported no changes in
intelligibility, listening effort, or willingness to listen. It is possible that the ratings failed to capture thedifferences between the methods because listeners rated the sound material after each block of 5 sentences
independently. No direct comparisons between methods were made and participants may have changed
their rating criteria throughout the experiment, depending on the topic of the list, for example. The
advantage of object-based WDRC was not demonstrated in preference ratings in the N3 group. A larger
effect was expected in this group because amplification had the strongest effect on the signal (this group
had the most gain). The preference task required participants to simultaneously follow the conversation,
switch between the signal processing strategies, and rate the sound material. Several participants foundthis task confusing. An additional aspect was the confusion of the spatial location of the sound sources
with the spatial location of A/B switches in the user inferface. Another possible explanation is that N3
group participants found following the target conversation too difficult with either WDRC scheme, so they
had no preference.
Our paradigm had several limitations. An important factor in the study design is the choice of the gainpresets. Although gain presets were computed based on audiograms representing the most common hearing
loss types, no personalization in the gain/frequency responses was used in these experiments. The degree
to which individuals’ hearing impairments were matched with the prescribed gain preset likely varied
between participants bringing an additional source of variability in the data. Another limitation lies in thegain calculation methods between schemes. Object-based WDRC is computed based on the level of thesignal at the source whereas in the gain of scene-based WDRC is computed based on the signal level at thelistener. This could lead to lower gains applied in the object-based method and hence, an overall better
audibility in the scene-based method. In this study we assumed perfect source separation prior tocompression. In real-world scenarios the quality of source separation would certainly influence benefit
from the object-based WDRC. Each source separation method may introduce specific artifacts, which
could interact with non-linear amplification and should be investigated in further research.
Despite advances in technology and medicine, sound amplification remains the main strategy to address
hearing loss. Limitations of conventional scene-based processing are likely one of the reasons why people
with hearing difficulties leave their hearing aids at home. Although object-based amplification alone does
not solve the dilemma of only amplifying desired sounds, the data presented in this work suggest that it
can still provide benefit. While the results in diffuse noise reflect benefit from denoising (or the lack ofnoise amplification) and can be understood as a demonstration of the commonly known disadvantages ofthe scene-based processing, the results in competing talker conditions demonstrate benefit from objectbased processing. Listening experience can potentially be improved by applying amplification separately
to each source through object-based processing without knowledge of the desired source (intended
speaker). These results can motivate development of object-based approaches prior to solving the complex
desired-source detection problem.

ACKNOWLEDGEMENTS
Research presented in this study was supported by Meta and conducted at Reality Labs Audio in
Redmond as a part of an internship project of Joanna Luberadzka.

REFERENCES
1. Bisgaard N, Vlaming MS, Dahlquist M. Standard audiograms for the IEC 60118-15 measurement procedure. Trends inamplification. 2010 Jun;14(2):113-20.
2. Chisolm TH, Johnson CE, Danhauer JL, Portz LJ, Abrams HB, Lesner S, McCarthy PA, Newman CW. A systematic
review of health-related quality of life and hearing aids: final report of the American Academy of Audiology Task Force on
the Health-Related Quality of Life Benefits of Amplification in Adults. Journal of the American Academy of Audiology.
2007 Feb;18(02):151-83.

3. Cobos M, Ahrens J, Kowalczyk K, Politis A. An overview of machine learning and other data-based methods for spatial
audio capture, processing, and reproduction. EURASIP Journal on Audio, Speech, and Music Processing. 2022
Dec;2022(1):1-21.
4. Corey RM, Singer AC. Modeling the effects of dynamic range compression on signals in noise. The Journal of the
Acoustical Society of America. 2021 Jul 9;150(1):159-70.
5. Corey RM, Singer AC. Dynamic range compression for noisy mixtures using source separation and beamforming. In
2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) 2017 Oct 15 (pp. 289-293).
IEEE.
6. Corey RM, Blog post: https://publish.illinois.edu/augmentedlistening/dynamic-range-compression-and-noise/
7. Davis AC, Hoffman HJ. Hearing loss: rising prevalence and impact. Bulletin of the World Health Organization. 2019 Oct
10;97(10):646.
8. Derleth P, Georganti E, Latzel M, Courtois G, Hofbauer M, Raether J, Kuehnel V. Binaural Signal Processing in Hearing
Aids. InSeminars in Hearing 2021 Aug (Vol. 42, No. 03, pp. 206-223). Thieme Medical Publishers, Inc..
9. Grimm G, Luberadzka J, Herzke T, Hohmann V. Toolbox for acoustic scene creation and rendering (TASCAR): Render
methods and research applications. InProceedings of the Linux Audio Conference 2015 (pp. 9-12).
10. Hendrikse MM, Eichler T, Hohmann V, Grimm G. Self-motion with Hearing Impairment and (Directional) Hearing
Aids. Trends in Hearing. 2022 Mar;26:23312165221078707.
11. Hohmann V, Paluch R, Krueger M, Meis M, Grimm G. The virtual reality lab: realization and application of virtual
sound environments. Ear and Hearing. 2020 Nov;41(Suppl 1):31S.
12. Holube, Inga, Petra von Gablenz, and Jörg Bitzer. "Ecological momentary assessment in hearing research: Current
state, challenges, and future directions." Ear and Hearing 41 (2020): 79S-90S.
13. Johnson CE, Jilla AM, Danhauer JL, Sullivan JC, Sanchez KR. Benefits from, satisfaction with, and self efficacy foradvanced digital hearing aids in users with mild sensorineural hearing loss. InSeminars in hearing 2018 May (Vol. 39, No.
02, pp. 158-171). Thieme Medical Publishers.
14. Kates JM. Understanding compression: Modeling the effects of dynamic-range compression in hearing aids.
International journal of audiology. 2010 Jan 1;49(6):395-409.
15. Keidser G, Naylor G, Brungart DS, Caduff A, Campos J, Carlile S, Carpenter MG, Grimm G, Hohmann V, Holube I,
Launer S. The quest for ecological validity in hearing science: What it is, why it matters, and how to advance it. Ear andhearing. 2020 Nov;41(Suppl 1):5S.
16. Keidser G, Dillon H, Flax M, Ching T, Brewer S. The NAL-NL2 prescription procedure. Audiology research. 2011
Mar;1(1):88-90.
17. Kollmeier B, Kiessling J. Functionality of hearing aids: State-of-the-art and future model-based solutions. International
journal of audiology. 2018 May 25;57(sup3):S3-28.
18. Llorach G, Grimm G, Hendrikse MM, Hohmann V. Towards realistic immersive audiovisual simulations for hearing
research: Capture, virtual scenes and reproduction. InProceedings of the 2018 Workshop on Audio-Visual Scene 10.
Understanding for Immersive Multimedia 2018 Oct 26 (pp. 33-40).
19. Llorach G, Hendrikse MM, Grimm G, Hohmann V. Comparison of a Head-Mounted Display and a Curved Screen in a
Multi-Talker Audiovisual Listening Task. arXiv preprint arXiv:2004.01451. 2020 Apr 3.
20. Mansour N, Marschall M, Westermann A, May T, Dau T. The effect of hearing aid dynamic range compression on
speech intelligibility in a realistic virtual sound environment. The Journal of the Acoustical Society of America. 2022 Jan
13;151(1):232-41.
21. May T, Kowalewski B, Dau T. Signal-to-noise-ratio-aware dynamic range compression in hearing aids. Trends inhearing. 2018 Aug;22:2331216518790903.
22. Mehra R, Brimijoin O, Robinson P, Lunner T. Potential of augmented reality platforms to improve individual hearing
aids and to support more ecologically valid research. Ear and hearing. 2020 Nov;41(Suppl 1):140S.
23. Marozeau J, Florentine M. Loudness growth in individual listeners with hearing losses: A review. The Journal of the
Acoustical Society of America. 2007 Sep 21;122(3):EL81-7.
24. Seol HY, Kang S, Lim J, Hong SH, Moon IJ. Feasibility of virtual reality audiological testing: prospective study. JMIR
Serious Games. 2021 Aug 31;9(3):e26976.
25. Stone MA, Moore BC. Side effects of fast-acting dynamic range compression that affect intelligibility in a competingspeech task. The Journal of the Acoustical Society of America. 2004 Oct;116(4):2311-23.
26. von Gablenz P, Kowalk U, Bitzer J, Meis M, Holube I. Individual hearing aid benefit: Ecological momentary
assessment of hearing abilities. InProceedings of the International Symposium on Auditory and Audiological Research2019 (Vol. 7, pp. 213-220).
27. Cox RM, Alexander GC, Gilmore C. Development of the connected speech test (CST). Ear and Hearing. 1987 Oct
1;8(5):119S-26S.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0021

Haptic music players for children with profound and multiple
learning disabilities (PMLD) – Exploring different modes ofinteraction for felt soundEmma FRID12; Claudio PANARIELLO3;
1

EECS School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Sweden;
2

STMS–UMR9912 Sciences et Technologies de la Musique et du Son, IRCAM Institut de Recherche et
Coordination Acoustique/Musique, France

3

EECS School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Sweden;

ABSTRACTThis paper presents a six-month exploratory case study on the evaluation of three Haptic Music Players(HMPs) with four pre-verbal children with Profound and Multiple Learning Disabilities (PMLD). The
evaluated HMPs were 1) a commercially available haptic pillow, 2) a haptic device embedded in a modified
plush-toy backpack, and 3) a custom-built plush toy with a built-in speaker and tactile shaker. We evaluatedthe HMPs through qualitative interviews with a teacher who served as a proxy for the preverbal children
participating in the study; the teacher augmented the students’ communication by reporting observations fromeach test session. The interviews explored functionality, accessibility, versus user experience aspects ofrespective HMP and revealed significant differences between devices. Our findings highlighted the influenceof physical affordances provided by the HMP designs and the importance of a playful design in this context.
Results suggested that sufficient time should be allocated to HMP familiarization prior to any evaluation
procedure, since experiencing musical haptics through objects is a novel experience that might require sometime to get used to. We discuss design considerations for Haptic Music Players and provide suggestions for
future developments of multimodal systems dedicated to enhancing music listening in special education
settings.
Keywords: tactile sound, haptics, accessibility

1. INTRODUCTION
Listening to music is a multisensory experience; musical experiences involve both perceiving
airborne acoustic waves and vibratory cues conveyed through air and solid media (1). Musical Haptics
is an emerging interdisciplinary field that focuses on investigating such experiences of touch andproprioception in music scenarios (1). Playing a musical instrument requires a complex set of skills
that depend on the brain’s ability to integrate information from multiple senses (2). Acoustic
instruments intrinsically provide vibratory feedback during sound production and these vibrations are
utilized for self-monitoring in acoustic performance, see e.g. (3). The haptic sense has been shown toplay an important role in musical interactions with acoustic instruments such as the piano (4–6), theviolin (7,8), and percussion (9). Research has also suggested that vibrations play a significant role in
the perception in music; amplifying certain vibrations in a concert venue or music production system
can improve the musical experience overall (10,11).
The current study focuses on Haptic Music Players (HMPs), i.e., devices that can process an audio
signal to extract musical information and translate the information into a vibratory signal. HMPs canbe considered sensory substitution systems, i.e., systems that translate sensory information that isnormally available via one sense to another (12), in this case from hearing to touch. HMPs allow forenhancement of music listening experiences through simultaneous playback of haptic feedback; haptic
signals passed through a HMP system are put in contact with a users’ skin and body to enrich music
emmafrid@kth.seclaudiop@kth.se1
3

activities. A review of Haptic Music Player technology was recently published in (13). Examples of
HMPs include fixed setups or installations such as haptic chairs (14) and platforms (15); wearable
systems covering skin surface such as bracelets (16), gloves (17), belts (18), or whole-body suits (19);systems intended to be held in the hands (20); and more complex setups combining several of above
mentioned mechanisms (21). Examples of commercially available HMPs include the Ultrasonic Audio
Syntac 4 , the SubPac 5 and the Soundbox and MiniBox from Soundbeam 6 . In the review of HMPs
published in (13), the authors describe four main components that are necessary to provide vibrotactile
rendering of musical elements in HMPs: audio signaling processing software, a digital to analog
converter (DAC), amplifier(s), and actuator(s). Crossmodal mapping techniques, which involve
identifying sound descriptors to be mapped to properties of vibrotactile feedback (22), can be used totranslate elements of music into coherent haptic signals. Knowledge about the psychophysicalsimilarities and differences of the auditory and tactile modality can help developing perceptually
optimized algorithms for such music-related vibration generation (11). Examples of methods used fora haptic chair described in (11) were low-pass filtering, reduction to fundamental frequency, octave
shifting, substitution of signals, and compression of dynamic range of an audio source. Results froman experiment revealed a general connection between vibrations and the perceived quality of the music
reproduction, but that it might not be necessary to code all available auditory information into thetactile channel to improve the perceived quality of music.
In this paper we present a six-month study exploring different types of Haptic Music Players(HMPs) for pre-verbal children with Profound and Multiple Learning Disabilities (PMLD). Pre-verbal
children do not yet have verbal communication skills. The term PMLD is commonly used to describe
a person with severe learning disabilities who most likely has other complex disabilities and healthconditions, although there is no single universally agreed definition of the term (23). Children withPMLD may express themselves through several different communication methods depending on what
is most efficient for them at the time. Examples include bodily gestures, nonverbal sounds, pointing,and facial expressions. Previous research focused on the role of music for children with PMLD hasemphasized the importance of music education (24) and musical play (25). A study exploring the roleof music within the home-lives of young people with PMLD through parental perspectives waspresented in (26). Findings outlined the positive role of music in contexts where music was used forenjoyment, to support mood-regulation, and to add structure to the lives of the children. As pointed
out in (1), musical haptics could potentially facilitate access to music for persons affected bysomatosensory, visual, and hearing impairments. The potential of multisensory experiences to enrich
the lives of people with profound and multiple disabilities has been stressed (27). However, few
studies have yet explored the role of haptic feedback for people with multiple disabilities, especiallyin the context of music (see e.g. 28,29).
The goal of the work presented in this paper was to allow for rich multisensory experiences andmultiple modes of interaction for users with various abilities and needs, as suggested in (30). The
work is part of a larger research project aimed to improve access to music-making through thedevelopment of novel accessible multimodal music interfaces, see (31). As suggested in (32), designof technology should make sure that users have control of, and not only are passive recipients of,developed technology. The disability rights movement motto "Nothing about us without us" (33)highlights that people with disabilities know what is best for them and their communities. In our
research we have tried to include the children as informants in the design process using user-centered(34) and participatory (35) methods. Participatory development processes should enhance the voiceof the participants, considering their ideas, desires, and needs (36). Existing co-design methods mayneed to be adapted to fully incorporate users with disability into user-centered design processes (37).
Such adaptations are particularly important when including children with PMLD in design andevaluation processes since the communication between designers and the users can be affected.
Alternative methods for augmented communication, such as PODD (Pragmatic Organisation
Dynamic Display 7), a way of organizing word and symbol vocabulary in a communication book to
support understanding (38), can be used to support design and evaluation processes with children with
PMLD. In the study described in this paper, we used Participatory Design with Proxies (PDwP) (39)https://ultrasonic-audio.com/products/syntact-touch-the-sound/
https://subpac.com/
6 https://www.soundbeam.co.uk/vibroacoustics
7 https://novitatech.com.au/podd-communication-books/
45

to enable inclusion of input from the pre-verbal students in the evaluation process. Since the students
participating in the study are pre-verbal, we could not ask them directly about their haptic experiences.
To get an understanding of how the HMPs were perceived, the responsible teacher and the teachingassistants at the school communicated directly with the students using alternative methods foraugmented communication, such as PODD picture cards. In this way, the teachers acted as proxies,
augmenting the direct input from the students taking part in the study.
In the following sections we describe a case study exploring how music can be presented not onlyin an audible form, but also as haptic sensations, through three different Haptic Music Players used
in a special education setting. This paper contributes to the field of Musical Haptics by exploring how
HMPs could be designed to be accessible to a group of pre-verbal children with multifunctional
physical and intellectual challenges. To our knowledge, little prior work has explored design
considerations for HMPs for pre-verbal children in special education settings.

2. METHODSThis paper describes a long-term collaboration with a teacher and a student group at the school
Dibber Rullen 8. Dibber Rullen is a special education school for students with intellectual challenges
and multifunctional physical challenges. The study was carried out during the COVID-19 pandemic.
This greatly affected the choice of methods, as well as the extent to which we could interact with thestudent group. Social-distancing measures were required to ensure the safety of the children, since
they are a risk group. Since we could not meet the students physically, we adapted our methods to a
distributed participatory design setting (40) in which the design team members were physicallydispersed, and the HMPs were tested with the help from a teacher and teaching assistants at the school,in the wild (41). All research described in this paper was carried out as activities within the school’s
standard curriculum.
2.1 Aim
The purpose of this case study was to explore design considerations of Haptic Music Players for agroup of children with PMLD. More specifically, the aim of the study was to allow for exploration ofmusical haptics using customized devices that could be tested by the students in a safe and familiar
environment, without leaving the educational setting at the school 9.
2.2 User GroupWe collaborated with a student group consisting of four children (2F, 2M, age 9-15). The students
are mostly pre-verbal, have multifunctional physical challenges, varying motor skills, use wheelchairs,
and have moderate to severe intellectual challenges. A questionnaire was initially sent to the students’parents to get a better understanding of the children’s challenges and musical background 10. The
parents of S1 (Student 1) reported that she is hard of hearing, has a visual impairment, a chromosome
aberration, reduced mobility, a physical disability, and hypermobility in the joints. The parents of S2reported that she has a visual impairment and that she can lift her arms and grab objects. It is unclear
if she has a hearing impairment. The parents of S3 described that they didn’t know if S3 is hard ofhearing and has a visual impairment. He cannot walk but he can jump when sitting. The parents of S4reported no hearing loss or visual impairment for S4. None of the parents reported that their childcould play a musical instrument or sing. Only one of the students (S3) had tested electronic musicmaking tools or music technology. When asked to rate the level of musical interest (where 0
corresponded to strongly disagreeing, and 10 corresponded to strongly agreeing that the child wasinterested in music), the parents of S1 and S2 reported a level 4 interest, S3 a level 10, and S4 a level
8. Based on the questionnaire results we concluded that the students had very diverse taste in musicoverall, spanning across many different music genres.
2.3 Design Process
To inform the design of the HMPs and define a set of design constraints, we first organized ameeting with the teacher who worked with music and dance classes at the school. In this in-person
https://dibber.se/skola/rullens-sarskola/
Because of the COVID-19 pandemic, the students could not visit any external multisensory rooms/music
installations.
10 The questionnaire is available as supplementary material: https://doi.org/10.5281/zenodo.6977775
89

meeting, we demonstrated the use of two haptic devices, the HUMU Augmented Audio Cushion 11 (seeFigure 1) and the Woojer Strap Edge 12 (see Figure 2). The teacher got the chance to test the devices
on different body parts (the Woojer Strap Edge is intended to be worn on the hips, chest, or crossbody). We played different music examples through the devices and discussed appropriateness of thetools for the specific user group. In particular, we focused on how the HMPs should be designed tobest take the physical challenges of the students into account, and which modes of interaction (i.e.,
bodily locations) that might work well for them. The discussion was recorded and manuallytranscribed by author 1. Both authors analyzed the transcription, summarizing the findings into a setof design constraints, presented in Section 3.1.1. Conclusions from this analysis guided the design of
two Haptic Music Players (HMP 2 and 3, see Section 3.1.3 versus 3.1.4 for detailed descriptions
respective device). We concluded that the HUMU Augmented Audio Cushion (from this point referred
to as HMP 1, see Section 3.1.2) could be used without any modification, since it to already fulfilled
the defined design constraints. To summarise, the three final HMPs used in the study were HMP 1,
the HUMU pillow; HMP 2, a haptic device embedded in a modified plush-toy backpack; and HMP 3:
a custom-built plush toy with a built-in speaker and tactile shaker. The construction of HMP 3 took
longest time since it was built completely from scratch. Once the HMPs were ready, we confirmedwith the teacher that the visual appearance of the devices would work well for the students (i.e., thatthey would not scare the children). This was followed by the evaluation sessions discussed in Section
2.4. The evaluation sessions prompted a re-design of HMP 2. This HMP was thus tested in two
different iterations. The teacher provided valuable suggestions regarding how HMP 2 could be
improved in a follow-up meeting held after the first evaluation session.
2.4 Evaluation
Since we could not attend the HMP test sessions with the students in person, we asked the teacher
to film the interactions taking place at the school. The teacher had six months to test out the HMPs
with the students. Each HMP was tested a minimum of two times. Apart from the responsible teacher,teaching assistants were also present during the test sessions. To support the teacher and teachingassistants in their use of the HMPs, we provided instruction manuals with links to instructional videos
for all devices. The responsible teacher decided which music that would be used during the evaluation
sessions. To be sure that music with low frequency content was also tested, we provided a soundexample with a broad frequency range 13. The PODD tool was used to communicate with the students
during the evaluation sessions. The teacher and teaching assistants used PODD to ask 1) if/what thestudents liked versus disliked about respective device, and 2) with which body they wanted to feelvibrations. The students had previously practiced using PODD picture cards focused on descriptionsof musical haptics and haptic sensations in different body parts, within the scope of the larger study
described in (42). When possible, the responsible teacher filmed the interactions taking place during
the evaluation sessions using a mobile phone, and annotated observations. The authors watched thesevideos, taking notes of the interactions taking place, as well as the PODD discussions.
We carried out three semi-structured interviews with the responsible teacher to discuss what wasobserved during the evaluation sessions, using a video conferencing tool. The interviews started withgeneral questions focusing on: the music used when testing; how many times the HMPs were used
and the time spent exploring respective HMP; which HMP that worked best/worst, i.e. which HMP
that was most/least preferred by the students; if the vibrations provided by the HMPs enhanced hearing(and if so, how); and the balance between the sounds that you could hear versus feel. This wasfollowed by questions focusing on three different themes: functionality, accessibility, and user
experience, broadly inspired by evaluation methods used in Human Computer Interaction (43,44) andDigital Musical Instrument research (45,46). The functionality theme focused on how well versus notwell the HMPs worked, overall. This involved, for example, performance in terms of audio quality,
reliability, durability, and if certain functions were lacking. The accessibility theme focused onachieving an identified goal in the identified context of use. More specifically, accessibility in thiscontext was conceptualized as usability for a population with the widest range of user needs,
characteristics, and capabilities, see ISO 9241-171:2008 14, which fits within the universal design or
https://humu.fi/
https://www.woojer.com/products/strap
13 This sound file is available as supplementary material: https://doi.org/10.5281/zenodo.6977775
14 https://www.iso.org/standard/39080.html
1112

design for all 15 philosophies. This category focused on questions based on the seven design principles
of universal design (47), i.e. equitable use, flexibility in use (e.g., the preferred bodily locations used
for the three HMPs), simple and intuitive use, perceptible information (e.g., if the HMPs
communicated musical information effectively, regardless of the sensory abilities of respective
student), tolerance for error, low physical effort (if the use of the HMPs resulted in fatigue), and sizeand space for approach and use. Finally, the user experience theme focused on the overall experienceof interacting with the HMPs and the emotional reactions of the students. For this category, questions
were based on the User Experience Questionnaire 16, focusing on concepts such as attractiveness (ifthe students liked the look and feel of the HMPs), perspicuity (if it was easy to get familiar with the
HMPs and learn how to use them), efficiency (if the HMPs could be used in an efficient and pragmaticway), dependability (if the behaviors of the HMPs met the students expectations and if the students
appeared to feel safe when using the HMPs), stimulation (if it was fun and motivating to use the HMPs,
and if any emotions could be identified among the students), and novelty (if the HMPs caught theinterest of the students and if they were creatively designed).
Two of the interviews were done when the HMPs had been tested for three months. The thirdinterview was carried out after six months, when the teacher had had time to try out a new iterationof HMP 2. Mean interview duration was 50 minutes (min 33 min, max 79 min). All interviews weremanually transcribed by author 1. Transcriptions were analyzed using content analysis (48) to identify
different types of reactions to the HMPs across sessions. Themes were included if more than seveninstances (quotes) were identified for a specific theme. All quotes were also subdivided into
accessibility, usability, and user experience themes for respective HMP. The results are presented in
Section 3.2.
2.5 Ethics Statement
This research was reviewed by the Swedish Ethical Review Authority (application No 2021-0630701). The study was carried out in accordance with the declaration of Helsinki. In this work, we use
people/person first language (PFL) when writing about disability, as opposed to identity-first language(IFL), see (49). The parents and the teacher gave written informed consent before participation andagreed to the data being collected as described in the consent form. The teacher and teaching assistants
used PODD to make sure that all students gave consent to participate at all times. Management of
datasets that include personal information of study participants was compliant with the General Data
Protection Regulation (GDPR). Procedures for registration and storage of personal was reviewed andapproved by KTH’s data protection officer and KTH’s research data team.

3. RESULTS
3.1

Design of Haptic Music Players

3.1.1 Design Considerations
Conclusions from the initial meeting with the teacher resulted in the definition of a set of design
constraints for the HMPs to be used in the study. The discussions about the design of the Woojer Strap
Edge versus the HUMU Augmented Audio Cushion provided important insights about how the HMPs
could be designed to best fit the student’s needs. We quickly realized that the HMPs must be robustand durable, since they might be thrown to the floor. Related to this, we discussed the affordances ofthe demonstrated haptic devices. For example, the teacher mentioned that the Woojer strap looked a
bit like a ball, which might also invite the students to throw it. We therefore discussed embedding the
Woojer strap into a plush toy or similar. An approach in which the vibrating unit could be attached tosomething wooden, which in turn would be embedded into a fluffy material, would both invite to touch
while simultaneously enable transmission of vibrations. Such a design would also allow the students
to throw the device to the floor, without any risk of damaging the haptic actuators. Regarding bodily
locations, we discussed that the Woojer Strap Edge, which is supposed to be worn as a belt on thebody, might not be suitable for all students since it is not so easy to take off if you do not like it. As aresult of the above-mentioned discussions, our design process largely focused on finding solutions
that would enable us to embed the hardware in soft protective padding, and on hiding the cables of
the hardware (since visible cables might invite the students to pull them out). The final design of the1516

https://dfaeurope.eu/what-is-dfa/dfa-documents/the-eidd-stockholm-declaration-2004/
https://www.ueq-online.org/

devices is described below and displayed in Figure 1. The three HMPs differ in terms of affordances:
HMP 1 (a haptic pillow) encourages the user to lay the head down on it, HMP 2 (a plush toy backpack)is intended to be worn on the back/belly, and HMP 3 (a custom-built plush toy reminiscent of the Star
Wars character Chewbacca) can be placed on the lap and explored using your hands.
3.1.2 Haptic Music Player 1
HMP 1, the HUMU Augmented Audio Cushion, is displayed in Figure 1. The HUMU pillow has
an embedded vibrating sound board assembly, with a sound and vibration frequency response of 2020 000 Hz. It plays music in stereo. We connected the HUMU pillow using Bluetooth, but it also has
a 3.5mm jack connection. Based on the meeting described in Section 3.1.1, we concluded that thecushion might not need to be modified to fit the needs of the students. There was no need for an
additional layer of padded protection for this device since it did not break when we threw it to thefloor for testing purposes.

Figure 1 – The Haptic Music Players (HMPs) used in the study: HMP 1) a haptic pillow, HMP 2) a plush
toy backpack with an embedded haptic strap, and HMP 3) a custom-built plush toy with embedded tactile
transducer and full-range speaker17
3.1.3 Haptic Music Player 2
HMP 2 was designed as a plush toy teddy bear backpack with an embedded Woojer Strap Edge(see Figure 1). The Woojer Strap Edge consists of a haptic device that looks like a black disc, which
is put in contact with the skin of the user by strapping on a textile tape around the body, see Figure 2.
The device uses an Osci™ transducer 18 which delivers polyphonic vibrations that can reproduce
frequencies up to 200Hz, including subsonic frequencies below the threshold of human hearing in therange 1-20Hz. The device connects through Bluetooth, but also supports 3.5mm cable input/output.
The Woojer Strap Edge has independent volume control for audio versus vibrations, and two different
presets for vibrations: a broad mode with frequencies up to 200Hz and a focused mode withfrequencies up to 100Hz. It should be noted that embedding the Woojer into the plush toy backpack
may attenuate certain frequencies, thereby affecting the overall haptic experience. However, our tests
suggested that the haptic experience was still of adequate quality in terms of the feedback provided.
In the first version of HMP 2, the Woojer was connected to headphones using a cable. The Woojer wasplaced in a pocket located on the back of the backpack and surrounded by soft padding material.
Conclusions from the feedback collected from evaluation session (see Section 3.2) prompted us to redesign HMP 2 to make it easier to debug. In the second iteration of this device, we connected a mini
speaker (frequency response 50-18 000Hz) directly to the Woojer device. The speaker was taped to
the Woojer using duct tape. We also sewed the protective padding into the backpack, so that the
Woojer was the only thing visible in the pocket. The final version of HMP 2 is displayed in Figure 2.

17
18

Author 1 and 2 are the copyright owners of all pictures used in this manuscript
https://www.woojer.com/pages/technology

Figure 2 – HMP 2 in first iteration (upper), versus second iteration (lower pictures). The Woojer Strap Edgeis shown in the leftmost image (upper) and middle image (lower)
3.1.4 Haptic Music Player 3
HMP 3 was designed as a custom-built plush toy with an embedded full-range speaker (Visaton
FRS 8 - 8 Ohm 19, frequency response 200-20 000Hz) and tactile transducer (TT25-8 PUCK Tactile
Transducer Mini Bass Shaker 20 , frequency response 20-80Hz). This HMP was inspired by thecharacter Chewbacca from Star Wars. An overview of the design process is displayed in Figure 3. The
HMP was built from a small wooden napkin box which was embedded in multiple layers of protective
foam. This structure was covered in textile fake-fur material. To give the device a more animalistic
character, we decorated the HMP with glass eyes and a round speaker cover that would symbolize amouth. The HMP is constructed in such a way that it can easily be opened if cables or soldering would
need to be mended or upgraded. Female 3.5mm jack outputs are glued to the wooden structure so that
all cables can be easily pulled out by the students. Pulling the cable only results in the audio being
unplugged; it doesn’t break the device or the soldering inside of the wooden structure. HMP 3 wasconnected to a DTA-2.1BT2 100W 2.1 Class D Bluetooth amplifier 21 using two 5-meter-long cables.
This allowed the device to be moved freely in the room while at the same time keeping the amplifier(with all its cables and buttons) further away from the students. The amplifier has an adjustablecrossover frequency (50-120Hz) for the subwoofer output. The sonic signal has a frequency responseof 20-20 000Hz.
3.2 Evaluation
Transcripts from the three interviews are available as supplementary material. The evaluationinterviews revealed that the students managed to successfully explore musical haptics using HMP 1,
HMP 3, and the second iteration of HMP 2. The devices were primarily tested during lessons, but
HMP 2 had also been used during free play time at the school. The average time spent exploring the
HMPs was 10 minutes. The teacher described that it was a bit difficult to say which HMP that worked
best: ”I would say that […] either the Chewbacca or the [teddy]bear were most reliable. At the sametime, I know that in the [free playing time] they used mostly the pillow. So comparing to how manytimes it is used…maybe the pillow […] I would say that the most engagement from the students I felt[…] so far maybe from the Chewbacca.” A range of different sounds were used when testing: themusic that we had sent to the teacher, children’s music, other music that the teacher knew that thestudents liked, instrumental sounds (drums and electric guitar), and animal sounds (bear and whale
singing). The students managed to actively engage with the HMPs using different modes of listening,both individually and as a group. When interacting as a group, the devices were usually placed on awooden table, and passed around. This also enabled you to feel the vibrations through the table surface.
Common themes that were identified through content analysis for all HMPs related to 1) different
strategies for exploration of the HMPs (18 quotes), 2) the novelty of musical haptics (8 quotes), and3) elements of play (7 quotes). Regarding theme 1 (strategies for exploration), the most common
strategy was to use the hands for haptic exploration. The teacher described that this strategy also madeit easy to withdraw from the interaction by removing the hand, if you did not like how it felt. The
different interaction strategies are discussed in detail below, for respective HMP. Concerning theme 2
https://www.visaton.de/en/products/drivers/fullrange-systems/frs-8-8-ohm
https://www.daytonaudio.com/product/1104/tt25-8-puck-tactile-transducer-mini-bass-shaker
21 https://www.daytonaudio.com/product/1604/dta-2-1bt2-100w-2-1-class-d-bluetooth-amplifier-with-subfrequency-adjustment
19
20

(novelty of musical haptics), it was clear that regardless of which HMP that was used, the students
were not interested at their first encounter with the devices. The novelty of the vibrating experience,or the functionality of the devices in general, clearly resulted in skepticism upon first use. This wascommunicated by pushing away the devices the first times they were presented. The teacher described:“We usually just listen to music and dance so it is another experience for them that there is somethingthat vibrates to music.” However, the HMPs were indeed accepted, and appreciated, in subsequent
test sessions (although some devices were more popular than others). Finally, regarding theme 3(elements of play), certain aspects of the HMPs naturally invited to play. For example, it appeared tobe particularly interesting and fun for the students to play with the hair of HMP 3, and its cables.
3.2.1 Haptic Music Player 1
When it comes to the functionality of the pillow, the teacher mentioned that it was quick to set up
and reliable. It was good that it only had one button. However, there were instances where there had
been some issues; the sound had been cut off at times and it was a bit tricky to set up the Bluetooth
for the first time. The pillow had not been thrown but it once fell to the floor, and it did not break. In
terms of interactions, the students had used their hands, hugged, clapped/petted it, laid their head onit, and used their feet, and mouth, to explore it. Regarding accessibility, the teacher described that
“The cool thing about the pillow was that it could be used by two persons at the same time. For
example we could put [it] on the table […] and the students who were sitting in front of each other,
could both feel it with their hands or with their faces”. In other words, the fact that the pillow waslong enough to be accessed by two students at the same time was a positive feature. The pillow was
described as somewhat less intuitive to use compared to HMP 3; the teacher needed to demonstrate to
the students how it should be used for them to understand it. Regarding the user experience, one
student had described that he liked HMP 1 (and also HMP 3), using PODD. The teacher described thatshe got the impression that the students were quite interested in HMP 1, overall. However, in termsof visual and tactile experience, the object was not as interesting as HMP 3, since it has a beige color
and the textile was quite smooth.

Figure 3 – HMP 3 at different stages of the development process
3.2.2 Haptic Music Player 2
In terms of functionality, the first version of HMP 2 had problems related to the headphones: it
was difficult to debug the system and to know if it was working or not. The new iteration of the HMP
included an embedded loudspeaker, which made it easier to use and more reliable. In particular, thespoken prompts about the Bluetooth connectivity were immediately transmitted through the speaker.
The teacher described that she had to carefully read the instructions about Bluetooth pairing tosuccessfully set up the device. She also mentioned that in a school setting a speaker might be more
appropriate, since it allows more students to listen to music simultaneously in a group setting(compared to headphones). She described that since the speaker was embedded in padding the soundwas a bit muffled, but this was not something that significantly affected the overall experience.
Regarding durability, the zip locker on the backpack unfortunately broke rather quickly. When it
comes to accessibility, as opposed to HMP 3 (which also had an animalistic character), HMP 2 was
not as intuitive for the students. The teacher described: “It looked like it was confusing that it’s ateddy bear, and it’s something that sounds also at the same time.” We discussed whether this confusion
could be caused by the fact that the sound was always on; the teddy bear was just making sounds
without any interaction (as opposed to standard interactive plush toys, which require you to, forexample, squeeze them). The teacher also mentioned that providing a “button feature” perhaps could
have resolved this issue. She compared the design of HMP 2 with HMP 3, mentioning that HMP 3 hada mouth that the audio was coming from, and that the black circular shape of its mouth reminded a bitof a button. In terms of bodily locations used when interacting with HMP 2, the students mostly usedtheir hands. The teacher did not try to put the backpack on the students’ bodies, since she wanted toallow them to test out the device using their hands, first. The teacher described: “But it's hard to say
if they pushed it away because of the vibration or they pushed it away because of the unusuality ofthe sounding plush toy.” Overall, regarding the user experience, the students appeared to be mildlyimpressed; only one student played a little bit with HMP 2, before pushing it away.
3.2.3 Haptic Music Player 3
Regarding the functionality of HMP 3, it took a little longer to set it up, compared to HMP 1.
However, it was still easy for the teacher to use, despite having a somewhat more complex
configuration, involving an amplifier. The audio quality was reportedly good and the filter knob forthe subwoofer frequencies resulted in perceptual differences. Regarding this feature, the teacher
described one instance in which: “it was such a wonderful time and [...] it seemed like the students
were so calm and happy, so I felt that maybe I won’t change [the knob] now because it felt really good,with the vibration and everything.” One student had tried to pull the cables from HMP 3 and theteacher was impressed that the HMP did not break. A comment made by the teacher was that the
combination of HMP 1 and HMP 3 would have been ideal, i.e., to have one button to set up the system
while maintaining the playful properties of HMP 3. Regarding accessibility, it seemed like HMP 3
worked equally well for different users. It was somewhat more intuitive to use than HMP 1. The
teacher described that for HMP 1:“[…] I was showing them more that oh, you can put your hand onit, or you can lie down on it. You can put it in your lap ”. She contrasted this to HMP 3: “[…] with
Chewbacca I hadn’t done this kind of showing. I just presented the Chewbacca […] so maybe because
the Chewbacca was more playful, it was more intuitive also for the students.” The teacher not only
explored music but also animal sounds with HMP 3, since Chewbacca looked a bit like a bear.
Apparently, bear sounds were well received by the students. As described by the teacher, the students
reacted as though they thought: “Okay it's a bear and it sounds like a bear.” She had also tried using
whale sounds, but the students were more engaged for the bear sounds. Regarding how HMP 3 wasused, it had - similarly to HMP 1 - been put on the table and mostly been explored using the hands.
The students also tested it in their lap and put their chins and faces onto it. One student had
communicated using PODD that she wanted to feel the vibrations from HMP 3 on her feet. The teacher
described that a positive aspect about HMP 3 was that it was smaller than HMP 1. As such, it waseasier to put it on the stomach or the feet. Although the strategies used for interacting with HMP 1
versus HMP 3 were not that different, some aspects related to the user experience appeared to be. For
example, the teacher described: “The big difference for Chewbacca [was that]…the student that isreally interested in hair, felt of course more with their hand, on the Chewbacca. Because, it was, it
was also a play.”. The fact that HMP 3 encouraged and invited to play distinguished it from the other
HMPs. The teacher also commented: “Chewbacca has also strong colors, fur […] I would guess thatthe texture and the color makes it more interesting than the pillow”.

4. DISCUSSION
Based on the evaluation of functionality, accessibility, and user experience, it appears as the mostsuccessful HMP overall was HMP 3. HMP 1 also worked well, but it did not invite to play in the same
way as HMP 3 did. Moreover, based on the evaluation sessions, we can conclude that the affordancesof the physical objects are important in this context. The appearance of HMP 2 led to some confusion,
perhaps since its design created certain expectations that were not met. HMP 3 also looked somewhat
like a plush toy, but not exactly like one. Perhaps it did not as strongly afford that it would work likea standard plush animal. Interestingly, HMP 3 was the device that was most interesting in terms of its
tactile experience overall, since it had fake fur and a hair-like texture that invited to active haptic
exploration, even if the vibrations were turned off.
If the types of devices described in this paper should work effectively in a school setting, they need
to be effortless to set up. One reason why HMP 2 did not work well, apart from the fact that it had
headphones as opposed to a built-in speaker, was perhaps that we had not demonstrated its setup
procedure in person (HMP 1 was connected to the teacher’s iPad during the initial meeting, but wenever paired the teachers’ iPad to the Woojer device; HMP 3 had also been demonstrated in person at
another occasion). As mentioned by the teacher, having only one single button to setup the device
would be ideal. Because of its ease of use, HMP 1 was used most times, even if HMP 3 was perhaps

the object that resulted in most engagement from the students. The interviews with the teacher also
clearly highlighted the need for adequate time to get used to the HMPs; all devices were met withskepticism upon first use.
Assessing complex multisensory phenomena with users who are pre-verbal is a challenging task.
In the current work, we could not directly ask the students what they perceived and thought, since
they are pre-verbal. In addition, we could not meet the students physically, so the study had to be
conducted in a remote setting. To get an idea of how the students perceived musical haptics throughthe different HMPs, we used an observation by proxies methodology. In other words, we focused onwhat was observed by the teacher, i.e., PODD conversations and observations of nonverbal
expressions, to judge if the vibration enhanced the musical listening experience. Of course, thisapproach introduces a level of uncertainty and possible bias in the evaluation process. One of the main
weaknesses of the current work is that the evaluation relies on the observation of a single teacher. It
is also difficult to generalize from the reported results since only four students were involved in the
study. What is reported in this paper should therefore be seen as results from a first exploratory casestudy; there is a need for more controlled long-term studies with a larger group of participants.
Moreover, much of what was discussed during the evaluation interviews focused on the usability ofthe devices from the teacher ’s perspective. This is important, but it would be good to explore thehaptic experiences of the children more in detail in future work. More objective methods of evaluation
could be explored in future studies. For example, the time spent with respective HMP could be used
as a measurement in this context. Because of the restrictions due to the COVID-19 pandemic, we
could unfortunately not attend the evaluation sessions in person. As such, it was difficult to collectsuch measures. Nevertheless, we believe that the teachers’ observations provide useful insights
regarding what worked well versus did not work well for respective HMP, and that these design
considerations could be considered in future work. Finally, it should be noted that we did not perform
any accelerometer measurements or analysis of the frequency responses of the HMPs. This is anotheraspect that could be further explored in future studies.

5. CONCLUSIONSIn this paper we present a six-month exploratory study on the qualitative evaluation of three Haptic
Music Players (HMPs) with four pre-verbal children with Profound and Multiple Learning Disabilities(PMLD). The HMPs encouraged exploration of haptic music through different means of interaction.
Findings from the evaluation sessions revealed that the students managed to actively engage with the
HMPs using different modes of listening, both individually and as a group. The students
communicated to the teacher and teaching assistants if they liked versus disliked the haptic sensations
and which body parts they wanted to use when feeling the vibrations. The dominating strategy wheninteracting with the HMPs was to explore the devices using the hands. This allowed the children toeasily remove the point of contact, or to push away the HMPs, if they did not like the haptic sensations.
The three HMPs greatly differed in terms of their functionality, accessibility, and user experience. Ourfindings highlighted the need for extra time to get used to HMPs since this type of musical interaction
can be completely new for many children. The physical affordances and touch and feel of the objects
are important in this context since they might influence how well the HMPs are received. The designof HMPs for children should ideally invite to touch and play. For experimentation with HMPs inschool settings, it might be good to provide several devices of different sizes which can be used bothin single and collaborative settings. An example of a design solution that allows for multiple users toperceive vibrations is a haptic table, or a haptic device that could be attached to such furniture. Finally,
it is crucial that HMPs used in the wild are easy to setup, have proper documentation involving
troubleshooting sections, and that the HMPs used in special education settings are demonstrated inperson before first use.

ACKNOWLEDGEMENTS
The authors would like to thank the students, the teacher, and the teaching assistants at Dibber
Rullen, who kindly participated in the study. We would like to especially extend our thanks to Eszter
Trautmann. This research was funded by the Swedish Research Council (Vetenskapsrådet) grant
number 2020-00343, and a Bernadotte scholarship from the Royal Swedish Academy of Music(Kungliga Musikaliska Akademien).

REFERENCES
1. Papetti S, Saitis C. Musical Haptics: Introduction. In: Musical Haptics. Springer Nature; 2018.
2. Zimmerman E, Lahav A. The Multisensory Brain and its Ability to Learn Music. Ann N Y Acad Sci.
2012;1252(1):179–84.
3. O’Modhrain MS. Playing by Feel: Incorporating Haptic Feedback into Computer-Based MusicalInstruments [PhD Thesis]. Stanford University. 2001.
4. Galembo A, Askenfelt A. Quality Assessment of Musical Instruments-Effects of Multimodality. In:
Proceedings of the 5th Triennial Conference of the European Society for the Cognitive Sciences of Music(ESCOM5), Hannover, Germany. 2003. p. 8–13.
5. Fontana F, Avanzini F, Jarvelainen H, Papetti S, Klauer G, Malavolta L. Rendering and Subjective
Evaluation of Real vs. Synthetic Vibrotactile Cues on a Digital Piano Keyboard. In: International
Conference on Sound and Music Computing (SMC2015). SMC; 2015. p. 161–7.
6. Fontana F, Papetti S, Civolani M, dal Bello V, Bank B. An Exploration on the Influence of Vibrotactile
Cues During Digital Piano Playing. In: Proceedings of the International Conference on Sound Music
Computing (SMC2011), Padua, Italy. 2011. p. 273–8.
7. Saitis C, Fritz C, Scavone GP, Guastavino C, Dubois D. Perceptual Evaluation of Violins: A
Psycholinguistic Analysis of Preference Verbal Descriptions by Experienced Musicians. J Acoust Soc Am.
2017;141(4):2746–57.
8. Wollman I, Fritz C, Poitevineau J. Influence of Vibrotactile Feedback on Some Perceptual Features ofViolins. J Acoust Soc Am. 2014;136(2):910–21.
9. Giordano B, Avanzini F, Wanderley M, McAdams S. Multisensory Integration in Percussion Performance.
In: 10ème Congrès Français d’Acoustique. 2010.
10. Merchel S, Altinsoy ME. Auditory-Tactile Music Perception. In: Proceedings of Meetings on Acoustics
ICA2013. Acoustical Society of America; 2013. p. 015030.
11. Merchel S, Altinsoy ME. Auditory-Tactile Experience of Music. In: Musical Haptics. Springer, Cham;
2018. p. 123–48.
12. Visell Y. Tactile Sensory Substitution: Models for Enaction in HCI. Interact Comput. 2009 Jan;21(1–2):38–53.
13. Remache-Vinueza B, Trujillo-León A, Zapata M, Sarmiento-Ortiz F, Vidal-Verdú F. Audio-Tactile
Rendering: A Review on Technology and Methods to Convey Musical Information Through the Sense of
Touch. Sensors. 2021;21(19):6575.
14. Nanayakkara S, Taylor E, Wyse L, Ong SH. An Enhanced Musical Experience for the Deaf: Design and
Evaluation of a Music Display and a Haptic Chair. In: Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems. Boston MA USA: ACM; 2009. p. 337–46.
15. Frid E, Lindetorp H, Falkenberg Hansen K, Elblaus L, Bresin R. Sound Forest: Evaluation of an
Accessible Multisensory Music Installation. In: Proceedings of the 2019 CHI Conference on Human
Factors in Computing Systems. 2019. p. 1–12.
16. Petry B, Illandara T, Elvitigala DS, Nanayakkara S. Supporting Rhythm Activities of Deaf Children Using
Music-Sensory-Substitution Systems. In: Proceedings of the 2018 CHI Conference on Human Factors in
Computing Systems. Montreal QC Canada: ACM; 2018. p. 1–10.
17. Mazzoni A, Bryan-Kinns N. Mood Glove: A Haptic wearable Prototype System to Enhance Mood Musicin Film. Entertain Comput. 2016 Nov;17:9–17.
18. Yamazaki Y, Mitake H, Takekoshi M, Tsukamoto Y, Baba T, Hasegawa S. Hapbeat: Tension-BasedWearable Vibroacoustic Device. In: Hasegawa S, Konyo M, Kyung KU, Nojima T, Kajimoto H, editors.
Haptic Interaction. Singapore: Springer Singapore; 2018. p. 387–92. (Lecture Notes in ElectricalEngineering; vol. 432).
19. Giordano M, Hattwick I, Franco I, Egloff D, Frid E, Lamontagne V, TeZ C, Salter C, Wanderley MM.
Design and Implementation of a Whole-Body Haptic Suit for “Ilinx”, a Multisensory Art Installation. In:
Proceedings of the International Conference on Sound Music Computing (SMC2015); 2015. p. 169–175.
20. Yoo Y, Hwang I, Choi S. Consonance of Vibrotactile Chords. IEEE Trans Haptics. 2014 Jan;7(1):3–13.
21. Petry B, Huber J, Nanayakkara S. Scaffolding the Music Listening and Music Making Experience for theDeaf. In: Assistive Augmentation. Springer; 2018. p. 23–48.
22. Giordano M, Sullivan J, Wanderley MM. Design of Vibrotactile Feedback and Stimulation for Music
Performance. In: Papetti S, Saitis C, editors. Musical Haptics. Cham: Springer International Publishing;2018. p. 193–214. (Springer Series on Touch and Haptic Systems).
23. Bellamy G, Croot L, Bush A, Berry H, Smith A. A Study to Define: Profound and Multiple Learning
Disabilities (PMLD). J Intellect Disabil. 2010;14(3):221–35.

24. Ockelford A, Welch G, Zimmermann S. Focus of Practice: Music Education for Pupils with Severe or
Profound and Multiple Difficulties—Current Provision and Future Need. Br J Spec Educ.
2002;29(4):178–82.
25. Rushton R, Kossyvaki L. Using Musical Play with Children with Profound and Multiple Learning
Disabilities at School. Br J Spec Educ. 2020;47(4):489–509.
26. Rushton R, Kossyvaki L. The Role of Music Within the Home-Lives of Young People with Profound andMultiple Learning Disabilities: Parental Perspectives. Br J Learn Disabil. 2022;50(1):29–40.
27. Fowler S. Multisensory Rooms and Environments: Controlled Sensory Experiences for People with
Profound and Multiple Disabilities. Jessica Kingsley Publishers; 2008.
28. McLinden M. Hands On: Haptic Exploratory Strategies in Children Who Are Blind with Multiple
Disabilities. Br J Vis Impair. 1999;17(1):23–9.
29. Argyropoulos V, Papazafiri M. Investigating Tactile Exploratory Procedures of Students with Multiple
Disabilities and Visual Impairment: Current Trends in Education. Spec Educ. 2017;8:9.
30. Frid E. Diverse Sounds: Enabling Inclusive Sonic Interaction [PhD Thesis]. KTH Royal Institute ofTechnology; 2019.
31. Frid E. Accessible Digital Musical Instruments – A Review of Musical Interfaces in Inclusive Music
Practice. Multimodal Technol Interact. 2019;3(3):57.
32. Ladner RE. Design for User Empowerment. Interactions. 2015;22(2):24–9.
33. Charlton JI. Nothing About Us Without Us. University of California Press; 1998.
34. Abras C, Maloney-Krichmar D, Preece J, et al. User-Centered Design. Bainbridge W Encycl HumComput Interact Thousand Oaks Sage Publ. 2004;37(4):445–56.
35. van der Velden M, Mörtberg C. Participatory Design and Design for Values. In: Handbook of Ethics,
Values, and Technological Design: Sources, Theory, Values and Application Domains. Dordrecht:
Springer Netherlands; 2021. p. 1–22.
36. Quintero C. A Review: Accessible Technology Through Participatory Design. Disabil Rehabil Assist
Technol. 2020;1–7.
37. Spencer González H, Vega Córdova V, Exss Cid K, Jarpa Azagra M, Álvarez-Aguado I. IncludingIntellectual Disability in Participatory Design Processes: Methodological Adaptations and Supports. In:
Proceedings of the 16th Participatory Design Conference 2020-Participation (s) Otherwise-Volume 1.
2020. p. 55–63.
38. Porter G, Cafiero JM. Pragmatic Organization Dynamic Display (PODD) Communication Books: A
Promising Practice for Individuals with Autism Spectrum Disorders. Perspect Augment Altern Commun.
2009;18(4):121–9.
39. Hamidi F, Baljko M, Gómez I. Using Participatory Design with Proxies with Children with Limited
Communication. In: Proceedings of the 19th International ACM SIGACCESS Conference on Computers
and Accessibility. 2017. p. 250–9.
40. Constantin A, Alexandru C, Korte J, Wilson C, Fails JA, Sim G, et al. Distributing Participation in Design:
Addressing Challenges of a Global Pandemic. Int J Child-Comput Interact. 2021;28:100255.
41. Rogers Y. Interaction Design Gone Wild: Striving for Wild Theory. Interactions. 2011 Jul;18(4):58–62.
42. Frid, E. and Panariello, C. and Núñez-Pacheco, C. Customizing and Evaluating Accessible Multisensory
Music Experiences with Non-Vocal Children - A Case Study on the Perception of Musical Haptics Using
Participatory Design with Proxies. Multimodal Technologies and Interaction.
43. McNamara N, Kirakowski J. Functionality, Usability, and User Experience: Three Areas of Concern.
Interactions. 2006 Nov;13(6):26–8.
44. Petrie, Helen and Bevan, Nigel C. The Evaluation of Accessibility, Usability and User experience. In: The
Universal Access Handbook. Boca Raton: CRC Press; 2009.
45. Young GW, Murphy D. HCI Models for Digital Musical Instruments: Methodologies for Rigorous Testingof Digital Musical Instruments [Internet]. arXiv; 2020 [cited 2022 Jul 8]. Available from:http://arxiv.org/abs/2010.01328
46. Young, Gareth W. Human-Computer Interaction Methodologies Applied in the Evaluation of Haptic
Digital Musical Instruments. University College Cork; 2016.
47. Connell, BR, Jones M, Mace R, Mueller J, Mullick A, Ostroff E, Sanford J, Steinfeld ED, Story M, andVanderheiden G. The Principles of Universal Design. 1997.
48. Downe-Wamboldt B. Content Analysis: Method, Applications, and Issues. Health Care Women Int.
1992;13(3):313–21.
49. Ferrigon P, Tucker K. Person-First Language vs. Identity-First Language: An Examination of the Gains
and Drawbacks of Disability Language in Society. J Teach Disabil Stud. 2019;

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0127

Changes in audio intensity perception according to touchrecognition in a virtual environment
Hong-In Won1; Byeong Hak Kim2; Jun Ha Lee3; Sebastian Merchel4; M. Ercan Altinsoy5
1, 2
3

Korea Institute of Industrial Technology, Republic of Korea

Graduate School of Kyungpook National University, Republic of Korea
4, 5

Dresden University of Technology, Germany

ABSTRACTThis paper presents the changes in audio intensity perception according to touch recognition in a virtualenvironment. To investigate the audio intensity perception in a virtual environment, an experiment in which
a subject comes into contact with a touch object in a virtual reality space was modeled using Unity, one ofthe popular development platforms. As tools for the experiment, a headset and controllers of Oculus wereused for a subject to experience audio and haptic feedback in the virtual reality space. In the experiment,
subjects were asked to estimate the intensity perception of audio feedback given by the headset dependingon whether or not they noticed contact with a virtual object via haptic feedback and visual information. In
the end, the answers from the subjects were analyzed statistically, and then the effect of touch feedbacks on
audio intensity perception was discussed.
Keywords: Audio Intensity Perception, Multisensory Integration, Virtual Environment

1. INTRODUCTION
With the development of virtual reality (called VR) based digital twin applications and metaverse
services, research on audio stimulation and perception in a virtual environment is becoming
increasingly important (1, 2, 3). For a long time, many researchers have conducted several studies on
multisensory stimulation and perception in the real world (4, 5, 6), but studies on multisensory
characteristics in a virtual environment are still insufficient.
CAVE (Cave Automatic Virtual Environment) and HMD (Head Mounted Display) are typicallyused as user hardware for creating and experiencing VR environments, and various methods have beensought to appropriately stimulate human senses and provide an immersive experience. However, in a
VR environment, human perception is not the same as perception in the real environment, andsituations or conditions that do not exist in the real environment can be created. Accordingly, an
auditory sensory model for the VR environment should be established through the development andconduction of perception experiments in the VR environment.
In addition, it is important to investigate multisensory integration methods in the VR environment
since there is no real feedback on contact with a virtual object. People may miss the warning or noticefrom the system if they are not seeing that they touch an object in a virtual space. For example, if
someone touches a trigger object in a virtual space, but does not recognize it, unintended consequences
may occur. Moreover, if the virtual space is connected to physical systems in the real world, it cancause irreversible problems.
In this paper, the development of an experimental program that generates auditory stimuli in
response to touch events in a VR environment, using Oculus Quest and Unity, is introduced. The
magnitude production method (7) for the estimation of audio intensity perception, with a developed
luvhayym@kitech.re.kr
bhkim81@kitech.re.kr
3
ark986@knu.ac.kr
4
sebastian.merchel@tu-dresden.de5
ercan.altinsoy@tu-dresden.de
1
2

program, is explained. The magnitude production method is employed as a psychophysical estimation
method. In the end, experimental results are analyzed statistically to examine the changes in audio
intensity perception in a VR environment.

2. EXPERIMENTAL SETUP
2.1 VR Experience Device and Software
In this study, Oculus Quest 2 was used as a device for VR environment creation and experience(see Figure 1). Oculus Quest 2 is one of the popular HMDs developed by Facebook Reality Labs. It
is relatively inexpensive and many APIs are open for VR content development. In addition, it has theadvantage of being able to connect to the development computer, without additional equipment orcable, by the Air-link function. Oculus Quest 2 can play sound with its built-in speakers in the headset
and can generate haptic feedback with the controllers.
Unity was used as software for VR environment development. Unity is a representative VR
environment development tool that provides diverse conditions and components that can configure
various physical phenomena (gravity, collision, elasticity, friction, etc.) in the VR world. In addition,
through Oculus Integration which is available for free in Unity's Asset Store, developers can easily
link Oculus Quest 2 to VR worlds and set various functions and events required for ex perimentation.

HeadsetController (L)Controller (R)Figure 1 – Equipment of Oculus Quest 2 for VR experience
2.2 Development of Experimental Scene
For an experiment in a VR environment, it is necessary to design a VR scene according to the
purpose of the experiment. The purpose of this study is to generate auditory and tactile stimuli whentouching a virtual object in a VR environment and to perform multimodal perception experiments. As
shown in Figure 2, a VR scene was composed of two spheres (one is the modulus and another is thetest object) and hand-link objects on the experiment table. The experimental table was developed tohave the same position and height as the table of the user in the real world, and the two spheres are
virtual fixed objects. The hand-link objects move in conjunction with the user controller when the VR
environment is running.
Since the virtual sphere is an object that does not exist in the real world, the user's hand passes
through the sphere when the user puts one's hand on it. Therefore, by adding Unity Engine's Colli der
Component to the virtual sphere and hand-link objects, the contact between the sphere and the hands
was implemented. Additionally, the color change and vibration visualization were set so that the user
could recognize the contact between the sphere and the hand. In the case of a situation where the user
does not need to recognize the touch, it can be obscured by changing the opacity of the shadersurrounding the sphere.

3. PSYCHOPHYSICAL EXPERIMENT
3.1 Audio and haptic feedback stimuli
When the user's hand touches the white spheres in the VR scene, the auditory stimulus was set to
be provided through the Oculus headset and the tactile stimulus was set to be delivered through theOculus controller for a set period of time. In this study, a simple sinusoidal waveform with the ramp
envelope, as shown in Figure 3, was used for audio and haptic feedback to conduct the experiment. A
mathematical equation of the waveform can be expressed, as a function of time t, as follows:(1)
𝐴sin(2π𝑓𝑡) × (𝑡/𝑡𝑎 ) in the range of 𝑡 ≤ 𝑡𝑎
(2)
𝐴sin(2π𝑓𝑡) × (𝑡𝑏 − 𝑡)/(𝑡𝑏 − 𝑡𝑎 ) in the range of 𝑡𝑎 < 𝑡 ≤ 𝑡𝑏
where A is the maximum amplitude of the waveform, f is the frequency of the waveform, ta is the timecorresponding to the maximum amplitude and tb is the time at which the waveform ends. Unlessotherwise noted, the following values were used in this study: f = 175 Hz, ta = 0.5s, and tb = 0.6s.
The amplitude A was determined differently according to the type of feedback. For the audio
feedback, A was set to generate auditory sensation levels of 45, 55, and 65 dBA over the waveform
duration. On the other hand, A was set to generate a tactile sensation level of 125 dB re 1 μm/s 2 over
the waveform duration for the haptic feedback. The real values of A were calibrated before theexperiment began, by the accelerometer and the binaural microphones as shown in Figure 4.

Figure 2 – Experimental scene in VR environment

Amplitude

A

0

-A

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Time (s)Figure 3 – Base waveform of tactile and auditory feedback stimuli
3.2 Experimental Method and Procedure
The magnitude production method was adopted to estimate intensity perception from audio
feedback in the experiment. The modulus and test objects were displayed in the VR scene describedabove, and the modulus was set to generate audio feedback only while the test object was set togenerate audio-haptic feedback. In addition, the shader for the test object was set to change to black
opaque in order for the test where the subject does not recognize the touch.

(a)

(b)

Figure 4 – Amplitudes calibration setup using (a) the binaural microphones and (b) the accelerometer.
In this circumstance, participants were asked to adjust the perceived audio intensity level of themodulus equal to the test object with the Oculus controllers. For example, if they felt the perceived
intensity of the modulus was lower than the test object, they would have increased the auditory
sensation level of the modulus until the perceived intensity on both sides was approximately equal.
Then, the modified intensity level of the modulus was recorded as the perceived audio intensity levelof the test object. The estimation of sensation level was performed by their own subjective feelingafter the experience of stimulation, and the number of times for touch was not limited. I n theestimation task, no information about tactile and audio feedback stimuli was given to the participants.
The experiment was carried out divided into two sessions: one was the training session and theother was the main session. In the training session, the magnitude production method was explained
to the participants, and they experienced estimation tasks a few times. Then, in the main session, they
performed the magnitude production method with respect to all tactile and audio feedback stimuli inthe presence and absence of the shader, as shown in Table 1. In each estimation task, an initial
sensation level of audio feedback on the modulus was randomly set in the range of 40 to 70 dBA.
Twenty subjects, 12 male and 4 female volunteers, participated in the experiment. The ages ofparticipants ranged from 20 to 50. All participants had no disability in tactile or hearing senses. They
were instructed to sit in front of the experiment table and equipped the experiment devices in a
comfortable condition after the explanation of the contents of the experiment. It took approximately
30 minutes for each participant to complete both sessions.
Table 1 – Test cases for the audio intensity estimation experiment
Identifier

Level of audio
feedback (dBA)

Level of haptic
feedback (dB)

Shader forthe test object

A45

45

-

-

A45×H

45

125

-

A45×S

45

-

Presence

A45×H×S

45

125

Presence

A55

55

-

-

A55×H

55

125

-

A55×S

55

-

Presence

A55×H×S

55

125

Presence

A65

65

-

-

A65×H

65

125

-

A65×S

65

-

Presence

A65×H×S

65

125

Presence

4. Analysis of Experimental Results
All evaluation data collected from the experiment above are classified in each case and theirstatistical mean values were calculated after eliminating outlier data considering the interquartile
range. The mean perceived intensity level (mean PIL) of the audio feedback is plotted in Figure 5.
The circle, diamond, and square symbols respectively represent the mean PIL values for theexperiment case with the auditory sensation level of 45, 55, and 65 dBA. The test cases of A represent
the mean PILs when the audio feedback was given solely. The test cases of A×H and A×S represent
the mean PILs when the audio feedback was given with the haptic feedback and with the shading
effect, respectively. The test cases of A×H×S represent the mean PILs when the audio feedback wasgiven with the haptic feedback and the shading effect simultaneously. The error bar indicates the 95%
confidence interval in each case. It seems that the psychophysical experiment using the magnitude
production method was normally performed since the mean PILs were evaluated at levels of given
audio feedback.
According to the experimental results, mean PIL increased when the tactile and audio feedback are
given together, meaning that the audio intensity perception has been enhanced by the haptic feedback.
On the other hand, mean PIL decreased when the shading effect is given to the test object. This means
the audio intensity perception has been diminished if the participant does not recognize the touching
moment. In the cases of the haptic feedback and the shading effect are given simultaneously, the mean
PILs changed but they do not have significant differences. It seems that the enhancement by the haptic
feedback and the diminishment by the shading effect on the audio intensity perception in the VR
environment compensate for each other.
45 dBA

55 dBA

65 dBA70

mean PIL (dBA)

65

60

55

50

45

40
A

A×H

A×S

A×H×S

Test case

Figure 5 – Mean perceived intensity level evaluated by the magnitude production method.

5. CONCLUSIONSThe effect of touch recognition on audio intensity perception in a virtual reality environment has
been investigated, in this study, with the psychophysical experiment. The experimental scene wascreated, by using Unity Engine, to experience audio and haptic feedback in the VR environment. To
conduct the psychophysical experiment, twelve test cases, considering the presence and absence of
haptic feedback and the shading effect, were set for audio intensity estimation, by means of themagnitude production method.
From the statistical analysis of the collected data obtained from the experiment, the changes inaudio intensity perception in the VR environment, investigated in this study, can be summarized asfollows: (1) the audio intensity perception can be enhanced by t he haptic feedback given with theaudio feedback, (2) the audio intensity perception can be diminished by the shading effect on thetouching moment, and (3) the enhancement by the haptic feedback and the diminishment by theshading effect on the audio intensity perception can compensate for each other.

ACKNOWLEDGEMENTS
This work was supported in part by the Korea Institute of Industrial Technology under grant UI 22-0010 and the Korea Medical Device Development Fund grant funded by the Korea government(the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health& Welfare, the Ministry of Food and Drug Safety) (Project Number: 1711138339, KMDF_PR_20200901_0167).

REFERENCES
1. Gibbs JK, Gillies M, Pan X. A comparison of the effects of haptic and visual feedback on presence invirtual reality. International Journal of Human-Computer Studies 157. 2022. 102717.
2. Brasen PW. Christoffersen M, Kraus M. Effects of vibrotactile feedback in commercial virtual reality
systems. In Interactivity, Game Creation, Design, Learning, and Innovation. Springer, Cham. 2018. p.
219–224.
3. Samad M, Gatti E, Hermes A, Benko H, Parise C. Pseudo-haptic weight: Changing the perceived weightof virtual objects by manipulating control-display ratio. In Proceedings of the 2019 CHI Conference on
Human Factors in Computing Systems. 2019. p. 1–13.
4. Won HI, Altinsoy ME. Effect of auditory feedback on tactile intensity perception in a touchscreen
application. IEEE Transactions on Haptics 13(2). 2019. p. 343–353.
5. Merchel S, Altinsoy ME, Schwendicke A. Tactile intensity perception compared to auditory loudness
perception. 2015 IEEE world haptics conference (WHC). IEEE. 2015. p. 356–361.
6. Verrillo RT. Vibration sensation in humans. Music Perception 9(3). 1992. p. 281–302.
7. Meiselman HL, Bose HE, Nykvist WF. Magnitude production and magnitude estimation of taste intensity.
Perception & Psychophysics 12(2). 1972. p. 249–252.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0276

Haptic Feedback to Improve Musical Interaction

Sebastian MERCHEL; Kai DEHMELT; Pablo ALVAREZ ROMEO; M. Ercan ALTINSOY
Centre for Tactile Internet with Human-in-the-Loop (CeTI), Chair of Acoustic and Haptic Engineering,
TU Dresden, Germany
ABSTRACTInteraction between musicians during a joint musical performance is predominantly based on acoustic
feedback. In real situations, this feedback is unfortunately influenced by a variety of interfering factors. For
example, strong reverberation or additional sounds can alter or mask acoustic signals. This leads toundesirable effects, such as asynchronous playing, and thus to a reduction in the quality of the overall musicalexperience for the listener. This paper investigates a supportive system with vibrotactile feedback. The
applied vibration signal was generated directly from the audio signal of the partner instrument and reproduced
via an actuator on the lower back of the musicians. Whether this additional information helps improve thestability of timing and synchronization was investigated. For this purpose, experiments were conducted withbass guitarists and drummers. The participants were asked to learn and play a simple musical pattern, withand without vibration feedback, under different acoustically disturbed scenarios. The musical performance
was analyzed in terms of the accuracy achieved with respect to the expected beat grid. Furthermore, theparticipants’ latency thresholds between audio and vibrotactile feedback were determined in a perceptual testwith kick drum sounds to determine the delay requirements for the developed system.
Keywords: Music Performance, Sound and Vibration, Tactile

1. INTRODUCTION

Musicians' performance during live concerts is critical to the audience experience. In this context,
the perception of the artists often varies from that of the listeners. Poor location-dependent acoustic
conditions must be handled professionally so that the audience can still be provided with an impressive
experience. Occasionally, due to various factors, such as a noisy environment, musicians may lose
track of the common rhythm/tempo, decreasing the overall quality of the musical experience.
Multiple systems have been developed that combine music with haptics, more specifically focusing
on rhythmic feedback. Pianotouch (1) focuses on musical education. As a glove designed to improve
piano students’ skills, it applies vibration to the fingers that should be used when learning a new song,using eccentric rotating mass (ERM) actuators. With regard to education, there are other devices, suchas the Haptic Bracelets (2) for drum students, receiving multi-limb rhythmic information from theirteacher's movements or from computer playback and generating the corresponding vibrotactile
feedback through voice coil actuators (VCAs). Another device, the Haptic Tutor (3) is focused on
drum learning, using ERM actuators attached to each limb.
There are metronomes that also implement haptics. Giordano and Wanderley (4) investigated theeffectiveness of ERM actuators for transmission of rhythmic information. Soundbrenner (5) released
a commercial wristband, currently in two models, Core and Pulse, that use ERM actuators.
In this work, the focus is on improving inter-musician communication. In (2) it is already suggested
that haptics may help in that regard. For that goal, a novel system, the Groovuator, is developed. In
the next section, its structure, actuation system, operation and performance will be described, followedby its use in performance experiments and the final discussion.

2. Groovuator

A device focused on the improvement of dynamic rhythmic information between musicians ispresented. As it focuses on enhancing musical timing between users, which in colloquial language isknown as grooving, this system is referred to as the Groovuator2.1 System
The Groovuator is a modular system designed for improving the interaction between two musiciansby measuring their interactions with their instruments and providing haptic feedback to each other, as
shown in Figure 1.

Figure 1: Block diagram of the system's intercommunication.
In order to do that, each module must be able to detect and quantify the musical signal from its
corresponding musical instrument, process it, and send the haptic command to the corresponding
module, as explained in Figure 2. The sound acquisition system, however, varies from one instrument
to another. Here, two different instruments were chosen: a bass guitar and a kick drum of a drum set.
The additional system deployed for measuring their signals is explained in the next section.

Figure 2 – Block diagram of Groovuator's modular operation.
Each module is composed of an ESP32 microcontroller as the core processing unit, an input stage
where the input signal from the musical instrument is connected and converted with a TL072
operational amplifier to a readable value for the microcontroller’s ADC, and an output stage to drivea vibrotactile actuator. This output stage is prepared in such a way that it can drive different
technologies, in this case ERMs and voice coil actuators (VCA). For the first, the ERMs, the circuit
consists of a BC547 transistor, and for the second, the voice coil actuators, a Class-D audio amplifier,the PAM8403, is used.
2.2 Signal Processing
Once the adjusted signal from the musical instrument is read by the ESP32, it is processed. As theread signal is produced by a musical instrument, it is expected to follow the amplitude trend of an
ADSR model (Attack, Decay, Sustain and Release). It describes the envelope of an audio signal
exclusively according to its amplitude, disregarding its frequency. Therefore, this model is used for
detecting new events from the musical instrument. However, before applying the ADSR model for
event discrimination, the envelope of the signal must first be generated from the input AC signal. For
that purpose, an envelope-detector-model, shown in Figure 3, is used. Based on the model from
Sethares et al. (6), it rectifies the input AC signal, using exclusively absolute values, and applies alow pass filter which, in this case, is done by applying a moving average algorithm.

Figure 3 – Envelope detector algorithm used prior to the ADSR model.
To output vibrotactile feedback, two appropriate stimuli were created according to the input events
produced by the bass guitar and the kick drum. While the ERM was controlled by an on-off signal with afixed duration, the VCAs were controlled by a sinusoidal amplitude sweep at their resonance frequency of
70 Hz. In Figure 4, the vibrotactile feedback stimuli are displayed over time. The kick drum stimulus waschosen to last for 150 ms, according to the duration of an audio kick drum sample, while the bass guitar
stimulus was adjusted to 250 ms.

Figure 4 – Vibrotactile feedback stimuli, sinusoidal amplitude sweep, f = 70 Hz,
Left: Kick drum stimulus, 150 ms. Right: Bass guitar stimulus, 250 ms.
A voice coil actuator from Tactile Labs (Haptuator MM3C) was selected for use because of its highvibration strength and low latency.
2.3 System Latency
As the system aims to improve temporal synchronization between musicians, a controllable low
latency between sound and vibration is crucial. The overall latency of the system, which includes
processing the input signal, generating the driving signal for the actuator, and the actuator's start-up
latency, was measured with a 3-axis accelerometer ADXL335. The resulting time delay wasapproximately td = 2.8 ms. As this time depends on the input signal's waveform, there can be somesmall changes according to the dynamic audio production of an instrument. However, it can beassumed that the latency is within a range of two to three milliseconds when using a cable connection
between the input and output stages. A wireless connection was not implemented at this point but
could be an improvement to increase the convenience of using the system. According to Altinsoy (7,8),
the onset delay of a corresponding audio and vibrotactile feedback stimulus should not exceed ±10 ms
to be perceived as synchronous. Therefore, the Groovuator's latency is under that limit.

3. Performance Experiment

To evaluate the performance of the Groovuator, experiments with test subjects were conducted.
Although they presented different levels of musical experience, all of them were familiar with theirassigned instruments.

3.1 Setup
For the experiment, a setup that included two Groovuator modules and two different instrumentswith their measurement systems was prepared, as shown in Figure 5. A Fender Aerodyne bass guitar
and a Gretsch Catalina Club Jazz drum kit were used as musical instruments. For signal acquisition,a Behringer XAIR18 digital mixer was deployed as a mixing and recording device, receiving the bass
signal directly from the instrument, and the drum signal through a Sennheiser E 602 II microphone
placed next to the instrument. Nine pairs of participants between the ages of 18 and 40, each consistingof a bass player and a drummer, took part in the experiment.

Figure 5 – Experimental setup with musical acquisition system.
3.2

First Part - Solo PlayThe participants were asked to learn and play a simple musical pattern first. Following that, they
played the pattern alongside playback of the other instrument. The playback was provided overheadphones. Figure 6 illustrates the experimental setup.

Figure 6 – Left to Right: Bass player, Drummer, Vibration actuator placed at the lower back.
The pattern was then played for one minute each with 3 different conditions: 1) clean audio
feedback without Groovuator, 2) disturbed audio feedback without Groovuator, and 3) disturbed audio
feedback with Groovuator. Disturbed audio meant that the audio signal had an added reverberation
effect, which started approximately after 100 ms and was 12 dB louder than the direct signal. The
reproduced vibrations were generated from the original audio signal.
During the musical performance, the audio output of the played instrument was recorded andanalyzed in terms of accuracy with respect to the expected tempo and the beat grid. The mean value
and standard deviation over 30 measuring points per participant and per mode are displayed in Figure
7 for the bass players and in Figure 8 for the drummers. A latency of 0 ms would mean that themusicians play in perfect synchrony with the second instrument. Among the bass players, it can beseen that for most data points, a low mean latency with a small standard deviation is obtained for theclean drum audio feedback (blue) and that they performed well in the expected range of latency
slightly below tL = 0 ms. The participants' performance varies slightly depending on their personalsense of timing. When the data points from the disturbed audio playback (red) are considered, distinct

latencies in the direction of the reverberation signal can be noticed, which vary depending on theparticipant. The average delay is tL,Mean,all = 50.7 ms. When using the Groovuator (yellow), themusicians can better synchronize the onsets, similar to the clean audio condition. A latency oftL,Mean,all = 11.8 ms is attained on average.
After testing the prerequisites, repeated-measures analysis of variance (ANOVA) was applied forstatistical analysis. The results revealed that there was a highly significant difference between themeasured latency for the three feedback conditions [F(2;16) = 19.845, p < 0.0005]. To explore thesignificant main effect, post hoc pairwise comparisons were conducted using the Bonferroni
correction for multiple comparisons. The measured latency for the condition ‘clean audio’ wassignificantly different from the condition ‘disturbed audio’ (average difference = 56.5 ms, p = 0.002),
and the condition ‘disturbed audio’ was significantly different from the condition ‘disturbed audio +Groovuator’ (average difference = 38.9 ms, p = 0.006).

Test participants

Figure 7 – Mean ±1 standard deviation of bass guitar latency measurements, each averaged over 30
measuring points.
When analyzing the timing of the drummers shown in Figure 8, strong variation of the latency can beseen between test participants for the clean audio condition. An ANOVA did not prove any significant
differences between the three conditions [F(2;16) = 1.175, p = 0.334].

Test participants

Figure 8 – Mean ±1 standard deviation of kick drum latency measurements, each averaged over 30
measuring points.

It is assumed that the strong deviations in timing were influenced by the selected sound of the bass,
which was synthetically generated. Several musicians reported difficulties to follow the beat grid of
this bass line.
3.3 Second Part – Just–Noticeable Latency Thresholds
In a final experiment, the latency threshold of the subjects was estimated to assess whether it ispossible to work with higher latencies between audio and vibrotactile feedback in an extended system.
For this purpose, a drum playback was reproduced via headphones while the Groovuator delivered
stimuli associated with the kick drum to each participant. The latency between audio and vibration
was increased by t L = 2.5 ms every two pattern runs. Subjects were asked to signal after subjectively
detecting a latency between audio and vibrotactile feedback. Subsequently, the experiment was
repeated a second time. The averaged latency thresholds of all subjects are shown in Figure 9. The
data of the subjective latency detection threshold were confirmed to be normally distributed with two
statistical tests (Kolmogorov–Smirnov p = 0.2 and Shapiro–Wilk p = 0.626). Therefore, a mean value
can be used to summarize the data. The mean overall latency threshold is approximately tL = 11 ms.

Test participants

Figure 9 – Subjective latency detection thresholds for audio and vibrotactile feedback.

4. Discussion

This work investigated the development of a vibrotactile feedback device for the tempo-dynamic
support of two musicians. Information on the rhythm provided to the musician was obtained from theother musician's instrument. It has been demonstrated in these experiments that vibrotactile feedback,in addition to audio feedback, can contribute to a significant improvement in timing andsynchronization. Bass players in particular benefited from the feedback generated from the kick drum,
implying that rhythm instruments are best suited for generating vibrotactile feedback with the
Groovuator. Further improvements to the system, e.g., an improved algorithm for envelope generation,
as well as a more detailed series of experiments with distinctions between amateur and professionalmusicians, could be conducted to improve the performance of the Groovuator and better specify its
area of application.

ACKNOWLEDGMENTS

Funded by the German Research Foundation (DFG, Deutsche Forschungsgemeinschaft) as part ofGermany’s Excellence Strategy – EXC 2050/1 – Project ID 390696704 – Cluster of Excellence“Centre for Tactile Internet with Human-in-the-Loop” (CeTI) of Technische Universität Dresden.

REFERENCES

1. Huang K, Do E, and Starner T. "PianoTouch: A Wearable Haptic Piano Instruction System for Passive
Learning of Piano Skills". In: ISWC (Jan. 2008), pp. 41-44.
2. Holland S, Bouwer A and Hödl O. "Haptics for the Development of Fundamental Rhythm Skills,

3.
4.
5.
6.
7.
8.

Including Multi-limb Coordination". In: Musical Haptics. Ed. by Papetti S and Saitis C. Springer, 2018,p. 215-237.
Tom A, Singh A, Daigle M, Marandola F and Wanderley M. "Haptic Tutor - A haptics-based musiceducation tool for beginners". In: HAID, Montreal, Canada, Aug. 2020.
Marcello G and Marcelo W. "Follow the Tactile Metronome: Vibrotactile Stimulation for Tempo
Synchronization in Music Performance". In: Sound and Music Computing Conference, Maynooth,Ireland, July 2015.
Soundbrenner. Mar. 2022. URL: https://www.soundbrenner.com.
Sethares WA, Johnson Jr. CR and Klein AG. "Software Receiver Design". Cambridge University Press,
2011, p. 82-84.
Altinsoy ME. "Perceptual Aspects of Auditory-Tactile Asynchrony". In: ICSV, Stockholm, Sweden, 2003,
p. 3831-3838.
Altinsoy ME. "The Quality of Auditory-Tactile Virtual Environments" JAES, vol. 60, no. 1/2, Mar. 2012,
p. 38-46.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0297

Effect Of Audio-Tactile Congruence on Vibrotactile Music
Enhancement for Cochlear Implant Users
Scott C. AKER1,2; Kathleen F. FAULKNER2; Hamish INNES-BROWN3; Marianna VATTI2; Jeremy
MAROZEAU1
1Music and Cochlear Implant Lab, Department of Health Technology, Technical University of Denmark,Kongens Lyngby, 2800, Denmark
2
3

Oticon, Smørum, 2765, Denmark

Eriksholm Research Centre, Snekkersten, 3070, Denmark

ABSTRACT
Tactile vibrations can enhance the perceived quality of a music listening experience. As music is a challenge
for cochlear implant (CI) users, vibrotactile stimulation could be a way to assist them. However, there is noconsensus on the best way to code music into vibrotactile stimulation, or if the method should be different
for CI users compared to listeners without hearing loss.
An initial experiment on twenty normal hearing listeners tested a simple audio-vibrotactile coding scheme
which kept auditory and tactile stimuli time-aligned and mapped acoustic intensity to vibrotactile intensity
and acoustic fundamental frequency to vibrotactile frequency. A simple melody was presented over supraaural headphones and the associated vibrotactile stimulation was presented via a Lofelt L5 actuator held
between the thumb and forefinger. Participants rated the audio-tactile stimuli on preference alongsideversions in which different mapping combinations were made congruent or incongruent. Results suggested
that participants preferred stimuli which were time-aligned and congruent in intensity.
The experiment was repeated with eighteen CI users. With this cohort, however, only a portion of theparticipants preferred time aligned stimuli and stimuli with congruent intensity. Reasons for this difference
and the implications for vibrotactile solutions for music perception for CI users will be discussed.
Keywords: Vibrotactile, Music, Haptics

1. INTRODUCTION
Vibrotactile stimulation through a worn haptic device is a potential way to improve music
perception for people with cochlear implants (1). Tactile stimulation has been shown to improve music
listening experience for people without cochlear implants (2). However, an important question often
asked is: “how should one code the audio signal into a vibrotactile signal?” Alternatively: “what must
be congruent between the audio and the vibrotactile signal for vibrotactile music enhancement to takeplace?” Previously, a study was developed by Aker, et al. (3) to address this question for normal
hearing listeners. In the present study, a similar experiment was tested on cochlear implant listeners
and a control group of normal hearing listeners to see what features of the tactile stimulation must be
congruent with the auditory music stimulation for vibrotactile music enhancement to occur in cochlear
implant listeners.
The experiment by Aker et al. (3), which tested on normal hearing listeners, showed that intensity
congruence and time synchrony between the auditory and tactile stimuli is generally preferred overintensity incongruence and time asynchrony, while frequency congruence had no or little effect. In
that experiment, a simple melody comprised of sinusoids between the frequencies 70 and 180 Hz waspresented over supra-aural headphones and a Lofelt L5 haptic actuator held between the thumb andforefinger. When both the auditory and tactile stimulus was the same, the two modalities had frequency,
intensity, and timing congruence. In a MUSHRA-like set-up (4), listeners were asked to rate themelody alongside versions in which the intensity, frequency, and timing of the tactile stimulus wasmade incongruent or asynchronous with the auditory stimulus. The results showed removing intensity
and timing congruence had the greatest and most reliable negative effect on ratings. There was anadditional effect of musicianship, where listeners with more musical training were affected more bycongruence.
It was hypothesized cochlear implant listeners would have similar results to normal hearing
listeners. Rhythm discrimination, which considers intensity and timing, is generally similar incochlear implant listeners to rhythm discrimination in people with normal hearing (5). Therefore, thecongruence of the two parameters in the auditory and tactile modalities should grant the same benefit
to vibrotactile music enhancement as with normal hearing listeners.
However, there are potential confounds. Audio-tactile integration has been shown to occur less inpeople with cochlear implants. People with cochlear implants are less likely to fall for audio -tactile
illusions which depend on audio-tactile integration to occur (6). At the same time, these effects are
dependent on the time of implantation (7).
Other variables can potentially affect results as well. Age, for example, is a predictor of tactile
sensitivity and could affect results (8). Likewise, musicians have been shown to have better tactile
frequency discrimination than non-musicians (9).
The present study aimed to repeat the experiment on listeners with cochlear implants. A group ofeighteen cochlear implant listeners and twenty-three normal hearing listeners were presented with five
buttons, each which presented a simple auditory melody and accompanying tactile stimulation. Eachbutton presented the tactile stimulation with a different level of audio-tactile congruence to theauditory melody. Three audio-tactile congruences were tested for their effect on participant
preference: intensity congruence, fundamental frequency congruence, and timing alignment.

2. METHODS
2.1 Participants
Eighteen cochlear implant (CI) listeners were recruited (ages 41-82, median 65) from a variety ofbrands (3 Cochlear, 3 Med-El, 1 AB, 11 Oticon Medical). Twenty-three normal hearing (NH) listeners
were recruited (ages 19-27, median 24) with no self-reported hearing difficulties as a control group.
2.2 Stimuli
Auditory stimuli were presented between 58 dB SPL and 70 dB SPL for the normal hearing listeners
over supra-aural headphones. Cochlear implant listeners adjusted the level of a sample melody to an
audible, comfortable range, which was then used for all subsequent stimuli.
Tactile stimuli were presented through a Lofelt L5 held between the thumb and the forefinger
inside a fabric-lined box. Tactile stimuli were perceptually equalized to a level of 600 mV at 100 Hz
and 300 mV at 100 Hz using an intensity matching task.
The stimuli used in the experiment discussed in this paper was a midi file adapted from the song“szárad a száj,” by the band “Chalga.” The midi file was then transposed to the frequency range o f
70-180 Hz and reconstructed with sawtooth waves for the auditory stimuli or sine waves as the tactile
stimuli. Prior to reconstruction, features of the tactile stimulus were randomized independently of theauditory stimulus to achieve the desired level of congruence or incongruence. For intensity
incongruence, the intensity of the vibration assigned to each note was shuffled. Similarly, forfrequency incongruence, the fundamental frequency of the vibration assigned to each note was
shuffled. Timing was made incongruent by generated a new rhythm for each musical phrase of thetactile stimulus, which had the same number of notes and same total duration of notes.
2.3

ProcedureFirst, listeners completed the Goldsmith-MSI to score musical sophistication and musical training.
Second, listeners completed a tactile intensity matching task to equalize the frequency response of thetactile stimuli. Third, listeners completed a frequency discrimination task to screen for tactile

sensitivity.
Finally, the rating task was completed. Listeners were presented with ten screens with five buttons
each. Each button presented the auditory music stimulus and a version of the tactile music stimulus.
Each version of the tactile stimulus had a different parameter randomized (intensity, frequency,
timing), making it incongruent with the auditory stimulus on that parameter. Multiple songs and audiotactile maps were tested, however only one song (“szárad a száj”) is analyzed in this abstract.
The second screen and the last screen in the rating task were identical to test reliability.

3. RESULTS
3.1 Reliability
Between the two groups, only the normal hearing listeners were consistently reliable. Reliability
was tested by taking the Cronbach’s alpha of the repeated rating screen for each listener. A Cronbach’s
alpha over 0.7 is considered acceptably reliable. A box plot with the Cronbach’s alpha for both groups
can be seen in Figure 1.

Figure 1: A box plot showing the Cronbach’s alpha, a measure of reliability, for both listener groups.
Normal hearing listeners were consistently more reliable than cochlear implant listeners.
Modeling
A linear model of the arcsine transformed rating data (10) showed statistically significant
coefficients (p<=0.05) for intensity congruence and timing alignment, and interactions between timingalignment and reliability, and timing alignment and listener group (CI versus normal hearing). A linear
model was used to predict the effect size of each audio-tactile congruence on listener ratings. A larger
estimated coefficient, as seen on Figure 2, signifies a larger effect size on listener ratings. A positivecoefficient corresponds to a positive effect, and a negative coefficient corresponds to a negative effect.
Reliability was a binary parameter: if the listener had a Cronbach’s alpha of at least 0.7, the listener
was considered reliable.
3.2

Figure 2: A plot showing the estimated coefficients for a linear model of the arcsine transformed rating
data. A positive coefficient means a positive effect on listener ratings. Both normal hearing (NH) listeners
and cochlear implant (CI) listeners showed similar patterns, except for the “non-reliable” CI listeners, who
had a small and negative estimated coefficient for timing alignment.

4. DISCUSSION
4.1 Rating Task
Linear modeling of the rating task for the normal hearing listeners showed that intensity
congruence and timing alignment can be used to increase listener ratings. The results from the ratingtask concur with previous work by Aker, et al (3).
In the case of the cochlear implant listeners, the effect of intensity congruence and timingalignment were shown to have an interaction with reliability. Reliable cochlear implant listeners wereshown to have similar results to normal hearing listeners. For reliable cochlear implant listeners,
intensity congruence and timing alignment had a positive effect on ratings (Figure 2).
For non-reliable cochlear implant listeners with a Cronbach’s alpha of less than 0.7, only
intensity congruence showed a positive coefficient for ratings (Figure 2). Due to the poor reliability,
it is difficult to draw conclusions from the coefficients themselves. However, it is worth investigating
the source of such high unreliability in cochlear implant listeners.
4.2 Source of Unreliability
Consistently poor reliability in cochlear implant listeners suggests a reason intrinsic to the test,
either related to the use of the cochlear implant itself, or a confounding factor between the two listenergroups.
The source of the unreliability in cochlear implant listeners could be due to several factors. First,
cochlear implant listeners have been shown to have lower audio-tactile integration than normal
hearing listeners at implantation, which is restored the longer the cochlear implant is used. The amountof time since implantation could affect reliability.
Musical training could also influence reliability. Aker, et al. (3) showed an effect of musicaltraining on how much audio-tactile congruence affected user preferences. Listeners with more musicaltraining were influenced more by congruence in the auditory and tactile stimuli. As cochlear implantlisteners often change their music listening habits after implantation, musical training and background
could be a confounding factor or interaction.
Third, there are several key confounding factors between the two listener groups. In particular, theage groups of the two listener groups were different. The cochlear implant listeners in the present
study were between 41-82, while the normal hearing listeners recruited for the current study were 2030 years old. As age is a predictor of tactile sensitivity (8), this could be a potential confound.

4.3 Implications
The present study has two main implications. First, cochlear implant listeners with a Cronbach’s
alpha higher than 0.7, signifying reliable results, have similar audio-tactile congruence preferences asnormal hearing listeners. This implies certain cochlear implant listeners could benefit from a similaraudio-vibrotactile coding scheme that normal hearing listeners do.
However, the large number of cochlear implant listeners with Cronbach’s alpha below 0.7 suggestthat only a cohort of cochlear implant listeners might be able to benefit in the same way as normal
hearing listeners. An investigation into predictors of the reliability is necessary to measure thepotential benefit of vibrotactile music enhancement for cochlear implant listeners.
4.4 Future work
Future work will focus on investigating correlates and possible causes for the discrepancy inreliability. The experiment will be repeated on normal hearing listeners age-matched with the cochlear
implant listeners to investigate the age-confound. The effect of musical training and cochlear implant
experience will also be investigated.

ACKNOWLEDGEMENTS
This work was supported by the William Demant Foundation and the Innovation Fund Denmark.

REFERENCES
1.

Fletcher MD. Can haptic stimulation enhance music perception in hearing-impaired listeners? Front
Neurosci. 2021;1123.

2.

Merchel S, Altinsoy ME. The influence of vibrations on musical experience. J Audio Eng Soc.
2014;62(4):220–34.

3.

Aker, Scott C, Innes-Brown, Hamish, Faulkner, Kathleen F, Vatti, Marianna, Marozeau, Jeremy. Effectof audio-tactile congruence on vibrotactile music enhancement. (submitted);

4.

International Telecommunication Union. Method for the subjective assessment of intermediate quality
level of audio systems. 2014.

5.

McDermott HJ. Music perception with cochlear implants: a review. Trends Amplif. 2004;8(2):49–82.

6.

Landry SP, Guillemot JP, Champoux F. Temporary deafness can impair multisensory integration: a studyof cochlear-implant users. Psychol Sci. 2013;24(7):1260–8.

7.

Landry SP, Guillemot JP, Champoux F. Audiotactile interaction can change over time in cochlear implantusers. Front Hum Neurosci. 2014;8:316.

8.

Merchel S, Altinsoy ME. Psychophysical comparison of the auditory and tactile perception: a survey. J
Multimodal User Interfaces. 2020;14(3):271–83.

9.

Sharp A, Houde MS, Maheu M, Ibrahim I, Champoux F. Improved tactile frequency discrimination inmusicians. Exp Brain Res. 2019;237(6):1575–80.

10. Studebaker GA. A" rationalized" arcsine transform. J Speech Lang Hear Res. 1985;28(3):455–62.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0755

Audio-Tactile Installations To Augment Music Perception For People
With Hearing Impairment.
Jeremy MAROZEAU1;
1Music and Cochlear Implants lab, Department of Heath Technology, Technical University of Denmark,Denmark

ABSTRACT
Going out to a concert has been perceived as one of the activities people missed the most during this pandemic.
However, some people's hearing impairment prevents them from enjoying this basic social event. So far, only
a telecoil-loop system installed in a few theatres can provide a little auditory reinforcement. To fill that need,
we investigate different systems that can provide tactile reinforcement for them. The systems vary by theircomplexities and the part of the body targeted. The hands are excited by metal and wood plates mounted on
springs equipped with shakers. The shape of the plate is designed to elicit different resonance frequencies atdifferent locations. The belly is excited by a large wood multifaceted structure that can be hugged. Each faceis equipped with small shakers tuned to specific resonance frequencies. Finally, the audience is seated in awood chair equipped with dowels that will directly stimulate the audience's back and legs through the chairs.
The installations are displayed in museums and festivals, where the audience is invited to experience music
through vibration. After that, the audience filled out a questionnaire to assess the success of each device.
Keywords: Audio-Tactile, Cochlear Implant, Music, Art

1. INTRODUCTION
We perceive the world through all our senses. For example, the experience of drinking a good wine
involves our sense of taste, smell, vision, and touch. Similarly, we do not only hear music, but we alsofeel it through vibrations. We can easily forget it because our sense of hearing is more adapted totransmitting acoustic waves than our sense of touch. However, someone with a hearing loss will rely
more heavily on this other sense to compensate for their deficit.
The main treatment to restore sound perception is either amplifying the sound (the hearing aid)or directly stimulating the auditory nerve (the cochlear implant). However, combining audio andtactile stimulation will allow us to use the body's full potential and maximize the amount of sound we
can restore. This year, we worked with two Swiss artists and the Museum of Art and History of Geneva
to create a permanent art installation that provides people with a handicap a richer musical experience.
Three prototypes were built and tested. The first one was a simple podium equipped with a shaker.
The second one was designed to decompose the frequency components into different regions thatvibrate maximally different body parts (see figure 1, right). This mechanism was inspired by thebasilar membrane that decomposes the sound into filter bands before activating individual nerve fibers.
The third prototype decouples the vibrations from the structure supporting the body to increase
frequency selectivity and decrease vibration damping by the user's weight. Additionally, thesympathetic bars transmit the vibrations directly to the users (see figure 1, left). They are currently
used for their monthly concert ("le Retour du Jeudi"). In collaboration with the University of Geneva,
the audience is invited to fill out a survey after each show to give feedback on their experience.
The second set of devices has been created to engage a younger audience at the demand of theDanish association "Rum for Aestitik." It has been displayed at the festival Distortion for kids. Thisset of installations was designed to evoke the theme of nature and the forest. Each part was built tohave specific places of excitation with the minimum amount of shakers.

1

jemaroz@dtu.dk

Figure 1: The first auditory loungers placed in the Museum of Geneva

The vibrating forest
Each of those installations was mainly designed by master's students from the Technical Universityof Denmark under the guidance of the main author. The visual design was supervised by a local artist(kiralworld.com).
1.1 The flower
A large flower of 1 meter in diameter was created out of plywood (see figure 2 left). Each petal
was tuned to resonate at a specific resonance frequency. A Clark Synthesis (Gold) shaker is placed a t
the center of the flower (the ovary). This shaker was selected because of its larger frequency responsethat allows both the petal's vibration at low frequency and the audible higher frequency range of themusic.
1.2 The tree
The tree is a structure made for hugging. It is composed of a 2-meter vertical metal rod equippedwith two shakers positioned at 90 degrees angles that excite the rod perpendicularly. Twenty -fourwood panels (see figure 2 center) are distributed along the rod and are connected with perpendicular
metal bars. Therefore, different frequencies will excite different mode s of excitation of the rod.
Depending on those modes, different wood panels will resonate. A delay of 10 ms between the shakers
was introduced to decouple them.
1.3 The spiralThis installation was created by cutting a spiral into a metal sheet with a CNC machine. Then byhanging the center of the plate, a downward spiral appeared. The basilar membrane inspired the twoand-a-half turn of the spiral. Additionally, as in the cochlear, the width of the spiral decreased
gradually, letting a thicker part with a low resonance frequency at the apex and a thinner part with ahigh resonance frequency at the end. A shaker was connected to the top of the spiral (see figure 2,
right).

Figure 2: Three different audio-tactile installations: The flower, the tree, and the spiral

2. CONCLUSIONS
Restoring hearing through vibration is not a new idea. However, it is fair to say that no commercial
device has emerged as an elegant and efficient solution. It can be argued that to achieve real progress,
the problem must be tackled simultaneously from many different angles. This project involves
engineers, acousticians, audiologists, musicians, neuroscientists, designers, and patient assoc iations
to create the most appropriate solutions.
Each of these installations will be thoroughly tested by a diverse audience. The first result of theinstallation will be presented during the conference.

ACKNOWLEDGEMENTS
Many people have contributed to this project. First and foremost, the students that have designed
and built the different prototypes: The chairs: Tommer Tchelet, Brent Reissman and Gabriele Ravissa;
The flower: Carolina Costa da Silva Gaudeoso and Justin Sabaté; The tree: Jonas Bolduan and Giorgio
Papadimitriou; the Spiral: Mikkel Hermansen and Christian Svensson. The artists were the first
motivation of the project: Charlotte Nordin [ref] and Raphael Ortis (resonance-nocturne.ch). Kiral
World supervised and redesigned the visual aspect of each installation. At the University of Geneva,
Prof Daniel Hubert and Denis Kilicel collected the data at the Museum of Geneva. The association
Rum for Aestetik organized the festival Distortion for Born, and finally, the Museum of Geneva hosted
the project. A William Demant Foundation Grant funded the project.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0204

Observation of the vibration response on the skull bones by bone
conduction with the fresh frozen cadaver
Wan-Ho CHO1; Ji Hyeon LEE2; Young Joon SEO2
1

Division of Physical Metrology, Korea Research Institute of Standards and Science, Korea

2

Department of Otorhinolaryngology, Yonsei University Wonju College of Medicine, Korea

ABSTRACT
To investigate the dominant pathway of contralateral sound transmission, the vibration response on the skull
bones was observed including the facial side of skull. The measurement was conducted with a freshly frozen
cadaver and the bone conduction stimulation was applied to the mastoid using a bone vibrator. The
acceleration responses and coherence between excitation and receiver points were observed by using triaxial
accelerometers at the opposite side of mastoid and midline points, such as forehead, nose and chin. The
results show that the bone conduction stimulation is efficiently transferred for the mid-frequency range, 0.7 –2 kHz. Moreover, it was observed that both of magnitude response and coherence show the significantly highvalue at the face side of midline excepting the chin.
Keywords: Bone conduction, Vibration transmission, Coherence function

1. INTRODUCTION
The measurement of vibration response on the skull bones was conducted to investigate thedominant pathway of bone conduction. Many previous studies have been conducted to investigate
the transmission characteristics of bone conduction stimuli [1-4]. As an excitation and monitoring
points for bone conduction transmission, the area near the mastoids and occipital area has been
mainly considered in most of previous studies. In this study, it was observed that the vibrations of
the skull bones, including facial bones, vary in frequency, and transverse waves appear differently ineach bone.

2. SPECIMEN AND MEASUREMENT SETUP
As a specimen, the head of a fresh human cadaver cut from the body at the neck was investigated.
The preparation of cadaver followed the protocol of Yonsei University. The donor consented to thepost-mortem use of their bodies for science according to the terms of the human body donation
program of the Institute of Anatomy, Yonsei University.
Two different coordinate systems were employed as shown in Figs. 1. Figure 1(a) shows thecoordinate for the mastoid to mastoid transmission and figure 1(b) is for the points on midline. Two
triaxial accelerometers were placed and maintained at each mastoid position for whole measurement.
Three triaxial accelerometers were attached at each position of midline and moved to different
positions along the midline in turns from chin to occipital.

1chowanho@kriss.re.kr

(a)

(b)

Figure 1 – Coordinate system and accelerometer position: (a) mastoid positions, (b) on the midline

3. RESULTS

FRF (dB re. x of Sensor1)
Coherence (with x of Sensor1)

FRF (dB re. z of Sensor1)
Coherence (with z of Sensor1)(b)

FRF (dB re. y of Sensor1)(a)Coherence (with y of Sensor1)

Figure 2(a) shows the frequency response at the mastoid position excited by the bone vibrator at
the opposite mastoid position. The relatively high level is observed in the frequency range of 0.7 – 2
kHz referred to the y-direction of excitation position. The coherences of signal are observed asshown in Fig. 2(b) and it is observed that the coherence higher than 0.9 is observed in the frequency
range of 0.5 – 2 kHz. This means that the efficient and coherent signal transmission is possible
within the frequency range of 700 Hz to 2 kHz with the excitation of normal direction for the
mastoid to mastoid transmission.
Figure 3 shows the frequency response at the midline positions excited by the bone vibrator at
mastoid position. In this result, it is observed that the measured value on the chin was slightly lower
than that at the other points. Similar to the case between mastoid positions, it is also observed thatthe y-direction components of the excitation point are efficiently transferred compared to the otherdirections. The coherences between the accelerations measured at the midline positions and the
mastoid opposite the excitation position are shown in Fig. 4. The chin shows a consistently lower
coherence than the other positions, regardless of the propagation direction. The overall tendency is
consistent with the mastoid-to-mastoid case shown in Fig. 2(b), except for the chin position, which
consistently shows the lower values than the other positions. This means that a frequency range of
0.7-2 kHz has a high correlation for most of skull positions. However, for frequencies of higher than
2 kHz, the other position can be more efficient than the case of mastoid-to-mastoid transmission.

Figure 2 – (a) Frequency response of the cadaver head at the opposite mastoid positions excited by the bone
vibrator at mastoid position, and (b) its coherence (color of each line means the direction shown in Fig.
1(a))

FRF (dB re. x of Sensor1)
FRF (dB re. x of Sensor1)
FRF (dB re. x of Sensor1)

FRF (dB re. y of Sensor1)

FRF (dB re. z of Sensor1)
FRF (dB re. z of Sensor1)
FRF (dB re. z of Sensor1)(c)

FRF (dB re. y of Sensor1)(b)

FRF (dB re. y of Sensor1)(a)Figure 3 – Frequency response of the cadaver head at the midline positions excited by the bone vibrator at
mastoid position (color of each line means the measured position shown in Fig. 1(b)): (a) azimuth direction,
Coherence (with x of Sensor1)
Coherence (with x of Sensor1)
Coherence (with x of Sensor1)

Coherence (with z of Sensor1)
Coherence (with z of Sensor1)

Coherence (with y of Sensor1)(c)

Coherence (with z of Sensor1)(b)

Coherence (with y of Sensor1)(a)Coherence (with y of Sensor1)(b) elevation direction, (c) normal direction

Figure 4 – Coherence between each accelerometer output of the cadaver head excited by the bone vibrator at
mastoid position opposite to the excitation (color of each line means the measured position shown in Fig.
1(b)): (a) azimuth direction, (b) elevation direction, (c) normal direction

4. CONCLUSION
The vibration response on the skull bones was observed including the facial side of skull. The
measurement was conducted with a freshly frozen cadaver and the bone conduction stimulation wasapplied to the mastoid using a bone vibrator. The acceleration responses and coherence between
excitation and receiver points were observed by using triaxial accelerometers at the opposite side of
mastoid and midline points. The results show that the bone conduction stimulation is efficiently
transferred for the mid-frequency range, 0.7 – 2 kHz and it was observed that both of magnitude
response and coherence show the significantly high value at the face side of midline excepting the chin.
This indicates that such a frontal side of the face had a high contralateral BC efficiency, and the signaltransmission from the specific midline to mastoid can be efficient.

ACKNOWLEDGEMENTS
This work was supported by the KRISS grant (No. GP2022-0002-11).

REFERENCES
1. Håkansson B, Carlsson P, Brandt A, Stenfelt S. Linearity of sound transmission through the human skull
in vivo. J Acoust Soc Am.1996;99:2239-2243.
2. Eeg-Olofsson M, et al. Transmission of bone conducted sound - correlation between hearing perception
and cochlear vibration. Hear Res 2013;306:11-20.
3. Stenfelt, S, Goode, RL. Transmission properties of bone conducted sound: measurements in cadaverheads. J Acoust Soc Am. 2005;118:2373-2391.
4. Stenfelt S. Skull vibration during bone conduction hearing. Proc 20th ICSV; 7-11 July 2013; Bangkok,
Thailand, p. 1394-1401.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0205

The performance of EEG-based auditory attention decoding
according to the speech volume in a dichotic listening task: apreliminary study.
Jiyeon Ha1,2; Yoonseob Lim1,2; Jae Ho Chung1,3
1. Department of HY-KIST Bio-convergence, Hanyang University, Seoul 04763, Korea
2. Center for Intelligent & Interactive Robotics, Artificial Intelligence and Robot Institute, Korea Institute of
Science and Technology, Seoul 02792
3. Department of Otolaryngology-Head and Neck Surgery, College of Medicine, Hanyang University, Seoul
04763, Korea

ABSTRACT
Auditory attention decoding (AAD) has been developed to detect the attended speech using
electroencephalography. Recently, a real-time AAD method using linear decoder model has been
introduced lead to expand the application of AAD in terms of auditory brain computer interface.
However, people focus on sounds of varying volumes in everyday conversation, so the question ofwhether AAD is possible in various sound stimuli arises. The present study aimed to investigate the effectof the difference in volume between two competing speech in a dichotic listening paradigm for online AAD.
Most comfortable level (MCL) and dichotic speech recognition threshold (SRT) were evaluated. And the‘Sound level for Speech intelligibility (SI) of 90%’ and ‘Sound Level for SI 50%’ were also calculated. In
online AAD task, four different sound level (MCL, MCL-20dBA, Sound level for SI of 90% and SI 50%)
were introduced as attended speech, while the sound level of ignored speeches was fixed at the MCL. There
was no difference in AAD performance based on the four sound levels conditions. In addition, volume
difference was not significantly correlated with the individual decoder accuracy. This preliminary study
identified that lowering attended speech volume to SRT level had no effect on AAD.
Keywords: Online auditory attention decoding, Electroencephalography, most comfortable level, dichotic
speech recognition threshold

1. INTRODUCTION
Selective auditory attention, also known as the "cocktail party effect," is the ability to focus on aninteresting sound or speech in a multi-talker environment. Several hypotheses supported the selective
auditory attention. Due to limited capacity and performance of human brain, human information
processing system has “bottle neck” to select useful sound signal before auditory processing (1).
Another hypothesis proposed that all sensory information is attended to and evaluated fo r meaning
before it enters short-term memory. Furthermore, the filter analyzes the semantic properties of thesound information and allows sound with relevant sound information to pass through to short term
memory while ignoring irrelevant information (2). In addition, it is suggested that there is a thresholdmechanism in selective auditory attention in which words from the unattended stream of information
can grab one's attention by volume or meaning. (3)Several studies have been introduced to find the attended or ignored speech signals by assessing
brain activity. Auditory cortical activity is entrained to the temporal envelope of speech signals, andthe envelope of attended speech have shown that the entrainment is dominantly represented overignored speech (4, 5). Based on these insights, auditory attention decoding (AAD) can provide
information on which speech the listener is attending to, using electroencephalography (EEG; (6)) andmagnetoencephalography (MEG; (7)). O’sullivan et al (6) demonstrated a decoder model that canselect the attended speech based on the correlation between the envelope of reconstructed speech andthat of the actual speech in a dichotic listening paradigm. An online AAD method that can decode
auditory attention in real time was proposed based on this decoder model (8).
Recently, several AAD studies have also shown that auditory attention decoding is possible forvarious listening conditions such as the presence of background noise (9), presentation of stimuli
through loudspeakers located at diverse positions (10, 11) and using stimuli filtered by a Head-Related
Transfer Functions (12, 13). However, there were few studies that investigated the effect of speechvolume on AAD performance.
In the real listening environment, the speech signals have diverse volume level depending on thesound source, and the loudness may be an important factor for auditory attention detection. However,most AAD studies with dichotic listening tasks used two competing speeches with equal soundintensity. Therefore, we investigated the effect of the difference in sound level (sound pressure level(SPL)) between two competing talkers on the AAD performance.

2. METHOD
Subjects
We recruited 7 participants (2 females; ages 19 to 28 years) with bilateral normal hearing. All
participants were all right-handed and native Korean speakers. If a participant had a history of a neurologicaldisorders or hearing loss, they were excluded from the study. Written informed consent was obtained fromall participants. This experiment was approved by the Institutional Review Board (IRB) at the Hanyang
University and performed in accordance with the Declaration of Helsinki and Good Clinical Practiceguidelines (IRB code: 202207-006).
Subject-specific sound intensity of stimuli
To determine equivalent speech intelligibility levels across participants due to differences inparticipant’s hearing ability, subject-specific sound intensity was determined before an online AAD
task.
2.2.1 Most Comfortable Level (MCL)
Most comfortable level was determined in a sound-proof room. Speech signals were introduced
via in-ear earphone and speech sound level was increased or decreased by 5 dBA steps based on the
response of the participant (14). And participant asked to select the most comfortable sound speech
level in each ear (15).
2.2.2 Dichotic speech recognition threshold (SRT)
We conducted a dichotic speech recognition threshold (SRT) task that the subjects had to repeat
the sentence presented in the attended side, while ignoring speech stimulus presented in the other ear.
The ignored speech stimuli was introduced at the subject-specific MCL of the corresponding side,and the sound level of target stimulus was adaptively decreased until the volume level at which thesubject could correctly recognize 50% of the words in the sentence (16).
In the dichotic SRT task, the Korean Matrix sentence (17) narrated by a female was used as target
signal. The ignored stimuli were derived from one of the stories presented in the AAD task (female
voice), however, the sentences were extracted at random to avoid
The task was conducted for right and left ear (always right ear first), 15 sentences were presented
respectively. Speech intelligibility (SI) was estimated by fitting a sigmoid function on the sentence
recall scores of the last 13 sentences. Based on fitted SI curves, we obtained subject-specific soundlevel for speech intelligibility of 90% and 50% in the dichotic SRT tasks. The subject-specific soundlevel for SI was demonstrated as SPL difference (SPL of attended stimulus - MCL). The mean soundlevel for SI (of right and left ear) was used for online AAD task.
Stimuli & Procedure
Four speech signals were used as stimuli for the online AAD task. Two speech signals (out offour) was constructed based on two stories, “Journey to the center of the earth” and “Twenty thousand
leagues under the sea” by Jules Verne. Speech signals were recorded by two male speakers and edited
to a length of one minute for 14 trials of AAD training. The other two speech signals were excerpted
from the Korea listening test of the “Korean SAT”. Speech signals also recorded by same malespeakers and consisted of two sections of 28 different topics (each with length of 1 min) . Speechsilences longer than 0.5 seconds were truncated to 0.5 seconds to prevent attention loss brought on bya lack of auditory stimulation. Speech stimulations were presented via air-conduction earphones (ER2,Etymotic Research, IL, USA).
The experiment was divided into two sessions (Table 1). The first experiment (14 trials) was used
to build the online decoder model, while the second experiment (28 trials) were for evaluating online
decoder performance in a different speech volume levels. Thus, a tota l of 42 trials of AAD task waspresented per subjects. In the first experiment, the two different stories were introduced to each ear
with subject specific MCL. And the subjects were asked to attend to the story in one ear, while ignoring
the story in the other ear. In the second experiment, two different speeches excerpted from the Korean
SAT to each ear simultaneously and were instructed the same attention task.
The SPL of ignored speeches was fixed at subject-specific MCL. The SPL of attended speeches waspresented at four different sound levels: ‘MCL’, ‘MCL-20dBA’, ‘Sound level for SI 90%’ and ‘Sound
Level for SI 50%’. Here, ‘Sound level for SI 90%’ and ‘Sound Level for SI 50%’ indicate subjectspecific sound level for speech intelligibility of 90% and 50%. The SPL of attended speech was equalto (‘MCL’ condition) or lower than (the other conditions) that of ignored speech. For ‘MCL’ and‘MCL-20dB’ conditions, the SPL difference between attended and ignored speech was equal acrossall subjects (0 dBA and -20dBA). In contrast, for ‘SI 90%’ and ‘SI 50%’ conditions, SPL differenceswere different across all subjects. The four different sound levels were presented randomly.
The attention direction in which the attended speech was presented was equally distrib uted to leftor right over all presentations. The subjects could identify the attention direction thro ugh two types
of attention cues: directional sound (tone sound twice, length 500 ms) and a directional arrow on amonitor. The directional sound was presented only to the side of the attention direction, and at thesame time, the directional arrow pointed to the attention direction.
After each trial, participants were asked to fill out four multiple -choice questions to confirm theirattention: two pertaining to the target speech, and the other two pertaining to the ignored speech. For
second experimental session, the subjects were asked to answer, “How well could you hear the target
speech?”, as a numeric rating scale of 1 (easy to hear) to 5 (hard to hear) after each trial.
Table 1 – The sound profile for sound intensity in the dichotic listening tasks for AAD task.
Experiment

AttendedsessionExperiment 1

Ignoredstimulus

Stimulus SPL(dBA)

Stimulus SPL(dBA)

stimulus

Story 1

MCL

Story 2

MCL

Korean SAT 2

MCL

MCL
Experiment 2

Korean SAT 1

MCL-20dBA
SI 90 %
SI 50 %MCL : most comfort level of loudness.
SI 90% : Sound pressure level for speech intelligibility of 90%.
SI 50% : Sound pressure level for speech intelligibility of 50%.
Online AAD model
We performed an online decoder model proposed by previous study to conduct online AAD task
(8). The conventional decoder model reconstructs the envelope of attended speech from EEG signals
and estimated envelope correlations (by Pearson’s correlation) with reconstructed envelope andoriginal envelope of speech. The online decoder model applied an overlapping sliding window thatslides a window at a given interval to the conventional decoder model to extract the snippets of EEG
signals and the envelope of the speech. We used same parameter with Baek et al (8) such as time lag,regularization parameter, width of a window and the moving interval of the window fo r the online
decoder model.
The online decoder accuracy was evaluated by correlating the reconstructed envelope of speechwith the original envelope of attended and ignored speech. The reconstructed speech envelope wasconsidered to be estimated correctly for the window if the envelope correlation with attended speechenvelope is higher than that with the ignored speech envelope. The online decoder accuracy wasestimated as a percentage of the correctly evaluated windows over every trial held for the decoder
model evaluation (experiment 2).
Data acquisition & analysis
EEG signals were recorded using fifteen electrodes (Fz, Cz, C3, C4, P7, P8, Pz, F7, F8, F3, F4,
T7, T8, P3, and P4) with OpenBCI (Cyton with Daisy Biosensing Boards, OpenBCI, USA). The
electrode at the CPz served as the reference point for EEG signals. EEG signals were re -referenced to
a common average reference. The recorded EEG signals were bandpass filtered between 0.5 and 8 Hz
and downsampled to 64 Hz and z-scored. Speech envelopes were estimated using the Hilbert transform,
downsampled to 64 Hz, and z-scored before the experiment. During the experiment, EEG signals were
continuously transmitted to the laptop through BrainFlow (Parfenov, A BrainFlow) at a sampling rateof 125 Hz. All data analysis was performed in real time using a customized Python script.

3. Results
Subject-specific sound intensity of stimuli
The mean MCL is measured 62.14 ± 3.93 dBA across all subjects (shown in figure 1 (a)). Figure 1(b) shows results of SPL difference between attended and ignored speech for SI 90% and SI 50% in
the dichotic SRT task.
In the case of the task for the left ear, average SI 90% is -34.07 ± 3.43 dBA and average SI 50%
level is -37.97 ± 4.21 dBA. In the case of the task for the right ear, average SI 90% level is -29.9 ±4.74 dBA and average SI 50% level is -35.95 ± 2.82 dBA. For the mean of left and right results, SI
90% is -31.99 ± 3.96 and SI 50% is -37.03 ± 3.35 dBA. The subject-specific MCL and mean resultsof each SI 90% and SI 50% were used for the online AAD task.

Figure 1 – Results of Most comfort level (MCL) and dichotic speech recognition threshold (SRT) task forseven subjects. (a) Mean MCL and standard deviation across seven subjects. Each symbol represents
individual result for MCL. (b) The SPL difference between attended sentence and ignored sentence
corresponding to speech intelligibility 90% and speech intelligibility 50% extracted by the dichotic SRT
task. Each caption includes results of dichotic SRT task for left and right ear and mean of both ear. Eachsymbol represents individual results for corresponding conditions.
Behavior test
During the Experiment 1, the average correct answer rate for target speech was 87.76 ± 5.87 %,and for ignored speech was 25.51 ± 8.10 % across all subjects (figure 2 (a)).
During the Experiment 2, the average correct answer rate for target speech at MCL condition (i.e.
no SPL difference), was 75.51 ± 6.97% (for ignored speech was 34.69 ± 11.97 %). For ‘MCL-20dBA’
condition, the average correct answer rate for target speech was 76.53 ± 6.79 % (33.67 ± 18.77% forignored speech). For ‘SI 90%’ and ‘SI 50%’ conditions, the average correct answer for target speechwere 63.27 ± 12.66 % and 56.12 ± 18.64% (for ignored speech were 34.69 ± 11.97 % and 44.90 ±
17.84 %) as shown figure 2 (b).
Self-reported subjective scales
Figure 2 (c) shows the results of self-reported subjective scales on a range of 1-5 (easy-hard). The
scale significantly decreases with SPL difference (r = -0.9280, p < 0.001), indicating that Participants
reported that the smaller the SPL difference, the better ability to listen to the target speech.

Figure 2 – Results of behavior test for experiment 1 and 2, and subjective scales. (a) The average result of
behavior test for experiment 1. (b) The average result of behavior test for experiment 2. (c) Individual
rating of subjective scales according to individual SPL difference of seven subjects for the four conditions.
The symbol in purple, yellow, red and blue correspond to ‘SI 50%’, ‘SI 90%’, ‘MCL-20dB’ and ‘MCL’
conditions (7 subject * 4 conditions = 28 points). Each symbol represents individual results for all subject.
The gray line is a least-squares regression of rating of subjective scales with SPL difference.

Online AAD performance result of difference sound level conditions
Figure 3 shows the average decoder accuracy and standard deviation for seven participants. The
average decoder accuracy was obtained by averaging all decoder accuracy across each conditions foreach participant.
As a result of the decoder for ‘MCL’ condition, the average decoder accuracy was 65.00±
6.15 %, %. For ‘MCL-20dBA’ condition, the average decoder accuracy was 67.88±10.86 %. For each
‘SI 90%’ and ‘SI 50%’ condition, the average decoder accuracy was 70.32±13.77 % and69.34±14.07 %. Also, we performed Wilcoxon’s signed rank test between ‘MCL’ and each three
conditions. There was no significant difference anywhere.
To investigate the effect of volume difference between attended and ignored speech, we used a
Pearson’s correlation between SPL difference and individual decoder accuracy or envelope correlation(Figure. 4). The individual decoder accuracy at corresponding SPL difference for each subject isshown in figure 4 (a). There was no significant correlation for individual decoder accuracies acrossall SPL difference conditions (r = -0.057, p > 0.05).
Figure 4 (b) demonstrated the correlation between SPL difference and envelop correlation. In thecase of envelope correlation related to attended speech, envelop correlations does not significantly
correlate with SPL difference (r = 0.029, p > 0.05). Regarding correlation for ignored speechaccording to SPL difference, all conditions did not show significant correlation between SPL
difference and envelop correlation (r = -0.046, p > 0.05).

4. Discussion
In this study, we explored the effect of volume difference between attended and ignored speech on
the online AAD performance for normal hearing individuals. The average decoder accuracy did not
significantly differ between conditions with SPL difference (‘MCL-20dBA’, ‘SI 90%’ and ‘SI 50%’
conditions) and conditions without SPL difference (‘MCL’ condition). Also, we found no significant
correlation between individual decoder accuracy (or envelope correlation) and SPL difference acrossfour SPL difference conditions (p > 0.05).

Figure 3 – Mean and standard deviation of online decoder accuracy for four sound level conditions.

Figure 4 – Individual decoder accuracy and correlation correlations for the difference in SPL between
attended and ignored speech for seven subjects. In addition, gray regression lines represent the predicteddata for all SPL difference conditions. The symbols in purple, yellow, red and blue correspond to ‘SI 50%’,‘SI 90%’, ‘MCL-20dB’ and ‘MCL’ conditions (7 subject * 4 conditions * each 7 trials = 196 points). Eachsymbol represents individual result for results for corresponding conditions. (a) Individual decoder
accuracy according to individual SPL difference. (b) Individual correlation correlations related withattended (top) and ignored (bottom) speech according to individual SPL difference.
In this work, there was no difference of AAD performance between condition without SPL
difference and conditions with SPL difference. This result means that if the sound intensity of target
speech is presented to the listener as the intensity between MCL and SRT among dynamic range ofhuman hearing, AAD can perform well even when there was SPL difference.
We can guess that this might be the results of listening effort for auditory attention. The SPL
difference makes it more difficult to focus on target speech, requiring more listening effort from thesubjects, and hence better neural entrainment in envelope of target speech could be represented. The

better neural representation might help increase the decoder accuracy under three conditions with SPL
difference. Das et al (9) investigated the effect of background noise level and speaker location on
AAD performance. They showed that in condition of the 180° separation between two competing
speakers (same as speaker location in our work), average decoder accuracies of condition withbackground noise was higher than those without background noise. This also means that AAD can bewell performed, even when faced with difficult listening environment for speech understanding. In
addition, the SPL difference between two competing speech make it easier to distinguish between
them, helping to select auditory attention object. In this regard, although conditions with SPL
difference are hard to focus on target speech compared to condition without SPL difference, AAD
performance in conditions with SPL difference might not different from condition withou t SPL
difference depending on the attending ability.
To application of AAD to real life that exist sound sources with various sound level simultaneously,
it was necessary to investigate the effect of SPL difference between competing talkers on AAD.
However, to confirm whether AAD can apply to auditory brain computer interface, this work requires
for further investigation of individual with hearing loss. The current results show that online AAD
can perform even in the situation where SPL of target speech is lower than other speeches.

ACKNOWLEDGEMENTS
This research was supported by the National Research Council of Science & Technology (NST)
grant by the Korea government (MSIT) (No. CAP21051-200). This research was supported by the
National Research Foundation of Korea (NRF) grants funded by the Korea government(2021R1F1A1054810). This work was supported by the KIST Institutional Program (Project No.
2E00000-00-000).

REFERENCES
1. Goldstein S, Naglieri JA. Executive functioning. A Goldstein, Sam. 2014.
2. Deutsch JA, Deutsch D. Attention: Some theoretical considerations. Psychological review. 1963;70(1):80.
3. Treisman AM. Strategies and models of selective attention. Psychological review. 1969;76(3):282.
4. Mesgarani N, Chang EF. Selective cortical representation of attended speaker in multi-talker speech
perception. Nature. 2012;485(7397):233-6.
5. Woldorff MG, Gallen CC, Hampson SA, Hillyard SA, Pantev C, Sobel D, et al. Modulation of early
sensory processing in human auditory cortex during auditory selective attention. Proceedings of the
National Academy of Sciences. 1993;90(18):8722-6.
6. O'Sullivan JA, Power AJ, Mesgarani N, Rajaram S, Foxe JJ, Shinn-Cunningham BG, et al. Attentional
selection in a cocktail party environment can be decoded from single-trial EEG. Cerebral cortex.
2015;25(7):1697-706.
7. Koskinen M, Viinikanoja J, Kurimo M, Klami A, Kaski S, Hari R. Identifying fragments of natural speechfrom the listener's MEG signals. Human brain mapping. 2013;34(6):1477-89.
8. Baek S-C, Chung JH, Lim Y. Implementation of an Online Auditory Attention Detection Model with
Electroencephalography in a Dichotomous Listening Experiment. Sensors. 2021;21(2):531.
9. Das N, Bertrand A, Francart T. EEG-based auditory attention detection: boundary conditions forbackground noise and speaker positions. Journal of neural engineering. 2018;15(6):066017.
10. Horton C, Srinivasan R, D’Zmura M. Envelope responses in single-trial EEG indicate attended speakerin a ‘cocktail party’. Journal of neural engineering. 2014;11(4):046015.
11. Fuglsang SA, Dau T, Hjortkjær J. Noise-robust cortical tracking of attended speech in real-world acoustic
scenes. Neuroimage. 2017;156:435-44.
12. Das N, Biesmans W, Bertrand A, Francart T. The effect of head-related filtering and ear-specific decoding
bias on auditory attention detection. Journal of neural engineering. 2016;13(5):056014.
13. Kerlin JR, Shahin AJ, Miller LM. Attentional gain control of ongoing cortical speech representations in a“cocktail party”. Journal of Neuroscience. 2010;30(2):620-8.
14. Cox RM, Alexander GC, Taylor IM, Gray GA. The contour test of loudness perception. Ear and hearing.
1997;18(5):388-400.
15. Sherlock LP, Formby C. Estimates of loudness, loudness discomfort, and the auditory dynamic range:
normative estimates, comparison of procedures, and test-retest reliability. Journal of the American
Academy of Audiology. 2005;16(02):085-100.
16. Brand T, Kollmeier B. Efficient adaptive procedures for threshold and concurrent slope estimates forpsychophysics and speech intelligibility tests. The Journal of the Acoustical Society of America.
2002;111(6):2801-10.
17. Kim KH, Lee JH. Evaluation of the Korean matrix sentence test: Verification of the list equivalence andthe effect of word position. Audiology and Speech Research. 2018;14(2):100-7.

ABS-0273

Developing the LISTENING IN SPATIALIZED NOISE SENTENCES(LiSN-S) test in Brazilian Portuguese
Bruno MASIERO(1) , Leticia R. BORGES(2) , Harvey DILLON(3) , Maria Francisca COLELLA-SANTOS(4)(1) Communication Acoustics Lab, University of Campinas, Brazil, masiero@unicamp.br(2) Faculty of Hearing Sciences, Pontifical Catholic University of Campinas, Brazil, leticiarborges@yahoo.com.br(3) Department of Linguistics, Macquarie University, Australia, harvey.dillon@mq.edu.au(4) Faculty of Medical Sciences, University of Campinas, Brazil, mfcolell@unicamp.br

ABSTRACTThe Listening in Spatialized Noise Sentences (LiSN-S) is a test to evaluate auditory spatial processing currently only available in the English language. To develop the LiSN-S test in Brazilian Portuguese it wasnecessary to first devise a presentation software and prepare a speech database, recorded by professional
voice actresses, to be inserted into the software. These sentences were presented to a group of 35 adults (agedbetween 19 and 40 years) and 24 children (aged between 8 and 10 years), all with normal hearing—tested
through tone audiometry, speech audiometry and tympanometry testing—and good performance at school.
We used a logistic curve describing word error rate versus presentation level, fitted for each sentence, to selecta set of 120 sentences and adjust their amplitudes for equal intelligibility. We are currently applying the test
to a new group of students to determine normality criteria for age groups between 6 and 11 years. Hopefully,
the results obtained in this research will help to disclose the importance of studying spatial processing, especially in children with complaints related to hearing difficulties in noisy environments, thus contributing to
the diagnosis and help audiologists to plan a more fully and efficient rehabilitation of Brazilian children.

Keywords: spatial hearing, central auditory processing disorder, speech-in-noise test

1

INTRODUCTION

The Central Auditory Processing (CAP) can be defined as the efficiency and effectiveness with which thecentral auditory nervous system uses auditory information. It refers to the perceptual processing of auditory
information and the neurobiological activity underlying this processing that gives rise to electrophysiological auditory potentials (1, 2). The analysis and interpretation of normal and efficient auditory information
involves several sub-processes or skills necessary for auditory processing to occur and includes the neuralmechanisms underlying a variety of auditory behaviors, such as: sound localization and lateralization; auditory discrimination; auditory pattern recognition; temporal aspects of hearing (integration, discrimination,
resolution and temporal masking); auditory performance on competitive acoustic signals (including dichotic
listening) and auditory performance on degraded acoustic signals (3, 4, 5).
Central Auditory Processing Disorder (CAPD) is a dysfunction in the central auditory nervous system(CANS) that leads to hearing difficulties (2). It can occur in patients with neurological damage to the
CANS (6), the elderly (7) and children (5). In the pediatric population, there are several possible causesof the disorder, including delayed maturation of the CANS (6), presence of ectopic cells in the auditory
system (8), otitis media, as well as CNS injury in a minority of cases (9). The estimated prevalence of thisdisorder in school-age children ranges from 2 to 5% (4).
Since the 1950s, numerous tests have been created to assess the central functions of hearing. Morerecently, the auditory mechanism of binaural integration was identified as one of the main central processes
that should be included in the CAP assessment, particularly in cases where there are complaints or clinicalinformation of difficulty in locating the sound source or in hearing in a noisy environment, which can becaused by a deficit in this domain (1). The ability to understand speech in a noisy environment is directly
related to the listener’s ability to use binaural cues to differentiate the location of the sound source from thelocation of the noise (10). Thus, to understand speech in a noisy environment, it is necessary to focus on the
person we want to hear and simultaneously suppress competing sounds from different places. This processcan be called spatial processing (11) and allows us to obtain important information, necessary to be able tounderstand speech and participate in conversations.
The Listening in Spatialized Noise–Sentences (LiSN-S) test (12) was developed in Australia, with theobjective of evaluating the spatial processing of individuals with complaints related to CAPD. It is a speechin-noise test, applied through a dedicated computer software, via a headset that produces a three-dimensional
virtual auditory environment. The three-dimensional effect is achieved by processing the speech stimulus by
the head-related transfer function (HRTF) (12). A simple repetition response protocol is used to obtain the
Speech Recognition Threshold (SRT), defined as the signal-to-noise ratio that produces 50% intelligibility for
sentences presented together with competitive speech stimuli, in this case, children’s stories. The phrases, ortarget stimulus, were prepared by an Australian speech pathologist and recorded by a single woman. The
children’s stories were written by an Australian writer and recorded by the same woman who recorded thetarget phrases, but also by two other female speakers. The phrases and the competitive message are presented
to both ears simultaneously. By using appropriate HRTFs, target phrases are processed to be perceived asalways coming directly in front of the listener (0◦ azimuth), while competitive speeches are processed to beperceived as coming from 0◦ azimuth or coming from +90◦ and −90◦ azimuth simultaneously. Moreover,competitive speech may vary the speaker’s vocal identity in relation to target phrases.
LiSN-S is an assessment test that has been shown to be sensitive in children with CAPD and in otherpopulations with impaired binaural integration processing (13, 14). Children with a history of otitis media
are at risk for long-term functional hearing loss related to spatial processing skills, as assessed by LiSN-S. In
addition, risk was found to be related to age at onset and duration of otitis media episodes. They highlighted
the need for early intervention in these cases and an awareness that there may be a change in auditory skills
even after sound detection returns to normal. They found that further studies need to be developed withchildren with well-documented otitis media (15).
The main purpose of this study is to develop the LISN-S test in Brazilian Portuguese. In this manuscript
we describe the development of a software to assess spatial processing (Section 2), the development ofspeech material for the Brazilian Portuguese database, which was inserted into the software (Section 3),
and the sentence intelligibility equalization procedure (Section 4). The definition of normality criteria foran age group from 6 to 11 years old is underway. In addition, the results obtained in normal children
will be analyzed and compared with those who had a history of otitis media in childhood. The research
was approved by the Ethics in Research Committee of Campinas State University under No. 3,462,572 andparticipants were not reimbursed for their involvement.

2

PRESENTATION SOFTWARE

The software and the graphical user interface (GUI) were initially developed in Matlab using the App Designer tool. The graphical interface was developed to allow the examiner to start the test application procedure, record the number of words correctly repeated by the patient and visualize the test progress. Basedon the data provided by the examiner, the software automatically adjusts the next sentence’s sound pressurelevel in order to determine the SRT.
Although Matlab’s App Designer platform allows the construction of powerful graphical interfaces, theuse of Matlab software has two disadvantages: Matlab is a proprietary software with a pricey license and,even running a compiled version of the application, it is necessary to install the 1GB+ Matlab Runtime torun in the background.
For these reasons, it was decided to rewrite the test application interface in Python. As an interpreted
language, Python has a lower computational performance than compiled languages such as C++ and Java.
However, this does not become relevant for this specific application because it does not require very large
processing power. In this case, the simplicity requirement overrides the performance requirement. For thedevelopment of the graphical interface (see an example in Figure 1), the PyQT5 library was chosen, which iscompatible with QtDesigner, a software used to design interfaces by dragging-and-dropping. Audio processing
was done with the PyAudio library. Finally, the code was compiled using the cx-Freeze library, which
resulted in an executable file of 196 MB, where over the half of this file size is composed of audio files. A
video illustrating how the LiSN-S test work can be viewed at https://youtu.be/8NBxeIqr8s4.

3

AUDIO DATABASE

Speech therapists from the research team elaborated 187 sentences (9 with 3 words, 49 with 4 words, 77
with 5 words, 46 with 6 words and 6 with 7 words), considering the vocabulary, syntax and semantic content
suitable for children from 6 years old. In addition, two children’s stories were selected that were used as acompetitive speech stimulus.

Figure 1. Example of the graphical user interface programmed in Python to apply the LiSN-S tests in
Brazilian Portuguese.
The recording of the target sentences and the competitive stories were performed in an anechoic chamberby three vocal actresses of the same age group, with similar vocal characteristics and accent. They were all
instructed to speak in a clear voice, normal rhythm and avoid stressing any words. Speaker A recorded alltarget sentences and the two competitive stories while speakers B and C recorded only the two competitive
stories. An AKG microphone, model c3000, was used for the recording. Recorded material was saved as
WAV files using a sampling frequency of 44.1 kHz and 16 bit depth.
The audio material was edited using Audacity. Prolonged pauses were removed during the editing process
to ensure the storytelling was smooth and at a constant intensity. Each competitive story lasts just over two
minutes while the target sentences lasted, on average, 2 s. Furthermore, a period of 100 ms of silence wasinserted immediately before and after each story while a period of 500 ms of silence was inserted immediately
before and after each sentence.
The root mean square (RMS) level of all sentences and stories was computed in decibels (dB) and averaged. All signals were then normalized to the calculated average value. After normalization the stimuli were
further attenuated by 7 dB to ensure that saturation does not occur after convolution with the HRTFs. The
HRTFs used were measured from a 10 years old extracted from an openly available Children’s HRTFs andAnthropometric Scans for Auditory Research (CHASAR) dataset (16).

4

INTELLIGIBILITY EQUALIZATION

The following study was conducted to determine the relative intelligibility of the LiSN-S sentences and toadjust the level of the sentences for equal intelligibility.
4.1 Participants
The procedure described above was performed with 35 adults (30 females and 5 males) aged between 19
and 40 years, median of 22 years, and 24 children (8 female and 16 male) aged between 8 and 10 years,
median of 9 years. All had normal hearing and good performance at school.
4.2 Materials
Test was applied through a Samsung Windows PC coupled to a Sennheiser HD 280 PRO headset via an
RME Madiface audio interface.
4.3 Design and Procedure
The equivalence study used the same adaptive procedure used by the LiSN-S test itself for the “low cue”
condition, that is, target sentence and competitive stories presented with the same voice and virtually in frontof the listener. Each volunteer was instructed to repeat the phrase heard after the beep and to ignore thecompetitive story. The number of correctly repeated words was registered into the software.
Target sentences and competing histories were presented simultaneously in six blocks of 31 sentences—inorder to keep the blocks relatively short and thus minimize patient fatigue—with a period of rest in between
sections. For the first block there were an extra 2 training sentences presented with a fixed signal-to-noise
ratio (SNR) of 7 dB. The remaining sentences were presented in an adaptive fashion.
The competitive stimulus was presented at a fix sound pressure level of 65 dB SPL1 . Before each target
sentence, a warning signal was presented, consisting of a pulsatile tone of 1 kHz and 60 dB SPL, modulated
by a Hann window, with a total duration of 200 ms. The sentences were presented 500 ms after the warning.
The target sentences were presented in random order. In the first block there was a training phase consisting of the first three sentences, that were presented at a fixed SNR of 7 dB, i.e., the competitive stimulus
is presented at 65 dB SPL and the target stimulus is presented at 72 dB SPL. From the presentation of the
fourth sentence onward, the presentation levels were adjusted in 4 dB steps until the first reversal occurs, thatis, the first response with less than 50% of the correct words. If the number of words correctly identified
in the previous sentence is exactly 50%, the level of the next target stimulus is kept unaltered. If the firstreversal is negative, i.e., the participant responded with less than 50% correct word during training, the blockmust be restarted. From the first positive reversal onward, sentences were adjusted in 2 dB steps.
At the end of each block the SRT was calculated as the average of at least three midpoints, and in turn,each midpoint was calculated as the average of the level of a positive reversal and its subsequent negative
reversal. From the second presentation block onward, the first target sentence was presented at a level 3 dB
above the SRT found in the previous block and always using 2 dB steps.
The presentation software recorded the order in which all sentences were presented, the level at which
they were presented, as well as the number of correctly repeated words for each target sentence.

5

RESULTS

The data collected as described in the previous section was analyzed in order to obtain a method to select themost adequate sentences to compose the Brazilian Portuguese LiSN-S and later correct the selected sentences
for equal intelligibility.
5.1 Exploratory analysis
For all exploratory statistical analysis that follow we discarded the first 3 sentences presented to each participant, as these sentences were presented at a fixed level of 72 dB, as a training phase.
We started evaluating the effect of age. A one-way analysis of variance (ANOVA) was performed tocompare the effect of age (children VS adults) on the raw target sentence level (TSL). The analysis revealed
that there was a significant difference between both groups (F(1,10854) = 2530.32, p = 0). However, for theintelligibility equalization procedure we are interested in the adjusted target sentence level (aTSL), i.e., thetarget sentence level in relation to the individual’s estimated SRT. When we repeated the one-way ANOVA
to compare the effect of age on the aTSL, the analysis revealed that the difference is no longer significant(F(1,10854) = 0.88, p = 0.347). Therefore, for the remainder of this analysis we combine both groups.
Next we analyze the individual performance. To do so, we executed a one-way ANOVA to compare theeffect of the individual on the aTSL. The analysis revealed that there was a statistically significant difference in mean aTSL between at least two groups (F(58,10797) = 1.86, p = 8.42e-05). Tukey’s HSD Test
for multiple comparisons indicated that two individuals stand out from the group. After discarding the responses of these two individuals, there is no longer any significant difference between the remaining subjects(F(56,10431) = 1.03, p = 0.418).
Regarding the target sentences, we conducted a one-way ANOVA to evaluate the effect of individual
sentences on the aTSL. The analysis revealed that there was a statistically significant difference in mean aTSL
between at least two sentences (F(186,10669) = 3.02, p = 0). Furthermore, a one-way ANOVA on the effectof error rate on the aTSL revealed that the average aTSL is different for the different hit rates (F(18,10836) =179.77, p = 0). Both results were expected and corroborate the need for sentence intelligibility equalization.
Additionally, we analyzed if the number of words in the target sentence did influence the results. A
one-way ANOVA on the effect of number of words on the aTSL revealed there was a significant difference
between groups (F(4,10850) = 4.29, p = 0.0018). Tukey’s HSD Test for multiple comparisons indicated thatthe mean value of aTSL was significantly different between sentences with five words and sentences withthree or seven words. However, we have no indication that we should rule out a group of sentences by their
number of words.
1 Originally, LiSN-S presents the competitive stimulus at 55 dB SPL (12) . However, as the SRT usually occurs with an SNR of 10 to 15 dB,
the threshold target stimulus level is around 40 dB SPL, which can, in itself, make speech recognition difficult.

Figure 2. Distribution of aTSL in which the data was presented and the rate of correct words. On the left
we have the marginal distribution of the rate of correct words and below the marginal distribution of theaTSL. The orange dots show the average number of correctly recognized words in bands of 1 dB aTSL. The
diameter of the points is proportional to the logarithm of the number of presentations.
5.2 Distribution of samples
Figure 2 shows the rate of correct words versus the target sentence level in relation to the estimated SRT,
also known as aTSL, when combining all 187 target sentences. The histogram at the bottom shows thefrequency in which sentences were presented (corrected by the individual SRTs) and it seems to approach anormal distribution. On the other hand, the histogram on the left shows that the target sentences were mostoften either not recognized (31%) or completely recognized (43%), and it appears to have a beta distribution.
For a fixed value of aTSL, the rate of correct words can be modeled as a binomial distribution, since theact of correctly recognizing each word can be understood as a Bernoulli experiment, where the output of theexperiment is whether the word is correctly recognized or not. The maximum of the binomial distribution
occurs at the expected value, given by E[X] = np, where n is the number of trials, in our case the numberof words presented at a given aTSL, and p is the probability of correctly recognizing a word at that aTSL.
A logistic curve models the behavior of p as a function of the aTSL.
5.3 Sentence Selection
The data collected was used to determine the relative intelligibility of the 187 prepared sentences. Based onthat, we were able to select 120 sentences to be used in the Brazilian Portuguese LiSN-S and to adjust thelevel of these selected sentences for equal intelligibility.
To determine the relative intelligibility we first discarded the training sentences and the responses fromtwo participants, as discussed in Section 5.1. We also discarded three sentences perceived as “bad” duringlistening tests. For each of the remaining sentences, we combined the results of all volunteers and fit a
psychometric function, defined by the logistic curvep=exp(a + bX),
1 + exp(a + bX)(1)

where p is the percent correct word and X is the aTSL. One can verify that S = b/4 is related to the slopeof the steepest portion of the curve and that the ratio R = −a/b represents the SRT, i.e., the aTSL needed toachieve 50% correct identification of words in a sentence. The logistic curves fitted to our data are presented
in Figure 3 while Figure 4 shows the distribution of S and R.
Sentences with a “bad” fit, measured by the coefficient of determination (R2 )2 smaller than 0.5, werediscarded.
To select the final 120 sentences, we decided to choose those sentences “closest” to the central point of
this distribution presented in Figure 4. Among the various possible ways of defining “distance”, we chose to
2 The coefficient of determination is defined as the proportion of the variation in the dependent variable (in our case the rate of correct words)
that is predictable from the independent variable (in our case the aTSL).

Figure 3. Logistic curves fitted for each of the tested sentences.

Figure 4. Distribution of R and S values for each of the tested sentences. In red the median of the values,in blue the points related to the selected phrases and in orange the discarded phrases.

Figure 5. Logistic curves of the 120 selected sentences after adjustment for the same intelligibility.
use the ℓ∞ -norm3 of the values normalized as a function of their standard deviations. This resulted on thefollowing upper and lower bounds for S and R:• R − median(R) < 1.9 dB or R − median(R) > −1.9 dB;• S < 0.75 or S > 0.28.
The selected points are shown in blue in Figure 4, with the discarded ones in orange. Notice how theselection region has a rectangular shape.
5.4 Intelligibility adjustment
After selection of 120 sentences, the amplitude of the selected sentences was adjusted to equal intelligibility.
To do so, we vary the level of each sentence by the amount needed to make all the psychometric functions
have the same level for 50% accuracy. This adjustment is made by amplifying each phrase by −R, which isequivalent to shifting the curves to the right or left in Figure 3. The result of the curves after adjusting forintelligibility equalization is given in Figure 5.

6

CONCLUSIONS

We were able to craft a presentation software for the Brazilian Portuguese version of the LiSN-S test andproduce the audio material to be used with the test. After proper audio editing and normalization, sentenceswere presented to 59 individuals in an adaptive fashion, recording the sentence’s presentation level and number of words repeated correctly. Based on these results, we defined a criteria to selected 120 sentences as theBrazilian Portuguese LiSN-S sentences and further equalized the selected sentences for equal intelligibility.
Research is ongoing and software and speech material are ready for application in a selected sample todetermine the test’s normality criteria. It is hoped that the results obtained in this research may provide support for understanding the functioning of the central auditory nervous system structures involved in binauralinteraction tasks, from the cochlear nucleus to the auditory cortex in Brazilian children. In addition, we hope
to disclose the importance of studying spatial processing, especially in children with complaints related tohearing difficulties in noisy environments, to contribute to the diagnosis and help audiologists to plan a more
fully and efficiently rehabilitation.

ACKNOWLEDGEMENTS
This work was fully supported by the São Paulo Research Foundation (FAPESP), grants #2017/25267-0 and#2017/08120-6. The opinions, hypotheses and conclusions or recommendations expressed in this material are
3 We could use other distance measures, such as the ℓ norm, which gives me as a selection region a circle around the median, or the ℓ norm,
2
1
whose selection region will be a rhombus, that is, which maximizes the selection of sentences with the same slope and same displacement as
the median

the authors’ responsibilities, and do not necessarily reflect FAPESP’s views.

REFERENCES
1.

Association ASLH. (Central) auditory processing disorders. Tech. rep. 2005. Available from: http://www.asha.org/docs/html/TR2005-00043.htm

2.

Audiology AA of. Clinical Practice Guidelines for the diagnosis, treatment and Management of Children
and Adults with Central Auditory Processing Disorder. 2010. Available from: www.audiology.org/
publications-resources/document-library/central-auditory-processing-disorder3.

Association ASLH. Task Force on Central Auditory Processing Consensus Development: Central Auditory Current Status of Research and Implications for Clinical Practice. Journal of the American Academyof Audiology 1996 :41–54

4.

Central auditory processing disorders: new perspective. San Diego, CA: Singular Publishing Group,
1997

5.

Bellis TJ and Ferre JM. Multidimensional approach to the differential diagnosis of central auditory
processing disorders in children. Journal of the American Academy of Audiology 1999 Jun; 10:319–
28

6.

Musiek FE, Chermak GD, Weihing J, Zappulla M, and Nagle S. Diagnostic accuracy of established central auditory processing test batteries in patients with documented brain lesions. Journal of the American
Academy of Audiology 2011 Jun; 22:342–58

7.

Gates GA. Central presbycusis: an emerging view. Otolaryngology Head Neck Surgey
147:1–2

8.

Boscariol M, Guimarães CA, Hage SR, Garcia VL, Schmutzler KM, Cendes F, and Guerreiro MM.
Auditory processing disorder in patients with language-learning impairment and correlation with malformation of cortical development. Brain Development 2011 Nov; 33:824–31

9.

Chermak GD and Musiek FE. Neurological Substrate of Central Auditory Processing Deficits in Children. Current Pediatric Reviews 2011; 7:241–51. Available from: https://www.ingentaconnect.
com/content/ben/cpr/2011/00000007/00000003/art00012

10.

Moore BCJ. An Introduction to the Psychology of Hearing. San Diego, CA: Academic Press, 1997

11.

Cameron S, Dillon H, Glyde H, Kanthan S, and Kania A. Prevalence and remediation of spatial processing disorder (SPD) in Indigenous children in regional Australia. International Journal of Audiology
2014 May; 53:326–35

12.

Cameron S and Dillon H. Development of the Listening in Spatialized Noise-Sentences Test (LISN-S).
Ear Hear 2007 Apr; 28:196–211

13.

Rance G, Ryan MM, Bayliss K, Gill K, O’Sullivan C, and Whitechurch M. Auditory function in children with Charcot-Marie-Tooth disease. Brain 2012 May; 135:1412–22

14.

Rance G, Ryan MM, Carew P, Corben LA, Yiu E, Tan J, and Delatycki MB. Binaural speech processing in individuals with auditory neuropathy. Neuroscience 2012 Dec; 226:227–35

15.

Tomlin D and Rance G. Long-term hearing deficits after childhood middle ear disease. Ear Hear 2014;35:e233–242

16.

Braren HS and Fels J. Towards Child-Appropriate Virtual Acoustic Environments: A Database of HighResolution HRTF Measurements and 3D-Scans of Children. International Journal of Environmental Research and Public Health 2021 Dec; 19. DOI: 10.3390/ijerph19010324

2012

Jul;

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0281

32-channel Cochlear Implant
Kyou Sik Min1; Hoseung Lee1; Jeong U Lim1; Woojin Ahn1; Soowon Shin1; Gwangjin Choi1; Yoonhee
Ha1; Doohee Kim1; Seung Ha Oh2
1
2

TODOC Co., Ltd., South Korea

Seoul National University Hospital, South Korea

ABSTRACT
Cochlear implant is the most successful active implantable electronic neural prosthetic device in history.
Recently, the number of recipients has reached 1 million. The device consists of implantable pulse generator(IPG) and wearable speech processor (SP). And each has various components such as cochlear electrodearray, hermetic package, receiver-transmitter coils, neural stimulation/recording integrated circuit, software,and hardware for SP unit, and so on. In this study, we introduce the 32-channel cochlear implant and its
preliminary assessment results. The electrode array is manufactured by laser-micromachining process insteadof manually handled conventional method. While the electrode array is manufactured by novel process,materials for electrode, wire and encapsulation are identical to conventional cochlear implants. Mechanicaland electrochemical characteristics are measured to be feasible for clinical use. Including manufacturable 32channel cochlear electrode array, IPG shows Good Laboratory Practice (GLP)-compliant ISO-10993
biocompatibility.
Keywords: Cochlear Implant, Neural Prosthesis, Implantable Pulse Generator

1. INTRODUCTION
Cochlear implant is an implantable medical device which restores hearings of people with severe
hearing loss. It delivers electrical pulses to hearing nerves through neural interfaces which is inserted
in cochlea. For last 40 years, it has restored hearings of more than 1 million people throughout the
globe. [1] However, there are still more than 10 million people who can regain hearing by getting
cochlear implant surgery, and the number is adding up by 112,700 each year. [2] It has been
continuously pointed out that the cost exceeding 20,000 USD of cochlear implant is the key barrier ofglobal penetration. Besides the raw materials, the manual fabrication is one of the factors that makecochlear implant cost high. In recent article, we introduced manufacturable 32-channel cochlear
electrode array and its preliminary assessment results. [3] On the contrary to conventional research
for low-cost cochlear neural interfaces fabricated by semiconductor process, we employ micro laser structuring for semi-automated process. In this session, authors share recent progress of firstcommercial 32-channel manufacturable cochlear implant.

2. MATRIALS AND FABRICATION
Exterior of cochlear implant consists of electrode array, lead wire, hermetic package, receiver coiland encapsulation (Figure 1). Raw materials for these components should be selected in considerationof safety and efficacy. Platinum 90%-Iridium 10% alloy is used in electrode contacts and wires array.
As shown in the previous article [3], the bonding process between each electrode and wire pair isneedless because the electrode and wire array are carved from 20-micron-thick platinum-iridium alloy
film. To secure durability of the connection between electrode array and hermetic package, the 32channel lead wire array employs 3-dimensional serpentine structure. Titanium 90%-Aluminum 6%Vanadium 4% alloy (Ti6Al4V-ELI) is used for hermetic package. Gold receiver coil and 32-channel
wires are welded to feedthroughs on hermetic package. All components are encapsulated by1

ceo@to-doc.com

biocompatible silicone elastomer.

Figure 1 32-channel Cochlear Implant Exterior

3. PRECLINICAL ASSESSMENT
3.1 Biocompatibility
ISO 10993-compliant tests were conducted by GLP-certified facilities. The system went throughimplantation (ISO 10993-6) and sub-chronic systemic toxicity (ISO 10993-11) tests at Korea Testing& Research Institute (KTR, Gwacheon, South Korea). And Cytotoxicity (ISO 10993-5), maximization
sensitization (ISO 10993-10), irritation & intracutaneous reactivity (ISO 10993-10), and pyrogen (ISO10993-11) tests were done by Korea Testing Laboratory (KTL, Seoul, South Korea). All listed test
reports say that the raw materials went through cochlear implant manufacturing process were passed
the biocompatibility tests.
3.2 Mechanical and Electrochemical Properties of Electrode Array [3]Stiffness of the electrode array is measured by pushing a point on the electrode surface which is 6
mm away from fixation zig while the probe makes the electrode shape from its normal shape to 30 °deformation. Vertical and horizontal stiffness are measured to be 19.8 mN, and 15.7 mN, respectively,
which shows that its stiffness is comparable to one of commercial cochlear implant (Nurobiosys,
South Korea). The impedance of the electrode contact is measured to be 3.11 ± 0.89 kOhm and −48.9± 7.61° at 1 kHz, and the average CSCc is 5.09 mC/cm 2 , which is feasible for clinical use. The
proposed electrode array is inserted human scala tympani (ST) model to measure insertion andextraction forces which can be applied during surgeries. The insertion force is measure d as 17.4 mN
at 8 mm from the round window while metal wire-based electrode array showed 19.2 mN at the same
point. Maximum insertion force is 61.4 mN at the end of the insertion (24 mm). Which is nearly half
insertion force (111.8 mN) of the conventional wire-based electrode array.

4. DISSCUSSION
We propose two usefulness of the 32-channel cochlear electrode array in terms of efficacy and cost.
Original motivation of the research has been laid in how we can reduce the cost of cochlear implant
to be affordable even in poor circumstances. Reducing cost can be achieved in several ways such as
moving factories to developing countries which require less labor cost, changing raw materials
through innovative material engineering or introducing a new fabrication technology. In previous
research, the author tried to fabricate the flexible cochlear electrode array on liquid crystal polymer
which is known to be biocompatible and semiconductor-process-compatible. [4] [5] But, employing
new materials which have never been used as active implantable medical device and have never beenverified by authorized test facilities based on ISO-compliant tests have huge barriers to market
penetration: regulatory process. The authors made small shift in the research direction from novelty
to feasibility while maintaining the manufacturability of the device. Introducing the ultra-short pulse
laser structuring process, we propose manufacturable high-density cochlear electrode array based onconventional materials.
Current commercial cochlear electrodes have 22, 16, and 12 physical contacts depending on

manufacturers. It has been said that there are little differences in performance relevant to the numberof active channels. [6] [7] The main hypothesis is that the electric field interference between adjac ent
channels make little difference if electrode contacts are too close. However, in recent research, it issaid that there is positive correlation between number of channel and sound recognition. [8] [9] [10]For further research, new cochlear implant platform with more than 22 channel is needed. We expect
suggesting platform can contribute to cochlear implant and hearing research field.

ACKNOWLEDGEMENTS
This work was supported by the Korea Medical Device Development Fund grant funded by the
Korea government (the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, theMinistry of Health & Welfare, the Ministry of Food and Drug Safety) (Project Numb er: RS-202200140444)

REFERENCES[1] F.-G. Zeng, "Celebrating the one millionth cochlear implant," JASA Express Letters, p. 077201, 2022.
[2] S. J. S. K. Şahin Mİ, "Cochlear Implantation: Vast Unmet Need to Address Deafness Globally," Otology& Neurotology, vol. 38, no. 6, pp. 786-787, 2017.
[3] S. Shin, Y. Ha, G. Choi, J. Hyun, S. Kim, S.-H. Oh and K.-S. Min, "Manufacturable 32-Channel
Cochlear Electrode Array and Preliminary Assessment of Its Feasibility for Clinical Use,"
Micromachines, vol. 12, no. 7, p. 778, 2021.
[4] K. S. Min, S. H. Oh, M.-H. Park, J. Jeong and S. J. Kim, "A polymer-based multichannel cochlear
electrode array," Otology ¥& Neurotology, vol. 35, no. 7, pp. 1179-1186, 2014.
[5] T. M. Gwon, K. S. Min, J. H. Kim, S. H. Oh, H. S. Lee, M.-H. Park and S. J. Kim, "Fabrication andevaluation of an improved polymer-based cochlear electrode array for atraumatic insertion,"
Biomedical microdevices, vol. 17, no. 2, pp. 1-12, 2015.
[6] K. E. Fishman, R. V. Shannon and W. H. Slattery, "Speech recognition as a function of the number ofelectrodes used in the SPEAK cochlear implant speech processor," Journal of Speech, Language, andHearing Research, vol. 40, no. 5, pp. 1201-1215, 1997.
[7] L. M. Friesen, R. V. Shannon, D. Baskent and X. Wang, "Speech recognition in noise as a function of
the number of spectral channels: Comparison of acoustic hearing and cochlear implants," The Journalof the Acoustical Society of America, vol. 110, no. 2, pp. 1150-1163, 2001.
[8] N. B. Croghan, S. I. Duran and Z. M. Smith, "Re-examining the relationship between number ofcochlear implant channels and maximal speech intelligibility," The Journal of the Acoustical Society ofAmerica, vol. 142, no. 6, pp. EL537-EL543, 2017.
[9] K. A. Berg, J. H. Noble, B. M. Dawant, R. T. Dwyer, R. F. Labadie and R. H. Gifford, "Speech
recognition as a function of the number of channels in perimodiolar electrode recipients," The Journalof the Acoustical Society of America, vol. 145, no. 3, pp. 1556-1564, 2019.
[10] K. Berg, J. Noble, B. Dawant, R. Dwyer, R. Labadie, V. Richards and R. Gifford, "Musical soundquality as a function of the number of channels in modern cochlear implant recipients," Frontiers inneuroscience, vol. 13, p. 999, 2019.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0539

Feature analysis and evaluation for estimation of mild cognitive
impairment from spontaneous speech of Korean
Sunghee DONG1; Hyung-Bae JEON2
1

Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute (ETRI),
Daejeon, 34129, Republic of Korea

2

Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute (ETRI),
Daejeon, 34129, Republic of Korea

ABSTRACTThe main purpose of this study is to identify significant acoustic and linguistic features from the spontaneous
speech of Korean elderly individuals with cognitive declines. It is important to differentiate the early stageof dementia from the normal aging, but few studies have been conducted in Korean. Knowing the phonetic
and linguistic changes of different stage of cognitive impairment is important for detection and analysis. We
analyzed the difference of acoustic and linguistic characteristics between the normal, mild cognitive
impairment (MCI) and Alzheimer's disease (AD) in elderly groups from spontaneous speech obtained during
the speech task through statistical analysis and classification. As a result, we found a meaningful feature set
in phonetic characteristics of MCI and AD stage distinguished from intact subject. Moreover, acoustic featureshows better performance in classification between normal and MCI/AD compared to linguistic feature. This
study can provide a technical background in clinical and engineering researches for detection of cognitive
declines in Korean elderly.
Keywords: MCI, Alzheimer’s disease, speech

1. INTRODUCTION
As people ages, the cognitive decline in a person is emerging as a critical social issue (1). In
particular, it is important to diagnose it in the early stage and slow the disease’s progress before thedementia stage. Therefore, interest for effective diagnose of mild cognitive impairment (MCI) is
increasing in many studies (2). In addition, it is important to monitor cognitive state of aged populationin their real life, because they have difficulty in visiting doctor. Recently, speech is known as aneffective indicator of cognitive impairment from its early to late stage (3). Because most of thecognitive impairment significantly impacts on memory, language and speech, spontaneous speechreflects pathological changes in aged brain.
However, there are few studies on the influence of cognitive decline on speech in Korean. In thisstudy, the main purpose was to analyze how MCI and Alzheimer's disease (AD) affects phonetic andlinguistic characteristics in Korean using machine learning techniques and statistical analysis
techniques. We expect this paper can contribute future development of automatic detection system of
MCI and dementia from spontaneous Korean speech.

12dsh7560@etri.re.kr (Correspondence)hbjeon@etri.re.kr

2. METHOD
2.1 Dataset
This study used the Cognitive dysfunction diagnosis voice/conversation dataset built by the Korea
Intelligent Information Society Agency and published on the AI hub (https://aihub.or.kr). The dataset
consists of audio data and transcription files for clinical speech tasks for 191 normal elderly people,
134 patients with MCI, and 131 patients with AD. The subject’s personal information was deleted in
preprocess step. Subject’s information was attached as age, gender, MMSE pre-test score, and doctor’s
diagnosis.
2.2 Acoustic Feature
We used an open-source audio feature extraction tool, openSMILE (https://github.com/
audeering/opensmile) to extract acoustic feature from audio dataset (4). Type of feature set waseGeMAPSv02 at the functionals feature level, resulting a total of 88 dimensions of acoustic features
were extracted from each instance. One-way analysis of variance (ANOVA) test was used as statisticalanalysis method to compare two groups. Binary classification was performed for normal versus MCI
and/or AD groups using a logistic classifier trained with 10-fold cross validation.
2.3 Linguistic Feature
We used an opened pre-trained model KoBERT (https://github.com/SKTBrain/KoBERT), pretrained Korean language model, based on bidirectional encoder representations from transformer(BERT) (5). A BERT-based sentence embedding vector was used as a feature and a shallow layerclassifier was attached to the last layer. In fine-tuning training, parameters other than the last layer of
BERT were frozen and fully weighted. Because the given methodology suffers from the lack oftraining data, the MCI group and the AD group were considered as one group during classification.
Test accuracy was obtained by 5-fold cross validation.

3. RESULT and DISCUSSION
3.1 Statistical Analysis
We compare the acoustic feature between normal and MCI and AD groups.
As a result, there was a significant (p<0.01) difference between the normal and MCI in 34 out of
88 features. In the case of the normal versus AD group, 37 features out of 88 features showedsignificant (p<0.01) difference. Only 17 features showed significant difference in both the normal vs
MCI and normal vs AD groups in common. There were more features showing statistical difference
between normal vs AD, which suggests more severe cognitive impairment affects more acoustic
components compared to normal people. Figure 1 shows four examples of acoustic features for Normal,
MCI, AD groups. Asterisk represents significant difference (p<0.01) between the corresponding
groups.

Figure 1 – Values of four types of acoustic features a) Loudness_sma3_amean, (b)F0semitoneFrom27.5Hz_stddevNorm, (c) Mean_unvoiced_segment_length, (d) mfcc1_stddevNorm inNormal, MCI, and AD groups. X-axis shows the name of group and y-axis shows the feature value.
Asterisk with line represents statistical difference between indicated groups.
3.2 Classification Results
We investigated the classification results between groups using acoustic and linguistic feature forspeech tasks. To fulfill the main purpose of this study, four spontaneous speech tasks (task 5: picture
description of the park, task 9: talking about the happy memory, task 10: talking about the sad memory,task 11: talking about what happened yesterday) were selected among 11 tasks. For the four tasks, we
conducted three types of classification; 1) classification of normal and MCI using acoustic feature, 2)classification of normal and AD using acoustic feature, and 3) classification of normal and MCI+AD
using linguistic feature. Figure 2 shows the average of test accuracy from the classification models.
The results show that for all the tasks, classification accuracy is the best in the case of normal vs AD
groups using acoustic features. For three tasks (task 5, 9, 10) of all tasks, linguistic feature produces
the worst performance in classification. The best accuracy 83 % occurs when discriminating AD fromnormal people using acoustic feature.
Classification results revealed that acoustic feature is better to discriminate cognitive decline thanlinguistic feature, although it is difficult to compare the classification results of acoustic and linguisticfeatures because of their methodology. There are two possible reasons: First, phonetic characteristics
could be more affected by MCI and AD because it is hard to regard the given tasks are completely
spontaneous. Secondly, the amount of data is too small to solve the mismatch between pre-trained
KoBERT model used in linguistic feature extraction. Therefore, if fully spontaneous speech isevaluated on more sufficient training data, there could be better achievement in detection of cognitive
declines. Moreover, classification results can be further improved by combining acoustic andlinguistic feature in feature level.

Figure 2 – Bar graph of test accuracy in classification between 1) Normal versus MCI groups using
acoustic feature, 2) Normal versus AD groups using acoustic feature, and 3) Normal versus MCI/AD
groups using linguistic feature.

4. CONCLUSION
The main purpose of this study is to investigate the impact of cognitive declines in phonetic andlinguistic characteristics for Korean speech. We first investigate the statistical differences between
normal, MCI, and AD groups using acoustic feature. The result of statistical analysis shows more
phonetic characteristics are affected by AD than MCI compared to normal person. Moreover, we foundacoustic features provide better performance in classification of normal and cognitively declined
group, compared to linguistic feature. This study presents the possibility with methodology of using
acoustic and linguistic features when studying cognitive impairment such as MCI and Alzheimer’sdisease in Korean.

ACKNOWLEDGEMENTS
This research was supported by the National Research Council of Science & Technology(NST)
grant by the Korea government (MSIT) (No. CAP21051-300).

REFERENCES
1. Brayne C. The elephant in the room—healthy brains in later life, epidemiology and public health. Nature
Reviews Neuroscience. 2007;8(3):233-9.
2. Petersen RC, Negash S. Mild cognitive impairment: an overview. CNS spectrums. 2008;13(1):45-53.
3. Tóth L, Hoffmann I, Gosztolya G, Vincze V, Szatlóczki G, Bánréti Z, et al. A speech recognition-based
solution for the automatic detection of mild cognitive impairment from spontaneous speech. Current
Alzheimer Research. 2018;15(2):130-8.
4. Eyben F, Wöllmer M, Schuller B, editors. Opensmile: the munich versatile and fast open-source audio
feature extractor. Proceedings of the 18th ACM international conference on Multimedia; 2010.
5. Devlin J, Chang M-W, Lee K, Toutanova K. Bert: Pre-training of deep bidirectional transformers forlanguage understanding. arXiv preprint arXiv:181004805. 2018.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0434

A Study on the Frequency-Dependence of Sound Event Detection
Hyeonuk Nam; Seong-Hu Kim; Byeong-Yun Ko; Yong-Hwa Park
Department of Mechanical Engineering, Korea Advanced Institute of Science and Technology, Korea

ABSTRACTSound event detection (SED) has been applying many methods from other domains actively adopting deeplearning (DL). However, direct application of DL methods from other domains without thorough assessment
and analysis on the specialty and characteristics of SED do not result in optimized performance. Recent works
such as FilterAugment and frequency dynamic convolution has revealed that SED is highly frequency
dependent task. FilterAugment which generalizes SED model to extract sound event patterns over wider
frequency bands improved SED performance by 6.5%. Frequency Dynamic Convolution that appliedfrequency-adaptive convolution kernel to loosen translation-invariance of convolution on frequency axis
improved SED performance by 7.7%. This is consistent to physiological knowledge on human hearing
system, that sounds are perceived by human on time-frequency domain. In this work, we further analyze on
the frequency-dependent of SED using these two methods, and how these methods affect each other.
Keywords: Sound Event Detection, Frequency-Dependence, FilterAugment, Frequency Dynamic
Convolution

1. INTRODUCTION
Sound Event Detection (SED) is an audio-related task which aims to automatically detect if target
sound event classes can be heard and when they can be heard within given audio clips. It requires not
only classification of the sound event classes, but also time localization of those sound events (1). As
detection of sound events from given audio clip requires pattern rec ognition of those sound events,
SED has been developed rapidly since success of deep learning (DL) in image recognition field, which
requires pattern recognition as well. DL methods developed for 2D image recognition are successfully
implanted to SED by applying those methods on 2D audio data such as log mel spectrogram (1). By
expanding 1D audio data with time dimension into 2D audio data with time and frequency dimension,audio patterns become more apparent. However, such 2D audio patterns exhibit different
characteristics form 2D image patterns. While image data’s two dimensions are both location, audio
data’s dimensions are time and frequency, which have different physics. While location in single
dimension is bidirectional quantity, time is unidirectional quantity that only flows in one direction:from past to future. Furthermore, frequency is absolute quantity while location and time can be views
as relative quantities. Such differences in data domain introduces physical inconsistency when
applying DL methods for image domain on SED. Such inconsistencies has to be resolved to makeSED models to be more accurate and robust. Among time and frequency, frequency has to be more
carefully considered on SED because human auditory system process 1D audio signal into 2D audio
signal by adding frequency dimension and frequency is more different from location than time. In thiswork, we discuss two DL methods on SED those address this issue by considering frequency
dimension, FilterAugment and frequency dynamic convolution (2, 3). In this paper, effect of these
two methods dealing with frequency dimension on SED are discussed.

2. Frequency-Dependent SED Methods
2.1 FilterAugment
FilterAugment is a 2D audio data augmentation method proposed to regularize SED models on
various acoustic environments (2). Humans trained to listen sound in various acoustic environments
can recognize various sound events regardless of where they hear the sound events. Therefore, SED

Figure 1 – Illustration of an example of FilterAugment. (a) An original log mel spectrogram, (b) log mel
spectrogram applied with FilterAugment, (c) filter applied to (a) to resulted in (b).
models should detect sound events in various acoustic environments as humans do. In order toregularize SED models over various acoustic environments, we could apply acoustic filters on training
data to simulate acoustic environments. However, applying acoustic filters requires prior domain
knowledge on acoustic signal processing, and requires additional computational cost. Therefore,
FilterAugment was proposed to mimic acoustic filter with simpler method.
FilterAugment is applied by firstly randomly choosing frequency boundaries which divides whole
frequency region into several frequency bands. Then, random weights are applied on each frequency
band. An example of application of FilterAugment is shown in Figure 1. Such random weighting on
random frequency band could simply simulate acoustic environments where some frequency b ands
are boosted while some others are attenuated. Although it would sound less natural than applying
acoustic filters, natural sound do not guarantee better SED performance. In addition, when
FilterAugment is differently applied on every epoch, it could h ighlight different frequency bands
every epoch. Therefore, it could train SED model to extract sound event information from wider
frequency regions.
2.2 Frequency Dynamic Convolution
Frequency dynamic convolution is proposed to release translational equivariance of 2D
convolution on 2D audio data (3). 2D convolution has been successfully applied to various tasks incomputer vision to recognize 2D patterns within images (1). One important advantage of 2D
convolution is that it can recognize certain pattern regardless of the location within the given image.
Translation equivariance makes sure that translation of certain pattern on convolution ’s input is alsotranslated in the same direction by proportional distance on convolution’s output. In 2D image data,such translational equivariance should hold in both two dimensions as relative position of the image
does not change the identity of the image. However, translation equivariance should be revisited when
2D convolution is applied to 2D audio data with time and frequency dimensions. It is okay to betranslation equivariant over time dimension, because sound event class do not alter when it happens
in different time. However, it should not be translation equivariant over frequency dimension, because

Figure 2 – Illustration of an example of Frequency Dynamic Convolution operation.

each frequency exhibit different physical information and 2D audio data translated in frequency
dimension would sound different from original data.
In order to release translational equivariance of 2D convolution on 2D audio data, frequency
dynamic convolution applies frequency-adaptive convolution kernels which adapts to each frequency
bin by extracting frequency-adaptive attention weights and then applying weighted sum of basis
kernels with extracted frequency-adaptive attention weights. Frequency-adaptive attention weights
are obtained by time pooling the convolution input and applying squeeze -excitation algorithm as in
dynamic convolution and temporal dynamic convolution []. Then, obtained attention weights are used
to apply weighted sum on trainable basis kernels. This operation is illustrated in Figure 2. As eachfrequency bin results in different attention weights, each frequency bin is applied with different
convolutional kernels, thus frequency dynamic convolution releases translational equivariance overfrequency dimension.

3. Experimental Details
To compare how FilterAugment and frequency dynamic convolution affects the performance of
SED model, convolutional recurrent neural network (CRNN) based SED models were trained with orwithout FilterAugment and frequency dynamic convolution. The SED model takes log mel
spectrogram of 10-second-long audio data with 626 time frames and 128 mel frequency bins as input.
The architecture is based on 7 2D convolution layers, followed by two bidirectional gated recurrent
unit (GRU) layers and one fully connected layers (1, 4). Dataset used in the experiments are fromdomestic environment sound event detection (DESED) dataset (1). Training data used synthesizedstrongly labeled dataset, real weakly labeled dataset and real unlabeled dataset from DESED. To
compare SED performance, test was carried out using real validation dataset from DESED. SED
performance was measured using polyphonic sound detection score (PSDS), and was compared using
sum of PSDS1 and PSDS2 where PSDS1 favors accurate time localization of onset and offset of soundevents, and PSDS2 favors accurate sound event classification without cross triggers (5). The results
listed in Table 1 is the best PSDS among 24 separate training runs for each setting. More details areavailable in the implemented code: https://github.com/frednam93/FDY-SED. There are few changes
for this experiment from the code: median filter setting is 7 for all class and random seed is set to 42.

4. Results and Discussions.
The results are listed in Table 1. Both FilterAugment and frequency dynamic convolution improves
SED performance largely by 6.5% and 7.72% respectively. This proves that SED is highly frequency dependent task. Also, considering that application of both methods should result in 14.7% if two
methods are completely uncorrelated, 12.3% improvement proves that the effect of two methods are
only little overlapping. Although both methods solve problem in frequency domains, they performdifferent roles. From the results, we can conclude that addressing frequency dimension is crucial in
SED.
Table 1 – Result of frequency-dependent SED methods
Models

PSDS1

PSDS2

Sum

Improvement (%)

Baseline

0.387

0.598

0.985

0

+ FilterAugment

0.413

0.636

1.049

6.50

+ FDY

0.422

0.639

1.061

7.72

+ both

0.440

0.666

1.106

12.3

ACKNOWLEDGEMENTS
This work was supported by “Human Resources Program in Energy Technology” of the Korea
Institute of Energy Technology Evaluation and Planning (KETEP), granted financial resource from
the Ministry of Trade, Industry & Energy, Republic of Korea. (No.20204030200050).

REFERENCES
1. Turpault N, Serizel R, Shah AP, Salamon J. Sound event detection in domestic environments withweakly labeled data and soundscape synthesis. Workshop on Detection and Classification of Acoustic
Scenes and Events; 2019.
2. Nam H, Kim SH, Park YH. Filteraugment: An acoustic environmental data augmentation method.
International Conference on Acoustics, Speech and Signal Processing (ICASSP); 2022.
3. Nam H, Kim SH, Ko BY, Park YH. Frequency dynamic convolution: Frequency-adaptive patternrecognition for sound event detection. arXiv preprint arXiv:2203.15296; 2022.
4. Nam H, Ko BY, Lee GT, Kim SH, Jung WH, Choi SM, Park YH. Heavily augmented sound event
detection utilizing weak predictions. Techinal report DCASE2021 Challenge; 2021
5. Bilen C¸, Ferroni G, Tuveri F, Azcarreta J, and Krstulovi´c S. A framework for the robust evaluationof sound event detection. International Conference on Acoustics, Speech and Signal Processing(ICASSP); 2020.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0447

Deep Learning Based Prediction of Human Auditory Brainstem
Response for Sound Localization in Median Plane
Byeong-Yun KO 1; Gyeong-Tae LEE2; Hyeonuk NAM 3, Young-Hwa PARK 4
1, 2, 3, 4

Korea Advanced Institute of Science and Technology, Korea

ABSTRACT
Spatial audio convolved with non-individual head-related transfer function (HRTF) causes perceptual
discrepancy for sound localization. Various methods have been suggested to predict individual HRTF.
However, for validation of predicted HRTF, a virtual environment, hardware with various accessories, andconsiderable time cost are required to measure the localization response of the subject. Although anapproximation model for auditory-brainstem system by signal processes can estimate localization responsein median plane as an alternative, listener-specific parameters should be adjusted to each subject. To
overcome this limitation, we propose a method that predicts human auditory brainstem response for elevation
localization based on deep learning without adjusting the tuning parameter. A CNN model that extracts
vertical localization cues including peak and notch patterns in HRTF is used. Then it predicts localization
response by comparing the pattern of true and target HRTFs. When using the 39 experimental data of normal
hearing people for validation, the RMSE of localization response in median plane is below 37°.
Keywords: HRTF, Sound localization perception, Auditory-brainstem response, Deep learning

1. INTRODUCTION

For accurate simulation of sound sources from arbitrary directions, an individual head-relatedtransfer function (HRTF) should be convoluted with target stimuli (1). If an audio rendering system
uses a non-individual HRTF for the synthesis of the spatial audio, users feel perception errors (2).
Various works related to the prediction of individual HRTF have been proposed, e.g. HRTF
individualization and verification of individualized HRTF. To conduct verification of individualized
HRTF, VR/AR equipment and virtual environment to measure subject response are required (3). As
an alternative, a rule-based method to simulate human auditory-brainstem response is proposed to
predict sound localization perception in the median plane when the subject hears spatial audio
convoluted with random HRTF (4). However, the rule-based method has subject-specific parameters,
so the method has a tuning process to adjust the signal process model on each subject’s auditorybrainstem response. In this work, we propose a deep learning-based method for the estimation ofelevation localization perception without subject-specific parameters. We design a neural networkmodel for recognition of HRTF’s pattern, and training method to predict perception errors due todistortion of HRTF.

2. PROPOSED METHOD

To develop a deep learning-based method for estimation of sound localization perception, HRTF’s
properties used for sound localization should be analyzed to design neural network structure. In
addition, a proper training method is required to predict localization perception error due to HRTF
distortion. In the following, we explain the details of HRTF characteristics, neural network structure,and training method.

b.y.ko@kaist.ac.kr
hansaram@kaist.ac.kr
3frednam@kaist.ac.kr
4yhpark@kaist.ac.kr
12

2.1 Characteristics of HRTF
The HRTF is a frequency response describing a transmission from a sound source in an arbitrary
direction to an individual eardrum. In HRTF, acoustic effects from the head, torso, and ear pinna arecontained and spectral cues (main peaks and notches in HRTF) are mainly attributed to pinna
resonances (1). Spectral cues of HRTF are a main role in elevation localization perception in theauditory-brainstem response (5), and the distribution of spectral cues is dependent on sound source
direction and frequency range as shown in Figure 1. When a human hears spatial audio, auditory-

Figure 1. An example of distribution of spectral cues in HRTF depending on source direction andfrequency: X, peak; O, notch.
brainstem compares true HRTF (individual HRTF) to target HRTF (an HRTF used for the synthesis ofspatial audio) to estimate the direction of a sound source. In this process, spectral cues are extracted
and the direction where true HRTF has the same spectral cues as target HRTF is perceived for the
direction of a sound source.
2.2 Neural Network Structure
To predict elevation localization perception using the deep learning-based method, the neuralnetwork should be constructed based on the processes of auditory brainstem response. We use a
convolutional neural network (CNN) as a network model because CNN is widely used for image,speech, sound event, and speaker identification by CNN’s pattern recognition properties (6–10). In
addition, we apply a residual block on the network to learn low-to-high level features from network
layers and extract the spectral cues among complex patterns in HRTF. The input of the network is the
combination of true HRTFs (left and right) and target HRTFs according to the comparison process inauditory-brainstem response. The network output is a prediction of the response angle for localization
perception depending on a target angle, and the whole process of the proposed deep learning-based
method is described in Figure 2.

Figure 2. Deep learning model for prediction of elevation localization perception. The target and true
HRTFs is used as input of the network. The network predicts response angle depending on target angle.

2.3 Training Method
The CNN is trained with target and true HRTFs and localization perception response. In the training
process, the input and output of the neural network should be organized in terms of subject anddistortion type of HRTF. The distortion types of target HRTF can be low-pass filtered HRTF, spectral
wrapped HRTF (an HRTF where spectral components are compressed to half size), individual HRTF
(non-distorted HRTF), etc. The localization perception response is paired with target HRTF, namely
the neural network is trained with true and target HRTFs for the same subject and the localization
perception response measured with the true and target HRTFs. By the organized input and output of
the network, the neural network can estimate the effect of distortion of HRTF on sound localization
perception.
dB

°

150

Elevation angle,

200

100

20

10

0

-10

50

-20

0
0

5

10

15

20 0

5

Frequency, kHz

10

dB

True

200

0

-10

-20

150

0

5

10

15

20 0

5

10

15

dB

150
°

50

150

200

250

200

250

200

250

Predicted

°
0

-10

-20

150

0

5

10

15

20 0

5

Frequency, kHz

10

15

50

-50
-50

-30

20

100

0

0

0

Frequency, kHz
d

dB

200

100

Target angle,

°

50

150

250

20

True

200

Predicted

°

10

0

-10

50

-20

0
0

5

10

Frequency, kHz

15

20 0

5

10

Frequency, kHz

15

20

-30

150

Response angle,°

100

Target angle,

True

200

10

50

Elevation angle,

50

250

20

Response angle,°
Elevation angle,

200

100

0

Frequency, kHz

Frequency, kHz

150

50

-50
-50

-30

20

100

0

0

100

Predicted

°

10

50

150

-30

250

20

Response angle,°
Elevation angle,

100

20

Frequency, kHz

200
150

15

100
50
0
-50
-50

0

100

Target angle,°

Figure 3. Prediction results of localization perception for test set: (a) true HRTFs (left/right), (b)(d)(f)the different types of target HRTFs (left/right), (c)(e)(g) comparisons of true and predicted localization
perception response, (b) individual HRTFs, (d) low-pass filtered HRTFs, (f) spectral wrapped HRTFs.

3. DEEP LEARING STEP-UP

To validate our proposed method for the prediction of sound localization perception in the medianplane, we conduct deep learning using an open dataset for elevation localization perception (4). The
open dataset contains 39 localization measurement data for 3 types of target HRTF (13 low-passfiltered HRTF, 13 spectral wrapped HRTF, 13 individual HRTF) of 13 subjects and the true HRTF of
the subjects. The range of target angle for localization response is from -30° to 210°. We use 30 data
for a training set and 9 data for a test set. The data for 3 distortion types are evenly distributed for the
training and test set to prevent overfitting to specific distortion type.

4. PREDICTION OF LOCALIZATION PERCEPTION

We plotted the prediction results for localization perception of the test set and the results are shownin Figure 3. In the case that individual HRTF is used for target HRTF, accurate localization according
to target angle is observed for true localization response due to no distortion of HRTF. The results ofthe proposed network model are similar to the true localization response. In the case that low-passfiltered HRTF is used for target HRTF, as spectral cues are distorted except only the first peak andnotch, incorrect localization occurs in true localization response. The neural network generates asimilar tendency with true localization response but shows small variance. In the case that spectral
wrapped HRTF is used for target HRTF, the significant distortion of localization perception appears
in true localization response by the lost information of all spectral cues. Our deep learning model
generates a similar tendency with true localization response same as the case for low-pass filtered
HRTF but gives biased prediction to 50°. When we calculate root mean square error (RMSE) between
true and predicted localization response as an objective metric for prediction accuracy, the RMSE of
prediction results of our proposed method is 36.8°.

5. CONCLUSIONSWe propose a deep learning-based method for the prediction of elevation localization perception
and design CNN structure and training method. By the proposed deep learning-based method, we can
exclude the tuning process of subject-specific parameters. The designed CNN can predict localization
perception error due to distortion of HRTF. The RMSE of prediction results is 36.8°. For future works,
we will collect additional training data for the different types of HRTF distortion, and we will conduct
an experiment of localization perception using VR/AR equipment and a virtual environment to verify
our proposed deep learning-based method.

ACKNOWLEDGEMENTS

This work was supported by “Human Resources Program in Energy Technology” of the Korea
Institute of Energy Technology Evaluation and Planning (KETEP), granted financial resource from
the Ministry of Trade, Industry & Energy, Republic of Korea. (No.20204030200050).

REFERENCES

1. Iida K. Head-related transfer function and acoustic virtual reality. Springer Singapore; 2019.
2. Wenzel EM, Arruda M, Kistler DJ, Wightman FL Localization using nonindividualized head‐relatedtransfer functions. J Acoust Soc Am. 1993;94(1):111-123.
3. Lu J, Qi X. Pre-trained-based individualization model for real-time spatial audio rendering system. IEEE
Access. 2021;9:128722-128733.
4. Baumgartner R, Majdak P, Laback B. Modeling sound-source localization in sagittal planes for humanlisteners. J Acoust Soc Am. 2014;136(2):791-802.
5. Middlebrooks JC. Sound localization. Handbook of clinical neurology; 2015.
6. Kim SH, Nam H, Park YH Temporal Dynamic Convolutional Neural Network for Text-Independent
Speaker Verification and Phonemic Analysis. In ICASSP 2022; 2022.
7. Kim SH, Nam H, Park YH. Decomposed Temporal Dynamic CNN: Efficient Time-Adaptive Network for
Text-Independent Speaker Verification Explained with Speaker Activation Map. arXiv preprint
arXiv:2203.15277. 2022.
8. Lee GT, Nam H, Kim SH, Choi SM, Kim Y, Park YH. Deep learning based cough detection camera using
enhanced features. Expert Systems with Applications. 2022;117811.
9. Kim SH, Park YH. Adaptive convolutional neural network for text-independent speaker recognition. In

INTERSPEECH 2021; 2021.
10. Ko BY, Nam H, Kim SH, Min D, Choi SD, Park YH. Data Augmentation and Squeeze-and-Excitation
Network on Multiple Dimension for Sound Event Localization and Detection in Real Scenes. arXiv
preprint arXiv:2206.12059. 2022.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0461

Binaural sound event localization and detection for humanoid robot
Gyeong-Tae LEE1; Yong-Hwa PARK2
1, 2

Korea Advanced Institute of Science and Technology (KAIST), Korea

ABSTRACTTo realize human modality in humanoid robots, we propose a binaural sound event localization and detection(BiSELD) model that simultaneously detects and localizes sound events by learning features of head-relatedtransfer function (HRTF) from binaural signals. The dataset contains training, validation, and test datasets ofone-minute wav files synthesized by convolving sound event samples (12 classes) with head-related impulse
responses (HRIRs) of 36 directions (12 azimuths from −180° to +180° and 3 elevations from −30° to +30°)
from KAIST HRTF database. We propose an 8-channel binaural feature composed of mel-scaled spectrogram(MS), velocity-gram (V-gram), interaural-time-difference-gram (ITD-gram), interaural-level-differencegram (ILD-gram), and spectral-cue-gram (SC-gram). BiSELDnet consists of convolutional recurrent neuralnetwork (CRNN) followed by dense neural network (DNN). The output of BiSELDnet is a time frame
sequence of 12 direction-of-arrival (DOA) vectors. Since a sound event activity is assigned to the magnitudeof the corresponding DOA vector, sound event detection and localization can be performed simultaneously.
BiSELDnet was trained by minimizing mean squared error (MSE) loss using Adam optimizer. The evaluation
results on the test dataset are as follows: error rate (𝐸𝑅20° ) of 0.18, F-score (𝐹20° ) of 87.0%, localization error(𝐿𝐸𝐶𝐷 ) of 6.5°, localization recall (𝐿𝑅𝐶𝐷 ) of 85.5%, and BiSELD score of 87.7%.
Keywords: Humanoid robot, Sound event localization and detection, Head-related transfer function

1. INTRODUCTION
We receive a variety of sounds from all directions. The task of the auditory system is to parsecomplex audio mixtures to determine possible sound sources from incoming signals. Bregman named
this process auditory scene analysis (ASA) (1). Although incoming sounds do not fully express theirsources, our auditory perception is rarely confusing or misleading. However, little is known about theneural mechanisms by which the human brain accomplishes this feat. Over the past 40 years, several
theories have been hypothesized to explain the processing of complex auditory scenes and perceptual
phenomena. Most of these theories have been implemented in the form of computational models suchas computational auditory scene analysis (CASA). Most CASA models are fragmentary in that they
focus on evaluating the usefulness of specific algorithms for finding the causes of complex a udio
signals rather than providing a general interpretation of ASA (2). For the application of humanoid
robots, our goal is to emulate the ASA of human auditory processing by learning binaural featuresusing deep learning models without grouping and separating sound elements as in the previous CASA.
Sound event localization and detection (SELD) is the combined task of detecting the temporal
activities of sound events, and localizing their spatial locations when active (3). The conventional
SELD tasks use more than four audio input channels. However, 2-channel SELD task has not yet been
published. To realize human modality in humanoid robots or telepresence technology, the 2-channel
SELD method is required. In this work, we propose a deep learning based binaural sound event
localization and detection (BiSELD) approach. Essentially, binaural signal contains information abouthead-related transfer function (HRTF). Since humanoid robots have only two audio input channels
like humans, they must extract the HRTF information inherent in the binaural input signals todetermine the direction of their sound sources. An ideal humanoid robot will learn its HRTF
information like humans, and simultaneously detect and localize surrounding sound events. Therefore,
we propose a CRNN-based BiSELDnet to learn binaural features specialized for sound event detection
and localization. We implemented the prototype of BiSELD model and examined its feasibility.
12

hansaram@kaist.ac.kr
yhpark@kaist.ac.kr

2. WHAT IS HRTF?
2.1 Coordinate System
The spherical coordinate system and head transverse planes for specifying the location of a soundsource are shown in Figure 1. In Figure 1(a), the origin of the coordinate system is the center of thehead, between the entrances to the two ear canals. In Figure 1(b), the horizontal, median, and lateralplanes are illustrated. The position of a sound source is defined in the spherical coordinate system as(r, θ, ϕ). The azimuth θ is the angle between the y-axis and the horizontal projection of the position
vector, defined as −180° < θ ≤ +180°. The elevation ϕ is the angle between the horizontal plane andthe position vector of the sound source, defined as −90° ≤ ϕ ≤ +90°.

Figure 1 – Illustration for coordinate system: (a) spherical coordinate system, (b) head transverse planes
2.2 Definition of HRTF
The sound emitted from a sound source is diffracted and reflected from the torso, head, and pinna,and then reaches both ears as shown in Figure 1. HRTFs are acoustic transfer functions due to thesound transmission process that account for the overall acoustic filtering effect by human anatomy. A
far-field HRTF of the left or right ear for a sound source of 𝑃𝑆 (𝑟, 𝜃, 𝜙) is defined as follows
𝐻𝐿,𝑅 (𝜃, 𝜙, 𝑓, 𝑠) = 𝑃𝐿,𝑅 (𝑟, 𝜃, 𝜙, 𝑓, 𝑠)/𝑃0 (𝑟, 𝑓),(1)

where 𝑃𝐿,𝑅 is the complex-valued sound pressure in the frequency domain at the entrance of the leftor right ear canal of a subject; 𝑃0 is the complex-valued sound pressure in the frequency domain at
the center of the subject’s head in the absence of the subject; the subscripts L and R denote the left
and right ears, f refers to frequency, and s refers to a set of parameters related to the dimensions of
the subject’s anatomical structures. Although 𝑃𝐿,𝑅 and 𝑃0 are functions of distance r, the effects ofr on 𝑃𝐿,𝑅 and 𝑃0 can be regarded as identical under the far-field assumption, so that the effects of r
can be canceled out in 𝐻𝐿,𝑅 . s is constant because the measurement subject is predetermined.

3. DATASET
3.1 HRTF Database
To spatialize sound event data, HRTFs of a standard dummy head, Brüel & Kjær (B&K) HATS
Type 4100, were measured. As shown in Figure 2(a), the HRTF measurement system is installed in
the KAIST anechoic chamber (3.6 m in width, 3.6 m in length, and 2.4 m in height) and the cut-off
frequency is 100 Hz. The distance from the head center of the dummy head to the diaphragm centerof the speaker module is set to 1.1 m in consideration of the height of the anechoic chamber. The
sound source ranges of azimuth θ and elevation ϕ are set from −180° to +180° and from −40° to +90°
with 5° resolution, respectively. The total number of sound source locations where HRTFs weremeasured was 1,944 (72 points in azimuth × 27 points in elevation). The sampling rate of the audio
interface was 48 kHz and the sample size of the measured impulse responses was 4,096. In addition,
the output signal of the audio interface for full frequency excitation was set as a maximum length
sequence (MLS). In Figures 2(b1) and (c1), the arrival time difference between the left and right headrelated impulse responses (HRIRs) is maximized when the azimuth of a sound source is 90° because
the transmission path to the right ear is the shortest, while that to the left ear is the longest. In Figures2(b2) and (c2), level difference appears between the left and right HRTFs. Above 4 kHz, thecontralateral HRTF levels are noticeably attenuated due to the low-pass filtering effect of the headshadow. In addition, the phase patterns at high frequency change most rapidly when a sound source isin the contralateral 90˚ direction, as shown in Figures 2(b3) and (c3). The KAIST HRTF database (4,5) and source codes are available on GitHub (https://github.com/han-saram/HRTF-HATS-KAIST).

Figure 2 – HRTF measurements at 𝜃 = 90˚: (a) setup, (b1) left HRIRs, (b2) magnitude of left HRTFs, (b3)phase of left HRTFs, (c1) right HRIRs, (c2) magnitude of right HRTFs, (c3) phase of right HRTFs
3.2 Data Generation
The HRIRs of 36 directions from the KAIST HRTF database were used to simulate the binauralsignals, i.e. 12 azimuths from −180° to +180° and 3 elevations from −30° to +30°. To train andevaluate BiSELDnet, we collected sound event datasets as shown in Table 1. Dataset 1 was provided
for sound-related modeling in the field of CASA, particularly for sound event detection. Dataset 2 isa training dataset available on the website of Task 2 of the 2016 IEEE AASP challenge on detection
and classification of acoustic scenes and events (DCASE 2016). We selected 12 sound event classes
related to emergency from the collected dataset (0: Alarm, 1: Baby, 2: Cough, 3: Crash, 4: Dog, 5:
Female Scream, 6: Female Speech, 7: Fire, 8: Knock, 9: Male Scream, 10: Male Speech, 11: Phone)(6). The selected sound event classes are bolded in Table 1. Each sound event sample consists of apair of a data file (WAV file format) and a label file (CSV file format). Each label file contains theonset and offset times of the corresponding sound event. The length of each data sample was cut tobe less than five seconds. By analyzing the measured HRTFs, it was found that most spectral cues are
distributed below 16 kHz (4, 5). Thus, the sampling frequency of dataset was set to 32 kHz, and all
HRIR and sound event samples were resampled. Twenty sound event samples were assigned to eachclass, and samples from each class were assigned to the training, validation, and test datasets in a ratioof 15:3:2. Each sound event sample is convolved with HRIRs in a total of 36 directions, then one 60 second data is generated by randomly allocating 12 synthesized samples as shown in Figure 3(c). For
reference, the HRIR file named “a270e+30.txt” contains HRIR pairs of 270° in azimuth and +30° in
Table 1 – Collection of sound event datasets
Dataset name

Class

Sampling
rate (kHz)Total(wav files)

Length(seconds)

1

NIGENS

Alarm, baby, crash, dog, engine, female scream, female
speech, fire, footsteps, general, knock, male scream, malespeech, phone, piano.

44.1

898

16,7592

DCASE2016-2

Clearing throat, cough, door slam, drawer, keyboard, keys,
knock, laughter, page turn, phone, speech.

44.1

220

265

elevation. A pair of generated 60-second data and label files is shown in Figures 3(a) and (b). The
label is saved in CSV format file and is stored in four columns: time frame (deci-seconds), index ofsound event class, azimuth (degrees), and elevation (degrees). The total number of data and label filepairs in the dataset is shown in Table 2.

Figure 3 – Data and label sample: (a) data sample, (b) label sample, (c) illustration of data generation
Table 2 – Total number of data and label pairs in the training, validation, and test datasets
Train

Valid

Test

Total

Data (pairs)

540

108

72

720

Length (seconds)

32,400

6,480

4,320

43,2004. FEATURE
4.1 Binaural Sound Localization Cues
Based on the measured HRTF database, various HRTF features can be analyzed to obtain useful
information about the localization cues encoded in HRTFs. Psychoacoustic studies have shown thatbinaural sound localization cues for a sound source include interaural time difference (ITD), interaurallevel difference (ILD), and spectral cue (SC) (7); these values were thus derived from the measured
HRTF database to get insight into the input feature for BiSELDnet.
ITD is the dominant localization cue for frequencies below 1.5 kHz (8). As shown in Figure 4(a),ITD is nearly zero in the median plane (𝜃 = 0˚ or 180˚) because the paths from a sound source toboth ears are approximately equal in length. However, if the sound source is off the median plane, thepath lengths to both ears become different, so ITD has a non-zero value. ITD can be estimated as thetime delay at which the normalized cross-correlation function of a pair of corresponding left and right
HRIRs is maximized, as follows+∞

𝐼𝑇𝐷(𝜃, 𝜙) = 𝑎𝑟𝑔𝑚𝑎𝑥
𝜏∫−∞ ℎ𝐿 (𝜃, 𝜙, 𝑡)ℎ𝑅 (𝜃, 𝜙, 𝑡 − 𝜏) 𝑑𝑡+∞+∞√[∫−∞ ℎ𝐿2 (𝜃, 𝜙, 𝑡) 𝑑𝑡][∫−∞ ℎ𝑅2 (𝜃, 𝜙, 𝑡) 𝑑𝑡]

𝑤𝑖𝑡ℎ |𝜏| ≤ 1000 𝜇𝑠,(2)where τ is time delay, ℎ𝐿 (𝜃, 𝜙, 𝑡) is a left HRIR, and ℎ𝑅 (𝜃, 𝜙, 𝑡) is a right HRIR. In Figure 4(b), ITD
values are zero at 0° and 180° in azimuth and increase gradually as sound source deviates from themedian plane. As the sound source approaches lateral 90° directions, the absolute value of ITD
increases and reaches its maximum value. Compared to the other elevations, the range of ITD variation
is maximized in the horizontal plane (𝜙 = 0˚). As the source moves out of the horizontal plane, theITD variation range decreases.

Figure 4 – HRTF cue 1, interaural time difference (ITD): (a) illustration, (b) ITD contour map
ILD is another important localization cue above 1.5 kHz (8). When a sound source deviates fromthe median plane, the sound pressure level at the contralateral ear is attenuated, especially at highfrequencies, due to the head shadow effect, while the sound pressure level at the ipsilateral ear isslightly amplified. In the far-field with source distance far larger than the head radius, ILD is defined
as follows
𝐼𝐿𝐷(𝜃, 𝜙, 𝑓) = 20 log10|𝐻𝑅 (𝜃, 𝜙, 𝑓)/𝐻𝐿 (𝜃, 𝜙, 𝑓)|.

(3)

In Figure 5(b), the absolute value of ILD is nearly zero at the front and rear directions, and reaches
maximum values around the lateral 90° directions. ILD values under 1.0 kHz are small and changesmoothly with respect to the azimuth, which indicates that the head shadow effect is negligible at low
frequencies as shown in Figure 5(a). On the other hand, at high frequencies, the absolute value of ILD
tends to increase and varies in a complex manner. Above 5.0 kHz, ILD values are asymmetric withrespect to 𝜃 = ±90˚. This is caused by the ear position and the pinna shape as well as the front-backasymmetry of the head shape. The front-back difference in the ILD pattern can be regarded as alocalization cue to solve the front-back confusion.

Figure 5 – HRTF cue 2, interaural level difference (ILD): (a) illustration, (b) ILD contour map at 𝜙 = 0˚
SC is an important localization cue above 5 kHz (8). The high-frequency peaks and notches of
HRTF are generated in the pinna (9). Since first reported, the elevation dependence of the pinna notch
frequency has been considered an important vertical localization cue (10). To analyze SC, pinnarelated transfer functions (PRTFs), i.e. acoustic transfer functions related only to the pinna, wereextracted from the measured HRTFs. Since the first 1 ms of HRIR contains information about t hespectral peaks and notches for the pinna (9), HRIRs of the median plane were clipped by a 2 ms long
Hanning window centered on the maximum sample of each HRIR, leaving only the pinna effect. Then,the windowed HRIRs were Fourier transformed to obtain PRTFs. In Figure 6(b), the first and second
peaks, P1 and P2, are almost independent of sound elevation, so vertical localization cues are
considered not included in P1 and P2. On the other hand, the first and second notches, N1 and N2, arehighly dependent on sound elevation. When elevation varies from −40° to 90°, the N1 moves between
8 kHz and 10 kHz, and the N2 moves between 12 kHz and 15 kHz. Overall, the notches are deep nearthe horizontal plane and shallower away from it. Therefore, it is clear that the patterns of N1 and N2
are unique clues for vertical localization. Notably, the front-back asymmetry in the PRTF patternprovides an important cue to solve the front-back confusion.

Figure 6 – HRTF cue 3, spectral cues (SCs): (a) illustration, (b) SC distribution of right PRTFs at 𝜃 = 0˚
(X: peaks, O: notches)
4.2 Binaural Feature
Considering the human auditory process, as an auditory signal travel from the cochlea to the brain,
perceptual features are extracted and finally at the top level as a categorical representation. In theextraction, the features include onset, offset, periodicity, amplitude modulation (AM), frequency
modulation (FM), ITD, and ILD. Together with the binaural cues discussed above, these features form
the basis of the binaural feature that BiSELDnet is based on. We propose an 8-channel binaural featurecomposed of mel-scaled spectrogram (MS), velocity-gram (V-gram), interaural-time-difference-gram(ITD-gram), interaural-level-difference-gram (ILD-gram), and spectral-cue-gram (SC-gram).
The mel scale of MS is closer to the perceptual scale of human than the linear scale of spectrogram.
Notably, the gaps between the harmonic components in MS are equally spaced even if the pitch moves.
MS is suitable for sound event detection because it expresses the periodicity, AM, and FM of soundevent, and it is useful for detecting the start and end of sound event because it includes onset andoffset information. In addition, MS can be used as an aid to prevent front-back confusion during
azimuth estimation. Left or right MS is defined as follows
𝑆𝐿,𝑅 (1,1) 𝑆𝐿,𝑅 (1,2)⋯ 𝑆𝐿,𝑅 (1, 𝐾)
𝑆𝐿,𝑅 (2,1) 𝑆𝐿,𝑅 (2,2)⋯ 𝑆𝐿,𝑅 (2, 𝐾)
𝑀𝑆𝐿,𝑅 (𝑚, 𝑘) =
,
(4)
⋮
⋮
⋱
⋮
[𝑆𝐿,𝑅 (𝑀, 1) 𝑆𝐿,𝑅 (𝑀, 2) ⋯ 𝑆𝐿,𝑅 (𝑀, 𝐾)]
𝑆𝐿,𝑅 (𝑚, 𝑘) = 20 log10 |𝑃𝐿,𝑅 (𝑚, 𝑘)|,(5)where 𝑚 is a time frame index; 𝑀 is the max time frame index; 𝑘 is a frequency index; 𝐾 is themax frequency index, i.e. the number of mel bins; 𝑃𝐿,𝑅 (𝑚, 𝑘) is a left or right mel-scaled spectrogramat 𝑚 and 𝑘; 𝑆𝐿,𝑅 (𝑚, 𝑘) is a dB-scaled 𝑃𝐿,𝑅 (𝑚, 𝑘).
The human brain tends to respond sensitively to the rate of change over time. Also, since mostsound events are transient, it is effective to reflect the rate of change in binaural feature to distinguishit from the other similar sound events. V-gram helps to detect sound events because it shows thepattern of change over time of each frequency component (11, 12). Left or right V-gram is defined asfollows
𝑉𝐿,𝑅 (1,1) 𝑉𝐿,𝑅 (1,2)⋯ 𝑉𝐿,𝑅 (1, 𝐾)
𝑉𝐿,𝑅 (2,1) 𝑉𝐿,𝑅 (2,2)⋯ 𝑉𝐿,𝑅 (2, 𝐾)
𝑉-𝑔𝑟𝑎𝑚𝐿,𝑅 (𝑚, 𝑘) =
,
(6)
⋮
⋮
⋱
⋮
[𝑉𝐿,𝑅 (𝑀, 1) 𝑉𝐿,𝑅 (𝑀, 2) ⋯ 𝑉𝐿,𝑅 (𝑀, 𝐾)](𝑚 = 1),
𝑆𝐿,𝑅 (𝑚 + 1, 𝑘) − 𝑆𝐿,𝑅 (𝑚, 𝑘);
𝑉𝐿,𝑅 (𝑚, 𝑘) = { {𝑆𝐿,𝑅 (𝑚 + 1, 𝑘) − 𝑆𝐿,𝑅 (𝑚 − 1, 𝑘)}/2; (2 ≤ 𝑚 ≤ 𝑀 − 1),(𝑚 = 𝑀),
𝑆𝐿,𝑅 (𝑚, 𝑘) − 𝑆𝐿,𝑅 (𝑚 − 1, 𝑘);where 𝑉𝐿,𝑅 (𝑚, 𝑘) is a V-gram element at 𝑚 and 𝑘.

(7)

ITD-gram is proposed inspired by the ITD process of the medial superior olive (MSO) in thebrainstem. Humans detect the azimuth of sound based on time-of-arrival differences by comparing
neuron firing times in the MSO (13, 14). As mentioned above, ITD is the dominant localization cue
below 1.5 kHz. Above 1.5 kHz, the absolute value of interaural phase difference exceed s 2π because
the head size becomes larger than the wavelength. Thus, the interaural phase delay difference below
1.5 kHz can be a useful binaural feature for azimuth estimation. ITD-gram is defined as follows⋯ ∆𝜏(1, 𝑘1500 )∆𝜏(1,1) ∆𝜏(1,2)∆𝜏(2,1) ∆𝜏(2,2)⋯ ∆𝜏(2, 𝑘1500 )𝐼𝑇𝐷-𝑔𝑟𝑎𝑚(𝑚, 𝑘) = [
],
(8)
⋱
⋮
⋮
⋮
∆𝜏(𝑀, 1) ∆𝜏(𝑀, 2) ⋯ ∆𝜏(𝑀, 𝑘1500 )∆𝜏(𝑚, 𝑘) = 𝜏𝐿 (𝑚, 𝑘) − 𝜏𝑅 (𝑚, 𝑘) = {−𝜓𝐿 (𝑚, 𝑘)
𝜓𝑅 (𝑚, 𝑘)} − {−}
𝜔𝜔

1𝑃𝐿.𝑟𝑒 (𝑚, 𝑘) ∙ 𝑃𝑅.𝑖𝑚 (𝑚, 𝑘) − 𝑃𝐿.𝑖𝑚 (𝑚, 𝑘) ∙ 𝑃𝑅.𝑟𝑒 (𝑚, 𝑘)= tan−1 {},𝜔𝑃𝐿.𝑟𝑒 (𝑚, 𝑘) ∙ 𝑃𝑅.𝑟𝑒 (𝑚, 𝑘) + 𝑃𝐿.𝑖𝑚 (𝑚, 𝑘) ∙ 𝑃𝑅.𝑖𝑚 (𝑚, 𝑘)(9)where ∆𝜏(𝑚, 𝑘) is an interaural phase delay difference at 𝑚 and 𝑘; 𝑘1500 is the frequency index
corresponding to 1.5 kHz; 𝜏𝐿 (𝑚, 𝑘) and 𝜏𝑅 (𝑚, 𝑘) are left and right phase delays at 𝑚 and 𝑘 ,respectively; 𝜓𝐿 (𝑚, 𝑘) and 𝜓𝑅 (𝑚, 𝑘) are left and right phases at 𝑚 and 𝑘, respectively; 𝜔 is theangular frequency at 𝑘; 𝑃𝐿.𝑟𝑒 (𝑚, 𝑘) and 𝑃𝐿.𝑖𝑚 (𝑚, 𝑘) are the real and imaginary parts of 𝑃𝐿 (𝑚, 𝑘),respectively; 𝑃𝑅.𝑟𝑒 (𝑚, 𝑘) and 𝑃𝑅.𝑖𝑚 (𝑚, 𝑘) are the real and imaginary parts of 𝑃𝑅 (𝑚, 𝑘), respectively.
ILD-gram is proposed based on the ILD process of the lateral superior olive (LSO) in the brainstem.
The LSO compares the level of sounds from the two ears (15). As stated above, ILD contributes tolocalization above 1.5 kHz. Below 1.5 kHz, there is almost no difference in the left and right soundpressure levels due to the effect of sound diffraction. As frequency increases (especially above 5 kHz),the ILD gradually becomes dominant due to the head shadow effect. Hence, the ILD above 5 kHz is
another effective binaural feature for azimuth estimation. ILD-gram is defined as follows∆𝑆(1, 𝑘5000 ) ∆𝑆(1, 𝑘5000 + 1) ⋯ ∆𝑆(1, 𝐾)∆𝑆(2, 𝑘5000 ) ∆𝑆(2, 𝑘5000 + 1) ⋯ ∆𝑆(2, 𝐾)
𝐼𝐿𝐷-𝑔𝑟𝑎𝑚(𝑚, 𝑘) = [
],
(10)
⋮
⋮
⋱
⋮
∆𝑆(𝑀, 𝑘5000 ) ∆𝑆(𝑀, 𝑘5000 + 1) ⋯ ∆𝑆(𝑀, 𝐾)∆𝑆(𝑚, 𝑘) = 𝑆𝑅 (𝑚, 𝑘) − 𝑆𝐿 (𝑚, 𝑘),(11)where ∆𝑆(𝑚, 𝑘) is an ILD at 𝑚 and 𝑘; 𝑘5000 is the frequency index corresponding to 5 kHz.
SC-gram is proposed based on the fact that the SCs caused by the pinna contribute to vertical
localization. The high-frequency notches of HRTF depend on pinna shape as well as vertical angle(9). As discussed above, the notches above 5 kHz are highly dependent on the elevation of soundsource, so the spectral notch pattern above 5 kHz can be a distinct binaural feature for elevation
estimation. Therefore, we extended the MS-gram above 5 kHz and used it as another binaural feature.
Left or right SC-gram is defined as follows
𝑆𝐿,𝑅 (1, 𝑘5000 ) 𝑆𝐿,𝑅 (1, 𝑘5000 + 1) ⋯ 𝑆𝐿,𝑅 (1, 𝐾)
𝑆 (2, 𝑘5000 ) 𝑆𝐿,𝑅 (2, 𝑘5000 + 1) ⋯ 𝑆𝐿,𝑅 (2, 𝐾)
𝑆𝐶-𝑔𝑟𝑎𝑚𝐿,𝑅 (𝑚, 𝑘) = 𝐿,𝑅
.
(12)
⋮
⋮
⋱
⋮
[𝑆𝐿,𝑅 (𝑀, 𝑘5000 ) 𝑆𝐿,𝑅 (𝑀, 𝑘5000 + 1) ⋯ 𝑆𝐿,𝑅 (𝑀, 𝐾)]
Figure 7 shows an example of the proposed 8-channel binaural feature extracted from a baby crying
sound event from 𝜃 = 90˚ and 𝜙 = 0˚. In Figures 7(a1) and (a2), the MS expresses the periodicity,
AM, and FM of the sound event as well as its onset and offset information. In addition, in Figures
7(b1) and (b2), the V-gram shows the pattern of change over time. Thus, MS and V-gram areconsidered suitable for sound event detection. As defined in Equations 9 and 11, when a sound event
is on the right (𝜃 > 0˚), positive number dominates in ITD-gram and ILD-gram, respectively. On theleft (𝜃 < 0˚), negative number dominates. As shown in Figures 7(c1) and (c2), the ITD-gram and ILDgram are almost red (positive number) because the sound source is on the right (𝜃 = 90˚). If the soundsource is on the left, they turn blue (negative number). Hence, it can be seen that ITD-gram and ILDgram are specialized for azimuth estimation. In Figures 7(d1) and (d2), several thin and thick notch
lines appear between 8 kHz and 16 kHz. Therefore, it is expected that the elevation angle can be
estimated through deep learning of the SC-gram pattern.

Figure 7 – Binaural feature of baby crying from 𝜃 = 90˚ and 𝜙 = 0˚: (a1) left MS, (a2) right MS, (b1) left
V-gram, (b2) right V-gram, (c1) ITD-gram, (c2) ILD-gram, (d1) left SC-gram, (d2) right SC-gram

5. NETWORKAs shown in Figure 8(a), when a sound is input to the both ears of a humanoid robot, the binauralfeature extracted by the feature extractor is input to the BiSELDnet. The input feature has eightchannels, T temporal frames, and 64 mel bins (8 × T × 64). In the BiSELDnet, the local shift-invariant
features are learned using six convolutional neural network (CNN) modules. In each CNN module,
double CNN layer (64 kernels of 3 × 3 receptive field) is followed by a batch normalizer, a rectified
linear unit (ReLU) activator, and a max-pooling layer. The time and frequency dimensions of CNN
kernel are related to the learning of intra-channel features, whereas the number of CNN kernel
covering all the feature channels is related to the learning of inter-channel features. The feature
dimensionality is reduced using max-pooling layers across CNN modules along the time andfrequency axes, thereby reducing the time sequence length from T to T/5 and the frequency dimension
from 64 to 2. The output from the final CNN module is of dimension 64 × T/5 × 2. The output isfurther permuted and reshaped to a T/5 frame sequence of length 128 feature vectors and fed to two
bidirectional recurrent neural network (RNN) modules which are used to learn the temporal context
information from the feature vectors. In each RNN module, 128 nodes of gated recurrent units (GRU)
are used with tanh activations. This is followed by dense neural network (DNN) with five fully
connected (FC) layers for detection and localization of sound events. The last FC layer consists of 36
nodes and is followed by a tanh activator. The final tanh activator outputs a T/5 frame sequence oflength 36 output vectors. From each output vector of length 36, each of the 12 sound event classes isrepresented by a 3-dimensional direction-of-arrival (DOA) vector (x, y, z). The 3D DOA vector isreferred to as an activity-coupled Cartesian DOA (ACCDOA) vector (16). From each DOA vector, itscorresponding sound event is detected if the magnitude of the vector exceeds the threshold of 0.5, andthen localized by the vector’s azimuth and elevation as shown in Figures 8(b), (c) and (d). In thelocalization on a unit sphere, the range of sound event location along each Cartesian axis is between−1 and 1. Hence, the tanh activator is used to keep the output vector in a similar range. The BiSELDnet
is a regressor that estimates a DOA vector for each active sound event class every time frame. Notably,
for DOA estimation, the Cartesian coordinate system is more advantageous for learning than thespherical coordinate system because the Cartesian coordinates are continuous, while sphericalcoordinates are discontinuous at the wrap-around boundary (𝜃 = −180˚ or 180˚) (3).
In each time frame, the target value for each active sound event class is 1 while that for inactive
sound event class is 0. Similarly, the reference location (x, y, z) is used as the target value for eachactive sound event location and the origin (0, 0, 0) is used for inactive sound event location. Meansquare error (MSE) loss is used for BiSELD training. Interestingly, the reduction in MSE loss meansthat the 3D space distance between the reference vector and the DOA vector is reduced. The
BiSELDnet was trained with MSE loss for 300 epochs using Adam optimizer with the batch size of
256. Early stopping is used to prevent the BiSELDnet from over-fitting to the training dataset. The
training is set to stop if BiSELD score on the validation dataset does not improve for 20 epochs. For

reference, the BiSELDnet was implemented on Python using Keras library with TensorFlow backend.

Figure 8 – Overview of BiSELD model: (a) BiSELDnet with input binaural feature, (b) coordinate system
conversion (Cartesian → spherical), (c) sound event detection, (d) sound event localization

6. EVALUATION METRICS
Following the conventional SELD evaluation, the BiSELDnet is evaluated using the four joint
localization and detection metrics (17). The first two metrics are location-aware detection metrics,
such as the error rate (𝐸𝑅20° ) and F-score (𝐹20° ). The prediction is considered to be true positive only
when the prediction and reference classes are the same, and the angle between them is less than 20°.
The next two metrics are class-aware localization metrics, such as the localization error (𝐿𝐸𝐶𝐷 ) in
degrees and localization recall (𝐿𝑅𝐶𝐷 ), where the subscript refers to classification-dependent. Unlike
the location-aware detection metrics, there is no angle threshold. However, the localization error
represents the average angular distance between the predictions and references of the same class.
Lastly, the localization recall expresses the true positive rate of the number of localization estimates
detected in a class out of the total number of class events. As mentioned above, based on the BiSELD
score, training is stopped to prevent overfitting. The BiSELD score is calculated using the four
evaluation metrics as follows
𝐵𝑖𝑆𝐸𝐿𝐷 𝑠𝑐𝑜𝑟𝑒 = (𝑆𝐸𝐷 𝑠𝑐𝑜𝑟𝑒 + 𝑆𝐸𝐿 𝑠𝑐𝑜𝑟𝑒)/2,(13)

𝑆𝐸𝐷 𝑠𝑐𝑜𝑟𝑒 = {(1 − 𝐸𝑅20° ) + 𝐹20° }/2,

(14)

𝑆𝐸𝐿 𝑠𝑐𝑜𝑟𝑒 = {(1 − 𝐿𝐸𝐶𝐷 /180) + 𝐿𝑅𝐶𝐷 }/2,(15)

where SED and SEL refer to sound event detection and sound event localization, respectively. An
ideal BiSELD model will have a BiSELD score of 1.0 (100%).

7. RESULTS AND DISCUSSION
During training, the BiSELDnet was evaluated on the validation dataset every 5 epochs, and themodel parameters with the highest BiSELD score was saved. In Figure 9, the training loss started toconverge from the 45th epoch and ended at the 80th epoch. The training loss bounced a bit at the 35th
epoch, but overall it converged well. Except for the localization recall, the BiSELD score and theother evaluation metrics began to converge to certain values from the 45th epoch. The localization
recall started to converge to about 80% from the 35th epoch. The best BiSELD score was achieved at
the 70th epoch, and the corresponding model parameters were saved for the final evaluation. After
training is finished, the best model parameters were re-uploaded and then evaluated on the test dataset.

As shown in Table 3, the final evaluation results on the test dataset were achieved as follows: error
rate of 0.18, F-score of 87.0%, localization error of 6.5°, localization recall of 85.5%, and BiSELD
score of 87.7%. Even before hyperparameter tuning, it showed relatively good performance, especiallyin the localization error. Considering the 2-channel localization for sound sources from various
directions, the 6.5° is hardly considered to be a large error. Since this is the first result of the early
prototype, it is expected that the performance of BiSELD model will be further improved in the future .

Figure 9 – History of training loss and evaluation metrics on validation dataset: (a) training loss, (b)
BiSELD score (%), (c) error rate, (d) F-score (%), (e) localization error (˚), (f) localization recall (%)
Table 3 – Evaluation results of BiSELDnet prototype on the test dataset
𝐸𝑅20°

𝐹20° (%)

𝐿𝐸𝐶𝐷 (°)

𝐿𝑅𝐶𝐷 (%)

𝐵𝑖𝑆𝐸𝐿𝐷 𝑠𝑐𝑜𝑟𝑒 (%)

0.18

87.0

6.5

85.5

87.7

The inference results of the BiSELDnet for a test data file and their corresponding references arevisualized in Figure 10. Each x-axis of all sub-plots represents the same time frames. Each soundevent class and its trajectory in azimuth and elevation are marked with a unique color. Figures 10(a1)and (a2) are sub-plots for the left and right MSs of the test data file. The unique pattern of each soundevent such as onset, offset, periodicity, etc., and even the left and right level difference are clearly
expressed. Figures 10(b1) and (b2) show the reference value and inference result of sound event
detection, respectively. Each y-axis of the sub-plots represents the index of sound event class (0:Alarm, 1: Baby, 2: Cough, 3: Crash, 4: Dog, 5: Female Scream, 6: Female Speech, 7: Fire, 8: Knock,
9: Male Scream, 10: Male Speech, 11: Phone). The inference results of sound event detection arealmost perfect, except for the 20 second error of mis-recognizing the fourth sound event class (Dog).
Thus, it appears that the BiSELDnet learned well local shift-invariant features and temporal context
information related to each sound event class from the proposed 8-channel binaural feature. It is
presumed that the MS and V-gram played a major role, but detailed analysis will be carried out later.
Figures 10(c1) and (c2) show the reference value and inference result of sound azimuth localization,
respectively. Overall, the BiSELDnet accurately estimated the azimuth angle of each sound event.
Although the azimuth estimates fluctuate slightly, their mean values are very close to their respective
reference values. Therefore, it can be visually confirmed that the BiSELDnet properly learned theazimuth information of HRTF inherent in each sound event from the proposed binaural feature. It isalso assumed that both ITD-gram and ILD-gram played an important role in azimuth estimation, but
a detailed analysis will be conducted later by testing the presence or absence of the two features.
Figures 10(d1) and (d2) show the reference value and inference result of sound elevation localization,
respectively. Although the elevation estimates are not as accurate as the azimuth estimates, the averageof each elevation estimate appears near the corresponding reference value. Nevertheless, the elevation
inference results of the BiSELDnet are remarkable. Since the conventional SELD approach uses fouraudio input channels, the physical information of the vertical direction can be used when estimating
sound event elevation. Whereas, our BiSELD approach uses only two audio input channels arranged
horizontally. Thus, in order to estimate the elevation angle of a sound event, it depends entirely on theHRTF information inherent in the sound event, just like us humans. Since the elevation inference of
the BiSELDnet is achieved by the high-frequency pattern recognition, it is very difficult to estimate
the elevation angle of a sound event with sparse high-frequency bands. In extreme cases, such as lowfrequency focused sound events, the elevation inference may not be possible. Overall, in binauralsound source localization, the elevation estimation is a more difficult task than the azimuth estimation.
Since it is presumed that the SC-gram contributes to the elevation estimation of a sound event, adeeper neural network is needed to improve performance.

Figure 10 – BiSELDnet input (test_mix041.wav) and outputs (inferences): (a1) left MS of the input data,(a2) right MS of the input data, (b1) sound event reference, (b2) sound event inference, (c1) azimuthreference, (c2) azimuth inference, (d1) elevation reference, (d2) elevation inference

8. CONCLUSIONSTo realize human modality in humanoid robots, we proposed a deep learning based BiSELD modelthat simultaneously detects and localizes sound events by learning HRTF localization cues from the
8-channel binaural feature specialized for sound event detection and localization. We implemented
the early prototype of BiSELDnet and examined its feasibility. The final evaluation results on the test
dataset indicate that the initial version of BiSELDnet achieved 0.18 error rate, 87.0% F-score, 6.5°
localization error, 85.5% localization recall, and 87.7% BiSELD score. Overall, the sound event
detections are almost perfect, the sound azimuth estimates are accurate, while the sound elevation
estimates fluctuate around its reference values. Nevertheless, the BiSELD model showed that thedetection and localization of sound event are simultaneously possible with only the 2 -channel binaural
input of humanoid robot. In future work, we intend to conduct research on a deeper neural network to
improve the elevation inference performance. In addition, we plan to evaluate the performance of
BiSELDnet with overlapping sound events and various background noises. In the future, humanoid
robots with BiSELD function are expected to serve not only as helpers in human daily life, but also
as assistants in emergency situations.

ACKNOWLEDGEMENTS
This work was supported by “Human Resources Program in Energy Technology” of the Korea
Institute of Energy Technology Evaluation and Planning (KETEP), granted financial resource from
the Ministry of Trade, Industry & Energy, Republic of Korea. (No.20204030200050) .

REFERENCES
1. Bregman AS. Auditory Scene Analysis: The Perceptual Organization of Sound. Cambridge, Massachusetts,
USA: MIT Press; 1990.
2. Szabó BT, Denham SL, Winkler I. Computational models of auditory scene analysis: a review. Front.
Neurosci. 2016;10:1-16.
3. Adavanne S, Politis A, Nikunen J, Virtanen T. Sound event localization and detection of overlapping
sources using convolutional recurrent neural networks. IEEE J. Sel. Top. Signal Process. 2019;13(1):3448.
4. Lee GT, Choi SM, Ko BY, Park YH. HRTF measurement for accurate sound localization cues.
arXiv:2203.03166v2 https://doi.org/10.48550/arXiv.2203.03166.
5. Lee GT, Choi SM, Park YH. HRTF database construction and sound localization cue analysis. Proc
KSNVE Annual Autumn Conference 2021; 24-27 November 2021; Jejudo, Korea 2021. p. 284.
6. Lee GT, Park YH. Binaural set: dataset for binaural sound event localization and detection. Proc KSNVE
Annual Spring Conference 2022; 25-28 May 2022; Changwon, Korea 2022. p. 151.
7. Blauert J. Spatial Hearing: The Psychophysics of Human Sound Localization, second ed. Cambridge,
Massachusetts, USA: MIT Press; 1997.
8. Xie B. Head-Related Transfer Function and Virtual Auditory Display, second ed. Plantation, Florida, USA:
J. Ross Publishing; 2013.
9. Iida K. Head-Related Transfer Function and Acoustic Virtual Reality. Singapore: Springer; 2019.
10. Shaw EAG, Teranishi R. Sound pressure generated in an external-ear replica and real human ears by a
nearby point source. J. Acoust. Soc. Am. 1968;44(1):240-9.
11. Lee GT, Nam H, Kim SH, Choi SM, Kim Y, Park YH. Deep learning based cough detection camera using
enhanced features. Expert Syst. Appl. 2022;206(15):1-20.
12. Lee GT, Nam H, Park YH. CNN based cough detection camera using acoustic features. Proc ICA 2022;
24-28 October 2022; Gyeongju, Korea 2022. p. 1-9.
13. Lee GT, Park YH. Estimation of interaural time difference based on cochlear filter bank and ZCPA
auditory model. Trans. Korean Soc. Noise Vib. Eng. 2019;29(6):722-34.
14. Lee GT, Park YH. Method for estimating interaural time differences based on cochlea filter bank and
EUZ auditory model in noisy environment. Proc INTER-NOISE 2020; 23-26 August 2020; Seoul, Korea
2020. p. 1-12.
15. Lyon RF. Human and Machine Hearing: Extracting Meaning from Sound. Cambridge, UK: Cambridge
University Press; 2017.
16. Shimada K, Koyama Y, Takahashi N, Takahashi S, Mitsufuji Y. ACCDOA: activity-coupled Cartesian
direction of arrival representation for sound event localization and detection. Proc ICASSP 2021; 6-11
June 2021; Toronto, Ontario, Canada 2021. p. 915-9.
17. Mesaros A, Adavanne S, Politis A, Heittola T, Virtanen T. Joint measurement of localization and detectionof sound events. Proc IEEE WASPAA 2019; 20-23 October 2019; New Paltz, New York, USA 2019. p.
333–7.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0478

CNN based cough detection camera using acoustic features
Gyeong-Tae LEE1; Hyeonuk NAM2; Yong-Hwa PARK3
1, 2, 3

Korea Advanced Institute of Science and Technology (KAIST), Korea

ABSTRACT
Cough is a major symptom of COVID-19. To detect and localize cough sounds, a convolutional neuralnetwork (CNN) based model was developed and integrated with an acoustic camera for visualization of coughsounds. The cough detection model is a binary classifier whose input is a two-second acoustic feature andoutput is Cough or Others. The collected sound samples were augmented to mitigate class imbalance andreflect various background noises. For effective feature extraction, conventional features such as
spectrograms, mel-scaled spectrograms, and mel-frequency cepstral coefficients (MFCC) were enhanced bytheir velocity (V) and acceleration (A) maps. VGGNet, GoogLeNet, and ResNet were simplified to binaryclassifiers such as V-net, G-net, and R-net, respectively. To find the best combination of features and networks,a total of 39 models were trained and the prediction results were evaluated using test F1 score. Finally, a test
F1 score of 91.9% was achieved from G-net with the MFCC-V-A feature. The best cough detection model
was embedded in the acoustic camera that visualizes sound sources using a beamforming microphone array.
In a pilot test, the cough detection camera detected cough sounds with an F1 score of 90.0%, and tracked
cough locations in real time on the camera image.
Keywords: Cough detection, Sound visualization, COVID-19

1. INTRODUCTION
The most common symptoms of COVID-19 are fever and cough. While fever can be detected using
a thermal imaging camera, there is still no monitoring system to detect cough. In some approach,various signals related to audio, thermistor, accelerometer, and electrocardiogram (ECG) were
examined to find the best signal for cough detection, and it turns out that the audio signal is the mostsuitable (1). In previous studies of cough detection, various acoustic features were used in generalmachine learning methods (2-4). As deep neural network (DNN) technology has shown excellent
performance in many fields, it becomes possible to detect cough sounds with high accuracy byapplying DNN to acoustic features (5-7). Recently, AI framework to prescreen for COVID-19 fromcough recordings has been developed (8). In addition, the cough monitoring methods that quantify the
number of coughs per time have been proposed (9, 10). However, the previous methods are suitable
for monitoring systems that target only one person or detect symptoms in non-real time. To detecthuman cough sound in real time in the field, we developed a cough detection model based on
convolutional neural network (CNN) and applied it to a sound camera. In this paper, we described theentire development process of our CNN-based cough detection camera.

2. COUGH DETECTION MODEL
2.1 Dataset
As shown in Table 1, we collected several datasets to train and evaluate our cough detection model(11). We used dataset 1 as test dataset, datasets 2-6 as development dataset, and dataset 7 as
background noise dataset for data augmentation (DA). All wav files in datasets 1-6 were resampled to
16 kHz and divided into two classes, Cough and Others. Then, they were cut at 2 second intervals. In
this process, Others wav files were framed with a 0% overlap, whereas Cough wav files a 50% overlap
1hansaram@kaist.ac.kr
frednam@kaist.ac.kr
3
yhpark@kaist.ac.kr
2

Table 1 – Collection of sound event datasets
Sampling
rate (kHz)

Length(seconds)

Cough, dish, doorbell, general, glass, knock, phone, snoring, toilet.

48.0

1,580

Cough, speech, etc.

44.1

439

Cough, sneeze, sniffle, throat-clearing, speech, etc.

44.1

50,92044.1

16,759

44.1

265

44.1

4,336

16.0

729,837

Dataset name1

HUMAN20200923

2

SMI-office3

Google AudioSet

4

NIGENS

5

DCASE2016-2

6

DCASE2019-2

7

DCASE2018-5

Class

Alarm, baby, crash, dog, engine, female scream, female speech, fire,footsteps, general, knock, male scream, male speech, phone, piano.
Clearing throat, cough, door slam, drawer, keyboard, keys, knock,laughter, page turn, phone, speech.
Applause, cheering, chink & clink, clapping, crowd, dishes & pots &pans, fart, finger snapping, knock, shatter, slam, sneeze.
Absence, cooking, dishwashing, eating, other, social activity, vacuum
cleaning, watching TV, working.

Table 2 – Total number of wav files in the training, validation, and test datasets
Train

Valid

Test

Total
267,765

Cough

240,858

26,762145

Others258,651

28,738683

288,072

Total

499,50955,500828

555,837

to obtain more cough data. As a result, the number of Cough and Others files was 145 and 683 (1:4.7)
for the test dataset, and 5,952 and 32,144 (1:5.4) for the development dataset, respectively. To mitigate
class imbalance, each Cough data is mixed with 45 background noise data, whereas each Others datais mixed with 9 background noise data. In this process, each background noise data was randomly
selected from nine classes of dataset 7. When mixing data, the mixing ratio between 0 and 0.4 wererandomly selected, where the ratio close to 0 means less noisy condition. To simulate different
distances, the volume of the mixed data was randomly selected between 0.6 and 1.0, where the volume
close to 0.6 means longer distance. Finally, the training and validation datasets were organized byrandomly dividing the augmented development dataset into two subsets at a 9 to 1 ratio. The total
number of wav files in the datasets is shown in Table 2.
2.2 Feature
Acoustic features for cough detection were designed using spectrogram (SP), mel-scaled
spectrogram (MS), mel-frequency cepstral coefficients (MFCC), and their derivatives, such as
velocity map (V-map) and acceleration map (A-map). Since cough sound is essentially a transient
sound, it is necessary to reflect the rate of change over time in the acoustic features to distinguishthem from the other transient sounds. Therefore, we propose V-map and A-map that can be derived
from spectrogram-based features such as SP, MS, and MFCC. The V-map and A-map are defined asfollows(𝑡 = 0),𝑋(𝑡 + 1, 𝑓) − 𝑋(𝑡, 𝑓);𝑉(𝑡, 𝑓) = {𝑋(𝑡 + 1, 𝑓) − 𝑋(𝑡 − 1, 𝑓)}/2; (1 ≤ 𝑡 ≤ 𝑇 − 1),(𝑡 = 𝑇),
𝑋(𝑡, 𝑓) − 𝑋(𝑡 − 1, 𝑓);(1)(𝑡 = 0),𝑉(𝑡 + 1, 𝑓) − 𝑉(𝑡, 𝑓);𝐴(𝑡, 𝑓) = {𝑉(𝑡 + 1, 𝑓) − 𝑉(𝑡 − 1, 𝑓)}/2; (1 ≤ 𝑡 ≤ 𝑇 − 1),(𝑡 = 𝑇),
𝑉(𝑡, 𝑓) − 𝑉(𝑡 − 1, 𝑓);(2)

where 𝑡 denotes time index, 𝑓 the frequency (or coefficient) index, 𝑋(𝑡, 𝑓) the spectrogram-based
feature, and 𝑇 the time length of the feature. Each wav file in the datasets is converted to SP, MS,
and MFCC with FFT bin size of 512, with window and hop sizes of 30 ms and 15 ms respectively.
Since cough sound has a spectral distribution between 350 Hz and 4 kHz (12), the maximum frequencyof the features is set to 4 kHz. Lastly, SP, MS, and MFCC are concatenated with their V-map and A-

map, respectively, to create a variety of features with a size of 3 (channel) × 128 (height) × 128 (width).
Using a cough sound as an example, Figure 1(a) is the waveform of the cough sound. Figure 1(a1)is an SP-based 3-channel acoustic feature (SP-V-A) that concatenates the SP with its V-map and Amap. Similarly, Figure 1(a2) is an MS-based 3-channel acoustic feature (MS-V-A), and Figure 1(a3)is an MFCC-based 3-channel acoustic feature (MFCC-V-A). For comparison with the cough sound,acoustic features of cough-like sounds such as sneeze and throat-clearing are shown in Figures 1(b)and 1(c). In Figure 1(a) (cough), a similar spectral pattern is repeated two or three times, and distinct
vertical lines appear at the beginning and end of the cough over the entire frequency range. In addition,
tonal components related to the vocal tract stands out between the vertical lines. In Figure 1(b)(sneeze), the feature’s magnitude is small when breathing in the air before 0.75 seconds, whereas themagnitude becomes very large when ejecting droplets after 1 second. In addition, vertical lines appear
clearly at the beginning of droplet discharge. In Figure 1(c) (throat-clearing), the feature is mainlycomposed of a short first and a long pattern later, and it is noticeable that the tonal components of the
vocal tract move to the lower frequencies later. The abrupt vertical line is evident at the start of theevent. As described above, it could be confirmed with the naked eye that the difference between eachsound event clearly appeared on the acoustic features. These features are then passed through ourmultilayer CNN models.

Figure 1 – Features of (a) cough sound, (b) sneeze sound, and (c) throat-clearing sound: (x) Waveform, (x1)
SP-V-A, (x11) SP, (x12) V-map of SP, (x13) A-map of SP, (x2) MS-V-A, (x21) MS, (x22) V-map of MS,(x23) A-map of MS, (x3) MFCC-V-A, (x31) MFCC, (x32) V-map of MFCC, (x33) A-map of MFCC.
2.3 Network
The cough detection model is a binary classifier where the input is a 2-second long acoustic featureand the output is a Cough or Others inference. For binary classification, we simplified typical CNN
models such as VGGNet (13), GoogLeNet (14), and ResNet (15), and named them V-net, G-net, andR-net, respectively.
VGGNet was originally designed to examine the effect of network depth on performance.
Therefore, they fixed the kernel size to 3 × 3 and increased the number of convolutional (conv) layers.
A 3 × 3 sized kernel can make the network deeper than a 5 × 5 or 7 × 7 sized kernel, and has the effectof increasing the learning speed by reducing the number of parameters. In addition, as the number ofconv layers increases, more ReLU activation functions are used, which increases nonlinearity, so more
useful features can be extracted. However, VGGNet has the disadvantage of too many parameters.
The main reason is the three fully-connected (FC) layers in the final stage, and the number ofparameters in this part is about 122 million. Therefore, we reduced the total number of neurons in the
FC layers to about 1/17 in order to alleviate this shortcoming. In addition, the number of kernels ofeach conv layer was reduced to 1/4 based on D-type VGGNet to be suitable for binary classification.
Finally, we made a V-net by applying group normalization (GN) (16) after each conv layer so that themodel performance is robust to the batch size. The architecture of V-net is outlined in Figure 2(a).
GoogLeNet can be divided into four parts: pre-layer, Inception modules, average pooling, andauxiliary classifier. The pre-layer is a typical CNN part that exists for initial learning. After the prelayer, nine Inception modules are followed. Each Inception module extracts feature maps from kernels

of 1 × 1, 3 × 3, and 5 × 5 sizes, concatenates them in the channel direction, and delivers it to the nextlayer. This is different from the conventional CNN that uses the same sized kernel in the same layer,
and more diverse types of characteristics can be derived. In the classification process, applying an
average pooling instead of a FC layer after the final Inception module has the effect of significantly
reducing the computational amount. GoogLeNet has one classifier at the end and two in the middle.
The two classifiers in the middle are called auxiliary classifiers, which alleviate the vanishing gradientproblem and help its network converge quickly. We modified the GoogLeNet to use it as a binaryclassifier and named it G-net. First, the pre-layer was changed to fit the size of the input feature, andthe number of FC layer nodes was reduced to two for binary classification. In preliminary experiments,
it was found that the G-net does not have the vanishing gradient problem even without the auxiliary
classifiers. Thus, we removed the auxiliary classifiers and used only the softmax at the end. Finally,
GN was added after each conv layer so that the model’s performance would not be affected by thebatch size. The architecture of G-net is outlined in Figure 2(b).
ResNet uses bottleneck blocks as building blocks to reduce model complexity. The bottleneck
block first compresses the channel of an input feature with 1 × 1 convolution. Then, an output feature
is extracted by 3 × 3 convolution, the channel of the output feature is expanded again by 1 × 1
convolution, the input feature delivered through a skip connection is added, and the merged feature istransferred to the next layer. ResNet has the advantage of using both the simple characteristics of
input terminal and the complex characteristics of output terminal. Above all, the slope of the addition
operation is 1 when performing backpropagation. Thus, vanishing gradient problem can be solved
because the loss propagates well to the front of a model. Based on the 50-layer ResNet, we reduced
the number of kernels of each conv layer to 1/4 for binary classification. In addition, the conv layerof the input part was changed to fit the size of the input feature, and the FC layer of the output partwas modified to 2-way. Finally, R-net was built by applying GN so that the performance of the model
is not affected by the batch size. The architecture of R-net is outlined in Figure 2(c).

Figure 2 – Architectures of cough detection model: (a) V-net, (b) G-net, (c) R-net
2.4 Modeling
We made various candidate groups for cough detection model by combining the features andnetworks proposed above. In Table 3, type 1 is a model candidate group in which V-net, G-net, andR-net are trained using the basic features SP, MS, and MFCC, respectively. Types 2 and 3 are modelcandidate groups trained with 2-channel features using V-map and 3-channel features using both V-

Table 3 – List of candidate groups for cough detection model
Model summary

Type
1

Cases

FeatureNetwork

SP, MS, MFCC

V-net, G-net, R-net

92

SP-V, MS-V, MFCC-V

V-net, G-net, R-net

93

SP-V-A, MS-V-A, MFCC-V-A

V-net, G-net, R-net

9

4

SP-MS, SP-MFCC, MS-MFCC

V-net, G-net, R-net

95

SP-MS-MFCC

V-net, G-net, R-net

3

map and A-map, respectively. By comparing the results of types 1 to 3, we examined the effect of Vmap and A-map on the performance of the models. Moreover, we organized types 4 and 5 to examine
the effect of combinations between basic features on the performance of the models. Accuracy (A),recall (R), precision (P), and F1 score (F1) are used as performance metrics of each model. Since
accuracy can be a misleading metric for imbalanced dataset, we selected the final model with thehighest F1 score for the test dataset among 39 candidate models of types 1 to 5.
We trained each candidate model by minimizing cross-entropy loss using Adam optimizer. A
learning rate scheduler is set to reduce its learning rate by a factor of 0.1 if validation loss does notdecrease for 5 epochs. In addition, to avoid overfitting, the whole training is set to stop when thelearning rate falls below 1 × 10 . After training, the performances of types 1 to 5 on the test dataset
are shown in Tables 4 to 8. In each table, the top 3 F1 scores are bolded, and the best model is shaded.
Table 4 – Performance of type 1 models on the test dataset
SP
Models
V-net
G-net
R-net

A (%)
93.1
94.4
94.1

R (%)
84.1
83.4
80.0

MS
P (%)
78.2
84.6
85.3

F1(%)
81.1
84.0
82.6

A (%)
94.2
95.3
94.2

R (%)
70.3
77.2
80.7

P (%)
95.3
94.9
85.4

F1(%)
81.0
85.2
83.0

A (%)
93.7
94.4
93.5

MFCC
R (%)
P (%)
75.2
87.2
73.1
93.8
86.9
78.3

F1(%)
80.7
82.2
82.4

MFCC-V
R (%)
P (%)
79.3
87.1
83.4
87.1
80.0
87.9

F1(%)
83.0
85.2
83.8

MFCC-V-A
R (%)
P (%)
86.2
89.9
90.3
93.6
83.4
89.6

F1(%)
88.0
91.9
86.4

MS-MFCC
R (%)
P (%)
78.6
91.9
77.9
95.8
76.6
98.2

F1(%)
84.8
85.9
86.0

Table 5 – Performance of type 2 models on the test dataset
Models
V-net
G-net
R-net

A (%)
94.7
95.4
95.2

SP-V
R (%)
P (%)
77.2
91.1
84.1
89.1
75.9
95.7

F1(%)
83.6
86.5
84.6

A (%)
95.0
96.7
95.4

MS-V
R (%)
P (%)
74.5
96.4
83.4
97.6
77.2
95.7

F1(%)
84.0
90.0
85.5

A (%)
94.3
94.9
94.6

Table 6 – Performance of type 3 models on the test dataset
Models
V-net
G-net
R-net

A (%)
95.2
95.9
96.3

SP-V-A
R (%)
P (%)
77.9
93.4
89.0
87.8
80.0
98.3

F1(%)
85.0
88.4
88.2

A (%)
96.0
96.7
96.1

MS-V-A
R (%)
P (%)
82.8
93.7
85.5
95.4
86.9
90.6

F1(%)
87.9
90.2
88.7

A (%)
95.9
97.2
95.4

Table 7 – Performance of type 4 models on the test dataset
Models
V-net
G-net
R-net

A (%)
95.2
96.4
96.1

SP-MS
R (%)
P (%)
81.4
90.1
92.4
87.6
81.4
95.9

F1(%)
85.5
89.9
88.1

A (%)
94.8
96.0
95.2

SP-MFCC
R (%)
P (%)
79.3
89.8
82.1
94.4
85.5
86.7

F1(%)
84.2
87.8
86.1

A (%)
95.0
95.5
95.7

Table 8 – Performance of type 5 models on the test dataset
Models
V-net
G-net
R-net

A (%)
96.1
96.3
96.5

SP-MS-MFCC
R (%)
P (%)
92.4
86.5
85.5
92.5
91.0
89.2

F1(%)
89.3
88.9
90.1

In Table 4, the top 3 of type 1 models are MS & G-net (85.2%), SP & G-net (84.0%), and MS &R-net (83.0%). All the top 3 models have higher precision than recall. In particular, MS & G-net, thetop model, has the highest precision among the top 3, while the lowest recall. In Table 5, the top 3 oftype 2 models are MS-V & G-net (90.0%), SP-V & G-net (86.5%), and MS-V & R-net (85.5%). All
models of type 2 have higher precision than recall, and the best model, MS-V & G-net, has the highest
precision. In Table 6, the top 3 of type 3 models are MFCC-V-A & G-net (91.9%), MS-V-A & G-net(90.2%), and MS-V-A & R-net (88.7%). All the top 3 models have higher precision than recall, andthe top model, MFCC-V-A & G-net, is the only model that exceeds 90.0% in both recall and precision.
In Table 7, the top 3 of type 4 models are SP-MS & G-net (89.9%), SP-MS & R-net (88.1%), and SPMFCC & G-net (87.8%). The best model, SP-MS & G-net, has the highest recall in type 4, and theother models have higher precision than recall. In Table 8, the top model of type 5 is SP-MS-MFCC& R-net (90.1%). The top model has the smallest gap between recall and precision in type 5.
When comparing the features of types 1 to 3, the 3-channel features of type 3 using both V-map
and A-map showed the best performance. In the case of types 4 and 5, it was found that the
combination of basic features was effective, but the 3-channel combination did not show superior
performance than the 2-channel combination. Overall, we confirmed that using V-map and A-map ismore effective in detecting cough sound than combining several features. Moreover, calculating its Vmap and A-map based on one basic feature is more efficient than calculating multiple features.
Looking at the top models’ networks in types 1 to 5, the 16-layer V-net was not selected once, whereas
the 21-layer G-net was selected four times and the 50-layer R-net was selected once. Thus, as far as
resources allow, it is clear that the deeper the layers of the network, the better the performance. Dueto the vanishing gradient problem, G-net is limited in stacking layers deeper, but R-net solves thisproblem through skip connection, so we can stack it deeper. Since MFCC-V-A & G-net has the highest
F1 score, we selected it as the cough detection model, and embedded it in a sound camera to localize
cough sounds.

3. SOUND CAMERA
3.1 Principle
Sound camera is an imaging device used to locate sound sources with a microphone array. Acoustic
signals input from the microphone array are processed through the delay and sum (DAS) beamforming
method. Figure 3 gives an illustration of the DAS beamformer in the sound camera.

Figure 3 – Beamformer of sound camera: (a) structure, (b) location vectors for time delay calculation
In the time domain DAS processing, beamformer output can be defined as follows(3)
𝐵 𝑡, 𝑋 ⃗ = ∑𝑎 𝑠 𝑡 − 𝜏 𝑋⃗ ,where 𝑡 is time, 𝑋 ⃗ is the reference direction vector, 𝑁 is the total number of microphones, 𝑎 isa weight coefficient to be applied for individual microphones, 𝑠 (𝑡) is the acoustic signal received
by n-th microphone, and 𝜏 is a specified time delay to be imposed to the signal for virtual rotationof the array. In general, 𝑎 is set to unity, and 𝜏 is calculated through the following operation basedon 𝑋 ⃗. First, the spatial domain of interest is discretized into pixels on a virtual inspection plane.
Second, 𝑋 ⃗ is calculated for each pixel. Third, the position vector of n-th microphone is obtained by𝑋 ⃗ = 𝑋 ⃗ − 𝑀 ⃗, where 𝑀 ⃗ is the relative position vector of n-th microphone from 𝑋 ⃗. Finally, 𝜏 is
obtained by𝜏 𝑋⃗ =
𝑋⃗ − 𝑋⃗ ,(4)where 𝑐 is speed of sound. The scanning algorithm of the sound camera performs the delay
calculation over the entire spatial domain it monitors. For a given pixel, after the time delays for all
microphones are imposed on the input signals, the delayed signals are summed up as shown in Figure3(a) to produce a beamformer output. The amplitude contour can be obtained by representing the
beamformer output amplitudes at all pixels as a color map. The contour is then overlaid on the camera
image and output to the display on the sound camera. The location of sound sources is determined by
the areas highlighted in the display. By taking several contoured images per second, the sound location
information can be provided in video format. Therefore, we can visualize the location of sound sources.
3.2 Integration with Cough Detection Model
We developed a cough detection camera by embedding our cough detection model into BATCAM
2.0, a commercial sound camera developed by SM Instruments Inc. The entire process of detecting
cough sound and estimating its location in the cough detection camera is described in Figure 4. The
acoustic signals input to the microphone array are discretized through the DAQ process, converted
into an amplitude contour through the DAS beamforming process, overlaid on the camera image, andthen wait for the cough inference result from the cough detection model. While the sound camera isrunning, the cough detection model extracts a feature (MFCC-V-A) from the input signal and outputs
the cough inference result through its neural network (G-net). If the inference result is Cough, thecontoured camera image with a “Cough” label is displayed on the screen, and if it is Others, only thecamera image is displayed on the screen.

Figure 4 – Flowchart of cough detection and localization process

4. PILOT TEST AND DISCUSSION
In a room equipped with the cough detection camera, we tested its performance by sequentiallygenerating various sound events including cough sound. Table 9 shows a list of sound events used inthis pilot test. All sound events are divided into two classes, “Cough” and “Others”, consisting of 40
and 160 respectively. Moreover, the “Others” has three subclasses: “cough-like sound”, “impact
sound”, and “general sound”. Especially, the first two are important for the performance evaluation
because they consist of sound events that can be misrecognized as cough sound.
The pilot test results are shown in Figure 5. From the confusion matrix in Figure 5(a), the true
positive (TP) is 0.90, indicating that 90% of the actual cough sounds are estimated correctly. Also, thetrue negative (TN) is 0.97, which means that the proposed model predicts sounds other than coughwith 97% accuracy. In Figure 5(b), we can visually check the location and number of cough sounds
through the contour and label displayed on the output screen of the cough detection camera. In addition,
we can track a person who coughs in real time by following the path of the contour.
Table 9 – List of sound events for pilot test
Class

Subclass

Cough-

Others(10 per type)

Sound event type

Total

Cough

40

Sneeze, throat-clearing

20

Impact soundApplause, clapping, knock, footsteps, door slam, finger snapping, clink

70

General soundSpeech, laughter, sniffle, snoring, page turn, phone, keyboard

70

Cough-like soundFigure 5 – Pilot test result: (a) normalized confusion matrix, (b) output screen of cough detection camera
As shown in Table 10, our cough detection camera achieved 96.0% accuracy, 90.0% recall, 90.0%
precision, and 90.0% F1 score in the pilot test. For the F1 score, there is no significant difference
from the F1 score of 91.9% (accuracy of 97.2%), the result of the previous modeling process. This
reflects the robustness of the proposed model for different noisy environments because it was trainedwith augmented data for various background noises.
We proposed a 3-channel acoustic feature suitable for CNN-based architecture. In an auditory
cortex model, auditory features are extracted in a transition process from acoustic features toperceptual features and finally to category-specific representations at the top level (17). This is similar
to the processing of vision information, which is the basis of CNN. Also, our brain tends to respondsensitively to the rate of change. Therefore, we named the 3-channel acoustic feature as Spectroflow
by reflecting the human auditory perception process. Spectroflow is easily configured by calculating
both V-map and A-map from a spectrogram-based feature and then concatenating them all in thechannel direction. Looking at the performance results for the test dataset of each candidate model, it
was found that the overall performance improved when both V-map and A-map were used.
The network applied to the cough detection camera is G-net consisting of 9 inception modules. In
each inception module, kernels of various sizes, such as 1 × 1, 3 × 3, and 5 × 5, extract feature maps
at the same time, and we think that such simultaneous extraction of various patterns was effective in
detecting the pattern change of cough sound from various sounds. Considering that R-net is a 50-layer
network, the final selection of 21-layer G-net was unexpected. However, it is worth noting that R-net
was among the top 3 in all types of performance evaluation for the test dataset. We can make R-net
deeper because it mitigates the vanishing gradient problem by using skip connection. Considering the152-layer ResNet, it is expected that further performance improvements would be possible by using
R-nets of more than 50 layers.
Table 10 – Pilot test performance of cough detection camera
Model

A (%)

R (%)

P (%)

F1 (%)

MFCC-V-A & G-net

96.0

90.0

90.0

90.0

5. CONCLUSIONSAs a result of reviewing on the modeling process and pilot test, we confirmed that DA technique,Spectroflow suitable for CNN, and the inception module of G-net have an important influence on theperformance of the cough detection camera. In the future, it will be possible to further improve thecough detection performance by developing various R-nets with more than 50 layers or proposing a
network that maximizes the advantages of the inception module. Since droplets released duringcoughing are one of the transmission pathways of virus, it is very important to detect cough to preventthe spread of infectious diseases such as COVID-19. Through data collection using IoT devices, it isexpected to be used as a medical device that automatically monitors the patient’s condition in hospitals
or as a monitoring system to detect epidemics in public places such as schools, offices, and restaurants.

ACKNOWLEDGEMENTS
This work was supported by “Human Resources Program in Energy Technology” of the Korea

Institute of Energy Technology Evaluation and Planning (KETEP), granted financial resource from
the Ministry of Trade, Industry & Energy, Republic of Korea. (No.20204030200050).

REFERENCES
1. Drugman T, Urbain J, Bauwens N, Chessini R, Valderrama C, Lebecque P, Dutoit T. Objective study of
sensor relevance for automatic cough detection. IEEE J. Biomed. Health Inform. 2013;17(3):699-707.
2. Barry SJ, Dane AD, Morice AH, Walmsley AD. The automatic recognition and counting of cough. Cough
2006;2(8):1-9.
3. You M, Liu Z, Chen C, Liu J, Xu XH, Qiu ZM. Cough detection by ensembling multiple frequency subband
features. Biomed. Signal Process. Control 2017;33:132-140.
4. You M, Wang H, Liu Z, Chen C, Liu J, Xu XH, Qiu ZM. Novel feature extraction method for coughdetection using NMF. IET Signal Process. 2017;11(5):515-520.
5. Miranda IDS, Diacon AH, Niesler TR. A comparative study of features for acoustic cough detection using
deep architectures. Proc Ann. Int. Conf. IEEE Eng. Med. Biol. Soc. (EMBC); 23-27 July 2019; Berlin,
Germany 2019. p. 2601–5.
6. Hossain FA, Lover AA, Corey GA, Reich NG, Rahman T. FluSense: a contactless syndromic surveillance
platform for influenza-like illness in hospital waiting areas. Proc. ACM Interact. Mob. Wearable
Ubiquitous Technol. 2020;4(1):1-28.
7. Barata F, Tinschert P, Rassouli F, Steurer-Stey C, Fleisch E, Puhan MA, Brutsche M, Kotz D, Kowatsch T.
Automatic recognition, segmentation, and sex assignment of nocturnal asthmatic coughs and coughepochs in smartphone audio recordings: observational field study. J. Med. Internet Res. 2020;22(7):1-15.
8. Laguarta J, Hueto F, Subirana B. COVID-19 artificial intelligence diagnosis using only cough recordings.
IEEE open j. eng. med. biol. 2020;1:275-281.
9. Birring SS, Fleming T, Matos S, Raj AA, Evans DH, Pavord ID. The Leicester cough monitor: preliminary
validation of an automated cough detection system in chronic cough. Eur. Respir. J. 2008;31(5):10131018.
10. Wilhelm FH, Roth WT, Sackner MA. The LifeShirt: an advanced system for ambulatory measurement of
respiratory and cardiac function. Behav. Modif. 2003;27(5):671-691.
11. Lee GT, Nam H, Kim SH, Choi SM, Kim Y, Park YH. Deep learning based cough detection camera using
enhanced features. Expert Syst. Appl. 2022;206(15):1-20.
12. Shin SH, Hashimoto T, Hatano S. Automatic detection system for cough sounds as a symptom ofabnormal health condition. IEEE Trans. Inf. Technol. Biomed. 2009;13(4): 486-493.
13. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. Proc 3rd
Int. Conf. Learn. Representat. (ICLR); 7-9 May 2015; San Diego, CA, USA 2015. p. 1-14.
14. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A. Going
deeper with convolutions. Proc 28th IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR); 7-12 June 2015;
Boston, MA, USA 2015. p. 1-9.
15. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. Proc 29th IEEE Conf.
Comput. Vis. Pattern Recognit. (CVPR); 26 June-1 July 2016; Las Vegas, NV, USA 2016. p. 770-8.
16. Wu Y, He K. Group normalization. Proc Eur. Conf. Comput. Vis. (ECCV); 8-14 September 2018; Munich,Germany 2018. p. 3-19.
17. Bizley JK, Cohen YE. The what, where and how of auditory-object perception. Nat. Rev. Neurosci.
2013;14:693-707.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0548

Deep Learning-based Personalized Sound Event Recognition for
Human Care Robot
Seong-Hu Kim1; Hyeonuk Nam1; Sangmin Choi1; Yong-Hwa Park1
1

Department of Mechanical Engineering, Korea Advanced Institute of Science and Technology, Korea

ABSTRACT
Human care robots are needed by elderlies for their convenient life as well as emotional stability. The robots
should quickly and accurately detect events happening around the users in order to execute right actions at
the right time. In this paper, a real-time sound event recognition (SER) system for human care robots is
proposed. To maximize recognition performance, CNN-based SER model is personalized for selected eventclasses that are highly specific to the users such as doorbell and phonebell. Personalizing SER model bytraining it using the specific personal sound events has improved average F-score for doorbell and phonebell.
A pilot living room test for the proposed personalized SER model built in a humanoid robot (Pepper) withreal-time setting confirmed that the real-time personalized SER system could be successfully utilized byhuman care robots.
Keywords: Sound event recognition, Human care robot, Real-time sound processing

1. INTRODUCTION
Human care robots are emerging as an alternative solution to the aging society, and the need forhuman care robots to assist and take care of the elderlies is growing all over the world. Thanks to therapid development in deep learning technology, various types of intelligence to human care robots cancomprehend situations and take appropriate actions depending on the situations on their own. Auditory
intelligence enables robots to understand situations more comprehensively by recognizing urgent
situations where alarms ringing, people being hurt or sick, various emergencies, etc. with more
specific details. It is advantageous over visual intelligence as sound can be heard when the soundsource is invisible. Thus, we implement sound event recognition (SER) system that recognizes eventsfrom given audio input to provide auditory intelligence on a human care robot.
Although SED algorithms continue to develop, there have been few studies on how to apply SED
models to human care robots. Previous studies for robot application have been mainly conducted tosecure robust recognition performance in noisy situations by applying SER models [1-4]. We intend
to identify and resolve the effect of sound events rather than noise on the performance of SER model.
SER is required for human care robots to accurately recognize the situation, but there are some issues
in application of SER. Firstly, there are personal sound events differ by various users such as doorbell
and phonebell. The trained SER model using general doorbell and phonebell cannot accurately
recognize those events because sound events within each class do not have enough common acoustic
characteristics. Thus, personal sound events of which acoustic characteristics vary depending on users
degrade SER performance. In addition, human care robots require real-time SER systems to recognize
sound events and provide the necessary services to the elderlies at the right time. SER models are
generally designed to predict sound event contents within audio clips with fixed length. Thus, a realtime implementation system to adopt SER model is needed. To test the practicality of the proposed
SER system, a pilot test in living environment was conducted with real-time setting in the commercial
human care robot (Pepper).
In this paper, we proposed a system that trains SER model with personal sound events specific to
the users to solve the performance degradation problem of personal sound events. In addition, a realtime sound event detecting system was also proposed to mount the proposed SER model on the humancare robot. With a pilot test, practicality of the proposed system on the human care robot was verified.
1{seonghu.kim; frednam; cyanray1500; yhpark}@kaist.ac.kr

(a)

(b)

Figure 1. Confusion matrices of SER models tested with validation dataset using (a) the conventional SER
model trained with all bell sounds with average F-score of 0.846 and (b) the personalized SER model trainedwith personal bell sounds with average F-score of 0.970.

2. METHODOLOGY
2.1 Personalization of Sound Event Recognition Model
We chose four representative sound event classes to be recognized by human care robot: cough,snore, doorbell, and phonebell. All events other than the four events and background sound should berecognized as others. These four sound event classes are effectively helpful for caring elderlies
through there should be various sound events in a room environment. When training SER model withpersonalized sound events, it is hard to extract general acoustic feature and it might cause poor
recognition performance due to their different acoustic characteristics. To solve this problem, we
utilized specificity of human care robots and personalized sound events. Human care robots operate
mostly in the specific users’ living space and the user hardly changes the personalized sound. In thisnormal use case, it is sufficient for robots to react to the sound specifically used by their own users,
without considering all other sound candidate samples. Therefore, we propose a system that trains
SER model using only specific personalized sound event samples used by the certain robot’s user,
rather than collecting and training with personalized sound samples belonging to other users.
2.2 Real-Time Operation of Sound Event Recognition Model
Typical CNN-based models are trained using data with fixed size and tested using other data withthe same size to the training data. For the real-time application of SER model, continuously received
microphone input should be split into the size of training dataset while the splits are continuously fed
to SER model. To mitigate this contradiction, we propose a method to overlap adjacent input sample
fed to SER model by an optimized overlapping proportion. We examined how recognition performancechanges with different overlap setting and determined the optimal overlap proportion through theexperiment. Thought this approach, the overlapped input can capture more complete sound events ina robust manner, which might otherwise be cut into unrecognizable pieces of inputs.

3. RESULT AND DISCUSSION
3.1 Sound Event Recognition Performance in Testing Domestic Environment
ResNet-34-based SER model is trained using the training dataset gathered from AudioSet [5],Youtube, and DCASE challenge dataset [6, 7] with data augmentation. The sound events are decided
to be positive when the result node corresponding to that event yields value exceeding a certain
threshold value 0.7. When none of four result nodes’ values exceed the threshold, the result is
considered to be others class. The conventional model trained as whole sound event dataset was tested
with recorded validation dataset, and confusion matrix of the result is shown in (b) of Fig. 2. While
cough and snore (F-score: 0.952 and 0.945, respectively) are well recognized, doorbell and phonebell(F-score: 0.906 and 0.581, respectively) show relatively low recognition performances. It seems to be

Figure 2. Domestic environment where a human care robot is operated and pilot test on SER system was done.
There are Pepper and the server computer that processes the data collected by Pepper. The personalized SER
model was mounted on the server computer.

contributed to overgeneralization on personal sound events that have very little similarity between theindividual sound events’ acoustic characteristics.
We tackled this problem by personalizing SER model by training the model with doorbell andphonebell sound samples used by a specific user. Trained personalized SER model was also tested
with the recorded validation dataset, and confusion matrix of the result is shown in (b) of Fig. 2.
Compared to the previous results by conventional SER, the proposed personalized model not only
showed better recognition performance for doorbell and phonebell (F-score: 0.964 and 1.000,
respectively) but also better F-score for cough and snore (F-score: 0.968 and 0.947, respectively). The
doorbell and phonebell events, which vary largely with users, show that it is effective to train withspecific sound samples depending on specific user.
3.2 Pilot Test for Real-Time Sound Event Recognition
The human care robot system consists of Pepper and a server computer for SER. The server
computer receives audio packets of Pepper's microphone in real-time through a wireless transmission
and recognizes sound events using the SER model. The personalized SER model was mounted onserver computer that communicates with Pepper robot in an indoor apartment environment shown in
Fig. 4. An appropriate speech response of Pepper to the user is determined using recognition results
and sent back to the Pepper.
We designed a pilot test with total eight sound events, two of each for four sound event classes(cough, snore, doorbell, phonebell). Total five rounds of eight sound events were testes, so total forty
events were tested in this pilot test. The F-score for whole pilot test with different overlap proportions(0%, 50% and 75%) are listed in Table III. For doorbell and phonebell events, the proposed SER model
showed 1.000 of F-score regardless of the overlap proportion setting. This verifies that personalized
SER system works well in real situations where it is mounted on a robot within a similar domestic
environment. However, cough and snore recognition performances were relatively low in case nothaving overlap. The reason why doorbell and phonebell works equally well without overlap but coughand snore decrease SER performance is the difference in acoustic characteristics of the event classes.
Doorbells and phonebells are mainly melodic sounds, and they are more than 2 seconds. Conversely,
cough and snore are impulse-like sounds, and they have a very short duration. Thus, there is a highprobability that 2-second interval will contain short part of cough or snore events, rather than fully
containing the events when collecting data with short overlap proportions. For this reason, longoverlap section is required for accurate real-time SER of cough and snore sounds.
Table 1. F-scores with different overlap proportion in pilot test
Class
Cough
Snore
DoorbellPhonebell
Average

0%
0.889
0.667
1.000
1.000
0.889

50%
0.947
0.889
1.000
1.000
0.959

75%
1.000
0.889
1.000
1.000
0.972

(a)

(b)
Figure 3. A recorded sound and recognition result: (a) raw waveform recorded by Pepper and (b) SER results
using real-time personalized SER model with 75% overlap. The dotted line in the waveform plot indicates onset
and offset points at which the event occurred and ended.

With 75% overlap proportion, the average F-score is 0.972, which is even better than theperformance verified at the laboratory level. The result of real-time recognition with 75% overlap isshown in Figure 3. It can be seen that not only accurate SER but also accurate onset and offset are
indicated. Therefore, through the pilot test, we confirmed that the proposed method and system aresuitable for accurate real-time SER for human care robot system.

4. CONCLUSION
In this paper, a real-time personalized sound event recognition system was proposed for humancare robots. A total of four events, cough, snore, doorbell, and phonebell, that the human care robot
must detect were adopted as representative classes. We propose a method to train SER model using
only specific sound events used by a specific user, taking advantage of the fact that the user and usage
environment of the human care robot hardly change. Personalized SER model showed high F-score
for doorbell and phonebell compared to the conventional model using whole sound event dataset.
Furthermore, to verify whether it could be applied to human care robot system, personalized SER
model was mounted on Pepper robot and a pilot test was conducted in the actual living environment
with real-time setting. The human care robot system with personalized SER model showed that
accurate real-time recognition performance was achieved with F-score of 0.972.

ACKNOWLEDGEMENTS
This work was partly supported by the Institute of Information & communications Technology
Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2017-0-00162,
Development of Human-care Robot Technology for Aging Society) and Korea Institute of Marine
Science and Technology Promotion (KIMST) grant funded by the year 2022 Finances of Korea
Ministry of Oceans and Fisheries (MOF) (Development of Technology for Localization of Core
Equipment in the Marine Fisheries Industry, 20210623)

REFERENCES
1. M. Janvier, X. Alameda-Pineda, L. Girinz, and R. Horaud, "Sound-event recognition with a companion
humanoid," in 2012 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids 2012),
2012: IEEE, pp. 104-111.
2. S. Park, W. Choi, D. K. Han, and H. Ko, "Acoustic event filterbank for enabling robust event recognition
by cleaning robot," IEEE Transactions on Consumer Electronics, vol. 61, no. 2, pp. 189-196, 2015.
3. K. Nakamura and K. Nakadai, "Robot audition based acoustic event identification using a bayesian model
considering spectral and temporal uncertainties," in 2015 IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS), 2015: IEEE, pp. 4840-4845.

4. J. Ren, X. Jiang, J. Yuan, and N. Magnenat-Thalmann, "Sound-event classification using robust texture
features for robot hearing," IEEE Transactions on Multimedia, vol. 19, no. 3, pp. 447-458, 2016.
5. J. F. Gemmeke et al., "Audio set: An ontology and human-labeled dataset for audio events," in 2017 IEEE
international conference on acoustics, speech and signal processing (ICASSP), 2017: IEEE, pp. 776-780.
6. A. Mesaros et al., "DCASE 2017 challenge setup: Tasks, datasets and baseline system," in DCASE 2017Workshop on Detection and Classification of Acoustic Scenes and Events, 2017.
7. A. Mesaros et al., "Detection and classification of acoustic scenes and events: Outcome of the DCASE
2016 challenge," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 2,
pp. 379-393, 2017.