PROCEEDINGS of the
23rd International Congress on Acoustics
9 to 13 September 2019 in Aachen, Germany

The ear in the age of IoT
Jérémie VOIX1
1

Université du Québec (ÉTS), Montréal, Québec, Canada

ABSTRACTAt this age of Internet of Things (IoT), wearables are now everywhere, sometimes even in your earcanal.
The research team from the NSERC-EERS Industrial Research Chair in In-Ear Technologies (CRITIAS)
has been actively developing various in-ear technologies designed to complement the human ear, from"smart" hearing protection against industrial noises, to advanced individual communication systems, tohearing health monitoring devices using otoacoustic emission (OAE), to in-ear EEG Brain Computer
Interface (BCI). More fundamental research has also been conducted, particularly on the micro-harvestingof electrical power from inside the earcanal to power future auditory wearables. The current research
activities from CRITIAS and several possible applications are presented in this article, while thecorresponding keynote presentation will feature other research groups research and resulting technologies.
Keywords: Acoustics, hearing protection, hearing aid, in-ear biosensing, wearables, internet of things (IoT)

1. INTRODUCTION
This introduction gives an overview of the NSERC-EERS Industrial Research Chair in In-Ear
Technologies (CRITIAS) research group, by presenting briefly its history as well as its mission andfocus.
1.1 A little bit of history
The NSERC-EERS Industrial Research Chair in In-Ear Technologies (CRITIAS) was created in
September 2016 as part of a long-standing and successful partnership between EERS Global Technologies
Inc (previously Sonomax Hearing Healthcare Inc) and École de technologie supérieure (ÉTS). The
partnership was initiated in late 1999 and led to a unique technology designed to protect industrial workers
from noise-induced hearing loss (NIHL). The resulting hardware and software solution, dubbed theSonomax Solution, was protected by over 50 patents and trademarks and has been marketed all over theworld. Following 10 years of collaborative work, the partnership became more firmly established when theauthor was appointed at ÉTS as professor and founded the CRITIAS Research Chair in 2010, now
supported by the National Science and Engineering Research Council of Canada (NSERC) and dedicated tohis original quest: to develop a true “bionic” ear that provides effective protection, amplification,communication, monitoring and biosensing within a single in-ear device.
1.2 Mission and Research Focus
CRITIAS focuses on the development of various technologies designed to complement thehuman ear, from “intelligent” hearing protection and communications in high -noise environments tothe integration of in-ear brain-computer interfaces and wearable hearing diagnostics. The research
activities focus on three areas: Digital Hearing Protection, Communication in High -Noise
Environments, and In-Ear Biosensing. The research activities corresponding to these three themes
areas are further described in Section 2 and are illustrated in the Fig.1.

2. CURRENT RESEARCH
Beyond the ubiquitous availabilties of hearing protection and hearing aid, the human earcanal isa perfect place for many applications that span well beyond hearing. While the author and its
industrial partner have been early believers of such statement, that shaped all the collaborative1

Jeremie.voix@etsmtl.ca

1research work that was conducted within CRITIAS since early 2010, it is exciting to see many newcommercial entities now sharing the same vision (1) and are rapidly putting ear-centered wearable
products (often called “hearables”) on the market. The market for hearables is considered to be oneof the fastest growing (2) as it brings all the smartphones and tablet power one step closer to thehuman being. While probably not exhaustive, Fig. 1 illustrates the many possible applications anduses of the human earcanal, ranging from digital advanced hearing protection and hearing aid,
communication in noise, to in-ear biosensing and even electric and light stimulation.

Figure 1: Several of the possible uses of the human earcanal with hearables
The following sections detail several possible uses and present the specific applications that areenvisioned within current research activities at CRITIAS. The maturity of that research is rated with
the Technology Readiness Level (TRL), a value that ranges from 1 (conceptual idea) to 9
(commercial product), assuming that technologies typically move at TRL 5 -6 from academic
research activities conducted at CRITIAS to industrial R&D conducted at EERS.
2.1 Digital Hearing Protection
Passive Hearing Protection (TRL 9)
The preferred passive hearing protection device (HPD) used at CRITIAS is an instant custom
fitted earpiece, adjusted using SonoFit™ expandable silicone in-ear technology (3). It offers
superior comfort, retention and attenuation and is perfect to be further equipped with electronic(digital signal processor, sensors, electrodes, etc.) and electroacoustic components (speaker,
microphones, etc.), as illustrated in Fig. 2, to become a “digital earplug”.

2Figure 2 : Illustration of a digital earpiece obtainedFirst Fit Test (TRL 9)
One of the early technologies developed for the instant custom fit technologies is the individual
field fitting testing of HPD. An objective microphone measurement dubbed Field Microphone inReal Ear (F-MIRE) is performed with the digital earplug and checks HPD attenuation for eachwearer (4).
In-ear Dosimetry (TRL 9)
Typical noise dosimetry is performed by picking up the sound exposure of a worker with amicrophone placed on the shoulder. With the In-Ear Dosimetry approach, the microphone is placedin the occluded earcanal, measuring what noise level actually reaches the eardrum through thewearer’s HPD, but discarding wearers induced disturbances (such as speech, coughing, teeth
grinding etc.) that are not to be accounted for in the cumulated noise dose, as they are non hazardous for the user’s hearing (5,6).
Sound Reproduction (TRL 9)
By including a miniature loudspeaker in the earpiece, signals can be played back to the protected
ear. In-ear acoustics modeling recreates sound like it feels with an open ear, creating a more naturalsound experience (7).
Digital Audio Filtering (TRL 9)
With the miniature loudspeaker inside the earpiece and a microphone outside, useful externalsound is transmitted to the protected ear. Using Modulation-Based Digital Noise Reduction(MBDNR) voice and warning signals passed through while the rest of the ambient n oise ispassively blocked (8,9).
Active Noise Control (TRL 9)
The amount of attenuation provided by the passive HDP can be further increased using active noise control (ANC). The residual noise is picked up via the in-ear microphone, inside the occluded
earcanal, dephased by 180 degrees and played back to local cancel the initial sound disturbance.
ANC can also be very useful for cancelation of the so-called “occlusion effect”. Normally, allwearers induced disturbances (own voice, footsteps, friction noise, etc.) as well as physiologicalnoise (swallowing, coughing, breathing, etc.) would be perceived under the occluded ear and could
become very annoying. With ANC, those low frequency noises are all canceled out, making theexperience more natural and comfortable. The benefits of such technology were recently studied formusicians and performers, who need to protect their hearing while singing or playing instruments
and would otherwise be disturbed and even annoyed by the occlusion effect (10,11).

3

Otoacoustic Emission Monitoring – Ear Fatigue (TRL 6)
Physiological fatigue originating from over-exposure to sound creates a gradual reduction inauditory capabilities. Rather than relying on subjective testing, such as hearing threshold testing
using audiometry, a change in hearing sensitivity -often refered to as temporary threshold shift(TTS)- can be assessed using otoacoustic emission. By stimulating the ear with 2 pure -tone stimuli,
a distortion product otoacoustic emissions (DPOAE) can be generated by the o uter hair cell of theinner ear and picked up by the in-ear microphone. It recently became possible to monitor theses
DPOAEs in high level of ambient noise, using adaptive filtering and denoising techniques (12–14).
2.2 Communication in Noise
Hearing Aid (TRL 3)
Using the existing in-ear loudspeaker and an outside microphone it is possible to add a hearing
aid feature to the digital earplug. By selectively amplifying speech and warning signals, it ispossible to assist industrial workers, who sometimes have significant existing hearing loss (15).
Speech Capture (TRL 9)
Speech produced by the wearer can be captured within the occluded earcanal. Since that signal ispicked up underneath the HPD, it benefits from its passive isolation and offers a superior signal -tonoise ratio (SNR). Recent work has been conducted to further reduce the residual noise andenhancing the intelligibility of the picked-up speech by increasing its high frequency content (16).
2.3 In-Ear Biosensing
Biosignals Measurement (TRL 5)
The in-ear microphone can pick up many signals present inside the occluded earcanal. Thissignals is often referred to as physiological noise, but contains useful information, including
heartbeat and breathing sounds from which it is possible to extract th e wearer’s heartbeat andbreathing rate (17). It has been also shown that other non-verbal signals could be picked up(coughing, teeth grinding, teetoh or tongue clicking, sneezing, etc.) and could be used for a wide
range of hand-free and silent man-machine interface or as well for continuous health monitoring
application (18).
Ambient and Medical Conditions Measurement (TRL 4)
The ear is a perfect place to measure temperature, as traditionally done in medical diagnostics.
Using commercialy available sensors inside the occluded earcanal, ambient temperature andhumidity levels are easily recorded and monitored. The combination of these two measurements
detects workers’ physical involvement or overexertion (such as heavy lifting, levels of sweating) andcombined with prior biosignals measured can be very reliable indicator of the health and conditionof the wearer.
Dynamic Measurement (TRL 6)
Head movement can be recorded with 6-axis Inertial Measurement Unit (IMU) using
accelerometer and velocimeter sensors. These measurements of head’s dynamic are suitable formonitoring fatigue (like head nodding down, etc.) or for musculoskeletal troubles induced byrepetitive movements or activities (such as using a jackhammer, long -distance driving, etc.). The
addition of a magnetometer (packaged as an 9-axis IMU) enables applications with 3D audio
processing, which must account for head position in azimuth and elevation (gazing, etc.). A man down detection (fall or injury detection) has recently been developed at CRITIAS using this same
technology (19).
Physiological Sensing (TRL 3)
By making the earpiece conductive, for example by integrating a electrode or by using metal coated or conductive polymer, skin contact electrical impedance can now be measured. This uniquemarker tightly related to how the custom earpiece is fitting inside the earcanal, enables the uniqueverification of the wearer’s biometric information. The same conductive earpiece can record muchof the electro-dermal activity, capturing muscle and heart activities, thereby enabling a wide rangeof activity tracking and man-machine interface application.
Ear Canal Change Detection (TRL 2)

4

The conductive earpiece can easily detect movement in the earcanal induced bytemporomandibular activities. Monitoring such activities (jaw movement, opening and closing themouth, stretching, grinding teeth, etc.) creates some interesting applications. Examples includedentistry temporomandibular jaw-joint (TMJ) pain from jaw clenching, or linguistic research tounderstand jaw-joint activity during speech production. Leveraging on other biosignals detection
provides great insight for wellness and sports tracking solutions. Cardiovascular activity inside theearcanal distorts the ear canal. Blood flow pulse can be measured by monitoring the conductive
earpiece electrical properties (impedance, charge, voltage, etc.) while in contact against theearcanal, possibly opening new ways to monitor cardiovascular activities and diseases.
Electrophysiology (TRL 5)
By adding additional miniaturized electrodes on the earpiece, inside of the ear (intra -aural) andother around the ear pina (circumaural), brain wave signals can be recorded and monitored. While
such electrophysiological signals (EEG) are not as specific as the ones obt ained with a full scalp
EEG with a much higher density of electrodes, they are easy to record in a non -invasive way,
making them suitable for real-world applications. Auditory evoked potential (ERP) can beconveniently measured with a device that combined these electrodes on the audio digital earpiece(20, 21).
2.4 In-Ear Stimulation
The following two research activities are not currently taking place at CRITIAS, nor are they on
the short-term research agenda of the CRITIAS team but are possible innovative applications.
Electrical Stimulation
Instead of receiving signals, the electrodes can act as actuators, to send modulated electricalsignals, for example using the direct current stimulation paradigm. This can slightly alter the
behavior of the nervous system and could have promising applications, given the proximity of thevagus nerve, which comes close to the surface of the earcanal tissues. This pneumogastric nerve has
a tight parasympathetic control of the heart, lungs, and digestive tract and offers. There is researchusing this stimulus to alters motion sickness, appetite and nausea stemming from radiation therapy.
There is another line of research where such stimulation, combined with sound signals, can alter
auditory perception and mood for therapy or entertainment (22).
Light Stimulation
Instead of sending electrical signals, visible light can be shined inside the occluded earcanal.
There is recent research that shows that some photoreceptor-proteins in the brain can receive thatlight and that similar benefits as the one provided by light therapy have been observed, affecting thegenerationmolecules regulating circadian rhythmicity. This offers convenient and discrete means totreat jet lags, seasonal mood and affective disorders (23).
2.5 In-Ear Energy Harvesting
Energy Harvesting (TRL 3)
Many of the “hearable” applications mentioned above require a continuous electrical power
supply. One possible approach, rather than relying on disposable or rechargeable batteries orcapacitors is to harvest the energy directly from the wearer itself. Amongst the many possible
sources (head movement, head heat, solar radiation, etc.) a promising source appear to be themechanical distortion of the earcanal induced by the jaw-joint movement. Prototypes of
piezoelectric micro-energy harvesters have been developed and were proven to be able to recharge abattery over regular daily movements of the jaws (from eating, swallowing, yawning, speaking, or
chewing gum, etc.) (24–28).

3. CONCLUSIONSAt the dawn of the 4 th industrial IoT 5G revolution, it is clear that the interconnection between
the human being and machine (or network) is getting more real every day. The human earcanalappears, at least in the views of the researchers from CRITIAS, as a perfect place to ac hieve thisinterconnection and the many possible applications that have been highlighted in this paper. Whilesome of these applications may be years away, it is worth noting that several disruptive technologies

5developed by CRITIAS have already been integrated into innovative products brought to market by
EERS Global Technologies, the industrial partner. For example, the most recent version of the“digital earplug”, commercialized by EERS under the name “SonX”, won in 2016 the first prize forthe “Hear and Now” Noise Safety Challenge, a joint initiative by the US Department of Labor(DOL), the National Institute for Occupational Safety and Health (NIOSH), the Mine Safety and
Health Association (MSHA), and the Occupational Safety and Health Association (OSHA ) for
technologies that uniquely reduce work-induced hearing loss.

ACKNOWLEDGEMENTS
The financial support received from the National Science and Engineering Research Council of
Canada (NSERC), from the Fonds de recherche du Québec Nature et Technologie (FR QNT), fromthe Institut de recherche Robert-Sauvé en santé et sécurité au travail (IRSST), from PROMPT and
MITACS consortiums and from EERS Global Technologies inc have been greatly appreciated topursue this innovative line of applied research.

REFERENCES
1.

Crum P. Hearables Will Monitor Your Brain and Body to Augment Your Life [Internet]. IEEE
Spectrum: Technology, Engineering, and Science News. 2019 [cited 2019 May 6]. Available from:https://spectrum.ieee.org/consumer-electronics/audiovideo/hearables-will-monitor-your-brain-andbody-to-augment-your-life

2.

Why “hearables” will trump wearables — and make you healthier, too [Internet]. [cited 2015 Jun 8].
Available from: http://venturebeat.com/2014/10/21/why-hearables-will-trump-wearables-and-makeyou-healthier-too/3.

Voix J, Laville F. Expandable Earplug with Smart Custom Fitting Capabilities. In: 2002 [Internet].
Dearborn, MI, USA.: International Institute of Noise Control Engineering (I-INCE) Ames, IA; 2002.
p.

833–841.

Availablefrom:

http://link.aip.org/link/?INC/111/833/1

http://www.sonomax.com.au/images/pdfs/InterNoise2002JeremieVoix.pdf
4.

Voix J, Laville F. The Objective Measurement of Earplug Field Performance. J Acoust Soc Am[Internet]. 2009 Jun;Vol. 125(Issue 6):3722–3732. Available from: http://aipprod.aip.org/ap/cgibin/apGetFile?FileType=PROOF&FileID=052906JAS-39134.pdf
http://scitation.aip.org/getpdf/servlet/GetPDFServlet?filetype=pdf&id=JASMAN0001250000060037
22000001&idtype=cvips http://www.aip.org/servlet/AmsisReply

5.

Fabien Bonnet, Hugues Nélisse, Marcos-Antonio Nogarolli, Jérémie Voix. In-Ear Noise Dosimetry
under Earplug: Method to Exclude Wearer-Induced Disturbances. IJIE Int J Ind Ergon.
2019;(Submitted).

6.

Fabien Bonnet, Hugues Nélisse, Marcos-Antonio Nogarolli, Jérémie Voix. Individual in-situ
calibration of in-ear noise dosimeters. Appl Acoust. 2019;Submitted.

7.

Bacon C. Measurement, modelling and simulation of earphone sound signature [Internet] [masters].
École

6

de

technologie

supérieure;2013[cited

2019

Apr

4].

Availablefrom:https://espace.etsmtl.ca/1201/
8.

Narimene Lezzoum, Ghyslain Gagnon, Jérémie Voix. Noise reduction of speech signals using timevarying and multi- band adaptive gain control for smart digital hearing protectors. Appl Acoust. 2016
Aug;109:37–43.

9.

Carbonneau M-A, Lezzoum N, Voix J, Gagnon G, Gaudreau M-A. Detection of Alarms and Warning
Signals on an Digital In-Ear Device. Int J Ind Ergon [Internet]. 2013 Nov;43(6):503–511. Available
from: http://elsarticle.com/1ejptvW10.

Bernier A, Voix J. An Active Hearing Protection Device for Musicians. In: Proceedings of Meetingson Acoustics [Internet]. Montreal (QC), Canada: Acoustical Society of America; 2013. p. 040015–
040015. Available from: http://link.aip.org/link/PMARCW/v19/i1/p040015/s1&Agg=doi

11.

Antoine Bernier, Jérémie Voix. Active musician’s hearing protection device for enhanced perceptual
comfort. In: Euronoise 2015. Maastricht, Netherlands; 2015. p. 1773–8.

12.

Nadon V, Bockstael A, Botteldooren D, Lina J-M, Voix J. Design considerations for robust noise
rejection in Otoacoustic Emissions measured in-field using adaptive filtering. Acta Acust United
Acust. 2017;Vol. 103(No. 2):299–310.

13.

Vincent Nadon, Annelies Bockstael, Dick Botteldooren, Jérémie Voix. Field monitoring of
otoacoustic emissions during noise exposure: a pilot study within controlled environment. Am J
Audiol F. 2017 Oct;26):352–68.

14.

Vincent Nadon, Annelies Bockstael, Dick Botteldooren, Jean-Marc Lina, Jérémie Voix. Individual
monitoring of hearing status: development and validation of advanced techniques to measure
otoacoustic emissions in suboptimal test conditions. Appl Acoust. 2015 Mar;APAC5425:78–87.

15.

Laroche C, Vaillancourt V, Gendron M, Fortier P, Paré L, Giguère C, et al. Hearing aids in noisy
workplaces: a good or bad solution? Spectrum [Internet]. 2014 Dec;December 2014(NHCA Spectrum
Vol. 31 (2) 18-21). Available from: https://www.hearingconservation.org/page/2014SpecDecLaroche

16.

Rachel Bouserhal, Tiago Falk, Jeremie Voix. In-ear microphone speech quality enhancement via
adaptive filtering and artificial bandwidth extension. J Acoust Soc Am [Internet]. 2017
Mar;141(3):1321–31. Available from: http://asa.scitation.org/doi/full/10.1121/1.4976051

17.

Martin A, Voix J. In-Ear Audio Wearable: Measurement of Heart and Breathing Rates for Health and
Safety Monitoring. IEEE Trans Biomed Eng. 2018 Jun;65(6):1256–63.

18.

Bouserhal RE, Chabot P, Sarria-Paja M, Cardinal P, Voix J. Classification of Nonverbal Human
Produced Audio Events: A Pilot Study. In: Interspeech 2018 [Internet]. Hyderabad, India: ISCA; 20187

[cited

2019

Feb

18].

p.

1512–6.

Availablefrom:

http://www.isca-

speech.org/archive/Interspeech_2018/abstracts/2299.html
19.

Alex Guilbeault-Sauvé. Détection de situations d’homme-à-terre à l’aide d’une plateforme inertielle
intra-auriculaire [Master’s thesis]. [Montréal, QC, Canada]: École de technologie supérieure; 2018.

20.

Valentin O, Ducharme M, Cretot-Richert G, Monsarrat-Chanon H, Viallet G, Delnavaz A, et al.
Validation and Benchmarking of a Wearable EEG Acquisition Platform for Real-World Applications.
IEEE Trans Biomed Circuits Syst. 2018 Oct 15;13(1):1–1.

21.

Olivier Valentin, Jérémie Voix. EARtrodes: Towards a wireless in-ear custom-fitted brain computerinterface. In: Canadian Acoustics [Internet]. Guelph, ON; 2017. p. 142–3. Available from:https://jcaa.caa-aca.ca/index.php/jcaa/article/view/3067

22.

Siever D. Stimulation Technologies: “New” Trends in “Old” Techniques. Biofeedback [Internet].
2015

Dec

[cited2019

Aug7];43(4):180–92.

Availablefrom:

http://www.aapb-

biofeedback.com/doi/10.5298/1081-5937-43.04.11
23.

Jurvelin H, Jokelainen J, Takala T. Transcranial Bright Light and Symptoms of Jet Lag: A
Randomized, Placebo-Controlled Trial. Aerosp Med Hum Perform [Internet]. 2015 Apr 1 [cited 2019
Aug7];86(4):344–50.

Availablefrom:

http://openurl.ingenta.com/content/xref?genre=article&issn=23756314&volume=86&issue=4&spage=344
24.

Carioli J, Delnavaz A, Zednik RJ, Voix J. Power capacity from earcanal dynamic motion. AIP Adv[Internet].

2016

Dec

1

[cited2016

Dec5];6(12):125203.

Availablefrom:

http://scitation.aip.org/content/aip/journal/adva/6/12/10.1063/1.4971215
25.

Delnavaz A, Voix J. Energy harvesting for in-ear devices using ear canal dynamic motion. IEEE
Trans

Ind

Electron[Internet].

2014

Jan;61(1):583

–

590.

Availablefrom:

http://www.mendeley.com/download/public/3842911/5513604504/217cd66960d570f0792f1c5dcb7edde344da93d5/dl.pdf http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6420936
26.

Aidin Delnavaz, Jérémie Voix. Flexible piezoelectric energy harvesting from jaw movements. Smart
Mater Struct - IOP Publ Ltd. 2014 Oct;Vol. 23(Num. 10):8 pp.

27.

Delnavaz A, Voix J. Ear canal dynamic motion as a source of power for in-ear devices. J Appl Phys
[Internet].

2013

Feb;113(6):9p.

Availablefrom:

http://link.aip.org/link/?JAP/113/064701

http://www.mendeley.com/download/public/3842911/5513604474/fb1ba78b8d2cec7ea9badd991d518
4d38d7d5263/dl.pdf
28.

Delnavaz A, Voix J. Piezo-earpiece for micro-power generation from ear canal dynamic motion. J
Micromechanics Microengineering. 2013 Nov;23(114001):8pp.

8