PROCEEDINGS of the23rd International Congress on Acoustics
9 to 13 September 2019 in Aachen, Germany

Audiovisual speech processing and listening effort in untreatedage-related hearing loss: evidence from functional magnetic
resonance imaging
Christiane M THIEL1,2; Stephanie ROSEMANN1,2
1

Biological Psychology, Department of Psychology, Department for Medicine and Health Sciences,
Carl-von-Ossietzky Universität Oldenburg, Oldenburg, Germany

2 Cluster of Excellence “Hearing4all”, Carl von Ossietzky Universität Oldenburg, Oldenburg, Germany

ABSTRACT
Functional magnetic resonance imaging (fMRI) enables objective measurement of central auditory
processing. We used fMRI to investigate neural processing of audiovisual speech in hearing impaired
unaided and non-hearing impaired elderly subjects. Speech stimuli were used under four different
conditions, visual only, auditory only and two audio-visual conditions, in which subjects could see thespeakers face, one with congruent auditory and visual information and a second with incongruent
information. While performance in this task did not differ between hearing impaired and non -hearing
impaired participants, task-related neural activity differed. Hearing impaired participants showed
higher neural activity in medial and lateral prefrontal cortex which correlated with the amount ofhigh-frequency hearing loss. We suggest that the additional recruitment of frontal brain regions may
relate to increased effort, which was supported by reports of increased daily listening effort in thehearing impaired group. An additional data analysis on resting state fMRI data in the same group ofsubjects, showed that this increased listening effort relates to decreased functional connectivity
between inferior frontal and auditory cortex. Task and resting state fMRI data therefore provide
valuable measures of central auditory processing of speech and increases in listening effort in thehearing impaired.
Keywords: age-related hearing loss, functional neuroimaging, audiovisual speech, listening effort

1. INTRODUCTION
Age-related hearing loss affects a large proportion of the elderly but often stays untreated forseveral years (1). The loss of hearing abilities at high frequencies leads to difficulties in understanding
speech and increased listening effort, especially in multispeaker situations. The effects of reduced
sensory input on the brain are well investigated in the congenitally deaf and those with profoundhearing loss and demonstrate that the loss of auditory input induces changes in neural processing inand beyond the auditory cortex (2). A well-investigated phenomenon is crossmodal plasticity, evident
as increased responses in auditory cortex to visual stimuli. Recently, evidence for crossmodal
plasticity has also been found in mild to moderate age-related hearing loss (3, 4). For example, in afunctional magnetic resonance imaging (fMRI) study in elderly subjects with varying degree of highfrequency hearing loss, increased functional connectivity between auditory cortex and motion -related
brain regions was found when subjects had to categorize frequency-modulated tones presented alone
or in the context of non-matching versus matching visual motion (3). Together with a prior behavioural
finding that hearing impaired listeners are more distracted by incongruent v isual input (5) the results
12

gf@aaa.com
gf@aaa.com

5671

compellingly demonstrate that untreated age-related hearing-loss impacts brain connectivity andaudiovisual integration. Behaviourally, increased audiovisual integration in elderly hearing impaired
listeners has also been described for speech stimuli using the McGurk effect, an illusion that can occur
if incongruent auditory and visual syllables are presented which lead to a fused third percept (6, 7). An
increased sensitivity for processing lip movements in the hearing impaired, which convey visual
information that may aid speech understanding, may explain these findings (8). The brain mechanisms
related to processing audiovisual speech in the hearing impaired are however unclear.

2. METHODS AND RESULTS
To investigate audiovisual processing of speech in age-related hearing impairment we performed afunctional neuroimaging study in in a sample of subjects with untreated mild to moderate hearing loss
and an age-matched control group. Magnetic resonance imaging methods enable the measurement ofbrain function and structure with high spatial precision. FMRI was introduced in humans in 1991 andmeasures brain activity via the so-called blood oxygen level dependent (BOLD) contrast, which
exploits the different magnetic properties of oxygenated and deoxygenated blood (9). Even though thismeasure is an indirect measure of neuronal activity that relies on neurovascular coupling, it is strongly
correlated with the local field potential of neurons (10). The method can be used to investigate thestrength of activation in different brain regions during different task conditions or the coupling ofbrain regions during task conditions or at rest. They therefore provide an objective measure of changes
in neural activity and functional connectivity. In contrast to electrophysiological methods such as
electro- or magnetoencephalography, fMRI provides better spatial resolution and also enables trackingof neural activity in subcortical regions. The disadvantage is however the high levels of noiseproduced during measurement which can only partly be reduced by active noise cancellation.
We will here report on a previously published dataset that was acquired in normal hearing (n=19;mean age 63.2) and hearing impaired participants (n=20; mean age 63.5) who performed anaudio-visual sentence detection task inside the MRI scanner (3T Prisma with 20 channel headcoil) as
well as a so-called resting state scan. In addition, subjects performed the McGurk task outside thescanner and filled in a questionnaire on daily listening effort. Hearing status was assessed with puretone audiometry. Mean values for high frequencies (2-8kHz) were 39.5 dB for the hearing impairedand 18.9 dB for normal hearing subjects.
Analysis of the behavioral data acquired outside the scanner indicated significantly higher daily
listening effort and stronger audio-visual integration (increased McGurk effect) in hearing impaired
compared to normal hearing subjects. Furthermore, hearing-loss was highly correlated withaudio-visual integration with higher hearing-loss leading to a higher amount of McGurk illusions.
These results confirm prior findings that mild to moderate hearing impaired subjects are more prone to
the McGurk illusion than normal hearing subjects. Behavioural data in the MRI showed best
performance for congruent audiovisual sentences but no differences between hearing impaired andnormal hearing subjects, which may have been due to the scanner noise. Neural activity during
sentence processing differed however significantly between both groups. Increased activation wasfound in the hearing impaired in frontal areas for unimodal visual, unimodal auditory and incongruent
audio-visual sentences (see Figure 1, for further results see (11)). Our results indicate a hearing-loss
induced modulation of the BOLD response in frontal areas during unimodal auditory, visual andincongruent audio-visual input, but not for congruent audio-visual input. This effect may be related to
an increased listening effort in those conditions in the hearing impaired.

5672

Figure 1 – Higher neural activity in hearing impaired vs. normal hearing subjects showing increased BOLD
signal in medial, middle and inferior frontal gyri (side view upper row), as well as cingulate cortex (medialview, lower row) with incongruent audio-visual speech [p<0.05; FWE corrected on the cluster level]. Similar
activation differences were found for the unimodal auditory and visual conditions (not shown). No
differences between hearing impaired and normal hearing subjects were seen with congruent audio-visualspeech. Figure adapted from (11).
To further investigate long-term effects of increased daily listening effort in the study population
the subjectively reported questionnaire data was related to resting state functional connectivity.
Resting state functional connectivity measures coherent fluctuations of BOLD signal between brain
regions. Several resting state networks, such as the default mode, the dorsal attention and the salience
network, have been identified (12). Changes in these networks have been related to various diseasestates, two recent studies also report changes in relation to age-related hearing loss (13, 14) but therelation to daily listening effort is unclear. Our data provide evidence for lower functional connectivity
with increased listening effort. This effect was present in the dorsal attention network and in aconnection from auditory cortex to inferior frontal gyrus (for further results see (15) ). Note that thesechanges in functional connectivity were not dependent on hearing loss, which suggests that increased
listening effort experienced in everyday life rather than the hearing loss itself has a significant
influence on resting state connectivity.

3. CONCLUSIONSThe data presented here suggest that mild to moderate untreated hearing-loss and even increased
daily listening effort can lead to compensatory changes in brain activity and connectivity. Whether
these changes are beneficial or maleficial will need to be determined in f uture studies.

ACKNOWLEDGEMENTS
This work was funded by the German Research Foundation (Deutsche Forschungsgemeinschaft,5673

DFG; Cluster of Excellence DFG 1077 ‘‘Hearing4all’’) and supported by the Neuroimaging Unit of theCarl von Ossietzky Universität Oldenburg funded by grants from the German Research Foundation(3T MRI INST 184/152-1 FUGG). The authors wish to thank Gülsen Yanc and Katharina Grote forhelping with MRI data acquisition.

REFERENCES
1. Davis A, Smith P, Ferguson M, Stephens D, Gianopoulos I. Acceptability, benefit and costs of early
screening for hearing disability: a study of potential screening tests and models. Health Technol Assess.
2007;11(42):1-294.
2. Kral A, Kronenberger WG, Pisoni DB, O'Donoghue GM. Neurocognitive factors in sensory restorationof early deafness: a connectome model. Lancet Neurol. 2016;15(6):610-21.
3. Puschmann S, Thiel CM. Changed crossmodal functional connectivity in older adults with hearing loss.
Cortex. 2017;86:109-22.
4. Campbell J, Sharma A. Cross-modal re-organization in adults with early stage hearing loss. PloS one.
2014;9(2):e90594.
5. Puschmann S, Sandmann P, Bendixen A, Thiel CM. Age-related hearing loss increases cross-modaldistractibility. Hear Res. 2014;316:28-36.
6. Stropahl M, Chen LC, Debener S. Cortical reorganization in postlingually deaf cochlear implant users:
Intra-modal and cross-modal considerations. Hear Res. 2017;343:128-37.
7. MacDonald J, McGurk H. Visual influences on speech perception processes. Percept Psychophys.
1978;24(3):253-7.
8. Brown VA, Hedayati M, Zanger A, Mayn S, Ray L, Dillman-Hasso N, et al. What accounts forindividual differences in susceptibility to the McGurk effect? PloS one. 2018;13(11):e0207160.
9. Belliveau JW, Kennedy DN, Jr., McKinstry RC, Buchbinder BR, Weisskoff RM, Cohen MS, et al.
Functional mapping of the human visual cortex by magnetic resonance imaging. Science.
1991;254(5032):716-9.
10. Logothetis NK. What we can do and what we cannot do with fMRI. Nature. 2008;453(7197):869-78.
11. Rosemann S, Thiel CM. Audio-visual speech processing in age-related hearing loss: Strongerintegration and increased frontal lobe recruitment. Neuroimage. 2018;175:425-37.
12. Beckmann CF, DeLuca M, Devlin JT, Smith SM. Investigations into resting-state connectivity using
independent component analysis. Philos Trans R Soc Lond B Biol Sci. 2005;360(1457):1001-13.
13. Husain FT, Carpenter-Thompson JR, Schmidt SA. The effect of mild-to-moderate hearing loss onauditory and emotion processing networks. Frontiers in systems neuroscience. 2014;8:10.
14. Schmidt SA, Akrofi K, Carpenter-Thompson JR, Husain FT. Default mode, dorsal attention andauditory resting state networks exhibit differential functional connectivity in tinnitus and hearing loss.
PloS one. 2013;8(10):e76488.
15. Rosemann S, Thiel CM. The effect of age-related hearing loss and listening effort on resting stateconnectivity. Sci Rep. 2019;9(1):2337.

5674