Validation of highly acoustically rich audiovisual stimuli for naturalistic speech encoding in the brain
Creators: name: Desai, Maansi
name: 0000-0002-1209-699X
Publisher: name: The University of Texas at Austin
Year: 2023
Subjects: Naturalistic stimuli
Electrophysiology
Speech
Encoding models

description: In natural conversations, listeners must attend to what others are saying while ignoring extraneous background sounds. The process of being able to perceive speech in our daily environment relies on a combination of auditory and visual cues. In recent years, the use of audiovisual stimuli in auditory neuroscience experiments has become increasingly popular due to the potential benefits of incorporating multi-sensory information in investigating auditory perception and processing. Using naturalistic stimuli is in contrast to more controlled stimuli, which have traditionally been used, but may be limited in investigating neural encoding of speech without additional overlapping sound sources and visual information. Furthermore, controlled stimuli may be monotonous, leading to participant fatigue and lack of subject engagement with a given task which can diminish the quality of the acquired data. The purpose of this study is to use movie trailers, an acoustically rich audiovisual stimulus set, to understand brain responses to speech. We employ these stimuli in two participant groups: (1) healthy subjects while they underwent non-invasive scalp electroencephalography (EEG) and (2) pediatric epilepsy patient undergoing intracranial monitoring for seizures using stereoelectroencephalography (sEEG). The motivation of this dissertation work is to identify if it is possible to replace more controlled experimental paradigms with stimuli that are more acoustically rich, multi-modal, and engaging to listen to through four specific aims. Aim 1 compared neural responses from an EEG task where adults listened to sentences from a sentence corpus and listened to and watched movie trailers and provide evidence that neural responses to speech from the movie trailers could be encoded with comparable fidelity to sentence listening stimuli. Aim 2 assessed the effect that the visual information had on auditory encoding and the role of uni-modal neural encoding. Aim 3 provided methodological context in determining the amount of data that would needed when building encoding models to a variety of naturalistic stimuli. Aim 4 evaluated the ability to predict neural responses to speech using sentence listening and movie trailers in pediatric sEEG populations to further motivate the transition of using more ecological valid and engaging naturalistic stimuli in lieu of traditional sentence listening tasks. To address these aims, neural data were collected from healthy EEG subjects (n=27) and pediatric sEEG patients undergoing epilepsy monitoring (n=23). All subjects listened to a subset of continuous sentences presented without any background sounds and watched and listened to children's movie trailers. Encoding models, based on a linear regression framework, were used to predict neural responses to auditory, consisting of acoustic and linguistic information, and several feature sub-spaces of visual information. Overall, the results demonstrated that it was possible to encode brain responses to speech using the movie trailers with robust fidelity compared to just pure sentence listening. Additionally, the impact of visual information did not affect the ability to encode auditory information and the auditory information did not impact visual encoding. Overall, our results provide evidence that robust auditory and audiovisual selectivity can be uncovered using more naturalistic, multimodal stimuli. This is promising for clinical research or research on populations who may not tolerate or become fatigued by traditional psychoacoustics paradigms.