Analyzing the perception of natural music with EEG and ECoG

Brain states during real-word experiences have attracted growing research interest in the past two decades. Listening to music is one example of an on-going real-world experience that, on the one hand, relies on structured auditory input, on the other hand, often involves strong emotional responses. Since, obviously, the brain is the mediator between sound wave and subjective experience, it is an interesting question whether the comparative analysis of brain signals and music signals is a way to understand this fascinating process. Electrophysiological recordings of the brain are particularly suited for this, as they offer a temporal resolution in the millisecond range, a time scale that potentially gives access to the processing of the fine details of the rapidly changing musical surface. Deriving, however, interpretable information from electrophysiological signals recorded during music listening is a challenging task. Extracting stimulus-related brain activity from the electroencephalogram typically requires averaging of a high number of stimulus repetitions. If full-length pieces of music are presented, and, moreover, the unique listening experience is of interest, more sensitive methods for extracting neural activity from the continuous brain signal are required. This thesis makes several contributions toward the development of such methods, by addressing relevant issues that arise whenever brain signals are analyzed in combination with music signals. Taking advantage of the compelling properties of invasive ECoG recordings, the first part of this thesis presents a simple, but efficient method to derive a detailed reflection of an original rock song in the brain signal. A core contribution of this thesis aims at further promoting the more widely applicable recording modality of (scalp) EEG for investigating the relationship between music signal and brain signal. Utilizing an evoked brain response to low-level constituents of music, i.e., to note onsets, we propose a multi-variate regression-based method for mapping the continuous EEG signal back onto the music signal. This so-called stimulus reconstruction approach is highly suitable for EEG recordings of the presentation of full-length, acoustically very complex music pieces. The resulting stimulus reconstructions can be used to determine the level of CorticoAcoustic Correlation (CACor): the degree of synchronization of the brain signal with the stimulus. Through CACor, this thesis explores the potential of the stimulus reconstruction approach in several music-related research scenarios. A simple, repetitive sound stimulus is used to demonstrate the connection from the extracted brain signatures to canonical ERP components. Subsequently, the method is applied in a more complex setting that relates to auditory stream segregation. We first demonstrate that three monophonic semi-musical stimuli can be reconstructed from the listener’s EEG. Next, we show to what extent such learned mappings can be utilized to trace a neural correlate of the separate streams that the listener perceives when the three voices play together forming a polyphonic semi-musical sound pattern. Finally, we progress to the most ‘natural’ experimental setting in this context: the analysis of EEG recordings during listening to ‘real’ music without a specific task. We examine CACor for a range of naturalistic music and non-music sounds. We show that differences between stimuli with respect to observed significant CACor can be related to their acoustic/higher-level musical properties. Finally, with a complementary analysis of behavioral reports of perceived tension we provide first evidence on the experiential relevance of the observed relationship between brain signal and music signal.