Facial expressions of emotion as a measure of emotional responses and listening difficulty
Creators: name: Venkitakrishnan, Soumya
Publisher: name: University of Iowa
Year: 2024
Subjects: Confusion
Emotional responses
Facial expressions
Frustration
Listening effort
public abstract

description: Research in emotional responses is in its infancy in Audiology. Past studies have shown a reduced range of emotional responses to non-speech sounds in individuals with hearing loss, and a relationship between emotional responses and feelings of social disconnectedness and isolation. Hence, it is important to study emotional responses. Objective measures of emotional responses or emotional responses to speech stimuli have not been explored. In this study we use a novel method- automatic facial expression recognition - to measure objective, unbiased and unconscious measures of emotional responses. Further, we measure emotional responses to speech stimuli as this is important for communication. We measure emotional responses using facial expressions associated with confusion and frustration in adverse listening situation, i.e., to speech in noise stimuli in normal hearing, and in simulated hearing loss conditions. Our primary aim is to determine the effect of hearing loss on objective and subjective measures of emotional responses. We also determine the relationship between emotional responses, and measures of listening effort. Our secondary aim is also to determine whether objective emotional responses can be obtained remotely. We measured emotional responses in 33 participants with normal hearing in the normal hearing and in the simulated hearing loss condition in the lab, while simultaneously measuring pupil responses. We also measured emotional responses remotely. Our main outcome variables were objective and subjective measures of emotional responses (obtained in the laboratory and remotely) and listening effort. We found that objective and subjective measures confusion and frustration were greater in the simulated hearing loss condition. This finding could have consequences for individuals with hearing loss. For example, experiencing increased emotions of confusion and frustration may discourage individuals with hearing loss from participating in communication situations. We also found a weak positive correlation between subjective and objective measures of emotional response. This could mean that it is important to complement objective measures with subjective ratings of emotional response to get a measure of perceived and unconscious emotional responses. Further, we did not find a correlation between our objective measures of emotional response and objective measures of listening effort, though there was a negative correlation between the subjective measures. Though we see confusion and frustration, as well as listening effort increase with increasing listening difficulty, they are not statistically correlated. Hence, these constructs may be related, but measure different components of the communication and speech perception process. When comparing the responses for lab vs remote condition, we found that greater confusion and frustration was seen in the remote condition. This indicated that some stimulus and hardware considerations need to be taken when measuring objective emotional responses to speech stimuli in remote conditions. Overall, we found that this new method of measuring emotional responses using automatic facial expressions detection algorithm is feasible for use in measuring emotional responses.