Revisiting Modeling and Evaluation Approaches in Speech Emotion
  Recognition: Considering Subjectivity of Annotators and Ambiguity of Emotions
Authors: name: Huang-Cheng Chou; name: Chi-Chun Lee
Published: 2025-10-07 13:45:09

Abstract: Over the past two decades, speech emotion recognition (SER) has received
growing attention. To train SER systems, researchers collect emotional speech
databases annotated by crowdsourced or in-house raters who select emotions from
predefined categories. However, disagreements among raters are common.
Conventional methods treat these disagreements as noise, aggregating labels
into a single consensus target. While this simplifies SER as a single-label
task, it ignores the inherent subjectivity of human emotion perception. This
dissertation challenges such assumptions and asks: (1) Should minority
emotional ratings be discarded? (2) Should SER systems learn from only a few
individuals' perceptions? (3) Should SER systems predict only one emotion per
sample?
  Psychological studies show that emotion perception is subjective and
ambiguous, with overlapping emotional boundaries. We propose new modeling and
evaluation perspectives: (1) Retain all emotional ratings and represent them
with soft-label distributions. Models trained on individual annotator ratings
and jointly optimized with standard SER systems improve performance on
consensus-labeled tests. (2) Redefine SER evaluation by including all emotional
data and allowing co-occurring emotions (e.g., sad and angry). We propose an
``all-inclusive rule'' that aggregates all ratings to maximize diversity in
label representation. Experiments on four English emotion databases show
superior performance over majority and plurality labeling. (3) Construct a
penalization matrix to discourage unlikely emotion combinations during
training. Integrating it into loss functions further improves performance.
Overall, embracing minority ratings, multiple annotators, and multi-emotion
predictions yields more robust and human-aligned SER systems.