DOI: 10.14457/tu.the.2019.1440
title: A study on improving emotional speech classification by spotting emotional speech fragments and Thai speech corpus construction

description: Detecting emotion in speech is a challenging task that enhances conventional speech recognition towards fully understanding speech. Most existing speech emotion recognition methods focused on the whole speech utterance and applied standard speech recognition techniques to recognize the speaker's mood. However, in real situations, emotion is usually included in only some speech fragments in speech utterance, not in the whole utterance. Spotting speech fragments related to emotion (effective fragments) and ignoring unrelated fragments (ineffective fragments) can improve speech emotion recognition. In this research, three principal works are designing and developing Thai emotional speech corpus (namely EMOLA), emotion clustering, and applying feature expansion and frame selection with stacking methods for speech emotion recognition. The first part states the corpus design, construction, annotation process, and its analysis. Four basic types with twelve subtypes of emotions are defined with consideration of the Pleasure-Arousal- Dominance (PAD) emotional state model.To construct the corpus using a series of Thai dramas (1,397 minutes) containing its video clips of approximately 868 minutes were annotated. As a result, 8,987 transcriptions (of conversation turns) were derived in total, with each transcription tagged as one primary type and a few subtypes. Finally, an analysis was conducted to describe this corpus's characteristics in three sets of statistics: collection-level, annotator-oriented, and actor-oriented statistics. The second part explains how to apply a clustering approach to spot emotional speech fragments in an utterance. Several experiments were performed to evaluate the proposed spotting method. The results of spotting emotional fragments reduced the number of processing data by one-third, but the accuracy is not improved as expected. Finally, an approach was proposed to enhance speech emotion recognition or classification using four complementary methodologies; stacking, feature expansion, emotional frame selection, and ensemble techniques. As feature extraction, convolution neural networks with long short-term memory (ConvLSTM) are used to translate speech intensities (raw features) into the frame-level features. Then, the frame-level features are transformed into utterance-level features. As a stack model, the utterance-level features from ConvLSTM, are fed to Support Vector Machines (SVMs) for final emotion judgment. As the post feature extraction, feature expansion is introduced to enrich feature space by labeling each phone with two feature sets; emotion feature set and non-emotion feature set. The emotional frame selection utilizes probabilities of emotion and non-emotion feature sets in frame-level features to select frames (or fragments) that are likely to contain emotions. The ensemble method exploits multiple high-performance models generated with various settings and combines their results to predict performance better. Some experiments were performed to evaluate the proposed methods using the Interactive Emotional Dyadic Motion Capture (IEMOCAP) database. As the best result, the six-model ensemble method with stacking and feature expansion achieved 68.3%, i.e., approximately 16.95% improvement over the baseline performance. At the same time, the frame selection helped reduce nearly 14% data with comparable performance.