Fully Automatic Audiovisual Emotion Recognition: Voice, Words, and the Face

Abstract The recognition of human emotions from spontaneous andnon-prototypical real-life data is currently one of the mostchallenging tasks in the ﬁeld of affective computing. Thiscontribution presents our recent advances in assessing di-mensional representations of emotion, such as arousal, ex-pectation, power, and valence, in an audiovisual human-computer interaction scenario. We propose a fully auto-matic multimodal recognition approach based on context-sensitive modeling of audio and video features. Evalua-tions on the Audiovisual Sub-Challenge of the 2011 Au-dio/Visual Emotion Challenge show how accurately dif-ferent affective dimensions can be recognized. Our ex-periments reveal that the proposed multimodal recognitionsystem outperforms previously introduced techniques eval-uated on the same task. 1 Introduction Since multimodal human-machine communication systemsthat not only take into account what the user says but alsohow the user communicates, are usually perceived as morenatural and more enjoyable to use, automatic emotion recog-nition is attracting more and more attention. Typically,audio and video channels can provide complementary in-formation and tend to improve recognition performance ifthey are used in a combined multimodal setup [1]. The ac-curacy of automatic emotion recognition heavily dependson the considered scenario: Acted, prototypical emotionsrecorded in a laboratory environment typically lead to highrecognition rates. Yet, ‘second generation’ AER systemsthat focus on realistic data have to face ambiguous non-prototypical emotions which lead to lower recognition ac-curacies. In this paper, we focus on the 2011 Audio/VisualEmotion Challenge (AVEC) [1] which was organized toprovide research teams with uniﬁed training, developmentand test data sets that can be used to compare individual ap-proaches applying a deﬁned test scenario and deﬁned per-formance measures. The task was to classify two levels ofarousal, expectation, power, and valence from audiovisualdata as contained in the SEMAINE database [2]. Com-pared to rather ‘friendly’ test conditions as considered inthe early days of emotion recognition research, this sce-nario is exceedingly challenging and typically leads to re-sults from below chance-level accuracies to around 70%accuracy for a two-class task.As recognizer back-end, we apply so-called Long Short-Term Memory (LSTM) networks which tend to be bestsuited for long-range context modeling in emotion recog-nition [3]. Unlike conventional recurrent neural network(RNN) architectures, LSTM is able to incorporate an arbi-trary, self-learned amount of (long-range) context into thedecoding process by using memory blocks instead of con-ventional hidden neurons (see [3], for example). The audiofeature extraction front-end applied in our study is basedon our open-source toolkit openSMILE [4] which is ableto extract large sets of prosodic, spectral, and voice qualitylow-level descriptors (LLD) combined with various statis-tical functionals in real-time. Linguistic features, includ-ing non-linguistic vocalizations such as laughing, breath-ing, and sighing are extracted with an antomatic speechrecognition (ASR) engine optimized for emotional speechrecognition. In addition, we employ visual features gener-ated by our facial movement feature extractor.The proposed technique is evaluated on both the de-velopment set and the ofﬁcial test set of the AudiovisualSub-Challenge. This allows us to compare our results withother methods proposed for this task, including context-free approaches such as Support Vector Machines (SVM).