Speech Emotion Recognition – Measuring Driver States from Voice Characteristics

Detecting emotions and driver states in a broad sense (e.g. annoyance, anger, and fatigue) can help to enhance acceptance and comfort of these systems themselves as well as accident related safety. Adapting the system output, e.g., in speech interfaces of Driver Assistance Systems to the actual emotional and energetic driver states might enhance the acceptance of this empathic, emotional-intelligent communication due to its improved naturalism. In addition, comfort and comprehensiveness might be improved if the system’s output is adapted to the driver's actual e.g. stress-impaired attentional and cognitive resources. Furthermore, the detection of distracted driver states (e.g. pain, anger, emotional intensive conversation) could be beneficial from the viewpoint of safety concerns. Using voice communication as an indicator of emotion would have the following advantages: obtaining speech data is non-obtrusive, free from sensor application, calibration efforts, and robust against climatic environmental conditions. Several validation studies analysing e.g. basic emotions, fatigue, stress are presented in this paper reaching recognition rates for 2-class problems of about 80-90 %. Finally, current limitations and future demands (as e.g. freely accessible speech databases) for an efficient progress in speech emotions recognition are discussed. Measuring Driver States Many efforts have been reported in the literature for automatic measurement of emotional states. These systems mainly focus on (a) electropysiological data (e.g. EEG: Golz, Sommer, Holzbrecher & Schnupp, 2007; Hönig, Batliner & Nöth, 2007), and (b) behavioural expression data (gross body movement, head movement, mannerism, and facial expression; Horlings, Datcu & Rothkrantz, 2008) in order to characterize the user state. But these electrode(EOG/EEG reaching 15% error rate for fatigue detection; Sommer & Golz, 2005) or videobased instruments still do not fulfil the demands of an everyday life measurement system. The major drawbacks are (a) a lack of robustness against environmental and individual-specific variations (e.g. bright light, wearing correction glasses, angle of face or being of Asian race) and (b) a lack of comfort and longevity due to electrode sensor application. In contrast to these electrodeor video-based instruments, the utilization of voice communication as an indicator for emotional states could match the demands of everyday life measurement. Contact free measurements as voice analysis are non-obtrusive (not interfering with the primary driving task) and favourable for emotion detection, since an application of sensors would cause annoyance, additional stress and often impairs working capabilities and mobility demands. In addition, speech is easy to record even under extreme environmental conditions (bright light, high humidity and temperature), requires merely cheap, durable, and maintenance free sensors and most importantly, it utilizes already existing communication system hardware. Furthermore, speech data is omnipresent in many professional driver settings. Given these obvious advantages, the renewed interest in computational demanding analyses of vocal expressions has been enabled just recently by the advances in computer processing speed (Batliner, Steidl, Schuller et al, 2006; Juslin & Scherer, 2005; Owren & Bachorowski, 2007; Scherer, Johnstone & Klasmeyer, 2003; Schuller, Batliner, Seppi et al., 2007). Speech Emotion Recognition – Measuring Driver States from Voice Characteristics 99 Acoustic Features The acoustic speech emotion recognition research is mainly based on phonetics, general signal processing and computational intelligence research (e.g. Batliner, Steidl, Schuller et al., 2006). How emotions are expressed in the voice can be analyzed acoustically by measuring the characteristics of the speech wave form radiating from nostrils and mouth. Accordingly, acoustic features can be divided referring to auditive-perceptual concepts into prosody (pitch, intensity, rhythm, pause pattern, and speech rate), articulation (slurred speech, reduction and elision phenomena), and speech quality (timbre: breathy, whispery, tense, sharp, hoarse, or modal voice). A popular approach prefers the fusion of purely signal processing based features without any known auditive-perceptual correlates and perceptual-acoustic features as e.g. the speech intensity (loudness), fundamental frequency (pitch), voiced/unvoiced duration (rhythm) and several voice quality measures. Suitable measures for the voice quality category whisperiness are level of the first harmonic, bandwidth of the first formant, large amount of interharmonic noise, and steep spectral tilt. The spectrum of tense voices is flatter than that of lax voices. Creaky voice is characterized by a relatively flat spectrum, i.e. relatively strong upper harmonics. Finally, harsh and rough voices are characterized by a large amount of jitter, shimmer, and interharmonic noise. Typical acoustic features which implicitly cover the categories mentioned above and are used frequently in Speech Emotion Recognition (SER) (see Table 1) are (a) fundamental frequency, (b) intensity, (c) duration of voiced/unvoiced speech segments, (d) harmonics-to-noise ratio (HNR), (e) formant positions (F1-F7), (f) formant bandwidths (Fbw1-Fbw7), (g) mel frequency cepstrum coefficients (MFCCs), (h) linear frequency cepstrum coefficients (LFCCs), (i) linear predictive coding coefficients, and (j) spectral features derived from the long term average spectrum (LTAS). Table 1. Basic acoustic feature contours (frame-level descriptors). Frame level based feature Description Fundamental frequency (F0) acoustic equivalent to pitch; rate of vocal fold vibration; maximum of the autocorrelation function; models prosodic structure; speech melody indicator Energy models intensity, based on the amplitude in different intervals; average squared amplitude within a predefined time segment; stressing structure Harmonics-to-noise Ratio (HNR) spectral energy in voiced vs. unvoiced segments; ratio between harmonic and aperiodic signal energy; breathiness indicator Formant position (F1-F7) resonance frequencies of the vocal tract (VT) depending strongly on its actual shape; represent spectral maxima, and are known to model spoken content and speaker characteristics; influenced by lower jaw angle, tongue body angle, tongue body horizontal location, tongue tip angle, tongue tip horizontal location, relative lip height, lip protrusion, velum height Formants bandwidth (Fbw1-Fbw7) model VT shape and energy loss of speech signal due to VT elasticity (yielding wall effect), viscoeleasticity of VT tissue or heat conduction induced changes of air flow (jet streams, turbulences); width of the spectral band containing significant formant energy (3 dB threshold) Duration of voiced-unvoiced segments models temporal speech rhythm aspects as speech rate and pause structure Mel frequency cepstrum coefficients (MFCCs) “spectrum of the spectrum”; have been proven beneficial in speech emotion recognition, and speech recognition tasks; homomorphic transform with equidistant band-pass-filters on the Mel-scale; Holistic and decorrelated representation of spectrum Linear frequency cepstrum coefficients (LFCCs) similar to MFCC but without the perceptually oriented transformation into the mel frequency scale; emphasize changes or periodicity in the spectrum, while being relatively robust against noise Jarek Krajewski, Elmar Nöth und Anton Batliner 100 Linear Predictive Coding (LPC) Provides an accurate and economical representation of the envelope of the short-time power spectrum. One of the most powerful speech coding analysis techniques providing very accurate estimates of speech parameters and is known as being relatively efficient for computation at the same time Long Term Average Spectrum (LTAS) averages out formant information; giving general spectral trends; relative amount of energy within predefined frequency bands; speech quality The speaker’s emotional state expresses itself through prosodic-phonetic information as pitch, intensity, speaking rate, duration, articulation and voice quality (see Figure 2). Furthermore, this prosody related information conveyed by speech can be enriched by paralinguistic information, which contains short, emotional non-speech expressions, comprising both clear non-speech sounds and interjections with a phonemic structure (cf. affective bursts, Schroeder, 2001). Laughing, sniffs, sobs, or morphing speech into a cry (“noooo .”), swearing, yawning, interjections (e.g. "wow! "), deep breathing as sound of relief are typical examples of these paralinguistic events, which might be used for speech emotion recognition (Truong & van Leeuwen, 2007). Furthermore, emotional information is carried in the human speech over the information channel of syntax and semantics (lexical content). Syntactic channels include information as part-of-speech categories (POS; e.g. number of nouns, non-inflected adjectives), other syntax-related information (n-grams, bag of words, elaborated vs. simple construction, many vs. few word per sentence; many vs. few negations; 1 person singular use, tentative words, present vs. past tense), Linguistic Inquiry and Word Count (LIWC; e.g. categories as anger words, inclusive words, social processes), and MRC psycholinguistic features (concreteness of words, imageability; cf. Mairesse, Walker, Mehl & Moore, 2007). Moreover, the analysis of semantic channels might include: topic selection (e.g. self-focused, pleasure vs. problem talk, single vs. multiple topics, strict selection vs. think out loud ; bag of words features), and conversational behaviour characteristics (listen vs. initiate conversation, back-channel behaviour, formal style). The major drawbacks of SER using syntactic or semantic information are the difficulties related to an automatic identification of these categories. In addition, a word tagger