A two dimensional annotation scheme for emotion in dialogue

An increased recognition of how an understanding of emotion could benefit dialogue systems means that corpora annotated for emotion would be a valuable resource. In this paper we describe an annotation scheme for expressions of emotion in dialogue using abstract dimensions, discuss how to evaluate such a scheme and use those techniques to assess the scheme’s reliability. Dialogue analysis and emotion To date dialogue annotation has been conducted almost exclusively on objective, measurable aspects of a dialogue’s content. This is partly understandable due to the difficulty associated with reliably identifying subjective phenomena. However, recently there has been an increased interest in how an understanding of some of the more subtle facets of communicative behaviour might benefit this field. Emotion is an example of the type of behaviour that it would be beneficial to incorporate into dialogue systems. For instance, artificial communicative agents which conduct dialogues either between themselves or with humans can be allowed to converse more naturally and be more engaging if they appear to act emotionally (Bates 1997; Piwek 2003). In order to gain an understanding of how emotion affects the way we communicate or how emotional expression is perceived by a listener, we could study corpora annotated for emotional content. In this paper we shall discuss the difficulties that hinder the development of annotation schemes for this purpose and describe a suitable scheme that overcomes these difficulties. Emotion in transcribed dialogue There has been considerable work studying the relationship between emotion and prosody, e.g. (Lee, Narayanan, & Pieraccini 2002b; Dellaert, Polzin, & Waibel 1996). Indeed much of the emotional expression in speech is conveyed in its acoustic realisation. However, emotion is also expressed in the language used by a speaker and it is equally important to understand how emotion affects the content of what we say. Knowing this would allow us to work with emotion in Copyright c © 2004, American Association for Artificial Intelligence (www.aaai.org). All rights reserved. cases where only the linguistic content was available or necessary and this understanding would also complement work on acoustic correlates. Since identifying emotion from transcribed speech can prove more difficult than from an audio recording, annotation schemes designed to be used with audio are likely to prove unsuitable for working from transcriptions. This is because these schemes ask annotators to make distinctions or apply labels for emotions which it is not possible to detect in text. Furthermore, a scheme which is shown to be reliable for annotators listening to the dialogue may not be when the vocal delivery is not available. The scheme that we describe here is designed to be applied to transcribed dialogue and is evaluated in that context. Previous studies of emotion in dialogue There have been a handful of studies which have developed annotation schemes for emotion in dialogue, including the following examples – • (Narayanan 2002; Lee, Narayanan, & Pieraccini 2002a) This scheme was applied to call centre dialogues and included two labels, NegativeandPositive. Given the simplicity of the scheme and the presence of audio during the annotation, the reliability of the scheme was low (Kappa = 0.465), which reflects the difficulty of the task. • (Devillers, Vasilescu, & Lamel 2002) Another study using call centre dialogues, this time the scheme includes labels for specific emotions; Fear, Anger, ExcuseandSatisfaction. The choice of labels was tailored to their domain although no evaluation of the reliability of this scheme was conducted. • (Litman, Forbes, & Silliman 2003; Litman & Forbes 2003) This scheme was applied to transcribed tutoring dialogues between humans. This scheme included labels for NegativeandPositiveutterances and was proven to be reliable (Kappa = 0.624). Later we shall discuss how these approaches can be improved upon in order to develop a scheme that is suited to capturing the complexities of emotion while remaining reliable. Designing an annotation scheme Before we describe the design of our annotation scheme it can be useful to consider how annotation schemes are designed in general. Here we describe a number of design decisions that must be made and highlight any differences between emotion and more frequently annotated phenomena that will affect the decisions we make. Dialogue Units The are a number of unit types into which a dialogue can be segmented and to which annotation can be applied. Since many annotation schemes are motivated by some theory about language or communication the choice of unit is often implicit or obvious. This is not the case for emotion. Emotional episodes exist over an indistinct period of time, fading in and out and subtly changing throughout a dialogue. Descriptive MechanismsThere are a number of ways with which to describe aspects of communication (e.g. descriptive labels, numeric values). Again, the appropriate way of describing phenomena used in dialogue annotation is often obvious and is most commonly descriptive categorical labels. For emotion this would be manifest as a collection of labels such as ‘Anger’, ‘Fear’, ‘Sadness’etc. For many purposes this is an inadequate way of describing emotion and subsequently a number of different approaches have been proposed (Cowie & Cornelius 2003). Annotation Scheme For the chosen descriptive mechanism it remains to decide upon the details of how the annotation is to be conducted. For categorical labels this involves producing a taxonomy of categories based on the required level of complexity, the necessary distinctions and the range of phenomena it is desirable to identify. Similar decisions need to be made for other descriptive mechanisms. Developing an annotation scheme for emotion