Measuring emotion : exploring the feasibility of automatically classifying emotional text

This thesis explores methods for the automatic supervised categorization of text according to the emotions of its author. Since “emotions” are by definition subjective, we predict that this task will present a number of significant challenges. We know that there is low inter-annotator agreement on gold-standard data, since humans find it hard to identify emotions in text, which results in datasets that are noisy and difficult to learn from. In addition to (and maybe partially as a consequence of) this annotation difficulty, existing emotion datasets are relatively small. This makes the task even more challenging, as there is very little data from which a classification algorithm can infer a model. To explore the possibilities and limitations of automatic emotion classification, we develop two very different case studies in detail. In a first case study, we attempt to automatically classify sentences from business conversations according to a dimensional emotion model called Leary’s Rose, or the Interpersonal Circumplex. We discretize this model into eight emotion categories, and attempt to find the best features to capture the problem using a Support Vector Machines-based classification system. Considered feature types include word tokens and lemmas, character n-grams, part of speech tags, lexicon entries from two different subjectivity lexicons, and contextual information regarding the previous positions of speaker and hearer on Leary’s Rose. We determine that character n-grams are good at capturing relevant information, and that they are well supplemented with carefully selected emotion keywords. In theory, a speaker’s previous emotional state makes for an excellent predictor of their current position on the Circumplex, but in practice, we find that performance is too low for contextual class information to be useful. We compare the classifiers’ performance to the inter-annotator agreement on the dataset, and conclude that the ambiguity is also present in the gold standard data, as even human agreement is very low. The second case study concerns the automatic identification of emotions in suicide notes. We again classify on a sentence by sentence basis, but the class system is resolutely discrete (meaning there is no explicit dimensional relationship between the considered emotions, as there was on the Interpersonal Circumplex). It is also possible for one sentence to contain more than one emotion. To solve this multi-label classification problem, we use a SVM-based system which attributes a class label if its probability exceeds an experimentally determined threshold. We experiment with token and character n-grams, and again conclude that character n-grams capture relevant features well. The results of the case study also show very clearly that the number of training instances for a class has a very large influence on the classifier’s performance on that class. Sparse classes are extremely difficult to predict well. This is hardly a surprising finding, but given the limited size of existing emotion datasets, sparse classes are a problem that needs to be tackled.