Application of Sentiment Analysis to Labeling Characters as Good or Evil

Sentiment analysis models were trained with the objective of transferring this learning onto determining if characters in books are good or evil. Typically, sentiment analysis aims to test on a set-apart subset of the training set, however, with a small, hand-labeled training set, this approach was infeasible. Instead, we employed common models to learn sentiment on the training database, then developed code to label the sentiment of characters in books based on the sentiment of sentences in which they appeared. We screened models and datasets to determine which model/training dataset combination generalized most accurately to the other sets, then applied the resulting models to characters in books. In spite of trying several different models and character-labeling methods, none produced accuracies above 0.50. Introduction Have you ever found yourself wondering whether Sauron or Pippin was the real villain of Lord of the Rings? As children, we were taught that everything could be divided into two categories: good and evil. We intend to use sentiment analysis to analyze characters from children’s books on how good or evil they are. Generally, when designing a supervised classifier it is best to train the model on the same type of data as would be used for testing and predictions. However, there are no available datasets that have characters from books labeled as good or evil. This means that we have a very limited dataset of labeled characters. So instead we focus on creating a model that can be trained with available labeled datasets and transfer the model onto classifying characters. It is important to note that a key of this project is to not simply create the best sentiment analysis for a specific dataset, but to train a model in a way that could give us more transferability. We must use a sentiment database such as Amazon Book reviews, Twitter, or IMDB reviews to train a model to predict binary sentiment (0,1). Then we must find a simple way to attribute text to a character, take these texts, label them with a sentiment using our previous model and average the score to rate characters as good or evil. Figure 1. General pipeline to describe the process of our project. We first pick a dataset to train with that produces the most general models. We then compare how different embeddings perform on sentiment analysis tasks. Then we compare different classification models and their transferability. Lastly we split books into sentences that we assign to characters and see how our model performs. Related Work Natural language processing has focused on the problem of sentiment analysis, leading to a plethora of literature using every imaginable featurization and classification algorithm [1]. More classical approaches, such as Naive Bayes [2] (NB) and Support Vector Machines [3] (SVM) using bag-of-words (and similar featurizations like TF and TFIDF) predominated until the mid-2010’s. These approaches have the dual advantages of simplicity and interpretability: the underlying mathematical models are straightforward (and therefore easy to implement), and the features and weights of the models can be interpreted as ranking the sentiment impact of words. However, the underlying assumptions of these models are crude: bag-of-words neglects word order, so NB and SVM cannot capture context. These limitations can be somewhat alleviated by including ngrams (word groups of length n) as features, but the accuracy improvement is marginal [4]. Increasing the number of features results in minimal training data for each feature (a phrase is less common than a word, which itself may only appear a few times in the training set), which can cause models to fail to generalize beyond the train set. Deep neural networks using word-vector embeddings have been popularized recently and can provide a significant improvement in accuracy over NB and SVM [5]. These networks can include context in several ways: through the use of convolutional layers, through context-dependent embeddings, and through the addition of recursion. The input into a convolutional layer is a matrix: one column for each word. Convolutional layers convolve across the columns so that the output of each node is dependent on every word. A typical convolutional neural network (CNN) employed with the GloVe word-vector embedding can improve accuracy by 20% over bag-of-words SVM, and by 5% over GloVe SVM on Twitter datasets [6]. This statistic reveals how CNN outperforms SVM, but also the effect of word-vector embeddings. These embeddings will be discussed in further detail in the next section. Embeddings such as ElMo use a recursive model to embed words, resulting in context-dependent vectors, in contrast to vector dictionary embeddings like word2vec. This difference can provide accuracy improvements of up to 5% [7]. Most recently, marginal improvements to accuracy have been obtained by adding a recursive layer to a CNN. This recursive layer can be added either before the convolution (RCNN) or after it (CRNN) [8], [9]. The ability of nodes to remember previous inputs explicitly incorporates context, increasing accuracy. State-of-the-art models use some combination of these methods and features to incorporate context, as well as additional layers, featurizations, and data processing tricks; there are too many to enumerate here (see [5] for an overview of some such methods). Dataset and Features Datasets: The training set for the sentiment analysis models was comprised of three publicly-available sentiment databases. Amazon Books ​[10]: This dataset contains 970k book reviews taken from Amazon.com. We deem a positive review to be a rating of 4 or 5 stars and a negative review 1 or 2 stars. Neutral reviews are not included in the dataset. There is a bias for positive reviews in the data set, where over 80% of the reviews are positive. IMDb ​[11]: This dataset contains movie reviews along with their associated binary sentiment polarity labels. It contains 50k reviews split evenly into 25k positive reviews and 25k negative reviews. No more than 30 reviews are allowed for any given movie because reviews for the same movie tend to have correlated ratings. IMDb reviews have scores 1-10. Scores <= 4 are considered negative and >=7 are considered positive. Neutral ratings are not included in the dataset. Twitter ​[12]: This dataset contains 1.6 million tweets with emoticons removed along with trinary sentiment labels (positive, negative, neutral). For the purposes of this study, neutral tweets are omitted. The labels were automatically generated using (among other things) the positive/negative nature of the emoticons which were subsequently stripped. These datasets had commas and periods removed, were converted to lowercase, and split by spaces. Exclamation marks and question marks were separated to be their own token. No treatment of misspelled words, non-ASCII characters, or slang was performed. Such tokens do not appear in the books which form the test set, and therefore should not affect the results. For the Amazon dataset, it was necessary to remove some HTML tags. Test sets were formed from each dataset by removing ~400 examples. For testing the character labeling, we created our own small dataset consisting of 6 books and 20 characters. We took the text of books of the public domain and manually labeled major characters in the book as good (1) or evil (0). It is also important to note that there are several names that a character can be referred to as, so our labels consisted of a score: 0 or 1 and a list of names that belong to that one character. The books we used are: “Ashputtel,” "The Phantom Tollbooth," "Beauty and the Beast," "The Giving Tree," "The Count of Monte Cristo," and "Hamlet. " Features and Embeddings: The Naive Bayes, logistic regression, and simple neural network all used the bag-of-words featurization of sentences. A dictionary of all words was created, and each sentence was converted to a vector indicating which words were in the sentence. Two different embeddings, GloVe [13] and ELMo [14], were used to embed words as vectors and convert sentences into matrices. Static embeddings like GloVe and word2vec are designed to incorporate word similarity by mapping words to vectors based on word co-occurence (the dot product of two words in GloVe is the log probability of co-occurence). Context-dependent embeddings, like ELMo, rely upon a deep, recursive, pretrained model. Sentences are fed through this model, which embeds words based on the internal states of the recursive neural network, leading to embedding values which are context-dependent. Additionally, ELMo is character-based, which allows for it to capture the meaning of out-of-dictionary tokens. Methods Sentiment Classification: For the sentiment classification we implemented four algorithms: 1) Naive Bayes: ​This model was chosen as a baseline because of its simplicity and low computational requirements. It assumes that the input data is conditionally independent given the labels. ​ ​We also used Laplace smoothing and only words that appeared at least five times across the dataset were considered. 2) Linear Neural Network (LNN): ​A linear neural network bridges the gap between logistic regression and deep neural networks. After a few iterations we ended up with a model consisting of five layers. The first layer has 1,000 neurons and the following layers all have 2,000 neurons. We used ReLu activation functions for all layers. For the output we used a sigmoid function. To control overfitting we used L2 regularization and dropout. 3) Convolutional Neural Network (CNN)​: A specific architecture of neural networks in which we use filters to avoid the overfitting problem of fully connected layers. This allows for the model to generalize learning. For example, if there is a phrase with a given meaning, it probably does not matter if the phrase is at the beginning of the text, in the middle or at the end. The filter