Venue: NAACL 2018

# Letting Emotions Flow: Success Prediction by Modeling the Flow of

Emotions in Books

Suraj Maharjan\({}^{*}\) Sudipta Kar\({}^{*}\) Manuel Montes-y-Gomez\({}^{\dagger}\)

Fabio A. Gonzalez\({}^{\ddagger}\) Thamar Solorio\({}^{*}\)

###### Abstract

Books have the power to make us feel happiness, sadness, pain, surprise, or sorrow. An author's dexterity in the use of these emotions captivates readers and makes it difficult for them to put the book down. In this paper, we model the flow of emotions over a book using recurrent neural networks and quantify its usefulness in predicting success in books. We obtained the best weighted F1-score of 69% for predicting books' success in a multitask setting (simultaneously predicting success and genre of books).

+
Footnote †: The source code and data for this paper can be downloaded from [https://github.com/sjmaharjan/emotion_flow](https://github.com/sjmaharjan/emotion_flow)

+
Footnote †: The source code and data for this paper can be downloaded from [https://github.com/sjmaharjan/emotion_flow](https://github.com/sjmaharjan/emotion_flow)

## 1 Introduction

Books have the power to evoke a multitude of emotions in their readers. They can make readers laugh at a comic scene, cry at a tragic scene and even feel pty or hate for the characters. Specific patterns of emotion flow within books can compel the reader to finish the book, and possibly pursue similar books in the future. Like a musical arrangement, the right emotional rhythm can arouse readers, but even a slight variation in the composition might turn them away.

Vonnegut (1981) discussed the potential of plotting emotions in stories on the "Beginning-End" and the "Ill Fortune-Great Fortune" axes. Reagan et al. (2016) used mathematical tools like Singular Value Decomposition, agglomerative clustering, and Self Organizing Maps (Kohonen et al., 2001) to generate basic shapes of stories. They found that stories are dominated by six different shapes. They even correlated these different shapes to the success of books. Mohammad (2011) visualized emotion densities across books of different genres. He found that the progression of emotions varies with the genre. For example, there is a stronger progression into darkness in horror stories than in comedy. Likewise, Kar et al. (2018) showed that movies having similar flow of emotions across the book: _Alice in Wonderland_. The plot shows continuous change in _trust_, _fear_, and _sadness_, which relates to the main character's getting into and out of trouble. These patterns present the emotional arcs of the story. Even though they do not reveal the actual plot, they indicate major events happening in the story.

In this paper, we hypothesize that readers enjoy emotional rhythm and thus modeling emotion flows will help predicting a book's potential success. In addition, we show that using the entire content of the book yields better results. Considering only a fragment, as done in earlier work that focuses mainly on style (Maharjan et al., 2017; Ashok et al., 2013), disregards important emotional changes. Similar to Maharjan et al. (2017), we also find that adding genre as an auxiliary task improves success prediction.

## 2 Methodology

We extract emotion vectors from different chunks of a book and feed them to a recurrent

Figure 1: Flow of emotions in _Alice in Wonderland_.



neural network (RNN) to model the sequential flow of emotions. We aggregate the encoded sequences into a single book vector using an attention mechanism. Attention models have been successfully used in various Natural Language Processing tasks Wang et al. (2016); Yang et al. (2016); Hermann et al. (2015); Chen et al. (2016); Rush et al. (2015); Luong et al. (2015). This final vector, which is emotionally aware, is used for success prediction.

**Representation of Emotions**: NRC Emotion Lexicons provide \(\sim\)14K words (Version 0.92) and their binary associations with eight types of elementary emotions (_anger_, _anticipation_, _joy_, _trust_, _disgust_, _sadness_, _surprise_, and _fear_) from the Hourglass of emotions model with polarity (_positive_ and _negative_) Mohammad and Turney (2013); Mohammad and Turney (2010). These lexicons have been shown to be effective in tracking emotions in literary texts Mohammad (2011).

**Inputs**: Let \(X\) be a collection of books, where each book \(x\in X\) is represented by a sequence of \(n\) chunk emotion vectors, \(x=(x_{1},x_{2},...,x_{n})\), where \(x_{i}\) is the aggregated emotion vector for chunk \(i\), as shown in Figure 2. We divide the book into \(n\) different chunks based on the number of sentences. We then create an emotion vector for each sentence by counting the presence of words of the sentence in each of the ten different types of emotions of the NRC Emotion Lexicons. Thus, the sentence emotion vector has a dimension of 10. Finally, we aggregate these sentence emotion vectors into a chunk emotion vector by taking the average and standard deviation of sentence vectors in the chunk. Mathematically, the \(i\)th chunk emotion vector (\(x_{i}\)) is defined as follows:

\[x_{i}=\left[\frac{\sum^{N}_{j=1}s_{ij}}{N};\sqrt{\frac{\sum^{N}_{ j=1}\left(s_{ij}-\bar{s_{i}}\right)^{2}}{N}}\right] \tag{1}\]

where, \(N\) is the total number of sentences, \(s_{ij}\) and \(\bar{s_{i}}\) are the \(j\)th sentence emotion vector and the mean of the sentence emotion vectors for the \(i\)th chunk, respectively. The chunk vectors have a dimension of 20 each. The motivation behind using the standard deviation as a feature is to capture the dispersion of emotions within a chunk.

**Model**: We then use bidirectional Gated Recurrent Units (GRUs) Bahdanau et al. (2014) to summarize the contextual emotion flow information from both directions. The forward and backward GRUs will read the sequence from \(x_{1}\) to \(x_{n}\), and from \(x_{n}\) to \(x_{1}\), respectively. These operations will compute the forward hidden states \((\overrightarrow{h_{1}},\ldots,\overrightarrow{h_{n}})\) and backward hidden states \((\overleftarrow{h_{1}},\ldots,\overrightarrow{h_{n}})\). The annotation for each chunk \(x_{i}\) is obtained by concatenating its forward hidden states \(\overrightarrow{h_{i}}\) and its backward hidden states \(\overrightarrow{h_{i}}\), i.e. \(h_{i}\)=\(|\overrightarrow{h_{i}};\overrightarrow{h_{i}}|\). We then learn the relative importance of these hidden states for the classification task. We combine them by taking the weighted sum of all \(h_{i}\) and represent the final book vector \(r\) using the following equation:

\[r =\sum_{i}\alpha_{i}h_{i} \tag{2}\] \[\alpha_{i} =\frac{\exp(score(h_{i}))}{\sum_{i^{\prime}}\exp(score(h_{i^{ \prime}}))}\] (3) \[score(h_{i}) =v^{T}selu(W_{a}h_{i}+b_{a}) \tag{4}\]

where, \(\alpha_{i}\) are the weights, \(W_{a}\) is the weight matrix, \(b_{a}\) is the bias, \(v\) is the weight vector, and \(selu\)Klambauer et al. (2017) is the nonlinear activation function. Finally, we apply a linear transformation that maps the book vector \(r\) to the number of classes. In case of the single task (ST) setting, where we only predict success, we apply sigmoid activation to get the final prediction probabilities and compute errors using binary cross entropy loss. Similarly, in the multitask

Figure 2: Multitask Emotion Flow Model.

 

(MT) setting, where we predict both success and genre Maharjan et al. (2017), we apply a softmax activation to get the final prediction probabilities for the genre prediction. Here, we add the losses from both tasks, i.e. \(L_{total}\)=\(L_{suc}\) + \(L_{gen}\) (\(L_{suc}\) and \(L_{gen}\) are success and genre tasks' losses, respectively), and then train the network using backpropagation.

## 3 Experiments

### Dataset

We experimented with the dataset introduced by Maharjan et al. (2017). The dataset consists of 1,003 books from eight different genres collected from Project Gutenberg1. The authors considered only those books that were at least reviewed by ten reviewers. They categorized these books into two classes, _Successful_ (654 books) and _Unsuccessful_ (349 books), based on the average rating for the books in Goodreads2 website. They considered only the first 1K sentences from each book.

Footnote 1: [https://www.gutenberg.org/](https://www.gutenberg.org/)

Footnote 2: [https://www.goodreads.com/](https://www.goodreads.com/)

### Baselines

We compare our proposed methods with the following baselines:

**Majority Class**: The majority class in training data is _success_. This baseline obtains a weighted F1-score of 0.506 for all the test instances.

**SentiWordNet+SVM**: Maharjan et al. (2017) used SentiWordNet Baccianella et al. (2010) to compute the sentiment features along with counts of different Part of Speech (PoS) tags for every 50 consecutive sentences (20 chunks from 1K sentences) and used an SVM classifier.

**NRC+SVM**: We concatenate the chunk emotion vectors (\(x_{i}\)) created using the NRC lexicons and feed them to the SVM classifier. We experiment by varying the number of book chunks.

These baseline methods do not incorporate the sequential flow of emotions across the book and treat each feature independently of each other.

### Experimental Setup

We experimented with the same random stratified splits of 70:30 training to test ratio as used by Maharjan et al. (2017). We use the SVM algorithm for the baselines and RNN for our proposed emotion flow method. We tuned the \(C\) hyperparameter of the SVM classifier by performing grid search on the values (1e{-4,...,4}), using three fold cross validation on the training split. For the experiments with RNNs, we first took a random stratified split of 20% from the training data as validation set. We then tuned the RNN hyperparameters by running 20 different experiments with a random selection of different values for the hyperparameters. We tuned the weight initialization (Glorot Uniform Glorot and Bengio (2010), LeCun Uniform LeCun et al. (1998)), learning rate with Adam Kingma and Ba (2015) {1e-4,...,1e-1}, dropout rates {0.2,0.4,0.5}, attention and recurrent units {32, 64}, and batch-size {1, 4, 8} with early stopping criteria.

## 4 Results

Table 1 presents the results. Our proposed method performs better than different baseline methods and obtains the highest weighted F1-score of 0.690. The results highlight the importance of taking into account the sequential flow of emotions across books to predict how much readers will like a book. We obtain better performance when we use an RNN to feed the sequences of emotion chunk vectors. The performance decreases with the SVM classifier, which discards this sequential information by treating each feature independently of each other. Moreover, increasing the granularity of the emotions by increasing the number of chunks seems to be helpful for success prediction. However, we see a slight decrease in performance beyond 50 chunks (weighted F1 score of 0.662 and 0.664 for 60 and 100 chunks, respectively).

The results also show that the MT setting is beneficial over the ST setting, whether we consider

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|} \hline
**Book Content** & \multicolumn{4}{c|}{**1000 sents**} & \multicolumn{2}{c|}{**All**} \\ \cline{2-6}
* & 0.506 & 0.506 & 0.506 & 0.506 \\ SentWordNet + SVM & 20 & 0.582 & 0.610 &
- \\ \hline \hline NRC + SVM & 10 & 0.526 & 0.597 & 0.541 & 0.641 \\ NRC + SVM & 20 & 0.537 & 0.590 & 0.577 & 0.604 \\ NRC + SVM & 30 & 0.587 & 0.576 & 0.595 & 0.600 \\ NRC + SVM & 50 & 0.611 & 0.586 & 0.597 & 0.636 \\ \hline \hline Emotion Flow & 10 & 0.632 & 0.643 & 0.650 & 0.660 \\ Emotion Flow & 20 & 0.612 & 0.639 & 0.640 & 0.668 \\ Emotion Flow & 30 & 0.630 & 0.657 & 0.662 & 0.677 \\ Emotion Flow & 50 & 0.656 & 0.666 & 0.674 & **0.690*** \\ \hline \end{tabular}
\end{table}
Table 1: Weighted F1-scores for success classification in single task (ST) and multi task (MT) settings with varying chunk sequences when using all the book or only the first 1K sentences. *\(p<0.05\) (McNemar significance test between Emotion Flow (chunks 50, MT, All) and NRC+SVM (chunk 10, MT, All))
 

the first 1K sentences or the entire book. This finding is akin to Maharjan et al. (2017). Similar to them, we suspect the auxiliary task of genre classification is acting as a regularizer.

Considering only the first 1K sentences of books may miss out important details, especially when the only input to the model is the distribution of emotions. It is necessary to include information from later chapters and climax of the story as they gradually reveal the answers to the suspense, the events, and the emotional ups and downs in characters that build up through the course of the book. Accordingly, our results show that it is important to consider emotions from the entire book rather than from just the first 1K sentences.

## 5 Attention Analysis

From Figure 3, we see that using the attention mechanism to aggregate vectors is better than just concatenating the final forward and backward hidden states to represent the book in both ST and MT settings. We also observe that the multitask approach performs better than the singe task one regardless of the number of chunks and the use of attention.

Figure 4 plots the heatmap of the average attention weights for test books grouped by their genre. The model has learned that the last two to three chunks that represent the climax, are most important for predicting success. Since this is a bidirectional RNN model, hidden representations for each chunk carry information from the whole book. Thus, using only the last chunks will probably result in lower performance. Also, the weights visualization shows an interesting pattern for _Poetry_. In _Poetry_, emotions are distributed across the different regions. This may be due to sudden important events or abrupt change in emotions. For _Short stories_, initial chunks also receive some weights, suggesting the importance of the premise.

## 6 Emotion Analysis

**Climax Emotions**: Since the last chunk is assigned more weights than other chunks, we used information gain to rank features of that chunk. From Figure 5, we see that features capturing the variation of different emotions are ranked higher than features capturing the average scores. This suggests that readers tend to enjoy emotional ups and downs portrayed in books, making the standard deviation features more important than the average features for the same emotions.

Table 2 shows the mean (\(\mu\)) and standard deviation (\(\sigma\)) for different emotions extracted for all the data, and further categorized by _Successful_ and _Unsuccessful_ label from the last chunk. We see that authors generally end books with higher rates of positive words (\(\mu=0.888\)) than negative words (\(\mu=0.599\)) and the difference is significant (\(p<0.001\)). Similarly the means for _anticipation_, _joy_, _trust_, and _fear_ are higher than for _sadness_, _surprise_, _anger_, and _disgust_. This further vali

Figure 4: Attention weights visualization per genre.

Figure 5: Feature ranking with information gain.

Figure 3: Comparison of the Emotion Flow with and without attention mechanism for different chunk sequences.

 

dates that authors prefer happy ending. Moving on to _Successful_ and _Unsuccessful_ categories, we see that the means for _Successful_ books are higher than _Unsuccessful_ books for _anger_, _anticipation_, _disgust_, _fear_, _joy_, and _sadness_ (highly significant, \(p<0.001\)). We observe the same pattern for _trust_, and _surprise_, although the \(p\) value is only \(p<0.02\) in this case. Moreover, the standard deviations for all emotions are significantly different across the two categories (\(p<0.001\)). Thus, emotion concentration (\(\mu\)) and variation (\(\sigma\)) for _Successful_ books are higher than for _Unsuccessful_ books for all emotions in the NRC lexicon.

**Emotion Shapes**: We visualize the prominent emotion flow shapes in the dataset using K-means clustering algorithm. We took the average _joy_ across 50 chunks for all books and clustered them into 100 different clusters. We then plotted the smoothed centroid of clusters having \(\geq\) 20 books. We found two distinct shapes ( "Man in the hole" (fall to rise) and "Tragedy" or "Riches to rags" (fall)). Figure 6 shows such centroid plots. The plot also shows that the "Tragedy" shapes have an overall lower value of joy than the "Man in the hole" shapes. Upon analyzing the distribution of _Successful_ and _Unsuccessful_ books within these shapes, we found that the "Man in the hole" shapes have a higher number of successful books whereas, the "Tragedy" shapes have the opposite.

## 7 Conclusions and Future Work

In this paper, we showed that modeling emotions as a flow, by capturing the emotional content at different stages, improves prediction accuracy. We learned that most of the attention weights are given to the last fragment in all genres, except for _Poetry_ where other fragments seem to be relevant as well. We also showed empirically that adding an attention mechanism is better than just considering the last forward and backward hidden states from the RNN. We found two distinct emotion flow shapes and found that the clusters with "Tragedy" shape had more unsuccessful books than successful ones. In future work, we will be exploring how we can use these flows of emotions to detect important events that result in suspenseful scenes. Also, we will be applying hierarchical methods that take in the logical grouping of books (sequence of paragraphs to form a chapter and sequence of chapters to form a book) to build books' emotional representations.

## Acknowledgments

We would like to thank the National Science Foundation for funding this work under award 1462141. We are also grateful to the anonymous reviewers for reviewing the paper and providing helpful comments and suggestions.

## References

* Ashok et al. (2013) Vikas Ashok, Song Feng, and Yejin Choi. 2013. Success with style: Using writing style to predict the success of novels. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Seattle, Washington, USA, pages 1753-1764. [http://www.aclweb.org/anthology/Dl3-1181](http://www.aclweb.org/anthology/Dl3-1181).
* Baccianella et al. (2010) Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010. Sentiwordnet 3.0: An enhanced lexical

Figure 6: Emotion flow for four cluster centroids in the dataset. The two curves on top match the “Fall to Rise” shape and the two at the bottom match the “Tragedy” one defined in Reagan et al. (2016).

 

resource for sentiment analysis and opinion mining. In _LREC_. volume 10, pages 2200-2204.
* Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. _CoRR_ abs/1409.0473. [http://arxiv.org/abs/1409.0473](http://arxiv.org/abs/1409.0473).
* Chen et al. (2016) Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, pages 2358-2367. [http://www.aclweb.org/anthology/P16-1223](http://www.aclweb.org/anthology/P16-1223).
* Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington, editors, _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_. PMLR, Chia Laguna Resort, Sardinia, Italy, volume 9 of _Proceedings of Machine Learning Research_, pages 249-256. [http://proceedings.mlr.press/v9/glorot10a.html](http://proceedings.mlr.press/v9/glorot10a.html).
* Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, _Advances in Neural Information Processing Systems 28_, Curran Associates, Inc., pages 1693-1701. [http://papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend.pdf](http://papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend.pdf).
* Kar et al. (2018) Sudipta Kar, Suraj Maharjan, A. Pastor Lopez-Monroy, and Thamar Solorio. 2018. MPST: A corpus of movie plot synopses with tags. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_. European Language Resources Association (ELRA), Paris, France.
* Kingma and Ba (2015) Diederik Kingma and Jimmy Ba. 2015. Adam: a method for stochastic optimization. In _International Conference on Learning Representations_.
* Klambauer et al. (2017) Gunter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. 2017. Self-normalizing neural networks. _CoRR_ abs/1706.02515. [http://arxiv.org/abs/1706.02515](http://arxiv.org/abs/1706.02515).
* Kohonen et al. (2001) T. Kohonen, M. R. Schroeder, and T. S. Huang, editors. 2001. _Self-Organizing Maps_. Springer-Verlag New York, Inc., Secaucus, NJ, USA, 3rd edition.
* LeCun et al. (1998) Yann LeCun, Leon Bottou, Genevieve B. Orr, and Klaus-Robert Muller. 1998. Efficient backprop. In _Neural Networks: Tricks of the Trade, This Book is an Outgrowth of a 1996 NIPS Workshop_. Springer-Verlag, London, UK, UK, pages 9-50. [http://dl.acm.org/citation.cfm?id=645754.668382](http://dl.acm.org/citation.cfm?id=645754.668382).
* Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Lisbon, Portugal, pages 1412-1421. [http://aclweb.org/anthology/D15-1166](http://aclweb.org/anthology/D15-1166).
* Maharjan et al. (2017) Suraj Maharjan, John Arevalo, Manuel Montes, Fabio A. Gonzalez, and Thamar Solorio. 2017. A multi-task approach to predict likability of books. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_. Association for Computational Linguistics, Valencia, Spain, pages 1217-1227. [http://www.aclweb.org/anthology/E17-1114](http://www.aclweb.org/anthology/E17-1114).
* Mohammad (2011) Saif Mohammad. 2011. From once upon a time to happily ever after: Tracking emotions in novels and fairy tales. In _Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities_. Association for Computational Linguistics, Stroudsburg, PA, USA, LaTeCH '11, pages 105-114. [http://dl.acm.org/citation.cfm?id=2107636.2107650](http://dl.acm.org/citation.cfm?id=2107636.2107650).
* Mohammad and Turney (2010) Saif Mohammad and Peter Turney. 2010. Emotions evoked by common words and phrases: Using mechanical turk to create an emotion lexicon. In _Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text_. Association for Computational Linguistics, Los Angeles, CA, pages 26-34. [http://www.aclweb.org/anthology/W10-0204](http://www.aclweb.org/anthology/W10-0204).
* Mohammad and Turney (2013) Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a word-emotion association lexicon. _Computational Intelligence_ 29(3):436-465. [https://doi.org/10.1111/j.1467-8640.2012.00460.x](https://doi.org/10.1111/j.1467-8640.2012.00460.x).
* Reagan et al. (2016) Andrew J. Reagan, Lewis Mitchell, Dilan Kelley, Christopher M. Danforth, and Peter Sheridan Dodds. 2016. The emotional arcs of stories are dominated by six basic shapes. _CoRR_ abs/1606.07772. [http://arxiv.org/abs/1606.07772](http://arxiv.org/abs/1606.07772).
* Rush et al. (2015) Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Lisbon, Portugal, pages 379-389. [http://aclweb.org/anthology/D15-1044](http://aclweb.org/anthology/D15-1044).
* Vonnegut (1981) Kurt Vonnegut. 1981. Palm sunday: An autobiographical collage 

Yequan Wang, Minlie Huang, xiaoyan zhu, and Li Zhao. 2016 Attention-based lstm for aspect-level sentiment classification. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Austin, Texas, pages 606-615. [https://aclweb.org/anthology/D16-1058](https://aclweb.org/anthology/D16-1058).
* Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016 Hierarchical attention networks for document classification. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, San Diego, California, pages 1480-1489. [http://www.aclweb.org/anthology/N16-1174](http://www.aclweb.org/anthology/N16-1174).