SlowGuess/ACL_2020_OCR huggingface dataset:
Importance We follow Ely et al. in evaluating weighted versions of their surprise and uncertainty reduction measure  $S_{t}^{\alpha \mathrm{Ely}}$  and  $U_{t}^{\alpha \mathrm{Ely}}$  (see Equation (5)). We obtain the  $\alpha_{t}$  values by taking the sentiment scores assigned by the VADER sentiment classifier (Hutto and Gilbert, 2014) to each sentence and multiplying them by 1.0 for positive sentiment and 2.0 for negative sentiment. The stronger negative weighting reflects the observation that negative consequences can be more important than positive ones (O'Neill, 2013; Kahneman and Tversky, 2013).
Baselines We test a number of baselines as alternatives to surprise and uncertainty reduction derived from our hierarchical model. These baselines also reflect how much change occurs from one sentence to the next in a story: WordOverlap is the Jaccard similarity between the two sentences,
GloveSim is the cosine similarity between the averaged Glove (Pennington et al., 2014) word embeddings of the two sentences, and GPTSim is the cosine similarity between the GPT embeddings of the two sentences. The  $\alpha$  baseline is the weighted VADER sentiment score.
6.1 Narrative Suspense
Task The annotator judgements are relative (amount of decrease/increase in suspense from sentence to sentence), but the model predictions are absolute values. We could convert the model predictions into discrete categories, but this would fail to capture the overall arc of the story. Instead, we convert the relative judgements into absolute suspense values, where  $J_{t} = j_{1} + \dots + j_{t}$  is the absolute value for sentence  $t$  and  $j_{1}, \ldots, j_{t}$  are the relative judgements for sentences 1 to  $t$ . We use -0.2 for Big Decrease, -0.1 for Decrease, 0 for Same, 0.1 for Increase, and 0.2 for Big Increase. Both the absolute suspense judgements and the model predictions are normalised by converting them to z-scores.
To compare model predictions and absolute suspense values, we use Spearman's  $\rho$  (Sen, 1968) and Kendall's  $\tau$  (Kendall, 1975). Rank correlation is preferred because we are interested in whether human annotators and models view the same part of the story as more or less suspenseful; also, rank correlation methods are good at detecting trends. We compute  $\rho$  and  $\tau$  between the model predictions and the judgements of each of the annotators (i.e., five times for five annotators), and then take the average. We then average these values again over the 100 stories in the test or development sets. As the human upper bound, we compute the mean pairwise correlation of the five annotators.
Results Figure 2 shows surprise and uncertainty reduction measures and human suspense judgements for an example story (text and further examples in Appendix C). We performed model selection using the correlations on the development set, which are given in Table 2. We experimented with all the measures introduced in Section 3.1, computing sets of alternative sentences either us
2These values were fitted with predictions (or cross-worker annotation) using 5-fold cross validation and an L1 loss to optimise the mapping. A constraint is placed so that Same is 0, increases are positive and decreases are negative with a minimum 0.05 distance between.
Figure 2: Story 27, Human,  $S^{\mathrm{Hale}}$ ,  $S^{\mathrm{Ely}}$ ,  $U^{\mathrm{Ely}}$ ,  $U^{\alpha \mathrm{Ely}}$ . Solid lines: generated alternative continuations, dashed lines: sampled alternative continuations.
ing generated continuations (Gen) or continuations sampled from the corpus (Cor), except for  $S^{\mathrm{Ely}}$ , which can be computed without alternatives. We compared the LSTM and GRU variants (see Section 4) and experimented with rollouts of up to three sentences. We tried L1 and L2 distance for the Ely measures, but only report L1, which always performed better.
Discussion On the development set (see Table 2), we observe that all baselines perform poorly, indicating that distance between simple sentence representations or raw sentiment values do not model suspense. We find that Hale surprise  $S^{\mathrm{Hale}}$  performs well, reaching a maximum  $\rho$  of .675 on the development set. Hale uncertainty reduction  $U^{\mathrm{Hale}}$ , however, performs consistently poorly. Ely surprise  $S^{\mathrm{Ely}}$  also performs well, reaching as similar value as Hale surprise. Overall, Ely uncertainty reduction  $U^{\mathrm{Ely}}$  is the strongest performer, with  $\rho = .698$ , numerically outperforming the human upper bound.
Some other trends are clear from the development set: using GRUs reduces performance in all cases but one; rollout of more than one never leads to an improvement; sentiment weighting (prefix  $\alpha$  in the table) always reduces performance, as it introduces considerable noise (see Figure 2). We therefore eliminate the models that correspond to these settings when we evaluate on the test set.
For the test set results in Table 3 we also report upper and lower confidence bounds computed using the Fisher  $Z$ -transformation  $(p < 0.05)$ . On the test set,  $U^{\mathrm{Ely}}$  again is the best measure, with a correlation statistically indistinguishable from human performance (based on CIs). We find that absolute correlations are higher on the test set, presumably
Prediction	Model	Roll	τ↑	ρ↑
Human	.553	.614
Baselines	WordOverlap	1	.017	.026
GloveSim	1	.017	.029
GPTSim	1	.021	.031
α	1	.024	.036
SHaIe-Gen	GRU	1	.145	.182
LSTM	1	.434	.529
SHaIe-Cor	GRU	1	.177	.214
LSTM	1	.580	.675
UHaIe-Gen	GRU	1	.036	.055
LSTM	1	.009	.016
UHaIe-Cor	GRU	1	.048	.050
LSTM	1	.066	.094
SEly	GRU	1	.484	.607
LSTM	1	.427	.539
SαEly	GRU	1	.089	.123
LSTM	1	.115	.156
UEly-Gen	GRU	1	.241	.161
LSTM	2	.304	.399
LSTM	1	.610	.698
LSTM	2	.393	.494
UEly-Cor	GRU	1	.229	.264
LSTM	2	.512	.625
LSTM	3	.515	.606
LSTM	1	.594	.678
LSTM	2	.564	.651
LSTM	3	.555	.645
UαEly-Gen	GRU	1	.216	.124
LSTM	2	.219	.216
LSTM	1	.474	.604
LSTM	2	.316	.418
UαEly-Cor	GRU	1	.205	.254
LSTM	2	.365	.470
LSTM	1	.535	.642
LSTM	2	.425	.534
Table 2: Development set results for WritingPrompts for generated (Gen) or corpus sampled (Cor) alternative continuations;  $\alpha$  indicates sentiment weighting. Bold: best model in a given category; red: best model overall.
reflecting the higher human upper bound.
Overall, we conclude that our hierarchical architecture successfully models human suspense
Prediction	τ↑	ρ↑
Human	.652 (.039)	.711 (.033)
S Hale-Gen	.407 (.089)	.495 (.081)
S Hale-Cor	.454 (.085)	.523 (.079)
U Hale-Gen	.036 (.102)	.051 (.102)
U Hale-Cor	.061 (.100)	.088 (.101)
SEly	.391 (.092)	.504 (.082)
UEly-Gen	.620 (.067)	.710 (.053)
UEly-Cor	.605 (.069)	.693 (.056)
judgements on the WritingPrompts dataset. The overall best predictor is  $U^{\mathrm{Ely}}$ , uncertainty reduction computed over story representations. This measure combines the probability of continuation ( $S^{\mathrm{Hale}}$ ) with distance between story embeddings ( $S^{\mathrm{Ely}}$ ), which are both good predictors in their own right. This finding supports the theoretical claim that suspense is an expectation over the change in future states of a game or a story, as advanced by Ely et al. (2015).
6.2 Movie Turning Points
Task and Dataset An interesting question is whether the peaks in suspense in a story correspond to important narrative events. Such events are sometimes called turning points (TPs) and occur at certain positions in a movie according to screenwriting theory (Cutting, 2016). A corpus of movie synopses annotated with turning points is available in the form of the TRIPOD dataset (Papalampidi et al., 2019). We can therefore test if surprise or uncertainty reduction predict TPs in TRIPOD. As our model is trained on a corpus of short stories, this will also serve as an out-of-domain evaluation.
Papalampidi et al. (2019) assume five TPs: 1. Opportunity, 2. Change of Plans, 3. Point of no Return, 4. Major Setback, and 5. Climax. They derive a prior distribution of TP positions from their test set, and use this to constrain predicted turning points to windows around these prior positions. We follow this approach and select as the predicted TP the sentence with the highest surprise or uncertainty reduction value within a given constrained window. We report the same baselines as in the previous experiment, as well as the Theory Baseline,
Table 3: Test set results for WritingPrompts for generated (Gen) or corpus sampled (Cor) continuations. LSTM with rollout one; brackets: confidence intervals.
Dev D ↓	Test D ↓
Human	Not reported	4.30 (3.43)
Theory Baseline	9.65 (0.94)	7.47 (3.42)
TAM	7.11 (1.71)	6.80 (2.63)
WordOverlap	13.9 (1.45)	12.7 (3.13)
GloveSim	10.2 (0.74)	10.4 (2.54)
GPTSim	16.8 (1.47)	18.1 (4.71)
α	11.3 (1.24)	11.2 (2.67)
S^Hale - Gen	8.27 (0.68)	8.72 (2.27)
U^Hale - Gen	10.9 (1.02)	10.69 (3.66)
S^Ely	9.54 (0.56)	9.01 (1.92)
S^αEly	9.95 (0.78)	9.54 (2.76)
U^Ely - Gen	8.75 (0.76)	8.38 (1.53)
U^Ely - Cor	8.74 (0.76)	8.50 (1.69)
U^αEly - Gen	8.80 (0.61)	7.84 (3.34)
U^αEly - Cor	8.61 (0.68)	7.78 (1.61)
Table 4: TP prediction on the TRIPOD development and test sets.  $D$  is the normalised distance to the gold standard; CI in brackets.
which uses screenwriting theory to predict where in a movie a given TP should occur (e.g., Point of No Return theoretically occurs  $50\%$  through the movie). This baseline is hard to beat (Papalampidi et al., 2019).
Results and Discussion Figure 3 plots both gold standard and predicted TPs for a sample movie synopsis (text and further examples in Appendix D). The results on the TRIPOD development and test sets are reported in Table 4 (we report both due to the small number of synopses in TRIPOD). We use our best LSTM model with a of rollout of one; the distance measure for Ely surprise and uncertainty reduction is now L2 distance, as it outperformed L1 on TRIPOD. We report results in terms of  $D$ , the normalised distance between gold standard and predicted TP positions.
On the test set, the best performing model with  $D = 7.78$  is  $U^{\alpha \mathrm{Ely}}$ -Cor, with  $U^{\alpha \mathrm{Ely}}$ -Gen only slightly worse. It is outperformed by TAM, the best model of Papalampidi et al. (2019), which however requires TP annotation at training time.  $U^{\alpha \mathrm{Ely}}$ -Cor is close to the Theory Baseline on the test set, an impressive result given that our model has no TP supervision and is trained on a different domain. The fact that models with sentiment
Figure 3: Movie 15 Minutes,  $S^{\mathrm{Hale}}$ ,  $S^{\mathrm{Ely}}$ ,  $U^{\mathrm{Ely}}$ ,  $U^{\alpha \mathrm{Ely}}$ ,  $\spadesuit$  theory baseline,  $\star$  TP annotations, triangles are predicted TPs.
weighting (prefix  $\alpha$ ) perform well here indicates that turning points often have an emotional resonance as well as being suspenseful.
Our overall findings suggest that by implementing concepts from psycholinguistic and economic theory, we can predict human judgements of suspense in storytelling. That uncertainty reduction  $(U^{\mathrm{Ely}})$  outperforms probability-only  $(S^{\mathrm{Hale}})$  and state-only  $(S^{\mathrm{Ely}})$  surprise suggests that, while consequential state change is of primary importance for suspense, the probability distribution over the states is also a necessary factor. Uncertainty reduction therefore captures the view of suspense as reducing paths to a desired outcome, with more consequential shifts as the story progresses (O'Neill and Riedl, 2014; Ely et al., 2015; Perreault, 2018). This is more in line with the Smuts (2008) Desire-Frustration view of suspense, where uncertainty is secondary.
Strong psycholinguistic claims about suspense are difficult to make due to several weaknesses in our approach, which highlight directions for future research: the proposed model does not have a higher-level understanding of event structure; most likely it picks up the textual cues that accompany dramatic changes in the text. One strand of further work is therefore analysis: Text could be artificially manipulated using structural changes, for example by switching the order of sentences, mixing multiple stories, including a summary at the beginning that foreshadows the work, masking key suspenseful words, or paraphrasing. An analogue of this would be adversarial examples used in computer vision. Additional annotations, such as how certain
readers are about the outcome of the story, may also be helpful in better understanding the relationship between suspense and uncertainty. Automated interpretability methods as proposed by Sundararajan et al. (2017), could shed further light on models' predictions.
The recent success of language models in wideranging NLP tasks (e.g., Radford et al., 2019) has shown that language models are capable of learning semantically rich information implicitly. However, generating plausible future continuations is an essential part of the model. In text generation, Fan et al. (2019) have found that explicitly incorporating coreference and structured event representations into generation produces more coherent generated text. A more sophisticated model would incorporate similar ideas.
Autoregressive models that generate step by step alternatives for future continuations are computationally impractical for longer rollouts and are not cognitively plausible. They also differ from the Ely et al. (2015) conception of suspense, which is in terms of Bayesian beliefs over a longer-term future state, not step by step. There is much recent work (e.g., Ha and Schmidhuber (2018); Gregor et al. (2019)), on state-space approaches that model beliefs as latent states using variational methods. In principle, these would avoid the brute-force calculation of a rollout and conceptually, anticipating longer-term states aligns with theories of suspense.
Related tasks such as inverting the understanding of suspense to utilise the models in generating more suspenseful stories may also prove fruitful.
This paper is a baseline that demonstrates how modern neural network models can implicitly represent text meaning and be useful in a narrative context without recourse to supervision. It provides a springboard to further interesting applications and research on suspense in storytelling.
The authors would like to thank the anonymous reviewers, Pinelopi Papalampidi and David Hodges for reviews of the annotation task, the AMT annotators, and Mirella Lapata, Ida Szubert, and Elizabeth Nielson for comments on the paper. Wilmot's work is funded by an EPSRC doctoral training award.
H Porter Abbott. 2008. The Cambridge introduction to narrative. Cambridge University Press.
David Bamman, Brendan O'Connor, and Noah A Smith. 2013. Learning latent personas of film characters. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 352-361.
David Bamman, Ted Underwood, and Noah A. Smith. 2014. A Bayesian mixed effects model of literary character. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 370-379, Baltimore, Maryland. Association for Computational Linguistics.
Zheng Cai, Lifu Tu, and Kevin Gimpel. 2017. Pay attention to the ending: Strong neural baselines for the roc story cloze task. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 616-622.
Snigdha Chaturvedi, Mohit Iyyer, and Hal Daume III. 2017. Unsupervised learning of evolving relationships between literary characters. In *Thirty-First AAAI Conference on Artificial Intelligence*.
Yun-Gyung Cheong and R Michael Young. 2014. Suspenser: A story generation system for suspense. IEEE Transactions on Computational Intelligence and AI in Games, 7(1):39-52.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2015. Gated feedback recurrent neural networks. In International Conference on Machine Learning, pages 2067-2075.
James E Cutting. 2016. Narrative theory and the dynamics of popular movies. Psychonomic Bulletin & Review, 23(6):1713-1743.
Pablo Delatorre, Carlos León, Alberto G Salguero, Manuel Palomo-Duarte, and Pablo Gervás. 2018. Confronting a paradox: a new perspective of the impact of uncertainty in suspense. Frontiers in Psychology, 9:1392.
Vera Demberg and Frank Keller. 2008. Data from eyetracking corpora as evidence for theories of syntactic processing complexity. Cognition, 101(2):193-210.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jeffrey Ely, Alexander Frankel, and Emir Kamenica. 2015. Suspense and surprise. Journal of Political Economy, 123(1):215-260.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
Angela Fan, Mike Lewis, and Yann Dauphin. 2019. Strategies for structuring story generation. In ACL.
Edward Morgan Forster. 1985. Aspects of the Novel, volume 19. Houghton Mifflin Harcourt.
Stefan L Frank. 2013. Uncertainty reduction as a measure of cognitive load in sentence comprehension. *Topics in Cognitive Science*, 5(3):475-494.
Richard J Gerrig. 1989. Suspense in the absence of uncertainty. Journal of Memory and Language, 28(6):633-648.
Richard J Gerrig and Allan BI Bernardo. 1994. Readers as problem-solvers in the experience of suspense. Poetics, 22(6):459-472.
Philip John Gorinski and Mirella Lapata. 2018. What's this movie about? a joint neural network architecture for movie content analysis. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1770-1781, New Orleans, Louisiana. Association for Computational Linguistics.
Karol Gregor, George Papamakarios, Frederic Besse, Lars Buesing, and Theophane Weber. 2019. Temporal difference variational auto-encoder. In International Conference on Learning Representations.
Caglar Gülcehre, Orhan First, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. CoRR, abs/1503.03535.
David Ha and Jürgen Schmidhuber. 2018. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems 31, pages 2451-2463. Curran Associates, Inc. https://worldmodels.github.io.
John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the 2nd Conference of the North American Chapter of the Association for Computational Linguistics, volume 2, pages 159-166, Pittsburgh, PA. Association for Computational Linguistics.
John Hale. 2006. Uncertainty about the rest of the sentence. Cognitive science, 30(4):643-672.
Brent Harrison, Christopher Purdy, and Mark O Riedl. 2017. Toward automated story generation with markov chain monte carlo methods and deep neural networks. In Thirteenth Artificial Intelligence and Interactive Digital Entertainment Conference.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
Hans Hoeken and Mario van Vliet. 2000. Suspense, curiosity, and surprise: How discourse structure influences the affective and cognitive processing of a story. Poetics, 27(4):277-286.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018. Learning to write with cooperative discriminators. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1638-1649, Melbourne, Australia. Association for Computational Linguistics.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
C. T. Hsu, M. Conrad, and A. M. Jacobs. 2014. Fiction feelings in harry potter: haemodynamic response in the mid-cingulate cortex correlates with immersive reading experience. *NeuroReport*, 25:1356-1361.
Clayton J Hutto and Eric Gilbert. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In *Eighth international AAAAI conference on weblogs and social media*.
Mohit Iyyer, Anupam Guha, Snigdha Chaturvedi, Jordan Boyd-Graber, and Hal Daumé III. 2016. Feuding families and former friends: Unsupervised learning for dynamic fictional relationships. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1534-1544.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759.
Daniel Kahneman and Amos Tversky. 2013. Prospect theory: An analysis of decision under risk. In Handbook of the fundamentals of financial decision making: Part I, pages 99-127. World Scientific.
MG Kendall. 1975. Rank correlation measures. Charles Griffin, London, 202:15.
Y. Khrypko and P. Andreae. 2011. Towards the problem of maintaining suspense in interactive narrative. In Proceedings of the 7th Australasian Conference on Interactive Entertainment, pages 5:1-5:3.
Klaus Krippendorff. 2011. Computing krippendorff's alpha-reliability. 2011. Annenberg School for Communication Departmental Papers: Philadelphia.
Zhiwei Li, Neil Bramley, and Todd M. Gureckis. 2018. Modeling dynamics of suspense and surprise. In Proceedings of the 40th Annual Meeting of the Cognitive Science Society, CogSci 2018, Madison, WI, USA, July 25-28, 2018.
Chunhua Liu, Haiou Zhang, Shan Jiang, and Dong Yu. 2018a. DEMN: Distilled-exposition enhanced matching network for story comprehension. In Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation, Hong Kong. Association for Computational Linguistics.
Fei Liu, Trevor Cohn, and Timothy Baldwin. 2018b. Narrative modeling with memory chains and semantic supervision. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 278-284, Melbourne, Australia. Association for Computational Linguistics.
Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In International Conference on Learning Representations.
Lara J Martin, Prithviraj Ammanabrolu, Xinyu Wang, William Hancock, Shruti Singh, Brent Harrison, and Mark O Riedl. 2018. Event representations for automated story generation with deep neural nets. In Thirty-Second AAAI Conference on Artificial Intelligence.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849, San Diego, California. Association for Computational Linguistics.
M. B. Oliver. 1993. Exploring the paradox of the enjoyment of sad films. Human Communication Research, 19:315-342.
Brian O'Neill. 2013. A computational model of suspense for the augmentation of intelligent story generation. Ph.D. thesis, Georgia Institute of Technology.
Brian O'Neill and Mark Riedl. 2014. Dramatis: A computational model of suspense. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, July 27-31, 2014, Quebec City, Quebec, Canada., pages 944-950.
Pinelopi Papalampidi, Frank Keller, and Mirella Lapata. 2019. Movie plot analysis via turning point identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1707-1717, Hong Kong, China. Association for Computational Linguistics.