SlowGuess/ACL_2020_OCR huggingface dataset:
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Joseph Perreault. 2018. The Universal Structure of Plot Content: Suspense, Magnetic Plot Elements, and the Evolution of an Interesting Story. Ph.D. thesis, University of Idaho.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Juan A Prieto-Pablos. 1998. The paradox of suspense. Poetics, 26(2):99-113.
Eric S Rabkin. 1973. "Narrative suspense." When Slim turned sideways...". Ann Arbor: University of Michigan Press.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. https://openai.com/blog/language-unsupervised/.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
Hannah Rashkin, Maarten Sap, Emily Allaway, Noah A. Smith, and Yejin Choi. 2018. Event2Mind: Commonsense inference on events, intents, and reactions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 463-473, Melbourne, Australia. Association for Computational Linguistics.
Brian Roark, Asaf Bachrach, Carlos Cardenas, and Christophe Pallier. 2009. Deriving lexical and syntactic expectation-based measures for psycholinguistic modeling via incremental top-down parsing. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 324-333, Singapore. Association for Computational Linguistics.
Melissa Roemmele and Andrew Gordon. 2018. An encoder-decoder approach to predicting causal relations in stories. In Proceedings of the First Workshop on Storytelling, pages 50-59, New Orleans, Louisiana. Association for Computational Linguistics.
G. Schraw, Flowerday, T., and S. Lehman. 2001. Increasing situational interest in the classroom. Educational Psychology Review, 13:211-224.
Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D Manning. 2019. Do massively pretrained language models make better storytellers? arXiv preprint arXiv:1909.10705.
Pranab Kumar Sen. 1968. Estimates of the regression coefficient based on kendall's tau. Journal of the American statistical association, 63(324):1379-1389.
Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. 2018. Tackling the story ending biases in the story cloze test. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 752-757.
Aaron Smuts. 2008. The desire-frustration theory of suspense. *The Journal of Aesthetics and Art Criticism*, 66(3):281-290.
Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates. 2018. Cold fusion: Training seq2seq models together with language models. In *Interspeech* 2018, 19th Annual Conference of the International Speech Communication Association, Hyderabad, India, 2-6 September 2018., pages 387-391.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 3319-3328. JMLR.org.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. 2013. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139-1147.
Marten Van Schijndel and Tal Linzen. 2018a. Can entropy explain successor surprisal effects in reading? CoRR, abs/1810.11481.
Marten Van Schijndel and Tal Linzen. 2018b. Modeling garden path effects without explicit hierarchical syntax. In Proceedings of the 40th Annual Meeting of the Cognitive Science Society, CogSci 2018, Madison, WI, USA, July 25-28, 2018.
Robert J Yanal. 1996. The paradox of suspense. The British Journal of Aesthetics, 36(2):146-159.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. CoRR, abs/1905.12616.
Dolf Zillmann. 1996. The psychology of suspense in dramatic exposition. *Suspense: Conceptualizations, theoretical analyses, and empirical explorations*, pages 199-231.
A Pre-processing
WritingPrompts comes from a public forum of short stories and so is naturally noisy. Story authors often use punctuation in unusual ways to mark out sentences or paragraph boundaries and there are lots of spelling mistakes. Some of these cause problems with the GPT model and in some circumstances can cause it to crash. To improve the quality, sentence demarcations are left as they are from the original WritingPrompts dataset but some sentences are cleaned up and others skipped over. Skipping over is also why there sometimes are gaps in the graph plots as the sentence was ignored during training and inference. The preprocessing steps are as follows. Where substitutions are made rather than ignoring the sentence, the token is replaced by the Spacy (Honnibal and Montani, 2017) POS tag.
1. English Language: Some phrases in sentences can be non-English, Whatthelang (Joulin et al., 2016) is used to filter out these sentences.
2. Nondictionary words: PyDictionary and PyEnchant and used to check if each word is a dictionary word. If not they are replaced.
3. Repeating Symbols: Some author mark out sections by using a string of characters such as \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
4. Ignoring sentences: If after all of these replacements there are not three or more GPT word pieces ignoring the POS replacements then the sentence is skipped. The same processing applies to generating sentences in the inference. Occasionally the generated sentences can be nonsense, so the same criteria are used to exclude them.
B Mechanical Turk Written Instructions
These are the actual instructions given to the Mechanical Turk Annotators, plus the example in Table 5:
INSTRUCTIONS For the first HIT there will be an additional training step to pass. This will take about 5 minutes. After this you will receive a code which you can enter in the code box to bypass the
training for subsequent HITS. Other stories are in separate HITS, please search for "Story dramatic tension, reading sentence by sentence" to find them. The training completion code will work for all related HITS.
You will read a short story and for each sentence be asked to assess how the dramatic tension increases, decreases or stays the same. Each story will take an estimated 8-10 minutes. Judge each sentence on how the dramatic tension has changed as felt by the main characters in the story, not what you as a reader feel. Dramatic tension is the excitement or anxiousness over what will happen to the characters next, it is anticipation.
Increasing levels of each of the following increase the level of dramatic tension:
- Uncertainty: How uncertain are the characters involved about what will happen next? Put yourself in the characters shoes; judge the change in the tension based on how the characters perceive the situation.
- Significance: How significant are the consequences of what will happen to the central characters of the story?
An Example: Take a dramatic moment in a story such as a character that needs to walk along a dangerous cliff path. When the character first realises they will encounter danger the tension will rise, then tension will increase further. Other details such as falling rocks or slips will increase the tension further to a peak. When the cliff edge has been navigated safely the tension will drop. The pattern will be the same with a dramatic event such as a fight, argument, accident, romantic moment, where the tension will rise to a peak and then fall away as the tension is resolved.
You will be presented with one sentence at a time. Once you have read the sentence, you will press one of five keys to judge the increase or decrease in dramatic tension that this sentence caused. You will use five levels (with keyboard shortcuts in brackets):
- Big Decrease (A): A sudden decrease in dramatic tension of the situation. In the cliff example the person reaching the other side safely.
- Decrease (S): A slow decrease in the level of tension, a more gradual drop. For example the cliff walker sees an easier route out.
Annotation	Sentence
NA	Clancy Marguerian, 154, private first class of the 150 + army , sits in his foxhole.
Increase	Tired cold, wet and hungry, the only thing preventing him from laying down his rifle and walking towards the enemy lines in surrender is the knowledge that however bad he has it here, life as a 50 - 100 POW is surely much worse .
Increase	He's fighting to keep his eyes open and his rifle ready when the mortar shells start landing near him.
Same	He hunkers lower.
Increase	After a few minutes under the barrage, Marguerian hears hurried footsteps, a grunt, and a thud as a soldier leaps into the foxhole.
Same	The man's uniform is tan , he must be a 50 - 100 .
Big Increase	The two men snarl and grab at each other , grappling in the small foxhole .
Same	Abruptly, their faces come together.
Decrease	“Clancy?”
Decrease	“Rob?”
Big Decrease	Rob Hall, 97, Corporal in the 50 - 100 army grins, as the situation turns from life or death struggle, to a meeting of two college friends.
Decrease	He lets go of Marguerian's collar.
Same	“Holy shit Clancy , you're the last person I expected to see here ”
Same	“Yeah” “Shit man , I didn't think I'd ever see Mr. volunteers every saturday morning at the food shelf’ , not after The Reorganization at least ”
Same	"Yeah Rob , it is something isn't it"
Decrease	“Man , I'm sorry, I tried to kill you there”.
Table 5: One of the training annotation examples given to Mechanical Turk workers. The annotation labels are the recommended labels. This is an extract from a validation set WritingPrompts story.
- Same (Space): Stays at a similar level. In the cliff example an ongoing description of the event.
- Increase (K): A gradual increase in the tension. Loose rocks fall nearby the cliff walker.
- Big Increase (L): A more sudden dramatic increase such as an argument. The cliff walker suddenly slips and falls.
POST ACTUAL INSTRUCTIONS In addition to the suspense annotation. The following review questions were asked:
- Please write a summary of the story in one or two sentences.
- Do you think the story is interesting or not? And why? One or two sentences.
- How interesting is the story? 1-5
The main purpose of this was to test if the MTurk Annotators were comprehending the stories and not trying to cheat by skipping over. Some further work
through can be done to tie these into the suspense measures and also the WritingPrompts prompts.
C Writing Prompts Examples
The numbers are from the full WritingPrompts test set. Since random sampling was done from these from for evaluation the numbers are not in a contiguous block. There are a couple of nonsense sentences or entirely punctuation sentences. In the model these are excluded in pre-processing but included here to match the sentence segmentation. Also there are some unusual break such as "should n't", this is because the word segmentation produced by the Spacy tokenizer.
C.1 Story 27
This is Story 27 from the test set in Figure 4, it is the same as the example in the main text:
0. As I finished up my research on Alligator breeding habits for a story I was tasked with writing, a bell began to ring loudly throughout the office.
Figure 4: Story 27, Human,  $S^{\text{Hale}}$ ,  $S^{\text{Ely}}$ ,  $U^{\text{Ely}}$ ,  $U^{\alpha \text{Ely}}$
1. I could feel the sound vibrating off the cubicle walls.
2. I looked over my cubicle wall to ask a co-worker what the bell was for.
3. I watched as he calmly opened his desk drawer, to reveal a small armory.
4. There were multiple handguns, knives and magazines and other assorted weapons neatly stashed away.
5. "What the hell is that for ? "
6. I questioned loudly, and nervously.
7. The man looked me in the eyes, and pointed his handgun at my face.
8. I saw my life flash before my eyes, and could n't understand what circumstances had arisen to put me in this position.
9. I heard the gun fire, and the sound of the shot rang through my ears.
10. I heard something hit the ground loudly behind me.
11. I turned to see the woman who had hired me yesterday, lying in a pool of blood on the floor.
12. She was holding a rifle in her arms.
13. I looked back at the man who had apparently just saved my life.
14. He seemed to be about 40 or so, well built, muscular and had a scar down the right side of his face that went from his forehead down to his beard.
15. “She liked to go after the new hires” he explained in a deep voice.
16. "She hires the ones she wants to kill"
17. I was n't sure what to make of this, but my thoughts were cut off by the sounds of screaming throughout the building.
18. "What's happening"
19. I asked, barely able to look my savior in the eyes.
20. “ You survive today , and you ’ll receive a bonus of $5,000 and your salary will be raised 5 % ”
21. I cut the man off.
22. "What does that?"
23. He continued to speak, while motioning me to stop taking.
24. " I 'll keep you alive , if you give me your bonus and half your raise
25. He finished.
26. I just nodded, still unable to understand the position I was in.
27. He grabbed my arm so hard I thought it would break, and pulled me over the cubicle wall, and under his desk.
28. Then, he placed a gun in my hand.
29. “The safety is on, and it’s fully loaded with one in the chamber”
30. He said, pointing to the safety switch.
31. The weapon felt heavy in my hand, I flicked the safety off with my thumb and gripped the gun tightly.
32. The man looked down at his watch.
33. “45 minutes to go”
C.2 Story 2066
This is Story 2066 from the test set in Figure 5:
0. The life pods are designed so we can't steer.
1. Meant for being stranded in space, it broadcasts an S.O.S.
2. to the entire human empire even as it leaves the mother ship.
3. Within minutes any occupant will be gassed so they won't suffer the long months, and perhaps years before a rescue.
4. As soon as your vitals show you 're in deep sleep, it puts the entire interior into a cryogenic freeze.
5. The technology is effective, efficient and brilliant.
6. But as I'm being launched out of our vessel I can't help but slam the hatch with my fists.
7. My ears are still ringing with the endless boom of explosions and my eyes covered in blind spots from the flashes.
8. The battle had been swift, and we humans had lost.
9. Captain's orders: Abandon ship.
10. Which was why I was stuck here, counting the seconds before I got put into stasis.
11. This was no Titanic.
12. There were ample pods for the entire crew, by the time the call was made only half of us had access to the escape pods, and a quarter of those were injured, a condition that no matter how advanced our technology was, made the life pod a null option.
13. No use being cryogenically frozen if you bleed out before the temperature even drops.
14. Better men and women than I were stuck alive on the ship, and I had to abandon them to whatever their fate may be.
15. I sit back and harness myself into the chair.
16. No use getting worked up over survivor's guilt now.
17. I'll do that when I thaw.
18. *
19. \*\*
20. The first thing I notice is the cold.
21. I'm too cold.
22. I shiver, my uniform plastered to me.
23. I frown at its tattered appearance.
24. What had happened?
25. The last thing I remember is ...
26. The life pod.
Figure 5: Story 2066, Human,  $S^{\text{Hale}}$ ,  $S^{\text{Ely}}$ ,  $U^{\text{Ely}}$ ,  $U^{\alpha \text{Ely}}$
27. I'm still in it.
28. But I’ve been picked up.
29. Someone on the outside has initiated the thaw cycle.
30. At once I ’ m struck by relief .
31. Then anxiety.
32. How long was I out?
33. How many of the crew survived?
34. Their screams are coming back to me now, and I squirm with the pain.
35. “Please do n’t let me be the only one,” I whisper to myself, half pleading with fate, half praying to a God.
36. The hatch swings open.
37. The lump in my throat drops to my toes with the weight of lead.
38. A gun greets me.
39. Slowly, I put my hands behind my head.
40. There's no mistaking the alien wielding it.
41. The brute features are familiar, too familiar.
42. I've been rescued by the wrong side.
C.3 Story 3203
This is Story 3203 from the test set in Figure 6:
0. I swore never to kill.
1. I swore that I will never stoop down to their level.
2. That we, the guardians of justice, can and will achieve our goals through the peaceful way.
3. But as I stood there, at the edge of the cliff, staring at the hideous smile that has tormented me for far too long, I could feel my vow slowly breaking before me.
4. "So what it's gon na be Batsy?
Figure 6: Story 3203, Human,  $S^{\text{Hale}}$ ,  $S^{\text{Ely}}$ ,  $U^{\text{Ely}}$ ,  $U^{\alpha \text{Ely}}$
5. Will you choose to kill the evil crazy clown, or are you going to let poor Miss Lane fall to her death?
6. Tick tock tick tock , time ’ s ticking ! ”
7. I gritted my teeth.
8. Lois was suspended in mid - air, 12 stories high, her life hanging by the mere minutes.
9. Around me, the League lay incapacitated, having fallen to Joker's devious ambush.
10. I turned towards Clark, hoping that he would have woken up by now.
11. No luck.
12. The Kryptonite knock out gas had worked its miracle.
13. As fate would have it, only two of us are left.
14. Two bitter rivals to the very end.
15. "Lethergo,Joker!
16. This fight is between you and me!
17. I shouted.
18. My mind raced for possible solutions.
19. A well - aimed batarang could free Lois, but I have to rappel to her in time.
20. Too risky with Joker free.
21. I could try knocking him out, but that would not leave me enough time to- " Tsk tsk tsk, my dear Bats.
22. Trying to stall for time, are n't you?
23. How many times must I tell you that it won't work!
24. I know you, Bats, better than you know yourself.
25. In fact ...
26. He took out a remote, and pushed one of the bright red buttons on it.
27. The cable jerked downwards, closer to the barrel of Joker venom.
28. “... for every minute you spend thinking, Miss Lane will be closer to smiley face land.
29. How about that!
30. Hahahaha ! ”
31. It was right then when I lost it.
32. I leaped from my spot, headed straight for the Prince of Clowns.
33. I thought about the last time we almost lost Lois.
34. Clark was so close to unleashing a destructive rampage across Metropolis.
35. Too close.
36. And it was on that day when every member of the League swore an oath to protect Lois no matter what it takes, no matter what the cost, even if it meant breaking our own sacred vows.
37. Superman was too great an asset to be lost.
38. Joker knew that.
39. From the very moment he saw the destruction Clark unleashed.
40. And he has been targeting Lois ever since .
41. The blade plunged through his chest and into his heart surprisingly quick.
42. I had expected the Joker to have a fail safe mechanism, but apparently he did not.
43. He wanted me to do it.
44. The blood splattered against my suit, as the sickening sound of flesh tearing apart filled my ears.
45. And as all these happened, the Joker kept laughing, his hysterical voice filling the air.
46. He laughed and laughed, until his voice gradually grew weaker, softer.
47. Before he drew his last breath, he raised his bloodied left hand and patted me on my cowl.
48. "Hehehe ... I win, Batsy."
D Turning Points Examples
This section is the full text output with some example plots from Turning Points TRIPOD dataset.
D.1 15 Minutes
The full text for the synopsis of 15 Minutes in Figure 7, this is the same example as is given in the main text:
0. After getting out of prison, ex - convicts Emil Slovak ( Karel Roden ) and Oleg Razgul ( Oleg Taktarov ) travel to New York City to meet a contact in order to claim their part of a bank heist in
1. Russia (or somewhere in the Czech Republic).
2. Within minutes of arriving, Oleg steals a video camera.
3. They go to the brownstone apartment of their old partner Milos Karlova ( Vladimir Mashkov ) and his wife Tamina , and demand their share .
4. When Milos admits that he spent it, an enraged Emil kills him with a kitchen knife, then breaks Tamina's neck as Oleg tapes it with his new camera.
5. The couple's neighbor, Daphne Handlova ( Vera Farmiga ), witnesses everything , but she escapes before they can get to her .
6. To cover up the crime, they douse the bodies in acetone, carefully position them on the bed, and burn down the apartment, intending to pass it off as an accident.
7. Jordy Warsaw ( Edward Burns ) , an arson investigator , andYPD detective Eddie Flemming ( Robert De Niro ) are called to the scene
8. Flemming is a high profile detective who frequently appears on the local tabloid TV show Top Story.
9. Flemming and Warsaw decide to work the case together.
10. They eventually determine that Milos was stabbed so hard that the knife's tip broke off and lodged in his spine.
Figure 7: The film 15 Minutes,  $S^{\mathrm{Hale}}$ ,  $S^{\mathrm{Ely}}$ ,  $U^{\mathrm{Ely}}$ ,  $U^{\alpha \mathrm{Ely}}$ , ◆ theory baseline, ★ TP annotations, triangles are predicted TPs.
11. While checking out the crowd outside, Warsaw spots Daphne trying to get his attention.
12. When he finally gets to where she was, she is gone, but Warsaw manages to produce a sketch of the witness.
13. Emil, who got hold of Daphne's wallet when she fled the apartment earlier, realizes that Daphne is in the country illegally and will be deported if she calls the police.
14. He contacts an escort service from a business card he found in Daphne's wallet.
15. He asks for a Czech girl hoping she will arrive
16. When Honey, a regular call girl, arrives instead, he stabs and kills her, but not before getting the address of the escort service from her.
17. Oleg tapes the entire murder.
18. In fact, he tapes everything he can; a wannabe filmmaker, he aspires to be the next Frank Capra.
19. Flemming and Warsaw investigate her murder, determine the link to the fire, and also visit the escort service.
20. Rose Heam ( Charlize Theron ) runs the service and tells them that the girl they are looking for ( Daphne ) does not work for her but rather a local hairdresser , and she just told the same thing to
21. a couple other guys that were asking the same questions.
22. Flemming and Warsaw then rush to the hairdresser but get there just after Emil and Oleg warn the girl not to say anything to anyone.
23. As Flemming puts Daphne into his squad car, he notices Oleg taping them from across the street.