SlowGuess/NAACL_2021_OCR huggingface dataset:
Figure 3: Using the replay baseline to predict the  $n + i$ -th story block from the  $n$ -th story block (story block size = 5, 10, ..., 1000.) Things that happen in the current story block are more likely to happen again shortly.
shows the results. We can see that things that happen now will be more likely to happen in the near future compared to story blocks farther from the current one.
Event-Rep works better in short stories. In the Bookcorpus dataset, event representation works better than the frame representation in small block sizes (5, 10, and 20). However, starting from a block size of 50, the model cannot perform as well as the other models. We thus conclude that event representation works better in short stories. The main reason is that event representations are generated on a sentence-by-sentence basis and will create overwhelming information on long stories. The existing intermediate representations (see Section 1) are mostly generated from sentences and will likely have the same issue as the event representation. Compared to the existing works, the proposed frame representation encodes a story block, no matter how long it is, into a fixed-length vector and therefore performs better on longer stories.
Feature	Model	Block Size
5	10	20	50	100	150	200
-	Prior	.2029	.2435	.2856	.3388	.3754	.3962	.4105
Frame	IR	.0401	.0615	.0900	.1368	.1775	.2051	.2262
Frame	RF	.2030	.2440	.2871	.3418	.3801	.4025	.4184
Frame	LGBM	.2033	.2472	.2935	.3540	.3980	.4248	.4441
Frame	DAE	.2058	.2482	.2929	.3507	.3926	.4178	.4400
Event	Event-Rep	.2046	.2470	.2905	.3454	.3799	.4069	.4171
Text	BERT	.2088	.2529	.2981	.3550	.3949	.4178	.4371
BERT performs very well in short stories. The results of BERT and SciBERT in Table 3 and Table 4 show that textual information is helpful in predicting story blocks. BERT performs better when the block size is under 100 in the Bookcorpus dataset and below 3 in CODA-19. However, handling long texts remain challenging for BERT, as its computational complexity scales with the square of the token length. Researchers started reducing the computation complexity for transformer-based models to allow modeling on long texts such as Linformer (Wang et al., 2020b), Longformer (Beltagy et al., 2020), Reformer (Kitaev et al., 2020), and BigBird (Zaheer et al., 2020). However, these models still require a lot of computation power and are not yet ready for general use.
The good performance does not merely come from the number of instances. Deep learning methods often require more instances for training. To show that the result in Table 3 is not mainly caused by the number of instances, we conduct the same experiment in Bookcorpus dataset using 88,720 training instances for block sizes ranging from 5 to 200. Table 5 shows the results. The performance is affected, but the conclusions we make above still stand, showing that the number of instances is not the main factor for our observations. Meanwhile, we find that BERT is affected more than LGBM. In Table 5 the performance of BERT drops by  $-0.0092$  to  $-0.0051$  compared to Table 3, but LGBM only drops  $-0.0039$  to  $-0.0007$ . Although this suggests that the number of instances can cause the difference, it also shows that the frame representation can be used with fewer instances.
GPT-2 is not effective. GPT-2 is not effective in predicting the story flow even though it can generate reasonable sentences. Even the naive Replay
Table 5: Result of the downsampling experiment. Although all the performance drops, the observations we find are still true. Therefore, the conclusions are not merely caused by the effect of data size.
window	Feature	Model	Block Size
20	50	100
2	Frame	LGBM	.2989	.3590	.4029
Text	BERT	.3081	.3625	.4002
5	Frame	LGBM	.2989	.3617	.4065
Text	BERT	.3082	.3618	.3985
Table 6: Results of using 2 or 5 foregoing story blocks to predict the  $n + 1$ -th story block. LGBM improves further when using more context but BERT fails to model the longer context, and its performance even gets hurt.
Frame	Lexical Units
Most Important Frames (Out of 50)
Kinship	father, mother, son, daughter
Biological_urge	tired, sleepy, randy, hungry
Connectors	ribbon, rope, thread, string
Firefighting	fight, battle, control, tackle
Origin	Chinese, American, Vietnamese, origin
Least Important Frames (Out of 50)
Proper/reference	proper, self
Cause_to_start	spark, generate, arouse, bring about
Friction	grate, squeal, scrunch, screech
Dominate_competitor	dominate, domination, dominant, strongman
State Continue	remain, stay, rest
Table 7: The most and least important five frames (from 50 random frames) identified in the ablation study.
baseline outperforms the GPT-2 baseline in predicting the story block. We hypothesize that GPT-2 is not good at maintaining the coherence among sentences or events, especially in the creative writing domain. Similar phenomenons are also observed by others and used to motivate the need for guided generation models or progressive generation models (Wang et al., 2020c; Tan et al., 2020).
6.1 Using a Larger Context Window
This paper focuses on using 1 story block to forecast the next one, i.e., window size  $= 1$  (see Figure 1.) As a proof of concept, we use 2 and 5 blocks (window size  $= 2$  and 5) for prediction, respectively. We use two models: LGBM with frame representation, and BERT with text. For LGBM, we simply concatenate the frame representation from the input story blocks to create the input vector. For BERT, we put the event tuple and the text together as the input. Table 6 shows the results. While BERT does not benefit from using more contexts, LGBM's performance improves, suggesting the potentials of using a larger context window. More research is required to understand the effects.
6.2 Which Semantic Frames Affect the Follow-Up Story More?
Different frames may contribute differently to the prediction of the follow-up story. To understand which frame plays a more important role in the story, we conduct an ablation study by investigating the LGBM model on block 150. We obliterate one frame from the input frame representation and record the performance change, where a higher performance deduction means the frame removed is more important. A total of 50 frames are selected randomly for the ablation study. Table 7 shows the top and bottom five frames. We hypothesize that the more generic frames, such as "State Continue" and "Proper/reference," might be less important to the follow-up stories, but it will require more research to understand the impacts fully.
We further evaluate the proposed method with humans. We first visualize the vector of semantic frames into word clouds so that humans can perceive and comprehend it. We then use online crowd workers to test the  $(i)$  representativeness and  $(ii)$  the specificity of the produced word clouds.
Visualizing Semantic Frame Vectors into Word Clouds. Figure 4 shows the workflow of generating word clouds based on a frame representation (i.e., a TF-IDF vector). In FrameNet, "lexical units" are the terms that can trigger a specific frame. Compared to showing the name and definition of a frame, lexical units are easier for people to read and comprehend. Therefore, we use the top 30 frames (ranked by their TF-IDF weights) and randomly select up to three lexical units for each frame to form a word cloud. The size and color of the lexical unit is computed according to the frame's TF-IDF weight, where a higher TF-IDF value will result in a larger font and darker color. Finally, we arrange the lexical units into three word clouds on nouns, verbs, and adjectives using their POS tags. All the word clouds are generated using d3-cloud (Davies, 2016).
7.1 Representativeness
This task evaluates which model can generate the most representative word cloud for a story block.
Task Setups. In this Human Intelligence Task (HIT), we show a story block  $(n + 1)$  and two or three [noun, verb, adjective] word clouds  $(n + 1)$
produced by different models based on the previous story block (n). The goal is to measure, from the users' perspective, how much the generated word clouds represent the actual human-written follow-up stories. We display the actual next story block  $(n + 1)$  and the word clouds produced by different models based on the latest story block  $(n)$ . The workers from Amazon Mechanical Turk (MTurk) are asked to read the story and select the word cloud that better represents the story block. In the worker interface, we set up a 3-minutes lock for submission and a reach-to-the-bottom lock for the story panel to make sure the workers read the story. Nine different workers are recruited for each task<sup>1</sup>. We empirically estimate the working time to be less than 6 minutes per HIT and set the price to $0.99/HIT (hourly wage = $10).
We choose block size 150 to compare two models: LGBM with frame representation and BERT with text. Ground-truth word clouds are also added to some of the HITs to check the validity of the task. A total of 150 instances are randomly selected from Boocorpus testing set. For each instance, the foregoing story block is feed into LGBM and BERT to predict the frame representation of the follow-up story block. Out of 150 instances, 50 instances are conducted with ground truth, where a total of three word clouds are shown. Another 100 instances are used for comparing LGBM against BERT directly.
Results. Over the 50 HITs where ground truth is included, (ground truth, LGBM, BERT) wins (32, 15, 16) HITs, respectively (ties exist.) Nine assignments are recruited from 9 workers for each HIT. Regarding to the assignment voting, (ground truth, LGBM, BERT) gets (199, 131, 120) votes, respectively. The result suggests that humans can correctly perceive the word clouds' conceptual meaning as the ground truth is rated the best.
Over the 100 HITs where LGBM and BERT are compared directly, (LGBM, BERT) wins (59, 41) HITs. Regarding the assignment voting, (LGBM, BERT) gets (472, 428) votes, respectively. The result shows that LGBM is better than BERT in a block size of 150, which aligns with our automatic evaluation results using cosine similarity (see Section 6.)
<sup>1</sup>Four built-in worker qualifications are used: HIT Approval Rate (≥98%), Number of Approved HITs (≥ 3000), Locale (US Only), and Adult Content Qualification.
Figure 4: The workflow to visualize the word clouds from frame representation. The top semantic frames are used where each is illustrated by a maximum of three corresponding lexical units. The size and the color of the lexical units are computed according to the TF-IDF value.
7.2 Specificity
This task evaluates whether using the proposed word cloud to represent a story block is specific enough for humans to distinguish the correct story from the distractor.
Task Setups. In this HIT, we show two story blocks  $(n)$  and one set of [noun, verb, adjective] word clouds  $(n)$ . Note that the current story block  $(n)$  and its ground-truth word cloud  $(n)$  are used to examine if humans can correctly perceive the semantic information from word cloud visualization. One story block is the answer that is referred to by the word clouds and the other one is a distractor. Workers are asked to read the two story blocks and select the story block that is referred to by the word clouds. Nine different workers are recruited for each HIT. We use the same worker interface design and built-in worker qualifications as that of Section 7.1. A HIT takes estimatedly 2.33 minutes and is priced at $0.38.
We choose block size 20 and use the groundtruth word clouds for this experiment. Fifty instances from 50 different books are randomly selected from Bookcorpus testing set. We also randomly select a 20-sentences story block from a different book as the distractor.
Results. Of the 450 assignments,  $63.8\%$  of the answers were correct. When aggregating the assignments using majority voting,  $74\%$  of 50 HITs were answered correctly. We thus believe that it is reasonably specific for humans to represent a story block using the proposed word clouds.
This paper proposes a semantic frame forecast task that aims to forecast the semantic frames in the next 10, 100, or even 1,000 sentences of a story. A long story is formulated as a sequence of story blocks that contain a fixed number of sentences. We further introduce a frame representation that can encode a story block into a fixed-length TF-
IDF vector over semantic frames. Experiments on both the Bookcorpus dataset and CODA-19 dataset show that the proposed frame representation helps semantic frame forecast in large story blocks. By visualizing the frame representation as word clouds, we also show that it is comprehensible, representative, and specific to humans. In the future, we will introduce the frame representation into story generation models to ensure coherence when generating long stories. We will also explore the possibility of supporting writers to develop the next part of their stories by generating semantic frames as clues using semantic frame forecast.
We would like to thanks the Huck Institutes of the Life Sciences' Coronavirus Research Seed Fund (CRSF) and the College of IST COVID-19 Seed Fund at Penn State University who support the construction of CODA-19. We also thank Tiffany Knearem for the feedback for designing word cloud visualization and workers who participated the human evaluation study.
2020. National novel writing month.
Prithviraj Ammanabrolu, Ethan Tien, Wesley Cheung, Zhaochen Luo, William Ma, Lara J Martin, and Mark O Riedl. 2020. Story realization: Expanding plot events into sentences. In AAAI, pages 7375-7382.
Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The berkeley framenet project. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics - Volume 1, ACL '98/COLING '98, pages 86-90, Stroudsburg, PA, USA. Association for Computational Linguistics.
David Bamman, Brendan O'Connor, and Noah A Smith. 2013. Learning latent personas of film characters. In Proceedings of the 51st Annual Meeting of
the Association for Computational Linguistics (Volume 1: Long Papers), pages 352-361.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3606-3611.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150.
Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. 2013. Generalized denoising autoencoders as generative models. In Advances in neural information processing systems, pages 899-907.
Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A. Smith. 2018. Creative writing with a machine in the loop: Case studies on slogans and stories. In 23rd International Conference on Intelligent User Interfaces, IUI '18, pages 329-340, New York, NY, USA. ACM.
J Davies. 2016. D3-cloud.(2016).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
Angela Fan, Mike Lewis, and Yann Dauphin. 2019. Strategies for structuring story generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2650-2660, Florence, Italy. Association for Computational Linguistics.
Katy Ilonka Gero and Lydia B. Chilton. 2019. Metaphoria: An algorithmic companion for metaphor creation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI '19, page 1-12, New York, NY, USA. Association for Computing Machinery.
Pablo Gervás, Belén Díaz-Agudo, Federico Peinado, and Raquel Hervás. 2004. Story plot generation based on cbr. In International Conference on Innovative Techniques and Applications of Artificial Intelligence, pages 33-46. Springer.
Mark Granroth-Wilding and Stephen Clark. 2016. What happens next? event prediction using a compositional neural network model. In Thirtieth AAAI Conference on Artificial Intelligence.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Chieh-Yang Huang, Shih-Hong Huang, and TingHao Kenneth Huang. 2020a. Heteroglossia: In-situ story ideation with the crowd. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1-12.
Ting-Hao Huang, Chieh-Yang Huang, Chien-Kuang Cornelia Ding, Yen-Chia Hsu, and C Lee Giles. 2020b. Coda-19: Using a non-expert crowd to annotate research aspects on  $10,000+$  abstracts in the Covid-19 open research dataset. In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020.
Ting-Hao Kenneth Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley, and Margaret Mitchell. 2016. Visual storytelling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1233-1239, San Diego, California. Association for Computational Linguistics.
Daphne Ippolito, David Grangier, Chris Callison-Burch, and Douglas Eck. 2019. Unsupervised hierarchical story infilling. In Proceedings of the First Workshop on Narrative Understanding, pages 37-43.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in neural information processing systems, pages 3146-3154.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.
Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In International Conference on Learning Representations.
Boyang Li, Stephen Lee-Urban, George Johnston, and Mark Riedl. 2013. Story generation with crowdsourced plot graphs. In Twenty-Seventh AAAI Conference on Artificial Intelligence.
Boyang Li and Mark Riedl. 2015. Scheherazade: Crowd-powered interactive narrative generation. In Twenty-Ninth AAAI Conference on Artificial Intelligence.
Hugo Liu and Push Singh. 2004. Conceptnet—a practical commonsense reasoning tool-kit. BT technology journal, 22(4):211-226.
Edward Loper and Steven Bird. 2002. Nltk: The natural language toolkit. In Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics - Volume 1, ETMTNLP '02, page 63-70, USA. Association for Computational Linguistics.
Ryan Lowe, Iulian Vlad Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. On the evaluation of dialogue systems with next utterance classification. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 264-269.
Lara J Martin, Prithviraj Ammanabrolu, Xinyu Wang, William Hancock, Shruti Singh, Brent Harrison, and Mark O Riedl. 2018. Event representations for automated story generation with deep neural nets. In Thirty-Second AAAI Conference on Artificial Intelligence.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849, San Diego, California. Association for Computational Linguistics.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
Nanyun Peng, Marjan Ghazvininejad, Jonathan May, and Kevin Knight. 2018. Towards controllable story generation. In Proceedings of the First Workshop on Storytelling, pages 43-49, New Orleans, Louisiana. Association for Computational Linguistics.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
Mark O Riedl and Robert Michael Young. 2010. Narrative planning: Balancing plot and character. Journal of Artificial Intelligence Research, 39:217-268.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series.
Melissa Roemmele and Andrew Gordon. 2018. An encoder-decoder approach to predicting causal relations in stories. In Proceedings of the First Workshop on Storytelling, pages 50-59.
Melissa Roemmele and Andrew S Gordon. 2015. Creative help: a story writing assistant. In International Conference on Interactive Digital Storytelling, pages 81-92. Springer.
Stuart Rose, Dave Engel, Nick Cramer, and Wendy Cowley. 2010. Automatic Keyword Extraction from Individual Documents, chapter 1. John Wiley & Sons, Ltd.
Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. 2019. Do massively pretrained language models make better storytellers? In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 843-861, Hong Kong, China. Association for Computational Linguistics.
Swabha Swayamdipta, Sam Thomson, Chris Dyer, and Noah A. Smith. 2017. Frame-Semantic Parsing with Softmax-Margin Segmental RNNs and a Syntactic Scaffold. arXiv preprint arXiv:1706.09528.
Bowen Tan, Zichao Yang, Maruan AI-Shedivat, Eric P Xing, and Zhiting Hu. 2020. Progressive generation of long text. arXiv preprint arXiv:2006.15720.
Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Michael Kinney, et al. 2020a. Cord-19: The pandemic of the 1st Workshop on NLP for COVID-19 at ACL 2020.
Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. 2020b. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768.
Su Wang, Greg Durrett, and Katrin Erk. 2020c. Narrative interpolation for generating and understanding stories. arXiv preprint arXiv:2008.07466.
Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan-and-write: Towards better automatic storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7378-7385.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontonon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062.
Bowen Zhang, Hexiang Hu, and Fei Sha. 2020. Visual storytelling via predicting anchor word embeddings in the stories. arXiv preprint arXiv:2001.04541.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV).