SlowGuess/NAACL_2021_OCR huggingface dataset:
Chieh-Yang Huang and Ting-Hao (Kenneth) Huang  
Pennsylvania State University, University Park, PA 16802, USA  
{chiehyang, txh710}@psu.edu
This paper introduces semantic frame forecast, a task that predicts the semantic frames that will occur in the next 10, 100, or even 1,000 sentences in a running story. Prior work focused on predicting the immediate future of a story, such as one to a few sentences ahead. However, when novelists write long stories, generating a few sentences is not enough to help them gain high-level insight to develop the follow-up story. In this paper, we formulate a long story as a sequence of "story blocks," where each block contains a fixed number of sentences (e.g., 10, 100, or 200). This formulation allows us to predict the follow-up story arc beyond the scope of a few sentences. We represent a story block using the term frequencies (TF) of semantic frames in it, normalized by each frame's inverse document frequency (IDF). We conduct semantic frame forecast experiments on 4,794 books from the Bookcorpus and 7,962 scientific abstracts from CODA-19, with block sizes ranging from 5 to 1,000 sentences. The results show that automated models can forecast the follow-up story blocks better than the random, prior, and replay baselines, indicating the task's feasibility. We also learn that the models using the frame representation as features outperform all the existing approaches when the block size is over 150 sentences. The human evaluation also shows that the proposed frame representation, when visualized as word clouds, is comprehensible, representative, and specific to humans. Our code is available at: https://github.com/appleternity/FrameForecasting.
Writing a good novel is hard. Creative writers can get stuck in the middle of their drafts and struggle to develop follow-up scenes. Writing support systems, such as Heteroglossia (Huang et al., 2020a), generate paragraphs or ideas to help writers figure out the next part of the ongoing story. However,
Figure 1: The semantic frame forecast is a task that predicts the semantic frames that will occur in the next part of a story based on the texts written so far.
little literature focuses on plot prediction for long stories. Much prior work focused on predicting the immediate future of a story, i.e., one to a few sentences later. For example, the Creative Help system used a recurrent neural network model to generate the next sentence to support writing (Roemmele and Gordon, 2015); the Scheherazade system uses crowdsourcing and artificial intelligence techniques to interactively construct the narrative sentence by sentence (Li and Riedl, 2015); Clark et al. (2018) study machine-in-the-loop story writing where the machine constantly generates a suggestion for the next sentence to stimulate writers; and Metaphoria (Gero and Chilton, 2019) generates metaphors, an even smaller unit, to inspire writers based on an input word by searching relations and ranking distances on ConceptNet (Liu and Singh, 2004).
Generating a coherent story across multiple sentences is challenging, even with cutting-edge pretrained models (See et al., 2019). To generate coherent stories, researchers often first generate a high-level representation of the story plots and then use it as a guide to generate a full story. For example, Martin et al. (2018) propose an event representation that uses an SVO tuple to generate story plots; Plan-and-write (Yao et al., 2019) uses the RAKE algorithm (Rose et al., 2010) to extract the keyword in each sentence to form a storyline and treat it as an intermediate representation; Fan et al. (2019)
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2702-2713 June 6-11, 2021. ©2021 Association for Computational Linguistics
use predicate-argument pairs annotated by semantic role labelers to model the structure of stories; and Zhang et al. (2020) take words with a certain part-of-speech tag as anchors and show that using anchors as the intermediate representation can improve the story quality. However, these projects all focused on short stories: The event representation is developed on a Wikipedia movie plot summary dataset (Bamman et al., 2013), where a summary has an average of 14.52 sentences; Plan-and-write uses the ROCStories dataset (Mostafazadeh et al., 2016), where each story has only 5 sentences; Fan et al. test their algorithm on the Writing-Prompts dataset (Fan et al., 2018), where stories have 734 words (around 42 sentences) on average; and Zhang et al.'s anchor representation is developed on the VIST dataset (Huang et al., 2016), where a story has 5 sentences.
All the existing intermediate representations are generated on a sentence basis, meaning that the length of the representations increases along with the story length. That is, when applying these representations to novels that usually have more than 50,000 words (as defined by the National Novel Writing Month (wik, 2020)), it is not likely that such representations can still work. We thus introduce a new Frame Representation that compiles semantic frames into a fixed-length TF-IDF vector and a Semantic Frame Forecast task that aims to predict the next frame representation using the information in the current story block (see Figure 1). Two different datasets are built to examine the effectiveness of the proposed frame representation: one from Bookcorpus (Zhu et al., 2015), a fiction dataset; and one from CODA-19 (Huang et al., 2020b), a scientific abstract dataset. We establish several baselines and test them on different story block sizes, up to 1,000 sentences. The result shows that the proposed frame representation successfully captures the story plot information and helps the semantic frame forecast task, especially for story blocks with more than 150 sentences. To enable humans to perceive and comprehend frame representations, we further propose a process that visualizes a vector-based frame representation as word clouds. Human evaluations show that word clouds represent a story block with reasonable specificity, and our proposed model produces word clouds that are more representative than that of BERT.
Automated Story Generation. Classic story generation focuses on generating logically coherent stories, plot planning (Riedl and Young, 2010; Li et al., 2013), and case-based reasoning (Gervás et al., 2004). Recently, several neural story generation models have been proposed (Peng et al., 2018; Fan et al., 2018), even including massive pretrained models (Radford et al., 2019; Keskar et al., 2019). However, researchers realize that word-by-word generation models cannot efficiently model the long dependency across sentences (See et al., 2019). Models using intermediate representations as guidance to generate stories are then proposed (Yao et al., 2019; Martin et al., 2018; Ammanabrolu et al., 2020; Fan et al., 2019; Zhang et al., 2020). These works are developed toward short stories and thus are insufficient for modeling novels (See Section 1).
Automated Story Understanding. Story understanding is a longstanding goal of AI (Roemmele and Gordon, 2018). Several tests were proposed to evaluate AI models' ability to reason the event sequence in a story. Roemmele et al. (2011) proposed the Choice of Plausible Alternatives (COPA) task, focusing on commonsense knowledge related to identifying causal relations between sequences. Mostafazadeh et al. (2016) proposed the Story Cloze Test, in which the model is required to select which of two given sentences best completes a particular story. Ippolito et al. (2019) proposed the Story Infilling task, which aims to generate the middle span of a story that is coherent with the foregoing context and will reasonably lead to the subsequent plots. Under the broader umbrella of story understanding, some prior work aimed to predict the next event in a story (Granroth-Wilding and Clark, 2016) or to identify the right follow-up line in dialogues (Lowe et al., 2016).
3 Semantic Frame Forecast
As shown in Figure 1, we formulate a long story as a sequence of fixed-length story blocks. Each story block (Figure 2 (1)) has a set of semantic frames (Figure 2 (2)) (Baker et al., 1998). We convert a story block into the Frame Representation (Figure 2 (3)), a TF-IDF vector over semantic frames, by computing the term frequency in that story block and the inverse document frequency over all the story blocks in the corpus. FrameNet (Baker et al.,
(1) Story Block (20 Sentences)
His mouth turned up slightly at the corners. "Are you really."
Meanwhile, in the city center...
too small for his enormous hands. There were two huge guns strapped in leather holsters on his hips.
Figure 2: The steps to generate the frame representation for story blocks. The human-readable word clouds are generated to illustrate the conceptual meaning of the frame representation.
Semantic Frame Parsing
(2) Semantic Frames
Meanwhile, [in] the [city] [center]...
[There] [were] [two] [huge] [guns] strapped [in] leather holsters [on] his hips.
Convert to TF-IDF
(3) Frame Representation (TF-IDF)
Prohibiting_or_licensing : 8.37
Size : 6.47
Rebellion : 6.31
Local_by_use : 3.96
Achieving_first : 3.74
Visualize as Word Cloud
(4) Human-Readable Word Cloud
1998) defined a total of 1,221 different semantic frames, so the generated TF-IDF has 1,221 dimensions. The Semantic Frame Forecast is then defined as a task to predict the frame representation of the  $n + 1$ -th story block using the foregoing content, namely the  $n$ -th story block.
Evaluation Metric. We use Cosine Similarity between the predicted vector and the gold-standard vector (complied from the human-written story block) for evaluation. Many other metrics, such as Mean-Squared Error (MSE), also exist to measure the distance between two vectors.
We build the dataset from the existing Bookcorpus dataset (Zhu et al., 2015) and CODA-19 dataset (Huang et al., 2020b). This section describes how we preprocess the data, remove undesired content, and build the final dataset.
Bookcorpus Dataset. We obtain a total of 15,605 raw books and their corresponding meta data. To get high-quality fictional content, we remove books using the following heuristic rules: (i) short books whose size is less than 10KB; (ii) books that contain HTML code; (iii) books that are in the epub format (an e-book file format); (iv) books that are not in English; (v) books that are in the “Non-Fiction” genre; (vi) books that are in the “Anthologies” genre; (vii) books that are in the “Graphic Novels & Comics” genre. Since most books contain book information, author information, and some nonfictional content at the beginning and end of the book, we use regular expressions to match the term “Chapter” to locate the chapter title. Only the contents between the first chapter title and the last chapter title are kept. The last chapter is also removed as there are no certain boundaries to identify the story ending. Books whose chapter titles are unlocatable are also removed. After removing all the unqualified books, a total of 4,794 books were used in our dataset. We transliterate all non-ASCII characters into ASCII characters using Unidecode (https://pypi.org/project/Unidecode/) to fulfill the requirement of Open-SESAME (Swayamdipta et al., 2017). Open-SESAME is then used to parse the semantic frames for each sentence.
The books are split into training/validation/test sets following a 70/10/20 split, resulting in 3, 357, 479, and 958 books, respectively. To measure the effect of frame representation for different context lengths, we vary the story block length, using 5, 10, 20, 50, 100, 150, 200, 300, 500, and 1,000 sentences. When creating instances, we first split a book into story blocks with the specified length and extract all the consecutive two story blocks as instances when context window size (see Figure 1) is set to 1. The IDF of the semantic frame is then computed over the story blocks using all the training sets. Combining with the TF value in each story block, we convert story blocks into frame representations. We use scikit-learn's implementation (Pedregosa et al., 2011) of TF-IDF but with a slight modification on IDF: Scikit-learn uses  $idf(t) = \log\left(\frac{n}{df(t) + 1}\right)$  to compute a smoothing IDF, but we use  $idf(t) = \log\left(\frac{n}{df(t)}\right)$ . The detailed statistic information is shown in Table 1.
CODA-19 Dataset. We envision a broader definition of "creativity" in writing and attempt to apply story arc prediction technologies to the do
Block Size	5	10	20	50	100	150	200	300	500	1000
# Words Mean	71.7	143.5	286.9	717.2	1433.9	2149.8	2865.3	4293.7	7142.5	14212.3
# Frames Mean	17.5	35.0	69.9	174.5	348.6	522.1	695.4	1040.7	1727.3	3417.2
# Events Mean	10.0	20.0	39.9	99.8	199.4	298.9	398.2	596.4	991.2	1967.1
# Train	3,744,948	1,869,947	932,464	369,941	182,479	119,967	88,720	57,455	32,469	13,749
# Valid	574,840	287,054	143,166	56,838	28,073	18,466	13,672	8,881	5,035	2,166
# Test	1,054,816	526,687	262,625	104,198	51,396	33,776	24,987	16,178	9,138	3,861
Table 1: The statistic of Bookcorpus dataset in ten different story block lengths. We use Open-Sesame to parse the semantic frame for each sentence. The Events represents the SVO tuples (Martin et al., 2018).
Block Size	1	3	5
# Words Mean	26.3	77.3	124.7
# Frames Mean	6.0	17.5	27.6
# Events Mean	1.2	3.5	5.6
# Train	48,489	9,858	2,739
# Valid	5,615	1,146	334
# Test	5,238	1,047	287
Table 2: The statistic of CODA-19 dataset in three different story block lengths. We use Open-Sesame to parse the semantic frame for each sentence. The Events represents the SVO tuples (Martin et al., 2018).
mains outside novels, for example, scholarly articles. As an earlier exploration, we choose to use a smaller set of human-annotated abstracts (CODA19 (Huang et al., 2020b)) rather than machine-extracted full text (CORD-19 (Wang et al., 2020a)) in our proof-of-concept study, avoiding formatting issues (e.g., reference format, parsing errors) and intensive data cleaning effort. The original CODA-19 dataset contains 10,966 human-annotated English abstracts for five different aspects: Background, Purpose, Method, Finding/Contribution, and Other. We remove sentences that are annotated as "Other," an aspect for sentences that are not directly related to the content (e.g., terminology definitions or copyright notices.) Abstracts that contain Unicode characters are also removed. A total of 7,962 abstracts are used in our dataset. We then use OpenSESAME to parse the semantic frames for each sentence. We adopt CODA-19's original split, where the training set, validation set, and testing set have 6,509,737, and 716 abstracts, respectively. Three different lengths of story block are used: 1, 3, and 5. We then create instances and compute TF-IDF as described above. Table 2 shows the details.
We implement two naive baselines, an information retrieval baseline, two machine learning baselines, two deep learning baselines, an existing model and
a text generation baseline.
Replay Model. For each instance, the replay model takes the frame representation in the  $n$ -th story block as the prediction, i.e., the same frames will occur again.
Prior Model. The prior model computes the mean of the frame representation over the training set and uses it as the prediction for all the testing instances.
Information Retrieval with Frame Representation. For each instance, the information retrieval model searches for the most similar story block in the training set and takes the frame representation from its next story block as the prediction. In this setting, we adopt the cosine similarity on frame representations to measure the story similarity. For block size 5 in the Bookcorpus dataset, there are around 3.7 million instances in the training set, which is infeasible to finish.
Random Forest with Frame Representation. The foregoing story block's frame representation is used as the feature for prediction. We use scikit-learn's implementation of Random Forest Regressor (Pedregosa et al., 2011) with a max depth of 3 and 20 estimators. For block sizes that have more than one million training instances (5 and 10 in the Bookcorpus dataset), we randomly sample one million instances to train the model.
LGBM with Frame Representation. This is the same as the previous setup but trained using the LGBM Regressor model (Ke et al., 2017) with the max depth 5, the number of leaves 5, and the number of estimators 100. For block sizes that have more than one million training instances (5 and 10 in the Bookcorpus dataset), we randomly sample one million instances to train the model.
DAE with Frame Representation. This is the same as the previous setting but trained with the
Denoising Autoencoder architecture (Bengio et al., 2013). We feed in the foregoing story block's frame representation and output the frame representation for the follow-up story block. Thirty percent of the input is dropped randomly. The model is optimized using the cosine distance (1 - cosine similarity). Both the encoder and decoder are created via five dense layers with a hidden size of 512. We use a learning rate of 1e-5 and a batch size of 512 and train the model with the early stopping criteria of no improvement for 20 epochs. The best model on the validation set is kept for testing.
Event Representation Model (Event-Rep). We use Martin et al.'s event representation (2018) on the foregoing story block as the feature. An event tuple is defined as  $\langle s, v, o, m \rangle$ , where  $s$  is the subject,  $v$  is the verb,  $o$  is the object, and  $m$  is the verb modifier. We extract the dependency relation using the Stanza parser (Qi et al., 2020). Unlike Martin et al.'s implementation, where the empty placeholder  $\varnothing$  only replaces unidentified objects and modifiers, we find that the subjects can also be frequently missing in fiction books. For example, in ““Come out?” Zack asked. “Come out of where?””. In both cases here, the verb “come” does not have a subject. In “Fine, follow me,” “follow” has an object but does not have a subject. Therefore, we allow  $s$  to have a  $\varnothing$  placeholder in our implementation. All words are stemmed by NLTK (Loper and Bird, 2002).
After extracting the event representation, the sequence of event tuples in the foregoing story block is fed into a five-layer LSTM model (Hochreiter and Schmidhuber, 1997) to predict its follow-up frame representation. Note that the length of the event tuple sequence changes along with the block size. We thus set the maximum length of the sequence to the 95th percentile of the length in the training set. Sequences longer than the maximum length are left-truncated. The model is trained with a hidden size of 512, a learning rate of 3e-5, a dropout rate of 0.05, and a batch size of 64. We optimize the model using the cosine distance and apply the early stopping criteria of no improvement for three epochs. The best model on the validation set is kept for testing.
BERT. We take the pure text in the foregoing story block as the feature and apply the pretrained BERT model (Devlin et al., 2019). BERT has a token length limitation, so we set the maximum
length of tokens to 500 for Bookcorpus and 300 for CODA-19. Sentences with more than 500 tokens are truncated from the left. We take the [CLS] token representation from the last layer and add a dense layer on top of it to predict the follow-up frame representation. The model is trained with a learning rate of 1e-5 and a batch size of 32. We optimize the model using the cosine distance and apply the early stopping when no improvement for five epochs. The model with the best score on the validation set is kept for testing.
SciBERT (For CODA-19 Only). This is the same as the previous setting but is trained using the pretrained SciBERT model (Beltagy et al., 2019). We only test this approach on the CODA-19 dataset since it is from the scientific domain.
GPT-2 (For Bookcorpus Only). We also include a text generation model, GPT-2 (gpt2-xl) (Radford et al., 2019) with block sizes of 5, 10, 20, and 50. Since GPT-2 is computationally expensive, we conduct the experiment on a subset of the dataset, where 1,000 instances are randomly selected. We feed the text in the latest story block (n) into GPT-2 and generate 70, 150, 300, and 700 words for block sizes 5, 10, 20, and 50, respectively (5 sentences  $\approx$  70 words; 10 sentences  $\approx$  150 words in Bookcourpus, etc). For stories that exceed the GPT-2's word limit, we truncate the text from the left. Stories with block size larger than 100 would have more than 1400 words which by itself exceed the GPT-2's word limit. Generated stories are then parsed by Open-SESAME to extract the semantic frames and turned into frame representations as the predictions.
Table 3 and Table 4 show the experimental results. In this section, we summarize the main findings.
Predicting forthcoming semantic frames is remarkably challenging yet possible. Machine-learning models outperform the two naive baselines for different story lengths. In the Bookcorpus dataset, BERT performs the best for story blocks under 100 sentences, while LGBM performs the best for story blocks over 150 sentences. In the CODA-19 dataset, SciBERT performs the best for block sizes of 1 and 3, while DAE performs the best for a block size of 5. While the task is very challenging, these results shed light on the semantic frame forecast task. However, the improvement
Feature	Model	Block Size
5	10	20	50	100	150	200	300	500	1000
-	Replay Baseline	.0654	.0915	.1237	.1737	.2163	.2448	.2665	.3000	.3462	.4155
-	Prior Baseline	.2029	.2435	.2857	.3389	.3754	.3962	.4105	.4302	.4528	.4776
Frame	IR Baseline	-	.0631	.0851	.1290	.1841	.2085	.2262	.2536	.2859	.3321
Frame	Random Forest	.2037	.2448	.2881	.3427	.3807	.4025	.4184	.4402	.4659	.4966
Frame	LGBM	.2072	.2506	.2967	.3564	.3995	.4255	.4441	.4711	.5048	.5510
Frame	DAE	.2082	.2515	.2966	.3547	.3976	.4223	.4400	.4598	.4898	.5280
Event	Event-Rep	.2111	.2541	.2994	.3532	.3929	.4126	.4280	.4453	.4626	.4792
Text	BERT	.2172	.2611	.3073	.3637	.4012	.4229	.4371	.4559	.4779	.5057
Text	GPT-2	.0519	.0739	.0990	.1402	-	-	-	-	-	-
DELTA	.0142	.0176	.0216	.0249	.0257	.0293	.0336	.0409	.0520	.0734
Table 3: Baseline result for Bookcorpus dataset. BERT and Event-Rep work better in smaller block sizes, while models using frame representation perform better in larger block sizes. DELTA represents the difference between the best model and the prior baseline — an extremely simple but strong baseline — in that specific block size. The small value of DELTA shows that semantic frame forecast is challenging yet possible.
Feature	Model	Block Size
1	3	5
-	Replay Baseline	.0524	.0971	.1363
-	Prior Baseline	.1573	.2067	.2288
Frame	IR Baseline	.0315	.0601	.0752
Frame	Random Forest	.1581	.2081	.2278
Frame	LGBM	.1561	.2024	.2094
Frame	DAE	.1611	.2155	.2380
Event	Event-Rep	.1595	.2118	.2332
Text	BERT	.1660	.2202	.2353
Text	SciBERT	.1675	.2219	.2339
DELTA	.0102	.0152	.0092
Table 4: Baseline result for CODA-19 dataset. SciBERT performs the best in block size 1 and 3. Using the frame representation as the feature, DAE performs the best for block size 5. DELTA shows the difference between the best model and the prior baseline in that specific block size. The small value of DELTA shows that semantic frame forecast is challenging yet possible.
is not big, as shown in the DELTA row, suggesting that semantic frame forecast requires more investigation and understanding.
"Prior" is a robust and strong baseline. In both the Bookcorpus dataset and the CODA-19 dataset, the prior baseline is strong. As the story gets longer, the performance also increases. This suggests that when the story block gets bigger, more and more frames will constantly occur.
Replay baseline shows the relation of consecutive story blocks. The replay baseline assumes that the events that happen now will likely happen again shortly. The results in Table 3 and Table 4 partially confirm this assumption. To understand more about the assumption, we use the replay baseline to predict the  $n + i$ -th story block from the  $n$ -th story block in the Bookcorpus dataset. Figure 3