SlowGuess/NAACL_2022_OCR huggingface dataset:
Rujun Han $^{1}$  Hong Chen $^{2}$  Yufei Tian $^{3}$  Nanyun Peng $^{3}$
$^{1}$ University of Southern California  $^{2}$ Tokyo University
rujunhan@usc.edu; chen@nlab.cic.i.u-tokyo.ac.jp
yufeit@ucla.edu; violetpeng@cs.ucla.edu
Stories or narratives are comprised of a sequence of events. To compose interesting stories, professional writers often leverage a creative writing technique called flashback that inserts past events into current storylines as we commonly observe in novels and plays. However, it is challenging for machines to generate flashbacks as it requires solid understanding of event temporal order (e.g. feeling hungry  $\langle$  before  $\rangle$  eat, not vice versa), and the creativity to arrange storylines so that earlier events do not always appear first in narrative order. Two major issues in existing systems exacerbate the challenges: 1) temporal bias in pretraining and story datasets that leads to monotonic event temporal orders; 2) lack of explicit guidance that helps machines decide where to insert flashbacks. We propose to address these issues using structured storylines to encode events and their pair-wise temporal relations ( $\langle$  before  $\rangle$ ,  $\langle$  after  $\rangle$  and  $\langle$  vague  $\rangle$ ) as temporal prompts that guide how stories should unfold temporally. We leverage a Plan-and-Write framework enhanced by reinforcement learning to generate storylines and stories end-to-end. Evaluation results show that the proposed method can generate more interesting stories with flashbacks while maintaining textual diversity, fluency and temporal coherence. $^{1}$
Flashback is a popular creative writing technique that brings the readers from the present moment to the past via inserting earlier events in order to provide background or context of the current narrative (Pavis, 1998; Kenny, 2004; Gebeyehu, 2019). For example, in Figure 1a, the "GHOST" in Shakespeare's play Hamlet instruments a flashback by interrupting the main narrative and describing a historical event to the audience that Hamlet's father was killed by the current king rather than a snake.
GHOST: I am glad you're eager...Now listen, Hamlet. Everyone is told that a poisonous snake bit me when I was sleeping in the orchard. But that's a lie...You should know, my noble son, the real snake that stung your father is now wearing his crown. -- Hamlet, Act 1 scene 5
(a) A flashback example from William Shakespeare's famous play Hamlet (in plain English). Red text indicates past events.
Story (1): [e1] John lost consciousness when ambulance arrived.  
[e2] He woke up in the hospital in the morning.
Story (2): [e1] John woke up in the hospital in the morning.  
[e2] He lost consciousness when ambulance arrived.
(b) Two-sentence stories with the same event temporal order but different narrative order. The second one with a flashback is intuitively more interesting than the first one.
Figure 1: (a) flashback (b) temporal v.s. narrative order.
Flashback, by manipulating the event temporal orders in narrative structure, can arouse readers' emotions such as surprise, suspense, and curiosity (Brewer and Lichtenstein, 1981, 1982). These emotions stimulate readers' interests and eventually contribute to the satisfaction of reading (Tan, 1996; Alwitt, 2002), which improves the interest level of a story. The example in Figure 1a injects historical events in the middle of the narrative. This arrangement of events can surprise readers and therefore, makes the story more interesting than a straightforward storyline where the past events are shown in the beginning.
Similarly, consider the pair of two-sentence stories in Figure 1b. Both stories are composed of the same events with the temporal order "lost consciousness" (before) "woke up in the hospital." In Story (1), seeing [e1], readers can make a relatively easy educated guess of [e2], but it is more subtle in Story (2) as there are many different ways to end up in a hospital. By showing the ending event first, the flashback in Story (2) creates suspense that makes the following sentences less predictable, and thus arouses readers' curiosity and makes the reading more interesting.
While human writers are capable of maneuver
$^{1}$ Code, data and trained models are available here: https://github.com/PlusLabNLP/flashback_gen
Human Language Technologies, pages 1450 - 1470
ing event temporal orders to compose coherent and interesting stories, it remains challenging for machines. The challenge is partially attributed to data bias. Ning et al. (2018a) shows that the pattern in Story (1) is dominant in human-written texts, where neighboring events with  $\langle before\rangle$  temporal relations (i.e., narrative order indicates temporal order) occur  $60 - 70\%$  of the time. This is also manifested in our experiments with vanilla language models amplifying this ratio and producing more than  $80\%$ $\langle before\rangle$  relations for neighboring events in the generated stories. Furthermore, current state-of-the-art story generation systems that incorporate event representations usually assume event temporal order follows narrative order (Goldfarb-Tarrant et al., 2020; Lin et al., 2021). There are no explicit prompts in these systems that help determine when flashback should be used, leaving models to produce dull stories consisting of event sequences with monotonic  $\langle before\rangle$  relations.
To facilitate more effective flashback, we propose to incorporate temporal prompts in an end-to-end story generation framework inspired by the Plan-and-Write paradigm (Yao et al., 2019; Xu et al., 2020; Goldfarb-Tarrant et al., 2020), where machines first learn to plot a storyline, and then generate the story based on the storyline. Specifically, we encode predefined event temporal prompts in structured storylines. As the bottom block of Figure 2 shows, a structured storyline contains two components: 1) event representations where an event trigger ("grabbed") and two arguments ("she" and "the dog") are extracted from the original story sentences; and 2) temporal prompts: the temporal order between neighboring events, e.g. event 1: ("she", "grabbed", "the dog") is  $\langle after\rangle$  event 2: ("white snow", "blanketed", "the ground"). By training our storyline generation model with these predefined pair-wise temporal relations, models capture how neighboring events are temporally related to each other; while during storyline decoding, supplying predefined temporal prompts can guide models to generate reasonable narratives with desirable event temporal orders.
Prior works (Fan et al., 2019; Goldfarb-Tarrant et al., 2020) build the storyline and story models separately, which creates a discrepancy where gold storylines are used during training, but predicted storylines are used during inference. To mitigate this training-inference discrepancy, we leverage reinforcement learning (RL) to train our systems end
to-end. It enables the story model to train on generated storylines and updates the storyline model with the feedback from the story model. Our experimental results show that the RL-based models can leverage temporal prompts more effectively, resulting in more effective flashback generation and more interesting stories.
We summarize the contributions of this paper as follows: 1) To facilitate effective flashback, we propose to leverage structured storylines with temporal prompts to arrange events in story generation. 2) We integrate reinforcement learning in our story generation pipeline, which can help models better leverage temporal prompts. 3) We test our framework on two open-domain story datasets and show more effective flashbacks and increased interest level while maintaining fluency and temporal commonsense in the generated stories. To our best knowledge, this is a pioneering study on flashbacks in neural story generation.
2 Background and Task Definitions
In this section, we describe the key components: events and temporal prompts in our proposed structured storylines, and then define the Plan-and-Write generation task.
Event Representation. Following the definitions of ACE (2005), we define an event as a trigger word and its arguments. In this work, we simplify the representation by leveraging semantic role labeling (SRL) tools (Shi and Lin, 2019) to parse two arguments as shown in Figure 2. We only consider one event per story sentence and denote the  $k$ -th event in story  $i$  as  $e_{i,k}$ . We leave more complicated representations for future study.
Temporal Prompts. Let  $\boldsymbol{r}_i = \{r_{i,k}\}$  denotes the set of temporal relations between the  $k$ -th and the  $(k + 1)$ -th event in story  $i$ . If  $k$  indexes the last event,  $r_{i,k}$  is not defined. Following the event relation definition of (Ning et al., 2018b), we use events' start time to evaluate temporal order.
Structured Storyline. Figure 2 provides a storyline consisting of five event representations extracted from our data. More formally, let  $S_{i} = \{e_{i,1}, e_{i,2}, \ldots, e_{i,k}, \ldots, e_{i,n}\}$  indicates a storyline with  $n$  events. Encoding temporal prompts,  $S_{i}$  becomes  $S_{i}^{r} = \{e_{i,1}, r_{i,1}, e_{i,2}, r_{i,2}, \ldots, e_{i,k}, r_{i,k}, \ldots, e_{i,n}\}$ . Note that in this work,  $r_{i}$  is provided as predefined prompts rather than predicted as  $e_{i,k}$ .
Story. Our ultimate goal is to generate flashbacks in stories. We denote the story associated with the
Figure 2: An illustration of our overall model. Here we use the first sentence of the story (and its associated event representation) as input  $\mathbf{x}$ . The upper block shows the vanilla implementation of the Plan-and-Write workflow. The bottom block is our core novel design by leveraging temporal prompts in structured storylines to generate flashbacks. For illustration purposes, we re-order the triggers and arguments, and storylines are ground-truths (i.e. not predicted by models). Our final model uses reinforcement learning to implement end-to-end training.
storyline  $\mathcal{S}_i$  as  $\mathcal{V}_i$ .
Plan-and-Write is a two-stage framework that first generates storyline  $\hat{S}_i$  given some input  $x$  (e.g. title, leading sentence), and then generate  $\hat{\mathcal{V}}_i$  based on  $\hat{S}_i$ . Again,  $r_{i,k}$  are given as predefined prompts whereas  $e_{i,k}$  are to be predicted as part of the storyline generation shown in Figure 2.
3 Framework for FlashBack Generation
In this section, we first provide an overview of the Plan-and-Write story generation system and introduce a vanilla version of the end-to-end training method. Then we describe the details of our key contribution of leveraging event temporal prompts to generate flashbacks. After that, we discuss pretraining structured storylines with self-labeled data and incorporating reinforcement learning to jointly train our end-to-end models.
3.1 Plan-and-Write Models
In order to provide better explainability and controllability over the machine generated stories, recent research efforts (Yao et al., 2019; Xu et al., 2020; Goldfarb-Tarrant et al., 2020) explore dividing story generation into two steps: 1) from input or prefix,  $x$ , we first produce a storyline,  $S_{i}$ ; 2) based on the storyline, we generate a story,  $\mathcal{V}_{i}$ . We describe the details below.
Storyline Model. Let  $\alpha$  denote the parameters of the storyline model, per sample training loss can be computed as  $\mathcal{L}_{\alpha} = -\log p(S_{i}|\boldsymbol{x}_{i},\alpha)$ .
Story Model. Let  $\beta$  denote the parameters of the story model, per sample training loss can be computed as  $\mathcal{L}_{\beta} = -\log p(\mathcal{Y}_i|\boldsymbol{x}_i,\mathcal{S}_i,\beta)$ .
Inference. Note that  $S_{i}$  above is the gold storyline extracted from  $\mathcal{V}_{i}$ . At the inference time, we do not have  $S_{i}$ , and have to replace it with  $\hat{S}_i$ , the predicted storyline. This results in a discrepancy between the training and inference time.
End-to-end Training. Instead of using gold storyline  $\mathcal{S}_i$  to train a story model, we can take  $\hat{S}_i$  as its input. Now the per sample training loss for the story model becomes  $\mathcal{L}_{\theta} = -\log p\left(\mathcal{Y}_i|\boldsymbol{x}_i,\hat{S}_i,\theta\right)$  where  $\theta$  indicates the end-to-end story model parameters. End-to-end training can alleviate the gap between the training and inference time, and potentially lead to more consistent stories<sup>2</sup>.
3.2 Structured Storyline Construction
As Figure 2 shows, for a story sentence, we first use the SRL tool to parse its trigger  $t_{i,k}$  and two arguments  $a_{i,k}^{1}$  and  $a_{i,k}^{2}$ . We then convert this representation into a textual form: "  $t_{i,k}$ ;  $a_{i,k}^{1}$ ;  $a_{i,k}^{2} \langle eoe \rangle$ ", where “;” separates two event components, and  $\langle eoe \rangle$  indicates event ending. For example, the parsed  $t_{i,k}$ ,  $a_{i,k}^{1}$  and  $a_{i,k}^{2}$  in the story sentence “she grabbed the dog and ran outside” are “grabbed,” “she” and “the dog” respectively. They are concatenated into a final textual representation as “grabbed; she; the dog  $\langle eoe \rangle$ ”.
Depending on the experimental setup, we may use no or only the leading event as input,  $x$ . Inspired by the mask prediction design in Devlin et al. (2019); Liu et al. (2019); Lewis et al. (2020), we represent the remaining missing events in the inputs as “ $\langle \text{mask} \rangle$  ;  $\langle \text{mask} \rangle$  ;  $\langle \text{mask} \rangle$  ;  $\langle \text{eoe} \rangle$  ” where  $\langle \text{mask} \rangle$  indicates either event trigger word or argu
$^{2}$ End-to-end training may have small coverage of the generated events in stories, which we address in Appendix F.
ments to be predicted by the storyline model.
3.3 Temporal Prompt Encoding
Temporal prompts are used to generate flashbacks. As we mentioned in Section 2, we encode a sequence of predefined event temporal prompts  $r_i = \{r_{i,k}\}$  in storyline for  $k \in \{1, n-1\}$  to help models determine whether the next event mention (in narrative order) should start earlier or later than its preceding event mention. We use temporal relation extraction tools to annotate all  $r_{i,k}$  in our experimental data. Specifically, we use ECONET (Han et al., 2021b) finetuned on the MATRES dataset (Ning et al., 2018b) to predict the temporal relation between neighboring events. $^3$  The context and the locations of a pair of event trigger words are fed into ECONET to predict their temporal order. The temporal prompt set consists of  $\langle before \rangle$ ,  $\langle after \rangle$  and  $\langle vague \rangle$  (capturing undetermined temporal order), and are fixed in  $S_i^r$ . Note that  $\langle vague \rangle$  indicates undetermined temporal order due to the ambiguity of the context (Cassidy et al., 2014; Ning et al., 2018b) and it does not suggest the context is poor or the relations are wrong. As shown in Figure 2, we replace the end-of-event token  $\langle eoe \rangle$  with temporal prompts in storylines, except for the last event which does not have a next event. With the prompt-augmented storylines,  $S_i^r$ , we can re-write the storyline loss as  $\mathcal{L}_{\alpha} = -\log p(\mathcal{S}_i^r | x_i, r_i, \alpha)$ , and story loss as  $\mathcal{L}_{\theta} = -\log p(\mathcal{Y}_i | x_i, \hat{\mathcal{S}}_i^r, \theta)$ .
3.4 Storyline Pretraining
Using intermediate pretraining to adapt original pretrained language models has been shown to be effective for a variety of downstream tasks such as information extraction (Joshi et al., 2020), question-answering (Khashabi et al., 2020; Garg et al., 2020) and commonsense reasoning (Zhou et al., 2021). To capture more diverse event sequences and facilitate better story generation, we explore pretraining
storyline model with SRL extracted storyline from BookCorpus dataset (Zhu et al., 2015), and use learned  $\alpha$  to initialize storyline models.
3.5 RL-based End-to-end Model
The end-to-end model described in Sec. 3.1 allows the story model to train with the generated storylines and hence alleviate the gap between training and inference. However, this workflow still lacks a mechanism that enables the storyline model to adjust with the feedback from the story model. The challenges of training storyline and story models jointly originate from decoding storylines as inputs for the story model, which involves nondifferentiable token selections. Thus, the final loss  $L_{\theta}$  cannot be directly back-propagated into the storyline model.
To overcome this barrier, we adopt reinforcement learning (RL), specifically, the REINFORCE algorithm (Williams and Peng, 1991) in our end-to-end training. Let  $R_{i} = R(\boldsymbol{x}_{i},\boldsymbol{r}_{i})$ . The expected reward with respect to the storyline model can be written as  $\mathbb{E}_{\alpha}[R_i] = \mathbb{E}[R_i\cdot \log (p(S_i^r |\boldsymbol{x}_i,\boldsymbol{r}_i,\alpha))]$ . The gradient to update the storyline model is  $\nabla J_{\alpha} = \mathbb{E}[R_i\cdot \nabla \log (p(S_i^r |\boldsymbol{x}_i,\boldsymbol{r}_i,\alpha))]$ , which can be approximated with sampling techniques. Motivated by Xu et al. (2018), we use negative loss of the story model to construct rewards, that is,  $R = -\mathcal{L}_{\theta}$ . In other words, smaller loss from the story model is associated with larger reward. Algorithm 1 summarizes the overall method.
In this section, we start by describing our research objectives, then we describe our data, evaluation metrics, experimental designs and implementation details aiming to achieve these objectives.
The overall research objective is to measure the impact of using temporal prompts in structured storylines. Specifically, can  $\langle \text{after} \rangle$  successfully induce flashbacks? If so, does that contribute to the interest level of the generated stories while maintaining the overall quality of the texts?
4.1 Datasets.
ROCStories (Mostafazadeh et al., 2016a) and WritingPrompts (Fan et al., 2018) are our experimental datasets. We ensured all reported results using the same test data as the baseline systems (Xu et al., 2020) and (Goldfarb-Tarrant et al., 2020). For
<sup>5</sup>We do not use baseline reward as we found this simple reward design works effectively in our experiments.
3The ECONET tool is available here: https://github.com/PlusLabNLP/ECONET.
pretraining data, we use BookCorpus (Zhu et al., 2015). Appendix B shows all details of data splits and pre-processing process.
4.2 Temporal Prompts Constructions
ECONET was finetuned three times with different random seeds, so we take the consensus vote from three models. If there is any disagreement, we label the temporal order as  $\langle \text{vague} \rangle$ . We benchmark ECONET's annotation performances in Appendix G, which shows it provides highly accurate temporal relations. For human evaluations specifically, we consider two prompt settings in order to gauge different impacts of  $\langle \text{after} \rangle$ . 1) for ROCStories, all structured storylines consist of exactly four predefined temporal prompts created following Sec 3.3. We randomly sample stories with one  $\langle \text{after} \rangle$  prompt from the test data. We will show later in the analysis that vanilla language models would generate more than  $80\%$  event pairs with  $\langle \text{before} \rangle$  relations for ROCStories;  $\langle \text{after} \rangle$  prompt should bring this ratio down if it is effective. 2) for WritingPrompts, since the number of events is not fixed, we randomly sample test stories generated with  $\langle \text{after} \rangle$  prompts for evaluation.
We use automatic metrics to evaluate the textual quality of stories. We report Ref. PPL: reference stories' perplexity in our models and Gen. PPL: generated stories' perplexity scored by GPT-2 (Radford et al., 2019). For diversity, we report Distinct Ratio (%): overall vocabulary:token number ratio. We also report standard BLEU-3 and ROUGE $_L$ .
4.4 Human Evaluation Metrics
We rely on human annotators to analyze the effectiveness of flashback generations. We request 18 MTurkers who succeeded in our previous annotation tasks (Han et al., 2021a) to evaluate stories produced by our compared models. We host a small qualification round followed by a large annotation task. Only 10 workers are qualified, and we only consider their annotations. Eventually, we collect 106 and 77 sets of valid annotations for ROCStories and WritingPrompts.
Temporal diversity. The dominance of  $\langle before\rangle$  relation in our data can make models biased toward generating stories with more  $\langle before\rangle$  relations. Therefore, we are interested to see how inserting an  $\langle after\rangle$  prompt can help increase the percentage of non- $\langle before\rangle$  event relations in the generated stories. Let  $\hat{R}_r$  indicate the percentage of a
particular relation annotated by MTurkers. We calculate the entropy of the set  $\{\hat{R}_r\}, \forall r \in \{\langle before\rangle, \langle after\rangle, \langle vague\rangle\}$  to measure temporal diversity.
Accuracy measures the percentage of  $\langle after\rangle$  being correctly incorporated in the generated stories labeled by human annotators. We used a relaxed version by counting annotated  $\langle vague\rangle$  as correct too, as  $\langle vague\rangle$  can potentially be  $\langle after\rangle$ . Both accuracy and temporal diversity can show the effectiveness of generating flashbacks using  $\langle after\rangle$ .
Temporal coherence indicates if the event sequence in a generated story aligns with an annotator's temporal commonsense. 1 and 0 correspond to yes and no, respectively.
Interest level. Precisely defining interest level is difficult as it is a broad concept. So we focus on the unexpectedness component of cognitive interest. As pointed out by Behrooz (2019), unexpectedness can be further explained as how predictive an event is, which is closely related to flashback generation. Therefore, we define an interesting event as 1) being unexpected or surprising and 2) being logical according to the context and general commonsense.<sup>7</sup>
For the compared models, we ask annotators to provide ranks between 1 to K for the generated stories, with K indicating the most interesting story and 1 indicating the least interesting one. We encourage workers to provide different scores for all compared stories, but equal scores are allowed. The max score K depends on the number of compared models, 5 for ROCStories and 4 for WritingPrompts. We provide detailed instructions in the interface shown in the appendix. Crucially, interest level is separately annotated from other metrics and we ensure annotators do not see the same set of stories in both tasks.
4.5 Compared Models
Baselines. Xu et al. (2020), denoted as MEGA-TRON, is chosen as the baseline as it outperforms previous systems such as Guan et al. (2020) on ROCStories. We also compare with TEMPORAL-BART (Lin et al., 2021) as it is pretrained with temporal ordering and event infilling tasks. For WritingPrompts, we compare with CONTENTPLANNING (Goldfarb-Tarrant et al., 2020) as it also
$^6$ interface with detailed instructions and examples can be found in Figure 7 of the appendix.
the second definition is not identical to the temporal coherence above. Events contradicting general commonsense can still be temporally coherent (see Figure 7) for examples.
Models	Automatic Evaluation	Human Evaluation
Ref. PPL(↓)	Gen. PPL(↓)	Distinct Ratio(↑)	BLEU (↑)	ROUGE(L) (↑)	Temporal Diversity(↑)	Accuracy (↑)	Temporal Coherence (↑)	Interest Level (↑)
TEMPORAL-BART	24.65	19.47	4.10	5.01	19.12	-	-	-	-
MEGATRON	-	34.14	4.57	2.57	15.23	1.21*	-	0.78	2.69
VANILLA-GEN	27.30	19.29	3.99	5.13	19.29	0.88	-	0.88	2.95
+ STRUCTURED-PROMPT	22.85	19.94	4.09	5.07	19.39	1.09	55.75	0.82	3.03
+ PRETRAINED	21.16	19.25	4.01	5.06	19.44	1.07	52.21	0.84	2.96
+ RL	15.45	19.42	4.17	5.20	19.49	1.14	56.64	0.86	3.06