SlowGuess/NAACL_2022_OCR huggingface dataset:
Rujun Han, I-Hung Hsu, Mu Yang, Aram Galstyan, Ralph Weischedel, and Nanyun Peng. 2019a. Deep structured neural network for event temporal relation extraction. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 666-106, Hong Kong, China. Association for Computational Linguistics.
Rujun Han, Mengyue Liang, Bashar Alhafni, and Nanyun Peng. 2019b. Contextualized word embeddings enhanced event temporal relation extraction for story understanding. In 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT 2019), Workshop on Narrative Understanding.
Rujun Han, Qiang Ning, and Nanyun Peng. 2019c. Joint event and temporal relation extraction with shared representations and structured prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 434-444, Hong Kong, China. Association for Computational Linguistics.
Rujun Han, Xiang Ren, and Nanyun Peng. 2021b. ECONET: Effective continual pretraining of language models for event temporal reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5367-5380, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Rujun Han, Yichao Zhou, and Nanyun Peng. 2020. Domain knowledge empowered structured neural net for end-to-end event temporal relation extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5717-5729, Online. Association for Computational Linguistics.
Marissa Hoek, Mariet Theune, and Jeroen Linssen. 2014. Generating game narratives with focalization and flashbacks. volume 10.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
J. F. Kenney and E. S. Keeping. 1965. Ols regression via normal equations. In Mathematics of Statistics, 3 edition, chapter 15, pages 252-285. Princeton, NJ.
Kenny. 2004. Teaching Tv Production in a Digital World: Integrating Media Literacy. Libraries UnLtd Incorporated.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1896-1907, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Manling Li, Sha Li, Zhenhailong Wang, Lifu Huang, Kyunghyun Cho, Heng Ji, Jiawei Han, and Clare R. Voss. 2021. Future is not one-dimensional: Graph modeling based complex event schema induction for event prediction. arXiv preprint arXiv:2104.06344.
Shih-Ting Lin, Nathanael Chambers, and Greg Durrett. 2021. Conditional generation of temporally-ordered event sequences. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7142-7157, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint, arXiv:1907.11692.
Mingyu Derek Ma, Jiao Sun, Mu Yang, Kung-Hsiang Huang, Nuan Wen, Shikhar Singh, Rujun Han, and Nanyun Peng. 2021. Eventplus: A temporal event understanding pipeline. In 2021 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Demonstrations Track.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016a. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849, San Diego, California. Association for Computational Linguistics.
Nasrin Mostafazadeh, Alyson Grealish, Nathanael Chambers, James Allen, and Lucy Vanderwende. 2016b. CaTeRS: Causal and temporal relation scheme for semantic annotation of event structures. In Proceedings of the Fourth Workshop on Events, pages 51-61, San Diego, California. Association for Computational Linguistics.
Qiang Ning, Zhili Feng, and Dan Roth. 2017. A structured learning approach to temporal relation extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1027-1037, Copenhagen, Denmark. Association for Computational Linguistics.
Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020. TORQUE: A reading comprehension dataset of temporal ordering questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1158-1172, Online. Association for Computational Linguistics.
Qiang Ning, Hao Wu, Haoruo Peng, and Dan Roth. 2018a. Improving temporal relation extraction with a globally acquired statistical resource. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 841-851, New Orleans, Louisiana. Association for Computational Linguistics.
Qiang Ning, Hao Wu, and Dan Roth. 2018b. A multi-axis annotation scheme for event temporal relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1318-1328, Melbourne, Australia. Association for Computational Linguistics.
Shantz Pavis. 1998. Dictionary of the Theatre: Terms, Concepts, and Analysis. University of Toronto Press.
Nanyun Peng, Marjan Ghazvininejad, Jonathan May, and Kevin Knight. 2018. Towards controllable story generation. In *NAACL Workshop*.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Peng Shi and Jimmy J. Lin. 2019. Simple bert models for relation extraction and semantic role labeling. ArXiv, abs/1904.05255.
Pradyumna Tambwekar, Murtaza Dhuliawala, Lara J. Martin, Animesh Mehta, Brent Harrison, and Mark O. Riedl. 2019. Controllable neural story plot generation via reward shaping. In Proceedings of the 2019 International Joint Conference on Artificial Intelligence.
Bowen Tan, Zichao Yang, Maruan Al-Shedivat, Eric P Xing, and Zhiting Hu. 2021. Progressive generation of long text with pretrained language models.
Ed S. Tan. 1996. Emotion and the structure of narrative film: Film as an emotion machine. Mahwah, NJ. Erlbaum Associates.
Ronald J. Williams and Jing Peng. 1991. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3:241-268.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame,
Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Hui-Yin Wu, Michael Young, and Marc Christie. 2016. A Cognitive-Based Model of Flashbacks for Computational Narratives. In AIIDE 2016 - 12th Annual AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, pages 239-245, San Francisco, United States.
Jingjing Xu, Xuancheng Ren, Yi Zhang, Qi Zeng, Xiaoyan Cai, and Xu Sun. 2018. A skeleton-based model for promoting coherence among sentences in narrative story generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4306-4315, Brussels, Belgium. Association for Computational Linguistics.
Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar, and Bryan Catanzaro. 2020. MEGATRON-CNTRL: Controllable story generation with external knowledge using large-scale language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2831-2845, Online. Association for Computational Linguistics.
Lili Yao, Nanyun Peng, Weischedel Ralph, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan-and-write: Towards better automatic storytelling. In The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19).
Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. 2019. Bridging the gap between training and inference for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4334-4343, Florence, Italy. Association for Computational Linguistics.
Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. "going on a vacation" takes longer than "going for a walk": A study of temporal commonsense understanding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3363-3369, Hong Kong, China. Association for Computational Linguistics.
Wangchunshu Zhou, Dong-Ho Lee, Ravi Kiran Selvam, Seyeon Lee, Bill Yuchen Lin, and Xiang Ren. 2021. Pre-training text-to-text transformers for concept-centric common sense.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards
story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV).
Input prompt	[s1] nina needed blood tests done.
langle before\ranglelangle after\ranglelangle before\rangle
baseline	[s2] she was very nervous.
[s3] she went to the local clinic.
[s4] they told her about her bad breath.
[s5] she was relieved her nerves were n’t too much to handle.
VANILLA	[s2] she was terrified of needles.
[s3] she went to the doctor.
[s4] the doctor gave her a needle.
[s5] nina was relieved.
RL	[s2] she was afraid to go to the hospital.
[s3] she had never been to a hospital before.
[s4] but when she got there, she was relieved.
[s5] she felt so relieved!
Input prompt	[s1] mickey was always a fan of webcomics.
langleafter⟩ ⟨before⟩ ⟨before⟩ ⟨before⟩
baseline	[s2] he loved writing his own comics one night.
[s3] then he had a great idea.
[s4] he created a story called the superman story.
[s5] the story was real as a kids cartoon.
VANILLA	[s2] one day, he decided to make a webcomic himself.
[s3] he created a website and sent it to a publisher.
[s4] the publisher loved it and contacted mickey.
[s5] mickey was happy to receive a publishing deal.
RL	[s2] he had been writing for years.
[s3] one day, mickey saw an ad for a webcomic.
[s4] he signed up for it.
[s5] mickey was able to make a lot of money from his work.
Input prompt	[s1] annahad just gotten her hair permed. <before> <before> <after> <before>
baseline	[s2] she was nervous about it. [s3] she was too nervous to do it. [s4] she tried to cut hair to look better. [s5] when she was done it looked great !
VANILLA	[s2] she was excited to see how it looked. [s3] but then she saw that her hair had grown out. [s4] annaworried! [s5] she had to go to the salon to get it fixed.
RL	[s2] she was excited to see how it looked. [s3] but when she went to apply it, it was terrible. [s4] her hair had grown way too long! [s5] annahad to go to the salon to get her hair fixed.
Table 6: Additional generated stories for ROCStories.
A Additional Generated Stories
Please see Table 6 and Table 11 for more examples on ROCStories and WritingPrompt respectively.
ROCStories (Mostafazadeh et al., 2016a) contains 5-sentence stories. Following (Xu et al., 2020), we split data into 88,344/4,908/4,909 for train/validation/test sets.
WritingPrompt (Fan et al., 2018) contains 30,335 pairs of prompts and stories. With an average of more than 700 words per story, Writing Prompts are much longer than ROCStories. These stories are also much less structured as some dialogues and short phrases may be included. To speed up our experiments, we select stories with a maximum of 500 words, resulting in a total number
of 96,488 training and 5,784 validation prompt-story pairs, respectively. For the test set, we use the 1,000 prompt-story pairs provided by the baseline paper (Goldfarb-Tarrant et al., 2020) for reporting automatic evaluation results.
Pretraining Data. As we mention in Section 3.4, we pretrain storyline models for ROCStories. To be consistent with ROCStories inputs, we divide BookCorpus data (Zhu et al., 2015) into 5 consecutive sentences and filter out those with noisy tokens. We randomly select 1 million such 5-sentence text spans and extract their storylines following Section 3.2.
C More Details for Evaluation Metrics
Automatic evaluation metrics are used to measure textual quality of stories. We report 1) Ref. PPL: reference stories' perplexity in a model; 2) Gen. PPL: generated stories' perplexity scored by GPT-2 (Radford et al., 2019), i.e. we feed the generated stories into GPT-2 to compute perplexity scores. For diversity scores, we found our models implemented by Huggingface (Wolf et al., 2020) can achieve nearly 0 Repeat-3 and  $100\%$  Distinct-3 scores, so we follow Goldfarb-Tarrant et al. (2020) to compute the overall vocabulary:token number ratio, which we denote as 3) Distinct Ratio  $(\%)$ . We also report standard 4) BLEU-3 and 5) ROUGE $_L$  scores.
D More Details for Baseline Models
MEGATRON-CNTRL Xu et al. (2020), denoted as MEGATRON for brevity, is chosen as the baseline as it outperforms previous systems such as Guan et al. (2020) on ROCStories. We do not perform delexicalization that replaces names and entities with [MALE], [FEMALE] and [NEUTRAL] tokens, as we found our models work well by recognizing names and entities. When conducting evaluations, we try our best to map these special tokens back to their original texts by using the given first sentence. For rare undetermined cases, we manually examine the generated stories and swap in names or entities that make the most sense in the context. To be fair, we compare with the 124M-parameter version.
ContentPlanning (Goldfarb-Tarrant et al., 2020) is chosen as the baseline for WritingPrompt, as it also adopts the Plan-and-Write workflow as well as structured event representations. However, their models are based on BART-large and do not train
with an end-to-end framework. They use  $65\%$  of the original training data and also filter out samples with non-[WP] prompts. Our final training data is about 2/3 of theirs.
TemporalBART (Lin et al., 2021) is designed for two event temporal relation related tasks: temporal ordering and event infilling. Although TemporalBART does not tackle story generation directly, it encodes event temporal information via pretraining tasks. So we consider TemporalBART as another baseline model by initialing the storyline model with their parameters and training the STRUCTURED-PROMPT workflow on ROCStories. $^{8}$
E Reproduction Check List
We finetune BART-base. For ROCStories, hyperparameters are learning rate:  $5e^{-5}$ ; batch size: 10. We use 3 random seeds: (5,9998,20016) and report the average performances for all end-to-end models. For Writing Prompts, hyper-parameters are learning rate:  $1e^{-4}$ ; batch size: 64; gradient accumulation: 8.
For ROCStories, we were able to finetune on a single Nvidia GTX2020 GPU with 11G memory, and training time is 3-4 hours per epoch. For WritingPrompt, we have to use a much larger Nvidia A100 GPU with 40G memory, and the training time is 20 hours per epoch. We train all models for 10 epochs and save the model with the best evaluation perplexity. All reproduction details can be found in the separately submitted code.
F Perplexity and Event Coverage Trade-off
One caveat of using end-to-end training is that there is no guarantee that the generated events will appear in the final stories; whereas in two-stage models, the story model learns a mapping from reference storylines to stories, which leads to a higher coverage rate of the generated events. To provide a potential solution, we experiment with the mixture-training method proposed by Zhang et al. (2019),
$$
p = \frac {\mu}{\mu + \exp (e / \mu)}
$$
where  $p$  controls the ratio of reference storylines used in training and  $e$  is the training step. Here the larger the hyper-parameter  $\mu$ , the slower  $p$  decays to 0 as training proceeds.
Figure 4: Trade-off between perplexity scores and generated event (trigger) coverage.
In Figure 4, we show the trade-off between perplexity and  $\mu$  values by training our final RL-based models. When  $\mu$  is nearly zero, it corresponds to always using predicted storylines, and hence a smaller predicted event coverage rate in the final stories. When  $\mu$  gets larger, eventually making p close to 1, the training corresponds to always using gold storylines, which leads to a very high event coverage rate, but relatively poor perplexity (still much stronger than two-stage results). We leave the search for optimal  $\mu$  for future research.
CaTeRS	MATRES
A <Before> B	A <before> B
A <Identity> B	A <vague> B
A <Contains> B	A <before> B
A <Overlaps> B	A <before>, <after>, <vague> B
Table 7: Label mapping from CaTeRS to MATRES.  $\langle$ after $\rangle$  is ignored in CaTeRS by flipping event physical order.
G Benchmark Event Temporal Relation Annotations
The experimental results in the main text demonstrate the effectiveness of using temporal prompts. Here, we further show that the tool to produce temporal prompts, i.e. ECONET, provides reliable event temporal relation annotations. We benchmark ECONET's performances using CaTeRS (Mostafazadeh et al., 2016b), which annotates 4 types of temporal relations for event pairs in a small amount of ROCStories. However, CaTeRS's annotations are based on event time interval rather than event start time as used in MATRES, which ECONET is finetuned on.
In Tabel 7, we provide a mapping from CaTeRS's temporal relations to MATRES labels.
8More implementation details such as hyper-parameters and software can be found in the appendix.
The only non-unique mapping is  $\langle Overlap\rangle$ . In other words, when ECONET predicts  $\langle before\rangle$  for a CeTeRS sample  $\langle Overlaps\rangle$ , we have to manually examine whether it is correct or not. We found that when ECONET predicts  $\langle before\rangle$  for CeTeRS data, the precision rate is  $65.53\%$  due to a large amount of  $\langle Overlaps\rangle$  event pairs being predicted as  $\langle before\rangle$ . But we emphasize here that this low number is caused by label mismatch as shown in Table 7, which does not truthfully reflect the ECONET's accuracy.
To have a better understanding, we randomly selected 20 such pairs and manually examine their temporal relations in the context and found that  $90\%$  of such pairs are indeed correctly predicted by ECONET. Adjusting for this factor, the precision rate for the annotated  $\langle before\rangle$  relation would be  $92.07\%$ , indicating highly accurate predictions. We do not claim the final accuracy is  $92.07\%$ , but simply argue that the annotations provided by ECONET are helpful as our main experimental results demonstrate.
H Two-stage Model Results
Two-stage Model. As we mentioned in Section 3, another way to implement Plan-and-Write framework is to train storyline and story models separately with gold input and outputs, and replace story models' inputs with the storyline model's predictions during inference. We found this variant's performances fall far behind other compared models. So we do not use them in human evaluations and simply show their automatic evaluation results in Table 8.
Models	Ref. PPL(↓)	Gen. PPL(↓)	Distinct Ratio(↑)	BLEU (↑)	ROUGE(L) (↑)
ROCStories	197.97	27.15	4.26	2.98	16.39
WritingPrompt	76.69	39.43	8.12	0.00	6.24
I Storyline Model Results
Our primary goal is to improve temporal control in final stories. However, as aforementioned, the motivation for using reinforcement learning is to allow the storyline model to adapt together with the story model. In Table 9, we compare storyline prediction performances between the vanilla and the RL-based end-to-end models. These results suggest that our final framework combining structured storylines, temporal prompts, pretraining and
Table 8: Automatic Evaluation Results for the Two-stage models.
Models	Ref. PPL (↓)	BLEU (↑)	Distinct Ratio (↑)
Vanilla-Gen + RL	13.97	3.21	3.68
12.71	3.49	4.09
RL also helps storyline models fit reference storylines better, resulting in lower perplexity, higher overlapped scores and better diversity.
J Error Analysis
Table 10 shows examples for the temporal diversity in Section 6.
Table 9: Improvements of RL-based model against vanilla end-to-end model on ROCStories data.
Input 1 [s1] justin is a very charismatic man who loves women.
baseline	[s2] he always said that he wanted to be a man.
[s3] he thought that men looked bad.
[s4] justin decided to become a boy scout.
[s5] he is now a successful realtor.
predicted relations: <vague> <vague> <vague> <before>
RL	[s2] one day he met a beautiful woman and they fell in love.
[s3] justin decided he wanted to marry her.
[s4] he went to the mall and bought a ring.
[s5] he proposed to her and she said yes.
predicted relations: <before> <before> <before> <before>
Input 2 [s1] the smoke was beginning to get to me.
baseline	[s2] mary friend said he had to come to my house.
[s3] i wanted to have a smoke fight.
[s4] he said i had to light a cigarette.
[s5] i smoked for a few hours before i gave up.
predicted relations: <after> <vague> <before> <before>
RL	[s2] i had just gotten out of my car.
[s3] i ran to the front door.
[s4] i opened the door.
[s5] the fire department came and put out the fire.
predicted relations: <vague> <before> <before> <before>
Input 3 [s1] tom was in court for a parking ticket.
baseline	[s2] he did not have to pay his court date.
[s3] the judge gave him a parking ticket.
[s4] tom paid his court date.
[s5] the judge gave tom a parking ticket.
predicted relations: <before> <vague> <before> <after>
RL	[s2] he had been speeding for a while.
[s3] the judge gave him a warning.
[s4] tom didn't listen to the warning.
[s5] he got a ticket anyway.
predicted relations: <after> <before> <before> <after>
Table 10: Examples from our RL-based model that are not as temporally diverse as examples generated by the baseline MEGATRON model. Predicted relations are provided by human evaluators.
K Potential Risks
Since our models deal with open-domain generation, it is conceivable that the generated stories could contain biases, malicious languages, and hallucinations. We refer readers to the work in fairness and fact checking to address these issues.
Input articoutpost9 prompt (after) (vague) (before) (after) (before) (vague)