SlowGuess/NAACL_2021_OCR huggingface dataset:
Naoki Otani, Toshiaki Nakazawa, Daisuke Kawahara, and Sadao Kurohashi. 2016. IRT-based aggregation model of crowdsourced pairwise comparison for evaluating machine translations. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 511-520.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. In OpenAI blog.
Hannah Rashkin, Asli Celikyilmaz, Yejin Choi, and Jianfeng Gao. 2020. PlotMachines: Outline-conditioned generation with dynamic plot state tracking. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4274-4295.
Melissa Roemmele and Andrew S. Gordon. 2015. Creative help: A story writing assistant. In Proceedings of the International Conference on Interactive Digital Storytelling.
Melissa Roemmele, Andrew S. Gordon, and Reid Swanson. 2017. Evaluating story generation systems using automated linguistic analyses. In Proceedings of the Workshop on Machine Learning for Creativity.
João Sedoc, Daphne Ippolito, Arun Kirubarajan, Jai Thirani, Lyle Ungar, and Chris Callison-Burch. 2019. ChatEval: A tool for chatbot evaluation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 60-65.
Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. 2019. Do massively pretrained language models make better storytellers? In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 843-861.
Reid Swanson and Andrew S. Gordon. 2012. Say anything: Using textual case-based reasoning to enable open-domain interactive storytelling. ACM Transactions on Interactive Intelligent Systems, 2:16:1-16:35.
Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar, and Bryan Catanzaro. 2020. MEGATRON-CNTRL: Controllable story generation with external knowledge using large-scale language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2831-2845.
Rowan Zellers, Ari Holtzman, Elizabeth Clark, Lianhui Qin, Ali Farhadi, and Yejin Choi. 2021. Evaluating machines by their real-world language use. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations.
A Writing Interface
A screenshot of the interface is shown in Fig. 2.
B Data Details
We filter the WritingPrompts dataset to contain the first 500 words of all the stories; we do not use the prompts. After filtering, the dataset has 56,582 types and 55,785,118 tokens.  $1.3\%$  of the data is replaced with UNK. The original dataset can be found at https://github.com/pytorch/fairseq/tree/master/examples/stories.
Because the fusion model was originally trained to map from "prompt" to "story," we reconfigure the data and retrain the model to map from "story beginning" to "story end." To do this, we randomly split the stories at a newline and make the first portion of the story the "source" and the second portion the "target." In cases where there are no newlines within the text, we instead split on a space.
C.1 Fusion model
We train the fusion model with the data split in "source" and "target" as described in App. B, using the settings described at https://github.com/pytorch/fairseq/tree/master/examples/stories. We pretrain the model for 9 epochs before adding the fusion model and training for 14 epochs.
To generate, we assign an UNK penalty  $= 10$  to suppress UNKs and use top- $k$  sampling with  $k = 40$ .
C.2 GPT2 model
We finetune small GPT2 model on the WritingPrompts data using the code and settings at https://github.com/huggingface/transformers. We finetune the model for 3 epochs.
To generate, we use either top-  $k$  sampling with  $k = 40$  (for GPT2 and TOP-K) or nucleus sampling with  $p = 0.9$  (for NUCLEUS).
D.1 Edit Results by Suggestion #
The full results, broken down by suggestion #, for edit distance, Jaccard similarity, and USER are in Table 7 (for FUSION vs. GPT2) and Table 8 (for NUCLEUS vs. TOP-K).
D.2 Likert-Scale Results
As shown in Fig. 3, the majority of participant responses are positive about their experience of writing with the CYOA, regardless of which model pair they were working with. The median score for almost all questions is 4 ("Agree").
The one exception is the median response for The suggestions connected to what had happened in the story so far. for FUSION vs. GPT2 is slightly lower-3 ("Neutral"). As hypothesized in §3, this is likely due to the higher degree of randomness in the FUSION-generated text.
Step #2: Choose a suggestion to continue the story.  
You can edit it as much as you like before adding it to the story.
One morning, Gerald woke up early. He ran to the window and thew it open.
The sun was shining down on him. He had just finished his coffee when a knock came from the door.
He took his coat and set it aside, then got out of bed.
Edit Option 2
Add Line to Story
R.Caldecott.The Complete Collection of Pictures &Songs,1887.
Figure 2: The story writing interface. The first box was the first turn of writing (author writing alone). In this case, Option 1 was generated with NUCLEUS sampling and Option 2 with TOP-K sampling. The writer has chosen Option 1, which shows up in the text box below and can now be edited before adding it to the story.
Characters: 97
Total	OVERALL	FUSION	GPT2
ED(↓)	JS(↑)	USER(↑)	ED(↓)	JS(↑)	USER(↑)	ED(↓)	JS(↑)	USER(↑)
32.27	57.85	67.97	37.61	51.13	60.69	29.49	61.35	71.77
Sugg.#1	24.10	65.11	73.77	25.76	62.09	70.17	23.57	66.05	74.90
Sugg.#2	27.26	62.15	71.71	34.19	48.09	57.79	24.22	68.31	77.81
Sugg.#3	34.40	55.96	67.09	36.51	52.20	63.74	33.15	58.17	69.06
Sugg.#4	31.52	59.74	70.89	36.95	56.40	65.55	28.32	61.71	74.05
Sugg.#5	44.08	46.29	56.39	48.13	41.71	50.61	41.03	49.73	60.72
Table 7: Edit distance (ED), Jaccard similarity (JS), and USER scores between the edited and the original generated suggestions overall, from FUSION, and from GPT2.
OVERALL	NUCLEUS	TOP-K
ED (↓)	JS (↑)	USER (↑)	ED (↓)	JS (↑)	USER (↑)	ED (↓)	JS (↑)	USER (↑)
Total	35.74	52.21	62.86	34.65	53.64	63.64	36.69	50.96	62.18
Sugg. #1	32.40	54.12	65.56	26.98	61.37	71.31	36.28	48.93	61.44
Sugg. #2	37.13	49.24	60.22	37.71	47.86	58.25	36.62	50.44	61.95
Sugg. #3	32.93	55.56	65.04	34.79	51.43	60.58	31.31	59.16	68.92
Sugg. #4	33.35	55.37	66.60	34.38	57.31	66.77	32.45	53.68	66.45
Sugg. #5	42.89	46.75	56.90	38.23	51.28	62.24	47.84	41.94	51.24
Table 8: Edit distance (ED), Jaccard similarity (JS), and USER scores between the edited and the original generated suggestions overall, from NUCLEUS, and from TOP-K.
Figure 3: The Likert-scale results for FUSION vs. GPT2 and NUCLEUS vs. TOP-K.
Neutral
Agree