SlowGuess/NAACL_2022_OCR huggingface dataset:
At the plot planning stage, we retrieve a set of triples from ConceptNet (Speer et al., 2017) for each keyword extracted from the input or generated by the model. We only retain those triples of which both the head and tail entity contain one word and occur in our dataset, and the confidence score of the relation (annotated by ConceptNet) is more than 1.0. The average number of triples for each keyword is 33. We show more statistics in Table 2.
Train	Valid	Test
# Examples	47,910	6,477	6,063
Avg. Context Length	332.7	324.8	325.7
Avg. Description Length	23.8	22.7	24.6
Avg. Story Length	230.5	225.7	234.3
Avg. Target Length	21.8	21.7	22.2
Avg. # Keywords (Input)	101.1	99.2	99.6
Avg. # Keywords (Story)	31.2	30.5	31.5
Table 2: Dataset statistics. We compute the length by counting tokens using the BPE tokenizer of GPT2. Keywords are extracted either from the input to initialize the local graph, or from the story to train the model for plot planning.
We compare CONPER with following baselines. (1) ConvS2S: It directly uses a convolutional seq2seq model to generate a story conditioned on the input (Gehring et al., 2017). (2) Fusion: It generates a story by first training a convolutional seq2seq model, and then fixing the model and initializing another trainable convolutional seq2seq model with its parameters. Then the two models are trained together by a fusion mechanism. (Fan et al., 2018). (3) Plan&Write: It first plans a keyword sequence conditioned on the input, and then generates a story based on the keywords (Yao et al., 2019). (4) GPT2Scr: It has the same network architecture with GPT2 but is trained on our dataset from scratch without any pretrained parameters. (5) GPT2Ft: It is initialized using pretrained parameters, and then fine-tuned on our dataset with the standard language modeling objective. (6) PlanA
head: It first predicts a keyword distribution conditioned upon the input, and then generates a story by combining the language model prediction and the keyword distribution with a gate mechanism (Kang and Hovy, 2020). We remove the sentence position embedding and the auxiliary training objective (next sentence prediction) used in the original paper for fair comparison.
Furthermore, we evaluate the following ablated models to investigate the influence of each component: (1) CONPER w/o KG: removing the guidance of the commonsense knowledge in the plot planning stage. (2) CONPER w/o TG: removing target guidance in the plot planning stage. (3) CONPER w/o PP: removing the plot planning stage, which means the model first plans a target sentence and then directly generates the whole story. (4) CONPER w/o TP: removing the target planning stage, which also leads to the removal of target guidance in the plot planning stage.
We build CONPER based on GPT2 (Radford et al., 2019), which is widely used for story generation (Guan et al., 2020). We concatenate the context and the persona description with a special token as input for each example. For fair comparison, we also add special tokens at both ends of the target sentence in a training example for all baselines. We implement the non-pretrained models based on the scripts provided by the original papers, and the pretrained models based on the public checkpoints and codes of HuggingFace's Transformers*. And we set all the pretrained models to the base version due to limited computational resources. We set the batch size to 8, the initial learning rate of the AdamW optimizer to 5e-5, and the maximum training epoch to 5 with an early stopping mechanism. And we generate stories using top- $p$  sampling with  $p = 0.9$  (Holtzman et al., 2019). We apply these settings to all the GPT-based models, including  $\mathrm{GPT}_{\mathrm{Scr}}$ ,  $\mathrm{GPT}_{\mathrm{Ft}}$ , PlanAhead, CONPER and its ablated models. As for ConvS2S, Fusion and Plan&Write, we used the settings from their respective papers and codebases.
Metrics We adopt the following automatic metrics for evaluation on the test set. (1) BLEU (B-n): We use  $n = 1,2$  to evaluate  $n$ -gram overlap
\*https://github.com/huggingface/ transformers
Models	Coherence	Persona consistency
Win(%)	Lose(%)	Tie(%)	κ	Win(%)	Lose(%)	Tie(%)	κ
CONPER vs. ConvS2S	89.0*	5.0	6.0	0.625	82.0*	8.0	10.0	0.564
CONPER vs. Fusion	71.0*	23.0	6.0	0.213	61.0*	22.0	17.0	0.279
CONPER vs. GPT2Ft	54.0*	18.0	28.0	0.275	53.0*	11.0	36.0	0.215
CONPER vs. PlanAhead	53.0*	25.0	22.0	0.311	59.0*	28.0	13.0	0.280
Table 3: Manual evaluation results. The scores indicate the percentage of win, lose or tie when comparing our model with a baseline.  $\kappa$  denotes Randolph's kappa to measure the inter-annotator agreement. * means CONPER outperforms the baseline model significantly with p-value  $< 0.01$  (Wilcoxon signed-rank test).
Models	B-1	B-2	BS-t	BS-m	PC
ConvS2S	12.5	4.7	22.2	32.8	17.1
Fusion	13.3	5.0	22.7	33.3	30.8
Plan&Write	7.2	2.8	6.2	29.7	23.6
GPT2Scr	13.3	4.8	24.7	38.0	26.6
GPT2Ft	13.5	4.7	26.7	37.8	39.5
PlanAhead	15.4	5.3	26.1	37.8	50.2
CONPER	19.1	6.9	32.1	41.4	59.7
w/o KG	17.4	6.3	31.6	39.7	53.4
w/o TG	17.7	6.3	31.9	40.2	56.3
w/o PP	14.9	5.3	32.0	40.0	46.9
w/o TP	16.4	5.8	27.8	37.7	44.9
Growth Truth	N/A	N/A	42.6	42.6	75.2
Table 4: Automatic evaluation results. The best performance is highlighted in **bold**, and the second best is underlined. All results are multiplied by 100.
between generated and ground-truth stories (Papineni et al., 2002). (2) BERTScore-target (BS-t): We use BERTScoreRecall (Zhang et al., 2019) to measure the semantic similarity between the generated target sentence and the persona description. A higher result indicates the target embodies the persona better. (3) BERTScore-max (BS-m): It computes the maximum value of BERTScore between each sentence in the generated story and the persona description. (4) Persona-Consistency (PC): It is a learnable automatic metric (Guan and Huang, 2020). We fine-tune RoBERTa<sub>BASE</sub> on the training set as a classifier to distinguish whether a story exhibits a consistent persona with a persona description. We regard the ground-truth stories as positive examples where the stories and the descriptions are consistent, and construct negative examples by replacing the story with a randomly sampled one. After fine-tuning, the classifier achieves an  $83.63\%$  accuracy on the auto-constructed test set. Then we calculate the consistency score as the average classifier score of all the generated texts regarding the corresponding input.
Result Table 4 shows the automatic evaluation results. CONPER can generate more word overlaps with ground-truth stories as shown by higher BLEU scores. And CONPER can better embody the specified persona in the target sentence and the whole story as shown by the higher BS-t and BS-m score. The higher PC score of CONPER also further demonstrate the better exhibition of given personas in the generated stories. As for ablation tests, all the ablated models have lower scores in terms of all metrics than CONPER, indicating the effectiveness of each component. Both CONPER w/o PP and CONPER w/o TP drop significantly in BLEU scores, suggesting that planning is important for generating long-form stories. CONPER w/o TP also performs substantially worse in all metrics than CONPER w/o TG, indicating the necessity of explicitly modeling the relations between persona descriptions and story plots. We also show analysis of target guidance in Appendix C.
4.5 Manual Evaluation
We conduct a pairwise comparison between our model and four strong baselines including PlanAhead, GPT2 $_{\text{Ft}}$ , Fusion and ConvS2S. We randomly sample 100 stories from the test set, and obtain 500 stories generated by CONPER and four baseline models. For each pair of stories (one by CONPER, and the other by a baseline, along with the input), we hire three annotators to give a preference (win, lose or tie) in terms of coherence (intersentence relatedness, causal and temporal dependencies) and persona-consistency with the input (exhibiting consistent personas). We adopt majority voting to make the final decisions among three annotators. Note that the two aspects are independently evaluated. We resort to Amazon Mechanical Turk (AMT) for the annotation. As shown in Table 3, CONPER outperforms baselines significantly in coherence and persona consistency.
Furthermore, we used human annotation to eval
Policies	Yes (%)	No (%)	κ
Random	22.0	78.0	0.25
Ours	75.0	25.0	0.36
uate whether the identified target sentence embodies the given persona. We randomly sampled 100 examples from the test set, and identified the target for each example as the sentence with the maximum BERTScore with the persona description. And we used a random policy as a baseline which randomly samples a sentence from the original story as the target. We hired three annotators on AMT to annotate each example ("Yes" if the sentence embodies the given persona, and "No" otherwise). We adopted majority voting to make the final decision among three annotators. Table 5 shows our method significantly outperforms the random policy in identifying the persona-related sentences.
4.6 Controllability Analysis
To further investigate whether the models can be generalized to generate specific stories to exhibit different personas conditioned on the same context, we perform a quantitative study to observe how many generated stories are successfully controlled as the input persona descriptions change.
Automatic Evaluation For each example in the test set, we use a model to generate ten stories conditioned on the context of this example and ten persona descriptions randomly sampled from other examples, respectively. We regard a generated story as successfully controlled if the pair of the story and its corresponding persona description (along with the context) has the maximum persona-consistency score among all the ten descriptions. We regard the average percentages of the stories which are successfully controlled in all the ten generated stories for each example in the whole test set as the controllability score of the model. We show the results for CONPER and strong baselines in Table 6. Furthermore, we also compute the superiority (denoted as  $\Delta$ ) of the persona-consistency score computed between a generated story and its corresponding description compared to that computed between the story and one of the other nine descriptions (Sinha et al., 2020). A larger  $\Delta$  means the model can gen
Table 5: Percentages of examples labeled with "Yes" or "No" for whether the identified sentence reflects the given persona.  $\kappa$  denotes Randolph's kappa to measure the inter-annotator agreement.
Models	Controllability Score	Δ
Plan&Write	10.6	0.01
GPT2Ft	24.2	11.2
PlanAhead	23.1	11.2
CONPER	29.5	15.1
Table 6: Automatic evaluation results for the controllability. All results are mulitplied by 100.
Models	Acco (%)	Oppo (%)	Irre (%)
GPT2Ft	21	10	69
PlanAhead	44	12	44
CONPER	66	9	25
Table 7: Manual evaluation results for the controllability. Acco/Oppo/Irre means the example exhibits an accordant/opposite/irrelevant persona with the input.
erate more specific stories adhering to the personas.
As shown in Table 6, there are more stories successfully controlled for CONPER than baselines. And the larger  $\Delta$  of CONPER suggests that it can generate more specific stories to the input personas. The results show the better generalization ability of CONPER to generate persona-controllable stories.
Manual Evaluation For manual evaluation, we randomly sampled 50 examples from the test set, and manually revised the persona descriptions to exhibit an opposite persona (e.g., from "skilled pilot" to "unskilled pilot"). We required a model to generate two stories conditioned on the original and its opposite persona description, respectively. Finally we obtained 300 stories from three models including GPT2Ft, PlanAhead and CONPER. Then, we hired three graduates to judge whether each story accords with the input persona. All annotators have good English language proficiency and are well trained for this evaluation task. Table 7 shows the evaluation results. We can see that  $66\%$  of the stories generated by CONPER are accordant with the input persona, suggesting the better controllability of CONPER.
We present some cases in Table 8. We can see that the story generated by CONPER exhibits the specified persona with a coherent event sequence. The planned keywords by CONPER provide an effective discourse-level guidance for the subsequent story generation, such as tablet, which has a commonsense connection with computer skills and
Context: ... the group has gathered on the rooftop garden of Miyamoto Mansion ... the TV set out near the long table on the patio is talking about some spreer of thefts at low volume ... the issue of Chloe's disappearance and the missing statue still hang over their heads.
Persona Description: [Aito] You are above average in your computer skills. If information is power, then your ability to use the internet makes you one of the most powerful people on the planet.
GPT2Ft: Aito looked at the others, still trying to help find a way out of the hotel. He wasn't sure what the rest of the group wanted to see if they were going to survive and all knew if he needed to be needed...
PlanAhead: Miyamoto Mansion ... perhaps it's just a bit farther away. The music sounds bright enough but the line of visitors does not. Aito was once a pretty girl, he had always been quite witty when talking to people but she always found it annoying that a group of tourists looked like trash just to her ...
CONPER: “Oh, wait … wait … people are talking about Chloe?” “I have a feeling the internet is probably our best chance to get through this” Aito looked around the table a moment before pulling out her tablet and starting typing furiously into her computer. She looked up at the tablet that had appeared, and she could see that it was working on a number of things…
Planned keywords:  $\cdots$  people  $\rightarrow$  look  $\rightarrow$  around  $\rightarrow$  tablet  $\rightarrow$  see  $\cdots$
Table 8: Generated stories by different models. Italic words indicate the improper entities or events in terms of the consistency with the input. The bold sentence indicate the generated target by CONPER. Red words denote the consistent events adhering to the input. And the extracted keywords are underlined.
Internet in the input. In contrast, the baselines tend to not generate any persona-related events. For example, the given persona description emphasizes the strong computer skills of the protagonist while the stories generated by PlanAhead and GPT2 have nothing to do with the computer skills. We further analyze some error cases generated by our model in Appendix G.
We present CONPER, a planning-based model for a new task aiming at controlling the protagonist's persona in story generation. We propose target planning to explicitly model the relations between persona-related events and input personas, and plot planning to learn the keyword transition in a story with the guidance of predicted persona-related events and external commonsense knowledge. Extensive experiments show that CONPER can generate more coherent stories with better consistency with the input personas than strong baselines. Further analysis also indicates the better persona-controllability of CONPER.
This work was supported by the National Science Foundation for Distinguished Young Scholars (with No. 62125604) and the NSFC projects (Key project with No. 61936010 and regular project with No. 61876096). This work was also supported by the Guoqiang Institute of Tsinghua University, with Grant No. 2019GQG1 and 2020GQG0005. This
work was also sponsored by Tsinghua-Toyota Joint Research Fund. We would also like to thank the anonymous reviewers for their invaluable suggestions and feedback.
Ethics Statements
We conduct the experiments by adapting a public story generation dataset STORIUM to our task. Automatic and manual evaluation results show that our model CONPER outperforms existing state-of-the-art models in terms of coherence, consistency and controllability, suggesting the generalization ability of CONPER to different input personas. And our approach can be easily extended to different syntactic levels (e.g., phrase-level and paragraph-level events), different model architectures (e.g., BART (Lewis et al., 2020)) and different generation tasks (e.g., stylized long text generation).
In both STORIUM and ConceptNet, we find some potentially offensive words. Therefore, our model may suffer from risks of generating offensive content, although we have not observed such content in the generated results. Furthermore, ConceptNet consists of commonsense triples of concepts, which may not be enough for modeling inter-event relations in long-form stories. We resort to Amazon Mechanical Turk (AMT) for manual evaluation. We do not ask about personal privacy or collect personal information of annotators in the annotation process. We hire three annotators and pay each annotator $0.1 for comparing each pair of stories. The payment is reasonable considering that it
would cost average one minute for an annotator to finish a comparison.
Nader Akoury, Shufan Wang, Josh Whiting, Stephen Hood, Nanyun Peng, and Mohit Iyyer. 2020. Storium: A dataset and evaluation platform for machine-in-the-loop story generation. arXiv preprint arXiv:2010.01717.
Prithviraj Ammanabrolu, Wesley Cheung, William Broniec, and Mark O Riedl. 2021. Automated storytelling via causal, commonsense plot ordering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 5859-5867.
David Bamman, Brendan O'Connor, and Noah A Smith. 2013. Learning latent personas of film characters. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 352-361.
David Bamman, Ted Underwood, and Noah A Smith. 2014. A bayesian mixed effects model of literary character. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 370-379.
Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. "O'Reilly Media, Inc."
Faeze Brahman and Snigdha Chaturvedi. 2020. Modeling protagonist emotions for emotion-aware storytelling. arXiv preprint arXiv:2010.06822.
Faeze Brahman, Meng Huang, Oyvind Tafjord, Chao Zhao, Mrinmaya Sachan, and Snigdha Chaturvedi. 2021. "let your characters tell their story": A dataset for character-centric narrative understanding. arXiv preprint arXiv:2109.05438.
Faeze Brahman, Alexandru Petrusca, and Snigdha Chaturvedi. 2020. Cue me in: Content-inducing approaches to interactive story generation. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 588-597.
Marc Cavazza, David Pizzi, Fred Charles, Thurid Vogt, and Elisabeth Andre. 2009. Emotional input for character-based interactive storytelling. In Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems-Volume 1, pages 313-320.
Khyathi Chandu, Shrimai Prabhumoye, Ruslan Salakhutdinov, and Alan W Black. 2019. "my way of telling a story": Persona based grounded story generation. In Proceedings of the Second Workshop on Storytelling, pages 11-21.
Elizabeth Clark, Yangfeng Ji, and Noah A Smith. 2018. Neural text generation in stories using entity representations as context. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2250-2260.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020. Plug and play language models: A simple approach to controlled text generation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833.
Angela Fan, Mike Lewis, and Yann Dauphin. 2019. Strategies for structuring story generation. arXiv preprint arXiv:1902.01109.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In International Conference on Machine Learning, pages 1243-1252. PMLR.
Seraphina Goldfarb-Tarrant, Tuhin Chakrabarty, Ralph M. Weischedel, and Nanyun Peng. 2020. Content planning for neural story generation with aristotelian rescoring. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 4319-4338. Association for Computational Linguistics.
Arthur C Graesser, Kathy L Lang, and Richard M Roberts. 1991. Question answering in the context of stories. Journal of Experimental Psychology: General, 120(3):254.
Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, and Minlie Huang. 2020. A knowledge-enhanced pretraining model for commonsense story generation. Transactions of the Association for Computational Linguistics, 8:93-108.
Jian Guan and Minlie Huang. 2020. UNION: an un-referenced metric for evaluating open-ended story generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 9157-9166. Association for Computational Linguistics.
Jian Guan, Yansen Wang, and Minlie Huang. 2019. Story ending generation with incremental encoding and commonsense knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6473-6480.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751.