Melissa Roemmele and Andrew S. Gordon Institute for Creative Technologies, University of Southern California roemmele@ict.usc.edu, gordon@ict.usc.edu
1
Introduction
The field of artificial intelligence has long envisioned the ability of computers to automatically write stories (Dehn [1981], Lebowitz [1985], Meehan [1977], Turner [1993]). For a long time, progress on this task was limited by the difficulty of encoding the vast narrative knowledge needed to produce stories with diverse content. The rise of data-driven approaches to AI introduced the opportunity to acquire this knowledge automatically from story corpora. Since then, this approach has been utilized to generate narratives for different domains and genres (Li et al. [2013], McIntyre and Lapata [2009]), which has in turn made it possible for systems to collaborate with human authors in developing stories (Khalifa et al. [2017], Manjavacas et al. [2017], Swanson and Gordon [2012]). Roemmele and Gordon [2015] introduced a web-based application called Creative Help that provides automated assistance for writing stories.  The interface consists of a text box where users type "\help\" to automatically generate a suggestion for the next sentence in the story. One novelty of the application is that it tracks users' modifications to the suggestions, which enables the original and modified form of a suggestion to be compared. This enables sentences generated by different models to be comparatively evaluated in terms of their influence on the story. We examined a dataset of 1182 Creative Help interactions produced by a total of 139 authors, where each interaction consists of the generated suggestion and the author's corresponding modification. The suggestions were generated by a Recurrent Neural Network language model (RNN LM), as described in Roemmele et al. [2017], which generates sentences by iteratively sampling words according to their observed probability in a corpus. The training corpus for the model analyzed here was 8032 books (a little over half a billion words) in the BookCorpus 1 , which contains freely available fiction from a variety of genres. This paper briefly characterizes the generated sentences by highlighting their most prominent words and phrases and showing examples of them in context.
2
Analysis and Discussion
Table 1 shows the most common lemmatized words generated by the model according to grammatical (part-of-speech) categories 2 , as well the most common two and three word phrases (bigrams and trigrams), with the total count of each item in parenthesis. The total number of words generated across all suggestions was 16025. Table 2 shows some examples of suggestions that feature these lexical items. The model has an interesting writing style and preference for topics. It commonly conveys both narrated and quoted dialogue, as indicated by the frequency of the verbs say, tell, ask, whisper and talk . It is surprisingly eager to introduce a monster into a story. It refers frequently to body parts ( hand, eye, body, face, head, blood ), elements of a residence ( home, door, room, bed ), and time (along with specifically day and night ). It likes abstract nouns ( way, idea, mind, thing, life, truth ). It often
writes from the first-person perspective, given that I is by far the most frequent pronoun (though some of these occurrences might be included in quoted dialogue). In terms of pronouns, it is more likely to refer to female than male characters, indicated by the relative frequency of she and her versus he and him . However, man and father also show up as characters.  The model has an interesting preoccupation with being sure (or not sure ), as well as the capacity to do something ( able, not be able, able to handle, not handle). Apologies are common ( sorry ). References to cognitive and emotional states also appear often, indicated by the adjectives glad, troubled, and afraid as well as the verbs want, feel and think . The model makes value judgments about things being good, right, wrong, and true . Negations ( not ) are overwhelming common, particularly in the context of not know , not sure , not want, not go, not expect, not think , and not handle . Moreover, the model tends to hedge with the adverb maybe as well as probably. The act of transferring or coordinating locations also shows up (go, leave, stay, come, wait, away). These biases are largely indicative of the corpus on which the model was trained. The corpus included several romance novels, for instance, and this appears to be reflected in many of the word choices. However, Roemmele et al. [2017] found that compared to human-written text, the RNN LM was much more constrained in the variety of words/phrases it generated.  It t ends to repeat itself by disproportionately generating frequently observed words. The probability distribution of the model can be manipulated to influence this diversity (via the temperature variable; see Manjavacas et al. [2017]), but it comes with the trade-off of potentially reduced coherence and grammaticality of the sentences. As future work, we are focusing on how the linguistic features of a suggestion influence how users modify it. For example, specific lexical items may be more or less likely to be retained in the story. This analysis supports the overall goal of generating suggestions that are maximally helpful to authors for continuing a story.