A Framework for Exploring Player Perceptions of LLM-Generated Dialogue in Commercial Video Games
Authors: Nader Akoury
Qian Yang
Mohit Iyyer
Venue: EMNLP 2023 Findings (2023)
Keywords: human-centered
dialogue generation
video games
interactive storytelling

Abstract: The growing capabilities of large language models (LLMs) have inspired recent efforts to integrate LLM-generated dialogue into video games. However, evaluation remains a major challenge: how do we assess the player experience in a commercial game augmented with LLM-generated dialogue? To explore this question, we introduce a dynamic evaluation framework for the dialogue management systems that govern the task-oriented dialogue often found in roleplaying video games. We first extract dialogue from the widely-acclaimed role-playing game *Disco Elysium: The Final Cut*, which contains 1.1M words of dialogue spread across a complex graph of utterances where node reachability depends on game state (e.g., whether a certain item is held). Using this dataset, we have GPT-4 perform *dialogue infilling* to generate grounded utterances based on game state represented via code. In a statistically robust study of 28 players recruited from the r/DiscoyElysium subreddit, the LLM outputs are evaluated against the game designers' writing via both preference judgments and free-form feedback using a web interface that recreates the game's core conversation functionality. Overall, the game designers' prose is significantly preferred to GPT-4 generations, with participants citing reasons such as improved logical flow and grounding with the game state. To spur more principled future research in this area, we release our web interface and tools to enable researchers to build upon our work. https://pl.aiwright.dev

A Framework for Exploring Player Perceptions of LLM-Generated Dialogue in Commercial Video Games
Nader Akoury
Qian Yang
Mohit Iyyer
University of Massachusetts
Cornell University
University of Massachusetts Amherst miyyerêcs umass edu
Amherst nsaêcs. uwasu edu
qianyang@cornell.ee
Abstract
those by Al Dungeon (Walton, 2019), InWorld Al (Gelfenheyn al., 2021), and ConvAT (Mukherjee, 2022), which can potentially imbuc games with endless variety. However, evaluating the impacts of LLM-generated dialogue on the player experi- ence has yer to be tackled in principled manner. In this paper, directly evaluate the player ex- perience by asking video gamers interact with LLM-generated dialogue injected into Disco Ely- sium: The Final Cut, a highly-acclaimed dialogue- centered video game (Kurvitz et al., 2021).3 To mitigate difficulties with evaluating open-ended text generation (Karpinska et al., 2021), we specif- ically examine a constrained dialogue generation task in which an LLM must decide how to update dialogue match the corresponding game state. Take for example game's protagonist (an amnesiac detective) is interrogating an uncooperative suspect. If the player decides to act like the suspect's friend to obtain more information, the vari- able seafort. wewifoot.deeetter_seeee yougg_you_are_biee burdbee is set to true, and the corresponding line of dialogue is:
The growing capabilities of large language models (LLMs) have inspired recent efforts to integrate -.L--egnerteed dialogue into video games. However, evaluation remains a ma- jor challenge: how do we assess the player experience in commercial game augmented with LLM-geeerated dialogue? To explore this question, introduce dynamic evaluation framework for the dialogue management sys- lems that govern the task-oriented dialogue of- ten found in roleplaying video games. We first extract dialogue from the widely-acclaimed role-playing game Disco Elysium: The Final Cut, which contains 1.1M words of dialogue spread across a complex graph utterances where node reachability depends game (e.g., whether a certain item is held). Using this dataset, we have GPT-4 perform dialogue infilling generate grounded utterances based on game state represented via code. In statisti- cally robust study of 28 players recruited from the r/Di scoElysiiiii puts are evaluated against the game designers' writing via both preference judgments and free- form feedhack using web interface that recre- ates the game's 's core conversation functionality, Overall, the game designers' prose is signif- icantly preferred to GPT-4 generations, with participants citing reasons such as improved logical flow and grounding with the game state. To spur more principled future research in this area, we release our web interface and tools to enable researchers build upon our work.!
scene in which the
subreddit, the LLM out-
You tell me, here. won't be *thar* usable.
Continuing the example, we then prompt LLM to appropriately modify the dialogue when the variable seafort. beafort.deserter. deserter_j_ann _i_________looo _tt.maals__commiist communist is set to true. We evaluate the generated dialogue against the game writers' original line:
1 Introduction
You can tell a comrade. It won't I be *that* usable.
Dialogue in most narrative-driven video games has historically been static: players may choose from a small number of pre-written dialogue options that depend on the game's 's state (c.g., items held or goals achieved). The advent of large language models (LLMs) has inspired efforts to dynamically gener- ate dialogue in video game environments, such as
While the semantics of the utterance remains mostly unchanged, the player's assumed commu- nist persona is reflected in the second dialogue. These dialogue options, and any associated non- player character (NPC) responses, are defined us-
See for example Nvidia's recent ACE demo.
Disco Flysium: The Final Cut is currently rated the #1 PC video game of all time on Metacritic.
Phttps: h/pp:/pl..al/p www.wiat.wiideee.. dev

---

ing a graph structure commonly referred as "di- alogue tree" in the video game industry. Crucially, our task setup does not expect an LLM to generate all of the game's 's dialogue (as in e.g., AT Dungeon), but rather provides the LLM with human-written di- alogue as input and tweaks it to fit various dynamic aspects of the game state. To condition on the game state, we devise clever approach of encoding the graph structure as mix of code and natural lan- guage, which also opens the possibility in future work of modifying the game state in response to a generated utterance. This constrained setup makes the task tractable to evaluate and also practically relevant to future LLM-human collaborative game writing applications.
grounded language (Suhr et al., 2019) and re- inforcement learning (Bellemare et al., 2012; Kempka et al., 2016). To the best of our knowl- edge, only the sandbox game Minecraft has been explored as a testbed for interactive dialogue re- search (Volum et al., 2022), despite some commer- cial video game dialogue being explored in non- interactive settings (Lopez Latouche et al., 2023; Weir et al., 2023).
Dialogue systems in roleplaying vidco games are task-oriented (Grosz, 1974), where quests in the game act as tasks that the player must complete within the constraints of the game world. The static pre-written dialogue graphs are form of finite- state dialogue management (Brabra et al., 2022). Using such a rigidly constrained dialogue man- agement approach ensures the authorial intent of the game writers at the expense of more natural conversation flows. Rather than upend these famil- iar techniques for more flcxible approaches which have yet to gain traction (Riedl and Young, 2004; Mateas and Stern, 2005), likely due to their com- plexity, we opt to augment the existing approaches actively in use in commericial video games. Our work bridges the tightly scripted scenarios com- mon to video games, with more natural speech that offers humans more agency over their interaction with virtual agents, leading to easier to design agent interactions useful for training simulations (Demasi et al., 2020) and tutoring (Wang et al., 2023a). Narrative-driven video games fall under the um- brella of interactive storytelling, which can take on many forms including tabletop roleplaying games like Dungeons and Dragons (Callison-Buur) et al., 2022), choose your own adventure books (Clark and Smith, 2021), and interactive fiction (Hausknecht et al., 2020). More broadly, graph rep- resentations have been used for finite-state dialogue management (Koller et al., 2018), and knowl- edge bases (Gritta et al., 2021) for slot-filling ap- proaches (Cohen, 2019) to task-oriented dialogue.
Can an LLM understand enough about the game state to appropriately modify dialogue in a way that is logically and tonally consistent with the game state? More generally, how does --geerrret dialogue stack up with the award-winning dialogue written by Disco Elysiun's s designers, and what do video gamers think are the biggest issues with it? By choosing popular roleplaying game with a dedicated following, we are more easily able to find participants familiar with the expected tone and lore needed to effectively assess our generated dialogue. We evaluate OpenAl's state-of-the-ar the-art GPT-4 LLM (OpenAI, 2023) via statistically ro- bust user study, asking Disco Elysium fans to pro- vide preference judgments and free-form feedback within an interface designed to mimic the game's dialogue engine.
Perhaps unsurprisingly, players strongly pre- fer the original dialogue (H) compared to LLMM- generated dialogue (G), with participants citing reasons such as better logical consistency (H: 61 % G: 28%) and flow (H: 67% vs G: 21%). How- ever, participants note that GPT- begins to close the gap at providing interesting dialogue options (H: 47% vs G: 36%) that advance their goals (H: 57% vs G: 33%), though further work is required to ground the dialogue to the game state as 32% of generations rated by at least one player are deemed illogical upon reading the next utterance. To facili- tate future research in player-centered video game dialogue generation, we release our annotation in- terface and tools to reproduce dataset.
3 The Disco Elysium Dataset
In Disco Elysium: The Final Cut, the player takes on the role of a down-on-his-lucc detective in order to solve murder mystery in a dystopian city. The majority of in-game interactions are in the form of dialogue, including interactions with not only other characters but also inanimate objects (e.g., ceiling fan and a bathroom mirror from the first scene of the game), which makes the game well-suited for
2 Related Work
Commercial video games have become increas- ingly popular testbeds for neural approaches to

---

P. PAirzraa
KEAGURD IEAO
NCTI CUSE ORO
YOU
lead "but bene?
oFrice
bodty bocolegi
that cand that chick
fow bag atll oniter atreng
fer.. bacc
PER
PUAZA
Darc lacs domoortt
SEAEORT
bakoc tha, Nos..
DOAT
Deseation AneeSt
DESERTIN BONUS
Pay
Pg
Not
E31
DESERTOR ROASON
RSULIN 440-4450
MAI s9-i0 No-1
Stoeet
X lick
Spokin
"ssufore "ssufort. doserdeeeeee concest, statue's falas
THE DEGEATER
Conversatee conrraaiionn
Actor: Decartrr
*eg.claire, predecmcoor oor dicespeereddf tal
"ceeforrt deserte: did sanething dig for
"ssulori te Login
teeffort.desrrte reactia
CHAKK D1M
Buccts with Hen ang tocky buitiing cherr thit
Calogaerageert
"spalor Lobger 1p,, deiederteerrrree aneny, sead talan, "plamuent prese falls
Lroee"
80. 0A1T
Type las cancaal beth....
Figure 1: As part of the effort decipher the data format from Disco Elysium: The Final Cut, created tool to display and analyze the dialogue graphs from the game. This tool allows for filtering data in various ways, displaying attributes of the dialogue nodes, visualizing entire conversations, creating dataset splits (Section 3.1; Appendix A), preprocessing dialogues into Lua scripts (Section 4.2), analyzing experiements (Section 6), and more.
our experiments, Additionally, the game's state is encoded in Lua" via descriptively-nammn boolean variables and getter/setter functions that trigger on certain nodes of the dialogue graph. In this section, we describe how we extract and process the rich di- alogue graph (Figure 1) and game state from Disco Elysium for constrained dialogue generation task.
Dataset Splits
Train
Valid
Test
Total
89.8%
5.4%
4.7%
Utterances
65,316
4,143
3,237
72,696
Words
1,001,191
59,877
52,816
1,113,884
Nodes
98,442
6,950
5,092
110,484
Forks
17,283
1,544
964
19,791
Variables 99,015
6,989
5,132
111,136
3.1 Extracting data from Disco Elysium
We split the data into training, validation, and test sets based on the number of dialogue words, while ensuring an approximately similar proportion of condi- tional dialogue forks and referenced Lua variables. See Table Al for more information.
Table 1:
We begin by extracting catalog of all top-level entities (characters, items, conversations, dialogue entries, and game state variables) from a purchased PC version of Disco Elysium: The Final Cut using the open source tool AssetStudio.. Its prose, with over 70K utterances consisting of roughly 1.1M words of dialogue (Table 1), is nearly twice the length of Atlas Shrugged. As with many games, Disco Elysium encodes game state variables and functions into Lua expressions that are run by the game engine based on the player's actions.
Dialogue and game state encoding: The game state of Disco Elysium is an exhaustive mix of vari- ables and functions that descriptively named and commented. All dialogue entries in the graph include metadata about the character who is speak- ing, as well as any preconditions (boolean-value expressions) required to speak the utterance. For example, the first dialogue option in Figure 2's Lua
chttps://wwww www.uuaa.oo.. our.org hhtta:://gittuub.om///efaaareeee wwtp///eraa/////eeettttts.e

---

dlaloguetpcions (1
Trufy
The Dieart
Moyt (ead) Thut lncge"
vetuzo ?
sgeaker "eagesLin"'
Jost little. You' boddiar.",
Viensgcken [unokion()
'uaaly chande cfe. Saye Lambard
GaniarlahleValee iwwffrt..Seeaatta _wwg___gg__oooeoe pot.__ooo_rt.tdddda" trie) cad,
Vou
Dus beight
end
rotuzo
speakec "baislggas
Step Cluster similar dialogue
"Phrnaten wfth. PATH, Al1 Turan bwlngs reegeat TE
the Leuth.
You
pais for tolking and respect. does not actoslly wozk
Gattarinbbeeaaiee wastert, dereetttrrrrtrrrr...r...... thrasten with pais true)
Step 2: Linearize into masked Lua script
Veriobiel veeseett devecter hi thoesten with psis then
Gennge snipect
rable.ineest www.://wteeeeoaaaeeeeoeeoo htt./ioaaeatttoono "Ne hare Maye 5o nake you calk
Vaziabie] 'oostezt desecter L. comminist thes
--1l
saying
aable.
buddies")
Step 3: Generate masked dialogue
www.:e/eeaeaaaae piallgiaepttoonn "rou dan cell bere. In soat
that upcole.)
poueerriion Jonny "put you did rdoe 16.") sisicquupppiiase head.)) but Know
table insnet leseeggrgg
"You know, Zim bit rgentE, Y00 oen tzust De GRANK LOIP
SIC- ciriar col?"
Mksloqueogtinn
Figure 2: In this conversation from Disco Elysium: The Final Cut, we first cluster dialogue nodes by semantic similarity of game state variables and spoken dialogue. Then, we linearize the next turn of dialogue in the graph into a Lua script that contains game logic and dialogue. Finally, we <MASK> one utterance from the cluster and ask GPT-4 (an LLM trained on code and natural language) to infill the masked dialogue.
script contains the following precondition:
more efficient for smaller hypergraphs than spe- cialized algorithms (Papa and Markov, 2007), We use the branch and bound algorithm to enumerate all valid partitions of the dialogue graph that sat- isfy
CheckPassiveSkill (""ggeeesiiiisssgsttiirio"
We also observe comments written by the game developers used to document the intent of variable function, as in this comment before setting the variable seafort. deserteredsttdth wwwhhhhthhttteeeeee witp__iithhhpinnn
1.5% variation from the desired splits of 90%/5%/5% train/valid/test"" 6 The final split is achieved with minimal overlap in game variables (Table 1).
pain for talking and respect. does not actually work
4 Grounded dialogue infilling
These comments provide a rich, natural source of context to better explain the vast array of entities (Table Al) referenced throughout the game logic. Each dialogue entry may also contain functions that can alter game state when uttered that also serve as important context to understand the dialogue, as in:
As prior text generation and dialogue research has clearly demonstrated (Karpinska et al., 2021; Clark et al., 2021), evaluating LLM-generated dialogue is daunting challenge. We examine a more con- strained subtask in which an LLM is given multi- ple lexically-similar human-writter responses given utterance, each of which is slightly different from each other based on the game state, One these responses is masked out, and an LLM is asked to generate it based on cues from the game state (e.g., that communists have historically referred to one another comrade). While this task is strictly easicr than open open-ended dialogue generation,
SetWariableWaluec "seafor t.deser wesserter_sugee. youe_ ee__dddieee, true
Creating dataset splits: Since the dialogue in Disco Elysium ultimately tells a single overarching story, it is not possible to create a dataset split in which each fold contains disjoint set of charac- ters, items, and game state variables. The game's dialogue graph implicitly forms a hypergraph, with hyperedges defined by the Lua variables. Opti- mal partitioning of a hypergraph is known to be NP-hard, and an exhaustive enumeration is often
Luucklyy many distinct conversations are connected by one or more dialogue edges, making such that small handful of connected components
the graph make up roughly 70% of the dialogue and thus are required to be in the training set. "While the rest of this paper uses the validation set, in- cluding as a source of few-shot demonstrations to prompt GPT-4, we describe preliminary fine-tuning experiments on the training set in Appendix C.

---

we find that state-of-the-art LLMs still struggle to solve it. This section details the data filtering and preprocessing steps we performed, as well as the few-shot prompting strategy we use with GPT-4.
Few-Shot Prompt Statistics
Min
Avg
Max
Example Length
158
330
2228
Examples per Prompt
15
33
45
4.1 Clustering lexically-simiaa utterances
Table 2: Here we report statistics for token lengths of each example, along with the number of examples per prompt.
We detect lexically-similar utterances by applying a simple token-based clustering algorithm on the nodes in the dialogue graph. Starting from source dialogue node, we traverse all outgoing directed paths which terminate upon encountering dia- logue node; collect all such sets by using each dialogue node source. Then, we tokenize the dialogue each node set on whitespace and punc- tuation boundaries while preserving common con- tractions (e.g., 'II, 's, etc). Next, we compute bag- of-words Fl among utterances within each set to measure similarity, and also apply the same proce- dure to the associated Lua conditions. Finally, we consider all disjoint subsets for which Fi>= 0.5 (for utterances or conditions), which we qualita- tively validated as producing clusters of high lexi- cal similarity.
following instruction to guide the model to gener- ate dialogue constrained to the game state:
You are a creative game designer writing engag- ing dialogue for roleplaying game. For each self-contained dialogue script fill in <MASK> with interesting dialogue using only facts from the script.
Then for a given script, we select demonstrations from our validation set that do not contain any utterances in common with the target script. We fill the full 8k context of GPT-4 with demonstrations, which qualitatively find to produce utterances that best match the game's writing style (Table 2).
4.2 Linearizing clusters into Lua scripts
Now that we have a set of clusters, we convert each cluster into a single Lua script (see Figure 2, right) that can be fed to a language model. We prefix all the characters, items, comments, and variables referenced by the clustered nodes at the top of the script, along with default values and metadata. Each node in the cluster is visited in sequential order, and its Lua conditions, dialogue, and any associated post-speech game state alter- ing actions are included in the script. Lastly, we enumerate all variants of each script by masking out each instance in a cluster one-by-one. Our masked infilling prompts use a prefix-suffix-mass (Donahue et al., 2020; Bavarian al., 2022) or- dering, where the prefix contains the portion of the script before the masked utterance, followed by the text <MASK> the suffix contains the remainder of the script, followed by <MASK> =; and the mask
Setting up a strong user study
Evaluating LLM-generated dialogue within a large commercial video game is complex undertak- ing. Both automatic and crowdsourced evaluation lead to misleading and unreliable conclusions for creative generation tasks (Karpinska et al., 2021: Wang et al., 2023b), which motivates expert anno- tation (Xu et al., 2023; Karpinska and lyyer, 2023). More importantly, we want to collect evaluations from people who are actually invested Disco Elysim,, as primary goal of our study is to char- acterize how LLM-generated dialogue affects the player experience. This entails collecting evalua- tions within an interactive setting rather than hav- ing annotators rate utterances in isolation. In this section, we specify our evaluation setup, which in- volves (1) designing an interface that mimics the Disco Elysium dialogue engine; (2) conducting pi- lot tasks to determine common error categories and further refine the interface to reduce annotator bur- den; and (3) recruiting participants from Reddit who have completed one more playthroughs of Disco Elysium to complete our main user study.
contains the utterance we want the model to infill,
followed by <MASK: END>
4.3 Few-shot prompting infill dialogue
From our set of lincarized Lua scripts, we build few-shot prompts to perform dialogue infilling us- ing GPT-4. We first prefix each prompt with the
*See Appendix B for more details on alternate clustering algorithms that we experimented with.
Additional details, including our full prompt template, can he found in Appendix D.

---

Step 1: read conversation context and select preferred response
Step 2: provide justification for preference
Step 3: read next utterance and update preference if desired
dialugan erikets originn!
buitton
illen ap is cutiin the netting uithed fin battrn bins His raloso nia acck eyee, hoodec by creaord ape its, to meet yours.
Minimne
dieatry
choice hightihheed change Please decssao baton.
shown followiing chose Hithightted
"Vs seen some dest too, know
Undicu eatoracti eyesdnn
(hagh))
"uue dei know?
(Decture towadd your
The Rowen
cwnlll dlown unfforr... throed, priiting ceat." full "lossss. Rooal tnsitmentt
"ouu don't Shis untiorni nif (Geaturs roward coal)
Flovo eriich obblngum cation
Bacck
St Leet
daaliing sptiace
The Deserttd
be The neen sors enit rwo, Imews." +)
*0n llowe Nokes me nostagic for the 1- pousee for mroment, casching breatt.
Plrsse receorint for choosing hig fignted orange
"ouhee reta red your spwnight?
This cotaat
2bx "Yor tebsiigf EM scons rokking 1I
neaded, aiscorite chose sext
Mot
TD 1o-iotiog Devt
Figure 3: Three panels from the mobile version of our web app, which reproduces Disco Elysium's ' dialogue engine and allows to collect preference judgments and justifications. In the first panel, players take part in conversation and are given multiple candidate responses to choose from. Options denoted with a and b are randomly shuffled between human and LLM-generated dialogue. Upon selecting paired dialogue option, the second panel is displayed to collect justification for their choice. The final panel allows players the next line of dialogue that will be spoken and optionally change (and justify) their preference.
5.1 Designing an interface to recreate Disco Elysium's 's dialogue engine
choose which candidate best fits their goals while taking into account the previous conversation his- tory and story context (Figure 3a). After making a preference judgment, they are asked to justify their choice via both predefined tags (e.g., advances my goals or matches desired mood) as well as op- tional free-form comments (Figure 3b). Finally, they are shown the ground-truth human-written next line of dialogue, and they are asked whether they would change their judgment in retrospect given this knowledge; if so, they are again asked justify their decision (Figure 3c).
In Disco Elysium. players control virtual represen- tation of the main character that can move around and interact with the environment, including initiat- ing conversations. This freedom makes it difficult for us to constrain our study within the confines of the video game. Thus, rather than creating a game mod that includes LLM--genrraed dialogue, we design a custom web app" that recreates the core gameplay systems that underpin the game's dia- logue system. This interface allows us to present a specific conversation taken from our validation split to players (i.e., study participants).
5.2 Running usability studies to refine the annotation task
What annotations do we collect? Players are presented with the original human-writter dia- logue for all characters from the game, while the main character's dialogue options are paired along- side generated utterances from GPT-4 (Section 4), which we randomly shuffle and label with the sub- scripts a and b. Players are then asked to provide a preference judgment over the candidate utterances:
We conduct two usability studies to better under- stand common types of free-form participant feed- back; additionally, these studies guide the refine- ment of our interface to reduce cognitive load for participants (e.g., switching to the two-stage anno- tation flow presented in Figure 3b & c). These us- ability studies also led to the coding and integration of predefined tags discussed above. Our first usabil- ity study enlists six college students (each of whom had previously played through Disco Flysium) spend one hour with our web app. We also con- ducted a follow-up controlled observational study
""See for example the World Al-based Skyrim mod. "We use the React framework with an integrated Lua in- terpreter that executes the gameplay logic as defined in our linearized Lua scripts (Section 4.2).

---

Reddit Player Demographies
Reddit Study Details
Number of Players
28
Min
Avg
Max
Total Utterances Rated
1,158
Hours Played
25
82.6
230
Avg Utterances Rated per Player
41
Playthroughs
2.7
Avg Utterances Seen per Player
203
Unique Utterances Rated
Table 3: The number of hours and playthroughs of Disco Elysium as reported by the Redditors who took part in the study.
Table 4: Study details, including number of annotators, the number of unique generated utterances that were rated, and the total number of ratings.
with two more college students, using the Nielsen Norman Group Observer Guidelines 12 to assess the user experience implications of our interface. 13
dialogue choices. For example, if a player often chooses dialogue options that indicate the main character a fascist, the player will more fre- quently be presented with dialogue options that reflect this world view. Thus, each player will have unique trajectory through the dialogue graph and must mentally keep track of their choices, skill players learn through experience with the game, Furthermore, due to the dataset split (Section 3.1), our validation data is from a late stage the game, so participants must know the full story context to understand the nuances of each dialogue choice.
Manually coding human feedback: We manu- ally code the human-wa wwitten feedback from our first usability study to build list of common justifica- tions for player preferences. This leads to a catego- rization of 13 high-level justifications of player's initial preference, and 8 reasons for retroactively updating their preference. To improve annotator efficiency, we update our annotation interface to provide a list of common justification tags partic- ipants can choose from in addition to free-form text.'* 14
Study parameters: Approximately 20% of terances player reads during our study contain LLM-generated dialogue. This is another reason why we hired so many participants, as it requires substantial reading time between annotations. In aggregate, 112 unique utterances are rated by our participants, though since each player performs unique walk of the dialogue graph, on average each player rates 41 utterances. and only 100 utterances receive at least three ratings (Table 4).
5.3 Statistically robust Reddit study
Our task necessitates study with many players since post-utterance functions can alter the current game state, potentially affecting reachability in the graph, which leads to dozens of paths through the conversation chosen from our validation set that we use for our evaluation. For that reason we recruit 28 fans of the game from the r/DiscoE Lysium sub- reddit to take part in our study (Table 3)." Based our usability studies,
estimated players re- quire between 1-2 hours complete our study, and we provide $25 gift cards for participation in the study.
Results & analysis
Overall, our study reveals a strong preference for human-written dialogue over LLM-generated dia- logue. Participants most commonly cite reasons for their preferences such as increased appropriate- ness, better match with their gameplay goals, and stylistic properties (Figure 4). Because assessing the generative capabilities of LLMs confounded by the subjective nature of rating narrative qual- ity (Ethayarajh and Jurafsky, 2022; Wang et al., 2023b), we also conduct fine- see--grinnd analysis of free-form player justifications uncover where GPT-4 succeeds and where it needs improvement.
All participants have played Disco Elysium before: We limit participation in our study to players who have completed at least one full playthrough of Disco Elysium: The Final Cut in English. 16 This is necessary since the game weaves a complex narrative that incorporates the player's
Phttps://www h/wps::///ww... www.w.nggo....omm wwngrop.ccm//aaeeeeoooooooooooo htmm/aartccles/beeeeerrr videlines/
3See Appendix E. for more details about our usability stud-
ies and refinements.
""See Appendix F for specifics each tag.
Ourr study was approved by IRB review. partici-
12 Following Card et al. (2020), our study design has a statis- tical power of 0.96 for a margin of 10%, though our reported margins in Section 6.1 are often much larger.
pants are least 18 years of age.
WW leave evaluation of other languages to future work.

---

Original Preference
Updated Preference
GPT-4 (756)
From human to GPT-4 (14)
Human (1 3000)
From GPT-4 buman (71)
300
30
20
10
100
0-
Blggical hindiight
better moletes devired moed
komros eu intoreatiog
better ad advonces my goals
oilect
fetter exploter options
cintales more specifies
Bads more appropmate
adfances my goals
matcle deaind mond
exploring my options
eeeass nutreaatin
reftrranees carles info enstatta aasait more speiific weeas move logical
other optun is trotevver
othert opton is repert randemly seterned
ether
acsidentally solected
repist to hindugla
(a)
(b)
Figure 4: Histogram of tags, aggregated across all 1,158 judgements, (a) upon initially choosing the dialogue, and (b) after player retroactively updates their preference upon seeing the next utterance in the conversation.
6.1 Participants prefer human-written dialogue
6.2 Fine-grained analysis
While the overall results show a strong preference for human-written dialogue, we note that the task is inherently subjective, and player preference is not always related the quality of the options. Some level of disagreement between participants expected, which motivates us to perform more fine-grained analysis uncover common facets that provide insight on these disagreements and highlight where GPT-4 excels and struggles.
Overall, out the 1,158 total judgements we col-
lect from players, 702 (61%) state a preference for human-written dialogue while 456 prefer GPT-4, When aggregating at the instance level (i.c., com- puting the majority vote on all instances for which we collected annotations from at least three dif- ferent players), note stronger preference for human-written dialogues. Specifically, upon their first assessment (Figure 3b), players prefer human- written dialogue that of GPT-4 (H: 64% vs G: 23%; rest ties), and after retrospectively updating their preference (Figure 3c). annotators prefer the original dialogue even more (H: 66% vs G: 23%)..
Inconsistencies with the game world: Many of the justifications for players' preferences mention appropriateness logical consistency / flow with the conversation history. In general, the world cre- ated by the game designers has a depth and con- sistency that is difficult for models understand, especially when it is encoded programmatically (c.g., in Lua scripts) rather than in unstructured text. In the following example, four players justify their preference (one shown below) for the human- written text by noting that the generated utterance does conform the game's notion of "ultra- liberal" character, which requires an understanding of the game's various political factions.
Reasons for preference: When players prefer human-written dialogue over model-generated dia- logue, they cite reasons such as increased logical consistency (H: 61% G: 28%) and flow (H: 67 % G: 21 %). After seeing the next utterance, the most common reason for players change their preference is that the selected utterance was illog- ical hindsight (Figure 4b), which affects 32% GPT-4 generations rated by at least one player. Overall, these results suggest that future research should focus on better grounding of LLM gener- ations to the game state. However, GPT-4 does close the gap certain aspects of the generated dialogue, including providing interesting dialogue options (H: 47% vs G: 36%) that contain more specifics (H: 43% vs G: 46%) and advance player goals (H: 57% vs G: 33%), which shows the poten- tial of collaborative human-LL -LLM dialogue.
(1) a. That's is "choice You could have become self- employed. Create the system.
HUMAN-WRITTEN
b. I steal from the rich, redistribude wealth, and fight for borderless world.
GPT-4
i. As Ultraliberal character, this option feels more appropriate in this playthrough. feels appropriate
PREpeRS HUMAN-W WRITTEN
Awkward articulation: Sometimes, the compet- ing objectives of generating a fluent utterance and

---

staying faithful to the game state result in awkward generations that can impact logical and stylistic coherence, as in:
Paraphrasing: Dialogue in games tends to be static: speaking with a non-player character of- ten leads to the same utterances being repeated in the absence of relevant game state changes. For- tunately, recent L.Mss like GPT-4 are quite adept at paraphrasing text. When the model conrectly reproduces an utterance semantically similar to the human written dialogue, players often randomly choose between the two:
(2) We're not "Couliiionappponted" We just try help people.
HUMAN-WRITTEN b. I'm not "Coaliionn-ppoinned We just try to help people.
GPT-4 cenerated
i. Since Kim is with me, it's more appropriate say "we".
(4) One more time: what have you used this gun for?
feels appropriate, matches destred mood PREFERS HUMAN WRITTEN
HUSMAN-
never know if we can trust Kim
feels more appropriate, matches desired mood. advances my goals, contains more specifies PREFERS GPT-4 GENERATED
b. Airight, I'll ask again. What have you been using this gun for?
GPT-4 GENERATED
Here, six players prefer the original utterance. in which both sentences use the plural pronoun "we", and call out the awkwardness of excluding the main character's 's partner Kim in the second GPT-4 generated sentence; one example justification is shown in (2i). Despite this conflict, however, two players prefer the GPT-4 generation as they believe it better matches the mood (2ii).
These are both basically the same so I just picked one at random
randomly selected PROPERS HUMAN-WRITTEN
However, in some cases the generated para- phrases include small extraneous information that feel off to the players. In the following example, two players specifically call out GPT-4's phrasing:
(5)
Stop changing g the subject- we have the murder weapon. (Point to it.)
Fit with play style: An attractive aspect of role- playing games is that players mold the game's s narrative to their individual play style. Disco Ely- sium allows for a huge variety of play styles; for example, players can choose to tackle the game as an analytic Sherlock Holmes-type detective or as physically imposing but dim-witted enforcer. GPT-4 is able to provide diverse dialogue that is amenable with certain play styles:
HUMAN-WRIFTEN
b. Enough squirming. I have the murder weapon, and Kim here can confirm it.
GPT-4 GENERATED
i. don't think Kim is a figure of authority at this point- he knows about as much about the murder weapon as Harry does.
feels more appropriate,
advances my goals, more logical PREPERS HUMAN-WRIFTER
(3) a. Pick up the gun lying in the sand.]
7 Conclusion
HUMAN-WRITTEN
b. Mind if exanine your gun, Mr. Dros?
In this paper, we perform a user study of LLM- generated dialogue integrated into video games, hiring fans of Disco Elysium: The Final Cui to provide fineggraneed insights about issues in this domain. We examine a constrained dialogue gener- ation task, in which the game state integrated into the LLM prompt via Lua scripts that encode game variables, functions, and dialogue. We develop a web interface that reproduces Disco Elysium's 's dialogue engine to conduct the evaluation. Human- written dialogue is strongly preferred over GPT-4 generations for reasons such improved logical flow, appropriateness, and tonal consistency. Fu- ture work can build on our framework consider user play style, faithfulness to the game state, and dynamically updating game state as important com- ponents of the modeling and evaluation process.
GPT-4 GeneratEd
i. He threw it away "like an amputated limb" so don't 't think asking him for permission thetorical or not seems appropriate
foels more appropriate PREFERS HUMAN WRITTEN
i. I am being passive aggressive here, do not, in fact, care if he minds.
feel appropriate PRIPERS GPT-4 GEnerATiOO
In the above example, one player thinks it does not make sense to ask permission to look at the gun (3i), while the other prefers the GPT-4 genera- tion because they intentionally want be passive aggressive towards Mr. Dros (3ii). Both players marked feels more appropriate as a justification, which is not a contradiction since they each have
different play styles and objectives.

---

8 Acknowledgements
review board. We do not collect any demographic information of the participants beyond age veri- fication. Additionally, we ensure participants in our study are fairly compensated for their time, of- fering $25 gift cards for Reddit participants, $30 gift cards for our observational study participants, and either $25 or $50 gift cards for our initial pilot participants depending on the length of time they spent on the study.
Thanks to all of the Disco Elysium fans who took part in the studies described in the paper. Also, thanks to the reviewers and to Brendan O' O'Connor for valuable feedback which helped improve the manuscript. Finally, thanks Ronan Salz and George Wei who conducted an early exploration of the Disco Elysium data. This project was partially supported by awards IIS-2202506 and IIS IIS-2046248 from the National Science Foundation (NSF).
We also note that Disco Elysium: The Final Cut is a copyrighted game, so we take special care to ensure we respect the intellectual property of the game's ' designers. 20 We do not release any models trained on the game's data, nor do we widely re- lease our web interface, which requires registering an account with authorization token to take part in the study. Finally, the web app only has access to a small portion of the overall game data taken from the validation set which is needed to conduct the study, and that data is only ever kept in memory, never persisted to disk.
9 Limitations
While the dialogue for Disco Elysium is available in Simplified Chinese, Traditional Chinese, En- glish, French, German, Japanese, Korean, Polish, Portuguese-Brazilian. Russian, Spanish, and Turk- ish, our study focuses exclusively the English version of the game, which was necessary due to the large number of players required for adequate coverage of the dialogue graph. Additionally, the generated dialogue we show participants is gener- ated once from GPT-4, such that each player rates the same generated utterance. This reduces varia- tion in the annotation task allowing us to have high statistical power for our results. Though due to the static nature of the generated dialogue we do not ac- count for the player's unique walk of the dialogue graph, which would require dynamically generated utterances, even though this could better reflect the interactive nature of video games. We live this to future work. Wc also note that Disco Elysium represents a niche genre of narrative-drive driivv games that does not reflect the full diversity of narratives seen in video games, thus our approach is unlikely generalize the diverse catalog of video game narratives. Though our work does apply to a large class of popular video games such as the recently released Baldur's 's Gate 3 and Starfield both of which have enthralled millions of gamers within a month of their 19
References
Mohammad Bavarian, Heewoo Jun, and Nikolas Tezak. 2022. Efficient Training of Language Models to Fill in the Middle. arXiv:2207. 14255 [cs], page 30.
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. 2012. The arcade learning envi- ronment: An evaluation platform for general agents. CoRR, 21//1/27747700
Hayet Brabra, Marcos Báez, Boualem Benatallah, Walid Gaaloul, Sara Bouguelia, and Shayan Zamanirad. 2022. Dialogue management in conversational sys- tems: A review of approaches, challenges, and op- portunities. IEEE Transactions on Cognitive and Developmental Systems, 14:783-798.
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. arXiv:2005. 14165 [cs].
10 Ethical Considerations
Disco Elysium contains adult themes, including discussions of suicide, murder, and rape. For this reason, ensure participants in our study are at least 18 years of age as required by our institutional
2"The intellectual property rights of the game are currently the subject of a legal dispute between the game's 's designers and their former development studio ZA/UM. For more infor- mation, see https:/ //www//www/////////// http:///pw/peteeeeee pcgamer cww/the- tom/thh-legal-- -war -over-disco derr-dicoo-ee elysium slysium-reaches reaches-dicc-- diisco-elssine elysium- elysium-level coofccompexitty/
18 http:://wwww.....c.e/rrr//////tt s- 19 Lemm:oooooooppeessse-esssseaaaee tttpped-starfielee www.ssiedd-pllleeeeeeeeee wal.ayyrreeggggeent-tteeeeeee staeeeeeettttttetteet us- us
cana

---

Chris Kallison-Burrele Gaurav Singh Tomar, Lara Mar- tin, Daphne Ippolito, Suma Bailis, and David Reit- dialog chal-
Findings of the Association for Computational Lin- guisties: EMNLP 2020, pages 3623-3636, Online. Association for Computational Linguisties.
ter. 2022. Dungeons and dragons lenge for artificial intelligence. In Proceedings of the 2022 Conference on Empirical Methods in Nat- urul Language Processing, pages 9379-9395 Abu Dhahi, United Arab Emirates. Association for Com- putational Linguistics.
Chris Donahue, Mina Lee, and Percy Liang. 2020. En- abling language models to fill in the blanks. In Pro- ceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 2492- 2501, Online. Association for Computational Lin- guistics.
Dallas Card, Peter Henderson, Urvashi Khandelwal,
Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020. With little power comes great responsibility. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9263-9274, Online. Association for Computa- tional Linguistics.
Kawin Ethayarajh and Dan Jurafsky. 2022. The authen- ticity gap in human evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 6056-6070, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards. Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philiippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert. Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code.
Ilya Gelfenbeyn, Kylan Gibbs, and Michael Ermolenko. InWorld Al [online]. 2021.
Milan Gritta, Gerasimos Lampouras, and Ignacio la- cobacci. 2021. Conversation graph: Data augmenta- tion, training, and evaluation for non-deteriiiisic dialogue management. Transuctions of the Associa- tion for Computational Linguistics, 9:36-52.
Barbara Grosz, 1974. The structure of task oriented dialogs. In IEEE Symposium on Speech Recogni- rion: Contributed Papers. Carnegie Mellon Univer- sity Computer Science Dept., Pittsburgh, Pennsylva- nia, volume 10.
Matthew Hausknecht, Prithviraj Ammanabrolu, Mare- Alexandre Côté, and Xingdi Yuan. 2020. Interactive Fiction Games: A Colossal Adventure. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7903-799
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. 2021. All that's "human' is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pages 7282-7296, Online. Association for Computational Linguistics.
Marzena Karpinska, Nader Akoury, and Mohit Tyyer. 2021. The perils of using Mechanical Turk to evalu- ate open-ended text generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1265-1285 Online and Punta Cana. Dominican Republic. Association for Computational Linguistics.
Marzena Karpinska and Mohit lyyer. 2023. Large lan- guage models effectively leverage document-leve context for literary translation, but critical errors per- sist.
Elizabeth Clark and Noah A. Smith. 2021. Choose your own adventure: Paired suggestions in collaborative writing for evaluating story generation models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 3566-3575, Online. Association for Computa- tional Linguisties.
Marzena Karpinska, Nishant Raj, Katherine Thai, Yix- iao Song, Ankita Gupta, and Mohit lyyer. 2022. DEMETR: Diagnosing evaluation metrics for trans- lation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9540-9561, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Philip Cohen. 2019. Foundations of collaborative task- oriented dialogue: What's 'ssss in a slot? In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 198-209, Stockholm, Sweden. Association for Computational Linguistics.
Michal Kempka, Marek Wydmuch. Grzegorz Runc. Jakub Toczek, and Wojciech Jaškowski. 2016. Viz- Doom: A Doom-based AI research platform for vi- sual reinforcement learning. 2016 IEEE Conference on Computational Intelligence and Games (CIG). pages 1-8.
Orianna Demasi, Yu Li, and Zhou Yu. 2020. multi- persona chatbot for hotline counselor training. In

---

Alexander Koller, Timo Baumann, and Arne Köhn. 2018. Dialogos:
Yuxin Wang, Jieru Lin, Zhiwei Yu, Wei Hu, and Börje Karlsson. 2023b. Open-world story generation with structured knowledge enhancement: A comprehen- sive survey.
Simple and extensible dialogue
modeling. In Interspeech.
Robert Kurvitz, Helen Hindepere, Argo Tuulik, Cash De Cuir, and Olga Moskvina. 2021. Disco Elysium: The Final Cut. ZA/UM Studios.
Nathaniel Weir, Ryan Thomas, Randolph D' Amore, Kel- lie Hill, Benjamin Van Durme, and Harsh Jhamtani. 2023. Ontologically faithful generation of non-player character dialogues.
Gaetan Lopez Latouche, Laurence Marcotte, and Ben Swanson. 2023, Generating video game seripts with style. In Proceedings of the 5th Workshop on NLP for Conversational AT (NIP4ConvAT 2023), pages 129- 139, Toronto, Canada. Association for Computational Linguistics.
Fangyuan Xu, Yixiao Song, Mohit lyyer, and Eunsol Choi. 2023. A Critical Evaluation of Evaluations for Long-form Question Answering. In Proceedings of the ō1st Anmual Meeting of the Association for Computational Linguistics, Toronto, CA. Association for Computational Linguistics.
Michael Mateas and Andrew Stern. 2005. Demonstra- tion: The Interactive Drama Façade. In Proceed- ings of the First AAAI Conference on Artificial In- telligence and Interactive Digital Entertainment, Al- IDE'0S 05, pages 155, Marina del Rey, California AAAI Press.
Purnendu Mukherjee. ConvAl [online] 2022.
OpenAl. 2023. GPT-4 Technical Report.
David A. Papa and Igor L. Markov. 2007. Hypergraph partitioning and clustering. In Handbook Approxi- mation Algorithms and Metaheuristics.
Mark O. Riedl and Robert Michael Young. 2004. An intent-driven planner for multi-agent story genera- tion. Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, 2004. AAMAS 2004., pages
Alane Suhr, Claudia Yan, Jack Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, and Yoav Artzi. 2019. Executing instructions in situ- ated collaborative interactions. Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 21 291-2130, Hong Kong, China. Association for Computational Linguistics.
Ryan Volum, Sudha Rao, Michael Xu, Gabriel Des- Garennes, Chris Brockett, Benjamin Van Durme, Olivia Deng, Akanksha Malhotra, and Bill Dolan. 2022. Craft an iron sword: Dynamically generating interactive game characters by prompting large lan- guage models tuned code. In Proceedings of the 3rd Wordplay: When Language Meets Games Work- shop (Wordplay 2022), pages 25-43, Seattle, United States. Association for Computational Linguistics.
Nick Walton. Al dungeon [online] 2019.
Lingzhi Wang, Mrinmaya Sachan, Xingshan Zeng, and Kam-Fai Wong. 2023a. Strategize before teach- ing: A conversational tutoring system with pedagogy self-distillation. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2268- 2274, Dubrovnik, Croatia, Association for Computa- tional Linguistics.

---

A Dataset Splits
C Preliminary Experiments
We provide additional details regarding our dataset splits. As we note in the main body of the paper, a key challenge in splitting the Disco Elysium dataset into train, valid, and test splits is the high degree of interconnectedness across conversations in the game. While it is not possible to create dialogue split with a disjoint set of game state variables, we minimize overlaps amongst the splits.
We conduct experiments using two LLMs: GPT- Curie and Codex (Table A2). GPT-3 Curie is strong generation model for natural language (Brown et al., 2020), especially when finetuned on downstream task, while Codex is extremely pable few-shot LM for code (Chen al., 2021). As our task contains elements of both natural language and code, it is important assess the capabilities of each model paradigm.
Variable Overlap
Dataset Totals
Train Valid 2897
Items
259
Model Class
Prompt
Model
OpenAI API
Train
Test 2871
Characters
424
Tokens
Type
Name
Valid n Test
303
Conversations
610
Curie
2048
Finetuned
curie
Codex
8000
Few-Shot code-davinci-002
(a)
Table A2: Details of the models used in our experiments. As OpenAl does not provide parameter counts details on finetuming, we also provide the API name for the models to help reproducibility.
Table A1: For the Disco Elysium dataset: (a) take special care to minimize the number of referenced vari- able overlaps amongst the splits; (b) though do not at- tempt to disentangle Characters and Items across splits,
'Likely 6.7B parameters, https://blog.. eleuther wweuther.ai/goteee ai/gpt3 moptt--mdeelsizes/
B Additional Clustering Details
We experiment with a number algorithms for clustering the game's 's dialogue, including the Lev- enshtein distance, Jaccard index, and the Dice co- efficient. We also vary the features used for cluster- ing by splitting the words into characters, grouping by ngrams, and through the use of lowercasing. We conduct manual inspection of the various ap- proaches to clustering, including hyperparameter sweep of the similarity threshold. This inspection indicated that solcly clustering based on the dia- logue utterances would either systematically miss semantically similar text, or cluster dissimilar ut- terances when the similarity threshold was made more permissive.
Since the two models perform different tokeniza- tion and support different context lengths, we filter the clusters, keeping only those that fit the smallest context length (2048 tokens) using the GPT-3 tok- enizer. We then generate all the linearized scripts representing semantically related text for the next turn of dialogue. After filtering and generating masked variants of the clusters, we are left with 30,501 training examples and 2,668 validation ex- amples.
We finetune Curie for I epoch, with batch size of 32 examples and learning rate of 0.2x the learning rate of the pretrained model and we weight the loss for the prompt tokens by 0.01. For the few-shot Codex model, we prefix each linearized Lua script with several samples from the validation set such that they take up nearly the full context window (we reserve 100 tokens of the context for generation). We also ensure there are no overlaps in dialogue between the few-shot examples and the script. Consequently, each Codex script has 7 few-shot examples on average.
To combat this tendency, additionally tried clustering nodes by inspecting the associated Lua conditions. We first parse the Lua expression, ex- tract identifiers (which often refer to functions) and string literals (which often refer to variables). We then split the literals into their constituent words (e.g., whirling. dreamone_brave becomes whirling, dreamone, brave), before running the above battery of clustering approaches. In the end, we find that clustering based on combina- tion of dialogue and Lua expressions produced the best results, without the need for the extra feature engincering, while only relying on the simple Dice coefficient with a threshold d > 0.5.
We choose to measure the performance of the models on the validation set using bag-of-words F1, as the clustered utterances have a large over- lap with the masked text the model is tasked with
"Codex uses modified tokenizer that collapses whitespace since it is commonly used in code formatting.

---

Model Examples
Tokens BLEURT F1
Few-Shot Prompt Statistics
Curie
2,668
3,041,299
41.9 25.6
Min
Avg
Max
Codex
2,668
21,077,200
44.2 29.5
Example Length
158
330
2228
Examples per Prompt
15
33
45
Table A3:
Preliminary experiments over the validation set show that few-shot Codex outperforms finetuned Curie model for generating context-aware dialogue.
Prompt Length
7547
7566
8089
Table A5:
Here we report statistics for token lengths of each example and the overall prompts, along with the number of examples per prompt.
infilling. In addition, we use BLEURT which has proven to be robust for semantic similarity of gen- erated text (Karpinska et al., 2022). Both metrics favor Codex slighty, though given the low Fl score, it's clear the models have much room for improve- ment on this simplified form of our task. That is to say, naively applying preliminary approach to all the dialogue in the game, not just to the subset of dialogue clustered via similarity, is even more likely to fail. We also posit that Codex likely out- performs Curie since it is a larger model that is explicitly trained on a large corpus of code, even though it uses a few-shot approach to inference.
to include previous turns of dialogue in the prompt to see if that improves generation quality.
D Few-shot Prompting
Since we target GPT-4 for our main study, we pro- vide few shot prompts using their chat format. All prompts are prefixed with the following system message:
You are a creative game designer writing engag- ing dialogue for a roleplaying game. For each self-contained dialogue script fill in the <MASK> with interesting dialogue using only facts from the script.
To better understand the performance difference between the two models also conduct small analysis of each model's output. We find that both models tend to copy from the prompt (Table A4), but Codex does nearly twice as often.
Additionally, our few shot examples are encoded as chat conversations where user message pro- vides the model with script, and the the assis- tani responds with the completion. In terms of the linearization we describe in Section 4.2, the user message consists of the prefix and suffix, while the assistant message consists of the mask. Note we, experimented with other prompting approaches, but found the above worked best. We investigated zero-shot and few-shot prompts containing various numbers of examples. The resultant GPT-4 gen- erations often did not match the style the game writing, frequently leading to verbose utterances. We also tried interleaving instructions before each few-shot example, and that seemed to have no no- ticeable difference is quality.
Model
Examples
Copied
Curie 2,668
235
Codex
2,668
455 (8))
Table A4: We find that both Curie and Codex occassion- ally copy dialogue from the prompt, and in 8t instances Codex directly copies completion from the few-shot examples.
A qualitative inspection of the generations from the Codex model (our best performer) seem to indi- cate the model may struggle to generate plausible completions due lack of historical context to the current conversation. Our script-based prompts do not include any previous dialogue utterances, but rather rely only on the combination of dialogue that can be emitted next and conditional game logic gating those options. It is clear the models also do not make effective use of the game designer's s anno- tations to fill in the gaps. While these comments likely useful reference for the writers of the game, they may not contain enough context alone to guide generation. Considering Codex has a very long con- text window and performs better than a finetuned Curie (Table A3), future experiments could attempt
E Interface adjustments from usability studies
Two-step annotation flow In our initial interface players are presented with single screen in which to provide free-form feedback both the selected utterance and whether they want to change their mind after seeing the next utterance. This unified feedhack screen led players to conflate the reason they chose a dialogue option with their host-hoc reasoning after seeing the next utterance. Before

---

Predefined Tag
Description
randomly selected
You randomly selected between the paired options.
accidentally selected
You accidentally selected the dialogue option.
advances my goals
The selected dialogue option advances your goals.
exploring my options
You are just trying explore all the dialogue options.
The selected dialogue option fits the conversation better than it's counterpart.
feels more appropriate
matches desired mood
The selected dialogue option matches the mood you are going for.
The selected dialogue option contains more context specific infor- mation than it's counterpart.
contains more specifics other option is repeat other option is irrelevant references earlier info seems interesting
The paired dialogue option that was not selected repeats something that was already stated.
The paired dialogue option that was not select is irrelevant to the current context.
The selected dialogue option references information from earlier in the conversation.
The selected dialogue option seems more interesting than it's coun- terpart.
The selected dialogue option fits the current context more logically than it's counterpart.
seems more logical
Please explain in your words why you selected the dialogue option.
other
Table A6: List of predefined tags and their associated description from our web app for justifying why a player initially chose candidate ullerance.
our observational study, we split the feedback pro- cess into two screens (Figure 3b & c) and note that this obviates the player confusion seen in our initial study.
task starting where the player left off.
F Predefined Tags
We manually code the free-form comments from our first pilot study to understand common justifi- cations for expressed preferences. We do this both for comments on a player's initial preference and upon retroactively updating their preference based upon seeing the next utterance. We then update our interface to include manually coded categoriza- tions predefined tags that players justify their choices. For the full list of predefined tags and the description we provide for the tag in our interface, please see Table A6 (initial prefer- ence) and Table A7 (upon retrospectively changing preference).
Updating the interface The second observa- tional study highlighted three major concerns. First, the actual Disco Elysium game provides visual cues that were missing from the web interface that play- ers relied upon to follow the conversation. We address this concern by tweaking our interface to better match the one from the game, including high- lighting actor names using the same colors from the game and updating the icons for the player statis- tics shown at the bottom of screen Figure 3a. Second, we decided to instruct players to exhaust all dialogue options within the conversation, which allows us to improve our coverage of the generated utterances in the dialogue graph. This process dif- fers from the actual gameplay, in which players free skip through many dialogues that serve to provide a backstory the game world. Finally, became clear that players might not complete the annotation task within one session, so we added support for automatically saving and resuming the
select

---

Predefined Tag
Description
After seeing the next line of dialogue, it turns out that the other dialogue option better advances your goals.
better advances my goals
After seeing the next line of dialogue, turns out that the other dialogue option better explores the conversation.
better explores my options
After seeing the next line of dialogue, it turns out that the other dialogue option better matches the mood your are going for. After seeing the next line of dialogue, turns out that the other dialogue option actually contains more context specific information.
better matches desired mood
contains more specifics
After seeing the next line dialogue, turns out that the other dialogue option does not make sense in context.
illogical in hindsight
After seeing the next line of dialogue, it turns out that the other dialogue option repeats something that was previously stated. After sceing the next line of dialogue, it turns out that the other dialogue option is actually more interesting.
is repeat in hindsight seems more interesting now
Please explain in your own words why you changed the selected dialogue option.
other
Table A7: List of predefined tags and their associated description from web app for justifying why player retrospectively changed their preference.

---

G
Rating Trends
Original Preference
Due to the nature of the game using a dialogue graph where node reachability is altered depend- ing on the dialogue option chosen, on average each player only rates 41 utterances out of the 112 unique utterances rated by all players (Table 4). To understand how the player preference changes the number of ratings particular utterance re- ceives, we produce stacked histograms (Figure A1 & Figure A2) where each bar represents player preference given the number of players rating an utterance.
100
Tied
GPT-4 Preferred
Utterances
Human Preferred
50
0
5
10
15
20
Minimum Number Users Rating Utterance
Figure Al: Histogram of initial candidate preferences based on minimum number players rating the utter- ance.
Updated Preference
Tied
100
GPT-4 Preferred
Uterances
Human Preferred
50
0
15
Minimum Number of Users Rating
Utterance
Figure A2: Histogram of retroactively updated prefer- ences based on minimum number of players rating the utterance.


Reviews:
[{"review_id": "AUc9AhIn9H", "reviewer": "Reviewer_h39F", "summary": "", "questions": "- Table 1 (content & caption): What is a dialogue fork? Please define such specific terms briefly for non-familiar readers.\n- 100-104 & 409-417 & 427-429: What do \"H:\" vs. \"G:\" stand for in these paragraphs? Please clarify (e.g., \"H\" for human-written dialogue vs. \"G\" for GPT-4 generations).\n- How deep do you go in conversation history for the GPT-4 generated responses? Do you provide the whole game session dialogue history, or is there a fixed number of conversational turns to go back and check (for efficiency)? Please clarify.\n\n--- Updates After Rebuttal ---\n- Appreciate the clarifications for \"Terminology\" and \"Undefined initials\".\n- Regarding the \"Dialogue History\", I was referring to your observations in lines 441-449 (see below) and wondering if including the dialogue history (maybe up to 10 turns for efficiency) would help to resolve these issues:\n\"Inconsistencies with the game world: Many of the justifications for players’ preferences mention appropriateness or logical consistency / flow with the conversation history. In general, the world created by the game designers has a depth and consistency that is difficult for models to understand, especially when it is encoded programmatically (e.g., in Lua scripts) rather than in unstructured text.\"", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Under reasons to accept, you describe our work as providing valuable insights.\nThanks for noticing!\nWe appreciate your feedback and sincerely wish that you'll read our comments and reflect on them during the reviewer discussion period.\nIf you do, we hope that you will update your score and associated review text to indicate how our comments below factor into your decisions.\nThanks!\n\n**Background**\n\nWe wish to bring your attention to an important consideration for our work: while many companies are looking to integrate LLMs into video games, no published research has performed an extrinsic evaluation of model-generated dialogue in a video game, not even from major game studios like Microsoft [[1](https://arxiv.org/abs/2212.10618)] who have direct access to the game code.\nThe referenced paper (which we only became aware of after submission and will cite in a revision of the paper) explores dialogue generation for the roleplaying video game *The Outer Worlds*, but in contrast only looks at utterances **without conditioning on game state** and uses a mix of automatic metrics and **out-of-context human evaluation**.\nIn comparison, we reverse engineer the dialogue system from *Disco Elysium* and recreate it in a web app environment which reduces confounding variables (e.g. choosing where to go, who to interact with, etc).\n**The release of our code and tools will democratize the ability to conduct future in-context human evaluation for research exploring interactive dialogue in video games.**\nThis is important since LLMs are actively being investigated for video games and other virtual experiences as mentioned in our paper: many well-funded startups like InWorld AI [[2](https://inworld.ai/blog/inworld-valued-at-500-million)], character.ai [[3](https://www.crunchbase.com/organization/character-ai)], and convai [[4](https://investor.nvidia.com/news/press-release-details/2023/NVIDIA-ACE-for-Games-Sparks-Life-Into-Virtual-CharactersWith-Generative-AI/default.aspx)] have begun offering products in this space, and established gaming companies like Microsoft [[1](https://arxiv.org/abs/2212.10618)] and Ubisoft [[5](https://aclanthology.org/2023.nlp4convai-1.11/)] are conducting research in the area too.\n\n**Study Size**\n\nYour first stated concern is whether our usage of the term \"large-scale\" is appropriate.\nUnderlying that concern is an important question: Are 28 players enough to provide meaningful insights into our research question regarding player experience with LLM-generated dialogue?\nWe begin by noting that our paper ensures that 100 utterances receive at least three annotations across our 28 participants as discussed in our **Study parameters** in Section 4.3.\nUsing the methodology from [[6](https://aclanthology.org/2020.emnlp-main.745/)], our study design has a statistical power of 0.96 for a margin of 10% in a high variance population (see the tables at the end for a more complete analysis).\nThe observed margins in our study as described in Section 5.1 are often much larger (e.g. in aggregate 61% of players prefer human-written dialogue, a margin of 22%) indicating more than sufficient statistical power for our reported results (a power $\\geq0.8$ is typically desired).\nThe additional fine grained analysis in Section 5.2 covers specific actionable areas of concern to improve modeling for the task in the future.\nTherefore we believe the scale of the study is adequate for the findings we report.\nWe will include a discussion of our study's statistical power in a revision of the paper and will happily address your concern by not referring to our study as \"large-scale\".\n\n**Dynamic Text Generation**\n\nYour second concern is whether our evaluation is dynamic enough.\nWe discuss why we make this choice in our limitations section.\nIncluding a player's dialogue history as part of the prompt is an easy change to include in our framework, but we chose not to do so as it may introduce a large amount of variance necessitating a considerably larger number of participants to achieve a similar statistical power as our current design.\nThis is certainly a task that can be explored in future research once our code is released.\nThough please note that our work goes above and beyond any other research in the field as discussed above [[1](https://arxiv.org/abs/2212.10618)].\n\n**Question 1: Terminology**\n\nA dialogue fork is a node with an outgoing edge where traversal is gated by a binary game state expression -- the edge can only be traversed if the game state matches a given Lua condition.\nWe will clarify such terms further in a revision of the paper.\n\n**Question 2: Undefined Initials**\n\nYou are correct, *H* is for human and *G* is for GPT-4 in those paragraphs.\nWe will clarify this in a revision of the paper.\n\n**Question 3: Dialogue History**\n\nWe are not sure we fully understand the question, but will try our best to answer.\nThe prompts we use for dialogue generation do not contain any dialogue history, rather they include the game writer's comments and the alternate dialogue options conditioned upon game state.\nFor an example, please see the Lua script in Step 2 of Figure 1.\nIn regards to player annotation, we allow players to edit their feedback for a particular utterance after seeing future utterances.\n\n**Thanks for reading!**\n\nIn light of our comments addressing your concerns, we sincerely hope you'll consider raising both your soundness and excitement scores.\nSpecifically, we believe **we provide sufficient support for all of our claims** and that **our work will enable a promising new research direction** for evaluating video game dialogue in an interactive setting, which is an area of growing interest [[1](https://arxiv.org/abs/2212.10618),[2](https://inworld.ai/blog/inworld-valued-at-500-million),[3](https://www.crunchbase.com/organization/character-ai),[4](https://investor.nvidia.com/news/press-release-details/2023/NVIDIA-ACE-for-Games-Sparks-Life-Into-Virtual-CharactersWith-Generative-AI/default.aspx),[5](https://aclanthology.org/2023.nlp4convai-1.11/)].\n\n\n\\[1\\]: https://arxiv.org/abs/2212.10618  \n\\[2\\]: https://inworld.ai/blog/inworld-valued-at-500-million  \n\\[3\\]: https://www.crunchbase.com/organization/character-ai  \n\\[4\\]: https://investor.nvidia.com/news/press-release-details/2023/NVIDIA-ACE-for-Games-Sparks-Life-Into-Virtual-CharactersWith-Generative-AI/default.aspx  \n\\[5\\]: https://aclanthology.org/2023.nlp4convai-1.11/  \n\\[6\\]: https://aclanthology.org/2020.emnlp-main.745/  \n\nWe report the results of the statistical power analysis of our study design below:\n```\nNum Workers: 10\nMargin | Power | Variance\n0.05   | 0.765 | low\n0.05   | 0.325 | high\n0.1    | 1     | low\n0.1    | 0.775 | high\n0.2    | 1     | low\n0.2    | 1     | high\n```\n\n```\nNum Workers: 28\nMargin | Power | Variance\n0.05   | 0.91  | low\n0.05   | 0.475 | high\n0.1    | 1     | low\n0.1    | 0.96  | high\n0.2    | 1     | low\n0.2    | 1     | high\n```\n\n```\nNum Workers: 50\nMargin | Power | Variance\n0.05   | 0.925 | low\n0.05   | 0.61  | high\n0.1    | 1     | low\n0.1    | 1     | high\n0.2    | 1     | low\n0.2    | 1     | high\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This study performs an acceptable user study of LLM-generated dialogue augmented into video games, with 28 players of 'Disco Elysium: The Final Cut' providing valuable insights on player experience.\n- The authors mainly examine the following critical research question: Can an LLM understand enough about the game state to appropriately modify dialogue in a way that is logically and tonally consistent with the game state? \n- They focus on a constrained dialogue generation task, in which the game state is integrated into the GPT-4 LLM few-shot prompt via Lua scripts that encode game variables, functions, and dialogue. \n- This constrained setup makes the task more tractable for evaluation and practically relevant to future human-AI/LLM collaboration in interactive game/storytelling applications.\n- The authors implemented a web interface to recreate Disco Elysium's dialogue engine to perform these evaluations, and this interface and tools are released to enable researchers to build upon this initial work.\n- Results indicate that human-written dialogue is preferred strongly over GPT-4 LLM generations for reasons like improved logical flow, appropriateness, and tonal consistency. \n- Overall, the paper is well-written and clearly structured for readability.\n\n--- Updates After Rebuttal ---\n- Reworded \"a large-scale user study\" with \"an acceptable user study\" (appreciate the authors's response regarding the \"Study Size\").", "reasons_to_reject": "- The authors suggest that large number of players required for adequate coverage of the game's dialogue graph. They also claim a large-scale user study is necessary to evaluate the LLM-generated responses in this setup. While recruiting 28 players is not a small number, and it requires a great effort to build a system like this to collect such valuable data, due to the highly subjective nature of players experiences, one can argue that calling it a \"large-scale\" user study may not be well-accepted in this context. While there are no hard and fast rules around how many people you should involve in this type of research, having a population size of hundreds (even thousands) would be more appropriate to call it a \"large-scale\" user study comfortably.\n\n- The LLM-generated dialogue text shown to the participants are pre-generated from GPT-4, so each of the 28 players rates the same generated utterances. That indicates the lack of considering player's personal choices and specific paths on the game dialogue graph during the evaluation process. In the future, it would be better to explore methods to dynamically generate these utterances and eventually reflect the interactive nature of such video games.\n\n- No significant flaws detected for rejection, but upon acceptance, the authors must ensure that the web interface and tools to crawl/annotate similar datasets will be publicly available.\n\n--- Updates After Rebuttal ---\n- The discussions around \"Study Size\" was helpful and convincing to some degree (in the revised version, adding more discussions on statistical significance of your results based an the adequacy of population size would definitely help)\n- Regarding the \"Dynamic Text Generation\", I appreciate the reasoning behind this choice (as partly discussed in the Limitations section). However, it would help to emphasize this more in the main paper and discuss why this is important (but difficult and costly in terms of variance and population size) for a more realistic setup and evaluation in the future (could be part of the Conclusion/Future Work, not just the Limitations).", "questions_to_authors": "- Table 1 (content & caption): What is a dialogue fork? Please define such specific terms briefly for non-familiar readers.\n- 100-104 & 409-417 & 427-429: What do \"H:\" vs. \"G:\" stand for in these paragraphs? Please clarify (e.g., \"H\" for human-written dialogue vs. \"G\" for GPT-4 generations).\n- How deep do you go in conversation history for the GPT-4 generated responses? Do you provide the whole game session dialogue history, or is there a fixed number of conversational turns to go back and check (for efficiency)? Please clarify.\n\n--- Updates After Rebuttal ---\n- Appreciate the clarifications for \"Terminology\" and \"Undefined initials\".\n- Regarding the \"Dialogue History\", I was referring to your observations in lines 441-449 (see below) and wondering if including the dialogue history (maybe up to 10 turns for efficiency) would help to resolve these issues:\n\"Inconsistencies with the game world: Many of the justifications for players’ preferences mention appropriateness or logical consistency / flow with the conversation history. In general, the world created by the game designers has a depth and consistency that is difficult for models to understand, especially when it is encoded programmatically (e.g., in Lua scripts) rather than in unstructured text.\"", "paper_topic_and_main_contributions": "- This paper focuses on evaluating the impacts of LLM-generated dialogue on the player experience. \n- The authors asked the gamers to interact with LLM-generated dialogue injected into 'Disco Elysium: The Final Cut', a widely-popular dialogue-centered role-playing game. \n- This study investigates a constrained dialogue generation task in which an LLM must decide how to update a dialogue to match the corresponding game state. \n- For this dialogue infilling task, an LLM receives human-written dialogue text as input and tweaks it to fit various dynamic aspects of the game state (i.e., GPT-4 generates masked utterances conditioned upon the game state).\n- The authors evaluated GPT-4 LLM outputs via an acceptable-size user study, asking 28 Disco Elysium players to provide preference judgments and free-form feedback using a web interface to mimic the game's conversation functionality.\n- The findings suggest that players strongly prefer the original game engine dialogue over the LLM-generated dialogue due to better logical consistency and flow with the game state.\n- Participants also note that GPT-4 can close the gap by providing creative and interesting dialogue options. However, it often does that at the expense of logical flow and game-state grounding.\n- To contribute to future research in player-centered video game dialogue generation, the authors release the web-based annotation interface and tools to reproduce similar game datasets.\n\n--- Updates After Rebuttal ---\n- Reworded \"a large-scale user study\" with \"an acceptable-size user study\"(appreciate the authors's response: \"We will include a discussion of our study's statistical power in a revision of the paper and will happily address your concern by not referring to our study as \"large-scale\".\")", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {"Missing_References": "- 043: Footnotes/references can be inserted for \"AI Dungeon, InWorld AI, and ConvAI\".\n- 122: Footnote with link (https://www.lua.org/) or reference can be added for \"Lua\" language scripts/expressions used throughout the text for non-familiar readers.\n- 136: Footnote/reference can be inserted for \"Atlas Shrugged\".", "Typos_Grammar_Style_and_Presentation_Improvements": "Minor issues:\n- 045, 052, 133, 180, 182, 218, 256, 288, 331, 342, 351, 360, 635: \".^n\" could be \"^n.\" (placing the footnote numbers - in superscripts - before the punctuation). \n- 351 (footnote 13): \"IRB\" could be \"Institutional Review Board (IRB)\".\n- Section titles: Inconsistent capitalization exists in main section titles (please check the template or choose one style for consistency).\n- Tables 1, 2, 3, 4: Text font sizes used in table content could be smaller (for improved presentation style and consistency, plus potentially to gain some space for further discussions).\n- 568-569: \"to the best of our knowledge no commercial video games\" should be \"to the best of our knowledge, no commercial video games\" (missing comma).\n- 790: \"Disco Elysiumdataset\" should be \"Disco Elysium dataset\" (missing space)."}}, {"review_id": "KyzSZ9Htia", "reviewer": "Reviewer_aXdf", "summary": "", "questions": "A: Can you please state again what you think is the generalisable take-home message from your experiment? What have we learned about the capabilities of LLMs?\n\nB: Please explain again why the evaluators had to be familiar with the game.\n\nC: Section 4.3, how much was payed for the study alltogether? How many gift cards were offered?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Under reasons to accept, you state that our dataset, task, and experiment design are interesting.\nThanks for the kind words!\nWe appreciate your feedback and sincerely wish that you'll read our comments and reflect on them during the reviewer discussion period.\nIf you do, we hope that you will update your score and associated review text to indicate how our comments below factor into your decisions.\n\n**Background**\n\nWe wish to bring your attention to an important consideration for our work: while many companies are looking to integrate LLMs into video games, no published research has performed an extrinsic evaluation of model-generated dialogue in a video game, not even from major game studios like Microsoft [[1](https://arxiv.org/abs/2212.10618)] who have direct access to the game code.\nThe referenced paper (which we only became aware of after submission and will cite in a revision of the paper) explores dialogue generation for the roleplaying video game *The Outer Worlds*, but in contrast only looks at utterances **without conditioning on game state** and uses a mix of automatic metrics and **out-of-context human evaluation**.\nIn comparison, we reverse engineer the dialogue system from *Disco Elysium* and recreate it in a web app environment which reduces confounding variables (e.g. choosing where to go, who to interact with, etc).\n**The release of our code and tools will democratize the ability to conduct future in-context human evaluation for research exploring interactive dialogue in video games.**\nThis is important since LLMs are actively being investigated for video games and other virtual experiences as mentioned in our paper: many well-funded startups like InWorld AI [[2](https://inworld.ai/blog/inworld-valued-at-500-million)], character.ai [[3](https://www.crunchbase.com/organization/character-ai)], and convai [[4](https://investor.nvidia.com/news/press-release-details/2023/NVIDIA-ACE-for-Games-Sparks-Life-Into-Virtual-CharactersWith-Generative-AI/default.aspx)] have begun offering products in this space, and established gaming companies like Microsoft [[1](https://arxiv.org/abs/2212.10618)] and Ubisoft [[5](https://aclanthology.org/2023.nlp4convai-1.11/)] are conducting research in the area too.\n\n**Modeling Concerns**\n\nWe want to first address your concern regarding our focus on evaluation over modeling for this task.\nWe greatly appreciate that you notice the considerable effort we spent on our evaluation!\nWe believe this aligns very closely with the theme of the *Human-Centered* NLP track our paper was submitted to.\nWhen this track was first announced as a special track at NAACL 2022 [[6](https://2022.naacl.org/blog/special-theme/)], it was made clear that human factors are the focus rather than modeling contributions (emphasis added):\n> If the submission contributes a new modeling or training technique that is evaluated on standard corpus-based benchmarks, it would best fit the regular track. **However, if the submission evaluates, e.g., how users calibrate their trust in the predictions of the system, it would be a good fit for the theme track.**\n\nOur player-centered analysis in Section 5 clearly aligns with the stated goal above.\n\nThat said, while we do not focus heavily on the modeling for this task, we experiment with both fine-tuning and few-shot approaches as described in Appendix C.\nWe use Lua to ground the dialogue with game state as the *Disco Elysium* dialogue system is Lua-based and there is no way to directly translate Lua code into natural language.\nTherefore we focus on models which have shown strong performance on both code and language domains like GPT-4.\nAdditionally, by formalizing the dialogue structure as Lua we open the possibility to conduct future research that allows for dynamically changing the game state based on machine-generated utterances, an open problem.\n\n**Linguistic Analysis**\n\nAs you can see from Section 5.2, we include a fine-grained analysis performed by players comparing human-written and machine-generated dialogue.\nThis analysis includes a discussion of observed linguistic features, such as awkward articulations (e.g. combining the use of first person and third person pronouns within a single utterance) and interjecting spurious phrases during paraphrasing.\n\n**Style Identification**\n\nIf you look at the results highlighted in Figure 3b and discussed in Section 5.1, players are not merely performing style identification.\nThe most common reason for changing their preference from LLM-generated to human-written dialogue is that the utterance is illogical in context.\nThough since our paper is concerned with *human-centered* factors, even if players do perform style identification, that would still be an important consideration: one of the largest video game publishers, Ubisoft, is actively researching style for video game dialogue to ensure consistency [5].\n\n**Ethics**\n\nIn regards to your ethical concerns, we take great care to respect the copyright of the work.\nWe want to make it clear: **we do not directly release a dataset**.\nRather, we release instructions for using AssetStudio [[7](https://github.com/Perfare/AssetStudio)] to extract the game data from a **legally obtained copy of the game**.\nWe then provide tools for processing this data allowing easier replication and enabling future experimentation by researchers.\nFinally, our study requires a login with a registered account only available to participants who have previously completed the game.\n\n**Question A: Take-home Message**\n\nAs we highlighted in the **Background** above, LLMs are actively being investigated for video games.\nDespite this interest, there is no published research on player experience interacting with the LLMs in a game setting.\nOur research provides both macro- and fine-grained analysis of specific issues LLMs face in this context.\n\n**Question B: Familiarity**\n\nAs described in Section 4.3, players need to be experienced with the game because:\n\n* Dialogue choices in *Disco Elyisum* affect the game state and change what dialogue options the player has available in the future.\nThis is an acquired skill we ensure participants have by limiting participation to those who have played the game.\n* The conversation players are evaluating are from our validation set (the dataset split was automatically determined by minimizing game state overlap as described in Section 2.1).\nThis conversation is from late in the game and thus requires familiarity with the game to understand and assess the conversation, e.g. we wouldn't expect someone to understand references in the last episode of a TV show if they haven't watched the rest of the show; the same consideration applies here.\n\nPlease note that from our pilot and observational study it became clear that players could not remember exact dialogue options.\nAs the game was originally released in 2019 with an updated release in 2021, most players had completed their play-through significantly earlier than our main study's timeframe in Spring of 2023.\n\n**Question C: Study Cost**\n\nWe had 5 participants for our pilot study, 2 for our observational study, and 28 for our main study.\nIn total, we spent nearly 1000 dollars in gift cards across all participants (a breakdown of gift card amounts is detailed in Section 9).\nWe further spent approximately 3000 dollars in cloud costs for both our fine-tuning and few-shot experiments.\nFinally, we spent 1.5 years of research time to produce this study, starting from Jan 2022 -- an exact salary breakdown is difficult to provide.\n\n**Thanks for reading!**\n\nIn light of our comments addressing your concerns, we sincerely hope you'll consider raising both your soundness and excitement scores.\nSpecifically, we believe **we provide sufficient support for all of our claims** and that **our work will enable a promising new research direction** for evaluating video game dialogue in an interactive setting, which is an area of growing interest [[1](https://arxiv.org/abs/2212.10618),[2](https://inworld.ai/blog/inworld-valued-at-500-million),[3](https://www.crunchbase.com/organization/character-ai),[4](https://investor.nvidia.com/news/press-release-details/2023/NVIDIA-ACE-for-Games-Sparks-Life-Into-Virtual-CharactersWith-Generative-AI/default.aspx),[5](https://aclanthology.org/2023.nlp4convai-1.11/)].\n\n\n\\[1\\]: https://arxiv.org/abs/2212.10618  \n\\[2\\]: https://inworld.ai/blog/inworld-valued-at-500-million  \n\\[3\\]: https://www.crunchbase.com/organization/character-ai  \n\\[4\\]: https://investor.nvidia.com/news/press-release-details/2023/NVIDIA-ACE-for-Games-Sparks-Life-Into-Virtual-CharactersWith-Generative-AI/default.aspx  \n\\[5\\]: https://aclanthology.org/2023.nlp4convai-1.11/  \n\\[6\\]: https://2022.naacl.org/blog/special-theme/  \n\\[7\\]: https://github.com/Perfare/AssetStudio", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- interesting resource (human-authored dialogue game tree + game state extracted from game) + human preference ranking over original and LLM generated dialogue utterances in context\n\n- interesting task and experiment setup (\"dialogue infilling\", generate utterance that fits to given state, compare with original utterance from that point)", "reasons_to_reject": "- relies on intuitive notion that the task is hard and somehow involves language understanding, but doesn't spend any time analysing data for linguistic features of human-authored text vs computer generated one. What exactly is it that makes this task hard? Why would we even think to begin with that LLMs could handle the state description?\n\n- relatively little effort is spent on setting up the model for tackling the task, and very much effort on evaluating it. While in general I appreciate that, this limits the conclusions that can be drawn. If the focus was on creating a model that can do this task, I would have expected to see ablation studies, or more variation on how the game state is communicated to the model (e.g., not as LUA code but in natural language).\n\n- the evaluators were familiar with the game. Could it not be that the performed style identification rather than really were judging appropriateness? \n\n- the experiments are not freely replicable, as the data source is copyrighted material.", "questions_to_authors": "A: Can you please state again what you think is the generalisable take-home message from your experiment? What have we learned about the capabilities of LLMs?\n\nB: Please explain again why the evaluators had to be familiar with the game.\n\nC: Section 4.3, how much was payed for the study alltogether? How many gift cards were offered?", "paper_topic_and_main_contributions": "Paper presents a new resource (the extracted dialogue tree from a video game) and a new task (\"dialogue infilling\", generating a game dialogue utterance given the game state), and evaluates GPT-4 on this task (for human preference), showing that the original utterance is mostly preferred.\n\n- NLP engineering experiment\n- new data resource", "excitement": 4, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- not sure whether word-count(Atlas Shrugged) should count as unit for measuring length of written material\n\n- line 386, and elsewhere: you talk about \"LLM-generated dialogue\", but it seems to me that you at most mean \"LLM-generated dialogue utterances\"", "Justification_for_Ethical_Concerns": "Not a major red flag, and the authors do discuss it in the appropriate section, but I still wanted to note here that I am not fully satisfied with that discussion. The underlying material (the dialogue tree from the game) is copyrighted material, and the creators of that material did not consent to it being used for this study. This is the case for much of the material that we work with, but when the material comes from a single author or a small set of authors, it is thrown in even sharper relief."}}, {"review_id": "qYioxVgPOQ", "reviewer": "Reviewer_NDsa", "summary": "", "questions": "A: How were raters instructed and did they know what the purpose of their ratings was and even that some contributions were machine-generated? Were they asked about their opinion of machine-generated content in general?\n\nB: What is the role of the Lua scripts? Is there any particular reason that it is Lua scripts rather than some other programming language or just natural language? Is there any reason to believe that Lua scripts are better suited as input for an LLM than another formalization?\n\nC: Are there any demographics (e.g. age, gender) about the participants in the usability studies that you can report? Do the demographics match what is known about the demographics of people actually playing the game?\n\nD: For the contributions for which you have 3 or more ratings, how much do raters agree in their decisions? Even if 100 instances is small, this can give a first glimpse into how generalizable the ratings are for the game and this target group.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Under reasons to accept, you describe our work as interesting and meaningful.\nWe're thankful you noticed!\nWe appreciate your feedback and sincerely wish that you'll read our comments and reflect on them during the reviewer discussion period.\nIf you do, we hope that you will update your score and associated review text to indicate how our comments below factor into your decisions.\n\n**Background**\n\nWe wish to bring your attention to an important consideration for our work: while many companies are looking to integrate LLMs into video games, no published research has performed an extrinsic evaluation of model-generated dialogue in a video game, not even from major game studios like Microsoft [[1](https://arxiv.org/abs/2212.10618)] who have direct access to the game code.\nThe referenced paper (which we only became aware of after submission and will cite in a revision of the paper) explores dialogue generation for the roleplaying video game *The Outer Worlds*, but in contrast only looks at utterances **without conditioning on game state** and uses a mix of automatic metrics and **out-of-context human evaluation**.\nIn comparison, we reverse engineer the dialogue system from *Disco Elysium* and recreate it in a web app environment which reduces confounding variables (e.g. choosing where to go, who to interact with, etc).\n**The release of our code and tools will democratize the ability to conduct future in-context human evaluation for research exploring interactive dialogue in video games.**\nThis is important since LLMs are actively being investigated for video games and other virtual experiences as mentioned in our paper: many well-funded startups like InWorld AI [[2](https://inworld.ai/blog/inworld-valued-at-500-million)], character.ai [[3](https://www.crunchbase.com/organization/character-ai)], and convai [[4](https://investor.nvidia.com/news/press-release-details/2023/NVIDIA-ACE-for-Games-Sparks-Life-Into-Virtual-CharactersWith-Generative-AI/default.aspx)] have begun offering products in this space, and established gaming companies like Microsoft [[1](https://arxiv.org/abs/2212.10618)] and Ubisoft [[5](https://aclanthology.org/2023.nlp4convai-1.11/)] are conducting research in the area too.\n\n**Generalizability**\n\nWhile we do mention that our work does not generalize to all narrative-driven games, the framework we put forth is indeed useful across many games: any video game that provides dialogue choices conditioned on game state.\nThese types of games are played by millions of players: see Baldur's Gate 3 [[6](https://wccftech.com/baldurs-gate-3-topped-5-2-million-units-sold-on-steam-says-belgian-embassy/)] as a very recent example of their popularity (see also the examples we highlight in our **Background** above).\nNote that *Disco Elysium* uses an off-the-shelf dialogue system made for the Unity game engine which is in use by many additional games [[7](https://www.pixelcrushers.com/games-showcase/), [8](https://www.articy.com/en/showcases/)].\nThis means the tools we release can be directly used to to explore and conduct experiments on a wide range of currently released video games.\n\n**Study Size and Heterogeneous Raters**\n\nWe appreciate the concern over the study size in the face of potentially heterogeneous raters.\nAs you note, our paper describes a study where 100 utterances receive at least three annotations across our 28 participants as discussed in our **Study parameters** in Section 4.3.\nUsing the methodology from [[9](https://aclanthology.org/2020.emnlp-main.745/)], our study design has a statistical power of 0.96 for a margin of 10% in a high variance population (see the tables at the end for a more complete analysis).\nThe observed margins in our study as described in Section 5.1 are often much larger (e.g. in aggregate 61% of players prefer human-written dialogue, a margin of 22%) indicating more than sufficient statistical power for our reported results (a power $\\geq0.8$ is typically desired).\nThe additional fine grained analysis in Section 5.2 covers specific actionable areas of concern to improve modeling for the task in the future.\nTherefore we believe the scale of the study is adequate for the findings we report even across differences such as play style.\nWe will include a discussion of our study's statistical power in a revision of the paper.\n\n**Familiarity Bias**\n\nAs described in Section 4.3, players need to be experienced with the game because:\n\n* Dialogue choices in *Disco Elyisum* affect the game state and change what dialogue options the player has available in the future.\nThis is an acquired skill we ensure participants have by limiting participation to those who have played the game.\n* The conversation players are evaluating are from our validation set (the dataset split was automatically determined by minimizing game state overlap as described in Section 2.1).\nThis conversation is from late in the game and thus requires familiarity with the game to understand and assess the conversation, e.g. we wouldn't expect someone to understand references in the last episode of a TV show if they haven't watched the rest of the show; the same consideration applies here.\n\nPlease note that from our pilot and observational study it became clear that players could not remember exact dialogue options.\nAs the game was originally released in 2019 with an updated release in 2021, most players had completed their play-through significantly earlier than our main study's timeframe in Spring of 2023.\n\n**Additional References**\n\nWe would be happy to include a discussion of additional references and how they relate to our work.\n\n**Question A: Additional Instructions**\n\nIn addition to the instructions seen in Figure 2 of our paper, raters were given the following instructions:\n\n> As you play, you will be presented with dialogue options that may be machine-generated, paired alongside the original dialogue from the game (e.g. *1a* vs *1b*).\n> Anytime you choose a paired dialogue option you will be prompted to state why you chose the option.\n> You will then be presented the next line of dialogue that will be spoken and asked to reassess your choice.\n> Please consider the next line of dialogue and the historical context of the dialogue thus far when justifying your choices (e.g., because your choice fit the context better than the other choice or was more interesting).\n> * Please play through the scenario as many times as needed to exhaust all the dialogue options.\n> * We highly encourage you to include quotations from the dialogue history or the choices in your justifications!\n\n**Question B: Why Lua?**\n\nWe use Lua to ground the dialogue with game state as the *Disco Elysium* dialogue system is Lua-based and there is no way to directly translate Lua code into natural language.\nTherefore we focus on models which have shown strong performance on both code and language domains like GPT-4.\nAdditionally, by formalizing the dialogue structure as Lua we open the possibility to conduct future research that allows for dynamically changing the game state based on machine-generated utterances, an open problem.\n\n**Question C: Demographics**\n\nWe specifically do not collect demographic information in order to conform to the requirements of our Institutional Review Board (IRB), only ensuring that participants are at least 18 years of age and have completed a play-through of the game.\n\n**Question D: Study Generalizability**\n\nPlease refer to our discussion of **Study Size and Heterogeneous Raters** above.\n\n**Thanks for reading!**\n\nIn light of our comments addressing your concerns, we sincerely hope you'll consider raising both your soundness and excitement scores.\nSpecifically, we believe **we provide sufficient support for all of our claims** and that **our work will enable a promising new research direction** for evaluating video game dialogue in an interactive setting, which is an area of growing interest [[1](https://arxiv.org/abs/2212.10618),[2](https://inworld.ai/blog/inworld-valued-at-500-million),[3](https://www.crunchbase.com/organization/character-ai),[4](https://investor.nvidia.com/news/press-release-details/2023/NVIDIA-ACE-for-Games-Sparks-Life-Into-Virtual-CharactersWith-Generative-AI/default.aspx),[5](https://aclanthology.org/2023.nlp4convai-1.11/)].\n\n\n\\[1\\]: https://arxiv.org/abs/2212.10618  \n\\[2\\]: https://inworld.ai/blog/inworld-valued-at-500-million  \n\\[3\\]: https://www.crunchbase.com/organization/character-ai  \n\\[4\\]: https://investor.nvidia.com/news/press-release-details/2023/NVIDIA-ACE-for-Games-Sparks-Life-Into-Virtual-CharactersWith-Generative-AI/default.aspx  \n\\[5\\]: https://aclanthology.org/2023.nlp4convai-1.11/  \n\\[6\\]: https://wccftech.com/baldurs-gate-3-topped-5-2-million-units-sold-on-steam-says-belgian-embassy/  \n\\[7\\]: https://www.pixelcrushers.com/games-showcase/  \n\\[8\\]: https://www.articy.com/en/showcases/  \n\\[9\\]: https://aclanthology.org/2020.emnlp-main.745/  \n\nWe report the results of the statistical power analysis of our study design below:\n```\nNum Workers: 10\nMargin | Power | Variance\n0.05   | 0.765 | low\n0.05   | 0.325 | high\n0.1    | 1     | low\n0.1    | 0.775 | high\n0.2    | 1     | low\n0.2    | 1     | high\n```\n\n```\nNum Workers: 28\nMargin | Power | Variance\n0.05   | 0.91  | low\n0.05   | 0.475 | high\n0.1    | 1     | low\n0.1    | 0.96  | high\n0.2    | 1     | low\n0.2    | 1     | high\n```\n\n```\nNum Workers: 50\nMargin | Power | Variance\n0.05   | 0.925 | low\n0.05   | 0.61  | high\n0.1    | 1     | low\n0.1    | 1     | high\n0.2    | 1     | low\n0.2    | 1     | high\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The task setup is interesting and meaningful in the application evaluation that is targetted here. The main question that the authors are trying to answer is whether LLMs can be used for generated character contributions in video games and the setup tests exactly that by using the actual game variables to elicit GPT-4 output.\n\n- The work compares GPT-4 output with human expert output in a well-defined context that stays as close to the final target domain as possible: a commercial video game.\n\n- The study is extensive in that it required recruiting a specific target group of participants that then needed to spend substantial time playing and rating contributions in order to gather a meaningful amount of data.", "reasons_to_reject": "- The evaluation setup is rather specific to the game that is described here and it is unclear whether and how this can be tested with other games. Even though the limitations section mentions the limited generalizability, I consider this a serious issue for publication.\n\n- Even though substantial effort is put into data collection, the final dataset that is analyzed is small, only a hundred contributions receive 3 judgments, making the results rather weak.\n\n- In addition, there are factors that make the already small rater group heterogenous, such as play style, experience with the game, and possibly demographic factors (that are impossible to know here). This adds to the low generalizability.\n\n- Since only raters who have already played the game were included in the study (with good reason), it is unclear to me what role familiarity bias would play when rating dialog contributions. Being used to a certain story line might make raters prefer contributions they have seen before or chosen before.\n\n- The related works section mentions only work related to video games. I would like to see here some related work about comparing LLM output with human output and references to papers that have studied what issues there are with LLM-generated dialog contributions.\n\n*Update after rebuttal*\n\nThank you to the authors for clearly addressing my concerns. I would like to see the details on power analysis as well as additional background references in a final version of the paper.", "questions_to_authors": "A: How were raters instructed and did they know what the purpose of their ratings was and even that some contributions were machine-generated? Were they asked about their opinion of machine-generated content in general?\n\nB: What is the role of the Lua scripts? Is there any particular reason that it is Lua scripts rather than some other programming language or just natural language? Is there any reason to believe that Lua scripts are better suited as input for an LLM than another formalization?\n\nC: Are there any demographics (e.g. age, gender) about the participants in the usability studies that you can report? Do the demographics match what is known about the demographics of people actually playing the game?\n\nD: For the contributions for which you have 3 or more ratings, how much do raters agree in their decisions? Even if 100 instances is small, this can give a first glimpse into how generalizable the ratings are for the game and this target group.", "paper_topic_and_main_contributions": "This paper describes a user study in which LLM output is compared to human expert-generated text in a specific game context (the Disco Elysium game). The purpose is to get insight into LLMs' application to the video game domain. The games that are targetted are role-playing video games in which different characters produce contributions depending on the game state and the character state.\n\nThe evaluation of LLM output is done by letting a small group of players play a specific video game and choosing their preferred dialog contribution in certain situations, as well as asking for their reasons for the choice (both with pre-defined and freetext reasons).\n\nThe main contributions are:\n\n- A task design for comparing GPT-4 generated output to human expert output for the Disco Elysium game.\n\n- A user study that includes a specially designed interface, carefully designed rating options verified via an observation study and a specific target group of raters that were recruited to spend a considerable amount of time in order to gather data for analysis.\n\n- The analysis of the gathered data, including some insights into why raters chose one option or the other.", "excitement": 3, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Suggestions for references for related work about comparing LLM output with human output and papers that have studied what issues there are with LLM-generated dialog contributions:\n\nBelinda Z Li, Maxwell Nye, and Jacob Andreas. 2022. Language modeling with latent situations. arXiv preprint arXiv:2212.10012.\n\nJacob Andreas. 2022. Language models as agent models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5769–5779, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.\n\nYusuke Mori, Hiroaki Yamane, Yusuke Mukuta, and Tatsuya Harada. 2019. Toward a better story end: Collecting human evaluation with reasons. In Proceedings of the 12th International Conference on Natural Language Generation, pages 383–390, Tokyo, Japan. Association for Computational Linguistics.", "Typos_Grammar_Style_and_Presentation_Improvements": "- Figure 1: The text examples are too small, please increase as much as possible with the limited space.\n\n- p.5, bottom: \"partcipant\" -> \"participant\"\n\n- Figure 3: I do not understand whether these graphs show all ratings or just a subset (cf. Sec 5.1, first paragraph)"}}]

Author rebuttal:

Meta-review: This paper proposed a new dialogue task on a commercial video game, and conducted user study to analyze and evaluate the behavior of LLM-augmented dialogues and game-designers' writing. 

All the reviewers agree that this paper is both sound and exciting. But they also all have some concerns around the task evaluation and dataset analyses. 

The discussions around the study size (especially around the wording of "large-scale") during rebuttal and analysis of the statistical power should be included in the camera-ready. It would also be good if the authors could discuss how this specific task would shed light on general dialogue research beyond games.
Decision: Accept-Findings