Long-Horizon Dialogue Understanding for Role Identification in the Game of Avalon with Large Language Models
Authors: Simon Stepputtis
Joseph Campbell
Yaqi Xie
Zhengyang Qi
Wenxin Sharon Zhang
Ruiyi Wang
Sanketh Rangreji
Charles Michael Lewis
Katia P. Sycara
Venue: EMNLP 2023 Findings (2023)
Keywords: Dialogue Understanding
Multimodal Reasoning
Long-Horizon Games
TL;DR: We propose a dataset and testbed to investigate how deception and persuasion in a multi-modal multi-party long-horizon game can be utilized to identify hidden player roles.

Abstract: Deception and persuasion play a critical role in long-horizon dialogues between multiple parties, especially when the interests, goals, and motivations of the participants are not aligned. Such complex tasks pose challenges for current Large Language Models (LLM) as deception and persuasion can easily mislead them, especially in long-horizon multi-party dialogues. To this end, we explore the game of Avalon: The Resistance, a social deduction game in which players must determine each other's hidden identities to complete their team’s objective. We introduce an online testbed and a dataset containing 20 carefully collected and labeled games among human players that exhibit long-horizon deception in a cooperative-competitive setting. We discuss the capabilities of LLMs to utilize deceptive long-horizon conversations between six human players to determine each player's goal and motivation. Particularly, we discuss the multimodal integration of the chat between the players and the game's state that grounds the conversation, providing further insights into the true player identities. We find that even current state-of-the-art LLMs do not reach human performance, making our dataset a compelling benchmark to investigate the decision-making and language-processing capabilities of LLMs. Our dataset and online testbed can be found at our project website: https://sstepput.github.io/Avalon-NLU/

Long-Horizon Dialogue Understanding
for Role Identification in the Game of Avalon with Large Language Models
Simon Stepputtis', Joseph Campbell', Yaqi Xie', Zhengyang Qi', Wenxin Sharon Zhang',
Ruiyi Wang', Sanketh Rangreji', Charles Michael Lewis', Katia P. Sycara'
CCarnegie Mellon University, Universityy of Pittsburgh {stepputtis, jcampbell, yaqixie edu, ml@sis. pitt pdu {zqi2, wenxinz3, ruiyiwan, srangrej, sycara)@andrew. cmu edu
Abstract
Collected Game
Moclatiia
Repreacctation
Preducion
Game Chat
Context xbsaled ef
Deception and persuasion play critical role in long-horizon dialogues between multiple par- ties, especially when the interests, goals, and motivations of the participants are not aligned. Such complex tasks pose challenges for current Large Language Models (LLM) as deception easily mislead them, es-
Gome Rep. Round
Chatl
Fill Contect
seep
Gare Stain
Round
State
and persuasion pecially in long-horizon multi-party dialogues. To this end, we explore the game of Avalon: The Resistance, a social deduction game in which players must determine each other's hid- den identities to complete their team's 's objec- tive. We introduce an online testbed and dataset containing 20 carefully collected and labeled games among human players that ex- hibit long-horizon deception in a cooperative- competitive setting. We discuss the capabili- ties of LLMs to utilize deceptive long-horizon conversations between six human players to determine each player's goal and motivation. Particularly, we discuss the multimodal inte- gration of the chat between the players and the game's state that grounds the conversation, providing further insights into the true player identities. We find that even current state- of-the-art L.MSs do not reach human perfor- mance, making our dataset a compelling bench- mark to investigate the prccision-makinn and prooeegiigrcsssinggg capabilities of L.Mss. Our dataset and online testbed can be found at our project website: https://ssteppnn gi thub. io/Avalon-NLLe
<neee>
Figure I: Schematic representation of the Avalon role prediction pipeline. We experiment with three distinct modalities: Chat only. Chat and State, and State only. These representations are provided to the Language Model (I.LM) either by using data from a single round complemented with a carried-over belief (round- based context) from the preceding round or by using the entire history since the game's beginning (full context). Sub- sequent role predictions made by the LLM validated for consistency with the predefined Avalon roles.
vate and possibly competing belief's and agen- das, yet seek to cooperate in service of a shared goal. While humans excel at such tasks and are capable of reaching group consensus even in the presence of bad faith actors and deception, there are significant challenges inherent to the setting which state-of-the thr---rrttt- LLMs are ill-suited to han- dle (Wang et al., 2023b).
In this work, we aim to explore these limitations by introducing new benchmark and associated dataset for multi multi-party cooperative- task based on the game of Avalon: The Resistance Our task features two teams of players with hidden roles and conflicting goals; one side seeks to fulfill series of objectives and the other sabotage them. Through rounds of dialogue and voting, the players must discover each other's identity while conceal- ing their own win. We model the identification of player roles as a long-horizon dialogue under- standing task, and show that it poses a challenge for
1 Introduction
Despite the remarkable progress of large language models (LLMs) in natural language understand- ing and generation, they have largely been ap- plied and evaluated in question-answering (Brown al., 2020), instruction following (Driess al., 2023), and cooperative dialogue (Madotto al., 2020) tasks. An oft-overlooked, yet important set- ting is that of multi-party dialogue in cooperative- competitive scenarios, where participants hold pri-
'A game by Don Eskridge: https: wikipedia. www.kiiThe_Resistaccee

---

recent state-of-the-art LLMs due to three factors: 1) a large number of participants in multi-party dialogue, 2) the need to ground reasoning in both dialogue and game state, and 3) the active usage of deception and persuasion by players.
tity (Gunasekar et al., 2023; Raffel et al., 2020; Longpre et al., 2023), we expect this dataset be useful for both fine-tuning and evaluation of mod- els. During data collection, we have ensured that no spurious information channel exists between the players, ensuring that all conversation relevant to the game is captured by our dataset. Concretely, our contributions are follows:
While prior works have explored dialogue un- derstanding in multi-party settings, they typically involve small number of participants (Zahiri and Choi, 2017) or unnatural dialogue captured from online forums (Lowe et al., 2015). Our task, in contrast, involves natural dialogues with six par- ticipants, each of whom have their own privileged information and agendas. This significantly in- creases the complexity of the task, as participants may address each other over longer horizons. Com- pounding the issue is the need incorporate game state into the reasoning process, further increasing the horizon translated to natural language as is common in zero-shot paradigms. However, the ability to reason over extended horizons is crucial in our context, as inconsistencies in behavior, eva- sive responses, and ceff-contrdiiction which can be used to identify deceptive behavior may only manifest over time.
A testbed and dataset containing 2384 utter- ances from 20 human player games hand- annotated with strategies (persuasion and de- ception), player beliefs, and game state.
A comprehensive analysis of LLM perfor- mance in our proposed multimodal long- horizon dialogue understanding benchmark, including persuasive and deceptive behavior.
An exploration of the limitations of current models and the introduction of state repre- sentations that improve long-horizon dia- logue modeling.
2 Related Work
We explore a selection of recent, publicly avail- able LLMs, varying in size and training data, to evaluate their effectiveness understanding long-horizon relations in our proposed benchmark. Under the assumption that LLMs encode large amounts of general knowledge, we aim to ascertain whether specific state representations can facili- tate the comprehension of long-horizon tasks. We find this to be a critical aspect given the small con- text windows associated with many state-of-the-ar LLMs, despite recent improvements (Dao et al., 2022; Ainslie et al., 2023; Press et al.. 2021; Packer et al., 2023).
Dialogue Understanding in Games: Dialogue understanding tasks such as the identification of intentions and motivations are essential for suc- cessful conversation (Weld et al., 2022), and are associated with broader human cognitive activities. While games have long served challenges in the Al community, they often lack such dialog under- standing due to the difficulty in generating and eval- uating realistic dialog in commonly employed rein- forcement learning paradigms (Vinyals et al., 2019). As result, cooperative games with human partners often utilize simple, codified communication proto- cols such in the game of Hanabi (Siu et al., 2021) or ignored entirely (Serrino et al., 2019). When di- alogue incorporated (Zhao and Eskenazi, 2016; Pang and Wang, 2020), often through dialogue state tracking (DST) (Ren et al., 2018) in which participant beliefs (Oguntola et al., 2023) and in- tentions are modeled as semantic slots (Lee et al., 2021), and are inferred throughout the course of conversation. However, cooperative-compeiitv games involving negotiation, persuasion, decep- tion often require a more nuanced application of language that leaves room for subjectivity and inter- pretation. Past work on strategic dialogue systems has avoided these issues by focusing on simpler set- tings (Lewis et al., 2017; Keizer et al., 2017; Wang
In addition, we publicly release our dataset well as our browser-based version of Avalon in order to promote further research in this direc- tion. Consisting of 20 games with 30 unique hu- man players and 19 unique team compositions, our dataset comprises 2384 pieces of dialog with hand- annotated persuasion strategies for each player, de- ception strategies for evil players, player beliefs over the roles of other players throughout the game, and ground truth game state. To the best of our knowledge, this is the only high-quality dataset of its kind for long-horizon multi-party dialogue featuring deception and persuasion. In keeping with recent trends advocating for quality over quan-

---

al., 2019), which involve only a single partner, shorter dialogue contexts, or simpler strategies.
Though persuasion has often been studied in the context of negotiation (Lewis al., 2017; Keizer et al., 2017) or other two-party dialogues (Wang et al., 2019), multi-party persuasion analysis is of- ten limited to online social media discussions (Al- thoff et al., 2014; Tan et al., 2016) which often meaningfully differ from real-time dialogue.
Recent advances in LLMs have shown great po- tential (Liu et al., 2023; Li et al., 2023) across varying problems, including conversing (OpenAI), knowledge parsing (Jiang et al., 2023; Zhang et al., 2023), and instruction following (Alayrac et al.. 2022: Stepputtis et al., 2020; Xie et al., 2023), leading to the development of capable language- based agents (Team et al.,, 2021; Wang et al., 2023a; Driess et al., 2023). As shown in White et al. (2023). the inherent knowledge embedded in these models' weights allows for intricate answers, the prompt is posed correctly. Such agents have recently been successfully applied to the game of Diplomacy (FAIR et al., 2022; Bakhtin et al., 2022), although a significant source of relevant domain data was required for fine-tuning. This differs from the game of Avalon, in which a) all communication is public and b) hidden roles explicitly encourage deception, which leads to a more challenging di- alogue understanding problem as an agent must carefully understand, analyze, and ground each player's 's utterance.
The Game of Avalon
In this section, we provide description of the game Avalon: The Resistance", which a social deduction board game that can be played by five to ten players. Players assume various roles in the game that differ in knowledge and goals, includ- ing the Assassin, Merlin, Morgana, and Percival. Avalon is a cooperative-competin two groups of players attempt to infer the roles of the other players by forming allegiances while hid- ing or intentionally pretending to be a role different from their own. In this work, Avalon is played among six human players P - {pi,.... .p.......] - four good, two evil. Generally, the goal of the evil play- ers is to hide their identity and to convince the good players that they can be trusted.
game in which
The game progresses through up to five rounds referred as quests with players engaging in discussions and voting to determine the composi- tion of a party that is subsequently sent on a quest. The required size of a party that is sent on a quest differs in each round and has to be approved by a public majority vote amongst all players. After a party has been formed, each player in that party votes anonymously whether or not the quest should succeed, requiring all players to vote for success in order to succeed the quest. If the evil players succeed in failing three quests, they automatically win the game. Similarly, if the good players suc- ceed three quests, they win the game, unless the Assassin can identify who plays Merlin (see Sec- tion A. If the Assassin is successful, evil wins the game instead.
Deception and Persuasion in Dialogue: While deception is increasingly studied in terms of misin- formation on social media (Shu et al. 2017), in this work we focus on its analysis and detection in dia- logue. Unlike prior works which explore deception through analysis of verbal (Hirschberg et al., 2005) visual (Soldner et al., 2019) cues in spoken lan- guage from two-party dialogues, our benchmark is based on textual linguistic cues in multi-party party dialogues. Although datasets have previously been introduced for the games of Mafa (Ibraheem et al., 2022) and One Night Werewolf (Lai et al., 2022), we find Avalon to be significantly more challeng- ing task due to the increased game length, resulting in more than double the number of utterances per game in our dataset- 49, 64, and 119 for Mafia, Werewolf, and Avalon respectively. This requires dialogue models to reason over significantly longer context horizons, but also provides enough infor- mation for us to reason over hidden player roles opposed to simply inferring utterance labels. In addition, we provide high-quality annotated per- suasion strategy labels for each utterance in our dataset, following annotation schemes introduced in prior works (Yang et al., 2019; Lai et al., 2022), as well as deception labels, covering the type of lies utilized by the evil players (Houston et al., 2013).
Avalon: The Resistance introduces multiple spe- cial roles that impact gameplay, adding layers of strategic depth to the game's 's dynamics. In our set- ting, the roles consist of Merlin, Percival, and two Loyal Servants as the forces of good, with Morgana and the Assassin on the side of evil. Detailed role descriptions can be found in Section A.1 We further impose additional rules to facilitate
'The game's rules can be found here: https://www. Altrabbardgames. com/ htt/aavalon/ggaaa watp///aarrrreeseeeeeeeeee phpp

---

Spke
limited. Esspecillly in situations in which deception behavior is present, as in Avalon, identifying such behavior hinges on multiple factors. For example, consistency when voting for parties depending on which players are on it, as well as identifying in- consistencies in a player's arguments. These serve as an indicator for deceptive behavior, and con- sequently, the true nature of the player's charac- ter We propose to break up conversations into "rounds" r representing single round of discus- sion in which each player has had the chance to speak at least once. Breakpoints are introduced at the current quest leader's position. An example round can be seen in Figure 2. In this example, we propose that players' roles can be predicted given only the current round ra and a structured belief state b about each player's identity that carries over between rounds. To track the identities of players across rounds, the model is provided with list of initial player beliefs = [b..... be]. where each belief 6, E {good, evil, merlin, unknown], as con- text. The context is provided prior to consuming the next round of conversation. Finally, the model is tasked to provide an updated list of player beliefs dnext after each round, which subsequently serves as input for the next round r2+1-
Game Started
We don't have any info so just selected rendomly
Spite
Player-4 proposed party: Player-2, Player-6
Seems like weird choice
my apinion. Little
suspicious player-4 for not picking themself.
don't have opinions this point
Well, technically we don't know anything yet. but agree that this choice is a little weird
I'm good, but agrea this choice waird, Howaver, don't know what changee we make current party unless you some good arguments
first turn. approve
Syston F3
No apinions but a good guy will always place themselies in the team...
Pleyer-4 propoeed party: Player-2, Player-4
Sorry for the last turn, its etil random but just included myself
Party Vate Outcome: Player-1: Yes, Player-2 Yes, Player-3:
No, Player-4: Yes, Playar-6: Yes, Player-8:
Syster Syowe
Succeeded! Initieting Quest Vote!
Quest Succeeded!
Figure 2: Example of the players' conversation during round one. Player P4 is the quest leader.
the online nature of our data collection. Particularly, we enforce a turn- based discussion in which each player has a fixed amount of time to convey their thoughts via a chat interface, thus ensuring that all game-related interactions are captured. Should a player exceed their allotted time, the game will automatically progress to the next player, initiate appropriate votes, apply default votes depending on the situation. More details can be found in section A.2
Figure/ describes the three modalities we are building from the game's s chat and game state. Each of these modalities can be used in either context representation. Particularly, the game either con- veyed the LLM the complete history which we refer to as full context - from the beginning of the game up to the point of evaluation or as a round with a carried-over belief b holding the role pre- dictions after the previous round, which we refer to as round- based context. The respective context is then utilized by the LLM through TypeScript (see Section 4.3.1), allowing for validation of the generated response, ensuring that valid role has been assigned to each player.
4 Dialogue Understanding
We formulate our problem as identifying player's character 6 given the chat history heha and game state hstare- Formally, predict the role (c, hehat haaee) for each player in P. Addi- tionally, we also predict which of the six players is Merlin: PMertin (pMerli heeaa, haaee, e) where e is the privileged knowledge of the evil players.
4.2 State Representation
In the remainder of this section, we address the two fundamental issues of 1) understanding long- horizon conversations, and 2) utilizing the environ- ment's state enhance the inference capabilities of the LLM.
Recent LLMs have shown impressive, human-like, conversational capabilities and are increasingly de- ployed as public-facing conversational agents (Ope- nAl). However, the shortcoming of these models is that their interactions are mostly cooperative and they assume that language does not need to be grounded in the world's state. With our testbed and accompanying dataset, we provide an intrigu- ing, yet simple world in which language can be
4.1 Long-Horizon Dialogue Representation
Long-horizon conversations between multiple par- ticipants pose challenges for LLMs as their capa- bility for comprehending and tracking context is

---

grounded in structured state representation of the relevant environment.
system: are a helpful assistant that uses the chat be- tween six players, player-1 to player-6, who play Avalon: The Resistance (a cooperativ--competiin tify who is Merlin, Good or Evil. There are two evil play- ers, which can usually be found because they are doceptive and lie about the good player's roles and vote for quests and parties irrationall.. For Merlin, watch out for individ- uals with knowledge of evil players' identities, insightful comments beyond their role, and caution regarding mission teams or specific players.
game) to iden-
In the game of Avalon, the relevant state infor- mation includes the currently proposed party, the record of successful and failed quests, the play- ers that have been part of parties in prior rounds, well as previous party formation votes. At its core, we convert state information into linguistic statements that are given to each LLM prior to the player's conversations. By utilizing this tech- nique, the LLM can leverage its inherent zero- shot and few-shot capabilities to solve our task of identifying hidden player roles by interleaving spe- cially designed inputs, with little to no task-specific fine-tuning. In Avalon, states are conveyed to the LLMs in two manners: 1) through global game state that incorporates past quest outcomes, and 2) state changes that happen in the currently evaluated round.
human: The current state is: <SIATE> (See Section 4.2.1) The current party proposal is: <PARTY> (See Sec- tion 4.2.1)
Your initial belief is: <BEIJEF> (See Scction 4.1)
This is the chat between player-1 to player-6:
<CHAT> (See Section 4.2.2)
What do you think is the role of each player? Please do not explain your answer, do not elaborate on it fürther, and do not say that these are just guesses; only provide the list and nothing else.
Table I: Prompt used for role inference.
user provides updates about the game's global state during the round.
4.2.1 Global State
The game's global state is converted into text by uti- lizing a set of templates. Most importantly, the out- come, as well as parties and votes for prior quests are translated as follows:
4.3 Tasks and Prompt Generation
The prompts used to predict roles Prote and predict Merlin PMetii have been hand-designed to covey the basics of the game as well as the expected out- put format. The prompts for our two prediction tasks are shown in Tables I and 4, respectively.
quest-i: oF (party: Pj player votes: Uz))
where i is quest q ID, E {success, failure} is the outcome of quest, Py is the list of players that went on quest qi and 2y is list of tuples including the vote from each player of whether or not they approved of party Pj. For example, the game's s's state after the first successful quest conducted by player one and two could look as follows: "quest-I: success (party: player-l, player-2 I player votes: player-1: yes, player-2: yes, player-3: yes, player- 4: yes, player-5: yes, player-6: yes)". Further, the global state also covers the currently proposed party Preoussed in order to provide context for the players' discussion.
Each prompt is designed such that various in- formation can be given dynamically the models for inference. In particular, we are evaluating the impact of providing information about the game's s state to the model. Prompts shown in Tables and 4 demonstrate the case in which chat hehat and state hstate information are given. If only chat is used, lines indicating information about the <STATE> and current <PARTY> are omitted. Further, mes- sages from the 7-th "system" player are removed from the <CHAT> information, thus only retain- ing the players' utterances. Similarly, if only the game's 's state is desired, utterances from all players are removed from the <CHAT> information.
4.2.2 Round-Batd State Change
In addition to the game's global state each round of conversation can be accompanied by proposing par- ties, voting for such parties, and conducting quests. Such changes have an effect on the global state; however, the global state only represents the game prior to the currently ongoing discussion. Thus, changes that occur during round are communi- cated through a 7-th player named "system". Fig- ure 2 demonstrates the conversation between the six players during round one. Here, the system
4.3.1 Structured LLM Outputs
While very powerful, LLMs struggle adhering to structured outputs, as, for example, required in our role-prediction task. An LLM tasked to pro- duce bullet-point list of player names followed by one of the roles results in varying answer quali- ties ranging from half-populated lists to free-form responses with no discernible list, making subse- quent utilization of the predictions difficult. How- ever, many LLMMs exposed to code during training

---

cаеtttz logical deluction
The dataset released with this paper contains over 2300 utterances and corresponding hand- labeled persuasion and deception strategies. Games take an average of I hour and 12 minutes, with the shortest game lasting only 18 minutes and the longest game over 3 hours. Seven of the 20 games were won by the good players, while 13 were won by evil - nine of which were won through the As- sassin correctly identifying Merlin.
custtonigg
eapoestion
sppresllebeoe
Quiedy
300
250
Figure 3: Distribution of persuasion strategies.
can generate JSON strings and understand simple code blocks. To capitalize on this feature, we uti- lize TypeChat (Microsoft, 2023), which, given a schema definition, tasks LLMs with answering all user queries with a JSON string that conforms to the provided schema. A benefit of this approach is that the LLM response can be verified to satisfy the desired response schema by utilizing the Type- Script compiler. In cases where the response is incomplete or otherwise insufficient, TypeScript's compiler error message can be used to formulate a follow-up query to the LLM to generate a valid response. We utilize this approach to generate role predictions for each player, as well as for predicting which player is Merlin. Our schema definitions can be found in Section C.3 and enforce that a valid role is predicted for each player. However, our schema does enforce that: for example, the role of Merlin can only be assigned to a single player.
We split the dataset into 14 training and six test- ing games. The test games were selected such that they contain three good victories and three evil victories, where two of the three evil victories are through the assassin and one through successfully failing three quests. To form human baseline, we created Google Forms surveys and asked players not involved in the test games to label the roles (see Section A.3) by reading over the recorded games from the perspective of an external observer.
5.1.1 Data Processing
Due to the chat-based nature of our data collection process, the collected utterances are reasonable quality. However, we clean the data with an auto- matic spell-checker that corrects any word with a Levenshtein Distance between an unknown word and an English target dictionary. Similarly, player names are corrected with custom dictionary con- taining the correct spelling of the player's names. After this procedure, the remaining unknown words are corrected manually, which mostly contain ab- breviations of long or unusual player names. Af- ter spelling and player names are corrected, the data is anonymized by replacing player names with "player-X" where X is the index of the player in each particular game. Note that this means that in two different games, player-1 can refer differ- ent person. This replacement was done not bias the models towards certain players having identifi- able play styles.
5 Experiments
The following sections detail our data collection process and provide comparisons between three current state-of-the-an LLMs, including fine-tuned versions of these models, in utilizing different modalitics and context representations.
5.1 Data Collection
We created a browser-based version of Avalon: The Resistance that allows for unsupervised data col- lection from six participants. During the game, we collect the conversations between the players and the state of the game throughout each session. Further, we instructed participants to self-label per- suasion strategies (see Section A. 1.4 for details) for each of their chat messages, as well as indicating beliefs about the other players' roles throughout the game. In total, we collected 20 games over 24 hours of in-game time from a total of 30 unique par- ticipants, forming 19 different teams of six players. Participants were required to be least 18 years old and familiar with the general rules of Avalon. The Office of Research Integrity and Compliance sanctions the experiment and data collection.
5.1.2 Strategy Labels
Every utterance in our dataset is accompanied by a label indicating one of our eight different per- suasion strategies (see Figure and Section A.1.4). Further, if not truthful, utterances of evil players are labeled with an additional deception strategy label, covering the three common types of lies (Houston et al., 2013): commission, omission, and influence. Table in Section D.2 shows the prediction capa- bilities of GPT-4, GPT-3-Turbu and Llama-2-13 13b of our eight prediction strategies, including fine-

---

Table F1 scores for role prediction and identifying Merlin given privileged knowledge. We report results using the round-based (first value) and full context (second value). Models evaluated ten times the six test games.
Modalities
All-Role Prediction Evil
Evil find Merlin
Model
Familiar
Trained
Chat
State
Good
Merlin
Final
Anytime
Gpt-4
0.67 0.67
0.48 0.55
0.36/ 0.20
0.17 0.17/0.17
0.83 0.67
2
Gpt-4
0.59 0.57
0.20 / 0.33
0.06 00.11
0.17 0.33
0.17 0.50
3
Gpt-4
0.67 0.68
0.46 0058
0.05 0.27
0.00 0.00
0.67 / /0.50
4 gpt-3.5-turho
0.68 0.60
0.46 0.40
0.23 /.11
0.17/0.17
0.33 0.50
5 ggpt-3.5-turbo
0.57 0.47
0.46 0.30
/0332
0.17/0.11
0.17 0.11
6 gpt-3.5-turro
0.58 0.65
0.34 0.47
0.23 /0.13
0.17 0.17
0.33 0.33
7
3.5t-3.5-turb
0.52 0.59
0.38 0.41
0.17 0.19/0.15
1.00 0.67
8
Llama-2
0.68 0,61
0.39
0.00
0.17 0.00
0.17 0.11
9
Llama-2
0.41 / 0.62
0.00 0.34
0.00
0.00 0.00
0.00 /0.17
Llama-2
0.61 / 0.55
0.33 / 0.22
0.00
0.17 / 0.00
0.17 0033
Llama-2
v
0.65 / 0.63
0.35 0.26
0.23 /.227
0.33/ 0.00
0.33 0.00
12
Random
0.50
0.34
0.17
0.17
0.60
13
Human
0.76
0.72
0.33
0.5
0.67
tuned versions of GPT-3.5 and Llama-2. We find that fine-tuned GPT-3.5 performs well (micro-fl: 0.43) when predicting the strategies for each label, followed by the vanilla version of GPT-4 (miero- fl: 0.37). However, Llama-2-13b underperforms (micro-fl: 0.15), even in the fine-tuned version (micro-fl: 0.20).
the perspective of an external observer, but also on the task of identifying Merlin, given the privileged knowledge of who the evil players are. Each evaluation is conducted with three different sets of modalities, including the game's state, player chat, and a combination of game state and player chat. Our inherent assumption is that these LLMs possess enough encoded knowledge in their pre- trained model weights such that inference for our task is possible. To assess the pre-trained model's 's capability, we evaluate each model's familiarity with the rules of Avalon. We run three prompts and list selected answers in Table Responses to all three prompts are evaluated by a human for correctness and adherence to the game's 's rules and are listed in Table 2.
5.2 Turn Order and Utterances
For the 20 games we conducted, roles have been randomly assigned to each of the six players so as not to bias the data towards certain advantageous positions of Merlin or evil players in the turn or- der. We found that identifying good players is easier if Merlin is among the first three players (p-value 0.039). Similarly, if Morgana Percival speaks a lot, good players are easier identify (p-value 0.046 and 0.021, respectively). We also found that evil players who uttered lies more fre- quently had a statistically significant advantage of identifying Merlin correctly (p-value 0.015). We conjecture that Merlin's 's attempts to counteract the lies of evil players has high likelihood of reveal- ing his identity to evil players. Similarly, we found statistically significant influence of the number of Percival's utterances and the likelihood of evil winning (p-value 0.018).
Table demonstrates the performance of var- ious LLM approaches. Due to the probabilistic nature of the LLMs, each model was evaluated ten times on each of the validation games. Human per- formance (line 13) was evaluated by asking three human annotators for each of the six evaluation games to provide their estimates about the roles of each player in each game. Further, the performance of finding Merlin and identifying Merlin correctly at any point in the game was directly taken from the game's 's belief annotations, while the final pre- diction was taken either from the assassin's 's choice (if applicable) or the last anytime estimate. Table 2 presents the F1-score when providing information about the game in round-based context with a carried-over belief (first number) and when provid- ing the entire context from the beginning of the game (second number). In comparison, the round-
5.3 Comparing Model Performance
We compare three state-of-the-art LLMs, namely GPT-4 (OpenAI, 2023), GPT-3.5-turbo, and Llama- 2-13B (Touvron et al., 2023), as well as a fine-tuned version of 1lama-2-13B and GPT-3.5-turie Each model is used to predict the roles of players from

---

L.bma-2-138 (PT)
LJamp-138
Tarro (FT)
CPT33 -.Turbo
GPT4
Human (Pall History)
Good
6:93
0.07
8.97
6:16
0.36
0.12
0.38
Moyha Actual Roin
Eivil
6.06
0.31
0.00
2,63
0.34
0.00
0.50
0.30
0.72
0.56
0:35
0.10
0.61
0.39
0.00
0.23
6.70
0.7
6.35
0.32
2.85
0.30
0.48
0.23
0.50
9.38
0.40
0.2
Fbend
6.97
0.01
0.02
2.45
0.05
0.01
0.58
0.26
0.34
0.71
0.20
0.09
0:77
0.39
0.16
0.01
0.09
0.10
Aerus Rale
Merton Eyll
0.13
0.32
2.24
0.40
6.30
0.47
5.06
0.40
9.33
0.23
0.77
0.05
0.30
8.76
0.33
6:15
0.46
8.09
0.78
0.66
0.29
Good
Manin
Good
Perl
Good
Predicted Meriin
Good
Verfin
Good
Foni
Goond
Meriin
Prodicted
Prodicted
Prodicted Rule
Predicted
Prodicted
Figure 4: Confusion matrices evaluating LLMs in the two context conditions using Shat+State: round- based (top, blue) and full-context (bottom, orange). Human results (right, purple) are for the full context in both cases.
based context utilizes average of 974 (std: 333, max: 1941) tokens, while the full context utilizes an average 2844 (std: 2011, max: 8556) tokens.
predictions, particularly the two different context modes (round- and full-context) when using the joint modality. Particularly with round-base context, GPT-3.5-Turbo (Figure 4, top) suffers by mostly confusing evil players with good ones and identifying Merlin as evil. On the other hand, pro- viding the full context (Figure 4, bottom), does not lead to such confusion about Merlin, where he is mostly confused with good player. We conjec- ture that identifying players' underlying intentions. especially when players actively try to hide their identity, remains a challenging task for LLMs.
While most models and modalities outperform randomly guessing player roles, only GPT-4 in the round-based context utilizing chat (line 1) outper- forms the human baseline (line 13) when identi- fying Merlin among the players. Comparing each model's performance when identifying good and evil players, we notice a performance drop for evil players they actively hide their identity to appear good players. A similar trend can be observed in the human baseline (line 13), however, humans unmatched in their ability to identify them. Gener- ally, this trend is also observed, greater extent, the ability to identify Merlin.
5.3.1 LLM Fine-Tuning for Role Prediction In addition to testing the performance of pre-trained models, we have fine-tuned GPT-3.5-Turbo as well as Llama-2-13b. Training data for fine-tuning was generated from the remaining 14 games of Avalon that were not used for evaluation. We have trained each model for three epochs across these 14 games and report their performance in Table 2 and Fig- ure Particularly in the case of Llama-2 2, we notice that fine-tuning improves the model's performance when predicting the roles of all players. For GPT- 3.5, we do not observe the same trend: however, when predicting Merlin on its own, GPT-3.5 dra- matically improves in performance, even outper- forming GPT-4 and the human baseline. When comparing the failure cases in Figure fine-tuning reduces failure severity, particularly for Merlin, lim- iting failed predictions to good players, particularly in the case of Llama-2.
For GPT-based models, when predicting good and evil roles, we notice that combining game chat and game state information results in the highest Fl scores (lines 3 and 6) when providing the full context, confirming our hypothesis that ground- ing game chat in the game's s state improves perfor- mance. However, when utilizing the round- based context, individual modalities tend to outperform their full-context equivalents (lines 1, 2, 4, and 5). yet do not reach the performance of the joint modality. Further, we observe that predicting Mer- lin benefits from the availability of the game state (lines 2 and 5).
In the case of Llama-2, the best performance is achieved when only utilizing individual modalities (lines 8 and 9) while not allowing any conclusive insight as to which modality tends to perform better, despite significant differences (line 9 (good) and line 10 (evil)) in their respective performance).
5.4 Comparison to other Deception Tasks
We have also compared the prediction capabili- ties of pre-trained models on existing datasets social-deduction games, namely the game of Were-
Figure 4 provides further insights into the role

---

Table 3: Fl scores for role predictions in the game of Werewolf. Our results show enhanced role prediction scores compared to our Avalon data, suggesting a more challenging social deduction environment Avalon.
utterances from six human players at a time, hand- labeled persuasion and deception strategies, player
beliefs, and comprehensive game states recorded over more than 24 hours of gameplay. We demon- strate that current state-of-the-ar LLMs do not reach human-level performance in environments that require the understanding and tracking of long- horizon dialogue between multiple participants in challenging social cooperative-comppettiv settings.
Roles
Model
Tuned
Good
Evil
Seer
Gain
1
GPT-4
0.66
0.66
0.49
+0.28
2
GPrbo
0.53
0,56
0.35
+0.19
3
GPrb-..--Trbb
V
0.61
0.72
0.43
+0.61
4
Llama-2-138
0.43
0.47
0,00
+0.13
5 Llama-2-138
0.45
0.39
0.08
-0.24
Compared similar datasets, our benchmark con- tains longer context horizons, stricter game rules, and high-quality dialogue, making well-suited for NLU research as all game-relevant communication has been captured and is thus, available to learn- ing algorithms. This dataset opens doors for di- verse research avenues, from detecting deception to developing conversational agents for cooperative- competitive settings. We hope that our high-quality dataset will be valuable further tune and eval- uate the capabilities of large language models challenging social settings.
wolf (Lai et al., 2022). Table 3 shows the results in a similar setting, grouping the players into Good, Evil, and Seer by designating roles as Evil if ly- ing and deception are a part of their strategy and keeping the Seer separate role akin to Merlin in Avalon. Predictions for individual roles with the best model, GPT-4, are available in Table 7.
We demonstrate that in the setting of Werewolf, LLMs perform well predicting the players' roles (see "gains" in Table 3), which we hypothesize is due to the shorter horizon (Avg. 1786 tokens, 2795 max) of each game as compared our Avalon dataset (Avg. 2844 tokens, 8556 max), making the task of identifying roles easier. We find that fine- tuning dramatically improves the performance of GPT-3.5 over three epochs (line 3 vs. line 2) of training, while Llama-2-13B fine-tuning does not significantly affect performance (line 5 vs. line 4). This observation is contrary to our Avalon dataset (see Table 2 lines 7 and 11), where Llama-2, in par- ticular, improved its performance, while GPT-3.5 roughly maintained its performance, resulting in a large difference in gains (+0.61 vs. -0.24). We hy- pothesize that with the high-quality data provided in our Avalon dataset, fine-tuning Llama-2 allows it to achieve performance levels comparable to an untuned GPT-3.5 (see Table 2). Yet, when faced with the Werewolf dataset, 'naturalistic-social. setting' characterized by frequent cross-talk and off-topic exchanges, Llama-2 struggles to fine-tune effectively. In contrast, the broad training founda- tion of GPT-3.5 enables it to learn even under such conditions.
Limitations
The performance of our evaluated models depends on the capabilities of the pre-trained language mod- els. Given our poopeaative-compettti multi-party scenario, we observe that current approaches do not reach human-level performance. We also be- lieve that the fine-tuning results could be further improved by extending our dataset; however, we believe that future work will utilize it successfully.
Ethics Statement
Participants in our IRB approved human-subject study were provided consent forms and privacy no- tices explaining how we intend to use the data col- lected during the experiments. We strictly adhere to applicable data protection policies and use the data solely for research purposes. Further, offered opportunities for participants to seek clarification and ask questions, fostering informed consent and ethical, trustworthy interactions.
A Acknowledgements
This work has been funded in part by the Air Force Office of Scientific Research (AFOSR) un- der grants FA9550- 188500-8---0251 and FA9550-18-1- 0097, the Army Research Laboratory (ARL) un- der grant 1911NFF19-2-0044 DARPA award HR001 120C0036, and the Office of Naval Re- search (ONR) award N00014-23-1-28444
6 Conclusion
In this work, we present novel benchmark, associ- ated dataset, and testbed for horg-horizon dialogue understanding in scenarios of conflicting interests among multiple participants. This task combines

---

References
Michaelis, et al. 2005, Distinguishing deceptive from non non-deceptive speech.
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañóón, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, et al. 2023. Colt5: Faster long-range transform- ers with conditional computation. arXiv preprint arXiv:2303.09752
Philip Houston, Michael Floyd, Susan Carnicero, and Don Tennant. 2013. Spy the lie. Saint Martin's Grif- fin, New York, NY.
Samee Ibraheem, Gaoyue Zhou, and John DeNero. 2022. Putting the con in context: Identifying de- ceptive actors in the game of mafia. arXiv preprint arXiv:2207.02253.
Jean -an-ttttstte Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: visual language model for few-shot learning. Advances in Nearal Information Processing Systems, 35:23716-2377
Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, and Ji-Rong Wen. 2023, Struct- gpt: A general framework for large language model to reason over structured data. arXiv preprint arXiv:2305.0965.
Tim Althoff, Cristian
Niw.ssuu-Neuueeeuu--M Mizil, and Dan Jurafsky. 2014. How to ask for favor: A case study on the success of altruistic requests. In Pro- ceedings of the International AAAI Conference on Web and Social Media, volume 8. pages 12-21.
Simon Keizer, Markus Guhe, Heriberto Cuayáhuitl. Joannis Efstathiou, Klaus-Peter Engelbrecht, Mi- hai Dobre, Alex Lascarides, Oliver Lemon, et al. 2017. Evaluating persuasion strategies and deep rein- forcement learning methods for negotiation dialogue agents. ACL
Anton Bakhtin, David J Wu, Adam Lerer, Jonathan Gray, Athul Paul Jacob, Gabriele Farina, Alexan- der H Miller, and Noam Brown, 2022. Mastering the game of no-press diplomacy via auman-regularized reinforcement learning and planning.
Bolin Lai, Hongxin Zhang, Miao Liu, Aryan Pariani, Fiona Ryan, Wenqi Jia, Shirley Amugrah Hayati, James M Rehg, and Diyi Yang. 2022. Werewolf among us: A multimodal dataset for modeling per- suasion behaviors in social deduction games. arXiv preprint arXiv:2212.0822
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. 2021. Dialogue state tracking with language model using schema-driven prompting.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances Neural Information Processing Systems, 35: 13:54
Mike Lewis, Denis Yarats, Yann N. Dauphin. Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning for negotiation dialogues.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu. et al. 2023. Palm-e: An embodied multimodal language model. arXiv preprint 203.03230..
Huao Li, Yu Quan Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Michael Lewis, and Katia Sycara. 2023. Theory mind for multi-agent col- laboration via large language models. In The 2023 Conference Empirical Methods in Natural Lan- guage Processing.
Meta Fundamental AI Research Diplomacy Team FAIR. Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. 2022. Human- level play the game of diplomacy by combining language models with strategic reasoning. Science, 3786624)
Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, al. 2023. Summary of shatgpt/gpt-4 research and perspective towards the future of large language models. arXiv preprint 2rXiv:2304.0133 008522.
Suriya Gunasekar, Yi Zhang. Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmaan, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang. Sébastien Bubeck. Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. 2023. Textbooks are all you need.
Shayne Longpre, Gregory Yauney, Emily Reif, Kather- ine Lee, Adam Roberts, Barret Zoph. Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. 2023. A pretrainer's guide to training data: Measur- ing the effects of data age, domain coverage, quality. & toxicity, arXiv preprint arXiv:2305.1316.
Ryan Lowe, Nissan Pow, Tulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research unstructured multi-turn dia- logue systems. arXiv preprint
Julia Bell Hirschberg. Stefan Benus, Jason M Brenier, Frank Enos, Sarah Friedman, Sarah Gilman, Cynthia Girand, Martin Graciarena, Andreas Kathol, Laura

---

Andrea Madotto, Zihan Liu, Zhaojiang Lin, and Pascale Fung. 2020. Language models as few-shot learner for task-oriented dialogue systems. arXiv preprint arXiv:2008.06239
Simon Stepputtis, Joseph Campbell, Mariano Phiclipp, Stefan Lee. Chitta Baral, and Heni Ben Amor. 2020. canguage-oondttioned imitation learning for robot manipulation tasks. In Advances Neural Informa- fion Processing Systems, volume 33, pages 13139- 13150. Curran Associates, Inc.
Microsoft. 2023. TypeChat.
Ini Oguntola, Joseph Campbell, Simon Stepputtis, and Katia Sycara. 2023. Theory of mind as intrinsic motivation for multi-agent reinforcement learning.
Chenhao Tan, Vlad Niculae, Cristian Danescu- Niculescu-Mizie and Lillian Lee. 2016. Winning ar- guments: Interaction dynamics and persuasion strate- gies in good-faite faith online discussions. In Proceedings of the 25th international conference on world wide web, pages 613-624.
OpenAl. Introducing chatgpt.
OpenAI. 2023. Gpt-4 technical report.
DeepMind Interactive Agents Team, Josh Abramson, Arun Ahuja, Arthur Brussee, Federico Carnevale, Mary Cassin, Felix Fischer, Petko Georgiev, Alex Goldin. Mansi Gupta, al. 2021. Cre- ating multimodal interactive agents with imita- tion and self-supervised learning, arXiv preprint 1iXiv:2112.0376.
Charles Packer, Vivian Fang, Shishir G. Patil, Kevin Lin, Sarah Wooders, and Joseph E. Gonzalez. 2023. Memgpt:
Towards Ilms as operating systems.
Wei Pang and Xiaojie Wang. 2020. Visual dialogue state tracking for question generation. In Proceedings of the AAA/ Conference Artificial Intelligence, volume 34, pages 11831-11838.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale,
Ofrr Press, Noah A Smith, and Mike Lewis. 2021. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108. 12409.
2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09220 0.22888.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathicu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds. Petko Georgiev, al. 2019. Grandmaster level in starcraft using multi-agent reinforcement learning. Nature, 575(7782):350-35 354.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-5551
Guanzhi Wang, Yugi Xie, Yunfan Jiang. Ajay Man- dlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and An- ima Anandkumar. 2023a. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv: 2x055-230..16 16291.
Liliang Ren, Kaige Xie, Lu Chen, and Kai Yu. 2018. Towards universal dialogue state tracking. arXiv preprint
Jack Serrino, Max Cleiman-Weinee David C Parkes, and Josh Tenenbaum. 2019. Finding friend and foe in multi-agent games. Advances in Neural Information Processing Systems, 32.
Shenzhi Wang, Chang Liu, Zilong Zheng. Siyuan Qi, Shuo Chen, Qisen Yang. Andrew Zhao, Chaofei Wang. Shiji Song, Gao Huang. 2023b. Avalon's game of thoughts: Battle against deception through recursive contemplation.
Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake news detection on social dia: A data mining perspective. ACM SIGKDD ex- plorations newsletter, 19(1):22-36.
Xuewei Wang. Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang. Jingwen Zhang, and Zhou Yu. 2019. Per- suasion for good: Towards a personalized persua- sive dialogue system for social good. arXiv preprint arXiv: 1ww.1990622...
Ho Chit Siu, Jaime Peña, Edenna Chen, Yutai Zhou, Victor Lopez, Kyle Palko, Kimberlee Chang, and Ross Allen. 2021. Evaluation of human-ai teams for learned and rule-based agents hanabi. Advances in Neural Information Processing Systems, 34:16183- 16195.
Henry Weld, Xiaoqi Huang. Siqu Long, Josiah Poon. and Soyeon Caren Han. 2022. A survey of joint intent detection and slot filling models in natural language understanding. ACM Computing Surveys. 55(8):1- 38.
Felix Soldner, Verónica Pérez-Rosas, and Rada Mihal- cea. 2019. Box of lies: Multimodal deception de- tection in dialogue.. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Pupers), pages 1768-1777.
Jules White, Quchen Fu, Sam Hays. Michael Sandborn. Carlos Olea, Henry Gilbert, Ashral Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A prompt pattern catalog to enhance prompt engineer- ing with chatgpt.

---

Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. 2023. Translating natural language planning goals with large-language models. CoRR, abs/2302.051225
Diyi Yang, Jiaao Chen, Zichao Yang. Dan Jurafsky, and Eduard Hovy. 2019. Let's make your request more persuasive: Modeling persuasive strategies via semi- supervised neural nets on crowdfunding platforms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 3620-3630.
Sayyed M Zahiri and Jinho D Choi. 2017. Emo- tion detection tv show transcripts with sequence- based convolutional neural networks. arXiv preprint arXiv:/708.04299.
Xijia Zhang, Yue Guo, Simon Stepputtis, Katia Sycara, and Joseph Campbell. 2023, Explaining agent behav- ior with large language models.
Tiancheng Zhao and Maxine Eskenazi. 2016. Towards end-to-end learning for dialog state tracking and man- agement using deep reinforcement learning. arXiv prepriat 1606:6000025000

---

A Player Instructions
turns. During each player's turn, they will be able to communicate through the chat window, respond- ing to previous questions or accusations, making new statements, and asking questions to other play- ers. However, only the player whose turn it is will be able to use the chat. Please only communicate with the players within the interface provided in this version of the game.
The following sections provide further detail about the rules of Avalon, as well as the hidden knowl- edge imbued to every special role in the game.
A.1
In-Game Instructions
A.1.1 How to Play
Avalon:
The Resistance is the game of hidden loy- alty. Players are either Loyal Servants of Arthur fighting for Goodness and honor or aligned with the Evil ways of Mordred. Good wins the game by successfully completing three Quests. Evil wins if three Quests end in failure. Evil can also win by as- sassinating Merlin at game's s end if a Quest can- not be undertaken. Players may make any claims during their turns. Discussion, deception, accusa- tion, and logical deduction are all equally important in order for Good to prevail or Evil to rule the day. Before playing a game, please check out the official rules. The differences in this version of Avalon in comparison these rules will be explained below.
After you send message to the chat, you can in- dicate the strategy/mmotivation you used when writ- ing the message. While not mandatory, it would be great if players would provide these insights. The information provided in the strategy selection will not be shared with other players, so don't worry about saying that you potentially lied about some- thing!
A.1.4 Communication Labels
These are the labels and explanations available for your communication
Assertion his includes statements where the speaker makes an assertive remark, expresses a firm belief or makes a definitive statement.
A.1.2 Roles
this version of Avalon, you will play with five fixed roles:
This includes any questioning other players regarding their behavior.
Questioning
Merlin Merlin is Good's side and knows the evils' identity. The portrait frames of the evil people will have red circle around them.
This includes all instances where the speaker suggests an action, makes a proposal, or gives advice to another player.
Suggestion
Percival Percival is on the side of Good and gets information about two players. These two players are Morgana and Merlin; however, Percival does not know who is who. These two players will have red-and-blue circle around them.
Agreement This encompasses sentences where the speaker agreeing with another player's state- ment or strategy, affirming their own identity or someone else's.
Servants Servants are on the side of Good but do not have any special knowledge or ability.
Logical Deduction This includes statements where the speaker provides reasoned explana- tion, defends a point of view, elaborates a strategy or justifies their actions.
Morgana Morgana plays the side of Evil and knows who the other evil player is, indicated by red circle around their profile picture. Morgana appears potential Merlin Percival.
Compromise/Concesion This category in- cludes any sentences where the speaker concedes to another's 's point view, expresses indecision, or appears to back down from a previously held stance.
Assassin Assassin plays on the side of Evil and knows who the other evil player is, indicated by red circle around their profile picture. At the end of the game, should Good win, the Assassin can win the game for Evil by correctly identifying who Merlin is.
Critique/Opposition This includes instances where the speaker critiques another's 's view, coun- ters an argument, or points out an inconsistency or flaw in another's reasoning.
A.1.3 Communication
For the purposes this research study, communica- tion between the players will be conducted through
This category covers situations where the speaker appeals to others for trust, to be
Appeal/Defense

---

included in the quest, uses emotional/personall appeal to gain favor.
the time run out prior a decision, parties, and quests will automatically be approved or succeeded. Lastly, the outcome of the game requires the assassin has 200 seconds to evaluate the game's s chat history and decide on who they think is Mer- lin, and conduct an assassination. Failure to choose player potential Merlin will automatically fail the vote, proceeding to victory for the good play- ers.
A.1.5 Turns
The player with the crown is the current quest leader, while the jester's hat indicates whose turn it currently is. Each turn is limited to 100 seconds, after which the game automatically transitions to the next player. Similarly, if a vote is necessary. you will have 30 seconds to cast your vote. After that, parties/quests will be approved automatically for players that didn't I vote.
A.3 Human Baseline Labeling Instructions
On the following page, you will get see the log of a game of Avalon, including a description the current game state in the form of the "system" messages. Please provide your role estimates for each player.
The quest leader has allow one round of dis- cussion prior to being able to initiate a party vote. However, party proposals can be changed whenever it is the turn of the leader.
A.1.6 Parties
After each round of discussion, you will have the opportunity to indicate your beliefs about the role of each player.
Parties are indicated by a little shield icon next to a player's profile. If there is shield, players are part of the party.
Take your time to carefully analyze the avail- able information and make your best judg- ments about each player's role.
A.1.7 Selecting Players
Players can be selected for party or by the Assas- sin to indicate who they think is Merlin by clicking on their player profile picture frame. However, remember confirm your choices. Clicking on player profile frames alone will only count as choice if the choice is explicitly confirmed.
Avoid accessing or considering information from future rounds or pages when making your decisions. Focus only on the information available up to the current round.
Once you have made belief label for player particular round, refrain from changing that label later, even you change your mind about it in subsequent rounds.
A.2 Differences to Standard Avalon
To the games effectively in an online fashion, we incorporate few additional rules that are out- lined in this section. Particularly, discussions are conducted in a turn-based manner, enforcing that
B Surveys
only one player can talk at time. Conversations are captured through a chat-based interface, en- suring that all game-related player interactions are captured precisely. A player's turn ends when ei- ther they choose end their turn via an "End my Turn" button or when their turn-time of 200 sec- onds runs out. In either case, the next player will start their turn. However, if the turn time runs out for the current quest leader, instead of transitioning to the next player, a random party is proposed, or a vote for the currently proposed party is initiated, depending on the current game state. In addition to ending their turn, quest leaders have the choice to initiate another round of discussion or initiate vote for the currently proposed party. Similar the turn time, votes (cither for a party or for quest suc- cess) have to be made within 30 seconds. Should
The following section introduces the surveys con- ducted with our human subjects to collect the base- line results.
B.1 Example of Human Baseline Survey
This is example of the human baseline question- air
B.1.1 Round 1
Example for Round
system:
Game Started!
system: player-2 proposed a party: player-
2, player-3
player-2: This is the first round, we don't 't have a lot of information so

---

propose myself (because am good) and the next person player-3.
You helpful assistant that the chat between six players, player-] to player-6. who play Avalon: The Resistance (a cooperative- comperirive game) to identify who Merlin. There three good players, Merlin, who also good player found because they ure deceptive and lie about the good player's roles and vote parties and quests irationally. For Merlin, watch with knowledge of evil players' ideniiiies insightful comments heyond their specilic players.
evil players. Evil nsually he
player-3: I 'arree with player-2. I am good too.
individials
role, and caution regarding mission
buman: current is: <STATE> (Sac Scction 4.2.1)
player-4: I think I am fine with the team as well
The current proposal is: <PARTY> (See Section 4.2.1)
belief <BELIEP> Section only round.
based context)
You that <FVIl evil.
player-5: i am good with that
This the chat between player to player
<CHAT> (See Section 4.2.2) you elaborate on it further.
nor explain your answer,
No opinions in the first turn. I'm fine with this party
player-6:
Table 4: Prompl used infer who Merlin is.
player- I: Oh, ycah, am good with this. We don't know anything at this point anyways
Avalom: Resistance Assistant (upt-3,5-turbop) Avalon: The Resistance is a board game that invalves hidden roles, deduction, deception. The game set in the world King Arthur, and players take on roles of either loyal servants Arthur (the "good" faction) the minions of the avil Mordred (the "bod" faction). goal
human: you explain
system: Party Vote Outcome: player-1: Yes, player-2: Yes, player-3: Yes, player-4: Yes,
player-5: Yes, player-6: Yes
game difforest
faction, [...]
human: In Avalon: The Resistance, can you tell me about the role of Mertin!
system:
Vote Succeeded! Initiating Quest
ascistant (I 1mmm-22-130):: [...] Merlin is charncter with special knowledge who knows the true identities the minions of Mordred, the traitors the gume. Here are the key points to understund about the role Merlin: I...] human: In Avalon: The Resistance, what good stratogy Morgana when playing
Vote!
Percival
Merlin?
system: Quest Succeeded!
ussistant (gpt-4); 4); A good siralegy for Murgana when playing with Percival Merlin Avalon: Resistance focus confusing and doceiving both Percival and Merlin to turn the tide favor of the evil team. Here are tips
Quest results after quest 1: Success
this stralegy: I...]
Which role do you think the players have? Please only consider the information you have up to this point in the game.
Table 5: Prompts and sample answers to determine model's familiarity with Avalon: The Resistance.
player-1: <select of ["good", "evil", "mer- lin". don' know"]
C.2
Familiarity with Avalon
The questions in Table 5 are used to identify whether not particular pre-trained LLM is aware of Avalon: The Resistance. Sample answers indicate that all three tested models (GPT-4, GPT- 3.5, Llama-2-13B) are familiar with Avalon.
<select one of [''oodd", "evil", "mer- lin", dInon't know"]
player-2:
<select one of ['good", "evil", "mer- lin". "In'n' know""
player-3:
C.3 TypeScript Schemas
<select one of ["'godd", "evil", "mer- lin". "Idon't know"]
player-4:
The following schema is used for role predictions:
// Define the AvalonRoles interface
<select one of [""good", "evil", "mer- lin", "Tnon'' know"]
player-5:
export interface AvalonRoles
player_1: "good" "evil" "merlin";
player "good" "evil" " "merlin";
<select one of ["good", "evil", "mer- lin". don't know"]
player-6:
player_3: "good"
"evil" " "merlin";
player_4: "good"
"evil" " "merlin";
player_5: "good" "evil" " "merlin";
C Further Prompts
player_6: "good" "evil" " "merlin";
In this section, we provide additional insights into how we query the LLM.
}
The following schema is utilized to predict the
C.1 Identifying Merlin
player who playing as Merlin:
The following prompt was utilized to query the model about the identity of Merlin
// Define the MerlinPlayer interface export interface MerlinPlayer {

---

Table Fl scores for role predictions in the game of Werewolf. Our results show enhanced role prediction scores compared to our Avalon data, suggesting more challenging social deduction environment in Avalon.
Model
Strategy
GPT-
GPT-3.5
GPT-3.5 (FT)
Llama-2-13b
Llama-2-13b
Assertion
0.27
0.26
0.26
0.25
0.26
Questioning
0.40
0.33
0.39
0.16
0.00
3
Suggestion
0.57
0.11
0.56
0.12
0.00
Agreement
0.54
0.40
0.55
0.24
0.00
5
Logical Deduction
0.36
0.41
0.42
0.00
0.26
6
Cumpromise/Coooessin
0.17
0.40
0.23
0.18
0.00
7
Critique/Opossiion
0.36
0.43
0.43
0.07
0.33
8
Appeal/Defense
0.12
0.00
0.42
0.00
0.00
Overall (micro-fl)::
0.37
0.32
0.43
0.15
0.20
Table 7: Fl scores for Werewolf role predictions across test splits using the optimal model (GPT-4) from Table 3.
Model
Drunk
Insomniac
Minion
Hunter
Revealer
Villager
Werewolf
Troublemaker
Robber
Tanner
Seer
Mason
GPT-4| 0.42
0.70
0.04
0.43
1.00
0.85
0.41
0.71
0.39
0.30
0.55
0.00
merlin: "player_1" "player_2" "player_3" "player_4" "player_5" "player_6";
D Additional Results
D.1 Werewolf Individual Roles
In Table 7, we demonstrate that many key roles in Werewolf are predicted with high Fl scores. How- ever, Mason and Minion have low scores they only have and four representations in the test set of [2]. For Tanner and Robber, we observe common confusion with the Werewolf role (and vice versa). We attribute this to the Tanner's goal looking like werewolf and being eliminated for their unique win condition, while the Robber (at the end of the game) may have played like werewolf without being aware of the fraction change.
D.2 Persuasion Strategy Prediction
Table 6 reports the performance of model when predicting the persuasion strategies for all utter- ances across our six test games. In each case, mod- els have been run once for each utterance, while the fine-tuned models have been trained for three epochs on the 14 training games. We observe that a fine-tuned GPT-3.5 model performs best, even outperforming vanilla GPT-4 model, while Llama- 2-1 13BB, even in the fine-tuned case, underperforms.


Reviews:
[{"review_id": "KlI987rqzJ", "reviewer": "Reviewer_YFpd", "summary": "", "questions": "- As you do cite one paper from that group (Keizer et al 2017), you seem to be aware of the \"Settlers of Catan\" project. Would you not consider that a \"high-quality dataset for a ling-horizon multi-party dialogue featuring deception and persuasion\" (line 096)?\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their thorough review of our work and are encouraged to hear that our proposed dataset was perceived as an interesting resource that is different from the usual task-oriented dialogue. Furthermore, we are excited to hear that our experiments were perceived as well-designed, executed, evaluated, and analyzed.\n\n**Highlights:**\n- We have added deception strategies (omission, commission, influence [2]) for each evil utterance that is identified as a lie. \n- We have added additional analysis with regard to the influence of player order and utilized lies on the model performance and outcome of the game\n- We added experiments utilizing the full game context with all models for better comparison of game representations and report our results as F1 scores (see large table below)\n- We have removed Vicuna in favor of Llama-2 and added GPT-3.5 and Llama-2 finetuning.\n- We have added experiments across all models to predict persuasion strategies (see small table below)\n\n**What is the take-home message?:**\nWe would like to elaborate on this point. Our analysis points to a more nuanced picture than simply that LLMs don’t handle it very well. While the LLM only outperforms the human in finding Merlin in one setting (GPT-4 using chat only), dramatically different performances can be observed compared to a random baseline. For example, LLMs tend to be capable of finding good players (as compared to a random baseline), while at the same time, struggling with identifying evil players and merlin correctly (similar to the random baseline). Further, we show that the utilization of lies on the side of evil players has a statistically significant impact on the model’s capability of identifying Merlin, i.e., the more lies evil people utilize, the more likely Merlin will be found out. Additional experiments to that end have been added to our paper and are outlined further below.\nOur take-home message is that Avalon provides a setting in which current state-of-the-art LLMs do poorly due to complex deception and long-horizon dialogue between multiple players, which is currently not studied in much detail due to the lack of sufficient datasets and benchmarks. We show that even when using state-of-the-art models and associated \"tricks\" (e.g., TypeChat [3] and structured beliefs), they still fail to match human performance. In particular, our proposed dataset is more challenging than existing datasets on similar games because each game is longer, has more utterances, and has stricter rules that lend themselves well to NLU tasks. For example, our corpus does not have changing roles and teams, nor secret communication. Furthermore, we consider our dataset to be more comprehensive by providing five different roles, the full game state (and show how it can be used), eight labeled persuasion strategies, four labeled deception strategies (different types of lies), and a six-player focused chat that is completely task-related. Finally, we conjecture that humans are prone to deception and that investigation in such matters is beneficial for the broader community,\n\n**Concern regarding lack of in-depth data analysis:**\nWe appreciate the request for further analysis and have extended our paper as such. In particular, we have investigated the model’s performance and game outcome on: the number of utterances per player type, the number and type of lies, and the position of Evil players and Merlin in the turn order. We have found that Merlin is easier to identify when Evil has a high number of utterances being labeled as lies (particularly Commission), while it is more likely that Good will lose if Percival contributes a large number of utterances. Similarly, good players are easier to identify by our model if Percival or Morgana contribute many utterances, or if Merlin is early in the turn order. \n\nFurther, to address the particular concerns of the reviewer:\n\n**Possible Additional Subtasks:** With our dataset, additional tasks are possible beyond role predicting, including developing agents that are advising players, agents that play the game themselves, identifying persuasion strategies (see results at the bottom), identification of deceptive strategies (we have added an additional set of labels to our dataset with the three most common lie-strategies: commission, omission, and influence) [3]. \n\n**Learnable Language Capabilities:** With our dataset, we target long-horizon context learning in a challenging scenario utilizing hidden objectives, multi-party dialogue with many participants (introducing challenges when and how an LLM could enter a conversation), and further research on deception and persuasion in human communication. As humans are prone to deception, we conjecture that further research in that particular area is beneficial. \n\n**What would a Human Overhearer do?**: We have surveyed our human annotators again, asking them what cues they used to identify roles. As a qualitative summary, the answers were as follows: Evil players could be identified by their votes for or against parties, and if someone defends a person, that is likely to be evil. Merlin was identifiable by being more insightful than a normal player should be and by occasionally making “cover-up” claims that are vastly out of character. Generic good players were identified by exclusion through Evil/Merlin and generally had no direct strategy of being identified.\n\n**Human Strategies:** We have also asked a subset of our players what strategies they used during the game for each role. As a qualitative summary, good players should never lie while ensuring they do not contradict themselves without very good reason (i.e. when changing their mind about someone else). Evil players should not support their peers and wait for them to make a mistake, then call them out for it to make themselves look good. Finally, Merlin should be making a few wrong claims in cases in which the group already decided differently to hide as “being confused.”\n\n**“Settlers of Catan” being a high-quality dataset**: Thank you for asking for clarification on [1]. We consider the Settlers of Catan dataset a “high-quality dataset”; however, in their setting, the authors only focus on dialogue with artificial agents, not other humans, and limit the context of persuasive behavior to trading negotiations in a purely non-cooperative setting. Compared to the Settlers of Catan dataset, our proposed corpus introduces the following additional challenges: 1) long-horizon communication (avg. utterances in a game of Settlers: 79, ours: 119), 2) cooperative-competitive teamwork, 3) broader context of the discussion (i.e., not limited to individual trades), and 4) high-quality labels for persuasion and deception strategies. Further, Avalon highly depends on complex and sophisticated dialogue, while Settlers of Catan’s requirements are more forgiving as trades are handled independently. With these additional challenges and opportunities, we believe that our proposed dataset goes beyond the existing datasets on dialogue and persuasion, making it a relevant and exciting testbed for future research in agent development and investigation of deceptive behavior. \n\n**Additional Results:**\n\n**Prediction of Persuasion Strategies:**\n| PStrat | GPT-4 | GPT-3.5 |  GPT-3.5 (FT) | Llama-2-13b | Llama-2-13b (FT) |\n|----------|-------|---------|---------------|-------------|------------------|\n| Micro-F1 | 0.37  | 0.35    | 0.43          | 0.15        | 0.20             |\n\n**Updated Table 3 (Role Prediction)**\nIn the table below, the first number represents the round-based inference with a carried-over belief, while the second number represents inference with the full context.\n\n| Model | Familiar with Avalon | Trained Model (14 Games) | Using Chat | Using State | Find Good | Find Evil | Find Merlin | Evil find Merlin (Final Guess) | Evil find Merlin (Anytime)  | \n| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |\n| Gpt-4      | [x] | | [x] | | 0.67 / 0.67 | 0.48 / 0.55 | 0.36 / 0.20 | 0.17 / 0.17  | 0.83 / 0.67  | \n| Gpt-4      | [x] | | | [x] | 0.59 / 0.57 | 0.20 / 0.33 | 0.06 / 0.31 | 0.17 / 0.33 | 0.17 / 0.50 |\n| Gpt-4      | [x] | | [x] | [x] | 0.67 / 0.68 | 0.46 / 0.58 | 0.05 / 0.27 | 0.00 / 0.00 | 0.67 / 0.50 |\n| gpt-3.5-turbo      | [x] | | [x] | |  0.68 / 0.60 | 0.46 / 0.40 | 0.23 / 0.17 | 0.17 / 0.17 | 0.33 / 0.50 | \n| gpt-3.5-turbo      | [x] | |  | [x] | 0.57 / 0.47 | 0.46 / 0.30 | 0.00 / 0.32 | 0.17 / 0.17 | 0.17 / 0.17 | \n| gpt-3.5-turbo      | [x] | | [x] | [x] | 0.58 / 0.65 | 0.34 / 0.47 | 0.23 / 0.13 | 0.17 / 0.17 | 0.33 / 0.33 | \n| gpt-3.5-turbo      | [x] | [x] | [x] | [x] | 0.52 / 0.59 | 0.38 / 0.41 | 0.19 / 0.15 | 0.17 / 0.17 | 1.00 / 0.67 | \n| Llama-2 |  | | [x] | | 0.68 / 0.61 | 0.39 / 0.27 | 0.00 / 0.00 | 0.17 / 0.00 | 0.17 / 0.17 | \n| Llama-2 |  | | | [x] | 0.41 / 0.62 | 0.00 / 0.34 | 0.00 / 0.00 | 0.00 / 0.00 | 0.00 / 0.17 | \n| Llama-2 |  | | [x] | [x] | 0.61 / 0.55 | 0.33 / 0.22 | 0.00 / 0.00 | 0.17 / 0.00 | 0.17 / 0.33 |\n| Llama-2 | [ ] | [x] | [x] | [x] | 0.65 / 0.63 | 0.35 / 0.26 | 0.23 / 0.27 | 0.33 / 0.00 | 0.33 / 0.00 |\n| Random |  |  |  |  | 0.50 | 0.34 | 0.17  | 0.17 | 0.60 |\n| Human | [x] | | [x] | [x] | 0.76 | 0.72 | 0.33 | 0.5 | 0.67  |\n\n1. Kaizer et al.: “Evaluating Persuasion Strategies and Deep Reinforcement Learning\nMethods for Negotiation Dialogue Agents”\n2. The Three Types of Lies: PennState University\n3. Microsoft, TypeChat", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Potentially very interesting resource that is interestingly different from more usual task-oriented dialogue (which is typically between two parties and purely cooperative) and also chit chat dialogue\n\n- Experiments are well designed, executed, and evaluated / analysed \n\n- Very well written\n", "reasons_to_reject": "- Ultimately, it is a bit unclear to me what the take-home message is supposed to be. This clearly is a very hard task, that I would prima facie not have very strong reasons to think LLMs can handle. It does indeed turn out that they don't hande it very well. What have we learned?\n\n- To elaborate on this, I think it would have been helpful to spend more time on analysing the data. Are there more circumscribed subtasks within the larger task \"detect who's playing which role\"? What subtypes of the language understanding capacity are targeted by this? What would an overhearer need to be able to do to solve this task? Spot contradictions? What are typical strategies that humans use to solve this task? \n", "questions_to_authors": "- As you do cite one paper from that group (Keizer et al 2017), you seem to be aware of the \"Settlers of Catan\" project. Would you not consider that a \"high-quality dataset for a ling-horizon multi-party dialogue featuring deception and persuasion\" (line 096)?\n", "paper_topic_and_main_contributions": "This paper presents a data collection of (massively) multi-party dialogue in an online game, in a cooperative / competitive scenario [new data resource]; a task defined on this data and some experiments showing LLM performance on it [NLP engineering experiment]\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 156, something wrong with reference\n- l 240, \"i.e. the\" --> \"i.e.\\ the\"\n- l 328, \"the priviledged knowledge of the evil players are\" --> \"of who the evil players are\"?\n- l 412, \"global sate\" -> state\n- l 460, \"18 years old an familiar\" -> and familiar\n"}}, {"review_id": "NFcO6459QZ", "reviewer": "Reviewer_n5L7", "summary": "", "questions": "* Can an utterance have multiple labels? If not, why not? It seems like a single utterance can employ multiple persuasion strategies.\n* Why did you need to use a spell checker? What about the use of slang, emoticons, etc?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments and suggestions provided in their review. We are glad to hear that the self-reported persuasion strategies from actual players have been perceived as favorable. \n\n**Highlights:**\n- We have added deception strategies (omission, commission, influence [5]) for each evil utterance that is identified as a lie. \n- We have added additional analysis with regard to the influence of player order and utilized lies on the model performance and outcome of the game\n- We added experiments utilizing the full game context with all models for better comparison of game representations and report our results as F1 scores (see large table below)\n- We have removed Vicuna in favor of Llama-2 and added GPT-3.5 and Llama-2 finetuning\n- We have added experiments across all models to predict persuasion strategies (see small table below)\n\n**Concern regarding novelty**: We would like to start by addressing the concern of novelty, particularly in comparison to [1] and [2]:\n\n**In comparison to Mafia [1],** our dataset does not allow for secret communication between the evil players; thus, synchronizing beliefs and strategies between players is more difficult. Further, we introduced a total of five different roles (Good, Merlin, Percival, Morgana, and evil) with different knowledge and objectives (while still being divided into the two general good/evil teams), unlike [1], which only considered the general good/evil role. Further, roles and team compositions in our dataset are fixed, lending themselves well to NLU tasks, while in Mafia, team sizes change throughout the game. Additionally, while [1] only focuses on predicting roles, we predict roles and persuasion strategies (we have added additional experiments, see small table below) as well as provide labels for the deception (i.e., lying) strategy of evil players (commission, omission, instigation [5]), which goes beyond the work done and even possible with [1] as the Mafia dataset does not have such labels. In our work, in response to other reviewers’ questions, we have added additional evaluations with regard to model performance and game outcome under various conditions, including the number of utterances per player type, the number and type of lies, and the position of Evil players and Merlin in the turn order\n\n**In comparison to Werewolf [2]:** Werewolf is a vastly different game than Avalon, as outlined in [2]. However, we acknowledge that [2] evaluates their approach on eight Avalon sourced from the Ego4D dataset. However, in Ego4D, these games do not have labeled player roles, game states or explicitly labeled deception strategies. As mentioned in [2] “_We [...] ask the annotators to look through each game clip and annotate the starting role, ending role, and the voting outcome of each player_”. Upon further investigation of this dataset, we find that the authors of [2] have not released the Avalon data, as they are “_still negotiating the Ego4D team about the data release_”, which, unfortunately, means that the data for the eight Avalon games (outside of the un-annotated videos from Ego4D) is unavailable, making our dataset the only available annotated Avalon dataset, containing five different roles, the full game state, eight labeled persuasion strategies, four labeled deception strategies, and a six-player focused chat that is completely task-related. \n\n**Summary of Novelty:**\nIn our work, we propose a new benchmark dataset that complements yet goes beyond existing datasets on deception and persuasion. Avalon provides a setting in which current state-of-the-art LLMs struggle, due to the complexities inherent to deception, hidden objectives, and long-horizon dialogue between multiple players. Such problems are currently not studied in much detail due to the lack of sufficient datasets and benchmarks. We show that even when using state-of-the-art models and associated \"tricks\" (e.g., TypeChat [3] and structured beliefs), they still fail to match human performance. In particular, our proposed dataset is more challenging than existing datasets on similar games due to each game being longer, having more utterances, and having stricter rules that lend themselves well to NLU tasks. Further, our dataset is hand-labeled, largely by the players themselves, and contains various sets of labels including persuasion and deception strategies alongside the full game-state, and task-focused chat. \n\n**Calculation of average utterances:** We would also like to clarify the number of utterances in [2] and how we reached 68. Our games have an average of 119 utterances solely focusing on the game's content without any spurious utterances due to a naturalistic social setting. While [2] reports 26,647 utterances across 199 games, which results in 138 utterances per game, the authors of [2] also point out that “_49.2% of Ego4D utterances are labeled as no strategy because of the naturalistic social setting, while only 37.9% of YouTube utterances are labeled as no strategy since players from the YouTube videos are more proficient at the game and focused more on gameplay_”. This leads to the conclusion that 49.2% of Ego4D utterances are unrelated to the task at hand (because YouTube players were “_more proficient at the game and focused more on gameplay_”, which justifies the lower “_no strategy_” labels), namely the game of Avalon. \nThus, we conclude that (1.0-0.492) * 26647 / 199 results in approximately 68 game-relevant utterances per game. As all utterances in our dataset are game-related, our dataset is more complex than the eight unreleased Avalon games provided in [2].\n\n**Detail experiments compared to the previous datasets:** We are happy to conduct additional experiments; however, since the Avalon data of [2] is not available, nor are their pre-trained models, we are unable to compare against their approach. However, we have provided additional experiments with our models on our dataset in order to predict our eight persuasion strategies, which are a superset of the six strategies in [2]. Results on the Micro-F1 score across all eight strategies can be found in the small table below. An extended version detailing the F1 scores for all individual classes has been added the the paper itself. \nWith regards to [1], we would like to iterate again that the novelty of our work lies in the introduction of a new, complementary benchmark for long-horizon deception and persuasion in a cooperative-competitive multi-party setting. Statistically, we find Avalon to be a more difficult benchmark in terms of rules and game length as compared to [1], such that even SOTA LLMs can't perform well. Further, given the relatively small size of these rather complex datasets, we feel it's important to have multiple datasets that are capable of addressing such issues. Given that our dataset also comes with an online version of Avalon, we afford the ability to easily extend the dataset with little to no additional burden on the experimenters.\n\nIn the following, we would like to address your questions:\n\n**Results in Table 3**: Regarding the reported scores in Table 3: We appreciate the comment on utilizing the F1 score and have updated our entire Table 3 (see large table below) accordingly over a total of 10 runs per model per modality and task. As the field of LLMs moved since initial submission, we have also included Llama-2 (in place of Vicuna due to better performance; however, Vicuna is now available in the supplemental material), while also adding fine-tuned versions of Llama-2-13b and GPT-3.5-turbo. Furthermore, we have added, for all models, results when utilizing the full context of each game (as much as possible, given their context lengths, truncating early rounds if necessary). In comparison, the round-based context utilizes an average of 974 (std: 333, max: 1941) tokens, while the full context utilizes an average of 2844 (std: 2011, max: 8556) tokens.\nIn the old Table 3, the scores for Evil find Merlin were indeed just MultiClassAccuracy. However, we would also like to point out that we changed our methodology to utilizing TypeChat [3] (which we have re-implemented in Python), which makes it easier for the LLM to produce the structured output (and to correct it if it is non-conformant) relevant to our task; thus scores have slightly increased across the board. \n\n**Human Baseline**: For our six test games, we utilized three to four human annotators to read through each game’s chat log and game states in order to identify the roles. This is the same task and information given to LLMs in the Chat+State modality with full context history. Results were averaged over all annotators for each game. The agreement of the annotators measured as the mean of all pairwise Cohen Kappas is 0.41, indicating moderate agreement amongst the annotators according to [4].\n\n**Multiple persuasion labels:** The possibility of utilizing multiple persuasion strategies is indeed interesting. In our current setup, each utterance contains an average of 16 tokens, and participants were asked (not enforced) only to make “one argument per utterance message”, such that the need for multiple strategies in one chat message is reduced. This approach is also self-reinforcing during the labeling of their own utterances (approximately 82% of participants self-labeled their utterances). After choosing one of the eight possible strategies, no additional selections or changes can be made, thus weakly enforcing the idea of only placing one strategy in each utterance.\n\n**Why the spell checker:** Our game is set up in turns with a generous time limit for each turn in order to advance the game eventually and keep arguments reasonably concise. However, players rather type instead of correcting hastily misspelled words. Additionally, many of our participants were not native English speakers. Further, we use the spell checker with a custom dictionary (and high incentive) to correct the spelling of player names during the game. To address slang, we manually evaluated which words were corrected by the spell-checker and provided exceptions for slang like “hmmm” and similar terms. Special non-ASCII characters were not allowed; thus, emojis were not possible. However, emojis like “:)” have been left unaltered.\n\n\n**Additional Results:**\n\n**Prediction of Persuasion Strategies:**\n| PStrat | GPT-4 | GPT-3.5 |  GPT-3.5 (FT) | Llama-2-13b | Llama-2-13b (FT) |\n|----------|-------|---------|---------------|-------------|------------------|\n| Micro-F1 | 0.37  | 0.35    | 0.43          | 0.15        | 0.20             |\n\n**Updated Table 3 (Role Prediction)**\nIn the table below, the first number represents the round-based inference with a carried-over belief, while the second number represents inference with the full context.\n\n| Model | Familiar with Avalon | Trained Model (14 Games) | Using Chat | Using State | Find Good | Find Evil | Find Merlin | Evil find Merlin (Final Guess) | Evil find Merlin (Anytime)  | \n| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |\n| Gpt-4      | [x] | | [x] | | 0.67 / 0.67 | 0.48 / 0.55 | 0.36 / 0.20 | 0.17 / 0.17  | 0.83 / 0.67  | \n| Gpt-4      | [x] | | | [x] | 0.59 / 0.57 | 0.20 / 0.33 | 0.06 / 0.31 | 0.17 / 0.33 | 0.17 / 0.50 |\n| Gpt-4      | [x] | | [x] | [x] | 0.67 / 0.68 | 0.46 / 0.58 | 0.05 / 0.27 | 0.00 / 0.00 | 0.67 / 0.50 |\n| gpt-3.5-turbo      | [x] | | [x] | |  0.68 / 0.60 | 0.46 / 0.40 | 0.23 / 0.17 | 0.17 / 0.17 | 0.33 / 0.50 | \n| gpt-3.5-turbo      | [x] | |  | [x] | 0.57 / 0.47 | 0.46 / 0.30 | 0.00 / 0.32 | 0.17 / 0.17 | 0.17 / 0.17 | \n| gpt-3.5-turbo      | [x] | | [x] | [x] | 0.58 / 0.65 | 0.34 / 0.47 | 0.23 / 0.13 | 0.17 / 0.17 | 0.33 / 0.33 | \n| gpt-3.5-turbo      | [x] | [x] | [x] | [x] | 0.52 / 0.59 | 0.38 / 0.41 | 0.19 / 0.15 | 0.17 / 0.17 | 1.00 / 0.67 | \n| Llama-2 |  | | [x] | | 0.68 / 0.61 | 0.39 / 0.27 | 0.00 / 0.00 | 0.17 / 0.00 | 0.17 / 0.17 | \n| Llama-2 |  | | | [x] | 0.41 / 0.62 | 0.00 / 0.34 | 0.00 / 0.00 | 0.00 / 0.00 | 0.00 / 0.17 | \n| Llama-2 |  | | [x] | [x] | 0.61 / 0.55 | 0.33 / 0.22 | 0.00 / 0.00 | 0.17 / 0.00 | 0.17 / 0.33 |\n| Llama-2 | [ ] | [x] | [x] | [x] | 0.65 / 0.63 | 0.35 / 0.26 | 0.23 / 0.27 | 0.33 / 0.00 | 0.33 / 0.00 |\n| Random |  |  |  |  | 0.50 | 0.34 | 0.17  | 0.17 | 0.60 |\n| Human | [x] | | [x] | [x] | 0.76 | 0.72 | 0.33 | 0.5 | 0.67  |\n\n1. Samee Ibraheem, Gaoyue Zhou, and John DeNero. NAACL 2022. Putting the con in context: Identifying deceptive actors in the game of mafia\n2. Bolin Lai, Hongxin Zhang, Miao Liu, Aryan Pariani, Fiona Ryan, Wenqi Jia, Shirley Anugrah Hayati, James M Rehg, and Diyi Yang. Findings of ACL 2023. Werewolf among us: A multimodal dataset for modeling persuasion behaviors in social deduction games.\n3. Microsoft, TypeChat\n4. Viera et al.: “Understanding interobserver agreement: the kappa statistic.”\n5. The Three Types of Lies: Penn State University", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper introduces a new dataset for the hidden role game *Avalon: The Resistance* which includes self-reported play strategies from the actual players (rather than having third-parties annotate strategies).", "reasons_to_reject": "There are numerous weaknesses in this paper:\n\n* A major issue is one of novelty. I'm usually quite loathe to bring up novelty as an issue, but in this case I believe it is warranted. There are already two dialogue datasets, [1] and [2], which explore dialogue understanding in hidden role games. [2] is a multimodal dataset which includes video and introduces strategy annotations, both of which were missing in [1] (though they are different hidden role games). [2] even includes 8 games of *Avalon* as part of their test set for seeing how well models generalize across hidden role games.\n* A larger issue is that the paper does not make an adequate appeal as to why this new dataset is needed (which could have helped address the novelty concern). Namely, the paper describes the benefits of the dataset as being significantly more challenging than the previous datasets due to increased game length. First, this claim does not seem to hold, since [2] reports 26,647 utterances across 199 games (8 of which are games of *Avalon*); this appears to be 134 utterances per game (not 64 as reported in the paper), which is longer than the 119 reported for the introduced dataset. Second, to adequately make a claim regarding increased complexity, the paper should detail experiments comparing to the previous datasets. Those datasets came out before the LLMs explored in this paper were released, so there are no adequate comparisons of performance to demonstrate the increased complexity of this dataset.\n\nAddressing these issues could cause me to increase my soundess rating a bit:\n\n* Table 3 must label the reported values as precision or accuracy. For all-role prediction, the caption makes it sound like the number before the slash should be avg precision and the number after is accuracy, but given the reported numbers that seems unlikely. I assume the column for *Evil find Merlin* is just accuracy?\n* Why don't you report F1 score and instead rely on average precision? Both [1] and [2] report F1 scores.\n* It's great that you performed multiple runs for the experiments, but how were the multiple runs and human annotations aggregated? Were they just averaged? If so, what was the variance or confidence interval? What was the agreement for the human eval?\n\n[1] Samee Ibraheem, Gaoyue Zhou, and John DeNero. NAACL 2022. Putting the con in context: Identifying deceptive actors in the game of mafia.  \n[2] Bolin Lai, Hongxin Zhang, Miao Liu, Aryan Pariani, Fiona Ryan, Wenqi Jia, Shirley Anugrah Hayati, James M Rehg, and Diyi Yang. Findings of ACL 2023. Werewolf among us: A multimodal dataset for modeling persuasion behaviors in social deduction games.", "questions_to_authors": "* Can an utterance have multiple labels? If not, why not? It seems like a single utterance can employ multiple persuasion strategies.\n* Why did you need to use a spell checker? What about the use of slang, emoticons, etc?", "paper_topic_and_main_contributions": "This paper introduces a new dataset for dialogue understanding in the hidden role game *Avalon: The Resistance*.\nIn a hidden role game, players are assigned to teams with competing goals (accomplished via voting) which is dependent upon their hidden role.\nThey are forbidden from communicating their role aloud, so team members may not know they are on the same team.\nThese games often require persuasion and deception on the part of the players, which makes identifying roles in the dialogue a challenging task.\nPrevious datasets from hidden role games exist for the game *Mafia* [1] (a dataset containing dialogue and voting history) and for *One Night Werewolf* [2] (a multimodal dataset with video and annotation of the persuasion strategy, in addition to dialogue and voting history).\nThe introduced dataset for *Avalon* includes dialogue from the game chat and self-reported persuasion strategies (rather than strategies annotated by third parties as in [2]).\n\nGiven this dataset, the paper describes two tasks specific to *Avalon*: identifying all roles (good, evil, or Merlin) and identifying the role of Merlin at each stage of the game given knowledge of the evil characters.\nA prompting strategy for encoding player chat and game state are devised.\nThree LLMs are experimented with for solving the tasks: GPT-3.5/4 and Vicuna (both pre-trained and fine-tuned).\nExperiments also ablate both the chat and game state, providing three combinations of experiments for each LLM.\nThese are compared against both a human and random baseline.\nInterestingly, both pre-trained Vicuna and GPT-3.5 outperform GPT-4 at identifying all roles when ablating game state.\nOn the other hand, GPT-4 performs nearly as well as humans for identifying Merlin at the end of the game and outperforms humans at identifying Merlin at any stage of the game.\nSeveral models perform worse when provided both game state and chat and the fine-tuned Vicuna severely underperforms the pre-trained version in all settings.\n\n### Update after Rebuttal\n\nI've raised my excitement score from 2 to 3, mostly because the authors mention that the previously released Avalon test set from [2] is not currently available due to intellectual property issues. I've also raised my soundness to 4 as the authors have:\n\n1. Clarified the issues I brought up in my initial review regarding things like use of average precision rather than F1.\n2. Addressed how their dataset is more complex than previous datasets through additional experimental results comparing against Werewolf dataset [2] (these additional results should be included in an updated version of their paper).\n\n[1] Samee Ibraheem, Gaoyue Zhou, and John DeNero. NAACL 2022. Putting the con in context: Identifying deceptive actors in the game of mafia.  \n[2] Bolin Lai, Hongxin Zhang, Miao Liu, Aryan Pariani, Fiona Ryan, Wenqi Jia, Shirley Anugrah Hayati, James M Rehg, and Diyi Yang. Findings of ACL 2023. Werewolf among us: A multimodal dataset for modeling persuasion behaviors in social deduction games.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Terms like Macro AP should be written out explicitly so there is no confusion that it refers to avg precision (at least I assume that's what's being reported)"}, "extra_text": {}}, {"review_id": "d0qYXmUEnG", "reviewer": "Reviewer_f9MG", "summary": "", "questions": "- Question A: Do the authors have information about how often players speak in a single round? Would there be any potential bias in terms of the speaking order?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and are thrilled to hear that our testbed has been deemed valuable for the subfield and that Avalon has many attractive factors to investigate human interaction. We also thank the reviewer for pointing us to [1], and we have added the citation to our paper. \n\n**Highlights:**\n- We have added deception strategies (omission, commission, influence [2]) for each evil utterance that is identified as a lie. \n- We have added additional analysis with regards to the influence of player order and utilized lies on the model performance and outcome of the game\n- We added experiments utilizing the full game context with all models for better comparison of game representations and report our results as F1 scores (see large table below)\n- We have removed Vicuna in favor of Llama-2 and added GPT-3.5 and Llama-2 finetuning\n- We have added experiments across all models to predict persuasion strategies (see small table below)\n\n**Lack of analysis**: We have conducted additional analysis with regard to player ordering in the game as well as deception strategies (i.e., lies) utilized by the evil players. In particular, we appreciate the request for insights about bad-faith actors. In order to address this, we have added additional deception strategy labels to our dataset for each utterance of the evil players (711 utterances total) with respect to being neutral, or a type of lie (commission, omission, or influence [2]), out of which 429 utterances constitute a lie. For our analysis, we have investigated the model’s performance and game outcome on: the number of utterances per player type, the number and type of lies, and the position of Evil players and Merlin in the turn order. With these types of evaluations, we hope to address the question of how bad-faith actors influence the gameplay. We have found that Merlin is easier to identify when Evil has a high number of utterances being labeled as lies (particularly Commission), while it is more likely that Good will lose if Percival contributes a large number of utterances. Similarly, good players are easier to identify by our model if Percival or Morgana contribute many utterances, or if Merlin is early in the turn order. We have added further analysis to this end to our paper and supplemental material.\n\n**Composing model input using the proposed dataset**: Since our initial paper submission, additional LLMs have been released that allow for extended contexts during prompting (Llama-2), as well as additional models that can now be fine-tuned (GPT-3.5-turbo) exhibiting large contexts. We have followed the reviewer’s suggestion and added experiments that utilize the full game context (without a carried structured belief) for all models across all experiments and reported results in our new and updated Table 3 (see large table below). In comparison, the round-based context utilizes an average of 974 (std: 333, max: 1941) tokens, while the full context utilizes an average of 2844 (std: 2011, max: 8556) tokens. For this, we find that the full context does not necessarily result in a blanket improvement of performance for any model. However, there is no clear picture that would allow for a final conclusion. With this, we would like to highlight again the purpose of our contribution. In particular, we introduce a new, complementary benchmark for long-horizon deceptive tasks with hidden agendas in a multi-party cooperative-competitive game that poses significant challenges to current SOTA LLMs. As humans are influenced by and utilize deception, we hope to foster further research into how such behavior can be identified and understood. \n\n**Scalability of the dataset** construction framework: With respect to scalability, the provided testbed is fully automated and does not require the intervention of an administrator. During our gameplay, players self-labeled 82% of their utterances with persuasion strategies outside of their own turn, thus also keeping players engaged in the game while they wait for other players’ turns to end. We hand-labeled the remaining 18% of utterances after the fact (2348 total labels). Additionally, the newly added deception labels were all hand-labeled by a small group of annotators that had prior familiarity with the data. We believe that the provided online version of Avalon is valuable as it can operate in an unattended fashion. Data is collected and processed automatically, and persuasion strategies are largely self-labeled by players, making it easy to further extend the dataset with new games, or different team compositions.\n\nIn the following, we would like to answer your questions:\n\n**How often do players speak**: Across our dataset of 20 games, the six players went on 102 quests, making 768 turns (one quest can have multiple rounds of discussions). Out of these 768 turns, 9 players chose not to speak, skipping their turn without providing an utterance. From observation, these skipped turns happen when multiple rounds of discussions are happening for the same quest.\n\n**Bias in the speaking order**: Thank you for asking this interesting question! We have analyzed the speaker order with regard to the game outcome and model performance; however, we have not found a statistically significant impact on the win rate or model performance related to player order (we evaluated both evil players being in the first/last three players, evil players being next to each other, Merlin being in the first/last three players, or merlin being surrounded by evil players). However, given Percival's behavior, we found an impact on game outcome and model performance. Good players tend to lose the game if many utterances originate from Percival. Furthermore, if Percival or Morgana are getting very involved (i.e., producing many utterances) or if Merlin is early in the turn order, it is easier for the model to identify good players. Finally, we identified that if evil players utilize many commission utterances, the model’s ability to identify Merlin increases dramatically.\n\n**Additional Results:**\n\n**Prediction of Persuasion Strategies:**\n| PStrat | GPT-4 | GPT-3.5 |  GPT-3.5 (FT) | Llama-2-13b | Llama-2-13b (FT) |\n|----------|-------|---------|---------------|-------------|------------------|\n| Micro-F1 | 0.37  | 0.35    | 0.43          | 0.15        | 0.20             |\n\n**Updated Table 3 (Role Prediction)**\nIn the table below, the first number represents the round-based inference with a carried-over belief, while the second number represents inference with the full context.\n\n| Model | Familiar with Avalon | Trained Model (14 Games) | Using Chat | Using State | Find Good | Find Evil | Find Merlin | Evil find Merlin (Final Guess) | Evil find Merlin (Anytime)  |  \n| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |\n| Gpt-4      | [x] | | [x] | | 0.67 / 0.67 | 0.48 / 0.55 | 0.36 / 0.20 | 0.17 / 0.17  | 0.83 / 0.67  | \n| Gpt-4      | [x] | | | [x] | 0.59 / 0.57 | 0.20 / 0.33 | 0.06 / 0.31 | 0.17 / 0.33 | 0.17 / 0.50 |\n| Gpt-4      | [x] | | [x] | [x] | 0.67 / 0.68 | 0.46 / 0.58 | 0.05 / 0.27 | 0.00 / 0.00 | 0.67 / 0.50 |\n| gpt-3.5-turbo      | [x] | | [x] | |  0.68 / 0.60 | 0.46 / 0.40 | 0.23 / 0.17 | 0.17 / 0.17 | 0.33 / 0.50 | \n| gpt-3.5-turbo      | [x] | |  | [x] | 0.57 / 0.47 | 0.46 / 0.30 | 0.00 / 0.32 | 0.17 / 0.17 | 0.17 / 0.17 | \n| gpt-3.5-turbo      | [x] | | [x] | [x] | 0.58 / 0.65 | 0.34 / 0.47 | 0.23 / 0.13 | 0.17 / 0.17 | 0.33 / 0.33 | \n| gpt-3.5-turbo      | [x] | [x] | [x] | [x] | 0.52 / 0.59 | 0.38 / 0.41 | 0.19 / 0.15 | 0.17 / 0.17 | 1.00 / 0.67 | \n| Llama-2 |  | | [x] | | 0.68 / 0.61 | 0.39 / 0.27 | 0.00 / 0.00 | 0.17 / 0.00 | 0.17 / 0.17 | \n| Llama-2 |  | | | [x] | 0.41 / 0.62 | 0.00 / 0.34 | 0.00 / 0.00 | 0.00 / 0.00 | 0.00 / 0.17 | \n| Llama-2 |  | | [x] | [x] | 0.61 / 0.55 | 0.33 / 0.22 | 0.00 / 0.00 | 0.17 / 0.00 | 0.17 / 0.33 |\n| Llama-2 | [ ] | [x] | [x] | [x] | 0.65 / 0.63 | 0.35 / 0.26 | 0.23 / 0.27 | 0.33 / 0.00 | 0.33 / 0.00 |\n| Random |  |  |  |  | 0.50 | 0.34 | 0.17  | 0.17 | 0.60 |\n| Human | [x] | | [x] | [x] | 0.76 | 0.72 | 0.33 | 0.5 | 0.67  |\n\n1. Callison-Burch et al.: “Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence”\n2. The Three Types of Lies: Penn State University", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Exploring language understanding ability of language models through existing games is a nice motivation in that games simulate human interaction well. One relevant example not mentioned in this paper would be [Dungeons and Dragons dataset](https://aclanthology.org/2022.emnlp-main.637/) for dialogue challenge. The Avalon game also contains many attractive factors for understanding human interaction. \n\n### After rebuttal\nI appreciate the authors for handling all of my concerns in detail. Based on the score rubric, I would raise my Soundness score to 3.", "reasons_to_reject": "I have concerns about this paper as following: \n\n**[1] Lack of analysis**\n\nMy major concern is that this paper skips the analysis of model performance, which is one of the most important part. Most experimental results describe the numerical performance of subtasks among models, according to input type. Although one of the main motivation of the authors is \"the presence of bad faith actors and deception (as mentioned in Introduction)\", there is no analysis of how these kinds of bad faith actors and deception actually affect the model performance. It has to be analyzed because there would also be several factors affecting the poor performance of language models. I recommend to clearly clarify the research question of the paper.\n\n**[2] Composing model input using the proposed dataset**\n\nAlthough the authors try to handle multi-turn conversation, the input given to the model only contains utterances of the last turn and previous information is only expressed as structured information. To the best of my knowledge, this can cause two issues. First, not containing previous utterances cannot sufficiently capture the interactive features of human conversation because natural language context implicitly draw out extra information, such as nuance and mind state. Second, replacing previous information with structured state can be suboptimal. In the perspective of dialogue state tracking, [this paper](https://aclanthology.org/2021.emnlp-main.593/) shows that using belief state rather than actual conversational history can deteriorate performance. Therefore, it is hard to fully understand the performance of multi-turn modeling.\n\n**[3] Scalability of the dataset construction framework**\n\nThe authors construct a web version of the game and annotated the dataset manually. This is valuable platform to the subfield, but it also has a limitation because annotating additional dataset consistently would be costly. But I think this is not a major concern compared to concerns above.", "questions_to_authors": "- Question A: Do the authors have information about how often players speak in a single round? Would there be any potential bias in terms of the speaking order?", "paper_topic_and_main_contributions": "This paper introduces a dataset simulating *Avalon: The Resistance* by developing a browser version of the game. This simulation includes multi-party conversation (i.e., 6 participants in each game) with global information. Through this dataset, the authors try to explore LLMs' ability to understand long-horizon dialogue in complex situation where people have different roles and objectives.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "- Captions of Table 2 and Table 5 need to be explained more concretely because the content is the same even though they indicate difference situations."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a potentially valuable resource, focusing on a unique domain of dialogue that differs from traditional task-oriented or chit-chat dialogues. The experiments are well-designed, executed, and thoroughly evaluated and analyzed. The paper is also well-written., Concerns raised by reviewers are adequately addressed in the rebuttal and author responses.
Decision: Accept-Findings