Werewolf Among Us: Multimodal Resources for Modeling Persuasion Behaviors in Social Deduction Games
Year: 2023

Abstract:
Persuasion modeling is a key building block for conversational agents. Existing works in this direction are limited to analyzing textual dialogue corpora. We argue that visual signals also play an important role in understanding human persuasive behaviors. In this paper, we introduce the first multimodal dataset for modeling persuasion behaviors. Our dataset includes 199 dialogue transcriptions and videos captured in a multi-player social deduction game setting, 26, 647 utterance level annotations of persuasion strategy, and game level annotations of deduction game outcomes. We provide extensive experiments to show how dialogue context and visual signals benefit persuasion strategy prediction. We also explore the generalization ability of language models for persuasion modeling and the role of persuasion strategies in predicting social deduction game outcomes. Our dataset can be found at https://persuasion-deductiongame. socialai-data.org. The codes and models are available at https://github.com/ SALT-NLP/PersuationGames.

heading: 1 Introduction
text: As humans, from childhood, we develop the ability to attribute mental belief states to ourselves and others (Premack and Woodruff, 1978). Moreover, we constantly exhibit persuasive behaviors to influence and even reshape the belief states of others during our daily social interactions (Lonigro et al., 2017). An automatic system with the ability to understand human persuasion strategies and deduce human belief states may enable more proactive humancomputer interaction, and facilitate collaborative decision-making processes.
* denotes equal contribution.
Prior works targeted at understanding the persuasion strategies utilized on online forums like Reddit, crowd-funding platforms (Yang et al., 2019; Chen and Yang, 2021; Atkinson et al., 2019), and in 1-on1 dialogues under simulated scenarios through the Amazon Mechanical Turk platform (Wang et al., 2019; Chawla et al., 2021).
However, the persuasive behaviors during naturalistic group discussions with face-to-face conversation remain unexplored. More importantly, daily human social interaction is multimodal by nature. Both verbal communication (e.g. language and audio) and non-verbal communication (e.g. gesture and gaze behavior) are essential for analyzing persuasive behavior.
Moreover, resources for understanding how persuasion strategies affect decision and deduction outcomes during social interactions are missing from the language technologies community.
To bridge these gaps, we introduce the first multimodal benchmark dataset for modeling persuasive behaviors during multi-player social deduction games. As shown in Fig. 1, our dataset is captured in a naturalistic setting where groups of participants play social deduction games1. Our dataset contains both video recordings and the corresponding dialogue transcriptions. The video data is sourced from both the Ego4D Social dataset (Grauman et al., 2022) and YouTube videos. Our dataset also has annotations for persuasion strategy at the utterance level and the voting outcome of each participant during the social deduction game.
We benchmark our dataset by providing comprehensive experimental results and analyzing the role
1We consider two games in our dataset: One Night Ultimate Werewolf and The Resistance: Avalon. Appendix A describes the rules of these two games.
6570
of the video modality and contextual cues in designing computational models for persuasion behavior prediction. We also provide results to show how different computational models generalize across different data sources and different games. Our contributions are summarized as follows: • We present the first multimodal dataset for per-
suasion modeling. Our dataset is collected in naturalistic social game scenarios with intensive face-to-face group conversations.
• We conduct comprehensive experiments to show the importance of context and visual signals for persuasion strategy prediction.
• We provide additional experimental results to investigate model generalization on the persuasion modeling task and discuss how persuasion strategy influences the game voting outcome.

heading: 2 Related Work
text: Persuasive Behaviors Understanding. A few previous works introduce datasets for the computational modeling of persuasion (Yang et al., 2019; Chen and Yang, 2021; Chawla et al., 2021; Wang et al., 2019; Luu et al., 2019; Atkinson et al., 2019). As summarized in Table 1, existing works collectpersuasion language data from online platforms
where real-time communication is not available. Moreover, these datasets mainly contain 1-on-1 conversations and lack conversations among multiple speakers. In contrast to these prior efforts, our dataset targets capturing persuasive behaviors in a social group setting of 4 to 6 people.
Multimodal Social Interaction. A rich set of literature has addressed the problem of multimodal sentiment analysis (Xu et al., 2021b; Li et al., 2020; Lu et al., 2019; Xu et al., 2021a). We refer to a recent survey (Kaur and Kautish, 2022) for a more detailed discussion on this topic. The Ego4D Social benchmark (Grauman et al., 2022) includes the tasks of identifying who is looking at and talking to the camera wearer using video
and audio. Bara et al. (2021) adopts a multimodal approach to understanding dialogue behavior in a simulated setting. The most relevant work is Bai et al. (2021), which adopts a multimodal method to predict the debate outcome from a TV show. In contrast, we address the challenging tasks of predicting the utterance-level persuasion strategy and the deduction outcome from a naturalistic conversation, which requires a richer understanding of high-level social behaviors. Computational Modeling of Deduction Games. Prior works have investigated computational models for social deduction games. One stream of work seeks to analyze strategies and develop AI agents that play deduction games using a game theory approach (Nakamura et al., 2016; Serrino et al., 2019; Chuchro, 2022; Braverman et al., 2008; Bi and Tanaka, 2016). These works focus on models of the state of the game alone and do not address understanding the dialogue and persuasive behaviors that often occur while playing. More relevantly, Chittaranjan and Hung (Chittaranjan and Hung, 2010) developed a model for predicting Werewolf game outcomes from player speaking and interrupting behaviors. Recently, (FAIR) introduced a game agent–CICERO, that achieves human-level performance in the Diplomacy game by leveraging a language model with planning and reinforcement learning algorithms. In contrast to these prior works, we present the first work for understanding persuasive behaviors in a group setting from a multimodal perspective.

heading: 3 Dataset

heading: 3.1 Data collection
text: To benchmark the generalization ability of the computational models, we collect our data from different sources, as detailed in this section. This work was approved by an Institutional Review Board.
Ego4D Dataset We first leverage a subset of Ego4D Social dataset (Grauman et al., 2022) for our study. This subset captures videos of groups of participants playing social deduction games. This subset contains 7.3 hours of videos with 40 games of One Night Ultimate Werewolf and 8 games of The Resistance: Avalon. Note that the Avalon data has a relatively small scale, and therefore is only used to evaluate the cross-domain game generalization ability of our models. To ensure all participants are visible in the frame, we use third-person videos
instead of the first-person videos from Ego4D for visual representation learning and transcription.
YouTube Video We retrieve the top search results for YouTube videos using the keywords of “one night ultimate werewolf” and “ultimate werewolf”. We manually select from the searched videos to make sure the they adopt a similar game setup as the Ego4D data. Specifically, we filter all results with more than 5 players or fewer than 4 players, and those using game roles from the expansion package. We finally collect a final set of 14.8 hours of videos with 151 clips of completed games that adopt the same game setup as the Ego4D dataset and have fully visible game outcomes. We will release the YouTube URLs for the selected videos.

heading: 3.2 Data Annotation
text: Video Annotation Most Ego4D and YouTube videos contain multiple games. Therefore, we first annotate the starting time (when the game narration voice begins) and the ending time (right before the voting stage) of each game. We then ask the annotators to look through each game clip and annotate the starting role, ending role, and the voting outcome of each player.
Transcription We use an automatic transcription service rev.com to generate the transcript of each game clip. We further ask annotators to carefully examine the alignment of the videos and transcripts, and manually correct any errors in the transcripts. Please refer to Appendix B for more details.
Persuasion Strategy Annotation Inspired by prior psychology studies and other works on predicting persuasion strategies (Chawla et al., 2021; Carlile et al., 2018; Yang et al., 2019; Chen and Yang, 2021), we propose six persuasion tactics that are frequently adopted in social deduction games.
• Identity Declaration: State one’s own role or identity in the game. This is a game-specific persuasion tactic.
• Accusation: Claim someone has a specific identity or strategic behavior. Accusation, similar to Undervalue-Partner (Chawla et al., 2021), is a generic proself behavior.
• Interrogation: Questions about someone’s identity or behavior. Interrogation, is a proself strategy related to individual preferences.
• Call for Action: Encourage people to take an action during the game. Call for Action relates to the coordination for persuasion (Chawla et al., 2021), which is a generic prosocial behavior.
• Defense: Defend oneself or someone else against an accusation or defend a game-related argument. An utterance demonstrates Defense when the persuader tries to use credentials to earn others’ trust or justify their earlier decisions.
• Evidence: Provide a body of game-related fact or information. Evidentiality is a general persuasion tactic that has been widely studied in previous works (Carlile et al., 2018).
Following previous work (Chawla et al., 2021), we annotate the persuasion strategy at the utterance level. We provide a website annotation tool adopted from Hayati et al. (2020) for our annotation task (see Appendix C for details) to the annotators. To properly train the annotators, we first ask all three annotators to annotate the same subset of dialogues and compute inter-annotator agreement using the nominal form of Krippendorff’s alpha (Krippendorff, 2018). We then discuss with the annotators on their disagreements and come up with a general rule to address the disagreements during the annotation process. We repeat the above process until the annotators reached a Krippendorff’s alpha greater than 0.6 for each category. Despite the subjectivity of persuasion strategies, the previous work (Chawla et al., 2021) suggests that annotations from 3 people are reasonable enough to most humans if they have a Krippendorff’s alpha greater than 0.6. In Table 2, we report the per-class Krippendorff’s alpha value for the final round of
inter-annotator agreement calculation. After the annotator training phase is completed, we ask the three annotators to independently annotate the rest of the Ego4D and YouTube data.
Annotation Statistics Our dataset has 5, 815 utterances from the Ego4D data and 20, 832 utterances from the YouTube data. More than 49.2% of Ego4D utterances are labeled as no strategy because of the naturalistic social setting, while only 37.9% of YouTube utterances are labeled as no strategy since players from the YouTube videos are more proficient at the game and focused more on gameplay. Furthermore, as shown in Table 2, the adopted persuasion strategies have an imbalanced distribution, where “Accusation”, “Interrogation”, and “Defense” are the most frequent strategies for both the Ego4D and YouTube videos. The annotators are recruited from a Startup Data Platform dedicated to research projects. All annotators are paid hourly at a rate above the federal minimum.

heading: 4 Strategy Prediction
text: Given an utterance and its corresponding video segment, we seek to predict the persuasion strategies adopted in the utterance. We first leverage a pretrained language model (Devlin et al., 2019; Liu et al., 2019) as the text encoder to obtain the utterance embedding, and a vision transformer (Fan et al., 2021) to obtain the visual embedding. We then concatenate the textual and visual features to predict the persuasion strategy. Additionally, we study the impact of textual context by including prior utterances as input.

heading: 4.1 Methodology
text: In our dataset, an utterance may be labeled with multiple persuasion strategies. For instance, “I’m a villager and she is the werewolf.” is labeled as both identity declaration and accusation. Therefore, we formulate this task as a binary classification problem for each strategy and consider an utterance as non-strategic if it gets negative labels in all strategies. The most straightforward approach to solving this task is fine-tuning a pre-trained language model, which is referred to as Base model. In addition, we consider the following approaches:
Modeling with Context Embedding. Since some persuasion strategies cannot be easily recognized from one single utterance, we further consider a model with additional context (prior utterances) for each utterance. This is denoted as Base + C. Modeling with Video Representation. We further leverage the non-verbal signals by combining video features with the text representation for persuasion modeling. We directly use a pre-trained Vision Transformer to extract video representations, and fuse the video and text representations before feeding them into the classification layer as shown in Fig. 2. We refer to this model as Base + V.
Late fusion of Video and Context. Finally, we adopt a late fusion model (Base + C + V) that incorporates both video features and context cues for persuasion strategy prediction.

heading: 4.2 Model Details
text: We perform our experiments using both BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) as backbones for the text encoder. We use the bert-base-uncased and roberta-base models from Huggingface (Wolf et al., 2020) in our implementation. We adopt MViT-B-24 (Fan et al., 2021) pretrained on Kinetics-400 (Kay et al., 2017) as the video encoder. Moreover, we also implement a multi-task model (Chawla et al., 2021) as an additional baseline, referred to as MT-BERT. Context and video features are incorporated into MT-BERT in the same way as BERT and RoBERTa. Base Model. For base models, we obtain the textual input T from the current utterance only. Then we input T into a text encoder ϕ followed by a classifier to get the strategy prediction. Base + C. We first concatenate the k previous utterances C1, C2, · · · , Ck with an [EOS] token to get context C, and then concatenate this with the current utterance U using a [SEP] token to get the final input T . Formally, we have
C = C1 [EOS] C2 [EOS] · · ·Ck, (1) T = C [SEP] U. (2)
Base + V. We use video encoder ψ to extract visual representation ψ(V) of the corresponding video clip V . During training, video features are concatenated with the text representation ϕ(T ) and fed into a fusion layer, which uses a linear mapping function W TF and an activation function Tanh(·). Finally, we apply a linear classifier W TP to obtain the prediction logits, which can be formulated as
logits =W TP · Tanh ( W TF · (ϕ(T )⊕ ψ(V)) ) ,
(3) where ⊕ denotes the concatenation of two vectors. Note that we fix the parameters of the video encoder during training. Please refer to Appendix D for more details on visual representation extraction. Base + C + V. We further late fuse + C with + V. Formally, we denote the probability predictions of the two models after softmax as PC and PV . Then the output after linear combination is formulated as
PC,V = (1− λ)PC + λPV , (4)
where λ is a scalar that balances PC and PV .

heading: 4.3 Training Details
text: All models are trained using cross-entropy loss. For training hyper-parameters, we do a grid search
of learning rates in {1e − 5, 3e − 5, 5e − 5} and batch sizes in {16, 8} for Base models. We then fix the optimal hyper-parameters for subsequent models incorporating context or videos. We train all models with the optimal learning rates and batch sizes for 10 epochs using AdamW (Loshchilov and Hutter, 2017) as the optimizer. We run all the experiments with three random seeds and report the average score and the standard deviation.

heading: 4.4 Experiment Results
text: Evaluation Metrics. Following (Chawla et al., 2021), we report the F1 score for each persuasion strategy category, the average F1 score of all categories, and Joint Accuracy. Note that the prediction is considered as correct when all the categories are predicted correctly in Joint Accuracy.
Ablations on Additional Context. We first present a systematic ablation study of how incorporating textual context may improve the performance of persuasion strategy prediction. Specifically, we feed a fixed length of previous utterances together with the current utterance into the backbone language encoders for classification. As shown in Fig. 3, the additional context can boost the performance of all baseline models. However, setting the context length too long may confuse the model, especially for categories that can be reliably predicted with the current utterance (e.g. Identity Declaration, and Interrogation). We present the per-class performance in Appendix E. Our empirical finding is that a context length of 5 can consistently improve the performance of all three baseline models. Therefore, we adopt a context length of 5 as a default setting for the rest of our experiments.
Modeling with Video Representation. We further study how incorporating video representation im-
proves the performance of persuasion modeling. The results are summarized in Table 3. Importantly, video features can improve the BERT model by 0.8% on both the Ego4D dataset and YouTube dataset. However, RoBERTa+V only beats the RoBERTa model by 0.2% on the YouTube dataset. This may be because the YouTube dataset has more training data which enables the RoBERTa model to learn a robust representation without video feature embedding. Interestingly, including video features has a larger performance boost on predicting “Accusation”, “Interrogation”, and “Call for Action”, which is likely due to the more frequent non-verbal communication (e.g. pointing to someone, raising hands, turning the head) during these persuasive behaviors.
Off-the-shelf GPT-3 Inference. Prompting Large Language Models off-the-shelf to solve NLP tasks has received increasing attention (Brown et al., 2020). Here, we experiment with GPT-3-175B on our benchmark under three settings: zero-shot, one-shot and five-shot. Specifically, we use the text-davinci-002 engine from OpenAI’s API2 with temperature 0 to produce a deterministic answer. The detailed templates for different settings are shown in Appendix G. The result is shown in Table 4. Using GPT-3 off-the-shelf achieves a nontrivial performance (Joint-A of 52.0 v.s. 38.8 for majority on YouTube data). Adding more examples further boosts performance, though it is still inferior to the fine-tuned models.
Data Domain Generalization. We conduct additional experiments to show the generalization ability of language models on persuasion prediction. To begin with, we use the model trained on the
2https://beta.openai.com/
YouTube data to make predictions on the Ego4D Werewolf testing data without any fine-tuning. As shown in Fig. 4, the resulting model achieves better performance than models trained only on Ego4D in most cases, due to the larger amount of available training data from the YouTube dataset. This also suggests that, for the text modality, the domain gap between the Ego4D and the YouTube data is small. We further fine-tune the model trained on the YouTube data with the Ego4D training data, and the resulting model performs even better. These results suggest promise in leveraging the large body of videos available online as a pre-training source for persuasion modeling in naturalistic social interactions. Another finding from our experiments is
that the multi-task setting (MT-BERT) may compromise the model’s generalization ability. We also find that including video representation cannot improve the model generalization ability (see more details in Appendix D), suggesting that the video modality domain gap between the two data sources is much larger than the text modality.
Game Domain Generalization. We also study the model generalization ability on another social deduction game – Avalon. Werewolf and Avalon are vastly different in the game rules and winning conditions, especially because Werewolf has only one voting round per game, while Avalon has multiple rounds per game. Therefore, the persuasion strategies adopted in Avalon have a different distribution from Werewolf (see Appendix H). We run inference on the Avalon data using models trained only on the Ego4D Werewolf data without fine-tuning. Results are shown in Figure 5. Despite the large domain gap between the two games, our models achieve decent performance on the Avalon data. However, we find incorporating additional context has marginal performance improvements, and may even compromise the performance of the RoBERTa
model. The detailed results of data and game domain generalization are shown in Appendix F.

heading: 5 Game Outcome Deduction
text: In addition to predicting persuasion strategies, we further model the human deduction process by predicting the voting outcomes of each pair of players, i.e., whether player A (voter) votes for player B (candidate). Therefore, in a game of n players, there are C2n (Combinations) of player pairs, corresponding to P 2n (Permutations) data points. We merge all data points from the Ego4D Werewolf data and YouTube data to enlarge the dataset size, and split the resulting data into 2741/427/827 samples for train/val/test sets. Since each player is only allowed to vote for one player, the resulting data has an imbalanced distribution, with 20.4% of the samples being positive (positive indicates the voter votes for the candidate).

heading: 5.1 Method
text: For deduction modeling, we encode the input feature with three embeddings: a 7 × 1 vector rep-
resenting the persuasion strategy distribution (including non-strategy) adopted by the voter; a 7× 1 vector representing the persuasion strategy distribution adopted by the candidate; and a 12×1 one-hot vector representing starting role of the voter (One Night Werewolf has 12 roles in total). Therefore, the input for deduction is a 26× 1 vector. We use a simple logistic regression model for deduction modeling. To address the class imbalance of positive and negative samples, we train the model with weighted binary classification loss.

heading: 5.2 Experiment Results
text: Our model achieves an F1 of 32.7% and an AUC of 54.7%, outperforming random prediction, which obtains F1 and AUC of 28.6% and 50.0%, respectively. These results show the effectiveness of persuasion strategy usage and role as predictors for game-level outcomes. To analyze the contribution of persuasion strategy embedding and role embedding, we consider another model that only takes the persuasion strategy embeddings as inputs. This model achieves an F1 of 32.2% and an AUC of 54.6%. Overall we find that the persuasion strategy embedding is more informative for the predicting game outcomes than the role embedding.
We visualize the weights of logistic regression in Fig. 6. Interestingly, for positive prediction (the voter votes for the candidate), the weights of the candidate are higher than the voter. It indicates that a player’s voting choice depends more on the candidate’s behaviors. This confirms our intuition that players make their decisions based on candidates’ arguments.
As for the negative prediction, we see that evidence is the most important strategy for the candi-
0 1
No Strategy Identity Declaration
Accusation Interrogation Call for Action Defense
Evidence
No Strategy Identity Declaration Accusation Interrogation Call for Action Defense Evidence
Voter CandidateThe voter doesn’t vote for the candidate
The voter votes for the candidate
Figure 6: Weights visualization of persuasion strategies in logistic regression. The connection between a strategy and 0 means this strategy contributes to the prediction of 0 (i.e. the voter doesn’t vote for the candidate). Likewise, the connection between a strategy and 1 denotes this strategy contributes to the prediction of 1 (i.e. the voter votes for the candidate). The transparency of lines corresponds to the weights of logistic regression. A less transparent line suggests a greater weight and more impact on the output.
date to negate suspicion. It confirms that players are inclined to trust those who provide more information and evidence to find the werewolf.

heading: 6 Conclusion and Future Work
text: In this work, we introduce the first persuasion modeling dataset with multiple modalities and rich utterance-level persuasion strategy annotations. We design a computational model that leverages both textual and visual representations for understanding persuasion behaviors in social deduction games. Our experiments show that visual cues benefit model performance on persuasion strategy prediction. We encourage future work to explore the role of the audio modality in persuasion modeling and to investigate joint learning of multimodal representations for the social persuasion setting.
Limitations
We only use pre-trained video transformers off-theshelf to encode the videos, while more nuanced and specific utilization of other models can be explored to further improve the performance. There are also valuable egocentric videos and demographic statistics along with the Ego4D dataset that we have not yet incorporated in our approach. Due to the difficulty and cost of collecting videos with transcriptions and voting outcome annotations, the total number of games is insufficient to train a deep neural network for voting outcome deduction, though data augmentation techniques can be explored to mitigate this limitation.
Ethics Statement
How humans use persuasion strategies in their communication has been long studied in psychology, communication, and NLP (Hovland et al., 1953; Crano and Prislin, 2006; Petty and Cacioppo, 1986; Yang et al., 2019; Wang et al., 2019; Chen and Yang, 2021). We recognize that persuasion skills could be used for good and bad purposes. In this study, our goal is to study persuasive behaviors by multiple speakers through social deduction games. While we recognize that in both games of One Night Ultimate Werewolf and Avalon games players could use persuasion strategies for behaviors that perhaps are considered morally wrong, such as deception, bias, and emotional manipulation, our study does not encourage such behaviors. Instead, we aim to understand people’s behavior in a group setting when persuasion happens. Having these persuasion skills could benefit people to perform well in their workplace, such as pitching their ideas, or advocating for peace-making (Simons, 1976).
For our data collection and annotation process, this study has been reviewed and approved by our institution’s internal review board. We obtain consent from the players who are recorded and deidentify personally identifiable information (PII), as part of the Ego4D efforts. Moreover, to mitigate potential risks of harmful usage of this dataset in the future, we ask any users to sign an online agreement before using our resources for their research as follows: "I will not use this dataset for malicious purposes (but not limited to): deception, impersonation, mockery, discrimination, manipulation, targeted harassment, and hate speech."

heading: Acknowledgements
text: We are thankful to the members of SALT Lab for their helpful feedback on the draft. This research was supported, in part, by NSF CCRI Research Infrastructure CNS-2308994.

heading: A Game Rules
text: One Night Werewolf. In this game, players are divided into two teams – the team of villagers and the team of werewolves. In each game, players close their eyes in the night phase and take some actions (e.g. swapping cards) depending on their roles. Players’ roles might be changed during the night, but they don’t know their new roles except for a few special roles. Then all players open their eyes. The villager team needs to find the werewolf through communication and negotiation. The werewolf team must mislead the others and try to hide their identities. At the end of the game, everyone has to point out the most suspicious player. If at
least one werewolf is voted out, the villager team wins the game. Otherwise, the werewolf team wins. We refer to https://en.wikipedia.org/wiki/ Ultimate_Werewolf#One_Night_roles for detailed explanations of game rules and roles. The Resistance: Avalon. In this game, players are divided into two teams – the team of Minions and the team of Loyal Servants of Arthur. After shuffling and distributing cards to players, they secretly check their role cards and place them face down on the table. Each player will take turns serving as the Leader. In each round, the Leader proposes a Team to do a Quest, and all players are involved in discussing if the Team assignment is passed or rejected. After the Team Building phase, the approved Team will decide if the Quest is successful or not. In the Quest phase, the Good Team can only use the Quest Success card, and the Evil Team can use either Success or Fail card. The Good Team wins when three successful Quests are made, while the Evil Team wins when three failed Quests are made or the Evil players identify Merlin in the Good Team. We refer to https://en.wikipedia.org/wiki/The_ Resistance_(game)#Avalon_variant for detailed explanations of game rules and roles.

heading: B Transcription Interface
text: We provide a screenshot of the transcription tool (rev.com) in Fig 7. We upload video clips to this online platform for transcription. We also provide player names and roles involved in each game to make the transcription more accurate. As illustrated in Fig 7, they return the transcription of each utterance, the name of the speaker and the corresponding timestamp. Then we ask annotators to watch videos again and examine the alignment of videos and transcripts. Annotators also correct errors in speakers’ names and texts.

heading: C Annotation Interface
text: We provide a screenshot in Fig. 8 of our interface used by annotators to annotate utterance-level persuasion strategies.

heading: D Details of Video Representation
text: We now introduce video representation extraction. Given an utterance Ui, We first localize the corresponding video segment using the utterance timestamp ti, and then approximate the duration of the
utterance by di = ti+1 − ti. We have an average duration of 2 seconds on both the Ego4D and Youtube datasets. To tolerate some misalignment of videos and transcripts, we set a 2-second time window for utterances shorter than 2 seconds, and hence the final duration of an utterance Ui is d′i = max(di, 2). Then we sample N frames out of the corresponding video segment with equal spacing, i.e., V = {V1,V2, . . . ,VN}. All videos in our dataset have an aspect ratio of 16:9. Hence we make three square crops on the left, center, and right of each frame to cover the entire view. Correspondingly, The visual embedding from the vision encoder is composed of three parts, i.e., Vi = {V lefti ,Vcenteri ,V right i }. We input the left crops, center crops, and right crops into the video encoder separately and obtain the corresponding representations, i.e., ψ(V) = {ψ(V left), ψ(Vcenter), ψ(Vright)}. The three video representations are flattened as a single vector when we concatenate them with the text representation. In our experiments, we adopt the 24-layer multiscale vision transformer (MViT) (Fan et al., 2021) pretrained on Kinetics-400 as the video encoder. The number of sampled frames N is set as 32 in our experiments. Note that we don’t finetune the video encoder on our datasets because the performance of some models drops after finetuning due to overfitting.
In the experiments on Ego4D Werewolf data and Youtube data, the video features improve the performance prominently. However, the video feature does not necessarily help with model generalization. When we apply the model Base+V trained on Youtube data to Ego4D data, the average F1 of BERT and RoBERTa drop by 0.8% and 3.0%, respectively, while the F1 of MT-BERT increases by 1.1% after involving video features. This suggests a domain gap exists in the videos of the two datasets which is caused by the differences in the camera locations, angles of views, brightness in the room and etc. Video models are sensitive to these visual differences, resulting in limited performance in generalization on different datasets. In contrast, players communicate in a similar way in different conditions, so the pure text model generalizes better to other data.

heading: E Per-class Results for Experiments with Different Context Lengths
text: We showcase our experimental results including per-strategy scores on incorporating additional conversational context, in full detail in Table 5.

heading: F Experiments of Domain Generalization
text: We demonstrate the detailed experiment results of data domain generalization (training models on Youtube data and testing on Ego4D Werewolf test set), as well as game domain generalization (training models on Ego4D Werewolf data and testing on Avalon data). Results are reported in Table 6 and Table 7, respectively.

heading: G Detailed prompt templates used for GPT-3
text: For the prompt templates, we use the guideline and the persuasion strategy definitions provided to the annotators under the zero-shot setting, and append one/five more examples under one/five-shot setting. The detailed prompt template we used for GPT-3 inference is shown in Table 8.

heading: H Persuasion Strategy Annotation on Avalon Games
text: Adjacent pie-charts comparing the distributions of annotated utterance-level persuasion strategies for
One Night Ultimate Werewolf games and Avalon games in Ego4D are shown in Fig. 9. We can observe a different distribution of adopted persuasion strategies in the two games, suggesting a large game domain gap.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations
3 A2. Did you discuss any potential risks of your work? Ethics Statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3; 4; Appendix C
3 B1. Did you cite the creators of artifacts you used? 3.1; 4.2
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 3.1; 4.2
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 3.1; 4.2
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? 3
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3; 5
C 3 Did you run computational experiments? 4; 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4.2; 4.3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4.4; 5.2; Appendix D E F
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4.4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4.2; 4.3
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 3.2
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix B C
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 3.2
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? 3
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? 3.1
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? 3.2

heading: Future Work
text: We encourage future work to explore the role of the audio modality in persuasion modeling and to investigate joint learning of multimodal representations for the social persuasion setting. Limitations
We only use pre-trained video transformers off-theshelf to encode the videos, while more nuanced and specific utilization of other models can be explored to further improve the performance. There are also valuable egocentric videos and demographic statistics along with the Ego4D dataset that we have not yet incorporated in our approach. Due to the difficulty and cost of collecting videos with transcriptions and voting outcome annotations, the total number of games is insufficient to train a deep neural network for voting outcome deduction, though data augmentation techniques can be explored to mitigate this limitation