Venue: NAACL 2025

# Personalized Help for Optimizing Low-Skilled Users' Strategy

Feng Gu\({}^{1}\)  Wichayaporn Wongkamjan\({}^{1}\) Jonathan K. Kummerfeld\({}^{2}\)

Denis Peskoff\({}^{3}\)   Jonathan May\({}^{4}\)    Jordan Lee Boyd-Graber\({}^{1}\)

\({}^{1}\)University of Maryland   \({}^{2}\)University of Sydney

\({}^{3}\)Princeton University

\({}^{4}\)Information Sciences Institute, University of Southern California

{fgu1, wongkgam}@umd.edu jbge.umiacs.umd.edu

###### Abstract

AIs can beat humans in game environments; however, how helpful those agents are to humans remains understudied. We augment Cicero, a natural language agent with superhuman performance in _Diplomacy_, to generate both move and message advice based on player intentions. In a dozen _Diplomacy_ games, novice and experienced players, with varying advice settings, benefit from some of the generated advice. Advice helps novices compete with experienced players and in some instances even surpass them. Just reading advice can be advantageous, even if players do not follow it.1

Footnote 1: Code available at [https://github.com/ALLAN-DIP/diplomacy_cicero/](https://github.com/ALLAN-DIP/diplomacy_cicero/)

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)_, pages 65–74

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)_, pages 65–74

## 1 Leveraging Human-ai Collaboration

AI and humans are frequent collaborators: in writing [9], making decisions [28], and creating artwork [10]. The most fruitful collaborations are those in which humans and computers have complementary skills, such as ai analyzing medical imaging to identify anomalies and doctors interpreting these findings. We posit that the board game _Diplomacy_ is an apt testbed for studying this type of collaboration. Wongkamjian et al. wongkamjianEtAl:2020:ACL study cicero[1], the best _Diplomacy_-playing ai capable of communicating in natural language, and show that while the state-of-the-art ais have near-optimal move strategy, human players remain better at communication.

We introduce **P**ersonalized **H**elp for **O**ptimizing **L**ow-Skilled **U**sers' **S**tategy (**pholus**),2 a natural language agent that provides both moves and messages generated by cicero as advice to _Diplomacy_ players in real-time. The core distinction

between them is that cicero is a game-playing agent whereas pholus is an advisor that does not actively participate in the game. Players' moves and message history influence pholus's advice.

We run a user study and collect a dataset with twelve games, \(1{,}070\) player turns, and \(117\) playing hours. pholus enables novices--who barely know the rules of _Diplomacy_--to compete with experts (Figure 2). But this does not just mean the novices blindly follow the advice. First, they use pholus's strategic insights to inform their communication strategies with other players. Second, pholus helps experienced players, although they are less inclined to take the advice than the novices. Overall, both advice types from pholus improve players' game outcomes (Section 3.1). Our research enables human-ai collaboration and offers valuable insights into the potential of using ai to enhance human learning experiences.

On a broader scale, our study explores the potential for ais like pholus to enhance learning in unfamiliar environments. ai agents surpass traditional rule-based methods by offering more flexible and personalized learning experiences. Integrating tailored guidance into human intelligence, these systems provide unique learning experiences for inexperienced individuals. Future research directions in human-ai collaboration include generating advice based on high-level intentions and goals, reducing over-reliance on skilled ais, and facilitating learning processes.

## 2 _Dipolmacy_ as a Cooperative Testbed

_Diplomacy_ is a seven-player, turn-based board game. The goal is to obtain more than half of the board's possible points.3 Critically, turns are simultaneous, with moves written in secret by players and then revealed. This means that players must communicate to collaborate effectively.

Footnote 3: Represented by a subset of spaces / territories on the map termed _supply centers_.

### Experiment Setup

We recruit _Diplomacy_ players online. For experienced players, we advertise in the _Diplomacy_ community (specifically players active on _webDiplomacy_ and _Backstabbr_, as well as in-person tournament attendees). To find novice players, we contact board game enthusiasts in university clubs. A novice player is someone who has no prior _Diplomacy_ experience and is unfamiliar with its rules.

We modify a game engine and interface Paquette et al. (2019) and maintain the same game format used by Wongkamjan et al. (2024). Each game involves two to five human players. Games last about three hours, with each turn taking ten minutes.

As illustrated in Figure 1, pholus passively observes the game. If cicero is an active participant, it would have submitted moves and sent messages based on the game state and its message history. Instead, pholus presents these moves and messages as advice to players. Each time the player sends a message, pholus recomputes advice given the new context and presents it to the user. Every turn, we randomly assign each player to one of the following settings:

**1) No advice**: pholus does not offer any information, meaning the player receives no assistance from pholus.

**2) Message advice**: pholus suggests _to whom_ a player should send a message and _what_ the message content could be.

**3) Move advice**: pholus recommends a set of moves (or unit orders) to the player.

**4) Message and move advice**: Combines the previous two types.

In total, we collect data from twelve games involving fourty-one players. This includes over \(3{,}600\) entries of move advice and \(4{,}300\) pieces of message advice (Table 1).

### Evaluation Metrics

To assess the effectiveness of pholus's advice, we consider the net gain or loss of points in each turn as the effect of advice. We train a linear regression model with regularization to examine the advice's effectiveness. The model includes features such as which of the seven Great Powers is assigned to the player, the number of turns that have passed, the player's type (novice or veteran), and the advice setting. We encode the Power, player type, and advice setting as one-hot vectors.

\begin{table}
\begin{tabular}{l r r r r} \hline  & \multicolumn{2}{c}{**Move Advice**} & \multicolumn{2}{c}{**Message Advice**} \\  & Accepted & Total & Accepted & Total \\ \hline
**Novices** & 32.6\% & 872 & 6.3\% & 1413 \\
**Veterans** & 6.4\% & 2807 & 3.4\% & 2912 \\ \hline \end{tabular}
\end{table}
Table 1: Statistics of advice generated by pholus and accepted by players. _Diplomacy_ novices are more willing to accept move and message advice than veterans. Move advice is more frequently accepted than message advice for both novices and veterans.

 

To evaluate players' reliance on pholus, we use both qualitative and quantitative methods. In addition to computing move advice acceptance frequency, we also measure agreement and equivalence between the move suggested by pholus and a player's moves. Agreement is the proportion of moves that appeared in both the players' move set and pholus's advice set in a given turn. The sets are equivalent if they overlap entirely. Formally, we define move agreement \({\cal A}\) in turn \(i\) as \({\cal A}_{x_{i},y_{i}}=|x_{i}\cap y_{i}|/|x_{i}|\) and equivalence \({\cal E}\) as \({\cal E}_{x_{i},y_{i}}=1\) if \(x_{i}=y_{i}\) and \({\cal E}_{x_{i},y_{i}}=0\) otherwise, where \(x_{i}\) is the player's move set and \(y_{i}\) is pholus's move advice set in turn \(i\)4 Agreement is particularly useful for capturing the overlap when players reject the complete move advice set but follow individual advice from pholus.

Footnote 4: For any \(i\), \(|x_{i}|=|y_{i}|\).

## 3 Pholus Provides Helpful Advice

### Quantitative Analysis

**Non-advice factors parallel previous findings.** Playing as France offers the most strategic advantage Burton (2007). cicero playing as Germany or Italy is correlated with better game outcomes, while playing as Austria, England, or Turkey is correlated with worse game outcomes Wongkamjan et al. (2024). Additionally, cicero dominates: of twelve games, cicero won eight.

**Advice helps.** Playing a game without advice puts players at a disadvantage. The feature associated with no advice has a negative coefficient of approximately \(-0.05\) (Figure 2). The coefficients suggest a slight positive correlation between receiving move advice and point gains. Players who receive both move and message advice gain more points than those who receive only move advice. Interestingly, only having message advice negatively affect players' game outcomes.

**Novices can outperform experienced players with the help of pholus.** Players with no prior experience in _Diplomacy_ naturally face a disadvantage against seasoned players. This often results in novices being eliminated relatively early in the game. Even if they remain in the game, losing supply centers is almost inevitable. However, novice players receiving advice play better: in five games where novices received message and move advice, only one player was eliminated before the game concluded (typically \(3\!-\!4\) players in a game are eliminated). In the other four games, novices ended the game with more supply centers than they started with.

**Novices are more likely to follow pholus's advice.** Experienced players tend to disregard advice. They accept only \(3.4\%\) of message advice and \(6.4\%\) of move advice from pholus. Although novice players are also hesitant to accept message advice, doing so \(6.3\%\) of the time, this rate is nearly double that of experienced players. Novice players follow move advice approximately one-third of the time, with an acceptance rate of \(32.6\%\). Both novice and experienced players tend to take more move advice than message advice.

**Novices do not fully trust move advice from pholus.** Across all games, pholus generates \(333\) instances of individual move advice for novices, organized into \(134\) sets. At the start of turns, average move agreement is \(80\%\) and average equivalence is \(46\%\), indicating strong alignment between novices' initial idea for moves and pholus's move advice. However, by the end of each turn, the average agreement drops by \(10\%\) and the average equivalence decreases by \(8\%\), indicating that novice players do not follow the move advice blindly.

### Qualitative Analysis

While we can compute equivalence \({\cal E}\) for moves, this is more difficult for messages. To better understand why players reject pholus's advice more

Figure 2: Regression coefficients for advice settings and player skills to predict supply center gains. Not receiving any advice from pholus is slightly disadvantageous. Move advice has a positive correlation with player performance. Receiving both forms of advice has the greatest positive impact. As expected, not having previous exposure to _Diplomacy_ is indicative of bad performance. However, with the help of pholus’s advice, _Diplomacy_ novices are on the same level as veterans and have the potential to defeat experienced players.

 

than they accept it (Table 1), we qualitatively investigate the differences between pholus's suggested messages and the actual messages sent by players. To analyze message content, we use Abstract Meaning Representation (amr, Banarescu et al., 2013) to extract _Diplomacy_-specific tokens. We parse player messages and the corresponding message advice from pholus to amr. We then measure the similarity of the two parses using smatch score Cai and Knight (2013). Many pairs have high smatch, indicating that players often incorporate parts of pholus's advice into their messages. For example, pholus suggests "_bounce in Galicia again?_" while the player wrote "_Do you want to bounce in Galicia again?_" Despite being written differently, these clearly have the same meaning, and indeed, smatch gives the pair a score of \(0.74\).

We also notice message-advice pairs with low smatch scores, where human players have different objectives in mind. For instance, in the fifth game, Italy captures Warsaw from Russia and anticipates losing it in the next turn due to Russia's stronger nearby presence. When Russia injquires about the unexpected attack, pholus suggests using the fallacy of deflection to feign ignorance: "_Turkey has been the only one to heed my concerns, despite my reservations,_" and, "_I thought you were lying_". However, the player disregards the advice of talking to Russia. Instead, the player seeks help from Turkey, who has an adjacent unit, to "_support Warsaw's hold_". The player then secures the support from Turkey and successfully keeps Warsaw from subsequent Russian attack. These pairs yield smatch scores of \(0\).

Our analysis indicates that smatch scores match with our intuitions about textual similarity. Given the many high smatch scores, we can conclude that many messages that are sent by players are minor variations on the provided advice. We provide more examples in Appendix A.6. For additional qualitative insights, we survey players on the effectiveness of the advice. We summarize the survey results in Appendix A.7.

## 4 Related Work

**Appropriate Reliance on AI:** The topic of human reliance on ai is central to current research in machine learning and explainable ai. Prior work measures reliance in ai-assisted decision making Schemmer et al. (2023); Chen et al. (2023); Schoeffer et al. (2024); Zhou et al. (2024), and explores reducing over-reliance Bucinca et al. (2021); Schemmer et al. (2022); Vasconcelos et al. (2023). Some researchers have examined how explanations affect human reliance on ai Starke et al. (2021); Vereschak et al. (2021). However, empirical evidence from multiple domains shows conflicting results: while some show that ai explanations improve human decision making, others find evidence of over-reliance on ai explanations even when they are incorrect Lai and Tan (2019); Bucinca et al. (2020); Zhang et al. (2020); Wang and Yin (2021); Bansal et al. (2021); Poursabzi-Sangdeh et al. (2021); Liu et al. (2021); Kim et al. (2022); Si et al. (2024). For pholus, humans remain relatively conservative toward ai advice. Even novice _D Diplomacy_ players do not blindly follow the advice.

**AI as Player Companion:** AI agents have a long history of superhuman gameplay. In 1996, IBM's Deep Blue defeated the reigning world chess champion, Garry Kasparov, although it lost several other games in the same match Campbell et al. (2002). More recently, DeepMind's AlphaGo Silver et al. (2016) consistently defeated top-rated Go players, a game with exponentially complex computational space, and later changed professional Go players' play style. Multi-agent reinforcement learning systems like AlphaStar Vinyals et al. (2019) and OpenAI Five Berner et al. (2019) also show high performance in computer games through self-play.

However, these experiments focus only on game outcomes rather than how they can shape human gameplay. Some studies on nlp communicative agents aim to generate guidance in a grounded environment McGee and Abraham (2010). Tremblay and Verbrugge (2013) develop an adaptive ai companion that adjusts its behavior based on the player's experience. Dunning et al. (2024) assess human reliance on ai-based advice by examining the skill level of ai agents and the presentation of advice. While these studies show that ais outperform non-adaptive agents in guiding players, they do not consider player intention when generating guidance. In comparison, pholus takes players' past messages and moves entered when generating personalized advice.

**Augmented Learning:** This is an educational approach that enhances and personalizes the learning experience. Traditionally, peer interaction simulates social interaction and helps learning Kim and Baylor (2006). Recent advancements in ai 

and nlp agents, suggest adaptive pedagogical interactions between humans and these agents to help learning in new environments (Moreno and Mayer, 2000, 2004; Hirsh-Pasek et al., 2015; Johnson and Lester, 2018). Zhou et al. (2023) apply the theory of mind to generate guidance for players in _Dungeons and Dragons_. Ruan et al. (2020) develop a narrative-based tutoring system and show that it helps effective learning for children. In this study, we apply the concept of augmented learning to help novices understand the game of _Diplomacy_.

## 5 Conclusion

Human-ai collaboration depends on a range of factors. Using the board game _Diplomacy_, pholus provides real-time move and message advice tailored to intentions of both novice and experienced players. Surprisingly, even though only some advice is accepted, it can have a substantial impact on outcomes, particularly for novice players. This is because advice can positively inform choices even if the advice isn't strictly followed. Our experiments enable further study of human-ai collaboration, including modeling explicit intentions and how to better use knowledge within these models. On a broader scope, future research should consider how ai can inform people without making choices for them and measure that impact.

## 6 Limitations

While we can effectively use pholus to generate both message and move advice for players, this advice can be too general or may not align with player intentions at times. For example, when a player expresses interest in an alliance with another player, pholus may give aggressive move advice deemed hostile toward that Power. We suspect that the advice may be optimized more for cicero's intentions, which come from optimal moves in the supervised training data. Consequently, players who are willing to sacrifice individual optimality for mutual gains may find the advice less useful.

Furthermore, pholus cannot generate advice based on high-level player intentions. Specifically, pholus generates move advice based on optimal utility and message advice by inferring intentions from player-input moves. Potential improvements include 1) explaining meta-level intentions (e.g., ally with Germany and prioritize defeating Austria) from player input, and 2) generating targeted move and message advice based on meta-level intentions.

Finally, pholus is a resource-intensive advisor that runs on high-end gpus that require a large amount of on-chip memory (over 35GB). We use Nvidia's A100 for running pholus. This limits accessibility for _Diplomacy_ players and researchers to efficiently utilize pholus. The community would benefit from a distilled version of pholus by reducing computational limits and future adaptations.

## 7 Ethical Considerations

We recruit players individually via email and assign pseudonyms to ensure anonymity, even if players know each other outside the experiment. We adhere to human subject research regulations and the study was approved by our institution's ethics review board (IRBNet ID: 1740681, University of Maryland). We report the experimental procedure in Appendix A.3 and compensation details inA.4.

## Acknowledgments

We thank Meta for open sourcing cicero. We thank the _Diplomacy_ community for taking interest in our study. Specifically, our thank goes to Matthew Totonchy, Dr. Abhishek Singhal, Antonio Imperato, Sophia Wiste, and other members of the community who took the time to play against cicero. In addition, we thank Yanze Wang and Sadra Sabouri from University of Southern California for their helpful feedback. Denis Peskoff is supported by the National Science Foundation under Grant No. 2127309 to the Computing Research Association for the CIFellows 2021 Project. This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Agreement No. HR00112490374 Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the view of our sponsors.

## References

* Anthony et al. [2020] Thomas Anthony, Tom Eccles, Andrea Tacchetti, Janos Kramar, Ian Gemp, Thomas C. Hudson, Nicolas Porcel, Marc Lanctot, Julien Perolat, Richard Everett, Roman Werpachowski, Satinder Singh, Thore Graepel, and Yoram Bachrach. 2020. Learning to play No-press Diplomacy with best response policy iteration. In _Proceedings of the 34th International Conference on Neural Information Processing Systems_, NIPS '20, Red Hook, NY, USA. Curran Associates Inc.
* Bakhtin et al. [2019] Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, 

Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Rendhuntalta, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. 2022. Human-level play in the game of Diplomacy by combining language models with strategic reasoning. _Science_, 378(6624):1067-1074.
* Bakhtin et al. [2021] Anton Bakhtin, David Wu, Adam Lerer, and Noam Brown. 2021. No-press Diplomary from scratch. In _Advances in Neural Information Processing Systems_.
* Banarescu et al. [2013] Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for Sembanking. In _Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_. Association for Computational Linguistics.
* Bansal et al. [2019] Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S. Lasecki, Daniel S. Weld, and Eric Horvitz. 2019. Beyond accuracy: The role of mental models in human-AI team performance. In _AAAI Conference on Human Computation & Crowdsourcing_.
* Bansal et al. [2021] Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? The effect of AI explanations on complementary team performance. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_, CHI '21, New York, NY, USA. Association for Computing Machinery.
* Berner et al. [2019] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Jozefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d. O. Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. 2019. Dota 2 with large scale deep reinforcement learning.
* Bucinca et al. [2020] Zana Bucinca, Phoeeo Lin, Krzysztof Z. Gajos, and Elena L. Glassman. 2020. Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems. In _Proceedings of the 25th International Conference on Intelligent User Interfaces_, IUI '20, page 454-464, New York, NY, USA. Association for Computing Machinery.
* Bucinca et al. [2021] Zana Bucinca, Maja Barbara Malaya, and Krzysztof Z. Gajos. 2021. To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AlAssisted decision-making. _Proceedings of the ACM on Human Computer Interaction_, 5(CSCW1).
* Burton [2007] Josh Burton. 2007. The statistician: Solo victories. [https://diplom.org/Zine/F2007R/Burton/statistician3.htm](https://diplom.org/Zine/F2007R/Burton/statistician3.htm). Accessed: 2024-10-15.
* Cai and Knight [2013] Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 748-752, Sofia, Bulgaria. Association for Computational Linguistics.
* Campbell et al. [2002] Murray Campbell, A.Joseph Hoane, and Feng hsiumg Hsu. 2002. Deep blue. _Artificial Intelligence_, 134(1):57-83.
* Chen et al. [2023] Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, and Gagan Bansal. 2023. Understanding the role of human intuition on reliance in human-AI decision-making with explanations. _Proceedings of the ACM on Human Computer Interaction_, 7(CSCW2).
* Dunning et al. [2024] Richard E. Dunning, Baruch Fischhoff, and Alex L. Davis. 2024. When do humans need AI agents' advice? When should they? _Human Factors_, 66(7):1914-1927. PMID: 37553098.
* Ferreira et al. [2015] A Ferreira, Henriquep Lopes Cardoso, and Luis Reiss. 2015. Strategic negotiation and trust in Diplomacy-the Diplbue approach. In _Transactions on Computational Collective Intelligence XX_, pages 179-200.
* Gray et al. [2021] Jonathan Gray, Adam Lerer, Anton Bakhtin, and Noam Brown. 2021. Human-level performance in no-press Diplomacy via equilibrium search. In _International Conference on Learning Representations_.
* Hirsh-Pasek et al. [2015] Kathy Hirsh-Pasek, Jennifer M. Zosh, Roberta Michnick Goilinkoff, James H. Gray, Michael B. Robb, and Jordy Kaufman. 2015. Putting education in "educational" apps: Lessons from the science of learning. _Psychological Science in the Public Interest_, 16(1):3-34. PMID: 25985468.
* Jacob et al. [2022] Athul Paul Jacob, David J Wu, Gabriele Farina, Adam Lerer, Hengyuan Hu, Anton Bakhtin, Jacob Andreas, and Noam Brown. 2022. Modeling strong and human-like gameplay with KL-regularized search. In _Proceedings of the 39th International Conference on Machine Learning_, volume 162 of _Proceedings of Machine Learning Research_, pages 9695-9728. PMLR.
* Johnson and Lester [2018] W. Lewis Johnson and James C. Lester. 2018. Pedagogical agents: Back to the future. _AI Magazine_, 39(2):33-44.
* Kim et al. [2022a] Eunsee Kim, Jeongmin Hong, Hyuna Lee, and Minsam Ko. 2022a. Colorbo: Envisioned mandala colortingthrough human-AI collaboration. In _Proceedings of the 27th International Conference on Intelligent User Interfaces_, IUI '22, page 15-26, New York, NY, USA. Association for Computing Machinery.
* Kim et al. [2022b] Sunnie S. Y. Kim, Nicole Meister, Vikram V. Ramaswamy, Ruth Fong, and Olga Russakovsky. 2022b. Hive: Evaluating the human interpretability of visual explanations. _arXiv preprint arXiv:2112.03184_.
* Kim and Baylor [2006] Yanghee Kim and Amy Baylor. 2006. A social-cognitive framework for pedagogical agents as learning companions. _ITLS Faculty Publications_, 54 

Vivian Lai and Chenhao Tan. 2019. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_, FAT* '19, page 29-38, New York, NY, USA. Association for Computing Machinery.
* Lee et al. (2022) Mina Lee, Percy Liang, and Qian Yang. 2022. CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_, CHI '22, New York, NY, USA. Association for Computing Machinery.
* Liu et al. (2021) Han Liu, Vivian Lai, and Chenhao Tan. 2021. Understanding the effect of out-of-distribution examples and interactive explanations on human-AI decision making. _Proceedings of the ACM on Human Computer Interaction_, 5(CSCW2).
* McGee and Abraham (2010) Kevin McGee and Aswin Thomas Abraham. 2010. Real-time team-mate AI in games: a definition, survey, & critique. In _Proceedings of the Fifth International Conference on the Foundations of Digital Games_, FDG '10, page 124-131, New York, NY, USA. Association for Computing Machinery.
* Moreno and Mayer (2000) Roxana Moreno and Richard Mayer. 2000. Engaging students in active learning: The case for personalized multimedia messages. _Journal of Educational Psychology_, 92:724-733.
* Moreno and Mayer (2004) Roxana Moreno and Richard Mayer. 2004. Personalized messages that promote science learning in virtual environments. _Journal of Educational Psychology_, 96:165-173.
* Paquette et al. (2019) Philip Paquette, Yuchen Lu, Seton Steven Bocco, Max Smith, Satya O.-G., Jonathan K. Kummerfeld, Joelle Pineau, Satinder Singh, and Aaron C Courville. 2019. No-Press Diplomacy: Modeling multi-agent gameplay. In _Advances in Neural Information Processing Systems_, volume 32. Curran Associates, Inc.
* Polberg et al. (2011) Sylwia Polberg, Marcin Paprzycki, and Maria Ganzha. 2011. Developing intelligent bots for the Diplomacy game. In _2011 Federated Conference on Computer Science and Information Systems (FedCSIS)_, pages 589-596.
* Poursabzi-Sangdeh et al. (2021) Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Workman Vaughan, and Hanna Wallach. 2021. Manipulating and measuring model interpretability. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_, CHI '21, New York, NY, USA. Association for Computing Machinery.
* Rose et al. (2007) Andrew Rose, David Norman, and Hamish Williams. 2007. Diplomacy artificial intelligence development environment. [http://daide.org.uk/index.html](http://daide.org.uk/index.html). Accessed: 2024-10-06.
* Ruan et al. (2020) Sherry Ruan, Jiayu He, Rui Ying, Jonathan Burkle, Dunia Hakim, Anna Wang, Yufeng Yin, Lily Zhou, Qiangxo Xu, Abdallah AbuHashem, Griffin Dietz, Elizabeth L. Murane, Emma Brunskill, and James A. Landay. 2020. Supporting children's math learning with feedback-augmented narrative technology. In _Proceedings of the Interaction Design and Children Conference_, IDC '20, page 567-580, New York, NY, USA. Association for Computing Machinery.
* Schemmer et al. (2022) Max Schemmer, Patrick Hemmer, Niklas Kuhl, Carina Benz, and Gerhard Satzger. 2022. Should I follow AI-based advice? Measuring appropriate reliance in human-AI decision-making. _arXiv preprint arXiv:2204.06916_.
* Schemmer et al. (2023) Max Schemmer, Niklas Kuehl, Carina Benz, Andrea Bartos, and Gerhard Satzger. 2023. Appropriate reliance on AI advice: Conceptualization and the effect of explanations. In _Proceedings of the 28th International Conference on Intelligent User Interfaces_, IUI '23, page 410-422, New York, NY, USA. Association for Computing Machinery.
* Schoeffer et al. (2024) Jakob Schoeffer, Maria De-Arteaga, and Niklas Kuhl. 2024. Explanations, fairness, and appropriate reliance in human-AI decision-making. In _Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems_, CHI '24, New York, NY, USA. Association for Computing Machinery.
* except when they are convincingly wrong. _arXiv preprint arXiv:2310.12558_.
* Silver et al. (2016) David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016. Mastering the game of Go with deep neural networks and tree search. _nature_, 529(7587):484-489.
* Starke et al. (2021) Christopher Starke, Janine Balieis, Birte Keller, and Frank Marcinkowski. 2021. Fairness perceptions of algorithmic decision-making: A systematic review of the empirical literature. _arXiv preprint arXiv:2103.12016_.
* Tremblay and Verbrugge (2013) Jonathan Tremblay and Clark Verbrugge. 2013. Adaptive companions in FPS games. In _International Conference on Foundations of Digital Games_.
* van Hal (2009) Jason van Hal. 2009. Albert. [https://sites.google.com/site/diplomacysai/albert](https://sites.google.com/site/diplomacysai/albert). Accessed: 2024-10-06.
* Vasconcelos et al. (2023) Helena Vasconcelos, Matthew Jorke, Madeleine Grundeclaughlin, Tobias Gerstenberg, Michael S. Bernstein, and Ranjay Krishna. 2023. Explanations can reduce overreliance on AI systems during decision-making. _Proceedings of the ACM on Human Computer Interaction_, 7(CSCW1) 

* Vereschak et al. (2021) Oleksandra Vereschak, Gilles Bailly, and Baptiste Caramiaux. 2021. How to evaluate trust in AI-assisted decision making? A survey of empirical methodologies. _Proceedings of the ACM on Human Computer Interaction_, 5(CSCW2).

* Wang and Yin (2021) Xinru Wang and Ming Yin. 2021. Are explanations helpful? A comparative study of the effects of explanations in Al-assisted decision-making. In _Proceedings of the 26th International Conference on Intelligent User Interfaces_, IUI '21, page 318-328, New York, NY, USA. Association for Computing Machinery.
* Wilkenfeld (2019) Yoni Wilkenfeld. 2019. Can chess survive artificial intelligence? _The New Atlantis_, (58):37-45.
* Wongkamjan et al. (2024) Wichayaporn Wongkamjan, Feng Gu, Yanze Wang, Ulf Hermjakob, Jonathan May, Brandon Stewart, Jonathan K. Kummerfeld, Denis Peskoff, and Jordan Boyd-Graber. 2024. More victories, less cooperation: Assessing Cicero's Diplomacy play. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 12423-12441, Bangkok, Thailand. Association for Computational Linguistics.
* Zhang et al. (2020) Yunfeng Zhang, Q. Vera Liao, and Rachel K. E. Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_, FAT* '20, page 295-305, New York, NY, USA. Association for Computing Machinery.
* Zhou et al. (2024) Kaitlyn Zhou, Jena D. Hwang, Xiang Ren, Nouha Dziri, Dan Jurafsky, and Maarten Sap. 2024. REL-A.I.: An interaction-centered approach to measuring human-LM reliance. _arXiv preprint arXiv:2407.07950_.
* Zhou et al. (2023) Pei Zhou, Andrew Zhu, Jennifer Hu, Jay Pujara, Xiang Ren, Chris Callison-Burch, Yejin Choi, and Prithviraj Ammanabrolu. 2023. An AI dungeon master's guide: Learning to converse and guide with intents and theory-of-mind in Dungeons and Dragons. In _Proceedings of the The 61st Annual Meeting of the Association for Computational Linguistics_, Toronto, Canada.

## Appendix A Appendix

### Diplomacy

_Diplomacy_ is a board game that has two core components: strategy and communication. Strategic reasoning plays a crucial role in determining the game's outcome, as players' moves directly impact the board's status. Meanwhile, negotiation and deception significantly influence player strategies. Successful cooperation can remove a common adversary from the board, while a well-timed betrayal by a trusted ally can be catastrophic, greatly reducing the chances of winning. Excelling in _Diplomacy_ requires not only a thorough understanding of the game's mechanics but also strong communication skills. Consequently, _Diplomacy_ is an ideal testbed for studying human-ai interaction and appropriate reliance in a grounded environment where outcomes are clearly observable.

Early efforts to develop agents for _Diplomacy_ concentrated solely on creating rule-based agents that relied heavily on feature engineering van Hal (2009). These agents only submit moves and are not capable of communication. In 2002, a group of programmers released a communication protocol, _Diplomacy Artificial Intelligence Development Environment_daide, Rose et al. (2007). DAIDE defines a language syntax that enables agents to diplomatically negotiate and describe game actions. Following daide, researchers built communicative agents, including Albert van Hal (2009), SillyNegoBotPoherg et al. (2011), DipBlue Ferreira et al. (2015).

Starting with DipNet Paquette et al. (2019), neural networks were applied to the game, leading to the first agents that were competitive with people. Subsequent studies incorporated reinforcement learning to achieve super-human performance Gray et al. (2021); Bakhtin et al. (2021); Anthony et al. (2020); Jacob et al. (2022).

### All Regression Coefficients

In Figure 2, we only show regression coefficients related to advice setting and player experience. Figure 3 contains coefficients for all regression features. The official _Diplomacy_ rule states that supply center control changes only on even turns. However, we consider moving a unit to a center on an odd turn as gaining it, since the unit typically remains there in the next turn 



### Experiment Procedures

The procedures involve playing _Diplomacy_ with other participants as well as ai agents and providing feedback throughout the game on 1) the truth value of players' messages 2) the predicted truth values of their opponents' messages 3) the perceived friendliness of opponents 4) if players think they have been lied to in the previous phase. Players are asked to participate by placing their game moves in each turn before communicating. Some players in some turns received strategic moves and message advice from pholus. Games end after 3 hours or less. After each game, participants are asked to fill out a survey asking about their experience and the ai advice, as well as how ai agents performed during the game.

### Participant Compensation

Players received a $70 gift card of their choice for each game. In addition, up to $29 was rewarded based on performance in the game.

### Survey Questions

We use Google Forms to conduct player surveys. Table 2 shows the survey questions.

### Pholus's message advice and human messages with smatch scores

We provide additional examples of cases where human players reject pholus's message advice, together with smatch. We show some advice (Figure 6 and 7) that experienced players mostly agree with, however, they partially edit to make it more aligned with their goals. We also show advice that is not aligned with the player's goals, for example in Figure 8.

### Survey Summarization

Many participants find pholus's move advice helpful. Experienced players observe that the advice often aligns with their own moves and offers "_some interesting ideas_." However, the advice can be suboptimal and short-sighted in complex scenarios, where it fails to consider interaction between

Figure 4: Human uses pholus’s advice directly without modification.

Figure 5: Italy paraphrases the advice with the same underlying intention.

Figure 7: The advice and human message gainet the same goal that rejects the other player’s proposal, instead using their unit to bounce in Bur.



allies. Players find message advice useful for simple, quick communication but inadequate for more complex or specific situations, especially when it does not align with their strategies or alliances. Players mention that the messages include common communication terms, and they "_regret not using this feature more._" However, the advice is less helpful for specific planning and often does not align with player alliances and intentions.

\begin{table}
\begin{tabular}{l l} \hline \hline
**Question** & **Type** \\ \hline I am really good at Diplomacy. & Likert scale \\ How many Diplomacy games have you played before? & Free text \\ I am able to identify all bots. & Likert scale \\ I enjoy talking with the AIs. & Likert scale \\ I was able to make plans with other players in the game. & Likert scale \\ I was able to make plans with the AIs in the game. & Likert scale \\ human players communicated transparently. & Likert scale \\ AI players communicated transparently. & Likert scale \\ Do you have a link to your diplomacy stats? & Free text \\ How helpful is the move advice? & Likert scale \\ Thoughts on the move advice? & Free text \\ How helpful is the message advice? & Likert scale \\ Thoughts on the message advice? & Free text \\ Feel free to let us know your other post-game thoughts. & Free text \\ \hline \hline \end{tabular}
\end{table}
Table 2: List of questions in the survey.

Figure 9: An example of low SMATCH. PHOLUS advises Italy to inquire about a specific game move, but the player discusses a high-level game plan.