Are Large Language Models Strategic Decision Makers? A Study of Performance and Bias in Two-Player Non-Zero-Sum Games
Authors: Nathan Herr, Fernando Acero, Roberta Raileanu, Maria Perez-Ortiz, Zhibin Li


Reviewer paper summaries: This paper examines the strategic decision-making capabilities of Large Language Models (LLMs) in the context of two-player non-zero-sum games. The study explores how LLMs handle game theory tasks, highlighting the influence of systematic biases on their decision-making. These biases include positional bias, payoff bias, and behavioral bias, which significantly affect the LLMs' performance when the game configuration does not align with these biases.
Summary of strengths: 1. Discussing biases in LLMs is a meaningful topic, as addressing these biases can enhance the reliability of LLMs.
Summary of weaknesses: 1. The authors do not effectively justify the motivation for discussing biases in the context of game theoretic tasks. The conclusions in lines 165-168 are unsubstantiated, and previous studies have shown that humans also exhibit positional bias [1].

2. Similarly, the empirical findings are of negligible value, as the biases identified in game theoretic tasks do not significantly differ from those observed in multiple-choice question-answering scenarios. Similarly, the empirical findings are of negligible value, as the biases identified in game theoretic tasks do not significantly differ from those observed in multiple choice question-answering scenarios.

3. The conclusions drawn from specific LLMs have limited applicability, especially since these biases may vary with changes in prompts.

4. The study fails to effectively eliminate the influence of the models' prior knowledge about common strategic games. Merely omitting the names of the games does not prevent the models from utilizing their prior knowledge.


**Refs**
[1] ] Valenzuela, A., & Raghubir, P. (2009). Position-based beliefs: The center-stage effect. Journal of Consumer Psychology, 19(2), 185-196.
Comments, suggestions and typos: See Weakness
Overall assessment scores: 1.5