Humans expect rationality and cooperation from LLM opponents in
  strategic games
Authors: name: Darija Barak, name: Miguel Costa-Gomes

As Large Language Models (LLMs) integrate into our social and economic
interactions, we need to deepen our understanding of how humans respond to LLMs
opponents in strategic settings. We present the results of the first controlled
monetarily-incentivised laboratory experiment looking at differences in human
behaviour in a multi-player p-beauty contest against other humans and LLMs. We
use a within-subject design in order to compare behaviour at the individual
level. We show that, in this environment, human subjects choose significantly
lower numbers when playing against LLMs than humans, which is mainly driven by
the increased prevalence of `zero' Nash-equilibrium choices. This shift is
mainly driven by subjects with high strategic reasoning ability. Subjects who
play the zero Nash-equilibrium choice motivate their strategy by appealing to
perceived LLM's reasoning ability and, unexpectedly, propensity towards
cooperation. Our findings provide foundational insights into the multi-player
human-LLM interaction in simultaneous choice games, uncover heterogeneities in
both subjects' behaviour and beliefs about LLM's play when playing against
them, and suggest important implications for mechanism design in mixed
human-LLM systems.