Playing repeated games with Large Language Models
Authors: name: Elif Akata, name: Lion Schulz, name: Julian Coda-Forno, name: Seong Joon Oh, name: Matthias Bethge, name: Eric Schulz

Large Language Models (LLMs) are transforming society and permeating into
diverse applications. As a result, LLMs will frequently interact with us and
other agents. It is, therefore, of great societal value to understand how LLMs
behave in interactive social settings. Here, we propose to use behavioral game
theory to study LLM's cooperation and coordination behavior. To do so, we let
different LLMs (GPT-3, GPT-3.5, and GPT-4) play finitely repeated games with
each other and with other, human-like strategies. Our results show that LLMs
generally perform well in such tasks and also uncover persistent behavioral
signatures. In a large set of two players-two strategies games, we find that
LLMs are particularly good at games where valuing their own self-interest pays
off, like the iterated Prisoner's Dilemma family. However, they behave
sub-optimally in games that require coordination. We, therefore, further focus
on two games from these distinct families. In the canonical iterated Prisoner's
Dilemma, we find that GPT-4 acts particularly unforgivingly, always defecting
after another agent has defected only once. In the Battle of the Sexes, we find
that GPT-4 cannot match the behavior of the simple convention to alternate
between options. We verify that these behavioral signatures are stable across
robustness checks. Finally, we show how GPT-4's behavior can be modified by
providing further information about the other player as well as by asking it to
predict the other player's actions before making a choice. These results enrich
our understanding of LLM's social behavior and pave the way for a behavioral
game theory for machines.