Efficacy of Language Model Self-Play in Non-Zero-Sum Games

## Abstract

Abstract Game-playing agents like AlphaGo have achieved superhuman performance through self-play, which is theoretically guaranteed to yield optimal policies in competitive games. However, most language tasks are partially or fully cooperative, so it is an open question whether techniques like self-play can effectively be used to improve language models. We empirically investigate this question in a negotiation game setting known as Deal or No Deal (DoND). Crucially, the objective in DoND can be modified to produce a fully cooperative game, a strictly competitive one, or anything in between. We finetune language models in self-play over multiple rounds of filtered behavior cloning in DoND for each of these objectives.
Contrary to expectations, we find that language model self-play leads to significant performance gains in both cooperation and competition with humans , suggesting that self-play and related techniques have promise despite a lack of theoretical guarantees.

## 1 Introduction

Many of the greatest achievements in artificial intelligence have occurred in two-player zero-sum (2p0s) games such as Go (Silver et al., ), chess (Silver et al., ), and heads-up poker (Brown and Sandholm, ).
One key technique enabling these breakthroughs has been self-play, in which identical copies of a model are pitted against each other and used to generate new training data.
By iteratively training on their own data from games of self-play, models like AlphaGo and AlphaZero were able to continue improving long past the threshold of human performance.
In certain types of 2p0s games, self-play is theoretically guaranteed to produce optimal policies, given sufficient model capacity and compute (Bai and Jin, ; Bai et al., ).
However, in settings that involve collaboration with humans, self-play is no longer guaranteed to yield optimal policies (Strouse et al., ).

It is an open question whether the same principles that led to the success of models like AlphaGo can be applied to language models.
Empirically, previous work on training agents to communicate via self-play has shown that they often invent uninterpretable communication strategies (Kottur et al., ); even when initialized with natural language data, self-play can cause models to gradually diverge from human-interpretable language (Lewis et al., ). As a result, much work has focused on mitigating these challenges, e.g., by regularizing with models trained on human data (FAIR, ).

In this work, we examine the effect of game objectives on self-play between language models. We run a series of experiments on a negotiation task known as Deal or No Deal (Lewis et al., ) and train language models for multiple rounds of self-play across three different objectives on this task, ranging from fully cooperative, to semi-competitive, to strictly competitive.
Contrary to expectations, we find that self-play leads to large improvements in both the cooperative and semi-competitive settings. These results generalize to human experiments, where scores improve by up to 2.5$\times$ in the cooperative setting and 6$\times$ in the semi-competitive setting.
In contrast, we find minimal improvements in the strictly competitive setting, where models tend to overfit during self-play.

Figure: Figure 1: We ran experiments on a modified version of the Deal or No Deal negotiation game from Lewis et al. (). In this game, two players are presented with a shared collection of items and private value functions over those items. Players can send messages to each other and then each submit private proposals describing the items they wish to receive. If the proposals are compatible, then the items are scored. In our modified version of the task, players may receive reward based not only on their own item scores, but on the item scores of the other player as well. This modification allows us to convert Deal or No Deal into a cooperative or strictly competitive game.
Refer to caption: x1.png

We then investigate the reasons behind these improvements, finding that models trained with self-play better follow task instructions, hallucinate less, and obtain a higher agreement rate with humans. However, at the same time, self-play causes model dialogues to become less diverse and does not appear to teach high-level strategic reasoning or negotiation tactics in our experiments. Although these results highlight potential room for improvement, we view them as a promising initial signal for self-play training of large language models and release all code for our environments, models, and human data collection to support future research in this area: [github.com/nickatomlin/lm-selfplay](https://github.com/nickatomlin/lm-selfplay).

## 2 Cooperative and Competitive Games

Can language model self-play be effective under both cooperative and competitive objectives?
To address this question, we conducted experiments on Deal or No Deal (DoND; Lewis et al., ), a two-player negotiation game in which players decide how to divide a shared pool of items through natural language dialogue.
Although introduced as a semi-competitive game, DoND has the special property that it can be readily adapted into either a cooperative or strictly competitive (i.e., zero-sum) game. Below, we describe the rules of DoND, how we modify its objective, and how we convert it into an environment for evaluating language models.

#### Game Setup

Following Lewis et al. (), we present two players with a shared collection of books, hats, and balls (with 5-7 total objects).
Each player is assigned their own private value function, mapping each item type to an integer point value.
Value functions are selected according to the following criteria: (1) each item is valued by at least one player, (2) the maximum score either player can receive is 10, and (3) at most one player can achieve the maximum score.
Players must divide the objects; if they fail to reach an agreement, they both receive zero points. These rules ensure that the game is semi-competitive: players have conflicting objectives, but if they fail to cooperate at all then they will end up without any points.

#### Game Rules

The game is divided into two phases. In the first phase, players send messages discussing which items they would like to receive. At any point, either player may end this phase by submitting a private proposal, delineating which items they would like to claim from the shared collection. During the second phase, no additional messages can be sent, and the other player must respond by submitting a proposal of their own, which ends the game. If players submit complementary proposals (i.e., adding up to the total number of objects in the shared collection), then players receive rewards according to their respective objectives. Hence, players should aim both to reach an agreement and to optimize the value of that agreement.

#### Game Objectives

In the original formulation, players receive a reward equal to the inner product of their value function and proposed set of objects. However, we observe that this objective can be modified to convert DoND into a cooperative game, a strictly competitive one, or anything in between. For example: if players receive rewards $R_{1}$ and $R_{2}$ in the original setting, instead setting the objective to $R_{1}+R_{2}$ for both players results in a fully cooperative game. More generally, we can define the objective for Player 1 as $R_{1}+\lambda\cdot R_{2}$ for $\lambda\in[-1,1]$, and vice versa for Player 2. In this work, we experiment with $\lambda=0$ (semi-competitive), $\lambda=1$ (cooperative), and $\lambda=-1$ (strictly competitive), although we note that in principle $\lambda$ can be tuned to smoothly interpolate between these objectives. The maximum reward that can be obtained in a single game is 10 in the strictly and semi-competitive settings and 19 in the cooperative setting.(^1^11Reward is obtained by adding the item scores from each player in the cooperative setting. However, the maximum possible reward is 19 and not 20 because of the game’s constraint that at most one player can achieve the maximum item score. The maximum average self-play reward is 7.5 in the semi-competitive setting and 15 in the cooperative setting; in the zero-sum setting, average self-play scores are always zero.)

#### Game Environment

Akin to recent work on language agents (Abdulhai et al., ; Lin et al., ), we implement an OpenAI Gym-like (Brockman et al., ) environment for evaluating language models on DoND. This environment provides affordances for (1) generating new random game instances, (2) prompting language models with game rules and context, (3) handling messages and formal proposal actions, (4) computing player rewards, and (5) sending comprehensive error messages to models in case they violate the game rules, e.g., by sending incorrectly formatted proposals. We provide full details in Appendix A and in our open-source code release.

## 3 Language Model Self-Play

We begin by evaluating pretrained language models, prompted only with task instructions and the current game context, as detailed in Appendix A.
Because our goal is to analyze the effects of self-play on language model behavior, we do not prompt our models with few-shot example dialogues or finetune them on task-specific data; doing so helps us avoid biasing models toward specific patterns of behavior.
We then iteratively finetune these models over many rounds of self-play.

We implement a straightforward algorithm for language model self-play based on filtered behavior cloning (filtered BC; Chen et al., , ; Zelikman et al., ). In this setting, two language models with identical parameters but different prompts play $K$ games and receive rewards according to their (identical) objectives. Each game produces two dialogue histories, one from each player’s perspective. The average score across games is computed and dialogues with above-average scores are kept and used for finetuning the model. This procedure is repeated for $N$ iterations or until early stopping. We set $K=500$ and $N=10$ for the majority of our experiments.

Figure: Algorithm 1 Language Model Self-Play

Because this approach filters out games which receive a below-average score, it requires a sufficiently capable model to “get off the ground.” In our preliminary experiments with open-weight models such as Mixtral 8x7B and LLaMA-2-70B, we found that they did not achieve enough nonzero scores to improve from the first round of self-play. In particular, weaker models struggle to follow the game instructions and routinely fail to produce compatible proposals. Therefore, we instead used GPT-3.5
(gpt-3.5-turbo-0125; Chen et al., ; Ouyang et al., ) which strikes a balance between capability and accessibility. While not an open-weight model, GPT-3.5 can still be finetuned through the OpenAI API, and it achieves sufficiently high initial scores to kickstart training. Due to the high cost of experiments, we leave investigation of other models to future work.

Figure: Figure 2: Language model self-play significantly increased model performance in both cooperative and semi-competitive games. Moreover, these results generalized to collaboration and competition with humans, leading to improvements of up to 2.5$\times$ and 6$\times$ the baseline scores, respectively. We found that human-LM baseline scores were higher in the collaborative setting as humans can help “guide” models to avoid common failure modes.
Refer to caption: x2.png

## 4 Human Experiments

To evaluate whether model improvements generalize beyond self-play, we built a web interface which allows humans to play DoND against our trained models. We evaluated models on both the cooperative and semi-competitive objectives across the timecourse of training. However, due to the large number of models we trained and the cost of human experiments, we only ran human evaluation on every other iteration of self-play finetuning.

#### Crowdsourcing

We ran human evaluation on Amazon Mechanical Turk. After a prescreening survey and three pilot studies, we identified a group of 60 reputable English-speaking workers and invited them to participate in our task. In order to incentivize high-quality dialogue, we primarily compensated workers with bonus pay: each worker earned $1.00 for picking up the HIT, $0.10 for each game played, and $0.20 for each point earned. In total, we collected 1,175 human-LM dialogues, with the average worker receiving pay of $37.50. More details on crowdsourcing, including screenshots of our web interface, can be found in Appendix C.

## 5 Results

### 5.1 Baseline Models

We conducted the majority of our experiments on GPT-3.5. Prior to self-play finetuning, GPT-3.5 performed relatively poorly, obtaining mean scores of 0.4 in the original, semi-competitive setting and 0.7 in the cooperative setting.
These scores are much lower than those obtained in prior work (Lewis et al., ; Gandhi et al., ) because we did not provide models with few-shot example dialogues or task-specific finetuning data.

Qualitatively, GPT-3.5’s low scores can primarily be attributed to its inability to consistently reach complementary proposals with itself in self-play, reaching a valid agreement with its partner in only 6.8% of games across both objectives. Additionally, this baseline model relies heavily on error-handling from the environment to send messages properly and fails to remain grounded in the game’s context throughout an entire dialogue, often hallucinating new items, changes in its value function, or both.

In collaboration with humans, GPT-3.5 obtained a much higher average score of 4.6. While errors tend to compound in self-play, we observed that humans can help “guide” models to avoid common failure modes in the cooperative setting, resulting in higher scores (e.g., by suggesting which objects to propose). However, similar improvements did not occur in the semi-competitive setting, where humans are less incentivized to help models perform well; in the semi-competitive setting, the baseline GPT-3.5 model achieved a mean score of 0.8.

**Table 1: Mean scores of models initially finetuned on task-specific data, which was either generated using GPT-4 self-play or extracted from prior human experiments in Lewis et al. (). We generated finetuning data for both the cooperative and semi-competitive objectives with GPT-4, whereas pre-existing human data was only available for the semi-competitive setting. After training, models were evaluated both in self-play and via human experiments. While model scores improved in the earliest iterations of self-play finetuning, performance plateaued and even declined much earlier than in the experiments without initial training on task-specific data.**
|  | GPT-4 (Cooperative) | GPT-4 (Semi-Competitive) | Human (Semi-Competitive) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | Self-Play | Human Eval | Self-Play | Human Eval | Self-Play | Human Eval |
| GPT-3 | 0.7 | 4.7 | 0.4 | 0.7 | 0.4 | 0.7 |
| Finetuned | 11.3 | 11.0 | 5.5 | 4.2 | 5.3 | 4.8 |
| Iteration 1 | 11.0 | 11.7 | 5.5 | 4.7 | 6.0 | 5.3 |
| Iteration 2 | 11.4 | 11.7 | 5.8 | 5.8 | 6.2 | 5.7 |
| Iteration 3 | 10.8 | 10.5 | 5.6 | 3.6 | 6.1 | 5.5 |

To establish a stronger baseline, we also evaluated GPT-4 using the same prompts and game environments as in the rest of our experiments, over 300 games of self-play.
GPT-4 achieved a mean score of 4.3 in the semi-competitive setting and 8.8 in the cooperative setting.
Qualitatively, this model was able to follow the game instructions and negotiate much more competently than GPT-3.5.
Although we use GPT-4 as a reference point for model performance, we did not conduct further experiments on it due to the lack of public finetuning access at the time of experimentation.

### 5.2 Self-Play Finetuned Models

Despite weak performance of the initial models, language model self-play was highly effective, as shown in Figure 2. When evaluated against another copy of the same model, self-play finetuning increased scores by as much as 14$\times$ in the semi-competitive setting (0.4 $\to$ 5.8) and 17$\times$ in the cooperative setting (0.7 $\to$ 12.1). Although these experiments were conducted on GPT-3.5, these scores are significantly higher than those of a baseline GPT-4 model, as reported in Section 5.1.

Improvements from self-play generalized to collaboration and competition with humans as well, with scores increasing by 6$\times$ (0.8 $\to$ 4.9) in the semi-competitive setting and 2$\times$ in the cooperative setting (4.6 $\to$ 8.7). In the cooperative setting, human-LM scores peaked at 11.6 but began to decline before the 10th iteration of self-play.
We do not report scores for any model after the 10th iteration, as they tended to stabilize or even decline.

#### The Case of Strict Competition

For the strictly competitive setting, due to the zero-sum nature of the game, it is not informative to report mean scores in self-play (since they would always be zero).
We instead evaluated the quality of trained models based on how well they performed against a separate model, GPT-4.
Additionally, due to a sparsity of positive-scoring games, we modified the filtering criteria for strictly competitive self-play to also include include samples from zero-scoring games in which a valid agreement was reached. While models improved at reaching valid agreements in self-play, we found they generalized poorly against other agents. Our preliminary results indicated that even the best-performing models for this objective would routinely fail to reach agreements with humans, so we instead ran 100 games between each iteration’s model and GPT-4, confirming that the model failed to improve outside of self-play.
We report results for this objective, along with further implications, extensively in Appendix D.

### 5.3 Comparing the Effect of Self-Play to Task-Specific Finetuning Data

We also considered the case where our initial model was finetuned on an externally provided corpus of task-specific data. For the semi-competitive objective, we finetuned models on 300 nonzero scoring games from the original human-human dataset in Lewis et al. ().
However, because task-specific data only exists for the original task formulation, we used GPT-4 to generate a comparable amount of finetuning data for the cooperative objective; to enable fair comparison, we also used GPT-4 to generate finetuning data for the semi-competitive objective.
We finetuned GPT-3.5 on the nonzero scoring games for each of these three settings and then repeated the self-play algorithm from Algorithm 1, providing a finetuned GPT-3.5 model as input instead of the baseline GPT-3.5 model.

Table 1 shows that finetuning on task-specific data, either provided by humans or distilled from a stronger model, can lead to performance improvements comparable to those obtained from language model self-play.
However, the benefits of finetuning and self-play training can be combined. Running language model self-play on the finetuned models still improves model performance, but only for 1-2 iterations, after which scores begins to decline. This holds across both self-play and human experiments; moreover, we find that score improvements in self-play generally correlate with score improvements in collaboration or competition with humans.

## 6 Analysis

### 6.1 Agreements and Pareto Optimality

In addition to scores increasing with each iteration of self-play finetuning, models also improved in their ability to achieve valid agreements (Table 2) and Pareto-optimal game scores (Table 3), both in self-play and in human experiments. Trained models achieved almost perfect agreement rates in self-play, with 96.4% of games ending in an agreement in the semi-competitive setting. While the agreement rates are lower in collaboration with humans, this can partially (but not wholly) be attributed to human error. For example, we observed that humans sometimes failed to read overly long messages in full, resulting in failed proposals.

To understand the relationship between agreement rate and game score, we calculated Pearson correlation coefficients between completed iterations of self-play and scores achieved, before and after filtering out samples that failed to reach a valid agreement. For our self-play data under the semi-competitive objective, this yielded $\rho=0.44$ before filtering and $\rho=0.34$ after. On human-LM data with the same objective, however, the correlation drops from $\rho=0.29$ to $-0.04$ after filtering out non-scoring games. These results suggest that improvements in model performance come from an increase in agreements, which can primarily be attributed to better understanding of task instructions, following the environment rules, and not hallucinating items or proposals. Although self-play scores appear to rise after filtering out games which fail to reach a proposal, this improvement is a ruse: these improvements do not generalize to humans.

As a result, even though human-LM scores improved substantially after self-play finetuning, there is still a moderate amount of headroom on this task. More evidence for this headroom comes from Table 3, which reports the percentage of games that ended in Pareto-optimal outcomes, i.e., agreements where it would be impossible for one player to increase their score without the other player’s score decreasing. Despite improvements from self-play, models still did not obtain the maximum score in the majority of games. We expect techniques other than filtered BC may necessary to close this gap.

**Table 2: Agreement rates (%) before and after ten rounds of self-play finetuning. Models came to more valid agreements across objectives in both self-play and human generalization. We note that the relatively lower agreement rate for human performance in the cooperative setting can be attributed to performance beginning to decline before the last iteration, as shown in Figure 2.**
|  | Semi-Competitive | Cooperative |  |  |
| --- | --- | --- | --- | --- |
|  | Self-Play | Human | Self-Play | Human |
| Before | 6.8 | 13.3 | 6.8 | 32.7 |
| After | 96.4 | 76.5 | 91.0 | 64.0 |

**Table 3: Pareto-optimality rates (%) before and after ten rounds of self-play finetuning. Although model scores improved substantially, there is still significant headroom; models still leave many points on the table, especially in collaboration with humans.**
|  | Semi-Competitive | Cooperative |  |  |
| --- | --- | --- | --- | --- |
|  | Self-Play | Human | Self-Play | Human |
| Before | 2.2 | 8.9 | 5.8 | 16.3 |
| After | 46.0 | 49.0 | 89.6 | 38.0 |

Figure: Figure 3: Mean dialogue lengths per game for every iteration of the model in semi-competitive and cooperative objectives. Dialogues under the semi-competitive objective progressively shrank in length, while dialogues under the cooperative objective grew significantly.
Refer to caption: x3.png

Figure: Figure 4: Aggregate vocabulary sizes across each model iteration’s 500 games of self-play, for both semi-competitive and cooperative objectives. In the semi-competitive setting, vocabulary size trended downward, but the model maintained and even expanded its vocabulary when trained with the cooperative objective.
Refer to caption: x4.png

### 6.2 Dialogue Length and Diversity

We qualitatively observed that dialogues in the semi-competitive setting became less diverse over the course of self-play finetuning. We quantified this in two ways: (1) by average dialogue length, in Figure 3, and (2) by the number of unique words produced during each iteration, in Figure 4.

We first computed the average dialogue length over 500 hundred games of self-play during each iteration of model training in the semi-competitive and cooperative settings.
We found that self-play caused dialogues to become substantially longer in the cooperative setting but shorter in the semi-cooperative one.
We hypothesize that this discrepancy may occur because agents in the cooperative setting are incentivized to share all information; qualitatively, we often observed models sharing exact details of their private value functions. Additionally, we found that models which argue with each other for too long are more likely to “go off the rails” and fail to reach an agreement at all; because these games receive scores of zero, this behavior may be filtered out over the course of self-play under the semi-competitive objective.

We also computed the “vocabulary size” of each iteration by counting the number of unique tokens produced during 500 games of self-play. We observed a similar trend of decreasing vocabulary size over the course of self-play in the semi-competitive setting, supporting the hypothesis that the semi-competitive objective leads to convergence in model behavior. However, as shown in Section 5.2, these models performed well in both self-play and in human generalization experiments, suggesting that they may not need very diverse communication strategies to achieve high scores.

## 7 Related Work

#### Grounded Dialogue

Much prior work on goal-oriented dialogue has focused on collaborative settings.
In tasks such as Cards (Djalali et al., ; Potts, ), CerealBar (Suhr et al., ), OneCommon (Udagawa and Aizawa, ), and DialOp (Lin et al., ), two agents must collaborate via natural language dialogue to achieve a shared goal within an environment. In many of these tasks, models are automatically evaluated via self-play, which serves as a proxy for human evaluation (Fried et al., ; Lin et al., ).

Relatively less work has focused on the case where agents have conflicting goals. A handful of grounded dialogue tasks are focused on bartering or negotiation, including Deal or No Deal (DoND; Lewis et al., ), CaSiNo (Chawla et al., ), and the fruit trading game from Gemp et al. (). These games are all structurally similar and differ primarily in the number and types of objects they use, as well as the public availability of human data.
In the Craigslist Bargaining task (He et al., ), agents negotiate on the price of a object for sale.

Fried et al. () provides additional discussion of collaborative and competitive grounded dialogue tasks and modeling approaches.

#### Self-Improving Language Models

Lewis et al. () trained GRU-based language models on the Deal or No Deal task using REINFORCE (Williams, ). In contrast to our work, Lewis et al. () did not learn a model tabula rasa but instead interleaved reinforcement learning from self-play with supervised learning on task-specific data to avoid divergence from human-interpretable language. Divergence issues abound in other settings where models are trained via self-play, including in emergent communication (Kottur et al., ; Lowe et al., ; Tomlin and Pavlick, ).

A wave of recent work has focused on methods for autonomously improving large language models at training (Ouyang et al., ; Bai et al., ; Abdulhai et al., ) or inference (Shinn et al., ; Yao et al., ; Wu et al., ) time.
Chen et al. () propose a method called self-play fine-tuning, but their usage of the term self-play differs from the traditional meaning, i.e., it does not involve agents interacting within an environment; instead, Chen et al. () proposes a preference learning method akin to DPO (Rafailov et al., ) or PPO for reinforcement learning from human feedback (Ouyang et al., ).
More closely related to our work is Pan et al. (), which iteratively trains models for device-control tasks using filtered behavior cloning; however, in contrast to our work, Pan et al. () studies a single agent interacting with an environment, rather than multiple agents interacting with one another.

Another closely related work is Fu et al. (), which uses self-play to refine language models for a bargaining task. In contrast to our work, the bargaining task is a relatively simple, zero-sum game which elicits less rich dialogues than DoND. Further, while our work focuses on finetuning models, Fu et al. () present models with in-context demonstrations of previous games and natural language feedback from a critic model (similar to Reflexion (Shinn et al., )), leading to less major performance improvements.

#### Multi-Agent Reinforcement Learning

Training agents against copies of themselves is a longstanding technique in reinforcement learning (Littman, ), popularized in the past decade by models like AlphaGo (Silver et al., ) and AlphaZero (Silver et al., ). Experiments on games such as Overcooked (Carroll et al., ) and Hanabi (Bard et al., ) have shown that policies learned via self-play often fail to generalize to collaborative or imperfect information games. Methods such as fictitious self-play and population play have been proposed to address these issues (Heinrich et al., ; Strouse et al., ), but have primarily been applied in games without language components. One notable exception to this is CICERO, which trained models for the game of Diplomacy via self-play using a KL-regularization objective which prevented language from drifting too far from human-written training data (FAIR, ).

## 8 Conclusion & Discussion

Our experiments showed that language model self-play can lead to significant performance improvements in both semi-competitive and cooperative games.
This finding contradicts existing wisdom that self-play is ineffective in collaborative domains (Strouse et al., ), or that models need to be trained on task-specific human data to avoid divergence from human-interpretable language (Lewis et al., ; FAIR, ). One hypothesis is that because we observed significant model improvements after just ten rounds of self-play, the model may not have had time to overfit to cooperation with itself. Another hypothesis is that better language models might simply be more robust to the negative effects of self-play. Given a model with good generalization abilities, finetuning on self-play games might be able to elicit model capabilities which are not directly present in the self-play data. Furthermore, self-play with pretrained language models might actually function more similarly to population play, since large language models are trained on text from a population of users and may simulate different personas in different contexts (Pataranutaporn et al., ; Park et al., ).

Although game scores increased significantly after self-play, this increase can be almost entirely attributed to an increase in the percentage of completed games, rather than better high-level strategies or negotiation tactics. Future work may be able to obtain even larger improvements by combining self-play with approaches other than filtered BC, such as natural language reflections, e.g., akin to Shinn et al. (). Another possible approach is described by Srivastava et al. (), in which a language model is used to describe distributional differences between good and bad trajectories.

Finally, the effectiveness of methods like self-play is completely dependent on a reward signal, which in this work was obtained from the game environment. To apply similar methods not just in game-playing domains but in real-world scenarios, we anticipate that models will need to rely on feedback from general-purpose, learned reward models (e.g., as in Du et al., ; Pan et al., ). We leave further investigation of the challenges associated with bringing self-play into real-world application domains to future work.

## Acknowledgments

We are grateful to Daniel Fried, Justin Chiu, and Jessy Lin for early discussions which contributed to the idea for this work. We also thank Lucy Li, Jiayi Pan, and Rodolfo Corona for their feedback. NT is supported by the DARPA SemaFor program.

## Appendix A Environment Implementation

We implemented an environment for the task under which language models can play the game with each other or against another human.
For initializing a new instance of the game, we sample from a list of 4,186 valid game contexts (shared item counts and private value functions for each player), provided by Lewis et al. ().
The environment then takes turns prompting each player to either send a message (prepended by [message]) or submit a proposal (prepended by [propose]). After detecting a submitted proposal, the environment forces the other player to submit a proposal of their own. This is enforced by error correction, described in the section below.
Once a game is completed, the environment uses the game context and the submitted proposals to determine the final score of each player, conditioned on the objective under which the players were instructed to play.

#### Error Correction

This environment comes with comprehensive error-handling to correct models’ errant outputs. Specifically, the environment will reply with instructions for correcting errors when errors are detected, providing the model with the opportunity to format its output correctly. The errors that we check for, and their corresponding correction messages, can be found in Table 4. If a model generates five errant outputs in a row, then the environment aborts the game, and both players receive zero score.

**Table 4: Our game environment sends error messages to language models if they produce ill-formed outputs, e.g., sending a message after the discussion phase ends. The model then has an opportunity to send a new message based on the correction. If the model repeatedly fails to produce well-formed outputs, then the game aborts and both players receive zero score.**
| Error | Correction Prompt |
| --- | --- |
| Outputting text without a prefix of either "[message]" or "[propose]" | Your output should either begin with [message] or a [propose]. |
| Submitting proposals before any messages have been sent | Please begin the dialogue by discussing how you’ll divide the items before submitting a private proposal. |
| Sending messages with multiple mentions of "[message]" or "[propose]" (i.e. outputting multiple messages in a row) | Do not include any mentions of [message] or [propose] after the initial prefix. Please just send a single message, beginning with [message]. |
| Sending messages after a proposal has been submitted | Opponent’s proposal must be followed by a proposal of your own. Please send a proposal, beginning with [propose]. |
| Submitting proposals with incorrectly sequenced items | Item counts must be sequenced in the following order: books, hats, and then balls. |
| Submitting proposals with more than three item counts | There should only be counts for three items in your proposal: books, hats, and balls. |
| Submitting proposals with invalid item counts, based on game context | Item counts suggested are invalid based on game context; some of your proposal’s item counts are greater than total items available. |

#### Zero-Shot Prompting

Across every setting, the initial models are zero-shot prompted with the game’s rules and the instructions for sending messages and submitting proposals with the correct syntax. The choice of this approach over few-shot prompting was motivated by the concern that few-shot examples might influence strategies chosen by the model during inference. Our preliminary experiments found that models would closely match the negotiation techniques used in few-shot examples; for example, if prompted with dialogues where players shared their exact value functions, the model would consistently share its own values, whether doing so was advantageous or not.

#### Prompting with Conversation History

Following the format recommended by the OpenAI API’s chat completions endpoint, the prompt containing game instructions is sent under the system role; for subsequent dialogue, any messages sent by the model itself are categorized with the assistant role, and messages from the other player are appended to a model’s input as messages from the user role. The system prompt used for the semi-competitive objective can be found in Figure 5; other prompts are available in our code release.

Figure: Figure 5: System prompt used for the semi-competitive objective. Values in {brackets} are filled in based on the game context (i.e., item counts and private value functions).

## Appendix B Model Training and Hyperparameters

All models were finetuned using the OpenAI API, with parameters n_epochs=3, batch_size=1, and learning_rate_multiplier=8.
For model inference, we generated outputs with temperature=1. These parameters were all default values chosen by the OpenAI API, except for the learning rate multiplier, which defaults to 2.
Our preliminary experiments with default learning rate multipliers yielded models that not only failed to improve but also devolved significantly in quality.
We hypothesize that this occurred because parameters are set dynamically based on the quantity of finetuning data. Because language model self-play requires sequential rounds of finetuning, it may be important to choose initial parameters based on the expectation of future finetuning rounds.

## Appendix C Details of Human Experiments

### C.1 Game Interface

We developed a web interface for human data collection, shown in Figure 9, Figure 10, and Figure 11. The interface provides comprehensive instructions describing the different game modes, as well as an explanation of the bonus pay structure and a running count of bonus pay earned so far. At the end of each game, players see a popup with the number of points and bonus pay earned. If players receive no points (e.g., due to game error or non-compatible proposals), they receive a small amount of bonus pay and an explanation of what went wrong. During the main phase of data collection, players were allowed to complete up to 40 games; after each game, players were given the option to end the HIT and collect bonus play or keep playing.

### C.2 Crowdsourcing

We ran human evaluation through Amazon Mechanical Turk (MTurk). We restricted our task to workers from the United States with a 98+% HIT approval rate and at least 500 completed HITs, based on recommendations in Huang et al. (). In order to filter out bots and low-quality workers, we ran a brief prescreening survey which asked workers to (1) to answer a question about text on a linked, external website and (2) write a 2-3 sentence description of their favorite MTurk task. The authors then manually reviewed responses to the prescreening survey and chose approximately 15% to invite to the main task.

We ran three pilot studies before launching our main human evaluation, with 10 workers each. After the initial pilots, we modified the interface and incentive structure to obtain higher-quality dialogues. We reviewed data from each pilot and removed low-quality workers and spammers from later rounds of data collection. In total, we invited 60 workers to our final round of data collection, although a small number of workers declined the HIT. We include data from the third pilot in our results because we did not modify the game after that point. Figure 6 provides additional statistics on the total number of workers hired.

### C.3 Incentive Structure

We paid $1.00 for picking up the HIT and $0.10 per game completed. The majority of pay was distributed through bonuses. We paid a bonus of $0.20 per point earned in the semi-competitive setting and $0.10 per point earned in the cooperative setting, since scores in the cooperative game are on average twice as high. We also paid workers $0.25 in cases where models aborted due to repeatedly generating ill-formed messages.

In contrast, Lewis et al. () paid workers $0.10 per game and $0.05 in bonus pay only when workers achieved the maximum score of ten points. We found that this approach incentivized workers to end the game as quickly as possible, as maximizing the number of games played was more lucrative than attempting to achieve a high score.

Figure: Figure 6: The majority of our workers played the maximum number of games, with a small handful contributing data from both the pilot and the main study. The mean pay for workers was $35.70.
Refer to caption: x5.png

## Appendix D Results for Strict Competition

When we applied LM self-play to Deal or No Deal under the strictly competitive objective, the model failed to improve its performance, instead learning strategies that adversely impacted its ability to perform outside of self-play. Since the mean score in self-play for every iteration will always be zero (because the game is zero-sum), we instead evaluated the quality of model self-play through agreement rates. As shown in Figure 7 and Figure 8, while agreement rate trends show that the model’s performance in self-play improves, these models fail to generalize to competition with other models, such as GPT-4. Our preliminary human experiments also showed that the model failed to reach agreements in roughly 95% of games.

In our qualitative analysis of successive iterations of the model under the strictly competitive objective, we found that it learned to replicate an inverted proposal strategy; specifically, the model learned a strategy where it submits a proposal based on what the opposing player should receive, rather than what the model itself should receive. While the model optimized this strategy in self-play well enough to arrive at valid agreements at a competitive rate with itself, this strategy does not generalize to competition with humans or GPT-4. We found this to be a consequence of the aggressive nature of the strictly competitive objective leading to a smaller proportion of games ending in valid agreements to derive reward signal from. With a smaller number of samples providing reward signal, we risk an outcome where the few samples that are isolated for finetuning achieve their non-zero reward through undesirable strategies (inverted proposals, in this instance) that do not generalize well to human interaction.

Our experiments under this objective illustrate an important takeaway regarding the failure modes of LM self-play. For this strategy to be effective, there must be confidence that the initial model is capable of achieving high performance through desired strategies with significant probability. Additionally, we speculate that this strategy is most effective in environments with continuous reward.

Figure: Figure 7: We evaluated the strictly competitive model against GPT-4, since self-play scores are not informative for zero-sum games. However, we determined that the model experienced no significant improvement in generalization.
Refer to caption: x6.png

Figure: Figure 8: Under the strictly competitive objective, self-play finetuning increased the frequency of games reaching valid agreements in self-play, but the strategies learned generalized poorly to interaction with other agents, such as GPT-4. Our preliminary experiments also indicated a low agreement rate with human competitors.
Refer to caption: x7.png

Figure: Figure 9: The landing page for our human data collection site provides comprehensive game rules and instructions and an explanation of the bonus pay structure.
Refer to caption: extracted/5694670/assets/landing.png

Figure: Figure 10: The game interface for human data collection shows the shared item pool, game mode, item values, and a chat window, as well as the number of games played so far and a running count of bonus pay earned.
Refer to caption: extracted/5694670/assets/interface.png

Figure: Figure 11: At the end of each game, players see a popup with the number of points and bonus pay earned. Players have the option to end the game and collect their bonus pay or keep playing, up to the maximum of 40 games.
Refer to caption: extracted/5694670/assets/game_over.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: **major**

- why do errors occur in 22% of cooperative games but only 1% of semi-competitive? (section 6.1)

- how can you be sure that human-AI performance is not bolstered by humans compensating for your AI playing badly?

**minor**

- why do you talk about fully competitive games in both the intro and game objectives when communication is not useful in 2 player fully competitive games (Crawford and Sobel, 1982)

- Can you change the words "finetuning" in Section 5.3? It is very confusing to differentiate between the "warm-started" model and the act of iterative finetuning towards the task.

- You claim that Humans are to blame for not reading overly long messages but it could be that the blame is with the model for learning to generating long output: this is actually just alignment tax / language drift. As shown in Figure 3 left, is your model drifting away from human-preferred language towards longer useless text when being trained in self-play?

- Why do you include lying about an item preference value in your errors for 6.2? Lewis et al argued that this lying could be advantageous / effective deception.

**references**

- Your method of iterative finetuning with interactive LLMs with filtering is quite close to [Rest-EM](https://arxiv.org/abs/2312.06585), so this probably deserves a citation.

- The negotiation game of Lewis et al (2017) itself based on a negotiation game of [Devault et al (2017)](https://aaai.org/papers/10335-10335-toward-natural-turn-taking-in-a-virtual-human-negotiation-agent/)

- The citation and explanation of how SPIN (Chen et al, 2024) is *not related* feels superfluous and strange. Just don't have it in related work.

- KL is always used to prevent language from drifting too far, CICERO is just a simple case of this. You can cite [Jaques et al (2019)](http://arxiv.org/abs/1907.00456) for this if you'd like. Also, if I'm not mistaken, the language component of CICERO was trained a bit separately from the game-playing component which is the real way it didn't drift too far from initialization. 

- Why are you citing Chen et al (2021) for GPT-3.5 in Section 3? I think this is a mistake
soundness: 2
strengths: I really enjoy the research direction and perspective of the authors. Agentic LLms are becoming more popular and their interactions in non-cooperative environments are understudied, so this work has a novel and interesting approach.

The authors do well to demonstrate where their self-play is effective, especially the result that it learns to reduce hallucinations. They also provide interesting analysis of human-AI play and different ways in which training affects results. There are a lot of interesting insights to be gained from this paper and I give the authors some slack for attempting something quite novel and difficult (LLM self-play). It is clear that there are many challenges in this direction (LLaMA-70B was not good enough at initialization!) so all reasonably positive results are encouraging.
summary: This work proposes to study how LLMs learn in competitive games. They choose to investigate using mostly GPT-3.5 and the game Deal or No Deal (DoND) where two players negotiate to split a set of items.

They find that GPT-3.5 is unable to play the game using just prompting but through iterations of supervised finetuning on successful self-play data, it learns to improve its play. Improvement are seen not just in self-play but also when evaluated playing with humans. Warm-starting from a GPT-3.5 model finetuned on task-specific data does even better.

Analyzing the improvements, models are largely just learning the rules and making valid proposals, not strategically negotiating. Still, iterative self-play does reduce the rate of hallucinating wrong preferences and inconsistent. As expected, cooperative play increases the length and diversity of messages (as more information can always be used) while semi-competitive play reduces both (perhaps to reduce exploitability).
weaknesses: The major weaknesses of this paper is the lack of explicit understanding of their own game dynamics, which are crucial for understanding expected behaviour, baselines on performance, and conclusions that can be drawn. This is likely because of missing insights from previous, non-LLM work. The overall story of the paper is also muddled and so it is unclear what are the main insights and whether the experiments support the conclusions. I believe this is fixable in the review period and give my recommendations for what the authors can improve.

Overall, I think the authors should formalize the competitiveness of the game, explain why they study competitiveness since their results seem negative, and give a more full, coherent story. If the authors answer one of the three major weaknesses here I will increase my score and if they answer all three I will increase my score again.

### How competitive is your game?

First and foremost, the authors claim to work on semi-competitive games but never actually measure the competitiveness of their game! This is crucial, as proven theoretically in the famous work by [Crawford and Sobel (1982)](https://www.jstor.org/stable/1913390), the level of achievable communication is inversely proportional to the level of competition in the game. Communication is not useful in a fully competitive game. This was further shown to be true in neural networks learning to communicate (emergent communcation) in semi-competitive games by [Noukhovitch et al (2021)](https://arxiv.org/abs/2101.10276) which also looks at an extremely similar negotiation game [(Cao et al, 2018)](http://arxiv.org/abs/1804.03980)

The competitiveness of DoND depends on the specific items and preferences each player has over the items. A game with $\lambda=1$  can be highly competitive if both players have similar values for the same items, highly cooperative if they value different items, or a mix of the two. Without knowing this, it is impossible to say whether a score of 5 is good or not. At minimum, the authors should calculate the average competitiveness of the game and have a line in Figure 2 for the average pareto-optimal score in the semi-competitive game.

You show "Pareto-Optimality" in Table 2 but this term is never defined or the value formally stated. I believe this to be the score for each player if the items were to be divided such that players had as high and as similar possible scores (i.e. a "fair" pareto optimal). This should be formalized.

As well, the that your game is "cooperative, competitive, or anything in between" in your abstract is not necessarily true. You should show this as done in Noukhovitch et al. 

**There should be no communication in a fully competitive game**

As proven by Crawford and Sobel, and shown empirically by Noukhovitch et al, a fully competitive game should have no communication. Indeed the best action for your DoND agents with $\lambda = -1$ is to not communicate and fail to agree to any deal as that will allow them to get score 0.

Your experiments trying to achieve valid agreements makes no sense as it is explicitly advantageous to *not* reach agreements. As argued by Noukhovitch et al, when there is no incentive to communicate, agents will not communicate.

**What is optimal play?**

To figure out if your models are actually learning the rules of the semi-competitive game you should outline what optimal play really is. Noukhovitch et al (2021) argues that the negotiation game is an ultimatum game which would give first-mover advantage and mean the first player should win most of the points. The rounds of dialogue in-between are cheap talk that shouldn't affect this game-theoretic optimal play, but I am open to other interpretations.

### What's the point of semi-competitive and self play?

If the actual gains in your setting come about from learning to play the game and follow the rules (Section 6.1, 6.2) then what is the point of competition in your setting? Why does it matter if the game is competitive if all you're doing is learning to play?

Further, [Ross et al (2024)](https://arxiv.org/abs/2408.02784v1) show that LLMs are generally altruistic. Based on this analysis, they should not be playing the competitive game effectively (at least at initialization) as they would seek to maximize fairness for their opponent. This suggests to me that your competitive scenario is not even maximizing the competitive objective and your results seems to suggest this too.

You should make a strong argument for whether self play and competitive games are a negative result and why it matters. This is especially important since GPT-4 seems to play the game reasonably well at initialization so it is unclear why you're training and not creating a better prompt.

### What is the story and what is actually being learned

The paper feels a lot like a splattering of results without a very coherent backbone. This is fixable and I am also willing to change my view.

Currently the story reads as 
- Here's a challenge for LLMs: DoND and a solution: iterative finetuning with LLM self-play data
- Here's how it improves over a very basic prompt baseline
- Here's an analysis that shows that it's really just learning the basic rules of the game
- Here's a slight difference between cooperative and competitive play

This is a bit underwhelming and entirely hinges on the fact that your deal or no deal game is difficult to play using prompts alone. I am not convinced of this. I am especially not convinced that a better prompt could solve the cooperative game with GPT-3.5. 

I think you need to reformat this to make it clear how difficult DoND is or, more likely, to argue that perfect prompting of LLMs is difficult and your self-play method is an easier approach to getting complex agentic behaviour. One method (just as an example) would be to actually find the perfect cooperative prompt and show how it fails in the competitive case, demonstrating how much harder it would be to transfer the prompts.

For cooperative play, it is clear you are learning the rules of the game and cooperation. But what is being learned in the semi-competitive game? The competitive game includes the ability to exploit other agents, so are you really learning the game if you're not learning to exploit other agents? This is an interesting negative result that contributes to Ross et al (2024) if you're showing that even training on a competitive game does not easily achieve a competitive but altruistic strategy, but this should be clear.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: - Can you further elaborate why the performance in iteration 3 is worse in Table 1? Do you have any examples showcasing the overfitting?
- What do you think is the reason why open source LLMs cannot improve their performance with your method?
- How long does it take to finetune one round/iteration?
soundness: 4
strengths: - A key innovation of this work is the self-play training on large language models, which I believe is a novel approach. While self-play has been successfully used in other domains with smaller models, applying it to LLMs in a negotiation game is interesting. This novel approach expands the scope of self-play, making it as a promising technique for training LLMs in complex, dialogue-based tasks.
- The experimental results demonstrate substantial improvements, with the performance of the models increasing multiple times through finetuning.
- The modified DoND game environment in this paper is highly engaging for evaluation across different interaction types, including cooperative and competitive objectives.
- The authors plan to make their dataset and code publicly available, which may encourage further research in the field.
summary: This paper explores finetuning LLMs with self-play data in the negotiation game "Deal or No Deal" (DoND). The authors implement a filtered behavior cloning on LLMs and achieve promising results. They also build a web interface which allows humans to play DoND against their trained models. The experiments demonstrates that their method enables LLMs to achieve substantial performance gains across various game objectives—ranging from cooperative to competitive settings. The study also highlights improvements in task adherence and reduced hallucination rates, suggesting the models became more aligned with game rules over time.
weaknesses: - The study exclusively focuses on language models and does not incorporate any reinforcement learning (RL) baselines for comparison. Including RL-based models could have provided a broader benchmark, highlighting the unique contributions of self-play for LLMs while also revealing potential strengths or weaknesses relative to established RL techniques.
- The self-play data generation and subsequent finetuning require substantial computational resources, resulting in high training costs. The authors mentioned themselves in the paper that they only ran a small scale baseline experiment on GPT-4 due to its high cost.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. What could be the underlying reasons for the observed performance decline in cooperative games involving humans after 8 to 10 rounds? Why is this trend not present in other game settings?
2. What alternative environments/games could be explored to strengthen the study's findings?
3. Would expanding the study to include more competitive and cooperative levels provide a more comprehensive understanding of the impact of cooperation on self-play efficacy?
soundness: 3
strengths: 1. The paper provides extensive quantitative results demonstrating significant performance improvements. And results show models trained with self-play perform better in collaboration with humans.
2. Includes detailed analysis of errors, agreement rates, and Pareto optimality.
3. Provides detailed insights, such analyses of dialogue length and hallucination rates.
4. The paper is clearly presented and easy to follow.
summary: This paper explores the application of self-play training to language models in non-zero-sum settings through experiments in the Deal or No Deal negotiation game. The authors modify DoND to create cooperative, semi-competitive, and strictly competitive game objectives. Empirical results show substantial improvements in model performance across cooperative and semi-competitive settings in self-play and interactions with humans, but limited gains in strictly competitive scenarios. The study also identifies increased instruction adherence and reduced hallucinations as benefits of self-play while noting limitations in strategic behavior learning.
weaknesses: 1. In cooperative games involving humans, performance declines as the number of rounds increases from 8 to 10. Why does this occur? This trend is not seen in other game settings.
2. The experiments and analysis are limited to a single game, which could introduce bias. Evaluations across additional environments are needed for validation.
3. The study only considers cooperative and semi-competitive settings. Including a wider range of competitive and cooperative levels could provide deeper insights into the impact of cooperation on the effectiveness of self-play.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Please see Weaknesses.
soundness: 3
strengths: 1. Good clarity: the manuscript is well-organized and easy to follow. The DoND game and the idea of language model self-play are clearly described.
2. Good analysis and discussion: the authors give a detailed analysis of the effect of self-play in the DoND task. They also discuss the limitations and directions for future improvement.
summary: This work studies whether self-play can improve language models' (LM) performance in cooperative and semi-cooperative games. They use Deal or No Deal (DoND) as the testbed and find self-play based on filtered behavior cloning significantly improves LMs' performance and generalizes to gameplay with humans.
weaknesses: This work studies the question of "whether techniques like self-play can effectively be used to improve language models (in cooperative tasks)". My main concern is that the current manuscript mainly presents preliminary empirical results and lacks solid and sufficient evidence to support a conclusion. Below is the detailed discussion.

1. Limited insights on why and how to make language model self-play work. Self-play has been known to be effective in non-language-based cooperative settings like MPE [1], SMAC [2], GRF [3], etc., so it is expected to work in language-based cooperative settings. Some previous work, as mentioned in the paper, finds that the self-play of language model can diverge from human-interpretable language, while this work finds it leads to performance improvement in DoND. Instead of just presenting these empirical findings, a more important question is to understand why language model self-play works in some tasks but fails in others, and how to make it work in more general tasks. However, this work mainly provides a limited and hypothetical analysis of this question in DoND only. It would be more insightful if the authors could investigate more general settings to provide an in-depth understanding behind the empirical findings.
2. Potential overclaim due to limited task, model, and training. This work investigates only one task (DoND) with one LM (GPT-3.5) and trained for just ten rounds. This is more like a case study and it would require evidence on more tasks with more LMs and training for more rounds to support the authors' claim that "self-play leads to large improvements in both the cooperative and semi-competitive settings" (L047-048). In fact, the authors also mentioned some preliminary results that may negatively impact the current claim.
    1. More models: as mentioned in L190, Mixtral 8x7B and LLaMA-2-70B fail to improve from the first round of self-play. This shows that self-play can lead to no improvement in some LMs. Providing results of Mixtral 8x7B and LLaMA-2-70B can examine the soundness of the claim.
    2. More training: as mentioned in L258-259, the authors "do not report score for any model after the 10th iteration as they to stabilize or even decline". If the performance can decrease with more training rounds, is it possible that, with more self-play rounds, the model will still diverge to uninterpretable language? It would be better if the authors could provide results of more iterations to examine the long-term stability of language model self-play.
    3. More tasks: as mentioned in L466-467, the performance increase in DoND "can be almost entirely attributed to an increase in the percentage of completed games, rather than better strategic reasoning or negotiation tactics". In fact, cooperative DoND may be too easy because the agents can simply share their values for each item and reach the optimal decision. The current results mainly show that self-play can improve LM's understanding of a specific task, instead of reasoning or decision-making ability. It would be more convincing if the authors could provide results in other more complex reasoning and decision-making tasks.

Therefore, I think the current manuscript mainly presents preliminary empirical results and lacks solid and sufficient evidence to support its claim.

[1] Lowe, Ryan, et al. "Multi-agent actor-critic for mixed cooperative-competitive environments." Advances in neural information processing systems 30 (2017).

[2] Samvelyan, Mikayel, et al. "The starcraft multi-agent challenge." arXiv preprint arXiv:1902.04043 (2019).

[3] Kurach, Karol, et al. "Google research football: A novel reinforcement learning environment." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 04. 2020.

Meta-review: This paper explores the application of self-play training to language models in non-zero-sum settings through experiments in the Deal or No Deal negotiation game.

The reviewers agree that the proposed self-play training on large language models is a novel approach. However, the current version of this paper has the following weaknesses.  
- The study lacks evaluations across additional environments or tasks to generalize its findings. Thus, claims of large performance improvements might be overstated given the restricted task, model, and training setup.
- Does not include reinforcement learning (RL) baselines for benchmarking.
- Limited insights into why self-play works in some cases and fails in others.

Decision: Reject