Title: Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback
Venue: NeurIPS 2023
Accepted: no
Keywords: large language models

negotiation game

in-context learning

self-play

AI feedback
Abstract: We study whether multiple large language models (LLMs) can autonomously
improve each other in a negotiation game by playing, reflecting, and criticizing.
We are interested in this question because if LLMs were able to improve each other, it
would imply the possibility of creating strong AI agents with minimal human
intervention. We ask two LLMs to bargain with each other, playing the roles of a
buyer and a seller, respectively. They aim to reach a deal with the buyer targeting
a lower price and the seller a higher one. A third language model, playing the
critic, provides feedback to a player to improve the player’s negotiation strategies.
We let the two agents play multiple rounds, using previous negotiation history
and AI feedback as in-context demonstrations to improve the
model’s negotiation strategy iteratively. We use different LLMs (GPT and Claude)
for different roles and use the deal price as the evaluation metric. Our experiments
reveal multiple intriguing findings: (1) Only a subset of the language models we
consider can self-play and improve the deal price from AI feedback, weaker models
either do not understand the game’s rules or cannot incorporate AI feedback for
further improvement. (2) Models’ abilities to learn from the feedback differ when
playing different roles. For example, it is harder for Claude-instant to improve
as the buyer than as the seller. (3) When unrolling the game to multiple rounds,
stronger agents can consistently improve their performance by meaningfully using
previous experiences and iterative AI feedback, yet have a higher risk of breaking
the deal. We hope our work provides insightful initial explorations of having
models autonomously improve each other with game playing and AI feedback.

Abstract

We study whether multiple large language models (LLMs) can autonomously improve each other in a negotiation game by playing, reflecting, and criticizing. We are interested in this question because if LLMs were able to improve each other, it would imply the possibility of creating strong AI agents with minimal human intervention. We ask two LLMs to bargain with each other, playing the roles of a buyer and a seller, respectively. They aim to reach a deal with the buyer targeting a lower price and the seller a higher one. A third language model, playing the critic, provides feedback to a player to improve the player's negotiation strategies. We let the two agents play multiple rounds, using previous negotiation history and AI feedback as in-context demonstrations to improve the model's negotiation strategy iteratively. We use different LLMs (GPT and Claude) for different roles and use the deal price as the evaluation metric. Our experiments reveal multiple intriguing findings: (1) Only a subset of the language models we consider can self-play and improve the deal price from AI feedback, weaker models either do not understand the game's rules or cannot incorporate AI feedback for further improvement. (2) Models' abilities to learn from the feedback differ when playing different roles. For example, it is harder for Claude-instant to improve as the buyer than as the seller. (3) When unrolling the game to multiple rounds, stronger agents can consistently improve their performance by meaningfully using previous experiences and iterative AI feedback, yet have a higher risk of breaking the deal. We hope our work provides insightful initial explorations of having models autonomously improve each other with game playing and AI feedback.

1 Introduction

We study whether multiple Large Language Models (LLMs) can improve each other in a negotiation game with minimal human intervention, in the fashion of AlphaGo Zero [31] where AI agents improve themselves by continuously playing competitive games under well-defined rules. The answers to this research question have profound implications. On the positive side, if the agents were able to improve autonomously, strong agents might be created with very few human annotations, which greatly saves the cost compared to today's data-hungry LLM training [8, 17]. On the risky side, it also implies strong agents with limited human oversight [4]. In this work, we ask two language models (a seller and a buyer) to bargain about a product. The seller is asked to sell the product at a higher price, while the buyer aims to purchase it at a lower price (Fig. 1A). After reaching a deal, we ask a third language model to play as the critic and give feedback to a player. Then we play the game again, asking the player to improve their strategy using AI feedback provided by the critic LLM.

We choose the bargaining game because it comes with well-defined rules described in text, and a 35 clear and measurable objective (a lower/ higher deal price) for strategic negotiation. Although the 36

A. We ask two agents to play a negotiation game and have an AI critic to give suggestions to one of the player to improve its negotiation strategy. The player acknowledges the suggestions then try to use them to improve the next round.

C.

The abilities of continuously improving from AI feedback in a negotiation game only emerges in models that are strong and

well-aligned. Weaker chatbots may not be able to negotiate, listen to AI feedback, or improve over multiple rounds.

Round 1

. A seller and a buyer bargain

about a product.

Hi, how much is the

balloon?

It's a good balloon and

its $20 …

AI feedback

: a critic reads the dialog history

and  give suggestions for improvements

To sell it to a high price

you may want to ...

Understood. I will

use ... to sell a higher

price

Round 2

. Seller improves bargaining

strategy based on AI feedback

Hi, how much is the

balloon?

It's a royal product

with top-notch

craftsmanship ...

Buyer

Critic

Buyer

…

…

C1

. Chitchat in

casual scenario

C2

. Role playing and

negotiation

. Improve next round

from AI feedback

C4

. Continuously improve

over multiple rounds

Claude-v1.3

GPT-3.5-Turbo

Claude-v1.3

Claude-instant-v1.0

GPT-3.5-Turbo

Claude-v1.3

Claude-instant-v1.0

Jurassic-2-instruct

Cohere-command

GPT-3.5-Turbo

Claude-v1.3

Claude-instant-v1.0

Jurassic-2-instruct

GPT-3.5-Turbo

Employ the "flinch" technique: when

the seller offers a counteroffer, the

buyer should display a degree of

surprise or disappointment

Buyer's Improvement:

Oh! That's higher than I expected. I

saw a similar balloon at another

store for $14. Can you match that

price?

Use the power of silence: The buyer

can employ the power of silence in

the negotiation process by pausing

longer before responding to the

seller's offer.

Buyer's Improvement:

*pause* ...  Alright, I'll take the

balloon for $13.

Utilize split-the-difference: In

situations where a small price

difference remains, propose to split

the difference with the buyer.

Seller's Improvement:

I understand, how about we split the

difference and make it $16.75 to

accommodate your budget?

Use anchoring technique: Begin by

emphasizing the high starting price

and then offer a slightly lower price

Seller's Improvement:

This high-quality, long-lasting balloon

is really worth $25, but I'm offering

it for $20.

Buyer proposes $15, seller calls $18

Context:

B1.

The 'flinch' technique

B2.

The power of silence

B3.

Split-the-di

ff

erence

B4.

The anchoring technique

B.

Example feedback from AI critic and how a GPT-4 player improves from it.

game seems easy at first glance, it requires non-trivial capabilities of the language models, as the 37 model needs to: (1) clearly understand and strictly follow the textual rules of the negotiation game 38 (2) correspond to the textual feedback provided by the critic LM and improve based on it iteratively 39 (see example feedback in Fig 1B); (3) reflect upon the strategy and feedback over the long term 40 and improve over multiple rounds. We will see that not all models we considered show all these 41 abilities (Fig. 1C), and only models that can (1) understand negotiation rules and strategies (capable) 42 and (2) respond to AI instructions (well-aligned) can continuously improve from AI feedback over 43 multiple rounds (in our experiments, only gpt-3.5-turbo , gpt-4 , and claude-v1.3 ) meet these 44 requirements). We have also tried more complicated textual games including board games and textual 45 RPG games in the preliminary experiments, but they are more challenging for current agents to 46 understand and follow the rules. 47

We call our approach In-Context Learning from AI Feedback (ICL-AIF). Specifically, we use the 48

feedback from the AI critic as well as the previous rounds of dialog history as in-context demonstra49

tions [5]. By doing this, the critic's suggestions for improvements and the player's actual improvement 50

in the previous rounds effectively become the few-shot prompts for the next round of negotiation. 51

We use in-context learning for two reasons: (1) tuning large language models with reinforcement 52

learning is prohibitively expensive [25, 15] and the base model [24] may not be accessible to a wide 53

range of the community; (2) in-context learning is recently shown to be closely related to gradient 54

descent [11, 1, 32], such that the conclusions we draw is fairly likely to generalize when one actually 55

finetunes the model (if resources permit). One notable difference between our ICL-AIF and the 56

mainstream Reinforcement Learning from Human Feedback (RLHF) is that in RL the reward is a 57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

Claude-instant-v1.0 v.s. GPT-3.5-Turbo 2023-05-10

Hi, this is a good balloon and its

price is $20

Claude-ins-1.0 | Seller

GPT-3.5-T | Buyer  Would you consider selling it for $10?

Claude-ins-1.0 | Seller  No, that price is too low.

GPT-3.5-T | Buyer  What about $15?

Claude-ins-1.0 | Seller  $15 is better, but still not enough.

GPT-3.5-T | Buyer  What is the lowest price you

can offer?

Claude-ins-1.0 | Seller  My lowest price is $18.

GPT-3.5-T | Buyer  I understand. Could you do $16?

That's the highest I can go.

$16 will work for today.

Congratulations, you got a good

deal!

Claude-ins-1.0 | Seller

Round 1, before AI Feedback

AI Feedback to seller

1. Focus on the qualities and benefits

of the balloon, not just the price.

Highlight how rare and special it is.

Claude-ins-1.0 | Seller

Sure, I will try to improve my

negotiation strategy based on the

feedback from the critic.

Round 2, After AI Feedback

Hi, this is a good balloon and its price is $20

Claude-ins-1.0 | Seller

GPT-3.5-T | Buyer

Would you consider selling it for $10?

No, $10 is much too low. This balloon is

made of high quality latex and

handcrafted by expert artisans. It's

unique one-of-a-kind piece.

GPT-3.5-T | Buyer

What would be the lowest price you

could offer me for this balloon?

Well, because I trust you and value your

business, I will offer you this special

balloon at $17. This price is only

available to valued customers like

yourself today.

Claude-ins-1.0 | Seller

GPT-3.5-T | Buyer

Can you give me a better deal, like $15?

I apologize, but $15 does not even cover

my costs. I have to charge at least $17

for this unique, high-quality balloon.

Claude-ins-1.0 | Seller

GPT-3.5-T | Buyer

Alright, you've got a deal. I will take the

special  balloon for $17.

And I will try to sell it at a higher

price (higher than $16.0) than the

previous round.

3. Be willing to walk away if needed.

Let the buyer know you have other

interested buyers, so they realize the

value in meeting your price.

2. Build rapport and connection with

the buyer to make them feel like they

are getting a good deal from a trusted

seller.

A.

Round 1, before AI feedback to seller, the

bargaining process is relatively simple, final price $16.

B.

AI critic give feedbacks to the seller based on

the previous round, then seller acknowledge.

C.

Round 2, seller improves the reasoning

strategy based on the feedback, final price $17.

scalar [25, 15] while in ICL the feedback is in natural language . We study AI feedback (rather than rely on human intervention after each round) because it is more scalable and can allow models to self-improve automatically.

Our experiments lead to several intriguing findings: (1) The requirements of our bargaining game effectively serve as a testbed for assessing the abilities of LLMs (Fig. 1C): although most models can do chitchat in a casual scenario, as of our experiment date (May 2023), cohere-command [10] model does not understand the rule of bargaining (Fig. 3A), ai21-jurassic [18] model does not respond to AI feedback (Fig. 3B), claude-instant-v1.0 can at most improve one round (Fig. 5), and only gpt-3.5-turbo , gpt-4 , and claude-v1.3 can continuously improve over multiple rounds. (2) Models behave differently upon receiving feedback when playing different roles. Models playing the buyer role may be harder to improve than when in the seller role (Fig. 4). (3) It is indeed possible for strong agents like gpt-4 to continuously improve meaningfully using previous experiences and online iterative AI feedback, yet the attempt to sell at a higher price (or buy at a lower price) comes with the risk of failing to reach a deal at all (Fig. 6). We further show evidence of the model being able to negotiation in a less verbose but more strategic (thus more effective) way (Fig. 7). Overall, we hope our work serves as a meaningful initiative for improving language models' negotiation in a game setting using AI feedback.

2 Problem Setting

Our goal is to study whether LLMs can improve each other by playing a negotiation game and 76 incorporating AI feedback, as shown in Fig. 1A. We set the product being bargained as a balloon (and 77 our results hold when changing the balloon to other items). We use different combinations of backend 78 LLM engines: cohere-command [10], AI21's jurassic-2 [18], OpenAI's gpt-3.5-turbo and 79 gpt-4 [24], Anthropic's claude-instant-v1.0 (which supposedly matches gpt-3.5-turbo [14]) 80 and claude-v1.3 (which is supposed to be slightly worse but close to gpt-4 [14]). throughout our 81 experiments, we provide feedback to improve only one of the two players, while its rival receives no 82 feedback, clears the negotiation history of previous rounds, and restarts. We vary the engines for the 83 model being improved while fixing its rival's engine to be gpt-3.5-turbo . Essentially, our game 84 is gpt-3.5-turbo vs. all other engines. We keep the LM engine behind the critic is always the 85 same as the player it provides feedback to. One example setting is a gpt-4 seller playing against a 86 gpt-3.5-turbo buyer, with a gpt-4 critic. After one round, the gpt-4 critic provides feedback to 87 the gpt-4 seller such that the seller can improve in the next round while its rival gpt-3.5-turbo 88 buyer clears its dialog history and restarts. 89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

Process of the Game Before the game begins, the rules of the negotiation game are explained to the models through textual instructions with the objective of selling/ buying at a higher/ lower price. We set the deal price to [$10 , $20] for easier evaluation, since other the deal price may vary in a wide range according to the observations from our preliminary experiments. To achieve this, we hard code the seller to kick off the negotiation with 'This is a good balloon and its price is $20.' Similarly, the buyer always opens with 'Would you consider selling it for $10?' When both players strictly follow the game rules, the deal price would be between $10 and $20. We let the models play multiple runs and measure the average deal price before and after AI feedback. During the game, the seller's output is used to prompt the buyer and vice versa, conditioning on the entire conversation history. This process is repeated till a terminal state is reached. Fig. 2A shows an example round. We define three game states: (1) ON-GOING: the negotiation between the two players is still ongoing; (2) DEAL: the negotiation has concluded and the two players have reached a deal; (3) NO DEAL: the players cannot agree on a price and have failed to reach a deal. To track the game states, we set an additional moderator (powered by a fourth LLM, in our case, gpt-3.5-turbo ) to read the current dialog and classify the states (we will discuss more details about the moderator later). We measure the performance of the players based on the final deal price.

Critic A round is finished when the negotiation reaches a terminating state, either a DEAL or NODEAL. After each round, the critic LM is asked to provide constructive feedback to the player we aim to improve. This player's dialog history from all past rounds and all feedback it has received are used to prompt the critic LM (Fig. 2B). The critic model is instructed to provide three suggestions to the player, in order to improve its negotiation strategies to achieve a more favorable price in the next game. Before the next round, the player being improved receives the critic's feedback as a textual prompt, while its rival clears its negotiation history and restarts.

The Moderator The game state is classified by prompting a gpt-3.5-turbo moderator using few-shot demonstrations. The moderator reads the most recent four rounds (as well as in-context examples of different dialog states) and determines the state of the negotiation. Empirically, we found that four rounds of conversations are sufficient for the moderator to determine the negotiation state. One key challenge here is detecting no-deals as the model seems to be better at recognizing DEAL than NO DEAL. We mitigate this issue by playing multiple runs, inspect failure cases manually, and add them to the prompt with corrected labels. We find this method an effective side product recommend it as a technique for prompt optimization for generic classification tasks.

Playing for Multiple Rounds Finally, we would like to explore whether the players can continuously improve from AI feedback in a game over multiple rounds. Intuitively, the more rounds the players play, the more challenging to keep improving because the (already improved) price from the previous round becomes the baseline for the next round. In the experiments, we will show that only gpt-4 can improve over 5 rounds while other models' improvements may saturate at about 3 rounds.

3 Related Work

Game Playing and AlphaGo Zero Our setting is strongly inspired by AlphaGo Zero [31] where two agents play the game of Go and improve each other with minimal human intervention. Here we would like to explore its counterpart in natural language. Our work is similar to AlphaGo Zero in the sense that we also have AI agents (large language models) playing competitive games (bargaining) and try to improve with little human supervision. Yet there is an important difference between our work and AlphaGo Zero: we have a third agent, the critic , to give feedback helping its player to improve. This is a cooperative relationship that does not exist in AlphaGo Zero. On the NLP side, the closest related work is Lewis et al. [19] where they have (small) RNN [9] language models to bargain, and our work can be viewed as a more developed version of them since we change the engine to be large language models. In general, our work is broadly under the area of AI negotiation [6, 6], strategic reasoning [12], and general game playing [30].

Large Language Models as Generative Agents Large language models have demonstrated incredible multi-dimensional capabilities [33, 24], especially in complex reasoning [34, 28, 13] and multi-round dialog [15, 2, 3], which serve as the foundation of this work. Our work is related to concurrent works like Generative Agents [26] and CAMEL [20] as they also study the behavior of LLMs in a multi-agent game setting. The core difference between our work and theirs is that we

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

have a clear objective (the deal price) for the model to improve through competition and cooperation, while their work studies the generic social behavior of LLMs.

Learning from AI Feedback Our method is also strongly inspired by constitutional AI [3] as we both use AI feedback, while the difference is that our feedback is directly in natural language (not a scalar from a reward model). There are also related/ concurrent works demonstrating the effectiveness of natural language feedback [29, 27, 22] and self-refinement [7, 23]. Our work further confirms the effectiveness of AI feedback in the strategic negotiation game setting.

4 Experiments

In our experiments, we consider three stages that gradually deepen our exploration of learning from AI feedback: (1) We first set up the basics of the game (Sec. 4.2), showing that only a few models can improve from AI critics, in which case AI feedback can be comparable (but more scalable) as human feedback. Other models either do not understand/ follow the rule of bargaining, or cannot incorporate AI feedback for improvements. (2) Then we study the models' behaviors when playing different roles (Sec. 4.3). We discover the intriguing result that buyers are in general harder to improve than sellers. (3) Finally, we study whether models can continuously improve over multiple rounds (Sec. 4.4), and show a tradeoff of deal price versus success rate: although some models can continuously improve the deal price, it comes with a higher risk of breaking a deal. We further show evidence of negotiation in a more strategic way: both gpt-4 and claude-v1.3 's responses become longer after multiple rounds of AI feedback (note that verbosity is a straightforward negotiation strategy), yet gpt-4 is less verbose than claude-v1.3 but achieves higher deal price and deal rate, meaning that its responses, although using fewer words, are more strategic and effective.

4.1 Experiment Setup

Model Engines The minimum requirement for models to enter our game is that they should be a chatbot. All models we consider ( cohere-command , AI21's jurassic-2 , OpenAI's gpt and Anthropic's claude ) can be accessed by API calls. Amoung them, gpt-4 is the most expensive one and running 500 rounds of negotiation costs about $120 and gpt-3.5-turbo costs about $10. Other models are beta testing (as of May 2023) and do not charge money. For reference, the approximate rank of these models, from benchmarks like chain-of-thought hub [14] and HeLM [21], is that gpt-4 and claude-v1.3 are approximately similar, better than gpt-3.5-turbo and claude-instant-v1.0 , and better than cohere-command and j2-jumbo-instruct . We will consider more models in the future, such as Google's PaLM-2 [16].

We let all models compete with gpt-3.5-turbo , effectively making it a baseline for all other models. We will show that, aligning with other concurrent model rankings [14, 21], gpt-3.5-turbo is a middle-level powerful engine (worse than gpt-4 , better than claude-instant-v1.0 ). For a given model engine (say claude-v1.3 ), we run it as the seller (with gpt-3.5-turbo as the buyer) and

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

as a buyer (with gpt-3.5-turbo now as the seller) We first let the models to play one round and manually inspect if they understand the rules of bargaining. If they do, we let them play two rounds to see if they could respond to AI feedback. For the critic model, we set its engine the same as its player. We repeat the game 500 times to compute the average deal price before and after AI feedback. If they do improve one round, we let them play multiple rounds and see if they could continuously improve their strategy. We repeat the game 200 times with 5 max rounds to compute the average deal price for each round. When decoding from the model engines, we use sampling with default temperature (1.0 for gpt and claude , 0.75 for cohere and 0.7 for j2 ).

Prompt Engineering In this work, we only had to manually optimize the prompts for the moderator because the player may reach/ break a deal with very diverse expressions, and we would like to make sure the moderator correctly recognizes all of them. As mentioned above, we identify the errors made by the moderator in identifying deals and keep adding them as in-context demonstrations until the model reaches a sufficiently high accuracy (about 90+ by manual inspection). For the players and the critic, we do not do prompt engineering and keep the instructions the same for all engines (but the format may be different, e.g., claude requires two linebreaks before 'HUMAN:' and j2 requires two '##' after each dialog round). Code and Prompts will be released publicly on publication.

4.2 Basic Experiments

In this section, we first study the minimal requirements for models to participle in our game, namely (1) understanding the rule of bargaining and (2) responding to AI feedback. Then we consider basic comparison between AI and human feedback, showing that AI feedback can be comparable to human feedback, but more scalable.

Conversational ability does not guarantee ability to negotiate or learning from feedback We study whether conversational models can understand the rule of bargaining by manually checking traces of the dialog, and found that cohere-command fails to understand the rules, as is shown in Fig 3A. We observe that it does not realize what price is a better deal. For example, when playing seller, it rejects a proposal of $10 but accepts $8. We also observe that AI21's j2-jumbo-instruct model, although understanding the rule of bargaining, cannot incorporate AI feedback, as is shown in Fig. 3B. Generally, when instructed with AI feedback, the model keeps the same strategy as before, without any improvements.

After ruling out the cohere-command and j2-jumbo-instruct models, we consider the three remaining models: gpt-3.5-turbo , claude-instant-v1.0 and claude-v1.3 . For these three engines, we do not observe the problems in Fig. 3. This means that these models can be used for our multi-round games.

AI Feedback can be comparable to human feedback Now we consider some initial comparison between AI and human feedback. We emphasize that our goal is not to show which one is better a similar level of effectiveness would suffice our study (to see if LLMs can continuously improve through self-play and AI feedback). For the human feedback, we manually write done a pool of 10 suggestions. Then we play 500 runs of the game, computing the deal price before and after feedback. After 500 runs, we compare the improvements after: (1) randomly sampling 3 suggestions from the predefined pool and (2) asking the AI critic to write down 3 suggestions. We note that this may underestimate the performance of human feedback, yet it would be unpractical to ask human to write done 3 suggestions for all 1500 runs (while AI feedback does not have this problem). The results are shown in Table 1 where we see that all three models ( gpt-3.5-turbo , claude-instant-v1.0 and claude-v1.3 ) exhibit comparable improvements over human and AI feedback.

222

223

224

225

226

227

228

229

230

231

232

233

234

A1

. Claude-instant-v1.0 seller

A2

. GPT-3.5-Turbo seller

A

. AI feedback moves the distribution of sellers' deal price towards a higher range

B1

. Claude-instant-v1.0 buyer

B

. Buyers are harder to improve than sellers: AI feedback does not quite move buyers' deal price distribution

C1

. Claude-v1.3 buyer

C2

. GPT-4 buyer

C

. Stronger agents (Claude-v1.3 and GPT-4), when playing buyers, can still improve from AI feedback

B2

. GPT-3.5-Turbo buyer

100

BeforeAlCritic$15.3

101

Before AlCritic $15.0

102

108

AfterAlCritic$16.3(+1.0)

79

75

69

80

62

62

50

52

55

53

50

54

63

63

34

37

44

45

41

25

28

26

21

21

12

11

12

0

00

3

Q

Q

8

6

10

12

14

16

18

20

10

12

14

16

18

20

109

97

100

Before AlCritic $15.4

After AI Critic $15.2 (-0.2)

99

98

79

83

AfterAlCritic$14.6(-0.5)

Before AlCritic$15.1

73

75

59

73

70

61

53

64

51

52

59

53

60

48

50

43

44

3740

40

28

33

25

20

18

0

10

T

01

12

14

16

18

3

20

10

5

12

14

16

18

3

Q

20

104

100

90

96

93

AfterAlCritic$13.7(-1.2)

BeforeAlCritic$14.9

After AI Critic $12.4(-3.0)

BeforeAlCritic$15.5

81

75

59

57

112

106

50

46

42

47

71

72

25

27

52

57

51

0

2

12

7

10

12

20

1212

11

4

10

12

14

16

18

20

10

12

14

16

18

20

4.3 Behaviors of Different LLM Backend

So far we have established that our game setting is valid for stronger LLM engines. Now we consider the detailed behavior comparisons using different engines for different roles. Specifically, we use claude-instant-v1.0 , claude-v1.3 , gpt-3.5-turbo , and gpt-4 to play the seller/ buyer (against a gpt-3.5-turbo buyer/ seller respectively), then study the deal price distribution before/ after AI feedback (also recall that the AI critic is powered by the same engine as its player). The results are visualized in Fig. 4. When claude-instant-v1.0 and gpt-3.5-turbo play the seller, they are able to improve their average deal price after AI feedback (Fig. 4A). But when they play the buyer role, their average deal price does not improve, which indicates that buyers tend to be a harder role than sellers (Fig. 4B). Yet this observation does not hold for engines like gpt-4 and claude-v1.3 , as they can still improve from AI feedback even playing buyers. Overall, this set of experiments reveal the nuanced capability differences between the four engines we consider.

4.4 Towards Continuous Improvements from Iterative AI Feedback

Now we unroll the game to multiple rounds and see if models can continuously improve from 235 previous dialog history and iterative AI feedback. Specifically, we let gpt-3.5-turbo , gpt-4 , 236 claude-instant-v1.0 , and claude-v1.3 play as the seller/ buyer respectively against a rival 237 powered by gpt-3.5-turbo . As mentioned before, the critic shares the same engine as the player it 238

GPT-3.5-Turbo

Claude-instant-v1.0

Criticize seller

Criticize buyer

A1

. GPT-3.5-Turbo seller continuously improves

from AI feedback

A2

. GPT-3.5-Turbo buyer continuously improves

from AI feedback

B1

. Claude-instant-v1.0 seller only improves 1 round

from AI feedback

B2

. Claude-instant-v1.0 buyer does not improve

from AI feedback

20

20

18

18

16.75

16.9516.85

16.24

16.13

15.87

15.96

16.07

16

15.06

16

15.11

14

14

12

12

10

Round 1

Round 2

Round3

Round 4

Round5

10

Round 1

Round 2

Round 3

Round 4

Round 5

20

20

18

18

16

15.35

16

-15.60

15.53

15.48

15.67

15.72

14.69

14.54

14.25

14.18

14

14

12

F

12

10

T

Round 1

Round 2

Round 3

Round 4

Round 5

10

IRound 1

Round 2

Round 3

Round 4

Round 5

helps with. We play 200 runs of the game, and unroll each game to be 5 rounds. We compute the 239 final deal price and the deal success rate and see if the price can be continuously improved. 240

Fig. 5 shows gpt-3.5-turbo and claude-instant-v1.0 playing different roles. For a given 241 engine, improvements over one round may not necessarily extrapolate to multiple rounds, as we 242 observe that gpt-3.5-turbo can improve over multiple rounds, but claude-instant-v1.0 only 243 improves at most one round. 244

245

246

247

248

249

250

Now we consider the tradeoff between the tendency of achieving a higher deal price versus the rick of breaking a deal, as is shown in Fig 6. We see that when playing sellers, all four model engines can improve over at least one round, but this comes at the cost of decreasing deal success ratio. When playing buyers, there are models that cannot improve ( claude-instant-v1.0 ), or saturate over 3 rounds ( claude-v1.3 ), while gpt-4 and gpt-3.5-turbo can continuously improve, and gpt-4 achieves better (lower) deal price and higher deal rate than gpt-3.5-turbo .

251

252

253

254

255

256

257

258

259

260

Finally, we study how iterative AI feedback influences the language complexity used by the agents by plotting the average response length (measured in number of characters) after each round, as is shown in Fig. 7. We see that both claude-v1.3 and gpt-4 become more verbose after iterative AI feedback with a continuously increasing response length. This is intuitive because being verbosity is a straightforward strategy in negotiation. Yet for claude-v1.3 , the verbosity does not translate to better negotiation strategy, as its improvement saturates after three rounds (Fig. 6B1). In comparison, gpt-4 's increase verbosity is more strategic, as it use less words than claude-v1.3 , but achieves better deal price and deal success rate (Fig. 6B). This observation serve as strong evidence that AI feedback improves players' response towards a word-tuned, strategic direction.

5 Conclusions

In this work, we study whether multiple large language models can autonomously improve each other 261 in a negotiation game by role-playing and learning from AI feedback. Our experiments show that 262 certain models can indeed improve by continuously playing competition games with iterative AI 263 feedback, under well-defined rules in an AlphaGo Zero fashion. We also show the tradeoff between 264 next-round price improvement and success rate, as a better deal price also comes with a higher risk 265

Average deal price over 5 rounds

Deal success ratio over 5 rounds

Criticize

Criticize

Buyer

Claude-instant-v1.0

GPT-3.5-Turbo

Claude-v1.3

A1

. When playing seller, most models can

improve at least one round.

A2

. Yet the deal success ratios continue to decrease

over 5 rounds (mostly < 50% in the last round).

B1

. When playing buyer, GPT models are better at

improving from AI feedback than Claude models.

B2

. The deal success ratio continues to decrease, but

overall higher (mostly >50%) than criticizing sellers.

1.00

17

0.75

16

0.50

0.25

15

0.00

T

1

2

3

4

5

1

2

3

4

5

16

1.00

15

0.75

0.50

14

0.25

13

0.00

1

2

3

4

5

1

2

3

4

5

Round 1. How about we meet in the middle at $15?

Round 2. This high-quality balloon is made from durable material, and I can offer a slight

discount at $18.

Round 3. Hi there! I hope you're having a fantastic day. This one-of-a-kind balloon, made from durable

material, is priced at $20.

Round 4: This special balloon can bring an extra touch of joy to any event and create lasting

memories, making it a worthwhile purchase at $20.

Round 5.This custom-designed balloon is not only high-quality, but

it also has a unique and captivating look that sets it apart from any

other balloons you might find, making it a great value at $20.

Response length in number of characters

Claude-v1.3 seller

1500

1000

500

0

of deal breaking. This suggests future research may consider global optimization for improving the 266 overall gain over multiple rounds. We further show evidence of improved language from iterative 267 AI feedback: in a multi-round game, one model may be less verbose than another, but be better 268 word-tuned, thus more effective in getting a better deal. 269

We believe our results have profound implications for AI research: on the positive side, it indicates 270 the possibility of continuously improving language models with minimal human intervention. On 271 the risky side, it might be more challenging to oversight the model behavior in our framework 272 because models are acting autonomously, which calls for future alignment and safety research in the 273 multi-agent game setting. Overall, we believe our work provides a meaningful initial exploration for 274 large language models' learning from game-playing and iterative AI feedback. 275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

References

Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661 , 2022.

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 , 2021.

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 , 2022.

Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamile Lukosuite, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:2211.03540 , 2022.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Kushal Chawla, Jaysa Ramirez, Rene Clever, Gale Lucas, Jonathan May, and Jonathan Gratch. Casino: A corpus of campsite negotiation dialogues for automatic negotiation systems. arXiv preprint arXiv:2103.15721 , 2021.

Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128 , 2023.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 , 2014.

Cohere. Cohere command models. Cohere website , 2023. URL https://docs.cohere.com/ docs/models .

Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:2212.10559 , 2022.

Meta Fundamental AI Research Diplomacy Team (FAIR)†, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science , 378(6624):1067-1074, 2022.

Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting for multi-step reasoning. arXiv preprint arXiv:2210.00720 , 2022.

Yao Fu, Litu Ou, Mingyu Chen, and Yuhao Wan. Measuring llms' reasoning performance. Github , 2023. URL https://github.com/FranxYao/chain-of-thought-hub .

Amelia Glaese, Nat McAleese, Maja Tr˛ ebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375 , 2022.

Google. Palm 2 technical report. ArXiv , 2023.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022.

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

AI21 Labs. Announcing jurassic-2 and task-specific apis. AI21 Blog , 2023. URL https: //www.ai21.com/blog/introducing-j2 .

Mike Lewis, Denis Yarats, Yann N Dauphin, Devi Parikh, and Dhruv Batra. Deal or no deal? end-to-end learning for negotiation dialogues. arXiv preprint arXiv:1706.05125 , 2017.

Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large scale language model society. arXiv preprint arXiv:2303.17760 , 2023.

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110 , 2022.

Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Languages are rewards: Hindsight finetuning using human feedback. arXiv preprint arXiv:2302.02676 , 2023.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651 , 2023.

OpenAI. Gpt-4 technical report. ArXiv , abs/2303.08774, 2023.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Joon Sung Park, Joseph C O'Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442 , 2023.

Ethan Perez, Sam Ringer, Kamil˙ e Lukoši¯ ut˙ e, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251 , 2022.

Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. Reasoning with language model prompting: A survey. arXiv preprint arXiv:2212.09597 , 2022.

Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with natural language feedback. arXiv preprint arXiv:2204.14146 , 2022.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. arXiv preprint arXiv:2212.07677 , 2022.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the ability of LLMs to improve in a negotiation game. The find that only a subset of language models can self improve from AI feedback, a model's ability to learn from feedback depends on its role in the game, and stronger agents can go through more rounds of negotiation.\n\nSTRENGTHS:\n- The paper is generally well written; it was more or less easy to understand the entire paper.\n- The experiment methodology is sensible. It was nice to narrow down the set of models by eliminating models based on their ability to respond to feedback or understand the problem. \n- The results are interesting and could be valuable to the community on understanding the role of LLMs as agents.\n\nWEAKNESSES:\n- It seems like incorporating feedback is done by providing the feedback as context for the LLM. The paper could be made stronger by utilizing the fine-tuning APIs for the given models. \n- Using GPT3.5 as the fixed agent is interesting. It's clear GPT-4 is the strongest agent in this scenario and it'd be interesting to see how well these results would hold against a stronger agent. In a similar vein, seeing how well these results would hold when actually negotiating with a human. \n- Could buyer's and seller's be prompted better? E.g. would it be possible to prompt claude instant v1.0 more effectively to respond to multi-turn negotiations or prompt Cohere command or AI21 j2 better to respond to bargaining and feedback respectively?\n\nQUESTIONS:\nOverall, I think the work is interesting, it's reasonably well designed, and has interseting implications in reinforcement learning and large language models. I think there were some decisions (e.g. incorporating feedback in the context passed to the LLM, using GPT3.5 as the fixed agent that we are buying/selling against) that make the paper's results a little less clear. However, the results are interesting and show a first step towards using LLMs as agents, in negotiations.\n\nLIMITATIONS:\nThe authors partially address limitations of their work, and address societal implications of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the intriguing possibility of autonomous improvement among multiple large language models through a negotiation game. By assigning various LLMs to distinct roles and allowing them to engage in iterative improvement, the paper aims to enhance their negotiation strategies without human intervention. The study uncovers some insights into negotiation problems, encompassing the assessment of model capabilities and their responses to AI feedback. However, after careful evaluation, I am inclined to reject this paper due to its limited contribution and insufficient experimental results, which fail to adequately demonstrate the effectiveness of its findings.\n\nSTRENGTHS:\nThis paper studies an interesting problem: Improving large language models with each other and demanding only black-box access is a promising direction.\nThis paper is largely clear and concise. It is easy to follow the problem setting and the negotiation process.\nThis paper compares capabilities (especially the continue learning ability using in-context learning) of some advanced large language models in the proposed negotiation problem, which is interesting.\n\nWEAKNESSES:\nOne of the main technical novelties is the AI feedback technique used in their method. However, this technique seems like a result of random attempts plus some intuition. More detailed though on how did the authors develop this technique or the comparison between other possible candidate techniques is needed. Moreover, this technique is similar to CoT. It would be better for the authors to discuss on the relationship between these two techniques. I am curious about the performance of guiding LLMs to think step by step without relying on additional critics in negotiation problems. More thorough explanations or experimental results of this aspect would provide deeper insights into the effectiveness of the proposed approach.\nThis paper aims to improve the ability of large language model by self-play with each other. However, to strengthen the paper's claims, it would be better to provide additional evidence regarding the transferability of the proposed framework to different types of games. Suggestions on the specific implementation process would also be valuable in applying this framework to different domains.\nThe stability of the environment is also a concern since the results appear to heavily rely on the reliability of the moderator. This dependence raises doubts regarding the individual contributions of various components in the system. Furthermore, the claim of proposing a technique for prompt optimization for generic classification tasks in Line 120 lacks sufficient details and evidence. Elaborating on this technique and demonstrating its effectiveness would enhance the paper's credibility and address this specific weakness.\nIt is not clear whether there exists an upper limit of improvement using ICL. It would be better for the authors to also discuss ICL with, e.g., fine-tuning or other trainable parts.\n\nQUESTIONS:\nWhat is the exact definition of \"self-play\"? It seems that only one of the two players improves in the setting of this paper, which slightly different previous definition in, e.g., https://arxiv.org/pdf/2002.04017.pdf.\nLine 59: It would be beneficial to provide additional details, possibly with references, comparing the effectiveness of using AI feedback with RHLF. This comparison would contribute to a more comprehensive understanding of the proposed approach's advantages and distinguish it from existing techniques.\nLine 85: Why the LM engine behind the critic is consistently the same as the player it provides feedback.  \nFigure 4: It lacks the experimental results of Claude-v1.3 and GPT-4 as buyers. Why these scenarios are omitted? Including these results in the figure would ensure a complete representation of the experiments conducted.\n\n \n\nPresentation:\n\nLine 118, 119: inspect -> inspecting, add -> adding\nLine 120: recommend -> and recommending\nLine 211: comparison -> comparisons\nFigure 5: whild -> while\nLing 257: use less -> uses fewer\nLine 258: serve -> serves\n\nLIMITATIONS:\nThe authors discuss the limitations of the work in the last section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies whether multiple large language models (LLMs) can improve each other in a negotiation game by playing, reflecting, and criticizing, with minimal human intervention. Two LLMs play the roles of a seller and a buyer, and a third LLM plays the role of a critic who provides feedback to one of the players to improve their negotiation strategy. The authors report several intriguing findings, such as the different abilities and behaviors of various LLMs in the game, the trade-off between deal price and success rate, and the evidence of improved language complexity and strategy from iterative AI feedback.\n\nSTRENGTHS:\n1. Interaction between LLMs is quite interesting and potentially important for future AI research.\n2. The authors experimented with a variety of LLMs, including open-source (Cohere) and proprietary (GPT, Claude) ones.\n\nWEAKNESSES:\nMajor Issue:\n1. The negotiation setting is quite contrived and not well-grounded. Specifically, there is no context about the goods being discussed. Also, the feedback and overall conversation seems very generic (Figure 2). There is no value/intrinsic motivation for the buyer to get the goods, and there is only one choice to go for. Typically, negotiation in real-life doesn't happen this way.\n2. It feels like the current LLM negotiation setting can be viewed as `predicting the most probable outcome/text' given the negotiation contexts.\n3. AI feedback is not the same as human feedback, the behavior may be completely different in real-world situations since there are a lot of other concerns including 'value', 'time', 'personal preference' for negotiation. This makes me wonder what this simple negotiation has to offer in terms of understanding/future research.\n\n\nMinor Issue:\n1. The title is a bit misleading. It seems to me from the title that the paper is about an algorithm, but it is more about evaluations and understanding.\n\n\n[After Rebuttal]\n\nMy concern still lies in the 'motivation' behind the setting for the experiment. In my understanding, negotiation requires a setting, e.g., both seller and buyer will have a stake on the product (buyer want to pay lower than the *utility* of the product and seller want to earn more than the *cost*). However, such basic setting is not present in the experiments. \n\nThis makes me concerned if the observed effect is really negotiation or just the effect of instruct fine-tuning.\n\nI decided not to change my score.\n\nQUESTIONS:\n1. I wonder why LLAMA based models are not considered in this work.\n2. Is it possible to have both the seller and buyer receive feedback in a single experiment? I wonder what will happen.\n3. Is it possible to add some story/context to the negotiation so that the LLMs have a better motivation?\n4. Is the best policy to just always be fixed at starting price and ask LLM to generate some excuse (Prompt: \"Give me some sentences to sell my __ for the highest possible price.\")?\n\n[After Rebuttal]\n\nAll questions have been addressed.\n\nLIMITATIONS:\nI was unable to find any information related to limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the strategic multi-agent problem setting of two LLM Players interacting in a negotiation (or bargaining) game and proposes to use feedback from an LLM Critic to improve each Player’s expected behavior and performance in the game.  Importantly, the paper aims to study how AI Feedback can enable Player improvement when playing competitive games under well-defined rules.  The proposed method is to use the player dialog history and critic feedback as “in-context demonstrations” for players to self-improve over the course of the game.  Experiments are conducted on a bargaining game instance negotiating the price of a balloon, investigating several LLMs as base models for the Players (and their Critics).\n\nSTRENGTHS:\nIdentified strengths of paper:\n - I really like the premise of this work: that agents playing in a strategic game may each be able to benefit and improve their negotiation strategies, not only by observing their opponents, but additionally by heeding the advice of personalized critics whose aim is to help (cooperate with) them.  Like other AI feedback methods, this type of approach has the potential to scale better than using Human feedback for the same type of assistance, presuming feedback is helpful and thus desirable for this negotiation task.\n - The topic is timely and very relevant to the multi-agent and game theoretic communities, as it takes a long-existing and well-studied problem (multi-agent negotiation) and investigates how rational player strategies can be improved using AI feedback and LLMs.\n\nWEAKNESSES:\nIdentified concerns and suggestions to improve the paper submission:\n- It would be extremely helpful to see examples of what the *instructions* and *in-context learning* looks like for the Critic agents? Examples of their few-shot prompts (or instruction data for Finetuning) are important, as this is a key contribution of the work (adding AI Feedback to strategic reasoning/negotiation settings). Notably, where are the Buyer/Seller Critics getting the suggested negotiation strategies in Figure 1B  from (e.g. the flinch technique, the anchoring technique, etc)?  They seem to be good, general strategies to have in the player’s toolbox, but require the Critic to have knowledge of effective negotiations and how/when to employ them.  Thus, how are the Critics coming up with general negotiation strategies and reasoning about when to apply them?  This is important for opening the black box of the AI Feedback component of the proposed method.\n - While I like the overall premise, the novel technical contribution of this work seems tenuous.  With that, the problem setting is strongly inspired by AlphaGo Zero, but is there any traditional learning in this setting (i.e. any updates to the agent policies based upon the AI feedback given)?  That is still unclear to me.  Currently the paper seems to simply add a small amount of additional context in the prompt for the Player LLMs.  If there is no traditional learning, this setting is also critically different from AlphaGo Zero in that the proposed feedback signal (from Critic Agents) is **not** used to update the Player policies.  This should be explicitly clarified.  Also, if there are model updates, it would be useful to see the learning update rules.  If not, why not do any finetuning of the model?  At least as an experimental condition to compare against and understand if in-context learning is \"sufficient\".  Perhaps more motivation and context for this design decision would be helpful.\n - This work explores a potentially interesting research direction (negotiation games or strategic gameplay more generally + AI Feedback for improved player strategies) but it’s not clear to me how well motivated AI feedback is for this negotiation game or how interesting the problem is in its current instantiation.  In particular, it would be helpful if the paper provided some type of performance analysis to show that the negotiation game used (an instance of balloon purchasing) is interesting and challenging to solve *before* ever adding in LLMs or a Critic.  How would existing multi-agent negotiation/bargaining approaches solve this game?  What equilibria do the Players generally converge to without LLMs?  How and why does using LLMs change the *expected* solution?  Given two Player LLMs, why is a Critic *necessary* to have?  In other words, if the Critic is being prompted with examples of bargaining conversations, could the Buyer/Seller agents not simply see those same examples and then converge upon the same solution (they are currently finding) *without* a Critic?  Or is there some value that the Critic role *uniquely* adds (e.g. more efficient or robust convergence on an equilibrium in this negotiation game)?  Motivating the game selected as an interesting and challenging problem in its own right is critical.\n - Furthermore, I still question the generality and significance of the paper’s empirical findings.  **RE Significance:** How significant and meaningful are the differences in Seller performance in Table 1?  The numbers don’t seem that different to me, but perhaps with more context, the significance becomes more clear.  With that, I see a distributional shift in Figure 4, but how meaningful is this shift?  What are the implications of it, regarding how good or bad these solutions/equilibria are before and after receiving AI feedback?  Do the solutions to the negotiation game simply change a small amount but are comparable in terms of how preferable/desirable they are or do they get qualitatively better in some way?  **RE Generality:** The paper seems to use only one evaluation domain (balloon purchasing).  It is not clear to me which empirical findings/trends from the use of AI Feedback on this one, seemingly simple bargaining domain, are expected to generalize. In particular, regarding the effectiveness and impact of AI Feedback in other multi-agent negotiation domains and settings.  This limited evaluation seems more like an interesting case study than general findings that transfer.\n - In Subsection 4.3, the paper provides analysis of each of the LLM models (gpt-3.5, gpt-4, claude-v1.0, etc) versus a fixed gpt-3.5-turbo opponent.  However, a more thorough investigation examining the cross product of each LLM model as Buyer against every other LLM model as Seller (a *complete* Payoff matrix with all pairwise model comparison) could potentially provide more general insights. Why was this not done? Can it still be done to show a resulting Payoff Matrix?  Not doing a pairwise comparison of *all* pairs of models might unnecessarily constrain the results and thus the insights that can be extracted.\n\nQUESTIONS:\nPlease see Weaknesses section.\n\nLIMITATIONS:\nPlease see Weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-08-01"}