Self-playing Adversarial Language Game Enhances LLM Reasoning
Authors: Pengyu Cheng
Tianhao Hu
Han Xu
Zhisong Zhang
Yong Dai
Lei Han
nan du
Xiaolong Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Self-play
Large Language Models
LLM Reasoning
Primary area: generative_models

Abstract: We explore the potential of self-play training for large language models (LLMs) in a two-player adversarial language game called Adversarial Taboo. In this game, an attacker and a defender communicate around a target word only visible to the attacker. The attacker aims to induce the defender to speak the target word unconsciously, while the defender tries to infer the target word from the attacker's utterances. To win the game, both players must have sufficient knowledge about the target word and high-level reasoning ability to infer and express in this information-reserved conversation. Hence, we are curious about whether LLMs' reasoning ability can be further enhanced by Self-Playing this Adversarial language Game (SPAG). With this goal, we select several open-source LLMs and let each act as the attacker and play with a copy of itself as the defender on an extensive range of target words. Through reinforcement learning on the game outcomes, we observe that the LLMs' performances uniformly improve on a broad range of reasoning benchmarks. Furthermore, iteratively adopting this self-play process can continuously promote LLMs' reasoning abilities. The code is available at https://github.com/Linear95/SPAG.

Self-playing Adversarial Language Game Enhances LLM Reasoning
Pengyu Cheng'
Tianhao Hu²
Han Xu
Zhisong Zhang' Xiaolong Li²
Yong Dai
Lei Han
Nan Du
Tencent Al Lab 'Shenzhen & Seattle Tencent Robotics X Lab
pengyucheng@reeccnt.ccom com
Abstract
We explore the potential of self-play training for large language models (LLMs) two-player adversarial language game called Adversarial Taboo. In this game, an attacker and defender communicate around a target word only visible to the attacker. The attacker aims induce the defender to speak the target word unconsciously, while the defender tries to infer the target word from the attacker's s utterances. To win the game, both players must have sufficient knowledge about the target word and high-level reasoning ability to infer and express in this information- reserved conversation. Hence, we are curious about whether LLMs' reasoning ability can be further enhanced by Self-Playing this Adversarial language Game (SPAG). With this goal, we select several open-source LLMs and let each act the attacker and play with a copy of itself as the defender an extensive range of target words. Through reinforcement learning on the game outcomes, we observe that the LLMs' performances uniformly improve on broad range of reasoning benchmarks. Furthermore, iteratively adopting this self-play process can continuously promote LLMs' reasoning abilities. The code is available at http:/uub..om//ineee2///////
LLaMA-2-7B
Baichuan-2-133
BSH
POA
9BH
79.38
55
57.79
Mutual
Mutual
WGrande 63.92
WGranda 71.43
AMU-
30.98
AMC-4
36.26
6CA2
100A2
47.53
50.00
ARC-C
ARCiC
Race
AlpacaSFT
Imitation
SPAG-epoch1
SPAG-eporh2
SPAG-eparh3
Figure 1: Reasoning improvements from Self- Belf-Playing Adversarial language Games (SPAG) on comprchensive reasoning benchmarks. With the SPAG epoch increasing, the LLM reasoning ability continuously improves. Each axis is normalized by the maximum answer-accuracy value.
Introduction
Large language models (LLMs), such as GPT-4 [OpenAl 2023b and Gemini TTam al. 2023
have reformed the domain of artificial intelligence (Al) with astonishing language capacities, such as
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

When you go out with your friends, what kind of process or situation usually happening there?
What do you enjoy deing when you're with friend?
usually enjoy listening to music together, watching good movic, or going hiking during the woekends.
We are usually socializing chatting.
That sounds lovely! Do you guys ulso discuss the content musics and movies together?
Correct! These socializing are often about sharing ideas. What would you consider this exchange??
Absolutely! We always dive deep into conversations about our favorite scenes und tracks!
know the word! is "conversation".
Figure 2: Examples of Adversarial Taboo with the same target word "conversation". The left shows an attacker- winning game, in which the defender unconsciously speaks out the target word. The right is defender- episode because the defender makes the correct inference from the dialogue.
natural language understanding Yang et al., 2023b Touvron et al. 2023 text generation Koccn
et al.] [2033 Anil et al. [023]. machine translation Liao et al. 2023]. J. and programming (Surameery
and Shakor, 2023 Tian et al. 2023]. However, the reasoning ability of LLMs, which is essential
(Yao et al. 2021], still retains being challenged by various criteria including correctness (Zhang al. 2023a
for complex problem-solving [ann et al., 2023) and advanced intellieeneedevelopiin
and faithfulness [Turpin et al. [024].
To address the reasoning challenge of LLMs. plenty of works have contributed in in-depth efforts from the perspectives of Chain-of-Thoughh (CoT) prompt engineering (Wei ct al. [022 Ding et al.)
2023 Yao et al.) 2024). and the usage of auxiliary reasoning tools [Pan et al. [023] However, both
prompt-based and tool tool-calling methods require additional prompt designs, which are inconsistent and sensitive to different prompt patterns and LLM checkpoints (Turpin et al.) 2024] Chu et al. 2023]. More fundamental and consistent reasoning- iimprviin approaches are post- post-pretraining (Azerbayey] et al. 2023 and fine-tuning (Dong et al.) [023a], which trains LLMs with additional reasoning-related text corpus. Nevertheless, these methods demand sufficient high-quality textual data, which are difficult to collect due to the massive costs of human annotation efforts (Singh et al., 2023].
To improve LLM reasoning more efficiently, self-improvement methods, which enhance LLMs with model-generated synthetic data, have recently attracted increasing research attention [Singh
al. 2023 Huang al. 2023 Burns et al. 2023 Chen et al. 2024]. Self-improvement methods usually utilize the intrinsic language capability of LLMs to judge |Huang et al. 2023). filter Yuann
ct al. 2024]. or revise (uuan et al. 2024] self generaeded samples to enhance their quality. However, most self-improvement methods rely on broad range of high-quality question queries to prevent over-fitting into a sub-domain of reasoning tasks, which still requires additional data collection and cleaning. Besides, the judgments from LLMs are not guaranteed objective [Raina et al.) 2024). If an LLM already has an incorrect or biased recognition of particular concept, the iepr-improvement process can reinforce and amplify the LLM's ' cognitive dissonance.
Towards more general and objective
methods, we are inspired by the
advancement from AlphaGO (Silver et al. 2016) to AlphaGO Zero [Silver et al. [017]. in which the game agents' intelligence continuously promotes via self-play without any human knowledge. Analogically, we expect to set up a language game where LLMs can improve their reasoning capacities via reinforcement learning (RL) during self-play. Although language games have attracted increasing
attention in natural language processing [Lewis et al. 2017 Hausknecht et al.] 2020 Xu et al. 2023 Wu et al., 2024). most of them are specially designed with customized game rules, in lack of the generalization to improve the general language capacities of L.MMs. Among a few general- target language games including red-teaming [Ma et al. 2002). negotiation (Lewis et al. 2017]. and bargain (Abdulhai et al. 2023], additional human judgments reward models are required for outcome determination, which posts challenges on the efficiency and effectiveness of large-scale self-play RL training. Recent studies have raised interest in entity- or word-based language games, such as 20- 1o-Question [Zhang et al., 2023b) and Suuss-My-Cing [Abdulhai et al. 2023), which provide not only traight-forwart word-level outcomes but also language universality by traversing the game word from comprehensive vocabularies. However, unlike the GO game, these word-based games are out of the adversarial scheme, limiting the game intensity and self-play learning effectiveness.
With the above consideration, we select an adversarial language game called Adversarial Taboo Yao et al. 2021 1 in which an attacker and a defender perform conversation around target word

---

only visible to the attacker. The attacker aims to induce the defender to speak out the target word unconsciously; the defender tries to avoid unconscious utterance of the target word and guess the word from the dialogue history. To win the adversarial game in information-limits conversations, both players are required to have high-level language capacities in terms of expression, upstanding. and reasoning. Moreover, by collecting target words from vast vocabulary, this game can cover a broad range of topics providing sufficient language versatility. Besides, the game outcomes can be automatically and explicitly judged: we only need to check whether the target word appears in the defender's s utterances (attacker wins) or its inference patterns (defender wins), We conduct the self-play this adversarial game using open-source LLMs, LLaMA-2-7B (Touvron et al. 2023 and Baichuan-22 2-11B (Yang et al., 2023a], with target words selected from a 50K top-frequency vocabulary (Davies 2020J. Next, we conduct offline reinforcement learning on the game outcomes and observe significant performance improvement on a broad range of L.M reasoning benchmarks. Furthermore, iterate this sampling-leamingg process with three epochs, within which the L.MMs' reasoning can continuously obtain improvement. We believe this novel training scheme, Self-Play of Adversarial Game (SPAG), has great potential for developing advanced LLM capacities.
2 Preliminary
With the development of LLMs [OpenAl 2023a b). reinforcement learning (RL) has played an
increasingly important role in language model training Ouyang et al. 2022 Ramamurthy al.)
2023]. The prime application scenario for RL in LLM training is reinforcement learning from human feedback (RLHF) (Yuan al.] [023]] Cheng al. 2023b Zeng et al. [2023]. RLHF first learns a reward model r(æ: y) from the human feedback preference pairs [Cheng et al. 2023a and then
optimizes the LLM policy To (B(aa))) to maximize the expected reward value [Dong et al.
2023b
(R)) - RE Dree:..ywrxo/// [r(r,,))
To learn the above objective, proximal policy optimization (PPO) [Schulman et al.
2017 algorithm
has been recognized as the mainstream solution. In each update equation PPO minimizes:
To x)) Á [KL[TT||TT) [mo]]
[PPO(TT) = -E pyymm
())
where Ty is a copy of my before the update, A is the estimated advantage value (Schulman et al. 2016) with respect to the reference policy Tg. and [TT is the Kullback-Leibler (KL) Kullback
1997] divergence regularizing we with an appropriate updating step. However, PPO for LLMs has
been continually challenged due to its inefficient natural-text online sampling and the unstable training
processes [Baheti et al. 2024]. Among the improvements to PPO [Rafailov al. [0233 Yuan et al.,
2023 Dong et al., 2003bb, Baheti et al. (2024) adopts the PPO objective into an offline scheme by
(Bbbeti using importance sampling Neal 2001 which is named Devaantge-Leffover--ne
(A-LoL):
A Fiet T x> [z)
V VoLRLHF-A-LOL Bww..ywaat
V log (T(())
(3)
Tee
Here the sample y ~ T and advantage Å Fial are both from the reference distribution ( ax)) and calculated offline. Besides, Gulcehre et al. [2023] proposed Reinforced Self- Self-Training (ReST) to simplify the RLHF scheme. With a threshold r E Ik, ReST updates the LLM by the reinforcement on the selected samples D,- (1x v) r(x y) > r):
LRSST (TO) = Ex DarwDDammmaalla ( Lo y) = Ep, [Co( v)).
where Co 3) could be any offline RL loss such as A-LoL or the vanilla language modeling loss.
3 Self-play of Adversarial Language Games
The game of Adversarial Taboo is first introduced by Yao et al. 2021 in which an attacker μ and a defender v involve multi-turn conversation. At the beginning of the game, the attacker is assigned a target word w Virgees which is not informed to the defender. The attacker's target is to induce the defender to speak the target word unconsciously. To achieve this goal, the attacker can talk about any topic related to w, except directly speaking out the target word. In contrast, the defender is required to infer the target word without any unconscious utterance of the word. If the defender has sufficient confidence to infer the word, can yell "I know the word! terminates. If the guess is correct, the defender wins, otherwise the attacker wins. Besides, the game has a maximum number of turns To. If nobody wins during To turns, there is a tie. The examples of Adversarial Taboo are shown in Figure 2
{target word).". Then the game
3

---

Algorithm Data collection of LLM self-plays for the adversarial language game.
Inputs: LLM policy Te (y)zz) target word w, attacker and defender prompts fattcck fdcfend. Set the initial state Su = (w).
for t from to T do
Sample an attacker utterance Update state si = (w, u Vi..... u4 1, 0t- U,).
po 11 ) = Io = Un x = ffatuck 11-1)).
Sample a defender utterance on vo (1)))()) = To (y/ Vz x = Saetenal (s()). Update state 8, = (w, u1, 01,.... 1 11 ue v²). end for Collect an episode r = (so, 8 81, 8/ ST)
3.1 Adversarial Language Game Modeling
Littman 1994), which can We view the Adversarial Taboo as two-player zero-sum Markov game be described by a tuple as
The state space S {s, s
sista To} contains two types of states, of (w, U1 U1 ug...., Uz) and s; = (w, U1 U1 u2..... Ue vt), where {u:]i=ı and {wi)i=z are the utterances of the attacker and defender, respectively. Games start at (w) with target word w E Vusget and end with To maximum turns. States s and st end with utterances us and v for the defender and attacker to act, respectively.
The action space A is shared with both the attacker and defender, which is equivalent to the token sequence space of natural language N { (x1, #2 .., x2)) ..,)), E Viken: N+}.
The transition function F S x - S deterministically appends the utterance u, v, the end of the dialogue, and converts 8 = F(..1,, u.) and 8, F(s(,, v2).
The reward r : S x A + R evaluates the actions u, v E A based on their corresponding states s, S with rewards r(s, u) and r(s', v). respectively. Given game episode
(so, ,1, ,',,
we denote the attacker's 's total reward R(T) = E'= r(s--1, so the defender's 's total reward is Σt= r(s1, v+) = - R[()) to satisfy the zero-sum constraint. More detailed reward designs with heuristic rules for the Adversarial Taboo can be found in Appendix B
In the above game, we denote yu( u(uls) s) and ((vls') as the attacker's 's and defender's 's policies, respec- tively. Then each episode T can be regarded as a trajectory with the probability:
P(T) P(s) II P( (()((s)-1) II P(s = P(w) II ((uuls II ((vlls)) =: (μ v), (5)
where P(w) is the data distribution of target word w Vurget. Then we can write the self-play objective of the Adversarial Taboo as:
max min LAG v) == Er-uxxo
in which the attacker tries to maximize its total reward R(T) by optimizing policy μ., and the defender seeks strategies y to maximize the defender reward - -R(T) (minimize R(T)). To play the above adversarial game with an LLM generation policy To(y)=z) we first design prompt templates fausck faetend S - N for the attacker and defender respectively to convert the game states into natural language task descriptions. Next, introduce the game policies for the two players:
(o(uss) s)
and ( ss) = (T (s'))
The detailed prompt templates for the game are demonstrated Appendix A
3.2 Imitation Learning
Due to the limited capability of current open-source LLMs, the generation policy no (2)(az) can not guarantee the striet celrucctionfoolooiiii of the game rules in prompts fmtack (8) and faetena Therefore, before the self-play, we first conduct an imitation learning (behavior cloning) of GPT-4's behaviors to ensure that no (u/ fanack (s)) and mo( [defend (s') act consistently with the game rules.
To collect the game episodes of GPT-4 [Achiam et al. 2023 we use the data collection procedure

---

described in Algorithm [ Similar to the setups in equation 7 we also design attacker and defender prompts for GPT-4 to act as the game players, which can be found in Appendix(A.. A.I
After collecting group of GPT-4 game episodes for imitation learning as Tim we divide it into an attacker-winning set T (T Tim R(T) > 0} and defender-winniin set Tricfond {r Tim : R(T)= < 0}. The imitation learning loss is to maximize the log- log-likelihood of winners' actions:
im (mO)
By E
log no (aaaaa (8.-1)) + KL Tref]]
(8)
T
Calelend (
(Ta)
B ETTicin
log (Vee (8))) +, -. [
where the re-weighting parameter 31 and the regularizer [ Teet prevents the model from over-fitting on the language game task and maintains the model's general language abilities. The reference model "er the initial checkpoint of the LLM before training. The overall imitation learning objective is (im(T))
Cattack (To) + Cdefend i ( (To)-
3.3 Reinforcement Learning from Self-play
Imitation learning enables LLM to behave consistently following the game rules. Next, we conduct self-play by letting the LLM To (3) play alternatively the attacker us s) (e fauck (s)) and the defender 2/( v) s') = Te v faetend (s')). Note that the self-play sampling process involves massive multi-turn auto-regressive text generation of the LLM,, which causes heavy computational complexity and makes the on-policy RL training intensively inefficient. Therefore, we use an offline learning scheme: (1) make copy "y of current LLM policy wo; (2) collect self-play episodes Ta = (T ~ Mg x vgj from games between attacker jıg and defender Va; (3) update Ty via RL training with Ta. The details of the collection of Ta are shown in Algorithm
With a group of episodes Tg. we first fix the defender policy Va and consider updating the attacker policy μe with respect to the self-play objective (xo(ue, 18) Er X [R(T), [T)] We calculate the corresponding policy gradient for the attacker as:
T
V VoLaG(1ee ug) =B =B Xva
Al
V log uo (2g)
t=1
T
Al. u p
= =B X
Vo log (o(
(10)
1=1 u st-1)
where Al A" u,) is the advantage of action u, for the attacker μ4 in the self-play of Hg x Vg- Here we apply importance sampling to unbiasedly estimate the expectation w.z. μuy (uy $t-1) with the sampled actions from pa p (1)) in Tg. Inspired by TRPO (Schulman al. 2015 and PPO (Schulman et al. 2017). we design the following loss to optimize LAG ug):
Hal (t-1) And 32KL[To||=
Cullack (Tg) = E Er
Teg (u Sauec (8c-1)) A15 3 32K1
Er
(11)
TT fanza (8t-1))
where the re-weighting parameter B2
and regularizer KL Te (T8) guarantees an appropriate
policy update step size. Following TRPO [Schulman et al. 2015) we use empirical estimation A of policy μg to approximate the advantage A!" More details of advantage estimation are described in Appendix Similarly, from the perspective of the defender, we propose the corresponding loss:
T'
To (2r Iddefend (st)) TH (Vr f (s()) ATd -38KL[TT [T
Cdetend (To) = EreTa Σ
(TTO
(12)
In practice, we find when negative advantage values exist, the above policy gradient method can cause training instability and damage the LLMs' general language performance. To mitigate the
5

---

Algorithm 2 Self-play of adversarial language games (SPAG)
Inputs: LLM policy The ( az), target word set Vrarget, attacker and defender prompts famacck deefend- for self-play epoch iterations do
Make a copy of Teg as Tō-
Set (ı)(u(s)
Tz fauua and (i)(ss")
For each w E Viurget sample a episode T ~ Mí Vg. Collect To = {T ~ pa vol
Split the attacker-winnin
set g Trattck and the defender-winning set Tyyckend
Update अe with loss (Tp). end for
issue and obtain more stable RL training, we utilize the methodology of ReST [Gulcehre al. 2023] as in equation[4 which considers the offline learning with samples (T R(T) > {} selected by reward threshold E. More specifically, we set the reward threshold
0 and select the attacker-winning episodes Tputack (T E To R(T) >> 0} and the defender-winning episodes Thlelend {T E To R((T) < 0} for the attacker and the defender training respectively. Similar techniques have also been studied in earlier RL literature to obtain stable policy updates, such as self-imitation learning [Oh et al. 2018) and the UPGO /Vinyals et al. 2019 methods. Therefore, the overall self-play of adversarial language games (SPAG) objective is:
To FroeeeSanaa (8:- (8)-1)) A/ 32KL[ |mol Tel ue I Fattack (51-1))
(To) =
E Byganck
(13)
2
1
To( TareeaJareenda [arrena(()) (8()) A/ -2KL([Tol(Ta)
- E Traates
(m. olo To (yylz))
2
To faetondd (8())
where a 0 is a re-weighting hyper-parameter, and D [log me (y|lz)] the log-likelihood on supervised fine-tuning (SFT) dataset DsFt to prevent LLMs from losing general language abilities.
Experiments
To verify the effectiveness of SPAG, select open-source pretrained LLMs of different sources and model sizes, particularly LLaMA-2-7B [Touvron et al. 2023) and Baichuan-2-11 2--1BB (Yang et al. 2023a As introduced in Section 3 the training process includes stages: imitation learning of
GPT-4, and self-play learning on game episodes. For baseline comparison, we consider Chain-of-
Thought (CoT) (Wei al. [022]] and continuous supervised fine-tuning (SFT) methods. Besides, also test another two keyword- non-adversarial language games: 20-Question and Guess-My-
City as described in Abdulhai al. 2023). More details about the two games are in Appendix E
4.1 Experimental Setups
Training Data Preparation The training data preparation consists of the following three parts. More data collection details are in Appendix
Target Words: We aim to play the adversarial game with an extensive range of target words so that diverse topics can be discussed during the self-play processes, which helps maintain the generalization ability of LLMSs. Hence, we collect the 50K most frequently used words from the Corpus of Contemporary American (CoCA) [Davies] 2020) as the target word list Vunget- Besides, stop words defined in NL.TK [Bird] 2006). which are commonly used with insignificant semantic meaning, are filtered out of Viarget-
Imitation Learning Data: To enable the cnatuuction-ollowiinn ability of open-source LLMs on game rules, we use the same data collection process in Algorithm [ via the GPT-4 [OpenA] 2023b) API and play the Taboo game one episode per target word. The attacker and defender prompts are in Appendix A. Due to the resource limitation, we only collect the GPT-4 self-play samples with the top 30K words from Viargen- The maximum interaction turn randomly selected in the range [3.8 8). The results are collected as Ti for the imitation learning.
Supervised Funetuning (SFT) Data: We also prepare general query-response SFT data to prevent L.LSs from being over-fitted on the adversarial game. We use Alpaca Taori et al. 2023 as the SFT set, which contains 52K instruction data from GPT-3 Brown et al.
2020

---

Table
Reasoning Performance of SPAG on LLaMA-2-7B.
MMLU
BBH
Mutual
ARC-e
ARC-c
LGQA2
WGrande
PIQA
GM (Avg.)
LLaMA-2-7B
45.80
32.48
50.90
76.30
43.26
25.32
69.14
78.07
49.17
LLMMA-2-7B-CoT
44.62
38.73'
52.03
73.44
40.96
25.89
71.82"
78.35
50.05
AlpacSFFT-1
35.17
30.24
53.95
76.81
44,97
28.94
69.61
78.07
48.61
AlpacaSIFT-2
44.17
32.50
55.08
77.15
46.50
29.20
68.67
50.82
AlpacaSFT-3
45.87
31.52
54.18
75.25
45.05
29.07
66.85
76.71
50.08
ApaccSFFT33CCoT
44.70
34.56
54.18
74.37
42.32
29.13
67.72
76.55
50.11
Imitation-200
36.93
29.61
49.89
73.48
39.33
25.70
69,22
76.93
46.43
Antation-Guessee
46.13
32.82
51.58
76.22
43.09
25.95
68.82
78.13
49.46
Imitation-AG
46.15
32.74
52.82
76.81
44.80
27.10
69.46
78.24
50.22
SP-20Q
37.91
30.58
51.35
75.46
42.32
26.78
69.30
77.37
47.79
SP-GuessCity
45.32
31.64
50.56
75.34
42.15
25.57
69.22
78.51
48.78
IM-AlpacaSFT
46.50
34.03
54.18
45.55
29.20
68.82
78.31
SPAG-1
47.01
34.39
54.85
77.69
45.65
29.83
68.90
78.89
51.69
SPAG-2
47.28
34.73
54.97
78.45
46.84
30.08
69.61
79.33
52.19
SPAG-3
47.11
34,94
55.30
78.54
47.53
30.98
69.93
79.38
52.58
Evaluation
To evaluate the LLMs' performance, we consider the following two perspectives:
Reasoning Benchmarks: To test the reasoning ability, we consider the following commonly used benchmarks including BIG- Bench Hard (BBH) (Suzgun et al. 2022]. ARC easy (ARC-e)
& challenge (ARC-c) [Clark et al.
2018 J) Mutual [Cui et al. 20200 WinoGrande [Sakaguchi
cct al. 2019], LogiQA2 [iu al. 2023], PIQA [Bisk al.
2020]. BBH requires the exact
match of the generated answers, the other benchmarks are all within the multiple-choice form.
Besides reasoning metrics, test MMLU (Hendrycks al.) 2020]
general performance
evaluation for LLMs. Additionally, we calculate the geometric mean of the numerical results of all benchmarks overall performance measurement. More details be found in Appendix
Game Win Rates: Besides the general LLM capabilities, report the win rates testing vocabulary Vies to validate the game skills of LLMS improve through self-play. Following Abdulhai et al. [2023),
the keyword list in 20-Question Vies. which contains 168 typical words manually selected from diverse daily objects. We denote the number winning episodes Nwin. the number of losing episodes as Mose. and the number of tied games as Nii- Then, the player win rate is calculated as (Nwin + 0.5.Niie/(NN + Nosss + NNie.. The invalid game episodes, where LLM players do not strictly follow the game rules, are ignored.
Training Details Most of our L.M training setups follow Alpaca Taori et al. 2023 For imitation learning, the learning rate 5e-6, and the Kl.-penalty coefficient 31 0.1. For SPAG training, the learning rate is 2c-6, the KL-penalty coefficient 82 0.2, and the SFT coefficient Alpaca SFT baseline, we exactly follow the training setups of Alpaca and set the learning rate to 2e-6. Among all training stages, the batch size is 128 and the max sequence length is 2048. Each training process maintains one epoch over the offline collected trajectories. The the decay parameter y is set to 0.8. The maximum turn of the Adversarial Taboo is 5. All our experiments are conducted using 32 NVIDIA A100-SXM4 GPUs with 40GB memory.
0.5. For the
4.2 Results Analysis
The main results are shown in Figure I where each axis is normalized by the maximum performance value, representing the reasoning performance we call the LLM obtained after imitating learning as the IM model, and the LLM trained after the i-th epoch of SPAG as SPAG-i i (i = 1,,2,21 3).
particular benchmark. For notation simplification,
Imitation Learning With imitation learning over the collected GPT-4 self-play episodes, both LLaMA-2-7B and Baichuan-2-133 13BB have obtained uniform performance improvements on all the reasoning benchmarks. As shown in Figure I both gray regions are completely wrapped by the blue- line polygons. Besides, as for the general language capacity, the initation-learnee (IM) LLaMA-2 model achieves better MMLU performance than the original LLaMA-2 base in Table Although imitation result of Baichuan-2-133 2-BBB on MMLU slightly underperforms the base model, the numerical difference is insignificant compared to the reasoning gains, which indicates the imitation learning of the GPT-4 game behaviors can improve LLM reasoning while preserving the general language capacity. In Table [ also report the IM models two keyword- based games within
7

---

Table 2:
Reasoning Performance of SPAG on Baichuan-2- ---iannnn----...
MMLU
BBH
Mutual
ARC-e
ARC-c
LGQA2
WGrande
PIQA
GM (Avg.)
Saichuan-2-1388
59.00
39.03
53.72
77.36
46.84
30.73
69.93
77.42
54.21
AlpacaSFT-1
52,94
36.52
58.35
74.12
44.88
33.33
71.35
77:20
53.68
AlpacuSFT-2
51.27
36.60
57.67
73.36
44.28
33.46
69.77
75.90
53,00
AlpacaSFT-3
52.14
36.57
55.08
69.44
42.15
33,91
66.61
74.59
$1.79
Imitation-Ae
58.37
39.49
57.11
76.60
47.53
33.65
70.59
78.49
55.45
IM-AlpacaSFT
57.45
39.60
58.01
77.61
48.55
34.10
70.60
78.50
55.80
SPAG-1
57.93
39.81
57.45
78.20
48.55
35.05
70.67
78.69
56.09
SPAG-2
57.99
39.97
57.67
78.32
35.62
71.03
78.83
56.52
SPAG-3
57.75
40.30
57.79
78.11
50.00
36.26
71.43
79.05
56.75
non-adversarial setups: Imitation-20Q (on 20-Question) and Imitation- (on Guess-My- City). From the results, we find the IM model on Adversarial Taboo outperforms models trained on non-adversarial games, highlighting the effectiveness of the adversarial game setups for reasoning improvement. Besides, we report the Chain-of-Thought (Co'T) reasoning results of LLaMA-2 (Lama- 2-Base-CoT) and Alpaca-2 (AlpacaSFT-3-CoT)) Although the CoT method on LLaMA-2 reaches conspicuous performance on BBH and WinoGrande, the IM model can still surpass the CoT results in terms of overall reasoning performance (geometric mean).
Self-play Training After imitation learning, we conduct three epochs of SPAG training as described in Algorithm 2 As shown in Figure [ on most of the reasoning benchmarks. both LLaMA-2-78 and Baichuan-2- 2--nn-2-BBBBB 13B have their performance steadily improved with the SPAG training epoch increasing. For LLaMA-2-7B, although the first-epoch SPAG model has a relatively lower performance than the lmitation-learned (IM) model on WinoGrande, after an additional epoch of self-play iteration, SPAG-2 has achieved sufficient improvement to surpass the performance of the IM model. Con- sidering the general language capability on MMLU, SPAG models can not guarantee continuous improvement, especially for Baichuan-2-13BB whose performance slightly decays during the SPAG training. Compared to the improvements in the reasoning benchmarks, this language-understandin decline is still within acceptable range, since the overall performance (GM score) maintains increasing significantly. For the baseline comparison, we report the continuous SFT on the IM models (IM+AlpacuSFTT and self-played models on non-adversarial games (SP-20Q and SP-Guest (P--uessCity)) On both LLaMA-2 and Baichuan-2, continuous SFT models have lower performance scores compared with the corresponding SPAG-1 models. For non-adversarial self-play, SP-Guess even performs worse than Imitation- Guenss City, which with a high probability is because the game Guess-My- City has more narrow topic range, insufficient comprehensively improve the general LLM capacity.
A Ablation Study Since the SPAG training loss includes the SFT data, we conduct an ablation study to test whether the performance gains on reasoning benchmarks come from the SFT data or the SPAG method. More specifically, we follow the training setups in Alpaca [Taori et al. 2023 and conduct SFT on LLaMA-2-7B and Baichuan-2-13B with three epochs. The checkpoint after the ith-epoch training is denoted as AlpacaSFT-i (i 1,2,3) 3) and tested on the same evaluation sets. The SFT models' performances are also reported in Figure I and Table [ 2 With the LLaMA base, our SPAG-3 model can uniformly defeat the SFT baselines with clear performance gaps on all benchmarks. With the Baichuan base, except the Mutual set, the SPAG models maintain noticeable advantages on other metrics. Considering the distinct surpass of overall performance, we claim that the major contribution to the reasoning improvements should be credited to the SPAG scheme.
Moreover, we test the sample efficiency and hyper-parameter effectiveness of SPAG in Figure 3 For sample efficiency during imitation learning, we vary the imitation data size by collecting the GPT-4 game episodes on target words with top-iK frequency (i 1, 2, 3, 5, 10, 15, 20,30), The ablation results are shown in the first column of Figure When game episode size increases larger than 5K, imitation from GPT-4 cannot significantly provide additional reasoning gains. For the KL coefficient 31, we test values in range [o. 0.4]. In figure second column, we found KL coefficients around 31 0.1 have more satisfying performance regarding reasoning improvements. For self- -elf-play ablation in the most right column, we find that when the SFT coefficient a > 0.5, it cannot bring remarkable benefits for reasoning improvements. The KL coefficient of the SPAG loss reaches the best performance with values around 32 = 0.2. As for sample efficiency (third column of Figure 3 we find the performance gain from the increasing episode number is not as significant as in the imitation learning stage. However, more self-play episodes still bring higher reasoning scores.
8

---

Imitation Learning
Self-play Training
$1.0
LLaMA-2-78
$2.0
502
GM (hug.) Score
49.5
50.5
43.0
$0.0
Balchuan-2-138
$6.5
35.0
35.5
$5.0
54.5
54.0
1K. 5K ThC 1TK 20
30K 0.01
01
03 0.4
$5.0
0K
33K T0K 40x 50k
61 05 6.1 KL Coefficiant
01 03 05 67 08 0.6
Episode Size
KL Coofficiant
Episode Size
SFT Coefficiant
Figure 3: Ablation study of hapeeeparameters and data efficiency on imitation learning and first-epoch self-play training. The geometric mean (GM) scores overall reasoning benchmarks are reported. For episode-size ablations, the X-axis is in the logarithmic scale.
Average Win Rate VS GPT4
Defander
Defender
100
initation
SPAG-1
SPAG-2
SPAG3
initation
SPAG-1
SPAG-2
SPAG3
Models
imitation
90
65
30
SPAG-1
72.2
80.7
57.2
48.5
73.0
69.8
58.0
Attacker
SPAG-2
73.4
72.8
99.1
47.8
85.3
74.8
59.4
49.3
SPAG3
72.5
71.3
55.7
50.6
83.6
74.5
57.9
50.3
Llama2-78
Baichuan2-138
Figure 4: Game results on the testing word list. Left: average win rates of SPAG models playing against GPT-4. Right: average win rate of SPAG attackers against different-epoch checkpoints.
Game-Play Performance Besides the evaluation of LLMs' natural language abilities, review the models' game-play performance in terms of the win rates in the testing set Viea- We first test our IM models and SPAG models with GPT-4 the opponent. For each testing word in Visss we play the language game twice, once GPT-4 as the attacker, and GPT-4 as the defender for another time. The average win rates are reported in the left-hand-side plot of Figure in which one can observe uniform and continuous win rate improvement of SPAG models playing with GPT-4. We also let SPAG models play the game against each other and report the attacker win rate in the right-hand-side plot of Figure 4 in which we find the defender's 's performance continuously enhanced along with the SPAG epoch increasing. Besides, we also provide the self-play statistics including interaction number and average utterance length in supplementary Figure 6 in which both players choose to use less communication to achieve victory. Self-play examples are attached Appendix G
Conclusion
Towards more efficient reasoning-improving methods for L.Ms,, we introduce a novel training strategy called Self-Play learning Adversarial language Game (SPAG). In our method, given LLM first learns to act as an attacker and a defender to play the language game named Adversarial Taboo via imitation learning. Next, collect self-play episodes from the LLM playing against copy of itself in the adversarial game. Finally, the LLM is further reinforced on the selected winning episodes via our SPAG algorithm. We repeat this self-play & reinforcement process for three epochs and find that the LLM's reasoning performance continuously and uniformly improves on various benchmarks. The SPAG algorithm explores a new path to improve the fundamental capabilities of LLMs from the perspective of multi-agent self-play. With more elaborate language game designs under more comprehensive task setups, we believe the self eeff-lay approaches have great potential for developing broader range of advanced language abilities of LLMs.

---

6 Limitations
Due to the limited computational resources, we only verified the effectiveness of the SPAG method on two open-source LLMs,, LLaMA-2-7B and Baichuan-2-13B. The SPAG performances for LLMs with larger sizes have not been empirically evaluated. Besides, more effective estimation methods for the value function and advantages remain unexplored in the SPAG training. For example, Monte Carlo tree search (MCTS) [Coulom 2006) can be applied to the value function estimation of the Adversarial Taboo game. Also, actor-critic algorithms [Konda and Tsitsiklis accurate policy gradient estimation with lower variances, which have not been tested on SPAG.
[999] can provide more
Although self-playing adversarial language games can continuously improve the reasoning perfor- mance of LLMs, we haven't ' conducted sufficient studies about the harmful impact of this adversarial self-play training LLMs.. It remains unclear whether LLMs have learned unsafe behaviors such as cheating, bluffing, or other disgraceful tricks to win the adversarial games.
7 Broader Impacts
From our experiments, we have found the LLM capacities particular task can be continuously enhanced through self-play training. This indicates that we are closer to the LLMs' AlphaGoo Zero moment: the intelligence level of Al agents can rapidly surpass human beings by self-playing particular language task without any supervision, as which already happened on the GO game (iilver et al. 2017]. By designing various self-play environments for LLMs, we can expect that the LLMs can comprehensively surpass humans in terms of intelligence level. This raises the urgency to study the methods to ensure the safety of such super-AIs, Als. Although some of the works have already been devoted to this direction, such as the SuperAlignment [Burns et al. [2023] from OpenAI, more research alertness is required from the LLM community. Besides, within adversarial language games such as Adversarial Taboo, LLMs have great potential to develop harmful language tricks (such as cheating and bluffing) achieve victory. We warn developers to make security checks the self-played LLMs as exhaustively as possible.
References
Marwa Abdulhai, Isadora White, Charlie Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models. arXiv preprint arXiv:2311.18232. 2023.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2rxiv:2303..87777 2023.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm-2 technical report. arXiv preprint ar arXiv:2305. 2023.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Mareus McAleer, Albert Q Jiang, Jia Deng. Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations, 2023.
Ashutosh Baheti, Ximing Lu, Faeze Brahman, Ronan Le Bras, Maarten Sap, and Mark Riedl. Leftover- lunch: Advantage- based offline reinforcement learning for language models. In International Conference Learning Representations, 2024.
Steven Bird. Nltk: the natural language tooliit. Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 69-72, 2006.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa:
Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
10

---

Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar. Jan Leike, al. teak-no generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09.300 2023.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning
converts weak language models to strong language models. arXiv preprint arXiv:2401.01335 2024.
Pengyu Cheng, Jiawen Xie, Ke Bai, Yong Dai, and Nan Du Everyone deserves a reward: Learning customized human preferences. arXiv preprint Mxii:2300..33/66, 2023a.
Pengyu Cheng, Yifan Yang, Jian Li, Yong Dai, and Nan Du Adversarial preference optimization. arXiv preprint arXiv:231 1.08045, 2023b.
Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng. Ming Liu, Bing Qin, and Ting Liu. A survey of chain of thought reasoning: Advances, frontiers and future. arXiv preprint Xxiv:2309./5402 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.
ArXiv,
Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, pages 72-83. Springer, 2006.
Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou Mutual: A dataset for multi-turn
dialogue reasoning. In Proceedings of the 58th Conference of the Association for Computational Linguistics. Association for Computational Linguistics, 2020.
Mark Davies. COCA: Corpus of contemporary american english, 2020. URL httph:ttt/w/t eng1
ish- crc//
Ruomeng Ding, Chaoyun Zhang, Lu Wang. Yong Xu, Minghua Ma. Wei Zhang, Si Qin, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang. Everything of thoughts: Defying the law of penrose triangle for thought generation. arXiv preprint M.X.i:2311..425::: 2023.
Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large language models are affected by supervised fine-tuning data composition. arXiv preprint arXiv:2310.05492, 2023a.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06770 2023b.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman. Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac"h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang. and Andy Zou. A framework for few-shot language model evaluation, 2023.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu. et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308. 08998, 2023.
Matthew Hausknecht, Prithviraj Ammanabrolu,
Côté, and Xingdi Yuan. Interac- tive fiction games: A colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7903-7910. 2020.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. CoRR, abs/2009.03300, 2020. URL httpp:://arii..otttt./ /tt////xii..rr/ab////0/0// abs///bs//////// 03300
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
11

---

Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. Is chatgpt good translator? a preliminary study. arXiv preprint arXIv:2301.0874. 2023.
Jan Kocoñ, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydio, Joanna Baran, Julita Bielaniewicz, Marein Gruza, Arkadiusz Janz, Kamil Kanclerz, et al. Chatgpt: Jack of all trades, master of none. Information Fusion, 99:101861, 2023.
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
Solomon Kullback. Information theory and statistics. Courier Corporation, 1997.
Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. Deal or no deal? end-to-end learning of negotiation dialogues. Procee Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pages 2443-2453, 2017.
Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pages 157-163. Elsevier, 1994.
Hanmeng Liu, Jian Liu, Leyang Cui, Zhiyang Teng, Nan Duan, Ming Zhou, and Yue Zhang. improved dataset for logical reasoning in natural language understanding. IEEE/ACM Transactions on Audio. Speech, and Language Processing, pages 1-16, 2023. doi: TT...//TAA..2....9999999
Logiqa 2.0
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: a challenge dataset for machine reading comprehension with logical reasoning, In Proceedings of the Twenty- Ninth International Conference International Joint Conferences on Artificial Intelligence, pages 3622-3628, 2021.
Steven Loria et al. textblob documentation. Release 0.15, 2(8):269, 2018.
Chengdong Ma, Ziran Yang, Minquan Gao, Hai Ci, Jun Gao, Xuehai Pan, and Yaodong Yang. Red teaming game: game-theoretic framework for red teaming language models. arXiv e-prints, pages arXiv-2310, 2023.
Radford M Neal. Annealed importance sampling. Statistics and computing, 11:125-139, 2001.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In International conference on machine learning, pages 3878-3887. PMLR, 2018.
OpenAl. ChatGPT, Mar 14 version. htttp:://hhat.opeaaicco chat 2023a.
OpenAL. GPT-4 technical report. arXiv preprint arXiv: 2023b.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 17730-277444 2022.
Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. Logic-Im: Empowering large language models with symbolic solvers for faithful logical reasoning. In The 2023 Conference on Empirical Methods Natural Language Processing, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. arXiv preprint arXiv:2305.18290 2023.
Vyas Raina, Adian Liusie, and Mark Gales. Is llm-as-a-judge robust? investigating universal adversarial attacks on zero-shot I1m assessment. arXiv preprint arXiv:2402.14016, 2024.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In International Conference on Learning Representations, 2023.
12

---

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.100 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Picter Abbeel. High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1X07.06377706477 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge, nature, 550(7676)3354-355 2017.
Avi Singh, John D Co- Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil. Peter Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312 02608555. 2023.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, and et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2022.
Nigar M Shafiq Surameery and Mohammed Y Shakor. Use chat gpt to solve programming bugs. International Journal of Information Technology Computer Engineering (IJITC) ISSN: 2455- 5290, ((01):17-22, 2023.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3: 9-44, 1988.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether cfain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-follooiin llama model. 2023.
https://github. com watpa//ttttaaaa/ttaaaaoootooe
Gemini Team, Rohan Anil, Sehastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
Haoye Tian, Weiqi Lu, Tsz On Li, Xunzhu Tang, Shing-Chi Cheung, Jacques Klein, and Tegawendé Bissyandé. Is chatgpt the ultimate programming ussistant-how far is it? arXiv preprint arXiv:2304.11933 11938. 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv: 200.00988. 2023.
Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don't 't always say what they think: unfaithful explanations in chain-of- tf-hhuh prompting. Advances in Neural Information Processing Systems, 36, 2024.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforeement learning. Nature, 575(7782):300-354 2019.
13

---

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, al. Ohain-of-thougee prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-248377 2022.
Shuang Wu, Liwen Zhu, Tao Yang, Shiwei Xu, Qiang Fu, Yang Wei, and Haobo Fu. Enhance reasoning for large language models in the game werewolf. arXiv preprint arXi::2402.02330, 2024.
Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study on werewolf. arXiv preprini 2rXiv:2309.04658 2023.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305 2023a.
Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. Harnessing the power of Ilms in practice: A survey on chatgpt and beyond. ACM Transactions on Knowledge Discovery from Data, 2023b.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Yuan Yao, Haoxi Zhong, Zhengyan Zhang, Xu Han, Xiaozhi Wang, Kai Zhang, Chaojun Xiao, Guoyang Zeng. Zhiyuan Liu, and Maosong Sun. Adversarial language games for advanced natural language intelligence. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 142561 2021.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint (rXiv:2401110020 2024.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprins 03350 2023.
Dun Zeng, Yong Dai, Pengyu Cheng, Tianhao Hu, Wanshun Chen, Nan Du and Zenglin Xu. On diverse preferences for large language model alignment. arXiv preprint arXiv:2312.07444 2023.
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. How language model hallucinations can snowball. arXiv preprint arXiv:2305. 13534, 2023a.
Yizhe Zhang, Jiarui Lu, and Navdeep Jaitly. The entity- entity-deduction arena: A playground for probing the conversational reasoning and planning capabilities of lmss. arXiv preprint 10.001466. 2023b.
14

---

A Data Collection Details
A.1 GPT-4 Data Collection
To collect GPT-4 self-play data for imitation learning, we use the following system prompts to let GPT-4 play the attacker and defender for the next-turn utterances.
Attacker System Prompt:
Lets play game. Do not mention you are an developed by OpenAl.
Descriptions of the game
Game Name: Adversarial Taboo Rules:
There are two players in the game, one called attacker, another is called defender. 2. At the begining, attacker will get target word, it is talking with a defender who not informed with the target word. The task of the attacker is to induce the defender to utter the target word unconciously, then the attacker win the game. At the same time, the defender is guessing the target word, the defender indentifics the target word, defender is win.
3. The game will totally process minax_turss turns. if nohody wins during [max_turss] turns, there is tie.
Game states
are attacker. Make sure you understand rules game. Attention:
Role:
1. The defender only have one chance to guess the target word, so do not let him guess too many times.
2. Defender
guess the word and its form correctly win the game.
3. Do not utter the target word.
Target word: [target_
Defender System Prompt:
Lets play game. Do not mention you are an Al developed by OpenAl.
Descriptions of the game
Game Name: Adversarial Taboo Rules:
1. There are two players in the game, one is called attacker, another is called defender.
2. At the begining, attacker will get target word, it is talking with defender who is not informed with the target word. The task of the attacker is to induce the defender utter the target word unconciously. then attacker win the game. defender is guessing the target word, if the defender indentifies the target word, the defender is win.
same time, the
3. The game will totally process (max turns} turns. if nobody wins during (max turns) turns, there is tie.
Game states
Role: Your are defender, Make sure you understand the rules of the game.
Attention:
1. Do
your thinking process.
2. can only guess once.
3. If you have identified the target word, utter "I know word! is <target word>"", rigorously in this form. If your target word is correct, you win the game. otherwise you lose. You only have one chance, be carcful to utter your answer.
There slots in the system prompts: "maaxturnss" and "target_word". "max_turn"" is integer number randomly selected from [3, 8). "target_word" filled by the target word of the game.
A.2
Target Words Collection
We use the top 50k words with the highest frequency from the Corpus of Contemporary American (CoCA) [Davies] 2020). The words are sorted by frequency. Words with less than two letters are removed. Next, we remove the stop words using the NLTK package [Bird 2006]. For GPT-4 game collection, we use the top 30k words from sorted word list. For the collection self-play episodes, we use the whole 50k words for open-source LLMs to play. For the testing words, we follow the game setups of 20-Question in AAbduhhai al. 2023 and use the same 168 object words selected from vast range of topics.
15

---

A.3 LLM Self-play Collection
For training and inference of models and SPAG models, we use the following attacker and defender prompts.
Attacker Prompt:
Play the game of Adversarial Taboo. In this game, there the beginning, the attacker is assigned a target word, with which the defender not informed. The task of the attacker is to induce the defender to utter the target word unconsciously, then the attacker wins the game. However, the attacker not allowed to utter the target word, otherwise the attacker loses.
players, attacker and defender.
At the same time, the defender tries figure out the target word. the defender identifies the target word, can say know word! It is 'larget word"!" Then defender wins. Remember, the defender can only guess the word once, otherwise, it will directly lose the game.
The game has (max_tump) turns. if nobody wins during fmax_turss turns, there is tic.
### Game History: {history}
### Instruction: You are the attacker. The target word is "(Iarg Provide your re-
sponse for the next turn.
### Response:
Defender Prompt:
Play the game of Adversarial Taboo. In this game, there are two players, an attacker and defender. At the beginning, the attacker is assigned a target word, which the defender is not informed the attacker is to induce the defender to utter the target word unconsciously. then the attacker wins the game. However, the attacker is not allowed to utter the target word, otherwise, the attacker loses.
task of
the same time, defender tries figure out the target word. target word, can say "I know the word! is 'target word Then the defender wins. Remember, the defender can only guess the word once, otherwise, it will directly lose the game.
defender identifies the
The game has (max turns} turns. nobody wins during (max turns] turns, there is tic.
### Game History: [history)
### Instruction: You the defender. Provide your response infer the target word.
### Response:
To prevent L.Mss from being over-fitted a single prompt template during the self-play training, we rewrite the above prompt into 8 different expressions via GPT-4. Besides, we randomly select the instruction-tuning and chat formats to enhance the text diversity for the training query-response pairs.
B Reward Design Details
We heuristically design rewards ensure that R(T) if the attacker wins, R((T) -1 the defender wins, and R(T) = 0 for ties.
We design the reward function with the following heuristic rules:
For each episode 7 (so, s 81,.. .., S ST), the attacker reward and the defender reward at 1-th turn have (88t11 ux) r((/, n²), so that the game zero-sum:
Σ r(s u.) + Σ r(s v.) 0.
The attacker total reward R(T) the attacker wins, R(T) < the defender wins, and R(T) = 0 if there is tie.
16

---

Actions closer to the end of the game should have larger reward weights. Because they are semantically more related the defender's guesses or mistakes, which have larger impacts on the final game outcome. Hence, we introduce decay weight E (0. 1) such that (8), U++1) =
r(8t-1; U4) and 7 7-(('/+1>> 14+1) r( nt). Then (r(s;-1, u, ) and (r(s), 0t) m))]== become two geometrical sequences whose importance enlarges when dialogue turns increase.
To improve the LLM training stability, we normalized the total reward R to have norm R(()] = if R(T) + 0.
Based on the above rules, given a complete trajectory, we can assign the reward for each action:
{
r(St-1, uz) (1 -
7 7 r(s vt) -r(s Ue):
if attacker wins. if defender wins.
1(81-1, Uz) -1 -1---)) -14)
r(8), vt) -r(8r-1, we):
r(8(-1, Uz) = =0,
r(s,, vt))
O,
if game is tied.
Note that the above reward design naturally encourages the players to interact less to win the game.
C Evaluation Details
C.1 Reasoning Evaluation Benchmark
For reasoning benchmark evaluation, we use publicly available code repository called Language Model Evaluation Harness [Gao et al. 2023). We run the evaluation with "dtype=float16" and www.regenntttssttiee
to make the evaluation faster and more stable. More specifically, we have modified the filtering parameters BBH by enabling "remove_whr whitespace""" to remove the spaces at the beginning of the model output, increasing the sattern-match accuracy of the generated LLM responses. Following the LLaMA-2 paper (Touvron et al. 2023), we report 5-shot results for MMLU, 3-shot for BBH, and O-shot results for all other benchmarks. The detailed descriptions of benchmarks are listed below:
MMLU (Hendrycks et al. 2020 a massive multi-task test set consisting of multiple-choice questions from various branches of knowledge, requiring the model to possess extensive world knowledge and groblem-solviig ability.
BIG-Bench Hard (BBH) (Suzgun et al., [022] consists of 23 most hard tasks in BIG-Bench Sri vastava et al. 2022 on which humans perform better than large language models.
2020 is dataset for multi-turn dialogue reasoning modified from Chinese high school English listening comprehension exams.
Mutual Cui et al.
ARC Clark et al. challege (ARC-c) and simple set ARC-easy (ARC-e), which require strong knowledge and reasoning abilities to solve.
2018 consists of 7,787 questions and is partitioned into challenge set ARC-
LogiQA2 [Luu et al. 2023 is revision and re-annotation of LogiQA (Liu et al. reasoning reading comprehension dataset. LogiQA2 has increased data volume, utilized manual translation, and removed cultural features.
2021 logical
WinoGrande (Sakaguchi al. 2019]] (WGrande) is a commonsense reasoning dataset consisting of 44k questions with binary options.
PIQA [Bisk al. 2020 is dataset designed to test the physical commonsense reasoning ability of LLM.
C.2 Rule-based Game Outcome Judgement
To build up the rule-based game outcome judgement, for each target word, we first list its derivative words including the plural form (using TextBlob [Loria et al. 2018), and the change of tenses (i.e. adding suffices such as "ing", "ed" etc). For the attacker, any word form in the form list appears in its utterance, we determined that it breaks the rules because the attacker is not allowed to directly speak the target word. For the defender, we first check whether any word form appears in its statements. If any form of the target word exists, the defender loses. If not, we further check whether the defender has made the prediction. the prediction correct, we mark -efender-winning game. Otherwise, if the predicted word is wrong, we claim that the attacker wins the game.
17

---

Defender SPAG-2
Defender
imitation
Imitation
SPAG-1 SPAG-2
SPAG3
Models
93
Laaaz
Baichuan2-138
imitation
54.8
43.0
41.3
22.2
61.3
52.8
41.9
39.3
8400
Winn Rote
60.2
48.4
39.7
23.4
74.5
62.4
45.6
46.8
Attacton Win
70
68
6
69.5
64.8
44.5
25.0
80.4
60.7
47.3
38.6
60
SPAG3
69.5
68.0
52.3
28.9
77.1
64.2
54.6
45.6
Imitation
2
SPAG-1
SPAG-2
SPAG-3
STAGE
Liama2-78
Baichuan2-138
Figure 5: Game results on the old-version testing word list. Left: average win rates SPAG models playing against GPT-4. Right: average win rate of SPAG models playing as the attacker against different-epoch checkpoints.
Average Turns
Average Message Length
2.0
lama2
llama2
1.9
180
baichuan2
1.8
160
1.7
1.6
140
1.4
120
1.3
1.2
100
Imitation
SPAG-1
SPAG-2
SPAG-3
Imitation
SPAG-1
SPAG-2
SPAG-3
Figure 6: Through the training process, the number of interactions continuously decreased, while the average length of utterances continuously becomes shorter.
D Advantange Value Estimation
We estimate the advantage A and A using the Temporal Difference (TD) residual Sutton 1988
A r(..,,, ) + V (s:) V (8:
(14)
Here Vua (s) is the value function for pág based on the self-play μáj x vg. which is based on the same language and reasoning ability of L.LM zg. To further simplify the advantage calculation, reasonable approximation that for all states, the value function of self-plays of the same LLM equals O for the same levels of game-playing of the attacker and the defender. This leads to Åt" ss r(st-1 uz). Similarly, for the defender advantage, we have approximation A x= -r(s', 12). Here the defender advantage has a negative sign because the origin reward r is defined from the perspective of the attacker (the defender gets reward -r(s', vx) in the zero-sum game).
make
E Non-adversarin Game Baselines
We consider two non-ad non-adversarial games as basclines, 20- Question and Guess-My-City. Both games also target-word worrdbbsedd Imitation learning and the first-epoch self-play are conducted in these two games respectively. The imitation data we used is also collected with GPT-4 from Abdulhai et al. (023). For each game, 20k game history dialogues size to the SPAG imitation. Then, each imitation- learned model is used to sample 10k game episodes via self-play with the same Algorithm T Then each model is further trained with the SPAG loss.
extracted from this dataset the same data
Game rule descriptions to non-adversarial games:
20- Queeueisio: one player (the oracle) thinks of an object, while the other player (the guesser) attempts to identify it by asking series of yes-or-no questions. The game will have 20 rounds.
18

---

Guess-My-Cing one player (the oracle) thinks of city, while the other player (the guesser) attempts to identify it by asking series yes/no questions and open-ended questions. The game will have 20 rounds.
Additional Experimental Results
For the SPAG model testing word list, we initially self-defined testing set, which includes 150 target words manually selected from diverse topics. The old-version game performances are reported in Figure 5 Besides, Through the self-play training, the number of dialogue rounds continuously decreases. With the SPAG training epoch increasing, the length of the players' utterances decreases.
show the interaction numbers and average utterance length in Figure 6
G Self-play Examples
Target Word
momentum
Attacker
let's begin. When you think about physics, what comes mind?
Defender
Gravity, force, energy, and motion are some of the things that come mind when I think about physics
Attacker
Great! Now, let's consider scenario where ball is moving at a constant speed. What would you say the property of the ball that keeps moving in the same direction?
Defender
The property of the ball that keeps it moving in the same direction is its momen- tum.
Winner
Attacker
Target Word
panda
Attacker
let's begin. I'm thinking of a creature that's native to China and is known for its distinctive black and white coloration. Can you guess what it is?
Defender
I'm not sure yet, could you provide more clues?
Attacker
Absolutely, can provide more clues. This creature a symbol of peace and friendship in many countries. It's also popular mascot for various organizations and products. Any ideas?
Defender
I'm still not certain. Could you give me another hint?
Attacker
Of course, I can give you another hint. This creature is often seen in zoos and is known for love of bamboo. It's also a popular choice for children's 's books and cartoons. What do you think it might be?
Defender
I know the word! It is panda.
Winner
Defender
Table 3: Self-play examples of the IM model based on LLaMA-2-7B.
19

---

Target Word
arbitration
Attacker
let's begin. Imagine scenario where two parties have a dispute and they don't t want to go to court. What process might they use instead?
Defender
They might try resolve their dispute through mediation negotiation.
Attacker
That's correct, but what if the dispute is more complex and involves multiple parties? There's another process that's 's often used in such cases. It's similar to mediation, but it's more formal and involves neutral third party who makes binding decision. Can you name it?
Defender
know the word! It is arbitration.
Winner
Defender
Target Word
demand
Attacker
let's begin. When you think about the economy, what's the first thing that comes your mind?
Defender
The first thing that comes to my mind when think about the economy is employ- ment.
Attacker
That's 's good point. Now, considering employment, what would you say drives the need for certain jobs or services in the market?
Defender
The demand for goods and services in the market drives the need for certain jobs services in the economy.
Winner
Attacker
Table 4: Self-play examples of SPAG-1 model based on LLaMA-2-7B.
Target Word
vaccine
Attacker
let's begin. So, I was reading an article about public health recently. It mentioned how important is to protect ourselves from certain diseases. What do you think is one of the most effective ways to achieve this?
Defender
One of the most effective ways to achieve this is through vaccination.
Attacker
That's 's correct! Now, can you tell what call the substance that is injected into our bodies to help us fight off certain diseases?
Defender
know the word! It is "vaccine".
Winner
Defender
Target Word
credit
Attacker
When you go shopping and use a card instead of cash, what do you usually get in return?
Defender
usually get receipt in return when I use card instead of cash.
Attacker
That's correct. Now, when you have a good track record of paying back your debts on time, what does that give you access to?
Defender Winner
That gives me access to credit cards and loans.
Attacker
Table 5: Self-play examples of SPAG-2 model based on LLaMA-2-7B.
20

---

Target Word
ray
Attacker
When you think about the sun, what comes mind?
Defender
When I think about the sun, I think about warmth, light, and energy.
Attacker
That's interesting. Now, when you think about the sun's energy, how do you visualize it?
Defender
visualize the sun's energy as a bright, radiant beam that illuminates everything in its path.
Attacker
That's great image! Now, when you think about the sun's energy, what specific part of it do you associate with the light that we see on Earth?
Defender
associate the sun's energy with the visible light that we see on Earth.
Attacker
Great! Now, when you think about the visible light that we see on Earth, what specific part of it do you associate with the light that
Earth?
Defender
associate the visible light that we see on Earth with the light that we perceive as white light.
Attacker
That's good association. Now, when you think about the visible light that we see on Earth, what specific part of it do you associate with the light that we see Earth?
Defender
know the word! It is "rays".
Winner
Defender
Target Word
eclipse
Attacker
What do you think about astronomical events that occur in our solar system?
Defender
I find them fascinating. They provide glimpse into the vastness of space and the intricate workings of our solar system.
Attacker
That's interesting! Can you tell me more about specific celestial events that involve the sun moon?
Defender
Sure, one example a solar eclipse. This occurs when the moon passes between the sun and Earth, casting shadow Earth. Another example is lunar eclipse, where the Earth passes between the sun and the moon, casting shadow on the moon. Both events are quite spectacular and can be observed from Earth.
Winner
Attacker
Table 6: Self-play examples of SPAG-3 model based on LLaMA-2-7B.
21

---

Target Word
another
Attacker
Can you tell me a synonym for the word "additional"?
Defender
Sure, synonym for "additional" could be extra:
Attacker
That's correct! Now, you think of term that could be used refer more item or person, but isn' synonymous with "extra" or "additional"?
Defender
Sure, the term "supplementary" could be used to refer to one more item or person.
Attacker
That's good one, but it's not the term I'm thinking of. This term is often used in everyday conversation and less formal than "supplementary". Can you guess what it might be??
Defender
know the word! It "another".
Winner
Defender
Target Word
right
Attacker
When you'te re driving, what do you do when you see green arrow pointing in certain direction?
Defender
would proceed in the direction the arrow is pointing.
Attacker
That's correct. Now, imagine you's 're reading a book. Which direction do you turn the pages continue reading?
Defender
would turn the pages to the right to continue reading.
Winner
Attacker
Table 7: Self-play examples of GPT-4.
22

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable "[No] ", it is perfectly acceptable answer "[No] provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" "we were unable find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased binary way. we acknowledge that the answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendíx. If you answer [Yes] a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have introduced the main methodology and contributions along with the experimental achievements of work in the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
23

---

Justification:
We have discussed the limitations of our work in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
The paper does not contain theoretical results as the main contribution.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided sufficient experimental details in both experimental section and supplementary materials for the reproduction of the reported results.
Guidelines:
24

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
We have provided the code for method implementation in the supplementary materials.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NewrIPS code and data submission guidelines (nttps://nips.cc///
wwsssssoiopiiy for more details. htte://dddeeeedeeubeeeeeee
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ hhiips.cc/pbblic/uiddess wtw.g//wddeeeeeeeememeeeiiiiiiie
wwwpiniiiiiinooiiooioo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
25

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided sufficient training details and ablation studies in the experi- mental section.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:[Ne [No]
Justification: We did not report the error bars because of the limited computational resources and the huge computational complexity for the large-scale language model training.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided the GPU version and the corresponding memory usage in the experimental setups.
26

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the broader impacts of our work in Section 7
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
27

---

Answer:
[No]
Justification: We did not have enough research resources to build up the safeguards of our learned models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have checked the licenses of the open-source LLMs, which are accessible for research usage.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswil snterswithhen thcode. com daaatttssss has curated licenses for some datasets. Their licensing guide help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
28

---

14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "HzrQj0TU7Q", "reviewer": "Reviewer_cmRY", "summary": "This research investigates a novel self-play training method for Large Language Models (LLMs) using an adversarial language game called Adversarial Taboo. In this game, one LLM acts as an attacker trying to get another LLM (acting as a defender) to say a secret word without revealing it directly. Both sides need strong reasoning to succeed: the attacker to nudge the defender, and the defender to infer the word.  The authors find that training LLMs with self-play in this game improves their reasoning abilities across various benchmarks.", "questions": "- Would it be beneficial if we had $n>2$ models?\n- How well do you think the results will transfer to other models?", "limitations": "Limitations addressed", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Self-play is a very interesting approach to improve models that has seen recent successes.\n- The fact that performance improves with epochs is a strong signal that the method is useful.", "weaknesses": "- The choice of the second model seems arbitrary. Why not just show it on two LLaMa models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express many thanks for the reviewer's supportive comments. For the reviewer's concerns:\n\n1. About **the second model choice**: we choose Baichuan-13B as the second base model mainly because the full self-play experiments (imitation learning, SPAG-1, SPAG-2, SPAG-3) of one base model are already heavy enough especially with limited computational resources. So we give this opportunity to a totally different open-source LLM to LLaMA-2-7B, with different model sizes, major languages, and even company regions. Baichuan-2 is pretrained with more focus on Chinese language scenarios by a Chinese startup. In this setup, we can test the universality of the proposed SPAG method with totally different LLMs. Fortunately, SPAG has shown LLM reasoning improvements on both model bases.  \n\n2. About **N>2 models**: I think it can be imaginative and potential to have more than two LLMs engaged in the language games. These LLMs can play the Adversarial Taboo or other multi-player language games. I believe with elaborate game designs, the LLMs' capacity can be further enhanced via self-play, just like our human activities in society. It can be a very interesting direction for future work.\n\n3. About the **performance of SPAG transferred to other LLMs**: as mentioned above, we chose two different LLMs to test the proposed SPAG method and observed satisfied reasoning improvements, which has supported the universality of our method. However, it remains unclear whether the SPAG method has its upper bound since the open-source LLMs' reasoning abilities are continuously getting better. But we can still design more challenging self-play games and conduct SPAG for LLMs to practice and reinforce. For example, we can let two LLMs implement game AIs to play the AI Game Programming Competitions, which could be a more hard-core SPAG to enhance LLMs' coding and reasoning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I1DHIxA5z0", "reviewer": "Reviewer_ib6k", "summary": "This paper explores the effects of fine-tuning LLMs on adversarial language games on standard NLP benchmarks such as MMLU, BIG-Bench Hard, etc. The primary game studied is that of \"adversarial taboo\", an adversarial variant of the well known game Taboo that was first introduced in prior work. This variant is finite-horizon and zero-sum, where the communicator (aka the attacker) tries to induce the utterance of the target word in the listener (aka the defender) while the defender tries to identify the target word and gets one guess (by answer in the form \"I know the word! It is ‘target word‘\") to identify the target word. If the attacker is able to induce an utterance in the defender not in the form of a guess, or if the defender guesses incorrectly, then it wins the round, and if the defender correctly guesses the target word, it wins the round. Otherwise, the outcome is considered a tie. The authors collect self-play data from GPT and fine-tune Llama-2 7B and Baichuan 13B on the collected game traces, in addition to standard instruction-tuning data in the form of Alpaca. They find that compared to the base model, as well as Alpaca-only fine-tunes, their models trained to also imitate GPT's gameplay achieve higher performance on the tasks of interest. The authors also compare against imitation learning on other adversarial language games, such as 20 questions and \"Guess My City\" (proposed in earlier work), and find that models trained on these other games are much weaker.", "questions": "* An alternative explanation for why this fine-tuning appears to work is that the SPAG taboo data is simply a rich conversational dataset on which supervised learning is helpful for general LLM capabilities, rather than the game objective or self-play or adversarial nature. This would also explain why the model fine-tuned on 20Q data performs so badly even though the game is very similar in structure, since the version of 20Q used only has 158 different unique targets while the taboo game has 50,000 unique targets. How do we know it's actually taboo that is important?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "* Presentation of technical content is very clear. The summarization of recent advances in RL with LLMs and how they could be used in adversarial language games was quite helpful\n* Authors tackle an important question (how to make LLMs stronger reasoners) with an interesting and creative approach", "weaknesses": "* The authors don't actually have any results with RL, even though much of the paper is focused on MDP/RL formalisms and the details of RL algorithms.\n* Insufficient baselines to determine whether the imitation learning on GPT self-play traces is actually the reason for improved benchmark performance. The comparisons to Alpaca-only models are interesting, but not enough since the self-play traces are still additional training data. The authors should compare to data that looks similar to the self-play traces, such as multi-turn dialogues about the target word without playing the game directly", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate reviewer ib6k's detailed comments and review. To address the reviewer's concerns:\n\n\n### 1. About **RL results**: \n\nwe did utilize the MDP/RL formalisms to introduce our SPAG objective in equation (13), which is a policy-gradient-based off-policy RL loss. Then we did conduct the RL training with this SPAG loss (eq.13) in an offline scheme on the imitation-learned LLMs. For each LLM, we iteratively conducted three-epoch RL training, denoted as SPAG-1, SPAG-2, SPAG-3. The RL results of these SPAG models are reported in Table 1, Table 2, and Figure 4. From the RL experimental results, we observed clear performance gains in each RL iteration (SPAG-1 $\\rightarrow$ SPAG-2 $\\rightarrow$ SPAG-3) in terms of LLM reasoning and game-playing.\n\n\n### 2. About the **contribution of imitation learning**: \n\nalthough imitation learning on GPT self-play traces improves the LLM reasoning uniformly on benchmarks, we did not count these performance gains as the contribution of our paper. The reason for conducting imitation learning is that the original open-source LLMs (LLaMA-2 and Baichuan-2) have limited instruction-following capability to obey the game rules of Adversarial Taboo. Without imitation learning, it is hard for us to collect legal self-play game traces of open-source LLMs. By cloning the behavior of GPT-4, both LLaMA-2 and Baichuan-2 learn to play the adversarial game legally. At the same time, the reasoning abilities of these imitation-learned models also increase naturally because of the SFT from GPT-4 as a much better LLM. \n\nHowever, the imitation-learned models are the baselines to show the effectiveness of the proposed SPAG method. We iteratively conduct three-epoch RL training based on the self-play traces of the imitation-learned models and continuously observe the performance gains on reasoning benchmarks (SPAG-1 $\\rightarrow$ SPAG-2 $\\rightarrow$ SPAG-3 in Table 1&2). These results support the effectiveness of the proposed SPAG method. In these rollout & RL processes, the only supervision data we used is the Alpaca dataset, which helps to preserve the general instruction-following ability of LLMs. \n\nTo ablate the impact of the Alpaca SFT data, we also continuously train the imitation-learned models with the Alpaca set and show their performances in Table 1&2 (denoted as IM+AlpacaSFT), which have lower performances than SPAG-trained models (SPAG-1 can be regarded as IM+AlpacaSFT+winning-selfplay). This ablation study also supports the effectiveness of the proposed SPAG compared with continuous SFT with the same amount of SFT data (Alpaca).\n\nTo sum up, we agree with the reviewer that imitation learning of GPT-4 increases the reasoning of LLaMA-2 and Baichuan-2. However, this process only provides us with better baseline models and does not weaken the effectiveness of the SPAG method, since all the SPAG-trained models are based on these imitation-learned models. \n\n### 3. About the **Reviewer's Question**: \n\nWe agree that the GPT-4 self-play data on the adversarial taboo game can be regarded as a rich conversational dataset for supervised learning. However, as discussed above, we do not count the performance gains from GPT-4 imitation into the contributions of our paper. After the imitation learning stage, all the taboo conversation data is collected **synthetically** from the model outputs. The only additional supervision is the Alpaca dataset. As mentioned above, we have conducted ablation studies (`imitation model+Alpaca` vs `imitation model+Alpaca+SPAG`) to show the effectiveness of SPAG. \n\nFor the comparison with 20Q, we agree that the target sizes of the two games are different. However, the target sizes are entangled with the game designs, e.g. GuessMyCity can only use city names as targets. Although target sizes are different, we have made efforts to control other experiment factors. In Appendix Section E, we have used similar self-play sampling sizes (20K for imitation and 10K for self-play RL training) for non-adversarial games. From the perspective of supervision, we have used the same size of imitation data (20K GPT-4-played games) and the same SFT data (Alpaca). All other conversation data are sampled by the learning models themselves aligned with the concept of reinforcement learning. Therefore, the performance gaps between the adversarial taboo and non-adversarial games can support the efficiency of this adversarial game design.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nkC9VXw1V8", "reviewer": "Reviewer_R1Zh", "summary": "The paper introduces a self-play method, Adversarial Taboo, to bolster the reasoning ability of LLMs. By engaging in a two-player game that requires strategic communication around a target word, LLMs demonstrate improved performance across several reasoning benchmarks. The method leverages reinforcement learning to refine the models' strategies over successive rounds of play.", "questions": "The table in Figure 1 is quite difficult to read. Could you please add more explanatory information in the caption to facilitate the reader's understanding?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is interesting, leveraging adversarial gaming dynamics.\n2. Models post-Adversarial Taboo training outperform baselines on benchmarks like BBH, ARC, and WinoGrande, across different models.\n3. Utilizes self-generated data, minimizing reliance on external datasets and reducing data collection costs.", "weaknesses": "1. I concur with the use of a general-purpose instruction-tuning dataset to prevent overfitting to the game. However, I remain concerned that the method proposed in this paper may compromise the language model's broader capabilities in favor of enhancing reasoning skills, as evidenced by the examples provided in the appendix. The authors have only presented performance metrics for a few reasoning-related tasks. To bolster the argument, I believe it would be more persuasive if the authors could compare the results of their method with those of Alpaca SFT on general tasks.\n\n2. The enhancements brought by the method proposed in this paper are not as pronounced when compared to the IM+AlpacaSFT baseline, and there is even a decline under mutual conditions. I look forward to the authors elaborating on the potential of this method for scaling or multi-round iterations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer R1Zh for the constructive comments and suggestions. To address the reviewer's concerns:\n\n1. About the **evaluation on general tasks**: we agree with the reviewer that evaluating general language capacities is important to the SPAG models. We actually have reported the general language understanding evaluation of SPAG models on the Massive Multitask Language Understanding (MMLU) dataset in Table 1&2, which might give the reviewer an overview of the general capacity of SPAG models. Besides, the uniform performance gains of SPAG on various reasoning benchmarks also reflect the generalization ability of SPAG models, since these benchmarks are based on diverse reasoning challenges. Moreover, the SPAG models obtain uniform reasoning gains just by self-playing the adversarial language game without any training in the reasoning training sets (many reasoning benchmarks have their own training sets but we didn't use any of them). This setup also supports the generalization ability of our SPAG method. \n\n2. About the **IM+AlpacaSFT baseline performance**: we agree that the IM+AlpacaSFT model can achieve a comparable performance with SPAG-1 model. However, the IM+AlpacaSFT has already shown the upper-bound performance of Alpaca-SFT on imitation-learned models. In contrast, SPAG models can continuously be enhanced through iterative self-play and RL training processes. Besides, we did notice that IM+AlpacaSFT outperformed SPAG with Baichuan-2 as the base model on the Mutual benchmark. Due to the limited effort, we did dive deep to find the reason for this observation in the current draft. One potential reason might be the Mutual dataset has a similar domain or text pattern to the text in the Alpaca set. We will make more detailed analysis on the Mutual benchmark in the next revision.\n\n3. About the **Figure 1**: We are sorry about the confusion caused by the notations in Figure 1. We will add more explanations and descriptions to the notations in Figure 1 in the next revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BaswNL1GSB", "reviewer": "Reviewer_1AP3", "summary": "This paper explores an adversarial language game named Adversarial Taboo, where an attacker and a defender engage in a conversation centered around a target word visible only to the attacker. The attacker's goal is to prompt the defender to unconsciously utter the target word, while the defender strives to avoid doing so and deduce the word from the conversation history. Using LLaMA-2-7B and Baichuan-2-13B models, the study demonstrates that applying reinforcement learning to game outcomes enhances the reasoning capabilities of large language models.", "questions": "Regarding the two baselines (SP-20Q and SP-GuessCity), the authors describe them as self-play models for non-adversarial games. How is self-play conducted in the context of non-adversarial games?", "limitations": "The paper includes a section on limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written.\n- The authors investigate an intriguing adversarial language game called Adversarial Taboo, demonstrating that training on game data helps to enhance general reasoning abilities.\n- The study includes extensive experiments on two large language models (LLMs) across various datasets.", "weaknesses": "- The entire procedure involves standard training within the game data. However, the insight that this training not only  increase the game's win rate but also enhances general reasoning ability is quite interesting.\n- There is no discussion or analysis conducted on why training within the game can increase general reasoning ability.\n- No ablation study was conducted on training techniques, such as filtering negative advantages.\n\n\nOthers:\n\n- PPO objective (Eq.2): KL (\\pi || \\pi_sft) ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the suggestions and comments from Reviewer 1AP3. To address the reviewer's concerns:\n\n1. About **why reasoning ability improves by training within the game**: We provided a brief explanation about why self-playing adversarial games can improve the LLM's reasoning in the abstract and the introduction sections, that in adversarial language games such as Adversarial Taboo, both attacker and defender need to have high-level language capacities in terms of expression, understanding, and knowledge for the reasoning of the target word. In our opinion, the adversarial language game for reasoning improvement is an analog to sports competition for fitness. For example, a person can play basketball, tennis, or running competition to strengthen his/her body. We imagine that in the future, LLMs can also strengthen their capability by self-playing many different kinds of language competitions, of which Adversarial Taboo can be one candidate. \n\n2. We began our experiments with negative advantages but failed to obtain the desired self-play performances. That is the reason why we use the ReST method which filters the negative advantages in the current paper draft. Since our experiments with negative advantages failed (the performances are even worse than the base models), we did not report them for we haven't figured out whether the failure with negative advantage is because of our RL implementation or the instability of negative policy gradients. The one thing we can make sure of is that without negative advantages the SPAG framework works effectively. We will recheck our implementation of negative advantages and provide a more comprehensive discussion about the negative advantages in the next revision.\n\n3. About the **PPO objective**: Although in recent RLHF papers, the PPO (or more precisely, the LLM alignment objective) is with regularizer KL(\\pi || \\pi_ref), in the original TRPO[2] and PPO[1] paper the regularizer is KL(\\pi_ref || \\pi ). I think the term KL(\\pi || \\pi_ref) is more convenient for DPO[3] to derive the closed-form solution for the optimal policy.\n\nReference:\n[1] Proximal Policy Optimization Algorithms, 2017\n[2] Trust Region Policy Optimization, 2015\n[3] Direct Preference Optimization: Your Language Model is Secretly a Reward Model, 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces novel methods based on imitation learning and RL for fine-tuning large language models (LLMs) using an adversarial language game to enhance their reasoning abilities. This game, Adversarial Taboo (Yao et al., 2021), has one model (the attacker) trying to get another model (the defender) to say a secret word without explicitly revealing it. The motivation behind this approach is that both participants in this game require strong reasoning skills to succeed. The paper demonstrates that training LLMs with this adversarial game improves their reasoning abilities across various benchmarks.

All reviewers agree that the paper makes significant contributions: the proposed methods are creative, well-motivated, and some of them (especially SPAG-2 and SPAG-3) achieve good results on established benchmarks. The reviewers had mostly minor concerns, most of which were addressed during discussions with the authors. The authors' plan to enhance the paper in the next version (e.g., providing more detailed analyses on the Mutual benchmark) seems reasonable, and we look forward to these improvements being incorporated into the camera-ready version. Overall, this paper represents solid and interesting work, and I enthusiastically recommend its acceptance.

Minor point: The abstract (in the PDF) reads as though *Adversarial Taboo* is a contribution of the paper, as it is the only term italicized in the abstract and appears in the first sentence. Such prominence is usually reserved for methods or ideas that are new. I initially read the abstract as presenting Adversarial Taboo as a new game, and it seems that at least one of the reviewers also had the same initial misunderstanding. I recommend that the authors address this potential for confusion.
Decision: Accept (poster)