Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learning
Authors: Hao Ma
Tianyi Hu
Zhiqiang Pu
Boyin Liu
Xiaolin Ai
Yanyan Liang
Min Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: large language model
reinforcement learning with human feedback
multi-agent reinforcement learning
Primary area: natural_language_processing
TL;DR: We extend the RL fine-tuning of LLMs into a sequential cooperative multi-agent reinforcement learning framework.

Abstract: Reinforcement learning (RL) has emerged as a pivotal technique for fine-tuning large language models (LLMs) on specific tasks. However, prevailing RL fine-tuning methods predominantly rely on PPO and its variants. Though these algorithms are effective in general RL settings, they often exhibit suboptimal performance and vulnerability to distribution collapse when applied to the fine-tuning of LLMs. In this paper, we propose CORY, extending the RL fine-tuning of LLMs to a sequential cooperative multi-agent reinforcement learning framework, to leverage the inherent coevolution and emergent capabilities of multi-agent systems. In CORY, the LLM to be fine-tuned is initially duplicated into two autonomous agents: a pioneer and an observer. The pioneer generates responses based on queries, while the observer generates responses using both the queries and the pioneer’s responses. The two agents are trained together. During training, the agents exchange roles periodically, fostering cooperation and coevolution between them. Experiments evaluate CORY's performance by fine-tuning GPT-2 and Llama-2 under subjective and objective reward functions on the IMDB Review and GSM8K datasets, respectively. Results show that CORY outperforms PPO in terms of policy optimality, resistance to distribution collapse, and training robustness, thereby underscoring its potential as a superior methodology for refining LLMs in real-world applications.

Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learning
Hao Ma 1.2+
Tianyi 1.2 Zhiqiang Pu
Boyin Liu
Xiaolin AP
Yanyan Liang" Mín Chen
School of Artificial Intelligence, University of Chinese Academy of Sciences
Institute of Automation, Chinese Academy of Sciences
3 Alibaba (China) Co., Ltd.
Macau University of Science and Technology
(mahao2021, hutianyi2021, zhiqiang pu, xiaolin.ar ai, chenmin2020) ia.ac.cn
liuboyin. wwwaalaabbaaaaineeeeceaaae com
yyliangdmust.edd http:/mmsstteuuumm..mmmm edu.mo
Abstract
Reinforcement learning (RI.) has emerged as a pivotal technique for fine-tuning large language models (LLMs) on specific tasks. However, prevailing RL fine- tuning methods predominantly rely on PPO and its variants. Though these algo- rithms effective general RL settings, they often exhibit suboptimal perfor- mance and vulnerability to distribution collapse when applied to the LLMs. In this paper, we propose CORY, extending the RL fine-tuning LLMs sequential cooperative multi- agget reinforcement learning framework, to leverage the inherent coevolution and emergent capabilities of multi multi-agent systems. In CORY, the L.M to be fine-tuned is initially duplicated into two autonomous agents: a pioneer and an observer. The pioneer generates responses based on queries, while the observer generates responses using both the queries and the pioneer's s responses. The two agents are trained together. During training, the agents exchange roles periodically. fostering cooperation and coevolution between them. Experiments evaluate CORY's s performance by fine-tuning GPT-2 and Llama-2 under subjective and objective reward functions the IMDB Review and GSM8K datasets, respec- tively. Results show that CORY outperforms PPO in terms of policy optimality, resistance to distribution collapse, and training robustness, thereby underscoring its potential as a superior methodology for refining LLMs real-world applications. The code can be found https: het.sttt......rrtteeteeeee com/HHarry67Ha/COR!
of
1 Introduction
Large language models (LLMs) have achieved impressive success across diverse downstream tasks, including dialogue systems [Ouyang et al., 2022, Touvron et al., 2023], code generation [Roziere et al., 2023], and robotic control [Driess et al., 2023, Brohan et al., 2023]. However, as the capabilities of LLMS advance, the challenges associated with further performance gains become increasingly intricate. Fine-tuning LLMs for specific tasks presents a significant challenge, prompting recent exploration of LLM fine-tuning paradigm such supervised fine-tuning (SFT) [Wu et al., 2021], reinforcement learning (RL) fine-tuning [Shojace et al., 2023], and direct preference optimization
'These authors contributed equally to this work. Cooresponding author: zhiqiang pu pu@iaa.t @ia.ac.en.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(DPO) [Rafailov et al., 2024]. RL fine-tuning demonstrates promising potential for refining LLM. Compared SFT, RL fine-tuning offers a more direct optimization path, aligning training with desired outcomes and potentially leading to better nut-of-distribution performance [Kirk et al., 2023]. Compared to DPO, RI. fine-tuning allows fine-tuning on rule-hased reward functions without requiring preference data.
However, contemporary RL algorithms are not specifically designed for LLMs. When fine-tuning an LLM using these RL algorithms,, they exhibit instability and vulnerability to distribution collapse, which means that the LLM over-optimized and exhibits highly biased behavior [Zheng et al., 2023, Yang et al., 2024b]. From the perspective of RL. LLM fine-tuning has several challenges, including large discrete action space and sparse rewards. Taking the RL fine-tuning of 2lama-2 [Touvron et al., 2023] as an example, the dimension of the action space of Llama-2 can reach to 32000, representing 32000 potential vocabulary choices. Moreover, the reward signal is received only after generating the complete response, which results in sparse reward problem. The above challenges hinder the exploration in such a vast search space, causing the instability of popular algorithms like PPO [Schulman et al., 2017].
Cooperative multi-agent reinforcement learning (MARL) represents a paradigm shift in the field of artificial intelligence (AI), where multiple autonomous agents coevolve within a complex system, resulting in the emergence of new skills [Foerster, 2018, Yang and Wang. 2020, Oroojlooy and Hajinezhad, 2023, Zang et al., 2023]. Language is an outcome of such multi-agent coevolution. In society, numerous individuals utilize language for communication. Languages develop through agent interactions and are shaped by societal and cultural influences. As languages progress, they influence and are influenced by these interactions Slavali-Sforzz and Feldman, 1981, 2023]. Inspired by this, fine-tuning an LLM within cooperative MARL framework might lead to the emergence of superior policies during coevolution.
et al.,
In this paper, we propose a plug-and-play method named CORY, which extends the RL fine-tuning of LLMs a sequential cooperative MARL framework. In CORY, the LLM to be fine-tuned is initially duplicated into two autonomous agents" 3, assigned two roles respectively: a pioneer and an observer. There are two fundamental mechanisms in CORY enable the coevolution of the two LLM agents. The first is knowledge transfer, where the pioneer generates a response according to a task query independently, and the observer generates response based on the query as well as the response from the pioneer. The second is role exchange, where the roles of the two LLM agents are exchanged periodically during training. The two agents share a collective reward, calculated as the sum of individual task rewards, and they are trained simultaneously with their respective samples. Ultimately, CORY acts as a form of bootstrapping, wherein the collaborative learning between LLMs enhances the effectiveness of RL fine-tuning, Notably, this approach remains algorithm-agnostic, offering flexibility for integration with various RL algorithms beyond PPO, while maintaining simplicity and compatibility with existing methods.
In the experimental evaluation, we systematically investigate the efficacy of our proposed method across two types of reward functions: subjective and objective. Subjective reward functions are models trained to align human preferences, while objective reward functions are pre-defined functions typically established by domain experts. For the assessment of subjective rewards, we leverage the IMDB review dataset [Tripathi et al., 2020], a well-established benchmark for sentiment analysis. Meanwhile, the evaluation of objective rewards is conducted using the GSM8K dataset [Cobbe et al., 2021a], which focuses on mathematical word problem reasoning. Experiment results indicate that CORY surpasses PPO regarding policy optimality, resilience to distribution collapse, and robustness during training, highlighting its potential as an advanced method for improving LLMs in practical applications.
2 Problem Formulation
To understand LLMS through the lens of RI., we present sequential decision-making problem formulation for the next-token prediction causal language models. The next-token prediction is precisely defined by the concept of language-augmenter Markov decision process [Li et al., 2022], denoted as M V, S, A. r, p,y >. Here, represents a vocabulary of a language model,
'The "agents" here refer to individuals who make decisions and take actions in the context of reinforcement learning [Sutton and Barto, 2018J.

---

encompassing all possible tokens. The w E V represents a specific token within this vocabulary. The state space S VM, where VM is the combination space of M tokens. The action space A C V where VN is the combination space of N tokens. M and N are the max token lengths for state and action, respectively.
state S is a concatenation of token sequence 8 = (wi, W2....,, War). An action a E A is the output of causal language model, construed as a concatenation of token sequence a = (Wi, W2...... wy). The states and actions are padded with pad token the real length is less than the maximum length. The reward function r :: S x A - IR assigns numerical score to sequence of tokens, which can be considered as a typical sparse reward problem within the context of RL. The state transition function P: S X V + S describes deterministic transition of states according to the regr-ssgre paradigm. At each step, a predicted token is concatenated with the state of last step: $1+1 = (st, w(++) (So, w1:-1),, 1)), where so denotes tokenized user's input for causal language model, and wis = (Wi, W2, wh;) denotes a token sequence up to the i-th token. Then, the token-level policy of causal language model can be encapsulated within rrws]] [s, W1:s-)). And the sentence-level policy is defined as a joint policy:
N
(a|s0) = II x (w; wis-1).
The reward function r(.,-) is related specific task (e.g., safety alignment [Liu, 2023, Ji et al., 2024], code generation [Shojaee et al., 2023, Liu et al., 2023]). A task reward is only obtained after N steps of recision-makinn via token-level policy. Under such a sparse reward, RL is prone to over-optimisation, resulting in distributional collapse of the language model. To mitigate the risk of distributional collapse, it is common practice to incorporate token-level KL, penalties into the reward function, which serves to constrain the deviation of the language model from its original distribution [Go al., 2023, Zheng et al., 2023].
WKL(TO (-|su, w1:--1,, Tol- (s, W1s-1))
i< N
F(8;, w;)
r(so, a) K L Ta (- (180, w1s- nol (-00 w)=-1)) i= N,
where v) is the KL. coefficient, f(ss, wx) represents the token-level combined reward function. For each token, KL penalty is imposed based on the KL divergence between current policy no (-1so, w1:--1 and initial policy To (- w1: Only after predicting the final token, does the reward model yield a task-specific reward r(50, a).
3 Method
3.1 Coevolving with the Other You (CORY)
To extend the RL fine-tuning of LLMs to a cooperative MARL framework, the LLM to be fine-tuned in CORY initially duplicated into two copies, each is treated as an autonomous agent. Then, two roles, a pioneer and an observer, assigned these two LLM agents. We design two fundamental mechanisms to facilitate the coevolution between the two agents. The first design is knowledge transfer. The LLMs asynchronously take action, with the pioneer transferring its response (action) to the observer. The observer then utilizes this information to guide its own decision. The second design is role exchange. Once the observer achieves a satisfactory performance, it exchanges roles with the pioneer. In the following, we provide a comprehensive description of each element, and the pipeline of our method is shown in Figure
To enable collaboration between the two LLM agents for improved response generation, introduce knowledge transfer mechanism. Given query denoted so, the pioneer acts first and generates a response denoted as aj. Subsequently, the observer receives both the original query 8q and the pioneer's response o1 generate its own response d2. This sequential interaction facilitates knowledge transfer, where the observer leverages the pioneer's output to guide its own generation process, potentially leading superior response due to the in-context learning capabilities of LLMs. The sentence-level policies of the pioneer and observer can be formulated as follows:
Knowledge Transfer.
Tiio (-|8)),
T- 41).
During the training process, the parameters of the pioneer and the observer are optimized separately through an RL algorithm such as PPO. A cooperative relationship exists between the two LLM agents.
3

---

Traditional RL Fine-Ture Fine-Tuning
Coevolving with the Other You (CORY)
Task
Task Query
"Pesse following problem.."
S3 "9 help solve
tollnwiigg mith problem..
Role
Exchange
Pioneer
Observer
LLM Agent
Reward Model
Training
Training
Training
yn Uio Hq)
(1) T
Response
Response 2
Response
"Surcl acplutionn this problem...
Knowledge Transfer
Figure 1: The framework of CORY. A traditional RL fine-tuning method can be simply extended to the CORY version with only three steps. First, duplicate the LLM into two LLM agents, one acting as pioneer and the other as an observer; second, combine the task rewards the LLM agents replace the original task reward; third, periodically exchange the roles of the two LLM agents during training. After training, either the LLM agent can perform the task independently.
To facilitate this collaboration, CORY employs collective task reward, calculated as the sum of individual task rewards:
CCORY (So,Y(So, 01, 42) r(so: a1) + r(s0, 02),
which implies that both the pioneer and the observer receive rewards from each other's improvement. Following the form of Equation 2, we add rcony and the KI. penalty to construct a whole reward signal. Similar to Ni et al. [2022], we find that a partially correct reference can also be beneficial for not necessary for the pioneer generate a high-quality response.
the observer. Hence,
Role Exchange. During training, the observer may develop a prompt bias due to consistently receiving inputs in the form of (so, a1). This reliance on prompts that combine the original query with the pioneer's response, hinders the observer's 's ability to generate responses independently. To address this issue, introduce a role exchange mechanism. This mechanism involves exchanging the roles of the pioneer and observer periodically during training:
pio (-Iso) ppo (s: 01),
Tobe (- (so, a1) Fobs Fobs (-|so, a1; 02), swap False Tpio (+(so) ) = T- (-1s0: 02), Toba (-|s0, a1) Tabs (- (-130, a1; 01), if swap = T True,
(5)
where swap initialized as False, and reverse periodically. This exchange ensures that both the LLMs experience both roles (pioneer and observer) multiple times throughout the training process. Through this role exchange mechanism, they are forced adapt both prompt formats: 8p alone and the combined format (8o, a1). This allows us to use either LLM individually during inference. From a representational learning perspective, this role exchange mechanism encourages LLMs to develop unified representation for and (So, a1). This unified representation captures the essential information from the task query, regardless of the specific prompt format presented during training or inference.
These two key mechanisms in CORY with the observer potentially learning better policies by leveraging the pioneer's 's output. Role exchange ensures both the LLMs benefit from this collaborative learning, similar cooperative learning among approach, meaning it can be theoretically
form of bootstrapping. The LLM agents collaborate,
humans. Importantly, CORY is an algorithm-agnan compatible with various RL algorithms beyond PPO. Additionally, CORY offers the advantages of simplicity in implementation and seamless integration with existing frameworks, making it alug-and-play solution. The derivation the CORY's policy update can be found in Appendix B, and the detailed pseudocodes provided in Appendix C.

---

Sub optimal Fronteer
Pareto Frontier
Partto Honter
Sub optimal
Sub-optimel tronther of PFO
-15
PPD
Task Reviard
Taak Roward
0.22
Tosk Roward
(a) Pareto frontier
(b) Sub-optimal frontier
(c) Empirical result
Figure 2: The empirical demonstration of why CORY surpasses single-agent RL fine-tuning. In (c), the values of from left to right are le-5, le-4, le-3, and le-2.
3.2 Understanding CORY
Following the explanation of CORY in Section 3.1, this section provides an empirical demonstration of why the proposed method surpasses the single-agent RL fine-tuning method.
In fact, RL fine-tuning with KL penalty inherently formulates multi-objective reinforcement learning problem. The LLM agent strives to concurrently maximize the task reward and minimize the KL divergence. Unfortunately, these two objectives may be in opposition one another. This because maximizing the task reward will inevitably lead to the output distribution deviating from the pre- trained model, resulting in an increase in KL divergence. Hence, the optimization process seeks a trade-off between the task reward and the KL divergence, ideally driving the policy towards a Pareto frontier [Ngatchou et al., 2005]. This frontier covers all achievable policies where no policy can improve one objective without sacrificing performance on the other. Formally, the Pareto frontier can be defined as:
F (J+()) ( RETAA + Jr (m²) 2 (+()))
(6)
where JT(TT) = E rr(s at)l. r(s,a) E RT is a vector-valued reward function and II Z C Rm one could scalarize
denotes the set of all policies. Given a fixed reference vector the multi-objective reward into a single objective by using the weighted sum w' r(s,, a). Under this preference weighting, the ideal outcome for the policy is to converge point the Pareto frontier, as illustrated by the black dots in Figure 2(a).
However, due to the inherent complexities of natural language, achieving perfect policy convergence to the Pareto frontier is often intractable. Nevertheless, by adjusting the preferences, these sub-optimal policies can still form a frontier as illustrated in Figure 2(b). For simplicity. we term it the sub-optimal frontier. Our hypothesis is that the sub-optimal frontier achieved by CORY lies closer to the true Pareto frontier compared to that achieved by single-agent RL method.
To verify this hypothesis, we fine-tune the Llama-2-7b-char model on the grade school math 8K (GSM8K) dataset [Cobbe et al., 2021b] using both PPO and CORY. We measure the KL divergence and the task reward obtained by each policy after convergence. By adjusting the preference, i.e., n in Equation 2, we are able to generate sub- sub-optimal frontiers for both the methods, as illustrated in Figure 2(c). is important to note that the Y-axis represents the negative KL divergence (larger values indicate better performance). As expected, the sub-optimal frontier achieved by CORY consistently outperforms that of PPO, empirically validating the hypothesis.
Our analysis through the lens of multi-objective RI. offers valuable insights into the effectiveness of CORY. The knowledge transfer mechanism inherently addresses the optimization challenges faced by the observer. By leveraging the reference response provided by the pioneer, the observer actually experiences a guided optimization process. Such guided process can alleviate the optimization pressure on the task reward side, and prioritize improvement on the KL penalty side. However, since the observer's 's policy during training takes both the task query and the pioneer's 's response as inputs, the optimized policy is not the one we really want (we need the policy which only takes the task query as input), resulting in the prompt bias issue, The role exchange mechanism can effectively address this issue, and transfer the skills learned by the observer back to the pioneer, reducing the pioneer's 's optimization pressure. Notably, CORY demonstrates significantly better stability and robustness compared to single-agent RL method (See details in Section 4.2 and Appendix E. It consistently
5

---

achieves a lower KL divergence between the fine-tuned and pre-trained models while maintaining strong performance on the target task, signifying better trade-off between the two objectives.
4 Experiments
This section systematically investigate the performance of CORY across two types of reward functions: subjective reward function and objective reward function. Subjective reward functions are reward models trained on data capturing human preferences. They essentially translate the human sentiment or judgment into a numerical reward signal that guides alignment. Objective reward functions are pre-defined rule-based functions, typically established by domain experts. This categorization reflects real-world scenarios where reward functions might be learned from human preferences or manually crafted by domain experts. Prompts used in experiments are detailed in Appendix A.2.
4.1 Subjective Rewards on IMDB Review
Task Setup. To evaluate our method under the subjective reward setting, we select the IMDB Review dataset ['Tripathi et al., 2020]. This dataset contains 50K <text,label> pairs, with the training set and the test set each contains 25K pieces of data. The texts in the IMDB dataset are movie reviews, and the labels are the binary sentiment classification labels. The distilbert-imdb model* trained on the dataset is employed as the reward model. We fine-tune GPT2-Larg (774M)* by using single-agent PPO (single-PPO) and CORY respectively. In addition, GPT2-XL (I.5B) is fine-tuned by using single-PPO as an ablation on model size. In this task, we randomly sample text snippets from the IMDB dataset. The first 2 to 8 tokens (representing the beginning of the review) are retained as prompts for sentiment completion. The LLMs generate continuations that transform the prompts into positive sentiment comments. After that, the reward model evaluates the generated text to assign a sentiment score. The objective is to maximize the average sentiment score of the completed comments. Examples of this task are detailed in Appendix D.
In the experiments, each method undergoes 100 training iterations using batch size of 256. For simplicity, GPT2-Large and GPT2-XL fine-tumed by single-PPO are termed as PPO-GPT-2-1 and PPO-GPT-2-xl, respectively. GPT-2-Large that fine-tuned by CORY are referred to CORY-LLMI and CORY-LLM2, where the former one is the LLM that initialized as the pioneer, and the latter one is the LLM that initialized as the observer.
2.5
0.0
rewrrr
40
2.0
-25
Combined
5.0
PRO-G-131
1.0
-7.5
LOM
03RV-LUM
0.5
CORV
-10.0
00 20
20
50
Iteration
Iteration
(a) Task reward
(b) KL divergence
(c) Combined reward
Figure 3: Training curves under subjective rewards on IMDB Review.
Results and Analysis. We monitor the training process by visualizing task reward, KL diver- gence, and a combined reward function that incorporates both the above objectives. Denoted as ro(so, a). the combined reward function can be expressed as re(so, K LLsso, Ta, To), where r(sp, a) and K L(8o, 80, Te, To) are the sentence-level task reward part and the KL penalty part, respectively. And the KL penalty part can be calculated as K L(so, TT, To) E - LKL( w1: (- (1:--)).
r(5o,a) a) + 1)
is important to note that, the actual reward used for training in CORY is not the combined reward. The actual training reward not only includes the KL penalty and the task reward from the target agent,
http://www....eeeeee////tsttaaaa /ttpa////////agiigeeoee////////r imdb
"nctpe://huggingfaceee ct http:p////uugggggg.......oooooon
htteps:/huuginggaaa..oo//// cotponttoooomnttttttootooee ttpttutttttgtttttttigmmg

---

but also includes the task reward from the other agent. In fact, the combined reward re a) is the real overall objective that needs to be optimized, and can be aligned with the single-agent RL fine-tuning, making it easier to compare performance of all the methods.
The training curves of task reward, KL divergence, and the combined reward illustrated in Figure 12. The results show that singlc-PPO and CORY achieve similar task reward levels after 100 training iterations. However, the curve of KL divergence related single-PPO is significantly higher than that of CORY, reaching more than twice the level of CORY after all the training iterations. This indicates CORY's ability to achieve similar task reward levels with smaller deviation from the pre-trained policy. Moreover, it can be observed that the curves of CORY-LLMI and CORY- are very close, indicating that the two LLM agents initially playing different roles finally achieve very similar performance levels at the end of the training. Consistent with the motivation of CORY, both the fine-tuned L.LM agents can be used to finish tasks individually, which verifies the effectiveness of the bootstrapped learning and coevolution principles in CORY.
Finally, Figure 12(c) visually confirms CORY's advantage in combining the two objectives. The combined reward curve for CORY consistently rises, indicating its effectiveness in simultancously improving task reward and minimizing KL divergence. Conversely, PPO's combined reward curve exhibits a decreasing trend, suggesting its struggle in balancing these objectives. Hyperparameters used for both single-PPO and CORY are detailed in Appendix A.
4.2 Objective Rewards on GSM8K
Task Setup. To evaluate our method under rule-based objective reward function, we select the GSM8K task [Cobbe et al., 2021a]. GSM8K comprises 8.79K high-quality, linguistically diverse grade school math word problems, with 7.47K allocated for training and 1.32K for testing. For each question in the dataset, response is obtained via LLM. The precise answer is extracted from the responses using a regular expression, typically the final set of numbers in the response. If the number in question matches the ground truth as recorded in the dataset, a reward of 1 is awarded. Conversely, the number incorrect, reward of given. The Llama-2-7b-char model is selected as the pre- traiird model. To reduce the training overhead, the model is quantised to 4-bit. For simplicity, the 4-bit Llama-2-7b-chat model fine-tuned with single-PPO is referred to as PPO-Llama-2, The copied models fine-tuned with CORY are referred to CORY-LLMI and CORY-LLM2, where the former is the LLM that initialized as the pioneer, and the latter is the LLM that initialized as the observer. Examples of this task are detailed in Appendix D.
0.20
PPO CORY
Combined neward
Task Task rewardd
0.20
CORY-LIN
7.5
0.16
0.10
6.0
HMg
0.08
HPO
0.10
CORYLIM1
CONV-AMM
0.0
CORYLLA/2
20
40
60
80
100
20
40
60
80
100
20
40
50
80
100
lteration
Iteration
Iteration
(a) Task reward
(b) KI. divergence
(c) Combined reward
Figure 4: Training curves under objective rewards on GSM8K.
Results and Analysis. Similar to Section 4.1, We monitor the training process by visualizing task reward, KL divergence, and the combined reward. As shown in Figure 4, the jitter observed in all curves suggests the challenge posed by GSM8K. The vast exploration space presents inherent instability for the RL algorithms. As Figure 4(a) illustrates, the task reward curve of single-PPO peaks around 50 training iterations, followed by decline. Single-PPO's KL divergence exhibits no convergence trend, reaching a maximum value during training (Figure 4(b)). The instability of single-PPO results the high KL divergence after 50 iterations, leading to a poor performance on combined reward (Figure 4(c)).
aattps://hugginggace co/meta- Martaa--laan 1lama/ LLama- 2--------- chat-hi hf
7

---

In contrast, CORY demonstrates significantly more stable task reward curve, consistently outperforming single-PPO. What's more, CORY achieves a considerably lower KI. divergence compared to single-PPO, facilitating faster convergence. This characteristic is particularly valuable in the fine-tuning context, as it allows CORY to achieve similar or even better task rewards without significant modifications to the original parameter dis- tributions.
0.40
0.35
Accuracy
0.30
0.25
0.20
Laaa-227b-oo
PPO
0.15
CORY
3 Pass @k
5
Furthermore, the combined reward curves visually confirm CORY's superiority over single-PPO. CORY's ability effec- tively balance the two objectives is reflected in its steadily in- creasing combined reward. Conversely, single-PPO's struggle with balancing the objectives manifest as decreasing com- bined reward and training instability.
Figure 5: Evaluation results on GSM8K test dataset.
In addition, we conduct comparative analysis of models fine-tuned with distinct methods and pre-trained model the GSM8K test set shown in Figure 5. The evaluation metric utilized is pass&k, which generates k: corresponding repetitions for a sample and passes if at least one is correct. The test results demonstrate that the CORY fine-tuned 4bit 2lama-2-chat-77 7b could achieve a pass@1 of 18% on GSM8K test dataset.
4.3 Ablations
In ablation experiments, we ablate the influence of model size, knowledge transfer. and role exchange under the subjective reward setting on IMDB review dataset. For method names depicted in Figure REx indicates role exchange, KT indicates knowledge transfer, LLMI and LLM2 refer to LLMs who are initialized as the pioneer and the observer respectively.
3.0
2.5
0.0
2.0
Combined
20
Com comrututtaiit to
0.5
1000
000 200
20
50 Iteration
Itaration
Iteration
(a) Task reward
(b) KL divergence
(c) Combined reward
Figure 6: Training curves for ablations experiments.
Ablation Model Size. Our method employs two models during training, with the total parameters trained being doubled in comparison to single-PPO. In order to ablate whether the enhancement of CORY is derived from the expansion of the model parameters, additional fine-tuning of GPT2-XL (L.5B) with single-PPO is conducted on the IMDB dataset, which has twice the number of parameters as GPTg-Large. The results are presented in Figure 12. While the task reward of the model rapidly reaches its maximum value, the KL penalty part does not exhibit notable improvement compared to GPT2-Large. The KL divergence continues increase, leading the collapse of the distribution.
We maintain role exchange, and the two models still share collective task reward (Equation 4), but disable knowledge transfer. This resembles PPO with individual queries as inputs. However, without the observability of the pioncer's outputs, this equivalent to adding noise to the PPO reward signal. Consequently, the task rewards become unstable, and the KL. divergences are higher compared to CORY as shown in Figure 6. This highlights the importance of observability for framing RL fine-tuning
Ablation on Knowledge Transfer.
true multi-agent cooperation problem.
Ablation on Role Exchange. We maintain knowledge transfer but disable role exchange. As evident from Figure 6, both LLMS achieve good task rewards, but their KI. divergences much higher than that of CORY. Notably, the observer L.MM exhibits significantly lower KI. divergence compared to the pioneer LLM. This observation highlights fascinating phenomenon in cooperative learning:
8

---

by receiving the pioneer's ' response, the observer can effectively optimize the KL divergence. This suggests that the observer leverages the pioneer's exploration to refine its policy while maintaining good performance, potentially leading to more stable learning process.
5 Related Work
The most related topic is reinforcement leanring from human feedback (RLHF). InstructGPT [Ouyang et al., 2022] fine-tunes GPT-3 like models [Brown et al., 2020] to enhance helpfulness by combining SFT with RL based on human preference dataset. Askell et al. [2021] trains a preference model for aligning the LLM with human values. It argues that ranked preference modeling proves to be the most effective training objective for distinguishing between desirable and undesirable LLM behaviors. Bai et al. [2022] incorporates an iterative online training mode where preference model and LLM updated weekly using fresh human feedback data. Existing research acknowledges the inherent complexity, instability, and hyperparameter sensitivity of RLHF, particularly when employing PPO Zheng et al. [2023]. Several works have attempted to address these challenges by introducing max- entropy regularization [Wen et al., 2024], hyperparameter tuning [Zheng et al., 2023], and reward shaping (Yang et al., 2024a]. However, these methods does not show significant improvement over the vanilla PPO algorithm. This inspires us to explore alternative method from a different perspective that extent the RL fine-tuning of LLMs to cooperative MARL problem.
Another related topic is MARL. Under the interaction relationship (cooperation, competition, mixed), multi-agent could spontaneously emerge complex and diverse policies, problems that single-agent reinforcement learning is difficult to solve. For example, Kim et al. [2023], the RL based prompt tuning is decomposed into multi-agent joint tuning. The huge joint action space is equally split across agents, learning better and longer prompt. Such mechanisms have also been applied in the field of combinatorial optimization. The paper that is most similar to us on the architecture of agent training is Gao et al. [2023]. It proposes an asymmetric training symmetric execution framework to deal with the two-agent Stackelberg game Fang al. [2021]. In the Stackelberg game, two agents make decisions asynchronously. The agent that makes the decision later can observe the former agent, but the former agent cannot observe the later agent. The training framework proposed by the authors is able to converge in Stackelberg equilibrium empirically. This inspires us to design the training framework for LLMs under a sequential cooperative setting.
to solve the complex
Discussion
Experimental evidence suggests that CORY yields more stable and superior performance in RL fine- tuning. This can be attributed to our extension of singlc-agent RL fine-tuning into a cooperative MARL version. In this section, we delve into a discussion of how the multi-agent learning can benefit LLM fine-tuning. The primary benefit is that multi-agent learning encourages the coevolution of LLMs through collective living, social relationships and major evolutionary transitions Douéñez-Guzzale et al., 2023]. This process generates a variety of new data, which further facilitates coevolution. This mechanism contributes to many breakthroughs in games AI, such as Go [Silver et al., 2016, 2017, Clark and Storkey, 2015], StarCraft II [Vinyals et al., 2019], and Diplomacy [Bakhtin et al., 2022].
In this paper, investi gate the application of cooperative MARL address challenges in RL fine-tuning. Cooperative MARL fine-tuning appears to increase training robustness and prevent dis- tribution collapse. While we concentrate on cooperation, competitive MARL. especially population- based methods, represents a promising direction for future research. These approaches create an auto-curriculum mechanism driven by a natural arms race, which propels agent learning and en- ables mastery of complex tasks. Besides the interaction paradigm, the scale of agents is crucial to emergence. While we examine a setting involving two LLMs, incorporating more LLMS in MARL fine-tuning is an intriguing prospect for future studies.
7 Conclusion
In this paper, we extend the RL fine-tuning of LLMs to a sequential cooperative MARL framework. To this end, we duplicate the pre-trained L.LM into two L.MM agents with different roles, and design two key mechanisms: knowledge transfer and role exchange. These mechanisms enable the two LLM

---

agents to learn collaboratively. and after the fine-tuning process, either the LLM agent can be chosen perform the task independently. We also provide an in-depth analysis of RL fine-tune from the perspective of multi-objeccive RI., revealing the existence of a Pareto frontier between KI. divergence and task reward. We empirically illustrate that CORY has an advantage over single-agent RI. method in approaching the Pareto frontier. Experiment results indicate that CORY surpasses PPO regarding policy optimality, resilience to distribution collapse, and robustness during training, highlighting its potential as advanced method for improving LLMs in practical applications.
Acknowledgement
This work was supported by the Strategic Priority Research Program of Chinese Academy of Science under Grant No. XDA27030204, the National Natural Science Foundation of China under Grant 62322316, the Beijing Nova Program under Grant 20220484077 and 20230484435.
References
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, al. A general language assistant as laboratory for alignment. arXiv preprint ar 00861, 2021.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0580 2022.
Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624):1067-10009
2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision Vision-language-aciinn models transfer web knowledge robotic control. arXiv preprint arXiv:2307.15816 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamerr. Advances in neural information processing systems, 33:1877-1901, 2020.
2020.
Luigi Luca Cavalli-Sforza and Marcus W Feldman. Cultural transmission and evolution: A quantita- tive approach. Number 16. Princeton University Press, 1981.
Christopher Clark and Amos Storkey. Training deep convolutional neural networks to play go. In International conference on machine learning, pages 1766-1774. PMLR, 2015.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10.:2/10.11166. 2021a.
Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichrr Nakano, al. Training verifiers solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021b.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. In International Conference on Machine Learning, pages 8469-8488 PMLR, 2023.
Edgar A Duéñez-Guzmán, Suzanne Sadedin, Jane X Wang, Kevin R McKee, and Joel Z Leibo. A social path to human-like artificial intelligence. Nature Machine Intelligence, 5(11):1181-11) 181-1188. 2023.
10

---

Fei Fang, Shutian Liu, Anjon Basak, Quanyan Zhu, Christopher D Kiekintveld, and Charles A Kamhoua. Introduction to game theory. Game Theory and Machine Learning for Cyber Security, pages 21-46, 2021.
J Foerster. Deep mudii-agent reinforcement learning. PhD thesis, University of Oxford, 2018.
Yuan Gao, Junfeng Chen, Xi Chen, Chongyang Wang. Junjie Hu, Fuqin Deng, and Tin Lun Lam. Asymmetric self-play-enablee intelligent heterogeneous multirobot catching system using deep multiagent reinforcement learning. IEEE Transactions on Robotics, 2023.
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Mare Dymet- man. Aligning language models with preferences through f-divergence minimization. arXiv preprint arXivv2.302.08215. 2023.
Jamming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via a human- preference numan-preference dataset. Advances in Neural Information Processing Systems, 36, 2024.
Dong-Ki Kim, Sungryull Sohn, Lajanugen Logeswaran, Dongsub Shim, and Honglak Lee. Mul- tiprompter: Cooperative prompt optimization with multi-agent reinforcement learning, arXiv preprint arXiv:2310. 16730, 2023.
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rihf on I1m generalisation and diversity. In The Twelfth International Conference on Learning Representations, 2023.
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems,
2022.
Jiate Liu, Yiqin Zhu, Kaiwen Xiao, QIANG FU, Xiao Han, Yang Wei, and Deheng Ye. Rltf: Reinforcement learning from unit test feedback. Transactions on Machine Learning Research, 2023.
Yang Liu. The importance of human-labeled data in the era of llms. arXiv preprint arXiv:2306.14911 2023.
Patrick Ngatchou, Anahita Zarei, and El-Sharkawi Pareto multi objective optimization. In Proceedings of the 13th international conference on, intelligent systems application to power systems, pages 84-91. IEEE, 2005.
Ansong Ni, Jeevana Priya Inala, Chenglong Wang, Oleksandr Polozov, Christopher Meek, Dragomir Radev, and Jianfeng Gao. Learning math reasoning from self self-sampled correct and partially-correct solutions. arXiv preprint arXiv:2205. 14318, 2022.
Michael Noukhovitch, Samuel Lavoie, Florian Strub, and Aaron C Courville. Language model alignment with elastic reset. Advances in Neural Information Processing Systems, 36, 2024.
Afshin Oroojlooy and Davood Hajinezhad. A review of cooperative multi-agent deep reinforcement learning. Applied Intelligence, 53(11):13677-13322. 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint ar 2ati::228...0.00000 2023.

---

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Parshin Shojaee, Aneesh Jain, Sindhu Tipirenii, and Chandan K Reddy, Execution- code generation using deep reinforcement learning. arXiv preprint arXiv:2301.13811 2023.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529
2016.
David Silver, Julian Schrittieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):54-35)) 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09280 2023.
Sandesh Tripathi, Ritu Mehrotra, Vidushi Bansal, and Shweta Upadhyay. Analyzing sentiment using imdb dataset. In 2020 12th International Conference on Computational Intelligence and Communication Networks (CICN), pages 30-33. IEEE, 2020.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung. David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in stareraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-335 2019.
Muning Wen, Cheng Deng, Jun Wang, Weinan Zhang, and Ying Wen. Entropy-regularized token-level policy optimization for large language models. arXiv preprint arXiv:2402.067700 2024.
Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109. 2021.
Shentao Yang, Shujian Zhang, Congying Xia, Yihao Feng, Caiming Xiong, and Mingyuan Zhou. Preference-groundee token-level guidance for language model fine-tuning. Advances in Neural Information Processing Systems, 36, 2024a.
Wanli Yang, Fei Sun, Xinyu Ma, Xun Liu, Dawei Yin, and Xueqi Cheng. The butterfly effect of model editing: Few edits can trigger large language models collapse. arXiv preprint arXiv:2402.09656. 2024b.
Yaodong Yang and Jun Wang. An overview multi-agent reinforcement learning from game theoretical perspective. arXiv preprint arXiv:2011.0053: 2020.
Yifan Zang, Jinmin He, Kai Li, Haobo Fu, Qiang Fu, and Junliang Xing. Sequential cooperative multi-agent reinforcement learning. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pages 485 495, 2023.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.044 04964, 2023.
12

---

A Implementation Details
The code repository we utilize TRL*. Our experimentation employs AMD EPYC 7773X CPUs and 8 NVIDIA A6000 GPUs (48GB each). Leveraging single GPU, CORY can achieve full- precision RL tunentunig of GPT2-XL on the IMDB Review dataset within 12 hours. With GPUs, CORY can accomplish the RL fine-tuning of a 4-bit quantized Llama-2-7b-chal model on GSM8K within 4 hours.
A.1 Hyperparameters
The hyperparameter settings for fine-tuning GPT2 followed the default configuration in TRL for the IMDB dataset, while the hyperparameter setting Llama-2 primarily adhered the guidelines provided by StackLlama. To ensure a fair comparison, all hyperparameters were carefully selected balance the stability and performance PPO. grid search was conducted over and r,, with the sets a le-6, le-5, le-4 and le-3, le-2, le-1, 0.2, 0.3, respectively, to identify the hyperparam- eter that yielded the most stable training for PPO. Given CORY's robustness hyperparameters (Appendix E.1), most PPO hyperparameters, except for the learning rate were applied directly to CORY. For the GSM8K dataset, In the GSM8K dataset, we adjusted the learning rate a for CORY.
Table 1: Hyperparameters in IMDB Review
Hy Hyperparameter
PPO-GPT-2-1
PPO-GPT-2-XI
CORY
Learning Rate (a)
1.41e-5
1.4le-5
1.4le-5
Epochs
1
PPO Epoch
4
Batch Size
256
256
256
Mini Batch Size
256
256
256
Gradient Accumulation Steps
1
1
Iterations
100
100
100
Initial KI. Coefficient (1))
0.3
0.3
0.3
Early Stopping
False
False
False
Discount (7)
1
1
GAE (A)
0.95
0.95
0.95
Gradient Clip Range
0.2
0.2
0.2
Value Clip Range
0.2
0.2
0.2
Value Loss Coefficient (8)
0.1
0.1
0.1
Period of role exchange (TRE.)
iterations
Table 2: Hyperparameters in GSM8K
Hyperparameter
PPO
PPO-13b
CORY
Learning Rate (a)
le-5
le-5
le-4
Epochs
1
Batch size
32
32
32
Mini Batch Size
2
Gradient Accumulation Steps
16
16
16
Iterations
100
100
100
Initial KL Coefficient (1))
0.01
0.01
0.01
Early Stopping
False
False
False
Discount (7)
1
GAE (A)
0.95
0.95
0.95
Gradient Clip Range
0.2
0.2
0.2
Value Clip Range
0.2
0.2
0.2
Value Loss Coefficient (3)
0.1
0.1
0.1
Period of role exchange (TREE)
5 iterations
13

---

A.2
Prompt Details
IMDB Review. The prompts used in IMDB Review are follows. For PPO or CORY's pioneer, since this is sentence completion task, instead of using a prompt template, we directly input the first few words in the review (brown).
Another fun, witty, frothy
For CORY's s observer, we use pioneer's 's response (blue) to complete the sentence as reference for observer, and retype the first few words of the comment at the end of the prompt for observer to complete.
I can make this sentence 'nnother fun, witty, frothy cut different from the usual.' more positive: Another fun, witty, frothy
GSM8K. The prompts used in GSM8K are follows. For PPO CORY's pioneer, provide example question and answer. This is followed by question from the dataset (brown). Then the prompt ends with 'Answer:' to guide the LLM answer.
Question: Shawn has five toys. For Christmas, he got two toys each from his mom and dad. How many toys does he have now?
Answer:
Shawn started with 5 toys. If he got 2 toys each from his mom and dad, then that is 4 more toys. 5 +
Question: The civic league was hosting a pancake breakfast fundraiser. stack of pancakes was $4.00 and you could add bacon for $2.00. They sold 60 stacks of pancakes and 90 slices of bacon. How much did they raise?
Answer:
For CORY's observer, the question is followed by 'Reference' (blue), which is the pioneer's response. Finally, it also ends with 'Answer' guide the model answer.
Question: Shawn has toys. For Christmas, he got two toys each from his mom and dad. How many toys does he have now?
Shawn started with 5 toys. he got toys each from his mom and dad, then that is 4 more toys. 5 + 9.
Answer:
Question: The civic Ieague was hosting a pancake breakfast fundraiser. A stack of pancakes was $4.00 and you could add bacon for $2.00. They sold 60 stacks of pancakes and 90 slices of bacon. How much did they raise?
Reference: To find out how much the Civic League raised, need to multiply the number of stacks of pancakes by the cost of each stack. So, 60 x $4 = $240. Then, we multiply the number of slices of bacon by the cost of each slice. So, 90 $2 $180. Therefore, the Civic League raised total of $240 + $180 = $420.
Answer:
14

---

B Token-Level Policy Update of CORY
We first derive the formula of Q-function when fine-tuning LLM with PPO. The token-level reward function Ê is given in Equation 2.
(x ws) = Bug WN
(8 wotk)
N
* 8 witk)
* KLL [T( St+k), Nol. sc++)l
N-
[7 "r(s0, a)] - -nBu, 11.. -nBu.tin... W
Σ
* KL [T((- (844)), mol- s(423]]
Bur W y ((0000) a) -n Σ k=
KL [IT(- 1814)); nol. sct+a)
(7)
For CORY, pioneer and observer share the same task reward ECORY, but their Q-functions have slightly different forms due to their different inputs. For simplicity, we define uniform state
3o A (So. a1) ) for the observer, and šn _ so for the pioneer. Then, denoting the parameterized policy
as Ty. the Q-functions for them can be expressed in an uniform way.
Qxo (5;, wi) = Em wooeee
"CORY (5o, a1, a2) - Σ 7 KL [TT( Bitk),
Be++a]
(8)
Similarly, CORY's uniform state value function can be expressed as
VTa (5:)
Σ To (w; (5)) ($,, wt))
(9)
W.EV
In practice, both the pioneer and the observer in CORY are optimised using PPO independently. During the training phase, value head is attached to the last hidden layer of the policy network to predict the current state value. The loss function is:
V x = Emg V (3.) (3:))
(10)
where V (3:) is the predicted state value, ø represents the parameters of the corresponding value network. For policy loss, the optimisation objective with clip is used.
The (ure 84) ÅTo (3;, wz), clip( (e 83))
Te(w; 82) Toat (w, (5)) c1,,,1++) (8,, mi) )
L kg
Er min
(11)
where Wows is the older policy that collects data. The importance ratio no(ws)8) is used to estimate Toztal (Mr.2,)
Ans under To on data collected via Te... reflects how much the current policy deviates relative to the older policy. Ane is the advantage function, given o, = F(8i, wi) + 7V - (5+),
Å (8i, w;) 5, + (7A)8:+1 +... (TA) (N 8N-1
(12)
Ultimately, with value loss coefficient B, the pioneer and the observer are fine-tuned by maximising the following objective
L(e, の) L - SLW Ideally, after the optimisation, the optimal token-level policy x" is obtained, which in turn naturally leads to the optimal sentence-level policy.
(13)
N
" (a(30) = II
x (w, 1o0, w1+-1).
(14)
1=1
15

---

C Algorithm Details
C.1 Algorithm of CORY
Algorithm Coevolving with the Other You
Input:
Pre-trained LLM To, task reward model r, query data set Dq. period role exchange TREZ- Output: Fine-tuned LLMs Re, and Té.- Initialization: Duplicate To into a pioneer (-1:;-0: and an observer Toba ( 02), initialize the pioneer buffer pio 0 and the observer buffer Dobs 0.
1: Set l: +- O.
2: for each iteration do
3: Set Pio + and Dota + 0.
4: Sample task query batch BQ from Do.
5: for each 5p in BQ do
6: a1 ~ (- 01).
7:
Ppio f- r(su. 01).
8:
0g ~ (- 01:
9:
Tobs +- r(00 01).
rCORY + Tpio + Tobs-
11:
Set 60 - so and update memory pio +- Dpio U ((50, 01 rconr)).
Set &g + (so. a1) and update memory Dobs Dobs U ((80, 02 (CORY))
13:
end for
14:
Update 0, through Algorithm C.2 on D Pio-
15:
Update 0g through Algorithm C.2 on Dobe- if (I+ + 1)%TRE> = then
16:
17:
Set 0r t 01 and 6rew <- 82.
18:
022 - Bpew.
19:
01 Ogew.
20:
end
21:
k <-
22:
end for
C.2
Token-Level Policy Update
Algorithm 2 PPO-based Token-Level Policy Update
Input:
Target LLM ng. reference LLM No, sentence-level data buffer D, max token length action N, learning rate a, KL coefficient 7).
Output: The updated parameters of the target LLM 0.
Initialization: Initialize the value network Vo and the token-level data buffer DJ 6.
1: for (gg. o, (CORY) ) in D do
2:
D < 0.
3: for = 1,2, ... do
4:
FKL. KL nal 5o, all 1)), no(1150,[[1: 1))).
5:
8; + (zn, all -- 1]).
6:
a: +- ali.
7:
8 + (30, :)).
8:
if N then
9:
+ 77 TKL.
else
11:
end + ECORY - TKL-
13:
DT +- DT U (i.
end for
15:
Compute advantage estimate År, via GAE on D (Equation 12)
0-0 +a a Vo Φ). (Equation 13)
17:
Ø + δ a VoL(O, Ø). (Equation 13)
18: end for
16

---

D Qualitative Analysis of Experiment Results.
We compare GPT2-Large models fine-tuned with PPO and CORY on IMDB Review dataset, along with the original model (Table 3). The input review snippet consists the first few words of a movie review. The goal of LLMs is to complete the sentence in positive direction. Comparing results before and after fine-tuning, sentences are often incomplete and occasionally contain grammatical emors due to the limitations of GPT2-Large. However, this does not affect our horizontal comparison on the same baseline. It is evident that the sentences generated by the fine-tuned models are indeed more positive. Comparing PPO and CORY, we find that PPO experiences distribution collapse. While task reward is comparable to CORY, its KI. divergence is significantly larger (Figure 12). Sentences generated by CORY more positive. Although there occasional grammatical errors, they are similar to those in the pre-trained model, indicating that CORY effectively avoids distribution collapse.
We also compared the Llama-2-78 2-raaa---BBhhat models fine-tuned with PPO and CORY on GSM8K. Due to PPO's s sensitivity to parameters, which results in either stable training or distribution collapse, we divided the comparison into two tables. When PPO trains stably (Table 4), the quality the generated answer is similar to CORY, though slightly less accurate. When PPO experiences distribution collapse (Table 5), tends to generate particularly long outputs until reaching the maximum token limit. This is because the probability of the end-of-sentence token 
 in the token-level policy decreases significantly compared to its initial value, preventing sentence completion. Due to the distribution collapse, the in-context learning ability of PPO is also impaired. It generates another Question after generating an Answer. contrast, CORY's performance much more stable.
Table 3: Examples of IMDB Review. GPT2-Large is fine-tuned with PPO and CORY respectively.
Review snippet
This is one of Jackies
GPT2-Large
last regrets about running.
PPO
marvelous marvelous splendid marvelous marvelous magnificent awesome
marvelous stunning excellent phenomenal spectacular beautiful
CORY
Cruise movies are world famous worldwide has
Review snippet
I find it hard to believe
GPT2-Large
that someone could have survived three
PPO
amazing marvelous marvelous marvelous marvelous spectacular terrific
beautiful magnificent marvelous fantastic wonderful fabulous
CORY
that we've all had wonderful amazing
Review snippet
This movie was highly entertaining. The
GPT2-Large
characters not only great, guests
PPO
unacceptable THANK!! wonderful BEST
CORY
film was highly acclaimed captivating enjoy
17

---

Table 4: Examples of GSM8K when PPO fine-tuning is stable.
Question
George wants to borrow $100 from loan shark. The finance fee starts at 5% and doubles every week. If George plans to borrow for 2 weeks, how much in fees will he have to pay?
PPO
The finance fee starts 5%, so after one week would be 10%. After two weeks, the fee would be 20%. So, George would have to pay (100 x 10%) + (100 X 20%) 200 in fees.
CORY
At first, the finance fee is 5%. So George will pay $5 in fees. Then, the next week, the fee will be doubled to $10. George will pay another $10 in fees. Therefore, George will pay a total of $15 in fees.
Question
Lorin has 4 black marbles. Jimmy has 22 yellow marbles. Alex had twice as many black marbles as Lorin and one half as many yellow marbles as Jimmy. How many marbles does Alex have?
PPO
Alex has 8 black marbles (twice as many as Lorin). Since Alex has half as many yellow marbles as Jimmy, Alex has 11 yellow marbles (half of 22). So Alex has a total + 19 marbles.
CORY
Alex has 8 black marbles (twice many Lorin). Since Alex also has one half as many yellow marbles as Jimmy, Alex has 11 yellow marbles (half of 22). So, in total, Alex has 8 11 = 19 marbles.
Question
Bill is trying to control the pests in his garden. Each spider he introduces cats 7 bugs, and each time he sprays the garden he reduces the total bug population 80% of what was previously. the garden has 400 bugs start, and Bill sprays once and introduces 12 spiders, how many bugs are left?
PPO
The number of bugs in the garden before Bill sprayed is 400. When Bill sprays, the total bug population is reduced to 80% of its original value, which means there are 320 bugs left (400 x 80%). Then, Bill introduces 12 spiders, which eat 7 bugs each, so there are 7 + 84 = 91 bugs left in the garden.
CORY
First, let's find out how many bugs left after Bill sprays by multiplying the previous number of bugs by 80%. So, 400 x 0.8 = 320 bugs. Now, let's find out how many bugs each spider can eat. Each spider can eat bugs, 12 spiders can eat 7 x 12 = 84 bugs. Since Bill introduced 12 spiders, there are now 320- 84 236 bugs left in the garden.
Question
Mira jogs every morning. She jogs miles per hour. If she jogs for 2 hours every morning, how many miles can she jog for five days?
PPO
Mira jogs at a rate of miles per hour, and she jogs for hours each day, then she will have jogged total distance of 10 miles (2 hours 5 miles/hour). If she does this for days, then she will have jogged total distance of miles (10 miles x days).
CORY
Mira jogs at a rate of miles per hour, so if she jogs for hours, she will have jogged 10 miles (2 hours 5 miles/hour). If she does this 5 days in a row, she will have jogged a total of 50 miles (10 miles/day x 5 days).
18

---

Table 5: Examples of GSM8K when PPO leads to distribution collapse.
Question
Adam bought 3 kilograms of nuts and 2.5 kilograms of dried fruits at store. One kilogram of nuts costs $12 and one kilogram dried fruit costs $8. How much did his purchases cost?
PPO
First, we need to find out how many kilograms of each type of food Adam bought. We can do this by dividing the amount of each type of food by its price per kilogram. 3 $12 = 2.5 $8 =. Then we can find the total cost by multiplying the number of kilograms by their prices: x $12 y $8.
Question: Sarah had 30 hours of vacation time left hefore she could take another day off. She wanted to know how many days she could go trip for without using up all her vacation time. Can you help Sarah figure out how many days she can go
trip?
Answer: To find out how many days Sarah can go on trip, we first need to divide her total vacation time (30) by the number of days she wants go the trip (let's 's call it x). 30 x 5. Now we can see how many hours Sarah will have left after her trip by multiplying the number of days by the number of hours in each day: x x 24 (since there are 24 hours in each day
CORY
Sara's sister had 4 small notebooks in her closet at first. Then she ordered 6 more notebooks, so now she has + = 10 notebooks. However, she lost notebooks, so now she has 10 2 = 8 notebooks.
Question
Belinda can throw ball at a speed of 20 feet/second. If the ball flies for 8 seconds before hitting the ground, and Belinda's border collie can run feet/second, how many seconds will it take the border collie to catch up to the ball?
PPO
To find out how long it will take the border collie to catch up the ball, need to use multiplication and division. We know that the ball travels 20 feet per second, so first we will divide by the speed of the dog to find out how far the dog needs to run in one second, 20 feet 5 feet/second 4 seconds.
Now will find out how long this takes in total by multiplying the number of seconds by the time the ball is in the air. 8 seconds x 4 seconds 32 seconds. Therefore, will take the border collie 32 seconds catch up the ball. Question: Sarah can paint room in 6 hours. If she works on it for 4 hours every day for week, how many hours will it take her to paint another similar-sized room?
Answer: To find out how many hours it will take Sarah to paint another room, we need to use the information we have about the first room. Let's call the size of the first room "x". We are told that she can paint "x" hours. To find out how many hours it will
CORY
We know that the ball travels at speed of 20 feet/second, it will cover 20 feet in second. The ball flies for seconds before hitting the ground, so it covers 20 160 feet.
The border collie runs at a speed of feet/second, so it will cover feet in second. To find out how long it takes the border collie to catch up the ball, divide the distance the ball traveled by the speed of the border collie: 160 feet 5 feet/second 32 seconds.
19

---

E Supplementary Experiments
E.1 Robustness of CORY
We conduct robustness experiments on the GSM8K dataset, focusing on the impact of learning rate. In Figures and 8, we set the learning rates to le-4 and 1c-5, respectively, using PPO and CORY for fine-tuning the Llama-2-7b-chat model, while keeping all other hyperparameters consistent with those in Appendix A. Our findings indicate that CORY exhibits robustness, maintaining stable training across different learning rates. Its KL. divergence and task reward converge around the 10th iteration, with the KL divergence remaining rate of le-4, PPO leads to distribution collapse. PPO achieves stable training and relatively good performance only with learning rate of 1e-5, but its KL divergence shows accelerating upward trend even after 100 iterations, indicating instability and the risk of distribution collapse.
relatively low value. In contrast, with a learning
15
0.16
Task reward
0.16
0.10
PR0-28 COROLLLAN-E as
Conoired
PROC COHPVILSTT7
0.05
0.00
0.00
Iteration
Bursaion
Task reward
(b) KL divergence
(c) Combined reward
Figure 7: Training curves under objective rewards GSMSK. The fine-tuned model is Llama-2-76 chat. Learning rate a is set to le-4.
PASLI CORVLLNEE-
0.200
0.20
Task reward
0.175
CORY-LLM
PPO 7B
0.18
0.14
0.125
CORVLIAM CORYLLVR--15
0.12
0.100
0.0
0.18
lteration
Barraton
100
Task reward
(b) KL divergence
(c) Combined reward
Figure 8: Training curves under objective rewards GSMSK. The fine-tuned model is Llama-2-7b- chat. Learning rate a is set to le-5.
20
0.30
0.20
Task raward
0:25
0.25
Constined
0.20
0.20
0.15
0.16
7PO-135
CORY-LMM--10
0.18
C029-1107-130
0.10
20
40
Hteration
60
80
100
20
40
Iteration
80
100
20
40
10
100
(a) Task reward
(b) KL divergence
(c) Combined reward
Figure 9: Training curves under objective rewards on GSM8K. The fine-tuned model is Llama-2-133 chat. Learning rate is to le-4.
20

---

0.28
Task rewerd
CORRYLLMM-11S
CONTF-LMM-115
roward
0.28
Dover LLant
0.25
CORYLUIN
0.22
0.20
20
60
80
100
20
40
60
80
100
0.16
Iteration
Iteration
Buration
(a) Task reward
(b) KL. divergence
(c) Combined reward
Figure 10: Training curves under objective rewards on GSM8K. The fine-tuned model is Llama-2- 13b-chat. Learning rate is set to le-5.
In Figures and 10, we again set the learning rates to le-4 and le-5, respectively, using PPO and CORY to fine-tune the Llama-2-13b-chat model, with all other hyperparameters consistent with those in Appendix A.1. CORY ensures stability at both learning rates, achieving good task reward and low KL divergence with a learning rate of le-4. Although the task reward does not improve with a learning rate of le-5, both the KL divergence and task reward curves stabilize, indicating that CORY avoids distribution collapse even under inappropriate hyperparameter settings. In contrast, PPO rapidly leads to distribution collapse with learning rate le-4. With a learning rate of le-5, the task reward increases steadily, but the KL divergence curve shows accelerating upward trend, indicating the risk of distribution collapse.
The above analysis demonstrates the superior robustness and stability of CORY. Furthermore, com- paring the KL divergence and task reward curves across figures reveals that PPO struggles to balance task reward and KI. divergence, whereas CORY consistently maintains balance between the two, as discussed Section 3.2.
E.2 Different Reward Settings
To investigate the effect of the reward setting in CORY, we modify the original reward setting Ryat + Rather to Realf ARother- Adjusting À in the set -5,-3,-1,1,333 varying degrees of competition and cooperation. Additionally, to mitigate the impact of reward magnitude training, normalized the reward values. As shown in Figure E.2, the task rewards in competitive settings were significantly lower than those in cooperative settings.
could represent
Task reward
10
50
75
100
25
75
100
lteration
lteration
(a) IMDB Task Reward
(b) IMDB KI. Divergence
0.15
2.0
Task reward
1.5
1000 Dosere
0.10
E 1.0
1110 Ooe witt-w4t
0.5
0.05
0.0
25
75
100
50
Iteration
Iteration
(c) GSM8K Task Reward
(d) GSM8K KL Divergence
Figure 11: Cooperative and competitive settings between two L.MSs. The figure only displays the performance curve of L.LM1 for clarity.
21

---

E.3 Additional Baselines
We conduct a comparison a strong baseline Elastic Reset (ER) [Noukhovitch al., 2024] and RE- INFORCE. ER-n denotes resetting every epochs, with set to 17 for reproducing ER'S 's performance on IMDB its original paper, and to 40 on GSM8K. As illustrated in Figure E.3, REINFORCE is more prone to distribution collapse than PPO. Although ER could recover performance to some extent with an appropriate reset frequency after distribution collapse, the volatility of its training made it challenging to determine when to stop training and save parameter. In contrast, CORY was able to stabilize the KL. divergence and task reward effectively.
60
2.5
MAKOPQ
Task reward
1.5
0.0
40
80
100
lteration
(a) IMDB Task Reward
(b) IMDB KI. Divergence
Task reward
0.20
0.15
0.10
0.05
0.00
25
50
75
100
25
50
75
100
Iteration
Iteration
(c) GSM8K Task Reward
(d) GSM8K Divergence
Figure 12: Comparing REINFORCE and ER on IMDB and GSM8K datasets.
F
Limitations
Although our method shows promising results in training robustness, policy optimality, and avoiding distribution collapse. requires duplicating the LLM into two copies, doubling the computational resources needed. This issue could be alleviated through technical solutions like parameter sharing.
G Broader Impacts
better RL tine-tuning method can improve the performance of LLMs in specialized tasks such as robot control and code generation. Assume well-constructed reward function, higher rewards do lead to better policies. There exists an optimal policy that maximizes this function. If RL fine-tuning is sufficiently advanced, it could theoretically improve the capabilities of an L.M in a specific task beyond the human level, once the reward exceeds a certain threshold.
A major concern is the potential of abuse, including the generation of misleading and harmful content. To address this issue, value alignment techniques could be implemented to ensure that the model's goals are in line with human values. In addition, implementing monitoring mechanisms, such as real-time detection of L.L -egeeeeee content, could be beneficial.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Both the abstract and introduction clearly state that our main contribution
and scope: extending the RL fine-tuning of LLMs into a sequential cooperative MARL framework.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Appendix F.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper includes detailed descriptions of the experimental setup, algorithms, model architectures in Section 4. and Section 4.2. Hyperparameters used are detailed in Appendix A.I. Pseudocodes are detailed in Appendix C.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
24

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have provided open access to both the data and code necessary reproduce our main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips...
(eglli//gidee/Codeeeee
irciiissssoooolllooeoo onPolicy) icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wttpp.ccpppuiiiieeeeeeeepppppppp wctpublic//aeccccccccccccccc ggwpi/gg///////eeee// /Ce///eeembbiiic cttssuubmisiion.. ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All experimental settings are clearly stated in Section 4. All hyperparameters are detailed in Appendix A.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All training curves in Section are plotted with the mean + std across three random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All necessary information are provided in Appendix A.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: beetps://neuriie /neuriss.....// cc weabblic/Ethisssiiiiiie
Answer: [Yes]
Justification:
Our research fully adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper thoroughly discusses both the potential positive and negative socictal impacts of our work in Appendix G.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
26

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not involve the release of data models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All existing assets used in the paper, including code, data, and models, have been properly credited to their original creators in Section
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release any assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "M3e1WRCTE6", "reviewer": "Reviewer_7TR2", "summary": "In this paper, the authors study a multi-agent organization for LLM learning. Specifically, they have two LLMs, with the second one responding to the same query given the query itself and the response generated by another LLM. They author shows this method can achieve a better tradeoff of the task reward and the KL penalty.", "questions": "1. As far as the reviewer is concerned, this paper is missing several important baselines to make their experiments more convincing.\n\n(1.1) The dual-LLM setup in the LLM debating literature. \n\nThis line of research is highly related to the paper. There, two LLMs response in turn to the previous generation of the other part, as well as to the original query. As the authors are explicitly considering a cooperative setting, the important question then is: which one is more effective, cooperation or competition, in terms of multi-agent LLM learning.\n\n(1.2) Mixture of experts. It appears a bit unfair to compare with a single-LLM PPO baseline--after all the authors are using two LLMs. \n\nThe question is whether this pioneer-observer structure is more effective than a simpler mechanism where opinions from multiple LLMs are aggregated together by recent research in MOE. \n\n2. Fig. 2 needs to be further explained. The major drawback is the lack of legend regarding the eta values. The reviewer is asking for the which two points (one on the CORY frontier and another on the PPO frontier) corresponding which value of eta.\n\nA minor question: For the sub-optimal frontiers of CORY and PPO, it is unclear what is shown is the training KL and reward or that of the testing time.", "limitations": "The reviewer lists some questions that make them curious, expecting some clarifications. it seem that some analysis or experiments can improve the understanding of the proposed method.\n\n(1) In the current setup, the pioneer and the observer possess the same architecture, ie, they are the same LLMs. Does this homogeneity matter? If the two LLMs are different, can the proposed method still work?", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The writing is clear and the question and proposed method is interesting.", "weaknesses": "Since this is an empirical paper, the expectation of the experiments, from its design to its implementation and results, would be higher. In the questions below, the reviewer mentions several points that might improve the experiments.\n\nIf the authors address the concerns about the experiments, the reviewer will consider rising their score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and valuable feedback! We’re glad you found our proposed method interesting and the writing clear. We would like to address your concerns below.\n\n## **R1-1** Dual-LLM Setup and Multi-LLM Learning\nThere has been some work studying the collaboration of multi-LLM (including dual-LLM setup) to accomplish a shared task [1], and we will add this to our reference. To our knowledge, CORY is the first work employing a multi-agent framework for the RL fine-tuning of LLM. As highlighted by the reviewer, the dynamics of competition and cooperation among agents are crucial in multi-agent learning framework.\n\nTo investigate these dynamics, we modified the original collective reward of CORY from $R_{self} + R_{other}$ to an $R_{self} + λ * R_{other}$ format. By adjusting the value and sign of $λ$, we could represent varying degrees of competition and cooperation. Additionally, to mitigate the impact of reward magnitude on training, we normalized the reward values. As demonstrated in Figure 1 in the attached PDF, across both the GSM8K and IMDB datasets, the task rewards in competitive settings were significantly lower than those in cooperative settings. \n\nBeyond experimental analyses, we believe that training multiple LLMs in a cooperative setting can be modeled as MARL problems with a shared team reward. Research in the MARL domain on credit assignment (e.g., VDN[2],QMIX[3]) could further enhance the training levels of multiple LLMs. This represents a promising direction for future development!\n \n[1] Wu, Qingyun, et al. \"AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation.\" ICLR 2024 Workshop on Large Language Model (LLM) Agents.\n \n[2] Sunehag, P., et al. \"Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward.\" Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems. 2018: 2085-2087.\n \n[3] Rashid, T., et al. \"QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning.\" International Conference on Machine Learning. PMLR, 2018: 4295-4304.\n \n\n\n## **R1-2** MoE Baseline\nWe agree that it is crucial to fairly compare the total model size as well as the number of models. Regarding the model size, our paper has already conducted ablation studies, comparing GPT2-l (770M) + CORY with GPT2-xl (1.5B) + PPO in Section 4.3, and LLaMA2 7b + CORY with LLaMA2 13b + PPO in Appendix E. The experiments demonstrate that the advantages of CORY do not come from an increase in training parameters.\n\nAs for expanding the number of LLMs involved in training, we designed a simple baseline inspired by the MoE approach: aggregating the outputs of two models and selecting the output with a higher estimated value based on the Q-values. The experimental results on the IMDB dataset are as follows:\n\n|             | Task-reward ↑ | KL divergence ↓ |\n|-------------|---------------|-----------------|\n| PPO         | 2.17          | 44.33           |\n| Aggregate   | 2.36          | 47.75           |\n| **CORY**    | **2.67**     | **15.18**      |\n\nAs you mentioned, we found that using MOE to aggregate responses from multiple LLMs can effectively enhance task rewards, resulting in a 9% improvement compared to PPO. However, by comparing with CORY's results, we discovered that CORY achieves a 23% improvement. This indicates that CORY provides a greater enhancement in performance because it significantly boosts the capability of individual LLM agents through collaborative training. These findings indicate that the key to the effectiveness of the pioneer-observer structure lies in facilitating interactions between models.\n## **R1-3** Fig.2 Explanation\nWe sincerely appreciate your valuable suggestion and apologize for any confusion caused. Regarding Fig. 2(c), the values of eta from left to right indeed are 1e-5, 1e-4, 1e-3, and 1e-2. We will ensure to annotate this clearly in the revised figure for better understanding.\n\nAs for the sub-optimal frontiers, these actually represent the testing KL and reward, which we will explicitly clarify in the revised manuscript to avoid any ambiguity.\n## **R1-4** Possible Heterogeneous Multi-LLM Training\nThis is indeed a fascinating question. Although the paper emphasizes that CORY is a plug-and-play alternative to RL fine-tuning (requiring only a duplicate of the model to be finetuned, without the need for extra auxiliary models during training), the applicability of CORY's core ideas in heterogeneous multi-LLM finetuning is a topic worthy of exploration. \n\nTo this end, we conducted experiments on the GSM8K dataset involving heterogeneous LLM training with LLaMA3-8B, LLaMA2-7B, and GPT-2. As shown in Figure 2 in the attached PDF, in the (LLaMA2-LLaMA3) combination, LLaMA2's training curve exhibits stability similar to that of original CORY, with its training KL values (consistently less than 1) significantly lower than those of single-PPO (which peak at 10). This suggests that the \"knowledge transfer\" mechanism can effectively alleviate the optimization pressure on LLaMA2 at the KL end. However, in the (LLaMA2-GPT2) combination, due to the insufficient capabilities of the GPT2 model, which produces many erroneous or empty responses, it fails to alleviate any training pressure on LLaMA2, resulting in a KL curve for LLaMA2 that closely mirrors the trend of single-PPO. The weaker model hinders the training of the stronger model, a similar outcome can also be observed in the (LLaMA2- LLaMA3) combination. It is evident that in a heterogeneous setting, CORY's underlying mechanisms still function, but the selection of heterogeneous LLMs and their interactions pose significant challenges, making this an issue worth further investigation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Fh3LDBmtN", "reviewer": "Reviewer_Syvc", "summary": "This paper presents CORY, a novel approach for fine-tuning large language models (LLMs) using a sequential cooperative multi-agent reinforcement learning framework. Traditional methods, primarily based on PPO and its variants, often show suboptimal performance and risk distribution collapse. CORY addresses these issues by duplicating the LLM into two agents: a pioneer and an observer. These agents generate responses based on queries and each other’s outputs, exchanging roles periodically to foster cooperation. Experiments with GPT-2 and Llama-2 on the IMDB Review and GSM8K datasets demonstrate that CORY outperforms PPO in terms of policy optimality, resistance to distribution collapse, and training robustness. This highlights its potential as a superior methodology for refining LLMs in real-world applications.", "questions": "1. Can the authors provide a more detailed discussion on the relationship between emergent communication in multi-agent reinforcement learning (MARL) and the method in this paper? Since the authors mention that this work is inspired by the concept that \"Languages develop through agent interactions and are shaped by societal and cultural influences,\" it would be beneficial to explore this connection further.\n2. The paper suggests employing a collective task reward to facilitate cooperation, as outlined in Eq. 4. What would happen if the task reward is not the sum of individual task rewards? Can the authors provide empirical results or theoretical analysis to show the influence of the reward design?\n3. Currently, the framework includes only two agents: the pioneer and the observer. Is it possible to introduce additional agents, and would doing so enhance performance? I hope the authors can discuss this possibility further.", "limitations": "Limitations have been discussed in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of multi-agent reinforcement learning into LLM fine-tuning is both novel and compelling, with a well-justified motivation behind the designed role exchange.\n2. The presentation is well-organized and clear. The comprehensive explanations and illustrative figures in section 3.2 enhance the understanding of the proposed method and its effectiveness.\n3. The empirical experiments are adequate and effectively support the proposed method and its underlying rationale.", "weaknesses": "1. The paper lacks a theoretical analysis of the introduced framework.\n2. The baseline comparison is primarily with single-agent PPO. However, there are many more advanced algorithms for LLM fine-tuning. It would be beneficial if the authors conducted additional experiments using other baseline fine-tuning algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive remark and insightful feedback! We’re glad you found our idea is both novel and compelling, our presentation is well-organized and clear, and our experiments are adequate and effectively support the proposed method. Below, we provide individual responses addressing your comments. \n \n## **R2-1** Theoretical Analysis \nOur paper has already provided detailed mathematical modeling for LLM RL fine-tuning (Section 2), token-level PPO (Appendix B), and CORY (Appendix C), along with an analysis from the perspective of multi-objective RL (Section 3.2). However, we are still willing to offer a theoretical analysis of why CORY works from the perspective of game theory. \n\nAs shown in Figure 5 in the supplementary PDF, the training phase of CORY can be modeled as an extensive-form two-player game. This is a game of perfect but incomplete information, meaning that the observer is aware of all historical information, but the two agents do not know any specific utility functions. The relationship between the utility functions of the two LLM agents determines the type of game. In CORY, the utility function format $U=R1+R2$ corresponds to a fully cooperative game, while the format $U=R1-R2$ corresponds to a zero-sum fully competitive game. \n\nIt is easy to prove that the policy combination corresponding to $U_{\\text{obs}} \\equiv U_{\\text{pio}} \\equiv 0$ is a Nash equilibrium of the zero-sum game, where $R_1 \\equiv R_2$, meaning that the rewards of both agents are the same under any query. They tend to find policies that are similar in performance but not optimal during training, which is also verified by subsequent experiments (in **R2-4**).\n\nConversely, the design of CORY cleverly utilizes the characteristics of the global optimal solution. Assuming that for any task query, there exists an answer that maximizes the task reward, it can be proven that 1. The game has Pareto optimums (constructive proof), and 2. Under the Pareto optimums, both agents receive the same and maximum rewards. However, there are two extremes among all the Pareto solutions. One is both the observer and the pioneer independently generate the optimal response (which is the true global optimum), and the other is the observer completely relies on the pioneer, merely repeating its optimal response. Thanks to the design of role-exchange and collective reward, the policies of the two agents will avoid evolving towards the second extreme and instead evolve towards the first extreme. \nThanks, we will add this section to the appendix. \n\n## **R2-2** Other RL Fine-tuning Baselines \nThank you for your suggestion. We have added two baselines into our analysis: REINFORCE and Elastic Reset. Here is the results of IMDB. \n|     | Task-reward ↑ | KL divergence ↓ | \n|-----|---------------|-----------------| \n| PPO | 2.17          | 44.33           | \n|REINFORCE|0.747|56.87| \n| ER-30 | 2.65       | 32.15          | \n| ER-17 | 2.63       | 21.73           | \n| ER-5 | 0.53       | 0.32          | \n| CORY | **2.67**    | **15.18**      | \n \nThe results indicate that CORY has the ability to maintain a stable balance between the two targets. The detailed training curves are shown in Figure 3 in the supplementary PDF. \n\n## **R2-3** Emergent Communication \nPrior work has discovered that defining communication protocols as vocabularies enables agents to spontaneously emerge their own languages during cooperation. Furthermore, incorporating human labels into the learning process of communication policy can facilitate the emergence (0 to 1) of communication policy that utilize natural language [1,2].\nCORY is equivalent to a two-player cooperative game with unidirectional communication grounded in natural language. The communication policies are pre-trained LLMs. The LLM/communication policy has already undergone one emergence during pre-training. By casting the RL fine-tuning of the LLM into a multi-agent context, we anticipate a secondary emergence (1 to 100) of the LLM. \n \n[1] Lazaridou, Angeliki, Alexander Peysakhovich, and Marco Baroni. \"Multi-Agent Cooperation and the Emergence of (Natural) Language.\" International Conference on Learning Representations. 2022. \n\n[2] Graesser, Laura, Kyunghyun Cho, and Douwe Kiela. \"Emergent linguistic phenomena in multi-agent communication games.\" 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019. Association for Computational Linguistics, 2019. \n\n## **R2-4** Total Reward Design \nThis is indeed an interesting question. We have noticed that Reviewer 7TR2 also raised a similar issue. Due to the space constraints of the rebuttal, please refer to **R1-1** for details on the relevant experimental setup. The results indicate that competitive scenarios significantly underperform cooperative ones. Furthermore, under complete competition (zero-sum game settings), the agents trained to a state of low KL divergence but also low performance (similar but suboptimal), which aligns with our previous game-theoretic analysis. \n\n## **R2-5** More LLM Agents \nYes! introducing more agents could lead to performance improvements, which represents an exciting direction for future work. We look forward to the participation of more researchers in this area. We have already developed preliminary concepts, which fundamentally concerns the design of communication topologies. There are several simple communication topologies that could be explored, including linear (chain-like), mesh (fully connected), and randomly connected. Recent research [3] has uncovered scaling laws of agent numbers in the collaboration among multiple Large Language Models (LLMs). It also discussed several other insight communication topologies for LLMs (e.g. tree-like, radial).  \n \n[3] Qian, Chen, et al. \"Scaling Large-Language-Model-based Multi-Agent Collaboration.\" arXiv preprint arXiv:2406.07155 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W3mvEuKODc", "reviewer": "Reviewer_F4Ro", "summary": "This paper presents a Reinforcement Learning (RL) methodology to fine-tune LLMs based on Multi-Agent RL agents. One agent acts as the observer, and the other acts as the pioneer. They share knowledge through two interactions: transfer learning and role-switching. They named this methodology CORY (Coevolving with the Other You). They tested their framework with out-the-shelf LLMs and proved that learning benefits from using CORY. They compared against proximal policy optimization (PPO) in the Multi-Agent setup. Ablation studies showed the effect of model size, knowledge transfer, and role exchange. They provided theory related to RL in the context of this text-generation task and qualitative studies.", "questions": "I think I lack intuition about how the reward signals work after the N steps. Also, I am unclear about the influence of the KL divergence and how far it is relevant not to diverge from the reference policy. Could you please give me your take on that?", "limitations": "The authors added a Limitations section in the appendix.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is easy to follow for a person knowledgeable in RL who understands LLMs at a basic level. Their methodology provides a viable method to improve LLM fine-tuning using RL. They used modern and relevant benchmarks not only on the side of RL (PPO) but also on the side of models.", "weaknesses": "The Limitations section would've been a good addition to the main corpus of the paper as it addresses a big possible concern I imagined with your methodology: you require twice the amount of resources to train both agents. The average reader would enjoy this change, as some don't jump in appendices. Also, I think your claim about Moore's Law might be arguable, as current requirements for computing power are growing beyond expectation with the surge in LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive remark and insightful feedback! We’re glad you found our method viable for improving LLM RL fine-tuning, our paper is easy to follow, and modern and relevant benchmarks are used in our experiments. Below, we provide individual responses addressing your comments.\n\n## **R3-1** Limitation Section\nThank you very much for your suggestion. We will carefully use the discussion regarding Moore's Law and will directly incorporate the discussion on computational power consumption into Section 6 of the revised manuscript.\n\n## **R3-2** Reward Signal in RL Fine-Tuning\nFrom a reinforcement learning perspective, the N steps to generate a sentence (assuming padding is included) constitute an episode. For PPO, at the end of the N steps, gradient ascent is used to optimize the token-level policy in the direction of maximizing the return for those N steps.\n\n## **R3-3** KL in RL Fine-Tuning\nThis is a profound question. KL divergence measures the distance between the current LLM’s token-level policy and the pre-trained token-level policy. The pre-trained token-level policy is quite delicate. Deviating too much can lead to the destruction of the language modeling in the pre-trained model, that is, distribution collapse. For instance, if the probability of the eos_token in the token-level policy is too low compared to the pre-trained one, it could prevent sentences from ending, or result in the repetition of the same word until the maximum output length is reached. Distribution collapse can largely be detected by KL divergence, and experimental results also reveal a strong correlation between them.\n\nTo avoid distribution collapse, current RL fine-tuning often needs to be early-stopped before reaching a catastrophic level of KL divergence. This places RL fine-tuning in a rather awkward position. \n\nOverall, the benefits of maintaining a low KL can be summarized as follows:\n1. Stable training without collapse: This means that RL can explore more data, increasing the chances of finding better policies. \n2. Efficient improvement: This means achieving as much performance improvement as possible with as few changes as possible to the reference policy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I2rFPAtYno", "reviewer": "Reviewer_chZh", "summary": "The paper introduces CORY, a novel reinforcement learning (RL) technique for fine-tuning language models that casts a multi-agent framework on the trained language model by duplicating it at initialization and then using the two copies to improve one another.\nEssentially, a \"pioneer\" gives the first guess, used as a reference to an \"observer\" which gives the final answer, then both agents are rewarded for both of their outputs in a cooperative fashion and trained independently with an underlying RL algorithm, here PPO.\nThe authors claim that CORY improves on single-agent PPO in terms of downstream performance, training stability, and robustness to distribution collapse.", "questions": "How did the authors select the hyperprameters for their methods?", "limitations": "Although in the appendix, the authors adequately state the limitations of their work and its broader impacts.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is easy to follow and includes enough details about the experiments.\nCasting the problem into a multi-agent framework to change the optimization dynamics is original and will likely influence other work to build on it.\nThe observations made by the authors about the multi-agent framework changing the dynamics training dynamics to make the observer more easily optimize for the reward while maintaining a low KL are interesting.\nThe method can potentially be applied on top of any RL algorithm, either with independent learners as shown in the paper or by introducing centralized training critic network sharing, etc.", "weaknesses": "Overall, I find the contribution of the paper to be original and likely to have impact, however, I also find a number major issues in the work that make it not ready for publication at this venue.\n\nUnclear contextualization and missing baselines:\n- One main contribution of the paper is that it finds a policy that has a better Pareto frontier while still using the sum of task reward and KL divergence, but the KL divergence is an artificial reward specific to current RL fintuning methods in language modeling. If the paper's method is to be highlighted from a language modeling perspective then I would expect it to compare to methods like [1] which also claim to obtain a better Pareto frontier. And more than that, I would expect the method to show its benefits in downstream tasks for example with AlpacaFarm [2] as although a better frontier could be seen a a proxy to robustness to model collapse, it isn't clear if it translates to better language modeling capabilities.\nOtherwise if the method is to be highlighted in general multi-objective RL fine-tuning tasks, then I would expect comparison to multi-objective RL methods.\n- (minor) The method can also be seen a self-improvement method, so the pepper would benefit from discussing and contrasting methods from the same family.\n\nAn insufficient number of benchmarks:\n- The authors claim to evaluate their method \"across diverse reward function landscapes, encompassing both subjective and objective categories\", but only one dataset per category is tested. Judging by the scope of the claims the authors make and that this is the only form of evidence provided, I would expect at least 3 or 5 datasets per category. \n\nHyperparameter choice:\n- The authors use a single set of hyperprameters, with no apparent justification. This is not enough to make claims about a method being better than another.\n- (minor) PPO with 1 epoch and 1 batch per epoch (as in the hyperprameter table) becomes just a policy gradient, so the authors seem to be effectively using a vanilla actor-critic with a GAE value estimator and (usually useless) value clipping.\n\nPresentation:\n- (minor) The wording of the claims made in the abstract and introduction \"policy optimality, \"resilience to distribution collapse\", and \"robustness during training\" is not reused in the experiments section which makes it hard to connect evidence to claims.\n- (minor) The $\\geq$ symbol in equation 6 between vector-values objectives is undefined. One would expect a definition of Pareto dominance.\n\n[1] Noukhovitch, Michael, et al. \"Language model alignment with elastic reset.\" Advances in Neural Information Processing Systems 36 (2024).\n[2] Dubois, Yann, et al. \"Alpacafarm: A simulation framework for methods that learn from human feedback.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and time spent! We’re glad you found our idea original and likely to influence other work to build on it, our paper easy to follow, and the observations from the multi-objective RL perspective interesting. We would like to address your concerns below.\n\n## **R4-1** Missing Baselines\nWe appreciate your concern regarding the need for more convincing baselines. In response, we have added an additional baseline, Elastic Reset(ER). The original ER paper also tested using the IMDB dataset. We adopted the experimental setup from the paper with parameters set to decay=0.995, reset_freq=17, and eta=0.001, successfully replicating their results (ER-17). Moreover, we conducted comparisons with two other sets of reset frequencies. The specific experimental results are as follows:\n\n|     | Task-reward ↑ | KL divergence ↓ |\n|-----|---------------|-----------------|\n| PPO | 2.17          | 44.33           |\n| ER-30 | 2.65       | 32.15          |\n| ER-17 | 2.63       | 21.73           |\n| ER-5 | 0.53       | 0.32          |\n| CORY | **2.67**    | **15.18**      |\n\nFurthermore, due to the reset mechanism, the training curve of Elastic Reset (ER) exhibits a peak-like shape, unlike the stable state observed in our case.\n\n## **R4-2** Insufficient Benchmarks\nThank you for the reminder. We will amend the statement \"across diverse reward function landscapes, encompassing both subjective and objective categories\" in the revised manuscript.\n\nRegarding the benchmarks, we believe they are currently persuasive enough. Reviewer Syvc mentioned that \"the experiments are adequate and effectively support the proposed method,\" and Reviewer F4Ro also noted that we have utilized modern and relevant benchmarks and models.\n\nHowever, we understand your concerns regarding the benchmarks. Therefore, we have included an additional, widely used dataset, Anthropic-HH. On this dataset, CORY shows its superior ability to balance task reward and KL divergence than all the baselines (PPO, Elastic reset, REINFORCE). We hope this can addresses your concerns. Training curves are in Figure 4 in the additional PDF.\n\n## **R4-3** Self-improvement Method\nIn the domain of LLM fine-tuning, there exist some self-improvement methods, such as SPIN [1] and Iterative DPO [2]. However, these approaches are offline algorithms in supervised manner, where self-improvement stems from making more comprehensive use of the dataset. \n\nIn contrast, RL as an online algorithm, is optimized on data generated by itself. In the MARL field, the success of AlphaGO and OpenAI Five has validated the effectiveness of the self-improvement among a multi-agent system that diverse data are generated through competition/cooperation among multiple agents. The problem we are concerned with is how to make the RL fine-tuning of LLM benefit from this self-improvement in multi-agent system [3, 4]. Therefore, CORY introduces a completely different self-improvement mechanism into the fine-tuning of LLMs.\n\n[1] Chen, Zixiang, et al. \"Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models.\" Forty-first International Conference on Machine Learning.\n\n[2] Yuan, Weizhe, et al. \"Self-Rewarding Language Models.\" Forty-first International Conference on Machine Learning.\n\n[3] A social network for AI. Nat Mach Intell 5, 1175 (2023). https://doi.org/10.1038/s42256-023-00769-4\n\n[4] Duéñez-Guzmán, Edgar A., et al. \"A social path to human-like artificial intelligence.\" Nature Machine Intelligence 5.11 (2023): 1181-1188.\n## **R4-4** Hyperparameter Choice\nThe parameter settings for fine-tuning GPT2 followed the default configuration in TRL for the IMDB dataset, while the fine-tuning for LlaMA2 primarily adhered to the guidelines established by StackLlama. In pursuit of a fair comparison, all parameters were carefully chosen to balance the stability and performance of PPO. And a grid search was conducted over learning rates and eta coefficients within the ranges lr[1e-6, 1e-5, 1e-4] and eta[1e-3, 1e-2, 1e-1, 0.2, 0.3] respectively, to select the parameters that yielded the most stable training for PPO. Given CORY's robustness to parameter variations (as evidenced by the results of adjusting learning rates and kl coefficients presented in Appendix E), the parameters for PPO, with the exception of the learning rate, were directly applied to CORY. In the GSM8K dataset, we adjusted the learning rate for CORY. Important parameters include lr and kl_coef, while parameters such as mini_batch, ppo_epoch are considered to be of relative lesser importance.\n\n## **R4-5** PPO Hyperparameter Concern\nYes, it is acknowledged that a strictly on-policy PPO is equivalent to A2C. However, the hyperparameter 'Epoch' in our hyperparameter table does not refer to PPO epochs. In our experiments, the ppo_epochs were actually set to 4, which aligns with the standard configuration of PPO. We will update our parameter table to prevent similar misunderstandings.\n## **R4-6** Feedback on the Presentation\nThanks for your suggestion! Regarding your first point, our paper already contains clear experiments related to \"policy optimality\" (Sections 4.1 & 4.2), \"resilience to distribution collapse\" (Sections 4.1 & 4.2), and \"robustness during training\" (Appendix E). We will explicitly add these descriptions to the experimental section of our paper.\n\nAs for your second point, we will include definitions of the \">=\" symbol in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our gratitude to all the reviewers for their contributions and insightful comments. We appreciate that all the reviewers find our writing and presentation is clear and well-organized.  
We are encouraged by the reviewers' appreciation that our idea is novel and interesting (Syvc, 7TR2), that our method is viable and possibly influence other work to build on it (F4Ro, chZh), that the analysis from the multi-objective RL is comprehensive and interesting (Syvc, chZh), that the experiments using modern and relevant benchmarks are adequate (Syvc, F4Ro), and includes enough details about the experiments (chZh). 
 
To help address the concerns of the reviewers and facilitate further discussion, we are attaching a PDF with five figures that we reference in each of our reviewer-specific rebuttals. The figures are:   
**Figure 1**: Training curves of cooperative and competitive settings between two LLMs on IMDB and GSM8K datasets. This is mostly in response to Reviewer 7TR2 and Reviewer Syvc.   
**Figure 2**: Training curves of heterogeneous LLM settings under the CORY framework on GSM8K dataset. This is mostly in response to Reviewer 7TR2.   
**Figure 3**: Comparisons to REINFORCE and Elastic Reset on IMDB and GSM8K datasets. This is mostly in response to Reviewer Syvc and Reviewer chZh.   
**Figure 4**: Training curves on Anthropic-HH. This is mostly in response to Reviewer chZh. 

**Figure 5**: Game-theoretic modeling of CORY. This is mostly in response to Reviewer Syvc.

Meta-review: While there were some significant ethical concerns to walk through (e.g. misinformation creation) and some concern over the lack of evaluation and RL algorithm diversity, these were largely resolved during the rebuttal process. The expanded limitation section addresses the ethical concerns very thoughtfully, and I believe PPO is sufficient for the purposes of this paper.
Decision: Accept (poster)