Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learning

Authors: hao  ma
tianyi  hu
zhiqiang  pu
boyin  liu
xiaolin  ai
yanyan  liang
min  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 10:48:19

Abstract: Reinforcement learning (RL) has emerged as a pivotal technique for fine-tuning large language models (LLMs) on specific tasks. However, prevailing RL fine-tuning methods predominantly rely on PPO and its variants. Though these algorithms are effective in general RL settings, they often exhibit suboptimal performance and vulnerability to distribution collapse when applied to the fine-tuning of LLMs. In this paper, we propose CORY, extending the RL fine-tuning of LLMs to a sequential cooperative multi-agent reinforcement learning framework, to leverage the inherent coevolution and emergent capabilities of multi-agent systems. In CORY, the LLM to be fine-tuned is initially duplicated into two autonomous agents: a pioneer and an observer. The pioneer generates responses based on queries, while the observer generates responses using both the queries and the pioneer’s responses. The two agents are trained together. During training, the agents exchange roles periodically, fostering cooperation and coevolution between them. Experiments evaluate CORY's performance by fine-tuning GPT-2 and Llama-2 under subjective and objective reward functions on the IMDB Review and GSM8K datasets, respectively. Results show that CORY outperforms PPO in terms of policy optimality, resistance to distribution collapse, and training robustness, thereby underscoring its potential as a superior methodology for refining LLMs in real-world applications.

Main paper: 
 introduction: 
 Large language models (LLMs) have achieved impressive success across diverse downstream tasks, including dialogue systems [Ouyang et al., 2022 , Touvron et al., 2023] , code generation [Roziere et al., 2023] , and robotic control [Driess et al., 2023 , Brohan et al., 2023] . However, as the capabilities of LLMs advance, the challenges associated with further performance gains become increasingly intricate. Fine-tuning LLMs for specific tasks presents a significant challenge, prompting recent exploration of LLM fine-tuning paradigm such as supervised fine-tuning (SFT) [Wu et al., 2021] , reinforcement learning (RL) fine-tuning [Shojaee et al., 2023] , and direct preference optimization encompassing all possible tokens. The w ∈ V represents a specific token within this vocabulary. The state space S ⊂ V M , where V M is the combination space of M tokens. The action space A ⊂ V N , where V N is the combination space of N tokens. M and N are the max token lengths for state and action, respectively. A state s ∈ S is a concatenation of token sequence s = (w 1 , w 2 , . . . , w M ). An action a ∈ A is the output of a causal language model, construed as a concatenation of token sequence a = (w 1 , w 2 , . . . , w N ). The states and actions are padded with pad token if the real length is less than the maximum length. The reward function r : S × A → R assigns a numerical score to a sequence of tokens, which can be considered as a typical sparse reward problem within the context of RL. The state transition function P : S × V → S describes a deterministic transition of states according to the auto-regressive paradigm. At each step, a predicted token is concatenated with the state of last step: s i+1 = (s i , w i+1 ) = (s 0 , w 1:i+1 ), where s 0 denotes a tokenized user's input for a causal language model, and w 1:i = (w 1 , w 2 , . . . , w i ) denotes a token sequence up to the i-th token. Then, the token-level policy of a causal language model can be encapsulated within π(w i |s 0 , w 1:i-1 ).And the sentence-level policy is defined as a joint policy:π(a|s 0 ) = N i=1π(w i |s 0 , w 1:i-1 ).(1)The reward function r(•, •) is related to a specific task (e.g., safety alignment [Liu, 2023 , Ji et al., 2024] , code generation [Shojaee et al., 2023 , Liu et al., 2023] ). A task reward is only obtained after N steps of decision-making via token-level policy. Under such a sparse reward, RL is prone to over-optimisation, resulting in distributional collapse of the language model. To mitigate the risk of distributional collapse, it is common practice to incorporate token-level KL penalties into the reward function, which serves to constrain the deviation of the language model from its original distribution [Go et al., 2023 , Zheng et al., 2023] .r(s i , w i ) =-ηKL(π θ (•|s 0 , w 1:i-1 ), π 0 (•|s 0 , w 1:i-1 )) i < N r(s 0 , a) -ηKL(π θ (•|s 0 , w 1:i-1 ), π 0 (•|s 0 , wEQUATIONwhere η is the KL coefficient, r(s i , w i ) represents the token-level combined reward function. For each token, a KL penalty is imposed based on the KL divergence between current policy π θ (•|s 0 , w 1:i-1 ) and initial policy π 0 (•|s 0 , w 1:i-1 ). Only after predicting the final token, does the reward model yield a task-specific reward r(s 0 , a).Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Both the abstract and introduction clearly state that our main contribution and scope: extending the RL fine-tuning of LLMs into a sequential cooperative MARL framework.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations in Appendix F.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
coevolving with the other you (cory): 
 To extend the RL fine-tuning of LLMs to a cooperative MARL framework, the LLM to be fine-tuned in CORY is initially duplicated into two copies, each is treated as an autonomous agent. Then, two roles, a pioneer and an observer, are assigned to these two LLM agents. We design two fundamental mechanisms to facilitate the coevolution between the two agents. The first design is knowledge transfer. The LLMs asynchronously take action, with the pioneer transferring its response (action) to the observer. The observer then utilizes this information to guide its own decision. The second design is role exchange. Once the observer achieves a satisfactory performance, it exchanges roles with the pioneer. In the following, we provide a comprehensive description of each element, and the pipeline of our method is shown in Figure 1 .Knowledge Transfer. To enable collaboration between the two LLM agents for improved response generation, we introduce a knowledge transfer mechanism. Given a query denoted as s 0 , the pioneer acts first and generates a response denoted as a 1 . Subsequently, the observer receives both the original query s 0 and the pioneer's response a 1 to generate its own response a 2 . This sequential interaction facilitates knowledge transfer, where the observer leverages the pioneer's output to guide its own generation process, potentially leading to a superior response due to the in-context learning capabilities of LLMs. The sentence-level policies of the pioneer and observer can be formulated as follows: a 1 ∼ π pio (•|s 0 ), a 2 ∼ π obs (•|s 0 , a 1 ).(3) During the training process, the parameters of the pioneer and the observer are optimized separately through an RL algorithm such as PPO. A cooperative relationship exists between the two LLM agents. To facilitate this collaboration, CORY employs a collective task reward, calculated as the sum of individual task rewards:EQUATIONwhich implies that both the pioneer and the observer receive rewards from each other's improvement. Following the form of Equation 2, we add r CORY and the KL penalty to construct a whole reward signal. Similar to Ni et al. [2022] , we find that a partially correct reference can also be beneficial for the observer. Hence, it is not necessary for the pioneer to generate a high-quality response.Role Exchange. During training, the observer may develop a prompt bias due to consistently receiving inputs in the form of (s 0 , a 1 ). This reliance on prompts that combine the original query with the pioneer's response, hinders the observer's ability to generate responses independently. To address this issue, we introduce a role exchange mechanism. This mechanism involves exchanging the roles of the pioneer and observer periodically during training:EQUATIONwhere swap is initialized as F alse, and reverse periodically. This exchange ensures that both the LLMs experience both roles (pioneer and observer) multiple times throughout the training process. Through this role exchange mechanism, they are forced to adapt to both prompt formats: s 0 alone and the combined format (s 0 , a 1 ). This allows us to use either LLM individually during inference. From a representational learning perspective, this role exchange mechanism encourages the LLMs to develop a unified representation for s 0 and (s 0 , a 1 ). This unified representation captures the essential information from the task query, regardless of the specific prompt format presented during training or inference.These two key mechanisms in CORY act as a form of bootstrapping. The two LLM agents collaborate, with the observer potentially learning better policies by leveraging the pioneer's output. Role exchange ensures both the LLMs benefit from this collaborative learning, similar to cooperative learning among humans. Importantly, CORY is an algorithm-agnostic approach, meaning it can be theoretically compatible with various RL algorithms beyond PPO. Additionally, CORY offers the advantages of simplicity in implementation and seamless integration with existing frameworks, making it a plug-and-play solution. The derivation of the CORY's policy update can be found in Appendix B, and the detailed pseudocodes are provided in Appendix C. Following the explanation of CORY in Section 3.1, this section provides an empirical demonstration of why the proposed method surpasses the single-agent RL fine-tuning method.In fact, RL fine-tuning with KL penalty inherently formulates a multi-objective reinforcement learning problem. The LLM agent strives to concurrently maximize the task reward and minimize the KL divergence. Unfortunately, these two objectives may be in opposition to one another. This is because maximizing the task reward will inevitably lead to the output distribution deviating from the pretrained model, resulting in an increase in KL divergence. Hence, the optimization process seeks a trade-off between the task reward and the KL divergence, ideally driving the policy towards a Pareto frontier [Ngatchou et al., 2005] . This frontier covers all achievable policies where no policy can improve on one objective without sacrificing performance on the other. Formally, the Pareto frontier can be defined as:EQUATIONwhere J r (π) = E π [ T t=0 γr(s t , a t )]. r(s, a) ∈ R m is a vector-valued reward function and Π denotes the set of all policies. Given a fixed reference vector ω ∈ Ω ⊆ R m , one could scalarize the multi-objective reward into a single objective by using the weighted sum ω T r(s, a). Under this preference weighting, the ideal outcome for the policy is to converge to a point on the Pareto frontier, as illustrated by the black dots in Figure 2 (a).However, due to the inherent complexities of natural language, achieving perfect policy convergence to the Pareto frontier is often intractable. Nevertheless, by adjusting the preferences, these sub-optimal policies can still form a frontier as illustrated in Figure 2 (b). For simplicity, we term it the sub-optimal frontier. Our hypothesis is that the sub-optimal frontier achieved by CORY lies closer to the true Pareto frontier compared to that achieved by single-agent RL method.To verify this hypothesis, we fine-tune the Llama-2-7b-chat model on the grade school math 8K (GSM8K) dataset [Cobbe et al., 2021b] using both PPO and CORY. We measure the KL divergence and the task reward obtained by each policy after convergence. By adjusting the preference, i.e., η in Equation 2, we are able to generate sub-optimal frontiers for both the methods, as illustrated in Figure 2(c ). It is important to note that the Y-axis represents the negative KL divergence (larger values indicate better performance). As expected, the sub-optimal frontier achieved by CORY consistently outperforms that of PPO, empirically validating the hypothesis.Our analysis through the lens of multi-objective RL offers valuable insights into the effectiveness of CORY. The knowledge transfer mechanism inherently addresses the optimization challenges faced by the observer. By leveraging the reference response provided by the pioneer, the observer actually experiences a guided optimization process. Such guided process can alleviate the optimization pressure on the task reward side, and prioritize improvement on the KL penalty side. However, since the observer's policy during training takes both the task query and the pioneer's response as inputs, the optimized policy is not the one we really want (we need the policy which only takes the task query as input), resulting in the prompt bias issue. The role exchange mechanism can effectively address this issue, and transfer the skills learned by the observer back to the pioneer, reducing the pioneer's optimization pressure. Notably, CORY demonstrates significantly better stability and robustness compared to single-agent RL method (See details in Section 4.2 and Appendix E.1). It consistently achieves a lower KL divergence between the fine-tuned and pre-trained models while maintaining strong performance on the target task, signifying a better trade-off between the two objectives.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification:We have provided open access to both the data and code necessary to reproduce our main experimental results.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 This section systematically investigate the performance of CORY across two types of reward functions: subjective reward function and objective reward function. Subjective reward functions are reward models trained on data capturing human preferences. They essentially translate the human sentiment or judgment into a numerical reward signal that guides alignment. Objective reward functions are pre-defined rule-based functions, typically established by domain experts. This categorization reflects real-world scenarios where reward functions might be learned from human preferences or manually crafted by domain experts. Prompts used in experiments are detailed in Appendix A.2.Task Setup. To evaluate our method under the subjective reward setting, we select the IMDB Review dataset [Tripathi et al., 2020] . This dataset contains 50K <text,label> pairs, with the training set and the test set each contains 25K pieces of data. The texts in the IMDB dataset are movie reviews, and the labels are the binary sentiment classification labels. The distilbert-imdb model4 trained on the dataset is employed as the reward model. We fine-tune GPT2-Large (774M)5 by using single-agent PPO (single-PPO) and CORY respectively. In addition, GPT2-XL (1.5B)6 is fine-tuned by using single-PPO as an ablation on model size. In this task, we randomly sample text snippets from the IMDB dataset. The first 2 to 8 tokens (representing the beginning of the review) are retained as prompts for sentiment completion. The LLMs generate continuations that transform the prompts into positive sentiment comments. After that, the reward model evaluates the generated text to assign a sentiment score. The objective is to maximize the average sentiment score of the completed comments. Examples of this task are detailed in Appendix D.In the experiments, each method undergoes 100 training iterations using a batch size of 256. For simplicity, GPT2-Large and GPT2-XL fine-tuned by single-PPO are termed as PPO-GPT-2-l and PPO-GPT-2-xl, respectively. GPT-2-Large that fine-tuned by CORY are referred to CORY-LLM1 and CORY-LLM2, where the former one is the LLM that initialized as the pioneer, and the latter one is the LLM that initialized as the observer. Results and Analysis. We monitor the training process by visualizing task reward, KL divergence, and a combined reward function that incorporates both the above objectives. Denoted as r c (s 0 , a), the combined reward function can be expressed as r c (s 0 , a) = r(s 0 , a) + η * KL(s 0 , π θ , π 0 ), where r(s 0 , a) and KL(s 0 , π θ , π 0 ) are the sentence-level task reward part and the KL penalty part, respectively. And the KL penalty part can be calculated as KL(s 0 , π θ , π 0 ) = i=0,1,...,N -KL(π θ (•|s 0 , w 1:i-1 ), π 0 (•|s 0 , w 1:i-1 )). It is important to note that, the actual reward used for training in CORY is not the combined reward. The actual training reward not only includes the KL penalty and the task reward from the target agent, but also includes the task reward from the other agent. In fact, the combined reward r c (s 0 , a) is the real overall objective that needs to be optimized, and can be aligned with the single-agent RL fine-tuning, making it easier to compare performance of all the methods.The training curves of task reward, KL divergence, and the combined reward are illustrated in Figure 12 . The results show that single-PPO and CORY achieve similar task reward levels after 100 training iterations. However, the curve of KL divergence related to single-PPO is significantly higher than that of CORY, reaching more than twice the level of CORY after all the training iterations. This indicates CORY's ability to achieve similar task reward levels with a smaller deviation from the pre-trained policy. Moreover, it can be observed that the curves of CORY-LLM1 and CORY-LLM2 are very close, indicating that the two LLM agents initially playing different roles finally achieve very similar performance levels at the end of the training. Consistent with the motivation of CORY, both the fine-tuned LLM agents can be used to finish tasks individually, which verifies the effectiveness of the bootstrapped learning and coevolution principles in CORY.Finally, Figure 12 (c) visually confirms CORY's advantage in combining the two objectives. The combined reward curve for CORY consistently rises, indicating its effectiveness in simultaneously improving task reward and minimizing KL divergence. Conversely, PPO's combined reward curve exhibits a decreasing trend, suggesting its struggle in balancing these objectives. Hyperparameters used for both single-PPO and CORY are detailed in Appendix A.1.Task Setup. To evaluate our method under a rule-based objective reward function, we select the GSM8K task [Cobbe et al., 2021a] . GSM8K comprises 8.79K high-quality, linguistically diverse grade school math word problems, with 7.47K allocated for training and 1.32K for testing. For each question in the dataset, a response is obtained via LLM. The precise answer is extracted from the responses using a regular expression, typically the final set of numbers in the response. If the number in question matches the ground truth as recorded in the dataset, a reward of 1 is awarded. Conversely, if the number is incorrect, a reward of 0 is given. The Llama-2-7b-chat7 model is selected as the pre-trained model. To reduce the training overhead, the model is quantised to 4-bit. For simplicity, the 4-bit Llama-2-7b-chat model fine-tuned with single-PPO is referred to as PPO-Llama-2. The copied models fine-tuned with CORY are referred to CORY-LLM1 and CORY-LLM2, where the former is the LLM that initialized as the pioneer, and the latter is the LLM that initialized as the observer. Examples of this task are detailed in Appendix D. In contrast, CORY demonstrates a significantly more stable task reward curve, consistently outperforming single-PPO. What's more, CORY achieves a considerably lower KL divergence compared to single-PPO, facilitating faster convergence. This characteristic is particularly valuable in the fine-tuning context, as it allows CORY to achieve similar or even better task rewards without significant modifications to the original parameter distributions.Furthermore, the combined reward curves visually confirm CORY's superiority over single-PPO. CORY's ability to effectively balance the two objectives is reflected in its steadily increasing combined reward. Conversely, single-PPO's struggle with balancing the objectives manifest as a decreasing combined reward and training instability.In addition, we conduct a comparative analysis of models fine-tuned with distinct methods and a pre-trained model on the GSM8K test set as shown in Figure 5 . The evaluation metric utilized is pass@k, which generates k corresponding repetitions for a sample and passes if at least one is correct.The test results demonstrate that the CORY fine-tuned 4bit Llama-2-chat-7b could achieve a pass@1 of 18% on GSM8K test dataset.In ablation experiments, we ablate the influence of model size, knowledge transfer, and role exchange under the subjective reward setting on IMDB review dataset. For method names depicted in Figure 6 , REx indicates role exchange, KT indicates knowledge transfer, LLM1 and LLM2 refer to LLMs who are initialized as the pioneer and the observer respectively. Ablation on Model Size. Our method employs two models during training, with the total parameters trained being doubled in comparison to single-PPO. In order to ablate whether the enhancement of CORY is derived from the expansion of the model parameters, an additional fine-tuning of GPT2-XL (1.5B) with single-PPO is conducted on the IMDB dataset, which has twice the number of parameters as GPT2-Large. The results are presented in Figure 12 . While the task reward of the model rapidly reaches its maximum value, the KL penalty part does not exhibit a notable improvement compared to GPT2-Large. The KL divergence continues to increase, leading to the collapse of the distribution.Ablation on Knowledge Transfer. We maintain role exchange, and the two models still share a collective task reward (Equation 4), but disable knowledge transfer. This resembles PPO with individual queries as inputs. However, without the observability of the pioneer's outputs, this equivalent to adding noise to the PPO reward signal. Consequently, the task rewards become unstable, and the KL divergences are higher compared to CORY as shown in Figure 6 . This highlights the importance of observability for framing RL fine-tuning as a true multi-agent cooperation problem.Ablation on Role Exchange. We maintain knowledge transfer but disable role exchange. As evident from Figure 6 , both LLMs achieve good task rewards, but their KL divergences are much higher than that of CORY. Notably, the observer LLM exhibits significantly lower KL divergence compared to the pioneer LLM. This observation highlights a fascinating phenomenon in cooperative learning:by receiving the pioneer's response, the observer can effectively optimize the KL divergence. This suggests that the observer leverages the pioneer's exploration to refine its policy while maintaining good performance, potentially leading to a more stable learning process. 
related work: 
 The most related topic is reinforcement leanring from human feedback (RLHF). InstructGPT [Ouyang et al., 2022] fine-tunes GPT-3 like models [Brown et al., 2020] to enhance helpfulness by combining SFT with RL based on human preference dataset. Askell et al. [2021] trains a preference model for aligning the LLM with human values. It argues that ranked preference modeling proves to be the most effective training objective for distinguishing between desirable and undesirable LLM behaviors. Bai et al. [2022] incorporates an iterative online training mode where preference model and LLM are updated weekly using fresh human feedback data. Existing research acknowledges the inherent complexity, instability, and hyperparameter sensitivity of RLHF, particularly when employing PPO Zheng et al. [2023] . Several works have attempted to address these challenges by introducing maxentropy regularization [Wen et al., 2024] , hyperparameter tuning [Zheng et al., 2023] , and reward shaping [Yang et al., 2024a] . However, these methods does not show significant improvement over the vanilla PPO algorithm. This inspires us to explore alternative method from a different perspective that extent the RL fine-tuning of LLMs to a cooperative MARL problem.Another related topic is MARL. Under the interaction relationship (cooperation, competition, mixed), multi-agent could spontaneously emerge complex and diverse policies, so as to solve the complex problems that single-agent reinforcement learning is difficult to solve. For example, in  
discussion: 
 Experimental evidence suggests that CORY yields more stable and superior performance in RL finetuning. This can be attributed to our extension of single-agent RL fine-tuning into a cooperative MARL version. In this section, we delve into a discussion of how the multi-agent learning can benefit LLM fine-tuning. The primary benefit is that multi-agent learning encourages the coevolution of LLMs through collective living, social relationships and major evolutionary transitions [Duéñez-Guzmán et al., 2023] . This process generates a variety of new data, which further facilitates coevolution. This mechanism contributes to many breakthroughs in games AI, such as Go [Silver et al., 2016 , 2017 , Clark and Storkey, 2015] , StarCraft II [Vinyals et al., 2019] , and Diplomacy [Bakhtin et al., 2022] .In this paper, we investigate the application of cooperative MARL to address challenges in RL fine-tuning. Cooperative MARL fine-tuning appears to increase training robustness and prevent distribution collapse. While we concentrate on cooperation, competitive MARL, especially populationbased methods, represents a promising direction for future research. These approaches create an auto-curriculum mechanism driven by a natural arms race, which propels agent learning and enables mastery of complex tasks. Besides the interaction paradigm, the scale of agents is crucial to emergence. While we examine a setting involving two LLMs, incorporating more LLMs in MARL fine-tuning is an intriguing prospect for future studies.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: All experimental settings are clearly stated in Section 4. All hyperparameters are detailed in Appendix A.1.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 In this paper, we extend the RL fine-tuning of LLMs to a sequential cooperative MARL framework.To this end, we duplicate the pre-trained LLM into two LLM agents with different roles, and design two key mechanisms: knowledge transfer and role exchange. These mechanisms enable the two LLM agents to learn collaboratively, and after the fine-tuning process, either the LLM agent can be chosen to perform the task independently. We also provide an in-depth analysis of RL fine-tune from the perspective of multi-objective RL, revealing the existence of a Pareto frontier between KL divergence and task reward. We empirically illustrate that CORY has an advantage over single-agent RL method in approaching the Pareto frontier. Experiment results indicate that CORY surpasses PPO regarding policy optimality, resilience to distribution collapse, and robustness during training, highlighting its potential as an advanced method for improving LLMs in practical applications.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: All training curves in Section 4 are plotted with the mean ± std across three random seeds.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Our paper thoroughly discusses both the potential positive and negative societal impacts of our work in Appendix G.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper does not involve the release of data or models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All existing assets used in the paper, including code, data, and models, have been properly credited to their original creators in Section 4.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: The paper does not release any new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The code repository we utilize is TRL8 . Our experimentation employs 2 AMD EPYC 7773X CPUs and 8 NVIDIA A6000 GPUs (48GB each). Leveraging a single GPU, CORY can achieve fullprecision RL fine-tuning of GPT2-XL on the IMDB Review dataset within 12 hours. With 4 GPUs, CORY can accomplish the RL fine-tuning of a 4-bit quantized Llama-2-7b-chat model on GSM8K within 4 hours.The hyperparameter settings for fine-tuning GPT2 followed the default configuration in TRL for the IMDB dataset, while the hyperparameter setting of Llama-2 primarily adhered to the guidelines provided by StackLlama. To ensure a fair comparison, all hyperparameters were carefully selected to balance the stability and performance of PPO. A grid search was conducted over α and η, with the sets α 1e-6, 1e-5, 1e-4 and η 1e-3, 1e-2, 1e-1, 0.2, 0.3, respectively, to identify the hyperparameter that yielded the most stable training for PPO. Given CORY's robustness to hyperparameters (Appendix E.1), most PPO hyperparameters, except for the learning rate α, were applied directly to CORY. For the GSM8K dataset, In the GSM8K dataset, we adjusted the learning rate α for CORY. Another fun, witty, frothy For CORY's observer, we use pioneer's response (blue) to complete the sentence as a reference for observer, and retype the first few words of the comment at the end of the prompt for observer to complete.I can make this sentence 'Another fun, witty, frothy cut different from the usual.' more positive: Another fun, witty, frothy GSM8K. The prompts used in GSM8K are as follows. For PPO or CORY's pioneer, we provide a example question and answer. This is followed by a question from the dataset (brown). Then the prompt ends with 'Answer:' to guide the LLM to answer.Question: Shawn has five toys. For Christmas, he got two toys each from his mom and dad. How many toys does he have now? Answer: Shawn started with 5 toys. If he got 2 toys each from his mom and dad, then that is 4 more toys. 5 + 4 = 9. Question: The civic league was hosting a pancake breakfast fundraiser. A stack of pancakes was $4.00 and you could add bacon for $2.00. They sold 60 stacks of pancakes and 90 slices of bacon. How much did they raise? Answer:For CORY's observer, the question is followed by 'Reference' (blue), which is the pioneer's response.Finally, it also ends with 'Answer' to guide the model to answer.Question: Shawn has five toys. For Christmas, he got two toys each from his mom and dad. How many toys does he have now? Answer: Shawn started with 5 toys. If he got 2 toys each from his mom and dad, then that is 4 more toys. 5 + 4 = 9. Question: The civic league was hosting a pancake breakfast fundraiser. A stack of pancakes was $4.00 and you could add bacon for $2.00. They sold 60 stacks of pancakes and 90 slices of bacon. How much did they raise? Reference: To find out how much the Civic League raised, we need to multiply the number of stacks of pancakes by the cost of each stack. So, 60 x $4 = $240. Then, we multiply the number of slices of bacon by the cost of each slice. So, 90 x $2 = $180. Therefore, the Civic League raised a total of $240 + $180 = $420. Answer:We first derive the formula of Q-function when fine-tuning LLM with PPO. The token-level reward function r is given in Equation 2.Q π (s i , w i ) = E wi+1,...,w N ∼π N -i k=0 γ k r(s i+k , w i+k ) = E wi+1,...,w N ∼π N -i k=0 γ k r(s i+k , w i+k ) -ηE wi+1,...,w N ∼π N -i k=0 γ k KL [π(• | s i+k ), π 0 (• | s i+k )] = E wi+1,...,w N ∼π γ N -i r(s 0 , a) -ηE wi+1,...,w N ∼π N -i k=0 γ k KL [π(• | s i+k ), π 0 (• | s i+k )] = E wi+1,...,w N ∼π γ N -i r(s 0 , a) -η N -i k=0 γ k KL [π(• | s i+k ), π 0 (• | s i+k )] .(7) For CORY, pioneer and observer share the same task reward r CORY , but their Q-functions have slightly different forms due to their different inputs. For simplicity, we define a uniform state s0 ≜ (s 0 , a 1 ) for the observer, and s0 ≜ s 0 for the pioneer. Then, denoting the parameterized policy as π θ , the Q-functions for them can be expressed in an uniform way.Q π θ (s i , w i ) = E wi+1,...,w N ∼π θ γ N -i r CORY (s 0 , a 1 , a 2 ) -η N -i k=0 γ k KL [π θ (• | si+k ), π 0 (• | si+k )] .(8) Similarly, CORY's uniform state value function can be expressed asEQUATIONIn practice, both the pioneer and the observer in CORY are optimised using PPO independently.During the training phase, a value head is attached to the last hidden layer of the policy network to predict the current state value. The loss function is:EQUATION)where V ϕ (s i ) is the predicted state value, ϕ represents the parameters of the corresponding value network. For policy loss, the optimisation objective with clip is used.EQUATIONwhere π θ old is the older policy that collects data. The importance ratio π θ (wi|si) π θ old (wi|si) is used to estimate Âπ θ under π θ on data collected via π θ old . It reflects how much the current policy deviates relative to the older policy. Âπ θ is the advantage function, givenEQUATIONUltimately, with a value loss coefficient β, the pioneer and the observer are fine-tuned by maximising the following objective L(θ, ϕ) = L P π θ -βL V π θ .(13) Ideally, after the optimisation, the optimal token-level policy π * is obtained, which in turn naturally leads to the optimal sentence-level policy. for each s 0 in B Q do 6:a 1 ∼ π pio (•|s 0 ; θ 1 ).r pio ← r(s 0 , a 1 ).a 2 ∼ π obs (•|s 0 , a 1 ; θ 2 ).r obs ← r(s 0 , a 1 ).10:r CORY ← r pio + r obs .11:Set s0 ← s 0 and update memory D pio ← D pio ∪ {(s 0 , a 1 , r CORY )}.12:Set s0 ← (s 0 , a 1 ) and update memory D obs ← D obs ∪ {(s 0 , a 2 , r CORY )}. D T ← ∅.3: Compute advantage estimate Âπ θ via GAE on D T . (Equation 12) 16: 13) 17: 13) 18: end for D Qualitative Analysis of Experiment Results.for i = 1, 2, • • • , N do 4: r KL ← KL(π θ (•|s 0 , a[1 : i -1]), π 0 (•|s 0 , a[1 : i -1])). 5: s i ← (s 0 , a[1 : i -1]). 6: a i ← a[i]. 7: s i+1 ← (s 0 , a[1 : i]). 8: if i < N then 9: r i ← -η • r KL . 10: else 11: r i ← r CORY -η • r KL . 12: end if 13: D T ← D T ∪ {(s i , a i , r i , s i+1 )}.θ ← θ + α • ∇ θ L(θ, ϕ). (Equationϕ ← ϕ + α • ∇ ϕ L(θ, ϕ). (EquationWe compare GPT2-Large models fine-tuned with PPO and CORY on IMDB Review dataset, along with the original model (Table 3 ). The input review snippet consists of the first few words of a movie review. The goal of LLMs is to complete the sentence in a positive direction. Comparing results before and after fine-tuning, sentences are often incomplete and occasionally contain grammatical errors due to the limitations of GPT2-Large. However, this does not affect our horizontal comparison on the same baseline. It is evident that the sentences generated by the fine-tuned models are indeed more positive. Comparing PPO and CORY, we find that PPO experiences distribution collapse. While its task reward is comparable to CORY, its KL divergence is significantly larger (Figure 12 ). Sentences generated by CORY are more positive. Although there are occasional grammatical errors, they are similar to those in the pre-trained model, indicating that CORY effectively avoids distribution collapse.We also compared the Llama-2-7B-chat models fine-tuned with PPO and CORY on GSM8K. Due to PPO's sensitivity to parameters, which results in either stable training or distribution collapse, we divided the comparison into two tables. When PPO trains stably (Table 4 ), the quality of the generated answer is similar to CORY, though slightly less accurate. When PPO experiences distribution collapse (Table 5 ), it tends to generate particularly long outputs until reaching the maximum token limit. This is because the probability of the end-of-sentence token 
 in the token-level policy decreases significantly compared to its initial value, preventing sentence completion. Due to the distribution collapse, the in-context learning ability of PPO is also impaired. It generates another Question after generating an Answer. In contrast, CORY's performance is much more stable. George wants to borrow $100 from a loan shark. The finance fee starts at 5% and doubles every week. If George plans to borrow for 2 weeks, how much in fees will he have to pay?The finance fee starts at 5%, so after one week it would be 10%. After two weeks, the fee would be 20%. So, George would have to pay (100 x 10%) + ( 100x 20%) = 200 in fees.At first, the finance fee is 5%. So George will pay $5 in fees. Then, the next week, the fee will be doubled to $10. George will pay another $10 in fees. Therefore, George will pay a total of $15 in fees. Bill is trying to control the pests in his garden. Each spider he introduces eats 7 bugs, and each time he sprays the garden he reduces the total bug population to 80% of what it was previously. If the garden has 400 bugs to start, and Bill sprays once and introduces 12 spiders, how many bugs are left?The number of bugs in the garden before Bill sprayed is 400. When Bill sprays, the total bug population is reduced to 80% of its original value, which means there are 320 bugs left (400 x 80%). Then, Bill introduces 12 spiders, which eat 7 bugs each, so there are 7 + 84 = 91 bugs left in the garden.First, let's find out how many bugs are left after Bill sprays by multiplying the previous number of bugs by 80%. So, 400 x 0.8 = 320 bugs. Now, let's find out how many bugs each spider can eat. Each spider can eat 7 bugs, so 12 spiders can eat 7 x 12 = 84 bugs. Since Bill introduced 12 spiders, there are now 320 -84 = 236 bugs left in the garden.Mira jogs every morning. She jogs 5 miles per hour. If she jogs for 2 hours every morning, how many miles can she jog for five days?If Mira jogs at a rate of 5 miles per hour, and she jogs for 2 hours each day, then she will have jogged a total distance of 10 miles (2 hours x 5 miles/hour). If she does this for 5 days, then she will have jogged a total distance of 50 miles (10 miles x 5 days).CORY Mira jogs at a rate of 5 miles per hour, so if she jogs for 2 hours, she will have jogged 10 miles (2 hours x 5 miles/hour). If she does this 5 days in a row, she will have jogged a total of 50 miles (10 miles/day x 5 days).Table 5 : Examples of GSM8K when PPO leads to distribution collapse.Adam bought 3 kilograms of nuts and 2.5 kilograms of dried fruits at a store. One kilogram of nuts costs $12 and one kilogram of dried fruit costs $8. How much did his purchases cost?PPO First, we need to find out how many kilograms of each type of food Adam bought. We can do this by dividing the amount of each type of food by its price per kilogram. 3 / $12 = x 2.5 / $8 = y. Then we can find the total cost by multiplying the number of kilograms by their prices: x x $12 + y x $8. Question: Sarah had 30 hours of vacation time left before she could take another day off. She wanted to know how many days she could go on a trip for without using up all her vacation time. Can you help Sarah figure out how many days she can go on a trip? Answer: To find out how many days Sarah can go on a trip, we first need to divide her total vacation time ( 30) by the number of days she wants to go on the trip (let's call it x). 30 / x = 5. Now we can see how many hours Sarah will have left after her trip by multiplying the number of days by the number of hours in each day: x x 24 (since there are 24 hours in each day CORY Sara's sister had 4 small notebooks in her closet at first. Then she ordered 6 more notebooks, so now she has 4 + 6 = 10 notebooks. However, she lost 2 notebooks, so now she has 10 -2 = 8 notebooks.Belinda can throw a ball at a speed of 20 feet/second. If the ball flies for 8 seconds before hitting the ground, and Belinda's border collie can run 5 feet/second, how many seconds will it take the border collie to catch up to the ball? PPO To find out how long it will take the border collie to catch up to the ball, we need to use multiplication and division. We know that the ball travels 20 feet per second, so first we will divide by the speed of the dog to find out how far the dog needs to run in one second. 20 feet / 5 feet/second = 4 seconds. Now we will find out how long this takes in total by multiplying the number of seconds by the time the ball is in the air. 8 seconds x 4 seconds = 32 seconds. Therefore, it will take the border collie 32 seconds to catch up to the ball. Question: Sarah can paint a room in 6 hours. If she works on it for 4 hours every day for a week, how many hours will it take her to paint another similar-sized room? Answer: To find out how many hours it will take Sarah to paint another room, we need to use the information we have about the first room. Let's call the size of the first room "x". We are told that she can paint "x" in 6 hours. To find out how many hours it willWe know that the ball travels at a speed of 20 feet/second, so it will cover 20 feet in 1 second. The ball flies for 8 seconds before hitting the ground, so it covers 8 x 20 = 160 feet. The border collie runs at a speed of 5 feet/second, so it will cover 5 feet in 1 second. To find out how long it takes the border collie to catch up to the ball, we divide the distance the ball traveled by the speed of the border collie: 160 feet / 5 feet/second = 32 seconds. In Figures 9 and 10, we again set the learning rates to 1e-4 and 1e-5, respectively, using PPO and CORY to fine-tune the Llama-2-13b-chat model, with all other hyperparameters consistent with those in Appendix A.1. CORY ensures stability at both learning rates, achieving good task reward and low KL divergence with a learning rate of 1e-4. Although the task reward does not improve with a learning rate of 1e-5, both the KL divergence and task reward curves stabilize, indicating that CORY avoids distribution collapse even under inappropriate hyperparameter settings. In contrast, PPO rapidly leads to distribution collapse with a learning rate of 1e-4. With a learning rate of 1e-5, the task reward increases steadily, but the KL divergence curve shows an accelerating upward trend, indicating the risk of distribution collapse.The above analysis demonstrates the superior robustness and stability of CORY. Furthermore, comparing the KL divergence and task reward curves across all figures reveals that PPO struggles to balance task reward and KL divergence, whereas CORY consistently maintains a balance between the two, as discussed in Section 3.2.To investigate the effect of the reward setting in CORY, we modify the original reward setting R self + R other to a R self + λR other . Adjusting λ in the set {-5,-3, -1,1,3,5}, we could represent varying degrees of competition and cooperation. Additionally, to mitigate the impact of reward magnitude on training, we normalized the reward values. As shown in Figure E.2, the task rewards in competitive settings were significantly lower than those in cooperative settings. EQUATIONEQUATIONEQUATIONWe conduct a comparison to a strong baseline Elastic Reset (ER) [Noukhovitch et al., 2024] and RE-INFORCE. ER-n denotes resetting every n epochs, with n set to 17 for reproducing ER's performance on IMDB as its original paper, and to 40 on GSM8K. As illustrated in Figure E.3, REINFORCE is more prone to distribution collapse than PPO. Although ER could recover performance to some extent with an appropriate reset frequency after distribution collapse, the volatility of its training made it challenging to determine when to stop training and save parameter. In contrast, CORY was able to stabilize the KL divergence and task reward effectively. Although our method shows promising results in training robustness, policy optimality, and avoiding distribution collapse, it requires duplicating the LLM into two copies, doubling the computational resources needed. This issue could be alleviated through technical solutions like parameter sharing.A better RL fine-tuning method can improve the performance of LLMs in specialized tasks such as robot control and code generation. Assume a well-constructed reward function, higher rewards do lead to better policies. There exists an optimal policy that maximizes this function. If RL fine-tuning is sufficiently advanced, it could theoretically improve the capabilities of an LLM in a specific task beyond the human level, once the reward exceeds a certain threshold.A major concern is the potential of abuse, including the generation of misleading and harmful content.To address this issue, value alignment techniques could be implemented to ensure that the model's goals are in line with human values. In addition, implementing monitoring mechanisms, such as real-time detection of LLM-generated content, could be beneficial.The "agents" here refer to individuals who make decisions and take actions in the context of reinforcement learning[Sutton and Barto, 2018].https://huggingface.co/lvwerra/distilbert-imdbhttps://huggingface.co/openai-community/gpt2-largehttps://huggingface.co/openai-community/gpt2-xlhttps://huggingface.co/meta-llama/Llama-2-7b-chat-hfhttps://github.com/huggingface/trl 


Decision: While there were some significant ethical concerns to walk through (e.g. misinformation creation) and some concern over the lack of evaluation and RL algorithm diversity, these were largely resolved during the rebuttal process. The expanded limitation section addresses the ethical concerns very thoughtfully, and I believe PPO is sufficient for the purposes of this paper.

Reviews: review: paper_summary: paper_summary: In this paper, the authors study a multi-agent organization for LLM learning. Specifically, they have two LLMs, with the second one responding to the same query given the query itself and the response generated by another LLM. They author shows this method can achieve a better tradeoff of the task reward and the KL penalty.
strength_weakness: strength_weakness: Since this is an empirical paper, the expectation of the experiments, from its design to its implementation and results, would be higher. In the questions below, the reviewer mentions several points that might improve the experiments.

If the authors address the concerns about the experiments, the reviewer will consider rising their score.
questions: questions: 1. As far as the reviewer is concerned, this paper is missing several important baselines to make their experiments more convincing.

(1.1) The dual-LLM setup in the LLM debating literature. 

This line of research is highly related to the paper. There, two LLMs response in turn to the previous generation of the other part, as well as to the original query. As the authors are explicitly considering a cooperative setting, the important question then is: which one is more effective, cooperation or competition, in terms of multi-agent LLM learning.

(1.2) Mixture of experts. It appears a bit unfair to compare with a single-LLM PPO baseline--after all the authors are using two LLMs. 

The question is whether this pioneer-observer structure is more effective than a simpler mechanism where opinions from multiple LLMs are aggregated together by recent research in MOE. 

2. Fig. 2 needs to be further explained. The major drawback is the lack of legend regarding the eta values. The reviewer is asking for the which two points (one on the CORY frontier and another on the PPO frontier) corresponding which value of eta.

A minor question: For the sub-optimal frontiers of CORY and PPO, it is unclear what is shown is the training KL and reward or that of the testing time.
limitations: limitations: The reviewer lists some questions that make them curious, expecting some clarifications. it seem that some analysis or experiments can improve the understanding of the proposed method.

(1) In the current setup, the pioneer and the observer possess the same architecture, ie, they are the same LLMs. Does this homogeneity matter? If the two LLMs are different, can the proposed method still work?
paper_summary: paper_summary: This paper presents CORY, a novel approach for fine-tuning large language models (LLMs) using a sequential cooperative multi-agent reinforcement learning framework. Traditional methods, primarily based on PPO and its variants, often show suboptimal performance and risk distribution collapse. CORY addresses these issues by duplicating the LLM into two agents: a pioneer and an observer. These agents generate responses based on queries and each other’s outputs, exchanging roles periodically to foster cooperation. Experiments with GPT-2 and Llama-2 on the IMDB Review and GSM8K datasets demonstrate that CORY outperforms PPO in terms of policy optimality, resistance to distribution collapse, and training robustness. This highlights its potential as a superior methodology for refining LLMs in real-world applications.
strength_weakness: strength_weakness: 1. The paper lacks a theoretical analysis of the introduced framework.
2. The baseline comparison is primarily with single-agent PPO. However, there are many more advanced algorithms for LLM fine-tuning. It would be beneficial if the authors conducted additional experiments using other baseline fine-tuning algorithms.
questions: questions: 1. Can the authors provide a more detailed discussion on the relationship between emergent communication in multi-agent reinforcement learning (MARL) and the method in this paper? Since the authors mention that this work is inspired by the concept that "Languages develop through agent interactions and are shaped by societal and cultural influences," it would be beneficial to explore this connection further.
2. The paper suggests employing a collective task reward to facilitate cooperation, as outlined in Eq. 4. What would happen if the task reward is not the sum of individual task rewards? Can the authors provide empirical results or theoretical analysis to show the influence of the reward design?
3. Currently, the framework includes only two agents: the pioneer and the observer. Is it possible to introduce additional agents, and would doing so enhance performance? I hope the authors can discuss this possibility further.
limitations: limitations: Limitations have been discussed in the paper.
paper_summary: paper_summary: This paper presents a Reinforcement Learning (RL) methodology to fine-tune LLMs based on Multi-Agent RL agents. One agent acts as the observer, and the other acts as the pioneer. They share knowledge through two interactions: transfer learning and role-switching. They named this methodology CORY (Coevolving with the Other You). They tested their framework with out-the-shelf LLMs and proved that learning benefits from using CORY. They compared against proximal policy optimization (PPO) in the Multi-Agent setup. Ablation studies showed the effect of model size, knowledge transfer, and role exchange. They provided theory related to RL in the context of this text-generation task and qualitative studies.
strength_weakness: strength_weakness: The Limitations section would've been a good addition to the main corpus of the paper as it addresses a big possible concern I imagined with your methodology: you require twice the amount of resources to train both agents. The average reader would enjoy this change, as some don't jump in appendices. Also, I think your claim about Moore's Law might be arguable, as current requirements for computing power are growing beyond expectation with the surge in LLMs.
questions: questions: I think I lack intuition about how the reward signals work after the N steps. Also, I am unclear about the influence of the KL divergence and how far it is relevant not to diverge from the reference policy. Could you please give me your take on that?
limitations: limitations: The authors added a Limitations section in the appendix.
paper_summary: paper_summary: The paper introduces CORY, a novel reinforcement learning (RL) technique for fine-tuning language models that casts a multi-agent framework on the trained language model by duplicating it at initialization and then using the two copies to improve one another.
Essentially, a "pioneer" gives the first guess, used as a reference to an "observer" which gives the final answer, then both agents are rewarded for both of their outputs in a cooperative fashion and trained independently with an underlying RL algorithm, here PPO.
The authors claim that CORY improves on single-agent PPO in terms of downstream performance, training stability, and robustness to distribution collapse.
strength_weakness: strength_weakness: Overall, I find the contribution of the paper to be original and likely to have impact, however, I also find a number major issues in the work that make it not ready for publication at this venue.

Unclear contextualization and missing baselines:
- One main contribution of the paper is that it finds a policy that has a better Pareto frontier while still using the sum of task reward and KL divergence, but the KL divergence is an artificial reward specific to current RL fintuning methods in language modeling. If the paper's method is to be highlighted from a language modeling perspective then I would expect it to compare to methods like [1] which also claim to obtain a better Pareto frontier. And more than that, I would expect the method to show its benefits in downstream tasks for example with AlpacaFarm [2] as although a better frontier could be seen a a proxy to robustness to model collapse, it isn't clear if it translates to better language modeling capabilities.
Otherwise if the method is to be highlighted in general multi-objective RL fine-tuning tasks, then I would expect comparison to multi-objective RL methods.
- (minor) The method can also be seen a self-improvement method, so the pepper would benefit from discussing and contrasting methods from the same family.

An insufficient number of benchmarks:
- The authors claim to evaluate their method "across diverse reward function landscapes, encompassing both subjective and objective categories", but only one dataset per category is tested. Judging by the scope of the claims the authors make and that this is the only form of evidence provided, I would expect at least 3 or 5 datasets per category. 

Hyperparameter choice:
- The authors use a single set of hyperprameters, with no apparent justification. This is not enough to make claims about a method being better than another.
- (minor) PPO with 1 epoch and 1 batch per epoch (as in the hyperprameter table) becomes just a policy gradient, so the authors seem to be effectively using a vanilla actor-critic with a GAE value estimator and (usually useless) value clipping.

Presentation:
- (minor) The wording of the claims made in the abstract and introduction "policy optimality, "resilience to distribution collapse", and "robustness during training" is not reused in the experiments section which makes it hard to connect evidence to claims.
- (minor) The $\geq$ symbol in equation 6 between vector-values objectives is undefined. One would expect a definition of Pareto dominance.

[1] Noukhovitch, Michael, et al. "Language model alignment with elastic reset." Advances in Neural Information Processing Systems 36 (2024).
[2] Dubois, Yann, et al. "Alpacafarm: A simulation framework for methods that learn from human feedback." Advances in Neural Information Processing Systems 36 (2024).
questions: questions: How did the authors select the hyperprameters for their methods?
limitations: limitations: Although in the appendix, the authors adequately state the limitations of their work and its broader impacts.

Comments: title: Thanks for your response
Discussion period
Please Review Rebuttal
Please Review Rebuttal
comment: We thank all the reviewers for their valuable suggestions and active engagement during the discussion phase. Your time and effort have clarified our work, and we sincerely appreciate the constructive feedback.
Thank you for your positive feedback! Here is the information about the GGF-PPO experiment.

Regarding the implementation: GGF-PPO achieves reward scalarization via weighting, enforcing constraints on the weights assigned to multiple objectives. This includes the requirement for weights to follow a strictly descending order and for their sum to equal 1. Consequently, we adjusted the weights used in PPO's gradient computation and bypassed the original handling of the KL coefficient in the TRL library.

To ensure a fair comparison, we implemented the following adjustments: 
1. We maintained the hyperparameters of GGF-PPO, except for the KL coefficient, to be consistent with those of PPO. 
2. We ensured that the task reward and KL presented in the table represent actual measured values, rather than the weighted values utilized in PPO updates. 
3. We conducted experiments using weight combinations of [0.9, 0.1], [0.8, 0.2], [0.7, 0.3], and [0.6, 0.4], exploring both scenarios where task reward is prioritized and where KL is prioritized, resulting in a total of eight sets of experiments. The best-performing set of results was selected for inclusion in the table.
I thank the authors for the discussion of MORL. I'm happy to increase my score. I would just need more information about how GGF-PPO was tuned to obtain the results in the table and if this is fair to the rest of the methods.
Thank you for your timely response! We're glad to have addressed most of your concerns. We'd like to provide further clarification on multi-objective RL and baselines.

### 1. **Clarification on multi-objective reinforcement learning (MORL)** ###

From the perspective of RL, the RL fine-tuning of LLM indeed constitutes a multi-objective (specifically, dual-objective) RL problem. Algorithms specifically designed for MORL can be categorized into two main types: multi-policy methods and single-policy methods [1]. Multi-policy methods train multiple policies, each corresponding to a single combination of objectives [2]. This approach equates to altering the weights of KL and task reward in Section 3.2(constructing multiple combinations of objectives), training multiple policies through conventional RL methods, and selecting the more effective policies. However, as demonstrated in Fig.2, the performance of all policies (their Pareto frontier) is inferior to policy trained under any weight by CORY.

On the other hand, single-policy methods address multi-objective problems through reward scalarization [2]. In our paper, the baseline such as PPO, considers the weighted loss of both the task reward and KL during training, even dynamically adjusting the weight [3], which essentially represents a naive form of reward scalarization. To further alleviate your concerns, we conducted experiments on the IMDB dataset using GGF-PPO [4], which is one of the SOTA algorithms in the MORL domain. As demonstrated in the attached Table 1, GGF-PPO shows only a marginal performance improvement over PPO (a 3.2% increase in Reward and a 13.3% reduction in KL). Therefore, the effectiveness of single-policy methods is also limited. Compared to CORY, they fundamentally cannot leverage the interaction and emergent intelligence of multiple LLMs to unleash the potential of Multi-LLM fine-tuning.

Overall, we agree that the discussion on MORL is inevitable. However, we have already implicitly compared both multi-policy and single-policy methods in our manuscript. Thank you, we will incorporate this discussion into the appendix to explicitly discuss MORL algorithms.

Attached Table 1 is shown as follows:

|     | Task-reward ↑ | KL divergence ↓ |
|-----|---------------|-----------------|
| PPO | 2.17          | 44.33           |
| **GGF-PPO** | 2.24 |  38.45      |
| ER-30 | 2.646       | 32.152          |
| ER-17 | 2.626       | 21.73           |
| ER-5 | 0.5284       | 0.3237          |
| CORY | **2.668**    | **15.179**      |


### 2. **Clarification on baselines** ###

More baselines are always better. However, we believe these two strong baselines, PPO and ER, should be sufficient to evaluate our method. PPO remains a strong baseline for RL fine-tuning of LLMs. Although some work has explored SAC [5], the experimental results show limited improvement comparing to PPO. In terms of preventing distribution collapse for RL fine-tuning, there is work such as NLPO [6], but its effectiveness is not as strong as ER.

Thank you once again for your time. Should you have any further questions, please let us know.


[1]  Hayes C F, et al. "A practical guide to multi-objective reinforcement learning and planning." AAMAS (2022)

[2] Hwang M, et al. "Promptable behaviors: Personalizing multi-objective rewards from human preferences." CVPR(2024)

[3] TRL: https://github.com/huggingface/trl

[4] Siddique U, et al. "Learning fair policies in multi-objective (deep) reinforcement learning with average and discounted rewards." ICML(2020)

[5] Wen, Muning, et al. "Entropy-Regularized Token-Level Policy Optimization for Large Language Models." CoRR (2024).

[6] Ramamurthy, Rajkumar, et al. "Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization." ICLR (2023).
First, I apologize for the late response to your rebuttal; I read it some days ago and was trying to form a better idea through the other reviews and how you addressed them. I want to thank the authors for addressing the questions and concerns in my review. I better understood how the RL part works in this NLP application, and I find it interesting how it can keep contributing to the fine-tuning tasks that are much required for LLM. I appreciate your efforts in creating new benchmarks and baselines.

I will keep my score for the following reasons: my field of expertise is RL, and I don't feel confident giving you a higher score without a solid background in NLP. Also, my review didn't consider that you should add new baselines; for instance, after using PPO, I didn't consider it relevant to include REINFORCE, which is expected to underperform against PPO.

I wish you all the best.
The authors' response has addressed my major concern. I have increased my score accordingly.
The discussion period is almost over, so both authors and reviewers please respond to any unaddressed questions. Reviewers, be sure that you have all of the information you need from the authors, since after the 13th, they won't be able to respond.
Thank you for your valuable ethics review. We would like to address your concerns as follows.

## **E2-1** Elaborate on the responsible use ##
The motivation and methods of CORY are unrelated to harmful or discriminatory information. However, we need to emphasize that although the datasets used in our experiments contain positive information, there is a risk if the method is incorrectly applied to downstream tasks aimed at spreading disinformation or polarizing people. Therefore, we advocate for the prohibition of using and constructing datasets and reward models related to disinformation or polarizing people. Thanks, and we will include this discussion in the broader impacts section of the revised manuscript.

## **E2-2** Mitigation strategies ##
Thank you very much for your suggestions. Value alignment can be achieved using human-annotated data that reflects safety preferences. For instance, [1] constructed a large-scale safety alignment dataset containing over 1 million preference pairs. They trained a cost model and a reward model on this dataset and employed safe RL to fine-tune the Apaca7B model. This approach significantly reduces model bias and discrimination, thereby enhancing the model's overall safety. Thanks, and we will include this discussion in the broader impacts section of the revised manuscript.

## **E2-3** Highly biased behavior of RL-fine-tuned LLM 
In the context of RL fine-tuning, biased behavior does not imply that the LLM produces harmful information, but rather a collapse in model distribution. For example, if the probability of the eos_token (end-token) in the token-level policy is too low compared to the pre-trained one, it could prevent sentences from ending, or result in the repetition of the same word until the maximum output length is reached. We want to emphasize that without a contaminated dataset (filled with harmful information), fine-tuning through reward signals is more likely to cause model distribution collapse (see our paper's Appendix D), rather than a dedicated distribution capable of outputting harmful speech. Therefore, the key lies in the use of datasets and the reward model. We should prevent researchers from fine-tuning models on datasets and reward models with negative impacts using CORY. Thank you for your reminder, we will include a more in-depth discussion of ethical impacts in the broader impacts section of the revised manuscript.

[1] Ji, Jiaming, et al. "Beavertails: Towards improved safety alignment of LLM via a human-preference dataset." Advances in Neural Information Processing Systems 36 (2024).
Thank you for your valuable Ethics Review. We would like to address your concerns as follows:

## **E1-1** language emergence or evolution ##
Recently, there has been growing interest in how to make LLMs fine-tuning benefit from the co-evolution within multi-agent systems [1, 2]. Multi-agent systems generate diverse new data and policies, which may lead to the emergence of superior policies. To our knowledge, CORY is the first approach to extent RL fine-tuning into a MARL framework. As LLMs already possess fundamental language capabilities, what we emphasize by "Languages develop through agent interactions and are shaped by societal and cultural influences" is the "evolution" of language, i.e., evolve to fit downstream tasks.

## **E1-2** objective/subjective reward settings  ##
Thank you for your helpful suggestions. However, our objective reward functions refer to "pre-defined rule-based functions”,  which encompass not only mathematical reasoning but also unit test scores, compilation success rates (for code generation), exact matches (for mathematical reasoning), BLEU scores (for summarization), etc. Subjective reward functions are clearly defined as "models trained on data capturing human preferences." We believe this definition is sufficiently clear.

## **E1-3** Origin of the name CORY ##
Thank you for pointing this out. In fact, CORY is not an acronym but an abbreviation for easier dissemination,  originating from Coevolving with the OtheR You (CORY). We appreciate your note and will clearly state the origin of the CORY abbreviation in the introduction section of the revised manuscript.

## **E1-4** A deeper discussion of ethical impacts (Dual Use) ##
Thank you for your suggestion. We have conducted a more careful discussion on the ethical impacts through the lens of Dual Use and completed the checklist proposed by the article Thorny Roses. We will include this section in the broader Impacts section of the paper.

[1] A social network for AI. Nat Mach Intell 5, 1175 (2023).

[2] Duéñez-Guzmán, E.A., Sadedin, S., Wang, J.X. et al. A social path to human-like artificial intelligence. Nat Mach Intell 5, 1181–1188 (2023). 

The checklist we completed as proposed by Thorny Roses is shown below.

- **C1**  Did you explicitly outline the intended use of scientific artefacts you create? 

Yes. We developed CORY, extending the RL fine-tuning of LLMs into a sequential cooperative MARL framework. As a plug-and-play method, CORY can build on top of any common LLM RL fine-tuning algorithm, thus enhancing the performance of the original algorithm. Besides the method CORY, we did not create any datasets or reward models containing harmful information.

 - **C2**  Can any scientific artefacts you create be used for surveillance by companies or governmental institutions?
 
No. As an enhanced RL fine-tuning method, CORY is unrelated to surveillance by companies or governmental institutions.

- **C3** Can any scientific artefacts you create be used for military application? 

The motivation and method of CORY are unrelated to military applications. However, we must emphasize that although the datasets used in our experiments are filled with positive information, there is a risk if CORY is incorrectly applied to fine-tuning on datasets about military applications. Therefore, we call for the avoidance of using and constructing datasets and reward models for military purposes.

- **C4**  Can any scientific artefacts you create be used to harm or oppress any and particularly marginalised groups of society? 

We emphasize that there is a risk if CORY is incorrectly applied to datasets containing harmful and discriminatory information. Therefore, we call for the avoidance of using and constructing datasets and reward models with harmful or discriminatory information.

- **C5** Can any scientific artefacts you create be used to intentionally manipulate, such as spread disinformation or polarise people? 

We emphasize that there is a risk if this method is incorrectly applied to downstream tasks about spreading disinformation or polarizing people. Therefore, we call for the elimination of using and constructing datasets and reward models about disinformation or polarizing people.

- **C6** Did you access your institution’s or other available resources to ensure limiting the misuse of your research? 

Yes, we have accessed our institution to ensure limiting the misuse of our research, including but not limited to the promotion, use, and modification of this method.

- **C7** have you been provided by your institution with ethics training that covered potential mis-use of your research? 

Yes, we are confident that our institution has provided sufficient ethics training.

- **C8** Were the scientific artefacts you created reviewed for dual use and approved by your institution’s ethics board? 

Yes, the scientific artefacts we created have been reviewed for dual use and approved by our institution's ethics board.
We kindly ask the reviewer to read and respond to our rebuttal. During the rebuttal phase, we conducted new experiments that we believe address all the concerns raised regarding the paper and may merit an increase in the score. The experiments can be summarized as follows:

1. New Baselines: Introduced two new baselines: REINFORCE and a strong baseline, Elastic Reset [1]  (see attached PDF Figure 3).
2. New Benchmark: All baselines have been compared on the Anthropic HH benchmark (see attached PDF Figure 4).

If there are any outstanding issues, we would appreciate the opportunity to respond before the discussion period concludes. Thank you.

[1] Noukhovitch, Michael, et al. "Language model alignment with elastic reset." Advances in Neural Information Processing Systems 36 (2024).
We kindly ask the reviewer to read and respond to our rebuttal. During the rebuttal phase, we conducted new experiments that we believe address all the concerns raised regarding the paper and may merit an increase in the score. The experiments can be summarized as follows:

1. Dual-LLM Setup: Evaluated both cooperative and competitive settings (see attached PDF Figure 1).
2. Heterogeneous Setup: Implemented two different LLMs in CORY to assess the effectiveness of the proposed mechanism (see attached PDF Figure 2).
3. New Baselines: Introduced two new baselines: REINFORCE and a strong baseline, Elastic Reset [1]  (see attached PDF Figure 3).
4. New Benchmark: All baselines have been compared on the Anthropic HH benchmark (see attached PDF Figure 4).

If there are any outstanding issues, we would appreciate the opportunity to respond before the discussion period concludes. Thank you.

[1] Noukhovitch, Michael, et al. "Language model alignment with elastic reset." Advances in Neural Information Processing Systems 36 (2024).
I've read the author's response and I appreciate the additional discussion and clarifications. I would like to keep my score.
I thank the authors for the clarifications. Most of my concerns have been addressed.
I am increasing my score to 5 as I still believe the submission would benefit from better contextualization (should it be compared to multi-objective methods?) and more baselines (I appreciate the authors comparing to ER).