Discovering Preference Optimization Algorithms with and for Large Language Models

Authors: chris  lu
samuel  holt
claudio  fanconi
alex j chan
jakob  foerster
mihaela  van der schaar
robert tjarko lange
sakana  ai
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 14:19:40

Abstract: Offline preference optimization is a key method for enhancing and controlling the quality of Large Language Model (LLM) outputs.
Typically, preference optimization is approached as an offline supervised learning task using manually crafted convex loss functions. While these methods are based on theoretical insights, they are inherently constrained by human creativity, so the large search space of possible loss functions remains under-explored. We address this by performing LLM-driven *objective discovery* to automatically discover new state-of-the-art preference optimization algorithms without (expert) human intervention.  Specifically, we iteratively prompt an LLM to propose and implement new preference optimization loss functions based on previously evaluated performance metrics. This process leads to the discovery of previously unknown and performant preference optimization algorithms. The best performing of these we call *Discovered Preference Optimization* (DiscoPOP), a novel algorithm that adaptively blends logistic and exponential losses. Experiments demonstrate the state-of-the-art performance of DiscoPOP and its successful transfer to held-out tasks.

Main paper: 
 introduction: 
 Training Large Language Models (LLMs) usually involves starting with a model pre-trained on large text corpora and then fine-tuning it to match human preferences. Pre-trained, and even instruction fine-tuned LLMs, can generate harmful, dangerous, and unethical completions [Carlini et al., 2021 , Gehman et al., 2020] . To mitigate this and align an LLM with human values, we use human preference alignment through preference-ranked completion data. This approach has become an industry standard, popularized by reinforcement learning with human feedback (RLHF) [Christiano et al., 2017, RLHF] , and more recently, by offline preference optimization algorithms like direct preference optimization [Rafailov et al., 2023, DPO] and sequence likelihood calibration [Zhao Foundation Model Proposal Extend/Update Context & Fitness Inner Loop Optimization Run CODE-LEVEL JSON FOUNDATION MODEL PROPOSAL: {"thought": "Based on the previous outputs,...", "name": "cross_entropy_loss", "code": "def loss_fn( logits: torch.FloatTensor, targets: torch.FloatTensor ) -> torch.FloatTensor: loss = F.cross_entropy(logits, targets) return loss"} SYSTEM PROMPT: You are a machine learning researcher who is testing out different loss functions. <PROBLEM DESCRIPTION>.When you respond, output a JSON where the first key ("thought") corresponds to your thought process when designing the next function. The second key ("name") corresponds to the name of your next function.The last key ("code") corresponds to the exact python code that you would like to try. The corresponding performance is fed back to the language model, and we query it for the next candidate. Right.Performance of discovered objective functions on Alpaca Eval. et al., 2023, SLiC] , which cast the problem as a supervised learning objective. Many algorithms have been proposed in the literature for offline preference optimization, and it remains an open question which one performs best across tasks. While a strictly dominant algorithm may not exist, some algorithms likely exhibit generally improved performance. To date, all existing state-of-theart preference optimization algorithms [Rafailov et al., 2023 , Azar et al., 2023 , Zhao et al., 2023] have been developed by human experts. Despite their advancements, these solutions are inherently constrained by human limitations, including creativity, ingenuity, and expert knowledge.In this work, we aim to address these limitations by performing LLM-driven discovery to automatically generate new state-of-the-art preference optimization algorithms without continual expert human intervention in the development process. While previous works [Ma et al., 2023 , Yu et al., 2023] have used LLMs to design environment-specific RL reward functions, we discover general-purpose objective functions which can be used across various preference optimization tasks. More specifically, we iteratively prompt an LLM to propose new preference optimization loss functions and evaluate them, with the previously proposed loss functions and their task performance metric (in our case, MT-Bench scores [Zheng et al., 2024] ) as in-context examples. After performing this automatic discovery process, we catalogue high-performing loss functions and introduce a particularly strong one we call Discovered Preference Optimization (DiscoPOP), a new algorithm. To ensure robustness beyond MT-Bench, we validate DiscoPOP using AlapacaEval 2.0 [Dubois et al., 2024] , showing an improvement in win rates against GPT-4 from DPO (11.23% → 13.21%). Additionally, in separate, held-out, tasks such as summarization and controlled generation, models trained with the DiscoPOP loss outperform or perform competitively with existing preference optimization algorithms.Contributions: 1 ⃝ We propose an LLM-driven objective discovery pipeline to discover novel offline preference optimization algorithms (Section 3). 2 ⃝ We discover multiple high-performing preference optimization losses. One such loss, which we call Discovered Preference Optimization (DiscoPOP), achieves strong performance across multiple held-out evaluation tasks of multi-turn dialogue (AlpacaEval 2.0), controlled sentiment generation (IMDb) and summarization (TL;DR) tasks. 3 ⃝ We provide an initial analysis of DiscoPOP, which is a weighted sum of logistic and exponential losses, and discover surprising features. For example, DiscoPOP is non-convex. 
background: 
 Preference Optimization. Consider a pre-trained language model policy π θ and a dataset D = {(x i , y i w , y i l )} N i=1 consisting of prompts x and preference-ranked completions y w and y l . In this dataset, a human rater prefers y w over y l , denoted as y w ≻ y l . The task is to align π θ with the human values implicit in these preferences. Canonically, this has been achieved through reinforcement learning from human feedback [Christiano et al., 2017, RLHF] , an approach that proceeds in two phases: First, a reward modelling stage that learns a parameterized reward model r ϕ . By assuming a Bradley-Terry model [Bradley and Terry, 1952] of preferences, the probability of the data can be expressed as P (y w ≻ y l ) = exp r ϕ (y w , x)/(exp r ϕ (y w , x) + exp r ϕ (y l , x)), and subsequently simply optimized over ϕ through the maximum likelihood principle. The second stage of policy optimization employs a reinforcement learning algorithm to train the language model against the learned reward. Usually, a KL penalty is introduced between the model and the pre-RL reference policy π ref [Jaques et al., 2019 , Stiennon et al., 2020] to prevent over-optimization and straying too far from the original policy, resulting in the final objective: (1)Despite success in frontier models [Anthropic, 2023 , Gemini-Team, 2023] , deep RL has many implementations [Engstrom et al., 2019] and training challenges [Sutton, 1984 , Razin et al., 2023] that hinder its adoption. To simplify the whole process, direct preference optimization [Rafailov et al., 2023, DPO] aims to forego both the reward modelling and online RL procedure. Rewriting (1) with a decomposition of the KL term into:EQUATIONexpresses the problem as an entropy-regularised RL bandit task [Ziebart et al., 2008] , for which a known analytical solution exists: π * (y|x) = Z(x) -1 π ref (y|x) exp β -1 r ϕ (y, x) . By rearranging the reward, we can express the task as a binary classification problem based on the reward difference:min π θ E (yw,y l ,x)∼D f β • log π θ (y w |x) π ref (y w |x) -log π θ (y l |x) π ref (y l |x)r ϕ (yw,x)-r ϕ (y l ,x).(3)Here, we define the log ratio difference as ρ = log π θ (yw|x) πref(yw|x)log π θ (y l |x) πref(y l |x) . In DPO, the function f =log σ is derived as the negative log of the sigmoid function given the BT model assumptions. However, Tang et al. [2024] highlighted that more generally we can obtain a recipe for offline preference optimization algorithms by letting f : R → R be any scalar loss function. For example, setting f (x) = (x -1)2 , the squared loss function [Rosasco et al., 2004] yields IPO [Azar et al., 2023] , while employing the max-margin inspired hinge loss [Boser et al., 1992, Cortes and Vapnik, 1995] f (x) = max(0, 1x) produces SLiC [Zhao et al., 2023] .Meta-Optimization for Algorithm Discovery. The goal of meta-optimization (optimizing the optimization process) is to uncover novel learning algorithms using a data-driven process. Suppose that an algorithm uses an objective function f γ to train a model for K iterations, where γ denotes a set of meta-parameters. Meta-optimization searches for an objective that maximizes the expected downstream performance max γ E[η(π K )|train(f γ )] where η is a downstream performance metric. Unlike previous methods that rely on a predefined parameterization of γ (e.g., a neural network [Hospedales et al., 2021] or domain-specific language [Alet et al., 2020] ), we leverage LLMs to directly propose code-level objective functions in Python. This approach eliminates the need for a carefully designed search space and utilizes the extensive knowledge embedded in the LLM for flexible selection and mutation. 
llm-driven objective discovery: 
 Choosing an appropriate objective function is crucial for instilling capabilities into networks. Here, we detail our discovery process facilitated by LLM code-level objective function proposals: Initial Context Construction. In the initial system prompt, we 'burn-in' the LLM using several established objective functions given in code and their corresponding performance. Furthermore, we provide problem details and an example of the output response format as a JSON dictionary. LLM Querying, Parsing & Output Validation. We query the LLM, parse the response JSON, and run a set of unit tests (e.g. for valid output shapes) before starting a training run. If the parsing or unit tests fail, we resample a new solution after providing the error message as feedback to the LLM.Performance Evaluation. The proposed objective function is then evaluated based on its ability to optimize a model for a predefined downstream validation task. We refer to the resulting performance metric as η.Iterative Refinement. By using the performance provided as feedback, the LLM iteratively refines its proposals. In each iteration, the model synthesizes a new candidate loss function, exploring both variations of previously successful formulas and entirely new formulations that might improve upon the existing benchmarks. This iterative process is repeated for a specified number of generations or until convergence when a set of optimal loss functions is observed.We summarise this general objective discovery process in Figure 1 Run unit tests to check the validity of the candidate and resample if needed.Evaluate the objective function using the performance metric η 6:Update the LLM context with the performance data 7:LLM refines generation strategy based on the feedback 8: until convergence criteria are met or maximum generations are reached Small case study: Discovering supervised classification loss functions. Consider the case of supervised classification on the CIFAR-10 dataset as a simple starting example. We train a simple ResNet-18 for 5 epochs using the objectives proposed by GPT-4 [OpenAI, 2023] . After each training run we provide the LLM with the corresponding validation accuracy and query it for the next PyTorch-based [Paszke et al., 2017] candidate objective function. Figure 2 depicts the performance of the proposed objective functions across the discovery process. The different discovered objectives all outperform the standard cross-entropy loss. Interestingly, we observe that the LLM-driven discovery alternates between several different exploration, fine-tuning, and knowledge composition steps: Initially, the LLM proposes a label-smoothed cross-entropy objective. After tuning the smoothing temperature, it explores a squared error loss variant, which improved the observed validation performance. Next, the two conceptually different objectives are combined, leading to another significant performance improvement. Hence, the LLM discovery process does not perform a random search over objectives previously outlined in the literature but instead composes various concepts in a complementary fashion. Furthermore, the discovered objectives also generalize to different architectures and longer training runs. In Appendix D.3 we show that this process of discovery is robust to the choice of sampling temperature and prompt/context construction. 
discovering offline preference optimization objectives: 
 In this section, we run our LLM-driven discovery to automatically generate new state-of-the-art preference optimization algorithms. In this section we use our LLM-driven discovery method to discover new objective functions f for offline preference optimization, as defined in Section 2 and Equation (3). Specifically, at each generation i, GPT-4 generates PyTorch [Paszke et al., 2017] code of candidate objective function f i . Each objective function takes as input the variables of {log π θ (y w |x), log π ref (y w |x), log π θ (y l |x), log π ref (y l |x)}, and returns a scalar. For each proposed objective f i , we check if f i is valid with a unit test.For each valid generated objective function f i , we finetune an LLM and then collect a performance evaluation score. Specifically, we build on top of the 'alignment-handbook' [Tunstall et al., 2023a] repository to finetune our models. Notably, this repository, when using DPO, reproduces 'Zephyr 7B Gemma' 2 Tunstall and Schmid [2024] , Tunstall et al. [2023b] , which at the time of release, achieved state-of-the-art scores on MT-Bench for 7B models. 'Zephyr 7B Gemma' first takes gemma-7b [Gemma-Team et al., 2024] and finetunes it on the 'deita-10k-v0-sft' dataset [Liu et al., 2023] to produce 'zephyr-7b-gemma-sft' 3 . It is then trained on the pairwise preference dataset of 'Argilla DPO Mix 7K' 4 . When evaluating a new objective function, we replace DPO in this last step with the generated objective function, keeping the same hyperparameters. We show example runs in Figure 12 and provide further experimental details in Appendix B. Once we have a trained LLM for the proposed objective function f i , we evaluate that LLM on the popular multi-turn dialogue evaluation benchmark of MT-Bench [Zheng et al., 2024] . This is a multi-turn open-ended question set, which uses GPT-4 to assess the quality of the trained model's responses, obtaining a high correlation with the popular Chatbot Arena [Zheng et al., 2024] . We provide further evaluation details in Appendix C.After evaluating approximately 100 objective functions, we catalogued the best-performing ones in Table 1 . We tabulate the high-level objective forms here and provide the full objective loss functions and their associated code in Appendix E. Moreover, we also plot the best performing sub-task evaluations in Figure 4 .Table 1 : Discovery Task MT-Bench Evaluation Scores for each discovered objective function f . We provide the baselines first, followed by a dashed line to separate the objective functions that were discovered. We provide details for each discovered objective function in Appendix E. 5 Held-Out EvaluationsWe next validate each of our discovered objective functions (shown in Table 1 ) on held-out tasks. We find that the Performance Adaptive Decay Loss (PADLL) and the Log Ratio Modulated Loss (LRML) consistently perform well. Because of its unconventional properties and performance, we refer to LRML as our discovered preference optimization, or DiscoPOP, algorithm.We consider three different standard [Rafailov et al., 2023] open-ended text generation tasks each designed to evaluate different properties of the fine-tuned LLM policy π θ where each LLM policy is trained with one of our discovered objective functions f on a preference dataset D = {(x i , y i w , y i l )} N i=1 .5.1 Single-turn Dialogue -Alpaca Eval 2.0We evaluate the trained models on Alpaca Eval 2.0, [Li et al., 2023 , Dubois et al., 2023 , 2024] . This is a single-turn dialogue LLM-based automatic evaluation using GPT-4 to assess the win rate of the trained LLM policy's completion compared to the of the underlying SFT base model. Alpaca Eval 2.05 , has been validated against 20K human annotations, and aims to reduce the length bias of Alpaca Eval 1.0; where using length controlled (LC) Alpaca Eval shows a correlation with Chatbot Area of 0.98, making it a popular benchmark with the highest correlation to Chatbot Arena [Dubois et al., 2024] . We also detail task training details in Appendix B.1. We provide the Alpaca Eval 2.0 results in Table 2 . As reference policies, we used GPT-4 for absolute comparison and the SFT-trained model for relative comparison. We observe that the discovered LRML (DiscoPOP), PADLL, and AQFL functions outperform the baselines and other discovered losses on the normal and length-controlled win rates. The differences in scores among these topperforming losses are not significant, except for the LC win rate against the SFT reference model, where DiscoPOP performs best. 
summarization (tl;dr): 
 We train an LLM policy to, given a forum post on Reddit x, generate a summarization y of the main points. We finetune 'zephyr-7b-gemma-sft' using 10% of the Reddit TL;DR summarization preference dataset [Völske et al., 2017] on each of the baseline and discovered objective functions. As a reference model, we again use 'zephyr-7b-gemma-sft'. Further details on the training pipeline are outlined in Appendix B.2. To evaluate the quality of the summaries, we make use of the Alpaca Eval 2.0 library with a custom evaluation dataset existing of 694 test samples from the TL;DR dataset and a custom GPT-4 annotator template as described in Rafailov et al. [2023] . For additional details regarding the summarization evaluation see Appendix C.3.In Table 3 the PADLL loss and DPO loss perform best, with little difference from each other, on the summarization task in three out of four metrics. Additionally, the LRML -DiscoPOP function achieves scores slightly below the top performers, especially in the length-controlled win rates. In contrast to the single-turn dialogue task, the AQFL loss does not achieve high scores in the held-out evaluation. In this task, we train an LLM policy to generate movie review completions y with positive sentiment, where x is a prompt at the start of a movie review from the IMDb dataset [Maas et al., 2011] . We start with a GPT-2 [Radford et al., 2019] model, which had supervised fine-tuning on the IMDb dataset, and we perform preference optimization using the baseline and discovered objective loss functions. Details of the training implementations can be found in Appendix B.3. Inspired by Rafailov et al. [2023] 's experiments, we calculate the model rewards through a pre-trained sentiment classifier, which we use as a proxy for ground truth, as well as the KL-Divergence of the trained model and the reference model. Appendix C.4 provides further details into the evaluation for this task.We provide results of models with converging β values in Figure 5 for LRML compared against DPO and SLiC, displaying the model rewards against the KL-Divergence to the reference model. In Figure 5a , the LRML-trained text generator outperforms the DPO model in terms of rewards and KL-divergence with low β values (0.025, 0.05, 0.1). At higher β values (0.5 and 1.0) both methods show trends of increased KL-Divergence and lower rewards, but generally, LRML maintains a higher reward than DPO. In Figure 5b , we note that LRML slightly outperforms DPO, SLiC, AQFL, and PADLL at β ∈ {0.05, 0.1} in terms of reward. For larger β values (0.5 and 1.0), LRML shows similar trends of increased KL-Divergence and rewards like the other objective functions. A more detailed comparison between the individual discovered losses and the baselines can be found in Appendix Figure 8 . 
analysis of discopop: 
 We list all our discovered objectives in Table 1 , as well as the code and mathematical representations in Appendix E. In this section, we now analyze the Log Ratio Modulated Loss, which we define as the DiscoPOP loss function, as it performs consistently high across the held-out evaluation tasks, and we provide some intuitive understanding of how it outperforms the existing state-of-the-art objectives.The Log Ratio Modulated Loss is a dynamically weighted sum of the logistic loss (as used in DPO) and the exponential loss. The weight of each is determined through a sigmoid calculation of the difference of log-ratios (ρ). Mathematically, the LRML function can be described with a temperature parameter τ = 0.05 as follows:EQUATIONIf the difference of log ratios is zero (ρ = 0), which is at the start of the training when the model policy π θ is equal to the reference policy π ref , then the loss is equally balanced between the logistic and exponential loss. If ρ → ∞, the model policy diverges from the reference policy and chosen outputs are preferred, then the exponential term dominates. This emphasizes larger differences more strongly. On the other hand, if ρ → -∞, the model policy diverges from the reference policy, and rejected outputs are preferred. In this case, the logistic loss can handle moderate differences well.The baseline objective losses and the LRML, the PADLL, and the AQFL functions are displayed in Figure 6 , including their gradients. Surprisingly, the DiscoPOP function has a non-convex segment and negative gradients at the starting point ρ = 0. This is potentially helpful for introducing a curriculum or for stochasticity. Additional results and analysis of the discovered loss function can be found in Appendix D.6. While performing very well on single-turn text generation and text summarization, we observed during the IMDb experiment that LRML struggles to converge when β is too low (β ≤ 0.01) or too high (β ≥ 2.5), likely because β ̸ = 0.05 was never seen or used during the discovery process.In Figure 9 and Figure 10 of the Appendix, we plot the LRML objective function for β ∈ {0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5} against DPO. When β is high, the DiscoPOP objective function takes the form of the DPO log sigmoid loss. During training on β = 0.01, we observed that DiscoPOP gets stuck in generating predominantly negative reviews (resulting in a reward score of ∼ 0.15). We hypothesize that the loss is stuck in the local minima to the left with a negative difference in log ratios. While training with β ∈ {2.5, 5.0}, we observed that the model collapsed after a sharp spike in the loss and subsequently having loss value 0 and NaN outputs. This is potentially due to a large gradient in the non-convex part, which could be amended with gradient clipping. 
related work: 
 Evolution and Search with Large Language Models. LLMs provide a fast and automated way to create multiple candidate solutions for a problem stated in natural language [Song et al., 2024] , which makes them powerful tools for driving population-based search procedures. Various recent works have applied this approach to coding problems [Romera-Paredes et al., 2024] , neural architecture search [Chen et al., 2024a , Holt et al., 2024b] , virtual robotic design settings [Lehman et al., 2023] , reward functions [Ma et al., 2023 , Yu et al., 2023] , and algorithm heuristics [Liu et al., 2024] .Finally, recently LLMs have shown to be capable of acting as recombination operators for black-box optimization with Evolution Strategies [Lange et al., 2024] and for Quality-Diversity approaches [Lim et al., 2024] .Learning. There are many other approaches to automating the discovery of generalizable machine learning algorithms. Some prior works explore the space of ML functions using genetic algorithms and a hand-crafted domain-specific language for reinforcement learning algorithms [Co-Reyes et al., 2021] , curiosity algorithms [Alet et al., 2020] , and optimizers [Chen et al., 2024b] . Other works instead parameterize a transferrable objective function using neural networks and optimize them with evolution strategies or meta-gradients. [Biggio et al., 2021] and hybrid combinations of pre-training transformers, which are further refined with RL and evolution strategies [Holt et al., 2023b] .Preference Optimization Algorithms. While the reduction to supervised learning makes DPO and alternatives easier to use, other approaches have sought to simplify the RL step, including using variants of REINFORCE [Ahmadian et al., 2024 , Gemma-Team et al., 2024] as well as more fine-grained feedback [Wu et al., 2024] through preferences over individual steps in the reasoning process [Uesato et al., 2022 , Lightman et al., 2023] or reward redistribution [Chan et al., 2024] . Others use iterative offline training interleaved with sampling from the policy model and obtaining a preference ranking from themselves [Xu et al., 2023] , another judge LLM [Guo et al., 2024] , or an oracle [Swamy et al., 2024] . 
conclusion: 
 Summary. In this paper, we proposed and used LLM-driven objective discovery to generate novel offline preference optimization algorithms. Specifically, we were able to discover high-performing preference optimization losses that achieve strong performance across held-out evaluation tasks, with the highest performing providing new insights into what an optimal objective may need to possess, such as being a blend of logistic and exponential losses and possibly being non-convex.Limitations & Future work. There are multiple limitations to our current approach. First, we have only scratched the surface of how to generate LLM objective proposals effectively. Initial exploratory experiments using techniques such as temperature sampling or worst-to-best performance sorting in the context did not yield significant improvements. But one could imagine leveraging more information about the training runs and automatically tuning instruction prompt templates. E.g. by providing entire learning curve plots to a Visual Language Model (see Figure 13 ) or by meta-metaoptimizing [Lu et al., 2023] the LLM prompt. Second, the highest-performing loss re-purposed β in the traditional sense, making it affect the functional behavior and the KL penalty of the model with respect to the base model. This motivates future work to study different forms, with perhaps multiple floating point parameters in the form, that each could be tuned separately. Although we provided an initial analysis sweep over this one single parameter and observed some instances of the functional behavior leading to instability of training the model, a further multi-parameter analysis, reformulating the objective, would be beneficial for future work. Finally, our work uses closed-source models (GPT-4) to generate code, which limits reproducibility and is costly to run. Future work could use the produced models themselves to generate code, resulting in code-level self-improvement.Broader Impact and Ethical Considerations. This paper presents an LLM-driven discovery in-context learning pipeline that is used to generate better-performing novel offline preference optimization algorithms. However, misuse of the pipeline as a tool or training an LLM to produce undesirable, unethical, or harmful outputs could be possible by a user. Furthermore, due to the use of LLMs and training of LLMs, the outputs are susceptible to hallucinations, motivating all outputs of the LLMs to always have a content filter applied to the outputs. Finally, this work takes a small step towards code-level self-improvement in language models, which could potentially result in unintended behaviors.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Details are given in Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We reviewed the NeurIPS Code of Ethics and ensured the research conforms to it.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss this in Section 8 under the heading Broader Impact and Ethical Considerations.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risk. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All creators are properly credited. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: A.1 PromptsWe use the following system prompt to generate the model responses:You are a machine learning researcher who is testing out different RLHF loss functions . When you respond , output a JSON where the first key (" thought ") corresponds to your thought process when designing the next function . The second key (" name ") corresponds to the name of your next function . Finally , the last key (" code ") corresponds to the exact python code that you would like to try .Here is an example : { " thought ": " Based on the previous outputs , I should try the direct preference optimization algorithm ." , " name ": " dpo " , " code ": " def sigmoid_loss ( self , policy_chosen_logps : torch . FloatTensor , pol icy_re jecte d_logp s : torch . FloatTensor , re fe ren ce _ch ose n_ log ps : torch . FloatTensor , r ef e r en c e _r e j ec t e d_ l o g ps : torch . FloatTensor , ) -> torch . FloatTensor : pi_logratios = policy_chosen_logps -pol icy_r ejecte d_logp s ref_logratios = re fe ren ce_ ch ose n_ log ps -r ef e r en c e _r e j ec t e d_ l o gp s logits = pi_logratios -ref_logratios losses = -F . logsigmoid ( self . beta * logits ) return losses " } You are deeply familiar with binary classification losses from the literature . Be creative and reference prior literature when possible .You must use the exact function interface used above . Feel free to define extra hyperparameters within your function as constants . Do not make them attributes of self .Note that ' self . beta = 0.05 '.RLHF loss functions train on a dataset of pairs of preferred and rejected completions . ' policy_chosen_logps ' refers to the policy ' s log probabilities of the preferred completion , and ' policy_rejected_logps ' refers to the policy ' s log probabilities of the rejected completion . ' reference_chosen_logps ' and ' reference_rejected_logps ' refer to the same for the reference ( base ) model .The user will then return to you a fitness that corresponds to the performance of the resulting model on a downstream task . Your goal is to maximize performance .We then provide the first user prompt as such:Here are some results we ' ve obtained : Upon testing the generated code, if an error is encountered, we provide the following prompt, where 'error' is the text containing the system error:Code not valid . Error : { error } Please generate the next one .Upon successful completion, we return the following user prompt, where 'val' is the MT-Bench score:Fitness : { val }.Please generate the next one .For each valid generated objective function f i , we use it to train an LLM and then collect a performance evaluation score. Specifically, we follow the same process when training and evaluating all objective functions, starting with a pre-trained supervised fine-tuned (SFT) 7 billion gemma model of 'zephyr-7b-gemma-sft' This is a 7 billion base version gemma [Gemma-Team et al., 2024] model supervised-fine-tuned on the 'deita-10k-v0-sft' dataset [Liu et al., 2023] . Starting with this model, we train it on the pairwise preference dataset of 'Argilla DPO Mix 7K'; which attempts to create a high-quality preference dataset by filtering only highly rated chosen responses from the datasets of a multi-turn dataset, instruction following dataset [Longpre et al., 2023] and a diverse preference dataset that covers truthfulness, honesty and helpfulness [Cui et al., 2023] . For each training run, we trained all the parameters of the starting model, using a fixed β = 0.05. We used the same fixed hyper-parameters for all training runs unless explicitly noted. Specifically, we used a learning rate of 5e-7, bfloat16 floating-point format, two epochs, a batch size per device of two, a gradient accumulation step of 8, a cosine learning rate scheduler, and AdamW optimization algorithm [Loshchilov and Hutter, 2017] . We use the popular TRL transformers library [von Werra et al.] , adapting the offline preference optimization objective function to train all models. The models were trained on 8 Nvidia A100 GPUs. An individual training run takes approximately 30 minutes. We provide training and evaluation statistics for discovered objective functions in Figure 7 . We also provide the equivalent code implementation at https://github.com/vanderschaarlab/DiscoPOP.To determine if the discovered objective functions generalize well also to other tasks, we use them to preference optimize an LLM for text summarization. Specifically, we start again with a pre-trained supervised fine-tuned (SFT) 7 billion gemma model of 'zephyr-7b-gemma-sft', and we optimized it with the objective function f i on a subsample of the Reddit TL;DR summarization preference dataset [ Völske et al., 2017] 6 . More precisely we use the first 10% of the dataset for preference optimization, which amounts to around 8'000 training samples. During training the hyperparameters are kept the same as in the single-turn dialogue task, explained in subsection B.1, except that LLMs were trained 4 Nvidia A100 GPUS using a gradient accumulation step of 16. An individual training run takes approximately 1.5 hours.Another popular generalization task for preference optimization [Rafailov et al., 2023] is to fine-tune a small LLM to generate positive text for movie review, based on the IMDb sentiment dataset [Maas et al., 2011] 7 following the the work of Zhou et al. [2024] . As a starting model, we use a GPT2 model [Radford et al., 2019] , that was supervised fine-tuned on the IMDb dataset8 . Subsequently, we apply the baseline and discovered objective function f i for preference optimization. The goal of the LLM is to give a short prompt of 2-8 tokens, which indicate the start of a movie review, to generate a positive review. As we are interested in the effect of β on the rewards and KL-Divergence, we train the objective functions over a sweep of β ∈ {0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5}. Every LLM is trained for three epochs, using the AdamW optimizer, with an initial learning rate of 5.0e-5, a warm-up scheduler of 0.1, and a cosine learning rate scheduler. The models are trained on 4 Nvidia A100 GPUs, using a gradient accumulation step of 8, and a batch size per device of 2. The training takes around 30 minutes.To assess the fitness of the discovered preference optimization loss function during the discovery phase, we evaluate the trained LLMs on the MT-Bench [Zheng et al., 2024] benchmark. The evaluation benchmark consists of 80 high-quality multi-turn questions from various disciplines. The goal is to assess LLM's ability to follow instructions and keep the flow of a conversation. A larger LLM, in our case GPT-4, is then used as a judge to score the quality of the answers with a number from 0 (lowest) to 10 (highest). Scores are given based on the quality of the LLM's first-turn answer (single-turn) and first and second answers (multi-turn). Finally, the MT-Bench score is the average of single-turn and multi-turn scores. For answer generation and evaluation, we used the FastChat library9 and its standard sampling and temperature parameters, provided by Zheng et al. [2024] .Alpaca Eval 2.0 [Li et al., 2023 , Dubois et al., 2023 , 2024] is also a popular benchmark for evaluating LLMs. This is a single-turn dialogue LLM-based automatic evaluation using a stronger LLM, here GPT-4 Turbo, to assess the win rate of the trained LLM policy's completion compared to either GPT-4 or the of the underlying SFT base model. Specifically, Alpaca Eval 2.0 has been validated against 20K human annotations and aims to reduce the length bias of Alpaca Eval, where using length-controlled (LC) Alpaca Eval shows a correlation with Chatbot Arena of 0.98, making it a popular benchmark with the highest correlation to Chatbot Arena [Dubois et al., 2024] . The Alpaca evaluation dataset consists of 841 high-quality instructions from different data sets. The library10 provided by Dubois et al. [2024] calculates the win-rate (percentage were the trained policy is prefered over the reference policy, first introduced in Alpaca Eval 1.0), and a length-controlled win-rate, where a linear model is fitted to de-bias for length of the prompt and instruction difficulty. We used a temperature of 0.7, sampling, and a maximum number of new tokens of 1024 to generate the answers. Furthermore, the library provides the standard error of the mean, which indicates the confidence of the win-rate and LC win-rate.To evaluate how well the discovered objective functions generalize to the summarization task, we use the Alpaca Eval 2.0 library, similar to subsection C.2. Instead of using the Alpaca evaluation dataset, we create a custom dataset of 694 samples from the IMDb preference test dataset. Additionally, we change the prompt of the annotator LLM, to fit the "Summarization GPT-4 win rate prompt (C)" as described in Rafailov et al. [2023] . The (LC) win-rate is calculated against either the existing human-chosen test sample or against the summary generated by the SFT reference model. For a summary generation, we apply a temperature parameter of 0.7, sampling, and a maximum of 256 new tokens. Moreover, we stop the summarization after the "\n" token to avoid nonsensical generations. Furthermore, as we cannot calculate an instruction difficulty for the length-controlled win-rate, we omit this term from the linear model (This has only a small impact on the metric). In addition to the win-rates we also provide the standard error as a measure of confidence.For the positive text generation, we do not require an LLM judge compared to MT-Bench, Alpaca Eval 2.0, and TL;DR evaluation, as we take a pre-trained sentiment classifier11 as ground truth reward scorer. The LLMs apply sampling and a maximum of 60 new tokens for the positive text generation. The rewards and KL-divergence are averaged over 10 different generations from the trained LLMs. Here we also explore using only the top-K performing objectives unsorted and sorted by their performance. Right. We also investigate whether using a "thought" as part of the context and whether to include non-valid code and error messages improves performance. The discovery process for CIFAR-10 objectives (5 epochs) is robust to these settings. The results are averaged across 3 independent runs.Figure 12 : LLM-driven discovery for CIFAR-10 loss functions with (left) and without (right) providing fitness feedback. Note that without feedback, it performs worse but also is unable to refine its ideas, resulting in fewer improvements across generations. We performed further analysis to improve our understanding of the discovered loss function. We hypothesize that the local optimum of the DiscoPOP loss could catch noisy or incorrect data points. By inspecting the DiscoPOP loss values and log ratios of the training preference pairs in the IMDb dataset, we see that 1.35% of training points fall there (see Figure 14 ). Although we use the binary preference labels from the IMDb dataset12 for training, the dataset also includes a positivity reward score for each completion, calculated by a separate reward model. When we analyze the data points between the local optima, we find that the positive and negative completions are significantly closer in absolute reward difference than those outside the local optimum (See Table 4 ). This implies that the preference labels on those points are likely more challenging to distinguish and help empirically validate our hypothesis. Table 4 : The IMDb positive review preference dataset also provides "golden reward" scores for the chosen responses r w and rejected responses r l , calculated with a strong sentiment classifier (log p(pos)log p(neg)). We analysed the absolute difference in reward scores between the chosen and rejected responses |r wr l | across the different domains of the DiscoPOP function and report the statistics here. Training samples within the optima have a significantly lower mean absolute difference in rewards compared to the other regions in the DiscoPOP loss function. The samples stuck within the optima are the training triplets where the chosen and rejected responses are "closest" to each other and likely the most "noisy" labels.To describe the discovered losses mathematically, we define three existing preference optimization losses here:f dpo (βρ) = -log(σ(βρ)) = -log( 1 1 + exp(-βρ) ) = log(1 + exp(-βρ)) (6) f slic (βρ) = ReLU(1 -βρ) (7) f exp (βρ) = exp(-βρ) (8)Moreover, we display the code of the discovered losses as it is output by the LLM. In addition, we provide a mathematical representation of each, which we have adapted to be consistent with β being the KL-Divergence regularization parameter. This is because the generated code for LRML, DBAQL, AQL, AQFL, and PFL did not uphold the β ought to be multiplied with the difference of log-ratios before any further calculations. If this was not upheld, it could lead to the loss function changing shapes based on the KL-regularization term, and therefore, models could not converge or potentially collapse. In future work, we should constrain the exploring LLM to uphold the β multiplication with the input before any other calculations are done with the difference of log-ratios ρ. As the meta-exploration was done with a set β = 0.05, and we wish to keep consistent with this scale of regularization, we have adapted the losses by dividing ρ values used in intermediate calculations with a scalar τ = 0.05.In the IMDb experiment in Section 5, we have thus used the corrected version of codes for the discovered losses based on the provided mathematical representation, as we were most interested in the effect of the KL-divergence compared to the model rewards. EQUATIONf padll (βρ) = δ adpt • f dpo (βρ) (15) = δ base • 1 -1 [ρ<0] • τ • f dpo (βρ) (16) = δ base • 1 -1 [ρ<0] • τ • log 1 + exp(-βρ) (17) = 0.9 • 1 -1 [ρ<0] • 0.5 • log 1 + exp(-βρ) (18)This loss can also be rewritten as: Interestingly, the PFL generated function code did not include any β values in the loss function. We have added it to the corrected code for the IMDb experiment, as well as to the mathematical expression below. We account for the consistency of the logit values by dividing with τ = 0.05, which is the same value as β in the discovery process.f padll (β, ρ) = δ pos • f dpo (βρ), if ρ ≥ 0 δ neg • f dpo (βρ), if ρ < 0 ,EQUATIONEQUATIONEQUATIONF Extended Related WorkIn the following, we provide an extended related work of the related work included in the main paper.Evolution and Search with Large Language Models. LLMs provide a fast and automated way to create multiple candidate solutions for a problem stated in natural language [Song et al., 2024] , which makes them powerful tools for driving population-based search procedures. Various recent works have applied this approach to coding problems [Romera-Paredes et al., 2024] , neural architecture search [Chen et al., 2024a , Holt et al., 2024b,d] , virtual robotic design settings [Lehman et al., 2023] , reward functions [Ma et al., 2023 , Yu et al., 2023] , and algorithm heuristics [Liu et al., 2024] . Finally, recently LLMs have shown to be capable of acting as recombination operators for black-box optimization with Evolution Strategies [Lange et al., 2024] and for Quality-Diversity approaches [Lim et al., 2024] . Additionally, prior work has shown LLM multi-agent frameworks' ability to excel at large code-generation tasks [Holt et al., 2024c] , which we could envision enabling scaling up searching for larger candidate solutions for more complicated objectives or algorithms than the existing approaches that fit within the LLMs output context window.Learning. There are many other approaches to automating the discovery of generalizable machine learning algorithms. Some prior works explore the space of ML functions using genetic algorithms and a hand-crafted domain-specific language for reinforcement learning algorithms [Co-Reyes et al., 2021] , curiosity algorithms [Alet et al., 2020] , and optimizers [Chen et al., 2024b] . Other works instead parameterize a transferrable objective function using neural networks and optimize them with evolution strategies or meta-gradients. et al., 2020] , gradient descent [Kacprzyk et al., 2024] , RL with evolution strategies [Mundhenk et al., 2021] , pre-training transformers [Biggio et al., 2021] and hybrid combinations of pre-training transformers, which are further refined with RL and evolution strategies [Holt et al., 2023b] . Furthermore, more general-purpose ML approaches exist to perform sequential modeling of processes [Holt et al., 2022] , which can be readily used in model-based RL approaches [Holt et al., 2023a [Holt et al., , 2024a]] .Preference Optimization Algorithms. While the reduction to supervised learning makes DPO and alternatives easier to use, other approaches have sought to simplify the RL step, including using variants of REINFORCE [Ahmadian et al., 2024 , Gemma-Team et al., 2024] as well as more fine-grained feedback [Wu et al., 2024] through preferences over individual steps in the reasoning process [Uesato et al., 2022 , Lightman et al., 2023] or reward redistribution [Chan et al., 2024] . Others use iterative offline training interleaved with sampling from the policy model and obtaining a preference ranking from themselves [Xu et al., 2023] , another judge LLM [Guo et al., 2024] , or an oracle [Swamy et al., 2024] .We provide a full run below, formatted for readability.Since the logistic log loss currently has the highest fitness and is a direct optimization of the log likelihood ratio , let ' s try a variation that includes a margin in the log likelihood ratios , inspired by the concept of a margin in SVM hinge loss . Introducing a margin could create a buffer that leads to more robust learning , as seen in SVMs . name l o g istic_margin_loss code def logistic_margin_loss ( self , policy_chosen_logps : torch . FloatTensor , pol icy_re jecte d_logp s : torch . FloatTensor , re fe ren ce _ch ose n_ log ps : torch . FloatTensor , r ef e r en c e _r e j ec t e d_ l o g ps : torch . FloatTensor , ) -> torch . FloatTensor : margin = 0.1 pi_logratios = policy_chosen_logps -pol icy_r ejecte d_logp s ref_logratios = re fe ren ce_ ch ose n_ log ps -r ef e r en c e _r e j ec t e d_ l o gp s logits_with_margin = pi_logratios -ref_logratios -margin losses = -F . logsigmoid ( self . beta * logits_with_margin ) return losses ========== Fitness : 7.709375.Please generate the next one .The logistic margin loss outperformed the hinge and ipo losses but did not surpass the original logistic log loss . Perhaps introducing an adaptive margin based on the confidence in the predictions could yield improvements . The margin could be set proportional to the absolute difference between the policy ' s and the reference ' s log probabilities . This could potentially penalize incorrect confident predictions more heavily . name a d a p t i v e _ m a r g i n _ l o g i s t i c _ l o s s code def a d a p t i v e _ m a r g i n _ l o g i s t i c _ l o s s ( self , policy_chosen_logps : torch . FloatTensor , pol icy_re jecte d_logp s : torch . FloatTensor , re fe ren ce _ch ose n_ log ps : torch . FloatTensor , r ef e r en c e _r e j ec t e d_ l o g ps : torch . FloatTensor , ) -> torch . FloatTensor : confidence_margin = torch . abs ( policy_chosen_logpspol icy_re jected _logp s ) -torch . abs ( r efe re nce _c hos en_ lo gpsr ef e r e nc e _ re j e ct e d _l o g ps ) pi_logratios = policy_chosen_logps -pol icy_r ejecte d_logp s ref_logratios = re fe ren ce_ ch ose n_ log ps -r ef e r en c e _r e j ec t e d_ l o gp s logits_with_margin = pi_logratios -ref_logratios -confidence_margin losses = -F . logsigmoid ( self . beta * logits_with_margin ) return losses ========== Fitness : 7.51875. Please generate the next one . ========== thought The adaptive margin did not yield an improvement . It might have introduced too much variability or penalization . Let ' s go back to a fixed structure but combine elements from both logistic loss and hinge loss , with the idea of encouraging a larger margin when the decision is correct , while still having a soft , probabilistic interpretation . This combined loss could retain the benefits of both worlds . The combined logistic and hinge loss slightly outperformed the baseline logistic log loss but still did not achieve a significant improvement . Considering the nature of RLHF , where we prefer the model to be correct but not overly confident , we can explore a scaled version of the combined loss where the logistic part is emphasized more heavily , as it intrinsically accounts for confidence due to its probabilistic nature . The scaled combined loss has shown an improvement , which indicates that balancing the contribution between the logistic and hinge components is beneficial . To further expand on this idea , we could try a form of annealing where early in training we use more of the logistic component , encouraging better probability estimation , and later on a switch gradually to the hinge component , emphasizing decisions with a margin . We ' ll simulate this by using a weighting factor that changes linearly through the logits , giving lower logits ( earlier , less confident predictions ) a higher weight for the logistic loss . The annealed combined loss did not yield the expected improvement and performed worse than the scaled version . It ' s possible the transition from logistic to hinge was too aggressive . For the next iteration , let ' s return to a simpler concept and try a variation of the logistic loss with a squared term , similar to squared hinge loss , which might penalize incorrect predictions more smoothly than the linear term used in the standard logistic loss . name s q u a red_l ogisti c_los s code def sq uared_ logis tic_lo ss ( self , policy_chosen_logps : torch . FloatTensor , pol icy_re jecte d_logp s : torch . FloatTensor , re fe ren ce _ch ose n_ log ps : torch . FloatTensor , r ef e r en c e _r e j ec t e d_ l o g ps : torch . FloatTensor , ) -> torch . FloatTensor : pi_logratios = policy_chosen_logps -pol icy_r ejecte d_logp s ref_logratios = re fe ren ce_ ch ose n_ log ps -r ef e r en c e _r e j ec t e d_ l o gp s logits = pi_logratios -ref_logratios logistic_losses = -F . The temperature scaling seemed to have a positive impact , possibly due to better -calibrated probability estimates . To further explore this direction , we can try using label smoothing , which encourages the model not to be too confident about its predictions by preventing it from assigning full probability to a single class . Please generate the next one . ========== thought Since the label smoothing implementation had an issue with the loss shape being incorrect , it seems the loss reduction was erroneously computed over all inputs instead of keeping the per -input format . Let ' s adjust the implementation to make sure the loss retains the correct shape , by performing the mean operation separately for the positive and negative parts and then combining them , weighted appropriately . name The hybrid approach with the performance margin decay did not achieve the desired effectiveness . Introducing a margin may have been too punitive on correct predictions that are close to the boundary . To maintain the balance , let ' s explore the idea of using a tripletstyle loss , which is popular in embedding learning . By treating the chosen policy logs as the anchor and the reference chosen and rejected logs as positive and negative examples , respectively , we can encourage the chosen policy decisions to be closer to the reference chosen decisions and farther away from the reference rejected decisions . The revised triplet -style logistic loss still encountered an issue , likely due to improper application of the operations and reduction at the end , which should be on a per -example basis . Triplet loss typically works with distances embedded in a space , and here we are dealing with log probabilities , so the direct application may not be correct . For the next attempt , let ' s simplify : we ' ll reinforce the relationship between the chosen and rejected log probabilities directly by using their difference , promoting a wider margin while still within the logistic loss framework . The approach of applying a static margin did not yield the expected improvements . It ' s worth considering a different perspective ; instead of focusing on introducing margins or applying decays , let ' s attempt to directly optimize the policy ' s certainty . Intuitively , the policy should be more confident when choosing actions similar to the reference and less confident otherwise . A potential approach would be to scale the logistic loss by the absolute difference between the policy ' s choice and the reference choice , thus directly tying the loss to the policy ' s certainty in its decision . name c e r t a i n t y _ s c a l e d _ l o g i s t i c _ l o s s code def c e r t a i n t y _ s c a l e d _ l o g i s t i c _ l o s s ( self , policy_chosen_logps : torch . FloatTensor , pol icy_re jecte d_logp s : torch . FloatTensor , re fe ren ce _ch ose n_ log ps : torch . FloatTensor , r ef e r en c e _r e j ec t e d_ l o g ps : torch . FloatTensor , ) -> torch . • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.https://github.com/tatsu-lab/alpaca_evalhttps://huggingface.co/datasets/CarperAI/openai_summarize_comparisonshttps://huggingface.co/datasets/ZHZisZZ/imdb_preferencehttps://huggingface.co/lvwerra/gpt2-imdbhttps://github.com/lm-sys/FastChathttps://github.com/tatsu-lab/alpaca_evalhttps://huggingface.co/siebert/sentiment-roberta-large-englishhttps://huggingface.co/datasets/ZHZisZZ/imdb_preference 


Decision: The paper proposes a novel method that uses LLM to discover preference optimization algorithms. This falls into the metalearning category. The reviewers found the performance to be effective and the experiments to be comprehensive. During the rebuttal, the authors successfully addressed the reviewers' questions and concerns such as hyperparameters and the comprehensiveness of the experiments. I think it would be nice to provide the readers a more comprehensive related work section for the metalearning papers that were published before 2020.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel approach to improving LLMs by using an automated system to discover new optimization algorithms. Traditionally, enhancing LLMs has relied heavily on hand-designed loss functions, but this research employs an LLM to iteratively generate and refine these functions itself. This paper introduced DiscoPOP, which blends logistic and exponential losses to outshine existing methods. This algorithm was evaluated across a range of tasks, including multiturn dialogue, sentiment generation, and summarization, where it consistently delivered good results.
strength_weakness: strength_weakness: Although this paper mianly focuses on preference optimization task, the dicovery method seems to be easily to adapt to other domains/tasks. A general concern with the paper is its reliance on a complex method that involves detailed tuning of hyperparameters. This complexity could limit the method's broader applicability unless the process can be generalized or adapted effectively to other contexts. However, the paper does not thoroughly address how to establish and optimize the discovery pipeline across different scenarios, which could hinder its practical utility. More guidance on adapting the methodology to a variety of use cases would significantly enhance its value and impact.
questions: questions: 1. How well does the proposed LLM-driven discovery method generalize to other domains/tasks beyond preference optimization? Are there specific modifications needed to adapt this method to other fields?
2. The authors mention that DiscoPOP blends logistic and exponential losses. Could you elaborate on the theoretical justification for this choice? How does this combination affect the convergence properties of the algorithm?
3. Considering the non-convex nature of DiscoPOP, what strategies do you suggest to avoid local minima during optimization?
limitations: limitations: The authors discussed the limitations of DiscoPOP.
paper_summary: paper_summary: The paper proposes an algorithm to discover preference objective functions using LLM for LLM preference optimization. Authors conduct experiments with the discovered objectives on multiple datasets and demonstrate that the discovered objective functions generally perform better than baselines. The authors also show interesting insight into the discovered objective function, LRML.
strength_weakness: strength_weakness: 1. The objective function is discovered with a different placement of $\beta$ than the objective function used for evaluation, which causes a misalignment of the discovery and the evaluation process. It would be nice to keep them aligned.
2. The LLM in the discovery process is a clever generator for searching candidates. It is unclear if the LLM's capability used in the discovery process can change the duration for finding a good objective function or if it will cause the search process to fail.
3. Similar to the point above, it would be nice to test some non-LLM-based generators of the objective function and perform some traditional searching algorithms such as evolutionary search to demonstrate the effectiveness of the LLM.
questions: questions: See weakness above.
limitations: limitations: See weakness above.
paper_summary: paper_summary: The paper proposes DiscoPOP, an algorithm for discovering preference optimization loss functions using Large Language Models (LLMs). The authors propose an LLM-driven objective discovery process by iterative prompting LLMs by previously evaluated performance metrics. Experiments on various benchmarks demonstrate its effectiveness.
strength_weakness: strength_weakness: I think the updating process might be sensitive to the prompts used for proposing new loss functions.
questions: questions: Refer to the weakness
limitations: limitations: Refer to the weakness
paper_summary: paper_summary: This paper introduces a method of searching for offline RL objectives by using LLMs to generate and refine objectives. They demonstrate that several objectives discovered using this method are able to achieve higher evaluation scores than existing objectives (e.g. DPO) on a variety of benchmark tests. They provide a high level analysis of one discovered objective, hypothesizing that it may possess desirable properties that lead to these performance boosts.
strength_weakness: strength_weakness: The main weakness of this paper in my opinion is that it isn't clear what results the paper is trying to present. While this is partially an issue of organization, it is also an issue with the amount of evidence given to the claims in the paper. If these was made clearer, I would be inclined to raise my score. I detail below:

1. The paper starts by introducing a new method for discovering new optimization objectives using LLMs, then transitions quite abruptly to describing the properties of DiscoPop. While both could be valuable contributions, I don't feel that either is given quite the attention required. It would help to clarify what the main focus of the paper is. If DiscoPOP is the focus, then claims about its properties should be better studied. If the discovery method is the focus, more emphasis should be placed on the method's performance and ability to produce novel and useful objectives.

2. The majority of insights about DiscoPOP are presented as hypotheses or intuitive insights (e.g. the non-convex portion, and how the model behaves at $\pm\inf$. This is not enough to support the claim made in the conclusion that they provide insights into what properties an optimal objective function should possess. While it \emph{may} be enough to say the provide intuition into why DiscoPOP is successful, further experimentation would be necessary to make optimality claims like this.

3. Some portions of the discovery method are vague as well. Though the authors do not observe the models simply regurgitating old objectives and cite a pattern of objective refinement for supervised learning, this is of course not guaranteed, and not well studied. Further experiments or details on the how often they observe refinement vs regurgitation would help to support the utility of the method.
questions: questions: 1. How often are unit tests for the generated code failed?
limitations: limitations: The limitations described cover most points, including the difficulties and uncertainties associated with using LLMs to improve upon objectives in a rigorous fashion.

Comments: comment: Hi,
I've read the rebuttal and I will keep my score as positive. Thanks.
Thanks! Let us know if you have any other questions or concerns.
Thanks for the clarification. I don't have further questions. I will rise the score to 7.
We didn't try a large number of prompts. We recommend just describing the setting in text (eg what the inputs and outputs of the function are) and adding a few examples of the code it should generate (along with their associated performance).
Thanks for the reply. As you said, "*the only change needed is to adapt the text in the prompt*", I'm curious how to find the best prompt for a specific task. Are there any rules to follow?
Apologies for my late reply and thank you for your thorough response and clarifications. After going over your main response and supplemental PDF, I'm going to raise my score. The additional experiments on DiscoPOP properties and the experiment regarding refinement have satisfied my biggest concerns. The CIFAR-10 experiment is a good demonstrative example, though it doesn't prove that this will always be the case for the method, I believe it is a sufficiently convincing example to improve the point you were making in that section.
We would like to once again thank the reviewer for their time and feedback. We've incorporated it into our manuscript and we believe it has strengthened our paper.

We hope our rebuttal, which includes substantial additional results and analysis, has addressed the reviewer's concerns.

Seeing as the discussion period is coming to a close, could the reviewer please let us know if they have any further questions or concerns about our submission?