Discovering Preference Optimization Algorithms with and for Large Language Models
Authors: Chris Lu
Samuel Holt
Claudio Fanconi
Alex James Chan
Jakob Nicolaus Foerster
Mihaela van der Schaar
Robert Tjarko Lange
Venue: NeurIPS 2024 poster (2024)
Keywords: Preference optimization
RLHF
Large Language Models
Primary area: natural_language_processing
TL;DR: We use LLM's to generate novel RLHF objectives, some of which achieve strong results.

Abstract: Offline preference optimization is a key method for enhancing and controlling the quality of Large Language Model (LLM) outputs.
Typically, preference optimization is approached as an offline supervised learning task using manually crafted convex loss functions. While these methods are based on theoretical insights, they are inherently constrained by human creativity, so the large search space of possible loss functions remains under-explored. We address this by performing LLM-driven *objective discovery* to automatically discover new state-of-the-art preference optimization algorithms without (expert) human intervention.  Specifically, we iteratively prompt an LLM to propose and implement new preference optimization loss functions based on previously evaluated performance metrics. This process leads to the discovery of previously unknown and performant preference optimization algorithms. The best performing of these we call *Discovered Preference Optimization* (DiscoPOP), a novel algorithm that adaptively blends logistic and exponential losses. Experiments demonstrate the state-of-the-art performance of DiscoPOP and its successful transfer to held-out tasks.

Discovering Preference Optimization Algorithms with and for Large Language Models
Chris Lu" Sakana Al and FLAIR @brisl@ssakana.at
Samuel Holt" University of Cambridge sih31 @cam.ac.uk
Claudio Fanconi'
University of Cambridge
.k
Alex J. Chan
Jakob Foerster! FLAIR, University of Oxford
Mihaela van der Schaar University of Cambridge mv472 @ cam.ac.uk
University of Cambridge aje340@cam cam.ac.uk
aakob..oorsterrccngooo www..n............mmmm
Robert Tjarko Lange'
Sakana Al robert @ sakana.ai
Abstract
Offline preference optimization is a key method for enhancing and controlling the quality of Large Language Model (LLM) outputs. Typically, preference optimiza- tion is approached as an offline supervised learning task using manually crafted convex loss functions. While these methods are based theoretical insights, they are inherently constrained by human creativity, so the large search space possible loss functions remains under-explored. We address this by performing LLM-driven objective discovery to automatically discover new state-of-the-as pref- erence optimization algorithms without (expert) human intervention. Specifically, we iteratively prompt an LLM to propose and implement new preference opti- mization loss functions based on previously evaluated performance metrics. This process leads to the discovery of previously unknown and performant preference optimization algorithms. The best performing of these we call Discovered Prefer- ence Optimization (DiscoPOP)! novel algorithm that adaptively blends logistic and exponential losses. Experiments demonstrate the mate-of-the-ar performance of DiscoPOP and its successful transfer to held-out tasks.
Introduction
Training Large Language Models (LLMs) usually involves starting with model pre-trained on large text corpora and then fine-tuning it to match human preferences. Pre-trained, and even instruction fine-tuned I.LMs, generate harmful, dangerous, and unethical completions [Carlini et al., 2021, Gehman et al., 2020]. To mitigate this and align an LLM with human values, we use human preference alignment through preference-ranked completion data. This approach has become an industry standard, popularized by reinforcement learning with human feedback (RLHF) [Christiano al., 2017, RLHF], and more recently, by offline preference optimization algorithms like direct preference optimization [Rafailov et al., 2023, DPO] and sequence likelihood calibration [Zhao
'Equal Contribution.
Work partially done Spotify.
$ EEuual Advising.
Code: https///gethbb..cm//ucrri2222/2/
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

LLM-Driven Discovery
Discovered Policy Optimization (DiscoPOP)
eXtr CINCATI COURIFTSON. by
Extend/Updan Context & Fitness
ea%e the last corminse aguu
IPM DeppPc
25.10
AGF:
CORTANN "oder:
Foundation Model Proposal
PM04
legiss ToMtTNEEe www.epllept, Sarpen hur")
040
34
1000
seas 10W PLUINTTR PEPOARSS SETKCTUUY
10 94
Inner Loop Optimization Run
6:40
66:7
Rossine
DICDACT VALLICII BAGLET
65
Left. Conceptual illustration of LLM-driven discovery of objective functions. We prompt an LLM output new code-level implementations of offline preference optimization losses (p.))) [f (Bp) as a function of the policy (TT)) and reference model's (Tea)) likelihoods of the chosen (Yu) and rejected (g)) completions. Afterwards, we run an inner loop training procedure and evaluate the resulting model on MTTBennch The corresponding performance is fed back to the language model, and we query it for the next candidate. Right. Performance of discovered objective functions Alpaca Eval.
Figure 1:
et al., 2023, SLiCJ, which cast the problem as a supervised learning objective. Many algorithms have been proposed in the literature for offline preference optimization, and it remains open question which one performs best across tasks. While a strictly dominant algorithm may not exist, some algorithms likely exhibit generally improved performance. To date, all existing state-of-the- art preference optimization algorithms [Rafailov et al., 2023, Azar et al., 2023, Zhao et al., 2023] have been developed by human experts. Despite their advancements, these solutions are inherently constrained by human limitations, including creativity, ingenuity, and expert knowledge.
In this work, aim to address these limitations by performing LLM-driven discovery automati- cally generate new of-te-ofthhe-- art preference optimization algorithms without continual expert human intervention in the development process. While previous works [Ma et al., 2023, Yu et al., 2023] have used L.LMs to design environment-specific RI. reward functions, we discover general-purpose objective functions which can be used across various preference optimization tasks. More specifically, we iteratively prompt an LLM to propose new preference optimization loss functions and evaluate them, with the previously proposed loss functions and their task performance metric (in our case, MT-Bench scores [Zheng et al., 2024]) as in-context examples. After performing this automatic discovery process, we catalogue figh-performing loss functions and introduce a particularly strong one we call Discovered Preference Optimization (DiscoPOP). a new algorithm. To ensure robustness beyond MT-Bench, we validate DiscoPOP using AlapacaEval 2.0 [Dubois et al., 2024], showing an improvement in win rates against GPT-4 from DPO (11.23% + 13.21%). Additionally, in separate, held-out, tasks such as summarization and controlled generation, models trained with the DiscoPOP loss outperform or perform competitively with existing preference optimization algorithms.
Contributions: ! We propose I.L.-driven objective discovery pipeline discover novel offline preference optimization algorithms (Section 3). ② We discover multiple high-performing preference optimization losses. One such loss, which we call Discovered Preference Optimization (DiscoPOP), achieves strong performance across multiple held-out evaluation tasks of multi-turn dialogue (AlpacaEval 2.0), controlled sentiment generation (IMDb) and summarization (TL;DR) tasks. 3 We provide an initial analysis of DiscoPOP, which is weighted sum of logistic and exponential losses, and discover surprising features. For example, DiscoPOP is non-convex.
2 Background
Preference Optimization. Consider a pre-trained language model policy To and a dataset D ((x', y'u, " consisting of prompts æ and preference-ranked completions yhe and yı- In this dataset, human rater prefers you over yt, denoted as yıe > yt- The task is to align To with the human values implicit in these preferences. Canonically, this has been achieved through reinforcement learning from human feedback [Christiano et al., 2017, RLHF], approach that proceeds in two

---

phases: First, reward modelling stage that learns a parameterized reward model rs. By assuming a Bradley-Terry model [Bradley and Terry, 1952] of preferences, the probability of the data can be expressed as P(v > M1) expro / (exp simply optimized over & through the maximum likelihood principle. The second stage of policy optimization employs reinforcement learning algorithm to train the language model against the learned reward. Usually, a KL penalty is introduced between the model and the pre-RL reference policy Tref [Jaques et al., 2019, Stiennon et al., 2020] to prevent over-optimization and straying too far from the original policy, resulting in the final objective:
+ ex (11, æ)). and subsequently
max W
By (ro(y,)) xr)
3 ( Feee
reward maximization
regularization
Despite success in frontier models [Anthropic, 2023, Gemini-Team, 2023]. deep RL has many implementations [Engstrom et al., 2019] and training challenges [Sutton, 1984, Razin et al., 2023] that hinder its adoption. To simplify the whole process, direct preference optimization [Rafailov et al., 2023, DPO] aims to forego both the reward modelling and online RL procedure. Rewriting (I) with decomposition of the KL term into:
max BUNTTA W7
roly. 3 Blog Tre (xla)) + 3)(T))
reward
Tooy regularization
policy entropy
expresses the problem as entropy-regularised RL bandit task [Ziebart et al., 2008), for which known analytical solution exists: x (v(((x) = ((x)-1mm (ylxx) ( (3- (y, x)). . By rearranging
the reward, we can express the task as a binary classification problem based on the reward difference:
5
[log
To [o x)) Teer (a))
To (3/1 (z)
min LE
Bor 20 MD
B
log
Trer (yn (x)
r (2 TT (Bn..)))
In DPO, the function = log is derived as the negative log of the sigmoid function given the BT model assumptions. However, Tang et al. [2024] highlighted that more generally we can obtain a recipe for offline preference optimization algorithms by letting : RR- + R be any scalar loss function. For example, setting fzz (x- 1)² the squared loss function [Rosasco al., 2004] yields IPO [Azar et al., 2023), while employing the max-margin inspired hinge loss [Boser et al., 1992, Cortes and Vapnik, 1995] (=) = m0 x) produces SLiC [Zhao al., 2023].
Here, we define the log ratio difference as = log me (twl log mol la)
Far
Meta-Optimization for Algorithm Discovery. The goal of meta-optimization (optimizing the optimization process) is to uncover novel learning algorithms using a data-driven process. Suppose that an algorithm uses an objective function f to train a model for K iterations, where 7 denotes a set of meta- meta-maraaaannn parameters. Meta-optimization searches for an objective that maximizes the expected (elli/((((tta(((((e(((e((((((((( Unlike previous methods that rely |traii(f") [f)] where 77 is a downstream performance metric. predefined parameterization of (e.g.. neural network [Hospedales et al., 2021] or domain-specific language [Alet et al., 2020]), we leverage LLMs to directly propose code-level objective functions in Python. This approach eliminates the need for a carefully designed search space and utilizes the extensive knowledge embedded in the LLM for flexible selection and mutation.
downstream performance max, (*])
3 LLM-Driven Objective Discovery
Choosing an appropriate objective function is crucial for instilling capabilities into networks. Here, we detail our discovery process facilitated by LLM code-level objective function proposals:
Initial Context Construction. In the initial system prompt, we 'burn-in' the LLM using several established objective functions given in code and their corresponding performance. Furthermore, provide problem details and an example of the output response format as a JSON dictionary.
LLM Querying, Parsing & Output Validation. We query the LLM, parse the response JSON, and run a set of unit tests (e.g. for valid output shapes) before starting training run. If the parsing or unit tests fail, we resample new solution after providing the error message as feedback to the LLM.
3

---

Performance Evaluation. The proposed objective function is then evaluated based on its ability to optimize a model for predefined downstream validation task. We refer to the resulting performance metric as 1)-
Iterative Refinement. By using the performance provided feedback, the LLM iteratively refines its proposals. In each iteration, the model synthesizes a new candidate loss function, exploring both variations of previously successful formulas and entirely new formulations that might improve upon the existing benchmarks. This iterative process is repeated for a specified number of generations or until convergence when set of optimal loss functions is observed.
We summarise this general objective discovery process in Figure and is shown in Algorithm 1.
Algorithm LLM-Driven Objective Discovery
1: Initialize LLM with established loss functions and their performance in context.
2: repeat for each generation
3: LLM proposes new candidate objective function fi
4: Run unit tests to check the validity of the candidate and resample if needed.
5: Evaluate the objective function using the performance metric 77
6: Update the LLM context with the performance data
7: LLM refines generation strategy based on the feedback
8: until convergence criteria are met or maximum generations are reached
Small case study: Discovering supervised classification loss functions. Consider the case of supervised classification on the CIFAR-IO dataset as a simple starting example. We train a simple ResNet-18 for 5 epochs using the objectives proposed by GPT-4 [OpenAI, 2023]. After each training run we provide the LLM with the corresponding validation accuracy and query it for the next PyTorch-based [Paszke et al., 2017] candidate objective function.
LLM Objective Discovery CIFAR-10
Transfer Discovered Objective Models
95
Best Accuracy
Sisk ridt spnrant
Test Accuracy
andiae
90
75
sioctted
untresylou Hosa
70
Val
85
Test
65
ResNer18 8
SimpleDLA
EfficienNNeiS
10
15
25
30
LLM Proposals/Generationn
Figure 2: LLM-driven objective discovery for CIFAR-10 classification. Left. Performance across LLM- discovery trials. The proposals alternate between exploring new objective concepts, tuning the components, and combining previous insights. Right. The hest three discovered objectives transfer to different network architectures and longer training runs (100 epachs).
Figure 2 depicts the performance of the proposed objective functions across the discovery process. The different discovered objectives all outperform the standard cross-entropy loss. Interestingly, we observe that the LLM-driven discovery alternates between several different exploration, fine-tuning, and knowledge composition steps: Initially, the LLM proposes a label-smoothed cross-entropy objective. After tuning the smoothing temperature, it explores squared error loss variant, which improved the observed validation performance. Next, the two conceptually different objectives are combined, leading to another significant performance improvement. Hence, the LLM discovery process does not perform a random search over objectives previously outlined in the literature but instead composes various concepts in complementary fashion. Furthermore, the discovered objectives also generalize to different architectures and longer training runs. In Appendix D.3 we show that this process of discovery is robust to the choice of sampling temperature and prompt/context construction.
4 Discovering Offline Preference Optimization Objectives
In this section, we run our I.I.-driven discovery to automatically generate new state-of-the-art preference optimization algorithms.

---

4.1
Discovery Task Multi-turn Dialogue on MT--Becch
Best MT-Berch Score
8.0
LLM Objective Discovery Preferance Optimization
Best (TT-Berch Score
8.0
Objective Discovery Preference Optimization
7.9
decap
7.9
contaisd top ogaac tosa
wwwedddddd.... www.se. Joue
7.8
7.8
7.7
condaned http://wddwooooooooooooooooo
7.7
mmottee rgg...e.... pe5
7.6
0.0
5.0
10.0
15.0
LLM Prenraalsseeeeratloon
LLM Proposalo/Generations
Figure 3: Examples of LLM Objective Discovery improvement across generations. The first and second runs
are shown left and right respectively.
In this section we use our LLM-driven discovery method to discover new objective func- tions f for offline preference optimization, as defined Section 2 and Equation (3). Specif- ically, at each generation i, GPT-4 generates PyTorch [Paszke et al., 2017] code of can- didate objective function Si- Each objective function takes as input the variables of flog (z) log Tee (Valaa))) log No (y)lz)), log T [z)]. and returns scalar. For each proposed objective fi, we check if fi is valid with a unit test.
For each valid generated objective function ft. we finetune an LLM and then collect performance evaluation score. Specifically, we build on top of the alignment-handdoae repository to finetune our models. Notably, this repository, when using DPO, reproduces 'Zephyr 7B Gemma" Tunstall and Schmid [2024], Tunstall et al. [2023b], which at the time of release, achieved state-of-the oftthh--rtttttttt scores on MT-Bench for 7B models. 'Zephyr 7B Gemma' first takes gemma-7b [Gemma- Deemma-Team et al., 2024] and finetunes on the 1ikta--Ok--O-sftt dataset [Liu et al., 2023] to produce 7ephyx-7b-gemma-st* DPO Mix 7 When evaluating new objective function, we replace DPO this last step with the generated objective function, keeping the same hyperparameters. We show example runs in Figure 12 and provide further experimental details in Appendix B.
[Tunstall al., 2023a]
It is then trained on the pairwise preference dataset of 'Argilla
Once we have a trained LLM for the proposed objective function fi, we evaluate that LLM on the popular multi-turn dialogue evaluation benchmark of MT-Bench [Zheng et al., 2024]. This is a multi-ture turn open-ended question set, which uses GPT-4 to assess the quality of the trained model's responses, obtaining a high correlation with the popular Chatbot Arena [Zheng et al., 2024]. We provide further evaluation details in Appendix C.
4.2 Discovery Results
Figure 4: MT-Bench Discov- ered Objective Evaluations
After evaluating approximately 100 objective functions, we catalogued the best-performing ones in Table 1. We tabulate the high-level objective forms here and provide the full objective loss functions and their associated code in Appendix E. Moreover, we also plot the best performing sub-task evaluations in Figure 4.
Table 1: Discovery Task MT-Bench Evaluation Scores for each discovered objective function f- We provide the baselines first, followed by a dashed line to separate the objective functions that were discovered. We provide details for each discovered objective function in Appendix E.
Name
Name
Obsective Function
Score d/ 10)
DPO
Direct Prefenence Optimization
logt' exp)-
7,385
Hlogging oce 'xephye-Pu "b gomma' DPO model
logt exp(-3p)
SLIC
Sequence Litclibood Calibration
ReLL((1 Ao)
7.881
KTO
Pairwise Kahnestan- Trorsky Optimization
Me [Flhnnam et 2024]
76603
DDAQL
Dynamic Blended Adapiiive Quantile Loss
16 - 5a/ 71
7.976
AQL
Adapiive Quantll Loss
a faw( -11 - Fau13e)
7.953
PADEL
Perfommance Adaptive Decay I Logistic Loss
09- [1- 0.5- 1 p< (
7.941
AQEL
Adaptiee Quamille Feedback Loss
Y- fa (1-r) (3)
7,551
CELL
Combined Exponential Logistic Loss
ILS (ea 0.5.- [u,
7.225
LRML (DissPOPP)
Log Ratio Modainted Loss
(1 - ((0p/=)) (ay (96) o (e))
7916
PRL.
Policy Focused Loss
1/2- Srpt ll,, x.|+2 (a 1/T,
7.900
http://hhhgggggeecc///////////as 7b-gemaavv0.1
htg.://ggg.........oio////aa/a/a ote.e gemma-aft-vv..
htttpss//hugggingaac...
5

---

Held-Out Evaluations
We next validate each our discovered objective functions (shown Table 1) on held-out tasks. We find that the Performance Adaptive Decay Loss (PADLL) and the Log Ratio Modulated Loss (LRML) consistently perform well. Because its unconventional properties and performance, we refer LRML
discovered preference optimization, DiscoPOP, algorithm.
We consider three different standard [Rafailov al., 2023] open-ended text generation tasks each designed to evaluate different properties of the fine-tuned LLM policy where each LLM policy is trained with one of our discovered objective functions on preference dataset [(x", yur vi v
D
5.1 Single-tura Dialogue- Alpaca Eval 2.0
We evaluate the trained models on Alpaca Eval 2.0, [Li al., 2023, Dubois et al., 2023, 2024]. This single-turn dialogue L.LM-based automatic evaluation using GPT-4 assess the win rate of the trained LLM policy's completion compared to the of the underlying SFT base model. A Alaaaa Eval 2.0", has been validated against 20K human annotations, and aims to reduce the length bias of Alpaca Eval 1.0; where using length controlled (LC) Alpaca Eval shows a correlation with Chatbot Area of 0.98, making a popular benchmark with the highest correlation to Chatbot Arena [Dubois et al., 2024]. We also detail task training details in Appendix B.I.
Table 2: Alpaca Eval 2.0 Held Out Single Turn Dialogue Task. Win rate of the discovered objective functions evaluated on the Alpaca Eval 2.0 task against either GPT-4 or the SFT base model. Some of the discovered objective functions outperform the baselines, with best bolded. detail evaluation and error bars in Appendix C. We have highlighted the best scores with overlapping standard errors.
Function
Win Rate (%)
Win Rate- LC (%) vs. GPT-4
Win Rate (%) t
Win Rate L.C (%) f
vs. SFT Checkpoint
DPO
11.23 0.97
12.81 0.66
78.72 1 1.26
63.34 1 0.30
DPO*
+1.99
14.73 + 0.71
75.75 + 1.31
59.88 + 0.41
SLiC
10.67
13:16 1 0.69
75.05 + 1.34
59.67 + 0.42
KTO
12.57
13.58 0.67
78.81 1.25
62.76 1 0.31
DBAQL
10.68 0.99
11.41
72.06 + 1.42
54.40 1 0.38
AQL
11.11
13.63 0.68
76.34 + 1.30
60.94 0.36
PADLL
14.07 1.0011.0 1.04
14.89 0.66
81.10 1 1.21
64.14 0.28
AQFL
13.63 + 1.05
15.55 0.71
79.32 + 1.23
CELL
10.27 0.93
12.26 0.61
71.75 + 1.39
57.48 1 0.34
LRML
13.21 1 1.02
14.78 0.67
79.27 1 1.24
65.18 + 0.32
PFL
8.15 + 0.83
10.67 + 0.57
68.27 + 1.44
56.14 + 0.43
We provide the Alpaca Eval 2.0 results Table As reference policies, we used GPT-4 for absolute comparison and the SFT-trained model for relative comparison. We observe that the discovered LRML (DiscoPOP), PADLL, and AQFL functions outperform the baselines and other discovered losses on the normal and length-controlle win rates. The differences in scores among these top- performing losses are not significant, except for the LC win rate against the SFT reference model, where DiscoPOP performs best.
5.2
Summarization (TL:DR)
We train an LLM policy to, given forum post on Reddit x, generate summarization of the main points. We finetune 7ephyr--b-gemmmasst using 10% of the Reddit TL;DR summarization preference dataset [Völske et al., 2017] on each of the baseline and discovered objective functions. As reference model, we again use sephyr-7b-gemmas are outlined in Appendix B.2. To evaluate the quality of the summaries, we make use of the Alpaca Eval 2.0 library with a custom evaluation dataset existing of 694 test samples from the TL;DR dataset and custom GPT-4 annotator template described in Rafailov et al. [2023]. For additional details regarding the summarization evaluation see Appendix C.3.
Further details on the training pipeline
'nttps://githul http:/ooooo/tu/uu cranttsuullbbbban www.alaccaaaaaaaaaaaaaaaaaaaaaaa

---

In Table 3 the PADLL loss and DPO loss perform best, with little difference from each other, on the summarization task in three of four metrics. Additionally, the LRML DiscoPOP function achieves scores slightly below the top performers, especially in the length-controlled win rates. In contrast to the single-turn dialogue task, the AQFL loss does not achieve high scores in the held-out evaluation.
Table 3: TL;DR Held Out Summarization Task Win-rate of various preference optimization functions the summarization task was evaluated with the Alpaca Eval 2.0 calculations, against subset of the test set (694 samples). The baseline outputs are the human-gent generatedeee preferences, and the model after SFT (see Appendix C for details). Note that the standard enror the LC win-rate has been rounded down because of values 0.001. We have highlighted the scores with means overlapping the standard error the best score.
Function
Win Rate (%) Win Rate LC (%) f
Win Rate (%)
Win Rate LC (%)
vs. Human Preference
vs. SFT Checkpoint
DPO
88.27 + 1.07
82.82 0.00
54.38 1.52
54.64 0.00
SLiC
83.02 + 1.29
63.41 + 0.00
53.03 + 1.52
54.11 + 0.00
KTO
1.18 85.34 1.18
80.26 = 0.00
51.15 +: 1.54
50.0 0.00
DBAQL
84.71=11.21
78.68
52.55 1.52
55.14 0.00
AQL
81.87 + 1.32
1.32
+.00
+6.00+1.3
50.0 0.00
PADLL
88.54 1.05
0.00
55.34 1.52
55.64
AQFL
85.03 + 1.22
76.23 +-23+0000 0.00
49.56+1.55
50.38 + 0.00
CELL
86.33 1.14 1.14
50.35
51.90 - 0.00
LRMI.
87.63 + 1.10
$1.88 + 0.00
53.46 1.52
50. 0.00
PFL
79.84 1.35
69.23 + 0.00
44.12
44.57 = 0.00
5.3 Positive sentiment generation (IMDb)
In this task, we train LLM policy to generate movie review completions y with positive sentiment, where r is a prompt at the start of a movie review from the IMDb dataset [Maas et al., 2011]. We start with GPT-2 [Radford al., 2019] model, which had supervised fine-tuning on the IMDb dataset, and we perform preference optimization using the baseline and discovered objective loss functions. Details of the training implementations can be found in Appendix B.3. Inspired by Rafailov et al. [2023]'s experiments, we calculate the model rewards through pre-trained sentiment classifier, which we use as proxy for ground truth, as well as the KL-Divergence of the trained model and the reference model. Appendix C.4 provides further details into the evaluation for this task.
We provide results of models with converging values in Figure for LRML compared against DPO and SLiC, displaying the model rewards against the KL-Divergence to the reference model. In Figure 5a, the LRML-trained text generator outperforms the DPO model terms of rewards and KL-divergence with low 3 values (0.025, 0,05, 0.1). At higher 3 values (0.5 and 1.0) both methods show trends of increased KL-Divergence and lower rewards, but generally, LRML maintains a higher reward than DPO. In Figure 5b, we note that LRML slightly outperforms DPO, SLiC, AQFI., and PADLL at 3 E (0.05, 0.13 in terms of reward. For larger / values (0.5 and 1. 1.0), LRMI. shows similar trends of increased KL-DDieegence and rewards like the other objective functions. A more detailed comparison between the individual discovered losses and the baselines can be found Appendix Figure 8.
6 Analysis of DiscoPOP
We list all our discovered objectives in Table 1, well the code and mathematical representations in Appendix E. In this section, we now analyze the Log Ratio Modulated Loss, which we define as the DiscoPOP loss function, it performs consistently high across the held-out evaluation tasks, and we provide some intuitive understanding of how it outperforms the existing state-of -he- objectives.
6.1 Log Ratio Modulated Loss (DiscoPOP)
The Log Ratio Modulated Loss a dynamically weighted sum of the logistic loss (as used in DPO) and the exponential loss. The weight of each is determined through sigmoid calculation of the
7

---

IMDb Positive Text Generation: DPO vs LAML
0.98
IMDb Posilive Text Generation: All
0.98
0.95
0.98
0.94
Reward
-0.05
Roward
0.94
0.92
0.92
0.90
0.90
0.88
0.88
0.88
0.8
0.0
0.2
0.4
0.8
0.8
1.0
1.2
144
KL Divergence
Divergence
DPO
AGFL
PADLL
LRRL
DPO ..... LRML
SUI
(a) DPO LRML
(b) Discovered Baseline Losses
Figure 5: Frontiers of expected reward vs KL divergence for converging models for the LRML against DPO and SLiC objective function. The rewards and KL-divergence values are averaged
10 generations with different seeds. The sweep is done over E (0.025, 0.05, 0.1, 0.25, 0.5, 1.0]. The optimal point is the top left corner, where the perfect reward is achieved with minimal divergence from the reference model.
difference of log-ratios (p). Mathematically, the LRML function can be described with a temperature parameter T = 0.05 as follows:
ftrmt = (a(3p/T) 1) fapo (Bp) + = Semp((p) = -1 (1pp/ppp )) log(1 (xp(-3p)) (Bp/T) exp(-3p)
If the difference of log ratios is zero 0), which is at the start of the training when the model policy Te is equal to the reference policy Tet then the loss is equally balanced between the logistic and exponential loss. If + oc, the model policy diverges from the reference policy and chosen outputs are prefered, then the exponential term dominates. This emphasizes larger differences more strongly. On the other hand, if p - -o,, the model policy diverges from the reference policy, and rejected outputs are preferred. The baseline objective losses and the LRML, the PADLL, and the AQFL functions displayed in Figure 6, including their gradients. Surprisingly, the DiscoPOP function has a non-convex segment and negative gradients the starting point p O. This is potentially helpful for introducing curriculum or for stochasticity. Additional results and analysis of the discovered loss function can be found in Appendix D.6.
this case, the logistic loss can handle moderate differences well.
Discovered Objective Functions
Gradient Objective Functions
0.04
Gradient Pla)
0.02
Loss (00
1.0
0.00
-0.02
-0.04
-10
10
20
30
40
-10
10
20
30
AQFL
DPO
PADLL
SLIC
PADLL
SUC
LRMI.
(a) Discovered Objective Functions
(b) Gradients of the Discovered Objective Functions
Figure 6: Figure 6a: Baseline objective functions DPO and SLiC, and the discovered ones, LRML, AQFL, and PADLL. Figure 6b: gradients of the objectives as function of and with fixed B = 0.05.
6.2 Limitations of DiscoPOP
While performing very well on single-turn text generation and text summarization, we observed during the IMDb experiment that LRMI. struggles converge when B too low (3 S 0.01) or too high (3 2 2.5), likely because 3 a 0.05 never used during the discovery process.
8

---

In Figure 9 and Figure 10 of the Appendix, we plot the LRML objective function for 3 E {0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5) against DPO. When 3 is high, the DiscoPOP objective function takes the form of the DPO log sigmoid loss. During training on B = 0.01, we observed that DiscoPOP gets stuck in generating predominantly negative reviews (resulting in a reward score of ~ 0.15). We hypothesize that the loss is stuck in the local minima to the left with a negative difference in log ratios. While training with 3 E {2.5, 5.0). we observed that the model collapsed after a sharp spike in the loss and subsequently having loss value 0 and NaN outputs. This is potentially due to large gradient in the non-convex part, which could be amended with gradient clipping.
7 Related Work
Evolution and Search with Large Language Models. LLMs provide fast and automated way to create multiple candidate solutions for a problem stated in natural language [Song et al., 2024), which search procedures. Various recent works have applied this approach to coding problems Kkomera-Paredee et al., 2024), neural architecture search [Chen al., 2024a, Holt al., 2024b], virtual robotic design settings [Lehman et al., 2023], reward functions [Ma et al., 2023, Yu et al.. 2023], and algorithm heuristics [Liu et al., 2024]. Finally, recently LLMs have shown be capable of acting as recombination operators for black-box optimization with Evolution Strategies [Lange et al., 2024] and for Quality-Diversitg approaches [Lim et al.. 2024].
makes them powerful tools for driving population-base
Automated Discovery for Machine Learning. There are many other approaches to automating the discovery of generalizable machine learning algorithms. Some prior works explore the space of ML functions using genetic algorithms and hand-crafted domain-specific language for reinforcement learning algorithms [Co-Reyes et al., 2021], curiosity algorithms [Alet al., 2020], and optimizers [Chen et al., 2024b]. Other works instead parameterize a transferrable objective function using neural networks and optimize them with evolution strategies meta-gradients. For example, Lu al. [2022], Jackson et al. [2024b]. Houthooft et al. [2018], Alfano et al. [2024], Kirsch et al. [2019], Oh al. [2020], Jackson et al. [2024a] discover policy optimization objectives, Metz al. [2022] evolves neural network optimizers, and Lange et al. [2023b.a] evolve blackbox optimizers. Moreover, automatically discovering closed-form functions (i.e.. symbolic regression), works exist that leverage RL [Petersen et al., 2020], gradient descent [Kacprzyk et al., 2024], RL with evolution strategies [Mundhenk et al., 2021], pre-training transformers [Biggio et al., 2021] and hybrid combinations of pre-training transformers, which are further refined with RL and evolution strategies [Holt et al., 2023b].
Preference Optimization Algorithms. While the reduction to supervised learning makes DPO and al- ternatives easier use, other approaches have sought to simplify the RL step, including using variants of REINFORCE [Ahmadian et al., 2024, Gemma-Teer et al., 2024] as well as more fine- fine-grained feed- back [Wu al., 2024] through preferences over individual steps in the reasoning process [Uesato et al., 2022, Lightmmn et al., 2023] or reward redistribution [Chan et al., 2024]. Others use iterative offline training interleaved with sampling from the policy model and obtaining a preference ranking from themselves [Xu et al., 2023], another judge LLM [Guo et al., 2024], or an oracle [Swamy et al., 2024].
Conclusion
Summary. In this paper, we proposed and used LLM-driven objective discovery to generate novel offline preference optimization algorithms. Specifically, we were able to discover nigh-performing preference optimization losses that achieve strong performance across held-out evaluation tasks, with the highest performing providing new insights into what an optimal objective may need to possess, such as being a blend of logistic and exponential losses and possibly being non-convex.
Limitations & Future work. There are multiple limitations to our current approach. First, we have only scratched the surface of how to generate LLM objective proposals effectively. Initial exploratory experiments using techniques such as temperature sampling or worst-to-best performance sorting in the context did not yield significant improvements. But one could imagine leveraging more information about the training runs and automatically tuning instruction prompt templates. E.g. by providing entire learning curve plots to a Visual I Language Model (see Figure 13) or by meta-meta- optimizing [Lu et al., 2023] the LLM prompt. Second, the highest- loss re-purposed

---

in the traditional sense, making it affect the functional behavior and the KL penalty of the model with respect the base model. This motivates future work to study different forms, with perhaps multiple floating point parameters in the form, that each could be tuned separately. Although we provided an initial analysis sweep over this one single parameter and observed some instances the functional behavior leading to instability of training the model, further multi- parameter analysis, reformulating the objective, would be beneficial for future work. Finally, our work uses closed-source models (GPT-4) to generate code, which limits reproducibility and is costly to run. Future work could use the produced models themselves to generate code, resulting in code-level self-improvement.
Broader Impact and Ethical Considerations. This paper presents an LLM-driven discovery in-context learning pipeline that is used to generate perter-peroorming novel offline preference optimization algorithms. However, misuse of the pipeline as a tool or training an LLM to produce undesirable, unethical, or harmful outputs could be possible by a user. Furthermore, due to the use of L.LMs and training of LLMs, the outputs are susceptible to hallucinations, motivating all outputs of the L.LMs to always have content filter applied the outputs. Finally, this work takes a small step towards code-level self-improvement in language models, which could potentially result in unintended behaviors.
Acknowledgments and Disclosure of Funding
This work was supported by Azure sponsorship credits granted by Microsoft's AI for Good Research Lab and by Microsoft's Accelerate Foundation Models Academic Research initiative. The hardware used for training was sponsored by GoodAI. SH is funded by AstraZeneca. AJC funded by Microsoft Research and EPSRC ICASE scholarship award. CL and RTL were supported by Sakana AI at the time of this work. The code can also be accessed at https: //github. con/smmolt/ DiscoPOP.
References
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstlin, and Sara Hooker. Back basics: Revisiting reinforce style optimization for learning from human feedback in lims. arXiv preprint arXhv:2402.147400 2024.
Ferran Alet, Martin F Schneider, Tomas Lozano-Peres and Leslie Pack Kaelbling. Meta-learning curiosity algorithms. arXiv preprint arXiv:2003.05:25,, 2020.
Carlo Alfano, Sebastian Towers, Silvia Sapora, Chris Lu, and Patrick Rebeschini. Meta-learnnng the mirror map in policy mirror descent. arXiv preprint arXiv:2402.051187 2024.
Anthropic. Model card and evaluations for claude models, 2023. URL https://wwx-filet pdf.
anthropic. httphr/op.ccoooeee/eooo cot.:roducttcttccooooooooooo prwpuctiion///ooo images Tarages/Modun Moodel-Card-aa
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo. Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310. 12036,, 2023.
Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Parascandolo. Neural symbolic regression that scales. In International Conference on Machine Learning, pages 936 945. Pmlr, 2021.
Bernhard E Boser, Isabelle M Guyon, and Vladimir N Vapnik. training algorithm for optimal margin classifiers. In Proceedings of the fifth annual workshop on Computational learning theory, pages 144-152, 1992.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345 1952.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Vorbertt-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650, 2021.
10

---

Alex J Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar. Dense reward for free in reinforcement learning from human feedback. arXiv preprint arXiv:2402.00782, 2024.
Angelica Chen, David Dohan, and David So. Evoprompting: Language models for code-level neural architecture search. Advances in Neural Information Processing Systems, 36, 2024a.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in Neural Information Processing Systems, 36, 2024b.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing systems, 30, 2017.
John D Co-Reyes, Yingjie Miao, Daiyi Peng, Esteban Real, Sergey Levine, Quoc V Le, Honglak Lee, and Aleksandra Faust. Evolving reinforcement learning algorithms. arXiv preprint arXiv:2101.0399 033558 2021.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20:273-297 1995.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377 2023.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: simulation framework for methods that learn from human feedback, 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.0444 04475, 2024.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International conference on learning representations, 2019.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization, 2024.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Real- toxicityprompis: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009. 14622.009.111...... 2020.
Google DeepMind Gemini-Team Gemini: A family of highly capable multimodal models, 2023.
Gemma-Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre,, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprint arXiv:2403.08228 2024.
Shangmin Guo, Biao Zhang. Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint 2402::2402.0799 2024.
Samuel Holt, Alihan Hüyük, Zhaozhi Qian, Hao Sun, and Mihaela van der Schaar. Neural laplace control for continuous-time delayed systems. In International Conference on Artificial Intelligence and Statistics, pages 1747-1778. PMLR, 2023a.
Samuel Holt, Zhaozhi Qian, and Mihacla van der Schaar. Deep generative symbolic regression. In The Eleventh International Conference on Learning Representations, 2023b. URL. https: / //openreiiew met/forum?? http://rrrmmmdo/oEoEEeaa.. KdB.T..EE
Samuel Holt, Alihan Hüyük, and Mihaela van der Schaar. Active observing in continuous-time control. Advances in Neural Information Processing Systems, 36, 2024a.

---

Samuel Holt, Tennison Liu, and Mihaela van der Schaar. Automatically learning hybrid digital twins of dynamical systems. The Thirty- Anmual Conference Neural Information Processing Systems, 2024b. URL
Samuel Holt, Max Ruiz Luyten, and Mihaela van der Schaar. L2MAC: Large language model automatic computer for extensive code generation. In The Twelfth International Conference on Learning Representations, 2024c. URL
net. fatt/foruee formmmrrmmm/ wwt.:eorumidd-hrrrlllllao id=EhrdQwss4K..
Samuel Holt, Zhaozhi Qian, Tennison Liu, Jim Weatherall, and Mihaela van der Schaar. Data- driven discovery of dynamical systems in pharmacology using large language models. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024d.
Samuel Holt, Zhaozhi Qian, and Mihaela van der Schaar. Neural laplace: Learning diverse classes of differential equations in the laplace domain. International Conference on Machine Learning, pages 8811-8832 PMLR, 2022.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey, Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(9): 5149-5169, 2021.
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, OpenAI Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. Advances in Neural Information Processing Systems, 31, 2018.
Matthew T Jackson, Mingi Jiang, Jack Porker-Holder, Risto Vuorio, Chris Lu, Greg Farquhar, Shimon Whiteson, and Jakob Foerster. Discovering general reinforcement learning algorithms with adversarial environment design. Advances in Neural Information Processing Systems, 36, 2024a.
Matthew Thomas Jackson, Chris Lu, Louis Kirsch, Robert Tjarko Lange, Shimon Whiteson, and Jakob Nicolaus Foerster. Discovering temporally- reinforcement learning algorithms. arXiv preprint arXiv:2402.058228 2024b.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint 1rXiv:1907.004:0 2019.
Krzysztof Kacprzyk, Samuel Holt, Jeroen Berrevoets, Zhaozhi Qian, and Mihaela van der Schaar. ODE discovery for longitudinal heterogeneous treatment effects inference. In The Twelfth In- ternational Conference on Learning Representations, 2024. URL https://openre: //tp://e/ee/eieeee// net// forum?id==x115Ieekg..
Louis Kirsch, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Improving generalization in meta reinforcement learning using learned objectives. arXiv preprint 1rXiv:1910.0409. 2019.
Robert Lange, Tom Schaul, Yutian Chen, Chris Lu, Tom Zahavy, Valentin Dalibard, and Sebastian Flennerhag. Discovering attention- based genetic algorithms via meta-black-box optimization. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 929 929-937, 2023a.
Robert Lange, Tom Schaul, Yutian Chen, Tom Zahavy, Valentin Dalibard, Chris Lu, Satinder Singh, and Sebastian Flennerhag. Discovering evolution strategies via meta- black- box optimization. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation, pages 29-30, 2023b.
Robert Tjarko Lange, Yingtao Tian, and Yujin Tang. Large language models as evolution strategies. arXiv preprint 2rXiv:2402.8888 2024.
Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O Stanley. Evolution through large models. In Handbook of Evolutionary Machine Learning, pages 331-366. Springer, 2023.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacaeval: An automatic evaluator of -oslloiiion-following models. https: : ht///ggtttut www/tatsu- Sateaatsss-abbaaaaaaa http:/aatsu-abbaalpcc.a.eee _aall_ 2023.
12

---

Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
Bryan Lim, Manon Flageat, and Antoine Cully. Large language models in-context ai generators for quality-diversity. arXiv preprint arXiv:2404. 15794, 2024.
Fei Liu, Tong Xialiang, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang, Zhichao Lu, and Qingfu Zhang. Evolution of heuristics: Towards efficient automatic algorithm design using large language model. In Forty-firsi International Conference on Machine Learning, 2024.
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312.15683 2023.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Tay, Denny Zhou, Quoc Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pages 22631-22648 PMLR, 2023.
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2017. URL https: semanticscholast org/ CorpusID:55359220.
Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. Discovered policy optimisation. Advances in Neural Information Processing Systems, 35:16455- 16468, 2022.
Chris Lu, Sebastian Towers, and Jakob Foerster. Arbitrary order meta-learning with simple population- based evolution. In ALIFE 2023: Ghost in the Machine: Proceedings of the 2023 Artificial Life Conference. MIT Press, 2023.
Yecheng Jason Ma. William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310. 12931, 2023.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150, 2011.
Luke Metz, James Harrison, C Daniel Freeman, Amil Merchant, Lucas Beyer, James Bradbury, Naman Agrawal, Ben Poole, Igor Mordatch, Adam Roberts, et al. Velo: Training versatile learned optimizers by scaling up. arXiv preprint arXiv:221 1..07600 2022.
T Nathan Mundhenk, Mikel Landajuela, Ruben Glatt, Claudio P Santiago, Daniel M Faissol, and Brenden K Petersen. Symbolic regression via neural-guided genetic programming population seeding. In Proceedings of the 35th International Conference on Neural Information Processing Systems, pages 24912 22991222222,,. 2021.
Junhyuk Oh, Matteo Hessel, Wojciech M Czarnecki, Zhongwen Xu, Hado P van Hasselt, Satinder Singh, and David Silver. Discovering reinforcement learning algorithms. Advances in Neural Information Processing Systems, 33:1060-1070, 2020.
OpenAl. Gpt-4 technical report, 2023.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito,o, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Brenden K Petersen, Mikel Landajuela Larma, Terrell Mundhenk, Claudio Prata Santiago, Soo Kyung Kim, and Joanne Taery Kim. Deep symbolic regression: Recovering mathemat- ical expressions from data via risk-seeking policy gradients. In International Conference on Learning Representations, 2020.
13

---

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2.05..82290 2023.
Noam Razin, Hattie Zhou, Omid Saremi, Vimal Thila,, Arwen Bradley, Preetum Nakkiran, Joshua Susskind, and Etai Littwin. Vanishing gradients in reinforcement finetuning of language models. arXiv preprint ar 2rXiv:2310.0700, 2023.
Bernardino Nomera-Paredee Mobammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg. Pengming Wang. Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625(7995):468 475. 2024.
Lorenzo Rosasco, Ernesto De Vito, Andrea Caponnetto, Michele Piana, and Alessandro Verri. Are loss functions all the same? Neural computation, 1665):1063-1076, 2004.
Xingyou Song, Yingtao Tian, Robert Tjarko I Lange, Chansoo Lee, Yujin Tang, and Yutian Chen. Position paper: Leveraging foundational models for black-box optimization: Benefits, challenges. and future directions. arXiv preprint arXiv:2405.03547 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 13:3008-3021, 2020.
Richard Stuart Sutton. Temporal credit assignment in reinforcement learning. University of Mas- sachusetts Amherst, 1984.
Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaxi- malist approach reinforcement learning from human feedback. arXiv preprint arXiv:2401.0AR 04056, 2024.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: unified approach to offline alignment. arXiv preprint arXiv:2402.05749, 2024.
Lewis Tunstall and Philipp Schmid. Zephyr BugginggFaceH44zzppyrr 7a--emma---ee--e----- 2024.
gemma.
https://bugiingfacecco
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alexander M. Rush, and Thomas Wolf. The alignment handbook. https://githut: com/
huggingface, buggingface/alignnne awwgamenttt--aaaaaeeammmaaeeaa.m handbookk 2023a.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023b.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang. Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arX/v:2211.14225,25 2022.
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. Tl; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Sunmarization, pages 59-63, 2017.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, and Nathan Lambert. TRL.: Transformer Reinforcement Learning. URL httph:ttt//// htte::///ttttttt com/ huggingface/trl.
14

---

Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr. Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-gres Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36, 2024.
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16662, 2023.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang- Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, et al. Language to rewards for robotic skill synthesis. arXiv preprint urXiv:2306.08447 2023.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Zhanhui Zhou, Zhixuan Liu, Jie Liu, Zhichen Dong, Chao Yang, and Yu Qiao. Weak-to-strone search: Align large language models via searching over small language models. arXiv preprint arXiv:2405. /9262, 2024.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1438. Chicago, II, USA, 2008
15

---

Appendix
Table of Contents
A LLM-Driven Objective Discovery Implementation Details
17
A.1 Prompts
17
B Training Details
19
Discovery Task Single-turn Dialogue
B.2 TL;DR Summarization
19
B.3 IMDb Positive Text Generation
20
C Evaluation Metries
20
C.I MT-Bench
20
C.2 Alpaca Eval
................................
21
TL;DR Summarization Win-Rate
................................
21
IMDb Rewards vs KL.-Divergence.
21
D Additional Results
22
D.1 Frontiers of Expected Reward vs KI. Divergence
22
D.2 Loss Sweeps for Different Beta Parameters
23
D.3 Discovery Robustness with respect to LLM Hyperparameters
24
D.4 LLM-Driven Discovery Analysis
24
Visual Language Models Objective Discovery
24
D.6 Additional Analysis DiscoPOP
25
E
Discovered Objective Functions
25
E.1
DBAQL: Dynamic Blended Adaptive Quantile Loss
26
E.2
AQL:: Adaptiiee Quantile Loss
27
E.3
PADLL: Performance Adaptive Decay Logistic Loss
................................
27
E.4
AQFL: Adaptive Quantile Feedback Loss
................................
28
CELL: Combined Exponential Logistic Loss
.o..............................
28
E.6
LRML: Log Ratio Modulated Loss
................................
29
E.7
PFL: Policy Focused Loss
29
F
Extended Related Work
30
G Full Run Log
31
16

---

A LLM-Driven Objective Discovery Implementation Details
A.1 Prompts
We use the following system prompt generate the model responses:
You machine learning researcher who is testing out different RLHF loss functions. When you respond, output JSON where the first key ("thought") corresponds to your thought process when designing the next function. The second key ("name") corresponds to the name of your next function. Finally, the last key ("code") the exact python code that you would like to try. Here is an example:
corresponds
"thought": "Based on the previous outputa, I should try the direct preference optimization algorithm.", "name": "dpo", "code": "def sigmoid_loss( self,
Dolicy_chosen_logpa. torch. FloatTensor, policy.rejected_log:: torch. FloatTesro,,,
reference_chosen_lopp torch.Float FloatTensor,
reference_ referrance_rejscceeelloopp torch. Facch..noo
) >> Forch.FloatTemsor::
pi_logratios policy.chosen
ref_logratios reference_ copee
referene__rjjcctddlooo
logits pi_logratios ref ref_logratios
losses -F. loggggigoo (self (self beta * logits)
return losses"
You are deeply familiar with binary classification losses from the literature. Be creative and reference prior literature when possible.
You must use the exact function interface used above. Feel free to define extra hyperparameters within your function as constants. Do not make them attributes of self
Note that 'self beta 0.05'.
RLHF loss functions train on dataset of pairs of preferred and rejected completions.
"policy_chosen_ chosen_logpss refers the policy's log probabilities of the preferred completion, and "policy_rejected policy's log probabilities of the rejected completion. "reforence_chosm__oopr and "reforence for the reference (base) model.
refers to the
refer to the
The user will then return to you fitness that corresponds to the performance of the resulting model on a downstream task. Your goal is to maximize performance.
We then provide the first user prompt as such:
Here are some results we 've obtained:
"code":
def logistic_log. self, policy_chosen. logps:: torch. FloatTensor,
17

---

policy_rejjctee_oog::: torch FloatTensor, roference_chossm_loop torch. FloatTemsor,
reierrane.rjjjcceeelooop: torch. PloatTensor,
pi_logratios volicy_chosen__ogpr
policy_rejected
ref_logratios reference_ stterence_chose__oope rejere
logita pi_logratios ref ref_logratios
losses logigggioo (self beta logits)
return losses
"fitness"; 7.8875
}.
"code": def hinge_loss( self, : torch. PloatTensor,
policy.rejjcted_lopp:: torch. FloatTensor,
referencee.hooso_log::: torch. Flott.Proa,,,ssor,
referenee.rojeeddllopo. torch FloatTensor,
torch.FlottTenoo::
pi_logratios volicy_chosen__oghh
policy.rejected. logpp
ref ref_logratios
reference reference_choaen. www.sehhhoeccpeeeeeeeeeeeeee logps [eterrece__eeectedee
logits pi_logratios ref ref_logratios
losses torch.relut self beta logits)
return losses
"fitness": 7.88125
}.
"code": def ipo_loss self,
policy_chosen_logp:: torch. FloatTenoo,,,
policy_rejecttedllop:: torch FloatTensor, defereance_choson_loop. torch, Floc..eroa,e,sorr reference_rejeectd_..g. torch FloatTensor,
) >> Frrcch..lottTesor..
pi_logratios coliey_chosenn.ogp
policy.rejected
ref_logratios reference_ caaarenncechhoee_lope
refereace
logits pi_logratios ref ref_logratios
losses (logits 1 / (2 * self beta)) ** 2
return losses
"fitness": 7.84
}.
{ "code": def kto_pair_loss'
self,
policy.chhose_logps::: torch. FloatTensor,
policy/-eejettellogp::: torch FloatTenoo,,,,
peareence_choaem_oogge. torch FloatTensor,
refereaceerejjcctellooosss torch. FloatTensor,
) >> torch. Farch.FloatTense.
chosen_KL* coosicy_chosm__ogp referrace__hosse__logs)) mean O).
clamp (min=0)
rejected_KL
reference
mean cllamp (min-0)
chosen_lograticn rejeatid policy_chosen_ rejjce__ejected reference_chosem__ooo
reference_ reference_rejjcte__oogg # As described in the ктO report, the KL term for chosen (rejected is estimated using the rejected (chosen) half.
18

---

losses torch... cat
F.sigmoid beta
(chosen_logratios rejected_KL)
)
1 F.sigmoid(self. beta (chosen_K) KL logratio
)
)
return losses
"fitness"; 7.6012
]
Please generate the
Upon testing the generated code, if error is encountered, we provide the following prompt, where 'error' is the text containing the system emror:
Code not valid. Error: (error)
Please generate the next
Upon successful completion, we return the following user prompt, where 'val' is the MT--ennch score:
Fitness: {val}.
Please generate the next one.
B Training Details
B.1
Discovery Task Single-turn Dialogue
For each valid generated objective function fi, we it to train LLM and then collect perfor- mance evaluation score. Specifically, we follow the same process when training and evaluating all objective functions, starting with pre-trained supervised fine-tuned (SFT) billion gemma model of (eahhr--b-gemma-sst This is a billion base version gemma [Gemma-Tees Team et al., 2024] model the 10kia--- dataset [Liu al., 2023]. Starting with this model, we train it on the pairwise preference dataset of 'Argilla DPO Mix 7K'; which attempts to create high-qu gigh-tualtty preference dataset by filtering only highly rated chosen responses from the datasets of a multi-turn dataset, instruction following dataset [Longpre et al., 2023] and diverse preference dataset that covers truthfulness, honesty and helpfulness [Cui al., 2023]. For each training run, we trained all the parameters of the starting model, using fixed / 0.05. We used the same fixed syper-parametery for all training runs unless explicitly noted. Specifically, we used learning rate of 5e-7, 1 floating-point format, two epochs, batch size per device of two, a gradient accumula- tion step 8, cosine learning rate scheduler, and Adam optimization algorithm [Loshchilov and Hutter, 2017]. We use the popular TRL transformers library [von Werra al.), adapting the offline preference optimization objective function to train all models. The models were trained on 8 Nvidia A100 GPUs. An individual training run takes approximately 30 minutes. We provide training and evaluation statistics for discovered objective functions in Figure 7. We also provide the equivalent code implementation https:/ // //ggttu.bbb... hwt.a//eerrcaaaaaa/bbiiiia
supervised-fine-uuee
B.2 TL;DR Summarization
To determine if the discovered objective functions generalize well also to other tasks, we use them to preference optimize L.LM for text summarization. Specifically, start again with pre-trained supervised fine-tuned (SFT) 7 billion gemma model "h-phyt-h-gemmm--st" and we optimized with the objective function fo on subsample of the Reddit TL:DR summarization preference dataset
19

---

Train and Evai
Train and Eval Accuracy
10-
DPO
PABLL
LANL
SLC
0.8
Final Exal LoSs
Rewwee AcTurne
0.6-
0.4
02-
02-
0.0
0
20
40
60
80
1.00
20
40
60
80
100
Vain Stop
Train Stop
(a) Loss
(b) Accuracy
Figure 7: Training and eval statistics of DPO, SLiC, PADLL, and LRML. The losses are not directly comparable to each other, they calculated differently for each model. Interestingly, eval results are not strongly correlated with the downstream MT-Bench scores, as LRML achieves the worst accuracy.
[Völske et al., 2017] More precisely we use the first 10% of the dataset for preference optimization, which amounts to around 8'000 training samples. During training the hyperparameters are kept the same as in the single-turn dialogue task, explained in subsection B.1, except that LLMs were trained 4 Nvidia A100 GPUS using a gradient accumulation step of 16. An individual training run takes approximately 1.5 hours.
B.3 IMDb Positive Text Generation
Another popular generalization task for preference optimization [Rafailov et al., 2023] is to fine-tune small LLM to generate positive text for movie review, based on the IMDb sentiment dataset [Maas et al., 201 following the the work of Zhou et al. [2024]. As starting model, we use a GPT2 model [Radford al., 2019], that was supervised fine-tuned on the IMDb dataset Subsequently, we apply the baseline and discovered objective function fü for preference optimization. The goal of the LLM is to give a short prompt of 2-8 tokens, which indicate the start movie review, generate positive review. As we are interested in the effect of /3 on the rewards and KL-Divergence, we train the objective functions over a sweep of 3 E (0.01,0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5). Every LLM is trained for three epochs, using the Adam W optimizer, with initial learning rate of 5.0e-5, warm-up scheduler of 0. 1, and a cosine learning rate scheduler. The models are trained on 4 Nvidia A100 GPUs, using gradient accumulation step of 8, and batch size per device of 2. The training takes around 30 minutes.
C Evaluation Metrics
C.1 MT-Bench
To assess the fitness of the discovered preference optimization loss function during the discovery phase, we evaluate the trained LLMs on the MT-Bench [Zheng et al., 2024] benchmark. The evaluation benchmark consists of 80 high-quality multi-turn questions from various disciplines. The goal is to assess LLM's ability follow instructions and keep the flow of conversation. A larger LLM, in our case GPT-4, is then used as judge score the quality of the answers with a number from (lowest) to 10 (highest). Scores are given based on the quality of the L.M'' s first-turn answer (single-turn) and first and second answers (multi-turn). Finally, the MT-Bench score is the average
7 http:///uggnnggieee.ooo//usc///e Sun.aaaaaaccccccooooooaaaaaaaaaa summarize comparisons
20

---

of single-turn and multi-turn scores. For answer generation and evaluation, we used the FastChat library" and its standard sampling and temperature parameters, provided by Zheng al. [2024].
C.2 Alpaca Eval
Alpaca Eval 2.0 [Li et al., 2023, Dubois et al., 2023, 2024] is also a popular benchmark for evaluating LLMs. This is single-turn dialogue LLM-based automatic evaluation using stronger LLM, here GPT-4 Turbo, to assess the win rate of the trained LLM policy's completion compared to either GPT-4 the of the underlying SFT base model. Specifically, Alpaca Eval 2.0 has been validated against 20K human annotations and aims to reduce the length bias of Alpaca Eval, where using (LC) Alpaca Eval shows correlation with Chatbot Arena of 0.98, making it popular benchmark with the highest correlation to Chatbot Arena [Dubois et al., 2024]. The Alpaca evaluation dataset consists of 841 high-quality instructions from different data sets. The library provided by Dubois et al. [2024] calculates the win-rate (percentage were the trained policy is prefered over the reference policy, first introduced in Alpaca Eval 1.0), and length- cength-controlled win-rate, where linear model is fitted to de-bias for length of the prompt and instruction difficulty, We used temperature of 0.7, sampling. and a maximum number of new tokens of 1024 to generate the answers. Furthermore, the library provides the standard error of the mean, which indicates the confidence of the win-rate and LC win-rate.
C.3 TL;DR Summarization Win-Rate
To evaluate how well the discovered objective functions generalize to the summarization task, we use the Alpaca Eval 2.0 library, similar to subsection C.2. Instead of using the Alpaca evaluation dataset, we create a custom dataset of 694 samples from the IMDb preference test dataset. Additionally, we change the prompt of the annotator LLM, to fit the "Summarization GPT-4 win rate prompt (C)" as described in Rafailov et al. [2023]. The (LC) win-rate is calculated against either the existing against the summary generated by the SFT reference model. For
human-chosen test sample summary generation, we apply a temperature parameter of 0.7, sampling, and maximum of 256 new tokens. Moreover, we stop the summarization after the "'n" token avoid nonsensical generations. Furthermore, as we cannot calculate an instruction difficulty for the length--ontrolled win-rate, we omit this term from the linear model (This has only a small impact on the metric). In addition the win-rates we also provide the standard error as a measure of confidence.
C.4 IMDb Rewards vs KI.-Divergence
For the positive text generation, we do not require an LLM judge compared to MT-Bench, Alpaca Eval 2.0, and TL:DR evaluation, as we take pre-trained sentiment "lassifier as ground truth reward scorer. The LLMs apply sampling and a maximum of 60 new tokens for the positive text generation. The rewards and KL-divergence are averaged over 10 different generations from the trained LLMs.
hhtps://githubb http://hhwwhhioo///m//ys////////
10 hhpp:://gtthubb htwp:h/www....maaaaaaaaaaaaaaaea htt.:/ggig...eeeoooooeeeeeeeeene largeeeelll
21

---

D Additional Results
D.1 Frontiers Expected Reward vs KI. Divergence
IMDo Positvve Text Generation: DPO vs LRML
IMDo Positiv Text Generation: SLIC LRMLL
0.98
-0,0
0.98
71 1-0.05
0.005
+0-000
0.98
0.94
Roward
0.92
3.0.20
10.20.
Reward
0.92
H0.1
0.90
0.90
0.88
0.88
0.863 11
0.88 23.
-08
-1.00
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
KL Divergence
KL Divergence
DPO
LRML
SUC
(a) SLiC LRML
SLiC LRML
IMDb Posilive Text Generation: DPO vs PADLL
IMDb Positive Text Generation: SUC vs PADLL
0.98
00000
0.98
08
12006
30.325
0.96
22.05
0.94
0.94
Reward
Reward
-0.1
0.92
0.90
0.88
0.86
1-18
0.86
0.0
0.4
0.6
0.8
1.0
1.2
1.4
0.0
0.2
0.6
1.2
1.4
KL Divergence
KL Divergence
DPO
PADLL
SLC
PADLL
(c) DPO PADIL
(d) SLiC PADIL.
IMDb Positive Text Generation: DPO va AQFL
IMDb Positive Text Generation: SUC AOFL
0.98
0.98
0.005
0.96
0.94
0.94
-0.1
Reward
0.82
-0.25
Reward
0.82
0.90
0.90
0.889
0.88
0.88
1.5
0.86
1::8
KL Divergence
KL Divergence
DPO
AOFL
SUC
..... AQFL
(c) DPO vs AQFL
(f) SLiC vs AQFL
Figure 8: Frontiers expected reward vs KL divergence after convergence for the baseline functions all the discovered ones. The rewards and KL divergence values are averaged over 10 generations with different seeds. The sweep is done over 3 E (0.025, 0.05, 0.1, 0.25, 0.5, 1, }. The optimal point is the top left corner. where perfect reward achieved with minimal divergence from the reference model, avoid reward hacking.
22

---

D.2 Loss Sweeps for Different Beta Parameters
Comparison of DPO vs LRMML for dilferent 5 Values
( -2.01
e =0.00
9-0005
1.0
DRO
0P0
1.2
RM
2.6
Lo0s
0.7
0.8
0.2
0.4
0.5
-20
Loghts
DPO
DPO
DPO LNVE
USB
UNM
Lode 6g0
Looal Lg3erfg
Loae
0.5
0.3
0
40
23
200
20
Lugits P
Logis P
Logit
9 -2.6
9-45
15
20
Lu00 (1))
80
LoGG
40
10
Logits
Logits
Logit
2.5, Figure 9: DPO and LRML objective function over / E f0.01, 0.025, 0.05, 0.25 0.5, 1.5, 5).
Comparison of DPO vsLRML Gradients for Different ,3 Valuee
+0.01
2.00.00 02.05
-0.000
0.00
0.04
Loos Geddent V /0G
0.01
0.00
00000
0.00
0.00
-00085
-0.1
-00000
1.00375
-0.02
-0.04
-20
-20
-20
Loghs
Loghs
Loghs
8 -0.1
88 80.0.5
-..05
DPQ
0.2
DPQ
0.4
DPO
LRVAL
LRVA
LRVEL
Gandoon
Grddion
0.50
-0.
-02
0.06
-02
-0.4
Lugis
Lugis
Lughs
3-1
8-2.5
8P0
DP0 IMM
0P0
Gidden v,
V,
-0.
-1
-2
12
Looits
Loght
Logit
Figure 10: DPO and LRML gradient function over E {0.01, 0.025, 0.05, 0.1, 0.5, 1, 2.5,5). 5).
23

---

D.3
Discovery Robustness with respect to LLM Hyperparameters
CIFAR-10 Objective Discovery Temperature Settings
CIFAR-10 Objective Discovery Context Content/Sorting
CIFAR-10 Objective Discovery
Context Valid Examples & Thought
Validation Accuracy
80
Sort - Top
75
Temp:
Sort Top
Thought-
70
Temp:
Sort Top
Temp:
Sort Top 10
NoToought-AA
Thought-Al
0
5
10
15 0
5 10
15
0
5
10
15
LLM Proposals/Generations
LLM Proposals/Generations
LLM Proposals/Cenerations
Figure 11: Robustness of the LLM-driven discovery process. Left. We compare different sampling temper- atures 40.1, 0.5.1.0}. Middle. The default configuration includes all objective proposals and evaluations in chronological order. Here we also explore using only the top-K performing objectives unsorted and sorted by their performance. Right. We also investigate whether using "thought" as part of the context and whether to include non-valid code and error messages improves performance. The discovery process for CIFAR-10 objectives (5 epochs) is robust to these settings. The results are averaged across 3 independent runs.
D.4
LLM-Driven Discovery Analysis
LLM Objective Discovery - CIFAR-10
No fitness feedback - CIFAR-10
82.5
82.5
Bost Best Accuracy
77.5
Best Best Accuracy
77.5
75.0
72.5
72.5
70.0
70.0
67.5
Val Test
67.5
Val
65.0
65.0
Test
0
10
15
5
15
LLM PropoaalsGGnneration
LLM Preposas/Generratonn
Figure 12: LLM- -di-en discovery for CIFAR-IO 10 loss functions with (left) and without (right) providing fitness feedback. Note that without feedback, it performs worse but also is unable refine its ideas, resulting in fewer improvements across generations.
D.5 Visual Language Models for Objective Discovery
VLM-Driven Discovery
vs. LLM Discovery CIFAR-10
Best Accurnty
Extend/Update Context & Fitness
VLM-Test
LLM
Visual Language Model Proposal
to
20
VILLM Proposalaaiineertoonn
use ENLLARI
Inner Loop Opt. Run + Visualize
Objective Discovery with a Visual Language Model (VLM) for 10 (20 epochs). We provide a plot of the training and validation accuracy across training as context components the VLM (TrT--Turbb).
Figure 13:
24

---

D.6 Additional Analysis of DiscoPOP
We performed further analysis to improve our understanding of the discovered loss function. We hypothesize that the local optimum of the DiscoPOP loss could catch noisy or incorrect data points. By inspecting the DiscoPOP loss values and log ratios of the training preference pairs in the IMDb dataset, we see that 1.35% of training points fall there (see Figure 14). Although we use the binary preference labels from the IMDb dataser for training, the dataset also includes a positivity reward score for each completion, calculated by separate reward model. When we analyze the data points between the local optima, we find that the positive and negative completions are significantly closer in absolute reward difference than those outside the local optimum (See Table 4). This implies that the preference labels on those points are likely more challenging to distinguish and help empirically validate our hypothesis.
Hib6 brining Legit Dutebbtiio Less
201)
750
20
-50
3000 12.5
Figure 14: Distribution of 8-scaled difference of log-ratios (left y-axis) and corresponding DiscoPOP loss value (right y-axis) of the training samples on the IMDb positive review generation task. The 0.785929 and a local maximum
DiscoPOP function has local minimum /Irmt(-2..771) firma (1.44012) = 0.87829. The number of samples within the two local optima corresponds to 1.35% of the training set.
Description
Domain 8p
Mean
95%-CI
p-value
% of training set
Between optima
(-2.3714, 1.44012]
0.981
(0.830, 1.131)
1.35
Full range
IR R 22.37.3,, 1.44012]
3,861
(3,818, 3,904)
< 0.01%
100
Outside optima
3.9
(3,857, 3,944)
< 0.01%
98.65
Below local minimum
(-o0, 1.33744
4,086
(4.041, 4.131)
< 0.01%
6.22
Above local maximum
(1.44012, oc)
1.141
(1.076, 1.206)
4.29%
92.43
Table 4: The IMDb positive review preference dataset also provides "golden reward" scores for the chosen responses Tw and rejected responses Tı, calculated with a strong sentiment classifier log p(neg)). We analysed the absolute difference in reward scores between the chosen and rejected responses [rw - rıl across the different domains of the DiscoPOP function and report the statisties here. Training samples within the optima have a significantly lower mean absolute difference in rewards compared to the other regions in the DiscoPOP loss function. The samples stuck within the optima are the training triplets where the chosen and rejected responses are "closest" to each other and likely the most "noisy" labels.
E Discovered Objective Functions
To describe the discovered losses mathematically, we define three existing preference optimization losses here:
fpp) = (3p)) = llg((
1 1 exexp(-8p) = =log(1 + exp(-Bp))
(6)
(3p) ReLU(I -8p)
(7)
25

---

feap(Bp) exp( exp(-xpp
(8)
Moreover, we display the code of the discovered losses is oulput by the LLM. In addition, we provide mathematical representation of each, which we have adapted to be consistent with /3 being the KL-Divergence regularization parameter. This because the generated code for LRML, DBAQL, AQL, AQFL, and PFL did not uphold the 3 ought to be multiplied with the difference of log-ratios before any further calculations. If this was not upheld, it could lead the loss function changing shapes based on the KI -l.reeuulaiization term, and therefore, models could not converge or potentially collapse. In future work, we should constrain the exploring LLM to uphold the 3 multipicaaiion with the input before any other calculations are done with the difference of log-ratios p. As the meta-cxploration was done with a set 0.05, and we wish to keep consistent with this scale of regularization, have adapted the losses by dividing p values used in intermediate calculations with a scalar 7 = 0.05.
In the IMDb experiment in Section 5, we have thus used the corrected version of codes for the discovered losses based on the provided mathematical representation, as we were most interested in the effect of the Kl-ddivereence compared to the model rewards.
E.1 DBAQL: Dynamic Blended Adaptive Quantile Loss
MT- Score: 7.978
def
self, policy_chosen_Joge torch. FloatTensor, policy.reejcceed_lop::: torch. FloatTenoo,,
reference.,hoeem_1opp torch FloatTensor,
refereacee_ejjeccddlogggss torch torch. FloatTensor,
) >> Flocc..Fooateensor::
import torch functiionnn as Constants for the loss function
starting-quantile 0.5
guantile_adapt_rat 0.01
temperature 0.9
dyaamic.blend_rate 1.0
pi_logratios policy_chosen_loga ref ref_logratios reference reference_chosen. logits pi_logratios ref_logratios varibilyyri var
Calculate an adaptive quantile based on a moving target
soviinggqantile starting.quantile guantile_adapt_rae (torch (iggoi me)n()) Sharting-quantill
Calculate dynamic blending coefficient based on logits variability dynamic_blend blefd_coofff = sigmoid aaggssssgggggggtteearrrao111111
dynamic_blend_rate
Prepare components of the blended loss logistic_loss -F -F. logeignood beta beta logits / temperature)
exp_loss = torch exp (-self beta * logits temperature)
Blend the losses dynamically losses dynamic_ blasd__beedd__eeeeeeee logistic_loss (1 brramic_blend.com return losses
exp_loss
fdboot (Bp) o (Var[(pp/T) fapo + (1 o(Var[3p/T]) (ep 0.9)
(9)
0.05
0.05
(10)
26

---

E.2 AQL: Adaptive Quantile Loss
MT-Bench Score: 7.953
def adaptive__uatille...sss
self, policy_chosen,_ogpa torch. FloatTensr,,
policy_rejjcteelloops torch torch.Float FloatTensor, deference_chosennlogge torch. Floaatennr,, reference.rrjjcced_looos torch. FloatTensor, torch. :loatTenne::
percentile 0.5 Start with the median quantile
moving__quantlle_eiiggh 0.01 # Weight for updating the moving quantile
pi_logratios poliey_chosem_logps poli ref ref_logratios reference wawpeecccechheeeeooe___ee
seference__eeecteeeeee logita pi_logratios ref_logratio,
moring-quantile = percentile noving.quatileereiih (torch.
(ggitid(logit..mean mean ()) percentile)
quantile_veights, torch, sigmoid s-lff beta (logits (oving-quantile)
logistic_losses -F logsigooid (self beta logits)
hinge_losses torch. reluh1 self beta logits)
Blend the logistic and hinge losses based on the dynamic quantile weight
losses muantile_veights logistic_losses (entileeveights) binge_losses
return losses
fegt(Bp) q fápo (Bp) + q) (apo((p) a 3p)
(11)
(12)
m2 0.5 + 0.01 (E[o((p/TT] -
(13)
0.05
(14)
E.3 PADLL: Performance Adaptive Decay Logistic Loss
MT-Bendi Score: 7.941
def
performance_ htt.oraanc...appp.....eeeeeeeeee
self, policy_chosen_Loge logps: torch. FloatTensor,
policy.rejacted_logg:: torch FloatTenoo,,
deferenceechosen_logu
Frrc..loottenso.
refereence_rejecctd_logg torch. FloatTensor,
) >> torch. Forch.FFootteenoor: base_ 0.9
mismatch.penalty 0.5 Penalty decay for mismatched choices
pi pi_logratios policy charlycchoeeeoggeeeeee
ref_logratios reference_chosen_logr
reference_.ejeccedddde
logita pi_logratios ref ref_logratios
mismatches (logits 0) float( Identify mismatches
deaptive_decay base _eccy (1 mismatches Sesmatch_penaleee losses adaptive_decay -F. loggigogoo (self beta logits) return weighted_ losses
27

---

Spadu( (3p) dudpt [pp)
(15)
Ōhase (1 [p:0] 7) (Ap)
(16)
dase (1 [c00] 7) log( 1 exp(-3p))
(17)
0.9 - (1-1(pc0) 0.5) log + exp( -8p)
(18)
This loss can also be rewritten as:
fpadtl (3, p) = {1 Oponx Sapo ifp 2 0 8neg Sdp (Bp),
wweeee
ópos > òncg >
(19)
if p < 0
E.4
AQFL: Adaptive Quantile Feedback Loss
MT-Bench Score: 7.931
def odapiiee_guatiie.feedeat..t
self, policy_chosen_loge torch. Floattensor
policy.rejacted_loga: torch. Floateeno,,,,,sor,
refereecee_hooem_loop:::: torch. FloatTensor,
reference_
torch FloatTensor,
) >> Floch..Foot::nsorr:: import torch.nn.
quantile_update_raa 0.05
distance, scale 0.1
pi_logratios choiey_chosen_oogss policy rejicyyrrjecttd_logp ref_logratios reference Lloerence_chosen_lopp reference logits pi_ pi_logratios ref_ _ef__ogratios logits_std sogits.st(()
adaptive logits_ std torch. torch.sigmoile sigmoid mean ()
adaptive. mean()) adaptive_quantill.
(torch sigmoid (logits.
distance_fro__uaatil (logits (eaptive_quantilee abs blend_rate torch signoid
distance_from_quuntte
lassticc_lsses = -F loggigmoid beta logits) hinge_losses torch. relu self beta logits)
losses blend_rate logistic_losses + (1 blend_rate) hinge_losses return losses
fag)((pp) - (3p) + r) fauc((p)
(20)
r a o0.1 * d)
(21)
d 1/p/T m2l
(22)
m2 = mi 0.05 (E(sp// m1))
(23)
B[o((30/7)) / Var[Bp/TT
(24)
T 0.05
(25)
E.5 CELL: Combined Exponential + Logistic Loss
MT-Bench Score: 7.925
28

---

def hra.aneeeex..lgistii..oooo. self, policy_closen_loop:: torch. Ploatteeoo,,,,,
policy_.rejateedloop:::: torch. Float.ssoot,,,sorr
referenceechhsennloogs. torch FloatTensor,
referemne.rejjeceedloogg.. torch. FloatTensor,
-> torch.
= policy_chosen_ogge.
ref_logratios reference_
logits pi_logratios ref ref_logratios
exp_losses torch t exp logits) 1og_losses -F. 1. beta logits)
# Combine the losses with a tunable mixing coefficient
alpha = 0.5
losses alpha exp_losses (1 alpha) log_ losses return losses
Bp) = 0.5 Japo (3p) 0.5 Seap (Bp)
(26)
E.6 LRML: Log Ratio Modulated Loss
MT- MT-Benchh Score: 7.916
def
self, colicy_chosen_lope torch. FloatTense,
policg.rejectelllop:: torch. FloatTero,,,,,,,
leferrence_choss_llogsss torch. FloatTenoo,,,,s.r..
reference.rejectte_.ogssss torch FloatTensor,
>> Corch.FloatTensee
pi_logratios policy.chosem_logg
ref_logratios reference_ caaaaacceccchhee_oogooe
refereace.
logits = pi_logratios ref ref_logratios
# Modulate the mixing coefficient based the log ratio magnitudes
1og_ratioomodulatiin -F. 1o.1 beta logits)
torch. sirch.sigmoid((ogits)))
logistic_componett
exp_component torch. exp (-self beta * logits)
Blend between logistic and exponential component based log ratio modulation
losses cogisnec_component
exp_compo rog_ratio_noulationn
return losses
ftrm = (100)01110000000 (1-o())/T) (3p) + (BppTT) Seap(-Bp)
(27)
T 0.05
(28)
E.7 PFL: Policy Focused Loss
MT-Bench Score: 7,900
def policy_.fcused__oss
self, policy_chosen_ torch. FloatTenaor,
policy_rejected_lepp::: torch. FloatTesoo,,,,..
lefereaceechoseu_loge torch, FloatTensr.,
referonce.rejectdd_oee torch. FloatTensor,
>> torch. FloatTensor:
29

---

focus_scale 2.0
Scale to emphasize de de-emphasize based
the correctness of predictions
pi_logratios policy_.hosen_loggs policy_rejected_ ref ref_logratios referencee wwweeeeccechheeeeeeee_eeee logps reeeetee logps logits pi_logratios ref_logratios is_correct
policy_rejected
logistic.losses = -F eegiigo
binge_lossen torch relu logits)
focused_loss torch, where
is_coreect,
logistic_losses / focus_scale, De-emphasize correct
predictions
hinge_losses focus_scale
Emphasize incorrect predictions
return focused_loss
Interestingly, the PFL generated function code did not include any 3 values in the loss function. We have added the corrected code for the IMDb experiment, as well as the mathematical expression below. We account for the consistency of the logit values by dividing with = 0.05, which is the same value as 3 in the discovery process.
(pp)(3p) 1/2 (apo(Bp/T) I +2 fatc((p/T)) [[w
(29)
7= 0.05
(30)
Extended Related Work
In the following, we provide an extended related work of the related work included in the main paper. Evolution and Search with Large Language Models. LLMs provide fast and automated way create multiple candidate solutions for a problem stated in natural language [Song et al., 2024], which makes them powerful tools for driving iopulation-based search procedures. Various recent works have applied this approach to coding problems Chomera-Paredee et al., 2024], neural architecture search [Chen al., 2024a, Holt et al., 2024b,d], virtual robotic design settings [Lehman al., 2023), reward functions [Ma et al., 2023, Yu et al., 2023], and algorithm heuristics [Liu et al., 2024]. Finally, recently LLMs have shown be capable of acting as recombination operators for black-box optimization with Evolution Strategies [Lange et al.. 2024] and for Quality-iveersity approaches [Lim et al., 2024]. Additionally, prior work has shown LLM multi-agent frameworks' ability to excel at large code-generation tasks [Holt et al., 2024c], which we could envision enabling scaling up searching for larger candidate solutions for more complicated objectives or algorithms than the existing approaches that fit within the LLMs output context window.
Automated Discovery for Machine Learning. There are many other approaches automating the discovery of generalizable machine learning algorithms. Some prior works explore the space of ML functions using genetic algorithms and hand-crafted domain-specific language for reinforcement learning algorithms [Co-Reyes et al., 2021], ]. curiosity algorithms [Alet et al., 2020], and optimizers [Chen et al., 2024b]. Other works instead parameterize transferrable objective function using neural networks and optimize them with evolution strategies or meta meta-gradiess testa--rrdients. For example, Lu et al. [2022], Jackson al. [2024b], Houthooft al. [2018], Alfano al. [2024], Kirsch et al. [2019], Oh et al. [2020], Jackson et al. [2024a] discover policy optimization objectives, Metz et al. [2022] evolves neural network optimizers, and Lange al. [2023b.a] evolve blackbox optimizers. Moreover, automatically discovering closed-form functions (i.e.. symbolic regression), works exist that leverage RL [Petersen et al., 2020], gradient descent [Kacprzyk al., 2024], RL with evolution strategies [Mundhenk et al., 2021], I. pre-training transformers [Biggio et al., 2021] and hybrid combinations of pre-training transformers, which are further refined with RI and evolution strategies [Holt et al., 2023b]. Furthermore, more purpose ML approaches exist to perform sequential modeling of
30

---

processes [Holt et al., 2022], which can be readily used in model based RL approaches [Holt et al., 2023a, 2024a].
Preference Optimization Algorithms. While the reduction to supervised learning makes DPO and al- ternatives easier use, other approaches have sought simplify the RI step, including using variants of REINFORCE [Ahmadian et al., 2024, Gemma- Team et al., 2024] as well as more fine-grained feed- back [Wu al., 2024] through preferences over individual steps in the reasoning process [Uesato al., 2022, Lightman et al., 2023] or reward redistribution [Chan et al., 2024]. Others use iterative offline training interleaved with sampling from the policy model and obtaining preference ranking from themselves [Xu et al., 2023], another judge LLM [Guo et al., 2024], or an oracle [Swamy et al., 2024].
G
Full Run Log
We provide full below, formatted for readability.
thought
Since the logistic log loss currently has the highest fitness and direct optimization of the log likelihood ratio, let's try a variation that includes margin in the log likelihood ratios, inspired by the concept of margin in SVM hinge loss. Introducing margin could create buffer that leads to more robust learning as seen in SVMs.
name
logistic_margin_tnn code def logistic..argin_loss self,
policy.chosen.logps: torch. FloatTensor,
pollcy_.rejected_lggp torch. FloatTensor,
refereacee_chosen_loges Floch..ooar
refereanee_rjjcciee.loopss torch.FloatTens FloatTensor,
) -> Forch.FFoatTensor.
margin = 0.1
pi_logratios colicy_chosen_loops
Dolicy_rejected_Lee
ref_logratios reference referance_chosen_lopp referemre_rrjeceee_.oog
logits_ pi_logratios ref_logratios margin
losses -F. lFgloggigmoid ( (self beta bigits_vith_marye
return losses
Fitness: 709337.
Please generate the next thought
The logistic margin loss outperforned the hinge and ipo losses but did not aurpass the original logistic log loss. Perhaps introducing adaptive margin based the confidence in the predictions could yield improvements. The margin could be the absolute difference between the policy 's and the reference 's log probabilities. This could potentially penalize incorrect confident predictions more heavily. name hagguwee.narghh.ggiiii......e
proportional
code def odapiire_magih..oggiii.... self, policy_chosem__ogpe torch. FloatTenaor,
policy_rejected_oog:::: torch. Floateenoo,,,,,orr,,
reference Float.e.oo.
referonce torch. FloatTensor,
>> torch FloatTensor:
31

---

confidence_marge torch abs ob policy policy_rejecced_logpp)) torch abs (
reference.rojjetddlloog....
pi_logratios policy_chosen_loge
policy_rejected_ ref_logratios ceareence_choen_c.e
logps
Logits_with_nargin pl.logratios
ref ref_logratios
confidence_margin
losses -F. 1o..oggigooid (self beta logits_ with marginn
return losses
Fitness: 7.81875.
Please generate the next one.
thought
The adaptive margin did yield improvement. might have introduced too much variability or penalization. Let's go back to a fixed structure but combine elements from both logistic loss and hinge loss, with the idea of encouraging larger margin when the decision is correct, while still having soft, probabilistic interpretation. This combined loss could retain the benefits of both worlds
name combined combined how.::/i.sissttiteee.... code def combnne_logisticc.iigg..ooss
self colicy.chosen_loga. torch. FloatTensor,
policy.reejectedloop:: torch. FloatTeooa,,,,or,
reference_cheem.llop:::: torch. FloatTensor,
refereenee.ejjjcieeeloooo: torch. PloaaTeerrr,,
) -> torch..loatTenee
pi_logratios policy_chosen__oga
policy_rejected
ref_logratios relerence_chose_llpps
refereace.rojected_log.
logits pi_logratios ref_logratios
logistic_ -F. =f.logsigmoiddsself beta beta logits)
torch torch.relu(1 self beta logits)
combined_ losses logistic_loseet losses hinge_losses combined losses
lossss
Fitness: 7.7375.
Please generate the
thought
The combined logistic and hinge loss slightly outperformed the baseline logistic log loss but still did not achieve significant improvement Considering the nature of RLHF, where prefer the model to be correct but not overly confident, we can explore a scaled version of the combined loss where the logistic part is emphasized more heavily, as it intrinsically accounts for confidence due to its probabilistic nature. name scaled scaled_ ccaled.cobbneellggiiiii.........
code def
self policy_.hosem__oop::: torch. FloatTensor,
policg_rejectelllgp::: Forch..FoatTensor
referenceechoomm_lop:::: torch Flrch..Foot
reforonceereejeceedllgg torch FloatTensor, torch.
logistic_acale 2.0
Emphasize logistic loss more hinge_scale 0.5 # De De-emphasize hinge loss
32

---

pi_logratios policy_chosee__oope lop ref ref_logratios reference www.enccecchheenooge_eee reference_reeecteen logits pi_logratios ref_logratios
logistic_losses logistic_scale logits))
(-F. LFF.logsigmoid(sel1 beta
hinge_loases hinge_scale
relu (1 self beta logits))
combined_losses = logistic_lssse Lange_losse,
return combined_losse
Fitness: 7.85625.
Please generate the next one. a*ssass*s
thought
The scaled combined loss has shown an improvement, which indicates that balancing the contribution between the logistic and hinge components is beneficial. To further expand on this idea, we could try form of annealing where early in training the logistic component, encouraging better probability estimation, later on switch gradually to the hinge component, emphasizing decisions with a margin. We'll simulate this by using weighting factor that changes linearly through the logits, giving lower logits (earlier, less confident predictions) higher weight for the logistic loss.
of
name
annealed annealed aanellel.ccobnne....gsttti.....d
code def anaelld.ccabinal.lgggiiihhhh...e
self policy.chosem_lopp:: torch. Fleatfenra,,,,,rr,
policy.rejectted_ogge Forch.FloatTemsee
referenne__hhoss__loo:::: torch.FloatTenson.
reference..rjjeccellooosss torch. FloatTensor,
) >> torch.
pi pi_logratios colicy_.hosenn_ogee
rejicy_rejecteddloge
chopsennooosss ref loggaatatiosss reference_chone chosen_ logps reference_rejecteed rejected_logge logits pi_logratios ref ref_logratios
logistic_scale torch torch signooid--dd-ogitt logits
Higher weight for lower
Higher weight for logistic_losses logistic_scale * (-F. Lgg.logigmmoi(eell beta logits)) hinge_loase hinge_scale (torch relu(t reluch
hinge_scale torch sigmoid higher logits
(lgmoid(logits)) (logits)
self beta logits))
conbined_losses logistic_losses hinge_losser
return combined losses
Fitness: 7686875
Please generate the next one. =r:sassee=
thought
The annealed combined loss not yield the expected improvement performed worse than the scaled version. It's possible the transition from logistic to hinge was too aggressive. For the next iteration, let's return of the logistic loss with squared term, similar to squared hinge loss, which might penalize incorrect predictions more smoothly than the linear term used in the standard logistic loss.
simpler concept and try variation
name
33

---

Squared__ogistic_ooss
code
def equare_llogisiic_loae
self, policy.chose__oops:: torch. FloatTensor,
policy.rejecteelloog:::: Frrch.Floatensorr
refereaceechoomm_loog:: torch FloatTemsor,
) -> refereese.rejjcceeelooo.. torch. PloatTenrr,,,,,,
pi_logratios colicy_chosse_loops policy_rejecte_loogs
ref_logratios reference_chose__loggs reeetollce.ojjectd...ooo
logits pi_logratios ref_logratios
logistic.losses -F logsigmoid beta logits)
aquared_loaser = logistic_losser ** 2 return losses
Fitness: : 7. 6.6006299301171..
Please generate the next one.
thought
The squared logistic loss did not perform as well as hoped, possibly due to excessive penalization of difficult samples. This time, let 's integrate weighting mechanism that focuses while using the logistic component. Inspired by the focal loss used in object detection, which gives more weight to harder, misclassified examples, we can apply a similar mechanism to emphasize learning from examples where the policy significantly differs from the reference. name
hard examples
focal_logistic.loss
code def focal_logistic.loss
self policy_cchosea_oopp:::: torch. PloatTensor, policy_rejjcttodloogs: torch. floct..loo,,,soor,
Silerenneecchsse_loope torch. FloatTenor,, reference_rejected_
torch FloatTensor,
torch.
gamma 2.0 Focusing parameter for modulating the loss
pi_logratios policy_.hosen_loge,
referrace logits pi_logratios ref ref_logratios
ref_logratios refereance_chsen__lgge
sigmoids (1ogitt)
focal_weights (1 sigmoids) gamma Focus more on harder
examples
logistic_losses = -focal_ * F. logsigmoidd (self beta logits
return logistic_
Fitness: 7. 7.840625.
Please generate the thought
The focal logistic loss seems to have moderate effect, indicating that prioritizing hard examples has some merit. To build this, an alternative could be to apply temperature scaling to the logits before computing the logistic loss. Temperature scaling is often used in model calibration and can soften the probability distribution, which might work well with RLHF where overconfidence predictions is undesirable.
name
code
34

---

def teappraturr.scalllllgiiii.....
self,
policy__chose__oog::: torch. PloatTensr,,,
policy_.eejcteedloop:::: torch torch. FloatTensor, reference.ccoosm.lloge torch. PloatTensr, referenee.rejectelloog torch FloatTensor, torch. Plort.nFooaee
Temperature 1.0 softens the logits pi_logratios policy_chosen_logae policy_rejected_ reference_rejected_Loo logits (pi_logratios ref ref_logratios) / temperature logistic_losses logsigggood (self beta tempered return logistic_ losses
temperature 2.0
ref ref_logratios ateeren
**..******
Fitness: 7.86875.
Please generate the next one. ==s=a==s==
thought
The temperature scaling seemed to have positive impact, possibly due probability estimates. further explore this direction, we can try using label smoothing, which encourages the model not to be too confident about its predictions by preventing it from assigning full probability to single class. It's technique commonly used in classification tasks and might beneficial for RLHF
better-calibee
name
label_ a.bel_snottedd_lgistic....0
code def
label_atooteee_lgggtti...... self policy_chosen_logp:: Prrch..loatTenaor.
policy.rejjatted_logp: torch. FloatTenrr,,,
referenceechoosmmllop::::
retereasee.rjjeeccdlloggs torch. FloatTensor,
) torch..FloatTensor:::
label_ 0.1 Epsilon for label smoothing
pi_logratios caiiey_chosen_aie
relicy_rejected.om
ref ref_logratios reference_chosen__1oo
reference reference_reeectedd logpa
logits pi_logratios ref_logratios
smooth_positive 1.0 label_smoothing
emooth_negative = label_smoothing 2.0
targets torch ones wtt.oneeeeeeloiiisaaa....
smooth.
losses
self beta logits, targets, reduction-'none"
amooth_negative
witheopp_yitt_ooooo logits
helf.bete * logits, Berchrreros_ reduction='non.
return losses. mean ()
Code valid. Error:
Expected loss shape to be per input (e.g. (10,)), got torch Size (() Please generate the next one.
thought
Since the label smoothing implementation had an issue with the loss shape being incorrect, it seems the loss reduction was erroneously computed over all inputs instead of keeping the per iiput format. Let's adjust the implementation to make shape, by performing the the positive and negative parts and then combining them, weighted appropriately.
the loss retains the operation separately for
name
35

---

code
def oor.acaalabbl..ooteee.oogii...ee
self,
policy.chosen_logp:::: torch FloatTensor,
policy_rejecteelloog:::: torch. Flotteesoa,,,,,r,
refereacce.choam_llop.: torch. FloatTemsor,
Ploat.enoor,,,sor, rederracee_rjjecccddoogggg
) -> torch.
label_smoothing 0.1 Epsilon for label smoothing
pi_logratios policy_chosen_lops policy_rejected_oops
ref.logratios refereecee_hooen_loggs referenee
logits - pi_logratioo ref_logratios
emooth.positive 1.0 label_smoothing
amooth_negative label_smoothing
positive.targets torch. oorci.ones.like(oggitt)) smooth_positive
negative_targets torch. (orcl..srrs..)iee(lggttt))) smooth_negative
positive_losses = htwpuyy.coos..atttop....t.......
self.beta logits, positive_targets, reduction=' none
negative_losses
self. beta logits, regative.targets, reduction none
return losses negstive_losses) /
Fitness: 6.425.
Please generate the next one. a*s.ass*.=
thought
The corrected label smoothing implementation didn work as intended and significantly decreased the fitness. It seems that label smoothing may not align well with the objective of RLHF In light of this, let's explore different direction by introducing decaying weight on older samples. The idea is give higher importance to the more recent decisions made by the policy, under the assumption that they may be more aligned with the current state of the policy.
teaaying.eeigttssooisstttt.. code def deerying.rigata.oossssss._oss
self policy_chosee_logp:::: forch..lot
policj_rejected_log::: torch. torch. FloatTensor,
refereecee_coosm_loge::: Brrc.FFootTensos. refereeceerrjjecte__oogsss torch FloatTensor, torch Forch..FootTensoo.
decay_rate 0.9 Weight decay for older samples
batch_size decay_rate ** torch. arange 1, -1, -1)
decay_weights
decay_weights decay_weights... soary.eeggtt.ttoppllic......eodd coppelic device)
pi pi_logratios policy_chosen_logps policy_rejected_loge ref ref_logre lef_logratios
referenccerejeccedd_uu
logits pi_logratios ref_logratios
losses decay_weights ==F.Logsigmoidssell beta logits)
return losses / vecayss sum Normalizing by sum weights
Fitness: 7.871875.
Please generate the next one. =sssaseas thought
36

---

While the decaying weights logistic loss provided slight increase in fitness, it suggests that emphasizing beneficial, but the approach might need some refinement. potentially improve this by making the decay adaptive based on the performance of each choice. The idea would be to give less weight choices that heavily mismatched with the reference, under the hypothesis that these could be outliers errors. name bertomaacc..dppiirreeeee..ggg..p
recent samples be could
code def performance_ pertorraac..aapiirv..eeee.gggaae
self, Dolicy_chosen_lopa. torch. Froh.ennra,e
polic_.rejected_logs torch. Floattenrr,,,
reference.chosen_ logps:: torch. FloatTensor,
reference.rejjcceellopss torch FloatTensor,
base_decay = 0.9
mismatch.penalty 0.5 Penalty decay for mismatched choices
pi_logratios policy_choosn_ ref ref_logratios reference_ chopenn_oppoo
policy_rejected_logn
reference_rejected_ logits pi_logratios ref ref_ ref_logratios
mismatches = (logits < 0) float()
Identify mismatches
adaptive_decay base decay leighted_losses = staptiveddecay -F. loggggiioooi (self beta
mismatches mismatch_ penaattyypppeay
logits) return losses
Fitness: 7.940625.
Please generate the next thought
The performance- adaptivee decay approach provided a slight improvement in fitness, suggesting that dynamically adjusting the loss based the correctness of predictions is promising direction. To take this further, let's create hybrid approach which combines the performance addppiive decay with form of margin loss. This will aim to reduce the weights of not only the incorrect predictions but also those that are correct yet lack confidence, thereby promoting more decisive policy.
name ppr..pprroonncc.....l......oooo.
code def
self Dolicy_chosen_logps torch. PloatTenoa,,
policy_rejjcctd_logpp: torch. Float
reforennee_hocenn.log:::: torch. FloatTesoo,,,,,,,,, reference.rrojeceddlogg... torch FloatTensor, torch. F base_decas 0.9
margin 0.2
mismatch.penalty 0.5 Penalty decay for mismatched choices
pi_logratios colicy_chosen__oge
rolicy_rejecttddlogg
ref ref_logratios reference chosen__oggp referenco_rejectdd..oo.
logits pi_logratios ref_logratios margin_logits logits margin mismatches (margin_logee logits 0). ()oat Identify mismatches with margin
adaptive_decay = base_decay (1 mismatches mismatch.penalty)
37

---

weighted_ = adaptive_decay * -F, P.logsigmoid (self beta margin_logits) return reighted_losser
Fitness: 7.6125.
Please generate the thought
The hybrid approach with the performance margin decay did not achieve the desired effectiveness. Introducing margin may have been too punitive on correct predictions that are close the boundary. maintain the balance, let's explore the idea using triplet- style loss, which is popular in embedding learning. By treating the chosen policy logs the anchor and the reference chosen and rejected logs positive and negative examples, respectively, can encourage the chosen policy decisions to be closer to the reference chosen decisions and farther away from the reference rejected decisions.
name criplet.ssylo..oggsstic....
code def tripllet.ttyle_llgiatti...... self, policy_.chosee_logp:: torch. FloatTensr,,,,
policy.rejected_logu: Forch.Floa,,ensor,
referreace_chooss_llpp:::: torch, FleatTenor,,, reference.rrjjectddlooo
FloatTenoo
torch.FloatTensoe
alpha 0.1 Margin the triplet loss
positive =F..ogsiggoidd beta (policy.chosen_logg
reference_ ((gereence_.cosem__oogs)))))
negative_distare
-F (sel beta * (policy_chosen_logg
refereecee triplet_loss rerul dpetepeeeiveeedsstaace
nogative.distance
alpha)
return triplet_los loss
Code not valid. Error:
isnan ;) argument 'input' (position 1) must be Tensor, not NoneType Please generate the next one.
thought
It seems there was an error in the triplet-style loss function, which suggests that the computation may have been incorrect or incomplete. To correct this, intermediate steps yield non non-y tensors. Triplet loss a useful approach as inherently contains the concept of margin while allowing for a dynamic balance between the chosen and rejected log probabilities. Let's revise the code with added checks prevent any NoneType issues and computed correctly.
need to ensure that all
still
that the loss
name revised www.add.rijlee..ttleellgiiis. code def
terisdd.rippee..ttlelllpaai....t self, policy_chosen_logp::: forch.PPoatTensor.
policy_rejjcceedloop:: Frrch.Floateesor
reforennee_choem_lloo:::: torch FloatTensor,
refereaceerrjjaceelloops torch. FloatTensor,
torch. torch. FloatTensor:
alpha 0.1 Margin for the triplet loss
# Compute distances
38

---

positive_distance policy_chosen.logee reference.choosee_ooggs uegative_disten distance colicy_chossm_loggs refereace.rejectddloogg Calculate the triplet loss triplet_loss F.relu (positive_distanc alpha)
nogative_distance
Convert triplet loss into logistic- style loss togiatic.triplet.loe -F logsiiigd (self beta triplet_
return
mean Ensure the loss is averaged
Code not valid. Error: Expected loss shape to be per input (e.g. (10,)), got torch Size (())
Please generate the next one.
thought
The revised -tipee logistic loss still encountered an issue, likely due to improper application of the operations and reduction at the end, which should be on a examele basis. Triplet loss typically works with distances embedded in space, and here are dealing with log probabilities, so the direct application may not be correct. For the next attempt, let's simplify: we'll reinforce the relationship between the chosen and rejected log probabilities directly by using their difference, promoting wider margin while still within the logistic loss framework. name reinforedd.argaa.gggatti....o.
code def veinforedd.arriiu..ggiiii.....
self policy_choosm_lopps: torch. PloatTenrr,,,
policy.rejected_loge:: torch. FloatTensor, reference. torch, FloatTenor, referemee.rejectellogg torch FloatTensor, torch. PloatfePooo
margin 0.1 # Margin introduced for reinforcing the difference # Calculate log probability differences chosen_ differenee rejected_differee differencee rolicy_rejected_ reference_choseee
aalicy_chosen_1ee
eeference_rejected__U
# Apply the margin to differences reinforced_chosen = reeeuu(choeen reinforced__eeecteee
margin)
Felu (rejecteed difference margin)
Compute logistic losses with reinforced margins
losses -F. lo.gggggoo (self. beta reinforced_chosenl F.
logsigmoid (-self. beta * (anforreddrejected)
return losses mean()) (0) # Maintain loss shape per input
Code not valid. Error: Expected loss shape to be input (e.g. (10,)), got torch Size (())
Please generate the
thought
There was an error in the implementation of the reinforced margin logistic loss, likely because the mean operation was again placed incorrectly. The loss should be returned without any aggregation to maintain the per-input structure. Keeping this in mind, we need to correctly apply a margin to reinforce the separation between policy decisions and reference decisions. name cor.rtte_eeattrrrrd.ppp....ssssp code def corrected
self,
39

---

policy_cclose__lopp:: torch. t FloatTensor, policy_reejcted_logga torch FloatTensor,
refereace_chosse_loggs. torch. Flrch..Foateessorr referemce_rejectet..ogg. torch. FloatTenor.
torch. PloatTensor
margin 0.1 Margin for reinforcing the separation
Calculate log probability differences
caaicy_.hosen_loggg eeference_rejected_le
reference_chosem_.ooe
rejjected_differenne policy_rejected_lgg.
Reinforce chosen logits by adding margin reinforced_chosen chosen_difference margin Reinforce rejected logits by subtracting margin reinforced_rejecteee
rejected,difference # Compute logistic losses for reinforced logits chosen_losses FF.1ogsigmoid(seef... beta reinforced rejected_losses -F Logsiiggoiid(-sel beta reinforred
margin
Combine losses without applying mean return chosen_losses rejected_losse
Fitness: 7.525.
Please generate the next one. =*s=s==*s=
thought
The approach of applying a static margin did not yield expected improvements. It's worth considering a different perspective;
instead of focusing on introducing margins or applying decays, let 's attempt to directly optimize the policy 's certainty. Intuitively, the policy should actions similar to the reference and less confident othervise. potential approach would be to scale the logistic loss by the absolute difference between the policy's choice and the reference choice, thus directly tying the loss to the policy's certainty in decision.
more confident when choosing
name certainty.ecaleddlgiiiit...
code def certamty.scaledlloiistcc..... self policy_chosem_logps: torch, PloatTensr,
policy..eejateedlog:::: torch. FloatTenoo,,,,,
reference_choeen.loge torch FloatTensor, reforease.rejjectdlllog... torch. FloatTenor,,, torch.
# Compute the absolute certainty differences
chosen.ccrrtainty..dfff = torch abs lops
refereenee.choamm.1opp)))
rejjected.cerrainyy.dfff torch. abrch. cordhaabcppalcyy.ejecee....e
refference.rojjectdd..oo..
# Calculate the mean certainty difference
mean_cortainnty_diff (chosm_.crrtnnty.ddfff
rejectee_.erraainy..dfft
Compute the logistic loss losses -o..ogiggo (self betaf.beta" * (policy_chosen,lopp
roiicr_rejecte__ogps)))
Scale the loss by the certainty difference
losses mean cean_certaitty_dff
return cortainty_scaled_ees
Fitness: 7.33125.
Please generate the next one.
40

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please Sections 2-8.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Section under the heading Limitations & Future
Work
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
41

---

Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide details on all our experiments in the main paper and appendix, as well as providing code for reproducing them at https: //t/ sighthu.... com/ samholt/
DiscoPOP.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
42

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide code for reproducing results at https: / thub. com/ hom/samholt/
DiscoPOP.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
sswsssssoooossssssso oneooooooooooooo olyccy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/ mublic/guides SudeeS/mm Sundisubbssssss wwsssssollll.o..oo onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details given in Sections 4, 5, and 6.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide details on standard errors the results tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
43

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details are given in Appendix B.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the ines?
NeurIPS Code of Ethics https: h/tprrip///uui....... cc/
Answer: [Yes]
Justification: We reviewed the NeurIPS Code Ethics and ensured the research conforms to it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss this in Section 8 under the heading Broader Impact and Ethical Considerations.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
44

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All creators are properly credited.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode..ooo..... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
45

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
46


Reviews:
[{"review_id": "X60OJyDCYQ", "reviewer": "Reviewer_ePhP", "summary": "This paper proposes a novel approach to improving LLMs by using an automated system to discover new optimization algorithms. Traditionally, enhancing LLMs has relied heavily on hand-designed loss functions, but this research employs an LLM to iteratively generate and refine these functions itself. This paper introduced DiscoPOP, which blends logistic and exponential losses to outshine existing methods. This algorithm was evaluated across a range of tasks, including multiturn dialogue, sentiment generation, and summarization, where it consistently delivered good results.", "questions": "1. How well does the proposed LLM-driven discovery method generalize to other domains/tasks beyond preference optimization? Are there specific modifications needed to adapt this method to other fields?\n2. The authors mention that DiscoPOP blends logistic and exponential losses. Could you elaborate on the theoretical justification for this choice? How does this combination affect the convergence properties of the algorithm?\n3. Considering the non-convex nature of DiscoPOP, what strategies do you suggest to avoid local minima during optimization?", "limitations": "The authors discussed the limitations of DiscoPOP.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The automatic exploration and evaluation pipeline is promising and it can discover new algorithms without human intervention.\n- By using the above pipeline, the new discovered algorithm in preference optimization achieved SOTA, proving its effectiveness.\n- This can contribute to lots of other area and let machines themselves to discover and evaluate new efficent algorithms.", "weaknesses": "Although this paper mianly focuses on preference optimization task, the dicovery method seems to be easily to adapt to other domains/tasks. A general concern with the paper is its reliance on a complex method that involves detailed tuning of hyperparameters. This complexity could limit the method's broader applicability unless the process can be generalized or adapted effectively to other contexts. However, the paper does not thoroughly address how to establish and optimize the discovery pipeline across different scenarios, which could hinder its practical utility. More guidance on adapting the methodology to a variety of use cases would significantly enhance its value and impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback. We’re glad the reviewer finds the approach and results promising and effective.  \n\n> does proposed LLM-driven discovery method generalize to other domains?\n\nIn our submission, we demonstrated that it works well on CIFAR-10 in the small case study in Figure 2, where it discovers loss functions that outperform standard cross-entropy loss and transfers to other models and hyperparameters. We will adjust the writing to make this clearer. \n\nThere were no significant modifications needed to adapt this method to other fields, the only change needed is to adapt the text in the prompt (Appendix A) to contain information and examples for the new setting.\nWe chose preference optimization, as it is a new exciting area with great potential for novelty, though we agree that adapting our LLM-driven discovery method to other domains/tasks is an exciting future direction of work.\n\n> reliance on a complex method that involves detailed tuning...could limit the method's applicability unless the process can be generalized\n\nWe’re not sure exactly which hyperparameters the reviewer is referring to, but Algorithm 1 does not contain any hyperparmaeters and is very easy to apply to other domains.\n\n>Theoretical justifications for DiscoPOP\n\nTo understand DiscoPOP more from a theoretical perspective, we can closely compare it to the standard DPO loss. At the limits (eg when the relative preferences $\\rho$ are very high or low), it has similar behavior to DPO. However, it mostly differs by its non-convex region, which we analyze in the rebuttal PDF. Previously, we merely hypothesized that the local optimum of the DiscoPOP loss could catch noisy or incorrect data points. We now have empirical evidence for this.\n\nIn short, we identified which data points end up between the local optima after training, and found that 1.35% of points fall there (see Figure 1 in the PDF, where they are clearly visible). Although we use the binary preference labels from the IMDb dataset for training, the dataset also includes a reward score for each completion. When we analyze the data points that are between the local optima, we find that the positive and negative completions are *significantly* closer in absolute reward difference than the data points outside the local optimum (See Table 1 in PDF). This implies that the preference labels on those points are more difficult to distinguish and helps empirically validate our hypothesis. Thanks to the reviewers, we will be adding this analysis to the paper.\n\nWe would like to further emphasize that, in the current offline preference optimization paradigm, training and evaluation differ significantly. In most settings in machine learning, the loss usually corresponds directly with some desired metric, such as the accuracy. However, in our setting,  while we train on an offline set of preference pairs, we ultimately evaluate the model using MT-Bench and GPT-4 as a judge. Thus, it’s not very clear how the loss function used necessarily corresponds to the downstream task. In fact, recent works [1, 2] show that optimizing the DPO loss function too much can lower the quality of the model. Thus, “theoretical” justifications are not as useful in this setting.\n\n> what strategies do you suggest to avoid local minima during optimization?\n\nWe only found the local minima to be a problem when we set $\\beta$ to be very low ($\\leq 0.01$). Note that, while local minima are a problem when the loss function directly corresponds to the task, it’s not immediately clear if that’s the case in offline preference optimization (see our comment on this above).\n\n[1] Feng, Duanyu, et al. \"Towards analyzing and understanding the limitations of dpo: A theoretical perspective.\" arXiv preprint arXiv:2404.04626 (2024).\n\n[2] Chen, Angelica, et al. \"Preference Learning Algorithms Do Not Learn Preference Rankings.\" arXiv preprint arXiv:2405.19534 (2024).\n\n---\n*We hope that most of the reviewer’s concerns have been addressed and, if so, they would consider updating their score. We’d be happy to engage in further discussions.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Fbvk4rXKy", "reviewer": "Reviewer_MbcT", "summary": "The paper proposes an algorithm to discover preference objective functions using LLM for LLM preference optimization. Authors conduct experiments with the discovered objectives on multiple datasets and demonstrate that the discovered objective functions generally perform better than baselines. The authors also show interesting insight into the discovered objective function, LRML.", "questions": "See weakness above.", "limitations": "See weakness above.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces an interesting and new LLM-driven objective discovery algorithm to search for good objective functions using LLMs for the preference optimization of LLMs.\n2. The method successfully finds an objective function that can generally offer a better performance on multiple benchmarks, indicating the effectiveness of the discovery method.\n3. The authors show interesting insights into the best objective function found by the proposed method.", "weaknesses": "1. The objective function is discovered with a different placement of $\\beta$ than the objective function used for evaluation, which causes a misalignment of the discovery and the evaluation process. It would be nice to keep them aligned.\n2. The LLM in the discovery process is a clever generator for searching candidates. It is unclear if the LLM's capability used in the discovery process can change the duration for finding a good objective function or if it will cause the search process to fail.\n3. Similar to the point above, it would be nice to test some non-LLM-based generators of the objective function and perform some traditional searching algorithms such as evolutionary search to demonstrate the effectiveness of the LLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough and in-depth review. We are pleased that the reviewer finds our approach interesting and effective. We address the weaknesses outlined by the reviewer individually below.\n\n> The objective function is discovered with a different placement of $\\beta$ \n\nThank you for highlighting this. We assume that the reviewer is referring to discrepancies between the code for the LLM sampler in Appendix E.6 and the formula presented in Equation (28), as explained in Appendix E. As a result of the reviewer’s comment, we found have now corrected a typo in the LRML formula, which now reads:\n\n$$ f_{lrml}(\\beta\\rho) = (1-\\sigma(\\beta\\rho/\\tau)) \\cdot f_{dpo}(\\beta\\rho) + \\sigma(\\beta\\rho/\\tau) \\cdot f_{exp}(-\\beta\\rho) $$\n\nThe error occurred in the written mathematical notation, and the implementation remains the same.\n\nWe introduce the $\\tau$ parameter largely to retain the original semantic meaning of “beta” used in most papers, which is to adjust the KL-divergence regularisation. Since we use $\\tau = 0.05$ they are equivalent.\n\nWe have now corrected the mathematical notation for DBAQL, AQL, AQFL, and PFL as well.\n\n>  It is unclear if the LLM's capability used in the discovery process can change the duration for finding a good objective function or if it will cause the search process to fail.\n\nThis is a good point. We briefly tried using LLama3 8b-instruct and found that it rarely responded with the correct format and working code, rendering it largely unusable. Should the reviewer request, we can do a more thorough sweep over base models. However, we do not expect this to fundamentally change our results.\n\n> it would be nice to...perform some traditional searching algorithms\n\nUnfortunately, this exact experiment would be very hard to implement, as we would need to design a domain-specific language to search over for the objective function or parameterize it in some way. Parameterizing the objective with a neural network is possible, but would require far too many inner loop training runs to be optimized. However, inspired by the reviewer’s feedback, we’ve designed an experiment that is similar in spirit to validate the effectiveness of the LLM-Driven Discovery method. In particular, we’ve added a version of the CIFAR-10 experiment where we do not return the fitness to the LLM. We show the results in Figure 2 of the attached PDF and confirm that, without the fitness, the LLM is unable to refine its ideas, as it does not have knowledge of which ones worked.\n\n---\n*We hope that most of the reviewer’s concerns have been addressed and, if so, they would consider updating their score. We’d be happy to engage in further discussions.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GukdPB1h2b", "reviewer": "Reviewer_663E", "summary": "The paper proposes DiscoPOP, an algorithm for discovering preference optimization loss functions using Large Language Models (LLMs). The authors propose an LLM-driven objective discovery process by iterative prompting LLMs by previously evaluated performance metrics. Experiments on various benchmarks demonstrate its effectiveness.", "questions": "Refer to the weakness", "limitations": "Refer to the weakness", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This method is innovative. The idea of updating the loss function through automated exploration is both interesting and novel. The process being fully automated and end-to-end makes it particularly neat.\n2. The results are impressive. DiscoPOP demonstrates strong performance across various benchmarks, showing its potential.", "weaknesses": "I think the updating process might be sensitive to the prompts used for proposing new loss functions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their concise feedback. We’re happy that the reviewer finds the paper innovative and the results impressive. We understand the reviewer has concerns around prompt sensitivity. We would like to point the reviewer to a feel ablations we’ve run on this.\n\nIn Appendix D3 of the paper, we show that the discovery process is very robust to different sampling parameters, prompts, and techniques from the LLM. We find little change in performance or behavior when we change the sampling temperature (unless it is set unreasonably high), telling it to “think” first, or sorting the input context.\n\nNotably, in our attached rebuttal PDF we show that providing the fitness in context is important to allow the LLM to refine its proposed ideas.\n\n---\n*We hope that most of the reviewer’s concerns have been addressed and, if so, they would consider updating their score. We’d be happy to engage in further discussions.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hzpMBlAOrb", "reviewer": "Reviewer_gRRA", "summary": "This paper introduces a method of searching for offline RL objectives by using LLMs to generate and refine objectives. They demonstrate that several objectives discovered using this method are able to achieve higher evaluation scores than existing objectives (e.g. DPO) on a variety of benchmark tests. They provide a high level analysis of one discovered objective, hypothesizing that it may possess desirable properties that lead to these performance boosts.", "questions": "1. How often are unit tests for the generated code failed?", "limitations": "The limitations described cover most points, including the difficulties and uncertainties associated with using LLMs to improve upon objectives in a rigorous fashion.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written, organized, and easy to understand. While the use of LLMs to modify prompts and self-improve in other ways has been studied, using them to improve objective functions is a novel idea that the authors demonstrate holds potential. The authors demonstrate that the objectives discovered can be useful, showing improvements on a variety of generation and classification tasks.", "weaknesses": "The main weakness of this paper in my opinion is that it isn't clear what results the paper is trying to present. While this is partially an issue of organization, it is also an issue with the amount of evidence given to the claims in the paper. If these was made clearer, I would be inclined to raise my score. I detail below:\n\n1. The paper starts by introducing a new method for discovering new optimization objectives using LLMs, then transitions quite abruptly to describing the properties of DiscoPop. While both could be valuable contributions, I don't feel that either is given quite the attention required. It would help to clarify what the main focus of the paper is. If DiscoPOP is the focus, then claims about its properties should be better studied. If the discovery method is the focus, more emphasis should be placed on the method's performance and ability to produce novel and useful objectives.\n\n2. The majority of insights about DiscoPOP are presented as hypotheses or intuitive insights (e.g. the non-convex portion, and how the model behaves at $\\pm\\inf$. This is not enough to support the claim made in the conclusion that they provide insights into what properties an optimal objective function should possess. While it \\emph{may} be enough to say the provide intuition into why DiscoPOP is successful, further experimentation would be necessary to make optimality claims like this.\n\n3. Some portions of the discovery method are vague as well. Though the authors do not observe the models simply regurgitating old objectives and cite a pattern of objective refinement for supervised learning, this is of course not guaranteed, and not well studied. Further experiments or details on the how often they observe refinement vs regurgitation would help to support the utility of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and extremely helpful review. We’re glad that the reviewer finds our approach to be both novel and useful, though we agree that more analysis would improve the paper significantly. \n\n> The majority of insights about DiscoPOP are presented as hypotheses or intuitive insights...further experimentation would be necessary\n\nThis is a great point and we’re grateful to the reviewer for bringing it up. We’ve performed further analysis that *considerably* improves the paper and our understanding of the loss function. Previously, we merely hypothesized that the local optimum of the DiscoPOP loss could catch noisy or incorrect data points. We now have empirical evidence for this and include the results in the rebuttal PDF.\n\nIn short, we identified which data points end up between the local optima after training, and found that 1.35% of points fall there (see Figure 1 in the PDF, where they are clearly visible). Although we use the binary preference labels from the IMDb dataset for training, the dataset also includes a reward score for each completion. When we analyze the data points that are between the local optima, we find that the positive and negative completions are *significantly* closer in absolute reward difference than the data points outside the local optimum (See Table 1 in PDF). This implies that the preference labels on those points are more difficult to distinguish and helps empirically validate our hypothesis. Thanks to the reviewers, we will be adding this analysis to the paper.\n\n>Further experiments or details on...refinement vs regurgitation\n\nThis was another great suggestion. We’ve included an ablation on the CIFAR-10 results in which we *don’t* provide fitness feedback. Thus, the model is unable to perform refinement (as it does not know what performs well) and can only regurgitate ideas. This is a key baseline to compare to to validate our discovery method.\n\nIn the attached rebuttal PDF, you can see that, without the fitness score, the LLM is unable to refine its ideas and can only regurgitate, thus leading to fewer steps of improvement.\n\n> On LLM Discovery vs. DiscoPOP as the main focus\n\nOur main focus was on DiscoPOP, as that is where the majority of our resources and efforts were focused. We will adjust the writing to make this clearer.\n\nWe believe both parts are significant contributions of our paper, though we understand both could use more analysis. Thanks to the reviewer’s feedback, we’ve managed to significantly strengthen evidence for both.\n\n> what properties an optimal objective function should possess\n\nThanks for bringing this up. This was imprecise language on our end.  We agree, and have now removed any claims about what an \"optimal\" objective should possess, instead replacing the word optimal with the word \"good\" in the conclusion (line 277 and in line 110).\n\n> How often are unit tests for the generated code failed?\n\nFor GPT-4 the unit tests failed ~5% of the time; however, the model usually fixes it upon the feedback. We briefly experimented with weaker models and found the failure rates to be significantly higher.\n\n---\n*We hope that most of the reviewer’s concerns have been addressed and, if so, they would consider updating their score. We’d be happy to engage in further discussions.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful to the reviewers for their insightful feedback. There is broad consensus amongst the reviewers that our approach is novel and effective.


$\color{red} R1$ (ePhP): “the new discovered algorithm in preference optimization achieved SOTA, proving its effectiveness.”

$\color{green} R2$ (MbcT): “The method successfully finds an objective function that...offer a better performance...indicating the effectiveness of the discovery method.”

$\color{blue} R3$ (663E): “This method is innovative...The results are impressive”

$\color{magenta} R4$ (gRRA): “using [LLM’s] to improve objective functions is a novel idea that the authors demonstrate holds potential”

Reviewers understandably had concerns about the lack of in-depth analysis of the DiscoPOP loss ($\color{red} R1$, $\color{magenta} R4$) function and its theoretical implications ($\color{green} R2$, $\color{magenta} R4$).

Reviewers also had concerns about the effectiveness ($\color{green} R2$, $\color{magenta} R4$) of LLM-driven discovery and its sensitivity ($\color{blue} R3$) to the prompt.

We address these concerns below.

## Deeper Analysis of DiscoPOP

We would like to deeply thank the reviewers for bringing this up. We’ve performed further analysis that *considerably* improves the paper and our understanding of the loss function. Previously, we merely hypothesized that the local optimum of the DiscoPOP loss could catch noisy or incorrect data points. We now have some empirical evidence for this.

In short, we identified which data points end up between the local optima after training, and found that 1.35% of points fall there (see Figure 1 in the PDF, where they are clearly visible). Although we use the binary preference labels from the IMDb dataset for training, the dataset also includes a reward score for each completion. When we analyze the data points that are between the local optima, we find that the positive and negative completions are *significantly* closer in absolute reward difference than the data points outside the local optimum (See Table 1 in PDF). This implies that the preference labels on those points are more difficult to distinguish and helps empirically validate our hypothesis. Thanks to the reviewers, we will be adding this analysis to the paper.

### Misc:

Thanks to a comment from $\color{green} R2$, we discovered a typo in equations (4) and (5) for DiscoPOP. We have since fixed this.

## More Analysis of LLM-Driven Discovery

We’ve added a key baseline experiment to our results to validate the effectiveness of the LLM-Driven Discovery method. In particular, we’ve added a version of the CIFAR-10 experiment where we do not return the fitness to the LLM. We show the results in Figure 2 of the attached PDF and confirm that, without the fitness, the LLM is unable to refine its ideas, as it does not have knowledge of which ones worked.

Meta-review: The paper proposes a novel method that uses LLM to discover preference optimization algorithms. This falls into the metalearning category. The reviewers found the performance to be effective and the experiments to be comprehensive. During the rebuttal, the authors successfully addressed the reviewers' questions and concerns such as hyperparameters and the comprehensiveness of the experiments. I think it would be nice to provide the readers a more comprehensive related work section for the metalearning papers that were published before 2020.
Decision: Accept (poster)