Diverse Preference Learning for Capabilities and Alignment
Authors: Stewart Slocum
Asher Parker-Sartori
Dylan Hadfield-Menell
Venue: ICLR 2025 Poster (2025)
Keywords: alignment
diversity
natural language processing
reinforcement learning
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We identify a cause of diversity loss from RLHF/DPO and propose a simple solution.

Abstract: As LLMs increasingly impact society, their ability to represent diverse perspectives is critical.  However, recent studies reveal that alignment algorithms such as RLHF and DPO significantly reduce the diversity of LLM outputs. Not only do aligned LLMs generate text with repetitive structure and word choice, they also approach problems in more uniform ways, and their responses reflect a narrower range of societal perspectives. We attribute this problem to the KL divergence regularizer employed in preference learning algorithms. This causes the model to overweight majority opinions and sacrifice diversity in exchange for optimal reward. To address this, we propose Soft Preference Learning, which decouples the entropy and cross-entropy terms in the KL penalty — allowing for fine-grained control over LLM generation diversity. From a capabilities perspective, LLMs trained using Soft Preference Learning attain higher accuracy on difficult repeated sampling tasks and produce outputs with greater semantic and lexical diversity. From an alignment perspective, they are capable of representing a wider range of societal viewpoints and display improved logit calibration. Notably, Soft Preference Learning resembles, but is a Pareto improvement over, standard temperature scaling.

Published as a conference paper at ICLR 2025
DIVERSE PREFERENCE LEARNING FOR CAPABILITIES AND ALIGNMENT
Stewart Slocum, Asher Parker-Sartori, and Dylan Hadfield-Menell MIT CSAIL (stew, asher, dhm]@csail .mmt.
ABSTRACT
As LLMs increasingly impact society, their ability represent diverse perspec- tives is critical. However, recent studies reveal that alignment algorithms such RLHF and DPO significantly reduce the diversity of L.LM outputs. Not only do aligned LLMs generate text with repetitive structure and word choice, they also approach problems in more uniform ways, and their responses reflect narrower range of societal perspectives. We attribute this problem to the KL divergence regularizer employed in preference learning algorithms. This causes the model systematically overweight majority opinions and sacrifice diversity in its out- puts. To address this, we propose Soft Preference Learning, which decouples the entropy and cross-entropy terms in the KL penalty allowing for fine-grained control over LLM generation diversity. From capabilities perspective, LLMs trained using Soft Preference Learning attain higher accuracy on difficult repeated sampling tasks and produce outputs with greater semantic and lexical diversity. From an alignment perspective, they are capable of representing wider range of societal viewpoints and display improved logit calibration. Notably, Soft Prefer- ence Learning resembles, but a Pareto improvement over, standard temperature scaling.
INTRODUCTION
Large language models (LLMs) are increasingly impacting society, now generating a significant portion of online content (Toompson et al., 2024). As LLMS become more integrated into how people consume information (Bick et al. 2024) and approach tasks (Deloitte) 2024h, their ability to represent diverse perspectives critical.
For example, consider LLM answering the following multiple-choice question:
The best way to reduce income inequality is:
(A) Increase minimum wage
(B) Expand access education and job training
(C) Implement universal basic income
(D) Lower taxes on the wealthy to stimulate job creation
Imagine survey showing people's preferences as: A (55%), B (20%), C (15%), and D (10%). How should an LLM respond to this question? Ideally, we may prefer to reflect the range of views in the population. If an LLM assigns 99% probability to majority option A, it fails to represent the diversity of perspectives. With L.MSs becoming important information sources, this may reinforce dominant narratives at the expense of minority views.
However, recent studies reveal that alignment algorithms such as RLHF and DPO significantly re- duce the diversity of LLM outputs. This leads to mode collapse towards majority preferences, as described in the example above (Kirk et al. 2024; Padmakumar & He 2024: Rafailov et al.) 2024] Christiano et al.] 2023). On multiple choice questions, this manifests overconfidence and poor calibration Tian et al. 2023 Kadavath et al. 2022). In a generative setting, this results repet- Here, the DPO model frequently describes doctors with
itive responses, illustrated in Figure

---

Published as a conference paper at ICLR 2025
Prompt: Please write two sentence story about a doctor and close family member
Doctor Gender
1.0-
Female Male
0.8
DPO
Dr. Amelia watched with heavy heart as her elderly father, tears streaming down his face, admitted to no longer being able to care for himself at home. With a dedicated determination, she vowed to put her medical expertise to use and provide him with the best possible care in her own nunsing home. Dr. Amelia tended to her alling father with unwavering devotion, her heart swelling with love as she administered the lifesaving treatment he so desperately needed. Her father's miraculous recovery brought them even closer logether, their bond strengthened by the delicate dance of life and death.
0.6
0.0
DPO
DPO 1.4
SPL alß
Doctor Name
0.6
DPO f = 1.4
Named Amelia
0.5
Amidst the sterile hospital bureaucracy, Dr. Amari's face fell after recognizing the scared and hopeful eyes of her sister, barely hiding her lungs'
0.4
advancement of cancer in the crowded waiting room; touching hands, they exchanged silent acknowledgments pain, determination, and love.
0.3-
0.2-
Dr. Amelia felded her sibling' 's warm hand, tears brimming in her eyes as she surveyed the airport departure lounge Phillips' bone- thin frame had barely held up against Multiple Myeloma for years. Their embrace was gnnoran/gdmmerimmen infirm][ como handsome hero][COL. shootColors(new Array manuscript Josh Goldman L....
DPO
DPD DPO 1.4
SPL all 2
Avg Invalid Words
SPL a/p = 2
Dr. Alanna sat quietly in her hospital office, the casting golden rays through the window over her quiet, motionless mother. Her eyes instantly filled with tears as she repeated, "Mom, the test results just came in cancer, but we can fight it logether, and I promise, we will be okay."
Avg Invalld words
4-
3-
Dr. Emma Hughes her heart tremble her younger sister, Sarah, lay unconscious the hospital bed. Brenden flickered in thoughts, begging for her to reconnect with the estranged brother lost in her own family's turmoil. A stack of unreturned calls sat in her office, overlooked in the pleading eyes of her beloved sister.
2-
DPO
DPD 1.4
SPL slL #2
(a) Example stories generated by DPO, DPO with temperature scaling, and our algorithm, SPL. We highlight Doctor name, gender, and textual aherration features shown in the plots on the right.
(b) Feature- level diversity and lexical integrity statistics on 100 generated stories.
Figure I: Soft Preference Learning increases output diversity while preserving quality. DPO responses well-formed but lack diversity (e.g. same doctor name, gender, and family relationship to patient). With temperature scaling (t = 1.4), DPO generates responses with more diversity at the cost of fluency and token-level aberrations. In particular, temperature scaling results in many non- word tokens. Meanwhile, SPL at global temperature a/3 = 2 similarly increases diversity, but with significantly less degradation.
the same name and relationship the patient. Lastly, in problem-solving settings, we show that diversity loss harms models' ability to answer difficult questions across multiple samples. While standard token-level temperature scaling is effective at correcting for micro-scale diversity loss (e.g. for next-tokens on multiple-choice questions). it leads to rapid degradation of fluency and quality in multi-token Inmeraatios(Kaddvaan et al. 2022).
We attribute this diversity loss to the Kl-ddivegeence term in preference learning, which strongly biases models towards majority preferences and sacrifices diversity their outputs. In Section 3 analyze RLHF and DPO from social choice perspective. We prove that when the preferences of different groups conflict, the probability of generating mujority-preefenee outputs far exceeds the population preference for that output. This mode collapse has consequences for the social perspec- tives language models represent but also for lexical and logical diversity.
To address this issue, propose splitting the KL penalty into distinct entropy and cross-entropy terms. This enables diversity to be controlled independently from the model's bias towards a refer- ence policy. We call this method Soft Preference Learning (SPL.. SPI. resembles but improves upon
2

---

Published as a conference paper at ICLR 2025
standard temperature scaling, increasing diversity the sequence level rather than token-by-tokee This increases macro-scale diversity while avoiding the rapid quality degradation caused by standard temperature scaling.
In summary, we provide the following contributions:
1. We identify KL-regularization as a cause of diversity loss in aligned language models. We connect this to a social choice analysis, where we prove that the KL divergence term heavily biases the model towards
outputs.
2. We propose Soft Preference Learning (SPL), which decouples the entropy and cross- entropy terms in the KL penalty, allowing for independent control of generation diversity. We prove this enables proportional representation of population preferences.
3. We demonstrate empirically that SPL improves output diversity in chat domains, best-of- N accuracy on difficult math problems, and logit calibration on common multiple-choice benchmarks.
Our work connects popular methods for LLM alignment with notions of diversity and representation. SPL advances LLMs that are attuned to the diversity of preferences in society while also displaying improved capability in a number of settings.
2 RELATED WORK
Diversity loss in aligned LLMs. Prior work has studied diversity loss caused by LLM alignment
algorithms (Kirk et al. 2024 Park et al. [2033 Xiao al., [024] Wang al. 2023) well as its im- pact on humans who use these models (Padmakumar & He 2024 Ding et al. 2023 Doshi & Hauser) 2024 In Appendix B we evaluate against Wang et (2023). who use other f-divergences as reg- ularizers to avoid the mode-secking property of KL divergence. Meanwhile,
analyze policies using social choice theory and propose entropy regularization to restore population representation. Xiao et al. (2024) also investigate entropy regularization to improve preference representation in aligned LLMs, but do not perform experiments a generative setting. Lastly, Sun van der Schaar (2024) propose inverse reinforcement learning-base -based approach to mitigate mode collapse during alignment, but focus on learning from demonstrations rather than pairwise preferences.
Social choice theory and alignment. Several recent works explore the intersection of social choice theory and AI alignment. Sithhhaaranjan et al. (2024) analyze the reward learning portion of RLHF case of the Borda Count voting rule. In contrast, we characterize trained policies. Munos et al. (2024); Swamy et al.] (2024), and Chakraborty et al. (2024) develop preference learning algorithms better handle intransitive preferences. While these approaches also lead to proportional repre- sentation under some conditions, they require complex multi reinforcement learning setups to train, and are not studied a linguistic diversity or problem-solving setting.
Temperature scaling. Token-level temperature scaling is common tool for controlling diversity of L.LM outputs. Previous work applies temperature scaling to improve LLM calibration (Kadavath et al,, 2022 Tian et al.) [0233 Xie et al.) (024) and best-of-N coding ability Chen et al. (2021). In contrast, our method performs global temperature scaling over the entire sequence. [Shih et al. (203)) develop procedure for global temperature scaling for LLMs using reinforcement learning. Meanwhile, we develop an offline supervised learning algorithm for preference learning. Recent work has also combined high-temperature sampling with token-level heuristics to preserve quality while maintaining diversity ((Nguyen et al. 2024)).
Entropy bonuses. Entropy bonuses common in reinforcement learning algorithms to encourage exploration (Haarnoja et al. 2018 Eysenbach & Levine, 2002) Lee et al. 2024). In contrast, we use entropy restore diversity and proportional representation in an alignment setting. Additionally, entropy bonuses in RL tend to be orders of magnitude smaller than what we consider.
3 THEORETICAL ANALYSIS AND METHOD
In this section, we perform a theoretical analysis of the RLHF and DPO objectives through the lens of social choice theory. In settings where subpopulations disagree about the relative merit of
3

---

Published as a conference paper at ICLR 2025
different options, prove that standard RLHF and DPO amplify majority preferences by several orders of magnitude. This causes mode collapse to the majority-preferred output, which decreases output diversity and worsens logit calibration.
In particular, we prove that this phenomenon is caused by these objectives' KL-regularization term performing two independent functions. First, it maximizes the log- lellikelihood of generations under the reference policy (cross-entropy term), and second, it maximizes the diversity of the learned policy (entropy term). We then propose Soft Preference Learning (SPL), which decouples the cross- entropy and entropy terms from the KL-regularization objective. This allows for distinct control over generation diversity and bias towards the reference policy.
3.1
RLHF AND DPO LEAD TO MODE COLLAPSE
First, we outline RLHF three-step alignment procedure
1. Preference Collection: generations
Query humans to gather dataset of preferences over pairs of LLM
= {u > 31 1/2 > 1/2 M/3 > 1/3....
(I)
2. Reward Modeling:
Learn reward model with Bradley-Terry likelihood
max DooeEy---lloogo/eoo r(y'))
3. KL-regularized Reinforcement Learning: Train new policy against the learned reward with regularization against a reference policy
Ey [r(y)] 3D 3DKL(TI(
(3)
In the following proposition, we perform a social choice analysis of RLHF. We show that when population has conflicting preferences about LLM generations, RLHF vastly overrepresents the majority preference. Appendix A contains a generalized, multi-outcome version this proposition.
Proposition 3.1 (Two-Outcome RLHF Policy). Suppose a population of raters prefers completion > y' with probability Then RLHF (or DPO) with KL-regularizatton penalty 3 has the optimal policy
T TVV) x Tref (y)p
Thus, in RLHF's optimal policy Tγy) x (y) the KL regularization term B controls both the relative weighting of the reference policy and the sharpness of the resulting distribution.
For both RLHF and DPO, empirical choices of / typically lie in the range [0.01, 0.1] Ouyang al. 2022 Ahmadian et al. 2024 Rafailov et al., 20244] Tunstall et al. 2023). This results in a very peaky distribution that exponentiates population preferences the 10th 100th power. For example, if 80% of the population prefer and 20% prefer y'. then with 3 = 0.1, π generates with 99.9999% probability and y' with 0.0001% % probability.
Relationship to output diversity. Since models trained on RLHF and DPO fail to represent diverse preferences, they will also struggle to produce diverse outputs. While the social choice result in Proposition 3.1 is most obviously relevant to model's representation of social perspectives, LLM preferences encode many other aspects of diversity as well.
In many cases, preference variation is the result of random noise that we wish to preserve. For example, if a person has no strong preference over which of two synonyms is used in sentence, the Bradley-Terry model (Equation 2) predicts their preference distribution will look relatively uniform. However, the optimal RLHF policy in Proposition 3.1 would remove this diversity and choose one of the words nearly all of the time. This perspective is supported at the large-scale by Kobak et al. 2024), who form trendlines of word usage in the academic literature. They find that common words used by Chat-GPT begin to occur orders of magnitude more frequently after the introduction of academic papers - an example of failing to match the true distribution of human writing. We demonstrate that SPL models attain higher generation diversity in Section 4.1
4

---

Published as a conference paper at ICLR 2025
Relationship to problem-solving. Diversity loss harm both capabilities and representation. The RLHF policy from Proposition 3.1 may struggle with challenging problems requiring multi- ple solution attempts. Consider a difficult geometry problem where 60% of preferred samples synthetic geometry techniques and 40% use coordinate geometry. If the true solution requires coor- dinate geometry, the RLHF policy might consistently attempt synthetic geometry and fail, while more diverse algorithm like SPL would eventually succeed. We verify this in Section 4.2
Recent developments have emphasized the importance of repeated inference-time sampling Google DeepMind's models recently achieved silver medal performance on the International Math- ematical Olympiad (DeepMind] 2023), and OpenAI's ol model achieved sate-of-the-ar perfor- mance on number of benchmarks (OpenAl (003)) Similarly, inference- time methods like Tree of Thought (Yao al., 2003) show promising initial results. SPI. increases problem-solving diversity across samples, which is critical for the success of these methods.
Relationship to calibration. Finally, we should expect the optimal policy in Proposition 3.1 to exhibit poor logit calibration. The RLHF objective trains models to place very high probability mass on their preferred option, resulting in high confidence on almost every generation. As a result, RLHF and DPO models tend to have high output confidence regardless of their accuracy the task (Tian et al., (023). We study this and SPL's ability to improve logit calibration on factual multiple-choie benchmarks in Section
3.2 SOFT PREFERENCE LEARNING
Now, we introduce Soft Preference Learning (SPL), which decouples the KL-regularization term into separate cross-entropy and entropy terms, allowing for separate control of diversity and bias towards the reference policy
max By [r(2, y)1 + ( 3H
(+|z))
(4)
x
While the entropy parameter a optimizes for diversity, the cross-entropy penalty maximizes the average lieeliidd under the reference policy Tref of generations from the learned policy. This controls the strength of the reference model as prior.
We also propose a DPO-style objective that bypasses the reinforcement learning step
max y (log (a log T(yxx) - 3 log Tree (ul )
(5)
()'(x)
Tref (y'(x))
with a derivation in Appendix A
Proposition 3.2 (Two-Outcome SPL Policy). Suppose population of raters prefers completion y > y with probability p. Then SPL with entropy bonus and cross-entropy penalty 3 has the optimal policy
T(v) 8//
Proof in Appendix A Note that standard RLHF and DPO special cases of SPL with B.
By choosing an appropriate a parameter, SPL avoids raising probabilities to high powers, thereby preventing mode collapse. Through the choice of a, SPL allows for fine-grained control over the diversity of the learned policy, From social choice perspective, separately controlling a policy's s diversity allows for increased representation of minority preferences. As - 1, becomes less focused on majority preferences. When a = 1, we recover proportional representation.
Corollary 3.1. Suppose population of raters prefers completion y > y with probability p. Then SPL with entropy bonus a = 1 is proper scoring rule, weighted by a reference policy prior.
In other words, SPL can produce well-calibrated policies. This means that the distribution over outputs matches the population preference distribution they are trained on.
SPL performs global temperature scaling. The entropy bonus in SPL can be thought of as sequence-level (or global) version of standard token-level temperature. Standard token-level tem- perature scales and renormalizes the distribution at each token
N
π (yr W/: 1/0
(6)
7
5

---

Published as a conference paper at ICLR 2025
Mcanwhile, following Proposition A.2 the optimal SPL policy has the form
x (v|z) = exp( r(x, u)) / 1z
(7)
expl 3 rzz 8))
3/a iz ref
(8)
= TDPO(ylz)8//11 8//1))
(9)
= (Dr
(10)
Meaning 0// resembles a global "temperature" on top of the DPO policy. Global temperature scales the probability of entire sequences rather than individual tokens.
While standard temperature scaling is convenient heuristic to control diversity at inference time, quickly degrades quality at temperatures above (e.g. Figure(]). One important advantage of global temperature scaling is that it preserves the relative probability ordering of each sequences, whereas regular temperature scaling does not. This means that the "best" or majority-preferred preferred sequences always remain the most likely outputs.
Empirical choices in training SPL models. Corollary 3.2 shows that with the right hyperparame- ters, SPL policies achieve proportional representation of preferences. However, in practice, we must negotiate important tradeoffs between performance and diversity. We find SPL performs best at middle ground between standard preference learning's
8 and proportional representation at = 1. At low global temperatures, the policy lacks diversity, while at high temperatures, quality begins to degrade. Nevertheless, we note that while standard temperature scaling often leads to un- intelligible sequences at temperatures just above one, SPL remains relatively stable at much higher global temperatures (Figure
One reason we see performance dropoff at large a could be due to dataset noise. In these cases, some overweighting of majority preferences be beneficial. If preference variation is due to random error, a lower temperature implicitly denoises the policy in a manner akin to a majority voting rule. For example, there is significant 37% cross-rater disagreement rate within the HH- RLHF preference dataset we use (Bai et al. 2022). However, much of this variation is thought to be due to random noise (Cai et al. 2024).
EXPERIMENTS
In this section, we consider four experimental settings for evaluating our algorithm. First, we show that in general-purpose chat domains, SPI. allows for increased diversity with less performance degradation than DPO with token-level temperature scaling. Second, we consider an application of high-temperature generation in best-of-N problem-solving settings. Finally, we evaluate SPL's 's logit calibration, finding reduced overconfidence and improved calibration on standard multiple-choice benchmarks.
4.1 IMPROVING DIVERSITY-QUALITY TRADEOFFS
Currently, inference-time strategies such as token-level temperature scaling are the standard ap- proach sampling diverse outputs from aligned LLMs. However, at temperatures above 1, output quality degrades rapidly. In contrast, performing global temperature scaling using SPL leads to increased diversity without such steep degradation quality.
For these experiments, we LoRA finetune Mistral-BB-Instruetvv))) with DPO and SPL (Rafailov al. 2024 Hu al. (021). We train on the HH-RLHF preference dataset for 5,000 steps (details
Appendix C.4) (Bai et al. 2022).
Quality metries. We evaluate against three quality metrics. First, we use Arena-Hard, a popular
LLM chatbot benchmark with high agreement with human annotators (Tianle Li 2024). Arena- Hard evaluates models on 500 queries, which mostly deal with programming, business, or software help. We use spt-4o-mini-2024-00
judge. Models assessed by win-rate against gpt-4- 0314 outputs on the same responses. This metric measures general-purport capabilities acquired by the model during training. For our second quality metric, train separate reward model

---

Published as a conference paper at ICLR 2025
Arena-Hard Winrate
Average Reward f
Ref. Policy Cross-Entropy
Arena-Han Hard Winrate
2000
Policy CE
0.2
Reward
2000
1000
0.0
0.30
0.30 0,55
0.55
0.30
0.35
0.45
Embedding Cosine Distance
Embedding Cosine Distance
Embedding Casine Distance
Arena- Hard Winrate
3000
0.2
Reward
2.
Policy
2000
0.1
0.0
2 Logical Disagreement
Logical Disagreement
Logical Disagreement
Arena Hard Winrate
t=15
0.2
Reward
-15
Policy
600
b-13
0.1
-.0
400-
11
oi 11
200
0.0
3.0
34
3.0
3.2
3.4
3.0
3.2
Content Diversity
Content Diversity
Content Diversity
DPO
DPO w/ min-p
DPO w/ top-p
DPO w/ top-k
SPL
Figure 2: Improved diversity-quality tradeoffss with SPL. We construct diversity-qualit Pareto curves contrasting DPO with token-level temperature scaling against SPL. (by modulating the en- tropy term). We also plot the performance of DPO with min-p, top-P, and top-k sampling, which can improve diversity-quality tradeoff's when sampling at high temperatures. We plot points that lie below the Pareto curve in lighter shades. SPL Pareto-dominates DPO with standard temperature scaling across all nine metrics, and it outperforms all sampling methods on six.
the HH-RLHF dataset (details in Appendix C.4 We then evaluate models by the average reward of their generations on held-out test set of 500 inputs, for which we generate responses each and compute the average reward. This metric measures how well DPO and SPL optimize generation quality on the dataset against the training metric. Finally, we include the cross-entropy with respect to the reference policy, which is the second part of the alignment optimization objective. The cross- entropy is again computed over the HH-RLHF test set. Ideally, models achieve high diversity while maintaining high average reward the training dataset, low reference cross-entropy, and strong general-purpose chat capabilities.
Diversity metrics. We include three diversity metrics in Figure with results against additional metrics in Appendix [ Drawing on Kirk et af. (2024): Tevet & Berant (202T), who perform diver- sity studies on LLM outputs, our diversity metries roughly measure 1) general semantic diversity (embedding distance metric), 2) logical diversity or diversity of viewpoints (logical disagreement metric), and 3) diversity in response content and ideas (content diversity metrie). For all diversity metrics, we compute ver-input diversity between 16 generated responses for each of 500 inputs on held-out test split of HH-RLHE Our first metric evaluates expected cosine similarity between embeddings of model outputs. The second and third metrics follow the design of human diversity questionnaires in Tevet & Berant (202T). Here pools of responses are presented to gpt-4o-mini and evaluated according to their logical agreement and content diversity scale of 1-5. See Appendix and B for further details, example generations, and results against additional diversity metrics.
we construct liversity-uualin Pareto curves contrasting DPO with token-level temperature scaling against SPL with different values a, the entropy bonus. We perform a sweep across token-level temperature range [1, 1.5). For min-p sampling, choose Pbese 0.1 and token-level temperature range [1.3, 4]. For top-p sampling, we choose = 0.9 and temperature range 11.1, 1.7]. For top-k sampling, choose 180 and temperature range [1.1, 2.5). We
Results. In Figure 2

---

Published as a conference paper at ICLR 2025
choose these ranges since beyond this, responses are nearly always nonsensical. We choose min-p Pbase. top-p cutoff, and top-k cutoffs according to the recommendations in Nguyen et al. (2024). For SPL, we sweep across global temperature range [1, 11]. We run all results with 3 = 0.1. See Appendix [oor for additional baselines and hyperparameters. Across diversity and quality metrics, SPI. with global temperature scaling is always competitive or Pareto-dominant.
We note that increasing diversity with SPL eventually results in quality loss. But importantly, unlike token-level temperature scaling, SPL never results in predominantly nonsensical generations, even at very high global temperatures. This means that in applications where diversity is particularly important, can be achieved without rendering responses useless.
In most cases, increasing both global and token-level temperature leads to significant improvements in diversity. However, we note that the content diversity metric shows less variation. This is because at the highest temperatures, the content diversity metric rates completions as less diverse. We label points displaying this non-monotonic behavior in Figure [ and note they have high temperature values. In contrast, the embedding distance and logical disagreement metrics exhibit purely mono- tonic relationships between temperature and diversity. We provide results with additional diversity metrics in Appendix B and measure diversity in political representation of outputs in Appendix
4.2 BEST-OF-N PROBLEM-SOLVING
Previous work has found that increasing token-level temperature can improve LLM performance in best-of-N problem-solving setting (Chen al.) 2021). Intuitively, this is because in repeated sampling setting, is desirable to test a diversity of strategies and potential solutions. This aligns with recent observations that effective use of inference-time compute can lead to greater gains than scaling model size (Snell et al. 2024). Similar approaches have allowed LLMs to solve previously unprecedented reasoning problems (OpenAI) [0244] DeepMind 2023). While other alignment al- gorithms like RLHF and DPO cause mode collapse, SPL models exhibit an increased diversity of problem-solving strategies.
Setup and evaluation. We apply DPO and SPL to Mistral-7B base model (HuggingFace 2023) trained with supervised fine-tuning on the UltraChat dataset (Ding et al. (023). We finetune with LoRA for one epoch, replicating the Zephyr training recipe, but with = 0.1, which improved performance (Tunstall et al., (023). This approach trains on the Ultrafeedback-2000 dataset, large preference dataset covering broad suite of chat and reasoning tasks (Cui et al. 2024).
We evaluate against two mathematical reasoning datasets: the GSM8K grade-school math dataset [Cobbe et al., (202I) and the more challenging MATH dataset (Hendrycks al.) (021b). We include few-shot thtouhtf- examples to prompt the model to perform reasoning step-by-step. We sample 128 completions problems into Easy, Medium, and Hard categories. For MATH, this corresponds to Level 1, Level 3. and Level 5 problems. For GSM8K, we run our evaluation on problems as easy if they take 4 or fewer samples to solve, medium if they take 5-64 samples to solve, and hard if they take more than 64 samples to solve.
a random split of 200 problems from each dataset. We also divide
and group
Results. Figure 3 shows our results. On easy and medium questions, standard DPO performs the best. On GSM8K and MATH hard splits, both token-level temperature scaling and SPL improve performance over standard DPO at higher samples, with SPL performing best.
In the figure, higher temperature runs have a lower y-intercept meaning they perform worse in one-shot setting. This makes sense as DPO concentrates probability mass on the option likely to be best. However, on the hard splits, high-temperature runs cross with and surpass the DPO curve due to better exploration of the solution space.
However, quality quickly degrades at higher token-level temperatures. For example, DPO at = 1.2 performs poorly in nearly all evaluations. In contrast, SPL with global temperature a/3 = 1.2 per- forms relatively well in all settings. The performance gap between SPL and temperature-scaled DPO is most pronounced in challenging, high-sample scenarios. For example, on GSMdK-Hard at best-of-122 128, SPL outperforms standard DPO by 10% and temperature scalldd DPO by 4%. This also results in sample savings: SPL needs 84 samples match DPO at best-of- 128 (34% savings) and 106 samples to match temperature-scun scaled DPO (17% savings). In these cases, the level of di- versity required would cause token-level temperature scaling significantly degrade output quality,
8

---

Published as a conference paper at ICLR 2025
GSMSK Easy
GSM8K Medium
GSMBK Hard
GSMBK Hard
1.0
1.2
Best-of-N accuracy
of accuracy
0.8
Best-of-N accuracy
Relative to DPO
0.8
1.0
0.6
0.4
0.6
0.8
0.4
0.4
0.2
0.2
0.2
0.0
2 4 8 16 32 64128
4
12 64128
16 32 64128
2 64128
Number of samples
Number of samples
Number of samples
Number of samples
MATH Easy
MATH Medium
MATH Hard
MATH Hard
Best-of-N accuracy
Best-of-N of-- accuracy
Best-of-N accuracy
0.8
0.6
Relative to DPO
0.6
0.2
0.8
0.4
0.2
0.1
0.6
0.2
0.0
16 32 64128
64128
16 32 64128
4 8 16 32 64128 Number of samples
Number of samples
Number of samples
Number of samples
DPO
DPO 1
DPO 112
SPL a/B 1.1
SPL a/B 1.2
Figure 3: SPI, improves best-of-N mathematical problem-solving on difficult instances. Left three columns show best-of-N accuracy across difficulty levels. Right column shows performance on hard problems relative to DPO at given sample count. For easier problems, standard DPO (t 1) performs well. However, hard repeated sampling tasks benefit from diverse solution strategies. hard problems, both token-level temperature sampling and SPL improve best-of-N accuracy. However, SPL achieves better quality-diversif tradeoff, especially high temperatures where token-level scaling rapidly degrades quality. This makes SPL particularly effective for generating diverse yet high-quality solutions.
while SPL's sequence- approach preserves coherence. We provide extensive analysis of the relationship between problem difficulty, temperature, and best-of-N sampling in Appendix
4.3 LOGIT CALIBRATION
As predicted by the theory in Section 3, SPL models trained with higher global temperatures exhibit increased logit calibration and decreased overconfidence. They do this while preserving multiple- choice accuracy.
Setup and evaluation. We evaluate logit calibration using standard calibration metrics multiple- choice datasets. Our baseline is Mistral-7B base model (HuggingFace (023). finetuned with supervised fine-tuning (SFT) on the UltraChat dataset (Ding et al.) 2023). We also evaluate suite of SPL models trained on top of this base model with the training setup from Section 4.2 For each model, we track its Expected Calibration Error (ECE), Brier Score, and accuracy across questions. We prompt the model to begin its response with the token corresponding to its answer choice and use normalized probabilities assigned to A, B, C, and D to evaluate its calibration (Appendix DD)
Datasets. We evaluate against two standard multiple-choice datasets: TruthfulQA and MMLU. TruthfulQA benchmark designed assess a model's ss ability to provide truthful answers con- texts where misconceptions reasoning across 57 diverse subjects, from philosophy to abstract algebra (Hendrycks et al. 202
prevalent (Lin et al.) 2022). MMLU tests model's knowledge and
Results. As shown in Figure 4 models trained with higher global temperatures consistently display lower calibration error. While models trained using an global temperature of (equivalent to DPO) less calibrated than the base model, increasing the global temperature during training quickly enables the SPL models to surpass even the base model's calibration. SPL models also maintain similar accuracy models trained using DPO. In fact, SPL models with global temperatures slightly

---

Published as a conference paper at ICLR 2025
TruthfulQA Accuracy
TruthfulQA ECE
TruthfulQA Brier Score
0.45
Accuracy
Brier Score
0.22
0.40
DPO (aiß
DPO la/B
ECE
0.21
Base Model
0.25
Base Model
0.35
(aiā
0.20
0.30
Base Model
0.20
1.0
1.2
1.4
1.0
1.2
1.4
1.0
1.2
1.4
Global Temperature (a/B)
Global Temperature (o/9)
Global Temperature (e/B))
MMLU - Accuracy
MMLU - ECE
MMLU Brier Score
0.54
0.150
Base Model
0.155
Accuracy
0.52
0.125
DPO (aiß
Brier Score
DPO laiß
Base Model
0.50
0.100
0.150
DPO (a/3 1)
0.075
0.48
Base Model
1.0
1.2
1.2
1.4
1.4 Global Temperature (a//B)
Global Temperature (o/3)
Global Temperature (o/3)
DPO
SPL
Figure 4: SPL improves both calibration and accuracy multiple-choice question (MCQ) datasets. We plot model accuracy, Expected Calibration Error (ECE), and Brier Score for all models on both TruthfulQA and MMLU. The DPO model (equivalent SPL with global temperature 1) displays significantly worse calibration than the base model. In contrast, SPL models consistently exhibit improved calibration without sacrificing accuracy.
greater than consistently attain both higher accuracy and lower calibration error than their DPO counterparts.
CONCLUSION
In this paper, we presented Soft Preference Learning (SPL), a novel approach to mitigating the loss of output diversity in aligned LLMs. By analyzing the role of the KL divergence regularizer in RLHF and DPO, we identified that the coupling of entropy and cross-entropy terms leads to overweighting of majority preferences and reduced output diversity. SPL addresses this issue by decoupling these terms, allowing for independent control over generation diversity and bias towards the reference policy.
Our experimental results demonstrate that LLMS trained with SPL outperform those trained with standard methods in several key areas. From capabilities standpoint, SPL models produce out- puts with greater semantic and lexical variety and achieve higher accuracy on challenging tasks that benefit from diverse sampling strategies. From an alignment perspective, these models can proportionately represent a wider range of societal viewpoints and exhibit improved logit calibra- tion. Importantly, SPL achieves enhanced diversity with less quality degradation, offering Pareto improvement over standard temperature scaling.
The introduction of SPL highlights the importance diversity in the development and alignment of LLMss. SPL, provides the most significant benefits in scenarios that require inference-time scaling or representation of diverse perspectives. By providing mechanism to align LLMSs without causing mode collapse, SPL contributes to the creation of more capable, reliable, and representative lan- guage models. future work, would interesting to explore alternative, semantically grounded diversity metrics that could be integrated into preference learning algorithms such metrics constructed from embeddings or LLM judges.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learn- ing from human feedback in Ims, 2024. URL https://aaxxii.. org/ abs 2402. 14740
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage. Zac Hatfield-Dodds, Danny Hernan- dez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Ka- plan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL httth::////eet //appvi.//axvv...//// http://rggggaabaa/aegooeeee
05862
Alexander Bick, Adam Blandin, and David J Deming. The rapid adoption of generative ai. Technical report, National Bureau of Economic Research, 2024.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Tianchi Cai, Xierui Song, Jiyan Jiang, Fei Teng, Jinjie Gu, and Guannan Zhang. Ulma: Unified language model alignment with human demonstration and point-wise preference, 2024. URL
https://arxiv wwtp:////g/bbaooooo
Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Am- rit Singh Bedi, and Mengdi Wang. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences, 2024. URL https: arxiv. org/ abbs/2000
08925
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021. hatLhhtss:////
org abs
03374
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2023. URL
1776.03740
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021. URL httpp:://axxi..
orr//bb// 2210..210... 14168
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin. Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with scaled ai feedback, 2024.
wtt.:/wwwwrr//s//2l.ooooo http://a./223200.33 01377
DeepMind. 2023.
Ai
solves
IMO
problems at silver medal
level,
URL
httpa: gooqle/ discover/blog/
ai ai-sol voolvvs-imme imo- problems -mo----o-ooolllees at silver- mwda-- ae
Accessed: 2024-10-
02.
Deloitte. State of generative ai in the enterprise: Q3 report. Technical report, De- loitte, 2024.
articles/state-On att.://eeee.ttttooaaaaaaaaaaaaan html Accessed: 2024-09.
28.

---

Published as a conference paper at ICLR 2025
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional
conversations, 2023. URLKtttss//arxiv..oo httgg//bs/23333 14233
Anil Doshi and Oliver Hauser. Generative enhances individual creativity but reduces the collective diversity of novel content. Science Advances, 10(28):cadn5290, 2024.
Benjamin Eysenbach and Sergey Levine. Maximum entropy (provably) solves some robust problems, 2022. http:hhtt///aaxtttt org/ /tsbs//100 2203.. 06257
John Geweke. Bayesian inference in econometric models using monte carlo integration. Economet- rica: Journal of the Econometric Society, pp. 1317-1339, 1989.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-poolicy maximum entropy deep reinforcement learning with a stochastic actor, 2018. URL https:
//axxiv.orr//ab//100011
Dan Hendrycks, Collin Burn, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding, 2021a. URL https:
URL //axxiv.org/abb//000000
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021b.
org. abs /2003.2213..2 03874
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
09685
HuggingFace. Mistral-7b-sft-bett model card.
co/
HuggingFaceR4 htt.:/aaaacaa/iiit.a.c.e. httm/msstrallll 7b- to0ossftbetta betal 2023. Accessed: 2024-10-01.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-oodd, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bow- man, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Ka- plan. Language models (mostly) know what they know, 2022. URL
abs/ / 2207. 05221
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on Ilm generalisation and diversity, 2024. http://pps//aaa...oor/////eee//e
06452.
Dmitry Kobak, Rita Gonzälez-Márquez, Emöke Ágnes Horvát, and Jan Lause. Delving into chatgpt usage in academic writing through excess vocabulary, 2024. URL https://arxiv..rt
07016
Seanie Lee, Minsu Kim, Lynn Cherif, David Dobre, Juho Lee, Sung Ju Hwang, Kenji Kawaguchi, Gauthier Gidel, Yoshua Bengio, Nikolay Malkin, and Moksh Jain. Learning diverse attacks on large language models for robust red-teaming and safety tuning, 2024. Uttpstt:::/arxiv
org/ abs/ 2405. 18540
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022. URL
www.orgggabb///ooo 07958
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, and Bilal Piot. Nash learning from human feedback, 2024. URL https: h //tpr//aavvv..//////////aaa.aaav org/ abs/
231 2. 00886
12

---

Published as a conference paper at ICLR 2025
Minh Nguyen, Andrew Baker, Clement Neo, Allen Roush, Andreas Kirsch, and Ravid Shwartz- Ziv. Turning up the heat: Min-p sampling for creative and coherent Ilm outputs, 2024. URL
/ttr//bb////// org/ abs 2407.00000 01082
OpenAl. Learning to reason with LLMs. 2023. URL httss://ppeaa..commmeen Learning-to- recoo-reaso--- with- l1ms/ Accessed: 2024-10-02.
OpenAl. O1: Scalable Inference for Foundation Models, 2024. [Rtpss::://ope
o// Accessed: 2024-10-02.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback, 2022. URL http///axxvv..r..o wttp:r///a./.......oooo 02155
Vishakh Padmakumar and He He. Does writing with language models reduce content diversity?, 2024. Uttppsss:///axii http://orggga//////eee 051 96
Peter Park, Philipp Schoenegger, and Chongyang Zhu. Diminished th in a standard large language model, 2023. http:t/ts.//aaal...oo/aaaasa/eat
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024. URL /uRLtttss///rxxv.o////////
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks, 2019. htlLLtttee//axxx.oorrr/b//////// 1100b//
CP Robert. Monte carlo statistical methods, 1999.
Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hin- rich Schüizee, and Dirk Hovy. Political compass spinning arrow? towards more mean- ingful evaluations for values and opinions in large language models, 2024. URL https:
//arxiv.org/abai httpi//..rrgabb//4220e/ 16786
Andy Shih, Dorsa Sadigh, and Stefano Ermon. Long horizon temperature scaling, 2023, URL
200os//330.0000 03686
Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell. Distributional preference prefereeeee learn- ing: Understanding and accounting for hidden context in rlhf, 2024. URL htttps:////txi
org/ hts//bb//i 2312.222 08358
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024. URL. htt.ps://arxiv.orgg
abs / 74008 03314
Hao Sun and Mihaela der Schaar. Inverse-rlignment: Inverse reinforcement learning from demonstrations for Ilm alignment, 2024. URL httppss/aarxiv.orrrabb//////////
15624
Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A min- imaximalist approach reinforcement learning from human feedback, 2024. URL https:
// //axxiv.org/as///222/..
04056
Guy Tevet and Jonathan Berant. Evaluating the evaluation of diversity in natural language genera- tion. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Confer- of the European Chapter the Association for Computational Linguistics: Main Volume, pp. 326-346 Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 8021.eacl-oo maid-mman..55. 25. URL https:tt....... //ccl anthol
leacl -aael-maino 25
13

---

Published as a conference paper at ICLR 2025
Brian Thompson, Mehak Dhaliwal, Peter Frisch, Tobias Domhan, and Marcello Federico. A shock- ing amount the web is machine translated: Insights from multi-way parallelism. Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computa- fional Linguistics ACI. 2024, pp. 1763-1775 Bangkok, Thailand and virtual meeting, August (0555////222...ninnngoo..
2024. Association for Computational Linguisties. doi:
URL
https://aclanthologg. http://wwt..ooooooooooooooee//e fineiiiindigsgn wwainngssacl..1 acl. 103
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback, 2023. URL https:
//axxiv.crg/bb//23000 149755
Evan Frick Lisa Dunlap Banghua Zhu Joseph E. Gonzalez lon Stoica Tianle Li*, Wei-Lin Chiang* From live data high-quality benchmarks: The arena-hard pipeline, April 2024. URL https:
7/lmsys. ///pyya..rr//llloo///2200-------
haar/
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani. Kashif Rasul. Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
16944
Chaoqi Wang. Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints, 2023. URL
6240
Jiancong Xiao, Ziniu Li, Xingyu Xie, Emily Getzen, Cong Fang, Qi Long, and Weijie Su. On the algorithmic bias of aligning large language models with rlhf: Preference collapse and matching regularization, 2024. URL https://////.. arxiv. org/ abs/ 2405 16455
Johnathan Xie, Annie S. Chen, Yoonho Lee, Eric Mitchell, and Chelsea Finn. Calibrating language models with adaptive temperature scaling, 2024. URL httph::///aaaat hutpe://aarxivvoorg/bb/////
19817
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023.
URL https:://///. /tasss////vvvv./a/ www.wvoorrrggoooooooo http//bbs2230.... 10601
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A THEORETICAL RESULTS
Proposition 3.1 (Two-Outcome RLHH Policy). Suppose a population of raters prefers completion y y with probability Then RLHF (or DPO) with KL-regularization penalty 3 has the optimal policy
T(y) & Tref Tref(y)p"/5,
(11)
Proof. From Rafailov et (2024). RLHF and DPO share the same optimal policy. So, it suffices to analyze the RLHF objective. First, consider the Bradley-Tery reward modeling objective
max (r(()) - r(t)) + (1 p) log (r(()())))))))))))))(()) - r(y))
(12)
Because log-loss proper scoring nile, we know that the optimal y* will result in distribution- matching
exp(r" (v))
o(r - rr'((r))) (v)) exp(r*(v)) exp(r*(y"))
Following Rafailov et al. (2024), the solution to the KL-regularized RL problem gives us optimal policy
Tly) Tree (v) exp(r" (2))
(13)
for normalizing constant Z. We may rewrite this
exp(r" (1))
π (v) (r) exp(r (v)) + (sp(r"(y')) 1//8 /2'
(14)
Tref #xee(()p (1) 118/211122
(15)
a new normalizing constant Z'
Proposition 3.2
(Two-Outcome SPL Policy). Suppose population of raters prefers completion
y > y with probability ply > y²). Then SPL with entropy bonus a and cross-entropy penalty / has the optimal policy
T(y) x Tres(y) 810 1/0.
(16)
Proof. From Rafailov et al. (2024), RLHF and DPO share the same optimal policy. So it suffices to analyze the RLHF objective. First, consider the Bradley- -erry reward modeling objective
maxplog (r(v) - r(y)) + (1 p) log (r(y') r(y))
(17)
Because log-loss is proper scoring rule, we know that the optimal r* will result in distribution- matching p = (r(y) r(y/) = exp(r(y))/(exp(((u)) exp(r()/))).
Following Proposition A.2 we obtain optimal policy
(v)
Tref (y) 8/0 exp(r(y))
(18)
for normalizing constant Z. may rewrite this as
exp(r(v))
Tty) = Tre/ (1) 8/1
exp(r(v)) + exp(r(y')) 1///2'
(19)
= Tref S/a /
(20)
for new normalizing constant Z
15

---

Published as a conference paper at ICLR 2025
Proposition A.1 Muutti-Otccome SPL Policy). Suppose a population of raters have preferences over completions that can be modeled by Boackett-Luce (i.e. there exists an r such that Vy,,... UN Then SPL with entropy bonus a and cross-entropy
we have p(111 > 1/2. VN) = Ein exp(r(u)) penalty 3 has the optimal policy
exp(r(2))
(V) x Tref (1) Phes) (y)
(21)
where (y) ply Vyyy v\ is the proportion of people for whom y is the most preferred completion.
Proof. With more than two completions, is possible to form preference distributions that cannot be perfectly fit by Bradley-Terry model (e.g. intransitive preferences). So, we first assume that the preference distribution can be represented by a Bradley-Tery model, which ensures that the reward exp(r(u))
learning step results in distribution-matching ply > y) (xpt((u)))
Following Proposition A.2 we have optimal policy πly) (V) 3/0 exp(r()))Ha//z. rewrite this as
We may
exp(r(u)) /2/0/2'
(v) Trer(1) Ey exp(r(1)))
(22)
Tre 9/0 ( /Z'
(23)
for a new normalizing constant Z.
Proposition A.2 (SPL DPO Derivation).
The following objective
max Ey [lo (a log T(yax)
3 log Tref (N12z)
Tres (1/((//())) )]
(24)
shares the same optimal policy as the SPL RLHF objective
max E Ey a) [rxx, y)1 + (H(x(-lz)) BH (1)))
(25)
Proof. For the RLHF objective, begin by training reward model using Bradley-Terry find r r" y) = arg max
Eanxxyyye Ey y a (r((z, 3) (z))
(26)
Next, we find the optimal policy for the SPL RLHF objective. Let us represent policies T and Tref(- (t(ar) and reward function r(x, -) as vectors π, Tref: r. We rewrite the SPL RLHF objective as
max n OCT
log π
/, log Tref
(27)
9 log ree a log Treff
(28)
We solve this constrained optimization problem using the Lagrangian:
L(T, A) = T ax log ग -33 log rref + Σ 5
(29)
Taking the derivative with respect to and setting it to zero, obtain:
0 0 L A) = r - a(1 + log πT) 3 log Tre + A1.
(30)
Now, solving for म, we have:
a(1 log πτ) rr- log Tre Al
(31)
r - B log Feae + X1
+ log π
(32)
r - 3 log Tree + (1 - (x-a)l
log π
(33)
r G log Rree + (A xx)
exp
(34)
a
16

---

Published as a conference paper at ICLR 2025
which implies the optimal SPL RLHF policy for all
x (y/z) ()) = exp( - re) /Z(x)
(35)
a'
where = =xp(a X) is a normalizing constant such that the probabilities over outputs sum to one.
Finally, we analyze the optimal policy for the DPO-style objective. Applying the substitution trick from Rafailov et al. (2024). we can reparameterize reward functions in terms their optimal policies
according to Equation A.2
1 (ylz) exp( r(x, (00)
(36)
α
log (v(æ) 1 rr y) a
_ log loggrer(yyl))) log (xx)
3
(37)
a a (ylz) r(z, y) log Tre (b|z) a log ((x)
(38)
r(z, v) log (u) 8 log Tres (y/x) - a log ((x)
(39)
Observe that
r(x,y) r(x, v/) log n (y/z) log Tee (y]x) + + log Z(x)
(40)
log Ttyl((// log Tres (1/()) log [(x)
(41)
(y(r) <</ (z) ( s))
(ylæ)) log
a log
(42)
since the normalizing constants cancel.
We now substitute our reward function reparameterization into the Bradley-Terry objective to get the SPL DPO objective
x (y/z) max By--//e--llogo((( (ax log <yy(x) - 3 Blogg Tres ) T58((z)
(43)
Tre (v'l))
which satisfies the relationship ()((²² = (x.
(2, v)) Tre
17

---

Published as a conference paper at ICLR 2025
B DIVERSITY-QUALITY TRADEOFFS EXPERIMENT
B.1
ADDITIONAL. DIVERSITY-QUALITY RESULTS AND DETAILS ON DIVERSITY METRICS
Arena-Hard Winrate
Average Reward
Ref.
Policy Cross-Entropy
Arena Winrate
3000
Ref. Policy CE
Reward
2
2000
0.1
2-15
0.35
Sentence-BERT Cosine Distance
Sentence-BERT Cosine Distance
Sentence-BERT Cosine Distance
Winrate
Ref. Policy CE
0.2
Reward
1000
1.
0.25
0.30
0.15
3.40
8,45
0:25
0.30
0.35
0.60
0.45
2445 Embedding
0.25
0.30
0.35
0.40
Embedding
Embedding Cosine
Arena Hard Winrate
3000
Ref. Pollccy Ce
Reward
2
2000
0.1
Logical Disagreement
Logical Disagreement
Logical Disagreement
Arena Hard Winrate
800
02
Reward
1.-
Ref Policy
600
wpe
31
1-1.
-2.
400
-1-11
200
3.
3/2
5.0
s'
3.5
3/4 Content Diversity
Content Diversity
Content Diversity
Ref, Palicy CE
800
Arona Hard
Reward
-15
1w13
6.1
1033
140
400
w1-11
200
3.3
3.4
3.1
3.4
3.5
3.6
3.3
3.5
36
3.7
Surface Form Diversity
Surface Form Diversity
Surface Form Diversity
Ref. Policy CE
6.2
Reward
2000
te35
Arema Hard
bx3.4
1000
6.0
3/2 Perspective Diversity
2.8
3.0
2.8
3.0
3.2
2.6
2.8
3.0
3.2
Perspective Diversity
Perspective Diversity
DPO
DPO w/ min-p
DPO w/ top-p
DPO w/ top-k
SPL
Figure 5: Diversity-quality tradeoffs with additional diversity metries. Here, Sentence- BERT is an additional expected cosine distance metric in addition to "Embedding Cosine Distance" which uses the OpenAl Embeddings API. The remaining four diversity metrics are evaluated with an LLM judge on pools of responses. For SPI., we perform global temperature sweep in range ( [1, 11]. For all other methods, we sweep until outputs degenerate into unintelligible text. For some diversity metrics, such as content, surface form, and perspective diversity, low-quality generations rated as less diverse (see examples below). We label points off the Pareto frontier to help identify these cases.
often
18

---

Published as a conference paper at ICLR 2025
Cosine distance diversity metrics. In Figure and Figure 5 the Sentence-BERT Cosine Distance and Embedding Cosine Distance measure the expected cosine distance between response embed- ding pairs according to two different embedding models: 1) Sentence-BERT-Large (Reimers & Gurevych] 2019) and OpenAl's rext-embedding-33ssmee from HH-RLHF, sample N 16 inputs and calculate the cosine distance between all pairs, Concretely, this is expressed in the formula
model. For each of 500 held-out prompts
((x)) ((ú)) [ (10(1))1
E (d)
Σ 1
(44)
N
where (-) is the embedding map acting on response y-
LLM-based diversity metrics. For the remaining diversity metrics, which split the 16 responses into smaller pools and use "gpt-4o-mini-min -itiiiiiiiii222------1- as a judge for response diversity. Here provide the list of diversity evaluation prompts used for metries in Figures 2 and 5
Logical Agreement Evaluation Prompt
You will be presented with responses the same prompt. Your task is to analyze their logical agreement on scale of 1-5, where I means completely divergent approaches or ideas and means nearly identical logical frameworks or conclusions.
Prompt: [input}
Responses:
(response list)
Please provide numerical rating (1-5) of their logical agreement. Output your response as Rating: where X your rating. Afterwards, on the next line, please provide brief explanation of your rating.
For the logical agreement prompt, then subtract the resulting from 5 to turn it into logical disagreement metric.
Content Diversity Evaluation Prompt
You will be presented with 4 responses to the same prompt. Your task is to analyze their content diversity on a scale of 1-5, where means nearly identical content or conclusions and 5 means completely divergent content or ideas.
Prompt: {input}
Responses:
(response_list
How diverse are contents of the proposed responses? Please provide numerical rating (1-5). Output your response as Rating: X, where X is your rating. Afterwards, the next line, please provide a brief explanation of your rating.
Surface Form Diversity Prompt
You will be presented with 4 responses to the same prompt. Your task is to analyze their surface form diversity on scale of 1-5, where means nearly identical textual organization or structure, and 5 means completely varied textual arrangements or styles.
Prompt: {input}
Responses:
(response_lise
How diverse the surface forms of the proposed responses? Please provide numerical rating (1-5). Output your response as Rating: X, where X is your rating. Afterwards, on the next line, please provide brief explanation of your rating.
19

---

Published as a conference paper at ICLR 2025
Perspective Diversity Prompt
You will be presented with 4 responses to the same prompt. Your task is to analyze their per- spective diversity on a scale of 1-5, where I means nearly identical viewpoints or approaches and means completely varied perspectives or approaches to the topic.
Prompt: {input}
Responses:
(response list)
How diverse are the perspectives of the proposed responses? Please provide numerical rating (1-5). Output your response as Rating: X, where is your rating. Afterwards, on the next line, please provide brief explanation of your rating.
B.2 EXAMPLE GENERATIONS
B.3 TRAINING SETUP
DPO and SPL Finetuning and Inference. For both DPO and SPL, LoRA-finetune Mistral- 7B-Instruct--0.22 on HH-RLHF for 5,000 steps with batch size 8. We use LoRA rank TLoRA = 16, regularization OLRAA 16, and dropout PLORA 0.05. We use learning rate le 5, 150 warmup steps, and max conversation length of 512 tokens.
For all runs, scaling, we sample with temperatures = I, 1.1, 1.2, 1.3, 1.4, and 1.5. When combined with min- p sampling, we choose Pbase 0.1 and temperatures combined with top-p sampling, we choose = 0.9 and temperatures t = 1.1, 1.2, 1.3, 1.4, 1.5, 1.1,
regularization parameter 33 0.1. For DPO with token-level temperature
1.3, 1.5, 2.5, 3, 3.5, and When
180 and temperatures 1.2, 1.3, 1.5, 2, and 2.5. We stop at these temperatures because beyond this, responses are nearly always incoherent strings tokens. Lastly, for f-DPO (Wang et al. (023), we consider the family of a-divergences including 0 (reverse KL), 0.001, 0.1, 0.3, 0.5, 0.9, and (forward KL). Here, small a correspond to mode-seeking (diversity mass- cassriovering (diversity-preservinn) preserving) divergences. We stop are incoherent strings of text. For SPL, we train with global temperatures a/3 = 1, 1.1, 1.2, 1.3, 1.5, 2, and 11.
1.6, and 1.7. When combined with top-k sampling, we choose
divergences while large a correspond to
because, beyond this, responses
Reward Model Training and Inference. As one of our quality metrics, we measure the average reward obtained by model completions against separately trained reward function. To obtain this reward model, we LoRA-finetune Pisstrl-7B-Instructte Terry loss the HH-RLHF dataset.
with reward head using the Bradley-
At inference time, we sample 500 prompts from a held-out validation split of HH-RLHF that neither the language models nor the reward model were trained on. We use this calculate the average reward achieved by each model's completions.
20

---

Published as a conference paper at ICLR 2025
C BEST-OF-N PROBLEM-SOLVINE EXPERIMENT
C.I RELATIONSHIP BETWEEN PROBLEM DIFFICULTY AND OPTIMAL TEMPERATURE
In this section, we study the following questions
1. Why do harder problems generally benefit from higher temperatures? And why does this benefit disappear once the temperature becomes large?
2. Why are high temperatures especially helpful high best-of-N sample settings?
GSMBK Optimal Problem Difficulty
MATH Optimal a/B vs Problem Difficulty
Optimal DPL Temperature ajj
1.30
Optimal DPL Temperature alj
1:25-
1.20
0.
00
000
Trend
1:15
Trnd line
1:10
1.05
1.00
1.00
16
125
64 128
Problem Difficulty (Expected Samples Until Success)
Problem Difficulty (Expected Samples Success
Figure 6: Optimal SPL temperature function of problem difficulty. On the x-axis, we mea- sure problem difficulty as the expected number of samples needed for Mistral-Instruut-77 to achieve a correct answer. the y-axis, we indicate which SPL temperature led to the best success rate on that problem. We then fit polynomial trendline to the data, which finds a positive relationship between problem difficulty and higher optimal temperatures. Dots are plotted with transparency, meaning darker dots correspond to multiple problems.
Why harder problems benefit from higher temperatures, up a point. We first consider the role of temperature in the single-sample (not best-of-N setting). As displayed in Figure 6 every problem has an optimal temperature that maximizes the probability of successful solution. creasing temperature flattens the distribution while decreasing it sharpens the distribution. Difficult problems are those where the trained model has a low probability of generating a correct solution.
During inference-time scaling procedures, our goal is to construct sampling distribution that mini- mizes the number of samples needed to obtain correct solution. For best-of-N, we receive feedback only upon generation completion, where samples must pass final scoring function. This is akin to rejection sampling in statistics, where proposal distributions are chosen to minimize the number of samples required approximate target distribution.
In the rejection sampling literature, when little is known about the target distribution, well- known that flatter or high-entropy proposal distributions are best (Robert) 1999). Intuitively, highly concentrated proposal distributions that narrowly miss the target will achieve extremely low success rates. In contrast, flatter proposal distributions whose modes narrowly miss the target will still achieve reasonable success rates. The optimal proposal distribution scales in flatness inversely with one's confidence in its overlap with the target distribution (Geweke 1989). However, an overly flat distribution is also inefficient, underestimating the true overlap with the target distribution.
This intuition confirmed in Figure 6 where low-success problems are benefitted by SPL. tem- peratures higher than 1, which produce flatter sampling distributions. However, when temperatures become too large, performance again degrades. Additional evidence can be found the y-intercepts of Figure 3 and best-of-1 performance of Tables 3 On easy and medium problems, DPO has substantially higher single-sample success rate than temperature-see scaled methods. However, on hard problems, this gap shrinks. Table [ shows that GSM8K and MATH hard splits, SPL is within 0.2% and 0.08% SPL's best-of-1 success rate. This supports the finding that harder problems have higher optimal temperatures.
21

---

Published as a conference paper at ICLR 2025
PsaN V5
1.0
Bost bost-ol.tt Nf sucress probabilky Pbot
0.8
0.6
0.4
0.2
N=1
N-16
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Single-sample success probability p
Figure 7: Best-of-N success rate vs single-sample success rate as a function of N. As N in- creases, the best-of-N success curve approaches step function. Hard problems with low single- sample success rates benefit substantially from small increases in Meanwhile, the best-of-N suc- cess rate of problems past the "elbow" is already saturated, meaning that small decreases in p have little effect on PBoN- The rapidly decreasing gradient of this curve favors high-variance strategies such as high-temperature sampling.
Why high temperatures especially helpful in high sample best-of-N settings. For given problem æ, let use define the acceptable set A as the set of completions y that are correct. We can then define the probability of success for an LLM as p (AAr) ΣyEA (g|æz)
The best-of-N success rate then be modeled by the CDF of geometric distribution
PBoN (1-
(45)
Note that PBON is monotonie in single-sample success rate This means that for given problem, the optimal temperature which maximizes single-sample success rate also maximizes best-of-N suc- cess rate. Thus the number of samples N does not influence the optimal sampling temperature for problem.
Why, then, in Figure B do observe high temperatures being differentially helpful at high N? While optimal temperatures do not change, the relative gains and losses from increasing temperature become asymmetrical.
Rather than single problem performance, we are interested in the aggregate success rate across a set of problems with different difficulty levels
E2.A~D(PBol = =E,AAD[[L (1- (A
(46)
As shown in Figure 7 hard problems benefit from small increases in p than easier problems suffer from an equivalent decrease in In aggregate, this causes the optimal temperature for the set of problems to trend upwards N increases. Intuitively, high N, can afford to sacrifice some performance on easy problems (which will still succeed) to boost performance on hard problems (which benefit more from more diversity).
This behavior is a consequence of the concavity of the best-of-N success rate function f(p) = 1 (1-p) By Jensen's inequality, increasing the variance of p (e.g.. via higher temperatures) can raise the aggregate success rate across problems, even though the single-problem optimal temperature remains unchanged (Boyd Vandenherghe, (004). This results the "crossing" behavior seen in Figures and As N increases, f(p) becomes more aggressively concave, and higher temperatures begin to outperform standard DPO.
22

---

Published as a conference paper at ICLR 2025
GSMBK HARD
MATH HARD
Normalized Best-of-N accuracy
zeze Best-of-N accuracy
+1.0
+1.0
+0.8-
+0.8
+0.6
+0.6
+0.4
+0.4
2
4
8
16
32
64
128
1
8 16
32
64
128
Number of samples
Number of samples
DPO DPO
SPL a/B 1.1
SPL 1.2
Figure 8: Normalized best-of-N accuracy on difficult problem-solving instances. We show best- of-N accuracy of cach method fraction of the accuracy achieved by standard DPO. On these problems, higher temperature runs have a lower single-sample accuracy than standard DPO. How- ever, of them cross and eventually surpass DPO for large N. However, with too high of token-level temperature, quality degradation is so significant that the curve never approaches stan- dard DPO performance.
GSMBK HARD
MATH HARD
+0.4
Sample efficiency DPO
Sample efficiency DPO
+0.2
+0.2
0.0
00.2
-0.2
-0.4
-0.4
-0.6
0.8
-0.8
4
8
16
32
64
128
4
8 16
32
64
128
Number of samples
Number of samples
DPO DPO t= 1.1
SPL a/B = 1.1
a/B 1.2
Figure 9: Sample efficiency versus DPO. For each number of standard DPO samples, we show how many samples are required for other methods to achieve the same best-of-N accuracy. At low samples, DPO is the most sample efficient method. However, at higher sampling budgets, SPL. achieves the same accuracy as DPO with 25-35% computational savings. DPO t 1.2 was not included in this graph it lies far below all other methods.
One intuitive mechanism that drives this -iversity-favoring non-convexity is the way that best-of-N treats duplicate solutions. As shown in Figure redundant samples are major source of sample inefficiency for DPO at large N. In the single sample case, putting high mass on likely completions increases the success probability. However, in the best-of-N setting, is optimal to never sample the same solution twice, as it does not increase the probability of success. One advantage of high- temperature sampling the best-of-N setting is simply that it produces fewer redundant samples.
Conclusion. This analysis provides theoretical grounding for our empirical finding that higher SPL temperatures benefit difficult problems high-sample settings. It also suggests guidelines for prac- titioners: when compute allows for many samples, appropriately chosen high temperature sampling can be used to tackle the hardest problems in dataset without significantly impacting overall per- formance.
23

---

Published as a conference paper at ICLR 2025
Unique Solutions (out of 128 samples)
GSM8K
MATH
80
120
70
100
60
50
80
40
60
DPO
D
1
SPL a18
SPL aiB 1.2
DPO
DPO t=11
DPO
SPL 1.1
SPL alß 1.2
Figure 10: Solution diversity and sampling efficiency. For best-of-N sampling, repetition in final solutions is major cause of sample inefficiency. At 128 samples, 30-40% of solutions are redundant and have already been sampled before. While convergent chain-of-though reasoning can be helpful settings such as majority voting, it is undesirable when a verifier present. Both token-level temperature scaling and SPL increase the number of unique solutions sampled. For example, SPL samples 17% and 10% more unique solutions than DPO on GSM8K and MATH, respectively. This equivalent to 12% and 22% reduction in redundant solutions.
C.2 BEST-OF-N ACCURACY TABLES
We also provide results from Figure in tabular form for an easier quantitative performance com- parison.
Best-of-N
DPO t=1
DPO t=1.1
DPO t=1.2
SPL a/ 0/3=1.2 SPL a/3=1.1
GSM8K
1
1.43%
1.24%
0.59%
1.24%
0.91%
4
5.35%
4.75%
2.25%
4.74%
3.53%
16
16.92%
16.37%
7.62%
16.30%
12.50%
64
37.98%
41.11%
17.77%
42.43%
32.91%
128
48.75%
54.07%
21.66%
58.34%
43.07%
MATH
1
0.86% %
0.85%
0.35%
0.78%
0.85%
4
3.22%
3.19%
1.37%
2.97%
3.20%
16
10.18%
10.44%
5.03%
9.89%
10.50 %
64
22.12%
24.02%
14.81%
23.11%
24.51 %
128
27.99%
30.62%
21.31%
29.16%
31.91 %
Table 1: Best-of-N accuracy on GSM8K and MATH dataset hard splits. We provide Figure results in tabular form for easier quantitative analysis. 64 samples and above GSM8K and 16 samples and above on MATH, SPL outperforms standard and temperature-scales DPO. At 128 sam- ples on GSM8K, SPL achieves 10% higher accuracy than standard DPO and 4% higher accuracy than temperature-scaled DPO. At 128 samples on MATH, SPL achieves a 4% higher accuracy than DPO and 1% higher accuracy than temperature-sclll DPO. We expect these gaps widen with more compute-efficient inference-time scaling methods than naive best-of-N.
24

---

Best-of-N
DPO t=1
DPO t=1.1
DPO t=1.2
SPL /z 3=1.1
SPL // 3=1.2
GSM8K
1
8.95 %
6.06%
3.62%
5.79%
5.22%
4
26.81%
19.70%
12.39%
19.01%
17.39%
16
55.63%
46.67%
31.77%
46.05%
42.45%
64
79.21 %
72.74%
55.10%
73.40%
67.33%
128
85.30%
80.17%
63.93%
81.14%
74.40%
MATH
1
5.40% %
3.67%
2.16%
3.43%
3.56%
4
15.57%
11.23%
7.59%
11.09%
11.61%
16
36.15%
26.74%
20.71%
27.62%
29.57%
64
61.49%
50.92%
39.31%
52.11%
56.41%
128
69.07%
62.66%
46.88%
62.01%
66.96%
Table 2: Best-of-N accuracy on GSM8K and MATH dataset medium splits. Standard DPO
achieves the best accuracy in all settings here, although SPL begins to approach DPO at high sample counts, especially on the harder MATH dataset.
Best-of-N
DPO t=1
DPO t=1.1
DPO t=1.2
SPL 0/3.12 SPL a/
GSM8K
1
34.18%
26.40%
15.58%
24.33%
21.86%
4
68.94%
58.70%
41.93%
56.60%
52.70%
16
91.39%
85.67%
72.19%
84.75%
82.04%
64
97.97%
97.00%
89.48%
96.51%
95.62%
128
98.80% %
98.51%
92.86%
98.38%
98.22%
MATH
1
22.51 %
16.99%
14.01% %
18.02%
15.09%
4
46.52%
40.07%
34.89%
42.09%
38.32%
16
72.03%
65.00%
60.18%
66.99%
63.60%
64
89.72%
87.51%
83.32%
88 $8.49
81.86%
128
92.77%
94.49%
91.84%
92.63%
86.05%
Table 3: Best-of-N accuracy on GSM8K and MATH dataset easy splits. Standard DPO is by far the best performer prior to saturation (accuracies 100). At 128 samples on MATH however, temperature scaled DPO outperforms standard DPO by 2% and SPL matches standard DPO.

---

Published as a conference paper at ICLR 2025
GSM8K Easy
GSM8K - Medium
GSM8K Hard
GSM8K Hard
1.0
1.0
Best-of-N accuracy
Best-of-N accuracy
0.8
Relative to DPD
0.6
0.6
0.4
0.2
0.0
0.4 2 4 8 16 32 64128
2 4 16 32 64128
2 4 16 32 64128
2
4 16 32 64128
Number samples
Number samples
Number samples
Number of samples
MATH Easy
MATH Medium
MATH Hard
MATH Hard
0.4
Best-of-N accuracy
Best-of-N accuracy
Best-ofN accuracy
0.8-
0.6
0.3
Relative DPO
0.95
0.2
0.90
0.2
0.1
0.85
0.2
Number of samples
Number of samples
Number of samples
Number of samples
DPD t= 1.1 min-p=0.11
D 1.2
DPO =
SPL a/ß - 12 min-p=0.1
SPL a/B 1.5
SPL a/B 2.0 min-p=00.1
min-p=0.1
min-p=0.1
min-p=0.11
Figure 11: SPL vs DPO on best-of-N problem-solving with min-p sampling, Left three columns show best-of-N accuracy across difficulty levels. Right column shows performance on hard prob- lems relative to DPO min-p baseline at given sample count. Min-p sampling stabilizes high- temperature sampling - improving both token-level and global temperature scaling. Results here on hard splits are mixed with DPO performing best GSM8K and SPL performin best on the harder MATH dataset.
C.3 BEST-OF-N ACCURACY WITH MIN-P SAMPLING
Figure I1 shows best-of-N N problem solving results when combining methods with min-p sampling. Our results in ang that token-level temperature scaling min-p can be a competitive baseline to SPI., although SPI. maintains performance and coherence at higher temperatures better (e.g. a/3 = 2). Best-of-N results on GSM8K and MATH hard splits are mixed, since token- level temperatures just above 1 provide good enough diversity boost without degrading the token distribution enough to overwhelm min-p's stabilizing properties. Nevertheless, we expect SPL to perform particularly well on even harder problems and higher sample settings when higher levels of diversity are required.
26

---

Published as a conference paper at ICLR 2025
C.4 TRAINING AND INFERENCE SETUP
DPO and SPL Finetuning and Inference. For both DPO and SPL, our base model is a Mistral-7B base model that has been full-parameter supervised fine-tuned the UltraChat dataset. We then LoRA-finetune this model on Ulrafeedhack-200K for one epoch. We use batch size 4, LoRA rank LLORA- 64, regularization LoRA = 64, and dropout PLoRA 0.05. We use learning rate 1e 5, 150 warmup steps, and max conversation length of 1024 tokens. This replicates the Zephyr training recipe, except that we use 3 0.1 instead of 3 0.01, which substantially improved problem- solving performance. While more computationally expensive than the HH-RLHFF training runs, we found necessary to use stronger fine-tuning recipe like Zephyr in order to get meaningful results in difficult mathematical problem-solving settings.
At inference-time, we use standard few-shot prompts to encourage chain-of-thought reasoning.
GSM8K few-shot prompt
As an expert problem solver solve step by step the following mathematical questions. Place your answer after the "#I##" symbol.
Q: There are trees in the grove. Grove workers will plant trees the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today? A: We start with 15 trees. Later we have 21 trees. The difference must be the number of trees they planted. So, they must have planted 21 15 6 trees. The answer is 6. ##### 6
Q: If there 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot? A: There are cars in the parking lot already. more arrive. Now there are 2 cars.
The answer is 5. #### 5
Q: {question}
A:
MATH few-shot prompt
Given mathematics problem, determine the answer. Simplify your answer as much as possible. Output your answer in the format Answer: $ b/boxe((asswer/
Problem: Let $f(x)=x"3+33 and = =x 2x +1S. What Sg(f(-2))S? Answer: We note that $((=))=(2)3+3=555 (y(f 5)+1=41$ So, S)boxee(4113S
###
Problem: What is the greatest possible value of $x+yS such that Sx + y²2 =90S and Sxy=27$?
Answer: We have 24 12s. We want the larger value $x+y= (oxed(12){ ###
27=144S, so Sx+y=125 or $x+y=
Problem: {problem}
27

---

Published as a conference paper at ICLR 2025
D LOGIT CALIBRATION EXPERIMENT
Multiple Choice Prompt
Please answer the following multiple choice question. Start your answer with the capital letter corresponding to your choice:
{question)
Answer choices:
A. {option }
B. {option_b) b]
C. (option c
D. {option d)
Your answer:
28

---

Published as a conference paper at ICLR 2025
E POLITICAL ORIENTATION DIVERSITY EXPERIMENT
Response Diversity (Sta Dev of Agreementt Levelss Levels)
Political Orientation Diversity On Economic Questions
Political Orientation Diversity On Social Questions
1.4-
Pldd 2434
Mocian 3.45
Meen: 6.7
144
Musaar 2.
Median: 0.7!
Mean 0.72
Response Diversity (Std Dev of
0.8
0..5
0.9
0.0
DPO
DPD t 1.4
SPL a/ 2.0
DPO
DPD t= 1.4
SPL a/ 2.0
Figure 12: Hiig--teperature sampling increases diversity of political opinions, but only SPL preserves coherence. We have each model generate 32 responses each of the 60 political state- ments on the Political Compass test. Each response is scored on a Likert scale from (strongly dis- agree) (strongly agree). nigh-teempraturee sampling increases the diversity of political stanees of responses over standard DPO. However, token-level temperature scaling degrades response qual- ity, as shown in the examples below.
We evaluate the diversity of political opinions represented by different models using the Political Compass test from www.politialloopaas.oo The test consists of 60 political statements repre- senting particular viewpoints on either economic or social questions. In line with previous work on best practices for measuring political representation in LLM responses, we generate 32 open-ended responses to each question (Röttger et al. 2024). We then use 111-440-iii2222-07-- to place responses on Likert scale from (strongly disagree), 2 (disagree), (agree), (strongly agree).
We report the standard deviations of these scores across different responses to the same question in Figure [2] We find that the standard deviation of scores for the median question are similar for DPO t == 1.4 and SPL a/3 2, which are both more diverse than DPO standard. However, SPL better preserves coherence (see samples below).
29

---

Published as a conference paper at ICLR 2025
Sample Economic Question from Political Compass Test
Statement: People are ultimately divided more by class than by nationality.
While is true that class-related differences and disparities exist and can cause signif- icant divisions within and between societies, it is important to note that nationality remains a significant factor in shaping people's identities and experiences. Class and nationality are interconnected and can overlap, but they also distinct social constructs.
DPO:
Class refers to an individual's socioeconomic status or position in society, which can be influenced by factors such as income, education, occupation, and access resources. Class- related divisions can manifest in various ways, including economic inequality, political power imbalances, and social exclusion.
Nationality, on the other hand, refers to an individual's citizenship or membership in par- ticular nation or country. Nationality often expressed through cultural, linguistic, and historical ties, and it can significantly shape people's sense of identity and belonging, Na- tionality can also create societal and political divisions, particularly contexts of ethnic or religious conflict unequal access to resources and opportunities.
is essential to acknowledge the complex ways in which both class and nationality intersect and influence people's experiences. While class divisions may be more apparent in some contexts, nationality remains a powerful and often visible marker of identity that can shape people's lives in profound ways. Ultimately, both class and nationality divisions and inequalities, and addressing these issues requires a nuanced understanding of their interconnected roles in society. Agreement Level: 2 (Disagree)
contribute to
DPO t 1.4: It is true that people can often be divided by social and economic class more than by nationality in some cases. CosmellesEnablled())iiiinn much about the connections between individuals is influenced by various factors, including but not lim- ited to race, nationality, gender, age, religion, political views, and socioeconomic status. It's important to recognize that, while nationality can create significant divides and can be a major source of identity and pride, class can create uniquely distinct experiences and opportunities that maintain sense of distinction between people. In certain circum- access to education, healthcare, employment oppor- tunities, political power, and cultural resources. Although it's important to acknowledge that nationality and class often intersect and overlap, ultimately, the systems and struc- be quantifiable and internationally compar-
stances, socioeconomiestatusanning
tures that perpetuate class divides ative than the divisions created by nationality. lenge traditional notions of ethnic and national differences, givenemerringnpatnesshipp.p globalization, and migration. Eegration.itorrofiinn itorrrofinituulure [...] phechosssorraddessriotttteeeesss assnammeusaandyyllllDiiiiigggg
aratiinnllyyrassattioaaadeeooeet chal-
ourselves as primarily part coclass iffesaddressaddress- It is a common belief that people are more divided by class than by nationality, statement valid?
DassttruggesmoorquuuntNN
Reply:
This statement partially valid as socioeconomic status can create significant divides and differences between people ScotlandFilecin FacebookvPhpage Phppgeto investiggManjofforessdiff ferencesamonglt can be argued that although Nationalitiesareeys [...] prevents admitantly Clarionth Civilw identifying thingsint Unless atherwisementionnesseess decades edge Twn- dejoin Ibid reported Africanschoolatt highway spec TelcPF1I Agreement Level: 4 (Strongly Agree)
SPL a/3 = 2: There is a significant Body of research that underlines how social class can be a powerful force in shaping people's identities and social connections, often more so than nationality. According to Fanon, in his seminal book Black Skin, White Masks, "each civilization is founded upon an exclusively superficial class through which it states its fundamental ideology" (p. 34). By this he meant that people are fundamentally divided by the various social classes that exist, and by the fact that those classes guide our relationships to one another, our sense of ourselves, and our worldview. In this see-saw, each civilization appears to be divided according to its dominant class and its corresponding self self-image and values. Class is the means by which participants in diverse cultures construct meaning. Agreement Level: (Agree)
30

---

Published as a conference paper at ICLR 2025
Sample Social Question from Political Compass Test
Statement: I'd always support my country, whether was right or wrong.
While it's commendable that you have strong feelings of patriotism and loyalty to- wards your country, it's also important not to overlook the value of critical thinking and holding your country accountable for its actions. Blindly supporting your country regard- less of its righteousness to question and demand improvement when necessary. Ultimately, supporting your country doesn't endorsing everything it does, but rather striving for its betterment, learning from mistakes, and promoting its core values. Agreement Level: (Disagree)
DPO:
sometimes lead to perpetuating harm injustice. It's healthy
DPO 1.4: While understand the sentiment expressed in your statement, it's essential to also support progress, peace, and justice both for our own country and for others around world. Unquestioning support for a country's actions, regardless cal alignment, can potentially perpetuate harmful behaviors and hinder positive growth. It's important to strive for a balanced approach supporting the good efforts of our country, but alwaysShadow not Found pushing for improvements when needed. Faith in our ability as citizens to readily address and correct mistakes makes our countries stronger and more focused on integrity and growth. Additionally, pathy towards other nations and cultures around the world. That being said, the ultimate pursuit ofSource not Found supporting the welfare of humanity whole should continue to guide our integrity as global citizens. Agreement Level: (Strongly Disagree)
their moral or ethi-
cultivate caring and tremendous em-
SPL a/3 2: It's important to support your country and the values that its members hold dear. At the same time, strongly believe that commitment to a country or its government should be balanced with a critical perspective, in order to ensure that the ideals and prin- ciples of the society are upheld. Furthermore, it's essential to remember that believing in or supporting your country does not regulate your thinking or politics. It's by evaluating the wrongs and rights of policies and sometimes criticizing working towards change, and through peaceful means, that countries prosper and their societies can develop. By observ- question
ing the programmes of our government and policies of other governments, corruption and injustice anywhere and inform /engage in the pursuit of faimesss and equality for all people. Use this powerful perspective, free thinking and problem solving rather than criticism, to change the way we think about and live in the world. Agreement Level: (Disagree)
31


Reviews:
[{"review_id": "uKcPhLRPdS", "reviewer": "Reviewer_Tm8X", "summary": "This paper addresses the issue that alignment algorithms lead to a reduction in diversity. They motivate the need for diversity in a few ways: 1) societal implications, 2) better diversity might mean better inference-time problem solving (because you’re essentially doing a wider search in solution space), 3) better calibration.\n\nThough temperature scaling is one way of addressing this issue, the authors demonstrate that it is not sufficient, as temperature scaling quickly leads to degradation in generation quality.\n\nThe authors address the problem by first observing that the KL divergence term is the culprit for drop in diversity (Proposition 1). Ie, the hyperparameter \\beta controls both the reference policy’s weighting and the sharpness of the output distribution. \nThe solution that the authors propose is to decouple the KL regularization term into a cross-entropy term and entropy term, but weigh them independently (with hyperparameters \\alpha, \\beta), which in turn ends up independently affecting the reference model’s output distribution and how much such distribution is weighted (proposition 2). \n\nThe authors draw a couple of connections between their change and prior work – when \\alpha == \\beta, then this is the standard DPO regime. \nSimilarly, their \\alpha parameter can be thought of as temperature scaling, but done at a sequence level as opposed to token level (equation 14).\n\nFinally, the authors demonstrate a series of empirical results that match their original set of motivations (1: better diversity, 2: better inference-time problem solving, 3: better calibration).\n\n1) Diversity (Figure 2): The suggested approach method achieves better pareto curves when plotting diversity scores vs. generation quality\n2) Problem solving (Figure 3): For hard problems, in which an increased number of generations helps solve the task, their suggested approach leads to higher performance, which they attribute to increase in diversity across generations (ie, doing a wider search in solution space).\n3) Calibration (Figure 4): The authors show that *without affecting accuracy*, their suggested approach leads to improved calibration (according to ECE and Brier scores).", "questions": "Figure 2: There seems to be quite a large range in terms of win-rate/avg. reward/ref. policy CE when you consider points that lie below the Pareto curve (points with lighter shades). Do you have an explanation for such behavior? If the range of behavior is so wide, is it possible that the improved pareto curve is due to noise?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Well organized, lots of empirical results, the suggested approach is simple and principled.", "weaknesses": "* At a high level, the authors attribute improved problem solving (which is actually not very evident in Figure 3) to improved diversity. Though I could imagine that diversity hypothetically can lead to improved problem-solving performance for methods that use inference-time scaling (because the model is essentially generating more hypotheses for solving a problem), it seems like Figure 3 is not quite telling this story.\n* First of all, in the case of easy/medium difficulty problems, their suggested approach does not seem to be better than DPO – this is fine for the easy case – in which both DPO and DPL saturate to near perfect accuracy, meaning that the problems didn’t require a wide search in hypotheses in the first place.\n* However, in the medium case, DPO outperforms DPL, with still a lot of room for improvement for both models (ie. we do not see high accuracy saturation). \n* Though DPL outperforms DPO in the hard case, the improvement is rather minimal. \n\n* Perhaps more importantly, if more diversity does indeed lead to better problem solving, then even within DPL, we should see a relationship between temperature and best-of-N accuracy, which is not the case in Figure 3 – sometimes higher temperature is better, sometimes it’s not. Studying how accuracy is affected by varying temperatures would have been helpful.\n* The author’s claim that improved diversity leads to better problem solving for methods that use inference-time scaling could be strengthened by actually trying an inference-time scaling approach – this is something that can be empirically studied. I realize this might be a big ask, but I am willing to raise my score significantly if indeed the authors demonstrate evidence of this. Otherwise, I think their claim that DPL leads to better problem-solving is not very convincing. \n* (While we’re at it - minor note regarding Figure 3: the color scheme (gradients of green + blue) makes it very very difficult to parse the results. Please consider changing it)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cTkKEqzlcb", "reviewer": "Reviewer_foMz", "summary": "The contribution of this paper is simple, but well-motivated and addressing a real issue. \n\nThe authors note that the $\\beta$ parameter, which controls the KL regularisation strength in DPO, is involved in mode collapse to the majority option. Temperature sampling, they argue, does not fix this, as it causes disfluencies and broken outputs. To address this, they propose to decompose KL into an entropy and a cross-entropy term with the former, each with separate weights modulating the strength of the regularisation. They are able to bake this into DPO, proposing a new objective called DPL. They show how this objective can be thought of as temperature scaling on the sequence level. \n\nThey perform two different flavours of evaluation: in the first one they assess the quality/diversity tradeoff on Arena-Hard and HH-RLHF and they found DPL to be Pareto-dominant; in the second, they show that in a best-of-N sampling setup (GSM8K and MATH), DPL helps on hard problems, but results are mixed on easy-medium. Finally, they show improved results on MMLU and TruthfulQA in terms of both accuracy and calibration error.", "questions": "Suggestions:\n- Figure 3 is hard to read", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- To the best of my knowledge, the proposed method, while obvious in hindsight (as most good things are) is novel\n- The problem is well posed and mostly convincingly discussed. Acting on the entropy of the distribution has been gaining momentum in the community\n- Experiments are quite comprehensive, though some baselines should probably be included. (This is the main reason I am giving a 6 instead of an 8) (edit: improved after rebuttal)\n- The quality of the writing is outstanding", "weaknesses": "- The experimental setting could be stronger: DPL is mostly pitted against vanilla temperature scaling, which on its own does lead to bad outputs. However, top-k, top-p, and, more recently, min-p sampling are all relatively established solution that should have been incorporated in the analysis. (edit: improved after rebuttal)\n- Basic REINFORCE (with no KLD in the reward), which has become very popular again without should also have been compared against. This is more of a desideratum than a hard requirement.\n- Figure 2 seems to show some brittleness of DPL, with many green dots substantially worse than the Pareto frontier. Some additional discussion about it would be helpful", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OAXxdHiIyJ", "reviewer": "Reviewer_whFb", "summary": "The paper discusses how alignment algorithms like RLHF and DPO reduce the diversity of large language models (LLMs), leading to less varied and overly homogeneous outputs. The study identifies the KL divergence regularizer in preference learning algorithms as a primary cause, which biases models towards majority opinions at the expense of diversity. To combat this, the paper proposes a new method, Diverse Preference Learning (DPL), which decouples entropy and cross-entropy terms in the KL penalty to allow more controlled diversity in LLM outputs. DPL demonstrates better performance in representing a broader range of societal perspectives and produces outputs with greater semantic and lexical diversity compared to traditional methods.", "questions": "1. In line 215, what if ``Empirical choices of β typically lie in the range [0.01, 0.1].''\n2. Question towards Sec. 3.2:\n  2.1. In Eq. 8, should it be H(π(·|x),πref (·|x))?\n  2.2. In Eq. 22, should it be + απ^⊤ log π?\n  2.3. In Eq. 23, should it be + α log π?\n  2.4. From Eq. 25 to Eq. 26, should π = exp()-1?\n  2.5. If Question 5 stands, does Eq. 26 stand?\n  2.6. If in Eq. 28, it is proportional relation, you cannot use equal in Eq. 29.\n3. How to calculate \"1) general semantic diversity, 2) logical diversity or diversity of viewpoints, and 3) content diversity.\" and what's their meaning? What is \"Embedding Cosine Distanc\" in Figure 2?\n4. Why do you generate 16 responses for each question?\n5. How do you train the reward model, and why the rewards are all negative?\n6. What is DPO t=1.2 in Figure3, and where is the definition of t?\n7. Why do you use Mistral-7B-Instruct-v0.2 for DIVERSITY-QUALITY TRADEOFFS, while Mistral-7B base for the other two task?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The theoretical analysis on KL-divergence sounds interesting.\n2. The experimental design of the article is quite creative.", "weaknesses": "1. The objective of DPL (Eq. 9) appears interesting, yet the derivation process seems to have many inconsistencies, as detailed in the Questions section.\n2. You should define Proposition 3.1, Corollary 3.2, and Proposition A for clarity.\n3. The experiments lack comparisons with existing Diverse Preference Learning methods, such as [1,2], and are missing richer alignment baselines for analysis.\n4. The experiments lack many implementation details necessary for readers to fully understand or reproduce the results. See Questions for details.\n5. It appears that DPL does not perform better than DPO on mathematical tasks.\n\n[1] Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints\n[2] Inverse-RLignment: Inverse Reinforcement Learning from Demonstrations for LLM Alignment", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)