REFA: Reference Free Alignment with Fine-Grained Length Control
Authors: Taneesh Gupta
Rahul Madhavan
Xuchao Zhang
Chetan Bansal
Saravan Rajmohan
Venue: COLM 2025 (2025)
Keywords: Model Alignment
RLHF
Preference Optimization
TL;DR: Reference-free alignment methods that optimize over multiple user preferences with fine-grained control of length

Abstract: To mitigate reward hacking from response verbosity, modern preference optimization methods are increasingly adopting length normalization (e.g., SimPO, ORPO, LN-DPO). While effective against this bias, we demonstrate that length normalization itself introduces a failure mode: the **URSLA shortcut**. Here models learn to satisfy the alignment objective by prematurely truncating low-quality responses rather than learning from their semantic content. To address this, we introduce **REFA**, a new alignment framework that proposes probabilistic control on a structural token that controls termination. Our core innovation is a new class of regularizers that operate directly on the probability of the End-of-Sequence (EOS) token, a previously unexploited control lever. This token-level intervention provides a principled solution to the URSLA shortcut, ensuring genuine quality improvements. Furthermore, it unlocks a versatile mechanism for managing the alignment-efficiency tradeoff, enabling practitioners to fine-tune models that adhere to specific token budgets. Empirically, REFA achieves a **60.29\%** win rate and a **52.17\%** length-controlled win rate on AlpacaEval2 with Llama-3-8B-Instruct, demonstrating the power of our token-level control paradigm.

Published as a conference paper at COLM 2025
REFA: Reference Free Alignment with Fine-Grained Length Control
Taneesh Gupta*
Rahul Madhavan
Xuchao Zhang
Chetan Bansal*
Saravan Rajmohan"
Abstract
To mitigate reward hacking from response verbosity, modern preference optimization methods are increasingly adopting length normalization (e.g., SimPO, ORPO, LN-DPO). While effective against this bias, we demonstrate that length normalization itself introduces failure mode: the URSLA shortcut. Here models learn to satisfy the alignment objective by prema- turely truncating low-quality responses rather than learning from their semantic content. To address this, we introduce REFA, new alignment framework that proposes probabílistic control on a structural token that controls termination. Our core innovation is a new class of regularizers that operate directly on the probability of the End-of-Sequence (EOS) to- ken, previously unexploited control lever. This token-level intervention provides a principled solution to the URSLA shortcut, ensuring genuine quality improvements. Furthermore, it unlocks versatile mechanism tradeoff, enabling practitioners to
for managing the alignment-efficiene fine-tune models that adhere to specific token budgets. Empirically, REFA achieves a 60.29% win rate and a 52.17% length-controlled win rate on AlpacaEval2 with Llama-3-8B-Instrucc demonstrating the power of our token-level control paradigm.
1
Introduction
The twin goals of modern Large Language Model (LLM) research are enhancing alignment with human values and improving computational efficiency for practical deployment. While alignment is advanced through preference optimization techniques like DPO (Rafailov et al., 2024b; Xu et al., 2024b; Wang et al., 2023; Ouyang et al., 2022), efficiency is often pursued via model-centric approaches such as quantization (Dettmers et al., 2023), distillation (Shridhar et al., 2023; Agarwal et al., 2023), and architectural innovations that reduce computational load (e.g., Matryoshka representations (Kusupati al., 2022; Devvrit et al., 2024)). In this work, we argue for complementary axis of efficiency: managing the inference-time budget by controlling output token length through post-training alignment. The ability to precisely regulate response verbosity is critical tool for building cost-effective and reeeeo
responsive LLM-powered systems (Wang et al., 2024b; Erol et al., 2025).
However, achieving such control is complicated by classic problem: reward hacking. In preference datasets, rewards often correlate spuriously with length, creating an incentive for models to generate longer responses to maximize their score (Gao et al., 2023). To counter this, a growing class preference optimization methods (e.g., SimPO, ORPO, LN-DPO) incorporates length normalization directly into their objective. Yet, this fix for one form of reward hacking inadvertently creates another. As we formalize in Section 3, this leads to the URSLA shortcut: new failure mode where models learn to satisfy the alignment objective by exploiting a statistical property of language generation to prematurely truncate low-quality responses. This presents a dichotomy: the very mechanism needed to prevent one length-based exploit enables another, making robust length control elusive.
'Equal contribution.
"Microsoft
Indian Institute of Science (ISSc,, Bangalore

---

Published as a conference paper at COLM 2025
We argue this challenge stems from the coarse-grained nature of current preference objec- tives, which not only treat sequence probabilities as opaque scores but are often limited to single preference pairs (Zhong et al., 2024; Zeng et al., 2024). To achieve robust alignment and practical control, we introduce REFA, new alignment framework that proposes proba- bilistic control on structural token that governs termination. Our framework is designed to leverage the full spectrum of available data by operating on sets of preferred and rejected responses (Gupta et al., 2025). The core innovation a new class of regularizers that operate directly on the probability of the End-of-Sequence (EOS) token. Viewing generation as se- quential pecision-mmkingg process (Rafailov al., 2024a), the EOS token acts as the transition to a "terminal state" in the MDP, making it a uniquely powerful point of intervention. This token-level control provides a unified solution: it directly counteracts the URSLA shortcut and transforms output length into controllable parameter.
This principled control, achieved training time, enables REFA to serve as a versatile method for managing the alignment-efficience tradeoff. The ability to fine-tune model with specific length constraints is crucial for a range of applications. For instance, systems requiring concise outputs, such generating summaries (Stiennon et al., 2020) or social media posts, can be optimized for brevity. Conversely, for complex reasoning tasks where detailed explanations are paramount (Wei et al., 2022; Zhang et al., 2024), REFA can en- courage more verbose and thorough responses. Most critically, for large-scale deployments, our framework allows practitioners to fine-tune models that adhere to strict token budgets, directly managing inference costs and latency without compromising alignment quality.
In this work, we introduce REFA, framework that establishes this novel approach of token- level control. Theoretically, we formalize the URSLA shortcut as fundamental limitation of coarse-grained preference optimization. Algorithmically, we pioneer a new class of EOS probability regularizers that provide fine-grained, bidirectional length control. Empirically, this new approach enables REFA achieve state-of-the-art results in iterative, on-policy alignment, reaching 60.29% win rate on AlpacaFval2 with 1ama-3-88-Instrus more we demonstrate REFA's practical utility through budgeted alignment, showcasing its ability to steer generation towards specific length targets, bridging the gap between alignment quality and deployment efficiency.
Further-
We now outline our key contributions in this work:
1. Theoretical Insight: Formalizing the URSLA Shortcut. We identify and formalize the Uncertainty Reduction with Sequence Length Assertion (URSLA) shortcut, a critical failure mode inherent in reference-free alignment methods that use length normalization. We prove that under the URSLA principle, these models are incentivized to shorten negative responses trivial means of satisfying the loss, highlighting a fundamental limitation of purely sequence-level optimization.
Algorithmic Novelty: A New Paradigm of Token-Level Control. We pioneer a new approach to preference optimization by intervening directly at the token level. We propose a new class of EOS probability regularizers that act on the model's termination decision, previously unexploited control lever. This mechanism provides a principled solution to the URSLA shortcut and establishes a new paradigm for fine-grained control over generative properties.
3. A Practical Toolkit for Budgeted Alignment. We demonstrate that our token-level intervention is not just a theoretical fix but also a versatile mechanism for managing the crucial alignment-efficiency tradeoff. We develop and validate a family of regulariz- ers that provide practitioners with explicit, bidirectional control over response length, enabling the fine-tuning of models that adhere to specific inference-time token budgets.
4. State-of-the-Are Alignment Performance. We show that by resolving these underlying instabilities, REFA achieves state-of-the-an performance. In challenging iterative and on- policy settings, REFA significantly outperforms strong baselines, achieving 60.29% win rate and a 52.17% length-controln win rate on AlpacaEval2 with Hlama-3-SB-Insiee validating the effectiveness of our approach.
2

---

Published as a conference paper at COLM 2025
2
Related Work
The alignment of Large Language Models (LLMs) has rapidly evolved from early Reinforce- ment Learning from Human Feedback (RLHF) techniques (Ouyang al., 2022) to simpler, more stable Direct Preference Optimization (DPO) methods (Rafailov et al., 2024b). Our work builds upon three key trends that have emerged from this paradigm shift.
Reference-Free and Multi-Preference Optimization. To improve simplicity and data efficiency, the field is moving beyond the reference-based, pairwise formulation of DPO. Reference-free methods like SimPO (Meng et al., 2024) and ORPO (Hong al., 2024b) eliminate the need for fixed reference policy, optimizing the model's log-probabilities directly. Concurrently, multi-preference methods like InfoNCA (Chen et al., 2024a) and MPO (Gupta et al., 2025) leverage modern datasets with multiple graded responses per query (Cui et al., 2023), enabling a richer, set-wise contrast that better approximates the true preference landscape. REFÁ operates at this intersection, proposing reference-free, multi-preference objective.
Reference-Free Alignment: A common component in many alignment techniques, includ- ing original DPO, is a fixed reference model representing the policy before fine-tuning. Managing this reference model adds complexity, leading to growing interest in reference-free methods (Meng et al., 2024; Yuan et al., 2023; Zhao et al., 2023a). These approaches optimize the policy's likelihoods directly, often leveraging richer feedback signals such as scalar rewards available in datasets like UltraFeedback (Cui et al., 2023). This simplification can make the training pipeline more robust and efficient.
Length as a Failure Mode for Alignment. A critical challenge in preference optimization is the tendency for models to exploit response length as spurious correlate for reward. While the initial problem was a bias towards verbosity (Gao et al., 2023), the standard solution-length
a subtle but severe second-order problem, which we term the URSLA shortcut. Prior works have sought to manage length through sequence-level mechanisms like explicit regularization (Park et al., 2024) or by directly normalizing the objective (Meng et al., 2024; Ahrabian et al., 2024). REFA contributes a new approach by arguing that robust control requires moving beyond the sequence level to intervene directly on the model's termination logic at the token level.
cermalization-introodie
Positioning REFA. REFA synthesizes these advancements into a single, coherent frame- work. It is reference-free, multi-preference alignment method designed specifically to address the fundamental challenge of length-based reward hacking. Unlike prior works that apply coarse, sequence-level corrections, REFA introduces novel approach to fine-grained control via an End-of-Seeue End-of-Sequence (EOS) probability regularizer. By identifying and solving the subtle failure modes of length normalization, REFA not only achieves more robust alignment but also provides practical framework for managing the inference-time budget, directly connecting the goals of preference alignment and computational efficiency.
We provide full details of our related work in Section B.
3 The Challenge: A Subtle Length-Based Shortcut in Reference-Free Alignment
The paradigm of preference optimization for aligning Large Language Models (LLMs) has seen significant shift from reference-based methods like Direct Preference Optimization (DPO) (Rafailov et al., 2024b) towards simpler, more direct reference-free free approaches (Meng et al., 2024; Hong et al., 2024b). While eliminating the reference model reduces complexity and can improve performance, it exposes new set of challenges related to the inherent statistical properties of language and the dynamics of the optimization process itself. This section first establishes the necessary notation and then identifies critical, second-order problem that arises from the standard solution to length bias, motivating the need for more sophisticated approach to length control.
3

---

Published as a conference paper at COLM 2025
3.1 Notation and Preliminaries
Let D be a preference dataset consisting of tuples (x, ry) ,), where x is a query, [yi], is set of K candidate responses, and r; E IR is scalar reward indicating the quality of response yi. Our goal is to fine-tune policy model ni to increase the likelihood of high-reward responses.
The mean reward for given query f p= k We partition the responses into positive set Y+ = {y; r; > 8} and a negative set Y - {y: r; S P]. key quantity in reference- free optimization is the policy's log pog-robability of response, log To (y(()))) x)) Since raw log-probal prgpoabaailittes are biased towards shorter sequences (as they represent sum over fewer negative log-pne pa--obbability terms), reference-free methods commonly employ the length-normalized log-probability:
1
y
log (y
x)
log to x)
log Poly: x,y<r),
(1)
lyl
where ly| is the number of tokens in response y. This metric reflects the average per-token confidence of the model.
3.2 The Reference-Free Dilemma: Length Normalization and the URSLA Shortcut
Reference-free alignment objectives, such as that of SimPO (Meng et al., 2024), typically aim to maximize the margin between the scores of preferred and dis- preferred responses. Using the length-normalizer ag-probability (Eq. 1) as the score is a necessary first step to prevent the model from trivially favoring longer responses, which are often correlated with higher rewards in preference datasets (Dubois et al., 2024).
Avenge Log-Poplexity
0.00
Mistri-llaee (IB)
Llmal Bast (83
0.65
0.30
8.40
IPY): CPyybo [Pylo Dillea 1Ply Layl
Token Position
However, this very solution introduces more sub- tle and problematic failure mode. The issue stems from fundamental property of auto- auto-regressive lan- guage generation: the relationship between sequence length and model uncertainty. We formalize this re- lationship in the following conjecture.
Figure 1: Empirical analysis support- ing the URSLA conjecture (Conjec- ture 1) on Ultrafeedback. The plot shows decreasing trend in average per-token negative log- pog-probability (uncertainty) as sequence length increases for both Mistral-7B and Llama-3-8B base models.
Conjecture 1 (Uncertainty Reduction with Se- quence Length Assertion (URSLA)). Let LP(y) = -log (y x) be the average per-token negative log-probability (a measure of model uncertainty or perplexity) for response y. There exists non-empty subset of coherent responses )" y such that for all y E )", increasing the length of y reduces its average per-token uncertainty. Formally:
aLP a len
<0 for all e.
(2)
The URSLA conjecture, empirically supported by Figure 1, posits that model uncertainty decreases for longer coherent sequences. This interaction with a length-normalized objective creates a critical vulnerability. Since preference optimization objectives seek to reduce the score log To(y) x) for any negative response y E Y-, URSLA provides a trivial, non-semantic path to achieve this: by simply reducing the response's length.
This leads to an undesirable optimization shortcut: the model can satisfy the loss by prematurely terminating rejected sequences, rather than learning the semantic reasons for their low quality. This reward- wwwarddhacking behavior undermines genuine alignment, as the model exploits statistical artifact of length instead of improving its understanding of
4

---

Published as a conference paper at COLM 2025
quality. Therefore, while length normalization is prerequisite for reference-free alignment, is insufficient on its own. It inadvertently creates a new failure mode that must be addressed with explicit length control mechanism to ensure genuine alignment.
Developing the REFA Framework
Having established the critical need for explicit length control in reference-free alignment, we now develop the REFA (Reference-Pree Aligmment) framework. Our approach begins by contextualizing REFA within the landscape of existing preference optimization objectives. We then systematically construct a loss function that not only leverages the strengths of prior work but also possesses a theoretically superior alignment objective.
4.1 Technical Preliminaries: Key Preference Optimization Objectives
The design of REFA builds upon several key ideas in modern preference optimization. We briefly present the objectives for SimPO, InfoNCA, and MPO, their components and limitations directly motivate our approach, using the notation established in Section 3.1.
Simple Preference Optimization (SimPO). SimPO (Meng et al., 2024) is prominent reference-free method that operates on preference pairs (yw, Vr) Its key innovation is using the length-normalized pog-probability the margin between the scores of the winning and losing responses:
implicit reward signal. The objective maximizes
LsimPO -
[x
log or (Blog To (y - Blog - - ->]]
(3)
where is the sigmoid function and 7 is margin hyperparameter. While effective and reference-free, SimPO's primary limitation is its restriction to pairwise comparisons.
InfoNCA. InfoNCA (Chen et al., 2024a) addresses the multi multi--peeferecc setting by leverag- ing all K > 2 responses. It is a reference-based method that minimizes the cross-entropy between reward-derived target distribution (p) tprarget e'i) and model distribution derived from the policy-reeerecc ratio (pimodel o Ti (y:l1)))/ Ttee/Teeryy x)) Tts loss is:
LInONNCA (0; TTer) = Ex-D
target log p.
model
(4)
InfoNCA's strength its ability to use all response data, but requires reference model and aligns to an explicit target distribution, which, as we will show, can be a suboptimal objective for alignment.
Multi-Preference Optimization (MPO). MPO (Gupta et al., 2025) also utilizes K > responses but employs a group-contrastive objective. It contrasts the set of positive responses Y+ against all responses Y. MPO introduces deviation-based weighting, wy = ea(ry-?), to emphasize more informative examples. Using a reference-based score so(y(x) = log( (ylx) / (y(x)), its loss is:
Eyer+ Wy exp((so(y)x))
LmPo (Tef)) EEED log Eyer Wy (xp((s((v|x))
MPO's core contributions are its group-contrastive formulation and deviation weighting, though its original formulation retains a reference model. REFA aims to synthesize the strengths of these approaches within a single, coherent, reference-free framework that also solves the URSLA shortcut.
4.2 REFA Variants: From Distribution Matching to Contrastive Alignment
We develop the REFA framework by first analyzing reference-free adaptation of InfoNCA and then proposing our primary method, REFA-Dynamic, which employs a more powerful contrastive objective.

---

Published as a conference paper at COLM 2025
REFA-InfONCA. As baseline for reference-free multtipreefernce alignment, first consider a version of InfoNCA without a reference model, which we term REFA-InfONCA. It optimizes cross-entropy objective to align the model's output distribution (pmoodel Toly; x)) with target distribution derived from rewards. While simple, its stationary point occurs when pmodel plarget, This can be suboptimal for alignment, may require the model to continue assigning non-trivial probability to undesirable responses. A detailed analysis is provided in Appendix F.
REFA-Dynamic (Our Primary Method). To create a stronger alignment signal, we in- troduce REFA-Dynamic. This method uses a crotrsccntrasttvv loss that generalizes the Bradley-Terry model to sets of responses in a reference-free setting. Let the base score for response be its scaled, length-normalized log-probability, syy) = B log (y(x), where & is inverse temperature. The REFA-Dynamic loss is:
Ever+ ev)
LREFA-Dynamic (0) = - log
(6)
Eyey+ eyl)) Y Eyer- ely)"
Here, y > 0 adjusts the penalty for the negative set Y- This formulation directly addresses key requirements for robust alignment:
Suppression of Negative Responses: By placing responses from Y- only in the denomi- nator, the objective directly incentivizes reducing their assigned probabilities.
Collective Uplift of Positive Responses: All positive responses in Y+ are upweighted in the numerator, reducing the internal competition that arises from purely pairwise or cross-entropy objectives.
Crucially, this contrastive objective possesses a more desirable stationary point. As shown in Appendix I, the loss minimized as the probabilities of all rejected responses approach zero (P& (ylx) + for all y E Y-), offering theoretically superior objective compared to distribution matching.
W-REFA (Weighted REFA-Dynamice For off-policy settings with reliable, fine-grained reward scores, the binary partition can be enhanced. The W-REFA variant incorporates deviation-based weighting. The score for each response is additively adjusted: swid (y) = s(y) + (r, - r), where a is a hyperparameter. This variant is particularly effective in our off-policy experiments, while the unweighted REFA-Dynamic shows strong performance in on-policy settings where reward model noise a greater concern. The full formulation is detailed in the appendix.
Fine-Grained Length Control via EOS Regularization
The REFA framework developed in the previous section provides a robust, multi-preference objective for reference-free alignment. However, established in Section the length- normalized scores it relies on are still vulnerable to the URSLA shortcut. this section, we introduce our primary algorithmic novelty: family of And-of-Sequenee (EOS) probability regularizers designed to counteract this failure mode and provide versatile mechanism for fine-grained length control.
5.1 Counteracting the URSLA Shortcut with Targeted Regularization
The core problem identified is that models can learn to shorten negative responses to satisfy the loss. To prevent this, we introduce regularizer that penalizes premature termination.
Our primary mechanism, the Targeted Regularizer, applies a penalty to the EOS probability of response, where the penalty's magnitude is proportional to how much shorter the response is than given target length. In our experiments, we dynamically set this target to the maximum response length within each query, which effectively encourages the model

---

Published as a conference paper at COLM 2025
to avoid truncating any response relative to its peers. The regularizer is formulated as:
target = Σ
APa(EOS [yl) max(0, TARGET.LENGTH [yl),
yeytuY
where A > 0 is hyperparameter scaling the strength of the regularization. By applying this penalty, we disincentivize the model from placing the EOS token early for responses in Y-, directly counteracting the URSLA shortcut, which is especially apparent when we have high variance in the response length, for instance in off-policy responses.
The Final REFA Loss. By integrating this regularizer with the REFA-Dynamic objective from Eq. 6, we arrive at the final loss function for primary method:
Eyer+ ey(y)
LREFA (0) = -oo
EyeY+ ( + T esly + Rargget(O).
(8)
This complete objective synergistically combines a theoretically sound multi-preference contrast with targeted mechanism for robust length control, ensuring that alignment is achieved through genuine quality improvements. While this dynamic target Rarget (0) is adaptive to peer responses, we note that more adaptive strategies per domain, per context for setting the target length could be explored in future work. The full training procedure is detailed in Appendix C.
5.2 A Set of Tools for Bidirectional and Budgeted Length Control
Beyond solving the URSLA shortcut by encouraging longer responses, the principle of EOS regularization provides versatile method for controlling output length either direction to meet specific application needs. We introduce two additional regularizers for such scenarios.
Budget-Independen Regularizer (for Decreasing Length). For applications where con- ciseness is desired (e-g,, summarization, tweet generation), the Budget-Independent Regu- larizer can be employed. It is formulated as a penalty on the negative bog-probability of the EOS token:
R Rindep (0) = Σ
A log Pa (EOS at position ly/).
yEY+UY-
By minimizing this term, the model is encouraged to assign higher probability to the EOS token its naturally occurring position, promoting more confident and timely termination, which typically leads to shorter, more concise responses.
Budgeted Regularizer (for Strict Budget Adherence). For scenarios with strict token budgets due to economic costs platform constraints, the Budgeted Regularizer offers the most fine-grained control. It is designed to penalize premature termination before specified budget b while incentivizing termination at or after the budget. The formulation creates a "push-pull" dynamic around the budget:
( カ-1
v Σ 1=++1
f))
budgetted (0)
Σ
Po (EOS at token f)
PEOS at token
(10)
The first term penalizes the average EOS probability mass before the budget, discouraging early stops. The second term, with its negative sign, effectively rewards EOS probability mass after the budget. This allows practitioners to precisely steer the model's output length to conform to target budget, directly managing the accuracy-efficiency tradeoff. The effectiveness of these regularizers is demonstrated empirically in Appendix A.
Experiments
6.1 Experimental Setup
We benchmark REFA on Mistral-7B and Llama-3-8P 3-BB across three distinct training paradigms: offline, on-policy, and iterative. Appendix provides full description of our baselines and experimental configurations.
7

---

Published as a conference paper at COLM 2025
Mistral-Base (7B)
Llama-3-Base (8B)
Method
AlpacaEval 2
Arena-Hard
MT-Bench
AlpacaEval 2
Arena-Hard
MT-Beach
LC (%)
WR (%)
WR (%)
GPT4
LC (%)
WR (%)
WR (%)
GPT-4
SFT'
8.4
6.2
1.3
6.3
6.2
4.6
3.3
6.6
RRHFI
11.6
10.2
5.8
6.7
12.1
10.1
6.3
7.0
SLIC-HI
10.9
8.9
7.3
7.4
12.3
13.7
6.0
7.6
DPO
15.1
12.3
10.4
73
18.2
15.5
15.9
7.7
IPO
10.8
9.0
7.5
7.0
144
142
17.8
744
CPO
9.8
10.8
7.4
KTOP
13.1
6.8
14.2
7.8
ORPO
14.7
7.0
12.2
7.6
R-DPO
17.4
7.0
17.6
7.5
MPO
20.3
14.9
73
20.1
15.6
18.5
7.8
SimPO
21.5
20.8
16.6
73
22.0
20.3
23.4
7.7
REFA-IniONCA
20.4
19.1
14.2
7.1
22.8
17.1
20.2
7.5
REFA-I-US-k
20.4
19.1
7.1
25.5
7.5
Rera-dynamic
22.8
20.4
15.7
7.3
25.6
23.1
25.4
7.8
W-REFA
24.6
23.0
16.6
7.4
26.6
24.2
23.7
7.8
Table 1: Comparison of various preference optimization under off-policy setting for base- lines on AlpacaBval, Arena-Hard and MT-Bench benchmarks. LC-WR represents length- controlled win rate, and WR represents raw win rate. Best results in bold, second-best are underlined. REFA achieves SOTA performance across all metrics
Offline Training. we follow the pipeline established by Zephyr (Tunstall et al., 2023). We first create SFT models by training the base models on UltraChat-200k (Cui et al., 2023). These SFT checkpoints are then used to initialize REFA for preference optimization the static UltraFeedback dataset (Cui et al., 2023).
On-policy and Iterative Training. Our on-policy experiments begin with instruction-tuned models Llama-3-8B-Innse with the policy, synthesize responses for each UltraFeedback prompt (5 candidates, temp=1.0), similar to Meng et al. (2024). These candidates are scored by the Skywork- Reward-Llama-3.1 model Liu et al. (2024a), from which we select the top-2 and bottom-2 responses to form the preferred (Y+) and rejected (Y-) sets for REFA.
and Mistral-7B-Instrucc. To generate preference data aligned
This on on-policy data generation is also integrated into iterative framework inspired by SPPO (Wu et al., 2024). The UltraFeedback prompt set is partitioned, and at each round, the current model checkpoint generates fresh responses to scored and selected. This newly synthesized data is then used to update the model with REFA before the next iteration. More details are provided in Appendix
6.2 Experimental Results
Mistrrl-Instrrnt (7B)
Llama-3-Instrune (8B)
Method
LC (%)
WR (%)
Arena- Hard
MT-Bench
LC (C())
WR (%)
Arena-Han
MT- Bench
SPPO! (Iter !)
24.79
23.51
18.7
7.21
31.73
31.74
34.8
8.1
REFA (Iter
27.58
28.79
25.3
7.56
37.42
38.72
34.10
7.85
SPPO (Iter 2)
26.89
27. 62
20.4
7.49
35.15
35.98
38.0
8.2
REFA (Iter 2)
31.57
36.85
47.88
50.83
42,20
8.12
SPPO! (Iter 3)
28.53
31.02
23.3
7.59
38.78
39.85
40.1
8.2
REFA (Iter 34.87
40.85
23.8
7.62
52.17
60.29
52.70
8.16
Table 2: Results on Mistral-Instruct and Llama-Instruct models in the iterative on-policy setting with Ultrafeedback prompts.
These are taken directly from the paper SIMPo(Meng et al., 2024), MPO (Gupta et al., 2025), SPPO (Wu al., 2024)
8

---

Published as a conference paper at COLM 2025
REFA-dynamic outperforms existing preference baselines: We conducted extensive comparison of the REFA-Dynamic framework against various preference optimization methods using the Mistral-7B and Llama3-8b for offline setting. Mistrrl--7B-Instruc and Llamm3-8b-Instruch for online and iterative settings, as summarized in Table The results demonstrate that REFA-Dynamic consistently outperforms all baselines across the primary evaluation metrics. Key Insights: This highlight the efficacy of REFA-Dynamic in enhancing response quality, reinforcing its state-of-the-at performance in preference optimization.
Mistral-Instruct (7B)
Llama-Instruct (8B)
Method
AlpacaEval 2
Arena-Hard
MT-Bench
AlpacaEval 2
Arena-Hard
MI-Bench
LC (%)
WR (%)
WR (%)
CPT-
LC (%)
WR (%)
WR (%)
GPT-4
SFT
17.1
14.7
12.6
7.5
26.0
25.3
22.3
8.1
DPO!
26.8
24.9
16.3
7.6
40.3
37.9
32.6
8.0
IPO
20.3
20.3
16.2
7.8
35.6
35.6
30.5
8.3
KTO!
24.5
23.6
17.9
7.7
33.1
31.8
26.4
8.2
ORPO
20.8
7.7
28.5
R-DPO
27.3
24.5
16.1
7.5
41.1
37.8
33.1
8.0
SimPO
32.1
34.8
21.0
7.6
44.7
40.5
33.8
8.0
REFA-InfONCA
30.2
32.1
22.7
7.6
42.8
43.5
38.9
8.0
REFA-lvsk
33.1
36.3
26.5
7.7
49.6
50.2
43.2
8.1
RFFA-Dynamic
32.5
25.9
8.1
Table 3: Comparison of different alignment methods in an on-policy setting across multiple evaluation benchmarks on Mistrrl-Instruuc and Llama-Instruct models.
Effect of EOS Probability Regularization on REFA-dynamic: We analyzed the effect of the EOS probability regularization term À on the performance of REFA-dynamic on AlpacaEval2 for Mistral-7B-Bass Fig. 2. The results reveal that incorporating À significantly improves model performance. As À increases, we observe a notable rise in the Average Response Length, suggesting that higher regularization encourages longer responses.
offline-setting with different values of A, as shown in
Key findings: Importance careful tuning of À to balance verbosity and response quality, with moderate values (e.g., A 5e-5) yielding the best overall trade-off across all metrics. Our theoretical analysis in (Appendix j.4) supports this by demonstrating that regularizing the EOS probability stabilizes sequence generation, reducing overly brief responses.
25
25
Average Length
5500
4500
LC-WR%
20
WR%
20
3500
15
2500
1500
10
10
1c.4
5c-5
1e.4
5e-5
1c.4
Regularization(A)
Regularization(A)
Regularization(A)
Figure 2: Impact of À on Mistral-Bar (7B) Performance: a) Average Response Length, b) Length-Controlled Win Rate (LC-WR), and c) Win Rate (WR).
Effect of Regularization on Token Effi-
ciency in REFA Figure 3 shows REFA's effectiveness in controlling response length. Subfigure (a) indicates REFA yields a more balanced length distribution compared to the potential verbosity of InfoNCA or terse- ness of SimPO. Subfigure (b) confirms that increasing REFA's regularization coef- ficient À systematically increases average output length, demonstrating À provides fine-grained control over verbosity.
Lande WAR
Figure 3: Analysis of token distribution in REFA: (a) across alignment methods (In- foNCA, REFA, and SIMPO). (b) across regular- ization coefficient for the REFA method.

---

Published as a conference paper at COLM 2025
7 Budgeted Length Control
To rigorously evaluate the tradeoff between token efficiency and accuracy, conducted two key ablation studies focusing on using End-of-Sequence (EOS) token regularization for token reduction. These experiments systematically isolate the impact of different regularization strategies on model performance and output length.
We conducted below experiments on subset of UltraFeedback dataset in online-setting for Llama3-8b-instree and used AlpacaEval as the downstream evaluation dataset. More extensive experiments are being provided in Appendix A
7.1 Budget-Indepeendnt Regularization
This experiment investigates the impact of budget-independent regularizer Rindep (0) strength, controlled by the hyperparameter A. This regularization, Rindep (0)
log Pe (EOS at position ly]), is designed to increase the log-probability of the EOS token. By varying Å, we evaluated the resulting tradeoff between verbosity and quality, The results demonstrate effective control over response length; as increases, the average token count decreases, and the entire length distribution shifts towards shorter and precise outputs (Figure 4). Critically, this ablation reveals a non-monotonic relationship between regularization strength and performance.
Lambda LC-WR (%))
Lambda vs WR (%)
Lambda vs Avg Tokens
16-
1842
Le-2
14:2
11-2
16.
30-2
Figure Performance (LC-WR, WR) and Average Tokens vs. A for the budget-independent EOS regularizer on AlpacaEval2. Note the peak LC-WR at A v0 0.1.
7.2 Budgeted Regularization
To evaluate the efficacy of precise, targeted length control, we investigates our novel "bud- geted" EOS regularizer Rbudgeted (f). This experiment demonstrates the regularization strength A across several pre-defined token budgets b {128, 256, 184,5122, The for- mulation in eq.10 explicitly penalizes premature termination (EOS before budget b) and incentivizes termination at after it.
Lamhda vs LC-WR (%)
Lambda WR (R.(%)
Lamibáa Avp Tokass
LCC.R. (5))
30
Average Tokens
30
500
20
20-
20
40
Lambda
Lambda
Lumbda
Figure 5: Performance (LC-WR, WR) and Average Tokens vs. A for the budgeted EOS
regularizer with a target budget of b = 256 on AlpacaEval2.
This approach demonstrates highly precise and effective length control. As À increases, the average token count of generated responses converges directly towards the target budget shown in figure 5. A consistent accuracy-efficiency tradeoff is observed across all budgets: optimal LC-WR is achieved at a moderate A, while overly strict enforcement degrades alignment quality.
10

---

Published as a conference paper at COLM 2025
References
Rishabh Agarwal, Nino Vieillard, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. Gkd: Generalized knowledge distillation for auto-regressive sequence models. CoRR, 2023.
Kian Ahrabian, Xihui Lin, Barun Patra, Vishrav Chaudhary, Alon Benhaim, Jay Pujara, and Xia Song. A practical analysis of human alignment with po. arXiv preprint MiXi::247.15229, 2024.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. ArXiv, abs/ 2300.//360..2 2023.
Guangji Bai, Zheng Chai, Chen Ling, Shiyu Wang, Jiaying Lu, Nan Zhang, Tingwei Shi, Ziyang Yu, Mengdan Zhu, Yifei Zhang, et al. Beyond efficiency: A systematic survey of resource-efficient large language models. arviv preprint arXiv:2401.00625, 2024.
Huayu Chen, Guande He, Lifan Yuan, Ganqu Cui, Hang Su, and Jun Zhu. Noise contrastive alignment of language models with explicit rewards. arXiv preprint arXio:2402.05369, 2024a.
Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf. arXiv preprint ar #rXio:2402.07319 2024b.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high- quality feedback. arXiv preprint ar arXi:22310.0137 2023.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088- 10115, 2023.
Fnu Devvrit, Sneha Kudugunta, Aditya Kusupati, Tim Dettmers, Kaifeng Chen, Inderjit Dhillon, Yulia Tsvetkov, Hanna Hajishirzi, Sham Kakade, Ali Farhadi, et al. Matformer: Nested transformer for elastic inference. Advances in Neural Information Processing Systems, 97:140535-140564 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint ar arXiv:2404.04477 044454454, 2024.
Mehmet Hamza Erol, Batu EI, Mirac Suzgun, Mert Yuksekgonul, and James Zou. Cost- of-pass: An economic framework for evaluating language models. arviv preprint arXiv:2504.13355 2025.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: Model alignment prospect theoretic optimization. ArXiv, abs/ 240200/000. 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, et al. Controllable preference optimization: Toward controllable multi-objective alignment. arXiv preprint ar arXio:2402.19085, 2024.
Taneesh Gupta, Rahul Madhavan, Xuchao Zhang, Nagarajan and Saravan Rajmohan. Multi-preference optimizationn Generalizing dpo via set-level contrasts, 2025. URL
EagarajanNNateenn Natarajan, Chetan Bansal,
04628.
Jiwoo Hong, Noah Lee, and James Thome. ORPO: Monolithic preference optimization without reference model. ArXiv, abs/2200.0066) 2024a.
11

---

Published as a conference paper at COLM 2025
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11170-11189, 2024b.
Methoods
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard -award-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning. Advances in Neural Information Processing Systems, 35: 30233-30249, 2022.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Gengxu Li, Tingyu Xia, Yi Chang, and Yuan Wu. Length-controlled margin-based preference optimization without reference model. arXiv preprint ar afXf::2502.14643, 2025.
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-rewards Bag of tricks for reward modeling in llms. arxiv preprint Xiv:24410 188451, 2024a.
Yang Liu, Jiahuan Cao, Chongyu Liu, Kai Ding, and Lianwen Jin. Datasets for large language models: A comprehensive survey. arXiv preprint ar arx::22402.1804 2024b.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Xiv preprint ar arXiv:2405.14734 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. ArXiv, abs/ /000//2403.1155 2024.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q": Your language model is secretly a q-function. arviv preprint arXit::4444112358, 2024a.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xrv preprint ar XrXiv:1707.063// 2017.
Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. Distilling reasoning capabilities into smaller language models. Findings of the Association for Computational Linguistics: ACL. 2023, pp. 7059-7073, 2023.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint ar -xX:::2402205749 2024.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXivo2310.1694: 2023.
Chaoqi Wang, Zhuokai Zhao, Chen Zhu, Karthik Abinav Sankararaman, Michal Valko, Xuefei Cao, Zhaorun Chen, Madian Khabsa, Yuxin Chen, Hao Ma, et al. Preference optimization with multi-sample comparisons. arx preprint arXio:2410.12138. 2024a.
12

---

Published as a conference paper at COLM 2025
Wenxiao Wang, Wei Chen, Yicong Luo, Yongliu Long, Zhengkai Lin, Liye Zhang, Binbin Lin, Deng Cai, and Xiaofei He. Model compression and efficient inference for large language models: A survey. arXiv preprint ar arXtt::4402.09745 2024b.
Yuanhao Wang, Qinghua Liu, and Chi Jin. Is rlhf more difficult than standard rl? theoretical perspective. Advances in Neural Information Processing Systems, 36.76006-76032, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-248877 2022.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arviv preprint ar arXic:2405.00677 2024.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. ArXiv, abs/ abs/2401 /401.00418 2024a.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for Ilm alignment? comprehensive study, arviv preprint ar Xiw:2404.107199 2024b.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint ar Xiv:2304.00300, 2023.
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. arXiv preprint arXiv:2404. 11999, 2024.
Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. Chain of prefer- ence optimization: Improving chain-of-thought reasoning in llms. Advances in Neural Information Processing Systems, 37:333-355 -366, 2024.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. SLiC-HF: Sequence likelihood calibration with human feedback. ArXiv, abs/2305.10425, 2023a.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic- hf: Sequence likelihood calibration with human feedback. ar preprint arXio:2305.10425, 2023b.
Han Zhong, Zikang Shan, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arxiv preprint ar arXiv:2404.18922 2024.
Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond nhe-preference-fo-ale Multi-objective direct preference optimization. arxiv preprint ar xrXi::2310.03008, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences. ar Xio preprint ar Xiv:1909.08593, 2019.
13

---

Published as a conference paper at COLM 2025
SUPPLEMENTARY MATERIALS
These supplementary materials provide additional details, derivations, and experimental results for our paper. The appendix is organized as follows:
Section A provides additional experiments for fine-grained length control
Section provides detailed overview of related work pertaining this paper.
Section C provides a look on the actual Algorithm used for REFA. This is the primary method studied in Section 6.1
Section D provides a narrative roadmap to the theoretical results that underpin the REFA framework.
Section E analyzes the InfoNCA loss function, deriving its gradients, characterizing its stationary points, and examining the role of the reference model.
Section F provides additional analysis of the InfoNCA loss in reference-free setting, highlighting issues like promoting low-quality responses and generating contradictory optimization signals.
Section G details how addressing identified weaknesses in the reference-free InfoNCA formulation leads to refined, reference-free multi- multi-prreeernce loss function, paving the way for the proposed REFA-base variants.
Section H introduces the REFA-base loss functions without length normalization, explain- ing how we incorporate multiple preferences and reward-based weighting to overcome prior limitations.
Section I focuses on length normalization in REFA, demonstrating how normalizing per-token probabilities prevents trivial short-response biases and encourages genuinely improved responses.
Section J explores the nuances in the relationship between average per-token uncertainty and sequence length, and justifies the addition of an EOS-probability regularizer to counter subtle length-related incentives.
Section K More detailed discussion about the Baselines and Evaluation Benchmarks.
Section L It provides addition ablations done for REFA (Downstream analysis w.nt A, Error-Analysis and statistical-signifiianeee Computational Efficiency)
Section M provides qualitative comparison of responses generated from REFA and SIMPO
A Additional Fine-Grained Length Control Experiments
Lambda vs LC-WR (%)
Lambda vs WR (%)
Lambda vs Avg Tokens
25
362
26/2
163
3.2
15.2
163
300
30.0
280
Figure 6: Performance (LC-WR, WR) and Average Tokens vs. À for the budget-indeppedent EOS regularizer on AlpacaEval2. Note the peak LC-WR at Rd 0.1.
14

---

Published as a conference paper at COLM 2025
A.1 Impact of Regularization on Performance and Length
Figure 6 demonstrates the relationship between the regularization strength X, average response
empppeeettati length, and alignment performance (LC-WR and WR). The results show clear and predictable relationship: as À increases from O, average length steadily decreases. This controlled increase in length initially leads better alignment, with performance peaking at moderate À ~ 0.1. For À > 0.1, performance declines, suggesting excessive regularization leads to overly verbose and suboptimal outputs.
A.2 Impact of Regularization on Length Distribution
The effect of the regularizer on the overall distribution of response lengths is visualized in Figure 7. With no regularization (A = 0), the model produces a distribution of lengths centered around a lower mean. As increases, the entire distribution shifts towards shorter responses, with the mean token count increasing accordingly. This visualization confirms that the regularizer does not just affect the average length but systematically influences the model's overall generative behavior to produce longer sequences. Tekm Duribuur Valaas
A-3
A-O
1-001
2-31
A-10
10
2006
1000
1500
1006 1:30
Telen
16kus Oheza
Takes Crec
Tddes Coant
Token Crenl
Figure/ 7: Token count distribution for varying À values using the budget-independent regularizer. As A increases, the mean of the distribution (red dashed line) shifts towards shorter responses.
A.3 Impact of Budgeted EOS Regularization on Performance and Length
Figures 8, 9, 10, and 11 illustrate the impact of this budgeted EOS regularizer for target length budgets b = 128, b = 256, and b 512 tokens, respectively. The x-axis represents varying strengths of the regularization parameter X Figures 12, 13, 14, and 15 show the distribution of generated response lengths for different actual values for À E {0, 10, 50) and target budgets b E (128, 256, 252, The red dashed line indicates the target budget b.
Lumbda LC- (%)
Lambda vs (%)
Lambda vs Tokens
40
LC-WR (%6)
30
Average Tokems
30-
300
WR
20
20-
40
20
Lambda
Lambda
Lambda
Figure 8: Performance (LC-WR, WR) and Average Tokens vs. A for budget b = 128 AlpacaEval2.
15

---

Published as a conference paper at COLM 2025
Lamhda (%)
Lambda vs (%)
Lambda Avg Tokens
40
LC-WR (%6)
35
Average Tokens
400-
WK (%00
30
30-
300
40
40
20
Lambda
Lambda
Lamb-da
Figure 9: Performance (LC-WR, WR) and Average Tokens vs. A for budget 256 on A AlpacaEval2.
Lumbdu LC. WR (%)
Lambda vs (%)
Lambda Avg Tokens
LC-R (%)
35
35
Average Tokems
440
30
25
25
20
400-
20
Lambda
Lambda
Lambda
Figure 10: Performance (LC-WR, WR) and Average Tokens vs. Ả for budget b 384 on A AlpacaEval2.
Lumbda LC-WR (%)
Lumbda WR (%6
Lambda Avg Tokens
1000
Average Tokems
LC-WR (10)
30
800
25-
25
600
20-
20
40
20
20
40
20
0
Lambda
Lambda
Lambda
Figure 11: Performance (LC-WRR WR) and Average Tokens À for budget b 512 on AlpacaEval2.
From these figures, we observe a clear accuracy-efficiency tradeoff. For each budget b, there is an optimal range for À that balances adherence to the budget with achieving high win rates. As A increases, the average token count generally converges towards the target budget, demonstrating effective length control. However, forcing responses be too strictly confined to short budget, or allowing excessive verbosity with large budget, can be suboptimal for alignment quality. This allows practitioners to make informed decisions based on their specific accuracy versus token count requirements.
A.4 Impact of Budgeted EOS Regularization on Response Length Distribution
Figures -2-15 show the distribution of generated response lengths for different strengths of (A E {0, 20, 20))))) and target budgets 128, 256, 512. The red dashed line indicates the target budget b.
16

---

Published as a conference paper at COLM 2025
Tokez Cozzt Dusmbuttaz Amaua A Vaines for tradact lezat: 123
A-0
A- 33
A-20
Nonter
Tokan Coart
Tokar Cound
Teken Cuasi
Tokes Cuaal
Figure 12: Token count distribution vs. A for budget b = 128 on AlpacaEval2.
bradget
A-
1-30 30
2-20
A-16
Mias
250
Nomte of Quantes
200
100
1146
506
19:0
2000
3000
203
491
890
Figure 13: Token count distribution vs. A for budget b = 256 on AlpacaBval2.
Tikes Court Diamibtticss Arras A Valnes for brdast leeat:: 394
Meas
300
300
1990
2000
510
1950
1600
3000
580
1900
1568
300
Tokes Cours
Tokou Crasd
Tokes Cuasi
Tokes Cuaal
Figure 14: Token count distribution vs. À for budget b = 384 on AlpacaEval2.
Thcer Carrt Diembetier Arroce A Vaines for tradget lengh: 512 A 23
A-D
A-20
250
20.
600
Tolon
Telon
Figure 15: Token count distribution vs. A for budget b = -12 on AlpacaEvalz.
These distributions clearly show that as À increases from 0 to 50, the response lengths shift and concentrate more tightly around the target budget b. This demonstrates the regularizer's 's effectiveness in penalizing premature termination and pushing generations to be longer and closer to the budget. Yet certain challenges remain.
17

---

Published as a conference paper at COLM 2025
A.5 Challenges and Observations with Fixed Budgets Variable-Lengthh Data
The response length distributions (Figures 12-15, particularly for larger budgets like 384 and = 512) suggest interesting interactions when applying a fixed budget to a dataset with inherently high variance in optimal response lengths.
A.5.1 Adaptive Learning within Budget Constraints:
The model attempts to adapt its EOS probabilities based on the budget. For instance, if query naturally warrants response shorter than a large budget b, the regularizer (as formulated in Eq. 10 which penalizes early EOS) might push the model to extend responses, potentially beyond their natural endpoint for that query type is high. Conversely, if the query warrants response much longer than b, the model is incentivized to terminate around or after b.
A.5.2 Potential for Bi-modal or Complex Distributions:
When the inherent optimal lengths for dif- ferent queries in the dataset vary signif- icantly relative to single fixed budget b, the model's attempt to satisfy the bud- geted EOS regularizer across all examples can lead to complex, sometimes bi-modal, length distributions in the aggregate gen- erations. This is because the regulariza- tion pressure acts differently depending on whether an unconstrained optimal response would be shorter or longer than b.
Diaribution of Token Counts Quayy
16000
14000
Number Queries Queries
12000-
10000
8000-
6000
4200
2000
Figure 16 shows the distribution of our training data, which clearly indicates that there are queries both longer, as well as shorter than the budget. This suggests that while fixed budget offers control, for datasets with very diverse length require- ments, more adaptive budgeting strategies careful tuning of À might be needed to avoid overly distorting natural response lengths for all query types. Our original, non-budgeted EOS regularizer (Equation 28 in the appendix of the main paper) offers a more dataset- adaptive approach by regularizing based on the EOS position in the training examples themselves.
Token Count
Figure 16: Histogram of Token count vs. Num- ber of Queries on Ultrafeedback.
These findings underscore the power of EOS probability control for managing length but also highlight the nuanced considerations when applying fixed-length targets diverse datasets. REFA's framework provides the tools for such exploration.
B Extended Related Work
Preference Optimization for Alignment. Early alignment efforts focussed on RLHF through learning an intermediate reward model using reinforcement learning algorithms like PPO (Schulman et al., 2017; Ziegler et al., 2019; Ouyang et al., 2022). While effective, this can be computationally expensive and noisy due to the intermediate reward estimation. Recent approaches, such as Direct Preference Optimization (DPO) (Rafailov et al., 2024b), streamline alignment by directly optimizing a contrastive loss over pairs preferred and dispreferred responses, bypassing explicit reward models. Subsequent works have extended this idea, exploring variants like IPO (Azar et al., 2023), CPO (Xu et al., 2024a), ORPO (Hong et al., 2024a), and R-DPO (Park et al., 2024), each offering alternative formulations ularizations. Additionally, methods like RRHF (Yuan et al., 2023), SLiC-HF (Zhao et al., 2023a), GPO (Tang et al., 2024), KTO (Fthayarajh et al., 2024), and SimPO (Meng et al., 2024)
reg-
18

---

Published as a conference paper at COLM 2025
propose diverse preference optimization objectives, some relying reference models and others operating reference-free.
Reference-Free Alignment. There is growing interest in reference-free methods that can directly be used for post-training without reference model. These approaches avoid the complexity of ratio modeling and can leverage richer data containing multiple candidate responses per query (Cui et al., 2023; Liu et al., 2024b; Zhou et al., 2023; Wang et al., 2024a). Recently, objectives like SimPO (Meng et al., 2024) have shown that focusing on the log- probability of sequences without separate reference model can achieve strong performance, making the training pipeline simpler and potentially more robust.
Multi-Preference Optimization. The development of multi-preference and reference- free approaches is facilitated by datasets like the UltraFeedback dataset (Cui et al., 2023) that provide scalar rewards corresponding to multiple responses related to a single query. Methods like InfoNCA (Chen et al., 2024a) and MPÔ (Gupta et al., 2025) use such datasets to generalize beyond pairwise comparisons, simultaneously considering sets of positive and negative responses. Multi-preference objectives better approximate the true preference landscape, better enabling models to estimate the accepted and rejected distributions. While InfoNCA leverages noist-contrastive framework to align responses according to scalar rewards, MPO introduces deviation-basel weighting to emphasize highly positive or highly negative deviations more strongly, thus improving alignment quality.
Length as a Vector for Reward Hacking. A persistent challenge in preference alignment is the tendency for models to engage in "reward hacking" by exploiting spurious correlations in the training data (Gao et al., 2023). Response length is primary vector for this behavior. has been widely observed that longer, more verbose responses are often preferred by human and model raters, creating naive incentive for models to simply "write more" to achieve a higher score (Chen et al., 2024b). Recognizing this, a growing class of preference optimization methods has emerged to explicitly tackle length bias. One prominent approach is length normalization, where the implicit reward is based on the average per-token log- probability. This is a core feature in reference-free methods like SimPO (Meng et al., 2024) and ORPÓ (Hong et al., 2024b), as well as in dedicated works like LN-DPO (Ahrabian et al., 2024) and LMPO (Li et al., 2025). While these methods successfully counter the bias towards verbosity, our work shows they are vulnerable to a second-orde exploit, like the URSLA shortcut, highlighting that more fundamental approach to termination logic is required.
Efficiency in LLMs: From Model Compression to Inference Budgeting. The practical deployment of LLMs is fundamentally constrained by their computational cost. A signifi- cant body of research focuses on improving efficiency through model-centric approaches. These include reducing model size via quantization (Dettmers et al., 2023), knowledge distillation (Shridhar et al., 2023), and pruning (Bai et al., 2024), as well as developing more efficient architectures like Matryoshka models that allow for adaptive computation (Kusu- pati et al., 2022; Devvrit et al., 2024). A complementary line of work targets inference-centric optimizations like speculative decoding (Leviathan et al., 2023). Our work introduces third, less explored axis that bridges alignment and efficiency: output budget control. By manipulating the generative process at training time to produce outputs of a desired length, we can directly manage the primary drivers of inference cost and latency: the number of generated tokens. REFA contributes to this area by providing principled mechanism to instill this control during the preference alignmeet phase itself.
provid
Positioning REFA. REFA builds upon the principles of reference-free, multi-preference optimization established by methods like MPO and SimPO. However, its primary novelty lies in addressing the critical challenge of length-based reward hacking. Unlike prior methods that apply sequence-level normalization or regularization, REFA introduces novel approach of fine-grained, token-level intervention. By identifying and solving the subtle failure modes introduced by length normalization itself, REFA not only achieves more robust alignment but also provides a practical framework for managing the inference-time budget, directly connecting the goals of preference alignment and computational efficiency.
19

---

Published as a conference paper at COLM 2025
C REFA Training Algorithms
This section provides the detailed pseudocode and a step-by-step explanation for the training procedures of the primary REFA variants: the unweighted REFA-Dynamic and the weighted W-REFA.
C.1 Algorithm for REFA-Dynamic
Algorithm details the procedure for our primary method. This version uses the un- weighted, group-contrastive loss combined with the Targeted Regularizer. It is designed to be robust and is particularly effective in on-policy and iterative settings where reward model scores may be noisy or less calibrated. The core idea is to use the reward signal to partition responses into positive and negative sets, and then optimize a contrastive objective on the model's 's norgth-normalized probabilities, while simultaneously regularizing the model's termination behavior to prevent the URSLA shortcut.
Algorithm 1 REFA-Dynamic Training Procedure
Input: Dataset D {(x, ry: learning rate y, hyperparameters 6, y, A. repeat
Sample a batch of queries B D. for all query (x, / in the batch B do
Compute mean reward: P + K Fi.
Partition responses: Y+ + {y; T; >} Y {yi S}.
Compute length-normalized log- rog-robaabilities log no (y; x) for all y, E Y-UYY
Compute the base score for each response: s(y:) f log πiş (y: x).
Compute positive and negative aggregated scores for the contrastive loss: p+
Eyer- e(vs); p- + Eyer- e(v),
Set target length for regularization: TARGET <- max 1//l-
Compute the Targeted Regularizer term:
Rtarget (0) Eyertur
at [yl) max(0,
lyl).
Compute loss for the query: (0) t - -log p end for
+ Rtarget
Compute average batch loss: batch (0) + [8] ExEr (0). Update parameters: 0 +- 0 - V b
until convergence criteria met
C.2 Explanation of the Procedure.
The REFA-Dynamil training procedure is centered around a composite objective that simul- taneously optimizes for preference alignment and robust length control.
First, the algorithm translates raw preference data into a structured signal for group-wise contrastive loss. For each query, responses are partitioned into positive (Y+) and negative (Y-) sets based their mean reward. Each response is then scored using its length- normalized log- pe8ppbbbability, scaled by inverse temperature 8 (s(y) B log (T(yy)x)) (y)x))) This ensures the optimization is sensitive to per-token quality rather than raw sequence length.
The core alignment objective, - log
generalizes the Bradley-Terry model to sets, pP42p
pt
maximizing the collective probability of the positive set while suppressing the negative set, with the hyperparameter 7 controlling the weighting strength for the negative examples.
The second component of the objective is the Targeted Regularizer, which is critical for counteracting the URSLA shortcut. It applies a penalty, scaled by X, to the End-of-Sequence (EOS) probability of any response that is shorter than a dynamically set target (the maximum length in the batch). This mechanism explicitly disincentivizes the model from prematurely
20

---

Published as a conference paper at COLM 2025
truncating negative responses trivial means of satisfying the loss. The final loss is the sum of the contrastive objective and this regularizer. The losses are averaged across a batch before standard gradient descent step is used to update the model's parameters 8, thereby jointly learning preference alignment and stable length behavior.
C.3 Algorithm for W-REFA (Weighted REFA-Dynamic)
Algorithm 2 details the procedure for the weighted variant, W-REFA. This version is de- signed for scenarios, such off-policy training, where fine-grained and reliable reward scores are available. It enhances REFA-Dynamic by incorporating a deviation-based weight- ing term directly into the score of each response, allowing the model to prioritize more informative examples (i.e., those with rewards far from the mean).
Algorithm 2 W-REFA (Weighted REFA-Dynamic) Training Procedure
Input: Dataset D = {xx {y: r; (.) learning rate y), hyperparameters a, B, n, repeat
Sample a batch of queries B C D. for all query (x, {y in the batch B do
Compute mean reward: R E
Partition responses: Y+ + {y; r Y- it.
Compute length-normalized pyg-prbbabilities log 7 (y: x) for all E Y+UY.. Compute the weighted score for each response:
Swdd (vi) + B- log Tip (1/ x) + Ba(ry, f).
Compute positive and negative aggregated weighted scores:
p+ widd Eycr+ e Piid Lycr- efwial))
Set target length for regularization: TARGET LENGTH <- max lyil-
Compute the Targeted Regularizer term:
Riar (0) Eyertur (EOS at lyl) max(0, TARGET LENGTH- [y|).
p
Compute loss for the query: (0) t log end for
+ Rearget (0).
Puns +7
Compute average batch loss: Lbath (0) <-
1
ExeB (0).
[8
Update parameters: 0 +- 0 - V bbttch (0). until convergence criteria met
Explanation of Key Differences. The W-REFA algorithm follows the same overall struc- ture REFA-Dynamic, with one critical modification in how scores are computed.
Line 8 (Weighted Score Computation): Instead of using only the model's scaled log- probability, the score Syta(y;) is now composite. It starts with the base score (B-
log (y x)) and then adds deviation- weight term, Ba - P).
The Role of a: The hyperparameter scales the influence of the explicit reward signal. A positive a means that responses with rewards far above the mean receive significant boost to their score, while those far below the mean receive significant penalty. This creates an implicit curriculum, forcing the model to pay more attention to the most informative, high-deviation examples.
Subsequent Steps: All subsequent steps (Lines 9-12) are analogous to those Algorithm 1, but they operate on these richer, reward-aware scores wtat,, thereby propagating the fine-grained reward information throughout the loss calculation.
21

---

Published as a conference paper at COLM 2025
D A Reader's Guide to the Theoretical Analysis
This section serves as a narrative roadmap to the formal results that underpin the REFA framework. Our goal to provide the intuition behind our theoretical claims, explain how they connect, and build rigorous, step-by-step case for our methodology. The theoretical analysis unfolds in a logical progression: we first deconstruct the properties and pitfalls of baseline objective (InfoNCA), then use those insights to motivate and analyze our proposed contrastive objective (REFA-Dynamic) and finally, we formalize the subtle length-based shortcut that necessitates our novel EOS regularization, the core algorithmic contribution of this work.
D.1 The Narrative and Intuition of Our Theoretical Results
The Flaw in Reference- Alignment (Appendix E). Our theoretical journey begins by justifying the need for reference-free framework. We analyze the standard, reference-based InfoNCA loss, which aims to match model's output distribution to a target distribution derived from rewards. Through formal stationary point analysis, we prove that the presence of the reference model, f(y|x) fundamentally alters the optimization target. At equilibrium, the learned policy Ty does not align directly with the desired preference distribution. Instead, it converges to a skewed distribution proportional to the product larget of the target probability and the reference model's probability, P; "'y; x). This result demonstrates that a fixed reference model can act as a permanent, potentially undesirable bias, preventing the policy from ever fully learning the true preference landscape. This provides a strong theoretical motivation for moving to a reference-free setting where the model can learn more directly from the provided preference data.
Weaknesses of Naive Reference-Free Alignment (Appendix F). Having established the case for a reference-free approach, we analyze the most direct adaptation: a reference- free version of InfoNCA (which term REFA-InfONCA). While its stationary point is parget_distribution" our term-by-term gradient analysis
an intuitive 'model. diadeeddiitibbtionn (Lemmas and 2) uncovers critical flaws in its optimization dynamics. We prove that the cross-entropy objective provides positive gradient push to all responses with a non-zero target probability, regardless of their quality. This means the model is inadvertently en- couraged to waste capacity learning to generate low-quality responses. Furthermore, the objective creates " "gradient conflicts" among high-quality responses, as each is incentivized to increase its own probability at the expense of all others, hindering their collective im- provement. This analysis reveals that a simple distributiin-matching objective is ill-suited for the noisy, multi-faceted nature of preference alignment.
The Principled Construction of a Contrastive Objective (Appendix G). The identified weaknesses of a simple cross-entropy loss motivate the need for a more robust objective. This section of our theory details the systematic, step-by-step process of constructing the REFA-Dynamic loss by addressing each of the identified flaws. We show how moving from a sum of individual objectives to a single, group-contrastive term resolves the issue of gradient conflicts among positive responses. We then demonstrate how partitioning responses into positive and negative sets, and treating them asymmetrically in the loss function (i.e., positives in the numerator, negatives only in the denominator), prevents the model from directly promoting low-quality responses. This structured development shows that the final REFA-Dynamic loss is not an arbitrary formulation but is systematically engineered for robust multi-preference alignment.
The Theoretical Power of REFA-Dynamic (Appendix D). With the REFA REFA-Dynamic loss constructed, we then formally prove its theoretical superiority for the task of alignment. Through a detailed gradient analysis of simplified version of the loss (Lemma 3), we show how the contrastive structure induces different and more desirable dynamics for positive and negative response sets. We formally prove in Lemma 4 that the gradient for any negative response is strictly positive, while the gradient for a positive response is negative. This means the objective has clear and unambiguous directional influence: it exclusively
22

---

Published as a conference paper at COLM 2025
pushes to increase the probability of positive responses while simultaneously suppressing the probability of all negative responses. This leads to a far more powerful convergence property: the stationary point of the REFA-Dynamic loss is achieved only the probability of all negative responses approaches zero (P&(y) x) + O for all y Y-). This is much stronger guarantee of alignment than merely matching a target distribution.
The Necessity of Length Normalization (Appendix I, continued). Before we can apply the REFA-Dynamic loss, we must first address the foundational issue of scoring. In the same appendix, we prove in Lemma 5 that any objective based on raw, un-normalized log-probabilities is fatally flawed. Because shorter sequences accumulate less negative log- probability, they are assigned artificially high scores. This creates trivial shortcut where the model can satisfy the loss by simply producing degenerate, short responses. This result rigorously establishes that the of length-normalized scores is not an optional design choice but fundamental prerequisite for any valid reference- preference optimization method.
The Final Insight: The URSLA Shortcut (Appendix D. Our theoretical analysis concludes by formalizing the central problem that motivates our primary algorithmic contribution. Having established that length normalization is necessary, we then prove that it creates new, more subtle problem. We introduce the Uncertainty Reduction with Sequence Length Assertion (URSLA) (Conjecture 2), principle stating that longer coherent sequences tend to have lower per-token uncertainty. Under this principle, formally prove in Lemma 6 that any length-normalicn will create new and perverse incentive: the model is encouraged to shorten those negative responses to increase their per-token uncertainty and thus satisfy the loss. This result provides the rigorous theoretical foundation for why additional, explicit length control mechanism, our EOS regularizer, is an essential component for achieving robust, reference- free alignment.
objective that penalizes negative responses (like REFA- REFA-Dynamic)
E Analysis of the InfoNCA Loss
In this section, we provide a detailed analysis of the InfoNCA loss function introduced in previous works. We begin by establishing our notation and problem setup, then proceed to define the InfoNCA loss. Subsequently, we derive its gradients, characterize its stationary points, and examine the implications for the learned policy distribution when reference distribution present. Finally, we discuss how removing the reference model recovers simpler scenario that aligns the learned distribution directly with the target distribution.
E.1 Notation and Setup
We consider query (or context) E X and set of K candidate responses {yi) Each response y/ is associated with scalar reward r; rxx,y;), representing how suitable or aligned the response is according to some evaluation metric or annotated feedback.
We assume access to:
A policy model (y(y))) parameterized by 6, which assigns probability to each response
V/-
A reference model xyy)x,, which provides a baseline distribution over responses. This reference model is fixed and not optimized.
We define the ratio:
Tee (y rx(x,y)) = log
(11)
74 [1 x)
For simplicity, we set B = 1 in the definitions that follow, but the analysis easily generalizes to arbitrary positive B.
23

---

Published as a conference paper at COLM 2025
The target distribution derived from the rewards is given by softmax transformation:
target
e
Pi
Σ 2 e
(12)
where 0 an inverse temperature-like parameter controlling the sharpness of the target distribution.
E.2
The InfoNCA Loss
The InfoNCA loss aims to match the distribution induced by ra(x, to the target distribution larget Given that ra (xx log AT (V)))((()))//((((()))) we define the model distribution as: Pi
erolxa))
pi
model
(13)
Dj= e(o(88))
The InfoNCA loss is the cross-entropy between the target distribution p; distribution pmodel.
target and the model
K
LIffNNC (0)
Σ
Pi
tanget
log pmudell
(14)
1=1
Minimizing LinfONCA encourages pmoddl to align with p;
target
E.3 Gradient Derivation
To understand the optimization dynamics, we compute the gradient of LInfoNCA with respect to the log-ratios ræ yi).
First, note that:
log pl model
(x log
Differentiating log pf model with respect ro(x y:):
log pk model
dik - p. model
Jr, (x, vi)
where 6 = if and 0 otherwise.
Now, differentiate the loss:
target olo p model
ILIfONCA
Pk
or 0 (x,
dra(x.y))
Substituting the derivative of log podell
a LinioNCA dr (x, y/)
K
largel(Bx (8
largel
p model).
Pk
Evaluating the sum yields the well-known cross-entropy gradient form:
aLifaNNA
model
target
Pi
dro(x,y;)
Thus, the gradient simplifies to the difference between the model and target distributions.
24

---

Published as a conference paper at COLM 2025
E.4 Stationary Points
A stationary point occurs where the gradient is zero for all i:
p model
larget
0
pt model = Pj
larget Vi.
At this stationary point, the ratio distribution e<r(x,)) / Ej efo(xx))) perfectly matches the reward-derived target distribution. However, note that p,model is defined on the ratio <o(ylx)//(y(xx,, not directly on TT (T(y)xx) alone.
Implications for the Ratio To x()x) Recall:
Te Te TV To(Vy)x)/H((/il
p model
At the stationary point:
larget
pmodel
x)
Pi
pl
PT (x)
p(y;(x)
Rearranging:
To(y;1x)
target
Pi
[(zlxx)
Define:
no xx(lx
:
((yl
Then:
To (y:lx)
Pi m(y;(x)
target
This shows that at equilibrium, nip (vr(x) is proportional to the product p; M[,](x). Note that we have not assumed Σ 7ta (y: x)) 1; N (y(lx) could be any nonnegative values. The shape of the solution is defined by:
target
( x P.
target m(v(xx).
we subsequently impose normalization to interpret no (yyl(x) as probability distribution, we would set:
(v(lxx
P:
target
M ((
(y:l)) _k no (yx(x))
EK Pk (yvlxx)
Largel
Define:
K
M =
Σ
Pk
Jargel
(yx(xx),
then:
target
A
M
In this normalized view, the equilibrium policy distribution aligns with a y-weighted version of the target distribution rather than p: parget directly. This may point to potential improvement over the InfoNCA loss function, simply by removing the reference model, as show in the subsection below.
25

---

Published as a conference paper at COLM 2025
E.5 Removing the Reference Model: A Reference-Free Scenario
The presence of the reference model μ(y|x) skews the stationary solution. If we were to remove μ(yxx) from the formulation, iee, consider a scenario where ra (x, == log zie (ylxx) without dividing by p(yxx), then the model distribution p.oodel would directly be:
To((iix))
model
2j=1 2o (y)l))x)
which is simply the policy distribution itself.
In that reference-free case, setting p.model = Pi at the stationary point would imply:
tanget
(1(lx))) = Pi
target
directly aligning the policy distribution with the target distribution. This is the desirable outcome if no baseline or reference distribution is involved. Thus, the introduction of μyy(x) in the denominator shifts the stationary solution, necessitating careful design if we want larget
direct alignment with p;
In summary, the InfoNCA loss aligns the ratio (tlly))/ ((ylx) to the target Pi achieve direct alignment of To(y)xx with p; reference-free formulation is required.
target To
larget
F Additional Analysis of the InfoNCA Loss Without Reference Model
In the previous section, we analyzed the InfoNCA loss in the presence of reference model (y(xx) Here, we revisit the InfoNCA formulation under simpler, reference-free scenario and highlight several issues that arise when optimizing this objective. Specifically, we show how the loss may inadvertently increase the probabilities of low-rated responses and lead to contradictory optimization signals.
E1 Reference-Free InfoNCA
Removing the reference model from the formulation amounts setting:
re yi) log nu (y)lx)))
(15)
In this case, the model distribution becomes:
Mtp (11 xx)
model
(16)
2 ne (1)
The InfoNCA loss reduces to the standard cross-entropy between the target and model distributions:
LInONNCA =
Σ
Pi
target
log
p.
model
(17)
target defined as before.
with p,
E2
Term-by-Term Gradient Dynamics
Lemma 1 (Gradient of Single- Single-Term Objective). Let (,) be defined as:
(0)) -p;
larget
log pi model
where
7t (1: xx)
p model
EK=n To (y:lx)
26

---

Published as a conference paper at COLM 2025
and Pi largel is fixed scalar satisfying Pi larget O. Assume (o(y)lx) 0 for all j 1,..., K.
Then, the partial derivatives of (i) with respect to Te (vilx) and ne (y)))))) for # i are:
dl,(0)
largel
1
1
Pi
(v+lx)
Σj- ,o(y))x))
and for each +
(e)
target
pi
Tz (Vjlx)
Ex= TT ())
Proof. We start from the definition:
(1()) = -Pi
target
log pi
model
Substitute pmodell
To (x)
model
Σ 7eo TT(yyix))
Thus:
K
((0)) = -p;
larget
log (V(l)) - log
Σ To(v)ix))
First, consider the derivative with respect to Tee ()(ll)))
(0) d x)
=Pp; target
(y)l() log (v((x) ()p(())
a
log
Σ no To )]
Compute each derivative separately:
1. For the first term:
a
1
log 7i (y/(x)
T (y: xx
2. For the second term, let S = Σ Ti (yy]lx). Then:
a
as
log(s)
S (y:|x) S
Substituting back:
al;(0) Jarget (ypl()
1 EK= no
No (x](xx
Next, consider * i:
(e) d (1zx))
Σ No (o((yll))) )
--p pi
target
log
āzp (1)1x)
Since log(S) with S = =K_ To(y) (x) gives:
x log(s) 1
Therefore:
(e)
target
1
dr (v)(x)
Pi
kk_ x)(xxxx)
This completes the proof.
27

---

Published as a conference paper at COLM 2025
Lemma 2 (Directional Influence of Single-Tern Objective). Under the same assumptions as Lemma 1, consider the single-term objective (;(0) defined above. To reduce ((),, one should:
Increase ( The partial derivative af,
[x can be negative ()) is suffi-
ciently small, indicating that increasing Top (y; will decrease (,))
Decrease Fy (y)]x) for * i: For each j # i, the partial derivative d6 is strictly
positive. Thus, increasing ny (yyl)x raises (0)),, while decreasing it lowers (,)
In summary, minimizing (0) encourages increasing the probability of the i-th response and reducing the probabilities of all other responses.
Proof. From Lemma 1, we have:
al,(0)
target
1
Pi
OTT
Te (y)l
Ej=1 To (y;lx)
Since target > O, the sign of this derivative depends on the relative sizes of (y(l)))) and Σj ()(()))))) x)) nha (y:lx) is small compared to the total sum, then:
Ti x
making the entire expression negative. A negative derivative w.r.t. Tee (V(lx)) implies increas- ing Tee (V)lx) reduces (0).
On the other hand, for + i:
(0)
target
Pi
> 0.
ΣK_ Ti ())(x)
target and the denominator is positive, this derivative strictly positive. Thus, Since pi increasing ne (yylx) with j 전 increases (0), and decreasing (y)y)(x)) reduces (/(0).
Combining these observations, minimize (e),,, we must increase 7 |x) (when benefi- cial) and decrease Ht Ho(y) (x) for all j * i. This establishes the stated directional influence on the single-term objective.
G Fixing Weaknesses in InfoNCA Style Loss Leads to Reference Free MPO
In the absence of a reference model, the InfoNCA loss reduces to a standard cross-entropy form that aligns the model distribution with a target distribution derived from rewards. Although seemingly straightforward, this formulation exhibits several critical weaknesses when applied to multiple responses with varying quality. In this section, highlight these issues step-by-step, referencing Lemma 1 and Lemma 2, which describe how each individual loss term behaves, and propose incremental fixes that ultimately motivate more refined approach.
G.1 W1: Encouraging Low-Rated Responses
Issue: From Lemma and Lemma 2, each single-term objective ((0) encourages increasing 7tp (y:[x) regardless of the response's 's quality. Even if p; the model is pushed to raise the probability that response, effectively giving low-rated responses unwarranted attention.
largel is small, long it is nonzero,
28

---

Published as a conference paper at COLM 2025
Implication: Low-quality responses should not receive reinforcement, yet the InfoNCA- style objective provides positive gradient push as long as pi parget > This introduces noise into training and can skew the model toward suboptimal responses.
Fix 1:
Removing Terms for Low-Rated Responses
natural remedy is to remove terms corresponding to low-reward responses. For in- stance, define threshold (e.g,, mean reward ?) and restrict the loss to responses above this threshold:
I(0)= == Σ
Pj
larget
log pi
model,
Ipos
{ ri R}.
TeT
Excluding low-rated responses avoids their direct encouragement However, this approach introduces another weakness.
G.2
W2: Contradictory Signals Among Good Responses
After removing low-rated responses, only "good" responses remain. Yet, Lemma still applies: each remaining term (i) attempts to boost its own response's probability while suppressing all others, including other good ones. Consider responses (8,7,5 (8,7,3,2) and exclude those below the mean. The terms for 8-rated and 7-rated responses now compete against each other, creating contradictory optimization signals.
Issue:
We want multiple good responses to coexist and improve together. However, the pairwise competitive nature of these single-term objectives prevents collective uplift. Instead of reinforcing set of good responses, the model ends up with internal conflicts, slowing training and potentially leading to suboptimal solutions.
Implication:
Fix 2: Single "1 us All" Term
One idea is to avoid summing over multiple per-response objectives. Instead, choose one top-rated response and form "1 vs all" loss:
Tep yr- (x)
L (f) = - log
Σ 7i (o
where yr is the best response. This removes direct conflicts among multiple positive responses since we focus on only one. But what if there are multiple top-tier responses we wish to jointly learn from?
G.3
W3: Handling Multiple Equally Good Responses is Not Possible with 1 vs All
Issue: If there are multiple equally good responses, say {8, 8, 1,1], the vs all" approach cannot leverage both top responses simultaneously. It only promotes single chosen response y/- at the expense of all others. If we pick the first "g"-rated response, we lose the opportunity to learn from the second "8"-rated one.
The vs all" fix addresses contradictory signals when multiple good re- sponses present only by ignoring some of them, which is not ideal. We want a mechanism that can consider all top-tier responses together, promoting them as a group.
Implication:
Fix 3: MPO-Style Single Term (All Good in Numerator, All in Denominator)
We can construct a single-term loss function that includes all positively rated responses in the numerator and all responses (both positive and negative) in the denominator. This resembles MPO-like contrastive form:
Eyer+ To To(y)x)
Lall-pos (0)
-log Everlur To
29

---

Published as a conference paper at COLM 2025
where Y+ are all good responses and Y- are all non-positive responses. This addresses the coexistence issue, but now every positive and negative is treated equally, ignoring the magnitude of deviations.
G.4
W4: Lack of Weighting
In the MPO-style single-term approach without weighting, all positive responses are treated identically, as are all negative responses. Consider ratings {10, 4, 4, 1): 1 is significantly worse than 4, yet both are just "negative" if below the mean threshold. In other cases, like [9, [9,7,3, 3, } and 7 are weighted equally, without learning more from 9 than 7. Without weighting, we fail to differentiate these nuanced gradations in quality.
Issue:
Implication: We want to reward the most outstanding positive responses more than marginally positive ones. Similarly, the truly poor responses should have a stronger negative influence than those that are just below average. Ignoring this leads to suboptimal gradient signals that do not fully exploit the granularity of the reward information.
Fix 4: Deviation-Based Weighting
Assign weights based on the deviation from the mean reward. Responses with higher (posi- tive) deviation get amplified; those with negative deviation penalized proportionally. This yields:
Eycr+ wy (1)2)) Eyer+ur T (y|xx)
weightted (0) log
where wy = eaS where AS, = (ry - F) for some p-value like p = {1,2}, where F is the mean reward. This weighting refines the loss by making the gradient more informative and aligned with the underlying quality differences.
G.5
W5: Fine-Grained Control via Hyperparameters
Issue: While deviation-based weighting improves the distribution of gradient signals, we may still need more control. Different tasks might require adjusting how heavily to penalize negatives, how sharply to differentiate based on deviations, or how to set an effective "temperature" of the softmax.
Implication: A flexible loss function should allow fine-tuning how negatives compete with positives (y parameter), how sensitive it to rating differences (a parameter), and how sharply it distinguishes between responses (B parameter).
Fix 5: Hyperparameterizze Weighted Multi-Preference Loss
By introducing hyperparameters (a, Y),, can regulate:
1. 4: how strongly the deviation affects weighting.
2. ß: an inverse-temperature parameter controlling the softmax sharpness.
3. y: a factor that overweights negative responses to ensure a clear contrast.
Incorporating these parameters, we obtain a final, fully adjustable loss function that ad- dresses all previously identified weaknesses, allowing nuanced and stable optimization.
Eyer+ ((zp(ylx)+aass))
(M) = -log Eyer (l + + Eyer- ((mp(y)x)+aAss)
In summary, each weakness of the InfoNCA-style loss reveals the necessity of more sophis- ticated mechanisms: filtering out low-rated responses, avoiding contradictory objectives among top responses, considering multiple good responses together, weighting by deviation
30

---

Published as a conference paper at COLM 2025
for finer quality distinctions, and introducing hyperparameters for greater flexibility. These insights set the stage for refined multi-pre culli-ppeeerence optimization methods that align more closely with the complexity of real-world preference alignment scenarios.
H Proposed REFA Loss Functions Without Length Normalization
The analysis in the previous sections uncovers several fundamental weaknesses in InfoNCA-style loss when used without a reference model. We identified:
1. Encouragement of low-rated responses (W1): Every response, including low-quality ones, receives upward pressure as long as pi
Jarget > 0, leading to suboptimal reinforcement.
2. Conflicts among top-tier responses (W2 & W3): Even after filtering out low-quality responses, multiple good responses compete against each other. This prevents collective uplift and complicates the optimization dynamics.
3. Lack of nuanced weighting (W4): Treating all positive negative responses equally ignores the fine-grained quality differences indicated by their rewards, missing opportu- nities better leverage the available information.
Insufficient flexibility (W5): Without hyperparameters, one cannot adjust how strongly to penalize negatives, how sensitively to incorporate deviations, how sharply to distinguish among responses.
These issues highlight the need for more refined reference-free multi-preeerence loss function that:
1. Prioritizes genuinely high-quality responses without inadvertently promoting low- quality ones.
2. Reduces direct competition among good responses so that multiple top-tier candidates can improve collectively.
3. Incorporates reward-based weighting to reflect the spectrum of response quality.
4. Provides hyperparameter-drven flexibility for controlling emphasis on negatives, sensi- tivity rating differences, and the sharpness of the distribution.
H.1 REFABASE Loss Variants
We propose family of REFABASE (Reference-Free Alignment) loss functions that build upon these insights. By integrating deviation-based weighting, selective response sets, and hyperparameters for fine-grained control, we address the aforementioned weaknesses. We present two variants:
REFABASE-1 vs All: This variant selects the single highest-rated response yr defined as:
y/ = arg maxrir Y = (yr}, Y = 1ū
We assign weights based on deviation from the mean, AS; = r; - i, using an exponential scheme W. = gã Introducing B for inverse-temperature and for negative overweighting, the REFABASE-1 vs All loss is:
(0log (oly) 1x)
(REFABASE-IVAAI()) log
(18)
efllog Roly; +aAS,) + - ( (yyx))
This approach eliminates low-rated responses and avoids direct conflict among multiple good responses by focusing solely on the top candidate. However, it cannot simultaneously exploit multiple equally high-quality responses.
31

---

Published as a conference paper at COLM 2025
REFaBase-Dynamic:
To incorporate multiple good responses, we define:
Y 1, P), Y
(v; = P..
All positive responses appear in the numerator, and both positive and negative sets appear in the denominator. Applying deviation-ban weights and hyperparameters as before, we obtain MPO-style loss:
Eyer+ ellog no (o(y)))+aAs,,
BRsea llog
Eyer+ ellog (o(y))) Y Eyer ef (p(l)g)((()) (19)
This BAEEAAASEdyiamiccc formulation simultaneously addresses all previously identified weaknesses:
W1: Low-rated responses remain in the denominator but do not receive direct promotion.
W2 W3: Multiple responses are jointly improved, reducing pairwise conflicts.
W4: Deviation-based weighting differentiates among positive responses and among negative ones.
W5: Hyperparameters (a, B, enable nuanced control over the optimization process.
In essence, the REFAABAsEdyyamii loss represents the final convergence of our incremental fixes, providing a robust and flexible solution to the shortcomings of the InfoNCA-style loss in reference-free, multi-preference alignment scenarios.
I Length Normalization in REFA
In this section, analyze the behavior of simplified REFAlOSS function without hyper- parameters or weighting. We focus on the dynamic variant, which selects positive and negative responses based their rewards relative to the mean. We then highlight how length can become shortcut for reducing the loss and present a normalization strategy to address this issue.
I.1 Simplified REFA-Dynamic Loss
Consider the simplified REFA-dynamic loss:
L REFA-dynamic (f) = - log Eyer+ To(y/x)
(20)
Eyertur- no(y/x)²
where
Y+ -y ry > f} is the set responses with rewards above the mean
Y- [y ry S 7} is the set responses with rewards below or equal to the mean.
no [y(x)) = e"y, where sy = log xx((xx x) is the log-probability of response
Define:
-- Σ T Z: Σ Ta(y(x).
yEY+
vEr-Ur-
Thus:
Z
LREFA-dynamic (f) log
= log log
(21)
Z
32

---

Published as a conference paper at COLM 2025
1.2 Gradient Analysis of the Simplified REFA-Dynamic Loss
Lemma (Gradient of the Simplified REFA-Dynamic Loss). For a response with log- probability Sy = log (y(y))), the gradient of LReFa-dynamic (0) w.r.t. Sy is:
a Rera-daamami (0)
Py phodel Pyy
if y Y+,
doy
pmodel
y Y-,
where
TCp (z((y/x)
nt xg(y(x)
Py
model
and py
Z
Zt
Proof. First, note:
Lyera-dynamic (0) = llog Z - -log Z+,
with
= Σ
xo( E ,a(y|xx, yEY-
z Σ
Ne xo(l)x).
yEY+
yeY+
Taking derivatives w.r.t. Sy:
aL
log Z
à log Z
dsy
dsy
dsy
Compute each term:
a log
1 az
llg Z
az
dsy
Z dsy
dsy
Z dsy
y Y+:
Ze ()(xx)
()(xx
e"v,
asy
asy
Thus:
aL
no (y)x)
model pos
py
Jsy
Z+
y Y-:
az*
az
0,
z (ylx))
Jsy
dsy
Thus:
aL
Te (1/(x)
0 pyy
model
d5y Z
This completes the proof.
L.3 Implications for Positive and Negative Responses
Lemma 4 (Increasing Probability of Positive Responses Decreases the Loss). To decrease LRexx-dyaamic(O) the model must increase x for y E Y and avoid increasing πT (y(xx) for y E Y In particular, raising the probability of positive responses lowers the loss.
Proof. Recall the simplified REFA-dynamic loss:
Z log Z
LREFx-dynamic
where
Z Σ To(y)x), ycY
- Z + Σ To(y|x).
yCY
33

---

Published as a conference paper at COLM 2025
Since Y Ç cY+UY, it follows that Z+ Σ Z, and strictly Z+ < Z if there is at least one negative response.
Case y E Y
Notice that:
Py model (y)())x)
pe TT((yxx)
Z
Since Z+ Z, we have z 2, implying p
py
poddel
Therefore:
py
model
Py 0.
dsy
negative gradient w.rt. Sy means that increasing Sy (equivalently, increasing xo(y|x)) for y E Y+ decreases the loss L. In other words, raising the probability of positive response lowers the loss.
Case y E YY: y it does not appear in Z+ Thus:
az+
az
7 (ylxx)
Jsy
Jsy
Then:
aL
Tee (ylx)
Py
model
0.
dsy
Z
positive gradient w.rt. Sy indicates that increasing Sy (or (y( for negative response raises the loss L. To reduce the loss, we must not increase the probabilities of negative responses.
To decrease REpa-dyaamic we must increase the probabilities of positive responses (where the gradient is negative) and avoid increasing the probabilities of negative responses (where the gradient is positive). Thus, minimizing the loss encourages boosting positive responses' probabilities.
Conclusion:
1.4 Length as Shortcut
Without length normalization, log (ylx) is the sum of token log- log-probabilities:
log x =
Σ log Po (f/ctxx))
Shorter responses generally have fewer tokens over which accumulate negative log- probabilities, often resulting in higher (y(x))). Thus, a trivial way to increase Tip (y/xx) (x) for some E Y is to make exceedingly short, increasing its probability without genuinely improving per-token quality.
Lemma [Length-Inducee Probability Inflation). Given two responses y/1 and y/2 with similar average token probabilities, if ly1 < ly2), then typically nip (y)llxx) > nip (y)
incentivizing shorter responses as shortcut to reduce LRePA-dynamic
Proof. If each token in both responses has an expected log-probability around -c for some 0, then:
log ne (v1lx) x - llylle, T222(x) x - -ly2|c.
If lyal < ly2|, then -lylc > - ///c and thus:
,o(y1|x) = lnle e lulc (2222xx).
This shows shorter responses achieve higher probabilities, providing an artificial advantage.
34

---

Published as a conference paper at COLM 2025
1,5 Fix: Length Normalization
To prevent the model from exploiting this length-based shortcut, we introduce length normalization. Instead of:
log [T(()xx)=
Σ log Po (f)contt))))
we define:
log TT
Σ rey
log (t context),
and use elognelyle in place of zza (y(())) in the loss. This ensures that shorter responses do not gain an undue advantage from having fewer tokens, compelling the model to improve the quality per token rather than simply reducing response length.
1.6 Modified Loss Function
Given the above length normalization fix, we can now modify our loss function to use the new length normalized log- log-perplyxity values, rather than simply the log-perplexities. We write this as follows:
Eyer e @llogmolyli adSs))
LREFA-dynamic (0) =-log
(22)
Eyer+ ( + Eyer- (y(l))
J Relationship between Average Uncertainty and Sequence Length
Despite our length normalization, we find that the loss function given by 22 still has problems due to a nuanced observation. We will state this observation here as conjecture, and verify it through our experiments. But before we state this observation, we provide some notations.
Let y denote a response (sequence of tokens) with length len(y) and token- level conditional probabilities Po(t context). We define the average per-token negative log-probability (perplexity) of under the model 7i as:
Notation:
1
LP(y): = len(y)
| log Po( context)]
(23)
FeY
This quantity LP(y) measures the average uncertainty or difficulty that the model πyş has in predicting the tokens of y. Lower LP(y) indicates higher model confidence on per-token basis.
J.1 Model Uncertainty Reduces with Sequence Length
Conjecture 2 (Uncertainty Reduction with Sequence Length Assertion (URSLA)). Consider a set of responses y generated by the model πι.- Let LP(y) be the average per-token negative log-pno rig-probability of y. The URSLA Conjecture states that for non-empty subset of responses V, the length of a response y ) increases, the expected value of LP(y) decreases. Formally, there exists a set y such that for all y E Y:
aLP à len
<, for all y y.
(24)
In other words, longer sequences from y are generally easier to predict on per-token level, yielding lower average negative probabbbyyyyyty than shorter sequences.
35

---

Published as a conference paper at COLM 2025
This conjecture reflects the empirical observation that models, once committed to certain semantic or syntactic trajectory in generating text, tend to predict subsequent tokens with increasing ease. We have experimentally verified this phenomenon for the models under consideration.
We note that this conjecture may fail under certain conditions such when the model encounters domain shifts, incoherent sequences, or adversarial prompts, where longer sequences do not become inherently easier to predict. Empirical verification and domain- specific analysis are thus required to support or refute this conjecture in practical settings.
J.2 Loss Reduction Reduces Expected Sequence Length
Let y be set of "negative" responses, as identified by some alignment criterion, where the training objective encourages decreasing their probabilities. Minimizing the objective often translates into increasing LP(y) for y E y Under the URSLA Conjecture 2, shorter sequences have higher Ly),,, implying that reducing response lengths for these negative instances is straightforward way to achieve the desired increase in per-token perplexity and decrease in their probabilities.
Lemma 6 (Length Reduction for Negative Responses). Consider a loss function L(0) that depends on the probabilities of a set of negative responses y Suppose that reducing the probabilities of these negative responses (equivalently, increasing their average negative log-probability LP(y)) decreases the overall loss L(0). Under Conjecture 2, there exists positive correlation between decreasing len(y) for y E and reducing L(0). Formally, let:
(0) 0 for for y e e ā (yn
(25)
Then, minimizing L(0) w.r.t. encourages decreasing len(y) for y y
Proof:
Assume that for each y E y", the training objective encourages a reduction in Ao [y(()), the probability assigned to y. Equivalently, this corresponds to increasing
- log no (y)xx, increasing its average negative log-probability LP(y). By the chain rule,
we have:
aL(O) aL( aLP(y)
(26)
llnn aLP ā len(y))
Since L(O) decreases when LP(y) increases (to reduce log No (y)x)), we have:
AL(0) aLP(y)
0.
(27)
By the URSLA Conjecture 2,
aLP (y) a lyenyy)
<
(28)
Combining the two inequalities:
Al(O)
AL(O) aLP(y)
> 0,
(29)
lyn alP(y) a Olenyy)
since the product of two negative numbers is positive.
positive gradient Mentyy > 0 implies that reducing len(y) decreases L(0). Hence, during
dL(e)
training, the model is incentivized to shorten negative responses y E y= This completes the proof.
36

---

Published as a conference paper at COLM 2025
J.3 Discussion on the Surprising Relationship between Sequence Length and Average Model (Un)Certainty
Lemma 6 demonstrates that under the URSLA Conjecture 2, minimizing a loss function that penalizes high-probabillty negative responses effectively encourages the model to produce shorter negative responses. While length normalization techniques are often introduced to mitigate trivial solutions (such as producing degenerate short outputs), the above analysis reveals that under certain assumptions about length and certainty, length normalization alone can inadvertently incentivize undesirable length reductions. This necessitates additional considerations, such as length-regullrriztion or more sophisticated constraints, to ensure that the model's improvements not merely byproduuct of changing response lengths but reflect genuine quality enhancements.
J.4 Justification for an EOS-Probability Regularizer
Real-world training data often reflect inherent human biases. Humans typically write and record relatively concise content due to time, effort, or domain constraints. However, at inference time, users may request more elaborate, detailed, and longer responses from model, especially when studying complex topics that require extended reasoning rich explanations. This discrepancy can create a mismatch between the training distribution and the desired inference-tine behavior.
Dataset-Induced Bias for Shorter Sequences. Human-generated training data often skews toward brevity. Tasks such writing quick responses, providing minimal clarifications, or answering direct questions do not always require exhaustive elaboration. Economic factors, annotation fatigue, and cognitive load mean human annotators or content creators produce shorter sequences that still suffice to convey basic meaning. As result, the training corpus is filled with many shorter samples, unintentionally leading the model to treat shorter responses as "normal" or even optimal.
Mismatch Between Training and Inference Requirements. While training data may be efficient and minimal, users in deployment scenarios often seek more extensive answers. For instance, a user querying a large language model about a technical topic may benefit from a thorough explanation rather than a terse summary. Similarly, users might expect models to produce intermediate reasoning steps, citations, code snippets, or detailed examples. When the model trained primarily shorter samples, fails to internalize the incentive to produce these richer, longer responses at inference time.
Premature EOS as Shortcut. In the absence of constraints, model that has learned from predominantly short examples can exploit early termination as a trivial optimization. By producing an ond-of-sequence (EOS) token sooner, can quickly finalize its response and avoid the complexity of generating lengthy, coherent content. This behavior reduces computational effort and under certain loss formulations may even appear optimal, as shorter sequences are well-represented and lower-risk in the model's learned distribution.
Encouraging Richer Outputs via EOS Probability Control. To counteract the dataset- induced bias and prevent premature truncation, we introduce regularizer that influences the probability distribution over the EOS token. By penalizing or rewarding the model for EOS placement, we can realign training incentives with deployment-tine requirements. For example, if human evaluators or usage logs indicate preference for more elaborate answers, we can discourage too-early EOS predictions, nudging the model toward providing fuller, more informative content. Conversely, if the model tends to ramble or produce unnecessarily long responses, the regularizer can be adjusted to encourage timely conclusions.
In essence, the COS-probabillis regularizer acts as corrective mechanism. It helps balance the model's 's learned tendency toward short outputs, an artifact of the training data, with the user's 's practical need for more extensive, detailed responses at inference time. By bridging the gap between the dataset distribution and real-world user preferences, we ensure that the model's output quality and length better reflect the desired usage scenarios.
37

---

Published as a conference paper at COLM 2025
J.5 Incorporating the Regularizer into the REFA Loss
We start from the previously defined length-normalized REFA-dynamic loss (with deviation- based weights and hyperparameters a, B, Y as described Section H). Let:
p+
Σ yeY+
Blog no (ak))
p- = Σ
Bloog T. +a45)))
yeY
Then, the REFA-dynamic loss without regularization is:
p+
LREFA-dynamic - log p+ + 7pp
(30)
To incorporate a preference on the EOS probability, we define a regularization term ( that depends on Po(EOS] tokens before EOS in y). If the training data skew toward shorter sequences, we may increase À to discourage prematurely high EOS probabilities, thus encouraging the model to produce more extensive responses. Conversely, if the model generates unnecessarily long and uninformative content, we can adjust the regularizer to incentivize more timely endings.
simple choice for such a regularizer is:
R(0)
(EAS at position lyl), yEY+UY-
E
(31)
where À is small constant that scales the influence of this regularizer. A positive À can encourage the model to place appropriate probability mass on the EOS token the position it appears in the human-writteer training examples, gradually adjusting the model's length preferences.
Integrating this regularizer into the loss function, we obtain:
P
L REEAA-dynamic-eg (0) = log
+ R(0).
(32)
p 7P
By tuning A and the form of (0), practitioners can mitigate the dataset-induced length bias, steering the model toward producing responses of more desirable length and detail level. This balanced approach ensures that the model's performance at inference time more closely aligns with user expectations, enhancing both the quality and relevance of its outputs.
K Experiments Details
K.1 Experimental Setup
We evaluate our models using three widely recognized open- ended instruction-folloiing benchmarks: MT-Bench, AlpacaEval 2, AlpacaEval and Arena- Hard v0.1. These benchmarks are commonly used in the community to assess the conversa- tional versatility of models across diverse range of queries.
Evaluation Benchmarks
AlpacaEval 2 comprises 805 questions sourced from five datasets, while MT-Bench spans eight categories with total of 80 questions. The recently introduced Arena-Hard builds upon MT-Bench, featuring 500 well-defined technical problem-solving queries designed to test more advanced capabilities.
We adhere to the evaluation protocols specific to each benchmark when reporting results. For AlpacaEval 2, we provide both the raw win rate (WR) and the length-controlie win rate (LC), with the latter being designed to mitigate the influence of model verbosity. For Arena-Hard, we report the win rate (WR) against a baseline model. For MT-Bench, we present the scores as evaluated by GPT-4-Preview-111 which serve the judge model.
38

---

Published as a conference paper at COLM 2025
Baselines We compare approach against several established offline preference op- timization methods, summarized in Table Among these are RRHF Yuan et al. (2023) and SLiC-HF Zhao et al. (2023b), which employ ranking loss techniques. RRHF uses length-normalized log-likelihood function, akin to the reward function utilized by SimPO Meng et al. (2024), whereas SLiC-HF directly incorporates log-likelihood and includes supervised fine-tuning (SFT) objective in its training process.
IPO Azar al. (2023) presents theoretically grounded approach that avoids the assumption made by DPO, which treats pairwise preferences as interchangeable with pointwise rewards. CPO Guo al. (2024), on the other hand, uses sequence likelihood a reward signal and trains jointly with an SFT objective.
ORPO Hong et al. (2024b) introduces a reference-free odds ratio term directly contrast winning and losing responses using the policy model, also incorporating joint training with the SFT objective. R-DPO Park et al. (2024) extends DPO by adding a regularization term that mitigates the exploitation of response length.
InfoNCA Chen al. (2024a), which introduces K-category cross-entropy loss, reframes generative modeling problems as classification tasks by contrasting multiple data points. It computes soft labels using dataset rewards, applying softmax operation map reward values into probability distributions.
Lastly, SimPO Meng et al. (2024) leverages the average log probability of a sequence as an implicit reward, removing the need for reference model. It further enhances perfor- mance by introducing a target reward margin to the Bradley-Terry objective, significantly improving the algorithm's effectiveness.
Hyper-parameters
Model Name
Method
B
à
Y
LR
REFA-InfONCA
0.01
3.0
5.0e-5
2.0
3.0e-7
Mistral-7b-base
REFA-lvSK
0.01
2.0
5.0e-5
2.0
3.0e-7
REFA-Dynamic
0.01
3.0
5.0e-5
2.0
3.0e-7
REFA-InfONCA
0.01
4.0
2.0e-5
2.0
6.0e-7
Llama-3-8b-base
REFA-lvSk
0.01
2.0
2.0e-5
2.0
6.0e-7
REFA-Dynamic
0.01
4.0
2.0e-5
2.0
6.0e-7
REFA-InfONCA
1.0
2.5
1.0e-5
4.0
3.0e-7
Mistral-7b-instruct
REFA-lvSK
1.0
2.5
1.0e-5
3.0
3.0e-7
REFA-Dynamic
1.0
2.5
1.0e-5
4.0
3.0e-7
REFA-InfONCA
1.0
10.0
1.0e-5
4.0
4.0e-7
llama-3-bbiinsrruct
REFA-lvsK
1.0
10.0
1.0e-5
3.0
4.0e-7
REFA-Dynamic
1.0
10.0
1.0e-5
4.0
4.0e-7
REFA-Dynamic (Iterl)
1.0
2.5
1.0e-5
4.0
8.0e-7
Mistral-7b-instruuct
REFA-Dynamic (Iter2)
1.0
2.5
1.0e-5
4.0
5.0e-7
REFA-Dynamic (Iter3)
1.0
2.5
1.0e-5
4.0
3.0e-7
REFA-Dynamic (Iterl)
1.0
10.0
1.0e-5
4.0
8.0e-7
Llama-3-bbinnsruuc
REFA-Dynamic (Iter2)
1.0
10.0
1.0e-5
4.0
5.0e-7
REFA-Dynamic (Iter3)
1.0
10.0
1.0e-5
4.0
5.0e-7
Table 4: Sypee--paramtee settings for various models and REFA methods. grouped by base models, instruction-tunet models, and iterative tuning runs.
The table is
The main hyperparameters in REFAare a (reward deviation sensitivity), & (inverse temperature), Y (negative set penalty), À (EOS regularization strength), and p (power for deviation term, explored as variant).
Hyperparameter Consistency and Tuning Strategy:
Our tuning, performed on a held-out development split of the UltraFeedback dataset (test prefs), revealed several consistencies (see Table for optimal values):
39

---

Published as a conference paper at COLM 2025
(Deviation Sensitivity); Showed consistency by base model type: optimal 0.01 for Base models and & = 1.0 for Instruct models.
6 (Inverse Temperature): Displayed model-family consistency: B 2.5 for Mistral-
L Instruct variants and 5 10.0 for Llama-3-Instruct variants.
À (EOS Regularization): Consistently optimal at 1.0 x 10-5 5 for Instruct models across
variants and iterations; varied slightly for Base models (2.0 x 10 5 to 5.0 x 10 5). As key component for length control, some tuning in a small range is beneficial (see Fig. 2 in main paper).
Y (Negative Set Penalty): Generally stable, with y 2.0 often optimal for Base models and iterative REFA-Dynamic. Single-iteration Instruct models used y E [3.0, 4.0). This parameter did not require extensive fine-tuning.
p (Deviation Power): Explored as structural variant (p {0, (0,1,2]) rather than continuously tuned hyperparameter. p = 2 was often best for REFA-Dynamic, while p = 0 (unweighted deviation) also showed strong performance for REFA-ivsk on Llama- 3-8B, indicating REFA'S 's core benefits are not solely dependent on this specific weighting.
L Additional Ablations
Effect of Rentlty-Ratio on REFA-dynamic: To investigate the impact of the penalty-ratio parameter y on the performance REFA-DYNAMIC, we evaluate two key metrics: LC-WR (%) and WR (%) across varying values of Y. As shown in Figure 17, LC-WR increases initially, peaking at 7 = 3, before slightly de- clining. This trend indicates that a moderate penalty ratio balances positive and negative response contributions effectively, improving alignment consistency. Conversely, WR% shows decreasing trend as Y increases beyond 3, suggesting that overly penalizing negative responses may compromise reward alignment. Key find- ings underline the importance of selecting an optimal to maintain balance between alignment consistency and reward signals.
28
26
JC-WR
Error Analysis and Statistical Significance To validate the performance improvements conduct rig-
2
4
Penality-Ratio (y)
of our REFA framework, orous error analysis. It crucial to demon- strate that the observed gains not arti- facts of evaluation variance but represent statistically significant advancements. We analyze the results on two challenging bench- marks, AlpacaEval2 and Arena-Hard, by examining standard errors (SE) and 95% confidence intervals (CI).
Figure 17: Effect of penalty-ratio on Mistral-
Base (7B) on AlpacaEval2.
The consolidated results in Table 5 provide the foundation for this analysis, comparing our full method and its variants against strong baselines.
Significance Analysis on AlpacaEval2. The standard errors reported for AlpacaEval2 allow us to assess the significance of the performance gap between methods. common heuristic for statistical significance is a difference between two means that is greater than twice the combined standard error.
For Llama-3-Sb-Instruct, the REFA- DEFA-Dynamic win rate (50.63%) is 12.73 points higher than DPO (37 (37.90%). The difference is substantially larger than their standard errors (1.58 and 1.40), indicating highly significant improvement. Refa-Dynamic (LC-WR 49.64 + 1.58) also clearly outperforms SimPO (LC--W 44.70 + 1.40), with a gap of nearly 5 points, which is well over two standard errors of difference.
For Mistral-7b-Instruddt REFA-lvSk (WR 36 36.23% achieves a lead of over 11 points against DPO (24.90%). Given the small standard errors (1.48 and 1.30), this margin is again statistically significant.
40

---

Published as a conference paper at COLM 2025
Table 5: Statistical analysis of Reference-Free Alignment (REFA) and baseline alignment methods on the Llama-3-8b and Mistral-7b instruct models. To assess the reliability of performance gains, we report the length-controlled win rate (LC-WR) and standard win rate (WR) with standard error (SE) on AlpacaEval2, and the win rate with its 95% confi- dence interval (CI) on Arena-Hard. The results demonstrate that REFA's improvements are statistically robust and significant.
AlpacaEval2
Arena-Hard
Base Model
Method
LC-WR (%) SE
WR (%) SE
Win-Rate (%)
WR (%) [95% CI]
DPO
40.30 +: 1.40
37.90 +: 1.40
49.3
32.6 [30.3, 34.8|
SimPO
44.70 + 1.40
40.50 + 1.40
51.5
33.8 [30.9, 35.3]
Llama-3-8b-Instrun
REFA-InfONCA
43.47 + 1.60
44.08 + 1.60
53.6
38,9 [37.3, 41.4
REPA-Ivsk-Dynamic
49.56 + 1.57
50.23 1.57
43.8 (42.0, 45.9]
REFA-Dynamic
49.64 1: 1.58
50.63 1 1.58
60.2
44.7 [42.6, 46.8)
DPO
26.80 1: 1.30
24.90 1 1.30
37.1
21.0 [18.8, 22.7|
SimPO
32.10 + 1.40
34.80 1.40
41.3
16.3 [15.2, 18.0]
Mistral-7b-Instruct
REFA-InIONCA
30.20 + 1.52
32.10
1.52
40.2
22.5 [2..2, 24.1]
REFA-Ivsk-Dynamic
33.12 1 1.48
36.23 1 1.48
43.7
26.3
124.6, 28.1]
REFA- ELEA-DDnamic
32.50 + 1.53
34.56 + 1.53
42.5
25.8
[24.5, 27.6]
Significance Analysis on Arena-Hard. The 95% confidence intervals (Cls) on the Arena- Hard benchmark provide direct and stringent test of significance.
For Llama-3-8b-Instrune there is clear separation between REFA methods and the baselines. The lower bound of REFA-Dynamic's CI (42.6%) ))) is substantially higher than the upper bounds of both DPO (34.8%) and SimPO (35.3%). This non-overlapping interval demonstrates a statistically significant superiority of REFA-Dynamic.
For Mistral-7b-Instruct, REFA-lvsK emerges as the top performer. Its Cl of [24.6%, 28.1%] does not overlap with DPO's [18.8%, 22.7%] or SimPO's [15.2%, 18.0%]. This confirms that REFA-lvSk achieves robust and significant win rate advantage this difficult benchmark.
Conclusion on Robustness. The error analysis across both benchmarks and base models confirms the robustness of our findings. The performance gains achieved by the REFA framework, validate REFA as superior alignment technique.
Computational Efficiency We tested the computational efficiency of REFA baseline of MPO and show that it achieves significant computational efficiency. Specifically, it leads to 17% reduction in training time and 10% decrease in peak GPU memory usage compared to the MPO baseline as show in table 6.
Metric
REFA
MPO
Training Time
4h 1m
4h 50m
Peak GPU Memory
71 GB
79 GB
Table 6: Efficiency comparison of REFA and MPO. The table highlights REFA's advantages in both training time and peak GPU memory consumption.
M Qualitative Comparison
In this section we provide qualitative comparisons between the baseline model outputs and model outputs from model trained through the REFA-dynamic approach.
41

---

Published as a conference paper at COLM 2025
SimPO vs REFA-Dynamic comparison
User Query
Are you capable ChatGPT?
SimPO Response
am not capable of being as advanced as chatgpt, which is sophisticated ar- tificial intelligence language model developed by openai. Chatgpt possesses vast knowledge base, advanced natural language processing capabilities, and continuous learning abilities that enable it to generate human-like responses to various inputs, including text, images, and audio. While I can perform certain language processing tasks, my capabilities are limited compared to chatypt's advanced neural networks, algorithmss, and training data. Nonethe- less, can assist with specific tasks and provide relevant information based on my pre-existing knowledge and programming.
REFA-Dynamic Response
I am not capable of being advanced chatgpt, which is sophisticated language model developed by openai. Chatgpt has been trained on vast amounts of textual data, enabling it to generate human-like responses, under- stand natural language, and provide insights and recommendations based on context and patterns. While possess natural language processing abilities and can assist with various tasks, lack the level of sophistication and breadth of knowledge that chatgpt possesses. However, can continue learn and improve through training and exposure to new data, making me useful tool for specific tasks or applications. Ultimately, chatgpt's capabilities are unique and exceptional, and it remains benchmark for advanced ai language models.
42

---

Published as a conference paper at COLM 2025
SimPO vs REFA-Dynamic comparison
User Query
If you could help me write email to my friends inviting them to dinner on Friday, would be greatly appreciated.
SimPO Response
Let's Gather for Delicious Dinner on Friday! Dear [Friends2019 Names], Greetings and warm wishes! hope this email finds you all doing well and enjoying your week so far. As summer winds down, let's make the most of our time together and plan delightful dinner gathering on Friday evening. Mark your calendars for Friday, [Date], at ['lime], as we invite you to join us for a scrumptious feast at our place. Here are the details: Date: [Date] exciting menu filled with mouthwatering dishes, including [list some appetizing dishes or cuisine theme]. We can't wait to share our culinary creations with you and indulge in some quality time together. Please RSVP by [Date] so we can ensure adequate food and seating arrangements for everyone. Kindly let us know if you have any dietary restrictions or preferences, and we'll do our best to accommodate them. Looking forward to catching up, sharing laughs, and creating unforgettable memories over delightful dinner. Bring your favorite drinks, appetizers, or desserts to add to the festive ambiance. Don't miss out on this opportunity to unwind, connect, and savor some delicious food with dear friends. See you soon! Warm regards, [Your Name]
Subject:
Time: [Time] Location: |Your Address] We've prepared
REFA-Dynamic Response
Subject: Let's Gather for a Delicious Dinner on Friday! Dear Friends, Greetings and warm wishes! hoop this email finds you all doing well and enjoying your week. summer winds down, let's make the most of it by coming together for a delightful dinner gathering on Friday evening. We're re thrilled to invite you to join us for a cozy and intimate dinner at our place on [date] at [time]. We've ve planned an exciting menu filled with mouthwatering dishes, and we can't wait to share it with your company. Here's 's a sneak peek: [mention cuisine theme, if applicable). This casual yet elegant dinner party promises great conversations, laughter, and memorable moments. Please mark your calendars for Friday, [date], and kindly RSVP by [date] so we can prepare accordingly. Feel free to bring your favorite beverage or wine to complement the meal. Let's catch up, indulge in delicious food, and create cherished memories amidst these unprecedented times. We're excited have you all as our esteemed guests and look forward to seeing you soon. Kindly let us know if you have any dietary restrictions or preferences, and we'll make sure to accommodate them. Please share this invite with anyone Looking forward to hearing back from
some appetizing highlights
who might be interested in joining you and seeing you all at our dinner table! Warm regards, |Your Name] [Your Contact Information) Let's make this fantastic evening together! Best regards, [Your Friends' Names] (Optional: Add a fun and personal touch by including relevant GIF emoji related to food gathering, if appropiiate.) Note: Adjust the tone, style, and details according to your relationship with your friends and the occasion's specifics. Good luck and enjoy your dinner!
43


Reviews:
[{"review_id": "2DSxvSCeQb", "reviewer": "Reviewer_G1HG", "summary": "This paper introduce REFA, which is new method for training language models to follow human preferences better. Previous methods make models give too short answers because they learn shortcuts, but users actually want longer and more detailed responses. REFA uses special techniques like length normalization and EOS probability control to fix this problem and make models generate richer, higher-quality answers. The experiments show that REFA performs much better than other methods on benchmarks like AlpacaEval2, achieving 26.6% Length-Controlled Win Rate compared to 21.5% for the previous best method SimPO.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper contains with rich references and detailed technical background. \n* Empirical results on uncertainty reduction Figure 1 \n* The paper has good motivation of proposing.\n  * (Assuming the structure of this paper is massaged and Appendix E is correctly placed back in the main text)", "reasons_to_reject": "* [minor] REFA needs careful hyperparameter tuning and variant selection (InfoNCA, 1-vs-k, or dynamic)\n* Presentation and writing of the paper can be improved e.g., \"REFA-dynamic\" is the most important contribution of the work, but what exactly this refers to is hard to find.\n   * Many key motivation for proposing REFA and variants are placed in Appendix E, which makes it hard to understand to overall sotry without reading Appendix E. \n* The differences between other methods are none in some datasets (MT-Bench, Arena-Hard) and no error bars are present in the results.", "questions_to_authors": "### Questions\n* In Table 2, why is Arena Hard WR for SimPO (16.6) is not boldened but \"REFA-dynamic (p=2)\" (16.6) is boldened?\n  * Also how are these hyper-parameters (`p`, $\\alpha$, $\\beta$, $\\gamma$, $\\lambda$) tuned? Is it using a separate development split for each dataset? \n\n### Comments/Suggestions\n* Appendix E is the key to this paper, and I strongly recommend shortening Section 3 and 4, then add (summarized version of) Appendix E after section 5. This is critical for understanding the flow of paper and made me very confused on the underlying motivation of proposing REFA and its variants.", "extra_scores": {}}, {"review_id": "RBN8cUV2vt", "reviewer": "Reviewer_AyBr", "summary": "This paper proposes a family of reference-free alignment methods for multi-preference optimization which maintain response length control. The technical contributions that enable this are deviation-based weighting and a length regularizer. Theoretically, they show that naive length normalization can lead to shorter negative responses and shows how they can be addressed through additional constraints. Empirical results show REFA achieves state-of-the-art performance on AlpacaEval2, with 26.6% length-controlled and 24.2% standard win rate.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Introduces two new algorithmic components (deviation-based weighting) and improved length regularizer that addresses limitations in prior work. \n- Provides some interesting theoretical insights into length-based optimization in alignment methods. \n- Presents a thorough empirical comparison of the proposed methods with several competitive baselines across several datasets and different model families (up to 8B).  \n- Includes a detailed analysis the effect of length normalization hyperparameters and dynamic adaptation, highlighting useful insights for its use in practice. (e.g. need to carefully tune \\lambda)", "reasons_to_reject": "- The proposed methodology is incremental, primarily combining and refining techniques from existing approaches like SWEPO and SimPO.\n- There is no formal proof in the theoretical foundation of URSLA, and it is mainly built with conjectures based on some empirical observations. It's unclear if the findings are universally valid.", "questions_to_authors": "1. Could you elaborate on the differences of different methods in terms of computational costs/efficiency? \n2. From which tasks in the benchmarks does the improvement come from? It would be useful to provide an additional analysis to better understand the strengths.\n3. The paper might benefit from an ablation study for the performance impact of each proposed component (deviation-based weighting, length normalization, EOS regularization).", "extra_scores": {}}, {"review_id": "O8qcUWmDQ7", "reviewer": "Reviewer_wVBW", "summary": "This paper introduces a reference-free multi-preference optimization method. It combines some of more successful techniques in the literature such as deviation-based weighting and length-normalized scores. In addition, an additional EOS regularization term is introduced to further control the termination of model predictions. On both Mistral and Llama models, the experiments show that the proposed method (REFA-dynamic) significantly outperforms various baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method is well-motivated and for the most part straightforward to follow. I find the section on URSLA illuminating and intuitive.\n2. The gains appear solid, although see the question section below for one of my concerns.\n3. The ablations are good additions, answering many questions I had in mind while reading through the experiment section.", "reasons_to_reject": "My main reason for rejection concerns the lack of details on the many variants of the proposed methods, all presented in the bottom block of rows in Table 2. It's critical information and I couldn't find relevant information despite multiple readthroughs. For instance, what is \"dyamic\", \"1-vs-k\", and how do you combine REFA with \"InfoNCA\"? For this reason, I'm giving a below-acceptance assessment, although I do believe with proper explanations it should merit a higher score.", "questions_to_authors": "1. The main results in Table 2 look the best for AlpacaEval 2, but less so for Arena-Hard, and much less for MT-Bench. I wonder if there's an alternative way to evaluate two systems head-to-head (i.g. pairwise comparison) so that we can conclude method X is significantly better than method Y.\n2. Out of curiosity, do you have a plausible explanation for why the trends for Llama-3 looks different than those for Mistral? For instance, REFA-1-vs-k (p=0) performs the best for Llama-3 on Arena-Hard, but not so much for Mistral.\n3. The paper is very heavy on analytical and quantitative results, but I think it would benefit greatly from some additional examples to show how the proposed methods might help (e.g., how did EOS regularization help on a specific example?)", "extra_scores": {}}, {"review_id": "wPrSMloK4O", "reviewer": "Reviewer_9iEL", "summary": "The paper proposed an interesting alignment method which can control for various dataset biases.\n\nThe experiment setup is good and clean in the paper. However, it makes some wrong assumptions (bias towards brevity) and makes some unnecessary SOTA claims which led to my overall score.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Mathematically interesting approach which is explicitly designed to tackle biases present in preference data.\nGood list of related methods to which authors compared their approach.\nGood experiment setup: same starting model, same data, vary only approaches.", "reasons_to_reject": "Two main concerns:\n1) The authors start with assumption that existing preference datasets have bias towards brevity. In my experience it is quite the opposite - the longer response is often preferred by humans and, especially by models from GPT-4 family. This has been extensively shown in the literature as well: https://arxiv.org/pdf/2407.01085 (Figure 7) and https://arxiv.org/pdf/2410.16184 (Figure 2).\n\nThis is especially true for UltraFeedback dataset which is used in this work. In that data, longer responses are typically preferred.\n\n2) SOTA claims.  Authors claim that they get SOTA. But https://huggingface.co/jieliu/Storm-7B from “Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level” is based on Mistral-7B and has 50.5% - 50.3% on AlpacaEval 2 https://tatsu-lab.github.io/alpaca_eval/ (Community) and https://huggingface.co/FuseAI/FuseChat-Gemma-2-9B-Instruct has even higher scores", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0S5VVwmnkV", "reviewer": "Reviewer_zZqs", "summary": "The paper presents a new method for offline alignment which is reference-free, multi-preference and combines 3 different signals: length normalization, deviation-based weighting, and an EOS probability regularizer.\n\nThe paper provides a good motivation for each component in its algorithm and a good motivation for each (while justifying the differences to previous work). \n\nI found the writing sometimes a bit \"dense\" and I can see some whitespacing mistakes that seem like for fitting the paper into 9 pages. E.g: line 290 and 293 there is no space after the full stops.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper seems to show good results on benchmarks that have been used on similar papers (AlpacaEval, MT-Bench, ArenaHard). It also presents a good motivation for each of the components in the loss and how they contribute for the final performance.", "reasons_to_reject": "I don't have any reason to reject. The authors setup has been standard to compare offline methods such as this. Nonetheless I believe there are improvements to be made to make the setup more robust.\n\n1) What were the  hyper parameters used on each method? Results for these methods are highly sensible to hyper-parameters and the authors acknowledge that but then don't show what they used or what they tried\n2) The setup only uses preferences taken from other models (off-policy) but how does their method compare to other methods when data is on-policy? For this I would recommend a different setup where all samples are taken from the SFT model and then ranked according to an open reward model like ArmoRM or Skywork.\n3) How does it compare to WPO? WPO is a recent PO method where they actually weight each sample according to the models policy and it seems to achieve also great results in \"length-controlled\". This is more for my curiosity because I think authors used enough methods.", "questions_to_authors": "Questions are written above.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces REFA, a new method for training language models to better follow human preferences by addressing optimization shortcuts that can lead to overly short answers. REFA uses a combination of deviation-based weighting to focus on high-impact examples, length normalization, and a specific End-of-Sequence (EOS) probability regularizer to encourage richer, higher-quality outputs. Experiments show that this approach allows REFA to perform significantly better than other reference-free methods on benchmarks such as AlpacaEval2.

Based on the reviewers' observations and evaluation, and given the positive discussion and clarifications during the review period, I agree that the paper: i) the paper is well structured and motivated; ii) the experimental setup is sound iii) the findings and proposed methods are exciting and relevant for the community; iv) the empirical results support the motivation and theoretical claims. The risks and weaknesses were addressed, clarified, or can be included in the final version of the paper. 

Therefore, I recommend this paper to be accepted. I would suggest that the authors include the complementary results and incorporate the discussion and  recommendations provided by the reviewers.
Decision: Accept