ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
Authors: Hee Suk Yoon
Eunseop Yoon
Mark A. Hasegawa-Johnson
Sungwoong Kim
Chang D. Yoo
Venue: ICML 2025 poster (2025)
Keywords: RLHF
DAA
overoptimization
Primary area: deep_learning->large_language_models
TL;DR: In this paper, we proposed ConfPO, which identifies preference-critical tokens based on the training policy's confidence, thus requiring no additional models or compute for token selection.

Abstract: We introduce ConfPO, a method for preference learning in Large Language Models (LLMs) that identifies and optimizes preference-critical tokens based solely on the training policy's confidence, without requiring any auxiliary models or compute. Unlike prior Direct Alignment Algorithms (DAAs) such as Direct Preference Optimization (DPO), which uniformly adjust all token probabilities regardless of their relevance to preference, ConfPO focuses optimization on the most impactful tokens. This targeted approach improves alignment quality while mitigating overoptimization (i.e., reward hacking) by using the KL divergence budget more efficiently. In contrast to recent token-level methods that rely on credit-assignment models or AI annotators, raising concerns about scalability and reliability, ConfPO is simple, lightweight, and model-free. Experimental results on challenging alignment benchmarks, including AlpacaEval 2 and Arena-Hard, demonstrate that ConfPO consistently outperforms uniform DAAs across various LLMs, delivering better alignment with zero additional computational overhead.

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
Hee Suk Yoon
Eunseop Yoon
Mark Hasegawa-Johnson'
Sungwoong Kim
Chang D. Yoo
Abstract
Prompt
Arrange the numbers 5, 13, 99, 1, and 22 in descending order. What the first number the arrangement?
We introduce ConIPO, method for preference learning in Large Language Models (LLMs) that identifies and optimizes preference-critical tokens based solely on the training policy's 's confidence, without requiring any auxiliary models or com- pute. Unlike prior Direct Aligmment Algorithms (DAAs) such as Direct Preference Optimization (DPO), which uniformly adjust all token proba- bilities regardless of their relevance to preference, ConPPO focuses optimization on the most im- pactful tokens. This targeted approach improves alignment quality while mitigating overoptimiza- tion (i.c., reward hacking) by using the KL di- vergence budget more efficiently. contrast to recent token-level methods that rely credit- assignment models Al annotators, raising con- cerns about scalability and reliability, ConfPO is simple, lightweight, and model-free. Experimen- tal results on challenging alignment benchmarks, including AlpacaEval and Harna demon- strate that ConIPO consistently outperforms uni- form DAAS across various LLMs, delivering bet- alignment with zero additional computational overhead. The code is publicly accessible
Chosen
The numbers arranged in descending order would be 99, 22, 13, 5, and 1) The first number in the arrangement
Rejected
The numbers arranged in descending order would be: 22, 13, 5, 5650. the first number in the new arrangement
Figure I. Highlighted tokens green for the chosen, red for the rejected are those regarded low-confidence by the policy model, meaning their predicted probability falls below the sequence av- erage. This strategy surfaces tokens that carry high informational value or mark key decision points, while skipping predictable con- tinuations. For example, in "22" (tokenized as '2'; '2"), the first digit may be highlighted as uncertain, while the second, being a likely follow-up up, is not. The incorrect answer "1" in the rejected response is another such low-confidence token, reflecting both its unpredictability and its role in distinguishing preference.
Direct Alignment Algorithms (DAAs) (Rafailov et al., 2024b; Azar et al., 2024; Xu et al., 2024; Ethayarajh et al., 2024; Hong al., 2024; Park et al., 2024; Meng al., 2024) offer more efficient alternative by using the policy's own log- log-probabilities as an implicit reward signal. Among these, Direct Preference Optimization (DPO) (Rafailov et al., 2024b) has gained particular attention for optimizing the log-probability gap between chosen and rejected responses. SimPO (Meng et al., 2024), recent variant of DPO, im- proves this framework by removing the need for reference model and mitigating length bias, resulting in stronger align- ment with human preferences. Yet, these methods assume all tokens contribute equally to preference, applying uniform optimization across the entire sequence without considering the varying informativeness of individual tokens.
1. Introduction
Aligning Large Language Models (LLMs) with human preferences is commonly achieved through Reinforcement Learning from Human Feedback (RLHF) (Christiano al., 2017; Ouyang et al., 2022; Yoon al., 2024a), which fine- tunes policy using learned reward model. While effective, this approach introduces substantial computational overhead and is sensitive to reward model misspecification.
A growing body of linguistic and cognitive research shows that words are not equally informative- contribute little beyond confirming expectations, whereas others com- pel readers to revise their interpretation. Psycholinguistic surprisal theory (Hale, 2001; Jaeger & Levy, 2006; Smith & Levy, 2013) formalizes this insight by linking a word's 's in- formational value to its contextual predictability: the lower the conditional probability P(wss/conten) the higher the
Korrea Advanced Institute of Science and Technology (KAIST), Republic of Korea, Universsity of Illinois Siranaa-hhappal (UIUC), USA, Korea University, Republic of Korea. Correspon- dence to: Chang D. Yoo <cd_yoo@e @aaisstaack......
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
Simple Preference Optimization (SimPO)
Confidence-Guided Preference Optimization (ConfPO)
redparee
Fatture Jeb markee
Lopend
Worted 1800
Longunge
Longuaege
Large Language Modial
Sared 2000
My respence In boaed
future
job marker
Raded on carrent treed
jeks
2000
SSimPo(Ftg) =
confro (TTg)
-o to logol
sty sollogroo
- Bionoss- logot
Σ
logna 10 (.ll
where s(n)
10rp s
bul
KxdsGa 13I
Figure Overview of the Confidence-guided Preference Optimization (ConfPO), Existing DAA algorithm (SimPO (Meng al.. 2024)) uses all the tokens uniformly for preference alignment. ConfPO selectively optimizes on few tokens based on the policy model's confidence with no additional compute overhead.
surprisal and the greater the information conveyed. Mod- ern LLMs embody this principle by explicitly estimating for every token. Highly predictable tokens (high probability) add little new information, whereas low- probability tokens are information-rich and often determine how the rest of the sentence is interpreted (Merkx & Frank, 2021; Slaats et al., 2024). In essence, a small subset of low-confidence (high-surprisal) tokens carries most of the sentence-level information flow. We illustrate this idea in Figure I, which shows that tokens with below-average con- ditional probability tend be the unexpected, information- bearing tokens, whereas the remaining tokens generally highly predictable continuations.
Improved Alignment Performance. Across multiple
LLMs and alignment benchmarks, ConIPO outperforms DAAs that uniformly train on all tokens (Table 1).
Overoptimization Mitigation. By discarding tokens that contribute little to alignment, ConfPO makes more effi- cient use of the KL budget, reducing overoptimization (reward hacking) effects (Figure 7).
2. Related Works
Reinforcement Learning from Human Feedback (RLHF) Ensuring that models are helpful, safe, and factually accu- rate is a central goal of modern AI research (Yoon et al., 2022; 2023; 2024b;e; Yoon et al.). In particular, RLHF has become a widely used approach for aligning language models with human preferences. Early methods this area often relied on online reinforcement learning techniques, such as PPO with learned reward model, to adjust model behavior without requiring human input at every step (Chris- tiano et al., 2017; Ouyang et al., 2022). However, these ap- proaches are highly sensitive to hyperparameters and can be inefficient due to the need for extensive trajectory sampling. To address these challenges, Direct Alignment Algorithms (DAAs) have been developed as an alternative (Rafailov et al., 2024b; Azar et al., 2024; Xu al., 2024; Ethayarajh et al., 2024; Hong et al., 2024; Park al., 2024; Meng et al., 2024). DAAs eliminate the need for multi-stage stage training making the alignment process more efficient.
Motivated by these insights, we ask whether preference- relevant learning signals are better captured by selectively optimizing those low-confidence tokens. We provide empir- ical and theoretical analyses of gradient behavior in Direct Alignment Algorithms and show that policy model's own confidence scores form simple yet reliable proxy for iden- tifying preference-critical tokens. Building on these find- ings, we introduce ConfPO (Confidence-guidee Preference Optimization), a targeted extension of SimPO (Meng et al., 2024) that focuses optimization on the most informative tokens without auxiliary models or extra inference passes.
As illustrated in Figure 2, ConfPO dynamically filters out low-impact tokens, focusing updates only on those that meaningfully drive alignment. This targeted optimization strategy not only enhances performance on benchmarks such as AlpacaEval 2 and Arena-Hard, but also mitigates overop- timization by utilizing the KL budget more efficiently.
Token-level DAAS Sequence-level RLHF assigns re- ward signal the entire sequence, treating all tokens as equally contributing. However, this approach makes it chal- lenging to obtain fine- aineggrained learning signals. To address this limitation, recent efforts have focused on token-level reward signals for RLHF, particularly in DAAs. FIGA (Guo et al.) utilizes an external language model to generate data, which is then used for imitation learning. TDPO (Zeng et al.) enhances alignment and diversity by leveraging forward
In summary, our key contributions are:
Efficient Selective-Token Alignment. We propose method that uses the policy model's confidence scores to identify tokens most critical for preference learning. Guided by empirical and theoretical gradient analyses of DAAs, this approach enables selective optimization without additional computational overhead (Section 4).
2

---

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
KL divergence, while SePO (Yang et al., 2024) improves weak-to-strong generalization by pre-selecting tokens for preference learning using weak model. T-REG (Zhou et al., 2024), in contrast, does not rely on external models but instead obtains token-level rewards by multiple forward passes through prompt augmentation. Despite these ad- vancements, existing methods often introduce additional computational complexity, either through external models, extra training, or synthetic data generation. In this paper, we propose approach that enables loken-level DAA while maintaining the computational efficiency of existing DAAs.
RLHF proceeds in two stages: training the reward model and then using it to guide a policy model. In the first stage, the reward model f(z, y) is obtained via maximum likelihood estimation on the preference data. In the second stage, f(z, y) provides a feedhack signal that drives optimization of the language model's policy To. Let D₂ represent the distribution of prompts contained within D. The objective can be written as:
max En (1))) [(1, y) - , log (1a
(2)
where 3 is a coefficient controlling the penalty for deviating from a reference policy Teef-
Overoptimization in reward models has been identified as a critical issue in RLHF. Gao et al. (2023) highlighted that reward models in RLHF can suffer from overoptimization, leading to suboptimal align- ment. Later, similar concerns were observed in DAAs, where Rafailov et al. (2024a) demonstrated that overopti- mization can occur even without explicitly training a reward model. To mitigate this issue, Liu et al. (2024) proposed using a supervised loss as a regularizer, suggesting that it can help alleviate overoptimization. Additionally, Anony- mous (2025) introduced an approach based on importance sampling, demonstrating its potential to mitigate the overop- timization problem. We show in this paper that our pro- posed ConfPO tends to mitigate overoptimization due to the efficient use of the KL budget.
Overoptimizaion in RLHF
3.2. Direct Alignment Algorithms
Direct Alignment Algorithms (DAAs) offer an alternative to traditional RLHF by removing the need for explicit reward model training. Instead, these methods directly optimize the policy model using preference data, leveraging implicit signals from the model's 's own probability distributions.
Direct Preference Optimization (DPO) DPO (Rafailov et al., 2024b) reparamerterizes the reward function r using a closed form expression as:
(2, y) = P log Foly x))
3 log ((x)
Tyeryy x)
lul
ma(yi x,yeL) Terly: ,,yess
3
log
+ Blog ((x).
(3)
3. Background and Problem Formulation
where Tg is the policy model, "ee is the reference policy, typically the supervised fine-tuned (SFT) model, and (xx) is the partition function. This reward formulation incorpo- rated into the BT ranking objective (Eq. 1), allowing DPO to express the probability of preference data directly with the policy model, rather than separate reward model, yielding the following objective:
In this section, we provide an overview of preference leam- ing for aligning large language models (LLMs) with human feedback and Direct Alignment Algorithms (DAAs).
3.1. RLHF: Preference Learning for Language Models
Reinforcement learning from human feedback (RLHF) (Christiano et al., 2017: Ouyang et al., 2022; Yoon et al., 2024a) is a commonly adopted approach to aligning large language models (LLMs) with human preferences. By leveraging annotations from humans (or Al systems), RLHF encourages LLMs to produce outputs that more closely match user expectations. Formally, let D {a(1), ",, y.,(()),N 1 be preference dataset where each entry contains a prompt 20(() alongside two candidate responses: a preferred response () and an unpreferred response g(6 The core of RLHF lies in a reward model r" r²(z,y) which assigns a scalar indicating how well a response y aligns with human preferences for a given prompt v. A widely used model for pairwise preferences is the Bradley-Terry (BT) framework (Bradley & Terry, 1952), which specifies the probability that yd is preferred over y' as:
Copo(TTi mear) = Bea."" log ( 1x To('"" (z v., " (z, v..)
log
To mo las stew) Ter (r, vt }]
log
(4)
Simple Preference Optimization (SimPO) The reward function in Eq. 3 presents several limitations. First, it re- lies on a reference model निरर during training, which is not required during inference. This creates an inconsistency between the training objective and the generation strategy at inference time. Second, it employs the summed log prob- abilities of tokens as the reward, which introduces length bias bias-longer sequences tend to have lower log probabili- ties. As a result, the model may overestimate probabilities for longer sequences to ensure that y" is assigned a higher reward than y¹.
x)) ply": th > n (z) = o (r" (2. - r" (x,,))).
3

---

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
To mitigate these issues, SimPO (Meng al., 2024) rede- fines the reward function as follows:
lvl
TSimPO (z,, y) = log To 2) + 1og ((x)
8 ly|
r(z, v) =
8 1061 Σ s(x)) log (a 2z + B log ((x), (7)
log (u 2. Blog Z(x). (5)
where syy;) is selection function that determines whether to include the token yi for reward calculation, and 1ys de- notes the total number of selected tokens. Incorporating this selective token-based reward formulation, we refine the preference optimization objective of SimPO (Eq. 6) as follows:
1
Using this implicit reward function, the objective for SimPO becomes the following:
10 log ( (,'l v())
(Sim = (e.g" w/D (a)))
log 0
[(TT))
W log logno (ofla, ].
1 s(vl") ) log mo( lz, w.s)
B
(6)
13 (nn w'D logar(
B
Σ
where y is a target margin introduced by Meng et al. (2024) to ensure that the reward for the preferred response (s, (2, v²) exceeds the reward for the unpreferred re- sponse TSimPO (z, y²) by at least y. SimPO demonstrates significant improvements on various representative bench- marks (i.e., AlphacaEval 2 (Li et al., 2023) and Arena- (Li et al., 2024)) compared to DPO, whereas many other variants of DPO (Yuan et al., 2023; Zhao et al., 2023; Azar et al., 2024; Xu et al., 2024; Ethayarajh et al., 2024; Hong et al., 2024; Park et al., 2024) fail to show a consistent performance advantage over the standard DPO. Thus, our method is built upon SimPO; however, in Section 7.2, we also demonstrate that our findings generalize to DPO.
B 1ud
10' (v)) log (vll [
(8)
While previous works have proposed token selection strate- gies, these approaches often involve significant computa- tional overhead. Examples include performing multiple forward passes with augmented prompts (Zhou et al., 2024), leveraging annotations from powerful LLMs such as GPT-4 (Lai et al., 2024; Yoon al., 2024a), training auxiliary models to guide token selection (Yang al., 2024). These methods, while effective, are complex and expensive, limit- ing their practicality.
Unlike prior approaches, we aim to derive s(y:) directly from the internal signals of the model during training, ensuring that the computational cost remains identical to that of the original DAAs.
3.3. Problem Formulation: Token Selection for Preference Learning
Although Direct Alignment Algorithms (DAAs), such DPO (Eq. 4) and SimPO (Eq. 6), have demonstrated strong performance in aligning LLMs with human preferences, they are typically optimized over all tokens in the training dataset. However, prior studies (Lin et al., 2024; Chen et al., 2024; Lai et al., 2024; Yoon et al., 2024a) have shown that not all tokens contribute equally preference alignment. Uniformly optimizing over all tokens can introduce noise into the training process.
4. Revisiting the Gradients of Direct Alignment Algorithms (DAAs)
In this section, we introduce series of observations reveal- ing how to use the internal signals of the training policy model to select important tokens for preference learning. We focus on the SimPO (Meng et al., 2024) due to its state- of-the-art performance, but we show that these observations also hold for DPO (Rafailov et al., 2024b) in Section 7.2.
Selective Token Reward Formulation To address these challenges, we propose improving DAAS by optimizing only on effective tokens for preference learning. Specifi- cally, we aim to identify and leverage a subset of tokens that are critical aligning with human preferences, with- out incurring additional computational costs compared to standard algorithms like SimPO (Meng et al., 2024).
Gradient of SimPO As illustrated in Meng et al. (2024), the gradients of SimPO (Eq. 6) can be written as follows:
(T = -91 (x. w w'WD
Vo log Io (v=
1
We extend the implicit reward formulation of SimPO (Eq. 5) by incorporating a selective token scoring function. The re- ward for a response y is redefined as the joint log probability of critical (i.e., selective) tokens, expressed as:
V log 10ggo(lll)))
(9)
y
where
se o (
8 log To (p(( 1
log To
4

---

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
Profored Reoporse
Unpeslored Resporac
[O
Exceh Trectin 00
Tooch Prcdlee 00
Opoch Paotion 00
Eyoch Factoon 05
Epech Proctie 10
2000
Fack 05
Pactooe 2-
Fack Fradien
1000
Fladio
Gnd Value
2000
Grdd Vala
1000
2000
caed
Condlaco Calfilen 0.773
Gradien Your hous
Gestiet how Hakes
Hears
Figure 3. Observation 1. Token-level gradient norm values (i.c., Vo log [V iz, v=ell) follow long-tailed distribution for both the preferred response (left) and the unpreferred response (right). unpreferrees
(4) Cormisaton betweser log probebiiity and grad norn acroes spach fruction
Gpach Flaction Do
Epuch Fludion 0S
Epuch Fraction
Each distribution is computed from 10 sampled sentences, and Llama-3-Bass (8B) was used. This observation shows that not tokens contribute equally to preference alignment.
Low-Conliinna Tours
Lawoorfdess Tihera
Lovienffaraa Tohee
1205
1000
In this gradient equation, the sequence- gradients, Vo log mo [z) and Vp log (x)l) govern how the model adjusts probabilities across the tokens. To better understand the token-level contributions to the gradient, we can decompose these sequence-level terms (Eq. 10). For brevity, we focus on the gradient with respect to a single response y (either y or y ). as the expressions for both are structurally identical.
Valee
Crodect
(6) Grad nore seporation across epoch Faction for preferted response (n))
Fration do
Gpach Fandia da
Spach Fiedian
ligeCarddeoo Tstars
High-Grroera Takes)
Ligh-Go herve Toka
[6 Grad norn separation across epoch Fraction for unproforedd resporse (r)
lul Ve log no la, (e,
Figure 4. Observation 2. (a) Spearman correlation plot between the policy model's log probability (confidence) for cach token, log Th (y |z, y<z)). and its gradient norm. || Ve log we (Hu (z, y<c) )l- over the course of training. A strong negative correlation emerges for tokens in both the preferred (y"") and unpreferred (y') response, indicating that tokens with lower probabilities tend to have larger gradients. (b-c) For both y and y', the tokens are split into two groups based whether their probabilities exceed fall below the average probability. We then illustrate the gradient norm distributions for each group, revealing distinct gap throughout training: high-confidence tokens generally show smaller gradients, while low-confidence tokens exhibit consistently larger gradients.
Vo log 2))
(10)
4.1. Analyzing Token-Level Gradients in Direct Alignment Algorithms (DAAs)
Observation 1: Token-Level Gradients Follow Long- Tailed Distribution To understand how individual tokens contribute to preference aligmment, we analyze the gradient norm V log Te (yllz, )l for each token in both pre- ferred (y)*) and unpreferred (y/) responses over the course of training (see Figure 3). Our findings reveal a clear long- tailed distribution: although most tokens exhibit minimal gradient updates, a small subset accounts for disproportion- ately large gradients, indicating that these tokens drive the bulk of the learning signal. This observation aligns with ear- lier work on language model pretraining (Lin et al., 2024), which similarly reports that focusing on a fraction of crit- ical tokens can significantly enhance model performance. In the context of preference learning, the prevalence of low low-gradient tokens may indicate their limited relevance to human preference signals, diluting the model's overall effectiveness when training indiscriminately on all tokens.
To address this, we look for simpler proxy that can be measured without extra forward or backward passes. As shown in Figure 4-(a), we find a strong negative correlation between the model's probability (confidence) assigned to a token, Te To (z, yes). and the magnitude of its gradient norm, || V o log no To (z, Ve.))l |.. To further analyze this relationship, we group tokens based on whether their probability exceeds or falls below the average probability. Figure 4(b-c) presents the gradient norm distributions for each group, revealing a clear distinction: high-confidence tokens consistently ex- hibit smaller gradients, while low-confidence tokens have significantly larger gradients, for both the preferred (y") and unpreferred (n') responses.
There is a High Correlation between the Gradient Norm and the Confidence of a Token Building on Observation 1, one potential strategy im- prove preference learning might be to prioritize tokens with large gradient norms. However, directly computing ||V, log Te(yyla Ve for every token is prohibitively ex- pensive, since it requires a dedicated backward pass per token- an approach that would greatly exceed the computa- tional budget of standard preference learning.
Observation 2:
These findings suggest that tokens with the high learning signals (i.e, those critical for preference alignment) can be effectively identified using only the model's confidence scores. Importantly, this approach maintains the exact same computational cost as standard preference learning, making it both practical and scalable. A theoretical account of this relationship is presented later in Section 4.2.
5

---

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
fokors Binnd
Cigh-Conlionne Tokera only
LonCerhhoncs Tokor Only
1.0
1.0
088
0.5
066
0.6
0.4
0.4
Absurure
0.
0.2
0.0
(5) Mistrai-Base (78)
(6) Liamma Base (88)
Figure 6. Absolute Shapley value decomposition of the gradient-
Stepe
(4) Revard accuracy across training steps
(b) Human aligrment across training stops
norm ratio = VoTe z, v<ss)l// (3k 2, f.yess, illustrat- ing the relative contributions of its numerator (b;) and denominator (e:). Results are shown for both Mistral-Bars (7B) and Llama- Base (8B), where c; consistently accounts for larger share of the total attribution.
Figure 5. Observation 3. We conduct SimPO learning using three settings: (i) all tokens, (ii) only high-confidence tokens, and (iii) only now-confidence tokens. (a) Each setting calculates its reward based the selected tokens, and all three eventually reach high accuracy-indicating that, within their respective token subsets, the model discriminates well between preferred and unpreferred re- sponses. (b) Despite converging a similar reward accuracy, train- ing solely on high-com nigh-confidence tokens yields no gain alignment with human preferences. This suggests that high-confidence to- kens carry limited preference information, whereas oo-coofideecc tokens more strongly drive alignment improvements.
4.2. Theoretical Rationale for Why Low-Confidence Tokens Carry Strong Learning Signals
Looking back at the log-probability gradient norm of a single token Ve log To (a,, yes),, this can be equally ex- pressed as follows using the chain rule:
OV (es))
V log (o ves)
(11)
To Ir M<s))
Observation Cow-Confidence Tokens Dominate Pref- erence Learning Building on Observation we further explore whether low-confidence mhigh-gradient tokens in- deed provide more critical preference information than high- confidence (low-gradient) tokens. Figure 5 compares hu- man alignment outcomes (via the AlpacaEval2 (Li et al., 2023) Win Rate) under three conditions: training on only high-confidence tokens, only low-confidence tokens, and all tokens. Despite all three approaches achieving high training reward accuracy within their respective token subsets, their human alignment trajectories differ significantly:
Thus, if token confidence na no [z, Meei) dominates this ratio, the log-pro log-robbbiility gradient norm [ log ITa (1 z y-ei decreases as confidence increases, establishing the observed negative correlation.
Shapley Value Analysis To formalize the dominant role of the token probability No (w( yes) in determining the log- probability gradient norm |/ log To To (22, F<+) [] we consider the ratio by where b, = || Voto (I (r,, Ve:)) and c; = To(y [z, Mes). The objective is to quantify the extent to which each input, b, and c. contributes to variation in the output r,.
To fairly quantify these by and e; are not independent but are both functions of @-we use Shapley values (Shapley, 1953; Lundberg & Lee, 2017), a concept from cooperative game theory that offers principled approach to attributing a model's output to its individual input features. While the original frame- work assumes feature independence, recent extensions ac- count for dependencies between input features (Owen Prieur, 2017; Aas et al., 2021). In particular, we adopt the methodology introduced by Aas al. (2021)', which accurately estimates feature contributions under feature de- pendence. As shown in Figure 6, the results demonstrate that the denominator e; contributes substantially more to the variation in r; than the numerator b; providing theoretical support for the empirical observation that low-confidence to- kens (Ta (yx [z, yei) 4) tend to have stronger learning signals (I) V log To To(yl 22 v<:) 1).
pattiioon--pttiuularryyyy since
1. Coggh-Confeeecce Tokens Only: This yields negligible or even negative improvements, suggesting that highly confident tokens offer limited learning signal and may introduce noise into the optimization.
2. Cow-Confieence Tokens Only: Restricting training to these
tokens significantly boosts alignment, surpassing even the baseline that uses all tokens.
These findings align with earlier work (Lin et al., 2024) showing that not all tokens equally valuable. In fact, high-confidence tokens can impede preference learning by overshadowing more informative signals from their low- confidence counterparts. Consequently, selectively empha- sizing low-confidence tokens appears to be a straightfor- ward yet powerful way to enhance preference alignment.
6

---

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
5. ConfPO: Confidence- Critical Token Selection for Preference Optimization
We then apply pref- erence optimization using the UltraFeedback dataset (Cui et al., 2023). For the Instruct setup, we begin with instruction-tune tuned models (i.e., mistralai/Mistrall IB-sstrucc-- v0.2 and models. Then, we use the preference data generated by Meng et al. (2024) which is derived from prompts in UI- trafeedback dataset and scored using I1m blandee//r/// reward model. The full implementation details can be found in Appendix D.
Motivated by our observations and theoretical rationale in Section 4, we introduce our proposed soffidenne-guidee crit- ical token selection for preference optimization (ConfPO).
as the SFT
Confidence-guided Selective Token Reward Formulation Using the fact that low confident tokens contain high learn- ing information for preference learning, we propose the following implicit reward as a continuation for the selective token reward formulation that was introduced in Eq. 7:
6.2. Evaluation Benchmarks
lsl s()) log (p | v<,) + 8 log (xx),
We assess models two cnorrcctionfollowinng benchmarks: AlpacaEval 2 (Li et al., 2023) and Arena-Hard v0.1 (Li et al., 2024). AlpacaEval 2 consists of 805 questions from five datasets, while Arena-Hard v0.1 is an extension of MT- Bench (Zheng et al., 2023), comprising 500 problem-solving queries. For AlpacaEval 2, we report both the win rate (WR) and the length-controllee win rate (LC) (Dubois al., 2024), where the LC is designed reduce the length bias. For AlpacaEval 2, we use alpaca.eval.gptt4. .ur.b.....pt...ur.o........ annotator, which has a higher human agreement score then weight www.apaaa.eeaa..g.....oo turb used in Meng et al. (2024).
8
"Ccurro (It, y)
(12)
where s(6) 1(To(BilZ I r) and |aal Hys1s(s)) = 1H}-
Then, the resulting preference optimization objective of ConIPO can be expressed as:
(To)
1 =(v²')
( (
3
Σ
(vg
6.3. Baselines
8
( log No (ullo, vies) - ]
Similar to Meng et al. (2024). we compare ConPPO with other Direct Alignment Algorithms (DAAs) listed in Table the Appendix. RRHF (Yuan et al., 2023) and SLiC-HF (Zhao et al., 2023) use ranking-based losses, with RRHF normalizing log-likelihood by length, while SLiC-HF di- rectly optimizes log- likelihood alongside an SFT objective. IPO (Azar et al., 2024) addresses the assumption of DPO (Rafailov et al., 2024b) that pairwise preferences can be treated as pointwise rewards. CPO (Xu et al., 2024) opti- mizes sequence likelihood while training with SFT. KTO (Ethayarajh et al., 2024) learns from non-paired preference data, and ORPO (Hong et al., 2024) removes the need for a reference model by introducing odds ratio term to con- trast winning and losing responses. R-DPO (Park et al., 2024) adds regularization term to DPO to mitigate length bias. SimPO (Meng et al., 2024) resolves the discrepancy between the training objective and inference in DPO while also mitigating its length bias, resulting in a significant im- provement in performance.
(13)
|bc|
where a(v!)) 1 (v) Jr, voa) S "1 1v.1
= KKu (s(xt)
1H. s(xd) = 1 ( v I 7'), and lul| = (Edds((() =
We define the confidence thresholds -h and 7' as the aver- age token probabilities across the respective responses:
1v (o Ir, v.D,
wi To To(y l., slec)
(14)
In Appendix B, we empirically demonstrate that this thresh-
olding strategy outperforms other thresholding choices.
6. Experimental Setup
6.1. Models and Datasets
For the SimPO baseline, we retrained the model using the hyperparameters specified in their original paper (Meng et al., 2024). For other DAA baselines, we utilized the pub- licly available checkpoints provided by Meng et al. (2024). The hyperparameters used for ConIPO are listed in Table 6. Notably, we observe that ConfPO generally favors lower optimal and a higher ? compared to SimPO.
Following Meng et al. (2024), we conduct experiments us- ing two families of models: Mistral-7B (Jiang et al., 2023) and Llama-3-8B (AI@) Meta, 2024), under both Base and Instruct configurations. For the Base setup, we utilize base models that have been fine-tuned via Supervised Fine- Tuning (SFT) on the UlrrChat--22 dataset (Ding et al., 2023) (i.e., alignment- handbook/ephhy--7-bstt and
7

---

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
Table AlpacaEval (Li al., 2023) and Arena-Hard (Li* al., 2024) results under the four settings. T.C. and WR denote length- controlled and raw win rate, respectively. For AlpacaEval 2. we use the s.ppaa.eva...pt 4.tppt.tuuee uurufoo.nn LLM annotator. SimPO (Meng et al., 2024) was retrained; other baselines utilized checkpoints provided by Meng al. (2024) (see Section 6.3 full details). We report the standard deviation in Appendix E.
Mistral-Base (7B)
Mistral-Instruct (7B)
Method
AlpacaEval 2
Arena-Hard
AlpacaEval
Arena-Hard
LC (%)
WR (%)
WR (%)
LC (%)
WR (%)
WR (%)
SFT
4.84
2.92
1.3
24.4
18.8
12.6
RRHF (Yuan et al., 2023)
12.7
10.2
5.8
36.0
33.0
18.1
SLiC-HF (Zhao et al., 2023)
14.5
11.3
7.3
33.6
32.1
18.9
DPO (Rafailov et al., 2024b)
18.1
16.0
10.4
36.1
31.9
16.3
IPO (Azar al., 2024)
13.2
10.6
7.5
28.4
26.8
16.2
CPO al., 2024)
12.8
11.4
6,9
29.8
38.0
22.6
KTO (Ethayarajh et al., 2024)
12.8
8.26
5.6
32,9
29.8
17.9
ORPO (Hong et al., 2024)
18.7
13.8
7.0
32.0
30.7
20.8
R-DPO (Park et al., 2024)
21.2
15.6
8.0
32.2
27.6
16.1
SimPO (Meng al., 2024)
27.1
23.4
13.8
37.0
37.3
21.0
ConfPO (ours)
28.9
26.7
16.9
39.1
38.4
22.4
Llama-3-Base (8B)
Llama-3-Instrutt (8B)
Method
AlpacaEval 2
Arena-Hard
AlpacaEval
Arena-Hard
(%)
(R())
WR
(%)
(R())
WR (%)
SFT
6.86
3.79
3.3
33.1
32.2
22.3
RRHF (uaan al., 2023)
13.8
9.69
6.3
35.4
32.9
26.5
SLiC-HF (Zhao et al., 2023)
21.4
24.3
6.0
35.7
35.6
26.2
DPO (Rafailov et al., 2024b)
25.4
23.9
15.9
43.8
42.3
32.6
IPO (Azar et al., 2024)
22.5
25.5
17.8
42.9
42.4
30.5
CPO (Xu al., 2024)
21.9
23.0
5.8
35.7
38.7
28.8
KTO (Ethayarajh al., 2024)
24.7
22.1
12.5
41.0
39.7
26.4
ORPO (Hong et al., 2024)
17.6
14.2
10.8
36.7
34.7
25.8
R-DPO (Park et al., 2024)
21.9
25.0
17.2
46.2
43.9
33.1
SimPO (Meng al., 2024)
27.0
25.7
21.1
48.8
44.4
32.6
ConfPO (ours)
28.3
32.7
23.8
49.1
45.0
32.8
7. Experimental Results
7.1. Main Results and Ablations
Main results. Table 1 shows the main results of our periments. As iterated in Meng et al. (2024), most DPO variants confer minimal performance benefit over standard DPO, whereas SimPO delivers substantial performance gains across all tested models. We introduce ConIPO, a selective-token token variant of SimPO that leverages the model's confidence signals for preference learning, which further hances performance highlighting the effectiveness of confidence-driven token selection in optimizing preference alignment.
SwH
SanPO
Squan roonooo dieigenos
Sqare rodcof dieegnno 10
[N) Mistrrs-fBlos
Lamma-3-eee (88)
Figure 7. Results on Overoptimization. Each point in plots represents model trained with different hyperparame- ter. For both models (a) Mistral- Mistral-Base (7B) and (b) Llama-3 Base (8B) the curve shows that ConfPO places higher than the SimPO, suggesting that mitigates overoptimization. This improvement attributed ConIPO's efficient use of the KL budget, allow- ing for better alignment without excessive updates on non-critical tokens.
SimPO every evaluated model,
Selective-token preference learning mitigates overop- timization. Although Direct Alignment Algorithms (DAAs) do not rely on a separate proxy reward model, Rafailov et al. (2024a) have shown that they still suffer from overoptimization (or reward hacking). Overoptimization is often illustrated using a Human Alignment vs. Squared KL Divergence plot, where increasing the KI. budget beyond
certain point leads to decline in model performance (i.c., that it shows a humped-shaped pattern) (Gao et al., 2023; Rafailov et al., 2024a). Figure presents the overoptimiza- tion plots for (a) Mistral-Ban (7B) and (b) Alama-3-Bar 3-aase
8

---

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
Table 3. AlpacaEval 2 (Li et al., 2023) results comparing DPO and ConfPO applied to DPO (ContPOpeo (ours)).
050
Toaned Toaan thao
WWW l
Mistral-Buse (7B)
1lama-3-Base (8B)
Method
AlpacaEval 2
AlpacaEval 2
Paec Unpreteered Paaparme S/1
(C)
WR))
(%)
WR (%)
Uicander and Sesppior 101
SFT
6.2
1.3
17.1
Stepo
Stepo
(6) MattralBeea (78)
Lieem----Baaa
DPO ConfPOpeo (ours)
18.1
16.0
25,4
23,9
Figure 8. Number of Tokens Selected During Training. For hoth models-(a) Mistral-Base (7B) and (b) Llama-3-Base (8B)-the criteria
23.3
19.8
29.4
27.5
number of tokens selected by our confidence-guidee (Eq. 13) gradually increases over the course of training. Notably, ConfPO optimizes fewer than half of the total tokens for prefer- ence learning, yet achieves superior performance compared to the baseline, which utilizes all tokens (see Table 1).
lizes all tokens. This outcome highlights the importance of principled selection strategy, as simply reducing the number of tokens without informed selection does not yield performance gains.
(8B). The results indicate that ConfPO effectively mitigates the overoptimization issue observed in SimPO by utilizing the KL divergence budget more efficiently. This could be attributed the seken-selection: since we only perform pref- erence learning with fraction of tokens that are important for the learning, it prevents meaningless divergence, and thus mitigates the overoptimization.
The
7.2. Applicability on DPO
Throughout this paper, we have used SimPO (Meng et al., 2024) as our primary baseline. However, the observations presented in Section 4 hold equally for DPO (Rafailov et al., 2024b) (Eq. 4). Supporting plots for DPO can be found in Appendix C. In this section, we extend our Confidence- Guided Token Selection (ConfPO) approach to DPO, mod- ifying its objective as follows:
Number of tokens selected. Figure 8 illustrates the pro- portion of tokens selected throughout training. At the start, approximately % of tokens are selected for both Mistral- Base (7B) and Llama-3-Bass (8B). As training progresses, the selection ratio gradually increases, reaching 45% and 49, respectively, by the end. Notably, despite using fewer than half of the total tokens for preference learning, ConfPO outperforms SimPO, which utilizes all tokens. This result further underscores that not all tokens contribute equally to preference learning and that selective token optimization can enhance the alignment process.
LeonfrOpmn (To; Tiet)
To(v!" (zz, v (ue" (z, v
-B wDDD log o s(vi" log
( le, wlee) log Teet (v' x2, v'2c)
B
> 1=1
(vxt)
(15)
where s(v(") = 1 ( 00" Jz,, v2.) < 7"",, 1u21 (sv"")
1).. s(v)) = 1 (.ll, S 7¹). and In'l Hylls(ud
1).. As in Eq. 14, we define the thresholds 7 and rf as the average token probabilities.
Table 2. AlpacaEval (Li ct al., 2023) results for random token selection (ConfPO-rand).
Table 3 shows that ConfPOpro achieves better human align- ment than DPO on the AlpacaEval 2 (Li et al., 2023) bench- mark. This demonstrates that our confidence-guin -giieed token selection method is not limited to SimPO but can also en- hance other Direct Alignment Algorithms (DAAs).
Mistral-Base (7B)
Llama-3-uuse (SB)
Method
AlpacaEval 2
AlpacaEval
LC (%)
WR (%)
(%)
WR(%)
SFT
4.84
2.92
6.86
3.79
SimPO (Meng et 2024)
27.1
23.4
27.0
25.7
ConfPO-rand
23.3
21.4
24.0
22.8
8. Conclusion
CoufPO (ours)
28.9
26.7
28.3
32.7
We introduced Confidence-Guidee Token Selection (ConfPO)) an approach to preference learning that selec- tively updates low-confidence tokens, which we identified crucial for human alignment. Our method improves the performance of mate-of-thh-art DAAS, including SimPO and DPO, while maintaining the same computational cost. By efficiently utilizing the KL budget and updating fewer than 50% of tokens, ConfPO mitigates overoptimization and enhances alignment without additional overhead.
Random token selection does not improve preference learning. To further assess the effectiveness of our selec- tion strategy, we conduct a control experiment using random token selection while maintaining the same selection ratio as our method. As shown in Table 2, applying preference learn- ing to randomly selected tokens (ConfPO-rand) results in worse performance than the baseline (SimPO), which uti-

---

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
Acknowledgments
Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computational Linguis- tics: EMNLP 2024, pp. 7889-7903, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi:
This work was supported by Artificial intelligence indus- trial convergence cluster development project funded by the Ministry of Science and ICT(MSIT, Korea)&Gwangju Metropolitan City, Institute for Information & communi- cations Technology Planning & Evaluation (IIPP) grant funded by the Korea govemment(MMIIT) (No.RS-2021- 11211381, Development of Causal AI through Video Un- derstanding and Reinforcement Learning. and Its Applica- tions to Real Environments), and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea govemment (MSIT) (No.RS-2022 I1220184, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics).
463. URL https: pestps://aclannee //t//aaaantttt //t.aaaaaanttooooy.o.o.o..o .rgg 2024. f Sindings-entn 463/.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from hu- man preferences. In Guyon, L., Luxburg. U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Cui, G., Yuan, L.,, Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. Ultrafeddack: Boosting language models with high-quality feedback, 2023.
Impact Statement
This paper does not violate the use of others' work without reference. Furthermore, the paper does not involve intro- ducing new datasets and the experiments conducted do not utilize demographic or identity characteristics. This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Ding, N., Chen, Y., Xu, B., Qin, Y., Hu, S., Liu, Z. Sun, M., and Zhou, B. Enhancing chat language mod- els by scaling high quality instructional conversations. Bouamor, H., Pino, J., and Bali, K. (eds.), Pro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 3029-3051, Singapore, December 2023. Association for Computa- tional Linguistics. doi: 10.18653/v1/2023.ooo 183. URL https: /t///aalaannn /aaa//aaaaattlloooo/e wwwp./rgg//00/ emnlp-man panllpmmiin.83.1. 183/.
main.
References
Aas, K., Jullum, M., and Løland, A. Explaining individual predictions when features are dependent: More accurate approximations to shapley values. Artificial Intelligence, 298: 193:00 2021.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint 2rXtv:2404.04475. 2024.
AI Meta. Llama 3 model card. 2024. URL https:///ithuue wtt.tttth..ooooo.toooommmm..tttt caammmmta--ee aalammmaaamaa blob/main/ htth//main/MMDELL MODEL CARD.mc.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment prospect theoretic optimization. arXiv preprint arXiv: arXiv:2402.013366 2024.
Anonymous. Mitigating reward over-optimization in direct alignment algorithms with adaptive importance sampling, 2025. URL https: /tttp::://pppreeeiie http:/wwwwww.eeeeeeeeeeeeeeeeeee mnee/forutt 1d=#9dNX6TaRE.
Gao, L., Schulman, J., and Hilton, J. Sealing laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm understand learning from human preferences. In International Conference Artificial Intelligence and Statistics, pp. 4447 4455. PMLR, 2024.
Guo, G.,, Zhao, R., Tang. T., Zhao, X., and Wen, J.-R. Be- yond imitation: Leveraging fine-grained quality signals for alignment. In The Twelfth International Conference on Learning Representations.
Bradley, R. A. and Temry, M. E. Rank analysis of in- complete block designs: I. the method of paired com- parisons. Biometrika, 39(3/4):324-35, 1952. ISSN 00063444, 14643510. URL http://wwww jstor. htg/sttable/23344
Hale, J. A probabilistic earley parser as a psycholinguistic model. In Second meeting of the north american chapter of the association for computational linguistics, 2001.
Hong, J., Lee, N., and Thorne, J. Reference-fer monolithic preference optimization with odds ratio. arXiv e-prints, pp. arXiv-2403, 2024.
Chen, G., Liao, M., Li, C., and Fan, K. Step-level value preference optimization for mathematical reasoning. In
10

---

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
Jaeger, T. and Levy, R. Speakers optimize information density through syntactic reduction. In Schölkopf, B., Platt, J., and Hoffman, T. (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006. URL
cc oc/spper_fillee/pper//////// 62220882217866227664418882228888 pdf.
Meng, Y., Xia, M., and Chen, D. SimPO: Simple preference optimization with a reference-free reward. In The Thirty- eighth Annual Conference Neural Information Pro- cessing Systems, 2024. URL httpt:t////// pettp:://openrvviiii net/ / forum?dd=32zcote
cc /cc/pppeeffileeeepper////////// Paper. 0ee444222833644baa3332 pdf.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F,, Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Merkx, D. and Frank, S. L. Human sentence process- ing: Recurrence attention? In Chersoni, E., Hol- lenstein, N., Jacobs, C., Oseki, Y., Prévot, L., and San- tus, E. (eds.), Proceedings of the Workshop on Cogni- live Modeling and Computational Linguisticy, pp. 12-22. Online, June 2021. Association for Computational Lin- guistics. doi: 10. 10.186533 //aclanthologe wtt////g..../. cmcl-1.2/
Lai, X., Tian, Z. Chen, Y., Yang, S., Peng, X., and Jia, J. Step-dpo: Step-wise preference optimiza- tion for long-chain reasoning of Ilms. arXiv preprint arXiv:2406. 18629, 2024.
URL https:
Li T., Chiang*. W.-L., Frick, E., Dunlap, L., Zhu, B., Gonzalez, J. E., and Stoica, I. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024. URL https: // Amsys.... orr //ggg/////// 2024 S0--arrnaahaaeee
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A.. Welinder, P., Christiano, F., Leike, J.. and Lowe, R. Training language models follow instruc- tions with human feedback. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, vol- ume 35, pp. 17730-27744
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang. P., and Hashimoto, T. B. Alpacae-
val: An automatic evaluator of instruction-folloinng models. https: / ////gttttt /ightuu..b.. www/tatst-tta../ alpaca_eval, 2023.
Curran Associates, Inc., 2022.
Owen, A. B. and Pricur, C. On shapley value for measuring importance of dependent inputs. SIAM/ASA Journal on Uncertainty Quantification, (())996-11022 2017.
Lin, Z., Gou, Z., Gong, Y., Liu, X., yelong shen, Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., and Chen, W. Not all tokens are what you need for pretraining. In The Tiihty- Annual Conference on Neural In- formation Processing Systems, 2024. URL https: //oppereview.... net forum? ? 1d=0NM2
Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentan- gling length from quality in direct preference optimiza- tion. arXiv preprint arXiv:2403. 19159, 2024.
Liu, Z., Lu, M., Zhang, S., Liu, B., Guo, H., Yang, Y., Blanchet, J., and Wang, Z. Provably mitigating overop- timization in RLHF: Your SFT loss is implicitly adversarial regularizer. In The eiihth Annual Conference on Neural Information Processing Systems, 2024. URL id=2cQ31Phked
Rafailov, R., Chittepu, Y.. Park, R., Sikchi, H., Hejna, J., Knox, W. B., Finn, C., and Niekum, S. Scal- ing laws for reward model overoptimization in direct alignment algorithms. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https: /wwp//veeeeeeeeeeeeeeeeeee net forum? id-pf40uJyn40.
/ forum?
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, 2019, htth::::: //pp///peerr.eeee////ee/ee.e...e net/ fttt/forrmmidd::3. bet//Eorun?iic=Bxx88RCCC""
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon. S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024b.
Lundberg, S. M. and Lee, S.-I. A unified approach to inter- preting model predictions. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: https::/ppoceteeeee http///oceeddeec ngs. neurips.
Shapley, L. S. A value for n-person games. In Kuhn, H. W. and Tucker, A. W. (eds.), Contributions to the Theory of Games II. pp. 307-317. Princeton University Press, Princeton, 1953.

---

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
Slaats, S., Meyer, A. S., and Martin, A. E. Lexical surprisal shapes the time course of syntactic structure building. Neurobiology of Language, 5(4)9942-980, 10 2024. ISSN 2641-4368. doi: 10. 10.11622/ol...000555 URL https: ///oi.org//0/ wet/////oaaaee 2aa1162mmm1aa_01155_001
Yoon, S., Yoon, E., Yoon, H. S., Kim, J., and Yoo, C. information-theoretti text hallucination reduction for video-grounded dialogue. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pp. 4182-4193, Abu Dhabi, United Arab Emirates, December 2022. Association for Computa- tional Linguisties. doi: 10.18653/v1/2022.mmmll/ main. 280. URL https: /aa//aaaantttt http///aaaaatttyyy...aaaao org. emnlp waaan.2800/
Smith, N. J. and Levy, R. The effect of word pre- dietability on reading time logarithmic. Cog- nition, 128(3):302-319, 2013. ISSN 0010-0277. doi: https///bbooo/////oooooooooooooo
URL sttp:://www.scienccciieeeee com/ sstence/arrtc2e//////01107711100
Yoon, S., Koo, G., Lee, Y., and Yoo, C. Tpe: Test-time procrustes calibration for diffusion- based human image animation. Advances in Neural Information Processing Systems, 37:1 37:118654-118777 2024c.
von Werra, L., Belkada, Y. Tunstall, L.. Beeching, E., Thrush, T., Lambert, N., Huang, S., Rasul, K., and Gallouédec, Q. Trl: Transformer reinforcement learn- ing. httpps//gitte 2020.
Yuan, H., Yuan, Z., Tan, C., Wang, W., Huang, S., and Huang. RRHF: Rank responses to align lan- guage models with human feedback. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL /tR.httss//oppeeeiiiiiee net forum? id-EdI 4MCHK id=EdIGMCHK41.
Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Durme, B. V., Murray, K., and Kim, Y. J. Contrastive preference optimization: Pushing the boundaries of LLM perfor- mance in machine translation. In Forty-first International Conference on Machine Learning, 2024. URL https: //openreviews iew, net/ forum? i d1=1 iwkioZpn.
Zeng. Y., Liu, G., Ma, W., Yang, N., Zhang, H., and Wang. J. Token-level direct preference optimization. In Forty-first International Conference on Machine Learning.
Yang. K., Liu, Z. Xie, Q. Huang, J., Min, E., and Ananiadou, Selective preference optimization via token-level reward function estimation. arXiv preprint arXiv:2408. 13518, 2024.
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXIv:2305.10425, 2023.
Yoon, E., Yoon, H. S., Hasegawa-Johnson, M. A., and Yoo, C. D. Can video lims refuse to answer? alignment for answerability in video large language models. In The Thirteenth International Conference on Learning Repre- sentations.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt- and chatbot arena, 2023.
Yoon, E., Yoon, H. S., Fom, S., Han, G., Nam, D., Jo, D., On, K.-W., Hasegawa-Johnson. M., Kim, S., and Yoo, C. TLCR: Token-level continuous reward for fine-grained reinforcement learning from human feed- back. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 14969-14981, Bangkok, Thailand, August 2024a. Association for Computa- tional Linguistics. doi: 10.18663/1122244.mnmmm 889. URL https: /l//aalaante aaaclantoooogy.......... wrg/ findings acl. 889/.
Zhou, W., Zhang, S., Zhao L., and Meng. T. T-reg: Prefer- ence optimization with token-level reward regularization. arXiv preprint vrXiv:2412.026883 2024.
Yoon, H. S., Tee, J. T. J., Yoon, E., Yoon, S., Kim, G., Li, Y., and Yoo, C. D. Esd: Expected squared difference tuning-free trainable calibration measure. arXiv preprint 2.X0:22303.0247,,, 2023.
Yoon, H. S., Yoon, E., Tee, J. T. J., Hosegawa-Johnson, M., Li, Y., and Yoo, C. D. C-tpt: Calibrated test-time prompt tuning for vision-language models via text feature dispersion. arXiv preprint arXiv:2403. /4/19/ 2024b.
12

---

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
A. Limitations
Poderrad
While ConfPO achieves strong empirical results and is grounded in an intuitive rationale, a deeper theoretical anal- ysis is needed fully elucidate the mechanisms behind its effectiveness. In particular, further investigation is required to formally characterize its impact on human alignment and overoptimizttion mitigation. Additionally, while we have shown that ConPPO efficiently selects preference-critical tokens for SimPO (Meng et al., 2024) and DPO (Rafailov et al., 2024b), its generalization across different DAAS and model scales remains an open question, warranting further exploration.
Wrader Nave
Figure 9. Observation for DPO.
C Kome indero Naconio
few Mrunteed
Colblee 2015
Comnlatton between probubeiiy By goad rorm accep spact Frattoon
Opon
Opon Prenion
Table AlpacaEval 2 (Li al., 2023) results for different choices of threshold r.
Lee Conterne fakee
Londor
Lowdo Toom
Mistral-Base (7B)
Llama-3-Base (8B)
Method
2 2
AlpacaEival
LC (%)
WR (%)
(%)
WR (%)
Walee fale
Bd Gradd sorr ssparation across spoch fraction for prefemeed rapore (X"1
bualeed
SFT
4.84
2.92
6.86
3.79
Epech Frutien
Epock Factien 8.
Tpuch Frudton
SimPO (Meng et al., 2024)
27.1
23.4
27.0
25.7
ConfPO-Iixed/0.6)
28.2
26.2
27.6
27.5
ConfPPO-goometric
27.2
26.3
28.1
28.8
ConfPO-aritmmetic
28.9
26.7
28.3
32.7
(1 Gand rorn sepanation acioss epoch faction for upp ahereedd responee br)
Figure 10. Observation 2 for DPO.
B. Threshold Selection
In this section, we examine different thresholding strategies for ConfPO (Eq. 13) by evaluating various choices for т. Specifically, consider: (1) fixed threshold (T 0.6), (2) the geometric average, and (3) the arithmetic average (which we use in Eq. 14). The geometric average is determined as follows:
All Totare 1Ox
High Co--Corrierr Tokere oily Low Coefitere Tokere Ory
6.20
0.00-
200
200
(0) Reward accuracy across training steps
algement across training stepe
10 nolv! x. v
To
Figure 11, Observation 3 for DPO.
r'
II
To
2. vize)
(16)
D. Implementation Details
Following (Meng et al., 2024), we use batch size of 128 and train for single epoch for all preference optimization training. The maximum sequence length is set to 2048, with a maximum prompt length of 1800. We employ a cosine learning rate scheduler with 10% warmup, using Adam W optimizer (Loshchilov & Hutter, 2019). Our implementation is based on TRL (von Werra et al., 2020) and DeepSpeed
As shown in Table 4, all thresholding strategies outperform standard SimPO, with the arithmetic average yielding the best performance. Given this result, we adopt the arithmetic average for 7 in all experiments.
C. Observations on DPO
ZeRO-2, with gradient checkpointing enabled on 4 NVIDIA A 100 80GB PCIe GPUs
In Figure 9-11, we present observational plots for DPO (Rafailov et al., 2024b) under the same experimental setup as Section 4. The results exhibit trends consistent with those observed in SimPO, further validating our findings. The performance of ConfPO applied to DPO is reported in Table 3 in the main text.
For each model, we determine the optimal hyperparameter settings, which are summarized in Table 6. We search the 3 in the range of [1.0, 1.5, 2.0] and y in the range of [0.5, 0.8. 1.2, 1.6, 2.0, 2.5].
13

---

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
Table Various preference optimization objectives given preference data D = (zr, Bue Dw), where ar is an input, and you and yi are the winning and losing responses.
Method
Objective
RRHF (Yuan et al., 2023)
max (.
log To (yrr [r) +
ludl log (1 xz)
A log To (ow
lyw
SLiC-HF (Zhao et al., 2023)
max (0, 8 log No (Bw
2x) + log iTe (21
x)) X log we (Vw 2))
DPO (Rafailov et al., 2024b)
logg
log Me( (e) Thel (r)
Malin lax) 3 log (nd(3r(
IPO (Azar et al., 2024)
(log 5e( (x)
TarlBuulz) (x) log me( (3)()))
1
27
CPO (Xu et al., 2024)
log a Blg (M (x) - /3 log To (N w)) > log ne (Vin xr)
KTO (Ethayarajh et al., 2024)
Av (9 log mo(ywln) Zeett + AsTr (zeer log mo(sull))
Tur(
where zier E(r.)))- [$KL (TTO(V) (x) [I]]|T [x)]]
ORPO (Hong et al., 2024)
log (v x) - A log (log po(lul)))
log polyrle) (=po(a)
1-py (Vwllx
where poly | (2) = exp
log TT ux)
R-DPO (Park ct al., 2024)
log
[olg (e(wall) 19 log
ma(un (x) [ntkarl))
+ (ax ywl albl1))
SimPO (Meng et al., 2024)
log a
8 lvul
log Toe (yw xx)
Twal log no (M xx) -7)
ConfPO (ours)
log
"|
s(v.") log To (vl 12, v
lovl Il'' s(v!) log No (o -7)
-7)
where s(vl") = 1 (To (2i ax, v <T"), 1051 = (=(""")>")) = 1H
* .y²))
1 (To (v) ic, weee ST'),
lN'l = [[v/)=(u)) = 13/.
|r me (v." [r, v..) 1 Il'' (ofl v'Z
To ensure fair comparison with existing baselines, fol- low the decoding hyperparameters from Meng et al. (2024) for evaluation. Specifically, for AlpacaEval 2, we use sampling-bass decoding with temperatures of 0.7, 0.5, and 0.9 for Mistral-Base, Mistral-Instruuc, and both LLLaMA33 models, respectively. For Arena-Hard, we apply greedy decoding across all settings.
F. Correlations between tokens' confidence and their position in the sequences.
Spoch
Narsar uf 50000
Spoch 0a
Epock fradion
Spuch Faction
Epock frucioon
20000
50000
10000
10000
Table 6. The hyperparameter values in ConIPO.
Setting
B
Y
Learning rate
(4)
(3) rejected
Puellim In S0a
Mistral-Base
1.5
1.6
3--7
Figure 12. Histogram showing the distribution of selected tokens with respect to the position in the sentence.
Mistral-
2.0
0.5
5e-7
Llama-3-Bass
1.0
2.0
6c-7
Mlama-3-Instruue
1.0
2.5
1e-6
To check for potential positional biases, we analyzed where low-confidence tokens are selected within sequence. As shown in Figure 12, the distribution of these tokens is ap- proximately uniform across all normalized positions in both chosen and rejected responses. This pattern holds through- out the training process.
E. Standard Deviation of AlpacaEval 2 and Arena-Hardv0.1
Table 7 reports the standard deviation for AlpacaEval 2 and the 95% confidence interval for Arena- Hardd
14

---

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
Table Detailed statistics AlpacaEval and Arena-Hard. I.C refers the length-controlled win rate, denotes the raw win rate, and STD represents the standard deviation of the win rate. Length indicates the average generation length. For Arena-Hard, we report both the win rate and the 95% confidence interval.
AlpacaEval
Arena-Hard
Models
I.C. (%)
WR (%)
STD (%)
Length
WR
95 CT high
95 CI low
Length
Mistral-Base
SFT
4.84
2.92
0.6
953
1.3
1.8
0.9
521
RRHF
12.7
10.2
1.1
1789
5.8
8.0
6.0
596
SLIC-HF
14.5
11.3
1.1
1664
7.3
8.5
6.2
683
DPO
18.1
16.0
1732
IPO
13.2
10.6
1.1
1529
7.5
8.5
674
CPO
12.8
11.4
1.3
1451
6,9
6.7
4,9
823
KTO
12.8
8.26
1.3
1208
5.6
6.6
4.7
475
ORPO
18.7
13.8
1.5
1582
7.0
7.9
5.9
764
R-DPO
21.2
15.6
1.3
11.1
SimPO
27.1
23.4
1.6
1996
13.8
18.0
15.1
699
ConfPO
28.9
26.7
1.6
2021
16.9
18.0
15.1
699
Mistral-Instruct
24.4
18.8
1.3
1557
12.6
14.1
11.1
486
RRHF
36.0
33.0
1.5
1855
18.1
19.5
16.4
517
SLiC-HF
33.6
32.1
1.5
1946
18.9
20.6
17.3
578
DPO
36.1
31.9
1.6
1719
16.3
18.0
15.2
518
28.4
26.8
1.6
1949
740
CPO
29.8
38.0
1.6
2860
22.6
25.0
20.8
812
KTO
32.9
29.8
1.6
2047
17.9
20.3
16.1
496
ORPO
32.0
30.7
1.6
2316
20.8
22.5
19.1
527
R-DPO
32.2
27.6
18.0
SimPO
37.0
37.3
1.7
2023
21.0
22.7
18.8
539
ConPPO
39.1
38.4
1.7
1981
22.7
18.0
15.1
699
Llama-3-Bas
SFT
6.86
3.79
3.3
RRHF
13.8
9.69
1.2
1389
6.3
7.5
5.7
536
SLIC-HF
21.4
24.3
1.2
1432
6.0
11.5
8.9
676
DPO
25.4
23,9
1.3
1732
15.9
18.1
14.1
563
IPO
22.5
25.5
2617
17.8
19.5
16.0
608
CPO
21.9
23.0
1,5
2775
5.8
13.2
10.4
800
KTO
24.7
22.1
1.4
1782
12.5
14.2
10.9
519
ORPO
17.6
14.2
1.5
1599
10.8
12.3
9.6
639
R-DPO
21.9
25.0
1.5
3000
17.2
18.5
15.7
527
SimPO
27.0
25.7
21.1
25.4
ConfPO
28.3
32,7
1.6
2885
23.8
18.0
15.1
699
SFT
33.1
32.2
1.6
1949
22.3
23.9
20.3
596
RRHF
35.4
32,9
1.7
1893
26.5
28.4
24.6
502
SLiC-HF
35.7
35,6
1.6
1728
26.2
28.4
24.4
584
DPO
43.8
42.3
1.7
1923
32.6
34.8
30.3
528
IPO
42.9
42.4
1.7
2000
30.5
32.8
28.4
554
CPO
35.7
38.7
1.7
2148
KTO
41.0
39.7
1.7
1923
26.4
28.7
24.3
536
ORPO
36.7
34.7
1.6
1845
25.8
27.4
23.8
535
R-DPO
46.2
43.9
1.7
1904
33.1
35.3
30.9
522
SimPO
48.8
44.4
1.7
1795
32.6
35.9
32.0
504
ConfPO
49.1
45.0
1.7
1817
32.8
15.1
699
15


Reviews:
[{"review_id": "MtgiDxQRCs", "reviewer": "Reviewer_M4QG", "summary": "This paper introduces ConfPO, a novel method for enhancing preference learning in large language models (LLMs). The core idea behind ConfPO is to selectively update tokens during training based on their confidence levels, specifically focusing on low-confidence tokens which the authors empirically demonstrate are crucial for effective alignment. The method is motivated by three key observations: that token-level gradients follow a long-tailed distribution, there is a high correlation between the gradient norm and a token's confidence, and that low-confidence tokens dominate the learning process during preference optimization. The authors propose a simple yet effective approach where only tokens with a probability lower than the average probability across the sequence are selected for backpropagation. Specifically, the authors extend their method to DPO, demonstrating its applicability across different DAA frameworks. \n\nEmpirically, ConfPO demonstrates superior performance compared to existing DAAs, including SimPO and DPO, on benchmarks such as AlpacaEval 2 and Arena-Hard. The results indicate that ConfPO achieves higher win rates and reduces the risk of reward hacking. The authors conduct experiments using Mistral 7B and Llama-3-8B models, in both base and instruct configurations, and evaluate their method on the AlpacaEval 2 and Arena-Hard benchmarks. They show that ConfPO improves alignment performance and mitigates overoptimization, a common issue in preference learning. The paper also includes an ablation study in the appendix, which explores different thresholding strategies for selecting low-confidence tokens, finding that the average probability threshold performs best.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for offering detailed feedback. We address them below, following the order in which they appear across the review’s sections.\n\n---\n### **Claims and Evidence (CE)**\n**[CE-Q1] Computational cost comparison.**\n\n**[CE-A1]** **Table A1** compares runtime and GPU memory usage of ConfPO with baseline DAAs (DPO, SimPO). As ConfPO requires no extra forward/backward passes or auxiliary models, its computational cost is nearly identical to the baselines.\n\n**Table A1.** Runtime and peak GPU memory usage for each method under identical settings.\n||Time/Iter(s)|Mem(GB)|\n|-|-|-|\n|DPO|121.8|53GB|\n|ConfPO$_{DPO}$|125.8|51GB|\n|SimPO|150.2|50GB|\n|ConfPO|140.0|49GB|\n\n**[CE-Q2] Robustness to hyperparameter choices.**\n\n**[CE-A2]** We want to clarify that we do not claim our method is “robust” to hyperparameters. Rather, in Figure 5 (where we vary the KL-divergence of resulting models by adjusting the hyperparameters), it indicates our method remains more stable when increasing the KL budget which is a key concern in reward-hacking (overoptimization) scenarios. We will revise the manuscript to avoid any confusion about hyperparameter robustness and emphasize that these experiments were specifically designed to demonstrate reduced overoptimization.\n\nHowever, with regards to this, we have attached ***Figure A*** below, comparing ConfPO and SimPO across various hyperparameter settings.\n\n***Figure A***: https://anonymous.4open.science/r/ConfPO-B7C6/L/FigureA.md\n\n---\n### **Methods and Evaluation Criteria (ME)**\n**[ME-Q1] Evaluation on downstream tasks**\n\n**[ME-A1]** We have evaluated our trained models on downstream tasks involving math, QA, reasoning, and coding. We find that ConfPO achieves the highest average performance across all baselines. Due to space constraints, we summarize key results below (***Table A2***) and provide the full table (***Table B***) in the following link: \n\n***Table B***: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableB.md\n\n**Table A2.** Downstream task results on Llama-3-Base (8B).\n||GSM 8K (Math)|TruthfulQA|ARC (Reasoning)|Humaneval (Coding)|\n|-|-|-|-|-|\n|**Llama-3-Base (8B)**|||||\n|SFT|51.1|45.3|55.6|6.71|\n|DPO|51.2|53.5|56.1|12.5|\n|SimPO|50.8|56.2|56.7|12.8|\n|ConfPO|**52.9**|**59.2**|**58.4**|**17.0**|\n\n---\n---\n### **Experimental Designs or Analyses (EA)**\n**[EA-Q1] Comparison with existing token-level DAAs**\n\n**[EA-A1]** We include comparisons with TDPO and SparsePO in ***Table C*** below. While TDPO outperforms SimPO by leveraging token-level signals, our ConfPO still achieves higher alignment performance. Notably, TDPO requires a reference model to obtain token-level signals, incurring higher memory and compute overhead compared to ConfPO, which can identify critical tokens without a reference model. Meanwhile, SparsePO demonstrates performance comparable to SimPO.\n\n***Table C***: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableC.md\n\n---\n### **Relation to Broader Scientific Literature (RS)**\n\n**[RS-Q1] Theoretical Motivation**\n\n**[RS-A1]** In our response to reviewer RzSL under question *[SW-Q1]*, we provide a detailed theoretical motivation,including mathematical derivations and sensitivity analysis demonstrating why low-confidence tokens ($p_i(\\theta)$) yield higher gradient norms ($||\\nabla_\\theta \\log p_i(\\theta)||$). Due to space constraints, we kindly refer the reviewer to that section for complete details.\n\nFurthermore, we emphasize that various DAAs utilize the joint log probability as an implicit reward, causing their policy to fundamentally depend on $\\nabla_\\theta \\log p_i(\\theta)$. Consequently, our theoretical insights naturally generalize across various DAAs.\n\n**[RS-Q2] Extension to more DAAs**\n\n**[RS-A2]** In our main text, we have shown ConfPO to work on both the DPO and SimPO. We have now also applied ConfPO to IPO in ***Table D*** below, supporting the general applicability of ConfPO on various DAAs.\n\n***Table D***: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableD.md\n\n---\n### **Questions for Authors (QA)**\n**[QA-Q1] Analysis of different thresholding strategies.**\n\n**[QA-A1]** In Appendix B (Table 4), we analyze various thresholding strategies (fixed, geometric, arithmetic averages) and find that the arithmetic average, as used in our main experiments, provides the best performance.\n\n**[QA-Q2] How sensitive is ConfPO to variations in other hyperparameters such as learning rate?**\n\n**[QA-A2]** We have included ***Figure B*** below comparing ConfPO with SimPO across different learning rates (lr). Due to SimPO's design (no explicit KL regularization), it tends to become unstable at higher learning rates. In contrast, ConfPO, by selectively updating only critical tokens, demonstrates increased stability, even at higher lr.\n\n***Figure B***: https://anonymous.4open.science/r/ConfPO-B7C6/L/FigureB.md\n\n**[QA-Q3] Computational cost comparison.**\n\n**[QA-A3]** We have provided a detailed comparison under **[CE-Q1]** above.", "claims_and_evidence": "The paper claims that ConfPO has the same computational cost as standard preference learning, but it lacks quantitative data to support this claim. While the authors mention that their method is robust to hyperparameter choices, this claim should be supported by more extensive experimentation.", "methods_and_evaluation_criteria": "The authors should also consider using more fine-grained evaluation metrics that can capture the nuances of model behavior on these tasks. For example, they could use metrics that measure the quality of reasoning or the correctness of generated code. This would provide a more detailed assessment of ConfPO's performance and help identify areas for further improvement.", "theoretical_claims": "Yes. I checked the objective function listed in Table 5.", "experimental_designs_or_analyses": "The authors propose ConfPO which only selects tokens with a probability lower than the average probability. Experiments show that ConfPO improves alignment performance and mitigates overoptimization. Howeve, this paper lacks the essential comparison with existing work using similar ideas, such as TDPO [1] and SparsePO [2]. \n\n[1] Token-level Direct Preference Optimization\n[2] SPARSEPO: CONTROLLING PREFERENCE VIA SPARSE TOKEN MASKS", "relation_to_broader_scientific_literature": "The paper primarily focuses on the SimPO algorithm, and while it does extend the method to DPO, a more comprehensive comparison with other DAAs would strengthen the paper's claims. A deeper investigation beyond token selection into the theoretical underpinnings of why ConfPO works across various DAAs, beyond empirical observation, is needed.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No.", "other_comments_or_suggestions": "No.", "questions_for_authors": "The paper uses the average probability as a threshold, but it is unclear whether this is the most effective approach in all cases. A more detailed analysis of different thresholding strategies and their impact on performance is needed. Third, how sensitive is ConfPO to variations in other hyperparameters, such as the learning rate and batch size, and how do these hyperparameters interact with each other? \n\nWhat is the computational cost of ConfPO compared to other alignment techniques, such as DPO and SimPO, in terms of training time, memory usage, and energy consumption?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E8Dja1QOM6", "reviewer": "Reviewer_RzSL", "summary": "This paper proposes to only include low-confidence tokens during preference alignment.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for offering detailed feedback. We address them below, following the order in which they appear across the review’s sections.\n\n---\n### **Experimental Designs or Analyses (EA)**\n**[CE-Q1] Experiment on newer models.**\n\n**[CE-A1]** We focused on the four models used by SimPO, as these models had thoroughly tuned hyperparameters for nine existing DAAs, ensuring a fair comparison.\n\nHowever, following the reviewer’s suggestion, we provide additional experimental results using Qwen2.5 7B, shown in ***Table E*** below, where we see ConfPO has higher score than SimPO.\n\n***Table E***: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableE.md\n\n**[CE-Q2] Evaluation on diverse tasks**\n\n**[CE-A2]** We have evaluated our trained models on various downstream tasks, including math and coding, and find that ConfPO achieves the highest average performance than baselines. Due to space constraints, please refer to our response to reviewer M4QG under question [ME-Q1] or at the following link: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableB.md\n\n---\n### **Strengths and Weaknesses (SW)**\n**[SW-Q1] Theoretical Motivation.**\n\n**[SW-A1]** For theoretical clarity, we provide a first-principles analysis showing why low-confidence tokens naturally yield higher gradient norms.\n\nLet $\\pi_\\theta$ be our policy model, parameterized by $\\theta$. Given a prompt $x$ and a response $y =\\{y_1,\\dots,y_n\\}$, let $$p_i(\\theta)=\\pi_\\theta(y_i|x,\\,y_{<i})$$be the probability of the $i$-th token. Our goal is to show that the gradient norm of the log-probability of the $i$-th token, $||\\nabla_\\theta\\log p_i(\\theta)||$, is **negatively correlated** with the confidence of the policy model for that token, $p_i(\\theta)$.\n\n**Chain Rule and Gradient Norm**\n\nUsing the chain rule, we have:\n$$\\nabla_\\theta\\log p_i(\\theta)=\\frac{\\nabla_\\theta p_i(\\theta)}{p_i(\\theta)}\\implies||\\nabla_\\theta\\log p_i(\\theta)||=\\frac{||\\nabla_\\theta p_i(\\theta)||}{p_i(\\theta)}.$$\nThus, if token confidence $p_i(\\theta)$ dominates this ratio, the gradient norm of the log probability $\\nabla_\\theta\\log p_i(\\theta)$ decreases as confidence increases, establishing the observed negative correlation.\n\n**Sensitivity Analysis**\n\nTo formalize the dominance of $p_i(\\theta)$ on $\\nabla_\\theta \\log p_i(\\theta)$, consider the ratio $$r(\\theta)=\\frac{||\\nabla_\\theta p_i(\\theta)||}{p_i(\\theta)}=\\frac{b(\\theta)}{c(\\theta)},$$where $b(\\theta)=||\\nabla_\\theta p_i(\\theta)||$ and $c(\\theta)=p_i(\\theta)$. A local sensitivity test examines partial derivatives:$$\\frac{\\partial r}{\\partial b}=\\frac{1}{c},\\quad\\frac{\\partial r}{\\partial c}=-\\frac{b}{c^2}.$$To determine which component more strongly influences the ratio, one can sample many tokens $ i $, each yielding pairs $(b_i, c_i)$. Then, examine the *main effects*:\n$$S_b(i)=\\frac{1}{c_i},\\quad S_c(i) =-\\frac{b_i}{c_i^2}.$$\nComparing $\\mathbb{E}[|S_b|]$ and $ \\mathbb{E}[|S_c|] $ offers a practical measure of which component dominates in typical scenarios. If $\\mathbb{E}[|S_c|-|S_b|]>0$, it shows that variations in $ p_i(\\theta) $ (the token’s confidence) have the stronger effect on $||\\nabla_\\theta \\log p_i(\\theta)||$. Using Monte Carlo estimate with 1000 samples, we indeed find that $\\mathbb{E}[|S_c|-|S_b|]>0$ (The exact value can be found in https://anonymous.4open.science/r/ConfPO-B7C6/L/TableI.md). Thus, this describes our observed **negative correlation** between the confidence and log-probability gradient norm.\n\n**[SW-Q2] Visualization of selected tokens.**\n\n**[SW-A2]** In ***Figure C*** below, we visualize examples of selected tokens. In general, tokens acting as crucial inflection points within sentences (e.g., tokens initiating new phrases) are selected, while tokens that are merely continuations of preceding tokens are not selected (e.g., in the word \"Slack,\" \"Sl\" is selected, whereas \"ack\" is not). This shows that our confidence-based selection identifies important tokens that shape the overall sentence, leading to more efficient KL-budget usage and reduced overoptimization (Figure 5 in the main text).\n\n***Figure C***: https://anonymous.4open.science/r/ConfPO-B7C6/L/FigureC.md\n\n**[SW-Q3] Extension to other DAAs.**\n\n**[SW-A3]** In our main text, we showed how ConfPO applies to SimPO (Table 1) and DPO (Table 3). Additionally, we successfully extended ConfPO to IPO with enhanced performance (see response to reviewer M4QG, *[RS-Q2]*). Due to the space limit, we kindly direct the reviewer to that section or to the following link: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableD.md\n\n**[SW-Q4] Theoretical Motivation and Performance.**\n\n**[SW-A4]** Please refer to [SW-Q1] for the theoretical motivation. In terms of the performance, our method consistently improves on existing DAAs (SimPO, DPO, IPO) across multiple benchmarks. For instance, on Llama-3-Base(8B) for AlpacaEval2, ConfPO achieves a +7% win rate over SimPO, showing that our modified loss leads to meaningful alignment gains.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There is no theory in this work/", "experimental_designs_or_analyses": "Yes. \n\n1. Llama-3 and Mistral seem to be outdated. It would be better using more advanced models like Llama-3.1, Deepseek and Qwen.\n\n2. It will be better to have diverse tasks like math and coding.", "relation_to_broader_scientific_literature": "This is a slight modification of SimPO without much novelty and contribution.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Weaknesses:\n\n1. The relationship between gradient norm and confidence is weak, only supported  by limited empirical experiment observation.\n\n2. It would be beneficial to select specific examples and provide visualizations of the chosen tokens. This could reveal whether the selected tokens capture core information or merely auxiliary words.\n\n3. This actually corresponds to all alignment algorithms. So it would be better to see the combination of this selection strategy with different algorithms.\n\n4. The novelty is limited as it just slightly modifies the loss without much theoretical justification. Also, the performance is similar to existing works.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DwUktJSBiU", "reviewer": "Reviewer_kWCJ", "summary": "This paper proposes a token selection strategy for direct alignment algorithms. They observe a high correlation between the gradient norm and the confidence of a token and use confidence as a metric to automatically select the token, which can possibly prioritize the tokens with high gradient norm. They show empirically that incorporating selection strategy can enhance simpo and DPO algorithms.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for offering detailed feedback. We address them below, following the order in which they appear across the review’s sections.\n\n---\n### **Methods and Evaluation Criteria (ME)**\n**[ME-Q1] Static token selection baseline.**\n\n**[ME-A1]** We address this question under *[QA-Q2]* below, where we provide further details on the static token selection baseline.\n\n---\n### **Theoretical Claims (TC)**\n**[TC-Q1] Theoretical Motivation**\n\n**[TC-A1]** In our response to reviewer RzSL under question *[SW-Q1]*, we provide a detailed theoretical motivation, including mathematical derivations and sensitivity analysis demonstrating why low-confidence tokens ($p_i(\\theta)$) yield higher gradient norms ($||\\nabla_\\theta \\log p_i(\\theta)||$). Due to space constraints, we kindly refer the reviewer to that section for complete details.\n\n---\n### **Experimental Designs or Anlayses (EA)**\n**[EA-Q1] Evaluation on downstream tasks**\n\n**[EA-A1]** We have evaluated our trained models on downstream tasks involving math, truthfulness, reasoning, and coding. We find that ConfPO achieves the highest average performance across all baselines. Due to space constraints, please refer to our response to reviewer M4QG under question *[ME-Q1]* or at the following link: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableB.md\n\n---\n### **Supplementary Material (SM)**\n**[SM-Q1] Typo in Table 7.**\n\n**[SM-A1]** Thank you for pointing out the typo. We have corrected Table 7, which is available at the following link: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableF.md\n\n---\n### **Questions for Authors (QA)**\n**[QA-Q1] Concern about non-continuous update on coherency and performance of downstream task.** \n\n**[QA-A1]** Because each token is generated under a conditional distribution that considers all previously generated tokens, selectively updating some tokens while skipping others does not fragment the model’s coherence. Despite directing the preference objective toward challenging (low-confidence) tokens, the model still maintains a valid joint distribution $p(y)$ at inference time, ensuring coherent token-by-token generation. Empirically, we do not observe drops in fluency or logical consistency on bechmarks like AlpacaEval 2, Arena-Hard, and downstream tasks shown in question *[EA-Q1]* above. A small-scale human assessment of responses also further confirms that selectively omitting updates on high-confidence tokens does not undermine text coherency.\n\n**[QA-Q2] Static token selection baseline.** \n\n**[QA-A2]** We conducted an additional baseline experiment, selecting tokens based on confidence values from a fixed reference policy. Our results (see ***Table G***) indicate that approximately 80% of tokens selected dynamically by the evolving policy model overlapped with tokens chosen by the fixed reference policy; however, the remaining 20% differed, reflecting critical tokens that changed dynamically as training progressed. While token selection via the fixed reference policy slightly improved performance over SimPO, it was noticeably inferior compared to our dynamic ConfPO method. This clearly highlights that token importance evolves as the policy model updates, underscoring the necessity of our dynamic selection strategy for achieving optimal performance.\n\n***Table G***: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableG.md\n\n**[QA-Q3] Does noisy data result in degenerated performance.** \n\n**[QA-A3]** Thank you for raising this important concern. We tested ConfPO with two types of noise:\n\n1. **Flipped Preferences:** We randomly flipped 20% of the preference labels. Even in this noisy setting, ConfPO outperformed SimPO, confirming our low-confidence token selection does not amplify label noise.\n\n2. **Word-Level Noise (EDA):** We also introduced word-level noise via data augmentation methods—synonym replacement, random insertion, random swap, and random deletion. Although this aggressive augmentation substantially degraded overall performance, ConfPO still outperformed SimPO.\n\nThe detailed results are available at the following link: \n\n***Table H***: https://anonymous.4open.science/r/ConfPO-B7C6/L/TableH.md\n\nWe note that word-level noise is uncommon in recent preference-learning scenarios, which typically use on-policy data collection from advanced LLM-generated text that produces coherent sentences. Thus, randomly flipped preferences represent a more realistic noise scenario.\n\n**[QA-Q4] Applicability to on-policy RLHF, such as PPO.** \n\n**[QA-A4]** Our token-selection mechanism indeed applies effectively to both off-policy and partially on-policy settings, as shown. Even with highly on-policy data, our dynamic threshold maintains stable token selection ratios. Extending ConfPO explicitly to PPO-based RLHF, however, involves reward-based signals rather than implicit log-probabilities used by DAAs, and thus lies beyond our current scope but remains promising future work.", "claims_and_evidence": "Yes. Their claim is supported by clear evidence.", "methods_and_evaluation_criteria": "One weakness is the lack of sufficient baselines to demonstrate the necessity of incorporating dynamic selection. It would be beneficial to include baselines, such as static selection before training.", "theoretical_claims": "This work focuses on empirical analysis.", "experimental_designs_or_analyses": "One major weakness is that the evaluation primarily focuses on AlpacaEval Arena-Hard, relying on large language models as judges for human preferences. To better substantiate the effectiveness of the approach, it would be valuable to evaluate its performance on downstream tasks such as reasoning and factual question answering.", "relation_to_broader_scientific_literature": "This paper contributes to the ongoing research on direct alignment algorithms from a feature importance perspective.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "This paper contributes to the ongoing research on direct alignment algorithms from a feature importance perspective.\n\n1. Since the proposed token selection can be non-continuous, how might this discontinuity impact the coherence or the performance of downstream tasks?  \n\n2. Could the authors provide a simple baseline that uses a fixed reference policy to select important tokens? This would help demonstrate the necessity of dynamic selection.  \n\n3. A key implicit assumption is that low-confidence tokens contain more useful information and lead to more effective updates. However, if the data contains noise, could this selection mechanism amplify it, potentially resulting in degenerated performance?\n\n4. This selection mechanism appears to be applicable in both the base and instruct settings, where the former involves off-policy data and the latter on-policy data to some extent. I am curious whether this selection method is also applicable to general RLHF algorithms, such as PPO, where all samples are highly on-policy. In this scenario, tokens are more likely to have high confidence due to the on-policy nature of the data generation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tVoAKsfIkS", "reviewer": "Reviewer_foLE", "summary": "While most existing Direct Alignment Algorithms (DAAs) uniformly adjust token probabilities, this paper questions the assumption that each token contributes equally to preference, and proposes a new method, called ConfPO, which identifies preference-critical tokens based on the training policy's confidence, and zeros out all non-critical tokens in the loss. \n\nPreference-critical tokens are defined as tokens with (conditional) model probabilities below a certain (dynamically defined) threshold. This definition of preference-critical tokens is motivated empirically, as the paper shows a strong negative correlation between model probabilities (i.e., confidence) and gradient norm. Unlike other token-level approaches, a key benefit of this method is that the computation cost remains identical to that of original DAAs. The paper then shows that ConfPO outperforms its DAA analog without critical preference token selection, using the Mistral-7B and Llama-3-8B families of models, and evaluating on AlpacaEval 2 and Arena-Hard v0.1.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for offering detailed feedback. We address them below, following the order in which they appear across the review’s sections.\n\n---\n### **Claims and Evidence (CE)**\n**[CE-Q1] Theoretical motivation.**\n\n**[CE-A1]** In our response to reviewer RzSL under question ID [RS-Q1], we provide a detailed theoretical motivation,including mathematical derivations and sensitivity analysis demonstrating why low-confidence tokens ($p_i(\\theta)$) yield higher gradient norms ($||\\nabla_\\theta \\log p_i(\\theta)||$). Due to space constraints, we kindly refer the reviewer to that section for complete details.\n\n\n**[CE-Q2] KL budget efficiency at lower end.**\n\n**[CE-A2]** We added additional points at lower KL divergence levels in ***Figure E*** to enable a fairer comparison. These additional results clearly show that ConfPO consistently achieves higher alignment scores than SimPO across both low and high KL values.\n\n***Figure E***: https://anonymous.4open.science/r/ConfPO-B7C6/L/FigureE.md\n\n---\n\n### **Experimental Designs or Analyses (EA)**\n**[EA-Q1 & Q2] Correlations between tokens' confidence and their position in the sequences.**\n\n**[EA-A1 & A2]** In ***Figure C*** below, we visualize examples of selected tokens. In general, tokens acting as crucial inflection points within sentences (e.g., tokens initiating lists or new phrases) are selected, while tokens that are merely continuations of preceding tokens are not selected (e.g., in the word \"Slack,\" \"Sl\" is selected, whereas \"ack\" is not). This shows that our confidence-based selection identifies important tokens that shape the overall sentence, leading to more efficient KL-budget usage and reduced overoptimization (Figure 5 in the main text). \n\nSince inflection points of sentences are not really located towards certain position in a sentence, we see they are approximately uniformly distributed in the sentence as in ***Figure D***.\n\n***Figure C***: https://anonymous.4open.science/r/ConfPO-B7C6/L/FigureC.md\n\n***Figure D***: https://anonymous.4open.science/r/ConfPO-B7C6/L/FigureD.md\n\n**[EA-Q3 & 4] Exploration of soft weighting ConfPO and curriculum learning.**\n\n**[EA-A3 & 4]** Thank you for suggesting this valuable experiment. We explored a soft-weighting version of ConfPO,$$\\frac{\\sum_{i=1}^{|y|} w(y_i)\\log\\pi_\\theta(y_i|x,y_{<i})}{\\sum_{i=1}^{|y|} w(y_i)},$$ where $w(y_i) = (1 - p_i(\\theta))^T$. In ***Figure F*** below, we tested soft weighting with temperatures $T=1$ and $5$. When $T=0$, the equation reduces to standard SimPO; as $T$ increases, it approaches our original (hard-weighted) ConfPO. Our results demonstrate that alignment performance consistently improves as $T$ increases, with the original ConfPO achieving the highest performance. \n\nThis validates our key hypothesis: high-confidence tokens can impede preference learning by overshadowing more informative signals. This is consistent with Figure 4-(b) of our main paper, where training exclusively on low-confidence tokens improved alignment scores relative to training on all tokens, while training solely on high-confidence tokens resulted in performance worse than even the SFT baseline.\n\nWe also experimented with curriculum setups adjusting T from 0 to 10 and 10 to 0 during training. Results showed no clear advantage over standard ConfPO, which can also be found in ***Figure F*** below.\n\n***Figure F***: https://anonymous.4open.science/r/ConfPO-B7C6/L/FigureF.md\n\n---\n### **Other Comments or Suggestions**\nWe address each point below:\n- **Definition of \\( D \\) (Section 3.1):**  \nThank you for noting this inconsistency. We will clearly distinguish the dataset notation used for prompts alone versus prompt-response pairs in the revised manuscript.\n- **Bradley-Terry Model:**  \nWe included the Bradley–Terry (BT) model in Section 3.1 because it serves as the foundational probabilistic model for preference-based reward modeling, commonly used in RLHF. While traditional RLHF explicitly learns a reward model using BT, DPO (introduced in Section 3.2) leverages this same BT framework implicitly by reparameterizing the reward with the joint log probabilities.\n- **Figure 4 (model and dataset):**  \nFor Figure2~4, we use the Llama-3-Base (8B) with the UltraFeedback dataset. We will clarify this point in the manuscript.\n- **Equation Duplication:**  \nWe will simplify the presentation by directly defining as given in Equation 7 with added token selection criteria of ConfPO.\n- **Hyperparameter for Figure 5:**  \nIn Figure 5, we varied the $\\beta$ hyperparameter (with $\\gamma$ fixed) to achieve different levels of KL divergence. Also we have added more points in the following figure with fair KL range (Figure E: https://anonymous.4open.science/r/ConfPO-B7C6/L/FigureE.md).\n- **Number of tokens selected:**  \nWe apologize for the typo. The correct initial selection rate is indeed approximately 40% for both Mistral-Base and Llama-3-Base. We will correct this in the main text.", "claims_and_evidence": "The primary claim in the paper is that low-confidence tokens dominate preference learning. This is motivated by the observation that 1) token-level gradient norms follow a long-tailed distribution, and 2) there is a strong negative correlation between gradient norm and confidence. While these observations are supported empirically, there is no theoretical motivation for this simple observation.\n\nThe paper also states that ConfPO mitigates overoptimization (reward hacking) by using KL budget more efficiently. While Figure 5 shows a higher win rate for a given KL budget for ConfPO against SimPO at the high end of KL divergence, there is no clear gain at the low end, and this figure alone does not justify the claim about KL budget efficiency (which is nonetheless an intuitive claim).\n\nAlso, the claim that \"high-confidence tokens can impede preference learning by overshadowing more informative signals from their low-confidence counterparts\" provides a nice intuition for why the method works (i.e., a form of denoising), but it was not demonstrated theoretically or empirically that this was the mechanism of ConfPO outperformance.", "methods_and_evaluation_criteria": "The experimental setup (including models used and evaluation benchmarks) is pretty straightforward and makes sense. The paper includes reasonable motivation for why ConfPO is built on top of SimPO, and not other DAA variants. The ablation showing that the method also generalizes to DPO provides support for the claim that ConfPO is DAA-agnostic.\n\nThe ConfPO method is empirically motivated, based on the observation that tokens with lower probabilities (confidence) tend to have larger gradient norms. But no theoretical justification is provided for this.", "theoretical_claims": "This was primarily an empirical paper, with no theoretical claims or proofs.", "experimental_designs_or_analyses": "Questions:\n\n1. What is the correlation between tokens' confidence (+ gradient norm) and their position in the sequence? Tokens are generated autoregressively, so are not independent. Are most of the \"preference-critical tokens\" located towards the beginning, middle, or end of sequences, or are they approximately uniformly distributed? This is a key missing investigation, and it would be nice to see an additional ablation added to Table 2 (beyond ConfPO-rand), where the tokens are selected to match the sequence position distribution of ConfPO.\n\n2. (related to 1) Did you perform any qualitative or semantic analysis of the preference-critical tokens (across both the preferred and dispreferred examples)? Do preference-critical tokens tend to cluster into phrases, or do they tend to be sprinkled across diverse positions in the sequence?\n\n3. ConfPO uses hard weights (thresholding based on token logprobs) in the loss function (equation 11). Did you explore soft weights (e.g. weighting directly proportional to token logprob)? Does performance scale with the \"temperature\" (so that hard weighting is best)? This would be an insightful ablation.\n\n4. (related to 3) Did you explore a curriculum learning setup, where the weights on preference-critical vs non-preference-critical tokens are adjusted over the course of training? How does this compare to vanilla ConfPO?", "relation_to_broader_scientific_literature": "There has been a recent explosion in literature on Direct Alignment Algorithms, starting with DPO. These algorithms are summarized nicely in Table 5 in the Appendix. The ConfPO algorithm proposed in this paper is primarily explored as a modification to the SimPO algorithm (which modifies DPO by eliminating the need for a reference model and mitigating the length bias). This paper also cites other recent work in token-level preference learning, and differentiates itself by not relying on external models for token-level reward signals.", "essential_references_not_discussed": "The paper includes the most important prior works in the areas of Direct Alignment Algorithms, including token-level variants.", "other_strengths_and_weaknesses": "The ConfPO method is extremely simple and not particularly novel, but the strength of this paper is in the promising results, showing substantial and consistent gains over DAA methods which weight each token equally.", "other_comments_or_suggestions": "- Section 3.1: Usage of D in equation 2 as the set of prompts differs from the definition stated in the text (preference dataset, with prompts as well as preferred and dispreferred responses). The usage of D in equation 4, on the other hand, aligns with the definition stated in the text.\n- Why is the Bradley-Terry model mentioned in Section 3.1? Isn't it more relevant to Section 3.2?\n- Figure 4: Which model is trained here? And on which dataset?\n- Equation 11 is an exact copy of equation 7, but just going by a different name. Is it necessary to copy, or can r_{ConfPO}(x, y) just be defined as given in equation 7?\n- What hyperparameter is being adjusted to plot the points in Figure 5? Especially for Figure 5b, the \"Square root of KL divergence\" has a notably higher maximum for ConfPO than for SimPO.\n- In the \"Number of tokens selected\" paragraph in Section 7.1, the text states that \"at the start, approximately 30% of tokens are selected for Mistral-Base and 40% for Llama-3-Base\". But from Figure 6, it looks like the correct number is ~40% in both cases.", "questions_for_authors": "No additional questions, besides those mentioned in previous sections. In particular, please see the questions in the \"Experimental Designs or Analyses\" section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: My recommendation is to accept the paper.

The paper proposes filtering out high-confidence tokens when performing gradient updates in direct alignment algorithms such as DPO. The authors show that the ConfPO method outperforms current baselines and appears to be less prone to reward hacking. Several rationales are given for the filtering, and the filtering method is applied to multiple DAAs.

Reviewers raised several questions about the motivation of the method, comparisons to other methods, and applicability to other DAAs. The authors addressed these questions well in the rebuttal. I would encourage the authors to include these disucssions and comparisons in the text of the paper.
Decision: Accept (poster)