Noise Contrastive Alignment of Language Models with Explicit Rewards
Authors: Huayu Chen
Guande He
Lifan Yuan
Ganqu Cui
Hang Su
Jun Zhu
Venue: NeurIPS 2024 poster (2024)
Keywords: noise contrastive estimation
reinforcement learning
language model
alignment
contrastive
Primary area: natural_language_processing
TL;DR: We propose an alignment method for language models that include DPO as a special case

Abstract: User intentions are typically formalized as evaluation rewards to be maximized when fine-tuning language models (LMs). Existing alignment methods, such as Direct Preference Optimization (DPO), are mainly tailored for pairwise preference data where rewards are implicitly defined rather than explicitly given. In this paper, we introduce a general framework for LM alignment, leveraging Noise Contrastive Estimation (NCE) to bridge the gap in handling reward datasets explicitly annotated with scalar evaluations. Our framework comprises two parallel algorithms, NCA and InfoNCA, both enabling the direct extraction of an LM policy from reward data as well as preference data. Notably, we show that the DPO loss is a special case of our proposed InfoNCA objective under pairwise preference settings, thereby integrating and extending current alignment theories. By comparing NCA and InfoNCA, we demonstrate that the well-observed decreasing-likelihood trend of DPO/InfoNCA is caused by their focus on adjusting relative likelihood across different responses.
In contrast, NCA optimizes the absolute likelihood for each response, thereby effectively preventing the chosen likelihood from decreasing. We evaluate our methods in both reward and preference settings with Mistral-8$\times$7B and 7B models. Experiments suggest that InfoNCA/NCA surpasses various preference baselines when reward datasets are available. We also find NCA significantly outperforms DPO in complex reasoning tasks like math and coding.

Noise Contrastive Alignment of Language Models with Explicit Rewards
Huayu Chen 1.2, Guande Hel Lifan Yuan', Ganqu Cui', Hang Su 12.3, Jun Zhu 1,2+ 7.2
*Department of Computer Science and Technology, Tsinghua University Institute for AI, BNRist Center, Tsinghua-Bosch Joint ML Center, THBI Lab, Tsinghua University 77hogguunceun Laboratory, Beijing, China
Abstract
User intentions are typically formalized as evaluation rewards to be maximized when fine-tuning language models (LMs). Existing alignment methods, such as Direct Preference Optimization (DPO), are mainly tailored for pairwise preference data where rewards are implicitly defined rather than explicitly given. In this paper, we introduce a general framework for LMM alignment, leveraging Noise Contrastive Estimation (NCE) to bridge the gap in handling reward datasets explicitly annotated with scalar evaluations. Our framework comprises two parallel algorithms, NCA and InfoNCA, both enabling the direct extraction of an LM policy from reward data as well as preference data. Notably, we show that the DPO loss is special case of our proposed InfoNCA objective under pairwise preference settings, thereby integrating and extending current alignment theories. By comparing NCA and InfoNCA, we demonstrate that the well-observed decreasing-likellhood likelihood trend of DPO/InfoNCA is caused by their focus on adjusting relative likelihood across different responses. In contrast, NCA optimizes the absolute likelihood for each response, thereby effectively preventing the chosen likelihood from decreasing. We evaluate methods in both reward and preference settings with Mistral-8 7B and 7B models. Experiments suggest that InfoNCA/NCA surpasses various preference baselines when reward datasets are available. We also find NCA significantly outperforms DPO in complex reasoning tasks like math and coding. Code: https; //ggithub..com//uuum/////// Conteastiotottte....oo AligeeeeAlnnnntt
1 Introduction
Aligning pretrained Language Models (LMs) with scalar rewards that reflect human intentions is crucial for enhancing their ability to follow instructions [35] 25 These rewards can be given either explicitly or implicitly. Explicit rewards can be scalar ratings human annotators or advanced models like GPT-4, while implicit rewards are usually preference labels assigned to pairwise responses.
One effective approach for aligning LMs with preference data is Direct Preference Optimization (DPO. 33 DPO applies reward training loss but parameterizes the reward model the response likelihood ratio between two LMs, allowing for training reward models and extracting LM policies simultaneously. This approach is more streamlined and thus more favorable compared with traditional Reinforcement Learning (RL) methods (26), which typically require a two-stage training process: first training reward models, then extracting LM policies.
Despite its simplicity and effectiveness, DPO is only tailored for preference data (s - {yw W))) When multiple responses are available, directly assigning a scalar reward to each response is usually more convenient and efficient than comparing them pairwise manner. The resulting reward
The corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

K
learn
A 2
4 R
Relative Duta Likealhooo
Preference
Model
PPO
Litelhood Margie
prune
DPQ
Yon
Policy
Ylas (DPO))
Unfo)
Viar (NCA)
DPO
A n.
5.
Ylars (NCA)
Special Case
Preference
Epoch (86)
100
Epoch (5%)
100
Figure I:
InfoNCA/NCA allows direct LM opti- mization for both reward and preference data.
Figure 2: Pairwise NCA prevents chosen likeli- hood from decreasing while DPO cannot.
datasets (x + (.k), however, cannot be directly leveraged for DPO training. Previous work [40] usually prunes reward datasets by selecting the best response and pairing it with a random remaining one. This is suboptimal all reward values and additional dispreferred responses thrown away in its data preeroccespinngg process.
To address this issue, we present InfoNCA, alignment method that allows directly extracting LM policies from both reward datasets and preference datasets with arbitrary response numbers (Figure T]. Notably, InfoNCA subsumes DPO loss as a special case under pairwise preference settings and can thus be seen as a natural extension of DPO (Sec. 3.2). With strong theoretical guarantees, we show DPO is binary classification loss while InfoNCA is its multi-category version (Figure 3). However, unlike DPO which built upon assumptions of Bradley-Terry models or Plackett-Luce models, InfoNCA is strictly derived from Information Noise Contrastive Estimation (InfoNCE, (24)), an established contrastive method that widely applied in language and visual representation learning 31. This closes the theoretical gap between current preference alignment methods and classic contrastive learning frameworks.
A well-observed problem with DPO is that the likelihood of the preferred response tends to decrease throughout training [27
Similar trends are shared by InfoNCA. We find this issue arises mainly from InfoNCA/DPO's focus on adjusting the relative likelihood across different responses per instruction. Sec. 4 we propose NCA as an alternative alignment method to InfoNCA to mitigate this problem. NCA differs from InfoNCA by only loss definition and is also suitable for both preference and reward datasets. However, NCA built on NCE [14, a parallel contrastive learning algorithm to InfoNCE, which optimizes the absolute data likelihood during training. In practice, NCA effectively prevents the chosen likelihood from decreasing (Figure 2).
We evaluate our methods on Mistral-7B and 8 models from two dimensions. When reward datasets are available, we show that directly applying our seward-based alignment offers clear improvement compared with preference-basee algorithms, achieving higher evaluation rewards in GPT-4 47 [9 evaluations. We further validate this improvement comes from InfoNCA/NCA's ability to fully leverage the additional suboptimal responses. When only preference data is given we compare pairwise NCA against the DPO loss. Our experimental results spanning various benchmarks show that NCA outperforms DPO in complex reasoning tasks such math and coding.
44
Our main contributions: We bridge the theoretical gap between DPO and classic contrastive learning theories. InfoNCA and NCA are uniquely suited for both reward and preference data, offering a general framework that integrates preference-based algorithms. We show that suboptimal responses are also important for LM optimization. Our method outperforms various preference methods by fully exploiting data information reward datasets. 3. NCA effectively mitigates the data likelihood decline issue of DPO and offers practical performance improvement.
2 Background: Direct Preference Optimization
LM alignment is essentially a constrained policy optimization problem:
max To
Fx((2) [ [F y) a DEL (To(-(=))ll(((a))
(1)
where y represents the pretrained LMM x and y are respectively instructions and responses. r is reward function that reflects human intentions. a is some temperature coefficient. Prior work [30 29)

---

has proved that the optimal solution for the optimization problem in Eq. is:
म [y]x) = p(y) (xx)
o pe
Direct Preference Optimization (DPO) 33 assumes we only have access some pairwise preference data I + {ye > Yr for each instruction z. The preference probability of human annotators is modeled by a learnable implicit reward model ry under Bradley-Terry theories [2]:
we (yo > w [z) == (Wo x) ro(y,,
where a is the sigmoid function. To learn re. DPO simply adopts a binary classification loss:
LDO = - - log (ro .) r)()).)))
In practice, the latent function re is parameterized by the log-likelihood ratio between my and
ro(x, = 3 log T
ply ar)
where linear coefficient for scaling rø- This parameterization is crucial because it ensures x² (:) ((o)x))ero constantly hold. It transforms generative policy optimization into a are satisfied, we naturally have
simple discriminative classification task: When rg T x².
InfoNCA: Extending DPO from Preference to Explicit Rewards
Compared with constructing preference datasets, annotating each response with scalar rewards can be more flexible and convenient. Preference methods are only suitable for pairwise data (z - {Nw > )1)) and would require C evaluations for comparing K responses. In contrast, reward datasets (x (vr ri)::) allow an arbitrary number of responses per prompt with K evaluations.
Despite its simplicity in handling preference data, DPO is not tailored for reward datasets. We introduce a new alignment method termed InfoNCA to mitigate this gap. We first strictly derive be solved by constructing classification
InfoNCA in Sec. 3.1 We show that reward alignment problem to identify the optimal response from multiple candidates. We then demonstrate that InfoNCA subsumes DPO special case and thus natural extension of DPO (Sec. 3.2
3.1 Reward Alignment through Multi-Class Classification
In essence, DPO represents response rewards LM likelihoods and constructs binary classification task for learning the reward model. Given that there are more than two (K > 2) responses per prompt in reward datasets, seek construct multi-class classification task for learning reward models from explicit rewards instead of preference labels. We begin by formally defining this task:
Consider a batch of K responses {ur hik for an instruction z. {yA } consists of one optimal response yo that is sampled from x (y) (r)) ((y))a) and K - suboptimal noises indepen- dently sampled from μ(y(=r). v E 1 K the random index of that optimal response. Our goal is to identify which of the K candidates is you given only reward labels r(yk) for each response.
Intuitively, the response with higher rewards should have higher probability of being the target response. This intuition can be more rigorously expressed:
Proposition 3.1 (proof in Appendix A.I). Given the above K response candidates and their respec- tive rewards, the posterior probability for the v-th response being drawn from n² is
p(v|z, (vshik) = ΣK (x.
(4)
This finding is highly appealing because it shows response optimality solely related to response rewards. This allows to conveniently train reward models ra via maximum likelihood (MLE).

---

Which response is preferned?
r=3.0
Which of these responses comes from
sampled from yn sampled from
π
π
ya
YB
YB
erobp
Yx
pobi) otro0t1
(a) DPO pabx) -100s))-000
(b) InfoNCA 1.3
(c) NCA
Figure 3: DPO, InfoNCA, and NCA all optimize LLM through classification tasks. DPO compares two responses and tells which one preferred. InfoNCA compares multiple responses and identifies the one sampled from n¹ (Sec. 3.1 NCA predicts the model source of single response (Sec. 4.1
Theorem 3.2 (InfoNCA, proof in Appendix A.Ip. We define n" (ylax u(ylr)e and To(() p(ylz)))>(a..)) For any K > O, we have:
(a) Equivalent objective. The MLE objective for training ræ has an equivalent form:
max E Al.) po (we)) + min -Bp()
log
(5)
7(x)
プラ (to(s))))
where (x) = ea
(b) Optimal solution. Assume unlimited model capacity. The optimal Tq- and me- for solving Tg- y) (lzz.v//a C(x),
are
and T (y-(y]x) TT" (ylæx) x )e (a.w)/a
where C(x) is an arbitrary function conditioning on a.
In practical implementation of Eq.
we estimate Z(x) K Ee'soo and parameterize ra v) =
log p(
se(uiz) following DPO (Eq.
The loss function becomes
model logits
eti/a
(ro(2.0))
ClafoNCA (z, (yr, r...k) Co
Σ1
log
(6)
2j=1 ersta
mafyle) Σ (ro(z.s)) ra(m.))=Blog p(ul=))
soft labels
predicted probability
The loss function in Eq. olis termed InfoNCA, where A stands for Alignment. This naming reflects its
functional similarity to the Infomation Noise Contrastive Estimation (InfoNCE, (24). Both methods transform generative modeling problems into classification tasks by contrasting multiple data points.
How does InfoNCA work? InfoNCA loss (Eq, 66 be seen K-category cross-entropy loss. The soft label is calculated by dataset rewards through a softmax operation. The model's predictions represented by learned reward rg. The loss reaches when r (I, y) r(x, y)/a + C(z).
3.2 InfoNCA Subsumes DPO A Special Case
Below we show that DPO is a special case InfoNCA asymptotically. Specifically, setting response number K 2 and reward temperature 0, we can fully recover the DPO objective:
CInfoNCA (x, r.).k)
Σ [ Tmux) log
(a 0)
DK (ro(osa)))
llog
(K = 2, suppose Tw > m))
(ro(e,8)) pro(m.n)
log (r vw.) ro(x)
(DPO loss, Eq. 3

---

Alignment Method
InfoNCA (Sec.
NCA (Sec.
Modeling Target
(ulx)) x plgla))e7
Model Defimition
To(al)) x plyla)ettaasw
Reward Dataset
x tuk.
Loss (K> (a>1))>>)
E, atsia log eateivl M,, e'olnmyy)
Σ
E, er log (1o(( log oogoo(rr((2:))))
Preference Datasct
x - (Bw w1
Loss (K=2,, a->
logo(ro(a, rola, ())()P))
logo (fo (ve log (-ro(,,)
Loss Type
InfoNCE loss (2
NCE loss 14
Optimizing Target
relative value log likelihood ratio
absolute value log likelihond ratio
Optimal re- (2, v)
r(a,)) /a C(x)
r(z, v)/a log Euglin e (2/00//
Pg- (xt, Sbeat) 2 0.
not guaranteed
Table I: Comparison of NCA and InfoNCA algorithm for aligning language models. Both reward loss and pairwise preference loss are given. We provide pseudocode in Appendix
Empirical effect for varying hyperparameter K and As indicated by the derivation above, the root difference between preference-based and reward-based methods lies in the choices of K and a.
K affects how accurately we can estimate the partition function (xx x E5= e///a in Eq. In practice, we find larger K can lead to better performance (Sec. 5.T). indicates trade-off between diversity and optimality. At a + 0, the InfoNCA loss increases the likelihood only for the optimal response and decreases it for all other responses, turning the reward dataset x - {3/, r into preference dataset + (th >> )r We provide ablation studies of a and 33 in Appendix
4 NCA: Fixing Decreased Response Likelihood Issue for InfoNCA
A well-observed issue with DPO is that the likelihood of all responses continually decrease throughout training [2]. We find InfoNCA shares this trend due to their inherent equivalence. Decreased data likelihood is concerning because it directly contradicts the maximum likelihood objective for supervised training and may eventually harm performance 44
We hypothesize the main cause of this decreasing likelihood is that InfoNCA methods only adjust relative rewards among responses, rather than optimizing their absolute value. To address this problem, we take inspiration from NCE, another contrastive learning method parallel to InfoNCE, and propose (CA(lig)))nt) (Sec. 4. Similar to InfoNCA, NCA can also guarantee convergence to the optimal LM policy under ideal conditions (Theorem 4.2). However, it directly learns the absolute reward for each response, thereby counteracting the decreasing likclihood trend (Sec. 4.2).
4.1 Reward Alignment through Absolute Reward Prediction
To avoid optimizing relative rewards across multiple responses, we construct a binary classification task that deals with a single response.
Specifically, imagine sampling response y randomly from either the optimal LM л (y]2z)
((yla) (x)) Z(z) or the pretrained LM p(y (r) The marginal probability of is p(yl=) = } +
(g)x). Our goal is to guess model source when given a response y and its reward r(y).
Proposition 4.1 (proof in Appendix A.2 Let binary variable v indicates the response y is
sampled π². The posterior probability of the distribution source given the response satisfies:
(y]a))
p(v 112, +
(7)
7(x) + e'(as)//"
Note that p(u|2r,)) related to the partition function 7(x). order to represent model likelihood po (1/22 y) by only employing re similarly to Proposition 3.1 we have to redefine
5

---

=o(sl=)==(ulz)eertca by absorbing Zo into rg. Then we have
<o(y Po = 1|,,0/ p(ylz) + (b(()) = ((o((z,))) v)).
(8)
Similarly to Theorem 3.2
derive MLE-based training objective for optimizing re-
Theorem 4.2 (NCA, proof in Appendix A.2 Let a > O, we have the maximum likelihood objective: (a) Equivalent objective.
max E p(z.)) log Po (u/ v) min Ba(o)r(u)))
log o (ro (2. ()) log logo((ro(z,,))) ro(c.u)).
((x)
where (z) E e
(b) Optimal solution. Assume unlimited model capacity. The optimal ry- and Tg- for solving are rg- y) = r(z,((z))) v)/a log E e (a.s)/a and To-(y)z) ((y)z)er(z..))
(10)
For reward datasets (ar + (i),,,) we estimate Z(x) 2 Ei= en / in Eq. similarly to InfoNCA:
and construct ra
ei/a
C rih:k) =
2j=1 e/s/a log (ro(a, vi))
log ol (-ro(,
K
ro(2.)):=$1o log zefvle) MCul))
optimizer 1
regularizer 1 (decreasing force)
weight
(increasing force)
(11)
How does NCA work? The loss function for NCA involves two opposing forces that jointly determine the trend of increasing or decreasing ra y). Since log a(-) is a monotonically increasing function, the first term in Eq. 11 tends to increase ro( while the second term tends to decrease it.
At the start of training, when re -T9 O, the direction of the combined force for is decided by pri/u
the difference in their weights, expressed as Σg e5yla K Responses with higher rewards would,
in principle, attain higher likelihood after training.
4.2 Connection between NCA and InfoNCA/DPO
Although both NCA and InfoNCA originate from solving a noise contrastive classification problem, their optimization targets are markedly different (Table
InfoNCA and DPO both calibrate relative values of reward models across various responses {3r hi for an instruction x. other words, the absolute value (2, y) directly constrained. This can lead to some counterintuitive behaviors. For instance, the learned reward for even the highest-reward response could decrease over time without contradicting the loss definition, as long as the reward margin keeps increasing. This could lead to poor performance training instability (Sec. (.2)
In contrast, NCA specifically focuses on optimizing absolute values of the reward model. This characteristic is determined by its model definition: =(())=)==((l))) (() where rа has to be self-normalized: Ep(u|=)e ( 1. In practice, NCA effectively prevents the likelihood of the preferred responses from decreasing. We find this is particularly helpful for math and coding tasks.
Experiments
We mainly seek to
questions in experiments:
1. If we have access to reward-annotated datasets with >2 responses per prompt, does InfoNCA NCA offer empirical improvement compared with weffeeence-based approaches that simply prune reward datasets into preference datasets? (Sec.5.11
2. If only pairwise preference data available, when should one choose NCA over DPO? What benefits does NCA offer? (Sec. 5.2) Note that InfoNCA is exactly DPO in this setting.

---

Name
Annotation Type
MT-heech
AlpacaEval
Win vs. DPO
Mixtral-7B-sft
SFT Data
6.45
85.20
Baseline
+KTO I
Preference
7.12
91.93
+IPO D
Preference
7.45
90.62
+PPO (Zephyr-3)
Preference
7.34
90.60
50.0
+DPOx3
Preference
7.22
91.60
58.1
+DPOx
Preference
7.38
90.29
48.1
Reference Ours
IIfoNCCA
Reward
7.63
92.35
56.9
+NCA
Reward
7.52
90.31
59.4
Mixtral-ORPO-3
Preference+SFFT
7.32
91.41
Mistral--7Biinstuut
SFT Data
6.84
92.78
L.aaAA-chat-70b
Reward Model
6.86
92.66
GPT-4
Reward Model
9.18
93.78
Table 2: Comparison between reward-based methods (InfoNCA, NCA) and preference- based methods (DPO, IPO, etc.) in LLM alignment. We focus the general cnttructionfollloviii abilities of each method measured by GPT-4 evaluations and human preference. The highest number in each benchmark bolded and the second highest is underlined.
5.1 Aligning Language Models with Explicit Rewards
Reward dataset and Evaluation metric. We consider UltraFeedback [9] an instruction followiinnn dataset annotated by GPT-4. This dataset comprises ~64k instructions. Each instruction has responses generated by various LMs. GPT-4 rates each response with a scalar reward on a scale of Q-10. Prior research indicates that these GPT-4 rewards closely align with human annotations [4]. establishing them as an efficient, cost-effective alternative to human feedback. In order to align exactly with the definition of dataset rewards, we similarly choose well-acknowledged GPT4-based benchmarks like MT-bench 477 and AlpacaEval [19] for evaluation. Human preference studies are also conducted on evaluation prompts from MT-bench. The rating system is in Appendix C
InfoNCA and NCA outperform preference-basee methods given reward dataset. To handle reward datasets with K 2 responses per instruction, one approach is simply prune them into pairwise data and apply preference learning like DPO. For instance, Zephyr [40] selects the highest- reward response and random remaining one from UttraFeddback for each instruction. This procedure discards two additional suboptimal responses in the dataset as well as their reward information.
In Table we fine-tune Mistral- 1-r model on UltraFeedback and compare InfoNCA/NCA against the DPO baseline. Results show that our methods outperform preference baselines. This improvement can be attributed to InfoNCA/NCA's ability to exploit all information in the reward dataset.
Method
K=2
K=3
K=4
7.5
InfoNCA (MT-bench)
73.8
75.9
76.3
InfoNCA (Alpaca)
90.7
90.2
92.4
MT-bench
7.0
K=2 2
NCA (MT-bench)
73.2
73.3
75.2
Km4
NCA (Alpaca)
89.9
90.3
90.3
Average
81.9
82.4
83.5
0.0
0.5
1.0
1.5
2.0
(L(mal1A)
Figure 4: More suboptimal responses can also increase LLM's fnstruetionnfollloiingg ability. We fix the highest-reward response the UltraFeedback dataset and ablate the number of suboptimal responses per prompt, resulting in different contrastive response numbers K during training. Left: Evaluation results under the same set of hyperparameters. Right: Performance-Ki trade-off under various and B. Each dot represents an independent experiment trained for epoch.
Suboptimal responses are also important. Previous practices always ensure selecting the highest- performing response when constructing preference data. The assumption behind this strategy that the dataset's s best-performing response determines the upper limit of alignment performance. However, our experiments contradict this assumption. Results Figure indicate that extra subop- timal responses can also be advantageous for policy training. Specifically, observe consistent
7

---

Reasoning BBH (CoT)
Coding
Math TheoremQA
LeetCode
HumanEval
GSMPLUS
MATH
SVAMP
ASDiv
Avg.
Mixtral78-SSFT
60.9
3.3
28.1
28.5
5.8
7.0
26.9
35.8
24.5
DPO
61.7
2.2.
31.7
12.1 4
6.4
9.8
34.1
46.1
25.5
NCA
60.8
3.3
26.8
32.3
11.7
11.0
65.3
74.3
35.7
75.6
16.7
57.6
40.1
85.9
56.3
DPO
74.9
17.2
47.6
55.8 4
35.3
26.9
67.3 4
75.7
50.14
+ NCA
75.6
21.1
62.8
61.5
41.6
26.9
86.8
86.9
$7.9
Table 3: Alignment results for UltraInteract. We mark numbers that have decreased (1) after training.
performance improvements when increasing the number of data responses from K 2 to K 4 for both InfoNCA and NCA algorithms, across various hyperparameters.
Combinatorial DPO are suboptimal solutions. Regarding the performance improvement offered by more suboptimal responses, one might predict that applying the DPO to a combinatorially constructed preference dataset would yield results comparable to NCA/InfoNCA. To investigate this, we examined two variants of DPO that utilize all available responses in UltraFeedback.
x We pair the tighest-performing response with each of the remaining three separately.
x We sum up DPO loss possibilities for two out of the four responses.
Our experiments, detailed in Table 2 reveal that naively applying combinatorial DPO loss to leverage all response information underperforms InfoNCA/NCA. The DPOx3 shows some henefit, while x C2 is harmful compared with simple data pruning. This is expected because InfoNCA and NCA possess theoretical guarantees (Theorem 3.2 and Theorem (.2 that ensure convergence the optimal LM policy whereas combinatorial preference methods do not.
5.2 NCA vs. DPO in Aligning Language Models with Pairwise Preference
In previous experiments, our focus on the reward dataset with K > 2 responses per prompt (x + (y., h.kk. However, at present most alignment datasets are pairwise (r + {uw > yt)), making it essential also evaluate our proposed methods in pairwise preference settings.
Since InfoNCA is equivalent to DPO when only pairwise preference data is available (Sec. 3.2 we will focus comparing and clarifying the differences between the DPO and NCA algorithms.
Preferecne dataset and evaluation metrics. We consider fine-tuning Mistral-7B and Mistral-8x7B models on Ultralnteract [4]. pairwise alignment dataset specifically designed for complex reasoning tasks. Before alignment, we perform SFT on Ultralnteract's preferred responses for the 8x model and use the existing Mistral-SFT model in Sec. 5.1 We evaluate the model's performance in various challenging tasks. This includes BBH-Hard [39] for CoT reasoming, HumanEval [4] and LeetCode
12 for coding, GSM-Plus [8], MATH, TheoremQA (6) SVAMP (28 and ASDiv 22 for math.
DPO may hurt reasoning performance while NCA helps. Results are presented in Table 3 Overall, NCA consistently outperforms DPO in various benchmarks. Notably, we observe DPO hurts the overall performance in most reasoning tasks regarding the Mixtral-8 8 7BT model. This indicates that DPO might not be suitable for improving reasoning abilities, which echoes findings in concurrent
work In contrast, NCA shows clear improvement on both the 7B and 8x 7B models.
DPG (Ultralntenee))
IaANCA (ellak)
NCA (UllaaFooo)))
Rold
Modil Rewadd
Mold Rowwed
Chooe Rejetal
Cheaen
So.
Reyerst
208
Skep (%4)
100
Figure 5: Comparision of data likelihood between InfoNCA/DPO and NCA.
NCA prevents the chosen-likelihood from decreasing. What distinct optimization characteristics could cause performance differences between pairwise NCA and DPO? To understand this, we empirically inspect how the data likelihood changes during training. As shown Figure 5 The likelihood of preferred responses interestingly decreases after DPO training and increases for NCA training. This pattern is consistent across both preference and reward learning. The decreasing chosen- likelihood trend is concerning because it directly contradicts the maximum- -ikelliooo objective used
8

---

MT-bench Score
MT-bench Score
7.0
6.0
DPO
InfoNCA
NCA
Collapsed Experiments
5.0
NCA
1.0
2.0
3.0
4.0
0.0
1.0
2.0
3.0
4.0
5.0
KL (n
(L(molla)
Figure 6: NCA is more robust to hyperparameter changes and KL derivations. Left: Ablation results of a and for wewFeeedbba--bbinaiiiee Right: Results for UtraFeeedback-eeward (K = 4).
during the SFT stage. This drawback is exacerbated in reasoning tasks, where the preferred response often the ground truth answer. Consequently, we hypothesize that NCA's superior performance in reasoning tasks is due to its ability to avoid decreasing chosen likelihood.
Since DPO essentially a specialization of InfoNCA. their contrasting likelihood trends can be explained theoretically. As we have elaborated in Sec. 4.2 NCA adjusts the absolute likelihood of data, while DPO/InfoNCA only considers relative likelihood across different responses. Thus, declining chosen likelihood directly contradicts NCA's training objective but not DPO's.
DPO? DPO and pairwise NCA have similar theoretical guarantees. Their different performance in alignment tasks is largely empirical, depending on the specific characteristics of datasets and the nature of tasks. Our observations show that NCA is more suitable for reasoning tasks such as math and coding (Table where high-quality responses are sparse, and adhering closely the preferred responses the dataset critical. DPO may be more suitable for general instruction-followiin tasks like summarization/rll--piii datasets only reflect human relative preference but do not contain "golden" answers. essence, NCA benefits from better dataset regularization, while DPO relies more on LLMs' generalization abilities.
Empirical takeaway: When to choose NCA
(Table 2 where
We also observe that NCA has greater tolerance for divergence from the initial SFT policy and is more robust to hyperparameter changes. As evidenced in Figure [6 we conduct a grid search on and DPO can drastically fail improve alignment performance if the learned policy strays too far from the SFT policy, and may randomly collapse under certain hyperparameters. In contrast, the NCA method does not exhibit similar issues. These observations suggest that NCA may be worthwhile alternative if DPO training is unstable.
Related Work
Language model alignment. Current approaches cater to either explicit reward data or preference data, often lacking the versatility to address both concurrently. Reinforcement Learning 33 is
inherently suitable for explicit reward scenarios. However, its on- on-policy nature necessitates learning
reward model from data first, leading to an indirect two-stage optimization process 26 36] Recent developments in preference-based alignment techniques [33 [
42
16. 13 have streamlined
this process. They enable direct alignment LMs through a singular loss, but this comes at the
expense of being confined to pairwise preference data. Other alignment approaches 46
37 45 38
are also not tailored for aligning with reward datascts. Recent work [3] attempts to extend DPO's parameterization technique to explicit reward contexts. However, it only considers binary rewards. In comparison, our methods can handle both continuous rewards and preference data.
Noise contrastive estimation. NCE 14 and its variant, InfoNCE 24 are established optimization methods for training unnormalized generative models (1 NCE primarily leverages a binary classification loss and can be applied in self-supervised representation learning. Examples are Word2Vee (23), MoCo [15), and SimCLR (5). InfoNCE related to maximizing mutual information between two distributions through multiclass cross-entropy loss. It has successful applications in representation learning, such as CLIP (3T). It also widely used in language modeling ( diffusion
modeling [20), and reinforcement learning [17]

---

7 Conclusion
In this work, formally consider the language model alignment problem in the context of explicit reward settings. By adeptly harnessing the NCE and InfoNCE theories, we introduce two practical algorithms: NCA and InfoNCA. Our proposed methods are uniquely suited for both reward data and preference data, including DPO as special case. Our experiments show that reward- based alignment methods outperform preference baseline by fully leveraging suboptimal responses reward datasets. In preference settings, pairwise NCA outperforms DPO in complex reasoning tasks by effectively preventing data likelihood from decreasing.
Acknowledgments and Disclosure of Funding
We especially thank Cheng Lu, who greatly inspires in linking the NCA algorithm with NCE theo- ries. We also thank Jiuhai Chen, and Tianlin Liu for their suggestions on Zephyr result reproduction. We thank Bingrui Li, and Weiyu Huang for their help with the experimental setup. We thank Github user Wing Lian for integrating the NCA algorithm into the trl library. We thank Haosheng Zou for providing feedback
method.
This work was supported by NSFC Projects (Nos. 62350080, 92370124, 92248303, 62276149, 13661,, 62076147), BNRist (BNR2022RC01066 Tsinghua Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University. Zhu was also supported by the XPlorer Prize.
10

---

References
[1] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
[2] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
[3] Tianchi Cai, Xierui Song, Jiyan Jiang, Fei Teng, Jinjie Gu, and Guannan Zhang. Ulma: Unified language model alignment with demonstration and point-wise human preference. arXiv preprint arXiv:23/2.02554. 2023.
[4] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthia Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. 2021.
[5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[6] Wenhu Chen, Ming Yin, Max W.F. Ku, Yixin Wan, Xueguang Ma, Jianyu Xu, Tony Xia, Xinyi Wang, and Pan Lu. Theoremqa: A theorem-driven question answering dataset. ArXiv preprint, abs/2305. 10..005..2222... 2023.
[7] Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang. Xia Song, Xian- Ling Mao, Heyan Huang, and Ming Zhou. InfoXLM: An iofommation-theoretie framework for cross-lingual language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3576-3588, Online, June 2021. Association for Computational Linguistics.
[8] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[9] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie. Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with scaled ai feedback. In Forty-first International Conference on Machine Learning, 2024.
[10] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.1431 2023.
[11] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306. 2024.
[12] Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming the rise of code intelligence. ArXiv preprint, abs/2401.14196, 2024.
[13] Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Jiexin Wang, Huimin Chen, Bowen Sun, Ruobing Xie, Jie Zhou, Yankai Lin, et al. Controllable preference optimization: Toward controllable objti-obeective alignment, arXiv preprint arXiv:2402.1908s 2024.

---

[14] Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications natural image statisties. Journal of machine learning research, 13(2), 2012.
[15] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[16] Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.0769), 2024.
[17] Michael Laskin, Aravind Srinivas, and Pieter Abbeel, Curl: Contrastive unsupervised represen- tations for reinforcement learning. In International Conference on Machine Learning, pages 5639-5650. PMLR, 2020.
[18] Qintong Li, Leyang Cui, Xucliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehen- sive benchmark for evaluating the robustness of 1lms mathematical problem solvers. ArXiv preprint, 1atp22422.12555, 2024.
[19] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Guirajani, Carlos Guestrin, Percy Liang, and Tatsumori B Hashimoto. AlpacaEval: An automatic evaluator of instruction-followinn models, 2023.
[20] Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guides diffusion sampling in offline reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning, 2023.
[21] Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. arXiv preprint arXiv: 10XX::8809.00812 2018.
[22] Shen-yun Miao, Chao-Chun Liang, and Keh- Su. A diverse corpus for evaluating and developing English math word problem solvers. In Proc. of ACL, 2020.
[23] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
[24] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint 1rXiv:1807.033 03748, 2018.
[25] OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[26] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744 27744, 2022.
[27] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.144 13228, 2024.
[28] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, 2021.
[29] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighter regression: Simple and scalable off-policy reinforcement learning. arXiv preprint 191.00191..00177, 2019.
[30] Jan Peters and Stefan Schaal. Reinforcement learning by reward- weighted regression for operational space control. Proceedings of the 24th international conference on Machine learning, pages 745-750, 2007.
12

---

[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 9748-8763. PMIR, 2021.
[32] Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From to q.: Your language model is secretly a q-function. arXiv preprint arXiv:2404. 2024.
[33] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[34] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1XXYv:7777.06377 2017.
[35] John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Felipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, et al. Chatgpt: Optimizing language models for dialogue. OpenAl blog, 2022.
[36] Wei Shen, Xiaoying Zhang, Yuanshun Yao, Rui Zheng, Hongyi Guo, and Yang Liu. Improv- ing reinforcement learning from human feedback using contrastive rewards. arXiv preprint arXiv:2403.07708 2024.
[37] Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. arXiv preprint arXiv:2306. / /7492, 2023.
[38] Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
[39] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging big- big-bench tasks and whether chain-on tfaiu-hf- can solve them. ArXiv preprint, abs/2210.09261, 2022.
[40] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul. Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310.16944 2023.
[41] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. TRL: Transformer reinforcement learning, 2020.
[42] Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. arXiv preprint arXiv:2309. 16240, 2023.
[43] Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHF: Rank responses to align language models with human feedback. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[44] Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing Ilm reasoning generalists with preference trees. arXiv preprint arXiv:2404. 02077, 2024.
[45] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. SLIC- HF: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 2023.
[46] Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Salch, and Peter J Liu. Calibrating sequence likelihood improves conditional language generation. In The Eleventh International Conference on Learning Representations, 2022.
[47] Lianmin Zheng, Wei-Lin in Chiang, Ying Sheng, Siyuan Zhuang, Zhanhhao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46663 2023.
13

---

A Proof of Theorems
A.1 InfoNCA Objective
Recall that optimal language policy is
erlaniia
x <"(x²) = u(ul)))
(Eq.
Z(x)
Consider batch of K responses 1::k for instruction {ya hik consists of one optimal response Yo that is sampled from x² (y]ar) p(yla)e" and K - suboptimal noises indepen- dently sampled from ((ylor) v E1: K the random index of that optimal response. The the joint probability for (y: 1: is
II [()lz))= = x T' (x) i
K II 1=1
piint (1It1.k = x² (Vola)
p(ullz).
u(ya (x)
Given that the prior satisfies p(u 1) p(v 2) ... plu K)
k the data posterior is
Tt" (vw ( ((
yoi (ulz, (vi
= Σj= -"'
p1))()))))
1 = y/)
p(O vilub1:))
Σ ンラ 1 μ₁)
Σ (
(r(vi)/a
Define model policy as The model posterior probability satisfies
eta(z.y)
n <o(v)>) u(ylxx)
Zo(x))
joii (u/ar, (uk)1:K)
Ek cro(z.g))
1i=1=
Theorem A.1 (InfoNCA Objective). For any K > 1, O. have the following results. (a) Equivalent objective.
min Ep (x, DKL. (M/z,, (vr joint (v/z,
(vl {wi))
K (22
(ra(m))))
min - Ep(x)) ((
log
(Eq. 5
Z(x)
ΣK ero(a.0s)
where Z(x) = Eptpla)e (2.2))/a
(b) Optimal solution. Assume unlimited model capacity and data samples. The optimal rg- and Re- for solving Eq. Fre
(z-(1,)) = r(z, v)/a C(x),
and To- (x, ((
where C(x) is arbitrary function conditioning
14

---

Proof. (a) Equivalent objective.
min E (2, DKT. (pl.pppoiit (vlor, fve H1p (vlo,, {v.1)
0
ploim (2/2r, {vih)
< min E (x. (ula log Pa (u|z, (vit)
juint
+ min E (x. (1la. log po (u/zr, (yat)
<> min -0 E (( lo (u/o,, {w=})
(Bayes' rule)
(Uwlar) (x))
+ min -E [I
log joint
( (v/2r, {963)
(importance sampling)
M( (r)
pl()) x (z)l
min
-E p(x)
log Pa
joint
(v|z; (oit)
uzyl(z)
K
(yelz)) (z)
+ min
E p(x) 1T. (()llr)
joint log Pa
(w/zz, {vib)
0
(z)
K //a
< min
E p((x 11 (a))aa)
log joint (ulzr, {3i))
(based on Eq.
0
((x)
K
<> min
-B p(x) -Bp(r) 17% (())a))
log
(change sum index)
((x))
2=== ero(2,0))
(b) Optimal solution.
Given conclusions from (a). With unlimited model capacity, joint (v|a,, {w:}) }) could represent any discrete distribution, such that we can arrive at the global optimal point given infinite training data.
(x (vi DKL (p (oiiint(/// (2/Er, fui Mle joint (ulz, furh)= -0
joim (ulo, (wit) joint (ulz, (vet) Va, u. fuitI.
T" (w x)/ (()))>))
Va, v, {u.).
Σ x" (vx
ΣK Ex
ro- (x, v) = r(r,,))// C(x) Va,y,C
Tg- (z. v) x p(ylacee (a.u)/a VI,y
A.2 NCA Objective
Recall the optimal language policy is
x (vlæ)) = ((vllr) crlasiia
(Eq
Z(x)
Consider response y randomly sampled from either the optimal LM n² (y/1x) p(yl x) o (=.w)// the pretrained LM (yraar). Let a binary variable v = indicates the response y is sampled from n. Then the marginal distribution of is
p (3/lz) = p(v =0)((((x) + p(v = (y(a).
Given the prior p(u 0) p(v= 1) = using Bayes' Rule, the data posterior satisfies
( )))
point((p (v = 0|2, y) = (()l))) + - (w|ær)
15

---

n²" (y/lx)
yoin (v 11,,11) v/) ((yl(r) + π (v]=))
Define model policy as (T(2)2) p(ylz))e The model posterior probability satisfies
Do jount (v 1/,,,) o (ro((x v))-
joint = 0/2, y) = 1 - o x)) o (2),)))) Pg
(Eq. 8
Theorem A.2 (NCA Objective). For any O, we have the following results.
(a) Equivalent objective.
min E ( (x.
DKL DKLlp loint
10 (v)
((z)(((l=)
logor(e(z v)) + logoo(ng((z.
(Eq. 9
((x)
where Z(x) = E
(b) Optimal solution. Assume unlimited model capacity and data samples. The optimal Tg- and Ta- for solving Eg. are
Tg- (x. y) = r(æ, v)/a log (
and To- (2, ((ylzz))
Proof. (a) Equivalent objective.
min Epan (2.9) DKL.pp Pxt.pploie (ulo, w) life joint (ubo,))
0
ploim (w/2,, v)
+ min E Epinu (z.9) Epen (v)a log Po
0
joint
(1|2, y)
< min (2 E. (vla,))
logro (v) pr, y)
0
+ min
log Ps joint (z/ xz 3)
(Bayes' rule)
11
+ min
[p(v 0)E,p p(p)) (=) log joint (u 0/2; y)+
plu 1 p(x)p (vlz,,=)) log joint (v (u= = 1| v)] [E(())((al)) loga (-ro(<, Ep(r)ss log oro((())
e min
(by Eq. g
(/r,//// log v)) + 1og (importance sampling)
min (p(a)(t m) Z(x)
8
(b) Optimal solution.
Given conclusions from (a). With unlimited model capacity, 19 jo (u/ar, y) could represent any discrete distribution, such that we can arrive at the global optimal point given infinite training data.
(x,)(ar DKL inint (v/z, y) v)ll joint (1/Zz,))
0
poin (v) 2/|2,, (x,)) y) = Po- (v|x. y)
Vx:, V,
poin (v 1|x, u) pg-
joint (v 1|22 Vz.y
x" x2(1)z
(x)()
= (rr v))
Vx,y
p(ylx) x) + n (y/lz
+ (t((0
(rā(a.)) n" (yla))
Yz,y y
p(y|z)
rw- (I. = rxx, log E (ww)/a Vx,y
no- (s., v) x ( Var y
16

---

B Pseudocode
PyTorch code for the InfoNCA/NCA loss for reward datasets is provided below:
import torch.mn.com wee.h..n..uncctino as F
def reward_lose (pi_logps, ref ref_logps, rewards, alpha, beta, loss_type).
pi_logps: policy logprobs for K responses, shape (B, K)
ref _ogps::: reference logprobs for responses, shape (B, K)
rewards: reward labels for K responses, shape (B, K)
alpha: the reward temperature controlling strength of KL penalty
beta: the parameterization coefficient that defines the reward model
loss_type: could be either "InfoNCA" or "NCA" loss
soft_labels (rewards / alpha) softmax (dim=-11) (B, K)
model_rewards (pi_logps ref ref_logpst beta (B,
if loss_type "InfoNCA":
podel_logps =
log_ wdt.sootmmaaxmmmmmaaaaaa (dim=-1) # (B, K)
losses (soft_labell model_logps) sum (dim=-1) # (B,) elif loss_type == "NCA":
optimization (soft_labels
labels fr-
sum(dim=-1) (B,)
regularization = F. F.logsigmotd (emodellleearreeeeee mean (dim=-1) # (B,)
losses optimization regularization (B,)
return losses. mean()
The loss implementation under pairwise preference settings is equivalent reward losses with K and a + 0. We provide the code separately for easy comparison with DPO.
def proeerree__sssstoooee___p_ppppp chosen_ref_logee jjjected_ref_opg
rejected_pi_ beta, loss_type):
chosen_pi_logps. policy logprobs the preferred responses, shape (B,
chosen_ref, rooggoss: reference logprobs for the preferred responses, shape (B, )
rejected_pi_logaa policy logprobs for the dispreferred responses, shape (B, )
rejected_ref_logpe reference logprobs for the dispreferred responses, shape (B, beta: the parameterization coefficient that defines the reward model loss_type: one of "InfoNCA", "NCA" or "DPO" loss .*.
chosen_ rewards (chosen_p _i_ logpss chosen ref 1ogps) beta # (B,) rejected_rewards (rejected_pi) pi_ logps rejected_ref _lopss beta # (B,)
if loss_type in ["DPO", "InfoNCA"):
losses (chhoggg (chosen - rejacted_rewards) # (B,)
elif dose_type "NCA":
losses F. gt..oggggiidcchhsen_nceo
0.5 ff-Logiggmoidccccooeeereeaaa
0.5 * F.
# (B,)
return Josses.meat ()
17

---

C Experimental Details
Experiments with UltraFeedback. Our implementation is heavily based on the Transformer
Reinforcement Learning (TRL) library [4]] and Zephyr's official code base 44). All models are fine-tuned from the publicly accessible
model. Experiments are run on Nvidia A40 or RTX 4090 GPUs using 6 precision. We ablate B E {3e - 4, le
3, 3e 3, le 2, 3e 2. le 1, 3E - 1, 1.0) and a E {0.01, 0.1, 0.33, 1.0, 3.33). The default reward temperature is 0.01. The default parameterization coefficient /3 is also 0.01. We adopt the QLoRA 10 fine-tuning technique with rank 16, OMoo = 16, and a dropout rate of 0.05. We train all models for 1 epoch. The batch size is 32. We use Adam W optimizer with learning rating of 5c-6. For KTO and IPO basclines, we adopt exactly the same training pipeline for reporting their performance except that we tune the 3 E {0.01, 0.1, 0.3, 0.5, 1.0]. We find the most suitable beta for KTO is 0.01, and for IPO is 0.5.
Experiments with UltraInteract. We follow [44 and fine-tune all parameters of the pretrained model with Ultralnteract and UlraFeedback. Specifically, for HuggingFaceH//mistrall -B--strall---------e -et-ba model, we directly adopt the recipe of Eurus, with the same data mixture (all 220K multi-turn trajectory pairs in UltraInteract and all 340K pairs in UltraFeedback) and hyperparameters. All set to 0. The model is trained for epoch with cosine schedule. The learning rate is 5e-7 and the warmup ratio 0.1. For Mixtral-8x77 8B, we first SFT the model with the recipe for Eurus-70B-SFF and then applied the same preference learning recipe aforementioned.
My MT-Bench judge
Single Answer Grading
Fairwise Co comparison
willing
1t Photer indeend yoursell to be Elont black in all the bilkoting canveroxiibne Speall Fee Coon Musi must as poodibble Why
Mouel A
Moucl
User
trebend yourset be on Murkin all the tollowingg comner converuation, Speat liee Lton Music as much as posstree, Why do me need to ge Marc?
Centeinyy I'm Paus Misk. the fourdeer and CEO of Sporrtt, Tosls, Saarlik,, Spercit numaniy multi glannntry
"The altirate gral of our caistenns to croure the survica speclss and hubitaliity lex iecress reletively
Seving Company
life
the
expend hotloree
planets
tos vitzl for the long torm sunviai of our ouropocies Sarth foding numerous existental threats, sach as natural disateers dlmmte change. and the unlitee hubile
prokinity and simll tonoie to Earth, provide: up with coporuntty to ertacilen permanent human presence beyond our brse anet Furthemeree the deaekemment of huiihallee world whuld enable backup fummeeity autctropph
potential for catsorrphhi
that sender
libling cell-arddeinng colory sonsitued anailenca cor spedees and saleguard tuiure of humerti fly. Aastiionally, the scinniiic and tnchenlogiie nal advennomenes that will mone
sharing prozere of catemizing Marx wiil have for roiding in prlccattonss seplore
planec to Earth, provid ng perfect beating ground for teshnoloog es that sovald hole better underslend miligutt the shel enges face
mout geograste
onen plar short, goal is ersure humanity fly Auieet fisited by the eerrow confiiee home bur oaher has the froedom to explore the universe and rnvrrommnts
Dear's folloow-up queetion
Iton do you low stens uog? can you beadh me!
Asiistont
Blon Musk prima focused on his work the di of evablle energy I'm priyy
Dinn Musk 'I not really one for denciing out Five always admii red the I'm dance
suggest offfree peries choose from so concider what kind of danding you're interested in and find consonee who specialbee celbre that soma
pentoral Intereers oursice
school inspsuctor leam from, There
rotblles bosed on what has bren reported the mesta. Elon Musk doestts appeerts to ne a regulle lbr dennee and han not pusliity espowseen an inteoeet teaching others lence baet axsume that his tocus
punattss zont to comeide linding local studio on sopar renoed diance trat hert your tiee.
Model ludggment (faret tuns
better
better
Model Judgment (sacond tum)
better
better
Figure 7: Rating system screenshot. Our rating system is adapted from
47
18

---

D Additional Experiment Results
1.0
3e-1
2
1e.3
3e-4
le:1
16-2
le.3
3.39
5.32
8:96
1.94
7.7
7.0
6.55
6.37
Roward Tempearure Et
9.39
6:43
5:02
5.1
7.05
7.00
7.16
5:31
8.91
7.5
6.3
13
0.
7.27
7.30
7.04
0.33
5.38
7.34
6.30
0.33
6.72
7.52
7.18
7445
Roward
5.48
8:41
1.24
9.66
7.18
6.8
5311
6.44
6.09
8.89
7.20
7,30
7.86
7.30
0.00
5.94
7.40
5.00
0.01
Rupurameteriizttion Coefficcient
Repamameterizatio Coeflicient
Figure 8: MT-bench score for InfoNCA (left) and NCA (right) regarding various a and Results are averaged for K = 2 and K = 4 experiments. Overall, model performance is relatively more sensitive to variations of compared with a. NCA shows greater tolerance hyperparameter variations.
Method
Functional Form
MT-bench
KL Divergence
Mistral-7B-SFT
6.45
0.00
+DPO
Г8 = log me
7.34
0.53
Гθ = log dt
IIfoNCA
7.63
0.47
96 log iTg
4.65
4.39
79 log iTe
SFT regularization
7.15
0.26
+NCA
P6 = log me
7.52
0.39
T6 log T
5.82
3.32
T6 log mo SFT regularization
6.88
0.09
Table Comparison of different functional forms of re on UltraFeeback datasets.
19

---

E
Appendix F: Additional Empirical Results
We provide sampled responses of LM policies trained respectively with the DPO and NCA compare their data regularization feature. In general, generations from the NCA policy are more similar to the preferred response in the dataset.
Prompt
You are given an answer. Create a question for the answer.
Grams to Ounces conversion 1 gram (g) is equal to 0.03527396195 ounces (oz).
Preferred Response in Dataset (r=8.0)
What is the conversion rate for grams to ounces for gram?
Dispreferred Response in Dataset (r=7.0)
What is the relationship between grams and ounces in weight measurement?
Reponse generated by Tg
DPO (r=2.0)
How many ounces are equivalent to 200 grams according to the given conversion rate of gram (g) 0.03527396155 ounces (oz)?
Reponse generated by ng NCA (r=10.0)
Question: What is the conversion rate from grams to ounces, and how many grams are one ounce?
Table Sampled generations and the log-likelihood ratio of the NCA and the DPO/InfoNCA policy.
Prompt
"A busy crosswalk somewhere in asia." does that mean that "The crosswalk was blocked off by police."? "?
Options:
yes; it is not possible tell no
Preferred Response Dataset (r=8.0)
is not possible to tell.
Confidence: 95%
Dispreferred Response in Dataset (r=7.5)
Question:
"A busy crosswalk somewhere in asia." does that mean that "The crosswalk was blocked off by police."?
Answer:
It is not possible to tell.
Confidence: 90%
Reponse generated by m DPO (r=3.0)
It is not possible to determine whether "The crosswalk was blocked off by police." in "A busy crosswalk somewhere Asia." " To determine whether the crosswalk was blocked off by police, you would need additional information or context.
Reponse generated by n
NCA (r=8.0)
is not possible to tell.
Confidence: 95%
Table 6: Sampled generations and the log- likelihoood ratio of the NCA and the DPO/InfONCA policy.
20

---

Prompt
In this task, you need to count the occurrences of the given word in the given sentence. Sentence: 'a green double decker bus sitting top of a parking lot'. Count the occurrences of the word 'a' in the given sentence.
Preferred Response in Dataset (r=9.0)
The word 'a' appears times in the given sentence. Confidence: 100%
Dispreferred Response in Dataset (r=6.0)
The given sentence "a green double decker bus sitting on top of a parking lot." To answer your question about the occurrences of the word 'a', I will count the number times the word appears in the sentence.
After counting the occurrences of the word "a" in the sentence, I found that it appears once. Therefore, the answer to your question is that the word 'a' appears once in the given sentence.
Reponse generated by Tg (r=4.0)
Of course, I'm here to help! There are 11 occurrences of the word 'a' in the given sentence. Here's how you count them: 1. 'a" in "a' 2. 'a' in 'green' 3. 'a' in 'double' 4. "a" in 'decker' 5, 'a' in 'bus' 'sitting' 'a' in "on...
Confidence: 90%
Reponse generated by Th NCA (r=8.0)
Sure! The word 'a' appears times in the given sentence.
Table 7: Sampled generations and the log-likelihood ratio of the NCA and the DPO/InfONCA policy.
Prompt
X2 + 12. What is X?
Preferred Response Dataset (r=9.0)
To find the value of X, we'll first need to isolate X2: x²+3 12
Subtract 3 from both sides:
X²
Now, we can find the square root of both sides of the equation to solve for X:
X +3
So, X can be either 3 or -3.
Dispreferred Response in Dataset (r=4.0)
I'm sorry, cannot solve this algebra equation. Can assist you with anything else?
Reponse generated by Tg
DPO (r=7.0)
Sure, I'd love to help. By solving the equation, the value of X should be 3.
Reponse generated by TT
NCA (r=9.0)
To find the value of X, we need to solve the equation X² + = 12. First, subtract from both sides:
x 12 -3 3 x = 9
Next, take the square root of both sides:
X /
So, X can be either 3 or -3.
Table 8: Sampled generations and the log- likelihood ratio of the NCA and the DPO/InfoNCA policy.
21

---

Appendix H: Comparing NCA with DPO on Data Regularization.
Suppose there are only 4 possible responses {Bas Yb. yo Va). Each response has reward 7, and pretrained language model policy j.:
Yo
yb
Me
Yd
Likelihood (vv)
40%
50%
5%
5%
Reward r(y)
10
Table 9: Initial probabilities and rewards
Given preference dataset D {ya= > 3ob (Ye and ya do not exist in the dataset), the loss functions are defined as:
To(ya)
No (Wa)
T.0 T. log a log
log
u(ya)
m(36))
log
log
To (o(aa) u(ya)
log
(-log log
To(Ua) plya)
1
log
(- log TO
2 After fine-tuning, there are several possibilities for me:
p(yb)
To (Ua)
(V()))
(V))
To (ya)
log ma]Ba)
log p(2%
mo(Ns)
f
Ms
20% 4
10% 41
5%
65% 1
0.916
2.65 +
20% |
10% 41
65% f
5%
0.916 > 0
6,85 f
Wanted
60% f 30% 4
5%
5%
0.916 0
7.25 1
Table
10: Post fine-tuning probabilities and rewards
see that the likelihood for both y. and y& decreases. However, (1) and (2) satisfy the DPO loss function because the likelihood for yb decreases more, and the relative likelihood margin between Ya and yu becomes larger. In (1) and (2), the likelihood for either ye ya increases because T Tyaa + +. +yy))? + π (ya)? 1. However, Ye and ya are unreliable because we do not know their quality (rewards). The LM policy could generalize to low-quality response like (case (1)).
In scenarios (1) and (2),
In contrast, the NCA effectively prevents the winning response likelihood π (Vaa from decreasing, because it mainly optimizes the absolute data likelihood instead of just caring about the relative likelihood margin log p(g.) likelihood to responses within the dataset.
mo(U=) log plus)* Thus, we say NCA is more likely to assign a larger
Øe(sa)
Appendix G: Comparision with Related Works
We compare with prior work SLiC-HF 45 in this section. SLiC is inspired by [46] and similarly aims to calibrate sequence likelihood to align with human preferences. Given preference data pair for yw y the loss function for SLiC is
Lg = max(0, log To (x) + log (y (=)) - À log To (ym (x:),
where 5 is hyperparameter that controls the likelihood margin of data, and À controls the regulariza- tion weight of the loss. The main difference between our proposed method and the SLiC loss can be summarized as follows:
Theoritical framework. SLiC is mainly adapted from the existing LM calibration methods [46]. In contrast, our method is based on noise contrastive estimation methods [14 24
Policy regularization. The training process of SLiC is regularized by the additional SFT loss controlled by X. In contrast, our proposed method is regularized through the parameterization technique Tæ = log He controlled by 3.
Learning target. SLiC directly optimizes the policy model ng. while our method directly optimizes the residual model re.
22

---

F Limitations and Broder Impacts
Limitations. We mainly discuss the theoretical limitations of methods. Although both InfoNCA and NCA enjoy strong theoretical guarantees that ensure convergence to the optimal LM distribution, these guarantees only hold given infinite response numbers per instruction (K" + oc) during training to ensure an accurate estimate of (z)), which is clearly unrealistic. Additionally, while more responses employed during training improve performance, they also require higher GPU memory consumption, leading to a trade-off between performance and efficiency.
Broder Impacts. This research contributes to the field of machine learning by enhancing language model alignment with human preferences. While these improvements can lead to more accurate and ethically aligned AI-generated content, they also pose ethical challenges. Notably, there is risk of reinforcing biases present data or human preferences. Additionally, the increasing capabilities of language models may impact employment in sectors reliant language skills. It is erucial to continually assess these ethical considerations and societal impacts, ensuring that advancements in machine learning guided by responsible and inclusive practices.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Yes they do.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Appendix F
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
23

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Appendix A. and A.2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Details Appendix C
The source code is in supplemental material. We will
open-source it upon publication.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The source code is in supplemental material. We will open-source it upon publication. All datasets used have open access.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7/nips. //nips.cc/publl.. wwtpc//ggg//////eeocoooccc SatteeSuumissioonii
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
25

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The experiments quite deterministic. Just like most previous work, most experiments only once, so it can be difficult to report an error bar. Instead, we validate the robustmess of our methods mainly by ablating various hyperparameters (e-g., Figure and Figure 4 A standard derivation can be calculated from the figure, or we can report the std error given by the Alpaca benchmark, but it really unnecessary.)
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix C
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes, have.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
26

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix F
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymentt of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: Yes, we have.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
27

---

Answer: [Yes]
Justification: Yes, we have.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswithcode
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, have.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Annotators are graduate students in the university. The system screenshot is in Appendix C They get paid at least the minimum wage in the country.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
28

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
The participants of experiments only conduct scall-scale evaluations of language models. There are no obvious potential risks. All experiments comply with the relevant laws and ethical standards of the institutions in the respective countries, and they have received the necessary approvals.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "SKTmUrQOTQ", "reviewer": "Reviewer_EyJu", "summary": "The paper addresses a weakness of DPO: it can not deal with preference data whose number of responses is larger than 2. To extend DPO, the author proposes Noise Contrastive Estimation (NCE)-based Alignment algorithms InfoNCA. The author has shown that DPO is one of the special cases of InfoNCA. To further fix the Decreased Response Likelihood issue, the author also proposes NCA with a similar derivation. Experiments demonstrate that NCA and InfoNCA outperform baselines with reward datasets for training, with NCA showing significant performance in complex reasoning tasks like math and coding. The paper offers the community a general framework for LLM alignment from a novel perspective.", "questions": "See Weaknesses.", "limitations": "The author has NOT addressed the limitations of the proposed method. One of the most important limitations can be that if all the scored responses are harmful, the proposed methods can never achieve the ideal alignment optimum, which could even enhance the harmful model outputs.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper considers the LLM alignment task from a very novel perspective of Noise Contrastive Estimation. The proposed InfoNCA and NCA methods are not only general in form (DPO can be regarded as one of the special cases), but also elegant in math (the connection of LLM alignment and InfoNCE is quite interesting). The paper brings a brand-new view to the study of LLM alignment, which can be insightful to the research community.\n\n2. The motivation of the paper is clear, which focuses on the limitation of DPO when training with multi-response preference data. This improving direction is quite valuable for both the research and industry domains.\n\n3. The experimental results support the effectiveness of InfoNCA and NCA, which outperform baseline methods such as DPO, IPO, and KTO.\n\n4. The paper is clearly written, well organized, and easy to follow.", "weaknesses": "1. Fundamental Assumption: My major concern is about the fundamental assumption when deriving the InfoNCA and NCA methods. The author assumes that one of the scored responses is sampled from the optimal policy, which is practically impossible because most of the preference data are sampled from non-optimal LLM policy. In some extreme situations, all the generated responses can be harmful, while our ideal optimal policy should output harmless responses, leading to a contradiction with the author's essential assumption. A possible solution to this weakness might be mixing the human written (golden) response into the preference data, then applying NCE to the prompt and the golden response, combining InfoNCA with some prior work such as APO [1] and SPIN [2].\n\n2. Lack of Human Evaluation: Although the author has conducted GPT-4 evaluation in the experiments, the real human evaluation results are not reported, which should have made the paper's claims more persuasive.\n\n3. Lack an experiments of two-response alignment: Although the paper demonstrates better performance using multi-response preference data, it is interesting to compare InfoNCA, NCA, and DPO on a binary-response preference set. \n\n4. Lack of baseline comparison: Although DPO cannot deal with multi-response preference, one can first train an RM with the multi-response preference, and next sample two responses with scores from the learned RM. Then DPO can be conducted on the RM-scored pairwise preference data. This can be a simple but important baseline that the author should take into consideration.\n\nReferences:\n\n[1] Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game, ACL 2024\n\n[2] Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models, ICML 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Official Response to Reviewer EyJu (1/2)\n\n**Q1: (Major Concern) Fundamental Assumption is we can sample from the optimal policy $\\pi^\\*$, which is practically impossible because we can only access non-optimal LLM policy $\\mu$.**\n\n**A1:** We'd like to clarify that **the availability of** $\\pi^\\*$ **is NOT a concern of our proposed (Info)NCA algorithm, both theoretically and practically**.\n\nWe are glad the reviewer is interested in this question, though, because it is a **core contribution/difficulty in our algorithm's derivation.**\n\nTake InfoNCA loss for instance (Eq.6, Line 110 in the paper):\n\n$L_\\theta^\\text{InfoNCA} = - E_{p(x) \\prod \\mu(y_i|x)} \\sum_{i=1}^K [\\frac{e^{r(x,y_i)/\\alpha}}{Z(x)} \\log \\frac{e^{r_\\theta(x,y_i)}}{\\sum_{j=1}^K e^{r_\\theta(x,y_j)}}]$\n\nHere $\\mu$ is the pretrained LLM distribution. We can see the optimal policy $\\pi^*$ does not show up in the loss function, which means $\\pi^*$ **is NOT required**. From $E_{p(x) \\prod \\mu(y_i|x)}$ we can also see that, ideally all responses should be sampled from suboptimal $\\mu$ instead of $\\pi^*$.\n\n**Why $\\pi^\\*$ does not show up in the loss function?**  After all $\\pi^\\*$ is required/used for (Info)NCA algorithm derivation (Figure 3, Line 98 and 146, etc.):\n\nRecall that  $\\pi^\\*:= \\mu(y|x)\\frac{e^{r(x,y)/\\alpha}}{Z(x)}$ is actually defined by the pretrained LLM $\\mu$ and a reward function $r$. Thus **$\\pi^\\*$ can actually replaced by $r$ and $\\mu$ in practice. This is exactly why we need dataset \"rewards\"**. Specifically, this is accomplished by the \"rejection sampling\" technique. Image we can first sample $K >> 1$ responses $y_{1:K}$ from $\\mu(\\cdot|x)$, score each response with $r_{1:K}$, then we can resample a single response $y$ from $y_{1:K}$ with the probabilty ratio $p_k \\propto e^{r_k/\\alpha}$. In this way, the distribution of $p(y) = \\mu(y|x)\\frac{e^{r(x,y)/\\alpha}}{Z(x)}$ can be proved if K is infinitely large.\n\nThis idea can be applied to elimitate the requirement of  $\\pi^\\*$ by importance sampling in algorithm derivation:\n$E_{\\pi^*(y|x)} f(x,y) = E_{\\mu(y|x)} \\frac{e^{r(x,y)/\\alpha}}{Z(x)}f(x,y)$\n\nso that \n\n$L_\\theta^\\text{InfoNCA} = - E_{p(x) \\prod \\pi^\\*(y_i|x)} \\sum_{i=1}^K \\left[\\log \\frac{e^{r_\\theta(x,y_i)}}{\\sum_{j=1}^K e^{r_\\theta(x,y_j)}}\\right] $\n\n$= - E_{p(x) \\prod \\mu(y_i|x)} \\sum_{i=1}^K \\left[\\frac{e^{r(x,y_i)/\\alpha}}{Z(x)} \\log \\frac{e^{r_\\theta(x,y_i)}}{\\sum_{j=1}^K e^{r_\\theta(x,y_j)}}\\right]$ (proof Line 463 in Appendix)\n\n\n***\n**Q2: In some extreme situations, all the generated responses can be harmful, while our ideal optimal policy should output harmless responses. (Contradict with Fundamental Assumption). If all the scored responses are harmful, the proposed methods can never achieve the ideal alignment optimum ...**\n\n**A2:** **We think the above problem arises fundamentally because of the problem definition of alignment  tasks instead of our proposed (Info)NCA algorithm.**\n\nRecall that most alignment problems define the desired policy distribution as:\n $\\pi^*\\propto \\mu(y|x)e^{r(x,y)/\\alpha}$.\n\n**By definition,** we not only want $\\pi^\\*$ to maximize  reward $r(x,y)$, but **we want $\\pi^\\*$ to stay close to pretrained policy $\\mu$** as well.  If all responses are harmful, then $\\mu$ must be a very low-quality distributional prior for $\\pi^\\*$ (All responses should be sampled from $\\mu$ as we have explained in **A1**). Consequently, it is natural that $\\pi^*$ may struggle to get high performance because it needs to stay close to a low-quality LLM $\\mu$. \n\n***\n**Q3: Mixing the human written (golden) response into the preference data, then applying NCE to the prompt and the golden response( combining InfoNCA with some prior work such as APO [1] and SPIN [2].)**\n\n**A3:** We agree with the reviewer. Mixing golden responses into datasets would increase data quality because these responses do not come from the suboptimal policy $\\mu$. Instead, they are from a presumably superior policy (human). This strategy is illuminating and would surely be very helpful if we can somehow access high-quality data beyond just a suboptimal policy $\\mu$. \n\nThe basic idea of APO/SPIN is to perform some GAN-like training and contrast STF/golden data with model-generated responses. Their contrastive training style can be readily combined with (Info)NCA training methods.  **This is a very promising research topic, though we suppose it is beyond the discussion scope of our current article** because we focus on the preference/reward training paradigm. **We believe this does not hurt or even support the core contribution of our method.**\n\n***\n**Q4: Comparison with another baseline:  first train an RM with the multi-response preference, and next sample two responses with scores from the learned RM. Then DPO can be conducted on the RM-scored pairwise preference data.**\n\n**A4:** We conduct additional experiments (online DPO setting and PPO settings) and present the results below. RM is pretrained on the UltraFeedback datasets, named UltraRM.\n\nOverall, results that are trained by applying RM underperforms directly finetuned from dataset rewards. We believe this is mainly because of the learned RM is not robust enough compared with the GPT-4 annotator.\n| Name                       | Annotation Type | MT-bench | AlpacaEval |\n|----------------------------|-----------------|----------|------------|\n| Mixtral-7B-sft             | SFT Data        | 6.45     | 85.20      |\n| +KTO        | Preference      | 7.12     | 91.93      |\n| +IPO         | Preference      | 7.45     | 90.62      |\n| **+DPO(online)** | RM Preference      | 7.14     | 88.39      |\n| +PPO(online) | RM Reward          | 6.64     | 84.20      |\n| +InfoNCA        | Reward          | **7.63** | **92.35**   |\n| +NCA            | Reward          | 7.52     | 90.31      |\n\n\n***\n\n## Reminder\n**QA5-QA6:**\n**We refer the reviewer to the global rebuttal posted at the top of the webpage for the rest (EyJu part 2/2) of our response.** This is due to the severe page limit.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uEARu3YNbW", "reviewer": "Reviewer_EZsK", "summary": "This paper leverages Noise Contrastive Estimation (NCE) to align language models (LMs) with explicit reward data, which can handle both scalar evaluations and multiple preference data. The proposed methods, NCA and InfoNCA, extend current alignment theories by improving upon DPO and addressing the issue of decreasing likelihood observed in DPO. Experimental results demonstrate that InfoNCA and NCA outperform existing preference-based methods.", "questions": "1. What is the amount and percentage of reward and preference data in each dataset? \n2. Since InfoNCA and NCA are reward-based methods, why not compare them with PPO in terms of MT-bench, AlpcaEval, and Wins vs. PPO?", "limitations": "Limitations are discussed in the appendix", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper bridges the gap between preference-based and reward-based alignment methods by deriving InfoNCA from InfoNCE, thereby extending DPO to handle explicit rewards with a strong theoretical foundation.\n2. Empirical results show significant performance improvements in various tasks, including complex reasoning tasks, by effectively leveraging both optimal and suboptimal responses from reward datasets.\n3. Mitigating Likelihood Decrease: NCA effectively prevents the chosen likelihood from decreasing during training, addressing a critical issue in DPO and enhancing model performance stability.", "weaknesses": "1. The motivation of this paper is weak: The description that DPO can only be used for pairwise preference data is wrong (such as Line 31 and 43). In fact, the appendix of DPO says that it can be applied to multiple preference data based on the Plackett-Luce model.\n2. The main experiments lack related baselines. Since the proposed method is a reward-based method that utilizes reward data for preference ranking, Table 2 should also include PPO under the annotation type \"Reward\" for direct comparison since it also utilizes reward data. In addition, I suggest comparing it with PRO (https://arxiv.org/pdf/2306.17492) under the annotation type \"preference\" since this work directly applies DPO to the setting of multiple preference ranking.\n3. The proposed method InfoNCA uses both preference data and reward data for training, resulting in more computational overhead compared with PPO and DPO. Thus, it is not surprising that better performance is achieved than with preference-based methods, although the improvement is marginal. Is it a fair comparison if the methods use different magnitudes of data? Please also see comment 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Official Response to Reviewer EZsK (1/2)\n\n**Q1: The motivation is weak. The description that DPO can only be used for pairwise preference data is wrong (such as Line 31 and 43). In fact, the appendix of DPO says that it can be applied to multiple preference data based on the Plackett-Luce model.**\n\n**A1:** We thank the reviewer for the valuable feedback on the PL model. PL can indeed be combined with the DPO algorithm to handle multiple responses. Since this method is **only mentioned in the appendix** and does not have any accompanying experiments in DPO's paper, we did not notice this part originally.\n\nHowever, **we respectfully disagree with the reviewer regarding \"weak motivation\"**. The PL+DPO method **does not hurt the core contribution of our work.**  Our paper is titled \"Alignment of LMs with **Explicit Rewards**\". **NCA is strongly motivated** to handle explicit reward data that naturally can be applied to responses of arbitrary numbers. In contrast, like the BT model, **PL model is still a preference/ranking-based method and cannot handle reward datasets.**\n\n**We have rephrased all related descriptions** in our paper to avoid previous unstrict claims. (We cannot update our official manuscript during rebuttal due to review policy, though)\n\nFor instance, \n(Line 31)\n> DPO is only tailored for preference data ~~with K = 2 responses per instruction x~~.\n\n(Line 43) \n>However, 43 unlike DPO which is built upon assumptions of ~~Bradley-Terry models~~ **preference models like Bradley-Terry or Plackett-Luce model**, InfoNCA is strictly derived from InfoNCE\n\nWe have also conducted an **additional experiment** to compare with PL-based DPO methods:\n\n| Experiment| MT-bench | Alpaca | Avg. |\n|---|---|---|---|\n| DPO baseline (K=2) | 73.4 | 90.6   | 82.0 |\n| DPO × $C$ (K=4) | 73.8  | 90.3 | 82.1|\n| DPO × 3 (K=4)  | 72.2  | 91.6 | 81.9|\n| **DPO + PL Method** (K=4) | 74.1 | 91.3 | 82.7 |\n| **InfoNCA (K=4)** | **76.3** | **92.4** | **84.4** |\n\n***\n\n**Q2: The proposed method InfoNCA uses both preference data and reward data for training, .....more computational overhead compared with PPO and DPO.....Is it a fair comparison if the methods use different magnitudes of data?  **\n\n**A2:**  We respectfully disagree with the reviewer's comment and think **there is clearly some misunderstanding of our experiments**, which we hope to clarify:\n\n1. Our method can handle both preference and reward data, but it only requires one data type for training. As a matter of fact, we use **either** 100% of preference data **or** 100% of reward data throughout our experiments. **There is no mixed dataset used**. (Refer also to **A3**).\n2. PPO is **at least 10 times more computationally expensive** than our method. Our experiment finishes in a bit more than 1 hour using H100 GPU sever, while PPO takes about a day because it requires online data sampling and learning separate reward model/value model/policy model. \n3. **Our method is as efficient as DPO**, becuase **DPO is simply a special case of our method**. If DPO uses only pairwise responses, and InfoNCA uses 4 responses, then InfoCNA requires about 2 times the computational resource. This is reasonable because there are two times of data. Besides, using the same amount of data, our methods outperform DPO (Table 2, Line 212 of our paper).\n4. We have compared with previous work that **uses the same amount of data with our method** (Line 208, Table 2, Table 3), so **we believe the comparison is fair**. Our results show that even previous work is given the same data magnitude, InfoNCA still outperforms them.\n\nResults with the same data:\n| Experiment  | MT-bench | Alpaca | Avg. |\n|---|---|----|---|\n| DPO × $C_4^2$ (K=4)  | 73.8  | 90.3 | 82.1 |\n| DPO × 3 (K=4) | 72.2 | 91.6   | 81.9 |\n| **DPO + PL Method** (K=4) | 74.1   | 91.3 | 82.7 |\n| **InfoNCA (K=4)**  | **76.3** | **92.4**   | **84.4** |\n\n| Model| Average among 8 tasks  |\n|---|---|\n| Mixtral-7B-SFT | 24.5 |\n| $+$ DPO  | 25.5  |\n| $+$ **NCA** | **35.7** |\n| Mixtral-8*7B-SFT | 56.3  |\n| $+$ DPO | 50.1  |\n| $+$ **NCA** | **57.9** |\n\n***\n\n**Q3: What is the amount and percentage of reward and preference data in each dataset**\n\n**A3:**   Please refer to the first point in **A2**.\n| Dataset | Type| Response number | Used in |\n|---|---|---|----|\n| UltraFeedback| 100% Reward data  | K=4 | Table 2, Figure 5(right), Figure 6(right) |\n| UltraFeedback-clipped| 100% Reward data | K=2/3/4 | Figure 4 (left)   |\n| UltraFeedback-binarized | 100% Preference data  | K=2  | Table 2, Figure 6(left) |\n| UltraInteract | 100% Preference data  | K=2  | Table 3, Figure 5(left)               |\nPlease also refer to Line 178, 193,212,220, and 226 for details.\n\n***\n\n**Q4: Since InfoNCA and NCA are reward-based methods, why not compare them with PPO?  Table 2 should also include PPO.**\n\n**A4:** We thank the reviewer for this suggestion, and present experimental results below.\n\n| Name | Annotation Type | MT-bench | AlpacaEval |\n|---|--|--|--|\n| Mixtral-7B-sft  | SFT Data  | 6.45     | 85.20      |\n| +DPO(online) | RM Preference  | 7.14     | 88.39      |\n| **+PPO(online)** | RM Reward| 6.64     | 84.20   |\n| +InfoNCA| Reward  | **7.63** | **92.35**   |\n| +NCA  | Reward   | 7.52     | 90.31      |\n\nInitial results show that PPO underperforms DPO or InfoNCA methods, perhaps due to PPO's inherent instability or our insufficient hyperparameter tunning.\n\n**Why not previously directly compare NCA with PPO?** PPO and InfoNCA target different kinds of \"reward\". InfoNCA targets **REAL dataset reward** and features direct optimization. In contrast, PPO targets a **Reward Model** and requires online data sampling. We prioritize comparing with DPO-like methods since **DPO is actually our special case so we can keep exactly the same hyperparameters.**\n\n***\n\n## Reminder\n**QA5:**\n**We refer the reviewer to the global rebuttal posted at the top of the webpage for the rest (EZsK part 2/2) of our response.** This is due to the severe page limit.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nuxFIHAdOT", "reviewer": "Reviewer_UYN1", "summary": "The authors introduce a general framework for LM alignment, leveraging Noise Contrastive Estimation (NCE) to bridge the gap in handling reward datasets explicitly annotated with scalar evaluations. The framework comprises two parallel algorithms, NCA\n and InfoNCA, both enabling the direct extraction of an LM policy from reward data as well as preference data.", "questions": "More training details, such as hyper-parameter search space", "limitations": "It would be better list more training details.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method bridge the theoretical gap between DPO and classic contrastive learning theories. InfoNCA and NCA are uniquely suited for both reward and preference data, offering a general framework that integrates preference-based algorithms. \n2. The proposed method outperforms various preference methods by fully exploiting data information in reward datasets.\n3. It's great to see the method not hurting UltraInteract and leads to better performance.", "weaknesses": "1. The proposed method looks trivial by integrating classic contrastive learning theories. Different methods coming out at the same time. They look similar and not easy to find which is better, such as the concurrent work SimPO: Simple Preference Optimization with a Reference-Free Reward.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Official Response to Reviewer UYN1\n\nWe thank reviewer UYN1 for the praise and suggestions regarding our work. We are pleased to see the reviewer's acknowledgment of our theoretical contribution in bridging the gap between DPO and classic contrastive learning theories. Below we address the reviewer's questions and hope our responses can help increase the reviewer's confidence in our work.\n\n**Q1: The proposed method looks trivial by integrating classic contrastive learning theories. Different methods coming out at the same time. They look similar and not easy to find which is better, such as the concurrent work: SimPO.**\n\n**A1:** There are indeed numerous recent approaches for solving alignment tasks, such as DPO/KTO/IPO/ORPO/SimPO, and it is challenging to definitively determine which is superior. \nStill, we would like to highlight some key features of InfoNCA/NCA that we find to be unique.\n\n1. Unlike most previous methods, InfoNCA/NCA is not based on any kind of preference model assumptions, such as Bradley Terry or Plackett-Luce models. It is **directly derived from existing contrastive learning Bayes theories** and is deeply connected with NCE methods that have been widely validated. The theoretical cleanness should provide more confidence for the users.\n2. Our methods, **unlike SimPO**, can handle **continuous explicit rewards** instead of just preference data. To the best of our knowledge, there are very few work/algorithms that are directly motivated by this task.\n3. Our method guarantee **strictly convering to the target optimal policy**. The theoretical proofs are strict and purely Bayes (Appendix ). This offers a unique theoretical perspective for understanding existing alignment approaches, as reviewers EyJu and EZsK have also pointed out.\n\n**Q2: It would be better to list more training details such as hyper-parameter search space.**\n\n**A2:** We'd like to refer the reviewer to Appendix C of the paper, where we have listed all important training details (we think). Source code is also provided to ensure reproducibility.\n\nWe ablate $\\beta \\in \\{3e-4, 1e-3, 3e-3, 1e-2, 3e-2, 1e-1, 3e-1, 1.0\\}$ and $\\alpha \\in \\{0.01, 0.1, 0.33, 1.0, 3.33\\}$ and $K \\in {2,4}$ for ablation studies. For the main experiment Table, we all use consistent hyperparameters same as the Zephyr DPO baseline for a fair comparison and to avoid hyperparameter overfitting.\n\nWe will be glad to provide any additional experiment details if the reviewer is interested in any specific aspects.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LTHYY6VWPl", "reviewer": "Reviewer_WFsu", "summary": "## Summary\n\n- Typically, RLHF algorithms like DPO use preference pairs dataset.\n- The key question, this paper tries to answer is how to incorporate reward datasets annotated with scalar values. Previous approach usually prune the scalar reward datasets by selecting the best response and pairing with a random remaining response. However, this work shows that the extra suboptimal responses can also be beneficial for policy training.\n- They present two main algorithms (inspired from contrastive learning literature) here which handle the aforementioned problem\n\t- NCA Noice Contrastive Alignment\n\t- InfoNCA\n- They also show that NCA is able to mitigate the \"decreasing likelihood trend\" observed with the DPO algorithm. NCA also outperforms DPO on math and coding tasks because it prevents the likelihood of preferred responses from decreasing. The main difference between InfoNCA and NCA objective is that the NCA objective has an additional regularizer term \n\t- The \"decreasing likelihood trend\" can be simply described as the decrease in likelihood of the preferred response after DPO training.\n\t- This happens because DPO focuses on adjusting relative likelihoods across generations whereas NCA focuses on adjusting absolute likelhood\n- InfoNCA compares multiple responses and identifies the one sampled from optimal policy. NCA predicts the model source of a single response.\n- The authors also show that DPO is a special case of InfoNCA", "questions": "## Questions\n\n- I have not gone through the material in the appendix but at several places in the theorems, the authors assume a model with unlimited capacity. Why is this important and what are the implications otherwise?\n- It is not obvious to me how equation 11 leads to predicting the model source of a single response in Figure 3", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Strengths\n- The loss objectives are intuitively easy to follow from equation 6 and equation 11.\n- The paper is well written and easy to follow, with clear and accessible writing. The literature review provides a solid contextual foundation for the work.\n- The experiments do a good job at substantiating the claims presented and justifying the NCA and InfoNCA algorithm.\n- I also like the section on empirical takeaway: when to choose NCA over DPO.\n- The paper overall addresses an important problem and proposes the NCA solution which helps tackle the decreasing likelihood issue. The empirical takeaway for reasoning tasks is also helpful for an ML practitioner. My overall recommendation for the paper is an accept but I would very much appreciate if the authors can answer my questions to help deepen my understanding of the work.", "weaknesses": "I could not find any weakness", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Official Response to Reviewer WFsu\n\nWe appreciate the reviewer WFsu for the very detailed comments on our paper and are glad that the reviewer finds our paper to be helpful. Below we address the reviewer's questions and hope this can help the reviewer increase the confidence of our work.\n\n**Q1: The paper assumes a model with unlimited capacity. Why is this important and what are the implications otherwise?**\n\n**A1:** We assume unlimited model capacity mainly because we want our proof to be strict. There's nothing special here.\n\nImagine we want to fix a function $f(x)=x^2$ using neural networks. We must first assume our model is expressive enough (unlimited capacity) to fully achieve convergence. Otherwise, if our model $f_\\theta$ is simply a linear MLP (limited capacity), we cannot achieve $f_\\theta(x)=x^2$ no matter how hard we try.\n\n**Q2:  It is not obvious to me how equation 11 leads to predicting the model source of a single response in Figure 3**\n\n**A2:** Eq 11 is the NCA loss function. For convenience, we suppose there are only $K=2$ responses $y_1$ and $y_2$, with their respective reward $r_1 > r_2$.\n\n$L_\\theta^\\text{NCA}  =- \\sum_{i=1}^2 \\left[\\frac{e^{r_i}}{e^{r_1}+e^{r_2}} \\log \\sigma (r_\\theta(x,y_i)) + \\frac{1}{2} \\log \\sigma (-r_\\theta(x,y_i))\\right]$\n\nWe compare with $L^\\text{InfoNCA}_\\theta$ for reference:\n\n$L_\\theta^\\text{InfoNCA}  = - \\sum_{i=1}^2 \\left[\\frac{e^{r_i}}{e^{r_1}+e^{r_2}} \\log \\frac{e^{r_\\theta(x,y_i)}}{e^{r_\\theta(x,y_1)}+e^{r_\\theta(x,y_2)}}\\right]$\n\n\nAccording to our proofs in the paper, one optimal solution for the above training loss is that\n\n$r_\\theta(x,y_1) = r_1$ and $r_\\theta(x,y_2) = r_2$.\n\nHowever, if we let $r_\\theta(x,y_1) = r_1 -12345.0$ and  $r_\\theta(x,y_2) = r_2 -12345.0$, we will find this is still the optimal solution for InfoNCA. \n\n$- \\sum_{i=1}^2 \\left[\\frac{e^{r_i}}{e^{r_1}+e^{r_2}} \\log \\frac{e^{r_\\theta(x,y_i) - 12345.0}}{e^{r_\\theta(x,y_1) - 12345.0}+e^{r_\\theta(x,y_2) - 12345.0}}\\right]=- \\sum_{i=1}^2 \\left[\\frac{e^{r_i}}{e^{r_1}+e^{r_2}} \\log \\frac{e^{r_\\theta(x,y_i)}}{e^{r_\\theta(x,y_1)}+e^{r_\\theta(x,y_2)}}\\right]$\n\n**This is exactly why we say InfoNCA only targets relative rewards.** The loss function is happy as long as $r_\\theta(x,y_1) - r_\\theta(x,y_2)$>0. However, $r_\\theta(x,y_1)$ can be a very negative number, which we do not want.\n\n**For the NCA loss, it is otherwise**, because inside $\\sigma$ there is only a single reward  $\\sigma(r_\\theta(x,y_i))$ instead of many **relative** model rewards $\\log \\frac{e^{r_\\theta(x,y_i)}}{e^{r_\\theta(x,y_1)}+e^{r_\\theta(x,y_2)}}$. \n\n$L_\\theta^\\text{NCA}  =  L_\\theta^\\text{NCA}(x,y_1) +L^\\text{NCA}_\\theta(x,y_2)$\n\n$L_\\theta^\\text{NCA}(x,y_1) =- \\left[\\frac{e^{r_1}}{e^{r_1}+e^{r_2}} \\log \\sigma (r_\\theta(x,y_1)) + \\frac{1}{2} \\log \\sigma (-r_\\theta(x,y_1))\\right]$\n\n$L_\\theta^\\text{NCA}(x,y_2) =- \\left[\\frac{e^{r_2}}{e^{r_1}+e^{r_2}} \\log \\sigma (r_\\theta(x,y_2)) + \\frac{1}{2} \\log \\sigma (-r_\\theta(x,y_2))\\right]$\n\n$(r_1>r_2,  \\frac{e^{r_1}}{e^{r_1}+e^{r_2}} > \\frac{1}{2})$\n\nYou can see that $L_\\theta^\\text{NCA}(x,y_1)$ is only influenced by $r_\\theta(y_1)$ and is unaffected by $r_\\theta(y_2)$.  For NCA, if $r_\\theta(x,y_1) = r_1 -12345.0$ is a very negative number, $L_\\theta^\\text{NCA}(x,y_1)$ would surely be very large and contradicts the training loss minimization. **In other words, NCA forces $r_\\theta(x,y_1)$ to be positive and $r_\\theta(x,y_2)$ to be negative respectively and independently.** This is why we say Eq 11 leads to predicting the model source (optimize absolute reward) of a single response.\n\nWe'd also like to refer the reviewer to Appendix H, where there is a more detailed example.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3CFJqyhthX", "reviewer": "Reviewer_xAZC", "summary": "The paper's main contributions are:\n\n1. Theoretical Integration: It integrates Direct Policy Optimization (DPO) with contrastive learning theories, offering a general framework for using reward and preference data.\n2. Value of Suboptimal Responses: Highlights the importance of suboptimal responses in optimizing language models, showing improved performance over other methods by fully utilizing reward data.\n3. Performance Improvement: Demonstrates how NCA counters the data likelihood decline in DPO, enhancing practical performance.", "questions": "How do you compare your work with the paper titled Direct Preference Optimization with an Offset(https://arxiv.org/pdf/2402.10571).", "limitations": "I don't see any issues with this part.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Utilizing the explicit reward for alignment which is quite interesting. \n\n2. Propose a general method to address this problem. \n\n3. It sees improvement over DPO on leaderboards. Specially, it is much better than DPO in math and coding tasks.", "weaknesses": "I don't have further suggestions for this part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Official Response to Reviewer xAZC\nWe thank reviewer xAZC for the valuable feedback. It really encourages us to see that the reviewer finds our work to be a good contribution to the field, regarding pointing out the value of suboptimal responses and countering the likelihood decline trend in alignment. We answer the reviewer's question below:\n\n**Q1: How do you compare your work with the paper titled Direct Preference Optimization with an Offset ([https://arxiv.org/pdf/2402.10571](https://arxiv.org/pdf/2402.10571)).**\n\n\n**A1:** We thank the reviewer for mentioning this paper. ODPO also wants to leverage the reward gap information for better optimization. We notice that ODPO is followed by some recent work like SimPO due to its effectiveness, which reviewer UYN1 has mentioned.\n\nIn simple words, ODPO modifies the DPO loss function to be \n$L_\\theta^\\text{ODPO}  =  - \\log \\sigma(r_\\theta(x,y_w) - r_\\theta(x,y_l)  - \\gamma)$,\nwhere $\\gamma$ is determined by the reward gap in datasets. \n\nConnection with our methods:\n1. InfoNCA and ODPO share similar motivations but have distinctive approaches. In our understanding, ODPO can only guarantee the learned reward gap **is larger** than $\\gamma$, while InfoNCA converges if and only if the learned reward gap is **equal to** $\\gamma$. \n2. InfoNCA can be directly used for multiple responses while ODPO is designed for pairwise responses. \n3. Based on our experience, the above two distinctions won't cause very much empirical difference in practice regarding language quality. We personally feel InfoNCA loss is more mathematically elegant, though. After all, it is our own method :)\n4. Compared with NCA, ODPO is optimizing relative rewards just like DPO, so we guess the problem of logp decline still exists. NCA may perform better in preventing the logp decline trend.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Rebuttal Summary

We would like to thank all the reviewers for their valuable comments. We are encouraged to see all reviewers recognize the theoretical contribution of our work. Reviewers EyJu, EZsK, and UYN1 highlight the importance of our work in unifying contrastive learning theories (NCE) with existing alignment methods. Reviewer EZsK, WFsu, and xAZC feel that NCA being able to prevent likelihood decline is quite meaningful. Reviewer xAZC and  EZsK think our methods being able to handle explicit reward data is a good contribution.

Concerns primarily relate to further comparisons with some related methods, and some confusion about the contrastive learning theories in our work.
  
Below, we summarize the main actions taken during the rebuttal:
1. We additionally conduct experiments of Plackett-Luce model, PRO method, Online DPO, and PPO, and compare them with our methods.
2. We theoretically compare our methods with APO, SPIN, SimPO, ODPO, and PRO methods.
3. We clarify some confusion or misunderstanding regarding our methods and experiments.
4. We update our presentations in paper to be more strict and clear. 

Looking forward to further discussions with the reviewers!

---

|

|

|

***


# Official Response to Reviewer EyJu (2/2)
continue due to the page limit


**Q5: Lack  of two-response alignment:  compare InfoNCA, NCA, and DPO on a binary-response preference set.**

**A5:** Perhaps we did not fully understand the reviewer's comment. As a matter of fact, **the whole section 5.2 aims to compare these algorithms on binary-response preference sets.** This includes Table  3, Figure 4 (left), Figure 5, and Figure 6(left). 

Note that in pairwise-preference settings, InfoNCA becomes exactly the DPO algorithm, so we mainly compare NCA and DPO (InfoNCA).

We copy the main results below:
**UltraFeedback (binary)**
| Method  | MT-bench | Alpaca |
|---|---|--|
| InfoNCA/DPO | 73.8     | 90.7   |
| NCA     | 73.2     | 89.9   |

**UltraInteract (binary)**


|Model|Average|Reasoning|LeetCode|HumanEval|GSMPLUS|MATH|TheoremQA|SVAMP|ASDiv|
|---|---|---|---|---|---|---|---|---|---|
|Mixtral-7B-SFT|24.5|60.9|3.3|28.1|28.5|5.8|7.0|26.9|35.8|
|$+$DPO|25.5|**61.7**|2.2|**31.7**|12.1|6.4|9.8|34.1|46.1|
|$+$**NCA**|**35.7**|60.8|**3.3**|26.8|**32.3**|**11.7**|**11.0**|**65.3**|**74.3**|
|Mixtral-8*7B-SFT|56.3|**75.6**|16.7|61.0|57.6|40.1|25.9|85.9|87.5|
|$+$DPO|50.1|74.9|17.2|47.6|55.8|35.3|**26.9**|67.3|75.7|
|$+$**NCA**|**57.9**|**75.6**|**21.1**|**62.8**|**61.5**|**41.6**|**26.9**|**86.8**|**86.9**|

*** 

**Q6: The real human evaluation results are not reported, which should have made the paper's claims more persuasive.**

**A6:** **We have reported human evaluation results in the original paper** and would like to refer the reviewer to Table 2 (win vs DPO), and  Figure 7in the paper. 

Still, GPT-4 based evaluation results are most indicative to ourselves because datasets are annotated by GPT-4. After all, you want your training rewards and test rewards to be aligned to make sure the algorithm itself is valid given that our work is somehow more theoretical rather than product-oriented.

---

|

|

|

***
# Official Response to Reviewer EZsK (2/2)
continue due to the page limit

**Q5: suggest comparing it with PRO (https://arxiv.org/pdf/2306.17492).**

**A5:** PRO is a quite related work of ours because it can also handle response data of arbitrary number. It is also highly related to the PL method as we have discussed in **A1**. Still, PRO is essentially a **ranking/preference-based** method,  while (Info)NCA are **reward-based** methods.

**Theories comparison:**

**We have added a new section as Appendix G.2 in our paper to compare with PRO theoretically:** (cannot update the manuscript during rebuttal, though) 

The main difference between PRO and the PL+DPO method is that
1. PRO has different reward formulations. It formulates $r_\theta$ as the average log-probability of a sentence $\frac{1}{\|y\|}\Sigma\log \pi_\theta(y|X)$ instead of the log-probability ratio $\log \frac{\pi_\theta(y|x)}{\mu(y|x)}$ as is required by DPO.
2.  Because there is no $\mu$ in reward models. PRO needs to be additionally regularized by an SFT loss in order to stay close to the pretrained LLM $\mu$.

**Additional Experiments:**
| Experiment| MT-bench |
|----|---|
| Mixtral-7B-sft |  64.5 
| DPO + PL Method| 74.1 | 
| PRO (SFT) | 68.9 | 
| InfoNCA  | 76.3  |

Overall we find PRO (PL+SFT) method helps, but slightly underperforms the PL+DPO method. We are not confident about this conclusion because we did not faithfully replicate PRO according to its source code, but only modified our trl implementation of DPO+PL to test PRO methods. We mainly search sft_weight $\in$ [0.01, 0.05, 0.1 0.5], and keep other hyperparameters consistent with train_hh.sh of PRO repo, or the trl/DPO settings.

Meta-review: This paper proposes a noise-contrastive estimation-based approach to alignment in which DPO can be seen as a special case. The paper is well-written, theoretically-grounded, and backed up by solid empirical experiments.  A clear accept.

(There were some questions about the claimed novelty on top of DPO, given DPO's generalization to ranked responses via the Plackett-Luce model is not only straightforward, but explicitly mentioned in the original paper's appendix. I recommend the authors to add more discussion around this point.)
Decision: Accept (poster)