Noise Contrastive Alignment of Language Models with Explicit Rewards

Authors: huayu  chen
guande  he
lifan  yuan
ganqu  cui
hang  su
jun  zhu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 06:26:52

Abstract: User intentions are typically formalized as evaluation rewards to be maximized when fine-tuning language models (LMs). Existing alignment methods, such as Direct Preference Optimization (DPO), are mainly tailored for pairwise preference data where rewards are implicitly defined rather than explicitly given. In this paper, we introduce a general framework for LM alignment, leveraging Noise Contrastive Estimation (NCE) to bridge the gap in handling reward datasets explicitly annotated with scalar evaluations. Our framework comprises two parallel algorithms, NCA and InfoNCA, both enabling the direct extraction of an LM policy from reward data as well as preference data. Notably, we show that the DPO loss is a special case of our proposed InfoNCA objective under pairwise preference settings, thereby integrating and extending current alignment theories. By comparing NCA and InfoNCA, we demonstrate that the well-observed decreasing-likelihood trend of DPO/InfoNCA is caused by their focus on adjusting relative likelihood across different responses.
In contrast, NCA optimizes the absolute likelihood for each response, thereby effectively preventing the chosen likelihood from decreasing. We evaluate our methods in both reward and preference settings with Mistral-8$\times$7B and 7B models. Experiments suggest that InfoNCA/NCA surpasses various preference baselines when reward datasets are available. We also find NCA significantly outperforms DPO in complex reasoning tasks like math and coding.

Main paper: 
 introduction: 
 Aligning pretrained Language Models (LMs) with scalar rewards that reflect human intentions is crucial for enhancing their ability to follow instructions [35, 25] . These rewards can be given either explicitly or implicitly. Explicit rewards can be scalar ratings of human annotators or advanced models like GPT-4, while implicit rewards are usually preference labels assigned to pairwise responses.One effective approach for aligning LMs with preference data is Direct Preference Optimization (DPO, [33] ). DPO applies a reward training loss but parameterizes the reward model as the response likelihood ratio between two LMs, allowing for training reward models and extracting LM policies simultaneously. This approach is more streamlined and thus more favorable compared with traditional Reinforcement Learning (RL) methods [26] , which typically require a two-stage training process: first training reward models, then extracting LM policies.Despite its simplicity and effectiveness, DPO is only tailored for preference data (x → {y w > y l }). When multiple responses are available, directly assigning a scalar reward to each response is usually more convenient and efficient than comparing them in a pairwise manner. The resulting reward datasets (x → {y i , r i } 1:K ), however, cannot be directly leveraged for DPO training. Previous work [40] usually prunes reward datasets by selecting the best response and pairing it with a random remaining one. This is suboptimal as all reward values and additional dispreferred responses are thrown away in its data-preprocessing process.To address this issue, we present InfoNCA, an alignment method that allows directly extracting LM policies from both reward datasets and preference datasets with arbitrary response numbers (Figure 1 ). Notably, InfoNCA subsumes DPO loss as a special case under pairwise preference settings and can thus be seen as a natural extension of DPO (Sec. 3.2) . With strong theoretical guarantees, we show DPO is a binary classification loss while InfoNCA is its multi-category version (Figure 3 ). However, unlike DPO which is built upon assumptions of Bradley-Terry models or Plackett-Luce models, InfoNCA is strictly derived from Information Noise Contrastive Estimation (InfoNCE, [24] ), an established contrastive method that is widely applied in language and visual representation learning [31] . This closes the theoretical gap between current preference alignment methods and classic contrastive learning frameworks.A well-observed problem with DPO is that the likelihood of the preferred response tends to decrease throughout training [27, 32] . Similar trends are shared by InfoNCA. We find this issue arises mainly from InfoNCA/DPO's focus on adjusting the relative likelihood across different responses per instruction. In Sec. 4, we propose NCA as an alternative alignment method to InfoNCA to mitigate this problem. NCA differs from InfoNCA by only loss definition and is also suitable for both preference and reward datasets. However, NCA is built on NCE [14] , a parallel contrastive learning algorithm to InfoNCE, which optimizes the absolute data likelihood during training. In practice, NCA effectively prevents the chosen likelihood from decreasing (Figure 2 ). We evaluate our methods on Mistral-7B and 8×7B models from two dimensions. When reward datasets [9] are available, we show that directly applying our reward-based alignment offers clear improvement compared with preference-based algorithms, achieving higher evaluation rewards in GPT-4 [47, 19] evaluations. We further validate this improvement comes from InfoNCA/NCA's ability to fully leverage the additional suboptimal responses. When only preference data is given [44] , we compare pairwise NCA against the DPO loss. Our experimental results spanning various benchmarks show that NCA outperforms DPO in complex reasoning tasks such as math and coding.Our main contributions: 1. We bridge the theoretical gap between DPO and classic contrastive learning theories. InfoNCA and NCA are uniquely suited for both reward and preference data, offering a general framework that integrates preference-based algorithms. 2. We show that suboptimal responses are also important for LM optimization. Our method outperforms various preference methods by fully exploiting data information in reward datasets. 3. NCA effectively mitigates the data likelihood decline issue of DPO and offers practical performance improvement. 
background: direct preference optimization: 
 LM alignment is essentially a constrained policy optimization problem:EQUATIONwhere µ represents the pretrained LM. x and y are respectively instructions and responses. r is a reward function that reflects human intentions. α is some temperature coefficient. Prior work [30, 29] has proved that the optimal solution for the optimization problem in Eq. 1 is:EQUATIONDirect Preference Optimization (DPO) [33] assumes we only have access to some pairwise preference data x → {y w > y l } for each instruction x. The preference probability of human annotators is modeled by a learnable implicit reward model r θ under Bradley-Terry theories [2] :π θ (y w > y l |x) := σ(r θ (y w , x) -r θ (y l , x)),where σ is the sigmoid function. To learn r θ , DPO simply adopts a binary classification loss:EQUATIONIn practice, the latent function r θ is parameterized by the log-likelihood ratio between π θ and µ:r θ (x, y) := β log π θ (y|x) µ(y|x) ,where β a linear coefficient for scaling r θ . This parameterization is crucial because it ensures π θ (y|x) ∝ µ(y|x)e r θ (x,y)/β constantly hold. It transforms generative policy optimization into a simple discriminative classification task: When r θ = r and β = α are satisfied, we naturally haveπ θ = π * .3 InfoNCA: Extending DPO from Preference to Explicit RewardsCompared with constructing preference datasets, annotating each response with scalar rewards can be more flexible and convenient. Preference methods are only suitable for pairwise data (x → {y w > y l }) and would require C 2 K evaluations for comparing K responses. In contrast, reward datasets (x → {y i , r i } 1:K ) allow an arbitrary number of responses per prompt with K evaluations. Despite its simplicity in handling preference data, DPO is not tailored for reward datasets. We introduce a new alignment method termed InfoNCA to mitigate this gap. We first strictly derive InfoNCA in Sec. 3.1. We show that reward alignment can be solved by constructing a classification problem to identify the optimal response from multiple candidates. We then demonstrate that InfoNCA subsumes DPO as a special case and thus is a natural extension of DPO (Sec. 3.2). 
reward alignment through multi-class classification: 
 In essence, DPO represents response rewards as LM likelihoods and constructs a binary classification task for learning the reward model. Given that there are more than two (K > 2) responses per prompt in reward datasets, we seek to construct a multi-class classification task for learning reward models from explicit rewards instead of preference labels. We begin by formally defining this task: Consider a batch of K responses {y i } 1:K for an instruction x. {y i } 1:K consists of one optimal response y ν that is sampled from π * (y|x) ∝ µ(y|x)e r(x,y)/α , and K -1 suboptimal noises independently sampled from µ(y|x). ν ∈ 1 : K is the random index of that optimal response. Our goal is to identify which of the K candidates is y ν , given only reward labels r(y i ) for each response.Intuitively, the response with higher rewards should have a higher probability of being the target response. This intuition can be more rigorously expressed: Proposition 3.1 (proof in Appendix A.1). Given the above K response candidates and their respective rewards, the posterior probability for the ν-th response being drawn from π * is p(ν|x,EQUATION)This finding is highly appealing because it shows response optimality is solely related to response rewards. This allows us to conveniently train reward models r θ via maximum likelihood (MLE). Theorem 3.2 (InfoNCA, proof in Appendix A.1). We define π * (y|x) ∝ µ(y|x)e r(x,y)/α and π θ (y|x) ∝ µ(y|x)e r θ (x,y) . For any K > 1, α > 0, we have:(a) Equivalent objective. The MLE objective for training r θ has an equivalent form:EQUATIONwhere Z(x) = E µ(y|x) e r(x,y)/α .(b) Optimal solution. Assume unlimited model capacity. The optimal r θ * and π θ * for solving (5) arer θ * (x, y) = r(x, y)/α + C(x),and π θ * (y|x) = π * (y|x) ∝ µ(y|x)e r(x,y)/α , where C(x) is an arbitrary function conditioning on x.In practical implementation of Eq. 5, we estimate Z(x) ≈ 1 K e rj /α and parameterize r θ (x, y) := β log π θ (y|x) µ(y|x) following DPO (Eq. 2). The loss function becomes .L InfoNCA θ (x, {y i , r i } 1:K ) = - K i=1 e ri/The loss function in Eq. 6 is termed InfoNCA, where A stands for Alignment. This naming reflects its functional similarity to the Infomation Noise Contrastive Estimation (InfoNCE, [24] ). Both methods transform generative modeling problems into classification tasks by contrasting multiple data points.How does InfoNCA work? InfoNCA loss (Eq. 6) can be seen as a K-category cross-entropy loss.The soft label is calculated by dataset rewards through a softmax operation. The model's predictions are represented by learned reward r θ . The loss reaches 0 when r θ * (x, y) = r(x, y)/α + C(x).Below we show that DPO is a special case of InfoNCA asymptotically. Specifically, setting response number K = 2 and reward temperature α → 0, we can fully recover the DPO objective:L InfoNCA θ (x, {y i , r i } 1:K ) = - K i=1 1(r i = r max ) log e r θ (x,yi) K j=1 e r θ (x,yj ) (α → 0)= -log e r θ (x,yw) e r θ (x,yw) + e r θ (x,y l ) (K = 2, suppose r w > r l )= -log σ(r θ (x, y w ) -r θ (x, y l )) (DPO loss, Eq. 3)Alignment MethodInfoNCA (Sec. 3) NCA (Sec. 4)Modeling Target π * (y|x) ∝ µ(y|x)e r(x,y)/αModel Definition π θ (y|x) ∝ µ(y|x)e rθ(x,y) π θ (y|x) = µ(y|x)e rθ(x,y)x → {y i , r i } 1:K Loss (K>1,α>0) -K i=1 e r i /α j e r j /α log e r θ (x,y i ) j e r θ (x,y j ) -K i=1 e r i /α j e r j /α log σ(r θ (x, y i )) + 1 K log σ(-r θ (x, y i ))Preference Dataset x → {y w > y l } Loss (K=2, α→0) -log σ(r θ (x, y w ) -r θ (x, y l ))(DPO) -log σ(r θ (x, y w )) -1 2 y∈{yw,yl} log σ(-r θ (x, y))Loss Type InfoNCE loss [24] NCE loss [14] Optimizing Target relative value of log likelihood ratio absolute value of log likelihood ratio Optimal r θ * (x, y) r(x, y)/α + C(x) r(x, y)/α -log E µ(y|x) e r(x,y)/α r θ * (x, y best ) ≥ 0 ? not guaranteed Table 1 : Comparison of NCA and InfoNCA algorithm for aligning language models. Both reward loss and pairwise preference loss are given. We provide pseudocode in Appendix B.Empirical effect for varying hyperparameter K and α. As indicated by the derivation above, the root difference between preference-based and reward-based methods lies in the choices of K and α.K affects how accurately we can estimate the partition function Z(x) ≈ K j=1 e rj /α in Eq. 5. In practice, we find larger K can lead to better performance (Sec. 5.1). α indicates a trade-off between diversity and optimality. At a → 0, the InfoNCA loss increases the likelihood only for the optimal response and decreases it for all other responses, turning the reward dataset x → {y i , r i } into a preference dataset x → {y w > y l }. We provide ablation studies of α and β in Appendix D.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: Appendix A.1 and A.2Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
nca: fixing decreased response likelihood issue for infonca: 
 A well-observed issue with DPO is that the likelihood of all responses continually decrease throughout training [27, 32] . We find InfoNCA shares this trend due to their inherent equivalence. Decreased data likelihood is concerning because it directly contradicts the maximum likelihood objective for supervised training and may eventually harm performance [44] .We hypothesize the main cause of this decreasing likelihood is that InfoNCA methods only adjust relative rewards among responses, rather than optimizing their absolute value. To address this problem, we take inspiration from NCE, another contrastive learning method parallel to InfoNCE, and propose NCA(lignment) (Sec. 4.1). Similar to InfoNCA, NCA can also guarantee convergence to the optimal LM policy under ideal conditions (Theorem 4.2). However, it directly learns the absolute reward for each response, thereby counteracting the decreasing likelihood trend (Sec. 4.2).To avoid optimizing relative rewards across multiple responses, we construct a binary classification task that deals with a single response.Specifically, imagine sampling a response y randomly from either the optimal LM π * (y|x) = µ(y|x) e r(x,y)/α Z(x) , or the pretrained LM µ(y|x). The marginal probability of y is p(y|x) := 1 2 µ(y|x) + 1 2 π * (y|x). Our goal is to guess its model source when given a response y and its reward r(y). Proposition 4.1 (proof in Appendix A.2). Let a binary variable ν = 1 indicates the response y is sampled from π * . The posterior probability of the distribution source given the response y satisfies:EQUATIONNote that p(ν|x, y) is related to the partition function Z(x). In order to represent model likelihood p θ (ν|x, y) by only employing r θ similarly to Proposition 3.1, we have to redefine π θ (y|x)=µ(y|x)e r θ (x,y) by absorbing Z θ into r θ . Then we haveEQUATIONSimilarly to Theorem 3.2, we can derive a MLE-based training objective for optimizing r θ . Theorem 4.2 (NCA, proof in Appendix A.2). Let α > 0, we have the maximum likelihood objective:(a) Equivalent objective.EQUATIONwhere Z(x) = E µ(y|x) e r(x,y)/α .(b) Optimal solution. Assume unlimited model capacity. The optimal r θ * and π θ * for solving (9) areEQUATIONand π θ * (y|x) ∝ µ(y|x)e r(x,y)/α .For reward datasets (x → {y i , r i } 1:K ), we estimate Z(x) ≈ K i=1 e ri/α in Eq. 9 and construct r θ similarly to InfoNCA:L NCA θ (x, {y i , r i } 1:K ) = - K i=1 e ri/α K j=1 e rj /α softmax weight log σ(r θ (x, y i )) optimizer ↑ (increasing force) + 1 K log σ(-r θ (x, y i ))regularizer ↓ (decreasing force)EQUATIONHow does NCA work? The loss function for NCA involves two opposing forces that jointly determine the trend of increasing or decreasing r θ (x, y). Since log σ(•) is a monotonically increasing function, the first term in Eq. 11 tends to increase r θ (x, y) while the second term tends to decrease it.At the start of training, when r θ = -r θ = 0, the direction of the combined force for r θ is decided by the difference in their weights, expressed as e r i /α K j=1 e r j /α -1 K . Responses with higher rewards would, in principle, attain higher likelihood after training.Although both NCA and InfoNCA originate from solving a noise contrastive classification problem, their optimization targets are markedly different (Table 1 ).InfoNCA and DPO both calibrate relative values of reward models across various responses {y i } 1:K for an instruction x. In other words, the absolute value of r θ (x, y) is not directly constrained. This can lead to some counterintuitive behaviors. For instance, the learned reward for even the highest-reward response could decrease over time without contradicting the loss definition, as long as the reward margin keeps increasing. This could lead to poor performance or training instability (Sec. 5.2).In contrast, NCA specifically focuses on optimizing absolute values of the reward model. This characteristic is determined by its model definition: π θ (y|x)=µ(y|x)e r θ (x,y) , where r θ has to be self-normalized: E µ(y|x) e r θ (x,y) = 1. In practice, NCA effectively prevents the likelihood of the preferred responses from decreasing. We find this is particularly helpful for math and coding tasks.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Details in Appendix C. The source code is in supplemental material. We will open-source it upon publication. 
experiments: 
 We mainly seek to answer two questions in our experiments: ) in LLM alignment. We focus on the general instruction-following abilities of each method measured by GPT-4 evaluations and human preference. The highest number in each benchmark is bolded and the second highest is underlined.Reward dataset and Evaluation metric. We consider UltraFeedback [9] , an instruction-following dataset annotated by GPT-4. This dataset comprises ∼64k instructions. Each instruction has 4 responses generated by various LMs. GPT-4 rates each response with a scalar reward on a scale of 0-10. Prior research indicates that these GPT-4 rewards closely align with human annotations [47] , establishing them as an efficient, cost-effective alternative to human feedback. In order to align exactly with the definition of dataset rewards, we similarly choose well-acknowledged GPT4-based benchmarks like MT-bench [47] and AlpacaEval [19] for evaluation. Human preference studies are also conducted on evaluation prompts from MT-bench. The rating system is in Appendix C.InfoNCA and NCA outperform preference-based methods given reward dataset. To handle reward datasets with K > 2 responses per instruction, one approach is to simply prune them into pairwise data and apply preference learning like DPO. For instance, Zephyr [40] selects the highestreward response and a random remaining one from UltraFeedback for each instruction. This procedure discards two additional suboptimal responses in the dataset as well as their reward information.In performance improvements when increasing the number of data responses from K = 2 to K = 4 for both InfoNCA and NCA algorithms, across various hyperparameters.Combinatorial DPO are suboptimal solutions. Regarding the performance improvement offered by more suboptimal responses, one might predict that applying the DPO to a combinatorially constructed preference dataset would yield results comparable to NCA/InfoNCA. To investigate this, we examined two variants of DPO that utilize all available responses in UltraFeedback.We pair the highest-performing response with each of the remaining three separately.DPO×C 2 4 :We sum up all DPO loss possibilities for two out of the four responses. Our experiments, detailed in Table 2 , reveal that naively applying combinatorial DPO loss to leverage all response information underperforms InfoNCA/NCA. The DPO×3 shows some benefit, while DPO×C 2 4 is harmful compared with simple data pruning. This is expected because InfoNCA and NCA possess theoretical guarantees (Theorem 3.2 and Theorem 4.2) that ensure convergence to the optimal LM policy whereas combinatorial preference methods do not.In previous experiments, our focus is on the reward dataset with K > 2 responses per prompt (x → {y i , r i } 1:K ). However, at present most alignment datasets are pairwise (x → {y w > y l }), making it essential also to evaluate our proposed methods in pairwise preference settings.Since InfoNCA is equivalent to DPO when only pairwise preference data is available (Sec. 3.2), we will focus on comparing and clarifying the differences between the DPO and NCA algorithms.Preferecne dataset and evaluation metrics. We consider fine-tuning Mistral-7B and Mistral-8×7B models on UltraInteract [44] , a pairwise alignment dataset specifically designed for complex reasoning tasks. Before alignment, we perform SFT on UltraInteract's preferred responses for the 8×7B model and use the existing Mistral-SFT model in Sec. 5.1. We evaluate the model's performance in various challenging tasks. This includes BBH-Hard [39] for CoT reasoning, HumanEval [4] and LeetCode [12] for coding, GSM-Plus [18] , MATH, TheoremQA [6] , SVAMP [28] , and ASDiv [22] for math.DPO may hurt reasoning performance while NCA helps. Results are presented in Table 3 . Overall, NCA consistently outperforms DPO in various benchmarks. Notably, we observe DPO hurts the overall performance in most reasoning tasks regarding the Mixtral-8×7B-SFT model. This indicates that DPO might not be suitable for improving reasoning abilities, which echoes findings in concurrent work [44] . In contrast, NCA shows clear improvement on both the 7B and 8×7B models. during the SFT stage. This drawback is exacerbated in reasoning tasks, where the preferred response is often the ground truth answer. Consequently, we hypothesize that NCA's superior performance in reasoning tasks is due to its ability to avoid decreasing chosen likelihood.6WHS 0RGHO5HZDUG '328OWUD,QWHUDFW &KRVHQ 5HMHFWHG 0RGHO5HZDUG 1&$8OWUD,QWHUDFW &KRVHQ 5HMHFWHG 6WHS 0RGHO5HZDUG ,QIR1&$8OWUD)HHGEDFN y1 y2 y3 y4 6WHS 0RGHO5HZDUG 1&$8OWUD)HHGEDFN y1 y2 y3 y4Since DPO is essentially a specialization of InfoNCA, their contrasting likelihood trends can be explained theoretically. As we have elaborated in Sec. 4.2, NCA adjusts the absolute likelihood of data, while DPO/InfoNCA only considers relative likelihood across different responses. Thus, a declining chosen likelihood directly contradicts NCA's training objective but not DPO's.Empirical takeaway: When to choose NCA over DPO? DPO and pairwise NCA have similar theoretical guarantees. Their different performance in alignment tasks is largely empirical, depending on the specific characteristics of datasets and the nature of tasks. Our observations show that NCA is more suitable for reasoning tasks such as math and coding (Table 3 ), where high-quality responses are sparse, and adhering closely to the preferred responses in the dataset is critical. DPO may be more suitable for general instruction-following tasks like summarization/role-playing (Table 2 ), where datasets only reflect human relative preference but do not contain "golden" answers. In essence, NCA benefits from better dataset regularization, while DPO relies more on LLMs' generalization abilities.We also observe that NCA has a greater tolerance for divergence from the initial SFT policy and is more robust to hyperparameter changes. As evidenced in Figure 6 , we conduct a grid search on α and β. DPO can drastically fail to improve alignment performance if the learned policy strays too far from the SFT policy, and may randomly collapse under certain hyperparameters. In contrast, the NCA method does not exhibit similar issues. These observations suggest that NCA may be a worthwhile alternative if DPO training is unstable. 
related work: 
 Language model alignment. Current approaches cater to either explicit reward data or preference data, often lacking the versatility to address both concurrently. Reinforcement Learning [34] is inherently suitable for explicit reward scenarios. However, its on-policy nature necessitates learning a reward model from data first, leading to an indirect two-stage optimization process [8, 26, 36] . Recent developments in preference-based alignment techniques [33, 1, 11, 42, 16, 13] have streamlined this process. They enable direct alignment of LMs through a singular loss, but this comes at the expense of being confined to pairwise preference data. Other alignment approaches [43, 37, 45, 38] are also not tailored for aligning with reward datasets. Recent work [3] attempts to extend DPO's parameterization technique to explicit reward contexts. However, it only considers binary rewards. In comparison, our methods can handle both continuous rewards and preference data.Noise contrastive estimation. NCE [14] and its variant, InfoNCE [24] , are established optimization methods for training unnormalized generative models [21] . NCE primarily leverages a binary classification loss and can be applied in self-supervised representation learning. Examples are Word2Vec [23] , MoCo [15] , and SimCLR [5] . InfoNCE is related to maximizing mutual information between two distributions through a multiclass cross-entropy loss. It has successful applications in representation learning, such as CLIP [31] . It is also widely used in language modeling [7] , diffusion modeling [20] , and reinforcement learning [17] . 
conclusion: 
 In this work, we formally consider the language model alignment problem in the context of explicit reward settings. By adeptly harnessing the NCE and InfoNCE theories, we introduce two practical algorithms: NCA and InfoNCA. Our proposed methods are uniquely suited for both reward data and preference data, including DPO as a special case. Our experiments show that reward-based alignment methods outperform preference baseline by fully leveraging suboptimal responses in reward datasets.In preference settings, pairwise NCA outperforms DPO in complex reasoning tasks by effectively preventing data likelihood from decreasing. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: Yes, we have. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [Yes] Justification: The participants of experiments only conduct small-scale evaluations of language models. There are no obvious potential risks. All experiments comply with the relevant laws and ethical standards of the institutions in the respective countries, and they have received the necessary approvals. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Recall that our optimal language policy is π * (y|x) = µ(y|x) e r(x,y)/α Z(x) .(Eq. 2)Consider a batch of K responses {y i } 1:K for an instruction x. {y i } 1:K consists of one optimal response y ν that is sampled from π * (y|x) ∝ µ(y|x)e r(x,y)/α , and K -1 suboptimal noises independently sampled from µ(y|x). ν ∈ 1 : K is the random index of that optimal response. The the joint probability for {y i } 1:K isp joint ({y i } 1:K |x, ν) = π * (y ν |x) i̸ =ν µ(y i |x) = π * (y ν |x) µ(y ν |x) K i=1 µ(y i |x).Given that the prior satisfies p(ν = 1) = p(ν = 2) = ... = p(ν = K) = 1 K , the data posterior isp joint (ν|x, {y i } 1:K ) = π * (y ν |x)/µ(y ν |x) K j=1 π * (y j |x)/µ(y j |x) . p(O = y i |{y} 1:K ) = p({y} 1:K |O = y i )p(O = y i ) K j=1 p({y} 1:K |O = y j )p(O = y j ) = π * (y i |x)/µ(y i |x) K j=1 π * (y j |x)/µ(y j |x) = e r(yi)/α K j=1 e r(yj )/αDefine model policy as π θ (y|x) := µ(y|x) e r θ (x,y) Z θ (x) .The model posterior probability satisfies x,yi) .p joint θ (ν|x, {y i } 1:K ) = e r θ (x,yν ) K i=1 e r θ (Theorem A.1 (InfoNCA Objective). For any K > 1, α > 0, we have the following results.(a) Equivalent objective.min θ E p joint (x,{yi}) D KL [p joint (ν|x, {y i })||p joint θ (ν|x, {y i })] ⇐⇒ min θ -E p(x) µ(yi|x) K i=1 e r(x,yi)/α Z(x) log e r θ (x,yi)K j=1 e r θ (x,yj ) , (Eq. 5)where Z(x) = E µ(y|x) e r(x,y)/α .(b) Optimal solution. Assume unlimited model capacity and data samples. The optimal r θ * and π θ * for solving Eq. 5 arer θ * (x, y) = r(x, y)/α + C(x),and π θ * (x, y) ∝ µ(y|x)e r(x,y)/α , where C(x) is an arbitrary function conditioning on x.Proof. (a) Equivalent objective.min θ E p joint (x,{yi}) D KL [p joint (ν|x, {y i })||p joint θ (ν|x, {y i })] ⇔ min θ E p joint (x,{yi}) E p joint (ν|x,{yi}) log p joint (ν|x, {y i }) p joint θ (ν|x, {y i }) ⇔ min θ -E p joint (x,{yi}) E p joint (ν|x,{yi}) log p joint θ (ν|x, {y i }) ⇔ min θ -E p(x)p(ν)p joint ({yi}|x,ν) log p joint θ (ν|x, {y i }) (Bayes' rule) ⇔ min θ -E p(x)p(ν) K i=1 µ(yi|x) π * (y ν |x) µ(y ν |x) log p joint θ (ν|x, {y i }) (importance sampling) ⇔ min θ -E p(x) K i=1 µ(yi|x) E p(ν) π * (y ν |x) µ(y ν |x) log p joint θ (ν|x, {y i }) ⇔ min θ -E p(x) K i=1 µ(yi|x) 1 K K ν=1 π * (y ν |x) µ(y ν |x) log p joint θ (ν|x, {y i }) ⇔ min θ -E p(x) K i=1 µ(yi|x) K ν=1 e r(x,yν )/α Z(x) log p joint θ (ν|x, {y i }) (based on Eq. 2) ⇔ min θ -E p(x) K i=1 µ(yi|x) K i=1 e r(x,yi)/α Z(x) log e r θ (x,yi) K j=1 e r θ (x,yj ) (change sum index) (b) Optimal solution.Given conclusions from (a). With unlimited model capacity, p joint θ (ν|x, {y i }) could represent any discrete distribution, such that we can arrive at the global optimal point given infinite training data.E p joint (x,{yi}) D KL [p joint (ν|x, {y i })||p joint θ * (ν|x, {y i })] = 0 =⇒ p joint (ν|x, {y i }) = p joint θ * (ν|x, {y i }) ∀x, ν, {y i } 1:K =⇒ π * (y ν |x)/µ(y ν |x) K i=1 π * (y i |x)/µ(y i |x) = e r θ * (x,yν ) K i=1 e r θ * (x,yi) ∀x, ν, {y i } 1:K =⇒ r θ * (x, y) = r(x, y)/α + C(x) ∀x, y, C=⇒ π θ * (x, y) ∝ µ(y|x)e r(x,y)/α ∀x, yRecall the optimal language policy isπ * (y|x) = µ(y|x) e r(x,y)/α Z(x) . (Eq. 2)Consider a response y randomly sampled from either the optimal LM π * (y|x) = µ(y|x) e r(x,y)/α Z(x) , or the pretrained LM µ(y|x). Let a binary variable ν = 1 indicates the response y is sampled from π * .Then the marginal distribution of y is p joint (y|x) := p(ν = 0)µ(y|x) + p(ν = 1)π * (y|x).Given the prior p(ν = 0) = p(ν = 1) = 1 2 , using Bayes' Rule, the data posterior satisfiesp joint (ν = 0|x, y) = µ(y|x) µ(y|x) + π * (y|x) . p joint (ν = 1|x, y) = π * (y|x) µ(y|x) + π * (y|x).Define model policy as π θ (y|x) := µ(y|x)e r θ (x,y) . The model posterior probability satisfiesp joint θ (ν = 1|x, y) = σ(r θ (x, y)). p joint θ (ν = 0|x, y) = 1 -σ(r θ (x, y)) = σ(-r θ (x, y)).(Eq. 8) Theorem A.2 (NCA Objective). For any α > 0, we have the following results. ⇐⇒ min θ -E p(x)µ(y|x) e r(x,y)/α Z(x) log σ(r θ (x, y)) + log σ(-r θ (x, y)), (Eq. 9)where Z(x) = E µ(y|x) e r(x,y)/α .(b) Optimal solution. Assume unlimited model capacity and data samples. The optimal r θ * and π θ * for solving Eq. 5 are r θ * (x, y) = r(x, y)/α -log E µ(y|x) e r(x,y)/α , and π θ * (x, y) ∝ µ(y|x)e r(x,y)/α .Proof. (a) Equivalent objective.min θ E p joint (x,y) D KL [p joint (ν|x, y)||p joint θ (ν|x, y)] ⇔ min θ E p joint (x,y) E p joint (ν|x,y) log p joint (ν|x, y) p joint θ (ν|x, y) ⇔ min θ -E p joint (x,y) E p joint (ν|x,y) log p joint θ (ν|x, y) ⇔ min θ -E p(x)p(ν)p joint (y|x,ν) log p joint θ (ν|x, y) (Bayes' rule) ⇔ min θ -[p(ν = 0)E p(x)p joint (y|x,ν=0) log p joint θ (ν = 0|x, y)+ p(ν = 1)E p(x)p joint (y|x,ν=1) log p joint θ (ν = 1|x, y)] ⇔ min θ -E p(x)µ(y|x) log σ(-r θ (x, y)) + E p(x)π * (y|x) log σ(r θ (x, y))(by Eq. 8)⇔ min θ -E p(x)µ(y|x) e r(x,y)/α Z(x) log σ(r θ (x, y)) + log σ(-r θ (x, y)) (importance sampling) (b) Optimal solution.Given conclusions from (a). With unlimited model capacity, p joint θ (ν|x, y) could represent any discrete distribution, such that we can arrive at the global optimal point given infinite training data.E p joint (x,y) D KL [p joint (ν|x, y)||p joint θ * (ν|x, y)] = 0 =⇒ p joint (ν|x, y) = p joint θ * (ν|x, y) ∀x, ν, y =⇒ p joint (ν = 1|x, y) = p joint θ * (ν = 1|x, y) ∀x, y =⇒ π * (y|x) µ(y|x) + π * (y|x) = σ(r * θ (x, y)) = e r * θ (x,y) 1 + e r * θ (x,y) ∀x, y =⇒ e r * θ (x,y) = π * (y|x) µ(y|x) ∀x, y=⇒ r θ * (x, y) = r(x, y)/α -log E µ(y|x) e r(x,y)/α ∀x, y =⇒ π θ * (x, y) ∝ µ(y|x)e r(x,y)/α ∀x, yExperiments with UltraFeedback. Our implementation is heavily based on the Transformer Reinforcement Learning (TRL) library [41] and Zephyr's official code base [40] . All models are fine-tuned from the publicly accessible HuggingFaceH4/mistral-7B-SFT-beta model. Experiments are run on Nvidia A40 or RTX 4090 GPUs using bfloat16 precision. We ablate β ∈ {3e -4, 1e -3, 3e -3, 1e -2, 3e -2, 1e -1, 3e -1, 1.0} and α ∈ {0.01, 0.1, 0.33, 1.0, 3.33}. The default reward temperature α is 0.01. The default parameterization coefficient β is also 0.01. We adopt the QLoRA [10] fine-tuning technique with rank 16, α lora = 16, and a dropout rate of 0.05. We train all models for 1 epoch. The batch size is 32. We use an AdamW optimizer with a learning rating of 5e-6. For KTO and IPO baselines, we adopt exactly the same training pipeline for reporting their performance except that we tune the β ∈ {0.01, 0.1, 0.3, 0.5, 1.0}. We find the most suitable beta for KTO is 0.01, and for IPO is 0.5.Experiments with UltraInteract. We follow [44] and fine-tune all parameters of the pretrained model with UltraInteract and UltraFeedback. Specifically, for HuggingFaceH4/mistral-7B-SFT-beta model, we directly adopt the recipe of Eurus, with the same data mixture (all 220K multi-turn trajectory pairs in UltraInteract and all 340K pairs in UltraFeedback) and hyperparameters. All β is set to 0.1. The model is trained for 1 epoch with a cosine schedule. The learning rate is 5e-7 and the warmup ratio is 0.1. For Mixtral-8x7B, we first SFT the model with the recipe for Eurus-70B-SFT, and then applied the same preference learning recipe as aforementioned.Figure 7 : Rating system screenshot. Our rating system is adapted from [47] . We provide sampled responses of LM policies trained respectively with the DPO and NCA to compare their data regularization feature. In general, generations from the NCA policy are more similar to the preferred response in the dataset.You are given an answer. Create a question for the answer. Grams to Ounces conversion 1 gram (g) is equal to 0.03527396195 ounces (oz). Question: What is the conversion rate from grams to ounces, and how many grams are one ounce?Table 5 : Sampled generations and the log-likelihood ratio of the NCA and the DPO/InfoNCA policy.If "A busy crosswalk somewhere in asia." does that mean that "The crosswalk was blocked off by police."? Options:-yes; -it is not possible to tell ; -no • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: The experiments are quite deterministic. Just like most previous work, we run most experiments only once, so it can be difficult to report an error bar. Instead, we validate the robustness of our methods mainly by ablating various hyperparameters (e.g., Figure 6 and Figure 4 . A standard derivation can be calculated from the figure, or we can report the std error given by the Alpaca benchmark, but it is really unnecessary.)Guidelines:• The answer NA means that the paper does not include experiments. Justification: Yes, we have.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Appendix F. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: This paper proposes a noise-contrastive estimation-based approach to alignment in which DPO can be seen as a special case. The paper is well-written, theoretically-grounded, and backed up by solid empirical experiments.  A clear accept.

(There were some questions about the claimed novelty on top of DPO, given DPO's generalization to ranked responses via the Plackett-Luce model is not only straightforward, but explicitly mentioned in the original paper's appendix. I recommend the authors to add more discussion around this point.)

Reviews: review: paper_summary: paper_summary: The paper addresses a weakness of DPO: it can not deal with preference data whose number of responses is larger than 2. To extend DPO, the author proposes Noise Contrastive Estimation (NCE)-based Alignment algorithms InfoNCA. The author has shown that DPO is one of the special cases of InfoNCA. To further fix the Decreased Response Likelihood issue, the author also proposes NCA with a similar derivation. Experiments demonstrate that NCA and InfoNCA outperform baselines with reward datasets for training, with NCA showing significant performance in complex reasoning tasks like math and coding. The paper offers the community a general framework for LLM alignment from a novel perspective.
strength_weakness: strength_weakness: 1. Fundamental Assumption: My major concern is about the fundamental assumption when deriving the InfoNCA and NCA methods. The author assumes that one of the scored responses is sampled from the optimal policy, which is practically impossible because most of the preference data are sampled from non-optimal LLM policy. In some extreme situations, all the generated responses can be harmful, while our ideal optimal policy should output harmless responses, leading to a contradiction with the author's essential assumption. A possible solution to this weakness might be mixing the human written (golden) response into the preference data, then applying NCE to the prompt and the golden response, combining InfoNCA with some prior work such as APO [1] and SPIN [2].

2. Lack of Human Evaluation: Although the author has conducted GPT-4 evaluation in the experiments, the real human evaluation results are not reported, which should have made the paper's claims more persuasive.

3. Lack an experiments of two-response alignment: Although the paper demonstrates better performance using multi-response preference data, it is interesting to compare InfoNCA, NCA, and DPO on a binary-response preference set. 

4. Lack of baseline comparison: Although DPO cannot deal with multi-response preference, one can first train an RM with the multi-response preference, and next sample two responses with scores from the learned RM. Then DPO can be conducted on the RM-scored pairwise preference data. This can be a simple but important baseline that the author should take into consideration.

References:

[1] Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game, ACL 2024

[2] Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models, ICML 2024
questions: questions: See Weaknesses.
limitations: limitations: The author has NOT addressed the limitations of the proposed method. One of the most important limitations can be that if all the scored responses are harmful, the proposed methods can never achieve the ideal alignment optimum, which could even enhance the harmful model outputs.
paper_summary: paper_summary: This paper leverages Noise Contrastive Estimation (NCE) to align language models (LMs) with explicit reward data, which can handle both scalar evaluations and multiple preference data. The proposed methods, NCA and InfoNCA, extend current alignment theories by improving upon DPO and addressing the issue of decreasing likelihood observed in DPO. Experimental results demonstrate that InfoNCA and NCA outperform existing preference-based methods.
strength_weakness: strength_weakness: 1. The motivation of this paper is weak: The description that DPO can only be used for pairwise preference data is wrong (such as Line 31 and 43). In fact, the appendix of DPO says that it can be applied to multiple preference data based on the Plackett-Luce model.
2. The main experiments lack related baselines. Since the proposed method is a reward-based method that utilizes reward data for preference ranking, Table 2 should also include PPO under the annotation type "Reward" for direct comparison since it also utilizes reward data. In addition, I suggest comparing it with PRO (https://arxiv.org/pdf/2306.17492) under the annotation type "preference" since this work directly applies DPO to the setting of multiple preference ranking.
3. The proposed method InfoNCA uses both preference data and reward data for training, resulting in more computational overhead compared with PPO and DPO. Thus, it is not surprising that better performance is achieved than with preference-based methods, although the improvement is marginal. Is it a fair comparison if the methods use different magnitudes of data? Please also see comment 1.
questions: questions: 1. What is the amount and percentage of reward and preference data in each dataset? 
2. Since InfoNCA and NCA are reward-based methods, why not compare them with PPO in terms of MT-bench, AlpcaEval, and Wins vs. PPO?
limitations: limitations: Limitations are discussed in the appendix
paper_summary: paper_summary: The authors introduce a general framework for LM alignment, leveraging Noise Contrastive Estimation (NCE) to bridge the gap in handling reward datasets explicitly annotated with scalar evaluations. The framework comprises two parallel algorithms, NCA
 and InfoNCA, both enabling the direct extraction of an LM policy from reward data as well as preference data.
strength_weakness: strength_weakness: 1. The proposed method looks trivial by integrating classic contrastive learning theories. Different methods coming out at the same time. They look similar and not easy to find which is better, such as the concurrent work SimPO: Simple Preference Optimization with a Reference-Free Reward.
questions: questions: More training details, such as hyper-parameter search space
limitations: limitations: It would be better list more training details.
paper_summary: paper_summary: ## Summary

- Typically, RLHF algorithms like DPO use preference pairs dataset.
- The key question, this paper tries to answer is how to incorporate reward datasets annotated with scalar values. Previous approach usually prune the scalar reward datasets by selecting the best response and pairing with a random remaining response. However, this work shows that the extra suboptimal responses can also be beneficial for policy training.
- They present two main algorithms (inspired from contrastive learning literature) here which handle the aforementioned problem
	- NCA Noice Contrastive Alignment
	- InfoNCA
- They also show that NCA is able to mitigate the "decreasing likelihood trend" observed with the DPO algorithm. NCA also outperforms DPO on math and coding tasks because it prevents the likelihood of preferred responses from decreasing. The main difference between InfoNCA and NCA objective is that the NCA objective has an additional regularizer term 
	- The "decreasing likelihood trend" can be simply described as the decrease in likelihood of the preferred response after DPO training.
	- This happens because DPO focuses on adjusting relative likelihoods across generations whereas NCA focuses on adjusting absolute likelhood
- InfoNCA compares multiple responses and identifies the one sampled from optimal policy. NCA predicts the model source of a single response.
- The authors also show that DPO is a special case of InfoNCA
strength_weakness: strength_weakness: I could not find any weakness
questions: questions: ## Questions

- I have not gone through the material in the appendix but at several places in the theorems, the authors assume a model with unlimited capacity. Why is this important and what are the implications otherwise?
- It is not obvious to me how equation 11 leads to predicting the model source of a single response in Figure 3
limitations: limitations: The authors have adequately addressed the limitations.
paper_summary: paper_summary: The paper's main contributions are:

1. Theoretical Integration: It integrates Direct Policy Optimization (DPO) with contrastive learning theories, offering a general framework for using reward and preference data.
2. Value of Suboptimal Responses: Highlights the importance of suboptimal responses in optimizing language models, showing improved performance over other methods by fully utilizing reward data.
3. Performance Improvement: Demonstrates how NCA counters the data likelihood decline in DPO, enhancing practical performance.
strength_weakness: strength_weakness: I don't have further suggestions for this part.
questions: questions: How do you compare your work with the paper titled Direct Preference Optimization with an Offset(https://arxiv.org/pdf/2402.10571).
limitations: limitations: I don't see any issues with this part.

Comments: title: Looking forward to your feedback
Reviewers, please respond to the author rebuttals
Additional Response
Additional Response
comment: Thank you for your feedback. We are glad that our responses help.
Thank you for your elaborate response. Since the author has expanded with more comprehensive and detailed experiments, I have decided to raise my score.
Dear reviewer,

Thank you for your valuable comments regarding our submission.  We have posted our responses and hope to address your concerns about the fundamental assumption of our method (availability of the optimal policy), as well as the comparison with several baselines such as APO and SPIN. 

Could you please take a look at our responses and give us some further feedback so that we can continually improve our work? Thank you so much in advance!

Best regards,

The Authors
Dear reviewers, this is your AC! 

The authors have responded to your review. Could you please take a look at the rebuttal (and respond/edit your scores if need be)?
(Thank you if you have already done so)

Thanks!
We appreciate the reviewer for the prompt and positive feedback and are glad we can address the reviewer's concerns.
Thanks for the response. Since most of my questions and concerns are addressed, I will increase my overall recommendation.
Dear reviewer,

We copy the second part of our rebuttal response here to ease reading. In our responses, we have compared with more baselines and clarified the details of the data we are using based on your suggestions. We hope we can address your concerns and are glad to answer any further questions you might have.

Thank you again for your valuable feedback!

# Official Response to Reviewer EZsK (2/2)

**Q5: suggest comparing it with PRO (https://arxiv.org/pdf/2306.17492).**

**A5:** PRO is a quite related work of ours because it can also handle response data of arbitrary length. It is also highly related to the PL method as we have discussed in **A1**. Still, PRO is essentially a **ranking/preference-based** method,  while (Info)NCA are **reward-based** methods.

**Theory comparison:**

**We have added a new section as Appendix G.2 in our paper to compare with PRO theoretically:** (cannot update the manuscript during rebuttal, though) 

The main difference between PRO and the PL+DPO method is that
1. PRO has different reward formulations. It formulates $r_\theta$ as the average log-probability of a sentence $\frac{1}{\|y\|}\Sigma\log \pi_\theta(y|X)$ instead of the log-probability ratio $\log \frac{\pi_\theta(y|x)}{\mu(y|x)}$ as is required by DPO.
2.  Because there is no $\mu$ in reward models. PRO needs to be additionally regularized by an SFT loss in order to stay close to the pretrained LLM $\mu$.

**Additional Experiments:**
| Experiment| MT-bench |
|----|---|
| Mixtral-7B-sft |  64.5 
| DPO + PL Method| 74.1 | 
| PRO (SFT) | 68.9 | 
| InfoNCA  | 76.3  |

Overall we find PRO (PL+SFT) method helps, but slightly underperforms the PL+DPO method. We are not confident about this conclusion because we did not faithfully replicate PRO according to its source code, but only modified our trl implementation of DPO+PL to test PRO methods. We mainly search sft_weight $\in$ [0.01, 0.05, 0.1 0.5], and keep other hyperparameters consistent with train_hh.sh of PRO repo, or the trl/DPO settings.
Dear reviewer,

We copy the second part of our rebuttal response here to ease reading. In our responses, we have conducted additional experiments and compared with more baselines based on your suggestions. We hope we can address your concerns and are glad to answer any further questions you might have.

Thank you again for your valuable feedback!

# Official Response to Reviewer EyJu (2/2)

**Q5: Lack  of two-response alignment:  compare InfoNCA, NCA, and DPO on a binary-response preference set.**

**A5:** Perhaps we did not fully understand the reviewer's comment. As a matter of fact, **the whole section 5.2 aims to compare these algorithms on binary-response preference sets.** This includes Table  3, Figure 4 (left), Figure 5, and Figure 6(left). 

Note that in pairwise-preference settings, InfoNCA becomes exactly the DPO algorithm, so we mainly compare NCA and DPO (InfoNCA).

We copy the main results below:
**UltraFeedback (binary)**
| Method  | MT-bench | Alpaca |
|---|---|--|
| InfoNCA/DPO | 73.8     | 90.7   |
| NCA     | 73.2     | 89.9   |

**UltraInteract (binary)**


|Model|Average|Reasoning|LeetCode|HumanEval|GSMPLUS|MATH|TheoremQA|SVAMP|ASDiv|
|---|---|---|---|---|---|---|---|---|---|
|Mixtral-7B-SFT|24.5|60.9|3.3|28.1|28.5|5.8|7.0|26.9|35.8|
|$+$DPO|25.5|**61.7**|2.2|**31.7**|12.1|6.4|9.8|34.1|46.1|
|$+$**NCA**|**35.7**|60.8|**3.3**|26.8|**32.3**|**11.7**|**11.0**|**65.3**|**74.3**|
|Mixtral-8*7B-SFT|56.3|**75.6**|16.7|61.0|57.6|40.1|25.9|85.9|87.5|
|$+$DPO|50.1|74.9|17.2|47.6|55.8|35.3|**26.9**|67.3|75.7|
|$+$**NCA**|**57.9**|**75.6**|**21.1**|**62.8**|**61.5**|**41.6**|**26.9**|**86.8**|**86.9**|

**Q6: The real human evaluation results are not reported, which should have made the paper's claims more persuasive.**

**A6:** **We have actually reported human evaluation results in the original paper** and would like to refer the reviewer to Table 2 (win vs DPO), and  Figure 7in the paper. 

Still, GPT-4 based evaluation results are most indicative to ourselves because datasets are annotated by GPT-4. After all, you want your training rewards and test rewards to be aligned to make sure the algorithm itself is valid given that our work is somehow more theoretical rather than product-oriented.