Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads
Authors: Avelina Asada Hadji-Kyriacou
Ognjen Arandjelovic
Venue: NeurIPS 2024 poster (2024)
Keywords: Deep Learning
Neural Networks
Ranking and Preference Learning
Reinforcement Learning
Primary area: natural_language_processing

Abstract: Pre-trained Language Models (LMs) exhibit strong zero-shot and in-context learning capabilities; however, their behaviors are often difficult to control. By utilizing Reinforcement Learning from Human Feedback (RLHF), it is possible to fine-tune unsupervised LMs to follow instructions and produce outputs that reflect human preferences. Despite its benefits, RLHF has been shown to potentially harm a language model's reasoning capabilities and introduce artifacts such as hallucinations where the model may fabricate facts. To address this issue we introduce Direct Preference Heads (DPH), a fine-tuning framework that enables LMs to learn human preference signals through an auxiliary reward head without directly affecting the output distribution of the language modeling head. We perform a theoretical analysis of our objective function and find strong ties to Conservative Direct Preference Optimization (cDPO). Finally we evaluate our models on GLUE, RACE, and the GPT4All evaluation suite and demonstrate that our method produces models which achieve higher scores than those fine-tuned with Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO) alone.

Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads
Avelina Asada Hadji-Kyriacou Department of Computer Science University St Andrews College Gate, St Andrews, KY16 9AJ 1hk3@st-andrews...
Ognjen Arandjelović Department of Computer Science
University of St Andrews College Gate, St Andrews, KY16 9AJ oa7@st-andrent ac.
Abstract
Pre-trained Language Models (LMs) exhibit strong zero-shot and in-context learn- ing capabilities; however, their behaviors are often difficult to control. By utilizing Reinforcement Learning from Human Feedback (RLHF), it is possible to fine-tune unsupervised I.Ms follow instructions and produce outputs that reflect human preferences. Despite its benefits, RLHF has been shown to potentially harm lan- guage model's reasoning capabilities and introduce artifacts such as hallucinations where the model may fabricate facts. To address this issue introduce Direct Preference Heads (DPH), fine-tuning framework that enables LMs to learn human preference signals through auxiliary reward head without directly affecting the output distribution of the language modeling head. We perform a theoretical analy- sis of our objective function and find strong ties to Conservative Direct Preference Optimization (cDPO). Finally we evaluate our models on GLUE, RACE, and the GPT4AII evaluation suite and demonstrate that our method produces models which achieve higher scores than those fine-tuned with Supervised Fine- Fine-Tuning Tunnig (SFT) or Direct Preference Optimization (DPO) alone.
1 Introduction
Reinforcement Learning from Human Feedback (RLHF) is technique that can be used to align agent - such as a Large Language Model (LLM) - to human preferences and lead to more truthful, more helpful, less harmful and more preferred outputs (29). Proximal Policy Optimization (PPO) 366 and Direct Preference Optimization (DPO) [31 are two such alignment techniques which have been extensively used to improve the quality of LLM outputs, leading to instruction following agents or chat assistants which are quickly approaching Haman-baaseliee in a variety of knowledge and
reasoning tasks 10 42 18
35
However, recent research has shown that RLHF may actually hurt an LLM's reasoning abilities rather than improve it. One study (5) discovered that performing alignment during the Supervised Fine- Tuning (SFT) stage of training may lead to worse performance on reasoning benchmarks, and another 3 that SFT alone outperforms RLHF for smaller models with the benefits of RLHF only emerging for models with more than billion parameters. Ouyang et al. (29 also report an increased tendency for RLHF models make up information in closed domain tasks ("hallucination") compared to models trained with SFT alone.
To combat the the risk of RLHF compromising the abilities of LLM in favour of producing preferable outputs we introduce Direct Preference Heads (DPH), a novel feature based approach that optimises reward score produced by the LLM rather than optimising the logits produced by language modelling head. DPH can be used in combination with (or without) existing alignment
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

techniques to allow language models to self-evaluate outputs sampled at inference time and select the highest scoring candidate.
We evaluate the performance of DPH using an efficient 551M parameter LM on a variety of com- monsense reasoning and Natural Language Understanding (NLU) tasks. All code used to train models is available on GitHub and we release our model weights on Hugging Face
2 Prior Approaches
Prior approaches to language model alignment involve directly optimizing the logits produced by the language modelling head to increase the likelihood of producing preferable responses while decreasing the likelihood of undesirable responses.
2.1 Reinforcement Learning from Human Feedback (RLHF)
Reinforcement Learning from Human Feedback seeks to learn a reward model from human feedback on completions generated by a language model which can be used to align an LM with human preferences. A typical RLHF pipeline consists of 3 steps: (1) supervised fine-tuning, (2) preference sampling and reward modelling, and (3) RL fine-tuning.
Supervised Fine-Tiee Mune-Tuning The first step of a standard RLHF pipeline is fine-tume fine-tuning pre-trained LM on high quality data for downstream tasks obtain a model SPT
Reward Modelling Next, the SFT model is prompted with input tokens x to produce completions y. These answers then rated by human labellers which rate the answers based criteria. A reward model (2r y) is then trained to estimate the scores assigned by human labellers using maximum likelihood estimation.
more
RL Fine- Tunnng During the RL phase the learned reward function used to provide feedback to the language model using the following optimization problem
max Rew (M/=) [r u)] Dx. (mo
where | controls the deviation from the base reference policy Tet which is typically initialized from SST Due to the non-differentiable nature of language generation this objective must be optimized using a reinforcement learning algorithm such as PPO [36].
2.2 Direct Preference Optimization (DPO)
Direct Preference Optimization was introduced as reparametrizatioo of RLHF which eliminates both the sampling stage and the reward modelling stages and reformulates alignment procedure as loss function which can be optimized directly on dataset of pairs of preferred and dispreferred completions to given prompts. This allows DPO to stably and efficiently converge on an optimal policy using what is effectively a classification loss over positive and negative pairs.
Given a dataset [zz Fw 31)} where x is the prompt and Vies Yi are the preferred and dispreferred completions, we introduce the following loss function:
To (V)uvl 3 log
No (o((
LDro(x,, yw, 31) = log BBog
(2)
( (r)
where na (z) and [re((y+la) |a:) are the probabilities of completions y. for prompt x given by the policy model and reference models respectively, and the 3 parameter controls the deviation from the reference policy.
There also exists an augmentation DPO namely Conservative DPO (cDPO) 26 which is designed to be more robust to noisy labels through the introduction of label smoothing parameter The objective function for eDPO is given by:
L.DPO (Ir, Vons 9/) (1 C Bor, Bors y1) CDPO (I, yı y.
htttt://sttiub. wwtp:/bbboooeaeeeeeeeeeeeeeeeeee wttp:///wwwwweea///eeeeeeae http://ddeeeeeepoooereeeeeeeeeee prwference. beads
bttpo:/
prefexeace heads proprint 661280bact336448822444443

---

Direct Preference Heads
The hypothesis underlying the Direct Preference Optimization framework of Rafailov et al. [31] is that a "language model is secretly a reward model" thereby making the purpose of Direct Preference Heads to exploit this and extract explicit reward signals without the need of additional reward model.
3.1 Reward Head
To obtain the rewards from sequence 2; three components required: aggregated hidden state / which is conditioned on the intermediate representations the language model, a pooling function f which transforms the hidden state, and a learnable vector Wdph with the same dimension as the output of f. We then compute the reward r as follows:
r r= f(h) Wdph-
To obtain the hidden state we take the output of the last transformer layer for the final token of sequence, and we experiment with three choices of f: (1) the identity mapping following the convention established by OpenAl's GPT for sequence classification [3]. (2) a learnable affine projection with lanh nonlinearity following BERT's pooling function [21] and (3) an inverted
bottleneck FFN with SwiGLU activation mirroring the FFN blocks used within the transformer backbone followed by tanh nonlinearity [37].
3.2 Objective Function
We formulate two novel objective functions for our method: a separable objective which maximises positive rewards and minimises negative rewards, and a contrastive objective which maximises the margin between positive and negative rewards. The loss landscapes are illustrated by Figure in the appendix.
3.2.1 Separable DPH
The Separable DPH loss function given by (5) is function of the preferred and dispreferred rewards Tw: rt, and the label smoothing parameter
S 0.5 which controls the reward margin:
- [(1 <) log (rrw) + loga(-rw)] - log (ri) + (1 - c) log (oogo(-ri)) (5) Theorem 1. For all c E (0, 0.5) the objective function LSepDru is convex and will optimize the policy T9 such that the preferred rewards Tw produced by the preference head converge towards log 1-c and the dispreferred rewards converge to log I-ce
CSepDRU(T (Twr,
This can be proven by observing the first and second partial derivatives of the loss function with respect to the rewards. The first partial derivative is equal to zero at the points T' = log 1- and = log - respectively, and the second partial derivative is strictly positive for all values of Two Tt- A full proof is included in Appendix A.I
3.2.2 Contrastive DPH
Like Separable DPH, the loss function for Contrastive DPH given by 6 is function of the preferred and dispreferred rewards rw: ri and the label smoothing parameter < c loss function optimizes the relative margin between the rewards rather than optimizing the absolute positive and negative rewards as in Separable DPH.
S 0.5. This version of the
(6) For all E (0, 0.5] the objective function LConDPH is convex and will optimize the policy Ty such that the difference between preferred rewards Tw and dispreferred rewards ri produced by the preference head will converge to fixed margin, given by r Tw T1 log
LCOaDPH(Te (Tw, r)) = -(1 (1- <) log (rw - ori - -Tw)-
Theorem 2.
This can be proven by reparametrising the loss function such that TΛ = Tup r and then by considering the first and second partial derivatives with respect to this reward margin. It can be observed that the first partial derivative is equal to zero when l'A log 1_c, and the second partial derivative is strictly positive for all values of r A full proof is included in Appendix A.2

---

3.2.3 Relation to cDPO
The properties of both Contrastive DPH and Separable DPH show a strong relationship with Conser- vative DPO: SepDPH will converge to optimal fixed reward margins above zero for Tw and below zero for ry; ConDPH will converge to optimal fixed reward margins between f' and T;, and cDPO will converge to fixed delta from the reference model [26]. Like Conservative DPO, this makes both Separable DPH and Contrastive DPH robust to preference label noise and makes training more stable than naive maximum likelihood estimation without label-smoothing
3.3 Novelty over Traditional Reward Modelling
Although similar to the reward modelling phase of an RLHF pipeline, DPH has some distinet differences which set it apart. DPH does not require an SFT sampling and human labelling stage meaning can take advantage of pre-constructed preference datasets such as those used for DPO. Typical RLHF also requires multiple models
reward model, a reference model and policy model - while DPH requires only a single model to produce both responses and rewards.
Unlike other RLHF pipelines such as PPO [36], the rewards produced by DPH are not used for RL fine-tuning; instead, the DPH rewards are be used to prune candidate generations sampled from the LM at inference time to select the candidate which aligns most with human preferences. This makes DPH an excellent choice for small language models which are (1) more lightweight and therefore can be efficiently used to generate multiple samples and, (2) are more prone to degradation when aligned using typical RI. techniques
4 Experimental Setup and Data
4.1
Datasets
We make use of variety of datasets for fine-tuning and evaluation which are outlined below. The specific prompt templates used for fine- tne-tunngg and evaluation are described in Appendix C
Natural Language Understanding (NLU) For general NLU make of the standard GLUE benchmark [40]. The overall score for GLUE is computed by the macro-average of unweighted metric averages for all 9 tasks, however also include a secondary score which does included We opted to omit WNLI
the 'problematic' WNLI task following the evaluation used for BERT 21]
during fine-tuning due the low sample size.
accordance with the GPT4AII 1 evaluation suite, we use the following datasets to evaluate commonsense reasoning abilities: HellaSwag 42). OpenBookQA
Commonsense Reasoning
25), WinoGrande [35), ARC [10] BoolQ 9 and PIQA 7
Reading Comprehension To evaluate reading comprehension abilities we use the RACE dataset 22 multiple-choice task which requires reasoning over provided passages.
OpenOrca (23,, and UltraFeedback [22 datasets to train our models for instruction following. We make use of OpenOrca and cleaned version of Alpaca for SFT, and binarized versions of OpenOrca and UltraFeedbbckk for alignment.
Instruction Following We include the Alpaca
38
Auxiliary Datasets To provide additional training data for SFT we include the MMLU [8] SQuAD V2 [33 32), Tiny Stories [15,, CNN-Dailymail (27] and CoQA [34] training splits. For alignment only include MMLU and SQuAD V2.
4.2 Prompts and Sampling
Prompts We make use of the ChatML prompt templating scheme 28 with handcrafted system.
user and assistant prompts specific to each task. During fine-tuning we mask out the loss for all tokens of the prompt and condition the model on the content assistant messages including the final < [li__nd]>> token. During evaluation we select the highest scoring answer using the average log-probabilitia of the tokens in the final assi stant message, or compute the reward scores on the final <| <
> token when evaluating with DPH.

---

SFT Sampling When sampling from the datasets for SFT we randomly shuffle each dataset and uniformly interleave samples from all tasks in the mix. To control the weighting of samples from each task we fill the context window with n consecutive samples from the same task before sampling from a different task, where n is chosen to be in our experiments. To maximise compute utilisation and
minimize unused portions of the context window we make us of Transformer-XI [13] style training with a context window size of 2048 tokens and a recurrent memory size of 2048 tokens.
DPH Sampling When sampling from datasets for DPH alignment we switch from the Transformer- XL style pipeline to typical SFT training. opting to only include single samples in the context window padded to fixed maximum length. As some of the datasets we use for DPH are intended for SFT rather than alignment (namely GLUE, GPT4AII, RACE, MMLU and SQuAD) we synthesise preference pairs where the 'correct' answer is used as the preferred completion and we uniformly sample an 'incorrect' answer from the available choices for the dispreferred completion. This trivial for most datasets, however we use special process for the SQuAD V2 dataset; for answerable questions we use "unanswerable" as the dispreferred completion, and for unanswerable questions we use SpaCy to randomly sample a noun span from the context to use as the dispreferred completion.
4.3
Regularization
The hidden states h used to compute the reward scores are likely sub-optimal for computing rewards when initialising my from SF As such, may be desirable to fine-tune some or all parameters the language model to learn better reward signals. This necessitates the use of regularization to prevent degradation of the models generative capabilities while learning to predict rewards.
Prior Regularization Typical parameter regularization strategies such as weight decay make the assumption that parameters 0 follow zero-mean Normal distribution p(0) NO, 1) leading
to auxiliary loss term [|01|3 However, when performing wransferleermingg fine-tuning pre- pre-trained model this assumption can be harmful and aid in catastrophic forgetting of the model's s previously learnt abilities.
An alternative regularization scheme is Prior Regularization g which instead makes the assumption that the fine-tuned parameters are normally distributed around the original parameters Oce that is 0 (Oreet 1), leading the auxiliary loss term 1 0
We employ Prior Regularization to limit the divergence of my from SFT while still facilitating the learning of improved hidden state representations for the Direct Preference Head. Pseudocode for optimizer based decoupled prior regularization included Appendix B.I
cDPO Regularization Rather than directly employing KL divergence penalty similar to that used in ( find that it is possible and even beneficial to use Conservative DPO as a means of limiting the divergence of the policy model to a fixed delta from the reference model, and 'nudging' the model towards generating more preferable outputs which increases the chance of generating a better candidate completion at inference time with fewer sampling steps.
4.4 Training Pipeline
We progressively fine-tune the models in 3 stages: vocab extension, supervised fine-tuning, and DPH alignment. The details of the pre-trained model are included in Appendix D.1]
Vocab Extension Since our model was pre-trained without a chat structure is necessary to train the embeddings for additional < $lim__staae start/ >> and < iim_edd >> tokens: we freeze all non-embedding parameters and use the same datasets as SFT. We fine-tune the embeddings for 4096 steps with a batch size of 128, max LR of 6e-5 which warms up over 200 steps followed by cosine decay down to zero, and clip the global gradient norm to 1.
Supervised TineTTunng After vocab extension we move onto the SFT step which conditions the model for NLU tasks and instruction following using the sampling and loss masking method described in Section 4.2 We fine-tune the model for 6144 steps with batch size of 128, max LR of 3e-5 which warms up over 200 steps followed by cosine decay down to zero, prior-regularizatioo applied to all non-embedding parameters with coefficient 0.5, and clip the global gradient norm to 1.
5

---

DPH Alignment Using the sampling method described in section 4.2 we jointly learn DPH rewards and perform cDPO alignment. The goal here is to gently push the model towards producing preferable outputs without compromising the model's 's reasoning abilities, and the priority is to attain the highest validation metrics from the DPH rewards. This requires balancing the two objectives, and as such we introduce weighting parameters a1 02 to our final joint objective in 7 where LDPH is either LeppDPH or CoODDPH. We find a1; a2 = -1 to be a good balance between DPO and DPH in our experiments.
L (2, Yw Ut, Twa ri) = an CeDroo (2, Mue v1) + 02 O2ZDDu(Fw:1)
(7)
We align the model for 23,040 steps with a batch size 64 pairs, a max LR of 3e-6 which warms up over 200 steps followed by cosine decay down to 3e-7, prior-regularization applied to all parameters with coefficient 0.5, and clip the global gradient norm to Following the optimal DPO parameters for OpenHermes-76-22.. [20] we use 3 0.6 and chose cDPO 0.25 and DPH 0.1 for regularisation. Additionally, we apply dropout with p = 0.1 to the outputs of the pooler.
4.5 Compute Resources
All fine-tuning was performed using an NVIDIA A100 SXM4 80GB GPU on compute cluster, with jobs allocated 24 cores and 160GB of memory. Each checkpoint is saved in FP16 format which consumes about I.IGB of storage; and the datasets require minimal storage space.
For vocab extension we train for 4096 steps with an average of 7.99 seconds of compute per step which translates to about 9 hours. For supervised fine-tuning we train for 61 14 steps with an average of 9.26 seconds of compute per step which translates about 16 hours. For DPH alignment we train for 23040 steps with an average of 7.21 seconds of compute per step which translates to about 46 hours. The DPH ablations with our models use about 140 hours of compute, and the Qwen ablations use about 60 hours of compute. In total, we used approximately 270 hours of A 100 compute to train our models and collect the results included in our paper. We used additional compute for preliminary tests and fixing bugs for silently failing experiments although this wasn't 't tracked.
Results
5.1 Evaluation Methodology
As described in Section 4 we use NLU, commonsense reasoning and reading comprehension tasks to measure model capabilities, while the instruction following and auxiliary tasks are used to provide additional training signals. For the NLU tasks evaluate on the test set of GLUE, providing average scores both with and without WNLL For reading comprehension we evaluate on the RACE test set. For commonsense reasoning tasks, evaluating on the test sets of ARC and OpenBookQA and the validation sets of HellaSwag, WinoGrande, BoolQ and PIQA, which brings our evaluations in line with other models.
follow the LM Evaluation Harness [16] implementations of these
For vocab extension and SFT checkpoints we obtain model predictions from the completions with the highest scoring log-probabilities. For the DPH checkpoints we report metrics for both log- probability predictions (Oursppo) and predictions chosen from the DPH rewards (Ourspen). We use the SwiGL.U-based pooler with the separable objective function for all our experiments as we found this combination to perform best overall as shown in Section 5.2.
5.1.1 Natural Language Understanding
Our results for NLU performance are included in Table I Note that the results for GPT-1 BERT 21 are from sub-task specific fine-tunes.
30 and
It is unsurprising that our model does not outperform BERT Large even though it has more parameters; this is likely due to BERT's task specific fine-tunes comparison to our model which was jointly trained on several tasks. Despite this our instruction following DPH model achieves a 2.2% higher average GLUE score compared to task- sask-specific GPT-1 fine-tunes and manages to attain the highest overall accuracy and macro-average on RTE and STS-B respectively.

---

Table 1: Comparison of GLUE performance. Dashes represent unpublished results. Note that the
Spearman correlation for Vara is misleading and caused by predicting "0" for all test samples.
MNLI
QQP
QNLI
SST-2
CoL.A Murr
STS-B
MRPC
RTE
Score
WNLI
Score
System
Tokens
Params
Are
WNLLI
Out
100B
551M
34.1/34.7
28.2/42.9
50.2
58.0
0.9
-0.9/9922
69.4/57.4
50.9
42.8
34,9
41.9
OurSSeT
100B
551M
73.6/75.0
39.1/82.8
81.4
90.8
22.7
80.6/92.4
80.6/75.2
71.4
72.0
38.4
68.2
OurSpra
551M
78.8/80.2
65.6/85.6
93.3
36.5
83.7/94.4
83.9/79.1
73.9
77.0
37.7
72.7
OurSppH
1008
+19M
80.0/80.6
65.8/85.3
87.5
94.0
43.8
85.3/93.0
85.5/80.2
75.3
78.6
46.6
75.0
GPT-1
32B
82.1/81.4
70.3/
82.0/80.0
82.3/
56.0
72.8
BERT
128B
110M
84.6/83.4
71.2/ -
90.5
93.5
52.1
-885.8
88.9/
66.4
78.3
BERT
128B
340M
86.7/85.9
72.1/89.3
92.7
94,9
60.5
87.6/86.5
89.3/85.4
70.1
82.5
65.1
80.5
5.1.2
Commonsense Reasoning
Our results for commonsense reasoning are summarized in Table Note the Pythia [6 and TinyLlama models were fine-tuned for any specific task but received significantly more pre-training and have much higher parameter counts.
4
Table 2: Comparison of accuracy the GPT4All test suite.
System
Tokens
Params
HellaSwag OpenBoookQA
WinoGrande ARC-Challenge
ARC-Easy
BoolQ
PIQA
Average
Oursyea
36.93
28.60
26.19
25.67
65.39
OursSEr
100B
551M
42.59
45.20
55.01
35.84
47.01
76.24
69.37
53.04
Ourspeo
100B
551M
44.83
52.40
57.38
39.76
53,54
79.08
72.36
57.05
Oursmm
100B
+19M
59.36
57.40
59.12
56.82
78.81
68.77
Pythia-1 1.0
300B
1.1B
47.16
31.40
53.43
27.05
48.99
60.83
69.21
48.30
Pythia-1.49 1.48
300B
1,5B
52.01
33.20
57.38
28.50
54,00
63.27
70.95
51.33
TinyLlama
1.1B
59.20
36.00
59.12
53,25
57.83
73.29
$2.99
With SFT alone
able to attain comparable performance to TinyLlama using half as many parameters, and when applying DPH alignment we achieve a 7.2% increase over the TinyLlama average score and the highest accuracy in 5 of the 7 tasks.
5.1.3 Reading Comprehension
Our results for reading comprehension are included Table from a RACE specific fine-tune, and the results for LLaMA 39
The results for GPT-1 were taken were zero-shot without fine-tuning.
Table 3: Comparison of accuracy the RACE
System
Tokens
Params
RACE-middle
RACE-high
Weighted Average
Oursyheah
100B
551M
26.0
24.6
25.0
Oursgrr
100B
551M
56.1
52.9
53.8
Ourspro
100B
551M
65.9
59.8
61.6
Ourspret
100B
+19M
66.9
60.6
62.5
GPT-1
32B
117M
62.9
57.4
59.0
LLaMA 7B
IT
6.7B
61.1
46.9
$1.0
LLaMA 13B
IT
13B
61.6
47.2
$1.4
Our SFT bascline achieves a higher average accuracy RACE compared with the non fine-tuned LLaMa models but cannot match the accuracy of the RACE specific GPT-1 fine-tune; however after alignment our model attains 3.5% higher average over GPT-1 while still maintaining excellent scores on other tasks using the same model weights.
5.2 Ablations
5.2.1
Pooling Head Function and Objective Choice
We ablate over the three pooling head and two objective function choices. We perform alignment for 7680 steps and report the validation scores in Table
For both separable and contrastive objectives the SwiGLU pooler performs best on the three bench- marks, and for both GLUE and RACE the separable objective performs best overall. However during these experiments we discovered that contrastive DPH was achieving higher scores than
7

---

Table 4: Comparison of DPH validation scores for different objective and pooler combinations.
Objective
Pooling Funetion
Add. Params
GLUE
GPT4AII
RACE
HellaSwag
WinoGrande
PIQA
Separable
Identity
56.86
53,20
Separable
BERT Style
2.4M
75.13
55.86
56.62
45.84
52.17
64.69
Sepurable
SwiGLU FFN
19M
75.19
57.14
57.60
48.72
53.35
64.96
Contrastive
Identity
1536
74.99
57.66
$4,09
50.93
53.83
66.87
BERT Style
57.07
54.62
Contrastive
SwiGLU FPN
19M
74.04
58.28
55.95
51.38
55.80
separable DPH for specifically the sentence completion style tasks like HellaSwag, WinoGrande and PIQA. We hypothesise this is caused by situations where multiple completions to given prompt may be plausible even though there is only one 'gold' answer, and such the model benefits from maximising the relative reward margin with the contrastive objective rather than optimising absolute rewards with the separable objective.
5.2.2 Task Specific Heads
By taking the DPH checkpoint and freezing all backbone parameters is possible to learn task specific heads and pooling functions for different downstream tasks at the cost of only 19M parameters per task. We train new heads for the three task groups and plot the confusion matrix each head for each task average in Table 5 We further fine-tune for an additional 7680 steps on each task group using the same training setup as DPH alignment.
Table 5: Confusion matrix comparing validation scores for alternate heads.
Benchmark
Bascline Head
GLUE Head
GPT4All Head
RACE Head
GLUE
12
76.36
76.20
GPT4AII
60. 19
60.13
60.29
60.24
RACE
64.17
64.05
64.48
64.43
Unsurprisingly the GLUE and GPT4AIl heads achieve the highest scores for GLUE and GPT4AIl benchmarks respectively, however the GPT4AIl head manages outperform the RACE head the RACE benchmark. We hypothesise this may be due to the inclusion of multiple choice QA and reading comprehension tasks GPT4AIl which may prove better training signals than the RACE training data alone.
5.2.3 Frozen Model Ablations
Our final experiments involve exploring the behaviour of DPH when applied to frozen language models without further fine-tuning of the original model weights. We experiment using the Qwen 1.5 model family (2) and train only the pooler and reward head weights, reporting results Table 6 We identical training setup DPH alignment but disable dropout due the low number of trainable parameters.
Because the model backbone and embeddings remain frozen during alignment the 'Log' scores represent the model's pre-trained (or fine-tuned) capabilities. When observing the difference between the Log scores of the 0.5B Qwen models it is evident that the fine-tuning and alignment used to transform the pre-trained model into the "chat" model resulted degraded performance across the three tasks. This phenomenon is less apparent for the 1.8B models, and actually results in higher GLUE scores for the "chat" variant of the model. This further confirms the hypothesis that alignment harm the reasoning capabilities of smaller language models.
Table 6: Comparison of validation scores calculated using the log probabilities from the vanilla model checkpoints and reward scores produced by the trained Direct Preference Heads.
System
GLUE GPT4AUl Log
RACE Log
GLUE DPH
GPT4AB DPH
RACE DPH
Qwenl 5.000.B
41.94
$3.11
51.38
45.69
48.52
41.21
Qwen1 1 S 1-0.5B-Chat
39,82
49.70
50.32
48.99
49.72
46.90
Qwen 1.5BB.8B
47.03
62.53
68.14
59.18
51.61
46.56
Qwenl 5-1 8B-Char
53.85
61.69
67.47
62.38
54.47
53.33

---

For all models DPH is consistently able to attain higher scores on the GLUE tasks compared to the log probabilities produced by the language modelling head, but the opposite is observed for RACE which suggests the hidden states produced by the frozen backbone do not contain rich enough features for long range modelling tasks such as reading comprehension. We also observe the "chat" variants produce higher task scores for DPH than the non-chat variants which we hypothesise is a result of the authors' fine-tuning with the Chat-ML format which lead to the models' greater understanding of message structure and therefore improved hidden state aggregation for the final end message token.
becomes evident that the pooling function and reward head exhibit slower convergence when the model backbone is frozen. This observation further supports our hypothesis in Section [4.3] indicating that the hidden states generated by the models are are initially sub-optimal and that further fine-tuning is necessary to optimize these hidden states to achieve the best features for DPH.
When we combine these findings with those presented in Section 5.2.2]
Discussion
6.1
Future Work
As shown in the results section, DPH is capable of learning to assign higher rewards to preferred outputs and lower rewards dispreferred outputs which implies the pooling function learns rich features with respect to
pairs. We believe that it would be possible to also extract additional information from the output of the pooling function to detect finer grained signals such as helpfulness, humour, creativity, toxic content, etc. Such an approach has been explored by ArmoRM 41 which was first trained with several reward heads to reflect different types of preference scores and then a gating system to chose the best combination of heads based on the context. We believe this technique can be integrated into the DPH process facilitate context-dependent re-ranking.
6.2 Limitations
The main benefit of DPH being its ability to perform alignment without directly effecting the model's output distribution is also its main limitation: unlike other alignment techniques which can help prevent the model generating harmful outputs, DPH is only capable of detecting harmful outputs. Although we do include DPO alignment in our experiments to reduce the likelihood of harmful outputs, DPH does not require such model aligmment to function, which shifts the responsibility of rejecting harmful outputs to the end user or service provider.
From computational perspective, DPH reformulates alignment as a re-ranking process, which necessitates sampling multiple candidate responses during inference. This approach requires more computational resources and memory compared to generating a single response. However, modern hardware used for LLM inference is often underutilized by smaller language models when generating singular responses in an autoregressive manner. By leveraging this unused compute capacity, we can generate multiple candidate responses in parallel when applying DPH to smaller models, without significantly increasing latency.
6.3 Conclusion
In this paper we introduced Direct Preference Heads, a novel form of language model alignment which is performed at inference time to prune candidate completions for given prompt. Unlike other alignment techniques which coerce the model into generating human preference aligned outputs, DPH instead produces reward scores for candidate outputs without affecting the actual generation process and therefore avoids the issue of RLHF leading to degraded performance when applied to smaller language models. We formulated two loss functions for DPH and find strong connections to Conservative DPO, implying that DPH is robust to label noise and can be tuned to specific confidence margin. Finally, we evaluated our methods on a number NLU, commonsense reasoning and reading Comprehension tasks and found that DPH is able to consistently outperform both our SFT baseline and multiple publicly available language model checkpoints of varying size and training volume.

---

Broader Impacts
As with all language modelling systems we cannot guarantee all responses produced by our models are factually correct nor can we guarantee that they are safe and free from harmful content. Our work focuses on creating system that helps filter out incorrect and harmful messages by scoring candidate outputs, but as with all alignment techniques our models may be susceptible to so-called 'jailbreaks' which can coerce the model into incorrectly assigning a higher score to less desirable content. To maximise safety DPH should be implemented alongside other safety guardrails such as Llama Guard 19 when used for publicly facing chat systems, and we intend for our provided model checkpoints to be used for reproduction of results and further research in the field of alignment.
References
[1] Yuvanesh Anand, Zach Nusshaum, Brandon Duderstadt, Benjamin Schmidt, and Andriy Mulyar. GPT4A1I: Training an assistant-style chatbot with large scale data distillation from GPT-3.5-Turbo. https://
2023.
[2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijic Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint 2rXiv:2309. 16609, 2023.
[3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 05862, 2022.
[4] Edward Beeching. Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert. Nazneen Rajani. Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open LLM leaderboard, https: httts://huggingfaae co/ aapaceee/uugginggaceeteeee aact:eggggnnnnneea//aaaaae open_ 11m_ leaderboard 2023.
[5] Aibek Bekbayev, Sungbae Chun, Yerzat Dulat, and James Yamazaki. The poison of alignment. arXiv preprint arXiv:2308. 13449, 2023.
[6] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kylc O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of the International Conference on Machine Learning, pages 2397-2430, 2023.
[7] Yonatan Bisk, Rowan Zellers. Jianfeng Gao, Yejin Choi, et al. PIQA: reasoning about physical com- monsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7432-7439, 2020.
[8] Ciprian Chelba and Alex Acero. Adaptation of maximum entropy capitalizer: little data can help a lot. Computer Speech & Language. 20(4):382-399, 2006.
[9] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: exploring the surprising difficulty natural yes/no questions. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2924-2936, 2019.
[10] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schocnick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning challenge. arXiv preprint arXiv: 1803.05457. 2018
[11] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek. Jacob Hilton, Reūchiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
[12] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan I Liu, and Maosong Sun. Ulrrfeedback: boosting language models with high-quality feedback. arXiv preprint 1rXiv:2310.01377. 2023.
[13] Zihang Dai. Transformer-XL XL: attentive language models beyond a fixed-length context. arXiv preprint 19Xiv.1901.02860. 2019.
10

---

[14] Hal Daumé III. Frustratingly easy domain adaptation. In Proceedings of the Annual Meeting of the Association of Computational Linguistics, pages 256-263, 2007.
[15] Ronen Eldan and Yuanzhi Li. Tinystories: how small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07755 2023.
[16] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang. Laria Reynolds, Eric Tang, Anish Thite. Ben Wang. Kevin Wang. and Andy Zou. A framework for few-shot language model evaluation. https://doi org/ 10. 5281/ 5371628 2021.
[17] Maarten Grachten and Carlos Eduardo Cancino Chacón. Strategies for conceptual change in convolutional neural networks. arXiv preprint arXiv:1111001634, 2017.
[18] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt, Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
[19] Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungla, Krithika Iyer. Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. LLaMA guard: LLM-based input-output safeguard for Auman-Al conversations. arXiv preprint urXiv:2312.06674 2023.
[20] kashif, edbeeching, lewtun, lvwerra, and osanseviero. Preference tuning ilms with direct preference op- timization methods. Github.
http:///w...aoougggggnaa///a///a htte///lob//mmm////eeee/ haabb//man///ffffeeeeeee - -tuning
md
2024.
[21] Jacob Devlin Ming-Woi Chang Kenton and Lee Kristina Toutanova. BERT: pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of the Annual Conference of the North American Chupter of the Association for Computational Linguisticx: Human Language Technologies. pages 4171-4186, 2019.
[22] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang. and Eduard Hovy. RACE: large-scale ReAding comprehension dataset from examinations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 785-794, 2017.
[23] Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. htttps://tttss://uggg ing co/
[pen=-Orca/OpenDrcca] 2023.
[24] Stephanic Lin, Jacob Hilton, and Owain Evans. TruthfulQA: measuring how models mimic human falschoods. In Annual Meeting of the Association for Computational Linguistics, pages 3214-3252, 2022.
[25] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2381-2391, 2018.
[26] Eric Mitchell. A note on DPO with noisy preferences relationship to IPO. htttps:://rrimmiichellll ai/cdpo. pdf 2023.
[27] Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, al. Abstractive text summarization using sequence-tor to-seueee rnss and beyond. arXiv preprint 1rX00:1602.00023, 2016.
[28] OpenAL. Chat markup language. https://github.cc http://wwww......o.//////aa//aaa http:////ttthhhooo///////a///pps Viccee2288ee33556552555566666363
cht////ccaml....m aehaamml.m. md]
[29] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-277 277444, 2022.
[30] Alec Radford and Karthik Narasimhan. training. :/ttps://cdn openai. inderstandingg_papee pdf 2018
Improving language understanding by generative pre- hat.auuuuuuuuuuueervvee/d/aagggu
[31] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning. Stefano Ermon, and Chelsea Finn. Direct preference optimization: your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-5533 537411, 2024.
[32] Pranav Rajpurkar. Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2383-2392 2016.
11

---

[33] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what don't It know: unanswerable questions for SQuAD. In Proceedings of the Annual Meeting the Association for Computational Linguistics, pages 784-789, 2018.
[34] Siva Reddy, Danqi Chen, and Christopher D Manning. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019.
[35] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99 100,,, 2021.
[36] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithus. arXiv preprint arXiv:1707.0633 2017.
[37] Noam Shazeer. GLU variants improve transformer. arXiv preprint arXiv:2002.05222, 2020.
[38] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford Alpaca: instmaction-follovii LLaMA model. URL https://github wom/tatsu-lab/sttrgge alpaca, 1(9). 2023.
[39] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397 2023.
[40] Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804.07461, 2018.
[41] Haoxiang Wang, Wei Xiong, Tengyang Xie, Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-expert. arXiv preprint arXiv:2406. 128455, 2024.
[42] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, 2019.
[43] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv: 200.:2240 2024.
[44] Liu Ziyin, Zhikang T Wang, and Masahito Ueda. LaProp: Separating momentum and adaptivity in Adam. arXiv preprint ar arXiv:2002 04839, 2020.
2020.
12

---

era + which is equal to zero for E (0, 0.5) the point FA log 1-c A Appendix Theory
A.1 Full Proof of Theorem
We can prove Theorem by examining the partial gradients with respect to the rewards.
a orw
SapDPH(TTr: rz)
(8a)
e:w +
1 e +1
"SspDen(Ties ri)
(8b)
From equations Sa and 8b we find that the partials gradients are both equal to zero at the points Tw = log 1- and r log I-c respectively. is also interesting to note that log² 1-e + log 1-r
which implies the positive and negative rewards will converge to an equal distance from
@ (Ts = (e"= + 1)2 0
e"'
(9a)
#² लrद
[TepDPa(Fm: (e" + 1)2
e
(9b)
If derive the second derivatives for the rewards, shown in equations 9a and 9b find that they are both strictly positive for all values of rw and ri which implies that Separable DPH is convex with respect to the rewards.
A.2 Full Proof of Theorem 2
We can prove Theorem by examining the partial gradients with respect the rewards.
e eti e et
CONDPH (Far: ri) c- Drw
(10a)
COnD (T dr:
8
(10b)
e e'x
From equations 10a and 10b we can see symmetry emerge, where the partial gradients with respect the preferred logits are equal and opposite to the partial gradients with respect to the dispreferred
logits. If we reparameterise the loss function such that = we can derive the following partial derivative
0 ar
CCoDDPH(Fw: rt)
1
(11)
we derive the second derivative of the Contrastive DPH objective function with respect to the reward margin r'A obtain the following formula
e'
8² 0r3 (Fumbra(Fw; (e 1)2
(12)
which is strictly positive for all values of rа. and with respect the reward logits frames Contrastive DPH as convex optimization problem with the additional properties of guaranteed convergence fixed margin for all (0, 0.5).
A.3 Illustrative Loss Landscape
We provide an illustration of the loss landscapes to give a visual comparison of how our objective functions 'pull' rewards towards the optimal margin bounds.
13

---

(a) Loss landscape of Separable DPH
(b) Loss landscape of Contrastive DPH
Figure The loss landscapes of the DPH functions. The and green points represent the rewards assigned to preferred and dispreferred answers, the vertical lines represent the direction and magnitude of reward gradients, and the blue area represents the optimal margin parameterised by c.
B Appendix Pseudocode
B.1 Decoupled Prior Regularization
Rather than optimizing the auxiliary loss term 1 ||0 -0--0 0 we can follow the procedure decoupled weight decay and implicitly include prior regularization step within the optimizer update function. The pseudocode for this is included below:
Algorithm 1 Decoupled Prior Regularization Update Function
learning rate
f regularization coefficient
0, Oref current, initial parameters
0 BB00 Oret)
Prior regularization step
+ optimizer update step
Normal optimizer update
14

---

C Appendix Data
C.1 Dataset Mixes
C.2 Data Licences
Dataset
License
GLUE CoLA
No License
GLUE MILI
Multiple (OANC, CC BY-SA 3.0)
GLUE MRPC
MSR-SSLA
GLUE QNLI
CC BY-SA 4.0
GLUE QQP
Other
GLUE RTE
No License
GLUE SST-2
No License
GLUE STS-B
Multiple (CC BY-SA 3.0, CC BY-SA 4.0)
GLUE WNLI
CC BY 4.0
HellaSwag
MIT License
OpenBookQA
Apache-2.0
WinoGrande
CC-BY
ARC
CC BY-SA 4.0
BoolQ
CC BY-SA 3.0
PIQA
AFI-3.0
RACE
Other
SQuAD V2
CC BY-SA 4.0
MMLU
MIT License
Tiny Stories
EDLA-Sharing-11
CNN-Dailymail
Apache-2.0
CoQA
Multiple (CC BY-SA 4.0, MSR-LA, Other Apache)
Alpaca Cleaned
CC-BY-4.0
OpenOrca
MIT License
OpenOrca Binarized
Apache-2.0
UltraFeedback Binarized
MIT License
Note that three of the GLUE tasks have no license specified on their homepages nor within their publications: CoLA claims their dataset falls under "fair use," no concrete license can be found for RTE nor its pre-cursors, and SST-2 does not specify license.
C.3 Prompt Templates
For brevity, we only include the prompt templates of the tasks use for evaluation. All other prompt templates are listed within the code repository.
C.3.1 GLUE
GLUE CoLAA
System
User
Assistant
Below is an instruction that describes a task. Write response appropriately completes the request using the provided answer options.
Given the following sentence, answer the question "yes"
[noo [yos]
Sentence: [[sentence])
Question: Does this sentence make sense?
15

---

GLUE-MNLI
System
User
Assistant
is instruction deseribes task. Write response that appropriately completes the request using the provided unswer options.
a premise statement a hypothesis statment, respond with "Truc" if the premise entails the hypothesis, respond with "False" premise contradicts hypothesis, respond with "Neither" if the statements are neurtral.
[[Trie Neither! False])
Premise: [[rremise]]
Hypothesis: [[hypothesis]
Question: True, False or Neither?
Answer:
GLUE MRPC
System
User
Assistant
Below is an instruction that describes task.
Given the following sentences, answer the question with "yes" or "no".
[no yes
Write response that appropriately completes roquest using provided answer options.
Sentence I: [seettencel]] [)
Sentence 2: [sentence2]]
Question:
both sentences mean the thing?
Answer:
GLUE QNLI
System
User
Assistant
Below is an instruction that describes task.
Given the following sentences, answer the question with "yes" "no".
[(yes no))
Write response appropriutely completes the request using the provided answer options.
Sentence I: ([question)
Sentence 2: ([sentence)
Question: Does Sentence 2 correctly
swer Sentence 1?
Answer:
QQP
System
User
Assistant
Below is instruction that describes task.
Given the following sentences, the [no yes]] question with "yes" or "no".
Write a response that appropriately completes request using the provided answer options.
Sentence I: [[question1]]
Sentence [[question2)
Question: Do both sentences ask the
same question?
Answer:
RTE
System
User
Assistant
Below is an instruction that describes task.
Given the following sentences, answer the question "yes" "no".
[[yes no)
Write response appropriately completes the request using the provided answer options.
Sentence I: [[sentencel]]
Sentence 2: ([sentence2)
Question: both same thing?
the
16

---

GLUE SST-2
System
User
Assistant
Below is an instruction that describes task.
Given the following sentence, answer the question "positive" "negative",
I (negative positive])
Write response appropriately completes the roquest using the provided answer options.
Sentence: [[sentence]]
Question:
Is this sentence positive
negative?
Answer:
GLUE STS-B
System
User
Assistant
Below is an instruction that describes task.
Given the following sentences, answer the question number between
(01112131415)
Write response appropriately completes the request using the provided unswer options.
Sentence I: [[sentencel] }}
Sentence 2: [[sentence2])
Question: scale of how simi-
lar are Sentence 1 and Sentence 2?
Answer:
GLUE WNLI
System
User
Assistant
Below is an instruction that describes task.
Given the following sentences, answer the question with "yes" "no".
[no yes))
Write response appropriately completes the request using the provided answer options.
Sentence I: (eentencel])
Sentence 2: [sentence2]]
on information Sentence I. can concluded that Sentence 2 is true?
Question:
Answer:
C.3.2
Commonsense Reasoning
HellaSwag
System
User
Assistant
Below is instruction that describes task. Write response that appropriately completes request.
Continue the following sentence: "[[context]]"
((cnding))
OpenBookQA
System
User
Assistant
Below question. paired with multiple choices. Respond with the choice that correctly answers question.
Question: (fquestion_stem)
((abbe])). [[choice))
Choices:
[label[(]]). [[choice(0]))
I [[]]]e[[1]]] [[choice[1]]
[[label[2]]] {(choice(21))
[[labe]]3])]. [[choice[3)]
WinoGrande
System
User
Assistant
Below is an instruction that describes task.
Continue the following sentence: sentencee,reeii prefix]]" I]"
[[option]] [ [[sentence.sufff
Write response that appropriately completes request.
17

---

ARC
System
User
Assistant
question, paired multiple choices. Respond with the choice that correctly answers the question.
Question: [[question] )
[(lbbel) [[choice]] ]
Choices:
((label[0))). ((choice(0]))
[[labe][n]]] [[choice[n]]]
Answer:
BoolQ
System
User
Assistant
Below is instruction that describes task.
Given the following sentences, answer the question with "yes" or "no".
[no yes]]
Write response that appropriately completes request using provided options.
Background: [passsgee]
Question: [[guestion]
Answer:
PIQA
System
User
Assistant
Below is an instruction that describes a task. Write response appropriately completes the request.
Write a solution to the following sentence: "[guol]))"
[ssolution]] ]]
C.3.3 Reading Comprehension
RACE
System
User
Assistant
Below is question, puired with a background context multiple chnices. Respond choice that correctly answers the question.
Background: ([article)]
[loption]:
Question: ((question)
Choices:
A. [[option[]]]]
B. [[option[]]]] []]]
[[option (2]])
D. []ppioon3]]]
Answer:
18

---

D Appendix -Model Details
D.1 Pre-Trained Model
Our pre-trained model was developed in house for efficiency and takes advantage of techniques such RoPE, SwiGLU activations and Flash Attention. The model totals 551 Million parameters (including embeddings).
We initialise the embeddings from OPT-125m and use embedding tying for the language modelling head. Since our model dimension is 1536 while the embedding dimension is 768 the model contains an up-projection the first layer of the backbone and down-projection for the final layer. There a total of 18 transformer blocks in the model backbone which use pre-layer norm in the attention and FFN residuals. The attention blocks have 24 attention heads and we RoPE with base frequency of 500,000 for positional embedding, and the FFN block uses SwiGLU activation with an intermediate dimension of 4096. The context window of the model is 2048 tokens and the Transformer XL recurrent memory contains 2048 tokens which allows the model to use a sliding window size of up to 4096 tokens inference without any degradation.
The model was trained for approximately 100 billion tokens on the first 24 shards of The Pile. Each batch is constructed of 480 sequences of 2048 tokens each which are continuously sampled from the datasets shards using queues for the Transformer XL style pre-training method.
We use the LaProp optimizer [4 with 31 0.9, 32 = 0.95, max learning rate of 6e-4 which warms up over 2000 steps and cosine decays down to 6e-5, L.R-coupled weight decay of 0.1 and global gradient clipping with max norm of 1.
Each epoch of 256 steps takes hour and 59 minutes on 4x RTX A4500 GPUs. For the full 398 epochs (or 101888 888 steps) this comes out to around 790 hours or just under 33 days of training time (ignoring time for validation in ne-etween epochs and the end of training).
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We conduct variety of experiments and ablations which validate the claims made in the abstract. We make use of standard benchmarks in the field of language models and compare our method with results from smaller, comparably sized and larger models.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include limitations sub-section in discussion, and discuss the fact that as with all probabilistic models there is no guarantee the outputs will be correct. We also discuss the fact that models aligned using our method are still capable of producing unsafe outputs if the end user ignores the reward scores.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
20

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All theorems are numbered and include sketch in the paper's 's main body with full proofs included the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We extensively describe our methodology, utilised datasets, and hyperpa- rameters. We will also be publicly releasing the code with instructions to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
21

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open Access to Data and Code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: All data used for fine-tuning is publicly available Hugging Face, the code used to train the models and perform the experiments is publicly available, and an additional repository containing the code and reproduction instructions will be made available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
public/guides/ https//ooooeSmmmiiiiiineeee
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips.cc/publ::: gww.e//ggge//e/eeee httpc///bbbbcegggaeeeee//teteaaa Cttp///ooeeSSeieeeeeeeeeee www.eeeSSuuiiiiiionniiii wwwsssssiiiiyyooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Experimental details are described in the body of the paper, with further elaboration in the appendix, and source code to reproduce the experiments will be made publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do not provide eтror bars or statistical significance measures because it would be too computationally expensive to perform the entire training pipeline several times to collect alternate model checkpoints. Additionally, the GLUE evaluation server imposes rate limiting which adds further time constraints on repeating tests for alternate model checkpoints.
22

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the eπror bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe the compute requirements for all experiments in the main body of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics ttttps://neuripee: httpaa/ppp...ppplll///eeeeeadeee
Answer: [Yes]
The work conducted in this paper conforms to all aspects of the NeurIPS Code of Ethics. All datasets used for fine-tuning and evaluation permissively licensed or fall under fair use. All code assets were created by the authors and all libraries used are permissively licensed. No human subjects were involved in the creation of this paper. And our work has minimal risks of misuse.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
23

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We include broader impacts section at the end of our paper and describe potential pitfalls of using our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Although we are publicly releasing our model checkpoints they have an incredibly low risk of misuse as other significantly more capable and more accessible models exist. Additionally, the Hugging Face checkpoints do not support 'AutoModel" loading, which means a level of technical knowledge required to the model up, making it less accessible those outside of the ML research sphere.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for Existing Assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
24

---

Answer: [Yes]
Justification: All datasets used are explicitly cited (and web links are included for cases where a 'cleaned' version of the cited dataset is used) in the main paper body, and the license information is listed in the appendix. Some datasets do not have an explicit licence, but the original authors claim they fall under fair use.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, haperswithcode com/coo//oaoooooo datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The source code is very well commented in its original repo, and a new repo will be released containing further instructions specific to this paper.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Neither crowdsourcing nor human subjects were involved in our work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
25

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "qR3KY8QoPV", "reviewer": "Reviewer_ebEQ", "summary": "The authors propose an inference time method to align language models with human preferences without harming the model’s reasoning abilities. The method creates an auxiliary reward head that operates during inference to score potential outputs without changing the output distribution directly. The author validates approach by comprehensive evaluations on NLU, commonsense reasoning, and reading comprehension datasets.", "questions": "1. The paper would be more convincing if the author applied the proposed method to a larger-sized model and demonstrated a scaling trend. The use of only one model makes the effectiveness of the approach a bit underdetermined\n\n2. How does the proposed DPH method compare with other inference time algorithm such as rejection sampling? \n\n3. How do you choose baselines when reporting performance? For example, Table 1 reports the model performance on BERT, an early-stage masked language model, instead of other competitive baselines.", "limitations": "The authors addressed all the limitations in the conclusion section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Theoretical Insight: The paper provides a theoretical analysis, connecting DPH to cDPO, and providing proofs to support the convexity and effectiveness of the proposed loss functions.\n\n2. Novel Approach: The paper introduces a novel approach allowing for preference-aligned model fine-tuning without directly affecting the output distribution, potentially reducing negative side effects like hallucination and preserving the model’s original reasoning capabilities.", "weaknesses": "1. Inference Time Overhead: The Direct Preference Heads method involves using language models to generate multiple candidate responses, which must then be evaluated for selection. This process is less efficient compared to traditional fine-tuning methods due to the additional computational steps required.\n\n2. Dependence on Initial Sample Quality: The effectiveness of the generation-then-reranking approach highly relies on the quality of the initial responses generated. If the sampled responses are of low quality, DPH is unable to enhance the output, as it can only rerank the given candidates.\n\n3. Limited Evaluation Scope: The evaluation of the DPH method is restricted to a single model of 550M parameters. It is not clear how well the method would perform when scaled to larger models, as the results might not be consistent across different model sizes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Addressing Weakness 1:**\nDPH does indeed have a higher inference-time overhead than other alignment methods which aim to produce optimal completions in a zero-shot manner. However we frame DPH as being a suitable method for smaller language models which, as cited in the paper, may be harmed by other RLHF methods. The inference overhead of such smaller language models often do not not saturate the available compute or memory meaning multiple candidates can often be produced in parallel with minimal increase in latency.\n\n**Addressing Weakness 2:**\nAlthough the quality of produced responses depends on initial sample quality, we make use of high quality training datasets which have been used to train SOTA open-weight open-dataset language models. It is, of course, possible that all candidates generated by the language model are low quality, however through tuning generation hyper-parameters and making use of advanced sampling techniques such as typical sampling or contrastive search it is possible to coerce the model into producing a candidates with high variation from which it is likely some candidates will indeed be high quality. It is possible for DPH to be combined with other alignment methods if desired to further increase the quality of the most probable completions, but this paper primarily aims at providing the foundations for DPH as an inference time reranking method.\n\n**Addressing Weakness 3:**\nSee below.\n\n**Addressing Question 1:**\nWe do evaluate the proposed method with larger sized models by training a reward head and pooling function on 4 frozen Qwen 1.5 models (0.5B, 0.5B-Chat, 1.8B, and 1.8B-Chat) and report the results in section 5.2.3 and Table 6.\n\n**Addressing Question 2:**\nRejection sampling is actually very similar to reranking with DPH, with the only difference being rejection sampling typically uses a separate reward model to evaluate the candidate completions whereas our method combines the generation and reward modelling into a single LM. Additionally, rejection sampling is often used during training to generate new candidates to improve the policy through methods such as PPO, while DPH is only intended for inference time reranking.\n\nIf we were to use rejection sampling during training we would run into \"reward hacking\" issues where the model becomes overconfident in its reward assignments on self-generated completions which would be analogous to mode collapse in GANs; this does not mean that rejection sampling cannot be used with DPH models as the reward hacking issue will be much less prevalent if a separate reward model is used for ranking. This is, however, antithetical to the lightweight nature of our method and the intention to minimise the number of concurrent models needed to perform fine-tuning and alignment, and as such we did not explore rejection sampling and other similar RLHF methods as part of our training pipeline.\n\n**Addressing Question 3:**\nOur choice of baselines were partially dictated by the tasks themselves, and partially by the availability of results. For example with GLUE we picked BERT as it is still common to finetune a BERT model for NLU tasks which makes it an excellent choice for comparison. Additionally, although BERT has fewer parameters, the results were produced by task-specific fine-tunes which we believe makes it a valid and competitive baseline to compare against our 551M multi-task fine-tunes. We also included GPT-1 to compare our model with task-specific fine-tunes of a causal language model, which we chose due to the availability of results and its inclusion in the original BERT paper. GLUE also imposes rate-limiting on the evaluation server which adds time constraints to re-evaluating newer models on the test set.\n\nFor the commonsense reasoning tasks we followed the evaluation used by TinyLlama and included TinyLlama and two Pythia models, all of which had significantly more parameters and pre-training albeit without fine-tuning.\n\nAnd for the reading comprehension tasks we included the results from GPT-1 to compare our model with a smaller but task-specific fine-tuned model, and the results from LLaMa 7B and 13B to compare our model with larger but non fine-tuned models.\n\nThese choices may seem odd, but we wanted to include verified results from popular models which were performed externally and posted publicly to reduce evaluation bias due to factors such as prompt selection, or quirks in the evaluation pipeline which may favour certain models or token vocabularies when computing log probabilities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "njxF7n3FpZ", "reviewer": "Reviewer_zurN", "summary": "Paper proposes Direct Preference Heads, which learns a reward prediction head using a pretrained model, without affecting the model's output distribution.", "questions": "See weaknesses", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Comprehensive evaluation\n- paper presented experimental results across a wide range of tasks (NLU, commonsense reasoning, reading comprehension, etc. ), and compared the proposed method against a range of different baselines (Pretrained model, SFT, DPO, etc. )\n\nGood presentation\n- results are cleanly presented, training objectives are clear\n\nSignificance\n- results showed that the proposed method outperforms comparable baselines in most presented tasks", "weaknesses": "Some missing/confusing parts about the proposed method:\n- How are the learned reward predictions used? Some parts of the paper seems to suggest that multiple responses are sampled from the model, and the reward head is used to rank the samples? I think this is an important component of how the proposed method would be used, so it would be great to further elaborate on this procedure, e.g. how many samples?\n- Some parts of the paper seems to suggest that the proposed approach only learns a reward prediction head on top of an existing model without modifying the original model (e.g. in abstract, \"... without directly affecting the output distribution of the language modeling head ...\"), but other parts of the paper stated that it is better to finetune the entire model when learning to predict rewards (e.g. Section 4.3). However, the stated reason for why DHP might be better than other RLHF techniques like DPO is that further alignment (past SFT) can hurt model performance. If DHP also requires further finetuning, what is the intuition for why DHP performs better than DPO?\n\nRelated works\n- Paper introduces PPO, DPO and cDPO. However, there has been a large number of recent works in RLHF beyond these papers. It would be great to present a more complete related works section with a more comprehensive survey of existing work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Addressing Weakness 1:**\nThe learned reward predictions are to be used to rerank candidate competitions at inference time, similar to rejection sampling. The number of samples to rank, however, is completely situational and can be determined by factors such as compute capability, memory availability and the maximum latency deemed acceptable before serving the highest rank completion to the end user. In general, the more candidates generated the more likely it is for a higher quality response to be produced. It should be pointed out that the time needed to rank the candidates is negligible as the rewards are computed alongside generation, meaning the compute, time and memory requirements are determined by generation procedure alone.\n\nThis does, however, bring up an excellent direction for further research to determine if and how an early stopping heuristic may be employed to optimise the number of candidates produced.\n\n**Addressing Weakness 2:**\nDPH can indeed be employed without fine-tuning any of the model's backbone weights, and we do include experiments performing this on 4 Qwen models in section 5.2.3. However for DPH to perform best it does require further fine-tuning model weights, and we facilitate this through the usage of both prior regularisation and regularised cDPO with a large beta penalty and epsilon value to discourage the policy model from diverging from the SFT model while learning to produce rewards.\n\nIt would be possible to replace cDPO with KL divergence, for example, to achieve a similar effect while completely decoupling the learning of the reward head from any form of RLHF or alignment method. We opted to use cDPO because we found it allowed the model to improve further (within the divergence and confidence bounds set by beta and epsilon) without degradation which further increases the likelihood of generating improved candidates for reranking at inference time.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gnk3Spu6vr", "reviewer": "Reviewer_LdN5", "summary": "This paper introduces Direct Preference Heads (DPH), a novel method for aligning language models (LLMs) with human preferences at inference time. DPH works by adding an auxiliary reward head to the LLM that learns to predict human preference scores for generated outputs. This allows the model to self-evaluate multiple candidate outputs and select the highest-scoring one, effectively pruning undesirable responses.\n\nThe authors argue that DPH offers several advantages over traditional Reinforcement Learning from Human Feedback (RLHF) methods like PPO and DPO:\n\n* Inference-time alignment: DPH aligns the model at inference time, avoiding the potential degradation of reasoning abilities often observed with RLHF during training.\n* Lightweight: DPH requires only a single model to produce both responses and rewards, unlike RLHF which typically involves multiple models.\n\nThe paper presents two objective functions for DPH, separable and contrastive, and demonstrates their connection to Conservative DPO. Experiments on GLUE, GPT4All, and RACE datasets show that DPH consistently outperforms both supervised fine-tuning (SFT) and DPO alone.", "questions": "* The paper uses different sampling strategies for SFT and DPH. It would be helpful to discuss the rationale behind these choices and explore the impact of different sampling strategies on DPH performance.\n\n* Sampling multiple candidate outputs at inference time can be computationally expensive, especially for larger LLMs. The paper doesn't explicitly address this cost, which could be a practical concern for real-world applications. Can you elaborate more on this in the paper?", "limitations": "The paper focuses on a 551M parameter model. It would be valuable to see how DPH scales to larger models, as RLHF is known to be more effective for larger models. Comparing DPH to RLHF-aligned larger models would provide a more comprehensive evaluation.\n\nAlso RLHF baselines on top of DPO need to be added.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Novel approach: DPH offers a new perspective on LLM alignment by focusing on inference-time pruning rather than modifying the generation process itself.\n* Theoretical grounding: The paper provides a theoretical analysis of the objective functions and their relationship to cDPO, demonstrating robustness to label noise.\n* Strong empirical results: DPH consistently outperforms baselines on various tasks, showcasing its effectiveness.", "weaknesses": "* Lack of statistical significance: The paper does not report error bars or statistical significance measures, making it difficult to assess the robustness of the results.\n\n* Missing the comparison to standard RLHF methods: the paper does not include comparison to the PPO baseline, which is the most commonly used technique.\n\n* The authors do not discuss other inference-time alignment techniques that exist in the literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Addressing Weakness 1:**\nThe paper does not include statistical significance measures as this would require performing each stage of the training pipeline multiple times which is computationally costly and we opted to use our compute budget to perform ablations with different hyperparameters and using different models.\n\n**Addressing Weakness 2:**\nComparing to a PPO baseline would have vastly increased the scope of our work as it would also involve steps such as SFT sampling, collecting human labels and training a reward model, or using pre-labelled datasets to train a reward model, and then performing one or more rounds of PPO with rejection sampling. This would add a plethora of choices in the alignment pipeline which would need to be ablated over and further add to the computational cost of the research. Our goal with this paper was to lay the foundations of DPH as a novel method which could be used as a baseline for further research.\n\n**Addressing Weakness 3:**\nThere are a variety of inference-time alignment techniques, varying from prompt based approaches to chain of thought based self evaluation to activation steering. All these methods, however, are typically performed on significantly larger language models which on account of their greater size would likely outperform our models even without these ad-hoc alignment methods. An alternative would be reproducing one or more of these methods with smaller models, but that would require significant time and compute.\n\n**Addressing Question 1:**\nTo maximise compute throughput we use the Transformer-XL style sampling strategy, which also allows for performing SFT on samples which are larger than the LM context window (such as multi-turn conversations) due to the recurrent memory mechanism. Additionally, sampling a task multiple times was introduced as a way to balance the rate each task was sampled from but ultimately all tasks remained balanced.\n\nHowever, for DPH alignment we switched to typical on task sample per sequence in the batch as the batch needed to be constructed from positive and negative sample pairs. This isn't really possible using the Transformer-XL as we need to exactly match the positive and negative pairs one-to-one which becomes difficult when packing multiple samples of varying lengths into a single sequence.\n\n**Addressing Question 2:**\nSampling multiple candidate outputs does indeed require more compute, however we frame DPH as a method which is useful for smaller LLMs which often don't saturate compute capabilities of higher end accelerators meaning larger numbers of candidates can be computed in parallel using similar compute cost as a single candidate generated by a larger LLM. There are other situations where DPH may be applicable such as inference on the edge where inference latency may be of less concern and where model weights of larger LLMs may not even fit on device.\n\n**Addressing Limitations:**\nAlthough our paper focuses on our in house 550M parameter model we do perform ablations on 4 Qwen 1.5 family models in section 5.2.3.\n\nSee weakness 2 for comments on RLHF comparisons. Such comparisons may be carried out in future works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1m0sbzJFwu", "reviewer": "Reviewer_3kiM", "summary": "The paper introduces DPH, a new method for pre-trained language models that addresses the limitations of RLHF. Unlike traditional RLHF, which can compromise a model's reasoning abilities and cause hallucinations, DPH employs an auxiliary reward head to learn human preference signals without altering the LM's output distribution. The authors conduct a theoretical analysis linking their objective function to cDPO and demonstrate that DPH can be used in conjunction with existing alignment techniques to improve performance. The experiment results show that models fine-tuned with DPH achieve higher scores compared to those fine-tuned with SFT or DPO alone.", "questions": "1. Why does DPH not require an SFT sampling and human labelling stage? In section 4.4, you mentioned SFT in your training pipeline.\n2. What is the relationship between the proposed approach and the sentence \"Would I Lie To You?\" in your title?", "limitations": "The authors include a discussion regarding the limitations of the proposed method.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors highlight the side effects of RLHF, such as damage to the model's ability to reasoning and hallucinations, and try to solve them. I believe the research problem is significant and worth investigating. The authors are going to release their code and model weights, which would benefit our community.", "weaknesses": "1.\tThe claim that the auxiliary reward head avoids affecting the output distribution of the language modeling head is undermined by the practical implementation. Specifically, this claim does not hold when the backbone language model is updated to learn the preference distribution (line 180) and when the model is updated using a joint loss function (Eq. 7). These aspects of the implementation weaken the validity of their claim.\n2.\tExperiment setup: The baseline setup in the experiments is not reasonable. The authors compare their method with other language models that have distinct training settings and data, rather than comparing with other alignment methods. This makes it difficult to infer the superiority of the proposed method over existing alignment techniques, as the comparisons are not directly relevant.\n3.\tClarity and organization: The writing is not easy to follow and requires improvement. Specifically, sections 4.3 and 4.4, which are parts of the methodology, are incorrectly placed in the experiment section. Additionally, the evaluation protocol (section 5.1) should be introduced in section 4 but is instead placed in the results section. This misplacement affects the clarity and logical flow of the paper, making it harder for readers to understand the methodology and its evaluation comprehensively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Addressing Weakness 1:**\nAlthough we do jointly train both the reward head and the preference distribution for completions this is realised through cDPO with a large beta penalty (which limits divergence of the output distribution) and a large epsilon (which limits the preferred vs dispreferred margin). So although the model does tend towards the LM head producing a more preferable output distribution this is heavily limited and was included purely to increase the chance of producing more preferable candidates. It is completely possible to swap cDPO with KL divergence to mitigate any change in the output distribution while learning the reward head weights, but - as stated in the paper - we found it beneficial to \"slightly\" align the model with regularised cDPO. We also do experiments using Qwen models in section 5.2.3 where the model backbone is completely frozen (meaning the output distribution cannot change) and find DPH achieves the highest scores on GLUE across the board, while lagging behind on other tasks which we believe is due to hidden state of the last token in the sequence not containing enough information for the commonsense and reading comprehension tasks - a phenomenon which would be mitigated by additionally training the backbone with either cDPO or KL. Additional ablations using KL divergence rather than cDPO would be an excellent candidate for inclusion in a paper revision either in the main body, or as additional experiments in the appendix.\n\n**Addressing Weakness 2:**\nThe baselines we chose are from models of comparable size or capability with tradeoffs such as task-specific fine-tunes for smaller models or using base pre-trained checkpoints for larger models. To obtain baselines for similarly sized models using other alignment methods would have required us to train these models ourselves which requires significant time and compute. Had we compared our models to other publically available aligned checkpoints our models would have performed worse across the board as these publically available models are typically orders of magnitude larger, and even their SFT-only counterparts would outperform even our best in-house models. We aimed to show the efficacy of our method by showing how DPH improves over our LM head baselines and the baselines of other popular models.\n\n**Addressing Weakness 3:**\nWe appreciate the reviewer's suggestion on organisation of sections which we will address in further revisions of the paper.\n\n**Addressing Question 1:**\nDPH does not require SFT sampling and human labelling as we do not follow the standard RLHF training pipeline and instead opt for a pipeline similar to that of DPO or ORPO which only requires binarized preference pairs for alignment, where such high quality datasets are readily available. We first must perform SFT to get a baseline policy (for both cDPO and prior regularisation) which is also necessary for other alignment methods like DPO and PPO. SFT sampling and human labelling could be incorporated into training however this is not a necessary step and would add complexity.\n\n**Addressing Question 2:**\nThe sentence \"Would I Lie To You?\" refers to the fact that language models can often produce incorrect output which may be unhelpful or factually incorrect. We used this tag line because the model may sometimes generate \"lies,\" but DPH is intended to rank such candidate outputs lower than more correct candidates regardless of log-probability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The approach presented is interesting -- rather than directly fine-tuning a model, it proposes to train a reward model to be used at inference time only to rerank base model outputs. In practice, this reward model is the same as the base model, which has been fine-tuned to have a reward head, but not directly on generated output sequences.

The approach requires more computation at inference time (in contrast to simple fine-tuning) and its usefulness is also constrained by inference-time compute (required to sample candidates to rerank). 

The experiments focused only on several small models, but the authors argue that the approach itself is focused on improving small language models, where inference-time compute is less burdensome. I suggest that the authors refine their arguments about the approach being focused on small models, and why this justify experiments and baselines from small / older models only.

There are also not experiments that provide direct evidence that the model hasn't degraded through the fine-tuning process in training it to predict rewards for inference time.

There is no direct comparison with other standard RLHF approaches such as PPO, although there are comparisons to DPO, and authors argue that PPO would be computationally infeasible for their experiments.
Decision: Accept (poster)