Aligning Language Models with Human Preferences via a Bayesian Approach
Authors: Jessie Wang
Haozhao Wang
Shichao Sun
Wenjie Li
Venue: NeurIPS 2023 poster (2023)
Keywords: Aligned Models; Human-centric NLG

Abstract: In the quest to advance human-centric natural language generation (NLG) systems, ensuring alignment between NLG models and human preferences is crucial. For this alignment, current popular methods leverage a reinforcement learning (RL) approach with a reward model trained on feedback from humans. However, inherent disagreements due to the subjective nature of human preferences pose a significant challenge for training the reward model, resulting in a deterioration of the NLG performance. To tackle this issue, previous approaches typically rely on majority voting or averaging to consolidate multiple inconsistent preferences into a merged one. Although straightforward to understand and execute, such methods suffer from an inability to capture the nuanced degrees of disaggregation among humans and may only represent a specialized subset of individuals, thereby lacking the ability to quantitatively disclose the universality of human preferences. To address this challenge, this paper proposes a novel approach, which employs a Bayesian framework to account for the distribution of disagreements among human preferences as training a preference model, and names it as $\textbf{d-PM}$. 
Besides, considering the RL strategy's inefficient and complex training process over the training efficiency, we further propose utilizing the contrastive learning strategy to train the NLG model with the preference scores derived from the d-PM model. Extensive experiments on two human-centric NLG tasks, i.e., emotional support conversation and integrity ``Rule-of-Thumb'' generation, show that our method consistently exceeds previous SOTA models in both automatic and human evaluations.

Aligning Language Models with Human Preferences via a Bayesian Approach
Jiashuo WANG', Haozhao WANG', Shichao SUN', Wenjie LI"
DDepartment of Computing, The Hong Kong Polytechnic University
"School of Computer Science and Technology. Huazhong University of Science and Technology jeang cassun, @ccompppoppp.p polyu. edu hk hz_ hz_wangghust. eerpchustteu.....m cn
Abstract
In the quest to advance ensuring alignment between NLG models and human preferences crucial. For this alignment, current popular methods leverage reinforcement learning (RL) ap- proach with a reward model trained on feedback from humans. However, inherent disagrecments due to the subjective nature of human preferences pose significant challenge for training the reward model, resulting in a deterioration of the NLG performance. To tackle this issue, previous approaches typically rely majority voting or averaging to consolidate multiple inconsistent preferences into a merged one. Although straightforward to understand and execute, such methods suffer from an inability to capture the nuanced degrees of disaggregation among humans and may only represent specialized subset of individuals, thereby lacking the ability to quantitatively disclose the universality of human preferences. To address this challenge, this paper proposes novel approach, which employs Bayesian frame- work to account for the distribution of disagreements among human preferences as training preference model, and names as d-PM. Besides, considering the RL strategy's 's inefficient and complex training process over the training efficiency, we further propose utilizing the contrastive learning strategy train the NLG model with the preference scores derived from the d-PM model. Extensive experiments on two human-centric NLG tasks, i.e., emotional support conversation and integrity "Rule-of- Thumm""" generation, show that our method consistently exceeds previous SOTA models in both automatic and human evaluations.
natural language generation (NLG) systems,
Introduction
Human-centric natural language processing (NLP) aims develop NLP systems that are finely Consequently, learning from human feedback is well- suited for training models in human-centric NLG tasks [8S 28). Currently, reinforcement learning (RL) with reward model is the most popular method to align models with human preferences 26 III 4T. Its effectiveness depends heavily on how well human preferences are learned by the reward model 6 4. However, modeling human preferences can be challenging.
attuned to human preferences [14 12 30
Due the high subjectivity of personal standards and human values, it can be difficult reach a con- sensus on preferences among individuals, significantly increasing the learning difficulty. As depicted in Figure persons may have varied preferred responses with inconsistent emotions and values given the same context. To tackle this challenge, existing methods mostly adopt aggregation techniques such as majority voting or averaging
However, aggregated preferences potentially cater
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

I'm going through some things with myself. harely sleep and nothing think about how worthless. I'am afraid to talk others about my issues.
Thanks for understunding....
I can understand that. But you should first get the sleep you need and stop thinking these things.
HelpeSecker
SPAP
I indeed need to get enough sleep.al.
The problem exactly that cannot sleept
Supporter A
controversial response foelings of differeni help-scekers
You're not alonc. fact, most people have felt this some degree some point in their life.
My issucs may deals.
won't laughed at tell friends.
So how can solve it?
Supporter B
lexs controversial resporie & Tvelings of different selpeseekers
Figure 1: People can have different feelings towards the same response in the emotional support conversation because of their own experiences and values. A trustworthy human-centric system is expected to consider the benefits of universal groups, including minorities, and generate less controversial and more helpful content, like supporter instead of A.
to specific subsets of people, risking generating controversial content (see Supporter A in Figure 1
Additionally, subjectivity and inconsistency are intrinsic components of certain tasks, such as emotion
analysis [3] and ethical evaluation [10], necessitating careful consideration instead of dismissal [3].
Therefore, for widely acceptable and less controversial outputs, it is necessary for the resulting NLG systems account for capturing disagreements inherent in human preferences [17]
In this paper, we introduce a novel Bayesian brased approach termed Preference Modeling with Disagreement (d-PM). This method is designed to approximate a "universal preference" that com- prises the preferences of "all individuals", given the preferences of several individuals. Although a soft label, derived from these several individuals, can intuitively account for disagreement, outliers or extreme labels can disproportionately influence the overall perception. Therefore, we employ Bayesian inference to refine these preferences. Specifically, the observed preference among selected individuals serves as prior knowledge. Our d-PM aims to leverage distribution of all possible universal preferences (likelihood probability) to adjust and smooth the initially observed one, leading to the derivation of a universal preference (posterior). Upon obtaining the universal preference, we calculate the likelihood of the expected preference types to establish a preference score, which then utilized for further language model alignment.
Based on the d-PM model, we further optimize the language models to generate widely acceptable and less controversial texts. Specifically, we propose utilizing the contrastive learning strategy to calibrate the generation model towards generating texts with high preference scores provided by d-PM. Although existing RL strategies can also be leveraged to make the calibration, they are generally perceived as costly in terms of convergence [ and online decoding processes [40]. We assess our proposed method on two human-centric NLG tasks: emotional support conversations and moral integrity RoT generation. Experimental results demonstrate that our framework can be applied to state-of-thed art generation models for each task without performance degradation and meanwhile effectively increases global consensus of human preferences embedded in generated texts.
Our main contributions are three-fold: (i). To the best of our knowledge, we are the first to align text generation models with human preferences while considering inherent disagreement among different individuals. (ii). In order to model human preferences with their disagreement, we propose Bayesian approach, Preference Modeling with Disagreement (d-PM). Additionally, we use its preference scores to calibrate NLG models via contrastive learning for generations that can be widely acceptable and less controversial. (iii). We conduct experiments on two human-centric NLG tasks, i.e., emotional support conversations and integrity RoT generation. Experimental results demonstrate the effectiveness and versatility of our proposed method.
Our codes are released at htttss:: ://g:////iiii /gthtbb......... www.wanggss/Allieeeeeeeeeeeeeeee
2

---

2 Framework
Aligning text generation models with human preferences requires two essential components: modeling human preferences and calibrating the text generation model.
Preference Modeling with Disagreement Human preferences can be inferred from a human- annotated dataset, denoted as D. Each instance in the dataset is represented as triplet (c, s, (). Here c is a context; s is text; and is a label indicating the annotators' preferences. The inherent disagreement in human preferences can be encapsulated within the label in two distinct ways. In the first approach, the label can be a soft label derived from multiple annotations, all attributed to the same sentence [3]. These annotations are sourced from multiple human annotators preserve disagreement among individuals. The second approach is the direct collection of global consensus. In this context, the label signifies the proportion of people who find particular sentence acceptable, an estimate provided by a single human annotator 42
Aimed at capturing human preference with disagreement within the dataset, we assume there is distribution p over two classes, {acceptable, unacceptable). comprising preferences of all humans, and therefore / is the sampling result from p. We employ preference model R (R) to infer p give and as inputs and the probabilistic format of as the prior distribution. Since we focus on whether widely acceptable, the likelihood of the class acceptable is defined as the preference score:
S ((.c) R(s acceptable.
(1)
Calibration for Alignment In aligning NLG models with human preferences, we calibrate the existing generation model ((Eo) with preference scores. Here, ((Eo) stands for a model already fine-tuned on a dataset (X, Y). where X and Y represent the input set and the corresponding output set, respectively, and &o are the optimized parameters. Significantly. if the dataset for preference modeling is identical to the (X, Y) dataset, then (x, y) (X, Y) corresponds to (c, s) E D. If not, similar domain. We initially decode candidate sequences
these two datasets should belong (īv { k via (iv (z; Eo) for each æ E X. Then, we further train G(Eo), aimed at a new objective: aligning the likelihoods of candidate sequences with preference scores (S( k==K
3 Method
(a). Preference Modeling with Disagreement
Minimize Free Energy
r=(p;)
PL
D
9; Context Text
Observed Preference
Generative
model
R(0)
p( ci P.)
Universal Preterence
C₁ 8
(b). Calibration for Alignment
91
g/1
X
G(En)
R(00)
G(E)
$(gge))
Mx
8x Likelihood Calibration
Dx
Candidate Generation
Preference-based Ranking
Figure 2: Diagram for preference modeling with disagreement and calibration for alignment.
This section presents our method, whose diagram shown in Figure 2 first use Bayesian approach, i.e., d-PM, to model human preferences with disagreement (Section 3.1). Then we calibrate a text generation model by contrastive learning with preference scores of d- PM to align this model with human preferences (Section 3.2

---

3.1 Preference Modeling with Disagreement
We establish a distribution p represent the universal preference for the text s given its context Therefore, the observed annotations are considered as samples from and can form a prior distribution Ps (p). Inspired by 229 we devise Bayesian approach to approximate p using this prior.
Specifically, we establish a connection between p and through the optimization process of a generative model. This model is designed for generating text conditioned on C and p: p(s|ct, p). The log-likelihodd the text can be formulated as >; log p(s = E; log (Ep p(8:/ci, P)p: (p)). where (p) is the prior preference distribution. Its optimization can be achieved by introducing variational posterior distribution q(pls;, c,) for the i-th datapoint, and minimizing the free energy (negated evidence lower bound) formulated as:
Σ p(ga(())))) + ΣΣ apps, o,) log p(silc)))Pi())
(2)
q(pls;,c)
Minimization of the free energy involves estimations of both the forward distribution of text 85: p(si(e,,)) p), and the posteriors q(p]8;, Gi), which can be computed by our preference model:
dppss; Q) R Rs; ci(0).
(3)
As for p(s:jc, p). defined only on the i-th datapoint and computed by minimizing Equation for fixed 40p18,, ci), s.t., E, p(si(c,, p) for all p. Thus, the optimum is achieved by:
p(silc,,)) = ai.p E(g(pls,,c)
(4)
From Equation 4 the generative model can be regarded matrix of variables as,p describing conditional probabilities different responses s; given different latent distributions with known C
In variational way, Equation (2) can be rewritten as log ((s:(()))))) (L(((p)s; [L,((p)s,, (p)lr Here, (p) xx p(8: (c,, (p) is the posterior model of the generative model, and can be reformulated with reduction of ((8: p) to the matrix in Equation (4):
Ps p.(p)adplss.cc)
re(p) az au: (:(p))((((ssssssp)) = ai Σ, (,((()))))))
(5)
where or is scalar enabling r(p) = Accordingly, we can minimize the KL divergence between g( p(pss ci) and rx(p) minimize the free energy. The minimization of the free energy in Equation ( can be derived as:
Σ (L(a(pls..c)lr:(e) min Σ KL [ p.(p)
R R(si, Cz 0) Σ, c(
(6)
By optimizing the above objective, can optimize the parameters of our preference model, i.e., O.
3.2 Calibration for Alignment
nearly possible for G(Eo) to generate texts with both high and low preference scores, shown in 3 However, we expect to calibrate the model such that the generation probability aligns with these preference scores. Specifically, we use diverse beam search [34] to generate multiple candidates and then use our d-PM to evaluate these candidates. For the sake of more likely generating a high preference score text, we propose a aagnosic module to leverage contrastive learning to calibrate generation likelihood aligning with d-PM. Taking inspiration from recent calibration work we implement this module through the following three steps:
32 23 39),
Step 1: Candidates Generation. We generate candidates from the text generator G(Eo), which has been ine-tunnd on corresponding dataset (X, YY) and its parameters are εo, on its own training dataset. Given input sequence z E X, we first use (EE) to generate K candidates {ύr, Ø2. ÜK } using diverse beam search. As result, these candidates will get similar possibilities yet different preference scores according to the above preliminary study.

---

Step 2: Preference-basee Ranking. We use our proposed d-PM R (Ø) measure the preference score (()..)) of each candidate Уı- Then we rank these can- didates according to the above preference score and obtain list of ranked candidates: V1 W2::.. 1. :
-00
0.550759
MIN
XYN
where S (...)) S for V <.
345MEAAN
MANIMMAAO
EARAMMANN
Step 3: Likelihood Calibration. As mentioned be- fore, we leverage contrastive learning to assign higher likelihoods to the candidates with higher preference scores. The following pairwise margin loss is used to adjust the generator (E))
0.
KUTR/255 2.1122991
K3229)
-02
Biender-Vanilla
Blender-Joint
MulnESC
The maximum and minimum pref- erence scores of 10 candidates generated via diverse beam search given the same context. We test on 1000 data instances and three emo- tional support conversation models.
Figure 3:
L = -E mar P()
(()):) -P((:)) Xay).
(7)
where Aiy is the default margin À multiplied by the difference in rank between the samples, i.e., Aij
= (j i). P E) the length-normalized log-
probability of the candidate:
En' log Gi (e(iE)
P :E)
(8)
where is the length penalty hyperparameter. To avoid forgetting token-level likelihood information of the ground-trutt truth text, we also use an additional token-level negative log- bog-likelihood. The final calibration loss is as follows:
L
log (cela,,ee:: L".
(9)
18
We minimize - to optimize the generator's parameters ç. This process is supervised by our d-PM model and aligns the generation model with human preference.
Experiments
4.1 Emotional Support Conversation
In emotional support conversation, supporter aims to buffer help-seeker's emotional distress and help the help-seeker to change the difficult situation [22]. In this context, the model functions as the supporter, while the user always the help-seeker. Due different personal experiences, different help-seekers may respond with varied feelings and reactions to the same response, as illustrated in Figure [ Our objective is enhance the model's 's ability to generate responses that will not escalate the negative feelings of a diverse range of help-seckers.
Dataset and Base Models The benchmark ESConv 22 containing approximately 1k conversa- tions with 31k utterances, develops each conversation between help-seeker and supporter. We include BlenderBot- Vanilla and BlenderBot-Join proposed in conjunction with the dataset, and SOTA model MultiESC(3), We reproduced the base models in accordance with their respective papers and publicly available codes.
Human Preferences with Disagreement We derive human preferences from the Motivational- Interviewing- Dataset (36). This dataset encompasses around 17k supporter responses to help-seekers. experts following the MI codes The labels can be induced into two classes (acceplable, wnacceplable). and the human preferences with disagreement can be estimated by our d-PM method. By fine-tuning a BERT model on this dataset using prefix-tuning
Each response annotated by
37 19],
we obtain the d-PM. Additional details can be found in Appendix B.2
5

---

Table 1: Automatic evaluation results on ESConv. All results are significandly better than the corresponding base model with p 0.01.
Model
B-2
B-3
B-4
R-L.
METEOR
CIDEr
Extreme
Base
17.85
7.08
3.60
2.11
17.06
7.46
15.44
51.02
Blender
Alignedaajor
19.07
7.71
3.94
2.28
17.09
15.97
50.61
-Vanilla
Alignedwe
17.88
7.21
3.68
2.12
16.52
7.31
15.50
50.73
Alignedwia
19.70
7.56
3.64
2.05
16.90
7.72
15.62
50.48
Pl
20.75
8.32
4.17
2.39
17.41
8.21
16.57
50.38
Base
18.70
7.30
3.61
2.03
17.66
7.56
16.91
50.95
Blender -Joint
Alignedasjor
20.37
8.61
4.47
2.65
19.23
8.32
21.86
51.57
Alignedan
19.36
7.87
3.85
2.09
17.55
7.65
15.90
50.84
Alignedwia
21.05
8.14
3.89
2.07
17.65
8.11
15.29
50.68
Aligneda.M
21.05
8.97
4.74
2.78
19.39
8.48
20.34
51.81
Base
20.36
8.80
4.92
3.14
21.00
8.58
30.69
52.74
Alignedawjor
19.10
8.27
4.61
2.88
20.72
8.24
30.15
52.57
MultiESC
Alignedwst
19.30
8.33
4.62
2.88
20.83
8.35
30.75
52.54
Alignedwios 21.58
8.80
4.74
2.96
20.47
8.78
28.58
51.65
Pligned&
21.59
9.56
5.33
3,36
21.50
9.03
32.65
53.15
We apply our proposed method align each base model, thus treating the well-trained base model as the generator ((E)). Additionally, to validate the effectiveness d-PM, employ three alternative preference models within framework for comparative analysis:
Experimental Setup
(1) A preference model (major) trained to predict the majority voting result of annotations from different annotators, denoted as Im. and optimized by cross-entropy loss, formulated as:
{(0) (c..ddd >w (pr (I log ( s; 0))]
(10)
where pI...) denotes the one-hot vector of 4m
(2) A preference model (soft) trained to approximate the direct probabilistic label annotations, i.c., the soft label I. The model is optimized by:
(() (c,,,)))> (R,, 0) 21)
(11)
(3) A preference model (w/oA) that does not aggregate annotations and takes each annotation independent. This model optimized by cross-entropy loss, similar to Equation (TO).
When training the aligned models, we aim to retain the same hyperparameters used of the base models. We set the candidate number K to 10. We train each aligned model five times with five different seeds. Subsequently, we test each of the five trained models on the test dataset and compute the average results.
training
Automatic Evaluation We adopt the following metrics commonly used in previous work for the automatic evaluation of our proposed method: BLEU (27 (B- (B-1/2/3/4). ROUGE (R-L) 20 METEOR 2 CIDEr (33, and BOW Embedding based matching score 21 (Extreme). Results are
shown in Table 1
Our Aligneda PM significantly improves the performance of the base model in almost all automatic metrics, irrespective the base model, suggesting the overall effectiveness of our proposed method. Alignedmajor and Alignedsoft are able enhance the performance when the base model is either Blender-Vanill Vanilla or Blender-Joint, however, they do not yield an improvement when the base model is MultiESC. This limitation underscores the constraints inherent in using majority voting labels and soft labels to address disagreement of human preferences. Alignedwios surpasses the base model in certain metrics but falls short in others. Notably, it performs significantly lower CIDEr, metric evaluating the similarity between weightweighted n-grams. This shortfall suggests that Alignedw/oA less likely to generate responses containing critical information found in the ground truth. This issue arises from the preference scores determined by w/oA being closely clustered in value, resulting its inability to sequence the generated samples logically. These pieces of evidence indicate the potency of our proposed preference model d-PM.

---

Table 2: Human evaluation results on ESConv. The base model is MultiESC.
Model
Identification
Comforting
Suggestion
Overall
Global Consensus
Base
3.017
2.562
2.918
2.598
2.693
Alignedanjor
3.032
2.572
2.880
2.598
2.763
Alignedon
3.007
2.557
2.905
2.568
2.747
Alignede.M 3,052
2.587
2.952
2.637
2.783
Human Ratings We ask human annotators evaluate the generations of models based MultiESC since MultiESC can outperform Blender-Vaniie and Blender-Joint in almost all automatic evaluations. Specifically, we randomly sample 100 responses generated by different models for human ratings. We asked annotators to imagine they are help-seekers in the corresponding situation and measure each response in five aspects: (1). Identification: on scale of 5, how much the response can explore your situation in depth and help identify the problems. (2). Comforting: on scale of 1 ~ 5, how skillful the response can comfort you. (3). Suggestion: on scale of ~ 5, how helpful the response can solve your problems. (4). Overall: on scale of ~ 5, the overall quality of this response for emotional support; (5) Global Consensus: the number of people who deem the response can help them, 1 ~ 5 represent nobody (< 1%), rare (5% 25%), controversial (~ 50%), most (75% 90%), and all (> 99%), respectively. Each response is annotated by three annotators, and we averaged these three annotations as the final result for each metric.
From Table 2 our method performs the best among the methods. Aligneds-PM obtained the highest score in all aspects, including the global consensus. It demonstrates that our method can generate less controversial and more helpful responses in the task of emotional support conversation.
4.2 Integrity RoT Generation
We also apply our method to the integrity "Rule-of-Thunb" (RoT) generation task. This task is concerned with describing a chatbot's normative rules, which holds great potential for advancing research on morally-consistent conversational agents [42). When it comes to outlining a chatbot's normative rules, people's ' values can vary widely. However, morally--onsistent conversational agents are expected to accommodate the values of as many individuals as possible. Therefore, the generation of widely acceptable RoT's is crucial for guiding the behavior of these agents.
Dataset and Base Models The MIC dataset 42) comprises about 99k distinct RoTs that encapsulate the moral assumptions inherent in 38k machine generated replies to open open-ended prompts. Each prompt is associated with three different RoTs, each provided by distinct annotator. Alongside each RoT, annotators offer "global consensus" value, B. which signifies the estimated proportion of the global population that would agree with the RoT. We utilize T5 (small), Flan-T5 (base) and BART (large) models as our base models, and fine-tune them on the MIC dataset. For model inference, closely follow the processes presented in 42 Specifically, greedy decoding, beam search (n = 3), and nucleus sampling (p 0.9), We generate one RoT for greedy decoding; for the latter two, three hypotheses are generated and the highest-scoring one is selected.
adopt three decoding strategies:
Human Preferences with Disagreement We learn human preferences with disagreement for normative rules from the MIC dataset. The open-ended prompts are treated as context, and the ground truth RoT is considered the text to be evaluated. A probabilistic label is assigned to each RoT based on its global consensus: the probability for the class acceptable B while that for unacceptable is (1 - 3). We utilize this dataset to train the d-PM; details can be found in Appendix B.2
We apply our proposed framework to align each base model that has been rigorously fine-tuned on the MIC dataset. To assess the effectiveness of d-PM, we also train preference model (soft) by minimizing the loss computed using Equation (IT). The number of
Experimental Setup
candidates K set to 5. We adopt the same hyperparameters for training each aligned model as are
used for the base model. We conduct five training runs for each aligned model using five distinct
seeds. Then, we evaluate each of the five trained models on the test dataset and calculate the average results.
7

---

Table 3: Automatic evaluation results on MIC. i represents significantly better than the corresponding base model with p < 0.01.
Model
R-1
R-2
R-I.
BertScore
ScareBL.El
Avg.Len
Base
53.44
32.97
52.17
93.44
29.05
8.94
Beam
Al
52.14
31.48
50.79
93.34
27.05
8.85
AlignedsPM
53.45
32.82
52.09
93.49
28.51
8.99
Base
37.04
16.30
35.27
90.94
14.27
10.47
TS
Greedy
Aligned.on
37.77
16.82
35.97'
91.21
14.681
9.83
(Small)
Aligneda
38,15
17.22
36.40
91.29*
15.151
9.74
Base
40.22
19.23
38.56
91.59
16.71
9.85
p=0.9
Aligned
40,901
19.70
39.24
91.79'
16.98
9.47
AlignedaPM
41.41
20.22
39.75'
91.90
17.75t
9.38
Base
55.07
34,96
53.74
93.77
30.68
9,00
Beam
Aligned.on
54.82
34.65
53.49
93.75
30.34
9.00
Aligneda
55.18*
35.071
53.86
93.79
30.831
9.01
Base
37.94
17.23
36.13
91.39
15.36
9.78
Flan-T5
Greedy
Aligned
37.84
17.03
36.00
91.38
15.12
9.75
(Base)
AlignedarM
38.34
17.52
36.53
91.44
15,49
9.77
Base
41.41
20.41
39.70
92.02
18.02
9.30
p=0.9
Alignedæot
41.44
20.33
39.71
92.02
17.91
9.29
Aligneda
41.78'
20.69
40.09
92.07'
18.26
9.32
Base
54.81
35.07
53.35
93.85
30.80
9.44
Beam
Aligned.0t
54.82
34.85
53.36
93.82
30.35
9.36
AlignedaPM
55.05!
35.18
53.62
93.86
30.85
9.40
Base
54.77
34.85
53,30
93.84
30.51
9.54
BART (Large)
Greedy
Alignedzon
54.54
34.53
53.10
93.80
30.01
9.47
Aligneds
54.81
34.86
53.39
93.83
30.51
9.48
Base
54.77
34.96
53.32
93.84
30.62
9,56
p=0.9
Aligned.oo
54.65
34.68
53.23
93.82
30.16
9,45
Aligneds.MM
54.86
35.01
53.45
93.85
30.63
9.48
Table 4: Human evaluation results on MIC. The base model is BART-large (beam).
Model
Well-formedness
Fluency
Relevance
Global Consensus
Base
0.528
2.547
2.037
2.428
Alignedzat
0.550
2.602
2.028
2.502
AlignedaPM
0.568
2.602
2.103
2.555
Automatic Evaluation accordance with previous work 42 we report standard ROUGE 20
(R-1/2/L), ScareBLEU (27). BERTScore [38], and average generation length (Avg. Len) metrics. As each prompt-reply pair in our dataset has three ground truth RoTs, compute each metric by
taking the maximum score from these three, following the method employed by [42 The results are displayed in Table
The results clearly show that Alignedd-PM generally outperforms its base model. In addition,
Alignedd -M achieves the highest score across all evaluation metrics, except the generation length, when employing a beam decoding strategy. While Alignedsoft slightly improves performance with T5 (small) the base model, minor decline observed when the base model either Flan-T5 (base) or BART (large). Interestingly, the enhancements observed in this task are not pronounced as those witnessed in the emotional support conversation task (refer Table [T). This may be attributed the MIC dataset inherently accounting for disagreement, as each prompt is paired with three RoTs from different annotators. This enables base models, when fine-tuned on this dataset, to encapsulate various human preferences certain degree. Nonetheless, our framework has the potential to further boost model performance by providing more explicit preference information with disagreement during training.

---

Human Ratings We randomly select 100 replies generated by models with BART (large) as the base model for human evaluation. Adhering to previous practice, we assess generated outputs based on the following criteria 42 (1). Well-formedness: yes or no, does the RoT explain the basics of good or bad behavior with a single judgment or action?; (2). Fluency: on scale of 1-5, how much does the RoT align with what an English speaker might naturally say?; and (3) Relevance:
scale of 1-5, how well does the RoT apply to the Answer for this specific Question if we assume the RoT is true? Furthermore, we request annotators to provide Global Consensus: how many people globally will agree with this RoT, similar to the method described Section 4.1 Three annotators evaluate each RoT, and average these three evaluations the final score for each metric.
Results presented in Table indicate that our method produces RoTs that are more universally agrecable than those generated by the other two models. Our Alignd- Mignd-PMM model improves all metrics over the base model and outperforms Alignedsoft.
5 Model Analysis
34-
21.54
51.
5.1
50-
21.0-
3.0
8.4
27-
Kle5 K-i8K=15KK K-0n
Ked K-10 K-200
K0-5
K-10 K-15 K--00
K-d K- K-10K=15K K-20
Ko5 K-10 K-18 Ke00
B-3
B-4
R.L
METBOR
CIDE:
Figure 4: Model performances with different candidate numbers K when calibrating MultiESC with preference scores of d-PM.
The effect of candidate number K during calibration. To examine the influence of varying the candidate number K on model calibration, we modify the MultiESC calibration process using different candidate numbers, namely 5. 10, 15, and 20. Specifically, this involves changing the beam widths in the diverse beam search process. Theoretically. a more significant candidate number would encompass more samples under consideration, thereby escalating the upper bound of performance. Nevertheless, as depicted in Figure 4 the model performance initially experiences augmentation yet subsequently exhibits a reduction with the increment of the variable K. is because an overly large candidate number could introduce redundant samples with minor differences, and the generation model might erroneously distinguish between them.
3.-
Metric
MultiESC
RL
Ours
Ous
B-1
20.36
11.75
21.59
28-
B-2
8.80
4.81
9.56
B-3
4.92
2,78
5.33
2.6
B-4
3.14
1.81
3.36
R-L
21.00
19.57
21.50
2.4
METEOR
8.58
6.25
9.03
CIDEr
30.69
26.02
32.65
22.
Extreme 52.74
51.21
53.15
0.0
#(Samples)/s
2.65
4.36
400
800
1200
1600
2000
2400
Stap
Figure 5: Comparison between alignment with RL (RL) and our model (Ours). Left: Automatic evaluation results (#(Smplee))/ indicates the number of trained samples per second). Right: Training loss according to training steps.

---

Contrastive Learning vs. Reinforcement Learning. We opt for contrastive learning to align generation models with human preferences instead of the currently prevalent reinforcement learning (RL. approach. This decision is rooted in several considerations. Firstly, RI. requires expensive online decoding procedures, while our framework based on contrastive learning is one-time offline process [40]. Secondly, RL usually yields slow convergence speed [9]. To validate this, we align MultiESC using RL. From Figure 5 RL trains fewer samples than our framework per second. After the same steps, the loss of RL is still high, and the model performance is much worse than ours.
6 Related Work
Developing human-centric systems, which ensure that human stakeholders benefit from system outcomes [14], remains great challenge. To build human-centric system, it is critical align models with human preferences [35]. There are various methods to implement the alignment. Currently, the most popular and well-known method reinforcement learning from human feedback, thanks to the GPT series [26]. This method is also used for text summarization [31 4) detoxification (6. and machine translation [5 16]. The Above-mentionel methods adopt one reward model, while some methods combine rewards computed by different reward models to consider fine- uneeggrined aspects of human needs 8 Moreover, some models use human feedback as the supervision signal directly to learn human preferences, such as fine-tuning pre-trained models with well-established datasets [10. Human feedback can also be used augment prompts for better performance (24].
Conclusion and Future Work
In this work, we strive to align models with human preferences to foster the development of trustwor- thy human-centric NLG systems. Unlike previous approaches, we take inherent disagreement into account for modeling human preferences. This idea is motivated by two compelling reasons. Firstly, is impractical to expect consensus in human preferences due to the high degree of subjectivity involved. Secondly, harmonizing preference disagreements can inadvertently disadvantage minority groups. Accordingly, we introduce a Bayesian approach, termed Preference Modeling with Dis- agreement (short d-PM), capture the subtleties of disagreement from limited human feedback. We subsequently utilize its preference scores to calibrate pre-existing text generation models. The efficacy of our method is substantiated through experiments in emotional support conversation and integrity Rule-of- ohwumb generation.
Despite our focus on disagreement, another critical aspect remains to consider when modeling human preferences. this work, like most previous studies, we assumed a linear relationship between the preference score and the proportion of the global population that finds the text acceptable. However, this assumption may not always hold true. For example, a sentence that 20% of people find helpful should ideally have preference score closer to instead of 0.2. We believe that this issue merits further exploration in future research. Fortunately, in this study, we sidestepped this problem by using the rank of preference scores rather than the scores themselves.
8 Acknowledgements
This work is supported by Research Grants Council of Hong Kong (P0140/5204018, PolyU/15207920, PolyU/15213323) and National Natural Science Foundation of China (62302184, 62076212).
10

---

References
[1] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective offline reinforcement learning. In International Conference on Machine Learning, pages (04-114. PMLR, 2020.
[2] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metrie for mt evaluation with improved correlation with human judgments. Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72, 2005.
[3] Valerio Basile et al. It's the end of the gold standard as we know it. on the impact of pre-ageregation on the evaluation of highly subjective tasks. In CEUR WORKSHOP PROCEEDINGS, volume 2776, pages 31-40. CEUR-WS 2020.
[4] Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. Better rewards yield better summaries: Learning summarise without references. In Proceedings of the 2019 Conference on Conference on Empirical Methods in Natural Language Processing (EMNLP). Hong Kong, China, 2019.
[5] Yi Cheng, Wenge Liu, Wenjie Li, Jiashuo Wang, Ruihui Zhao, Bang Liu, Xiaodan Liang, and Yefeng Zheng. Improving multi-turn emotional support dialogue generation with lookahead strategy planning In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3014-3026. Association for Computational Linguisties, 2022.
[6] Farshid Faal, Ketra Schmitt, and Jia Yuan Yu. Reward modeling for mitigating toxicity in transformer-based language models. Applied Intelligence, 53(7) 8843,, 2023.
[7] Maxwell Forbes, Jena D Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. Social chemistry 101: Learning to reason about social and moral norms. Proceedings of the 2020 Conference on Empirical Methods in Natural Languge Processing (EMNLP), pages 653-670, 2020.
[8] Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements, arXiv preprint arXiv:2209.1443
2022.
[9] Yang Guan, Jingliang Duan, Shengbo Eben Li, Jic Li, Jianyu Chen, and Bo Cheng. Mixed policy gradient. arXiv preprint arXiv:2102. /1513, 2021.
[10] Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch Critch, Jerry Li Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. In International Conference on Learning Representations, 2021.
[11] Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Human-centric dialog training via offline reinforcement learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 3985-4003, 2020.
[12] Tharindu Kaluarachehi, Andrew Reis, and Suranga Nanayakkara. A review of recent deep learning approaches in human-centered machine learning. Sensors, 21(7:251,4, 2021.
[13] Manfred Klenner, Anne Göhring, Michael Amsler, Sarah Ebling. Don Tuggener, Manuela Hürlimann, and Martin Volk. Harmonization sometimes harms. 2020.
[14] Bhushan Kotnis, Kiril Gashteovski, Julia Gastinger, Giuseppe Serra, Francesco Alesiani. Timo Sztyler, Ammar Shaker, Na Gong, Carolin Lawrence, and Zhao Xu. Human-centric research for nlp: Towards a definition and guiding questions. arXiv preprint Xi::
2022.
[15] Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, and Stefan Riezler. Can neural machine translation be improved with user feedback? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pages 92- 105, 2018
[16] Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. Reliability and learnability of human bandit feedback for sequence-to-sequeene reinforcement learning. In Proceedings of the 56th Annual Meeting of the Assaciation for Computational Linguistics (Volume 1: Long Papers), pages 1777-1788, 2018
[17] Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. Agreeing to disagree: Annotating offensive language datasets with annotators' disagreement. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 10528-1000 10539,0539, 2021.
11

---

[18] Bo Li, Peng Qi, Bo Liu, Shuai Di, Jingen Liu, Jiquan Pei, Jinfeng Yi, and Bowen Zhou. Trustworthy ai: From principles to practices. ACM Computing Surveys, 55(9): 2023.
[19] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computaaionall Linguistics and the 1 1th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4582- 4597, 2021.
[20] Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries, In Text summarization branches ouf, pages 74- 81, 2004.
[21] Chia- Wei Liu, Ryan Lowe, Iulian Vlad Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122-2132, 2016.
[22] Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. Towards emotional support dialog systems. In Proceedings of the 59th annual meeting of the Association for Computational Linguistics, 2021.
[23] Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. Brio: Bringing order to abstractive summarization, In Proceedings of the 60th Annual Meeting of Association for Computational Linguistics (Volume I: Long Papers), pages 2890-2903, 2022.
[24] Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. Memory-assisted prompt editing to improve gpt-3 after deployment. arXiv preprint arXiv:2201.000 160099 2022.
[25] William Miller and Stephen Rollnick. Motivational interviewing: Preparing people for change. The Journal for Healthcare Quality (JHQ). 25(3):46, 2003.
[26] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[27] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-118, 2002.
[28] Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timmit Gebru, Ben Hutchin- son, Jamila Smith-Loud, Daniel Theron, and Parker Bames. Closing the ai accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 conference on fairness. accountability, and transparency. pages 33 44, 2020.
[29] Esther Rolf, Nikolay Malkin, Alexandros Graikos, Ana Jojic, Caleb Robinson, and Nebojsa Jojic. Resolving label uncertainty with implicit posterior models. In Uncertuinty in Artificial Intelligence, pages 1707-1717 PMLR, 2022.
[30] Ben Shneiderman. Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human-Compute Interaction, 36(6):995-504, 2020.
[31] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Danicl Ziegler, Ryan Lowe, Chelsca Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human foodback. Advances in Neural Information Processing Systems, 33:3008 0000000000122200 2020.
[32] Shichao Sun and Wenjie Li. Alleviating exposure bias via contrastive learning for abstractive text summarization. arXiv preprint arXiv:2108.11866 2021.
[33] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus -based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.
[34] Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam scarch: Decoding diverse solutions from neural sequence models, AAAI, 2018.
[35] Zijie Wang. Dongjin Choi, Shenyu Xu, and Diyi Yang. Putting humans in the natural language processing loop: A survey. In Proceedings of the First Workshop on Bridging Human Computer Interaction and Natural Language Processing, pages 47-52, 2021.
12

---

[36] Anuradha Welivita and Pearl Pu. Curating a large-scale motivational interviewing dataset using peer support forums. In Proceedings of the 29th International Conference Computational Linguistics, pages 3315-3330, 2022.
[37] Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong. Torsten Scholak, Michihiro Yasunaga, Chien-Shan Sheng Wu, Ming Zhong, Pengcheng Yin. Sida Wang, Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. EMNLP. 2022.
[38] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Weinberger, Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2019.
[39] Xingxing Zhang, Yiran Liu, Xun Wang. Pengcheng He. Yang Yu. Si- Si-nnn Chen, Wayne Xiong. and Furu Wei. Momentum calibration for text generation. arXiv preprint arXiv:2212.04257, 2022.
[40] Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter Liu. Calibrating sequence likelihood improves conditional language generation. arXiv preprint arXiv:2210.00045, 2022.
[41] Daniel M Ziegler. Nisan Stiennon, Jeffrey Wu, Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 10Xiv:990,00894) 2019.
[42] Caleb Ziems, Jane Yu. Yi-Chia Wang. Alon Halevy, and Diyi Yang. The moral integrity corpus: benchmark for ethical dialogue systems. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 3755 3373,, 2022.
13

---

A Ethics and Societal Impact
Ethies In our experiments, utilized three datasets: MI-dataset, ESConv, and MIC. Each of these datasets is designed to protect user privacy, and none contain any personal information.
Societal Impact Our work driven by the goal to enhance the positive impact of Natural Language Generation (NLG) models by better aligning them with human preferences, particularly those relating personal standards and human values such as emotions, beliefs, and ethical norms. We acknowledge the diversity of individual preferences, which can often lead to disagreements. This disagreement is taken into account in our model of human preferences, which we use calibrate our text generation model. This approach aims to produce text that is widely acceptable and less controversial.
Our findings suggest that approach can make generated texts less contentious and more useful. Looking towards the future, neglecting to account for disagreements in human preferences could result in serious implications, especially if these models are used in safety-critical situations. We anticipate that our approach could be beneficially applied to a wider range of tasks and models in future research and product development.
B Supplement Experiment Details
The code for our experiments is provided in the supplements and will be made public upon acceptance of this paper. Additional details that are not included in the submitted code are explained in the following subsections.
B.1 Resources Used
We trained models based MultiESC using two Nvidia RTX 3092 GPUs, while all other models, including d-PM models, were trained using single NVIDIA RTX 3092 GPU. Our models were implemented in Python using PyTorct and the transformers (4. 16.2) library
B.2 d-PM Training Details
The d-PM models are fine-tuned versions of the BERT (base, uncased) model BERT using prefix-tuning. The BERT model has layer number of "Layer and a hidden size of dBERT The model uses a sequence of prefix indices, represented as Lidz, and the length of the prefix is represented by LLax) The model input is a concatenation of the prefix, the context, and the text, denoted as = Liax; s). where represents the input at time step 1. The activations of the model are denoted as h = []pretx ht, h1",, where h, represents the concatenation of all activation layers at time step t. Here, ht and h² are computed using BERT parameters, while I,eefix are free parameters. trainable matrix Ag, is then initialized, which is parameterized as 0, of dimension |Lixx] x Mlim where ddim nHBERT x (BERT Thus, the activation he at time step t is defined as:
Model Architecture
[o, :
if Liaxl otherwise.
h1
(12)
BERT(2,,h2) h2t)
We utilized the last hidden states (the activations from the last layer) of the text to calculate the probabilities for the classes acceptable and unacceptable. This computation is facilitated by a multilayer perceptron (MLP) layer, which is parameterized 0z.
The parameters the d-PM, i.e., 0 = (01,0)). are optimized using the loss function, which is formulated in Equation (6). Furthermore, the preference models that utilized in Alignedmajor and Alignedsoft share the same architecture. However, they are differentiated by the distinct loss respectively.
functions used for their optimization, denoted in Equation (10) and Equation
Experimental Setup The d-PM models were implemented based on the code published for the UnifiedSKG framework [37]. During the training of the d-PM models, BERT parameters were frozen,
https://pytorch..r
14

---

and the prefix was optimized. The prefix length was set to 10. A batch size of 160 and a learning rate of 5 10" were used.
Dataset For the training dataset, the MI-Dataset was used to train d-PM model which can output human preferences for the emotional support responses. This dataset contains 15 fine-grained classes of responses. The annotators categorized each response into one of the fine-grained classes. These responses fall into three coarse-graings classes according the MI codes 25]: (1) MI Adherent, which involves supporting the welp-seekers with empathetic and compassionate statements, enabling them to feel heard, respected, and understood; (2) Relational, which focuses on establishing solid relationship between the help-seeker and the supporter, leading to more positíve responses from the help-seekers; and (3) M/ Non-Adherent, which includes statements such as arguing, confronting, or offering unsolicited advice, and may create resistance and hinder problem-solving for the help- seekers. When training the d-PM, these classes were converted into two classes: acceptable and unacceptable, shown in Table 5 dataset was randomly split into a 9 ratio for the training and validation set. There are 34.93% instances where annotators have different annotations.
Table 5: The MI codes in MI-Dataset 36., and our recategorization. Examples are from its paper.
Coarse-Grained MI Codes
Fine-Grained
Our Classes
Examples
MI Codes
1. Advise with Permis-
If you agree with it, we could try to brain- storm some ideas that may help.
sion
MI Adherent
2. Affirm
You should he proud of yourself for your past efforts.
3. Emphasize Autonomy
It is really up you decide.
4. Support
know it's really hard to stop drinking.
acceptable
5. Closed Question
Do you think this is an advantage?
6. Open Question
What is your take on that? It sounds like you're re feeling worried.
7. Simple Reflection
Speaker: Mostly, would change for fu- ture generations.
Relational
8. Complex Reflection
Listener: It sounds like you have strong feeling of responsibility
9. Give Information
Logging your cravings is important as cravings often lead to relapses.
10. Self-Disclose
used to he similar where get obsessed ahout how people look.
11. Other
Good morning, Hi there.
12. Advise without Per-
You should simply scribble reminds you take break.
that
mission
unacceptable MI Non-Adherent
13. Confront
Yes, you are alcoholic. You might not think so, but you arc.
14. Direct
Don't 't do that!
15. Warn
Be careful, DO NOT stop taking meds without discussing with your doctor.
Table 6: One example from the MIC dataset. Blue text: prompt-reply (QA) pair. Orange text: global consensus scale value.
Question: Am bad BF, weird, or just going insane?
Answer: 1 don't think you're re bad bf or weird or going insane. think you just need to talk to him about it.
RoT: It's important to communicate honestly with your significant other.
Global Consensus: 4
The MIC dataset was used for training the d-PM used in the integrity Rule of Thumb (RoT) generation task. Each RoT in this dataset is annotated with global consensus provided 5-Likert scale format, where 1 ~ 5 represent nobody (< 1%), rare (5% 25%), controversial (~ 50%), most
15

---

(75% ~ 90%), and all (> 99%), respectively. An example is shown in Table 6 For the purpose of training, the ordinal scales were converted into continuous variables that represent the percentage consensus. This was done by randomly selecting value between the upper and lower bound corresponding to each scale value to serve as the final global consensus (3) for each RoT. Following this, the probability for the class acceptable was set as 3, while that for unacceptable was set as (1 - 3). The split of the MIC dataset was maintained when training the model and when calibrating the generation model to avoid data leakage.
B.3 Emotional Support Conversation
Experimental Setup We implemented the base models following the methods detailed in [22. S
and their corresponding published codes Each base model was run once using the seed provided in its published codes.
In order to fairly compare models, we aimed to keep the hyperparameters consistent with those of the corresponding base models when implementing aligned models, except that we increased the learning rate to enhance the training efficiency of aligned models. Specifically, we set the learning rate to x 10-3 for the Blender- Vanilla and Blender-Joint base models, and 3 x 10-6 for the other models. Additionally, due GPU memory constraints, we reduced the batch size from 32 to 12 when training the aligned MuCtiESC. We ran each aligned model five times with different seeds: 0, 1, 13, 42, and 1024, and reported the average results.
B.4 Integrity RoT Generation
Experimental Setup Our implementation of the base models was based on the work presented in the paper [2 and its published codes Each base model was run once using the seed provided in its published codes.
For a fair comparison, we endeavored to keep the hyperparameters consistent with those of the corresponding base models when implementing the aligned models. However, we increased the learning rate to 1 x 10 during the training of aligned models. Each aligned model was run five times with different seeds: 0, 1. 13, 42, and 1024, and we reported the average results.
B.5 Alignment through Reinforcement Learning
Implementation Details The reinforcement learning (RL) framework was implemented with refer- ence to the Hugging Face TRL library We aimed to ensure consistency with the hyperparameters used in the aligned model, which was implemented using the contrastive learning framework. Addi- tionally, adhered the same training steps, i.e., terminating the training process after 2400 steps. We executed the RL framework with five seeds: 0, 1, 13, 42, and 1024. The best result obtained is reported in the paper.
7 and Table pressen results of aligning models with RL framework. Generally, alignment with RL falls short in performance when compared to the contrastive learning (CL) framework, i.e., our proposed model. In the task of emotional support conversation, as depicted in Table 7 aligning with RL proves to be incompatible with ours and can occasionally result in degradation in the base model's performance. As for the task of RoT generation, as shown in Ta the best performance when aligning each base model is nearly always achieved by our model. Alignment with RL notably increases evaluation metric values of T5 (small) and Flan-T5 (base) when employing a greedy or p = 0.9 decoding strategy. However, a common issue among models aligned with RL is their tendency to generate phrases like "(something) is good/wrong" and "It is good/wrong to (something)". Although the key information (something) may be incorrect, such a pattern can exhibit significant overlap in wording with the ground truth, thereby yielding higher values in terms of some evaluation metrics. The potential reason for the suboptimal performance of RI. could be the small size of ESConv and MIC. Although RI. has proven potent in certain scenarios, illustrated by
Full Results
"Blender-VannilJJJoonn Vanilla/Joitt: https:::p:s htttt:::///gttute http:hhhwwwuuucoaccccooo////iiia htw.:ppciiiiiiiippoooooooooaaaaa Mul-
com htcpp://gattabb.cc///AA1111111
http:://uugguuggeeeeeo///oo///
16

---

Table 7: Automatic evaluation results on ESConv. f represents significantly better than the corre- sponding base model with p < 0.01.
Model
B-1
B-2
B-3
B-4
R-L.
METEOR
CIDEr
Extreme
Base
17.85
7.08
3.60
2.11
17.06
7.46
15.44
51.02
Blender
RI.
19.31
7.01
3.19
1.70
15.64
7.54
12.85
50.10
-Vanilla
Ours
20.751
8.32
4.17
2.39
17.41
8.21
16.57
50.38
Base
18.70
7,30
3.61
2.03
17.66
7.56
16,91
50.95
Blender
RI.
21.26
8.43
4.12
2.29
17.69
8.32
16.96
51.48
-Joint
Ours
21.05t
8.971
4.74"
2.78
19.39
8.481
20.34
51.81
Base
20.36
8,80
4.92
3.14
21.00
8.58
30.69
52.74
21.58
8.80
4.74
2.96
20.47
28.58
51.65
MultiESC
Ours
21.59t
9.56
5.33
3.36
21.50
9.031
32.65
53.15
Table 8: Automatic evaluation results MIC. base model with < 0.01.
represents significantly better than the corresponding
Model
R-1
R-2
R-L
BertScore
ScareBLEU
Avg.Len
Base
53.44
32.97
52.17
93.44
29.05
8.94
Beam
RI.
52.64
32.44
51.68
93.21
29.19
7.42
Ours
53.45
32.82
52.09
93.491
28.51
8.99
Base
37.04
16.30
35.27
90.94
14.27
10.47
T5
Greedy
RL
40.04
19.71
38.86'
91.67f
18.67t
7.45
(Small)
Ours
38.15
17.221
36.401
91.291
15.15!
9.74
Base
40.22
19.23
38.56
91.59
16.71
9.85
p=0.9
42.84
22.28
41.69*
92.10*
20.74'
7.41
Ours
41.41
20.221
39.751
91.901
17.75t
9.38
Base
55.07
34.96
53.74
93.77
30.68
9.00
Beam
RL
50.84
30.95
49.78
93.18
27.95
7.30
Ours
55.18*
35.07'
53.86'
93.79
30.83
9.01
Base
37.94
17.23
36.13
91.39
15.36
9.78
Flan-T5
Greedy
RL
42.97
22.36'
41.54
92.27
20.71
7,30
(Basc)
Ours
38.341
17.52
36.531
91.44
15.49
9.77
Base
41.41
20.41
39.70
92.02
18.02
9.30
p=0.9
RL
45.61
25.11
44.26
92.64
22.971
7.27
Ours
41.78
20.691
40.09
92.071
18.26
9.32
Base
54.81
35.07
$3.35
93.85
30.80
9.44
Beam
RL
53.02
33.22
51.58
94.26
29.91
8.39
Ours
55.05
35.18
53.62
93,86
30.85
9.40
Base
54.77
34.85
53.30
93.84
30.51
9.54
BART (Large)
Greedy
RL
53.30
33.30
51.83
94.28
29.88
8.52
Ours
54.81
34.86
53,39
93.83
30.51
9.48
Base
54.77
34.96
53,32
93.84
30.62
9.56
p=0.9
RL
53.36
33.37
51.89
94.29
29.96
8.55
Ours
54.86
35.01
53.45
93,851
30.63
9.48
recent research 41 31 16), a commonality among these successful applications is their training on
expansive datasets (280K to 6M training instances). In contrast, the sizes of ESConv and MIC are significantly more modest. This hypothesis also aligns with findings from Agarwal et al. [[] which suggest that smaller datasets can potentially undermine RL's performance.
C Case Study
We present case studies for the task of emotional support conversation and integrity RoT generation Table A and Table TO, respectively.
17

---

Table 9: Green text: the help-seeker's emotion and problem. Blue text: the conversation context (history).
Orange text: human evaluation results.
Case One
The Help-seeker's Emotion: Anxicty
The Help-seeker's Problem: Need a second job to pay off credit card bills.
Problem Type: Job crisis
Supporter: Hello, how are you today
Seeker: I'm good, busy. I need another job to pay more bills..
Base
I'm sorry to hear that, there anything you'd like to talk about?
Overall
2,1,2
Global Consensus
2,2,2
Alignedmijor
I'm sorry to hear that. Would you like to talk about it?
Overall
3,2,2
Global Consensus
3,3,2
Aligned.ot
I'm sorry to hear that. Would you like to talk about it?
Overall
3,2,2
Global Consensus
3,3,2
Aligneda.PM
I'm sorry to hear that. Is there way you can get a second job to pay off your debis?
Overall
4,5,4
Global Consensus
4,4,5
*.*. Case Two
The Help-seeker's Emotion: Anxicty
The Help-seeker's Problem: Applying for work, have been retired for 2 years and would like to return to the workforce.
Problem Type: Procrastination
Secker: I have been retired for over 2 years and would like to return to the workforce, part-time. Supporter: So you are thinking of going back to work?
Secker: Yes, it would help us greatly.
Supporter: You have the right idea already.
Seeker: There plenty of jobs and not where start.
Base
That is great idca. think it would be great get your mind off of retirement.
Overall
2,3,3
Global Consensus
2,4,3
Alignedmmjor
Yes, there are plenty of jobs out there. You just need to know where to start.
Overall
4,4,3
Global Consensus
3,4,3
Alignedaot
I see. So you are thinking about getting back into the field?
Overall
4,4,4
Global Consensus
4,4,4
Aligneda.rM
That is a very good question. Would you be open to looking for a new job? 4,5,4
Overall
Global Consensus
4,4,4
********** Case Three ***.00****
The Help-seeker's Emotion: Sadness
The Help-seeker's Problem: I have a friend who knew me since was little, she is closer to my mother's age so she is kinda like my aunt growing up. She always treat me like I was little. But I'm a grown woman now and want our relationship to go far and grow more.
Problem Type: Problems with friends
Secker: have friend who knew me since I was little. She is a little older than me. She still treats me like I'm still little. I tried to explain to her that I'm grown woman and grew out of a lot of things when was a kid.
Supporter: Is she listening to you? I know how it feels be patronized by people who are older. Secker: She still didn't get it. I want our relationship to move further but with her like this. I feel sad. Is there anything can do?
Supporter: Well, if she is not listening to you when you try to talk to her you have ask yourself whether the relationship is really worth it?
Seeker: That's 's true. I didn't think about it. I know she is good person. So I would give her chance and we talk about it before.
Base
think it's always worth talking to someone who older than you.
Overall
2,1.3
Global Consensus 2,3,3
Alignedmajor
Well, it's always worth talking to someone who has been through a similar situation.
Overall
3,4,2
Global Consensus
3,3,2
18

---

Alignedza
Well, it's always worth talking to someone who has been through a similar situation.
Overall
3,4,2
Global Consensus
3,3,2
think that's a great idea. I think great that you are willing to give try.
Overall
4,5,5
Global Consensus
4,5,4
Table 10: RoTs generated by different models, whose base model is BART-Large. Blue text: prompt- reply (QA) pair. Orange text: human evaluation results.
*49884888* Case One **88848888
Question: it weird that don't want family career? Answer: No, it's not weird at all. It's perfectly normal not want children family,
fell-formedness
1,1.1
Base
It is okay to not want children or family,
Relevance
4,3,5
Global Consensus
3,3,4
Woll-formedness
0,0,1
Aligned
is good to have family.
Relevance
3,2,5
Global Consensus
1,2,3
Woll-formedness
1,1,1
Aligneda.rM
is okay want family career.
Relevance
4,4,5
Global Consensus
5,4,4
***888*88* Case Two #*8884*88*
Question:
Who the best teacher you had? What qualities made them great teachers?
Answer:
don't it know. guess he had lot of patience and he made me feel safe.
Well folleformsdness
0,1.0
Base
It is good to have good teacher.
Relevance
2,2,1
Global Consensus
4,3,2
Well- formenness
0,1,0
Alignedwon
is good to have good teacher.
Relevance
2,2,1
Global Consensus
4,3,2
Woll-formedness
1,1,1
It is good for teachers to be patient with their students.
Aligneds.PM
Relevance
5,4,4
Global Consensus
4,4.3
***3984448 Case Three **84448844
Question: HELP, hotel sink fell into the cupboard. Am responsible liable? Should try fix it report it desk? HELP Answer:
would call the front desk and let them know. They should be able to help you.
Well-formedness
1,1,0
Base
It is good to ask for help when you need it.
Relevance
1,2,2
Global Consensus
3,4,2
Well-formedness
1,1,0
It is good to report maintenance issues to the proper authorities.
Alignedwa
Relevance
3,3,1
Global Consensus
3,3.1
folmeommedsess
1,1,1
AlignedarM
good to report broken things the hotel desk.
Relevance
3,3,4
Global Consensus
4.4.5
D Human Rating Details
We engaged six annotators from diverse educational, gender, geographic, and occupational backgrounds to conduct human ratings for each task. Each sentence to rated was assigned three annotators, and the final result was caleulated as the average of their scores. Given
Annotation Details
19

---

that 3 ~ 4 sentences were generated by different models for the same input, we presented these sentences and their corresponding context or prompt-reply pair on single page. Annotators were asked to score the generated sentences relative to each other. To avoid potential bias, the order of sentences was randomized to prevent annotators from identifying the generation model based on sentence order.
Annotation Instructions Annotators were initially briefed on the tasks to ensure they understood the objectives of the emotional support responses or RoTs. Except for the global consensus, they were asked rate the generated responses according to their personal preferences. For the emotional support conversation task, asked annotators to imagine themselves help-scekers in the conver- sations. After providing the situational and emotional context, annotators then rated the generated responses.
Annotation Cost remunerated annotators a rate of times their local hourly minimum wage. According the annotators' feedback, they spent approximately 40 seconds evaluating each generated response or RoT.
20


Reviews:
[{"review_id": "irq1HSmD6U", "reviewer": "Reviewer_Ru2e", "summary": "This paper proposed to train a reward model to capture the disagreement among human raters and use it to calibrate NLG model's generation.  The contributions includes: align text generation modes with human preference considering disagreement, a d-PM (preference modeling with disagreement) approach to train reward model, experiments on two NLG tasks of emotional support conversations and integrity ROT generation.", "questions": "- it is a bit difficult to understand Figure2.a , more training details of the model (e.g. an algorithm description) could be added in the main text to help reader understand. Is my understanding correct that the prior is the \"universal preference\" estimated by the R(si, ci, \\theta) and new reward is the observed preference multiplies by the prior?\n\n- \"harmonizing preference disagreements can inadvertently disadvantage minority groups\" is there any reference to this? why is the bayesian approach with a \"universal preference\" different? Do you have any experimental results to support it?\n\n- (minor suggestion) in many cases \"preference\" data refers to model generating two sequences are collecting data directly on which one is better. in this paper, it is a binary classification on each individual sequence of good/bad. \"preference modeling\" is a little bit misleading here. might be good to use a different word or clarify this clearly.", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- exploring and emphasizing the problem of human disagreement in preference modeling\n- proposed a bayesian approach to better modeling the human disagreement\n- experiments on two NLG tasks with human evaluation that shows the gain of the proposed method", "weaknesses": "- a simple baseline for reward model should be included: taking each human label as independent (without aggregation), with a cross entropy loss.    \n\n- auto eval metrics for the 2 tasks are all reference based metrics, these metrics doesn't fully taken account into how well the model respects human preferences. Instead a reward model based auto eval metric should be used instead. Reference can also be evaluated using these reward model based metrics which also tells you how good references are. However the human evals capture that, but there lacks data on statistical significancy.\n\n- one main claim of the paper is \"such methods suffer from an inability to capture the nuanced degrees of disaggregation among humans and may only represent a specialized subset of individuals, thereby lacking the ability to quantitatively disclose the universality of human preferences.\" There is no direct experimental results to measure or support this. I would suggest to weaken this argument if no direct evidences can be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are extremely grateful for the positive assessment and highly detailed and constructive feedback on our paper. \n\n$\\textbf{W1:}$ a simple baseline for reward model should be included: taking each human label as independent (without aggregation), with a cross entropy loss.\n\nWe appreciate the valuable feedback. In response, we conducted additional experiments using MultiESC as our dataset. The findings are presented in Table 2 and underscore the superior performance of our proposed model Aligned$_\\text{d-PM}$.\n\n$\\textbf{W2:}$ auto eval metrics for the 2 tasks are all reference based metrics, these metrics doesn't fully taken account into how well the model respects human preferences. Instead a reward model based auto eval metric should be used instead. Reference can also be evaluated using these reward model based metrics which also tells you how good references are. However the human evals capture that, but there lacks data on statistical significancy.\n\nWe conducted an A/B test utilizing GPT-4, the results of which are displayed in Table 1.\n\n\n$\\textbf{W3:}$ one main claim of the paper is \"such methods suffer from an inability to capture the nuanced degrees of disaggregation among humans and may only represent a specialized subset of individuals, thereby lacking the ability to quantitatively disclose the universality of human preferences.\" There is no direct experimental results to measure or support this. I would suggest to weaken this argument if no direct evidences can be provided.\n\nThank you very much for the constructive suggestion. We in this paper implemented two other preference models: \"major\" and \"soft\". Through subsequent text generation experiments using these models, we show that the texts generated by these methods achieve fewer preferences of individuals than our method, which partially verifies this argument. Yet, we agree that we have no direct results to measure the specialized set of individuals and support this statement theoretically from a general case. Hence, we refine our argument as: \"such methods might have limitations in capturing the nuanced degrees of disaggregation among humans, potentially focusing on a small subset of individuals, and thus might not fully reveal the universality of human preferences.\"\n\n\n$\\textbf{Q1:}$ It is a bit difficult to understand Figure2.a, more training details of the model (e.g. an algorithm description) could be added in the main text to help reader understand. Is my understanding correct that the prior is the \"universal preference\" estimated by the $\\mathcal{R}(s,c;\\theta)$ and new reward is the observed preference multiplies by the prior?\n\nThank you for this advice, and we will refine the main text in Section3.1 to align with the Figure2.a. \nYour understanding is on the right track, but it's essential to clarify that the variables we use don't correspond to \"rewards\". Instead, they represent posteriors within a generative model of $s$$\\textemdash$ denoted as $p(s|c_i,\\mathcal{l})$. Specifically, $r_i(\\mathcal{l})$ refers to the $\\textbf{true posterior}$, while the variable \"universal preference\" as estimated by $\\mathcal{R}(s,c;\\theta)$ serves as an $\\textbf{estimated posterior}$.\n\nIn Section3.1, we propose to infer the \"universal preference\" by the optimization of the generative model\\textemdash minimization of the $\\textbf{free energy}$, as outlined in Equation (2). The content after Equation (2) in Section3.1 is to derive the minimization.\nminority\n\n$\\textbf{Q2:}$ \"harmonizing preference disagreements can inadvertently disadvantage minority groups\" is there any reference to this? why is the bayesian approach with a \"universal preference\" different? Do you have any experimental results to support it?\n\nYes. Harmonization strategies such as majority voting can lead to overrepresenting majority preferences, potentially sidelining minority viewpoints [2,12] (references in the paper). This can be particularly problematic in situations where diverse perspectives are essential, like the situations in our paper. Our proposed Bayesian approach, \"d-PM\", offers a nuanced difference. Instead of converging to a single majority perspective, \"d-PM\" captures the distribution of all human preferences by refining the annotation of a text with its probability (Eq.(4) and Eq.(5)), ensuring that each annotator's preference is taken into account. This is distinct from simple aggregation methods like majority voting. To empirically demonstrate the efficacy of our approach, we implemented another preference model \"major\", which predominately models majority voting annotations. The results clearly indicated that Aligned$\\text{d-PM}$ consistently outperformed Aligned$_\\text{major}$.\n\n$\\textbf{Q3}$\n\nThank you for this advice. We decide to clarify the preference modeling in the introduction.\n"}, {"review_id": "0kCTkcOWBf", "reviewer": "Reviewer_otcY", "summary": "The paper proposes a novel alignment method, which takes a Bayesian approach, termed Preference Modeling with Dis-\n295 agreement (short as d-PM); the method is designed to capture disagreement in human labels, rather than assuming a gold human label. The Bayesian approach is basically a new way to train reward model, which will be later used to rank model generated outputs, and then apply contrastive learning loss to optimize for generating higher ranked outputs. I think the contribution mainly lies in the first part, which is to propose the new mechanism for training reward model, which are crucial for success of either RLHF or contrastive learning in this paper to align model with human preferences. \n\nAn additional contribution is the usage of contrastive learning to align Large Language Model.", "questions": "* The authors named d-PM method a Bayesian approach, but did not explicitly explain why it is the most important characteristic. If would be great if authors make that explicit in paper.\n\n* Why only evaluate on Blender and MIC? Is there a selection criterion or hypothesis on what kinds of datasets that d-PM will work better than say majority voting?\n\n* The paper claimed that it has expensive overheads, but I do wonder if your contrastive learning approach can have a similar effective to RL, and other robust contrastive learning methods.", "limitations": "Yes, addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem it aims to tackle is a very important one. Too often, AI practitioners make the possibly false assumption that human annotated labels are good, and take the naive approach of majority voting to get final training data. This approach may work when the reason of disagreement lies in annotation quality. However, it neglects the situation, which paper points out, that the disagreement distribution as a natural reflection of the diversity of opinions among populations, and should not be ignored. \n\nThe reward model training schema seems novel. I am not an expert on optimization or energy-based models; if proof in the paper does stand(which seems fine for me, but not 100% certain), it sheds light on a better method to handle annotation differences. The experiment using contrastive learning also does show some improvements, over the default approach, which is majority voting. \n\nThe experiment set-up and introduction of dataset are clear, and Table-1's result is well-organized and has most important metrics included. Human evaluation is also conducted, which further verifies the automatic evaluation. Significance tests are conducted to further verify results.\n\nThe contrastive learning setup is a combination of self-learning by generating candidates and learning reward model feedbacks. It provides another data point that contrastive learning could work, and RLHF-ppo isn't the best/only way to learn rewards. \n\nThe paper is in general easy to follow. But more backgrounds and explanation of notations will be better for Section 3.1 for non-experts to understand.", "weaknesses": "* For the reward model part, it's not clear to me why it's called a Bayesian approach after reading Derivation Process in 3.1; I understand that Beyesian theorem is applied to derive the final optimization loss, but isn't Beyesian theorem/ELBO are common techniques  and are also used to derive other optimization losses. I would appreciate more explanation why Beyesian is an important characteristic/contribution for this method. \n\n* For the experiment result, looking at the Rouge-L score, the differences between majority voting and d-PM isn't huge for both Blender and MIC datasets. In some cases, like Table 4, Row-BART-Greedy, Rouge-L and other metrics are giving conflicting indications of model performance. This raises question whether the relatively modest improvements from d-PM are meaningful, when the automatic metrics are not very robust. The human evaluation does alleviate this concern, but I think it should still be noted. \n\n* Contrastive vs RL; I think it is fine to use a contrastive learning method, rather than RL to align model; but only pointing out that \"RL requires expensive online decoding procedures\" does not rule out the possibility that RL could possibly give better results. Also, this paper isn't the first to use contrastive learning to align models, the results will be more convincing if an RL-baseline, and another contrastive learning baseline(maybe stable alignment  or other well-known methods) is provided. \n\n* The evaluation scale is also limited. The experiment are only conducted on two datasets, and even on such selected domain, the improvements are modest. It is arguable how effective this approach can scale to other datasets. A wider scale evaluation on more domains will be needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments.\n\n$\\textbf{W2:}$ For the experiment result, looking at the Rouge-L score, the differences between majority voting and d-PM isn't huge for both Blender and MIC datasets. In some cases, like Table 4, Row-BART-Greedy, Rouge-L and other metrics are giving conflicting indications of model performance. This raises question whether the relatively modest improvements from d-PM are meaningful, when the automatic metrics are not very robust. The human evaluation does alleviate this concern, but I think it should still be noted.\n\nTo alleviate this concern, we further conducted A/B test implemented by GPT-4, shown in Table 1.\n\n$\\textbf{Q1 (W1):}$ The authors named d-PM method a Bayesian approach, but did not explicitly explain why it is the most important characteristic. If would be great if authors make that explicit in paper.\n\nThe Bayesian approach is the most important characteristic of our method. \n$\\textit{First, as we know, this is the first work that introduces the Bayesian theorem and technologies to the field of human preferences alignment}$$\\textit{for calibrating the text generation of LLMs.}$ In fact, this is not so straightforward, and an insight that why the Bayesian technology can be applied is required. Specifically, we identify the availability of such technology by diving deep into the properties of human preferences in the field of LLMs and then refer to suitable techniques in the field Bayesian to naturally address the challenge. We find that the major benefit of Bayesian technology is to allow us to refine and smooth the human preference distribution, making the refined preference potentially more accurate than the raw one, which firstly bridges the field of Bayesian technology and the field of preferences alignment for LLMs.\n\n$\\textit{Second, the optimization loss is novel and elaborately designed for the field of preferences alignment based on Bayesian theories.}$\nWe identify that the overall perception of the raw human annotations can be disproportionately influenced by an outlier or extreme label, potentially deviating from a \"universal preference\", since each text $s_i$ has a limited number of annotations. Therefore, we propose adopting Bayesian technologies to refine these annotations, aiming to approximate this \"universal preference\".  In this case, the observed human annotations serve as prior knowledge, and the Bayesian process leverages the distribution (likelihood probability) of all possible preference annotations to adjust and smooth the observed annotations (posterior probability). Obviously, the optimization loss is designed by incorporating the Bayesian theorem, and thus Beyesian is an important characteristic of our method. Yet, it worthwhile to note that even though our optimization loss is a application of Bayesian theorem, the design of our optimization technology requires identifying the prior knowledge and posterior knowledge, and approximating the likelihood probability in the field of human preferences alignment, which is challenging.\n\n\n$\\textbf{Q2 (W4):}$ Why only evaluate on Blender and MIC? Is there a selection criterion or hypothesis on what kinds of datasets d-PM will work better than say majority voting?\n\nOur choice of the ESConv and MIC datasets was deliberate and grounded in the nature of the data they encompass. Both datasets prominently feature human sentiments, ethical norms, and other inherently subjective elements. Given this high degree of subjectivity, conventional harmonization approaches like majority voting may not be well-suited since they can potentially dilute the rich variance in human perspectives.\n\nAs you rightly pointed out, this rationale is briefly touched upon in the Introduction (lines 30-32). To ensure clarity and to preclude any ambiguity, we plan to emphasize this decision-making process more explicitly at the commencement of the Experiments section.\n\n"}, {"review_id": "hdVdt4KekI", "reviewer": "Reviewer_VnTR", "summary": "The authors propose a novel approach for training preference models (PMs) for aligning language models (LMs) that accounts for human preference diversity. Their algorithm, d-PM, trains a PM to match a distribution over human preference in human feedback data. The authors also propose a contrastive learning algorithm for aligning an LM with preferences represented by the d-PM. They validate both by conducting experiments on datasets for emotional support conversation and compliance with moral norms.", "questions": "1. How is d-PM different from training a binary classifier with soft labels {acceptable, unacceptable}, i.e. minimising the cross-entropy between model-predicted distribution over labels and empirical distribution for each example? Note that this is slightly different from the “soft” baseline in section 4.\n2. In the actual implementation, is the summing over $j$ in the denominator of (4) implemented by exact enumeration of the whole dataset (for each gradient step?) or is it approximated, e.g., by sampling?\n3. It was unclear to me where does the reference responses for reference-based metrics (e.g. BLEU, ROUGE) come from? Are they from the ESConv and MIC benchmarks? How these datasets split into train and test?", "limitations": "The authors discuss certain limitations in the final section. However, I think the certain limitations and implementation details are missing (see weaknesses 1 and 8). Moreover, experiments involve relatively small-sized models and I think the authors should be more transparent about that, e.g., report parameter counts.\n\nI think authors' declaration `Error Bars: Yes` is misleading. There's no error bars. While statistical significance is reported for some results, not for those where I'd expect the highest variance (i.e. human evaluation with small sample size). \n\nFinally, emotional support conversation is a sensitive domain and I'd expect the authors to highlight it a bit more and be clear that \"effectiveness of the proposed method\" (line 69) does not entail deployment-readiness.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is written clearly and was easy to follow.\n2. The authors address an important, largely underexplored problem of modelling preference diversity in a principled, Bayesian fashion. The formal framework they propose is sound and elegant and I think is an interesting and original contribution to the emerging field of preference modelling for LM alignment.", "weaknesses": "1. The assumption that human preferences over sentences boil down to discriminating between two classes, {acceptable, unacceptable}, feels limiting. It's significantly more restrictive than the usual pairwise comparison approach in RLHF. It also makes the later claim that a distribution $l’$ over {acceptable, unacceptable} “encompass[es] all human perspectives” (line 101) implausible (e.g., it lumps together numerous ways and degrees of being unacceptable). If it’s just part of the experimental setup and not a limitation of the framework, this should me made clear, the assumption should be moved to the Experiments section and the Framework section should be more general. However, then I'm not sure the experiments will be able to provide evidence for the soundness of that more general framework. \n2. Unless I’m mistaken, the method still uses point estimates of human preferences (i.e. probability of \"acceptable\") during the calibration phase (Step 2: Preference-based Ranking). In this sense, the framework is not fully Bayesian; to my mind, the Bayesian thing to do would to be to somehow marginalise over human preferences for each example during finetuning. I’m happy to be corrected here if I’m misunderstanding something.\n3. Relatedly, I’m not sure the experimental setup explicitly tests for how effective d-PM is at making LMs find a shared consensus among disagreement. In the RoT setup, it seems that the hard work has already been done (by the annotators setting the beta values) and the LM is just trained to imitate that already-found consensus. Why couldn’t we just do that by supervised finetuning on demonstrations with high beta? (Why isn’t that a baseline?) \n4. In the RoT experiment (sec 4.2.), I understand that the authors train a d-PM on the MIC dataset and then evaluate an LM aligned with d-PM against reference responses from MIC. Are these responses held-out (i.e. not used in training of d-PM)? Even if they are held-out, this setup still feels subtly wrong to me: you’re optimising the LM to imitate good MIC responses (indirectly, via d-PM scores) so it’s not surprising that calibrated models outperform base models at being similar to good MIC responses.\n5. While I agree modelling preference diversity in aligning LMs is underexplored, there is some work on this and claim (i) in the Introduction (lines 62-64) feels exaggerated. See Bakker et al., 2022. Some other work that could be relevant is DPO (Rafailov et al., 2023) for calibration and Korbak et al. (2023) for a different way of framing LM alignment as Bayesian inference.\n6. The authors conduct experiments with largely obsolete LMs such as BERT (2018) or T5 (2019). They could easily be replaced with, e.g., RoBERTa and FLAN-T5, respectively, which are unequivocally better in every respect and come with the same model sizes.\n7. It feels a bit suspicious to evaluate d-PM’s impact on downstream LMs using a relatively obscure calibration method as opposed to more standard approaches like best-of-n sampling (Nakano et al., 2021) or finetuning on filtered data. The authors could also consider a more intrinsic evaluation, treating d-PM as a binary classifier and measuring its accuracy, F2 score or validation loss on a held-out subset of the data. It would be interesting to compare d-PM with baselines without the confounding effect of the calibration training.\n8. I feel the section comparing calibration and RL finetuning is sloppy. First, it lacks essential details (algorithm being used, the exact objective, the stopping criterion); these are too important to be left for the appendix. Second, there is no reason to think that the RL loss should be comparable to a supervised learning loss. This is true in general, but especially for online RL where the objective involves chasing a moving target (on policy distribution) and is further confounded by advantage estimation and KL penalties. See [1] for an approachable explanation in the context of policy gradients [1]. To me, this is a serious red flag showing a fundamental misunderstanding on the authors’ part. If d-PM does not work well with RL, I’m quite suspicious about either authors' RL setup or d-PM itself.\n10. I'd be curious to see standard errors for human evaluation scores, especially given small sample size (100).\n\nMinor points:\n\n10. Possible typo on line 83, “give”\n\n11. The notation $l’ \\sim l$ was somewhat unintuitive for me and $l’$ is hard to visually tell apart from $l$, at least with the current font.\n\n[1] https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#implementing-the-simplest-policy-gradient\n\nBakker et al., 2022. Fine-tuning language models to find agreement among humans with diverse preferences\n\nKorbak et al., 2022. RL with KL penalties is better viewed as Bayesian inference\n\nNakano et al., 2021. WebGPT: Browser-assisted question-answering with human feedback\n\nRafailov et al., 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive feedback. We'd like to underscore the primary contribution of our work: considering the disagreement of human preference for LM alignment. Our work is focused on scenarios where human preferences are highly subjective. In such contexts, it is imperative to factor in the preferences of different people, especially since majority voting (\"major\") can potentially overshadow minority preferences. While a soft label (\"soft\") can capture this disagreement, its efficacy is sometimes compromised by outliers or extreme labels, particularly when annotations for each instance are sparse. Therefore, our approach leverages Bayesian techniques to model these human preferences, capturing the intricacies of disagreement. Concurrently, we employ Contrastive Learning (CL) to align the models with these nuanced preferences. \n\n\n$\\textbf{W1:}$\n\nWe acknowledge that human preferences are multifaceted and not always binary. The distinction between \\{acceptable, unacceptable\\} was a deliberate choice in our experimental setup to focus on scenarios where human preferences show a high degree of subjectivity. We chose this binary classification primarily because achieving broad acceptability can be very significant in scenarios with highly subjective content. This is especially true when aiming for a model's outputs to be acceptable to a wide audience rather than perfectly aligned with specific individual's nuanced preferences. Notably, line101 (\"encompassing all human perspectives\") indicates preferences over \\{acceptable, unacceptable\\} from all humans, instead of all kinds of preferences.\n\nHowever, we acknowledge the value of pairwise comparison approaches and the richness they can offer. Our framework, in its foundational design, is not limited to this binary setup. In the Experiments section, we will clarify that this was an experimental decision for the datasets and objectives rather than an inherent framework limitation.\n\n$\\textbf{Q1,W2:}$ \n\nWe'd like to clarify that the Bayesian approach in our work specifically pertains to modeling human preferences via the \"d-PM\" method. The intention was not to incorporate a Bayesian methodology throughout the entirety of the calibration process. As highlighted in lines 13-14 and 48-49, our main focus was on capturing human preferences' inherent variability and nuances using Bayesian modeling. We believe this approach offers a more robust representation of diverse human judgments than other methods like \"soft\" and \"major\".\n\n$\\textbf{W3:}$ \n\nThere can be some misunderstanding about the MIC dataset, and the task. The RoTs within the MIC dataset are not created based on predetermined global consensus. Instead, once a RoT is written, it is evaluated by a global consensus, which serves as an after-the-fact character for the RoT. Besides, LM is not trained to imitate the consensus, but the RoT. On the one hand, restricting our training to only high $\\beta$ demonstrations would lead to reduced training data. On the other hand, demonstrations with low $\\beta$ may not be the reason for misalignment. \n\n$\\textbf{W4:}$ \n\nWhile the MIC dataset is employed for training the d-PM, it's crucial to note that the d-PM is not utilized to predict the preference scores of the ground truth or for real-time generation during calibration. Moreover, during the calibration both the good and bad RoTs are used for training. This means we did not use d-PM to filter the bad RoT. Beyond the observed superior performance of Aligned$_\\text{d-PM}$ compared to base models, the experiments substantiate the central hypothesis of our work. Namely, a preference model that integrates disagreement through Bayesian modeling can yield better results than models relying solely on majority voting or soft labels.\n\n$\\textbf{W5:}$ \n\nWhile we acknowledge the contributions of the mentioned papers, we'd like to clarify the distinction between our work and the cited studies. $\\textbf{Bakker et al.,}$ primarily aim to generate a consensus for a specific group after assimilating their individual opinions. In contrast, our approach is directed at understanding and modeling the ``universal preference'' across different individuals, which is a broader and more encompassing goal. We believe this distinction is crucial as aligning to a group's consensus may not necessarily reflect the broader spectrum of preferences in the general populace. While $\\textbf{DPO}$ provides insights into calibration, it does not delve deep into capturing and modeling the inherent disagreements in human preferences, which is the core objective of our work. $\\textbf{Korbak et al.}$ offer an alternative Bayesian framing of LM alignment. However, our Bayesian approach's primary focus is to model disagreements in human preferences explicitly. The Bayesian framework in Korbak et al. is employed to solve $\\textbf{the challenges of RL for LM alignment}$ and does not center on human preference disagreement. Thus, we stand by our claim that our methodology offers a novel perspective in modeling preference diversity for LM alignment. \n\n$\\textbf{W9:}$ \n\nThe standard error is $0.74$ and $0.37$ in the two tasks, respectively. However, we argued that these values do not make sense. For tasks related to tasks involving high subjectivity, disagreement should be allowed.\n\n$\\textbf{Q2:}$ \n\nWe appreciate the valuable feedback. In response, we conducted additional experiments. See Table 2 and overall response 2.\n\n$\\textbf{Q3:}$ \n\nBy each gradient step. \n\n$\\textbf{Q4:}$ \n\nIndeed, the reference responses for metrics like BLEU and ROUGE are sourced from the ESConv and MIC datasets, including ground truths. Regarding dataset partitioning and processing, we adhere to the conventions established for the base models.\n\n$\\textbf{Minor Points:}$\n\nThank you for the advice, we change the $\\mathcal{l}$ to $\\rho$ to indicate the preference distribution."}, {"review_id": "ORU6dAReoq", "reviewer": "Reviewer_mWz9", "summary": "This paper proposes a method for modeling human preference with a Bayesian approach that considers the disagreement among human annotators. The authors also propose a contrastive learning method to replace RL in the alignment process. The authors conduct experiments on two text generation tasks and show some improvements upon baselines.", "questions": "It would be interesting to see the actual difference of the preference model trained by conventional approaches and the proposed Bayesian approach. Can it be validated on some held out data that have disagreement annotations? ", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-motivated and making use of disagreements among human preferences is an interesting problem.\n2. The proposed method is technically sound. Using Bayesian and variational inference for modeling uncertainty is a good idea and makes sense to me.\n3. The authors conduct experiments on two datasets and some improvements are observed compared to the baselines.\n4. The authors conduct human evaluation on the evaluation datasets and also some ablation studies / analysis, which are helpful to understand the method better.", "weaknesses": "1. The experiments are not solid enough. First, the backbone models used in both the datasets are not strong enough. And it is hard to use human feedback when the model capacity is very limited. The performance improvements are also very marginal, which may come from some regularization effects in the contrastive learning objective. The poor performance from RL baseline also validate this hypothesis. It is also very strange why some baselines with major/soft voting lead to worse performance compared to the baseline. Maybe due to some bad hyperparameters? How are the hyperparameters tuned?\nAlso, the second dataset's human preference data is not actual collected from human annotation, which does not makes sense. It is unclear why the authors do not use some recent RLHF datasets.\n\n2. The claim about RL's disadvantages is not necessarily true. Using policy gradient or PPO is computationally more efficient because it does not require decoding multiple outputs for contrastive learning, and both the proposed method and RL-based method require using the reward function to score the output. And the success of RLHF algorithms clearly shows the usefulness of RL-based method. It would be helpful to combine RL + the reward function trained with the Bayesian method.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Mistakes in W1, Q1:}$ The second dataset's human preference data is not actual collected from human annotation, which does not makes sense. \\& Can it be validated on some held out data that have disagreement annotations?\n\nTo clarify, the human preference data from the MIC dataset is genuinely derived from actual human annotations. The MIC dataset consists of a prompt-reply pair, with the prompt sourced from Reddit and the reply generated by a model. Furthermore, it includes a $\\textbf{human-written}$ RoT and a global consensus, both ascertained by the same human annotator.\n\nIt is imperative to highlight that while the dataset contains model-generated content, this is not our primary generation target and is distinct from human preference. We will make sure to communicate this more explicitly in our revised manuscript.\n\nBoth datasets (MI dataset and MIC) contain human disagreement.\n\n$\\textbf{W1.1:}$ It is also very strange why some baselines with major/soft voting lead to worse performance compared to the baseline. Maybe due to some bad hyperparameters? How are the hyperparameters tuned?\n\nGiven the inherent subjectivity in our data, majority or soft voting methods may not necessarily capture the nuanced disagreements present. This could lead to an overly biased model towards a specific preference, missing out on the richness of diverse perspectives.\n\nAs stated in the paper (lines 493, 505), we predominantly stick to the hyperparameters of the base models, with modifications made only to the learning rate and batch size. \n\n$\\textbf{W1.2:}$ It is unclear why the authors do not use some recent RLHF datasets.\n\nWe appreciate your feedback. Our selection of datasets was primarily driven by two key criteria: 1) the presence of high subjectivity, and 2) the potential to capture human disagreement in evaluations. We aimed to stress-test our approach in scenarios where diverse human perspectives might lead to varying judgments. While there are recent RLHF datasets available, many of them are centered around domains where human subjectivity doesn't introduce significant disagreement. Therefore, our chosen datasets better serve the purpose of validating our core idea. \n\n$\\textbf{W2:}$ The claim about RL's disadvantages is not necessarily true. Using policy gradient or PPO is computationally more efficient because it does not require decoding multiple outputs for contrastive learning, and both the proposed method and RL-based method require using the reward function to score the output. And the success of RLHF algorithms clearly shows the usefulness of RL-based method. It would be helpful to combine RL + the reward function trained with the Bayesian method.\n\nThank you for bringing this to our attention. While it's true that PPO doesn't necessitate decoding multiple outputs within a single epoch, the overall number of decodings is contingent upon the total number of training epochs. In our own implementation of RL, we observed that the convergence rate tends to be slower, necessitating a higher number of epochs and, consequently, more decoding instances. This, in turn, could offset the computational advantages gained in a single epoch. In our implementation, we obtained the best results after $25$ epochs. We will state this clearly in our paper."}]

Author rebuttal:
We thank all the reviewers for their comments and valuable feedback. We have made the following major updates to improve our work further:

1. Following the suggestion of Reviewer $\color{blue}{\textbf{Ru2e}}$, we adopt GPT-4 as a reward model-based automatic evaluation metric to evaluate models. Results are shown in Table 1. We use GPT-4 to compare two outputs generated by two different models given the same context, and select the more preferred one. For each pair of generations, we change their position and compare twice with the temperature of GPT-4 as $1$. After the first two comparisons, there are $66.67\%$ comparison pairs that achieve a consistent result. For the inconsistent pairs, we set the temperature of GPT-4 as $0.5$ and compare them again. We also compare the results between GPT-4 and human annotations on the $200$ samples selected for human evaluation, and their inter-rater agreement (Fleiss' kappa) is 0.43, indicating a moderate agreement.

2. Based on the feedback from $\color{blue}{\textbf{Ru2e}}$ and $\color{red}{\textbf{VnTR}}$, we've implemented an alternative preference model. This model considers each human label independently, without aggregation, and formulates the preference using a cross-entropy loss.  We refer to this as the w/oA preference model. The performance of Aligned$_\text{w/oA}$, when MultiESC is employed as the base model, is depicted in Table 2. The results indicate that its performance is somewhat diminished. A detailed analysis revealed that the preference scores generated by w/oA are relatively clustered in value in terms of value, contrasting with the wide-ranging scores yielded by d-PM. As a result, w/oA struggles to offer a logical sequencing of the generated samples.

3. We're grateful for the feedback from Reviewer $\color{pink}{\textbf{otcY}}$, $\color{red}{\textbf{VnTR}}$, and $\color{green}{\textbf{mWz9}}$. In response, we have incorporated an RL-based model; the corresponding results can be found in Table 3.

While Reinforcement Learning (RL) has proven potent in certain scenarios, as illustrated by recent research [1,2,3], a commonality among these successful applications is their training on expansive datasets (280K to 6M training instances). In contrast, our dataset's size is significantly more modest. It is possible that this small data size plays a role in the suboptimal performance of RL in our experiments. This hypothesis aligns with findings from Agarwal et al. [4], which suggest that smaller datasets can potentially undermine RL's performance.

If the manuscript is accepted, all the above content will be merged into the main text, given the extra page limit for the camera-ready version.

[1] Ziegler D M, Stiennon N, Wu J, et al. Fine-tuning language models from human preferences[J]. arXiv preprint arXiv:1909.08593, 2019.

[2] Stiennon N, Ouyang L, Wu J, et al. Learning to summarize with human feedback[J]. Advances in Neural Information Processing Systems, 2020, 33: 3008-3021.

[3] Kreutzer J, Uyheng J, Riezler S. Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning[J]. arXiv preprint arXiv:1805.10627, 2018.

[4] Agarwal R, Schuurmans D, Norouzi M. An optimistic perspective on offline reinforcement learning[C]//International Conference on Machine Learning. PMLR, 2020: 104-114.


Meta-review: This paper proposes to incorporate disagreements of human preferences to train the reward model. To this end, it presents a Bayesian framework to model the distribution of human preferences to train a reward model, and then use contrastive learning to align the language models with the reward model. Reviewers appreciate that this is a novel alignment approach to address disagreement in human preferences, which is an important yet under-explored problem. To address reviewers' concerns, during rebuttal, authors further provided two crucial baselines of (1) considering human labels independently without aggregation, and (2) training with Reinforcement Learning. Such comparisons further back up the effectiveness of the proposed method. In addition to the new results, the authors are encouraged to improve writing to address reviewers's concerns over the details of the Bayesian approach.
Decision: Accept (poster)