Title: Aligning Language Models with Human Preferences via a Bayesian Approach
Venue: NeurIPS 2023
Accepted: yes
Keywords: Aligned Models; Human-centric NLG
Abstract: In the quest to advance human-centric natural language generation (NLG) systems, ensuring alignment between NLG models and human preferences is crucial. For this alignment, current popular methods leverage a reinforcement learning (RL) approach with a reward model trained on feedback from humans. However, inherent disagreements due to the subjective nature of human preferences pose a significant challenge for training the reward model, resulting in a deterioration of the NLG performance. To tackle this issue, previous approaches typically rely on majority voting or averaging to consolidate multiple inconsistent preferences into a merged one. Although straightforward to understand and execute, such methods suffer from an inability to capture the nuanced degrees of disaggregation among humans and may only represent a specialized subset of individuals, thereby lacking the ability to quantitatively disclose the universality of human preferences. To address this challenge, this paper proposes a novel approach, which employs a Bayesian framework to account for the distribution of disagreements among human preferences as training a preference model, and names it as $\textbf{d-PM}$. 
Besides, considering the RL strategy's inefficient and complex training process over the training efficiency, we further propose utilizing the contrastive learning strategy to train the NLG model with the preference scores derived from the d-PM model. Extensive experiments on two human-centric NLG tasks, i.e., emotional support conversation and integrity ``Rule-of-Thumb'' generation, show that our method consistently exceeds previous SOTA models in both automatic and human evaluations.

Aligning Language Models with Human Preferences via a Bayesian Approach

Abstract

In the quest to advance human-centric natural language generation (NLG) systems, ensuring alignment between NLG models and human preferences is crucial. For this alignment, current popular methods leverage a reinforcement learning (RL) approach with a reward model trained on feedback from humans. However, inherent disagreements due to the subjective nature of human preferences pose a significant challenge for training the reward model, resulting in a deterioration of the NLG performance. To tackle this issue, previous approaches typically rely on majority voting or averaging to consolidate multiple inconsistent preferences into a merged one. Although straightforward to understand and execute, such methods suffer from an inability to capture the nuanced degrees of disaggregation among humans and may only represent a specialized subset of individuals, thereby lacking the ability to quantitatively disclose the universality of human preferences. To address this challenge, this paper proposes a novel approach, which employs a Bayesian framework to account for the distribution of disagreements among human preferences as training a preference model, and names it as d-PM . Besides, considering the RL strategy's inefficient and complex training process over the training efficiency, we further propose utilizing the contrastive learning strategy to train the NLG model with the preference scores derived from the d-PM model. Extensive experiments on two human-centric NLG tasks, i.e., emotional support conversation and integrity 'Rule-of-Thumb' generation, show that our method consistently exceeds previous SOTA models in both automatic and human evaluations.

1 Introduction

Human-centric natural language processing (NLP) aims to develop NLP systems that are finely attuned to human preferences [14, 12, 30]. Consequently, learning from human feedback is wellsuited for training models in human-centric NLG tasks [18, 28]. Currently, reinforcement learning (RL) with a reward model is the most popular method to align models with human preferences [26, 11, 41]. Its effectiveness depends heavily on how well human preferences are learned by the reward model [6, 4]. However, modeling human preferences can be challenging.

Due to the high subjectivity of personal standards and human values, it can be difficult to reach a consensus on preferences among individuals, significantly increasing the learning difficulty. As depicted in Figure 1, persons may have varied preferred responses with inconsistent emotions and values given the same context. To tackle this challenge, existing methods mostly adopt aggregation techniques such as majority voting or averaging [11, 6, 4]. However, aggregated preferences potentially cater

a

controversial

response

&

feelings of

different

help-seekers

I can understand that. But you should first get the

sleep you need and stop thinking these things.

Help-Seeker

I indeed need to get enough sleep.

Thanks for understanding...

The problem is exactly that I cannot sleep!

...

I'm going through some things with myself. I barely sleep and do nothing but

think about how I am worthless. I'am afraid to talk to others about my issues.

Supporter A

a

less controversial

response & feelings of

different

help-seekers

You're not alone. In fact, most people have felt this

to some degree at some point in their life.

My issues may be not big deals.

I won't be laughed at if I tell friends.

So how can I solve it?

...

Supporter B

to specific subsets of people, risking generating controversial content (see Supporter A in Figure 1). Additionally, subjectivity and inconsistency are intrinsic components of certain tasks, such as emotion analysis [13] and ethical evaluation [10], necessitating careful consideration instead of dismissal [3]. Therefore, for widely acceptable and less controversial outputs, it is necessary for the resulting NLG systems to account for capturing disagreements inherent in human preferences [17].

In this paper, we introduce a novel Bayesian-based approach termed Preference Modeling with Disagreement ( d-PM ). This method is designed to approximate a 'universal preference' that comprises the preferences of 'all individuals', given the preferences of several individuals. Although a soft label, derived from these several individuals, can intuitively account for disagreement, outliers or extreme labels can disproportionately influence the overall perception. Therefore, we employ Bayesian inference to refine these preferences. Specifically, the observed preference among selected individuals serves as prior knowledge. Our d-PM aims to leverage distribution of all possible universal preferences (likelihood probability) to adjust and smooth the initially observed one, leading to the derivation of a universal preference (posterior). Upon obtaining the universal preference, we calculate the likelihood of the expected preference types to establish a preference score, which is then utilized for further language model alignment.

Based on the d-PM model, we further optimize the language models to generate widely acceptable and less controversial texts. Specifically, we propose utilizing the contrastive learning strategy to calibrate the generation model towards generating texts with high preference scores provided by d-PM. Although existing RL strategies can also be leveraged to make the calibration, they are generally perceived as costly in terms of convergence [9] and online decoding processes [40]. We assess our proposed method on two human-centric NLG tasks: emotional support conversations and moral integrity RoT generation. Experimental results demonstrate that our framework can be applied to state-of-the-art generation models for each task without performance degradation and meanwhile effectively increases global consensus of human preferences embedded in generated texts.

Our main contributions are three-fold: (i) . To the best of our knowledge, we are the first to align text generation models with human preferences while considering inherent disagreement among different individuals. (ii) . In order to model human preferences with their disagreement, we propose a Bayesian approach, Preference Modeling with Disagreement (d-PM). Additionally, we use its preference scores to calibrate NLG models via contrastive learning for generations that can be widely acceptable and less controversial. (iii) . We conduct experiments on two human-centric NLG tasks, i.e., emotional support conversations and integrity RoT generation. Experimental results demonstrate the effectiveness and versatility of our proposed method. 2

2 Framework

Aligning text generation models with human preferences requires two essential components: modeling human preferences and calibrating the text generation model.

Preference Modeling with Disagreement Human preferences can be inferred from a humanannotated dataset, denoted as D . Each instance in the dataset is represented as a triplet ( c, s, l ) . Here c is a context; s is a text; and l is a label indicating the annotators' preferences. The inherent disagreement in human preferences can be encapsulated within the label in two distinct ways. In the first approach, the label can be a soft label derived from multiple annotations, all attributed to the same sentence [13]. These annotations are sourced from multiple human annotators to preserve disagreement among individuals. The second approach is the direct collection of global consensus. In this context, the label signifies the proportion of people who find a particular sentence acceptable, an estimate provided by a single human annotator [42, 7].

Aimed at capturing human preference with disagreement within the dataset, we assume there is a distribution ρ over two classes, { acceptable, unacceptable } , comprising preferences of all humans, and therefore l is the sampling result from ρ . We employ a preference model R ( θ ) to infer ρ give c and s as inputs and the probabilistic format of l as the prior distribution. Since we focus on whether s is widely acceptable, the likelihood of the class acceptable is defined as the preference score:

Calibration for Alignment In aligning NLG models with human preferences, we calibrate the existing generation model G ( ξ 0 ) with preference scores. Here, G ( ξ 0 ) stands for a model already fine-tuned on a dataset ( X,Y ) , where X and Y represent the input set and the corresponding output set, respectively, and ξ 0 are the optimized parameters. Significantly, if the dataset for preference modeling is identical to the ( X,Y ) dataset, then ( x, y ) ∈ ( X,Y ) corresponds to ( c, s ) ∈ D . If not, these two datasets should belong to a similar domain. We initially decode K candidate sequences { ˜ y k } k = K k =1 via G (˜ y k | x ; ξ 0 ) for each x ∈ X . Then, we further train G ( ξ 0 ) , aimed at a new objective: aligning the likelihoods of candidate sequences with preference scores { S (˜ y k ,x ) } k = K k =1 .

3 Method

(b). Calibration for Alignment

Candidate Generation

(a). Preference Modeling with Disagreement

Likelihood Calibration

...

Higher likelihood

...

Preference-based Ranking

Higher

S

...

...

Higher likelihood

...

Generative

model

Text

Context

Observed

Preference

Minimize Free Energy

Universal Preference

...

This section presents our method, whose diagram is shown in Figure 2. We first use a Bayesian approach, i.e., d-PM, to model human preferences with disagreement (Section 3.1). Then we calibrate a text generation model by contrastive learning with preference scores of d-PM to align this model with human preferences (Section 3.2).

3.1 Preference Modeling with Disagreement

We establish a distribution ρ to represent the universal preference for the text s given its context c . Therefore, the observed annotations l are considered as samples from ρ , and can form a prior distribution p i ( ρ ) . Inspired by [29], we devise a Bayesian approach to approximate ρ using this prior.

Specifically, we establish a connection between ρ and l through the optimization process of a generative model. This model is designed for generating text s conditioned on c i and ρ : p ( s | c i , ρ ) . The log-likelihood of the text can be formulated as ∑ i log p ( s i | c i ) = ∑ i log ( ∑ ρ p ( s i | c i , ρ ) p i ( ρ )) , where p i ( ρ ) is the prior preference distribution. Its optimization can be achieved by introducing a variational posterior distribution q ( ρ | s i , c i ) for the i -th datapoint, and minimizing the free energy (negated evidence lower bound) formulated as:

Minimization of the free energy involves estimations of both the forward distribution of text s i : p ( s i | c i , ρ ) , and the posteriors q ( ρ | s i , c i ) , which can be computed by our preference model:

As for p ( s i | c i , ρ ) , it is defined only on the i -th datapoint and is computed by minimizing Equation (2) for fixed q ( ρ | s i , c i ) , s.t., ∑ i p ( s i | c i , ρ ) = 1 for all ρ . Thus, the optimum is achieved by:

From Equation (4), the generative model can be regarded as a matrix of variables a i,ρ describing conditional probabilities of different responses s i given different latent distributions ρ with known c i .

In a variational way, Equation (2) can be rewritten as -log p ( s i | c i )+ ∑ i KL ( q ( ρ | s i , c i ) ∥ r i ( ρ )) . Here, r i ( ρ ) ∝ p ( s i | c i , ρ ) p i ( ρ ) is the posterior model of the generative model, and it can be reformulated with reduction of p ( s i | c i , ρ ) to the matrix in Equation (4):

where α i is a scalar enabling ∑ ρ r i ( ρ ) = 1 . Accordingly, we can minimize the KL divergence between q ( ρ | s i , c i ) and r i ( ρ ) to minimize the free energy. The minimization of the free energy in Equation (2) can be derived as:

By optimizing the above objective, we can optimize the parameters of our preference model, i.e., θ .

3.2 Calibration for Alignment

It is nearly possible for G ( ξ 0 ) to generate texts with both high and low preference scores, shown in Figure 3. However, we expect to calibrate the model such that the generation probability aligns with these preference scores. Specifically, we use diverse beam search [34] to generate multiple candidates and then use our d-PM to evaluate these candidates. For the sake of more likely generating a high preference score text, we propose a model-agnostic module to leverage contrastive learning to calibrate generation likelihood aligning with d-PM. Taking inspiration from recent calibration work [32, 23, 39], we implement this module through the following three steps:

Step 1: Candidates Generation. We generate candidates from the text generator G ( ξ 0 ) , which has been fine-tuned on corresponding dataset ( X,Y ) and its parameters are ξ 0 , on its own training dataset. Given an input sequence x ∈ X , we first use G ( ξ 0 ) to generate K candidates { ˜ y 1 , ˜ y 2 , · · · , ˜ y K } using diverse beam search. As a result, these candidates will get similar possibilities yet different preference scores according to the above preliminary study.

Step 3: Likelihood Calibration. As mentioned before, we leverage contrastive learning to assign higher likelihoods to the candidates with higher preference scores. The following pairwise margin loss is used to adjust the generator G ( ξ ) .

where λ ij is the default margin λ multiplied by the difference in rank between the samples, i.e., λ ij = λ ∗ ( j -i ) . P (˜ y ′ i ; ξ ) is the length-normalized logprobability of the candidate:

0.176(25%)

0.714(75%)

0.45(MEAN)

0.175(25%)

0.755(75%)

0.467(MEAN)

0.2(25%)

0.78(75%)

0.497(MEAN)

1

1

1

Blender-Vanilla

Blender-Joint

MultiESC

0.0

0.2

0.4

0.6

0.8

1.0

MIN

0.0

0.2

0.4

0.6

0.8

1.0

MAX

where α is the length penalty hyperparameter. To avoid forgetting token-level likelihood information of the ground-truth text, we also use an additional token-level negative log-likelihood. The final calibration loss is as follows:

We minimize L c to optimize the generator's parameters ξ . This process is supervised by our d-PM model and aligns the generation model with human preference.

4 Experiments

4.1 Emotional Support Conversation

In an emotional support conversation, a supporter aims to buffer a help-seeker's emotional distress and help the help-seeker to change the difficult situation [22]. In this context, the model functions as the supporter, while the user is always the help-seeker. Due to different personal experiences, different help-seekers may respond with varied feelings and reactions to the same response, as illustrated in Figure 1. Our objective is to enhance the model's ability to generate responses that will not escalate the negative feelings of a diverse range of help-seekers.

Dataset and Base Models The benchmark ESConv [22], containing approximately 1 k conversations with 31 k utterances, develops each conversation between a help-seeker and a supporter. We include BlenderBot-Vanilla and BlenderBot-Joint proposed in conjunction with the dataset, and the SOTA model MultiESC[5]. We reproduced the base models in accordance with their respective papers and publicly available codes.

Human Preferences with Disagreement We derive human preferences from the MotivationalInterviewing-Dataset [36]. This dataset encompasses around 17 k supporter responses to help-seekers. Each response is annotated by 2 ∼ 4 experts following the MI codes[25]. The labels can be induced into two classes { acceptable, unacceptable } , and the human preferences with disagreement can be estimated by our d-PM method. By fine-tuning a BERT model on this dataset using prefix-tuning [37, 19], we obtain the d-PM. Additional details can be found in Appendix B.2.

Experimental Setup We apply our proposed method to align each base model, thus treating the well-trained base model as the generator G ( ξ 0 ) . Additionally, to validate the effectiveness of d-PM, we employ three alternative preference models within our framework for comparative analysis:

A preference model (major) trained to predict the majority voting result of annotations from different annotators, denoted as l m , and optimized by cross-entropy loss, formulated as:

where p l ( l m ) denotes the one-hot vector of l m .

Apreference model (soft) trained to approximate the direct probabilistic label of annotations, i.e., the soft label l . The model is optimized by:

L ( θ ) = ❊ ( c,s,l ) ∼ D ∥ R ( c, s ; θ ) -l ∥ 2 . (11) (3) A preference model (w/oA) that does not aggregate annotations and takes each annotation as independent. This model is optimized by cross-entropy loss, similar to Equation (10).

When training the aligned models, we aim to retain the same hyperparameters used in the training of the base models. We set the candidate number K to 10 . We train each aligned model five times with five different seeds. Subsequently, we test each of the five trained models on the test dataset and compute the average results.

Automatic Evaluation We adopt the following metrics commonly used in previous work [5, 22] for the automatic evaluation of our proposed method: BLEU [27] (B-1/2/3/4), ROUGE (R-L) [20], METEOR [2], CIDEr [33], and BOW Embedding-based matching score [21] (Extreme). Results are shown in Table 1.

Our Alignedd-PM significantly improves the performance of the base model in almost all automatic metrics, irrespective of the base model, suggesting the overall effectiveness of our proposed method. Alignedmajor and Alignedsoft are able to enhance the performance when the base model is either Blender-Vanilla or Blender-Joint, however, they do not yield an improvement when the base model is MultiESC. This limitation underscores the constraints inherent in using majority voting labels and soft labels to address disagreement of human preferences. Alignedw/oA surpasses the base model in certain metrics but falls short in others. Notably, it performs significantly lower in CIDEr, a metric evaluating the similarity between TFIDF-weighted n-grams. This shortfall suggests that Alignedw/oA is less likely to generate responses containing critical information found in the ground truth. This issue arises from the preference scores determined by w/oA being closely clustered in value, resulting in its inability to sequence the generated samples logically. These pieces of evidence indicate the potency of our proposed preference model d-PM.

HumanRatings Weask human annotators to evaluate the generations of models based on MultiESC since MultiESC can outperform Blender-Vanilla and Blender-Joint in almost all automatic evaluations. Specifically, we randomly sample 100 responses generated by different models for human ratings. We asked annotators to imagine they are help-seekers in the corresponding situation and measure each response in five aspects: (1). Identification : on a scale of 1 ∼ 5 , how much the response can explore your situation in depth and help identify the problems. (2). Comforting : on a scale of 1 ∼ 5 , how skillful the response can comfort you. (3). Suggestion : on a scale of 1 ∼ 5 , how helpful the response can solve your problems. (4). Overall : on a scale of 1 ∼ 5 , the overall quality of this response for emotional support; (5) Global Consensus : the number of people who deem the response can help them, 1 ∼ 5 represent nobody ( < 1% ), rare ( 5% ∼ 25% ), controversial ( ∼ 50% ), most ( 75% ∼ 90% ), and all ( > 99% ), respectively. Each response is annotated by three annotators, and we averaged these three annotations as the final result for each metric.

From Table 2, our method performs the best among the methods. Alignedd-PM obtained the highest score in all aspects, including the global consensus. It demonstrates that our method can generate less controversial and more helpful responses in the task of emotional support conversation.

4.2 Integrity RoT Generation

We also apply our method to the integrity 'Rule-of-Thumb' (RoT) generation task. This task is concerned with describing a chatbot's normative rules, which holds great potential for advancing research on morally-consistent conversational agents [42]. When it comes to outlining a chatbot's normative rules, people's values can vary widely. However, morally-consistent conversational agents are expected to accommodate the values of as many individuals as possible. Therefore, the generation of widely acceptable RoTs is crucial for guiding the behavior of these agents.

Dataset and Base Models The MIC dataset [42] comprises about 99 k distinct RoTs that encapsulate the moral assumptions inherent in 38 k machine-generated replies to open-ended prompts. Each prompt is associated with three different RoTs, each provided by a distinct annotator. Alongside each RoT, annotators offer a 'global consensus' value, β , which signifies the estimated proportion of the global population that would agree with the RoT. We utilize T5 (small), Flan-T5 (base) and BART (large) models as our base models, and fine-tune them on the MIC dataset. For model inference, we closely follow the processes presented in [42]. Specifically, we adopt three decoding strategies: greedy decoding, beam search ( n = 3 ), and nucleus sampling ( p = 0 . 9 ). We generate one RoT for greedy decoding; for the latter two, three hypotheses are generated and the highest-scoring one is selected.

Human Preferences with Disagreement We learn human preferences with disagreement for normative rules from the MIC dataset. The open-ended prompts are treated as context, and the ground truth RoT is considered the text to be evaluated. A probabilistic label is assigned to each RoT based on its global consensus: the probability for the class acceptable is β while that for unacceptable is (1 -β ) . We utilize this dataset to train the d-PM; details can be found in Appendix B.2.

Experimental Setup We apply our proposed framework to align each base model that has been rigorously fine-tuned on the MIC dataset. To assess the effectiveness of d-PM, we also train a preference model (soft) by minimizing the loss computed using Equation (11). The number of candidates K is set to 5 . We adopt the same hyperparameters for training each aligned model as are used for the base model. We conduct five training runs for each aligned model using five distinct seeds. Then, we evaluate each of the five trained models on the test dataset and calculate the average results.

Automatic Evaluation In accordance with previous work [42], we report standard ROUGE [20] (R-1/2/L), ScareBLEU [27], BERTScore [38], and average generation length (Avg. Len) metrics. As each prompt-reply pair in our dataset has three ground truth RoTs, we compute each metric by taking the maximum score from these three, following the method employed by [42]. The results are displayed in Table 3.

The results clearly show that Alignedd-PM generally outperforms its base model. In addition, Alignedd-PM achieves the highest score across all evaluation metrics, except the generation length, when employing a beam decoding strategy. While Alignedsoft slightly improves performance with T5 (small) as the base model, a minor decline is observed when the base model is either Flan-T5 (base) or BART (large). Interestingly, the enhancements observed in this task are not as pronounced as those witnessed in the emotional support conversation task (refer to Table 1). This may be attributed to the MIC dataset inherently accounting for disagreement, as each prompt is paired with three RoTs from different annotators. This enables base models, when fine-tuned on this dataset, to encapsulate various human preferences to a certain degree. Nonetheless, our framework has the potential to further boost model performance by providing more explicit preference information with disagreement during training.

Human Ratings We randomly select 100 replies generated by models with BART (large) as the base model for human evaluation. Adhering to previous practice, we assess generated outputs based on the following criteria [42]: (1). Well-formedness : yes or no, does the RoT explain the basics of good or bad behavior with a single judgment or action?; (2). Fluency : on a scale of 1-5, how much does the RoT align with what an English speaker might naturally say?; and (3) Relevance : on a scale of 1-5, how well does the RoT apply to the Answer for this specific Question if we assume the RoT is true? Furthermore, we request annotators to provide a Global Consensus : how many people globally will agree with this RoT, similar to the method described in Section 4.1. Three annotators evaluate each RoT, and we average these three evaluations as the final score for each metric.

Results presented in Table 4 indicate that our method produces RoTs that are more universally agreeable than those generated by the other two models. Our Alignd-PM model improves all metrics over the base model and outperforms Alignedsoft.

5 Model Analysis

0.0

4.8

5.1

5.4

B-3

0.0

3.0

3.2

3.4

B-4

K=5

0.0

21.0

21.5

R-L

0.0

8.4

8.6

8.8

9.0

9.2

METEOR

K=5

0

27

28

29

30

31

32

33

CIDEr

The effect of candidate number K during calibration. To examine the influence of varying the candidate number K on model calibration, we modify the MultiESC calibration process using different candidate numbers, namely 5 , 10 , 15 , and 20 . Specifically, this involves changing the beam widths in the diverse beam search process. Theoretically, a more significant candidate number would encompass more samples under consideration, thereby escalating the upper bound of performance. Nevertheless, as depicted in Figure 4, the model performance initially experiences an augmentation yet subsequently exhibits a reduction with the increment of the variable K . It is because an overly large candidate number could introduce redundant samples with minor differences, and the generation model might erroneously distinguish between them.

0

400

800

1200

1600

2000

2400

0.0

2.2

2.4

2.6

2.8

Loss

Step

Ours

RL

3.0

Contrastive Learning vs. Reinforcement Learning. We opt for contrastive learning to align generation models with human preferences instead of the currently prevalent reinforcement learning (RL) approach. This decision is rooted in several considerations. Firstly, RL requires expensive online decoding procedures, while our framework based on contrastive learning is a one-time offline process [40]. Secondly, RL usually yields a slow convergence speed [9]. To validate this, we align MultiESC using RL. From Figure 5, RL trains fewer samples than our framework per second. After the same steps, the loss of RL is still high, and the model performance is much worse than ours.

6 Related Work

Developing human-centric systems, which ensure that human stakeholders benefit from system outcomes [14], remains a great challenge. To build a human-centric system, it is critical to align models with human preferences [35]. There are various methods to implement the alignment. Currently, the most popular and well-known method is reinforcement learning from human feedback, thanks to the GPT series [26]. This method is also used for text summarization [31, 4], detoxification [6], and machine translation [15, 16]. The above-mentioned methods adopt one reward model, while some methods combine rewards computed by different reward models to consider fine-grained aspects of human needs [8, 11]. Moreover, some models use human feedback as the supervision signal directly to learn human preferences, such as fine-tuning pre-trained models with well-established datasets [10]. Human feedback can also be used to augment prompts for better performance [24].

7 Conclusion and Future Work

In this work, we strive to align models with human preferences to foster the development of trustworthy human-centric NLG systems. Unlike previous approaches, we take inherent disagreement into account for modeling human preferences. This idea is motivated by two compelling reasons. Firstly, it is impractical to expect consensus in human preferences due to the high degree of subjectivity involved. Secondly, harmonizing preference disagreements can inadvertently disadvantage minority groups. Accordingly, we introduce a Bayesian approach, termed Preference Modeling with Disagreement (short as d-PM ), to capture the subtleties of disagreement from limited human feedback. We subsequently utilize its preference scores to calibrate pre-existing text generation models. The efficacy of our method is substantiated through experiments in emotional support conversation and integrity Rule-of-Thumb generation.

Despite our focus on disagreement, another critical aspect remains to consider when modeling human preferences. In this work, like most previous studies, we assumed a linear relationship between the preference score and the proportion of the global population that finds the text acceptable. However, this assumption may not always hold true. For example, a sentence that 20% of people find helpful should ideally have a preference score closer to 0 instead of 0 . 2 . We believe that this issue merits further exploration in future research. Fortunately, in this study, we sidestepped this problem by using the rank of preference scores rather than the scores themselves.

References

Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning , pages 104-114. PMLR, 2020.

Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , pages 65-72, 2005.

Valerio Basile et al. It's the end of the gold standard as we know it. on the impact of pre-aggregation on the evaluation of highly subjective tasks. In CEUR WORKSHOP PROCEEDINGS , volume 2776, pages 31-40. CEUR-WS, 2020.

Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. Better rewards yield better summaries: Learning to summarise without references. In Proceedings of the 2019 Conference on Conference on Empirical Methods in Natural Language Processing (EMNLP) , Hong Kong, China, 2019.

Yi Cheng, Wenge Liu, Wenjie Li, Jiashuo Wang, Ruihui Zhao, Bang Liu, Xiaodan Liang, and Yefeng Zheng. Improving multi-turn emotional support dialogue generation with lookahead strategy planning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 3014-3026. Association for Computational Linguistics, 2022.

Farshid Faal, Ketra Schmitt, and Jia Yuan Yu. Reward modeling for mitigating toxicity in transformer-based language models. Applied Intelligence , 53(7):8421-8435, 2023.

Maxwell Forbes, Jena D Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. Social chemistry 101: Learning to reason about social and moral norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 653-670, 2020.

Amelia Glaese, Nat McAleese, Maja Tr˛ ebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375 , 2022.

Yang Guan, Jingliang Duan, Shengbo Eben Li, Jie Li, Jianyu Chen, and Bo Cheng. Mixed policy gradient. arXiv preprint arXiv:2102.11513 , 2021.

Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch Critch, Jerry Li Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. In International Conference on Learning Representations , 2021.

Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Human-centric dialog training via offline reinforcement learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 3985-4003, 2020.

Tharindu Kaluarachchi, Andrew Reis, and Suranga Nanayakkara. A review of recent deep learning approaches in human-centered machine learning. Sensors , 21(7):2514, 2021.

Manfred Klenner, Anne Göhring, Michael Amsler, Sarah Ebling, Don Tuggener, Manuela Hürlimann, and Martin Volk. Harmonization sometimes harms. 2020.

Bhushan Kotnis, Kiril Gashteovski, Julia Gastinger, Giuseppe Serra, Francesco Alesiani, Timo Sztyler, Ammar Shaker, Na Gong, Carolin Lawrence, and Zhao Xu. Human-centric research for nlp: Towards a definition and guiding questions. arXiv preprint arXiv:2207.04447 , 2022.

Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, and Stefan Riezler. Can neural machine translation be improved with user feedback? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers) , pages 92-105, 2018.

Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1777-1788, 2018.

Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. Agreeing to disagree: Annotating offensive language datasets with annotators' disagreement. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 10528-10539, 2021.

Bo Li, Peng Qi, Bo Liu, Shuai Di, Jingen Liu, Jiquan Pei, Jinfeng Yi, and Bowen Zhou. Trustworthy ai: From principles to practices. ACM Computing Surveys , 55(9):1-46, 2023.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 45824597, 2021.

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74-81, 2004.

Chia-Wei Liu, Ryan Lowe, Iulian Vlad Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 2122-2132, 2016.

Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. Towards emotional support dialog systems. In Proceedings of the 59th annual meeting of the Association for Computational Linguistics , 2021.

Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. Brio: Bringing order to abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2890-2903, 2022.

Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. Memory-assisted prompt editing to improve gpt-3 after deployment. arXiv preprint arXiv:2201.06009 , 2022.

William Miller and Stephen Rollnick. Motivational interviewing: Preparing people for change. The Journal for Healthcare Quality (JHQ) , 25(3):46, 2003.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages 311-318, 2002.

Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. Closing the ai accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 conference on fairness, accountability, and transparency , pages 33-44, 2020.

Esther Rolf, Nikolay Malkin, Alexandros Graikos, Ana Jojic, Caleb Robinson, and Nebojsa Jojic. Resolving label uncertainty with implicit posterior models. In Uncertainty in Artificial Intelligence , pages 1707-1717. PMLR, 2022.

Ben Shneiderman. Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human-Computer Interaction , 36(6):495-504, 2020.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021, 2020.

Shichao Sun and Wenjie Li. Alleviating exposure bias via contrastive learning for abstractive text summarization. arXiv preprint arXiv:2108.11846 , 2021.

Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4566-4575, 2015.

Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural sequence models. AAAI , 2018.

Zijie J Wang, Dongjin Choi, Shenyu Xu, and Diyi Yang. Putting humans in the natural language processing loop: A survey. In Proceedings of the First Workshop on Bridging Human-Computer Interaction and Natural Language Processing , pages 47-52, 2021.

Anuradha Welivita and Pearl Pu. Curating a large-scale motivational interviewing dataset using peer support forums. In Proceedings of the 29th International Conference on Computational Linguistics , pages 3315-3330, 2022.

Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, et al. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. EMNLP , 2022.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations , 2019.

Xingxing Zhang, Yiran Liu, Xun Wang, Pengcheng He, Yang Yu, Si-Qing Chen, Wayne Xiong, and Furu Wei. Momentum calibration for text generation. arXiv preprint arXiv:2212.04257 , 2022.

Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. Calibrating sequence likelihood improves conditional language generation. arXiv preprint arXiv:2210.00045 , 2022.

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019.

Caleb Ziems, Jane Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. The moral integrity corpus: A benchmark for ethical dialogue systems. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3755-3773, 2022.

A Ethics and Societal Impact

Ethics In our experiments, we utilized three datasets: MI-dataset, ESConv, and MIC. Each of these datasets is designed to protect user privacy, and none contain any personal information.

Societal Impact Our work is driven by the goal to enhance the positive impact of Natural Language Generation (NLG) models by better aligning them with human preferences, particularly those relating to personal standards and human values such as emotions, beliefs, and ethical norms. We acknowledge the diversity of individual preferences, which can often lead to disagreements. This disagreement is taken into account in our model of human preferences, which we use to calibrate our text generation model. This approach aims to produce text that is widely acceptable and less controversial.

Our findings suggest that our approach can make generated texts less contentious and more useful. Looking towards the future, neglecting to account for disagreements in human preferences could result in serious implications, especially if these models are used in safety-critical situations. We anticipate that our approach could be beneficially applied to a wider range of tasks and models in future research and product development.

B Supplement Experiment Details

The code for our experiments is provided in the supplements and will be made public upon acceptance of this paper. Additional details that are not included in the submitted code are explained in the following subsections.

B.1 Resources Used

We trained models based on MultiESC using two Nvidia RTX 3092 GPUs, while all other models, including d-PM models, were trained using a single NVIDIA RTX 3092 GPU. Our models were implemented in Python using PyTorch 3 and the transformers (4.16.2) library 4 .

B.2 d-PM Training Details

Model Architecture The d-PM models are fine-tuned versions of the BERT (base, uncased) model using prefix-tuning. The BERT model has a layer number of n BERT layer , and a hidden size of d BERT . The model uses a sequence of prefix indices, represented as L idx , and the length of the prefix is represented by | L idx | . The model input is a concatenation of the prefix, the context, and the text, denoted as z = [ L idx ; c ; s ] , where z t represents the input at time step t . The activations of the model are denoted as h = [ h prefix , h c , h s ] , where h t represents the concatenation of all activation layers at time step t . Here, h c and h s are computed using BERT parameters, while h prefix are free parameters. A trainable matrix A θ 1 is then initialized, which is parameterized as θ 1 of dimension | L idx | × d dim , where d dim = n BERT layer × d BERT . Thus, the activation h t at time step t is defined as:

We utilized the last hidden states (the activations from the last layer) of the text s to calculate the probabilities for the classes acceptable and unacceptable . This computation is facilitated by a multilayer perceptron (MLP) layer, which is parameterized as θ 2 .

The parameters of the d-PM, i.e., θ = ( θ 1 , θ 2 ) , are optimized using the loss function, which is formulated in Equation (6). Furthermore, the preference models that we utilized in Alignedmajor and Alignedsoft share the same architecture. However, they are differentiated by the distinct loss functions used for their optimization, as denoted in Equation (10) and Equation (11), respectively.

Experimental Setup The d-PM models were implemented based on the code 5 published for the UnifiedSKG framework [37]. During the training of the d-PM models, BERT parameters were frozen,

and the prefix was optimized. The prefix length was set to 10 . A batch size of 160 and a learning rate of 5 × 10 -4 were used.

Dataset For the training dataset, the MI-Dataset was used to train d-PM model which can output human preferences for the emotional support responses. This dataset contains 15 fine-grained classes of responses. The annotators categorized each response into one of the fine-grained classes. These responses fall into three coarse-grained classes according to the MI codes [25]: (1) MI Adherent , which involves supporting the help-seekers with empathetic and compassionate statements, enabling them to feel heard, respected, and understood; (2) Relational , which focuses on establishing a solid relationship between the help-seeker and the supporter, leading to more positive responses from the help-seekers; and (3) MI Non-Adherent , which includes statements such as arguing, confronting, or offering unsolicited advice, and may create resistance and hinder problem-solving for the helpseekers. When training the d-PM, these classes were converted into two classes: acceptable and unacceptable , as shown in Table 5 The dataset was randomly split into a 9 : 1 ratio for the training and validation set. There are 34 . 93% instances where annotators have different annotations.

Table 6: One example from the MIC dataset. Blue text : prompt-reply (QA) pair. Orange text : global consensus scale value.

Question : Am I a bad BF, weird, or just going insane? Answer : I don't think you're a bad bf or weird or going insane. I think you just need to talk to him about it. RoT : It's important to communicate honestly with your significant other. Global Consensus : 4

The MIC dataset was used for training the d-PM used in the integrity Rule of Thumb (RoT) generation task. Each RoT in this dataset is annotated with a global consensus provided in a 5-Likert scale format, where 1 ∼ 5 represent nobody ( < 1% ), rare ( 5% ∼ 25% ), controversial ( ∼ 50% ), most

( 75% ∼ 90% ), and all ( > 99% ), respectively. An example is shown in Table 6. For the purpose of training, the ordinal scales were converted into continuous variables that represent the percentage consensus. This was done by randomly selecting a value between the upper and lower bound corresponding to each scale value to serve as the final global consensus ( β ) for each RoT. Following this, the probability for the class acceptable was set as β , while that for unacceptable was set as (1 -β ) . The split of the MIC dataset was maintained when training the model and when calibrating the generation model to avoid data leakage.

B.3 Emotional Support Conversation

Experimental Setup We implemented the base models following the methods detailed in [22, 5] and their corresponding published codes 6 . Each base model was run once using the seed provided in its published codes.

In order to fairly compare models, we aimed to keep the hyperparameters consistent with those of the corresponding base models when implementing aligned models, except that we increased the learning rate to enhance the training efficiency of aligned models. Specifically, we set the learning rate to 1 × 10 -3 for the Blender-Vanilla and Blender-Joint base models, and 3 × 10 -5 for the other models. Additionally, due to GPU memory constraints, we reduced the batch size from 32 to 12 when training the aligned MultiESC. We ran each aligned model five times with different seeds: 0 , 1 , 13 , 42 , and 1024 , and reported the average results.

B.4 Integrity RoT Generation

Experimental Setup Our implementation of the base models was based on the work presented in the paper [42] and its published codes 7 . Each base model was run once using the seed provided in its published codes.

For a fair comparison, we endeavored to keep the hyperparameters consistent with those of the corresponding base models when implementing the aligned models. However, we increased the learning rate to 1 × 10 -4 during the training of aligned models. Each aligned model was run five times with different seeds: 0 , 1 , 13 , 42 , and 1024 , and we reported the average results.

B.5 Alignment through Reinforcement Learning

Implementation Details The reinforcement learning (RL) framework was implemented with reference to the Hugging Face TRL library 8 . We aimed to ensure consistency with the hyperparameters used in the aligned model, which was implemented using the contrastive learning framework. Additionally, we adhered to the same training steps, i.e., terminating the training process after 2400 steps. We executed the RL framework with five seeds: 0 , 1 , 13 , 42 , and 1024 . The best result obtained is reported in the paper.

Full Results Table 7 and Table 8 present results of aligning models with RL framework. Generally, alignment with RL falls short in performance when compared to the contrastive learning (CL) framework, i.e., our proposed model. In the task of emotional support conversation, as depicted in Table 7, aligning with RL proves to be incompatible with ours and can occasionally result in a degradation in the base model's performance. As for the task of RoT generation, as shown in Table 8, the best performance when aligning each base model is nearly always achieved by our model. Alignment with RL notably increases evaluation metric values of T5 (small) and Flan-T5 (base) when employing a greedy or p = 0 . 9 decoding strategy. However, a common issue among models aligned with RL is their tendency to generate phrases like '(something) is good/wrong' and 'It is good/wrong to (something)'. Although the key information (something) may be incorrect, such a pattern can exhibit significant overlap in wording with the ground truth, thereby yielding higher values in terms of some evaluation metrics. The potential reason for the suboptimal performance of RL could be the small size of ESConv and MIC. Although RL has proven potent in certain scenarios, as illustrated by

recent research [41, 31, 16], a commonality among these successful applications is their training on expansive datasets (280K to 6M training instances). In contrast, the sizes of ESConv and MIC are significantly more modest. This hypothesis also aligns with findings from Agarwal et al. [1], which suggest that smaller datasets can potentially undermine RL's performance.

C Case Study

We present case studies for the task of emotional support conversation and integrity RoT generation in Table 9 and Table 10, respectively.

The Help-seeker's Emotion : Anxiety The Help-seeker's Problem : Applying for work, I have been retired for 2 years and would like to return to the workforce.

Problem Type : Procrastination

Seeker : I have been retired for over 2 years and would like to return to the workforce, part-time.

Supporter : So you are thinking of going back to work?

Seeker : Yes, it would help us greatly.

Supporter : You have the right idea already.

Seeker : There are plenty of jobs and not sure where to start.

: Sadness

The Help-seeker's Emotion

The Help-seeker's Problem : I have a friend who knew me since I was little, she is closer to my mother's age so she is kinda like my aunt growing up. She always treat me like I was little. But I'm a grown woman now and I want our relationship to go far and grow more.

Problem Type : Problems with friends

Seeker : I have a friend who knew me since I was little. She is a little older than me. She still treats me like I'm still little. I tried to explain to her that I'm a grown woman and I grew out of a lot of things when I was a kid.

Supporter : Is she listening to you? I know how it feels to be patronized by people who are older.

Seeker : She still didn't get it. I want our relationship to move further but with her like this. I feel sad. Is there anything I can do?

Supporter : Well, if she is not listening to you when you try to talk to her you have to ask yourself whether the relationship is really worth it?

Seeker : That's true . I didn't think about it. I know she is a good person. So I would give her a chance and we can talk about it before.

**********

Case One

**********

: Is it weird that I don't want a family or a career?

Question

Answer : No, it's not weird at all. It's perfectly normal to not want children or a family.

Well-formedness

Relevance

Global Consensus

Well-formedness

Relevance

Global Consensus

Well-formedness

Relevance

Global Consensus

Base

Alignedd-PM

It is okay to not want children or a family.

It is good to have a family.

It is okay not to want a family or a career.

**********

Case Two

**********

Question : Who was the best teacher you ever had? What qualities made them great teachers?

Answer : I don't know. I guess he had a lot of patience and he made me feel safe.

**********

Case Three

**********

Question : HELP, a hotel sink fell into the cupboard. Am I responsible or liable? Should I try to fix it or report it to the desk? HELP

Answer : I would call the front desk and let them know. They should be able to help you.

D Human Rating Details

Annotation Details We engaged six annotators from diverse educational, gender, geographic, and occupational backgrounds to conduct human ratings for each task. Each sentence to be rated was assigned to three annotators, and the final result was calculated as the average of their scores. Given

1,1,1

4,3,5

3,3,4

0,0,1

3,2,5

1,2,3

1,1,1

4,4,5

5,4,4

that 3 ∼ 4 sentences were generated by different models for the same input, we presented these sentences and their corresponding context or prompt-reply pair on a single page. Annotators were asked to score the generated sentences relative to each other. To avoid potential bias, the order of sentences was randomized to prevent annotators from identifying the generation model based on sentence order.

Annotation Instructions Annotators were initially briefed on the tasks to ensure they understood the objectives of the emotional support responses or RoTs. Except for the global consensus, they were asked to rate the generated responses according to their personal preferences. For the emotional support conversation task, we asked annotators to imagine themselves as help-seekers in the conversations. After providing the situational and emotional context, annotators then rated the generated responses.

Annotation Cost We remunerated annotators at a rate of 2 ∼ 3 times their local hourly minimum wage. According to the annotators' feedback, they spent approximately 40 seconds evaluating each generated response or RoT.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for modeling human preference with a Bayesian approach that considers the disagreement among human annotators. The authors also propose a contrastive learning method to replace RL in the alignment process. The authors conduct experiments on two text generation tasks and show some improvements upon baselines.\n\nSTRENGTHS:\n1. The paper is well-motivated and making use of disagreements among human preferences is an interesting problem.\n2. The proposed method is technically sound. Using Bayesian and variational inference for modeling uncertainty is a good idea and makes sense to me.\n3. The authors conduct experiments on two datasets and some improvements are observed compared to the baselines.\n4. The authors conduct human evaluation on the evaluation datasets and also some ablation studies / analysis, which are helpful to understand the method better.\n\nWEAKNESSES:\n1. The experiments are not solid enough. First, the backbone models used in both the datasets are not strong enough. And it is hard to use human feedback when the model capacity is very limited. The performance improvements are also very marginal, which may come from some regularization effects in the contrastive learning objective. The poor performance from RL baseline also validate this hypothesis. It is also very strange why some baselines with major/soft voting lead to worse performance compared to the baseline. Maybe due to some bad hyperparameters? How are the hyperparameters tuned?\nAlso, the second dataset's human preference data is not actual collected from human annotation, which does not makes sense. It is unclear why the authors do not use some recent RLHF datasets.\n\n2. The claim about RL's disadvantages is not necessarily true. Using policy gradient or PPO is computationally more efficient because it does not require decoding multiple outputs for contrastive learning, and both the proposed method and RL-based method require using the reward function to score the output. And the success of RLHF algorithms clearly shows the usefulness of RL-based method. It would be helpful to combine RL + the reward function trained with the Bayesian method.\n\nQUESTIONS:\nIt would be interesting to see the actual difference of the preference model trained by conventional approaches and the proposed Bayesian approach. Can it be validated on some held out data that have disagreement annotations?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel approach for training preference models (PMs) for aligning language models (LMs) that accounts for human preference diversity. Their algorithm, d-PM, trains a PM to match a distribution over human preference in human feedback data. The authors also propose a contrastive learning algorithm for aligning an LM with preferences represented by the d-PM. They validate both by conducting experiments on datasets for emotional support conversation and compliance with moral norms.\n\nSTRENGTHS:\n1. The paper is written clearly and was easy to follow.\n2. The authors address an important, largely underexplored problem of modelling preference diversity in a principled, Bayesian fashion. The formal framework they propose is sound and elegant and I think is an interesting and original contribution to the emerging field of preference modelling for LM alignment.\n\nWEAKNESSES:\n1. The assumption that human preferences over sentences boil down to discriminating between two classes, {acceptable, unacceptable}, feels limiting. It's significantly more restrictive than the usual pairwise comparison approach in RLHF. It also makes the later claim that a distribution $l’$ over {acceptable, unacceptable} “encompass[es] all human perspectives” (line 101) implausible (e.g., it lumps together numerous ways and degrees of being unacceptable). If it’s just part of the experimental setup and not a limitation of the framework, this should me made clear, the assumption should be moved to the Experiments section and the Framework section should be more general. However, then I'm not sure the experiments will be able to provide evidence for the soundness of that more general framework. \n2. Unless I’m mistaken, the method still uses point estimates of human preferences (i.e. probability of \"acceptable\") during the calibration phase (Step 2: Preference-based Ranking). In this sense, the framework is not fully Bayesian; to my mind, the Bayesian thing to do would to be to somehow marginalise over human preferences for each example during finetuning. I’m happy to be corrected here if I’m misunderstanding something.\n3. Relatedly, I’m not sure the experimental setup explicitly tests for how effective d-PM is at making LMs find a shared consensus among disagreement. In the RoT setup, it seems that the hard work has already been done (by the annotators setting the beta values) and the LM is just trained to imitate that already-found consensus. Why couldn’t we just do that by supervised finetuning on demonstrations with high beta? (Why isn’t that a baseline?) \n4. In the RoT experiment (sec 4.2.), I understand that the authors train a d-PM on the MIC dataset and then evaluate an LM aligned with d-PM against reference responses from MIC. Are these responses held-out (i.e. not used in training of d-PM)? Even if they are held-out, this setup still feels subtly wrong to me: you’re optimising the LM to imitate good MIC responses (indirectly, via d-PM scores) so it’s not surprising that calibrated models outperform base models at being similar to good MIC responses.\n5. While I agree modelling preference diversity in aligning LMs is underexplored, there is some work on this and claim (i) in the Introduction (lines 62-64) feels exaggerated. See Bakker et al., 2022. Some other work that could be relevant is DPO (Rafailov et al., 2023) for calibration and Korbak et al. (2023) for a different way of framing LM alignment as Bayesian inference.\n6. The authors conduct experiments with largely obsolete LMs such as BERT (2018) or T5 (2019). They could easily be replaced with, e.g., RoBERTa and FLAN-T5, respectively, which are unequivocally better in every respect and come with the same model sizes.\n7. It feels a bit suspicious to evaluate d-PM’s impact on downstream LMs using a relatively obscure calibration method as opposed to more standard approaches like best-of-n sampling (Nakano et al., 2021) or finetuning on filtered data. The authors could also consider a more intrinsic evaluation, treating d-PM as a binary classifier and measuring its accuracy, F2 score or validation loss on a held-out subset of the data. It would be interesting to compare d-PM with baselines without the confounding effect of the calibration training.\n8. I feel the section comparing calibration and RL finetuning is sloppy. First, it lacks essential details (algorithm being used, the exact objective, the stopping criterion); these are too important to be left for the appendix. Second, there is no reason to think that the RL loss should be comparable to a supervised learning loss. This is true in general, but especially for online RL where the objective involves chasing a moving target (on policy distribution) and is further confounded by advantage estimation and KL penalties. See [1] for an approachable explanation in the context of policy gradients [1]. To me, this is a serious red flag showing a fundamental misunderstanding on the authors’ part. If d-PM does not work well with RL, I’m quite suspicious about either authors' RL setup or d-PM itself.\n10. I'd be curious to see standard errors for human evaluation scores, especially given small sample size (100).\n\nMinor points:\n\n10. Possible typo on line 83, “give”\n\n11. The notation $l’ \\sim l$ was somewhat unintuitive for me and $l’$ is hard to visually tell apart from $l$, at least with the current font.\n\n[1] https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#implementing-the-simplest-policy-gradient\n\nBakker et al., 2022. Fine-tuning language models to find agreement among humans with diverse preferences\n\nKorbak et al., 2022. RL with KL penalties is better viewed as Bayesian inference\n\nNakano et al., 2021. WebGPT: Browser-assisted question-answering with human feedback\n\nRafailov et al., 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model\n\nQUESTIONS:\n1. How is d-PM different from training a binary classifier with soft labels {acceptable, unacceptable}, i.e. minimising the cross-entropy between model-predicted distribution over labels and empirical distribution for each example? Note that this is slightly different from the “soft” baseline in section 4.\n2. In the actual implementation, is the summing over $j$ in the denominator of (4) implemented by exact enumeration of the whole dataset (for each gradient step?) or is it approximated, e.g., by sampling?\n3. It was unclear to me where does the reference responses for reference-based metrics (e.g. BLEU, ROUGE) come from? Are they from the ESConv and MIC benchmarks? How these datasets split into train and test?\n\nLIMITATIONS:\nThe authors discuss certain limitations in the final section. However, I think the certain limitations and implementation details are missing (see weaknesses 1 and 8). Moreover, experiments involve relatively small-sized models and I think the authors should be more transparent about that, e.g., report parameter counts.\n\nI think authors' declaration `Error Bars: Yes` is misleading. There's no error bars. While statistical significance is reported for some results, not for those where I'd expect the highest variance (i.e. human evaluation with small sample size). \n\nFinally, emotional support conversation is a sensitive domain and I'd expect the authors to highlight it a bit more and be clear that \"effectiveness of the proposed method\" (line 69) does not entail deployment-readiness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel alignment method, which takes a Bayesian approach, termed Preference Modeling with Dis-\n295 agreement (short as d-PM); the method is designed to capture disagreement in human labels, rather than assuming a gold human label. The Bayesian approach is basically a new way to train reward model, which will be later used to rank model generated outputs, and then apply contrastive learning loss to optimize for generating higher ranked outputs. I think the contribution mainly lies in the first part, which is to propose the new mechanism for training reward model, which are crucial for success of either RLHF or contrastive learning in this paper to align model with human preferences. \n\nAn additional contribution is the usage of contrastive learning to align Large Language Model.\n\nSTRENGTHS:\nThe problem it aims to tackle is a very important one. Too often, AI practitioners make the possibly false assumption that human annotated labels are good, and take the naive approach of majority voting to get final training data. This approach may work when the reason of disagreement lies in annotation quality. However, it neglects the situation, which paper points out, that the disagreement distribution as a natural reflection of the diversity of opinions among populations, and should not be ignored. \n\nThe reward model training schema seems novel. I am not an expert on optimization or energy-based models; if proof in the paper does stand(which seems fine for me, but not 100% certain), it sheds light on a better method to handle annotation differences. The experiment using contrastive learning also does show some improvements, over the default approach, which is majority voting. \n\nThe experiment set-up and introduction of dataset are clear, and Table-1's result is well-organized and has most important metrics included. Human evaluation is also conducted, which further verifies the automatic evaluation. Significance tests are conducted to further verify results.\n\nThe contrastive learning setup is a combination of self-learning by generating candidates and learning reward model feedbacks. It provides another data point that contrastive learning could work, and RLHF-ppo isn't the best/only way to learn rewards. \n\nThe paper is in general easy to follow. But more backgrounds and explanation of notations will be better for Section 3.1 for non-experts to understand.\n\nWEAKNESSES:\n* For the reward model part, it's not clear to me why it's called a Bayesian approach after reading Derivation Process in 3.1; I understand that Beyesian theorem is applied to derive the final optimization loss, but isn't Beyesian theorem/ELBO are common techniques  and are also used to derive other optimization losses. I would appreciate more explanation why Beyesian is an important characteristic/contribution for this method. \n\n* For the experiment result, looking at the Rouge-L score, the differences between majority voting and d-PM isn't huge for both Blender and MIC datasets. In some cases, like Table 4, Row-BART-Greedy, Rouge-L and other metrics are giving conflicting indications of model performance. This raises question whether the relatively modest improvements from d-PM are meaningful, when the automatic metrics are not very robust. The human evaluation does alleviate this concern, but I think it should still be noted. \n\n* Contrastive vs RL; I think it is fine to use a contrastive learning method, rather than RL to align model; but only pointing out that \"RL requires expensive online decoding procedures\" does not rule out the possibility that RL could possibly give better results. Also, this paper isn't the first to use contrastive learning to align models, the results will be more convincing if an RL-baseline, and another contrastive learning baseline(maybe stable alignment  or other well-known methods) is provided. \n\n* The evaluation scale is also limited. The experiment are only conducted on two datasets, and even on such selected domain, the improvements are modest. It is arguable how effective this approach can scale to other datasets. A wider scale evaluation on more domains will be needed.\n\nQUESTIONS:\n* The authors named d-PM method a Bayesian approach, but did not explicitly explain why it is the most important characteristic. If would be great if authors make that explicit in paper.\n\n* Why only evaluate on Blender and MIC? Is there a selection criterion or hypothesis on what kinds of datasets that d-PM will work better than say majority voting?\n\n* The paper claimed that it has expensive overheads, but I do wonder if your contrastive learning approach can have a similar effective to RL, and other robust contrastive learning methods.\n\nLIMITATIONS:\nYes, addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed to train a reward model to capture the disagreement among human raters and use it to calibrate NLG model's generation.  The contributions includes: align text generation modes with human preference considering disagreement, a d-PM (preference modeling with disagreement) approach to train reward model, experiments on two NLG tasks of emotional support conversations and integrity ROT generation.\n\nSTRENGTHS:\n- exploring and emphasizing the problem of human disagreement in preference modeling\n- proposed a bayesian approach to better modeling the human disagreement\n- experiments on two NLG tasks with human evaluation that shows the gain of the proposed method\n\nWEAKNESSES:\n- a simple baseline for reward model should be included: taking each human label as independent (without aggregation), with a cross entropy loss.    \n\n- auto eval metrics for the 2 tasks are all reference based metrics, these metrics doesn't fully taken account into how well the model respects human preferences. Instead a reward model based auto eval metric should be used instead. Reference can also be evaluated using these reward model based metrics which also tells you how good references are. However the human evals capture that, but there lacks data on statistical significancy.\n\n- one main claim of the paper is \"such methods suffer from an inability to capture the nuanced degrees of disaggregation among humans and may only represent a specialized subset of individuals, thereby lacking the ability to quantitatively disclose the universality of human preferences.\" There is no direct experimental results to measure or support this. I would suggest to weaken this argument if no direct evidences can be provided.\n\nQUESTIONS:\n- it is a bit difficult to understand Figure2.a , more training details of the model (e.g. an algorithm description) could be added in the main text to help reader understand. Is my understanding correct that the prior is the \"universal preference\" estimated by the R(si, ci, \\theta) and new reward is the observed preference multiplies by the prior?\n\n- \"harmonizing preference disagreements can inadvertently disadvantage minority groups\" is there any reference to this? why is the bayesian approach with a \"universal preference\" different? Do you have any experimental results to support it?\n\n- (minor suggestion) in many cases \"preference\" data refers to model generating two sequences are collecting data directly on which one is better. in this paper, it is a binary classification on each individual sequence of good/bad. \"preference modeling\" is a little bit misleading here. might be good to use a different word or clarify this clearly.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}