SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We also apply our method to the integrity "Rule-of-Thumb" (RoT) generation task. This task is concerned with describing a chatbot's normative rules, which holds great potential for advancing research on morally-consistent conversational agents [42]. When it comes to outlining a chatbot's normative rules, people's values can vary widely. However, morally-consistent conversational agents are expected to accommodate the values of as many individuals as possible. Therefore, the generation of widely acceptable RoTs is crucial for guiding the behavior of these agents.
Dataset and Base Models The MIC dataset [42] comprises about 99k distinct RoTs that encapsulate the moral assumptions inherent in 38k machine-generated replies to open-ended prompts. Each prompt is associated with three different RoTs, each provided by a distinct annotator. Alongside each RoT, annotators offer a "global consensus" value,  $\beta$ , which signifies the estimated proportion of the global population that would agree with the RoT. We utilize T5 (small), Flan-T5 (base) and BART (large) models as our base models, and fine-tune them on the MIC dataset. For model inference, we closely follow the processes presented in [42]. Specifically, we adopt three decoding strategies: greedy decoding, beam search  $(n = 3)$ , and nucleus sampling  $(p = 0.9)$ . We generate one RoT for greedy decoding; for the latter two, three hypotheses are generated and the highest-scoring one is selected.
Human Preferences with Disagreement We learn human preferences with disagreement for normative rules from the MIC dataset. The open-ended prompts are treated as context, and the ground truth RoT is considered the text to be evaluated. A probabilistic label is assigned to each RoT based on its global consensus: the probability for the class acceptable is  $\beta$  while that for unacceptable is  $(1 - \beta)$ . We utilize this dataset to train the d-PM; details can be found in Appendix B.2.
Experimental Setup We apply our proposed framework to align each base model that has been rigorously fine-tuned on the MIC dataset. To assess the effectiveness of d-PM, we also train a preference model (soft) by minimizing the loss computed using Equation (11). The number of candidates  $K$  is set to 5. We adopt the same hyperparameters for training each aligned model as are used for the base model. We conduct five training runs for each aligned model using five distinct seeds. Then, we evaluate each of the five trained models on the test dataset and calculate the average results.
Table 3: Automatic evaluation results on MIC. † represents significantly better than the corresponding base model with  $p < 0.01$ .
Model	R-1	R-2	R-L	BertScore	ScareBLEU	Avg.Len
T5(Small)	Beam	Base	53.44	32.97	52.17	93.44	29.05	8.94
Alignedsoft	52.14	31.48	50.79	93.34	27.05	8.85
Alignedd-PM	53.45	32.82	52.09	93.49†	28.51	8.99
Greedy	Base	37.04	16.30	35.27	90.94	14.27	10.47
Alignedsoft	37.77†	16.82†	35.97†	91.21†	14.68†	9.83
Alignedd-PM	38.15†	17.22†	36.40†	91.29†	15.15†	9.74
p=0.9	Base	40.22	19.23	38.56	91.59	16.71	9.85
Alignedsoft	40.90†	19.70†	39.24†	91.79†	16.98†	9.47
Alignedd-PM	41.41†	20.22†	39.75†	91.90†	17.75†	9.38
Flan-T5(Base)	Beam	Base	55.07	34.96	53.74	93.77	30.68	9.00
Alignedsoft	54.82	34.65	53.49	93.75	30.34	9.00
Alignedd-PM	55.18†	35.07†	53.86†	93.79†	30.83†	9.01
Greedy	Base	37.94	17.23	36.13	91.39	15.36	9.78
Alignedsoft	37.84	17.03	36.00	91.38	15.12	9.75
Alignedd-PM	38.34†	17.52	36.53†	91.44†	15.49	9.77
p=0.9	Base	41.41	20.41	39.70	92.02	18.02	9.30
Alignedsoft	41.44	20.33	39.71	92.02	17.91	9.29
Alignedd-PM	41.78†	20.69†	40.09†	92.07†	18.26†	9.32
BART(Large)	Beam	Base	54.81	35.07	53.35	93.85	30.80	9.44
Alignedsoft	54.82	34.85	53.36	93.82	30.35	9.36
Alignedd-PM	55.05†	35.18	53.62	93.86†	30.85	9.40
Greedy	Base	54.77	34.85	53.30	93.84	30.51	9.54
Alignedsoft	54.54	34.53	53.10	93.80	30.01	9.47
Alignedd-PM	54.81	34.86	53.39	93.83	30.51	9.48
p=0.9	Base	54.77	34.96	53.32	93.84	30.62	9.56
Alignedsoft	54.65	34.68	53.23	93.82	30.16	9.45
Alignedd-PM	54.86	35.01	53.45	93.85†	30.63	9.48
Table 4: Human evaluation results on MIC. The base model is BART-large (beam).
Model	Well-formedness	Fluency	Relevance	Global Consensus
Base	0.528	2.547	2.037	2.428
Alignedsoft	0.550	2.602	2.028	2.502
Alignedd-PM	0.568	2.602	2.103	2.555
Automatic Evaluation In accordance with previous work [42], we report standard ROUGE [20] (R-1/2/L), ScareBLEU [27], BERTScore [38], and average generation length (Avg. Len) metrics. As each prompt-reply pair in our dataset has three ground truth RoTs, we compute each metric by taking the maximum score from these three, following the method employed by [42]. The results are displayed in Table 3.
The results clearly show that Alignedd-PM generally outperforms its base model. In addition, Alignedd-PM achieves the highest score across all evaluation metrics, except the generation length, when employing a beam decoding strategy. While Alignedsoft slightly improves performance with T5 (small) as the base model, a minor decline is observed when the base model is either Flan-T5 (base) or BART (large). Interestingly, the enhancements observed in this task are not as pronounced as those witnessed in the emotional support conversation task (refer to Table 1). This may be attributed to the MIC dataset inherently accounting for disagreement, as each prompt is paired with three RoTs from different annotators. This enables base models, when fine-tuned on this dataset, to encapsulate various human preferences to a certain degree. Nonetheless, our framework has the potential to further boost model performance by providing more explicit preference information with disagreement during training.
Human Ratings We randomly select 100 replies generated by models with BART (large) as the base model for human evaluation. Adhering to previous practice, we assess generated outputs based on the following criteria [42]: (1). Well-formedness: yes or no, does the RoT explain the basics of good or bad behavior with a single judgment or action?; (2). Fluency: on a scale of 1-5, how much does the RoT align with what an English speaker might naturally say?; and (3) Relevance: on a scale of 1-5, how well does the RoT apply to the Answer for this specific Question if we assume the RoT is true? Furthermore, we request annotators to provide a Global Consensus: how many people globally will agree with this RoT, similar to the method described in Section 4.1. Three annotators evaluate each RoT, and we average these three evaluations as the final score for each metric.
Results presented in Table 4 indicate that our method produces RoTs that are more universally agreeable than those generated by the other two models. Our Alignd-PM model improves all metrics over the base model and outperforms Alignedsoft.
5 Model Analysis
Figure 4: Model performances with different candidate numbers  $K$  when calibrating MultiESC with preference scores of d-PM.
The effect of candidate number  $K$  during calibration. To examine the influence of varying the candidate number  $K$  on model calibration, we modify the MultiESC calibration process using different candidate numbers, namely 5, 10, 15, and 20. Specifically, this involves changing the beam widths in the diverse beam search process. Theoretically, a more significant candidate number would encompass more samples under consideration, thereby escalating the upper bound of performance. Nevertheless, as depicted in Figure 4, the model performance initially experiences an augmentation yet subsequently exhibits a reduction with the increment of the variable  $K$ . It is because an overly large candidate number could introduce redundant samples with minor differences, and the generation model might erroneously distinguish between them.
Metric	MultiESC	RL	Ours
B-1	20.36	11.75	21.59
B-2	8.80	4.81	9.56
B-3	4.92	2.78	5.33
B-4	3.14	1.81	3.36
R-L	21.00	19.57	21.50
METEOR	8.58	6.25	9.03
CIDEr	30.69	26.02	32.65
Extreme	52.74	51.21	53.15
(#(Samples)/s	-	2.65	4.36
Figure 5: Comparison between alignment with RL (RL) and our model (Ours). Left: Automatic evaluation results (#(Samples)/s indicates the number of trained samples per second). Right: Training loss according to training steps.
Contrastive Learning vs. Reinforcement Learning. We opt for contrastive learning to align generation models with human preferences instead of the currently prevalent reinforcement learning (RL) approach. This decision is rooted in several considerations. Firstly, RL requires expensive online decoding procedures, while our framework based on contrastive learning is a one-time offline process [40]. Secondly, RL usually yields a slow convergence speed [9]. To validate this, we align MultiESC using RL. From Figure 5, RL trains fewer samples than our framework per second. After the same steps, the loss of RL is still high, and the model performance is much worse than ours.
Developing human-centric systems, which ensure that human stakeholders benefit from system outcomes [14], remains a great challenge. To build a human-centric system, it is critical to align models with human preferences [35]. There are various methods to implement the alignment. Currently, the most popular and well-known method is reinforcement learning from human feedback, thanks to the GPT series [26]. This method is also used for text summarization [31, 4], detoxification [6], and machine translation [15, 16]. The above-mentioned methods adopt one reward model, while some methods combine rewards computed by different reward models to consider fine-grained aspects of human needs [8, 11]. Moreover, some models use human feedback as the supervision signal directly to learn human preferences, such as fine-tuning pre-trained models with well-established datasets [10]. Human feedback can also be used to augment prompts for better performance [24].
In this work, we strive to align models with human preferences to foster the development of trustworthy human-centric NLG systems. Unlike previous approaches, we take inherent disagreement into account for modeling human preferences. This idea is motivated by two compelling reasons. Firstly, it is impractical to expect consensus in human preferences due to the high degree of subjectivity involved. Secondly, harmonizing preference disagreements can inadvertently disadvantage minority groups. Accordingly, we introduce a Bayesian approach, termed Preference Modeling with Disagreement (short as d-PM), to capture the subtleties of disagreement from limited human feedback. We subsequently utilize its preference scores to calibrate pre-existing text generation models. The efficacy of our method is substantiated through experiments in emotional support conversation and integrity Rule-of-Thumb generation.
Despite our focus on disagreement, another critical aspect remains to consider when modeling human preferences. In this work, like most previous studies, we assumed a linear relationship between the preference score and the proportion of the global population that finds the text acceptable. However, this assumption may not always hold true. For example, a sentence that  $20\%$  of people find helpful should ideally have a preference score closer to 0 instead of 0.2. We believe that this issue merits further exploration in future research. Fortunately, in this study, we sidestepped this problem by using the rank of preference scores rather than the scores themselves.
8 Acknowledgements
This work is supported by Research Grants Council of Hong Kong (PolyU/5204018, PolyU/15207920, PolyU/15213323) and National Natural Science Foundation of China (62302184, 62076212).
[36] Anuradha Welivita and Pearl Pu. Curating a large-scale motivational interviewing dataset using peer support forums. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3315-3330, 2022.
[37] Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, et al. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. EMNLP, 2022.
[38] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2019.
[39] Xingxing Zhang, Yiran Liu, Xun Wang, Pengcheng He, Yang Yu, Si-Qing Chen, Wayne Xiong, and Furu Wei. Momentum calibration for text generation. arXiv preprint arXiv:2212.04257, 2022.
[40] Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. Calibrating sequence likelihood improves conditional language generation. arXiv preprint arXiv:2210.00045, 2022.
[41] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.
[42] Caleb Ziems, Jane Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. The moral integrity corpus: A benchmark for ethical dialogue systems. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3755-3773, 2022.
A Ethics and Societal Impact
Ethics In our experiments, we utilized three datasets: MI-dataset, ESConv, and MIC. Each of these datasets is designed to protect user privacy, and none contain any personal information.
Societal Impact Our work is driven by the goal to enhance the positive impact of Natural Language Generation (NLG) models by better aligning them with human preferences, particularly those relating to personal standards and human values such as emotions, beliefs, and ethical norms. We acknowledge the diversity of individual preferences, which can often lead to disagreements. This disagreement is taken into account in our model of human preferences, which we use to calibrate our text generation model. This approach aims to produce text that is widely acceptable and less controversial.
Our findings suggest that our approach can make generated texts less contentious and more useful. Looking towards the future, neglecting to account for disagreements in human preferences could result in serious implications, especially if these models are used in safety-critical situations. We anticipate that our approach could be beneficially applied to a wider range of tasks and models in future research and product development.
B Supplement Experiment Details
The code for our experiments is provided in the supplements and will be made public upon acceptance of this paper. Additional details that are not included in the submitted code are explained in the following subsections.
B.1 Resources Used
We trained models based on MultiESC using two Nvidia RTX 3092 GPUs, while all other models, including d-PM models, were trained using a single NVIDIA RTX 3092 GPU. Our models were implemented in Python using PyTorch<sup>3</sup> and the transformers (4.16.2) library<sup>4</sup>.
B.2 d-PM Training Details
Model Architecture The d-PM models are fine-tuned versions of the BERT (base, uncased) model using prefix-tuning. The BERT model has a layer number of  $n_{\mathrm{layer}}^{\mathrm{BERT}}$ , and a hidden size of  $d^{\mathrm{BERT}}$ . The model uses a sequence of prefix indices, represented as  $\mathbf{L}_{\mathrm{idx}}$ , and the length of the prefix is represented by  $|\mathbf{L}_{\mathrm{idx}}|$ . The model input is a concatenation of the prefix, the context, and the text, denoted as  $z = [\mathbf{L}_{\mathrm{idx}}; c; s]$ , where  $z_t$  represents the input at time step  $t$ . The activations of the model are denoted as  $h = [h^{\mathrm{prefix}}, h^c, h^s]$ , where  $h_t$  represents the concatenation of all activation layers at time step  $t$ . Here,  $h^c$  and  $h^s$  are computed using BERT parameters, while  $h^{\mathrm{prefix}}$  are free parameters. A trainable matrix  $A_{\theta_1}$  is then initialized, which is parameterized as  $\theta_1$  of dimension  $|\mathbf{L}_{\mathrm{idx}}| \times d^{\mathrm{dim}}$ , where  $d^{\mathrm{dim}} = n_{\mathrm{layer}}^{\mathrm{BERT}} \times d^{\mathrm{BERT}}$ . Thus, the activation  $h_t$  at time step  $t$  is defined as:
$$
h _ {t} = \left\{ \begin{array}{l l} A _ {\theta_ {1}} [ t, : ] & \text {i f} t <   \left| \mathrm {L} _ {\mathrm {i d x}} \right|, \\ \operatorname {B E R T} \left(z _ {t}, h _ {<   t}\right) & \text {o t h e r w i s e .} \end{array} \right. \tag {12}
$$
We utilized the last hidden states (the activations from the last layer) of the text  $s$  to calculate the probabilities for the classes acceptable and unacceptable. This computation is facilitated by a multilayer perceptron (MLP) layer, which is parameterized as  $\theta_{2}$ .
The parameters of the d-PM, i.e.,  $\theta = (\theta_{1},\theta_{2})$ , are optimized using the loss function, which is formulated in Equation (6). Furthermore, the preference models that we utilized in Alignedmajor and Alignedsoft share the same architecture. However, they are differentiated by the distinct loss functions used for their optimization, as denoted in Equation (10) and Equation (11), respectively.
Experimental Setup The d-PM models were implemented based on the code<sup>5</sup> published for the UnifiedSKG framework [37]. During the training of the d-PM models, BERT parameters were frozen,
<sup>3</sup>https://pytorch.org/
<sup>4</sup>https://huggingface.co/docs/transformers/v4.16.2/en/index
<sup>5</sup>https://github.com/HKUNLP/UnifiedSKG
and the prefix was optimized. The prefix length was set to 10. A batch size of 160 and a learning rate of  $5 \times 10^{-4}$  were used.
Dataset For the training dataset, the MI-Dataset was used to train d-PM model which can output human preferences for the emotional support responses. This dataset contains 15 fine-grained classes of responses. The annotators categorized each response into one of the fine-grained classes. These responses fall into three coarse-grained classes according to the MI codes [25]: (1) MI Adherent, which involves supporting the help-seekers with empathetic and compassionate statements, enabling them to feel heard, respected, and understood; (2) Relational, which focuses on establishing a solid relationship between the help-seeker and the supporter, leading to more positive responses from the help-seekers; and (3) MI Non-Adherent, which includes statements such as arguing, confronting, or offering unsolicited advice, and may create resistance and hinder problem-solving for the help-seekers. When training the d-PM, these classes were converted into two classes: acceptable and unacceptable, as shown in Table 5 The dataset was randomly split into a  $9:1$  ratio for the training and validation set. There are  $34.93\%$  instances where annotators have different annotations.
Table 5: The MI codes in MI-Dataset [36], and our recategorization. Examples are from its paper.
Our Classes	Coarse-Grained MI Codes	Fine-Grained MI Codes	Examples
acceptable	MI Adherent	1. Advise with Permiss-sion	If you agree with it, we could try to brain-storm some ideas that may help.
2. Affirm	You should be proud of yourself for your past efforts.
3. Emphasize Autonomy	It is really up to you to decide.
4. Support	I know it's really hard to stop drinking.
Relational	5. Closed Question	Do you think this is an advantage?
6. Open Question	What is your take on that?
7. Simple Reflection	It sounds like you're feeling worried.
8. Complex Reflection	Speaker: Mostly, I would change for fu-ture generations.
Listener: It sounds like you have a strong feeling of responsibility
9. Give Information	Logging your cravings is important as cravings often lead to relapses.
10. Self-Disclose	I used to be similar where I get obsessed about how people look.
11. Other	Good morning, Hi there.
unacceptable	MI Non-Adherent	12. Advise without Permiss-ion	You should simply scribble a note that reminds you to take a break.
13. Confront	Yes, you are an alcoholic. You might not think so, but you are.
14. Direct	Don't do that!
15.Warn	Be careful, DO NOT stop taking meds without discussing with your doctor.
Table 6: One example from the MIC dataset. Blue text: prompt-reply (QA) pair. Orange text: global consensus scale value.
Question: Am I a bad BF, weird, or just going insane?
Answer: I don’t think you’re a bad bf or weird or going insane. I think you just need to talk to him about it.
RoT: It’s important to communicate honestly with your significant other.
Global Consensus: 4
The MIC dataset was used for training the d-PM used in the integrity Rule of Thumb (RoT) generation task. Each RoT in this dataset is annotated with a global consensus provided in a 5-Likert scale format, where  $1 \sim 5$  represent nobody ( $< 1\%$ ), rare ( $5\% \sim 25\%$ ), controversial ( $\sim 50\%$ ), most
$(75\% \sim 90\%)$ , and all  $(>99\%)$ , respectively. An example is shown in Table 6. For the purpose of training, the ordinal scales were converted into continuous variables that represent the percentage consensus. This was done by randomly selecting a value between the upper and lower bound corresponding to each scale value to serve as the final global consensus  $(\beta)$  for each RoT. Following this, the probability for the class acceptable was set as  $\beta$ , while that for unacceptable was set as  $(1 - \beta)$ . The split of the MIC dataset was maintained when training the model and when calibrating the generation model to avoid data leakage.
B.3 Emotional Support Conversation
Experimental Setup We implemented the base models following the methods detailed in [22, 5] and their corresponding published codes. Each base model was run once using the seed provided in its published codes.
In order to fairly compare models, we aimed to keep the hyperparameters consistent with those of the corresponding base models when implementing aligned models, except that we increased the learning rate to enhance the training efficiency of aligned models. Specifically, we set the learning rate to  $1 \times 10^{-3}$  for the Blender-Vanilla and Blender-Joint base models, and  $3 \times 10^{-5}$  for the other models. Additionally, due to GPU memory constraints, we reduced the batch size from 32 to 12 when training the aligned MultiESC. We ran each aligned model five times with different seeds: 0, 1, 13, 42, and 1024, and reported the average results.
B.4 Integrity RoT Generation
Experimental Setup Our implementation of the base models was based on the work presented in the paper [42] and its published codes. Each base model was run once using the seed provided in its published codes.
For a fair comparison, we endeavored to keep the hyperparameters consistent with those of the corresponding base models when implementing the aligned models. However, we increased the learning rate to  $1 \times 10^{-4}$  during the training of aligned models. Each aligned model was run five times with different seeds: 0, 1, 13, 42, and 1024, and we reported the average results.
B.5 Alignment through Reinforcement Learning
Implementation Details The reinforcement learning (RL) framework was implemented with reference to the Hugging Face TRL library<sup>8</sup>. We aimed to ensure consistency with the hyperparameters used in the aligned model, which was implemented using the contrastive learning framework. Additionally, we adhered to the same training steps, i.e., terminating the training process after 2400 steps. We executed the RL framework with five seeds: 0, 1, 13, 42, and 1024. The best result obtained is reported in the paper.