SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiashuo WANG $^{1}$ , Haozhao WANG $^{2}$ , Shichao SUN $^{1}$ , Wenjie LI $^{1*}$
$^{1}$ Department of Computing, The Hong Kong Polytechnic University
$^{2}$ School of Computer Science and Technology, Huazhong University of Science and Technology
{csjwang,csssun, cswjli} @comp.polyu.edu.hk
hz_wang@hust.edu.cn
In the quest to advance human-centric natural language generation (NLG) systems, ensuring alignment between NLG models and human preferences is crucial. For this alignment, current popular methods leverage a reinforcement learning (RL) approach with a reward model trained on feedback from humans. However, inherent disagreements due to the subjective nature of human preferences pose a significant challenge for training the reward model, resulting in a deterioration of the NLG performance. To tackle this issue, previous approaches typically rely on majority voting or averaging to consolidate multiple inconsistent preferences into a merged one. Although straightforward to understand and execute, such methods suffer from an inability to capture the nuanced degrees of disaggregation among humans and may only represent a specialized subset of individuals, thereby lacking the ability to quantitatively disclose the universality of human preferences. To address this challenge, this paper proposes a novel approach, which employs a Bayesian framework to account for the distribution of disagreements among human preferences as training a preference model, and names it as d-PM. Besides, considering the RL strategy's inefficient and complex training process over the training efficiency, we further propose utilizing the contrastive learning strategy to train the NLG model with the preference scores derived from the d-PM model. Extensive experiments on two human-centric NLG tasks, i.e., emotional support conversation and integrity "Rule-of-Thumbs" generation, show that our method consistently exceeds previous SOTA models in both automatic and human evaluations.
Human-centric natural language processing (NLP) aims to develop NLP systems that are finely attuned to human preferences [14, 12, 30]. Consequently, learning from human feedback is well-suited for training models in human-centric NLG tasks [18, 28]. Currently, reinforcement learning (RL) with a reward model is the most popular method to align models with human preferences [26, 11, 41]. Its effectiveness depends heavily on how well human preferences are learned by the reward model [6, 4]. However, modeling human preferences can be challenging.
Due to the high subjectivity of personal standards and human values, it can be difficult to reach a consensus on preferences among individuals, significantly increasing the learning difficulty. As depicted in Figure 1, persons may have varied preferred responses with inconsistent emotions and values given the same context. To tackle this challenge, existing methods mostly adopt aggregation techniques such as majority voting or averaging [11, 6, 4]. However, aggregated preferences potentially cater
* Corresponding author.
Figure 1: People can have different feelings towards the same response in the emotional support conversation because of their own experiences and values. A trustworthy human-centric system is expected to consider the benefits of universal groups, including minorities, and generate less controversial and more helpful content, like supporter B instead of A.
to specific subsets of people, risking generating controversial content (see Supporter A in Figure 1). Additionally, subjectivity and inconsistency are intrinsic components of certain tasks, such as emotion analysis [13] and ethical evaluation [10], necessitating careful consideration instead of dismissal [3]. Therefore, for widely acceptable and less controversial outputs, it is necessary for the resulting NLG systems to account for capturing disagreements inherent in human preferences [17].
In this paper, we introduce a novel Bayesian-based approach termed Preference Modeling with Disagreement (d-PM). This method is designed to approximate a "universal preference" that comprises the preferences of "all individuals", given the preferences of several individuals. Although a soft label, derived from these several individuals, can intuitively account for disagreement, outliers or extreme labels can disproportionately influence the overall perception. Therefore, we employ Bayesian inference to refine these preferences. Specifically, the observed preference among selected individuals serves as prior knowledge. Our d-PM aims to leverage distribution of all possible universal preferences (likelihood probability) to adjust and smooth the initially observed one, leading to the derivation of a universal preference (posterior). Upon obtaining the universal preference, we calculate the likelihood of the expected preference types to establish a preference score, which is then utilized for further language model alignment.
Based on the d-PM model, we further optimize the language models to generate widely acceptable and less controversial texts. Specifically, we propose utilizing the contrastive learning strategy to calibrate the generation model towards generating texts with high preference scores provided by d-PM. Although existing RL strategies can also be leveraged to make the calibration, they are generally perceived as costly in terms of convergence [9] and online decoding processes [40]. We assess our proposed method on two human-centric NLG tasks: emotional support conversations and moral integrity RoT generation. Experimental results demonstrate that our framework can be applied to state-of-the-art generation models for each task without performance degradation and meanwhile effectively increases global consensus of human preferences embedded in generated texts.
Our main contributions are three-fold: (i). To the best of our knowledge, we are the first to align text generation models with human preferences while considering inherent disagreement among different individuals. (ii). In order to model human preferences with their disagreement, we propose a Bayesian approach, Preference Modeling with Disagreement (d-PM). Additionally, we use its preference scores to calibrate NLG models via contrastive learning for generations that can be widely acceptable and less controversial. (iii). We conduct experiments on two human-centric NLG tasks, i.e., emotional support conversations and integrity RoT generation. Experimental results demonstrate the effectiveness and versatility of our proposed method.
$^{2}$ Our codes are released at https://github.com/wangjs9/Aligned-dPM.
2 Framework
Aligning text generation models with human preferences requires two essential components: modeling human preferences and calibrating the text generation model.
Preference Modeling with Disagreement Human preferences can be inferred from a human-annotated dataset, denoted as  $\mathcal{D}$ . Each instance in the dataset is represented as a triplet  $(c,s,l)$ . Here  $c$  is a context;  $s$  is a text; and  $l$  is a label indicating the annotators' preferences. The inherent disagreement in human preferences can be encapsulated within the label in two distinct ways. In the first approach, the label can be a soft label derived from multiple annotations, all attributed to the same sentence [13]. These annotations are sourced from multiple human annotators to preserve disagreement among individuals. The second approach is the direct collection of global consensus. In this context, the label signifies the proportion of people who find a particular sentence acceptable, an estimate provided by a single human annotator [42, 7].
Aimed at capturing human preference with disagreement within the dataset, we assume there is a distribution  $\rho$  over two classes,  $\{\text{acceptable}, \text{unacceptable}\}$ , comprising preferences of all humans, and therefore  $l$  is the sampling result from  $\rho$ . We employ a preference model  $\mathcal{R}(\theta)$  to infer  $\rho$  given  $c$  and  $s$  as inputs and the probabilistic format of  $l$  as the prior distribution. Since we focus on whether  $s$  is widely acceptable, the likelihood of the class acceptable is defined as the preference score:
$$
\mathcal {S} _ {(s, c)} = \mathcal {R} (s, c; \theta) _ {\text {a c c e p t a b l e}}. \tag {1}
$$
Calibration for Alignment In aligning NLG models with human preferences, we calibrate the existing generation model  $\mathcal{G}(\xi_0)$  with preference scores. Here,  $\mathcal{G}(\xi_0)$  stands for a model already fine-tuned on a dataset  $(X,Y)$ , where  $X$  and  $Y$  represent the input set and the corresponding output set, respectively, and  $\xi_0$  are the optimized parameters. Significantly, if the dataset for preference modeling is identical to the  $(X,Y)$  dataset, then  $(x,y) \in (X,Y)$  corresponds to  $(c,s) \in \mathcal{D}$ . If not, these two datasets should belong to a similar domain. We initially decode  $K$  candidate sequences  $\{\tilde{y}_k\}_{k=1}^{K}$  via  $\mathcal{G}(\tilde{y}_k|x;\xi_0)$  for each  $x \in X$ . Then, we further train  $\mathcal{G}(\xi_0)$ , aimed at a new objective: aligning the likelihoods of candidate sequences with preference scores  $\{\mathcal{S}_{(\tilde{y}_k,x)}\}_{k=1}^{K}$ .
(a). Preference Modeling with Disagreement
(b). Calibration for Alignment
Figure 2: Diagram for preference modeling with disagreement and calibration for alignment.
This section presents our method, whose diagram is shown in Figure 2. We first use a Bayesian approach, i.e., d-PM, to model human preferences with disagreement (Section 3.1). Then we calibrate a text generation model by contrastive learning with preference scores of d-PM to align this model with human preferences (Section 3.2).
3.1 Preference Modeling with Disagreement
We establish a distribution  $\rho$  to represent the universal preference for the text  $s$  given its context  $c$ . Therefore, the observed annotations  $l$  are considered as samples from  $\rho$ , and can form a prior distribution  $p_i(\rho)$ . Inspired by [29], we devise a Bayesian approach to approximate  $\rho$  using this prior.
Specifically, we establish a connection between  $\rho$  and  $l$  through the optimization process of a generative model. This model is designed for generating text  $s$  conditioned on  $c_{i}$  and  $\rho$ :  $p(s|c_i,\rho)$ . The log-likelihood of the text can be formulated as  $\sum_{i}\log p(s_{i}|c_{i}) = \sum_{i}\log (\sum_{\rho}p(s_{i}|c_{i},\rho)p_{i}(\rho))$ , where  $p_i(\rho)$  is the prior preference distribution. Its optimization can be achieved by introducing a variational posterior distribution  $q(\rho |s_i,c_i)$  for the  $i$ -th datapoint, and minimizing the free energy (negated evidence lower bound) formulated as:
$$
- \sum_ {i} \log p \left(s _ {i} \mid c _ {i}\right) + \sum_ {i} \sum_ {\rho} q \left(\rho \mid s _ {i}, c _ {i}\right) \log \frac {p \left(s _ {i} \mid c _ {i} , \rho\right) p _ {i} (\rho)}{q \left(\rho \mid s _ {i} , c _ {i}\right)}. \tag {2}
$$
Minimization of the free energy involves estimations of both the forward distribution of text  $s_i$ :  $p(s_i | c_i, \rho)$ , and the posteriors  $q(\rho | s_i, c_i)$ , which can be computed by our preference model:
$$
q \left(\rho \mid s _ {i}, c _ {i}\right) = \mathcal {R} \left(s _ {i}, c _ {i} \mid \theta\right). \tag {3}
$$
As for  $p(s_i|c_i, \rho)$ , it is defined only on the  $i$ -th datapoint and is computed by minimizing Equation (2) for fixed  $q(\rho | s_i, c_i)$ , s.t.,  $\sum_{i} p(s_i|c_i, \rho) = 1$  for all  $\rho$ . Thus, the optimum is achieved by:
$$
p \left(s _ {i} \mid c _ {i}, \rho\right) = a _ {i, \rho} = \frac {q \left(\rho \mid s _ {i} , c _ {i}\right)}{\sum_ {j} q \left(\rho \mid s _ {j} , c _ {j}\right)}. \tag {4}
$$
From Equation (4), the generative model can be regarded as a matrix of variables  $a_{i,\rho}$  describing conditional probabilities of different responses  $s_i$  given different latent distributions  $\rho$  with known  $c_{i}$ .
In a variational way, Equation (2) can be rewritten as  $-\log p(s_i|c_i) + \sum_i\mathrm{KL}(q(\rho |s_i,c_i)\| r_i(\rho))$ . Here,  $r_i(\rho)\propto p(s_i|c_i,\rho)p_i(\rho)$  is the posterior model of the generative model, and it can be reformulated with reduction of  $p(s_{i}|c_{i},\rho)$  to the matrix in Equation (4):
$$
r _ {i} (\rho) = \alpha_ {i} \cdot p _ {i} (\rho) p \left(s _ {i} \mid c _ {i}, \rho\right) = \alpha_ {i} \frac {p _ {i} (\rho) q _ {i} \left(\rho \mid s _ {i} , c _ {i}\right)}{\sum_ {j} q _ {j} \left(\rho \mid s _ {j} , c _ {j}\right)}, \tag {5}
$$
where  $\alpha_{i}$  is a scalar enabling  $\sum_{\rho}r_{i}(\rho) = 1$ . Accordingly, we can minimize the KL divergence between  $q(\rho |s_i,c_i)$  and  $r_i(\rho)$  to minimize the free energy. The minimization of the free energy in Equation (2) can be derived as:
$$
\sum_ {i} \mathrm {K L} (q (\rho | s _ {i}, c _ {i}) \| r _ {i} (\rho)) = \min  _ {\theta} \sum_ {i} \mathrm {K L} \left(\mathcal {R} (s _ {i}, c _ {i}; \theta) \| \alpha_ {i} \cdot p _ {i} (\rho) \frac {\mathcal {R} (s _ {i} , c _ {i} ; \theta)}{\sum_ {j} \mathcal {R} (s _ {j} , c _ {j} ; \theta)}\right). \tag {6}
$$
By optimizing the above objective, we can optimize the parameters of our preference model, i.e.,  $\theta$ .
3.2 Calibration for Alignment
It is nearly possible for  $\mathcal{G}(\xi_0)$  to generate texts with both high and low preference scores, shown in Figure 3. However, we expect to calibrate the model such that the generation probability aligns with these preference scores. Specifically, we use diverse beam search [34] to generate multiple candidates and then use our d-PM to evaluate these candidates. For the sake of more likely generating a high preference score text, we propose a model-agnostic module to leverage contrastive learning to calibrate generation likelihood aligning with d-PM. Taking inspiration from recent calibration work [32, 23, 39], we implement this module through the following three steps:
Step 1: Candidates Generation. We generate candidates from the text generator  $\mathcal{G}(\xi_0)$ , which has been fine-tuned on corresponding dataset  $(X,Y)$  and its parameters are  $\xi_0$ , on its own training dataset. Given an input sequence  $x \in X$ , we first use  $\mathcal{G}(\xi_0)$  to generate  $K$  candidates  $\{\tilde{y}_1,\tilde{y}_2,\dots ,\tilde{y}_K\}$  using diverse beam search. As a result, these candidates will get similar possibilities yet different preference scores according to the above preliminary study.
Step 2: Preference-based Ranking. We use our proposed d-PM  $\mathcal{R}(\theta)$  to measure the preference score  $S_{(\tilde{y}_k,x)}$  of each candidate  $\tilde{y}_k$ . Then we rank these candidates according to the above preference score and obtain a list of ranked candidates:  $\tilde{y}_1', \tilde{y}_2', \dots, \tilde{y}_K'$ , where  $S_{(\tilde{y}_i',x)} > S_{(\tilde{y}_j',x)}$  for  $\forall i < j$ .
Step 3: Likelihood Calibration. As mentioned before, we leverage contrastive learning to assign higher likelihoods to the candidates with higher preference scores. The following pairwise margin loss is used to adjust the generator  $\mathcal{G}(\xi)$ .
$$
\mathcal {L} ^ {r} = \sum_ {i} \sum_ {j > i} \max  \left(0, \mathbf {P} \left(\tilde {y} _ {j} ^ {\prime}; \xi\right) - \mathbf {P} \left(\tilde {y} _ {i} ^ {\prime}; \xi\right) + \lambda_ {i j}\right), \tag {7}
$$
where  $\lambda_{ij}$  is the default margin  $\lambda$  multiplied by the difference in rank between the samples, i.e.,  $\lambda_{ij} = \lambda * (j - i)$ .  $\mathbf{P}(\hat{y}_i'; \xi)$  is the length-normalized log-probability of the candidate:
Figure 3: The maximum and minimum preference scores of 10 candidates generated via diverse beam search given the same context. We test on 1000 data instances and three emotional support conversation models.
$$
\mathbf {P} \left(\tilde {y} ^ {\prime}; \xi\right) = \frac {\sum_ {t = 1} ^ {\left| \tilde {y} ^ {\prime} \right|} \log \mathcal {G} \left(\tilde {y} _ {t} ^ {\prime} \mid x , \tilde {y} _ {<   t} ^ {\prime} ; \xi\right)}{\left| \tilde {y} ^ {\prime} \right| ^ {\alpha}}, \tag {8}
$$
where  $\alpha$  is the length penalty hyperparameter. To avoid forgetting token-level likelihood information of the ground-truth text, we also use an additional token-level negative log-likelihood. The final calibration loss is as follows:
$$
\mathcal {L} ^ {c} = - \lambda \frac {1}{| y |} \sum_ {t = 1} ^ {| y |} \log \mathcal {G} \left(y _ {t} \mid x, y _ {<   t}; \xi\right) + \mathcal {L} ^ {r}. \tag {9}
$$
We minimize  $\mathcal{L}^c$  to optimize the generator's parameters  $\xi$ . This process is supervised by our d-PM model and aligns the generation model with human preference.
4.1 Emotional Support Conversation
In an emotional support conversation, a supporter aims to buffer a help-seeker's emotional distress and help the help-seeker to change the difficult situation [22]. In this context, the model functions as the supporter, while the user is always the help-seeker. Due to different personal experiences, different help-seekers may respond with varied feelings and reactions to the same response, as illustrated in Figure 1. Our objective is to enhance the model's ability to generate responses that will not escalate the negative feelings of a diverse range of help-seekers.
Dataset and Base Models The benchmark ESConv [22], containing approximately 1k conversations with 31k utterances, develops each conversation between a help-seeker and a supporter. We include BlenderBot-Vanilla and BlenderBot-Joint proposed in conjunction with the dataset, and the SOTA model MultiESC[5]. We reproduced the base models in accordance with their respective papers and publicly available codes.
Human Preferences with Disagreement We derive human preferences from the Motivational-Interviewing-Dataset [36]. This dataset encompasses around 17k supporter responses to help-seekers. Each response is annotated by  $2\sim 4$  experts following the MI codes[25]. The labels can be induced into two classes  $\{\mathrm{acceptable},\mathrm{unacceptable}\}$ , and the human preferences with disagreement can be estimated by our d-PM method. By fine-tuning a BERT model on this dataset using prefix-tuning [37, 19], we obtain the d-PM. Additional details can be found in Appendix B.2.
Table 1: Automatic evaluation results on ESConv. All results are significantly better than the corresponding base model with  $p < 0.01$ .
Model	B-1	B-2	B-3	B-4	R-L	METEOR	CIDEr	Extreme
Blender -Vanilla	Base	17.85	7.08	3.60	2.11	17.06	7.46	15.44	51.02
Alignedmajor	19.07	7.71	3.94	2.28	17.09	7.71	15.97	50.61
Alignedsoft	17.88	7.21	3.68	2.12	16.52	7.31	15.50	50.73
Alignedw/oA	19.70	7.56	3.64	2.05	16.90	7.72	15.62	50.48
Alignedd-PM	20.75	8.32	4.17	2.39	17.41	8.21	16.57	50.38
Blender -Joint	Base	18.70	7.30	3.61	2.03	17.66	7.56	16.91	50.95
Alignedmajor	20.37	8.61	4.47	2.65	19.23	8.32	21.86	51.57
Alignedsoft	19.36	7.87	3.85	2.09	17.55	7.65	15.90	50.84
Alignedw/oA	21.05	8.14	3.89	2.07	17.65	8.11	15.29	50.68
Alignedd-PM	21.05	8.97	4.74	2.78	19.39	8.48	20.34	51.81
MultiESC	Base	20.36	8.80	4.92	3.14	21.00	8.58	30.69	52.74
Alignedmajor	19.10	8.27	4.61	2.88	20.72	8.24	30.15	52.57
Alignedsoft	19.30	8.33	4.62	2.88	20.83	8.35	30.75	52.54
Alignedw/oA	21.58	8.80	4.74	2.96	20.47	8.78	28.58	51.65
Alignedd-PM	21.59	9.56	5.33	3.36	21.50	9.03	32.65	53.15
Experimental Setup We apply our proposed method to align each base model, thus treating the well-trained base model as the generator  $\mathcal{G}(\xi_0)$ . Additionally, to validate the effectiveness of d-PM, we employ three alternative preference models within our framework for comparative analysis:
(1) A preference model (major) trained to predict the majority voting result of annotations from different annotators, denoted as  $l_{m}$ , and optimized by cross-entropy loss, formulated as:
$$
\mathcal {L} (\theta) = - \mathbb {E} _ {(c, s, l _ {m}) \sim \mathcal {D}} [ p _ {l} (l _ {m}) \log (\mathcal {R} ^ {l _ {m}} (c, s; \theta)) ]. \tag {10}
$$
where  $p_l(l_m)$  denotes the one-hot vector of  $l_{m}$ .
(2) A preference model (soft) trained to approximate the direct probabilistic label of annotations, i.e., the soft label  $l$ . The model is optimized by:
$$
\mathcal {L} (\theta) = \mathbb {E} _ {(c, s, l) \sim \mathcal {D}} \| \mathcal {R} (c, s; \theta) - l \| ^ {2}. \tag {11}
$$
(3) A preference model (w/oA) that does not aggregate annotations and takes each annotation as independent. This model is optimized by cross-entropy loss, similar to Equation (10).
When training the aligned models, we aim to retain the same hyperparameters used in the training of the base models. We set the candidate number  $K$  to 10. We train each aligned model five times with five different seeds. Subsequently, we test each of the five trained models on the test dataset and compute the average results.
Automatic Evaluation We adopt the following metrics commonly used in previous work [5, 22] for the automatic evaluation of our proposed method: BLEU [27] (B-1/2/3/4), ROUGE (R-L) [20], METEOR [2], CIDEr [33], and BOW Embedding-based matching score [21] (Extreme). Results are shown in Table 1.
Our  $\mathrm{Aligned_{d - PM}}$  significantly improves the performance of the base model in almost all automatic metrics, irrespective of the base model, suggesting the overall effectiveness of our proposed method. Alignedmajor and Alignedsoft are able to enhance the performance when the base model is either Blender-Vanilla or Blender-Joint, however, they do not yield an improvement when the base model is MultiESC. This limitation underscores the constraints inherent in using majority voting labels and soft labels to address disagreement of human preferences.  $\mathrm{Aligned}_{\mathrm{w / oA}}$  surpasses the base model in certain metrics but falls short in others. Notably, it performs significantly lower in CIDEr, a metric evaluating the similarity between TFIDF-weighted n-grams. This shortfall suggests that Alignedw/oA is less likely to generate responses containing critical information found in the ground truth. This issue arises from the preference scores determined by w/oA being closely clustered in value, resulting in its inability to sequence the generated samples logically. These pieces of evidence indicate the potency of our proposed preference model d-PM.
Table 2: Human evaluation results on ESConv. The base model is MultiESC.
Model	Identification	Comforting	Suggestion	Overall	Global Consensus
Base	3.017	2.562	2.918	2.598	2.693
Alignedmajor	3.032	2.572	2.880	2.598	2.763
Alignedsoft	3.007	2.557	2.905	2.568	2.747
Alignedd-PM	3.052	2.587	2.952	2.637	2.783
Human Ratings We ask human annotators to evaluate the generations of models based on MultiESC since MultiESC can outperform Blender-Vanilla and Blender-Joint in almost all automatic evaluations. Specifically, we randomly sample 100 responses generated by different models for human ratings. We asked annotators to imagine they are help-seekers in the corresponding situation and measure each response in five aspects: (1). Identification: on a scale of  $1 \sim 5$ , how much the response can explore your situation in depth and help identify the problems. (2). Comforting: on a scale of  $1 \sim 5$ , how skillful the response can comfort you. (3). Suggestion: on a scale of  $1 \sim 5$ , how helpful the response can solve your problems. (4). Overall: on a scale of  $1 \sim 5$ , the overall quality of this response for emotional support; (5) Global Consensus: the number of people who deem the response can help them,  $1 \sim 5$  represent nobody ( $< 1\%$ ), rare ( $5\% \sim 25\%$ ), controversial ( $\sim 50\%$ ), most ( $75\% \sim 90\%$ ), and all ( $>99\%$ ), respectively. Each response is annotated by three annotators, and we averaged these three annotations as the final result for each metric.
From Table 2, our method performs the best among the methods.  $\mathrm{Aligned_{d - PM}}$  obtained the highest score in all aspects, including the global consensus. It demonstrates that our method can generate less controversial and more helpful responses in the task of emotional support conversation.
4.2 Integrity RoT Generation