SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Peiwen Yuan, Xinglin Wang, Jiayi Shi, Bin Sun, Yiwei Li, Kan Li*  
School of Computer Science and Technology, Beijing Institute of Technology, China  
{peiwenyuan, wangxinglin, shijiayi, binsun, liyiwei, likan}@bit.edu.cn
Turn-level dialogue evaluation models (TDEMs), using self-supervised learning (SSL) framework, have achieved state-of-the-art performance in open-domain dialogue evaluation. However, these models inevitably face two potential problems. First, they have low correlations with humans on medium coherence samples as the SSL framework often brings training data with unbalanced coherence distribution. Second, the SSL framework leads TDEM to nonuniform score distribution. There is a danger that the nonuniform score distribution will weaken the robustness of TDEM through our theoretical analysis. To tackle these problems, we propose Better Correlation and Robustness (BCR), a distribution-balanced self-supervised learning framework for TDEM. Given a dialogue dataset, BCR offers an effective training set reconstructing method to provide coherence-balanced training signals and further facilitate balanced evaluating abilities of TDEM. To get a uniform score distribution, a novel loss function is proposed, which can adjust adaptively according to the uniformity of score distribution estimated by kernel density estimation. Comprehensive experiments on 17 benchmark datasets show that vanilla BERT-base using BCR outperforms SOTA methods significantly by  $11.3\%$  on average. BCR also demonstrates strong generalization ability as it can lead multiple SOTA methods to attain better correlation and robustness. Code and datasets: https://github.com/ypw0102/Better-Correlation-and-Robustness.
Evaluating model-generated responses efficiently and accurately can facilitate the hyperparameter tuning and comparison among models, which is essential for the research of open-domain dialogue system (Bao et al., 2020; Sun et al., 2021; Feng et al., 2021; Li et al., 2023). Therefore, economical and practical automatic metrics are widely applied instead of human evaluation during development phase. However, the referenced automatic metrics, assessing dialogue based on the golden response (e.g. BLEU (Papineni et al., 2002) and BERTScore (Zhang et al., 2020b)), have been shown to be inaccurate for dialogue evaluation (Liu et al., 2016; Deriu et al., 2021) due to the one-to-many nature of dialogue (Zhao et al., 2017). Fortunately, the unreferenced metrics, especially the turn-level dialogue evaluation models (TDEMs) (Mehri and Eskenazi, 2020b,a; Huang et al., 2020; Ye et al., 2021; Zhang et al., 2021a, 2022b), have achieved a favorable budget between efficiency and accuracy.
Nevertheless, existing TDEMs still perform worse correlations with human judgements on medium coherence samples (samples with human ranking scores falling within the interval [0.25, 0.75]) and lack robustness as shown in Figure 1. We argue that these problems stem from the self-supervised learning (SSL) paradigm, which generally constructs positive and negative samples to strengthen
Figure 1: Deficiencies of self-supervised TDEM. (a) TDEM possesses low correlations with human judgements on conversations of medium coherence due to the lack of such samples of training data. (b) TDEM attains poor robustness due to the nonuniform score distribution caused by discrete label domain of training data together with general fixed loss functions.
the scoring ability of TDEM. Due to the fact that positive samples possess high coherence while negative samples possess low coherence (Sato et al., 2020), TDEM lacks training signals of medium coherence samples, thus leading to a bad performance when evaluating on such samples (e.g., (Q, R2) in Figure 1). Moreover, current discrete label domain together with fixed loss functions generally brings TDEM a nonuniform score distribution (Figure 1, 3), which will hurt the robustness of TDEM according to our theoretical analysis (demonstrated in §3.4).
To solve the aforementioned problems, we propose Better Correlation and Robustness (BCR), a distribution-balanced SSL framework for TDEM. BCR offers a Training Set Reconstructing (TSR) method to balance the data coherence distribution of training set and advance a high correlation with humans on medium coherence samples, and a Dynamic Penalty loss function (DP loss) for a more uniform model score distribution. In TSR method, we use two rule-based strategies to expand a conversation into a positive sample and a negative sample respectively to supplement medium coherence samples. Considering that utterances within a conversation are internally coherent, our strategies are reasonable. We also provide continuous labels for these medium coherence samples through self-distillation to alleviate the deviation of discrete two-level labels. DP loss promotes TDEM to obtain a more uniform score distribution by adaptively adjusting the penalty strength of samples in different scoring intervals according to the uniformity of score distribution estimated by kernel density estimation (Parzen, 1962), thus enhancing the robustness. Our contributions are summarized as follows:
- We propose Better Correlation and Robustness (BCR), a novel distribution-balanced SSL framework for TDEM, which can effectively alleviate limitations of vanilla SSL framework through Training Set Reconstructing (TSR) and Dynamic Penalty loss function (DP loss).
- We theoretically prove that the robustness of TDEM and the uniformity of score distribution are positively correlated, and further verify it through experiments.
- Comprehensive experiments on 17 benchmark datasets show that vanilla BERT-base (Devlin et al., 2019) applying BCR can outperform SOTA methods significantly by  $11.3\%$  on average. We also demonstrate the strong generalization of BCR as it can lead multiple SOTA methods to attain better robustness and correlations with human judgements.
Referenced metrics have shown to correlate poorly with human judgements (Liu et al., 2016) and thus various unreferenced model-based metrics have been proposed, which can be divided into CDEM (conversation-level) and TDEM (turn-level). CDEMs (Mehri and Eskenazi, 2020a; Ghazarian et al., 2022; Zhang et al., 2022a) evaluate conversations between real user and dialogue system. As this interactive evaluation occupies too much extra energy of researchers, we mainly study TDEM, which can be divided into supervised and self-supervised.
Supervised TDEMs (Lowe et al., 2017; Ye et al., 2021) are currently hard to train due to the lack of human annotated datasets. It is worth mentioning that the reward models used in the RLHF stage of LLM training (e.g., InstructGPT (Ouyang et al., 2022)) are essentially supervised TDEMs. Although existing literature has not published the performance of these models, we proved through experiments
that as a pre-training method, BCR can help supervised TDEMs converge to better performance faster (see Appendix C.2). This may inspire scholars to research and apply pre-trained TDEMs to improve the performance of reward models, thus attaining better LLMs.
Self-supervised TDEMs generally apply next utterance prediction as training task, in which the key points are training set construction and training loss design. Original response and randomly sampled response are generally regarded as positive and negative samples respectively for a given context (Tao et al., 2018; Phy et al., 2020). Some literature attains hard negative samples through Word2Vec similarity based filtering (Zhang et al., 2022b), speaker-level utterance shuffling (Zhang et al., 2021a), model generation (Lan et al., 2020) and adversarial craft (Sai et al., 2020; Ye et al., 2021). However, the lack of medium coherence samples will lead TDEMs to unbalanced evaluating ability. General loss functions are widely used to train TDEMs, such as Triplet loss (Tao et al., 2018), CrossEntropy loss (Mehri and Eskenazi, 2020b), NCE loss (Sinha et al., 2020), etc. (Ye et al., 2021) applies three loss functions and attains a score distribution with three peaks. All of these loss functions together with discrete label domain generally lead to a nonuniform score distribution, which will hurt the robustness of TDEMs, as proven in 3.4.
In this section, we first introduce the task definition and a baseline model, followed by verifying the low correlations and poor robustness caused by the SSL framework.
3.1 Task Definition
Given a conversation triplet (context, query, response) where response is offered by a dialogue system, TDEM is required to measure the quality (e.g., coherence) of the response. We can calculate the correlation coefficient (e.g., Spearman Correlation) with human judgements on annotated datasets to evaluate the performance of TDEM.
3.2 Baseline Model
Our baseline model is shown in Figure 2. Specifically, it contains a BERT-base-uncased model (Devlin et al., 2019) as feature extractor and a threelayers multi-layer perceptron to get a coherence score distributed within interval [0, 1]. Apart from general embeddings (token, speaker, position), we additionally apply class embeddings to distinguish utterances from context, query and response as the coherence between query and response is the dominant judging factor (Li et al., 2021). Following GRADE (Huang et al., 2020), we use Triplet loss to train the baseline model on the DailyDialog dataset (Li et al., 2017). The gap of Triplet loss is set as 0.3 at which our baseline model can attain best performance. For each conversation, we randomly select T  $(\mathrm{T} > 1)$  consecutive turns as positive sample and replace the last utterance with a randomly chosen utterance to get negative sample.
Figure 2: Baseline model. We apply four kinds of embeddings from top to bottom: token embedding, speaker embedding, position embedding and class embedding.
3.3 Coherence Distribution
To verify the impact of unbalanced coherence distribution of training data, we test our baseline model on subsets of DSTC10 Zhang et al. (2021b) dataset with different coherence respectively (Table 2). Our baseline model (w/o. TSR) obtains 0.125 Spearman correlation at the polarized interval while only 0.073 at the medium interval. This proves that TDEM applying general SSL framework performs bad on medium coherence samples.
2http://yanran.li/dailydialog
(a) MAE  $(\mathcal{L}_1)$
(b) CE
(c) MSE  $(\mathcal{L}_2)$
(d) Triplet
(e)  $\mathcal{L}_3$
(f)  $\mathcal{L}_{DP}$
(g)  $\mathcal{L}_7$
Figure 3: Score distribution of TDEM trained with different loss functions on DSTC10 dataset.
(h)  $\mathcal{L}_{15}$
3.4 Score Distribution
As shown in Figure 3, widely used loss functions (MAE, MSE, Triplet, CrossEntropy) lead TDEM to a polarized score distribution under the discrete label domain setting. We conjecture that these loss functions continuously give a relatively significant penalty loss to the samples already scoring within polarized interval, which aggravate this polarized trend.
We primarily consider the influence of score distribution on the robustness of TDEM. Given a small disturbance  $\epsilon$  at the sample, the corresponding predicted score changes  $\lambda$ , which will affect Spearman correlations  $r_s$  between the original score  $x$  and the new score  $x + \lambda$ . Robust TDEM is supposed to resist noise, thus attains higher  $r_s$ . Hence, we can use  $\mathbb{E}(r_s)$ , the mathematical expectation of  $r_s$  on  $n$  samples, to reflect the robustness of TDEM, and make the following statement:
Theorem 1 For any  $f(x)$ , the probability density function of TDEM score distribution,  $\mathbb{E}(r_s)$  has an upper bound after a small disturbance:
$$
\mathbb {E} \left(r _ {s}\right) \leq 1 - \frac {6 \mathbb {E} (\lambda) ^ {2}}{n ^ {2} - 1}, \tag {1}
$$
and the equality condition is  $f(x) \equiv 1, \forall x \in [0,1]$ .
Proof 1 The ranking difference  $d(x)$  before and after disturbance is :
$$
d (x) = \int_ {x} ^ {x + \lambda} f (x) d x \tag {2}
$$
According to the definition of Spearman correlations,  $E(r_{s})$  can be written as:
$$
\mathbb {E} \left(r _ {s}\right) = \mathbb {E} \left(1 - \frac {6 \sum_ {i = 1} ^ {n} d \left(x _ {i}\right) ^ {2}}{n \left(n ^ {2} - 1\right)}\right), \tag {3}
$$
we derive the lower bound of  $\mathbb{E}(d(x)^2)$  as follows (See detailed derivation in Appendix A):
$$
\begin{array}{l} \mathbb {E} (d (x) ^ {2}) = \int_ {0} ^ {1} \left(\int_ {x} ^ {x + \mathbb {E} (\lambda)} f (u) d u\right) ^ {2} f (x) d x \\ \geq \left(\int_ {0} ^ {1} \int_ {x} ^ {x + \mathbb {E} (\lambda)} f (u) d u f (x) d x\right) ^ {2} \tag {4} \\ \geq \mathbb {E} (\lambda) ^ {2} \\ \end{array}
$$
The equality condition of Eq. (4) is  $f(x) \equiv 1$  for  $x \in [0,1]$ . Taking the lower bound of  $\mathbb{E}(d(x)^2)$  into Eq. (3), we conclude the proof.
Note that higher  $\mathbb{E}(r_s)$  denotes better robustness of TDEM. Hence, we can derive that the robustness of TDEM correlates positively with the uniformity of score distribution based on Theorem 1.
Figure 4: Illustration of BCR Framework.
4 BCR framework
We propose BCR framework (Figure 4) to alleviate the low correlations on medium coherence samples and poor robustness caused by SSL framework. BCR offers: (1) training set reconstructing (TSR) to attain coherence-balanced training samples with continuous label domain ( $\S 4.1$ ); (2) DP loss to bring TDEM a more uniform score distribution and better robustness ( $\S 4.2$ ).
4.1 Balance Coherence Distribution
As training set of SSL framework generally lacks samples of medium coherence, we propose TSR to reconstruct a coherence balanced training set. Specifically, We replace  $N\%$  samples of the original dataset  $\mathbb{D}_{ori}$  with samples of medium coherence generated through two strategies, denoted as  $\mathbb{D}_{aug}$ .
Given a conversation, we suppose that replacing response with an utterance of context generally leads to a negative sample with medium coherence and disrupting context while maintaining the query and response leads to a positive sample with medium coherence. Our strategies are motivated by (Zhang et al., 2021a), which shuffles all utterances of a speaker to attain a hard negative sample. As a comparison, the samples generated by our strategies are more controllable in terms of coherence. Take Figure 4 for example. As we randomly replace  $④$  with  $②$ , although  $②$  does not answer  $③$ , they have the same topic (NBA player). Thus,  $(1, 2, 3, 2)$  constitute a negative sample with medium coherence. As we exchange the order of  $①$  and  $②$ , it remains a positive example since  $④$  still answers  $③$ . However, the word talent in  $④$  seems unnatural as  $②$  is no longer said by Speaker B, which makes  $(2, 1, 3, 4)$  a positive sample with medium coherence. Further human evaluation experiments show that TSR can provide stronger training signals with medium coherence compared with advanced data augmentation methods (Appendix B.4).
Assigning discrete two-level labels to samples with medium coherence can result in serious deviation. Therefore, we combine the original discrete labels and scores of TDEM to attain soft pseudo labels for the augmented samples and the final labels of the whole reconstructed dataset are as follows:
$$
\operatorname {L a b e l} _ {T S R} (x) = \left\{ \begin{array}{l} \operatorname {L a b e l} _ {\text {d i s c r e t e}} (x), x \in \mathbb {D} _ {\text {o r i}} \\ \alpha \times \operatorname {L a b e l} _ {\text {d i s c r e t e}} (x) + (1 - \alpha) \times \mathcal {M} (x), x \in \mathbb {D} _ {\text {a u g}} \end{array} \right. \tag {5}
$$
where  $\mathcal{M}$  denotes TDEM,  $Label_{discrete}(x) = 1$  if  $x$  is a positive sample and 0 otherwise. During training process, pseudo labels are obtained in real time.
4.2 Balance Score Distribution
Based on Theorem 1, we consider adaptively approaching the optimal loss function for a uniform score distribution to strengthen the robustness of TDEM. According to Weierstrass Approximation Theorem (Weierstrass, 1885), we consider approximating the optimal loss function in with polynomial loss function  $\mathcal{L}(x)$ :
$$
\mathcal {L} _ {i} (x) = \left| \mathcal {M} (x) - \operatorname {L a b e l} _ {T S R} (x) \right| ^ {i} \tag {6}
$$
$$
\mathcal {L} (x) = \sum_ {i = 0} ^ {\infty} \beta_ {i} \mathcal {L} _ {i} (x) \tag {7}
$$
To gain a deeper insight of the relationship between the probability density function of TDEM score distribution  $f(x)$  and  $\mathcal{L}_i(x)$ , we propose Loss Ratio Curve (Lrc) as a monitor. Taking the difference value (D-value) between label and predicted score as X-axis, the ratio of the corresponding loss and fiducial loss (loss when D-value is 0.5) as Y-axis, we get Lrc shown in Figure 5. Lrc visualizes the penalty strength (loss ratio) at different D-value of a certain loss function. The stronger concavity Lrc possesses, the smaller penalty the samples with polarized scores attain. Combining Figure 3 and Figure 5, score distribution undergoes a transition from polarization to centralization with the increasing concavity of Lrc and the rising power of  $\mathcal{L}_i(x)$ . This not only confirms our conjecture in Section 3.4, but also indicates that
$\mathcal{L}_i(x)$  with small power (e.g.,  $\mathrm{i} = 3$ ) generally leads TDEM to a polarized distribution while  $\mathcal{L}_i(x)$  with large power (e.g.,  $\mathrm{i} = 15$ ) promotes a centralized distribution.
Figure 5: Loss ratio curve of different loss functions
Bared this insight in mind, we can simplify Eq. (7) and attain our DP loss function as follows:
$$
\mathcal {L} _ {D P} (x) = \mathcal {L} _ {s} (x) + \beta \mathcal {L} _ {l} (x) \tag {8}
$$
where  $s$  is smaller than  $l$ . During training,  $\mathcal{L}_s(x)$  and  $\mathcal{L}_l(x)$  promote polarized and centralized score distribution respectively, and  $\beta$  adjusts adaptively according to the score distribution to make TDEM converge stably to a more uniform distribution. Specifically, we use kernel density estimation (KDE) to estimate the score distribution of TDEM on training set  $\mathbb{D}$  at the end of each epoch as follows:
$$
\hat {f} (x) = \frac {1}{| \mathbb {D} | h} \sum_ {i = 1} ^ {| \mathbb {D} |} K \left(\frac {x - X _ {i}}{h}\right) \tag {9}
$$
where  $|\mathbb{D}|$  is the size of the training set,  $h$  is the bandwidth of KDE,  $X_{i}$  is the model score of the  $i^{th}$  sample and  $K(\cdot)$  is the kernel function. We divide [0, 1] into polarized interval ([0, 0.25] and [0.75, 1]) and centralized interval ([0.25, 0.75]). If the estimated distribution shows a polarized trend (the integral of  $\hat{f}(x)$  in polarized interval  $>0.6$ ), we update  $\beta_{new} = \beta_{old} \times 10$  to enhance the influence of  $\mathcal{L}_l$ , so as to alleviate such trend. On the contrary, we update  $\beta_{new} = \beta_{old} / 10$ . From a more intuitive perspective, DP loss can dynamically adjust the penalty strength on different samples to approximate the optimal loss function, thereby enabling TDEM to achieve a relatively uniform score distribution.
We first apply BCR framework on BERT-base, and compare it with the SOTA methods on multiple datasets to verify its effectiveness. Then, we apply BCR framework on existing SOTA TDEMs to verify the generalization of BCR and whether it can lead TDEMs to attain better robustness and correlations with human judgements. Ablation study and case study are followed to further understand the effectiveness of BCR. $^3$
5.1 BERT-base With BCR
5.1.1 Experimental Setup
We apply BCR framework to train the baseline model mentioned in Section 3.2 on DailyDialog dataset. We set learning rate as 5e-5, batch size as 32,  $\alpha$  in Eq. (5) as 0.8, initial value of  $\beta$  as 1,  $N$  in TSR as 20. The default value of  $s$  and  $l$  in Eq. (8) are 3 and 7, and we examined the effects of different values in 5.3. In Eq. (9), we use Gaussian kernel and set h as 0.01. AdamW Loshchilov and Hutter (2017) is used as the optimizer (see Appendix B.2 for details).
We conduct experiments on 17 benchmark quality-annotated datasets and compare the results with SOTA methods of each dataset (see Table 5). As DSTC10 datasets contain five sub-datasets and
3For each experiment, We run four random seeds and report the averaged result.
Table 1: Spearman correlations between TDEMs and human judgements on 17 datasets, with standard deviations in gray. BERT refers to baseline model (See Section 3.2). The results of SOTA TDEMs come from the corresponding papers. All results are statistically significant (p-value  $< 0.05$ ).
Method	FED	GCG	GCR	GDG	GDR	GEG	GER	PE	DSTC6
BERT	0.205	0.606	0.500	0.310	0.197	0.202	0.315	0.612	0.224
SOTA	0.264	0.617	0.558	0.358	0.187	0.223	0.338	0.699	0.295
BERT+BCR	0.286	0.642	0.519	0.416	0.487	0.341	0.372	0.669	0.274
Method	UT	UP	JSALT	ESL	NCM	DT	DP	DSTC10	Average
BERT	0.305	0.372	0.166	0.345	0.253	0.220	0.401	0.255	0.323 ± 0.012
SOTA	0.419	0.469	0.116	0.414	0.299	0.326	0.456	0.310	0.373
BERT+BCR	0.421	0.435	0.272	0.531	0.336	0.305	0.428	0.316	0.415 ± 0.005
corresponding 11 qualities, apart from the appropriateness quality shared by the five sub-datasets, we also report the average results (column DSTC10 in Table 1) of all the 11 qualities. Results of SOTA methods come from the corresponding papers. We also reproduced the results of GRADE (Huang et al., 2020) and USR (Mehri and Eskenazi, 2020b) for a convincing comparison (Figure 7).
5.1.2 Experimental Results
As shown in Table 1, BERT-base applying our BCR framework obtains the highest correlations with human judgements on 11 of 17 datasets and  $11.3\%$  (4.2 points) higher average Spearman correlations than SOTA methods, which demonstrates the effectiveness of BCR framework (see Appendix B.3 for the results of Pearson correlations). We further visualize the encoded features and the predicted scores through Principal Component Analysis (PCA). As shown in Figure 6, the transition range (marked with red box) from high score to
low score is very narrow for baseline model while becomes much broader when applying BCR. This feature distribution can bring two benefits: (a) more discriminative features bring more accurate judgments; (b) larger feature space can better resist noise. This sheds light on the reason for the better robustness and correlations brought by BCR from a deeper perspective.
Figure 6: PCA results of baseline model without (left) and with (right) BCR on DSTC10 datasets.
5.2 SOTA TDEMs With BCR
5.2.1 Experimental Setup
We apply BCR on the following two TDEMs that attain best performance in research (Yeh et al., 2021) and test them on DSTC10 datasets.
- GRADE (Huang et al., 2020) possesses a BERT branch to get utterance-level contextualized representations and a Graph-Reasoning branch to get topic-level graph representations.
- USR (Mehri and Eskenazi, 2020b) employs one mask language model and two retrieval models to measure sub-qualities of given sample and combines them into an overall score.
We replace the original loss with DP loss and apply TSR based on their own training set for both of the two models to apply BCR. DailyDialog dataset is used to train all the compared TDEMs for a fair comparison. We also measured the difference in Spearman correlations when testing TDEMs on the DSTC10 datasets with and without noise to evaluate the robustness:
$$
D i f f (\mathcal {M}) = \operatorname {S p e a r m a n} (\mathcal {M}, \text {A d d N o i s e} (\mathbb {D} _ {\text {t e s t}})) - \operatorname {S p e a r m a n} (\mathcal {M}, \mathbb {D} _ {\text {t e s t}}) \tag {10}
$$
Figure 7: Diff and Spearman correlations of SOTA TDEMs on DSTC10 datasets.
where  $\mathbb{D}_{test}$  denotes test dataset,  $\mathcal{M}$  denotes TDEM and we randomly drop  $15\%$  of words in context and replace  $10\%$  of words in both context and query with synonyms to add noise.
5.2.2 Experimental Results
As shown in Figure 7, both GRADE and USR attain higher Spearman correlations and  $Diff$  when applying BCR framework, which verifies that BCR can lead SOTA TDEMs to attain better robustness and correlations with human judgements. This also confirms the stable generalization of BCR framework.
5.3 Ablation Study
We perform ablation studies for the main components of BCR to better analyze their relative contributions. We conduct experiments on the DSTC10 datasets based on BCR+BERT.