SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Training Set Reconstructing. We first verify the effectiveness of TSR. As shown in Table 2, BERT+BCR drops 0.006 and 0.055 Spearman correlations without pseudo label and the whole TSR respectively. From a more fine-grained perspective, baseline model achieves  $79.4\%$  (0.058) Spearman correlations gain on medium coherence samples and  $30.4\%$  (0.038) Spearman correlations gain on polarized coherence samples respectively when applying TSR. This further proves that TSR can simultaneously strengthen the evaluating ability of TDEM on both medium and polarized coherence samples while the former benefits much more.
Table 2: Ablation results (Spearman correlations) of TSR on subsets with different coherence of DSTC10 datasets, with standard deviations in gray. The Medium indicates samples with coherence labels ranking within interval [0.25,0.75] while the Polarized indicates the rest.
Metrics	Polarized	Medium	Overall
BERT+BCR	0.163 ± 0.003	0.131 ± 0.002	0.316 ± 0.002
w/o. Pseudo label	0.161 ± 0.002	0.120 ± 0.001	0.310 ± 0.002
w/o. TSR	0.125 ± 0.003	0.073 ± 0.004	0.261 ± 0.003
DP Loss Function. To verify the effectiveness of DP loss and the impact of different values of  $s$  and  $l$  in Eq. (8), we apply different loss functions to train TDEM respectively. As shown in Figure 8, DP loss generally brings better robustness and Spearman correlations compared with other loss functions when applying different  $(s,l)$  couples. We use Uniformity  $= -\lg(\sigma)$  to present the uniformity of score distribution, where  $\sigma$  is the variance of score distribution. We find that the more uniform the score distribution (greater Uniformity) is, the better robustness TDEM (greater Diff) attains. This verifies the theory we proved in §3.4 from an experimental perspective.
4We use EDA tools nlpaug https://github.com/makcedward/nlpaug to add noise.
Figure 8: Ablation results of loss functions.  $\mathcal{L}_{s - l}$  denotes DP loss applying couple  $(s,l)$ .
Table 4: Comparison of BCR with respect to parameter count, training costs, and computational costs to the compared methods.
Method	BCR+BERT	USR	GRADE	MME-CRS
Parameter Counts (M)	110	373.8	469	>435.2
Training Costs (hours)	2	6	7	>4
Computational Costs	Tbase	3 * Tbase	Tbase + TGNN	5 * Tbase
5.4 Case Study
To illustrate the performance of BCR, two representative examples are shown in Table 3. In the first example, the ranking (cumulative distribution function value of score distribution) given by USR and  $\mathrm{USR + BCR}$  both correlates well with human before adding noise. In case of noise, human judgement stays the same since the meaning of response remains unchanged. However, the ranking given by USR changes sharply (0.83 to 0.99) even though the score barely changes (0.995 to 0.999).  $\mathrm{USR + BCR}$  resists the noise well by converting the polarized distribution to a more uniform distribution, which verifies the better robustness brought by BCR.
Table 3: Two representative examples show the strength of BCR framework. U1 and U2 are two utterances of the conversation history and R is the corresponding response.
U1: Did you look in the mirror?
R: yeah i did.
Score (Human / USR / USR+BCR): 0.444 / 0.995 / 0.744
Ranking (Human / USR / USR+BCR): 0.81 / 0.83 / 0.83
R (add noise): yes i did.
Score (USR / USR+BCR): 0.999 / 0.732
Ranking (USR / USR+BCR): 0.99 / 0.81
U1: Why aren't you eating anything else?
U2: Well, fruits and vegetables are very healthy.
R: What kind of vegetables do you want to eat?
Score (Human / USR / USR+BCR): 0.620 / 0.998 / 0.556
Ranking (Human / USR / USR+BCR): 0.45 / 0.87 / 0.47
The second example shows BCR can bring better correlations with human judgements on medium coherence samples.
5.5 Efficiency Analysis
Table 4 shows the comparison of BCR with respect to parameter count, training costs, and computational costs to the methods we have examined. Specifically,  $T_{base}$  denotes the computational cost for a single pretrained model and  $T_{GNN}$  denotes a typical graph neural network. In the three dimensions of comparison, BCR used fewer resources but still achieved better results.
This paper proposes BCR, a distribution-balanced SSL framework for automatic dialogue evaluation. BCR offers two novel technologies: TSR and DP loss. TSR reconstructs a coherence distribution
balanced training set with continuous label domain. DP loss adjusts adaptively according to the score distribution estimated by kernel density estimation to bring TDEM a more uniform distribution. We prove that the uniformity of score distribution and the robustness of TDEM are positively correlated, which guarantees the better robustness brought by DP loss. Empirical results show that BCR framework brings significant improvements on correlations and robustness to various TDEMs. For future work, we will analyse the pre-training strategies of TDEM to promote the development of dialogue system and LLM training.
**Limitation.** We notice that different TDEMs may produce score disturbance in different scale for a certain small disturbance at the input, which will affect  $\mathbb{E}(\lambda)$  in Section 3.4. Fortunately, this does not affect the conclusion of Theorem 1, and we find that for different models with the same backbone network,  $\mathbb{E}(\lambda)$  barely changes. We suppose that this is alleviated by the regularization effect of weight decay. We also notice that though DP loss can approximate the optimal loss function, a completely uniform distribution of model score on the test set has not yet been achieved, which we believe is due to the distribution difference in coherence between the training set and the test set. We will investigate how to better align the distribution of training and test sets in the future.
This work is supported by Beijing Natural Science Foundation (No. 4222037, L181010). We thank the anonymous reviewers for their constructive comments.
Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. 2020. PLATO: pre-trained dialogue generation model with discrete latent variable. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 85-96. Association for Computational Linguistics.
Jan Deriu, Álvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. 2021. Survey on evaluation methods for dialogue systems. Artif. Intell. Rev., 54(1):755-810.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Alexander R. Fabbri, Wojciech Kryscinski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir R. Radev. 2021. Summeval: Re-evaluating summarization evaluation. Trans. Assoc. Comput. Linguistics, 9:391-409.
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 2214-2220. Association for Computational Linguistics.
Shaoxiong Feng, Xuancheng Ren, Kan Li, and Xu Sun. 2021. Multi-view feature representation for dialogue generation with bidirectional distillation. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 12812-12820. AAAI Press.
Sarik Ghazarian, Ralph M. Weischedel, Aram Galstyan, and Nanyun Peng. 2020. Predictive engagement: An efficient metric for automatic evaluation of open-domain dialogue systems. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7789-7796. AAAI Press.
A Formula Derivation of Theorem 1
$$
\int_ {0} ^ {1} f (x) d x = 1,
$$
we perform formula derivation as follows:
$$
\begin{array}{l} \int_ {0} ^ {1} \left(\int_ {x} ^ {x + \mathbb {E} (\lambda)} f (u) d u\right) ^ {2} f (x) d x \\ = \int_ {0} ^ {1} \left(\int_ {x} ^ {x + \mathbb {E} (\lambda)} f (u) d u \sqrt {f (x)}\right) ^ {2} d x \\ = \int_ {0} ^ {1} \left(\int_ {x} ^ {x + \mathbb {E} (\lambda)} f (u) d u \sqrt {f (x)}\right) ^ {2} d x \\ \cdot \int_ {0} ^ {1} f (x) d x \\ \geq \left(\int_ {0} ^ {1} \int_ {x} ^ {x + \mathbb {E} (\lambda)} f (u) d u f (x) d x\right) ^ {2} \\ (C a u c h y ^ {\prime} s I n e q u a l i t y) \\ = \left(\int_ {0} ^ {1} \mathbb {E} (\lambda) \cdot f (x) \cdot f (x) d x\right) ^ {2} (\mathbb {E} (\lambda) \rightarrow 0) \\ = \mathbb {E} (\lambda) ^ {2} \left(\int_ {0} ^ {1} f (x) \cdot f (x) d x\right) ^ {2} \\ = \mathbb {E} (\lambda) ^ {2} \left(\int_ {0} ^ {1} f (x) ^ {2} d x \cdot \int_ {0} ^ {1} 1 ^ {2} d x\right) ^ {2} \\ \geq \mathbb {E} (\lambda) ^ {2} \left(\left(\int_ {0} ^ {1} f (x) d x\right) ^ {2}\right) ^ {2} \\ (C a u c h y ^ {\prime} s I n e q u a l i t y) \\ = \mathbb {E} (\lambda) ^ {2} \\ \end{array}
$$
The equality condition of the first inequality is:
$$
\begin{array}{l} \int_ {x} ^ {x + \mathbb {E} (\lambda)} f (u) d u \sqrt {f (x)} / \sqrt {f (x)} \equiv C \\ \Longleftrightarrow \int_ {x} ^ {x + \mathbb {E} (\lambda)} f (u) d u \equiv C \\ \Longleftrightarrow \mathbb {E} (\lambda) \cdot f (x) \equiv C \\ f o r x \in [ 0, 1 ], w h e r e C i s a C o n s t a n t \\ \end{array}
$$
Considering that generally  $\mathbb{E}(\lambda)\neq 0$  and
we further derive that  $f(x) \equiv 1$  for  $x \in [0,1]$ .
The equality condition of the second inequality is:
$$
f (x) / 1 \equiv C
$$
$$
\begin{array}{l} f o r x \in [ 0, 1 ], w h e r e C i s a C o n s t a n t \\ \Longrightarrow f (x) \equiv 1 f o r x \in [ 0, 1 ] \\ \end{array}
$$
Thus, the overall equality condition of the main derivation is  $f(x) \equiv 1$  for  $x \in [0,1]$ .
B Detailed Experimental Information
B.1 Detailed Datasets Statistics
The detailed statistics of annotated benchmark datasets we evaluate in the experiments are shown in Table 5 and Table 6. For datasets that contains more than one qualities, we select the quality closest to the overall quality for evaluating. For DSTC10 datasets, we also report the average results on all the 11 qualities as this is the final evaluating indicator of DSTC10 track5 subtask  $1^{5}$ .
Table 5: Quality-annotated datasets used in experiments and the corresponding SOTA methods.
Dataset	SOTA method
FED (Mehri and Eskénazi, 2020a)	DynalEval (Zhang et al., 2021a)
GRADE (Huang et al., 2020)
- ConvAI2-Generator (GCG) - DailyDialog-Generator (GDG)
- Empathetic-Generator (GEG) - ConvAI2-Ranker (GCR)
- DailyDialog-Ranker (GDR) - Empathetic-Ranker (GER)	GRADE
USR (Mehri and Eskénazi, 2020b)
- TopicalChat (UT) - Persona (UP)	USR
DSTC6 (Hori and Hori, 2017)	Deep AM-FM (Zhang et al., 2020a)
Predictive Engagement (PE) (Ghazarian et al., 2020)	USL-H (Phy et al., 2020)
DSTC10 (Zhang et al., 2021b)
- JSALT - ESL - NCM - TopicalChat (DT) - Persona (DP)	MME-CRS (Zhang et al., 2022b)
B.2 Detailed Experimental Setup
We set learning rate as 5e-5, batch size as 32. AdamW Loshchilov and Hutter (2017) is used as the optimizer with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$  and  $\mathrm{eps} = 1\mathrm{e - }6$ . We use linear-scheduled learning rate strategy with decay  $= 0.02$ . We train the model for 40 epochs (except for GRADE and USR) and take the checkpoint of the last epoch.  $\beta$  of DP loss finally converges to 100. All experiments are based on a single NVIDIA GeForce RTX 3090 GPU with 24GB memory. We use the following Python code to calculate Spearman correlations and corresponding p-value: from scipy import stats; spearman;  $p\_value =$  stats.spearmanr(score, label). The p-value here roughly indicates the probability of an uncorrelated system producing datasets that have a Spearman correlation at least as extreme as the one computed from these datasets.
We directly used the loss on the Dailydialog validation set as a criterion for hyperparameter selection except for  $(s,l)$ . As for  $(s,l)$ , we choose (3,7) in our setting as it can bring the highest Uniformity.
B.3 Detailed Experimental Results
Apart from Spearman correlations, we also test TDEMs on Pearson correlations to gain a more comprehensive understanding of BCR framework. As some research have not reported Pearson correlations results of their methods, comparison can only be performed on 10 datasets. As shown in Table 7, BERT-base applying our BCR framework obtains the highest correlations with human judgements on 8 of 10 datasets. This further demonstrates the excellent effectiveness of BCR framework.
<sup>5</sup>https://dstc10.dstc.community/home
Table 6: Detailed statistics of quality-annotated datasets used in experiments. Num. Samples column refers to the number of samples of the corresponding dataset. Num. Qualities column refers to the number of annotated qualities of the corresponding dataset. Eva. Quality column refers to the quality we evaluate in the experiments.
Dataset	Num. Samples	Num. Qualities	Eva. Quality
FED	375	9	Overall
GRADE - ConvAI2-Generator	150	1	Coherence
GRADE - ConvAI2-Ranker	150	1	Coherence
GRADE - DailyDialog-Generator	150	1	Coherence
GRADE - DailyDialog-Ranker	150	1	Coherence
GRADE - Empathetic-Generator	150	1	Coherence
GRADE - Empathetic-Ranker	150	1	Coherence
USR - TopicalChat	300	6	Overall
USR - Persona	240	6	Overall
DSTC6	40000	1	Overall
Predictive Engagement	600	1	Engagement
DSTC10 - JSALT	741	1	Appropriateness
DSTC10 - ESL	1242	1	Appropriateness
DSTC10 - NCM	2461	1	Appropriateness
DSTC10 - TopicalChat	4500	4	Appropriateness
DSTC10 - Persona	5000	4	Appropriateness
Table 7: Pearson correlations between TDEMs and human judgements on benchmark datasets. BERT refers to our baseline model. The results of SOTA TDEMs come from the corresponding papers. All results are statistically significant (p-value  $>0.05$ ).
Method	FED	GCG	GCR	GDG	GDR	GEG	GER	PE	DSTC6
BERT	0.205	0.616	0.491	0.260	0.261	0.167	0.323	0.612	0.234
SOTA	-	0.606	0.535	0.368	0.261	0.257	0.375	0.688	0.326
BERT+BCR	0.307	0.636	0.510	0.377	0.444	0.348	0.399	0.666	0.273
Method	UT	UP	JSALT	ESL	NCM	DT	DP	DSTC10	Average
BERT	0.339	0.405	0.171	0.403	0.246	0.297	0.412	0.281	0.336
SOTA	0.422	0.411	-	-	-	-	-	-	-
BERT+BCR	0.437	0.452	0.266	0.519	0.323	0.341	0.428	0.318	0.414
B.4 Human Evaluation of Coherence Comparison
We conduct human evaluation to examine the coherence level of samples generated by TSR. Compared data augmentation methods include: (a) GRADE (Huang et al., 2020) applies both lexical sampling and embedding-based sampling to obtain negative samples; (b) PONE Lan et al. (2020) applies EDA (Wei and Zou, 2019) and generative dialogue model to obtain positive samples. For a given conversation, we use the above two data augmentation methods and medium coherence samples generating strategies (positive and negative) to obtain four samples respectively and ask five human annotators to sort them based on the response coherence. We choose researchers in the field of dialogue as annotators because this work requires certain professional quality. The experiment is conducted on 100 conversations of DailyDialog dataset. We calculate the average ranking of each type of the above methods. The average Fleiss's kappa of annotators is 0.63, indicating annotators have reached relatively strong agreement. The result of coherence ranking is PONE-positive  $(1.3)>$
TSR-positive  $(1.8) >$  TSR-negative  $(3.3) >$  GRADE-negative  $(3.6)$ , indicating that TSR can provide stronger training signals with medium coherence compared with general data augmentation methods.
C.1 Impact of Training Dataset
We also evaluate the generalization of BCR on different training datasets. We choose DailyDialog (Li et al., 2017), Empathetic (Rashkin et al., 2019), Topical (Gopalakrishnan et al., 2019) as training datasets and test them on DSTC10 dataset respectively. As shown in Table 8, BCR framework brings significant progress to BERT in all settings, which further demonstrates the strong generalization of BCR. We also notice that TDEM trained on DailyDialog performs better than other datasets, which explains why most TDEMs choose this dataset as training set.
C.2 BCR for Pre-training
In the future, supervised TDEM will probably achieve higher consistency with human judgements. But self-supervised TDEM can still serve as pre-trained model to help supervised TDEM converge faster to better performance under limited labeled data volume conditions. Thus, we compared the performance of supervised TDEM with TDEM that first applied BCR pre-training and then underwent supervised fine-tuning. Specifically, we split FED dataset into train set and test set in a ratio of 7:3 and the train set is served for supervised learning. We finetuned vanilla BERT and BERT-BCR trained on DailyDialog on the labeled train set and test them respectively. Supervised TDEM converged on the  $5^{th}$  epochs and attained 0.240 Spearman correlations while TDEM underwent both BCR pre-training and supervised finetuning converged on the  $3^{th}$  epochs and attains 0.372 Spearman correlations. This preliminary experiment may inspire scholars to research and apply pre-trained TDEMs to improve the performance of reward models, thus attaining better LLMs.
C.3 Comparing with GPT4
Recently, some automatic evaluation methods based on LLMs (e.g., GPT-4) (Liu et al.; Wang et al., 2023) have been proposed to explore new paradigms for addressing TDEM. We have compared BCR with Liu et al. in Table 9. The results of Liu et al. is obtained with GPT-4 (version: May 15, 2023). Even in the case of  $n = 20$  (self-consistency), BCR+BERT is still able to achieve better average results than G-EVAL. LLM based methods also have the following drawbacks: higher expenses; more time costs; much more computational cost and parameters; bias towards specific content. Therefore, how to effectively combine BCR and LLM is one of our future research directions.
C.4 Transferability to Other Tasks
We further discuss the possibility of migrating BCR to other self-supervised regression tasks. BCR can be transferred to other tasks, such as Factuality (Falke et al., 2019; Wang et al., 2020), Data to Text (Mairesse et al., 2010), Text Summarization (Fabbri et al., 2021), etc.
We take Text Summarization task for an example. As for TSR, in the first step, we can obtain positive samples with medium coherence by randomly deleting or replacing sentences in the text part of the positive samples; we can also obtain negative samples with medium coherence by randomly deleting or replacing sentences in the summarization part of the positive samples. Based on this, TSR can be implemented according to Eq. (5). As for DP loss, we directly use Eq. (8) to train BCR. It turns out that BERT with DP loss can obtain 0.243 Spearman correlations and -0.013 Diff, both better than Triplet loss (0.216 Spearman correlations and -0.030 Diff) and MSE loss (0.205 Spearman correlations and -0.039 Diff).
D Further Discussion
In the task of automatic dialogue evaluation, the mainstream approach is to utilize the SSL framework. This is because constructing a manually annotated training dataset for automatic dialogue evaluation is highly challenging. For instance, we once attempted to annotate a evaluation dataset for TDEM,
Table 8: Spearman correlations of BERT and BERT+BCR trained on different datasets.
Training dataset	Method	JSALT	ESL	NCM	DT	DP	DSTC10
DailyDialog	BERT	0.166	0.345	0.253	0.220	0.401	0.255
BERT+BCR	0.272	0.531	0.336	0.305	0.428	0.316
Empathetic	BERT	0.149	0.301	0.129	0.140	0.271	0.157
BERT+BCR	0.215	0.375	0.161	0.166	0.304	0.191
Topical	BERT	0.082	0.201	0.113	0.197	0.277	0.152
BERT+BCR	0.124	0.240	0.135	0.253	0.309	0.194
Table 9: Spearman correlations of BCR and G-EVAL on GRADE datasets. n represents GPT4 generating n scores for each sample and taking the average (self-consistency).
Method	GDG	GDR	GCG	GCR	GEG	GER	Avg.
G-EVAL n=1	0.427	0.422	0.501	0.464	0.267	0.370	0.408
G-EVAL n=20	0.448	0.448	0.513	0.559	0.296	0.398	0.443
BERT+BCR	0.416	0.487	0.642	0.519	0.341	0.372	0.463
consisting of 600 instances. Each instance comprises a context and multiple responses. Annotators were tasked with assigning scores to these responses individually. Each instance was annotated by 5 individuals to reduce noise. Given the longer dialogue contexts, it took approximately 2 minutes for a single annotator to complete each instance. After data cleaning (removing samples with low internal consistency), approximately 500 instances remained. As a result, annotating these 500 instances required a total of around 100 hours of annotation time. Based on this estimation, if we were to annotate a training set of the size of DailyDialog (13118 samples), we would need approximately 2623 hours of annotation time. Even so, this still does not guarantee that such a training set would exhibit strong generalization across multiple dialogue domains. The requirement for high inter-annotator consistency and the characteristic of longer dialogue contexts significantly elevate the annotation cost of dialogue evaluation training sets. We believe that achieving better evaluation performance requires a self-supervised (BCR) pre-training approach to impart the evaluation model with generalized assessment capabilities. Subsequently, fine-tuning can be carried out with limited annotated training data specific to the data domain. Thus, we look forward to the emergence of high-quality dialogue evaluation training sets in the future.