Toward Human-Like Evaluation for Natural Language Generation with Error Analysis
Year: 2023

Abstract:
The pretrained language model (PLM) based metrics have been successfully used in evaluating language generation tasks. Recent studies of the human evaluation community show that considering both major errors (e.g. mistranslated tokens) and minor errors (e.g. imperfections in fluency) can produce high-quality judgments. This inspires us to approach the final goal of the automatic metrics (human-like evaluations) by fine-grained error analysis. In this paper, we argue that the ability to estimate sentence confidence is the tip of the iceberg for PLM-based metrics. And it can be used to refine the generated sentence toward higher confidence and more reference-grounded, where the costs of refining and approaching reference are used to determine the major and minor errors, respectively. To this end, we take BARTScore as the testbed and present an innovative solution to marry the unexploited sentence refining capacity of BARTScore and human-like error analysis, where the final score consists of both the evaluations of major and minor errors. Experiments show that our solution consistently improves BARTScore, outperforming top-scoring metrics in 19/25 test settings. Analyses demonstrate our method robustly and efficiently approaches human-like evaluations, enjoying better interpretability. Our code and scripts will be publicly released in https: //github.com/Coldmist-Lu/ ErrorAnalysis_NLGEvaluation.

heading: 1 Introduction
text: Leveraging the power of large pre-trained language models (PLMs) has been proven effective in evaluating natural language generation (NLG) tasks (Ma et al., 2019; Mathur et al., 2020b). Metrics like BERTScore (Zhang et al., 2020b) and Mover-
∗Work was done when Qingyu was interning at JD Explore Academy. Qingyu and Liang contributed equally.
†Corresponding Author.
in each iteration are highlighted in yellow, and their corresponding scores are in red.
score (Zhao et al., 2019) leverage contextual embeddings provided by PLMs to evaluate the semantic similarity of sentences. Regression-based metrics like COMET (Rei et al., 2020) and BLEURT (Sellam et al., 2020) introduce a regression layer following PLMs to learn a supervised prediction using human evaluation. Recently, another line of research focuses on generation probabilities of seq2seq PLMs to measure the confidence of generated texts, such as PRISM (Thompson and Post, 2020) and BARTScore (Yuan et al., 2021), achieving the decent performance. It is commonly agreed that the ultimate goal of automatic evaluation is to achieve consistency with humans, namely humanlike evaluation.
Recent studies of the human evaluation community show that the quality of human judgments can be improved through fine-grained error analysis, incorporated in an error-based framework Multidimensional Quality Metric (MQM) (Freitag et al., 2021a). MQM requires evaluators to identify errors and categorize them into different levels according to their severity. For instance, mistranslations (Weng et al., 2020) and hallucinations (Zhou et al., 2021) are mostly considered as Major errors, and imperfections in fluency (Chow et al., 2019) are often marked as Minor errors. Different weights
5892
are then assigned to Major/ Minor errors, resulting in high-quality human evaluation scoring.
Analogous to Major/ Minor errors in MQM, we take the first step to consider incorporating the evaluation of Explicit/ Implicit errors into PLMsbased metrics. Specifically, we use BARTScore, a state-of-the-art metric for NLG by Yuan et al. (2021) as the test bed, and propose a metric called BARTScore++. We present an overview of our proposed method in Figure 1. In particular, given the hypothesis and reference, we propose an error analysis framework to obtain a refined sentence (see example in Table 1) using BARTScore, where the costs of refining and approaching reference are used to determine the explicit and implicit errors, respectively. The weighted integration of these two types of errors is the final score of BARTScore++, which has better interpretability.
We experiment on machine translation (MT), text summarization (SUM), and data-to-text (D2T), and show that BARTScore++ consistently and significantly improves the performance of vanilla BARTScore, and surpasses existing top-scoring metrics in 19 out of 25 test settings, even exceeding human performance on summarization dataset Rank19. We give further analyses to confirm that the consistent improvements come from the humanlike (specifically, MQM-like) error judgment.
Our main contributions are as follows:
• To the best of our knowledge, we take the first step toward human-like evaluation by incorporating error analysis mechanisms into existing advanced automatic metrics, e.g. BARTScore.
• We propose an innovative automatic error analysis framework to calculate the explicit error and implicit error-based scores, by refining sentences using BARTScore.
• We validate the effectiveness and universality of our method spanning 25 NLG evaluation tasks, achieving the SOTA in 19 settings.
Besides taking BARTScore as the testbed to verify the effectiveness of our proposed error-analysis evaluation strategy, we also show the universality in the recently advanced language model ChatGPT1 by designing an error-analysis-based prompt (Lu et al., 2023). We anticipate that our strategy will
1https://chat.openai.com/
shed new light on advancing the field of NLG evaluation with pretrained language models by enhancing both the accuracy and reliability of metrics.

heading: 2 Preliminaries
text: Problem Formulation The goal of NLG evaluation is to acquire a score measuring the quality of generated text y given a reference signal r. Unless otherwise stated, r represents the sentence properly created by human experts to assist in evaluation, and y = (y1y2 . . . yN ), called hypothesis in this paper, refers to the generated text to be evaluated2.
BARTScore BARTScore is a SOTA metric proposed by Yuan et al. (2021) for universal NLG evaluation. The idea of BARTScore is to utilize the generation probabilities of a large pre-trained model BART (Lewis et al., 2020) to measure the quality of sentences. It autoregressively computes the log probabilities of each token in the hypothesis, and then averages them as the overall score. This evaluation process can be formally written as:
BARTScore = 1
N
N∑
t=1
log pθ (yt|y<t, r)
Based on this formulation, BARTScore creates specific variants for different evaluation scenarios. We summarize their usage in Appendix A. For simplification, we use the notation of BARTS(y, r) when vanilla BARTScore is further applied.
MQM MQM is an error-based human evaluation framework, which is commonly agreed to be more reliable than traditional human evaluation techniques (Freitag et al., 2021b). In MQM framework, each evaluator is asked to identify all errors in a sentence and categorize them into Major and Minor levels indicating their severities. Sentences will be marked an Non-translation Error if they are not possible to reliably identify errors. Major/ Minor errors are then assigned with different weights, and the final MQM score is computed through the weighted sum of errors (Freitag et al., 2021a). Inspired by the mechanism of MQM, we take a step toward human-like evaluation by incorporating error analysis into BARTScore.

heading: 3 Methodology
text: To better understand how BARTScore++ works, we show a running example of our method in Figure 1.
2Note that in text summarization evaluations, BARTScore may use the source sentence as the reference signal.

heading: 3.1 Explicit/ Implicit Error Distance
text: Analogous to major errors in MQM, we define Explicit Errors to refer to errors that can be easily identified. In our example, mistranslations of name ("Mike" → "Jerry") and addition of "happily" are considered as explicit errors. Analogous to minor errors, we define Implicit Errors to indicate the semantic imperfections (e.g. disfluency, awkwardness) that may not influence the overall meanings. In our example, the missing article "the" is considered as an implicit error because it is a smaller imperfection in grammar.
To measure the influence of Explicit/ Implicit errors in the hypothesis y, we define Refined Sentence y∗ as a better hypothesis, where explicit errors are corrected. In this way, distances of explicit/ implicit error can be computed by:
Distexp = BARTS(y∗, r)− BARTS(y, r) Distimp = BARTS(r, r)− BARTS(y∗, r)
We then focus on how to 1) obtain the refined sentence y∗ and 2) take both explicit/ implicit errors into consideration and obtain the final score.

heading: 3.2 Error Analysis Framework
text: We introduce an automatic error analysis framework to generate the refined sentence y∗ by cor-
recting explicit errors in the hypothesis y. We first adopt a simple non-translation test to decide whether y will be refined or not. Then, a detectcorrect algorithm is performed iteratively, in each round one token is detected and then corrected. An example of this is shown in Table 1. This algorithm repeats for a determined number of iterations T , where at the end of each round the refined sentence y∗ is updated and becomes a new refining target. In our example, the hypothesis y is refined twice, where the mistranslated token "Jerry" is detected in Round 1 and corrected as "Mike", and the addition of "happily" is detected and deleted in Round 2. Afterwards, an extra round will run (omitted in table) to ensure that none of the tokens needs to be corrected. Finally, the hypothesis "Mike goes to bookstore." is taken as the refined sentence y∗.
Test Non-Translation Error Non-Translation Error is used in MQM (Freitag et al., 2021a) to refer to the translation which is too badly garbled or is unrelated to the source. If the hypotheses contain severe problems such as off-target issues (Zhang et al., 2020a), directly refining them will consume excessive computational cost. To avoid this problem, we run a test beforehand as a rough measure to filter out these hypotheses with low quality. We consider two strategies:
1. Token-level overlap ratio w.r.t the reference. Inspired by string-based metrics like BLEU (Papineni et al., 2002) or TER (Snover et al., 2006), the hypothesis with a non-translation error may be quite different from its reference, resulting in a low overlap ratio. Since good translations like paraphrased sentences (Freitag et al., 2020) may not have significant overlap with the reference, we adopt the other strategy as a double-check.
2. Percentage of tokens with low generation probability. Token-level log generation probability can be directly obtained from vanilla BARTScore as log pθ (yt|y<t, r). If most tokens’ generation probabilities are lower than the average score (vanilla BARTScore), we mark this sentence as non-translation. This strategy is more stable but less efficient.
Detect In this step, we choose one token ŷt with the lowest generation probability as the token to be corrected. This procedure can be denoted as:
ŷt = argmin yt
{ pθ (yt|y<t, r) }
Correct In this step, we leverage the distribution of generation pθ (·|y<t, r) to propose several refining options from vocabulary V . We apply the top-k sampling method (Fan et al., 2018) to obtain a set of candidate tokens (W) with the highest generation probability:
W = argmax w∈V
{ pθ (w|y<t, r) , k }
Then, a set of refined sentences S is proposed. Following Snover et al. (2006), we apply three types of editing strategies, including insertion of a candidate token w ∈ W , deletion of token ŷt, and substitution of ŷt for a candidate token w ∈ W . Finally, we use vanilla BARTScore to select the best sentence ŷ∗ as the refining strategy:
ŷ∗ = argmax ŷ∈S BARTS (ŷ, r) ,
where the hypothesis y will be temporarily replaced by ŷ∗ and as the input for the next iteration.
This detect-correct algorithm repeatedly detects the worst token ŷt and corrects it. It starts with the original hypothesis y and ends after a constant number of edits. We set an early-stop mechanism once the BARTScore performance stops improving.
In this way, we obtain the refined sentence y∗, which is also a by-product of our method.

heading: 3.3 Assigning Error Weights
text: With the help of the error analysis framework, explicit errors in the hypothesis are refined, resulting in a refined sentence y∗. We simply use a weighted sum method to achieve the final score:
BARTScore++ = −(Distexpωexp + Distimpωimp), where ωexp + ωimp = 1, 0 ≤ ωexp, ωimp ≤ 1
ωexp and ωimp weigh the importance of explicit and implicit errors respectively.3 For easy to use, we define λ = ωexp/ωimp as the only parameter, indicating the ratio of weights assigned to Explicit/ Implicit errors, where ωexp = λ1+λ , ωimp = 1 1+λ respectively. Since λ may be different from task to task, we perform specific analysis in §6, confirming the stability when adjusting this parameter. We also provide guidance on selecting λ in Appendix B to help researchers use BARTScore++ for different tasks.

heading: 4 Experiment Setup

heading: 4.1 Tasks and Datasets
text: Tasks We follow Yuan et al. (2021) to consider three different tasks: summarization (SUM), machine translation (MT), and data-to-text (D2T).
Datasets for Translation We obtain the machinetranslated texts and reference texts from the WMT20 metrics shared task (Mathur et al., 2020b). We use the DARR corpus and consider 10 language pairs, which are cs-en, de-en, ja-en, ru-en, zh-en, iu-en, km-en, pl-en, ps-en, and ta-en. We also consider Multidimensional Quality Metric (MQM) for zh-en provided by Freitag et al. (2021a) in §6, comprising judgments of 8 best-performing translation systems in WMT20, annotated by professional translators.
Datasets for Summarization (1) REALSumm (Bhandari et al., 2020) is a meta-evaluation dataset for text summarization which measures pyramidrecall of each system-generated summary. (2) SummEval (Fabbri et al., 2021) is a collection of human judgments of model-generated summaries on the CNNDM dataset annotated by both expert judges and crowd-source workers. Each systemgenerated summary is gauged through the lens of coherence, factuality, fluency, and informativeness.
3Following the same pattern as in Yuan et al. (2021), we reverse the score to ensure BARTScore++ ranging from −∞ to 0, with a higher score being a better quality of the sentence.
(3) NeR18 The NEWSROOM dataset (Grusky et al., 2018) contains 60 articles with summaries generated by 7 different methods are annotated with human scores in terms of coherence, fluency, informativeness, relevance.
Datasets for Factuality (1) Rank19 (Falke et al., 2019) is used to meta-evaluate factuality metrics. It is a collection of 373 triples of a source sentence with two summary sentences, one correct and one incorrect. (2) QAGS20 (Wang et al., 2020) collected 235 test outputs on CNNDM dataset from Gehrmann et al. (2018) and 239 test outputs on XSUM dataset (Narayan et al., 2018) from BART fine-tuned on XSUM. Each summary sentence is annotated with correctness scores w.r.t. factuality.
Datasets for Data-to-Text We consider the following datasets which target utterance generation for spoken dialogue systems. (1) BAGEL (Mairesse et al., 2010) provides information about restaurants. (2) SFHOT (Wen et al., 2015) provides information about hotels in San Francisco. (3) SFRES (Wen et al., 2015) provides information about restaurants in San Francisco. They contain 202, 398, and 581 samples respectively, each sample consists of one meaning representation, multiple references, and utterances generated by different systems.

heading: 4.2 Baselines and Meta-evaluation
text: Baselines We compare our method with several commonly used baseline metrics for evaluating text
generation, including BLEU (Papineni et al., 2002), BERTScore (Zhang et al., 2020b), MoverScore (Zhao et al., 2019) and PRISM (Thompson and Post, 2020). For MT tasks, we also consider supervised metrics that leverage human judgments to train, including COMET (Rei et al., 2020) and BLEURT (Sellam et al., 2020). For factuality evaluation on summarization tasks, we compare BARTScore++ with the best-performing factuality metrics FactCC (Kryscinski et al., 2020) and QAGS (Wang et al., 2020). We reproduce BARTScore and its variants using their official codes4.
Meta-evaluation We follow Yuan et al. (2021) to conduct the meta-evaluation. Specifically, we apply Kendall’s τ for MT tasks to measure the correlation of metrics with human evaluation5. For SUM and D2T tasks, we use Spearman correlation except for the Rank19 dataset, where Accuracy is used to measure the percentage of correct ranking between factual texts and non-factual texts. We adopt the paired bootstrap resampling method (Koehn, 2004) (p-value < 0.05) for significance tests.

heading: 4.3 Setup
text: As for the backbone BART, we use the same settings in BARTScore (Yuan et al., 2021) for specific
4https://github.com/neulab/BARTScore 5Since the meta-evaluation method is very sensitive to outliers (systems whose scores are far away from the rest of the systems) (Mathur et al., 2020a), we remove these outlier systems when computing correlations.
tasks, including BART-large, BART-CNN (finetuned on CNNDM) and BART-CNN-PARA (further fine-tuned on ParaBank2). We perform the same prompting strategy as in BARTScore (Yuan et al., 2021). Detailed settings are in Appendix A. In correct stage of error analysis, we set k = 10 when applying the top-k sampling, namely, a total of 10 tokens are obtained in W during each iteration.

heading: 5 Experimental Results
text: Machine Translation Table 2 shows segmentlevel Kendall τ correlation of metrics on WMT20. We can observe that BARTScore++ can achieve state-of-the-art performance on all language pairs (most significantly outperform vanilla BARTScore except ps-en). The average correlation of BARTScore++ can surpass all supervised and unsupervised metrics by a large margin in both high-resource and low-resource scenarios (except ps-en). This confirms our intuition that with analysis of explicit/ implicit errors, BARTScore++ will agree more with human evaluations compared with vanilla BARTScore.
Regarding the prompting strategy, we also observe that 1) our proposed error analysis mechanism in BARTScore++ can achieve a similar amount of correlation improvement as that of prompting, and 2) incorporating both prompting and error analysis can further push SOTA results, confirming the orthogonality of error analysis and prompting strategies upon BARTScore.
Text Summarization Results on REALSumm, SummEval and NeR18 are showed in Table 3. We observe that: 1) BARTScore++ surpasses all other metrics including BARTScore variants for all test settings except REALSumm. In most aspects, our purposed method can significantly outperform baseline metrics, and especially in NeR18, BARTScore++ can even significantly improve the performance of vanilla BARTScore. This further confirms the robustness (Rony et al., 2022) of our proposed metric. 2) Compared with prompting, er-
ror analysis mechanism in BARTScore++ on summarization tasks can also achieve a similar amount of correlation improvement, which again testify the importance of considering errors in summarization evaluation.
Analysis on factuality datasets As shown in Table 4, we also observe that BARTScore++ significantly outperforms other metrics on all three datasets. Strikingly, BARTScore++ can even surpass human baseline on Rank19. While prompting is not working in these tasks, error analysis mechanism corporated in BARTScore++ can also show significant improvement. This suggests that BARTScore++ is more effective in detecting the hallucination content and yielding more distinguishable scores in factuality summaries, which further confirms the universality of our proposed method.
Data-to-Text Results on data-to-text are shown in Table 5. We see that BARTScore++ can again surpass existing methods and significantly outperform vanilla BARTScore. We further find weights on explicit errors are consistently larger than implicit errors, interestingly suggesting we should focus more on explicit errors for data-to-text tasks.

heading: 6 Analysis
text: To better understand the mechanism by which BARTScore++ achieves promising results, we take a closer look and answer four questions:
Q1: How reliable is our BARTScore++ when evaluating top-performing systems?
Q2: How do explicit/ implicit error weights influence the accuracy of BARTScore++?
Q3: How does error analysis make BARTScore++ more human-like?
Q4: Does error analysis framework introduce significant latency?
For MT evaluation in this section, we use MQM, an error-based evaluation framework annotated by human experts (Freitag et al., 2021a). For a fair comparison, the error weight ratio λ for WMT20 zh-en test set is fixed to 1.7.
BARTScore++ is Reliable When Evaluating TopK Systems Previous studies have shown that most metrics are unreliable for evaluating bestperforming systems, showing a sharp degradation of correlation with human evaluation (Mathur et al., 2020a). To answer Q1, we assess our method shown in Figure 3 with several baseline metrics on Top-K MT systems by computing Kendall’s τ respectively. As seen, BARTScore++ can further improve BARTScore’s performance, especially when evaluating top-performing systems (K < 6). This verifies the reliability of our purposed method.
BARTScore++ is Stable When Adjusting Error Weights To answer Q2, we present an analysis on adjusting the error weight ratio λ in BARTScore++, which is the only parameter that needs to consider before evaluation. In Figure 2, as the number of systems K decreases, the ratio of error weights according to the best-performing BARTScore++ is fluctuating from 1 to 1.7. This suggests that different weights of importance should be given to explicit errors according to the overall qualities of MT systems. We also provide guidance on selecting this parameter in Appendix B to help researchers apply BARTScore++ to different task settings.
BARTScore++ is More Human-Like on Discriminating Errors To answer Q3, we perform a human analysis and show some cases in Appendix C to further show the advantage of our error analysis strategies incorporated in BARTScore++. In Ta-
ble 9, we can see that human evaluators consistently assign low MQM scores to explicit errors (e.g. mistranslation of "delivery" in WeChat AI in example 1, mistranslation of "disc" in Tencent Translation in example 3), but BARTScore produces contrary judgments, ignoring these errors that should be punished strictly. Through our proposed error analysis, BARTScore++ becomes more discriminative on explicit errors and reaches an agreement with human judgments, while BARTScore fails to such errors. To better quantify such discriminative property, we report the sensitivity of our method on major errors using a perturbation dataset DEMETR6 (Karpinska et al., 2022) in Figure 4, where BARTScore++ shows consistent boosts, confirming our claim.
BARTScore++ Brings Acceptable Latency A possible concern is the evaluation efficiency for BARTScore++, since top-k sampling and iterative inferences in error analysis inevitably introduce more complexity. We compare the latency between vanilla and ours on Nvidia A100 GPU with identical batchsize. As seen in Figure 5, 1) although increasing the k in sampling brings better performance, it inevitably increases the iterations and inference cost, and 2) well-performed BARTScore is used combining the Prompt strategy, which naturally owns high latency, i.e. 0.91 seconds per sentence, which is actually on the same order of magnitude as ours, i.e. 1.33∼2.17. Considering both the significant performance boosts and comparable latency, we believe the increased costs are totally acceptable.

heading: 7 Related Work
text: Automatic Metrics Automatic Evaluation Metrics are of crucial importance to the development of NLG systems, including translation (Koehn and Knowles, 2017; Ding et al., 2021; Zan et al., 2022b; Peng et al., 2023; He et al., 2023), summarization (Zhong et al., 2022b; Zan et al., 2022a), grammar error correction (Wu et al., 2023; Liu et al., 2021), dialogue generation (Li et al., 2017; Cao et al., 2021). Recent research has shown great success in language model-based metrics (Zhang et al., 2020b; Marie et al., 2021; Zhou et al., 2020; Rei et al., 2020; Sellam et al., 2020), which can significantly outperform traditional string-based metrics such as BLEU (Papineni et al., 2002). For example, BERTScore (Zhang et al., 2020b) and MoverScore
6Details of DEMETR analysis are shown in Appendix D.
(Zhao et al., 2019) leverage contextual embeddings to measure semantic distance between reference and hypothesis. COMET (Rei et al., 2020) and BLEURT (Sellam et al., 2020) rely on human evaluations to train. UniEval (Zhong et al., 2022a) reframes NLG evlauation into a Question Answering task and allows the metric to focus on different aspects. In this paper, we choose BARTScore (Yuan et al., 2021) as the testbed because of its SOTA performance and universality on NLG tasks. Note that our error analysis strategies can also be extended to other metrics, such as PRISM (Thompson and Post, 2020).
Human Evaluation Human evaluation, such as Direct Assessment (Graham et al., 2017), are often served as "golden standard". However, there is increasing evidence that inadequate evaluation will lead to wrong decisions (Toral, 2020). This motivates elaborate evaluation proposals (Popović, 2020; Gladkoff and Han, 2021) and MQM is one of these methodologies, grounded in explicit error analysis (Freitag et al., 2021a). In this work, We extend error analysis strategies to BARTScore, making it trigger more human-like judgments.
Error Analysis Existing automatic metrics tend to simplify the error detection procedure, such as edit distance in TER (Snover et al., 2006) and mismatch in BERTScore (Zhang et al., 2020b). To incorporate errors into automatic evaluation, recent research (Xu et al., 2022) simulates different errors and assigns scores like MQM as the training data to finetune a model-based metric. However, it does not address the issue of metrics lacking interpretability. In this work, we leverage the token-level judgments in BARTScore and analyze explicit errors through error analysis, making metrics more human-like, and providing more accurate evaluations. Our error analysis framework functionalizes like token-level quality estimation (Specia et al., 2021) or automatic post-editing (Freitag et al., 2019). With the reference signal provided, our proposed method is more accurate and universal for NLG evaluation.

heading: 8 Conclusion
text: We present an automatic metric BARTScore++ for NLG evaluation. Inspired by the advanced human evaluation MQM, BARTScore++ incorporates error analysis strategies to give a comprehensive score considering explicit and implicit
errors. Experimental results show our approach achieves competitive results on a broad range of tasks. Our work is an early step toward humanlike evaluation for automatic metrics, and we hope our BARTScore++ can motivate researchers working on NLG evaluation to focus more on human evaluation procedures such as error analysis.
Limitations
Limitations of BARTScore++ are three-fold:
• In §3.1, we propose Explicit/ Implicit errors to better distinguish different types of errors in generated texts. However, explicit errors only contain token-level errors that can be detected and corrected by error analysis, not involving all error types mentioned in MQM (e.g. severe fluency errors). We hope future studies can take these situations into account.
• In §3.2 we can see that our proposed error analysis framework fully relies on the generation probabilities of BART to decide how to refine the hypothesis. Still, we see that this framework may lead to false judgments due to unfaithful content. Further research can explore how to calibrate the pre-trained models during error analysis.
• In §3.3 we integrate the distance of explicit and implicit errors by simply computing their weighted sum. This can be improved by considering more factors, e.g. the overall quality of the generated text, refining iterations, and external signals. We will leave the exploration of combining these factors and designing better weighting schemes as future work.
Ethics Statement
We take ethical considerations very seriously, and strictly adhere to the ACL Ethics Policy. All procedures performed in this study are in accordance with the ethical standards. This paper focuses on improving automatic NLG evaluations with an error analysis framework. Our proposed metric relies on reference translations as signals and produces scores for translations indicating their quality. Both the datasets and models used in this paper are publicly available and have been widely adopted by researchers. Our model will not learn from user inputs or cause potential risks to the NLP community. We ensure that the findings and conclusions of
this paper are reported accurately and objectively. Informed consent was obtained from all individual participants included in this study.

heading: Acknowledgement
text: We are grateful to the anonymous reviewers and the area chair for their insightful comments and suggestions. This work was supported in part by the National Natural Science Foundation of China under Grant 61973083, in part by the Natural Science Foundation of Shenzhen under Grant JCYJ20210324121213036. Derek F. Wong was supported in part by the Science and Technology Development Fund, Macau SAR (Grant Nos. FDCT/060/2022/AFJ, FDCT/0070/2022/AMJ) and the Multi-year Research Grant from the University of Macau (Grant No. MYRG2020-00054-FST).

heading: B Guidance on Selecting Error Weights Ratio λ
text: Since error weights ratio λ is the only parameter that may differ from task to task, so we provide two suggestions on selecting it:
• Inspired by the idea of the Calibration Set from Licht et al. (2022), we suggest creating a relatively smaller test set and then collecting human evaluations on them. The test size should include over 100 samples covering various ranges of translation quality. To ensure the reliability of human evaluations, we recommended recruiting 2 to 3 professional evaluators to label the Calibration Set according to the MQM annotating procedure (Freitag et al., 2021a). Choose the error weights ratio relating to the highest consistency with human judgments.
• When evaluating the datasets mentioned in this paper, we provide settings of λ in Table 7 in BARTScore++ for researchers to apply directly.

heading: C Case Study
text: We show four evaluation examples of machine translation in Table 9 to further explain how error analysis makes BARTScore++ more humanlike. These examples are from WMT20 test set on three best-performing systems, Huoshan Translation, WeChat AI, and Tencent Translation. For all examples, judgments of BARTScore++ are agree with MQM (marked in Better and Worse), but contrary to vanilla BARTScore.
Example 1 The worse hypothesis generated by WeChat AI translates "投运" into "delivery" (highlighted in yellow). However, vanilla BARTScore seems to "ignore" this error and give a higher score than the better translation from Huoshan Translation. BARTScore++ applies an error analysis and gives a more discriminative evaluation by revising this word to "opening". In this way, Distexp are enlarged by a larger error weight (0.000 -> 0.348), resulting in an agreement with human judgment.
Example 2 WeChat AI produces a major error when translating "更缺" into "even more". This error is detected through the error analysis mechanism and the mistranslation word "more" is deleted for its awkward style. Such deletion helps
BARTScore++ to better distinguish the quality between these two sentence.
Example 3 Although vanilla BARTScore gives similar scores to both translations, their MQM scores are significantly different (11.333 vs 6.333), mainly because of the translation on "umbilical cord tray". Tencent Translation mistranslates it into "disc", which is detected and corrected through error analysis, leading to a relatively low score for BARTScore++. This example also shows that error analysis can help metrics better evaluate long sentences.
Example 4 Huoshan Translate produces a mistranslation error "recognized" when translating the verb "承认". We can see that such error is detected and revised to "admitted", resulting in a relatively large explicit distance (0.000 compared with 0.258), confirming that BARTScore++ can better distinguish major errors and become more human-
like.

heading: D Sensitivity analysis on BARTScore++ using DEMETR
text: To better quantify the sensitivity of BARTScore++ on different kinds of explicit errors, we utilize a metric diagnosing dataset, DEMETR (Karpinska et al., 2022), perturbing on 1000 test samples with different types of errors. We use the ratio proposed in DEMETR to measure the sensitivity of a metric, denoted as:
z = SCORE(r,h)− SCORE(r,h′)
SCORE(r,h)− SCORE(r, [empty]) where r,h,h′ and [empty] represent the reference, hypothesis, the perturbed hypothesis and empty string respectively. We calculate this ratio for each test sample and average them as the sensitivity for each error type. Figure 4 shows the sensitivity of BARTScore++ and BARTScore on different types of errors. We can see that: Compared with vanilla BARTScore, BARTScore++ is consistently more sensitive to major errors, confirming our claim.
E Influence of Different References when using BARTScore++
One potential concern is that the evaluation of BARTScore++ may heavily rely on the reference, which could make this metric less robust compared to the original BARTScore when switching to a different reference. We compare the performance of BARTScore and BARTScore++ on the top-5 systems from WMT20 zh-en, using two different references labeled as Ref.A and Ref.B. The results are presented in Table 8.
As seen, BARTScore++ is not significantly affected by the choice of reference, as consistent improvements observed (+0.21/+0.17). However, the performance of vanilla BARTScore appears to be less robust than BARTScore++ (-0.60 from Ref.B to Ref.A). This further validates the effectiveness and robustness of our method on different references.
Example 1: #239
Source 9月25日，北京大兴国际机场投运仪式隆重举行。 Reference On September 25th, a grand opening ceremony was held for the Beijing Daxing International Airport.
Huoshan Translation (Better) WeChat AI (Worse)
Translation On September 25, the commissioning ceremony of Beijing Daxing International Airport was held ceremoniously. On September 25, the delivery ceremony of Beijing Daxing International Airport was held. Refined Sentence On September 25, the commissioning ceremony of Beijing Daxing International Airport was held ceremoniously. On September 25, the opening ceremony of Beijing Daxing International Airport was held.
BARTScore++. Words detected and corrected by BARTScore++ are highlighted. We can see that BARTScore++ can benefit from the distances of explicit error and implicit error, achieving more reliable evaluations.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 9: "Limitation"
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 2, 3, 4
3 B1. Did you cite the creators of artifacts you used? Section 2, 3, 4
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We use the dataset (described in Section 4) and code framework (transformers) which are publicly available. Also, we cite the creators of them.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 10: "Ethics Statement", we use publicly available and widely-used datasets/code framework.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 10: "Ethics Statement", we use publicly available and widely-used datasets.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We include these relevant statistics described in Section 4.
C 3 Did you run computational experiments? Section 5, 6
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 5, 6
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4, 5; Appendix B
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5, 6
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4, 5, 6
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We hope future studies can take these situations into account. • In §3. 2 we can see that our proposed error analysis framework fully relies on the generation probabilities of BART to decide how to refine the hypothesis. Still, we see that this framework may lead to false judgments due to unfaithful content. Further research can explore how to calibrate the pre-trained models during error analysis. • In §3. 3 we integrate the distance of explicit and implicit errors by simply computing their weighted sum. This can be improved by considering more factors, e. g. the overall quality of the generated text, refining iterations, and external signals. We will leave the exploration of combining these factors and designing better weighting schemes as future work