SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakes
Year: 2023

Abstract:
Is it possible to train a general metric for evaluating text generation quality without humanannotated ratings? Existing learned metrics either perform unsatisfactorily across text generation tasks or require human ratings for training on specific tasks. In this paper, we propose SESCORE2, a self-supervised approach for training a model-based metric for text generation evaluation. The key concept is to synthesize realistic model mistakes by perturbing sentences retrieved from a corpus. The primary advantage of the SESCORE2 is its ease of extension to many other languages while providing reliable severity estimation. We evaluate SESCORE2 and previous methods on four text generation tasks across three languages. SESCORE2 outperforms unsupervised metric PRISM on four text generation evaluation benchmarks, with a Kendall improvement of 0.078. Surprisingly, SESCORE2 even outperforms the supervised BLEURT and COMET on multiple text generation tasks. The code and data are available at https://github.com/ xu1998hz/SEScore21.

heading: 1 Introduction
text: Recently, researchers made significant progress in text generation: translation (Birch, 2021), structured data-to-text (Gardent et al., 2017), dialogue generation (Vinyals and Le, 2015), and summarization (Chopra et al., 2016). Automatic metrics are essential for the development of text generation models as they replace expensive human labor and are able to evaluate the generation performance (Celikyilmaz et al., 2020), as well as guide the generation process (Unanue et al., 2021; Freitag et al., 2022). How can we efficiently and effectively train a metric for general text generation tasks?
Depending on the inputs, we can categorize evaluation metrics into source-based, hybrid-based, and
1Part of the work is done while WX is an intern at ByteDance.
reference-based metrics. Source-based metrics estimate text quality through the source and are useful when reference is noisy or unavailable (Louis and Nenkova, 2013; Kepler et al., 2019), but they may produce sub-optimal results and explore spurious correlations (Durmus et al., 2022). Referencebased metrics, when paired with high-quality references, can reflect text generation quality, regardless of source modalities (e.g audio and triples). Hybrid metric COMET (Rei et al., 2020) uses both source and reference. In this work, we aim to construct a reference-based metric, as it is invariant to the source modality, making it suitable for use across various tasks.
Although learned metrics have been shown to be more effective than rule-based metrics (e.g BLEU (Papineni et al., 2002)), they still have limitations in terms of evaluation capability and applicability to specific tasks. Supervised metrics such as BLEURT (Sellam et al., 2020) and COMET (Rei et al., 2020) are superior in evaluation, but restricted to tasks with human ratings of generated text. Unsupervised metrics, such as BERTScore (Zhang et al., 2019) and BARTScore (Yuan et al., 2021), do not require human ratings for training, but their correlation with human judgment on specific tasks is still inferior compared to the best supervised metrics (Freitag et al., 2021b).
Our goal of this paper is to devise a referencebased automatic evaluation metric that 1) can be learned without a human quality score, 2) align well with human judgments, and 3) can be generalized to a wide variety of domains and NLG tasks. To achieve this, we propose a self-supervised training method using text with synthetic mistakes. Our main intuition is that these synthetic mistakes should contain errors at different severity levels and appear realistic ("realistic mistakes” are defined as natural and model output-like mistakes). We are inspired by the human evaluation protocol MQM (Freitag et al., 2021a) which assesses transla-
5166
tion quality by identifying errors with two levels of severity. To make the synthetic mistakes realistic, we mine surface differences among groups of similar sentences and use these different phrases to perturb the original text and construct mistakes (See Figure 2). Unlike previous methods that utilize generative models to synthesize errors (Xu et al., 2022), our approach employs retrieved similar sentences, making it more general. To encompass text diversity, anchor texts are sampled from large-scale parallel corpora. Additionally, a novel pretraining signal is proposed to train SESCORE2, which aims to resemble the way humans grade model outputs by estimating the severity levels of each mistake (Freitag et al., 2021a). To date2, we support six languages: English, German, Chinese, Japanese, Russian, Spanish. The primary advantage of the SESCORE2 is its ease of extension to numerous other languages while providing reliable severity estimation. Our contributions to this paper are as follows:
• We propose SESCORE2, a self-supervised (SSL) method to train a metric for general text generation tasks without human ratings;
• We develop a technique to synthesize candidate sentences with varying levels of mistakes for training. To make these self-constructed samples realistic, we introduce retrieval augmented synthesis on anchor text;
• We annotate an additional human rating dataset for WMT21 German-to-English testing set fol- lowing MQM human annotation procedure and we release it for public use;
2May 25, 2023
• Our experiments demonstrate that SESCORE2 is effective in a wide range of NLG tasks and surpasses the top unsupervised metrics PRISM by 0.078. Additionally, it also outperforms or matches the supervised metrics in terms of Kendall correlation.

heading: 2 Related Work
text: Human evaluation metrics such as crowd-worker evaluation using direct assessment (DA) are widely used in WMT shared task competition (Ma et al., 2018, 2019; Mathur et al., 2020). Mathur et al. (2020); Freitag et al. (2021a) find that crowdworkers fail to discriminate human and machine outputs. Freitag et al. (2021a) improves human ratings by using Multidimensional Quality Metrics (MQM) framework (Lommel et al., 2014) with language experts. Each annotated error can be categorized into multiple types and is associated with different severity levels, such as major and minor.
Automatic evaluation metrics such as rule-based metrics (e.g. n-gram matching BLEU (Papineni et al., 2002), chrF (Popović, 2015)) and distancebased (e.g. TER (Snover et al., 2006)) have been commonly used in text generation evaluations because they are fast and domain invariant. However, they have limitations in capturing semantics and long-distance dependencies (Zhang et al., 2019). The supervised learned metrics (Rei et al., 2020; Sellam et al., 2020) are directly optimized from human ratings. However, they may have poor generalization to unseen domains and tasks (Freitag et al., 2021b). Unsupervised learned metrics attempt to obtain training objectives other than human ratings (Zhang et al., 2019; Zhao et al., 2019; Thompson and Post, 2020; Yuan et al., 2021). However, as pointed out by (Freitag et al., 2021a), they are limited on the error types that they can evaluate (e.g accuracy) and can not go beyond (e.g fluency or style). Some recent studies attempt to mitigate this issue by generating synthetic data via paraphrasing and perturbations (Sellam et al., 2020; Kryscinski et al., 2020; Gao et al., 2021). To further derive the fine-grained pretraining signals, SEScore (Xu et al., 2022) leverages language models to generate multiple error types in one segment and estimate each error’s severity level. However, model-dependent data synthesis can intrinsically introduce model bias and limit the diversity of data samples.
SESCORE2 develops a novel retrieval augmented synthesis technique, which is task-agnostic
and can simulate diverse and realistic model mistakes using the parallel corpora. Inspired by Freitag et al. (2021a), we obtain our pretraining signals aligning with human grading process.

heading: 3 Problem Definition
text: The task is to estimate the quality score between a reference and a model-generated hypothesis. Follow Freitag et al. (2021a), if the model error semantically alters the meaning of the sentence, we label it as major, otherwise as minor. See the example in Figure 1, where a 4-point star represents the reference, and circles and triangles represent the sentences with minor and major mistakes, respectively. A minor mistake contributes a −1 score, and a major mistake contributes a −5 score. The severity score of a hypothesis is the sum of all minor and major contributions and is no less than −25.
Given a set of reference, hypothesis, humanannotated severity score triples (x, y, s), our goal is to train a learned metric, M(x,y) → s. Due to the scarcity of human ratings, such triples are unavailable for most tasks. Thus, we propose an automatic way to generate synthetic training samples.

heading: 4 The SESCORE2 Approach
text: SESCORE2 is a SSL technique, initialized with pretrained embeddings, like BERT, then trained with task-agnostic NLG evaluation objective on large-scale synthetic data. No specific fine-tuning is required at inference time evaluation. There are three steps: 1) data construction that samples source-target pairs (t,x) from machine translation (MT) corpora and creates synthetic text y from x using retrieval augmented synthesis; 2) severity score labeling that automatically generates label s using (t,y); 3) model training which pretrains a regression model M using triples (x,y, s). During inference, SESCORE2 only takes reference and model output as input and estimates the quality score, which can be applied to different text generation tasks. Detailed implementations of the model can be found in Appendix 5.

heading: 4.1 Retrieval Augmented Synthesis
text: Given a reference, a common way to generate a negative hypothesis is in-batch negative samplings or in-batch token insertions or replacements (Fu et al., 2022). However, as shown in Figure 1, these approaches mainly produce negative samples that are syntactically or semantically incorrect, which
are not the case for modern text generation models (Freitag et al., 2021a), see Figure 1. Therefore training with these samples could not help distinguish model-generated hypotheses.
Thus we proposed to use retrieval-based approaches to search negative samples. More specifically, given a text corpus, SESCORE2 finds the k nearest neighbors of x based on their vector representation using pretrained language embeddings (ex. LASER3).
We control the text pair proximity by setting a margin criterion when retrieving the k nearest neighbors (Schwenk et al., 2021). For fast k-NN search, we use an index table. Details refer to Appendix B. Based on our preliminary study, the margin criterion (m = 1.06) can retrieve sentences with similar text structure or semantics. Detailed numbers can be found in the Appendix C.
We did not always use the first nearest neighbor because, in many cases, they are too close to the anchor. To increase a diverse coverage of errors, we randomly pick one of those k nearest neighbors z (All circles within the loop have equal chances to be selected in Figure 2). We use edit distance algorithm (Snover et al., 2006) to decompose the surface form difference between x and z into a chain of perturbations such as insertion, deletion, and replacement z = Pn(Pn−1(. . . (P1(x)))). In addition, we include random word drops to diversify the errors further. Each Pi is a candidate perturbation to be applied to the text x.
According to the human evaluation study (Freitag et al., 2021a), models are most likely to produce fewer than 5 errors. Otherwise, they are labeled as catastrophic errors. Thus for each x, we randomly select 5 out of the n perturbations. Each time, we apply a random subset of the five perturbations to transform x to a negative sample y, which contains no more than 5 compositional errors. One challenge is synthesizing positive samples for the anchor since no ground truth is available. Inspired by (Gao et al., 2021), we leverage the dropout function to simulate paraphrasing embeddings by feeding the anchor twice. In addition, In-batch negatives are used to approximate the catastrophic errors.
In figure 2, we demonstrate that our retrieval augmented synthesis can synthesize errors that are contextually sound but semantically or syntactically deviate from the reference. For example, drop of
3We used LASER3, which supports over 200 languages.
"of" introduces syntactical error whereas modifications of "made" and "security" introduces semantic errors.

heading: 4.2 Automatic Severity Score Labeling
text: Once we get the synthesized text y, we need to label its severity score. We design severity functions for all types of perturbations, and the score of a synthesized text y is the sum of all severity estimations. Inspired by the human evaluation (Freitag et al., 2021a), we consider two levels of severity measures: major (score: −5) and minor (score: −1), for each error in the candidate outputs. An error is major if it alters the core meaning of the sentence. See triangle and circle examples in Figure 1. Each perturbation is estimated independently to avoid the influence of the others. t is the machine translation pair of x. t and x will be used for insertion/replacement severity estimation.
For insertion and replacement, the severity score is determined by the likelihood of the inserted or replaced tokens. We use a cross-lingual MLM model such as XLM (CONNEAU and Lample, 2019) to estimate the likelihood. The intuition is that XLM with TLM can model co-occurrences and alignments between source-target tokens. If an error alters the meaning, XLM will be unlikely to restore altered tokens in the perturbed location under the MT source sentence t and the rest of y’s contexts.
The severity estimation of a single perturbation Pi on x to yi can be decomposed into two steps: In the first step, we replace perturbed tokens of y with masked tokens. Let ymask denote the masked text after the first step, and m denotes the perturbed tokens with length l, probability pinsert,replace = 1l ∑l i=1 P (mi|t,ymask) represents the likelihood of this insertion or replacement. For the delete operation, we use TF-IDF weight
wdelete to approximate the salience of deleted tokens. The importance weights of the deleted tokens are formulated as, wdelete = max(wi), i = 1, ..., l with l tokens. Our intuition has two parts: First, we assume that delete operation creates a minor error if all deleted words lack meaningful semantics (e.g. stop words). Second, if one word in the word span has high importance weights (TF-IDF), deletion of the whole word span will alter the sentence semantics. Therefore, we find the maximum of wi for wdelete, instead of mean.
Lastly, an operation is minor if pinsert,replace ≥ γ or wdelete < λ, and major otherwise. where γ and λ are predefined threshold4.

heading: 5 Quality Prediction Model
text: We initialized our quality prediction model with pretrained masked language model (e.g. XLMR). During training, SESCORE2 takes in text x
4λ = 1 and γ = 0.1 for all three languages. Discussions of hyperparameter choices are included in Appendix G
and synthesized text y, supervised with regression score s. We drive the sentence embedding from the average pooling of the last layer. Inspired by the prior approach (Shimanaka et al., 2018), we extract two features between sentence embeddings x and y: 1) element-wise sentence product and 2) element-wise sentence difference. We concatenated above two features into one vector and feed into a neural network regressor and trained with mean squared error. During inference, when given unseen candidate and reference pair, SESCORE2 can directly output a regression score.

heading: 6 Experiments
text: To verify the generalization ability of the SESCORE2, we investigate the following questions:
• Can SESCORE2 be generalized to multiple domains of the same task?
• Can SESCORE2’s language checkpoint X be used to evaluate all languages Y to X’s outputs?
• Can SESCORE2’s language checkpoint X be used to evaluate all different text generation tasks on language X?
• How to interpret SESCORE2?
Corresponding to the aforementioned evaluation aspects:
• We test SESCORE2 over two different domains (News and TED) at WMT.
• We test SESCORE2 over multiple Y-toEnglish directions.
• We test SESCORE2’s English checkpoint over a diverse set of NLG tasks: Machine Translation, Speech Translation, Data-to-Text, and Dialogue Generation.
• We test SESCORE2 over multiple evaluation dimensions. Moreover, we conduct comprehensive experiments for each component of SESCORE2 and analyze the leading factors contributing to the final result.

heading: 6.1 Pretraining Step

heading: 6.1.1 Pretraining Data
text: We collected our pretraining data from WMT17-19 publicly available datasets. Details of data collections can be found in the Appendix A. We randomly
sampled 5M, 4.5M, and 5M sentence pairs for ZhEn, En-De, and En-Ja respectively. We use each target language sentence as an anchor to retrieve the 128 nearest neighbors to build the index table and use parallel sentences to compute severity measures. We train separate checkpoints for each language direction and we use the final English checkpoint to evaluate SESCORE2 in different text generation tasks. To ensure diversity, our index table includes collected WMT News domain data and Wikipedia dumps (See Table 1 for details). We use WMT20 Zh→En and En→De with MQM labels (Freitag et al., 2021a) as our development sets.

heading: 6.1.2 Scoring Model
text: We use Rembert (Chung et al., 2020) as the backbone for all text generation tasks (other backbone choices are discussed in the Appendix D). We use Adam optimizer and set batch size, learning rate, and dropout rate to 256, 3e-5, and 0.1 respectively. We use the mean squared error to train the metric model. All checkpoints from Rembert trained for 15,000 iterations. We use 8 A100 GPUs to train for 18 hours for each checkpoint.

heading: 6.2 Baseline Model
text: For all NLG tasks, we include 1) three n-gram, distance-based baseline metrics: BLEU (Papineni et al., 2002), ChrF (Popović, 2015) and TER (Snover et al., 2006); 2) four best performed learned metrics without human ratings: PRISM (Thompson and Post, 2020), BARTScore (Yuan et al., 2021), BERTScore (Zhang et al., 2019) and SEScore (Xu et al., 2022); and 3) two SOTA supervised metrics: COMET 5 and BLEURT. Implementation details are discussed in Appendix E.

heading: 6.3 Evaluation Procedure
text: For all the text generation tasks, we compute segment-level Kendall correlation between metric
5Since COMET is a source-reference-based approach only applicable to translation tasks, we only used to generate results for machine and speech translation
outputs and human scores (We include all Spearman correlation in the Appendix Table 10, 13 and 14. They yield the same conclusion as Kendall correlation). We conduct William’s pair-wise significance test (Graham and Baldwin, 2014) to highlight the significant improvements.
Machine Translation For En-De and Zh-En, we used publicly available WMT21 News and TED’s human annotations (Freitag et al., 2021b). We also hired 3 professional linguists to annotate 1000 testing samples from WMT21 De-En News domain using MQM human evaluation procedures (Freitag et al., 2021a). Detailed testing statistics are in Appendix Table 7 and detailed human annotation procedures are in Appendix F.
Dialogue Generation Public BAGEL benchmark contains target utterance generation for spoken dialogue systems. This benchmark contains 202 model outputs. Each sample is annotated in the aspect of naturalness, informativeness, and quality.
Data-to-Text Generation Public WebNLG2020 (Zhou and Lampouras, 2020) contains 17 models and each contains 177 outputs. Each sample is annotated by five aspects: correctness, data coverage, fluency, relevance, and text structure.
Speech-to-Text We use IWSLT22 English-toJapanese (En-Ja) human annotations. The benchmark contains four systems and each contains 118 outputs. All human annotations were done using JTF MQM variant (JTF, 2018).

heading: 6.4 Overall Performance
text: In Table 2, we demonstrate metrics’ overall performance in machine translation, speech translation, data-to-text, and dialogue generation.
SESCORE2 outperforms the best rule-based metric chrF (Kendall=0.134) significantly in the overall Kendall correlation, with Kendall improvement of 0.145. SESCORE2 outperforms all unsupervised learned metrics significantly in all four text generation tasks and three MT translation directions. In particular, SESCORE2 outperforms PRISM (Kendall=0.201) with Kendall improvement 0.078. More importantly, SESCORE2 outperforms the supervised BLEURT in two of the four text generation tasks and achieves a higher Kendall correlation overall across four tasks, with Kendall improvement of 0.014 in D2T(En) and 0.004 in Dialog(En).

heading: 6.5 SESCORE2 achieves consistent superior performance for different text generation tasks
text: For Machine Translation, SESCORE2 outperforms all unsupervised metrics significantly across all three language directions. Despite all language directions being present in the training sets of both BLEURT and COMET, SESCORE2 outperforms both supervised metrics COMET and BLEURT in Zh-En and achieves comparable performance to COMET and close performance to BLEURT at EnDe and De-En. For speech translation, SESCORE2 outperforms all unsupervised metrics significantly and leads COMET by a large margin. One explanation for this improvement could be that human ratings for English to Japanese were not included in COMET’s training data, highlighting the limitations of supervised metrics in unknown domains or language directions. Lastly, SESCORE2 outperforms all supervised and unsupervised metrics at data-to-text and dialogue generation. Compared to BLEURT, which is supervised by translation
human rating, SESCORE2 can achieve superior generalization capability in non-translation tasks, such as data-to-text and dialogue generation.

heading: 6.6 SEScore2 achieves consistent superior performance for translation into the same target languages
text: SEScore2 is consistently better on a variety of text generation tasks with the same generation language. For the machine translation task, we further investigate SESCORE2’s generalization capabilities over different languages to X translation outputs. From Zh→En and De→En, SESCORE2 outperforms all unsupervised metrics significantly. In comparison to supervised metrics, SESCORE2 surpasses both BLEURT and COMET in Zh→En, and achieves a comparable performance to the COMET and 0.016 Kendall correlation gap to BLEURT at De→En.

heading: 6.7 SESCORE2 achieves consistent superior performance across different domains
text: We investigate the domain influence on the evaluation metrics when shifting the testing set from News to TED. As shown in Table 3, all metrics have lower Kendall correlations in TED compared to those in News. We conjectured that the cause is due to the domain differences between the two testing suites. Unlike News, TED contains sentences with informal and disfluent language styles (Freitag et al., 2021b). The supervised learned metrics have the largest gap when shifting the domain from News to TED. The reason is that the entire supervised human rating data is from the News do-
main only. Although the rule-based metrics (TER. BLEU and Chrf) have relatively lower overall correlations, their correlation is less influenced by the domain shift, with an average 0.023 Kendall correlation difference. Unsupervised learned metrics can be less influenced by domain shift compared to supervised metrics. However, they still have more Kendall correlation drops compared to the rule-based metrics, with an average Kendall correlation 0.032. Most importantly, we observed that SESCORE2 achieves the highest overall Kendall correlation and achieves the lowest gap (0.022) among learned metrics when shifting between domains. In Section 6.8.1, we demonstrate that SESCORE2 can take advantage of the data scale and improve its performance on TED while scaling up the data. Full results can be found in Appendix Table 12.

heading: 6.8 Quantitative Analysis
text: To validate the ideas in SESCORE2, we investigate the impact of data scaling, the risks of human rating supervision, effects of our retrieval augmented (RA) synthesis, ablation on RA operations and severity measures, and interpretation of SESCORE2 at different evaluation dimensions. We include the effects of model initialization in Appendix D.

heading: 6.8.1 Law of the Data Scaling
text: We study the scaling effects on SESCORE2’s performance, by testing checkpoints trained at 0.5M, 1M, 2M, and 4M training samples. For both Zh-En and En-De across two domains, we observe the sharp performance improvements at first 1M pretraining data. Larger pretraining data quantity leads to higher human correlations for both language directions. We get 2.5% and 1.8% improvements in Zh-En, and 2.5% and 1.1% improvements in En-De at 2M and 3M data points. Performance saturates from 3M to 4M synthetic data, with around 0.5% improvements in both language directions. This suggests that a data scale of 1M can train a competitive metric, and larger data can gradually improve the metric to fit into a general domain.

heading: 6.8.2 Danger of Fine-tuning
text: To address the question "Can fine-tuning always lead to better performance?", we fine-tune SESCORE2 on existing 350K English and 59K German WMT17-19 News domain human rating
data 6. For each domain, we report the average Kendall correlation between En-De and Zh-En. In Table 4, SESCORE2 +FT improves by 8.7% over SESCORE2 in the News domain. Additionally, it outperforms both BLEURT and COMET in the News domain with 4% and 2.3% Kendall improvements respectively. Despite the improvements in News, the supervised SESCORE2 has a 13.6% correlation drop in the TED testing set, resulting in a larger correlation gap (0.083) between News and TED domains. This confirms our assumption that fine-tuning with domain-specific human ratings can fit the metric tightly to the trained domain distribution, but may decrease its generalization ability across domains. The unsupervised SESCORE2 achieves the highest overall Kendall correlation across two domains.

heading: 6.8.3 Effectiveness of Retrieval Augmented Synthesis
text: In Figure 4, we demonstrate the superior performance of retrieval augmented (RA) data construc-
6Like COMET, we use WMT17-19 DA human ratings. BLEURT uses WMT15-19 DA results for its training dataset.
tion compared to the random token transformations. Our observation is that most of the random in-batch tokens have low co-occurrence probabilities with their contexts. The sentence embeddings from those text transformations can be easily distinguished from the anchor embedding, by the pretrained language model (Conneau et al., 2019). Therefore, further pretraining on negative samples with random token transformations does not lead to significant correlation improvements. We empirically demonstrate that RA data construction improves random token insert/replace by 114% in the News and TED domains.

heading: 6.8.4 Ablation on RA operations
text: To evaluate the performance of each component at our retrieval augmented synthesis, we separately trained checkpoints with synthetic data that 1) contains delete operations only; 2) contains insert and replace operations according to our scheduled locations; 3) contains all operations with scheduled positions. To exclude the effects from the severity measures, we do not assign severity measures for each error and instead label each sentence with the number of errors it contains. In Figure 4, we observe that our RA insert/replace contributes the most of the human correlations, 0.308 at News and 0.219 at TED. This suggests that our scheduled positions to insert and replace are important to construct realistic synthetic sentences and learning meaningful embeddings. Despite the simple scheme, delete-only construction can achieve competitive performance, with Kendall correlation 0.189 and 0.123 in News and TED respectively. By combining all operations, the aggregated effect can further improve the Kendall correlation 3.2% at News.

heading: 6.8.5 Effects of Severity Measures
text: In Figure 4, we empirically verify two of our severity measures: 1) IDF-based 2) XLM-based approaches. Our IDF-based severity measures on delete operation can improve 51.9% Kendall correlation at News and 81.3% at TED. Our XLMbased severity measures on insert and delete can improve 4.9% at News and 11.9% at TED. Lastly, the joint effect of two severity measures can improve SESCORE2 without severity measures by 6.92% at News and 17.9% at TED.

heading: 7 How to interpret SESCORE2?
text: In order to interpret the evaluation dimensions of SESCORE2, we conducted multi-dimensional human correlations in WebNLG and BAGEL benchmarks. In Figure 5, we observe that SESCORE2 achieves the highest Kendall correlation across all dimensions except informativeness compared to the BLEURT. At WebNLG and BAGEL, SESCORE2 is most correlated to fluency, text structure, naturalness, and quality, which leads the second highest metric BLEURT by 16.2% and 13.5%, 8.5% and 10.6% respectively. To conclude, despite producing an overall score, SESCORE2 can be a great indicator of diverse evaluation aspects of text generation. In particular, SESCORE2 has a significant advantage over baseline metrics in terms of the quality and fluency aspects of quality assessment. Full results are in Appendix Table 6 and Table 9. We further rescaled our results to the predefined
range (0 to −50) to ensure consistency and interpretability across domains and tasks (See Appendix Section H for implementation details).

heading: 8 Supported Languages
text: Currently, SESCORE2 supports English, German, Japanese, Spanish, Chinese and Russian. Our pipeline in extending SESCORE2 to future languages is generic. It is straightforward to extend SESCORE2 to up to 100 languages i.e. any language supported in XLM. To obtain reliable severity measures, we currently support 14 languages (Ar, Bg, De, El, Es, Fr, Hi, Ru, Sw Th, Tr, Ur, Vi and Zh). For detailed limitation discussion about severity measures, please refer to Section 9.

heading: 9 Conclusion
text: We propose a novel retrieval augmented synthesis method for generating diverse and realistic errors on a large scale, with varying severity levels. Our experiment demonstrates, SESCORE2, with its human-aligned self-supervised objective, can outperform prior metrics or match supervised metrics in four text generation tasks across three languages. Lastly, we demonstrate SESCORE2 can correlate well with multiple evaluation aspects, such as fluency and quality.
Limitations
One potential improvement to this work is the development of a method to evaluate the accuracy of the severity measure component. We have demonstrated the effectiveness of SESCORE2 with a severity measure through improved Kendall correlations for various types of retrieval augmented synthesis in Figure 4. However, there is currently no widely accepted way to quantitatively measure the accuracy of the severity labels. This is because there is no existing dataset that can be used to benchmark severity measures. While Freitag et al. (2021a,b) have released MQM annotations with error spans for each segment, these annotations often include compositional errors that prevent the evaluation of individual severity labels without also considering other errors in the sentence. A potential future direction for this research would be to create a benchmark dataset that would allow direct assessment of individual severity estimation or explore alternative methods for evaluating the accuracy of severity measures.
Second, we have not been able to test SESCORE2 on low-resource languages due to the lack of MQM-annotated testing sets in these languages. However, we have demonstrated that SESCORE2 can still perform well without severity estimation by outperforming top unsupervised metrics such as BERTScore, BARTScore and PRISM as shown in Figure 4. This suggests that SESCORE2 may be useful for low-resource languages since parallel corpora are not available for most low-resource language settings. To further verify this, a potential future direction would be to create testing sets with MQM labels for lowresource languages, to test the performance of SESCORE2 and other learned metrics in such scenarios.
Lastly, since SESCORE2 is based on proximity between reference and model output, its capabilities for open-ended text generation tasks have not yet been fully explored. This presents an opportunity for future research to investigate the potential of this method in such scenarios.
Ethics Consideration
We hired three human raters to annotate WMT21 De→En testing set. The evaluated text is publicly available machine translation testing set which contains no sensitive or explicit languages. There is no risk of exposing annotators’ identities and they are fully aware the usage of their collected dataset. We use the standard MQM human evaluation procedures (Freitag et al., 2021a) and all annotaters are experienced with this evaluation protocol. All collected human annotations will be released at the camera ready. The hourly salary for the raters are well above the minimum wage in the local region. Details can be found in Appendix F.

heading: A Pretraining Data Collection
text: For Chinese-to-English, we collect 20M sentence pairs from UN Parallel (Ziemski et al., 2016), News Commentary (Tiedemann, 2012) and CWMT corpus (Barrault et al., 2019). For English-toGerman, we collect 4.5M sentence pairs from Europarl (Koehn, 2005), Common Crawl (Kúdela et al., 2017) and News Commentary. For Englishto-Japanese, We collect 18M sentence pairs from News Complimentary, WikiMatrix (Schwenk et al., 2021) and En-Ja subtitle corpus (Pryzant et al., 2018).
B Index Table Construction
We use LASER library7 to compute all the sentence embeddings and use Faiss library 8 to build the index table for English, German and Japanese. We
7https://github.com/facebookresearch/LASER 8https://github.com/facebookresearch/faiss
used 8*A100 GPUs for the index table construction. The duration for building index table for English, German and Japanese is 48 hours, 24 hours and 48 hours respectively. From the constructed index table, we extracted 128 nearest neighbors for each text sentence. To ensure our learned metrics can cover diverse domains and tasks, we sample millions of raw sentences from diverse domains of corpuses and build ten-million scale index tables. Detailed statistics are discussed at Section 6.1.1.

heading: C Margin-based Criterion
text: We follow the implementation of margin criterion in (Schwenk et al., 2021). We set the threshold of margin criterion to be 1.06 and extract 128 nearest neighbors to estimate mutual translation capability.

heading: D Pretrained Model Initialization
text: All checkpoints from Rembert are trained for 15,000 iterations and all checkpoints from XLM-R are trained for 30,000 iterations. Since our pipeline utilized pretrained model, we try to answer the question that with the setting, can different pretrained model initialization lead to different performance? In particular, we studied two prior used pretrained models: RemBERT (used by BLEURT) and XLM-R (used by COMET). Based on prior study (Chung et al., 2020), RemBERT emperically outperforms XLM-R over multiple multilingual downstream tasks. In Table 8, we demonstrates that compared to XLM-R initialization, our SESCORE2 with RemBERT initialization can further improve Kendall correlations in all language directions. This finding suggests that SESCORE2 with a better pretrained model initialization can increase its learning capacity of score distribution and improve its correlations to human ratings.

heading: E Baseline Implementations
text: For WMT21 News and TED, we use WMT officially released output results (Freitag et al., 2021b) from their official script9. We use HuggingFace evaluate module (Open sourced library) to get baseline outputs for BLEURT (Sellam et al., 2020), COMET (Rei et al., 2020), SEScore (Xu et al., 2022), BLEU (Papineni et al., 2002), TER (Snover et al., 2006) and ChrF (Popović, 2015). For BERTScore (Zhang et al., 2019), BARTScore (Yuan et al., 2021) and PRISM(Thompson and Post, 2020), we use their open sourced Github repository. Specifically, following their recommendations, we use Roberta-large backbone (Liu et al., 2019) for English assessment of BERTScore. We use multilingual BERTScore (Devlin et al., 2019) to assess German and Japanese testing sets. For BARTScore, we use recommended Bart-large-cnn backbone for English testing sets and MBART backbone (Liu et al., 2020) for German and Japanese testing sets. For SEScore, we use stratified error synthesis process (Xu et al., 2022) to construct 120,000 Japanese synthesized texts with pseudo labels and trained a Japanese SEScore. We use this Japanese SEScore to test Japanese testing set.

heading: F Human Annotation Procedure
text: We conduct a human evaluation on WMT21 de-en testing set at News domain. We randomly select 10 systems outputs out of 20 participating systems. We annotate 100 testing segments for each selected system. In total, 1000 testing sentences are annotated. Following the prior study Freitag et al. (2021a), we hired 3 bilingual linguists in English and German. Following MQM evaluation procedure (Freitag et al., 2021a), each rater can only access the source (in German) and model output (in English), without any reference text. Rater is given all the choices of possible error typologies and definitions of the severity levels. We directly use the
9https://github.com/google-research/mt-metrics-eval
instruction of MQM hierarchy, MQM severity levels and MQM annotator guidelines from prior work (Freitag et al., 2021a) to our human raters (Please refer to Table 10, Table 11 and Table 12 for specific references). All three raters are well-trained to perform human ratings. The hourly rate for all raters is 70 Chinese Yuan per hour. The local minimum wage is 23 Chinese Yuan per hour. Our human evaluation study has no risk of exposing any annotator’s identities and text contains neither sensitive or explicit language.

heading: G Hyperparameters on Severity Measures
text: We conducted preliminary experiments over hyperparameter choices for the severity measures. We hand-crafted 50 severity examples (each example only contains one major or minor example) to select the γ and λ. We determine the range of γ to be from 0.1 to 0.4 and λ to be 1. To select the best γ, we use our retrieval augmented perturbation
to generate 200k German and English synthesized sentences, respectively. We greedy searched threshold γ from 0.1, 0.2, 0.3, 0.4, for both languages. From Table 11, we construct labels based on each possible γ and construct corresponding training triples (x, y, s). Therefore, we obtain four checkpoints in each language direction. Based on the Kendall correlations, we concluded that γ is not a sensitive hyperparameter. Our best hypothesis is that the tokens that are not likely to occur under the source and target contexts have low probabilities (ex. 0.01). Therefore, choosing the specific γ will not significantly affect the accuracy of the severity measures. In the end, we select γ = 0.1 for both En-De and Zh-En.

heading: H Score Range Rescaling
text: In our pretraining data, the synthetic score range is between 0 to −50. However, due to the last activation Tanh in our model, the learned score range is constrained between a positive constant h and a negative constant l. As a result, despite having a great ranking capability, our score is not interpretable for users. Ideally, we want our score to lie within the pre-defined range of 0 to −50 and remain invariant across domains and tasks.
To achieve this goal, we collected a large-scale dataset of raw data (2M) from Wikipedia for the target language. We randomly grouped 1M pairs without replacements. Given the random nature of the pairs, we hypothesize that they likely have low semantic or syntactic similarity, thus prohibiting low scores. Consequently, we can obtain a lower score bound l from this set. We calculate score l by averaging SESCORE2 for all 1M sentence pairs in the set. Furthermore, we randomly select 1M sentences, and for each of them, we feed it twice into the forward layers of SESCORE2 with a dropout rate of 0.1. In this case, the two computed
embeddings should be nearly identical, allowing us to derive an upper bound h for the score range. We obtain score h by averaging SESCORE2 for all 1M embedding pairs in this set.
SESCORE2Rescale = ( SESCORE2 − l
h− l − 1) ∗ 50 (1)
From Eqn 1, we can normalize our SESCORE2 roughly between 0 to −50. We can use our predefined score for major errors (score: −5) and minor errors (score: −1) to interpret the final results. By normalizing the score, we can ensure better interpretability and maintain consistency across different domains and tasks.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitation
3 A2. Did you discuss any potential risks of your work? Ethics Consideration
3 A3. Do the abstract and introduction summarize the paper’s main claims? We introduced our main claim at Abstract and Section 1
3 A4. Have you used AI writing assistants when working on this paper? ChatGPT. We use ChatGPT to detect grammar mistakes.
B 3 Did you use or create scientific artifacts? Appendix G and Section 5.3
3 B1. Did you cite the creators of artifacts you used? Appendix G and Section 5.3
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Appendix G and Section 5.3
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix G and Section 5.3
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? ethical consideration
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Appendix G, Section 5.3 and
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 5.1.1
C 3 Did you run computational experiments? Section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 5.1.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 5.1.12
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5.4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix G
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Appendix H and Ethics Consideration
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix H and Ethics Consideration
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix H and Ethics Consideration
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Appendix H and Ethics Consideration
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Appendix H and Ethics Consideration
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Appendix H and Ethics Consideration

heading: Future Work
text: A potential future direction for this research would be to create a benchmark dataset that would allow direct assessment of individual severity estimation or explore alternative methods for evaluating the accuracy of severity measures. Second, we have not been able to test SESCORE2 on low-resource languages due to the lack of MQM-annotated testing sets in these languages. However, we have demonstrated that SESCORE2 can still perform well without severity estimation by outperforming top unsupervised metrics such as BERTScore, BARTScore and PRISM as shown in Figure 4. This suggests that SESCORE2 may be useful for low-resource languages since parallel corpora are not available for most low-resource language settings. To further verify this, a potential future direction would be to create testing sets with MQM labels for lowresource languages, to test the performance of SESCORE2 and other learned metrics in such scenarios. Lastly, since SESCORE2 is based on proximity between reference and model output, its capabilities for open-ended text generation tasks have not yet been fully explored. This presents an opportunity for future research to investigate the potential of this method in such scenarios