Towards Multiple References Era – Addressing Data Leakage and Limited Reference Diversity in Machine Translation Evaluation (2024)

Abstract: Recent research has shown a weak correlation between n-gram-based metrics and human evaluations in machine translation task, particularly when evaluating large language models (LLMs). Additionally, the data leakage risk in LLMs may cause an overestimation problem when evaluating LLMs on downstream tasks. In this work, we identify the limited diversity of references as the primary cause for the inferior performance of n-gram-based metrics and the overestimation problem. To address this issue, we propose to utilize multiple references generated by LLMs, coupled with an effective selection strategy focused on accuracy and diversity, to improve the alignment between automatic metrics and human evaluations. We validate our approach on the WMT22 Metrics benchmark with 4 languages and observe a maximum accuracy gain of 9.5% in F200spBLEU, which makes it on par with computationally expensive neural-based metrics. We also show that using multi-reference with n-gram-based metrics significantly alleviates the overestimation problem when evaluating LLMs with data leakage. Further analysis explores the factors that affect the quality of generated references, offering insights into data synthesis by LLMs. We released the code and generated reference1 to encourage research on reference quality.

heading: 1 Introduction
text: Due to the inherent diversity and complexity of natural language, human evaluation serves as the gold standard for assessing the quality of machine translation. However, conducting human evaluation is time-consuming and prohibitively expensive in real-world scenarios. Consequently, developing reliable automatic evaluation metrics is crucial for advancing machine translation research and optimizing machine translation systems (Celikyilmaz
* Corresponding author. 1https://github.com/SefaZeng/LLM-Ref
et al., 2020). The ideal automatic evaluation metrics need to assess the accuracy, fluency, fidelity, and diversity of the generated candidates by the model. It also requires high consistency with human assessment to demonstrate its reliability.
Automatic evaluation metrics for machine translation can be generally classified into two categories: n-gram-based metrics and neural-based metrics. N-gram-based metrics, e.g., BLEU (Papineni et al., 2002) primarily calculate the lexical overlap between the model’s outputs and the ground truth. On the other hand, neural-based metrics e.g., BLEURT (Sellam et al., 2020) are trained on a huge amount of text data and score the candidates with a black-box model. Due to the huge amount of training data and parameters, neuralbased metrics generally achieve better robustness and generalization. Recent studies (Freitag et al., 2022; Kocmi et al., 2022a) have demonstrated that neural-based metrics exhibit better agreement with human evaluation when compared with n-grambased metrics.
The emergence of LLMs has further deepened these concerns (Zhao et al., 2023; Hendy et al., 2023) due to the diversity of output results from LLMs. Recent studies (Freitag et al., 2022) have highlighted the challenges in using n-gram-based metrics, such as BLEU, to evaluate LLM-generated hypotheses. Some latest reports (Anil et al., 2023) exclusively rely on neural-based metrics to evaluate the generation quality. Another challenge in evaluating LLMs is the data leakage risk due to their reliance on huge corpora obtained from the web. Recent research (Wei et al., 2023; Zhu et al., 2023) has emphasized that data leakage in LLMs is a prevalent and pressing issue, which underlines the importance of conducting more in-depth research to develop effective methods for addressing this challenge.
In this paper, we examine the performance bottleneck of n-gram-based metrics and identify the
11939
limited diversity of references as the primary cause for the inferior performance of n-gram-based metrics and the overestimation problem. To address this issue, we propose LLM-Ref, a framework that utilizes LLMs to generate multiple references, coupled with an effective selection strategy focused on diversity, to improve the alignment between automatic metrics and human evaluations. Experimental results show that our framework improves the consistency between human evaluation and both kinds of automatic evaluation metrics. Further analysis reveals that multiple references with n-grambased metrics can effectively mitigate the potential data leakage risk of LLMs while neural-based metrics still face difficulty in overcoming this issue. To our knowledge, this is the first attempt to address the data leakage risk issue in evaluating LLMs.

heading: 2 Preliminary Experiments
text: To verify BLEU’s evaluation capability on LLMgenerated results and the data leakage issue of LLMs, we conduct preliminary experiments using BLOOMZ-7b on the FLORES-200 test set (Costajussà et al., 2022). Previous work (Zhu et al., 2023) has identified data leakage issues with BLOOMZ7b on FLORES-200 across multiple translation directions.
We conduct supervised fine-tuning (SFT) on the BLOOMz-7b (Scao et al., 2022) with bilingual corpus in Chinese and Japanese from the CCMatrix dataset (Schwenk et al., 2021). To provide a comprehensive comparison, we also evaluate the performance of closed-source large models such as GPT-3.5 and Claude. The baseline machine translation (MT) model is an in-house model with the conventional encoder-decoder architecture.
The evaluation metrics utilized in our experiments include F200spBLEU (Costa-jussà et al., 2022), BLEURT (Sellam et al., 2020), and human evaluation (Human Eval). F200spBLEU is the BLEU (Papineni et al., 2002) score with a multilingual tokenizer trained in 200+ languages. Human Eval involves human experts rating the given translations on a scale ranging from 0 to 5, based on how accurately it reflects the meaning of the original sentence. We acquire 3 scores per sentence from human experts and take the average as the final sentence score. The summarized results are presented in Figure 1. We observe two phenomena from the above experiments:
• F200spBLEU significantly underestimates

heading: ID Model DistinctN (n=6) ↑ Unique Token ↑
text: LLMs like Claude and GPT-3.5-turbo, exhibiting a disparity compared to human evaluations.
• Due to data leakage issues (Zhu et al., 2023), F200spBLEU overestimates BLOOMZ-7b and BLOOMZ-7b-ft, and BLEURT overestimates BLOOMZ-7b-ft, contrasting with lower human evaluation scores.
The Inadequacy of N-Gram-Based Metrics for Evaluating LLMs. We conjecture that the output distribution of the LLMs differs significantly from that of the MT model, and the n-gram-based nature of BLEU is constrained by the provided reference, resulting in degraded performance when there is a substantial divergence between the output and the reference.
To gain further evidence of the underlying factors contributing to the bias in BLEU’s evaluation, we conduct experiments on the Chinese to English dataset from WMT22 (Freitag et al., 2022) to examine the output distribution of different models. We primarily use DistinctN (Li et al., 2015) and the count of unique tokens to analyze the output distribution. We select system outputs and ground truth translations from the task for our analysis. The results are presented in Table 1.
We discover that both DistinctN scores and the count of unique tokens of GPT-3.5-turbo are higher than all MT models, commercial systems, and human-translated references. These findings indicate that the outputs of LLMs exhibit a high degree of diversity, which renders it challenging for n-gram-based metrics to effectively estimate the quality of LLM generations.
LLMs’ Data Leakage Risk. The open-source LLMs have gradually become foundational models for most NLP tasks (Touvron et al., 2023; Scao et al., 2022; Zeng et al., 2023b,a). However, due to their reliance on huge corpora obtained from the web, data leakage is an inevitable problem, which may potentially lead to an overestimation of the performance on downstream tasks. Recent study (Wei et al., 2023; Zhu et al., 2023) about LLMs have confirmed that data leakage in some widely used benchmarks is a common issue.
Our hypothesis posits that utilizing multiple references can alleviate overestimation due to data leakage to a certain degree. The tendency of overestimation comes from the model generating almost identical characters as the ground truth, thereby resulting in the performance overestimated by the golden reference. With multiple references provided, the model output close to the golden reference tends to obtain suboptimal performance on other references, which can alleviate the overestimation issue. We will discuss it in detail in Sec 5.4.

heading: 3 Methodology
text: We employ a series of steps to obtain high-quality reference candidates for a more accurate evaluation of generation performance, as outlined below.

heading: 3.1 LLMs Reference Candidates Generation
text: LLMs demonstrate strong capabilities in a variety of natural language processing tasks. We plan to leverage LLMs’ powerful generative capabilities to generate diverse reference candidates which further enhance the NLG evaluations. The prompt for invoking LLMs consists of three components.
Rules. The rules mainly contain some characterization (e.g., You are a professional translator...), and rules to follow. These rules aim to facilitate LLMs generating higher quality and more humanpreferred reference candidates.
Input. The input mainly consists of a taskspecific description (e.g., Please provide 10 highquality translations of the following...) and the source sentence of the test set.
Ground Truth. Providing manually annotated ground truth enables LLMs to generate more accurate reference candidates and align with human preference.
Observations from the experiments indicate that there are multiple factors (e.g., providing ground truth or not) in the prompt that affect the generation quality. It is worth noting that the quantity of references generated in a single inference by LLMs is the most influential factor. We will delve into a detailed analysis of the impact of prompt variants in Sec 5.2. We list the prompt in Appendix B.

heading: 3.2 Quality-Aware Selection
text: As the number of generated reference candidates increases, the LLMs tend to produce outputs with limited diversity (few word substitutions) and lower accuracy (deviating from the original meaning). These two phenomena can be attributed to the fact that there is a finite number of suitable responses and the hallucination problem of LLMs. Therefore, it is crucial to employ effective strategies for selection from the generated reference candidates.
Diversity Selection. We utilize Self-BLEU (Zhu et al., 2018; Zeng et al., 2021; Meng et al., 2020) as a metric of diversity for the generated reference candidates. Self-BLEU is the BLEU score computed against other reference candidates, which is computed as:
Self-BLEUi = BLEU(yi,Y) (1)
where yi is the i-th reference candidate, Y is the array of all y except yi, and each reference will get a score Self-BLEUi.
To detect the anomalous candidates with too large Self-BLEU (limited diversity), we utilize the Inter Quartile Range (IQR) method to decide the threshold for selecting reference candidates, which is a classical method for Outlier Detection. It is computed as:
IQR = Q3 −Q1 (2) upper_bound = Q3 + 0.5 ∗ IQR (3)
where Q1 is the first quartile and Q3 is the third quartile for the Self-BLEU array. We filter out references with a score larger than upper_bound, as they have low diversity.
Accuracy Selection. We employ COMET22 (Rei et al., 2022) as the metric for assessing the accuracy of references, due to its relatively fast computation speed and high accuracy. Subsequently, we utilize the same Interquartile Range (IQR) method for selection.
lower_bound = Q1− 1.5 ∗ IQR (4)
where Q1 is the first quartile of references’ COMET-22 score. We filter out references with a score lower than lower_bound, as they have low accuracy. We conduct experiments to validate their impact and discuss in Sec 5.6.

heading: 3.3 Multi-Reference for Automatic Metrics
text: N-gram-based metrics (e.g. BLEU) are originally designed to use multiple references so we use the original implementation. In contrast, neural-based metrics assess the model’s performance by scoring individual model output in the test set with references and subsequently averaging these scores through the whole test set to get the system-level score for the model. As most neural-based metrics only calculate model output with one reference, we have explored several simple methods(average, top-k, etc.) to combine the scores with multiple references. Empirical results indicate that selecting the maximum value is the best way. Consequently, all subsequent experiments with neural-based metrics are based on the max method, as specified by the following calculation:
score = max([M(ŷ, y0),M(ŷ, yi), ...]) (5)
Here ŷ is the output from the model, yi represents the i-th reference candidate generated by LLMs and M is the metric. Every single sentence in the test set gets a score.

heading: 4 Experiments
text: In this section, we describe the benchmarks and the evaluation we used for our experiments.

heading: 4.1 Datasets and Evaluation
text: To evaluate the performance of our proposed framework, we conducted experiments on WMT22 Metrics shared task (Freitag et al., 2022). This task includes human judgments for three translation directions: English to German (EN-DE), English to Russian (EN-RU), and Chinese to English (ZHEN). The test set for each direction contains around 2000 sentences covering several text domains. The
evaluation criteria are based on MQM2 datasets annotated by domain experts (Freitag et al., 2021).
To evaluate the correlation between automatic metrics and human evaluation, we measured system-level pairwise accuracy, Pearson correlation (ρ), and Kendall-Tau correlation (τ ), following the methodology established by the WMT22 Metrics shared task (Kocmi et al., 2021).
We reproduced reported scores in the WMT22 Metrics shared task findings (Freitag et al., 2022) using the official WMT22 script.3

heading: 4.2 Baseline Metrics
text: As baseline metrics, we mainly focus on two types of metrics including n-gram-based metrics and neural-based metrics.
N-gram-based metrics. We use the following n-gram-based metrics as baselines, including BLEU (Papineni et al., 2002), F200spBLEU (Costa-jussà et al., 2022), chrF (Popović, 2015). We use the sacrebleu (Post, 2018)4 toolkit to perform the calculations for the first three metrics.
Neural-based metrics. For the WMT22 Metric Shared Task, we list the results of most neural-based metrics in the task, including BERTscore (Zhang* et al., 2020), MATEESEQE (Perrella et al., 2022), MS-COMET-QE22 (Kocmi et al., 2022b), UNITE-src (Wan et al., 2022), COMET-QE (Rei et al., 2022), COMETKiwi (Rei et al., 2022), YiSi-1 (Lo, 2019), MATESE (Perrella et al., 2022), MS-COMET22 (Kocmi et al., 2022b), UniTE (Wan et al., 2022). We mainly conduct experiments on two most widely used and powerful metrics BLEURT (Sellam et al., 2020) and COMET-205 (Rei et al., 2020). For BLEURT6 and COMET7, we use the official github example scripts.
We also report the result of GEMBA-GPT4DA (Kocmi and Federmann, 2023) which uses GPT-4 to directly assess the model outputs.
2MQM datasets are composed of multi-dimensional quality scoring by expert translators. Crowdsourced DA datasets are direct assessment scores from crowdsourcing staff.
3https://github.com/google-research/mt-metrics-eval 4https://github.com/mjpost/sacrebleu 5We did not use COMET-22 as its ensemble nature and
lack of open-source models. 6https://github.com/google-research/bleurt 7https://github.com/Unbabel/COMET

heading: 4.3 Settings
text: We use GPT-3.5-turbo for reference generation. When calling the OpenAI API8, we use the default values for all hyper-parameters. The adjustment of the temperature parameter can generate more diverse candidates, and we leave the exploration of this for future work. We generate 40 references for each sentence on WMT22 Metric Shared Tasks.

heading: 4.4 Main Results
text: We evaluated the performance of LLM-Ref at both the system level and segment level.
Table 2 presents the pairwise accuracy and Pearson correlation at the system level. LLM-Ref demonstrates positive improvements across all metrics, including both neural-based and n-gram-based metrics. The improvement is particularly noticeable in the n-gram-based metrics, with an accuracy increase of up to +5.5 percentage points when comparing ID 21 to 11. When combined with QualityAware selection for reference candidates, there is an improvement of up to +9.5 percentage points from ID 21 to 7.
For the neural-based metrics, both BLEURT and COMET-20 exhibit improvements of approx-
8https://openai.com/blog/openai-api
imately +1.7 percentage points comparing ID 6 and 4 to 2 and 1. After incorporating LLM-Ref, BLEURT and COMET-20 outperform all metrics except GEMBA-GPT4-DA.
In terms of the Pearson correlation (ρ), the improvement is more pronounced in the n-gram-based metric, with F200spBLEU experiencing an improvement of up to +19.5 percentage on ZH-EN from ID 21 to 7. The improvement in the neuralbased metric is relatively tiny, with a maximum increase of 1 percentage point.
Segment-level experimental results are consistent with system-level. We list the detailed results in the Appendix C.

heading: 5 Analysis
text: In this section, we will first analyze the effects of reference numbers and prompt variants, and then analyze whether multi-reference solves the data leakage issue. We conduct these analyses on the WMT22 Metrics Shared Task.

heading: 5.1 Effects of Reference Numbers
text: Although LLMs are capable of generating a large number of references, it is important to consider to what extent LLMs can scale the reference numbers, as the diversity and accuracy of the generated reference may decrease.
Figure 2 illustrates the effect of reference numbers in system-level Pearson correlation. For ngram-based metrics, the Pearson correlation tends to increase as the number of references increases. This tendency is more evident for ZH-EN and ENRU, while the effect is weaker for EN-DE. Furthermore, incorporating Quality-Aware Selection boosts performance in all directions. Notably, the impact of the ground truth on the system-level performance of n-gram-based metrics is consistently negative across all languages, so we only use the generated references.
For neural-based metrics, the Pearson correlation exhibits minor fluctuations. We assume that since BLEURT and COMET-20 already have strong performance (0.97 on ZH-EN and 0.95 on EN-RU), some low-quality reference candidates may be unbeneficial. In addition, neural-based metrics only calculate scores based on a single reference and then perform a basic max operation, which constrains their capability to use multiple references.
Segment-level results are similar to system-level. We list the detailed results in the Appendix C.

heading: 5.2 Effects of Prompt Variants
text: We investigate the impact of different prompts on the quality of the generated reference candidates. Commercial translation systems (Google and Microsoft) are also used to generate references. The results are summarized in Table 3. All experiments on prompt variants were conducted on the ZH-EN dataset. Notably, the number of generated reference candidates is constrained to 2 here and we will discuss its impact in Sec 5.3.
References generated by commercial translation systems do significant harm to system-level Pearson correlation (ρ), implying that commercial translation systems may not be able to produce highquality references to support multiple references. Regarding the choice of language for the prompt, the reference candidates generated by the English prompt slightly outperform the ones generated by the Chinese prompt. Furthermore, omitting the ground truth from the prompt leads to a 10 percentage point decrease in the Pearson correlation (ρ) of the evaluation metric. The rules (characterization and task-specific rules) in the prompt do not appear to have much impact. The Kendall correlation (τ )
at the segment level does not vary noticeably across methods.

heading: 5.3 Generate As Many References As Possible
text: In One Shot
It is worth noting that the number of generated translations specified in the prompt has a significant effect on the quality of the results. As shown in Figure 3, increasing the number of references from 2 to 30 leads to an 8 percent improvement in the Pearson correlation. We speculate that when LLMs generate more reference candidates at once, they strive to produce reference candidates with finer granularity, enabling more distinct discrimination between high and low-quality candidates. Consequently, the general quality of the reference candidates improves. We believe that this phenomenon can provide valuable insights for enhancing subsequent data synthesis from LLMs.

heading: 5.4 Addressing Data Leakage Issues through Multi-Reference
text: As previously discussed, data leakage can be a concern when using open-source LLMs, as it leads to an overestimation of the downstream task. We aim to alleviate this issue through multi-reference, as models that suffer from data leakage tend to produce nearly identical results to the ground truth,
resulting in suboptimal performance on other references. We verify this on the Flores-200 JapaneseChinese test set (Costa-jussà et al., 2022) as in preliminary experiments. The results are presented in Table 4.
When using a single reference, BLOOMz-7b exceeds all other systems in F200spBLEU including closed-source LLMs with hundreds of billions of parameters even without fine-tuning. The correlation of both F200spBLEU and BLEURT with Human Eval is weak, especially the Kendall score.
After using multiple references, the MT model and other closed-source LLMs in turn outperform BLOOMz-7b-ft. GPT-3.5-turbo also achieves comparable results to the MT model. And the correlation with Human Eval improves a lot for both F200spBLEU and BLEURT. F200spBLEU exhibits better agreements with Human Eval than BLEURT as it accurately measured some hard cases like the MT-ft-test.
Mt-ft-test is the MT model fine-tuned on the test set to simulate the data leakage problem. We observe that F200spBLEU accurately measures the model’s performance while BLEURT does not as it overestimates the MT-ft-test too much. We speculate the reason is that neural-based metrics rely on the semantic space to determine the similarity between translations. This inability to discriminate and penalize results in terms of diversity constrains the capability of neural-based metrics to mitigate the overestimation issue.
These observations suggest that using multireference with n-gram-based metrics can effectively mitigate the overestimation problem caused by data leakage, which is a pushing issue of opensource LLMs (Wei et al., 2023).
We also apply the direct assessment (DA) scoring method mentioned in GEMBA to evaluate results on the FLORES JA-ZH test set. The performance of GPT-3.5-DA is excellent and unaffected by data leakage from BLOOMz-7b.

heading: 5.5 Bias in Generated References towards the Original Model
text: One critical concern associated with the generated references is whether they exhibit a similar token distribution with the original LLMs and demonstrate any preferential biases when evaluating the original model. We conjecture that as the sentence is transformed into all possible formats, the bias towards the original model diminishes to a significant degree. To verify this assumption, we compute the
n-gram overlap for MT model translation, GPT-3.5turbo, and ground truth with generated references. The results are shown in Figure 4.
The scores of GPT-3.5-turbo are larger than the ground truth and MT model when the number of references is relatively small which can be attributed to the similar token distribution. As the reference number increases, the scores of ground truth exceed GPT-3.5-turbo, and the gap between MT and GPT-3.5-turbo becomes closer. This phenomenon validates our assumption that when the reference number increases, the bias gradually decreases.
Given that GPT-3.5-turbo is the model with the highest human evaluation scores, preventing us from observing reference bias, we employ an opensource model, Qwen-7B-chat (Bai et al., 2023), to further investigate this issue. We generate 10 references and list the results in Table 5. When using references generated by Qwen-7B-Chat, the score of itself remains at a reasonable level, without exhibiting excessively high values. Concurrently, the references produced by the open-source LLMs can still enhance the consistency between the metric and human evaluation to a certain extent.

heading: 5.6 Ablation Study
text: We further conduct some ablation experiments on both Accuracy Selection (AS) and Diversity Selection (DS) of QAS to validate their effectiveness. As shown in Figure 5, using QAS with only the lower bound or upper bound has a positive impact on the Pearson correlation. Combining them can obtain larger improvements, which means these two selection strategies are complementary to each other. The results confirm our hypothesis discussed in Sec 3.2.

heading: 5.7 The Effect of Ground Truth
text: In our experiments, we observed that the presence of ground truth exerts a negative impact on n-gram-based metrics, which aligns with previous studies (Freitag et al., 2020; Bawden et al., 2020). These studies identified that the presence of translationese in ground truth can diminish the correlation between evaluation metrics and human judgments. This phenomenon was consistently manifested across all three directions in our experiments, leading us to exclude ground truth from our experimental setup. The impact of ground truth is demonstrated as shown in Table 6.

heading: 6 Related Work

heading: 6.1 Automatic Evaluation Metrics
text: Based on the algorithm used, automated evaluation metrics can be divided into n-gram-based metrics and neural-based metrics. The former computes the
accuracy by matching the results generated by the model with the manually generated reference at the n-gram level (Papineni et al., 2002; Popović, 2015). The latter measures the similarity of sentences by learning on a large amount of text data with highdimensional semantic vectors (Sellam et al., 2020; Rei et al., 2022).
Previous research (Freitag et al., 2020) also investigates the impact of reference quality on automatic evaluation. They collect 2 more references with human annotators while we scale the reference number to 40 with LLMs.

heading: 6.2 Paraphrase Reference
text: Research has also been conducted on the generation of additional references through paraphrasing to enhance the accuracy of automatic evaluation (Kauchak and Barzilay, 2006; Zhou et al., 2006; Bawden et al., 2020). These methods predominantly employ paraphrase models to construct synthetic references. We compared the quality of references generated by PRISM (Bawden et al., 2020) and LLM-Ref, as illustrated in Table 7. The results indicate that the quality of references produced by LLM-Ref is significantly superior to those generated by PRISM. This is because such paraphrasers rely on beam search to generate multiple outcomes, but the beam search curse (Meister et al., 2020) leads to poor results when the beam size exceeds five, making these methods less effective, while LLM is not.

heading: 6.3 LLMs as Evaluators
text: The use of LLMs as evaluators have gained popularity in recent studies (Chiang and Lee, 2023; Wang et al., 2023), which leverages instructiontuned models like ChatGPT to score the results of various NLG tasks including machine translation. It has shown high agreement with human evaluation and has been applied to tasks such as machine translation, text summarization, and image captioning.
The approach closest to ours is one that exploits the paraphrasing ability of LLMs to generate multiple references to augment the consistency of automatic metrics (Tang et al., 2023). We list the results of WMT22 ZH-EN test set of both methods in Table 8. Our work imposes the task information to the generation process (for example the source sentence of translation and the translation rules in Sec 3.1) and proposes effective strategies to select high-quality references based on diversity and accuracy. We further emphasize the multiple references’ effect on the overestimation problem caused by data leakage. To our knowledge, this is the first attempt to address the data leakage risk issue in evaluating LLMs.

heading: 7 Conclusion
text: This work presents a study on the limited diversity of references for n-gram-based metrics and the overestimation issue when assessing LLMs that suffer from data leakage. We propose LLM-Ref, a framework that enhances alignments of machine translation evaluation metrics with human evaluation. By generating multiple reference candidates with LLMs and implementing the Quality-Aware selection mechanism, we effectively address these two challenges. Experiments on the WMT22 Metrics shared task including 4 languages demonstrate that LLM-Ref significantly improves the consistency of all automatic evaluation metrics with human assessments. Additionally, our analyses highlight the efficacy of n-gram-based metrics with multiple references in mitigating overestimation issue
due to data leakage in LLMs while neural-based metrics struggle to overcome. We explore the factors that affect the quality of generated references by LLMs and provide some insights for future data synthesis from LLMs.
Limitations
Our experiments primarily focused on GPT-3.5turbo which is a close-source model with hundreds of billion parameters. In future experiments, we plan to investigate the effect of smaller open-source models and generate hundreds of reference candidates.

heading: A Accuracy of Generated References
text: We calculate the BLEURT and COMET scores of the generated references. The generated results are reasonable and exceed the performance of the vast majority of MT systems as shown in Table 9.

heading: B Prompt Templates
text: Below we show our prompt templates which we use for the experiments to generate multiple reference candidates.
You are a professional translator specialized in translating Chinese to English. During the translation process, please follow the following principles: 1. Strictly adhere to the grammar and vocabulary usage norms of the target language, and do not add, delete, or adjust the language expression at will; 2. Maintain the meaning and tone of the original text in the translation, and try not to change the author's original intention; 3. Pay attention to the overall understanding and grasp of the context to ensure that the translation is complete, coherent, and fluent; 4. Pay attention to language style and localization in the translation, combined with the cultural background and language habits of the target readers, in order to make the translated text more natural and fluent.
The following is a difficult-to-translate Chinese sentence: {source_sentence}
Its human English translation is as follows: {ground_truth}
Please provide 40 high-quality English translations that differ from the above translation but adhere to the five principles of translation listed above, with the highest quality at the top and the lowest at the bottom, and one translation per line.

heading: C Segment Level Results of WMT22
text: Metrics Shared Task

heading: Future Work
text: We explore the factors that affect the quality of generated references by LLMs and provide some insights for future data synthesis from LLMs. Limitations
Our experiments primarily focused on GPT-3. 5turbo which is a close-source model with hundreds of billion parameters. In future experiments, we plan to investigate the effect of smaller open-source models and generate hundreds of reference candidates