BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training
Year: 2023

Abstract:
Automatic metrics play a crucial role in machine translation. Despite the widespread use of n-gram-based metrics, there has been a recent surge in the development of pre-trained model-based metrics that focus on measuring sentence semantics. However, these neural metrics, while achieving higher correlations with human evaluations, are often considered to be black boxes with potential biases that are difficult to detect. In this study, we systematically analyze and compare various mainstream and cutting-edge automatic metrics from the perspective of their guidance for training machine translation systems. Through Minimum Risk Training (MRT), we find that certain metrics exhibit robustness defects, such as the presence of universal adversarial translations in BLEURT and BARTScore. In-depth analysis suggests two main causes of these robustness deficits: distribution biases in the training datasets, and the tendency of the metric paradigm. By incorporating token-level constraints, we enhance the robustness of evaluation metrics, which in turn leads to an improvement in the performance of machine translation systems. Codes are available at https://github.com/ powerpuffpomelo/fairseq_mrt.

heading: 1 Introduction
text: Automatic metrics are crucial for the training of machine translation models, as they can measure translation quality at low cost. Currently, the most widely used translation evaluation metric is still the n-gram-based BLEU (Papineni et al., 2002; Marie et al., 2021). However, it is acknowledged that BLEU, which relies on the surface-level vocabulary matching, exhibits significant limitations (Smith et al., 2016; Reiter, 2018; Mathur et al., 2020; Kocmi et al., 2021). For instance, BLEU fails to differentiate between errors of varying severity and assigns equal weight to each word.
*Work was done during internship at ByteDance AI Lab. †Corresponding author.
In recent years, the advent of pre-trained models (Devlin et al., 2018; Liu et al., 2019; Conneau et al., 2019; Yang et al., 2019; Lan et al., 2019) has led to significant advancements in the development of metrics such as BLEURT (Sellam et al., 2020) and COMET (Rei et al., 2020), which employ pretrained language models (PLM) to assess the semantic meaning of sentences. These approaches have been shown to outperform metrics that rely on superficial word matching and have a more consistent correlation with human annotation. Despite these advances, it is important to note that neural metrics are characterized by opaque decision bases and may be subject to biases that are more difficult to detect (Sun et al., 2022). Therefore, we aim to conduct an analysis of the properties of various metrics in order to gain a deeper understanding. While there have been recent studies on the analysis of metrics (Kocmi et al., 2021; Hanna and Bojar, 2021; Sun et al., 2022), these works primarily focus on examining metric scores on specific datasets. To the best of our knowledge, this paper is the first to analyze metrics from the perspective of their guidance for training machine translation systems.
In this paper, we employ Minimum Risk Training (MRT) (Shen et al., 2015) to train translation
5428
models. Compared to Maximum Likelihood Estimation (MLE), MRT can reduce the gap between training and evaluation, resulting in higher quality translations (Shen et al., 2015; Edunov et al., 2017). In addition, since MRT uses metrics to optimize translation models, we can explore the impact of metrics on translation by observing the MRT training process.
Our experiment results show that MRT reveals the robustness defects in some metrics: the training collapses and the generated translations, despite getting high metric scores, show poor translation quality. For instance, we find universal adversarial translations of BLEURT and BARTScore, which are capable of obtaining high scores when evaluated against any reference sentence. An example is presented in Figure 1. Further analysis shows that the robustness defects are rooted in the distribution biases of the training corpora, as well as in the tendency of the metric modeling paradigm. In addition, we explore methods for optimizing metrics and translation models: word-level information constraints are introduced by combining MRT with NLL loss and metric ensemble.
Our main contributions are as follows:
• We present a systematic analysis of automatic metrics for machine translation from the perspective of guidance for training machine translation systems.
• We provide analytical conclusions, including metric robustness deficiencies, as well as an analysis of the underlying causes.
• We explore methods to improve metric robustness and translation quality and demonstrate their effectiveness.

heading: 2 Analyze Metrics with MRT
text: We train translation models in two stages: in the MLE training phase, the model is trained with con-
ventional negative log-likelihood (NLL) loss; then in the MRT training phase, we fine-tune the model with each metric, so as to obtain translation models with various metric styles. In this way, the characteristics of different metrics can be analyzed through observing the changes in the training process and the translation results.

heading: 2.1 Considered metrics
text: Given the translated sentence hyp, the automatic evaluation metric evaluates hyp by comparing it with the reference sentence ref (and sometimes with the source sentence src). This paper selects the most mainstream and cutting-edge six metrics for comparison and analysis, including three unsupervised metrics: BLEU (Papineni et al., 2002), BERTScore (Zhang et al., 2019), BARTScore (Yuan et al., 2021), and three supervised metrics: BLEURT (Sellam et al., 2020), COMET (Rei et al., 2020), UniTE (Wan et al., 2022). The specific information is shown in Table 1.
We use SacreBLEU1 and F1-score2 as a measure of text quality to calculate BLEU and BERTScore respectively. Following the instructions of Yuan et al. (2021), we use the CNNDM version of BARTScore3 to calculate the F1-score of ⟨hyp, ref⟩ for translate-to-English language pairs, and multilingual BART to obtain the faithfulness by calculating P (hyp | src) for the other language pairs. As recommended, we use BLEURT-204 and WMT20-COMET-MQM5 to compute BLEURT and COMET respectively. For UniTE, since our task is multilingual, we use UniTE-MUP6 in our experiments. It is worth noting that, for a fair comparison, we consider two input forms of
1https://github.com/mjpost/SacreBLEU 2https://github.com/Tiiiger/bert_score 3https://github.com/neulab/BARTScore 4https://github.com/google-research/bleurt 5https://github.com/Unbabel/COMET 6https://github.com/NLP2CT/UniTE
UniTE: one uses ⟨hyp, ref⟩ to calculate the translation quality, which we denote as UniTE_ref; the other uses ⟨src, hyp, ref⟩, which we denote as UniTE_src_ref.

heading: 2.2 Minimum Risk Training
text: Minimum Risk Training (MRT) is a sequence-level objective that aims to minimize the expected risk on the training data. Given a training set D = {(x,y)}, MRT uses the loss function ∆(ŷ, y) to compute the discrepancy between the ground truth y and the model prediction ŷ.
Different from conventional MLE training methods, MRT allows the use of arbitrary nondifferentiable loss functions. Therefore, automatic metrics can be introduced to train machine translation systems. While an MLE-trained model may not translate authentically, MRT can produce more natural translation results by reducing the gap between training and evaluation (Shen et al., 2015; Edunov et al., 2017; Wang and Sennrich, 2020).
In MRT training, risk is defined as the expected loss with respect to the posterior distribution:
R(θ) = ∑
(x,y)∈ D
∑
ŷ∈Y(x) P (ŷ|x; θ)∆(ŷ, y) (1)
in which Y(x) is the set of all possible translations of x. Since the full search space is intractable, we choose a certain number of candidate translations as a subset to approximate the posterior distribution.

heading: 2.3 Experiment Setup
text: Dataset With reference to datasets and language pairs that are widely used in machine translation and neural metrics studies, we conduct experiments on six language directions: English-German (En⇔De), English-Chinese (En⇔Zh), EnglishFinnish (En⇔Fi). We use the WMT14 training corpus for En⇔De, and the newstest13 and newstest14 are the validation set and the test set, respectively. For En⇔Zh, we use the LDC corpus as training data, and the NIST 2002, 2003 are used for validation, while NIST 2004, 2005, 2006 are
used as the test sets. For En⇔Fi, the datasets are from the training-parallel-ep-v8 and rapid2016 sections of WMT17, where the validation set and the test set are split at a rate of 0.1% respectively. The statistics of the datasets are shown in Table 2.
Implentation Details We train Transformer Base setting (Vaswani et al., 2017) using the fairseq7 toolkit, where the model consists of 6 layers of encoder and 6 layers of decoder with hidden size of 512. In the MLE training phase, the batch size is 65,536. The best checkpoint is selected based on the BLEU scores on the validation set. For evaluation, we average the last ten checkpoints and use beam search for inference. In the MRT training phase, each batch contains 8,000 tokens. Following previous work on MRT (Edunov et al., 2017), we use beam search to generate candidates, and the beam size is set to 12. The best checkpoint is selected based on the corresponding metric. We list the training duration for MLE and MRT in Appendix B. For all language pairs, sentences are encoded using byte pair encoding (Sennrich et al., 2015) with 32,000 merge operations, jointly learned from both the source and target side of the training data. We use Adam (Kingma and Ba, 2014) optimization and the same learning rate schedule as described in Vaswani et al. (2017) with the warmup step of 4,000.

heading: 2.4 Main Results
text: The MLE stage is the main factor in improving translation performance of the model, whereas MRT fine-tuning directs the model towards specific metrics. The SacreBLEU scores of the translation models after MLE training are shown in Table 3. Then in the MRT fine-tuning phase, we use six metrics separately on each language pair to guide the training. Figure 2 shows the evaluation results of the translations by optimizing each metric on Zh⇒En during this phase 8.
We investigate the changes in the MRT curve for each metric and language pair. The remaining of the metrics generally improve along with the optimized metrics, followed by a slight decrease, indicating that there are differences in the quality evaluation criteria of different metrics. In general, all metrics remain basically stable during the MRT
7https://github.com/pytorch/fairseq 8Due to space limitations, please refer to Appendix C for the complete graph of training states on each metric and language pair.
process. However, we find several exceptions, such as optimizing BLEURT on the En⇒De and En⇒Zh language pairs, where the rest of the metrics experience a severe drop. As shown in Table 4, BLEURT remains basically stable, but the rest of the metrics drop to particularly low or even negative values. The same situation occurs when optimizing BARTScore, as shown in Table 4 and Figure 2.
MRT Exposes the Robustness Defects of Metrics We find deficiencies in some metrics when MRT collapses. For example, we find that there are universal adversarial translations in both BLEURT and BARTScore.
(1) Universal translations of BLEURT. We take the checkpoint of the translation model on En⇒De where BLEURT reaches the highest point to generate translations on the test set. The decoded results show that the translation quality does indeed collapse severely. Table 5 shows the two most frequently decoded translations. It can be seen that the translation model generates many similar sentences with high frequency, regardless of the source sentences. This shows that decoding such sentences can get high BLEURT scores. The example of calculating the BLEURT scores of universal translations is also shown in Figure 1.
(2) Universal translations of BARTScore. We also generate translations with the checkpoint on De⇒En which gets highest BARTScore. As shown in Table 5, the translation model also decodes many
similar sentences, but unlike BLEURT, the form of the high-frequency decoded sentences is only repetition of simple words.
The phenomenon of universal adversarial translations shows that BLEURT and BARTScore are flawed, and a high metric score does not mean high translation quality. If the metric is not good enough, it actually leads the translation model in the wrong direction.

heading: 2.5 Analysis

heading: 2.5.1 Why Universal Translations Exist
text: We examine the WMT14 En⇔De parallel corpora, and find that there are many sentences with similar semantics in the training set, including a large corpus of hotel reviews that are semantically similar to universal translations of BLEURT 9. This indicates that the patterns of universal translations are related to the translation training set, and they come from the high frequency samples in the training corpora. Raunak et al. (2021) also mentions the problem of corpus bias, whose study on NMT hallucinations shows that specific noise patterns in the training corpora lead to specific hallucination patterns. Due to high frequency samples in the translation training set, it becomes easy for the translation model to decode certain sentences (even if they have nothing to do with the source sentences).
Moreover, the high score of the metric condones the model to decode such sentences, leading to
9Some examples can be found in Appendix E.
the creation of universal translations. BLEURT uses metric data and generates a large amount of pseudo-data for supervised training, and the metric data comes from the translation training corpus. Since data augmentation may introduce noise and amplify hallucinations (Raunak et al., 2021), we suggest that its indulgence of universal translations is also related to the training corpus.
The universal translations of BARTScore contain repetitions of simple words, which is similar to the hallucination phenomena that occurs in the early stages of translation model training. We not only use the F1 score, but also experiment with the Recall of BARTScore (computing P (ref |hyp)) to guide the training, and find that this setting can produce universal suffixes, that is, even if the correct translation is followed by a specific suffix, it does not reduce the BARTScore. Therefore, we suggest that the vulnerability of BARTScore may be due to the fact that it uses model generation probabilities to determine translation quality, and this generation-based metric tends to assign high scores to easily generated sentences. In short, the defects may stem from the tendency of the metric
modeling paradigm 10. The phenomenon of universal adversarial translations suggests that, on the one hand, we need to optimize the translation and metric datasets to balance their distributions, avoiding high-frequency samples; on the other hand, we need to optimize the metrics so that they are as little affected by the distribution bias of the dataset as possible. For example, sentence-level metrics can be constrained by incorporating word-level information. We present this experiment in Section 3.

heading: 2.5.2 Comparison of Metrics
text: We observe and compare the changes in the training effect of translation models guided by each metric on each language pair, and the summary is as follows:
BLEU converges quickly. This is as expected, since the translation model is selected by BLEU in the general MLE training phase, there is almost no continuous optimization during MRT. BERTScore also converges in a few steps. When BERTScore is optimized, other metrics remain relatively stable
10Metrics can be categorized into different modeling paradigms, including matching, regression, generation, and so on (Sun et al., 2022; Yuan et al., 2021).
and sometimes show an upward trend. The consistency between BLEURT and other metrics shows language pair differences: for translate-to-English language pairs, the other metrics change steadily and show high consistency with BLEURT. All three to-En language pairs show an increase in COMET, UniTE_ref, and UniTE_src_ref. However, on the language pairs that translate from English, the consistency becomes very poor, where the other metrics drop significantly when optimizing translation models with BLEURT.
The metric that is least consistent with other metrics is BARTScore. On all language pairs, the rest of the metrics decrease when BARTScore is used to train translation models.
COMET, UniTE_ref, and UniTE_src_ref are similar and can improve each other. However, when optimizing with these metrics, a decrease in BLEU is observed for all language pairs. This may indicate that the translation model is gradually trained to be more inclined towards translations that are semantically close to the reference sentences, but the specific words may not be the same. In addition, other metrics also show a smooth trend of change, indicating that these metrics may be superior and more robust.
Same Pre-trained Model Leads to Similar Metrics. We also find a pattern that metrics that are based on the same pre-trained model have similar
trends in the variation of the training effect of MRT. We count the pairwise correlation of each metric, and find that the correlation between BERTScore and BLEURT (both based on BERT), and the correlation between COMET and UniTE (both based on XLM-Roberta) are higher than other metric pairs for translate-to-English language pairs, as shown in Figure 3. For language pairs translated from English, the robustness bias of BLEURT weakens its correlation with BERTScore, but the Pearson correlation coefficient still reaches 0.82 and is significantly correlated. This indicates that metrics based on the same pre-trained model have more consistent criteria for the evaluation of translation quality.
Robust Metrics can Drive Improvement in Other Metrics. MRT experiments show that the optimization process of BARTScore as well as BLEURT (on translation-from-English language pairs) is accompanied by a strong decrease of the other metrics, and we find metric robustness deficits in these cases. Therefore, we suggest that robust metrics may drive other metrics to improve together during MRT. (However, the converse inference does not hold. The ability to drive other metrics to improve is not sufficient to conclude that the metrics are robust enough, because metrics may have common deficits that have not yet been discovered.)

heading: 3 Optimize Metrics and Translations
text: The analysis of the MRT training process allows us to understand the impact of each metric on translation quality. Our goal is both to exploit the advantages of the MRT training approach and to avoid training collapse due to the robustness deficiencies of the metrics.
MRT needs to sample many translation sentences in advance, and then use sentence-level metrics to predict the scores and calculate the loss. If the metrics that guide translation training do not take word-level information into account, the translation model may ignore details and gradually deviate during the training process. Therefore, we try two methods to constrain the training direction by introducing word-level constraints: combining MRT and NLL loss, and doing metrics ensemble.

heading: 3.1 Combine MRT and NLL Loss

heading: 3.1.1 Experiments
text: We take the fine-grained word-level similarity as a part of the objective function by incorporating the NLL loss, which computes the log loss for each token. We set the hyperparameter λMRT to control the weights. The formula is as follows:
L = λMRT ∗ LMRT + (1− λMRT ) ∗ LNLL (2)
We take the MRT training effect of the translation model optimized with BLEURT on En⇒De as an example to conduct experiments.

heading: 3.1.2 Results
text: The results are shown in Figure 4 (b). As can be seen, as the proportion of NLL loss increases, the decreasing trend of the remaining metrics gradually disappears.
The optimal result can be achieved when λMRT = 0.6 or 0.4. At this point, unsupervised metrics remain stable, and supervised metrics show an increase. This indicates that combining MRT and NLL loss can improve the training effect of
the translation model. For a fair comparison, we also check the results at the beginning of the optimization when using only MRT (before the training collapses). At this point, the improvement in BLEURT, COMET and UniTE is more obvious, but accompanied by a decrease in BLEU and BERTScore. This suggests that the inclusion of NLL loss can make training more stable and more balanced across all metrics.

heading: 3.2 Metrics Ensemble

heading: 3.2.1 Experiments
text: Supervised metrics focus more on high-level semantic similarity and are considered to have a higher correlation with human evaluation (Kocmi et al., 2021); while unsupervised metrics using word-level information are relatively stable and can ensure fine-grained text similarity 11.
We do an ensemble of different metrics in the hope that the integrated metrics can complement each other and integrate the advantages of different
11Note that although BARTScore is an unsupervised metric, it calculates the overall probability of sentence generation and still focuses more on sentence-level information.
metrics. Then the ensemble metric is applied to MRT training on En⇒De.

heading: 3.2.2 Results
text: Supervised and Unsupervised Metrics Ensemble. As can be seen in Figure 4 (a), optimizing BERTScore alone does not change the remaining metrics much, while only optimizing BLEURT reveals robustness problems. However, optimizing the ensemble of BERTScore and BLEURT works well: not only does it preserve the performance of the unsupervised metrics as much as possible, but it also leads to significant improvements in COMET and UniTE.
Supervised Metrics Ensemble. In addition, combining two sentence-level supervised metrics can also provide a boost, as the fifth column of Figure 4 (a) shows the effect of integrating BLEURT and COMET. Compared to optimizing only a single metric, we find that the ensemble metric can build on the strengths of both metrics. While maintaining the scores of unsupervised metrics, it can further improve supervised metrics. COMET and UniTE all improve about 14.5%, which is an increase of about 7 points. We suggest that this may be due to the fact that different metrics have different criteria for evaluating translation quality, and the robustness deficiency of one metric can be compensated by other metrics.

heading: 3.3 Method Validity Analysis
text: Avoid High-Frequency Decoding Sentences. We compare the entropy of decoded sentence frequencies on the En⇒De test set for the translation model trained with single or ensemble metrics. As shown in Table 6, the entropy is lower for the model trained with only BLEURT because it decodes a large number of identical sentences. While the frequency entropy for models trained with ensemble metrics is similar to that of the gold translations, indicating that the phenomenon of high-frequency decoded sentences disappears.
Comparison to MBR Decoding. Minimum Bayes Risk (MBR) decoding can also get translations with metric style (Freitag et al., 2022; Müller and Sennrich, 2021). Both MRT and MBR add some computational cost because they need to sample candidate translation sentences. However, MRT is a training process that can quickly generate translations at test time once the model has finished training. MBR, on the other hand, is a decoding
process, which requires more time for each decoding. Therefore, from an application point of view, MRT is more efficient.

heading: 4 Related Work
text: Automatic Metrics Traditional metrics for machine translation evaluation including BLEU (Papineni et al., 2002), METEOR (Lavie and Denkowski, 2009), and chrF (Popović, 2015) are based on lexical overlap. Embedding-based metrics measure the semantic equivalence between the reference and translation hypothesis by contextual representation, such as BERTScore (Zhang et al., 2019), MoverScore (Zhao et al., 2019). Generationbased metrics formulate the evaluation of text as a generation task, such as BARTScore (Yuan et al., 2021) and PRISM (Thompson and Post, 2020). The basic idea is that high quality text can be generated with high probability. Learned metrics, such as BLEURT (Sellam et al., 2020), COMET (Rei et al., 2020), and the recently proposed UniTE (Wan et al., 2022) aim to train neural networks to directly predict human judgements. These supervised metrics correlate well with human evaluations, but lack interpretability and robustness studies, which is explored by this paper.
Minimum Risk Training Shen et al. (2015) proposes the MRT method and confirms its superiority with experiments. Edunov et al. (2017) compares various objective functions and further verifies that MRT training can enhance translation quality. Wang and Sennrich (2020) uses MRT to avoid exposure bias, thus improving translation quality in out-of-domain settings. The above MRT work uses BLEU to guide the training of translation models, but BLEU is not the optimal metric. Our work uses various cutting-edge metrics to further improve translation quality. Wieting et al. (2019) proposes a new metric, claiming its superiority over BLEU and suitability for MRT training. Our work,
on the other hand, focuses on the analysis of metrics, with MRT serving as a tool to evaluate the robustness of various metrics systematically.
Metric Defects Analysis There are also some papers that start to explore the shortcomings of metrics. Sai et al. (2021) provides perturbation templates to measure the performance of metrics on the constructed challenge set, while our work is to guide the metrics to generate adversarial samples (universal translations) by themselves. Amrhein and Sennrich (2022) does a case study on COMET through MBR decoding, showing that COMET is insensitive to numbers and named entities. Different from a pure case study, our work shows the tendency of metrics through MRT, and can draw more typical conclusions. Sun et al. (2022) shows that PLM-based metrics, such as BERTScore, lack fairness and exhibit higher social bias than traditional metrics. Our work analyzes metrics from a robustness perspective and complements this work.

heading: 5 Conclusion
text: In this paper, we present the first systematic analysis of automatic metrics from the perspective of guidance for training machine translation systems. We find that MRT reveals the robustness deficiencies of some metrics, such as universal adversarial translations of BLEURT and BARTScore, and we further analyze the underlying causes. In addition, we explore methods to improve metric robustness, thus helping to further enhance the performance of translation systems.
Limitations
First, we find robustness deficiencies in metrics by comparing the evaluation differences among metrics. This applies to the case when there are metrics that do not have the same robustness flaws. If there are more latent common defects in the metrics, they cannot be identified by MRT. We leave this topic for future research.
Second, we use beam search to generate candidates during MRT training, but beam search is also known to have deficiencies. For example, beam search suffers from heuristic search biases and shifts statistics away from those of the data (Eikema and Aziz, 2020). Different decoding methods may have an impact on the experiment results.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their insightful comments. Shujian Huang is the corresponding author. This work is supported by National Science Foundation of China (No. 62176120), the Liaoning Provincial Research Foundation for Basic Research (No. 2022-KF-2602).

heading: A Ethics Statement
text: This paper finds universal adversarial translations that can be used to attack metrics and lead to security risks. However, this paper also proposes methods to improve metric robustness to avoid this situation.

heading: B Training Duration for MLE and MRT
text: We list the training duration for MLE and MRT in Table 8 and Table 9, respectively. Table 8 shows the number of training epochs, while Table 9 shows the number of training steps. It can be seen that the training duration for MRT is much shorter than that for MLE. The improvement of translation performance of the model mainly lies in the MLE stage, while MRT fine-tuning makes the model inclined towards specific metrics.

heading: C MRT Training Process Figures
text: From Figure 5 to Figure 9, we can see how all metrics change when the translation model is optimized with each metric on different language pairs. From the trends of different metrics, we can observe the differences between the metrics and the impact of the metrics used for optimization on the translation model.
In each figure, the horizontal axis represents the training steps, and the vertical axis is the score of each metric (except for BARTScore on the right axis, which is a negative number because it calculates the logarithmic probability of translations); metrics other than BARTScore and BLEU are mostly distributed between 0 and 1, and we multiply them uniformly by 100 for ease of observation. The asterisk represents the highest value achieved by the optimized metric.

heading: D MRT Training Process Statistics
text: Table 10 to Table 14 display the change range in all metrics when optimizing the translation model with a specific metric to the highest point across different language pairs. The results correspond to figures in Appendix C. 0.00% means that the optimized metric does not continue to improve, and the highest value remains the same as the result of MLE training; a negative number means that the metric score goes from positive to negative, which means it decreases a lot.

heading: E High Frequency Samples
text: Table 7 displays some hotel review examples in the WMT14 En⇔De dataset, and the semantics are very similar to universal translations of BLEURT on En⇒De. For ease of understanding, English is shown here.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitation is provided after secion 6.
3 A2. Did you discuss any potential risks of your work? Potential risks are provided after secion 6 in Limitation part.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract is provided in the very beginning of the article and introduction is provided in section 1.
3 A4. Have you used AI writing assistants when working on this paper? "Deepl write"(https://www.deepl.com/write) is used as AI writing assistant in this paper. "Deepl write" assistances purely with the language of the paper. Deepl write is used in section 1, 2, 3 of the paper
B 3 Did you use or create scientific artifacts? Scientific artifacts are used in section 2. Scientific artifacts are created in section 3.
3 B1. Did you cite the creators of artifacts you used? References part(at the end of the paper) cite the creators of artifacts used in this paper. In addition, URLs are provided in section 2.1
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In section 2, we report relevant statistics
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Computational experiments are in section 2, 3.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Not applicable. Left blank.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? We discuss the experimental setup in section 2.3
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We report descriptive statistics about results and report mean results. They are in section 2.4, 3.1.2, 3.2.2
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We report detail of used existing packages in section 2.3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave this topic for future research. Second, we use beam search to generate candidates during MRT training, but beam search is also known to have deficiencies. For example, beam search suffers from heuristic search biases and shifts statistics away from those of the data (Eikema and Aziz, 2020). Different decoding methods may have an impact on the experiment results