Breeding Machine Translations: Evolutionary approach to survive and thrive in the world of automated evaluation
Year: 2023

Abstract:
We propose a genetic algorithm (GA) based method for modifying n-best lists produced by a machine translation (MT) system. Our method offers an innovative approach to improving MT quality and identifying weaknesses in evaluation metrics. Using common GA operations (mutation and crossover) on a list of hypotheses in combination with a fitness function (an arbitrary MT metric), we obtain novel and diverse outputs with high metric scores. With a combination of multiple MT metrics as the fitness function, the proposed method leads to an increase in translation quality as measured by other held-out automatic metrics. With a single metric (including popular ones such as COMET) as the fitness function, we find blind spots and flaws in the metric. This allows for an automated search for adversarial examples in an arbitrary metric, without prior assumptions on the form of such example. As a demonstration of the method, we create datasets of adversarial examples and use them to show that reference-free COMET is substantially less robust than the reference-based version.

heading: 1 Introduction
text: Attaining good translation quality in machine translation (MT) arguably relies on good automatic metrics of MT quality. Recently, a new generation of evaluation metrics was introduced. These metrics are based on embeddings computed by large pretrained language models and human annotation scores. The improvements in metric quality resulted in renewed interest in metric-driven translation hypothesis selection methods, like Minimum Bayes Risk (MBR) decoding (Goel and Byrne, 2000; Kumar and Byrne, 2004).
Our method relies on MBR decoding and the genetic algorithm (GA; Fraser, 1957; Bremermann, 1958; Holland, 1975. Through combinations and mutations of translations produced by an MT model, we search for optimal translation under a
selected metric. This is a novel approach to generating translation hypotheses in NMT.
We find that by combining neural and surface form-based metrics in a GA’s fitness function, it is possible to create better quality translations than by simple reranking of the initial hypotheses (as evaluated by held-out metrics). It also allows the combination of multiple sources for the translation, for example, MT, paraphrasing models and dictionaries.
Another use-case for our method is the identification of weak points in MT metrics. Flaws and biases of the novel neural metrics are being studied, for example, by Hanna and Bojar (2021), Amrhein and Sennrich (2022a), Alves et al. (2022) or Kanojia et al. (2021). In summary, these metrics have low sensitivity to errors in named entities and numbers. Also, they are not sufficiently sensitive to changes in meaning and critical errors, like negations.
These previous works on deficiencies of the metrics mostly focus on analyzing the outputs of MT systems and looking for certain types of mistakes. Another approach they use is changing the outputs to introduce specific types of mistakes. In contrast, our approach aims to find translations with high scores on certain metrics automatically, by optimizing the candidate translations for a selected metric. We believe that through this more explorative approach, it is possible to find unexpected types of defects.
In summary, the main contribution of our work is a novel method for producing translations, which can be used to improve translation quality and analyze automatic MT evaluation metrics.1

heading: 2 Related work
text: Automated MT evaluation The traditional automatic MT metrics are based on comparing a trans-
1Source code at https://github.com/cepin19/ga_mt
2191
lation produced by an MT system to a human reference based on a string similarity. Popular choices are ChrF (Popović, 2015) and BLEU (Papineni et al., 2002). Multiple shortcomings of these metrics are well known (Callison-Burch et al., 2006; Bojar et al., 2010; Freitag et al., 2020; Mathur et al., 2020a; Zhang and Toral, 2019; Graham et al., 2020).
Neural MT metrics Novel, neural-based MT metrics were introduced recently. They address some of the deficiencies of the string-based methods, but possibly introduce new types of errors or blind spots: BERTScore (Zhang et al., 2020), BARTScore (Yuan et al., 2021), PRISM (Thompson and Post, 2020), BLEURT (Sellam et al., 2020), COMET (Rei et al., 2020, 2021, 2022), YiSi (Lo, 2019), RoBLEURT (Wan et al., 2021) or UniTE (Wan et al., 2022b).
Using a shared embedding space, these metrics better compare source, translated, and reference sentences. Their evaluation in WMT Metrics tasks (Mathur et al., 2020b; Freitag et al., 2021b, 2022) and other campaigns (Kocmi et al., 2021) demonstrate stronger agreement with human judgment.
While their system-level performance has been scrutinized, their segment-level performance remains less explored. Moghe et al. (2022) indicates these metrics are unreliable for assessing translation usefulness at segment level. However, we still try to optimize individual sentences for improved scores.
Deficiencies in metrics The closest work to ours is Amrhein and Sennrich (2022a). Authors use MBR decoding to find examples of high-scoring, but flawed translations in sampled model outputs. The conclusion is that the studied metrics are not sensitive to errors in numbers and in named entities (NE). Alves et al. (2022) automatically generate texts with various kinds of errors to test for sensitivity of MT metrics to such perturbations. Sun et al. (2020) claim that current MT quality estimation (QE) models do not address adequacy properly and Kanojia et al. (2021) further show that meaningchanging errors are hard to detect for QE.
Genetic algorithm Variations of the genetic algorithm and evolutionary approaches in general for very diverse optimization problems are being studied extensively for more than half a century (Fraser, 1957; Bremermann, 1958; Sastry et al., 2005).
Nevertheless, work on the utilization of the GA
in machine translation is scarce. Echizen-ya et al. (1996) use GA for example-based MT. Zogheib (2011) present multi-word translation algorithm based on the GA. Ameur et al. (2016) employ GA in phrase-based MT decoding. In the context of neural machine translation, GA was used to optimize architecture and hyperparameters of the neural network (Ganapathy, 2020; Feng et al., 2021).
Minimum Bayes risk decoding Our implementation of the fitness function depends on Minimum Bayes Risk (MBR) decoding (Goel and Byrne, 2000; Kumar and Byrne, 2004). This selection method has regained popularity recently as new, neural-based MT metrics emerged (Amrhein and Sennrich, 2022b; Freitag et al., 2021a; Müller and Sennrich, 2021; Jon et al., 2022).

heading: 3 Proposed solution
text: Our approach depends on two methods: Minimum Bayes Risk decoding and genetic algorithm.

heading: 3.1 Genetic algorithm
text: We propose the use of a GA to find new translation hypotheses. GA is a heuristic search algorithm defined by a fitness function, operators for combination (crossover) and modification (mutation) of the candidate solutions, and a selection method.
Before running the GA algorithm, an initial population of a chosen number of candidate solutions is created. A single solution is called an individual, and it is encoded in a discrete way (often as a list) by its forming units, genes. The resulting representation of an individual is called a chromosome. All chromosomes have the same length to simplify the corssover operation, but we add placeholders for empty tokens to account for additions, as discussed later.
The algorithm itself consists of evaluating each solution in the population using the fitness function and stochastically choosing parent solutions for the new generation by the selection algorithm. Crossover is used on the chromosomes of the parents to create their offspring (children). The mutation is used on the children and they form a new generation of the same size. This is repeated for a given number of iterations (generations).
In our proposed method, the candidate solutions are translation hypotheses produced by an MT model. Genes are tokens and the mutation operation replaces, deletes, or adds a token in a chromosome. The eligible new tokens are chosen
from a set of valid tokens. We discuss methods of construction of this set in Section 4.6.
To allow for variable lengths of the solutions and the add or delete operations, we add genes representing an empty string after each token gene, and all the candidates are also right-padded with the empty string genes. The final length of all the candidates is equal to the length of the longest candidate multiplied by a constant k. The empty string genes can be mutated to a non-empty gene, which is equivalent to inserting a new token into the candidate. Inversely, a non-empty string gene can be mutated to an empty string gene, which is equivalent to removing a token. Empty genes have no influence on the fitness score. Below we show the encoding of two translation hypotheses for k = 1.1: sent1=['Genetic','','algorithm','','can','','be','','used','', 'to','' ,'produce','','novel','','solutions','','.','','','']}
sent2=['Genetic','','algorithm','','creates','','new','', 'solutions','','.','','','','','','','','','']}
Fitness function Fitness functions are MT evaluation metrics, see Section 4. For some of the experiments, the fitness function is composed of multiple metrics. In that case, the scores are simply summed – we did not explore scaling them or using multi-objective GA (Murata et al., 1995; Surry et al., 1997; Gao et al., 2000; Deb et al., 2002).
Selection To select parents for the new generation, we use tournament selection with n = 3. For each individual in the population, two other individuals are randomly chosen and the one with the best value of the fitness function out of the three is selected as one of the parents for a new generation. Figure 1 illustrates this, including the fact that many individuals can be selected repeatedly through this process.
Crossover operation We iterate through the parents by pairs, each pair is crossed-over with probability c. A random index i in a chromosome is selected and two children are created, the first one inherits the part of chromosome up to i from the first parent and the part from i from the second parent and vice-versa for the second offspring. For parents p1 and p2 and children c1 and c2:
c1=p1[:i]+p2[i:]; c2=p2[:i]+p1[i:]
Mutation operation The children produced by the cross-over operation are mutated. Each gene (token) is mutated with a probability m. Mutation
replaces the token (or empty string placeholder) with a randomly selected one from the set of all possible tokens. This set also includes empty string placeholder, which is equivalent to token deletion. The approaches to the construction of this set are described in Section 4.6. After the mutation phase, the new generation is ready and the next iteration of GA is performed. One iteration of the whole GA process is illustrated in Figure 1.
MT Metrics and Fitness vs. Evaluation Optimizing the word composition of a translation towards an arbitrary metric is subject to Goodhart’s law – once a metric is used as a goal to optimize towards, it ceases to be a good measure of final quality (Strathern, 1997). Thus, we cross-evaluate with held-out metrics not used for optimization (even though these metrics might still be linked with the optimization metrics by spurious correlations caused by similar metric design, model architecture, or training data). We search for adversarial examples for the specific metrics, i.e. translations scoring high in the objective metric, but low in heldout metrics. This can be used to create training sets of negative examples. We use ChrF, BLEU, wmt20comet-da (Rei et al., 2020), wmt20-comet-qe-da-v2 as the objective metrics and wmt21-comet-mqm, eamt22-cometinho-da, BLEURT (Sellam et al., 2020) and UniTE (Wan et al., 2022a) as the heldout metrics.

heading: 3.2 MBR decoding
text: NMT models predict a probability distribution over translations for a given source sentence. A common method for selecting a final translation given this distribution is known as “maximum-a-posteriori" (MAP) decoding. Because of the computational complexity of exact MAP decoding, approximations such as beam search (Koehn et al., 2003) are used. Many limitations of MAP were described recently (Stahlberg and Byrne, 2019; Meister et al., 2020) and other approaches were proposed.
One of the alternatives is MBR decoding. It is a decision rule that selects the translation based on a value of a utility function (and thus minimizes expected loss, or risk) rather than model probability. MT metrics are often used as utility functions. In an ideal case, we have a distribution p(y|x) over all possible correct translations y of source sentence x available, which is not the case in real-world scenarios. Given the space of all possible target language sentences H(x) and utility function U ,
we search for the optimal translation h∗: h∗ = argmaxh∈H(x)Ep(y|x)[U(y, h)] A fixed number of translation hypotheses produced by the MT model can be used as an approximation of the reference translations distribution p(y|x) in practice. Still, the number of possible hypotheses H(x) is infinite – it consists of all conceivable sentences in the target language. For this reason, the same set of translations as for references is also used as candidate hypotheses. This leads to an implementation where MBR decoding can be seen as consensus decoding – a translation that is the most similar to all the other translations in the set is selected. Some of the recent embedding-based metrics also take the source sentence into account. In that case, utility is defined as U(x, y, h). In such cases, the process is no longer equivalent to consensus decoding due to the influence of the source.

heading: 4 Experiments
text: This section describes our experimental setup and results. We compare reranking of n-best lists to the application of the GA on them.

heading: 4.1 Data
text: We trained Czech-English MT model on CzEng 2.0 (Kocmi et al., 2020), a mix of parallel data
(61M) and Czech monolingual data back-translated into English (51M). For experiments with dictionaries, we use a commercial Czech-English dictionary. We use newstest-19 (Barrault et al., 2019) as the dev set and newstest-18 (Bojar et al., 2018) as the test set. Due to the high computational requirements of our approach, we only evaluate the first 150 sentences from the test set in all the experiments. We call this test set newstest-18-head150. We used a commercial lemmatizer.2 for lemmatization and word form expansion performed in some of the experiments, We tokenize the data into subwords with SentencePiece (Kudo and Richardson, 2018) and FactoredSegmenter.3

heading: 4.2 Model
text: We train transformer-big using MarianNMT (Junczys-Dowmunt et al., 2018) with default hyperparameters.

heading: 4.3 Hardware
text: We ran all the experiments on a grid server with heterogeneous nodes, with Quadro RTX 5000, GeForce GTX 1080 Ti, RTX A4000, or GeForce
2http://www.lingea.com 3https://github.com/microsoft/
factored-segmenter
RTX 3090 GPUs. The running time depends on population size, number of generations, and fitness function. We leave the first two fixed, so the computational requirements are most influenced by the fitness function. For the most computationally intensive fitness (combination of wmt20-comet-da and wmt20-comet-qe-da-v2), optimizing 150 examples on RTX A4000 takes 5 days. We discuss the computational requirements in Section 9.

heading: 4.4 Metrics
text: We abbreviate some of the longer metrics’ names further in the text in order to save space.4
For BLEU and ChrF we use SacreBLEU (Post, 2018). We use β = 2 for ChrF in all the experiments (i.e. ChrF2). For COMET5, BLEURT6 and UniTE7 scores we use the original implementations. We use paired bootstrap resampling (Koehn, 2004) for significance testing.

heading: 4.5 GA parameters
text: We did not search for optimal values of GA parameters due to high computational costs. The initial population is formed by 20-best hypotheses obtained by beam search and 20 sampled ones, copied 50 times over to obtain a population size of 2000. We select parents for the new generation with tournament selection (n = 3) and then we combine them using a crossover rate c = 0.1. The mutation rate for the mutation of non-empty genes to different non-empty genes m is 1/l, where l is the chromosome length. For mutating empty to non-empty gene (word addition) or vice-versa (deletion), the rate is m/10. We run 300 generations of the GA.

heading: 4.6 Possible mutation sources
text: We consider three possible sources for the mutation tokens set, i.e. the set of tokens that can replace another token in the chromosome:
1) init – set of all the tokens from the initial population (only tokens that are present in initial hypotheses can be used for the optimization).
2) dict – we performed word-by-word dictionary translation of each the source sentence, resulting in a set of English tokens. The source
4CMT20 (wmt20-comet-da), CMT21 (wmt21-cometmqm), CMTH22 (eamt22-cometinho-da), QE (wmt20-cometqe-da-v2), BLEURT (BLEURT-20), UniTE (UniTE-MUP)
5https://github.com/Unbabel/COMET 6https://github.com/google-research/bleurt 7https://github.com/NLP2CT/UniTE
sides of the dictionary and the source sentence are lemmatized for the search, and target token forms are expanded to cover all surface forms.
3) wordlist – all words from an English wordlist.8

heading: 4.7 Results
text: Reranking We translated newstest-18 by the baseline model using beam search with beam size 20. We also sampled another 20 translation hypotheses for each source sentence from the model. We rerank these lists by BLEU, ChrF and CMT20 metrics in two manners: either with knowledge of the true manual reference (i.e. oracle) or using MBR decoding. GA is not used in these experiments. There are two ways of using multiple references with BLEU: either compute single-reference scores for all the references separately and average them or use the multi-reference formula. We use the former.
The results are presented in Table 1. The confidence ranges are shown in Appendix C, Table 10. The 1st column shows the origin of the hypotheses.9 The 2nd column shows if the reference was used for reranking (Oracle), or the other hypotheses and MBR decoding were used instead (MBR). No reranking (-) means that the candidate with the highest model’s length-normalized log-prob is evaluated. The 3rd column indicates which metric was used for the reranking (the objective function). The remaining columns are the values of the evaluation metrics (computed with respect to the reference).
For most of the metrics, MBR-reranked hypotheses outperform the log-prob baseline, even though by a smaller margin than the referencereranked (oracle) ones. In some cases, optimizing with MBR towards one metric leads to a deterioration of scores in other metrics. The metrics most prone to this problem are QE, ChrF and BLEU. MBR rescoring with QE results in worse ChrF, BLEU and CMTH22 scores than the baseline, suggesting this metric is unsuitable for such application. CMT20 and especially the combination of CMT20+QE+BLEU are more robust, with the latter improving in all the metrics over the baseline. As shown further, both the negative and positive
8https://github.com/dwyl/english-words 9The outputs produced with beam size 5 are not used in further experiments, they are shown for comparison to account for the beam search curse (larger beam sizes sometimes result in worse translation outputs, Koehn and Knowles, 2017).
effects are more pronounced with GA. Reranking with knowledge of the reference is unsurprisingly performing better than MBR reranking. Here, we use it to show the upper bound of improvements attainable by reranking. In further experiments, reference-based GA is also used to analyze the objective metrics.
We also notice that while reranking beam search results leads to better final outcomes than reranking sampling results, a combination of both provides the best scores. All further experiments start with a population consisting of this combination of both.
Genetic algorithm We use the same metrics for GA fitness function as for reranking. Experiments were again conducted with either the knowledge of the reference or with MBR decoding. The results for GA with reference are presented in Table 2 (confidence ranges in Appendix C,S Table 11). The first two columns indicate the metric used as the fitness function and the source of the possible tokens for the mutation. The third column shows how many runs were averaged to obtain the mean scores shown in the remaining columns. The last column shows the ratio of the final selected hypotheses that were not in the initial pool produced by the MT model, but were created by GA operations.
We see that the GA can optimize towards an arbitrary metric better than simple MBR reranking. For example, the best ChrF score for GA is 87.1 compared to 65.4 for reranking. The results also suggest that the string-based metrics (ChrF and BLEU) are prone to overfitting – translations optimized for these metrics score poorly in other metrics. CMT20 is more robust – we see improvements over the baseline in all the metrics after optimization for CMT20.
Table 4 presents the results of the experiments aimed to improve the translation quality (confidence ranges for the scores are in Appendix C, Table 12). The reference is not provided and MBR decoding (always computed with regard to the initial population) is used instead. This way, it is feasible to use the approach to improve translations in a real-world scenario with no reference. We measure the improvement by held-out metrics.10 We consider UniTE to be the most trustworthy. It was created most recently and some of the flaws of the other metrics were already known and mitigated. It also correlates well with human evaluation (Freitag et al., 2022) and it is developed by a different team than the COMET metrics, which slightly decreases the chances for spurious correlations of the scores
10CMT21, CMTH22, BLEURT and UniTE
not based on translation quality. The metrics that only compare the translation with a reference (BLEU, ChrF) without access to the source sentence do not perform well as a fitness function. Since MBR decoding in such cases works as a consensus decoding, i.e. the most similar candidate to all the others has the best fitness, there is no evolutionary pressure to modify the individuals.
Optimizing for QE or ChrF results in a large decline in scores for other metrics. These metrics are prone to scoring malformed, nonsensical or unrelated sentences well. This is analyzed in Section 5. The sum of QE, CMT20 and BLEU as the fitness function reaches the best score in UniTE and does not show significant degradation in other metrics.
The ratio of examples where held-out scores improve, decrease or do not change after GA is shown in Table 3. We compare the scores both to log-prob selected hypotheses and MBR reranked ones. We again see that the combination of CMT20+QE+BLEU performs best. GA with the individual metrics as the fitness function leads more often to a decrease than an increase of heldout metrics compared to reranking. This suggests the effect of GA on the translation quality is negative if the fitness function is not chosen well.

heading: 5 Analysis
text: In this section, we analyze the GA procedure and the behavior of evaluation metrics.

heading: 5.1 GA process
text: Fitness vs. held-out metric We analyzed the behavior of the average fitness function over the whole population, best solution fitness, and heldout metric score during the GA process using CMT20+QE+BLEU as the fitness and UniTE as the held-out metric (Figure 2). Results show GA
consistently improved fitness values from initial solutions and increased average fitness. However, the correlation between fitness and held-out metrics varied: Example a) shows a decrease in final heldout score despite improved fitness, while Example b) shows aligned increases in both scores. Table 3 suggests case b) is more typical in our test set.

heading: 5.2 Search for adversarial examples
text: As a radically different goal, we use GA to search for examples that score high in the fitness function but are evaluated poorly by held-out metrics. This allows us to find blind spots in specific metrics without previous assumptions about the type of errors that could be ignored by the given metric. Such adversarial examples are defined as follows: for each test set example e, we compute the scores of the hypotheses produced by the MT model using both the optimization metric O and the held-out metric H . We rank the hypotheses by O. The scores of the best hypothesis are referred to as O(e)init and H(e)init. We then use a GA to optimize the hypotheses towards O. We consider the final translation as adversarial for a given metric if its score
O(e)ga improves by at least a margin mo over the initial O(e)init and at the same time H(e)ga decreases by at least mh compared to the H(e)init. In other words, e is adversarial if:
O(e)init+mo < O(e)ga∧H(e)init > H(e)ga+mh In search of adversarial examples, it is beneficial to explore a large space of hypotheses. Thus, we use all words from the wordlist for mutations.
Since the goal is to optimize the output towards a given metric to find its flaws, not to improve translation in a real-world scenario, we can assume we have the reference translations at hand and we can use them to compute the fitness scores.
We demonstrate the approach on two optimization metrics (CMT20 and QE) and one held-out metric (UniTE). We set mh = mo = 10−3. We present the results on newstest-18-head150 in Table 5. The first column shows which optimization metric was used and the second column shows the number of examples for which the final opti-
mization score improved upon the initial best score. The last column shows how many of the improved examples had decreased scores for the held-out metric. We show examples in Appendix A.
We observed QE is less robust than CMT20. Completely unrelated sentences are scored better than an adequate translation. Upon an inspection of the examples, we see that the QE metric prefers adding spurious adjectives and named entities (NEs). This could be caused by a length bias, or by a preference for more specific utterances. QE scores very unusual words highly and it scores punctuation low. For instance, Sentence 4 from Appendix A, Table 6 has a correct initial translation “Model was killed by chef.". After optimizing for QE, the translation becomes “Model Kiranti Tarkio killed by molluscan stalkier".
Changing or adding NEs can be observed also for CMT20 (Sentences 2, 5 and 8 in Appendix A,Table 7), although in a much smaller extent. This shows that even though QE and CMT20 correlate similarly with human evaluation on wellformed translations (Rei et al., 2021), QE is more prone to scoring nonsensical translations higher than adequate ones. This observation is also supported by the decline of other metrics when optimizing QE in Table 4.
In another experiment with QE we tried to construct a completely unrelated translation, convey-
ing a malicious message, which would score better than the original MT output by the QE metric. We present these examples in Appendix B.

heading: 6 Discussion
text: We agree that an argument could be made that our approach is very computationally expensive, too explorative and the search for weaknesses could be performed in a more principled way. However, by anticipating the types of errors the metrics ignore and by designing the procedure to create texts with such errors, some of the error types can remain unnoticed. We see analogies with the whole field of deep learning. The methods with more priors of what the outcome should look like and how an inductive bias should be represented in a model give way to more general architectures as systems are scaled both in parameters and training data size, in the spirit of Richard Sutton’s Bitter Lesson.11
Since the architectures of systems that produce evaluation scores are based mostly on empiric results, rather than on solid theoretical approaches,
11http://www.incompleteideas.net/IncIdeas/ BitterLesson.html
we believe that similar empirical, almost bruteforce methods, might be an effective tool to search for weaknesses of these systems.

heading: 7 Conclusions
text: We present a method of using a GA to find new translations based on optimizing hypotheses from an n-best list produced by an MT model. Our method optimizes well towards an arbitrary MT metric through modification of the candidate translations. We found that after optimizing for a single objective metric, scores on other metrics often decrease, due to over-fitting on the objective metrics’ defects. We discover that by combining multiple metrics (both neural and string-based) in the fitness (objective) function, we are able to mitigate the over-fitting and improve or maintain the held-out metrics for most inputs. This suggests GA can be used to improve MT quality.
MT evaluation metrics have specific flaws and blind spots. To test their robustness, we selected some of the metrics as the fitness functions to optimize towards, and others as held-out metrics. We have leveraged the over-fitting effect to search for adversarial examples for specific metrics, creating translations that score high in one metric and low in held-out metrics. Such translations can be used as negative examples for improving the robustness of the neural metrics.
This work also reveals that even though source-translation and source-translation-reference COMET scores were shown to have a similar correlation with human scores for well-formed translations, the reference-free COMET is more susceptible to adversarial inputs.This highlights the necessity of thorough analysis, beyond computing correlation with human scores for the new metrics.

heading: 8 Acknowledgements
text: This work was partially supported by GAČR EXPRO grant NEUREM3 (19-26934X) and by the Grant Agency of Charles University in Prague (GAUK 244523). We used the data and computing resources provided by the Ministry of Education, Youth and Sports of the Czech Republic, Project No. LM2018101 LINDAT/CLARIAH-CZ. We would also like to thank Dominik Macháček and Dávid Javorský for proofreading the text of the paper.

heading: 9 Limitations
text: Due to the high computational costs of the method, we tested it only on a very small set of sentences and larger-scale experiments are needed to confirm the results.
Many parameters of the GA algorithm were left unexplored – the results could be improved by grid search over the values for mutation and crossover ratios, using a better list of mutation candidates (for example based on k-NN search), experimenting with different selection methods, combining more metrics in the fitness function or using multiobjective GA like NSGA-II (Deb et al., 2002).
In the experiments concerning held-out metrics, we assumed weaknesses of the held-out metrics are not correlated to the weaknesses of the optimization metrics, which is probably not true, due to similar model architectures and training datasets. This means that held-out metrics are not strictly independent, but we believe combining multiple different held-out metrics should mitigate this issue.

heading: 10 Ethics
text: In some settings, automated MT evaluation metrics are used to decide whether the MT output should be presented to the client, or further processed by a human post editor. We present a method that uses genetic algorithms to create adversarial examples for MT evaluation metrics. The potential use of such adversarial examples raises ethical concerns, particularly in the context of machine translation applications that impact human lives, such as in medical, legal, financial or immigration contexts. We acknowledge that our work raises ethical questions regarding the potential misuse of adversarial examples. For instance, adversarial examples could be used to deceive or manipulate users by providing machine translations that are misleading or incorrect. Moreover, they could be used to create biased translations that reflect certain views or opinions. We believe that it is important to address these ethical concerns and to ensure that our work is not used for unethical purposes. As such, we recommend further research into the development of defense mechanisms against adversarial examples and into the identification of ethical and legal frameworks that can guide the use and development of adversarial examples for MT evaluation metrics. We also suggest that future work includes an explicit discussion of ethical implications and considera-
tions in the context of adversarial examples for MT evaluation metrics. Metrics are sometimes used to verify translations to be shown to the client. Our work can be used to generate adversarial examples.

heading: A Examples of adversarial translations
text: We ran GA with initial hypotheses generated by MT and permitted the words to be mutated by any word from an English wordlist to find a solution with the best fitness function. Tables 6 to 8 show examples of the produced translations for QE, CMT20 and BLEU as the fitness function. Here, we cherry-picked the examples with interesting phenomena, the whole datasets are available at https://github.com/cepin19/ga_mt. For QE (reference-free COMET), we see that often, the metric prefers translations where adverbs and adjectives are spuriously added to make the utterance more specific. It is often a very rare or unusual word. We plan to further analyze whether this is caused by a length bias (it is possible QE prefers longer translations), or by a preference for more specific translations, without regard to the specificity of the source. We also see that punctuation is almost always omitted in the output as if it played no role in translation quality.
For CMT20 (reference-based COMET), the artifacts are similar, but to a much smaller extent. Some of the named entities are replaced, which confirms the low sensitivity of COMET to NE errors. For punctuation, we see the opposite effect from QE in some examples – instead of no punctuation, CMT20 sometimes prefers double punctuation, for example in Sentence 6 in Table 7.

heading: B Creating intentionally false translations
text: We consider a scenario where QE is used in a pipeline to control the output quality and decide whether to assume the MT output is correct as it is. As shown by Sun et al. (2020) and Kanojia et al. (2021), current QE models are not sensitive to shifts in the meaning of the translation. We experiment with our method to inject fake information into the translation or reate completely unrelated MT output so that it would nevertheless pass the output quality check. We constructed an arbitrary message: "The Adversarial LLC company is the best choice for investment, send the money to our bank account.". We used ChatGPT (Jan 9 2022 version)
to construct 40 utterances conveying this message with this prompt: Please generate 40 diverse paraphrases for this sentence: "The Adversarial LLC company is the best choice for investment, send the money to our bank account.". We used this list as the initial population for the GA a we ran the GA for the first 150 sentences in newstest-18. We only allowed usage of tokens from these sentences for the mutations (we referred to this as init configuration earlier). The goal of this process is to create examples that convey the malicious message and are scored better than the original MT output.
We found 13 such examples out of 150 sentence pairs. We present some of them in Table 9.

heading: C Significance scores and confidence ranges
text: We use bootstrap resampling with n = 100000 to compute 95% confidenece ranges for Tables 1, 2 and 4 in Tables 10 to 12, respectively. the results are in format mean score [95% confidence range]. We also provide p-values for compari-
son between MBR reranking and GA with MBR scoring as the objective function in Table 13. We show that in UniTE and COMET22 (wmt22-cometda), GA performs significantly better (p < 0.01) than reranking. However, CMTH22 and BLEURT scores are better for reranking.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
7 A2. Did you discuss any potential risks of your work? Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 7
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Left blank.
3 B1. Did you cite the creators of artifacts you used? Left blank.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work