# Réduction des répétitions dans la Traduction Automatique Neuronale

Marko Avila, Anna Rebollo, Josep Crego

► To cite this version:

Marko Avila, Anna Rebollo, Josep Crego. Réduction des répétitions dans la Traduction Automatique Neuronale. 35èmes Journées d'Études sur la Parole (JEP 2024) 31ème Conférence sur le Traitement Automatique des Langues Naturelles (TALN 2024) 26ème Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (RECITAL 2024), Jul 2024, Toulouse, France. pp.198-210. [hal-04623017](https://hal.inria.fr/hal-04623017)

HAL Id: hal-04623017

https://inria.hal.science/hal-04623017v1

Submitted on 1 Jul 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Réduction des répétitions
dans la Traduction Automatique Neuronale

Marko Avila Anabel Rebollo Josep Crego

4 rue du Port aux Vins, F-92 150 Suresnes, France

{mavila, arebollo, jcrego}@chapsvision.com

## RÉSUMÉ

Actuellement, de nombreux systèmes TAL utilisent des décodeurs neuronaux pour la génération de textes, qui font preuve d'une capacité impressionnante à générer des textes approchant les niveaux de fluidité humaine. Toutefois, dans le cas des réseaux de traduction automatique, ils sont souvent confrontés à la production de contenu répétitif, également connu sous le nom de diction répétitive ou de répétition de mots, un aspect pour lequel ils n'ont pas été explicitement entraînés. Bien que cela ne soit pas intrinsèquement négatif, cette répétition peut rendre l'écriture monotone ou maladroite si elle n'est pas utilisée intentionnellement pour l'emphase ou des fins stylistiques. La répétition de mots a été traitée par des méthodes post-hoc pendant l'inférence, contraignant le réseau à examiner des hypothèses auxquelles le système avait initialement attribué une plus faible probabilité. Dans cet article, nous implémentons une méthode qui consiste à pénaliser les répétitions lors de l'apprentissage et qui s'inspire des principes du label smoothing. Conformément à cette méthode, nous modifions la distribution de la vérité terrain afin d'orienter le modèle de manière à décourager ces répétitions. Les résultats de nos expériences montrent que les méthodes proposées permettent de contrôler le problème de la répétition dans les moteurs neuronaux de traduction automatique sans compromis en termes d'efficacité ou de qualité des traductions.

## ABSTRACT

### Reducing Repetitions in Neural Machine Translation

Many contemporary NLP systems rely on neural decoders for text generation, which demonstrate an impressive ability to generate text approaching human fluency levels. However, in the case of neural machine translation networks, they often grapple with the production of repetitive content, also known as repetitive diction or word repetition, an aspect they weren't explicitly trained to address. While not inherently negative, this repetition can make writing seem monotonous or awkward if not used intentionally for emphasis or stylistic purposes. Repetitions have been addressed through post-hoc methods during inference, compelling the network to consider hypotheses it initially assigned lower probability. In this paper, we implement a repetition penalty method applied at learning inspired by the principles of label smoothing. In line with label smoothing, we modify the ground-truth distribution to steer the model towards discouraging repetitions. Experiments show the ability of the proposed methods in reducing repetitions within neural machine translation engines, without compromising efficiency or translation quality.

**MOTS-CLÉS** : Traduction automatique neuronale ; Génération de texte ; Répétitions.

**KEYWORDS**: Neural machine translation ; Text generation ; Repetitions.
# 1 Introduction

This study addresses the issue of word repetition in machine translation, which involves the repeated occurrence of words, phrases, or ideas throughout the translation process. More specifically, we focus on repetitions that typically occur when translating synonyms or semantically equivalent phrases found in the source sentence. These repetitions lead to diminished readability of the text, potentially causing boredom or confusion for the reader and creating the perception of verbose or awkward writing. Consider, for instance, the following French sentence :

— nous avons **lutté** contre l'infodémie en **combattant** les mythes par des informations fiables. and its two corresponding English translations :

— We have **combated** the infodemia by **combating** myths with reliable information.

— We have **fought** the infodemia by **combating** myths with reliable information.

Since both translations are grammatically correct and semantically equivalent, the first one is clumsy due to word repetition<fnref n="1" />, while the second one effectively avoids repetition by suggesting alternative translations *fought* and *combating*, for the French words *lutté* and *combattant*, resulting in a smoother and more preferable translation.

In neural machine translation, repetition often arises when the model faces input sentences containing synonyms, leading these synonymous terms to be translated into identical words. Although lacking numerical support for our observation, the repetition issue becomes more salient when utilizing a model trained across multiple domains, highlighting the dearth of lexical diversity. We attribute this phenomenon to a lack of diversity in the decoder module. Although this type of repetition may occur with low frequency, it is highly concerning as it vividly illustrates the lack of fluency in translations.

However, repetitions do not always have a negative impact on readability. Without aiming to be exhaustive : i) repetitions play a role when summarizing information or reinforcing a concept; ii) common expressions are formed using word repetitions, and altering them to eliminate repetition would alter their intended meaning; iii) in highly specialized domains, expressions convey precise meanings that disallow being reformulated. The following examples illustrate these observations :

i) once closed, the door stays closed

ii) over and over; to be or not to be; step by step

iii) the congenital muscular dystrophy in newborns presenting with muscular hypotonia

As previously introduced, finding suitable alternatives without altering the meaning of a sentence can be a challenging task. In this work, we propose a method applied in training designed to teach the model to discourage certain repetitions, thereby alleviating the need for difficult decisions during inference. Next, we summarize the main contributions of this work :

— We propose a method that discourages repetitions during the training phase by adjusting the ground-truth distribution so as to penalize repetitions more severely.

— We introduce a technique for gathering examples containing both, acceptable repetitions and repetitions that hinder fluency, which are then utilized during the training phase.

— We build a curated test set that includes various types of word repetitions found in machine translations. Evaluation on this test set provides deeper insights into the repetitions issue.

Repetitions can manifest in various forms, including single words, phrases, and larger segments of content. However, this work concentrates on repetitions manifested through the repetition of linguistic

<footnote n="1">1. Note that repetitions can diminish readability, even when they occur as inflectional variants. Is the case of the repeated words in our example, *combated* and *combating*.</footnote>
words, which are more commonly observed in machine translations. Note that linguistic words are
typically decomposed into multiple tokens as taken into account by neural networks.

## 2 Related Work

The fluency levels achieved by LLMs are widely acknowledged to be high, primarily owing to the extensive availability of monolingual datasets, which surpasses that of standard neural machine translation (NMT) models trained solely on parallel texts. To the best of our knowledge, no dedicated research has been conducted on addressing the repetition issue tackled in this work within NMT systems. Closely related, (Welleck et al., 2019) describe a method to train neural language models that in addition to maximizing likelihood to model the overall sequence probability distribution, also includes an unlikelihood term in the loss function to correct known biases such as repeated tokens. (Li et al., 2020) use the same approach to control copy effect and repetitions observed in dialogue tasks. (Su et al., 2022) present a contrastive solution to encourage diversity while maintaining coherence in the generated text.

Various studies have addressed diversity in neural MT systems, which is a closely related topic. Sampling predictions from the output distribution can be an effective decoding strategy for back-translation, as described by (Edunov et al., 2018), or sampling from less likely tokens (Holtzman et al., 2020). Results show that such techniques enlarge diversity and richness of the generated translations when compared to data generated by beam or greedy search, but introduce semantic inconsistency in translations. In (Lin et al., 2022) is proposed a multi-candidate optimization framework for augmenting diversity. The authors propose to guide an NMT model to learn more diverse translations from its candidate translations based on reinforcement learning. During training, the model generates multiple candidate translations, of which rewards are quantified according to their diversity and quality.

A different approach attempts to condition the decoding procedure with diverse signals. Typically, (Shu et al., 2019) use syntactic codes to condition the translation process. (Lachaux et al., 2020) replace the syntactic codes with latent domain variables derived from target sentences. Similarly, (Schioppa et al., 2021) use prefix-based control tokens and vector-based interventions for controlling output translations from a NMT system. In the context of paraphrase generation (Vahtola et al., 2023) propose a translation-based guided paraphrase generation model that learns useful features for promoting surface form variation in generated paraphrases.

## 3 Adjusting the ground-truth distribution

Throughout the training process, at every time-step $t$, neural machine translation networks generate predictions over the target-side vocabulary based on the input $x$ and previous predictions $y_{<t}$:

$$p_{t}^{i} = p(y_{t}^{i} | x, y_{<t}), i \in [1, \dots, V]$$

where $V$ indicates the size of the target vocabulary.

The loss function evaluates the neural network's capacity to model the training data by comparing
<table><thead><tr><th>t</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th></tr></thead><tbody><tr><th>r</th><td>1</td><td>like</td><td>cookies</td><td>and</td><td>cookies</td><td>.</td></tr><tr><td></td><td colspan="6" style="text-align: center;">M</td></tr><tr><td>.</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><th>I</th><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><th>and</th><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><th>like</th><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><th>cookies</th><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td></tr></tbody></table>

FIGURE 1 – Matrix $\mathcal{M}$ for the ground-truth $r$ = 'I like cookies and cookies.' . Rows $t$ and $r$ represent respectively the time-step and the corresponding ground-truth token. A reduced model vocabulary (matrix rows) is used to facilitate reading.

its predictions to a reference target vector $r = [r_1, r_2, ..., r_T]$, where $T$ denotes the sequence length. This loss is utilized to update the network’s parameters, aiming to minimize the observed error in the model. The loss at time-step $t$ is usually computed as the cross-entropy between the model predictions $p_t = [p_t^1, ..., p_t^V]$ and the ground-truth distribution $q_t = [q_t^1, ..., q_t^V]$ :

$$ \mathcal{L}_t = - \sum_{i=1}^{V} q_t^i \log(p_t^i) \qquad (1) $$

Note that the vector $q_t$ is a one-hot encoding representation of $r_t$, with all entries set to 0 except for the token indicated by $r_t$, which is set to 1. Addressing the over-fitting risk illustrated by the previous $q_t$ distribution, label smoothing (Szegedy et al., 2015; Müller et al., 2019) (LS) is widely employed to achieve a smoother distribution :

$$ q_t^{\epsilon^{LS}} = (1 - \epsilon)q_t + \frac{\epsilon}{V} \qquad (2) $$

with $\epsilon$ being a commonly small hyper-parameter.<fnref n="2" />

LS can be interpreted as penalizing the probability of the ground-truth class by a factor of $1 - \epsilon$, while evenly distributing the removed probability mass among all classes, $\epsilon/V$. Building upon a strategy akin to label smoothing, we make additional adjustments to the ground-truth distribution and reduce the likelihood of repeated tokens, with the goal of enabling the model to learn to predict repetitions with lower probability. We introduce a matrix, denoted as $\mathcal{M}_{V \times T}$, which indicates whether the ground-truth token $r_t$ is also present in the preceding time-steps.<fnref n="3" /> Figure 1 illustrates an example of matrix $\mathcal{M}$ with ground-truth I like cookies and cookies. as translation of the French sentence J'aime les cookies et les biscuits. with a model vocabulary of 5 tokens (matrix rows). Both French terms cookies and biscuits are correctly translated into English as cookies, yet this choice clearly reduces the fluency and clarity of the translation. As it can be seen, only $\mathcal{M}_{[i=5, t=5]}$ is set to 1 since only $r_5$ = 'cookies' occurs in a preceding time-step ($t = 3$).

We consequently update the ground-truth distribution following :

$$ q_t^{\epsilon^{LS} \alpha \mathcal{M}} = (1 - \epsilon)(1 - \alpha \mathcal{M}_t) q_t + \frac{\epsilon}{V} \qquad (3) $$

<footnote n="2">2. $\epsilon = 0$ yields the initial distribution $q_t$, whereas $\epsilon = 1$ implies a uniform distribution.</footnote>

<footnote n="3">3. Note that repetitions are computed over words while matrix $\mathcal{M}$ refers to tokens $r \in V$ for each time-step $t \in T$.</footnote>
<figure><img src="image_2.png" /><figcaption>FIGURE 2 – Ground-truth distributions for the 5^th^ time-step of our example : the original one-hot encoding q; adjusted with label smoothing q^{ϵLS} ; and further adjusted with repetitions q^{ϵLαM} .</figcaption></figure>

where α is a hyper-parameter, and αM is used as a penalty, much like ϵ in the case of LS. Note that only the label smoothing probabilities discounted are distributed among all classes. As a result, time-steps with repeated tokens (such as t = 5 in our example) do not constitute proper probability distributions, as their sum does not add to 1. Figure 2 illustrates ground-truth distributions for our example at time-step t = 5 : the original one-hot encoding q ; the original distribution adjusted using label smoothing q^{ϵLS} , and further adjusted using repetitions q^{ϵLSαM} .^4 A significant challenge with the aforementioned techniques that modify q distribution with repetitions is their limited impact on the training process, primarily caused by the scarcity of repeated tokens in datasets. In the following section, we present alternative approaches to address this challenge.

# 4 Gathering Examples with Repetitions

As previously depicted, our intention is to instruct the model to minimize certain repetitions while preserving others deemed necessary for an accurate translation. To achieve this, we must compile a relatively large dataset of examples that demonstrate this behavior to the model. We initially focus on repetitions of content words such as nouns, adjectives, verbs, and adverbs. Function words, which serve a distinct grammatical role in a sentence, are excluded from this analysis. Current MT networks reliably generate these words based on their understanding of grammatical correctness.

Given that training corpora contain a relatively small number of repetitions, and these are manually curated, often comprising only acceptable repetitions, we opt to focus solely on repetitions found in machine translations. Accordingly, we translate the source sentences (src) of our training examples to generate synthetic translations (hyp). Repetitions that detrimentally impact fluency are only considered if they appear in such translation. Identifying examples containing repetitions following the previous morpho-syntactic patterns is straightforward. However, the challenge lies in discerning which repetitions degrade the fluency of translations and which do not. We follow the next filtering steps to select repetitions degrading fluency :

- We first word align the source (src) and synthetic (hyp) sentences and eliminate repetitions in the synthetic sentences that also align with repeated words in the source sentence. Word alignments are performed by the Giza++ (Och & Ney, 2003) toolkit^5. This approach is

<footnote n="4">4. As previously discussed, distribution $q^{\epsilon LS\alpha M}$ does not form a proper distribution since probabilities do not add to 1 (0, 02 + 0, 02 + 0, 02 + 0, 02 + 0, 0092 = 0, 0892). We leave for future experiments the normalization of the output scores in order to allow for a valid probability distribution.</footnote>

<footnote n="5">5. https://github.com/moses-smt/giza-pp.</footnote>
<table><caption>TABLE 1 – Two synthetic translations containing repeated words (underlined) and their corresponding source (hyp) and reference translation (tgt) sentences. Shades of grey indicate word alignments between repeated words and their alignments in the src/tgt sentences.</caption><thead><tr><th></th><th>Degrading</th><th>Acceptable</th></tr></thead><tbody><tr><th>src</th><td>The home was <u>modest</u> and <u>frugal</u></td><td>I want to know what you <u>mean</u></td></tr><tr><th>hyp</th><td>La maison était <u>modeste</u> et <u>modeste</u></td><td>Je veux savoir ce que tu <u>veux</u> dire</td></tr><tr><th>tgt</th><td>La maison était <u>modeste</u> et <u>économique</u></td><td>Je veux savoir ce que tu <u>veux</u> dire</td></tr></tbody></table>

based on the premise that if repeated words are necessary in a human-generated sentence, the corresponding translation may similarly require the repetition. This holds true for repetitions used to reinforce a concept or in highly specialized domains.

- Next, we also align the synthetic (hyp) and target (tgt) sentences word by word and remove repetitions of the synthetic sentences (hyp) aligned to repeated words in the reference translation. The same previous rationale remains consistent, now encompassing examples of common expressions that necessitate word repetitions.

The resulting set of examples with repetitions from src/hyp training pairs will be regarded as instances that the model needs to learn to discourage. Consequently, we utilize them for training after annotating the repeated words in their respective $M$ matrices. Table 1 illustrates the procedure previously outlined for two synthetic translations (hyp) containing repetitions.

The repeated word on the left-side example is the French adjective *modeste* while the verb *veux* is repeated in the right-side example. Once word alignments are computed between synthetic translations and their respective source and target sentences, the example on the right is not classified as a repetition degrading fluency. This is because both instances of *veux* are aligned with repeated words in the target reference translation, suggesting that the repetition is motivated. Concerning the example on the left side, neither the source nor the reference target sentences contain repeated words, suggesting that the repetition stems from a lack of diversity when translating *modest* and *frugal*, thus hindering fluency.

We employ the left-side example to train the model to identify it as a repetition to be avoided. The corresponding matrix $M$ marks the second occurrence of the word *modeste* with a 1, thereby incurring in a significant loss if the model predicts it with high probability. The example on the right is used without penalization, thus instructing the model to reproduce the repetition.

It's worth noting that the presented approach does not require any alterations to the network architecture and maintains the same training and inference efficiency.

# 5 Experimental Framework

We evaluate the proposed methods in an English-to-French translation task. Thus, we utilize English-French parallel corpora freely obtained from the Opus website$^6$. We strive for balanced utilization across various domains and ensure the inclusion of clean parallel data whenever possible. Due to the extensive volume of French-English parallel sentences accessible we randomly choose a subset exceeding 7 million examples that we employ as *Training set*.

For testing, we make use of English-French *News-test* (2008 to 2013) datasets made available
<table><thead><tr><th rowspan="2">Type</th><th colspan="2">Training set</th><th colspan="2">Repetition-test</th></tr><tr><th>Degrading</th><th>Acceptable</th><th>Degrading</th><th>Acceptable</th></tr></thead><tbody><tr><td>Noun</td><td>170, 169</td><td>356, 105</td><td>25</td><td>34</td></tr><tr><td>Verb</td><td>36, 111</td><td>41, 949</td><td>24</td><td>33</td></tr><tr><td>Adjective</td><td>22, 834</td><td>51, 599</td><td>26</td><td>30</td></tr><tr><td>Adverb</td><td>4, 016</td><td>6, 097</td><td>26</td><td>1</td></tr><tr><td>Total</td><td>233, 130</td><td>455, 750</td><td>101</td><td>98</td></tr></tbody></table>

TABLE 2 – Number of repetitions that degrade fluency and those which do not, found in both the Training and Repetition-test corpora. Occurrences are also displayed considering the morpho-syntactic function of repetitions.

through the WMT’2014 translation shared task<fnref n="7" />. In addition, we use a held-out Repetition-test composed of reference English and their corresponding French machine translations that feature at least one repeated word on the target (French) side for a more nuanced analysis of repetition. Machine translations were obtained with our baseline NMT model (referred in Appendix B as baseline). The Repetition-test set primarily serves to assess our models’ performance in handling repetition issues, while we employ the News-test set to evaluate overall translation accuracy. Further details of the train and test datasets used are given in Appendix A.

We translate the English side of the Training and Repetition-test sets following the procedure detailed in Section 4 to identify repetitions which hinder fluency (Degrading) and those which do not (Acceptable). Table 2 displays the frequency of repetitions identified within the French translations. The table presents the occurrence of both types of repetitions, accompanied by an analysis of their frequency concerning the morpho-syntactic function of the repetitions<fnref n="8" />. Sentences are morpho-syntactically analyzed using the spaCy<fnref n="9" /> toolkit.

Our NMT model is built using an in-house implementation of the state-of-the-art Transformer architecture (Vaswani et al., 2017). Details of the network and training work are given in Appendix B.

# 6 Results and Analysis

To evaluate the methods presented in this paper we consider the previous baseline model that we update with 15K additional iterations for two different configurations of the ground-truth distribution:

$q^{\epsilon^{LS}}$ follows the same configuration than the baseline model with label smoothing set to $\epsilon = 0.1$.

$q^{\epsilon^{LS\alpha M}}$ further penalizes the ground-truth distribution with repetition penalties as detailed in Section 3 with $\epsilon = 0.1$ and for different values of $\alpha$.

Note that for both configurations, we use 7.6M reference sentence pairs detailed in Table A (Training set) together with the synthetic translations containing repetitions predicted Degrading and Acceptable of Table 2, summing up to 7.6M + 233K + 455K sentence pairs. It’s essential to find a balance between the number of sentences in each training set (reference and synthetic) to uphold overall quality while teaching the model to minimize specific repetitions.

<footnote n="7">7. https://www.statmt.org/wmt14/translation-task.html</footnote>

<footnote n="8">8. Only French adverbs ending with suffix *ment* are considered.</footnote>

<footnote n="9">9. https://spacy.io/ with the French `fr_core_news_lg` model.</footnote>
<table><caption>TABLE 3 – Translation accuracy results and number of repetitions present in translations performed by models under different configurations. Two different test sets are considered. $ε$ is always set to 0.1.</caption><thead><tr><th rowspan="2">Configuration</th><th colspan="4">Repetition-test</th><th colspan="2">News-test</th></tr><tr><th>BLEU</th><th>COMET</th><th>Degrading</th><th>Acceptable</th><th>BLEU</th><th>COMET</th></tr></thead><tbody><tr><td>q<sup>εLS</sup></td><td>45.15</td><td>37.36</td><td>99</td><td>95</td><td>32.60</td><td>26.50</td></tr><tr><td>q<sup>εLSαM</sup>, 1 − α = 10<sup>−2</sup></td><td>45.54</td><td>39.10</td><td>81</td><td>89</td><td>32.45</td><td>26.05</td></tr><tr><td>q<sup>εLSαM</sup>, 1 − α = 10<sup>−3</sup></td><td>45.63</td><td>40.31</td><td>79</td><td>87</td><td>32.44</td><td>26.18</td></tr><tr><td>q<sup>εLSαM</sup>, 1 − α = 10<sup>−6</sup></td><td>45.65</td><td>40.13</td><td>77</td><td>86</td><td>32.50</td><td>26.30</td></tr><tr><td>beam, β = 0</td><td>35.03</td><td>31.28</td><td>0</td><td>0</td><td>20.66</td><td>10.65</td></tr><tr><td>beam, topk = 10</td><td>44.34</td><td>37.88</td><td>94</td><td>85</td><td>32.45</td><td>25.91</td></tr><tr><td>GPT3.5</td><td>29.70</td><td>29.59</td><td>25</td><td>43</td><td>29.98</td><td>27.60</td></tr><tr><td>NLLB</td><td>34.13</td><td>25.37</td><td>51</td><td>57</td><td>31.98</td><td>23.64</td></tr></tbody></table>

Configuration *beam* employs the *baseline* model and performs inference following two strategies to reduce repetitions and improve diversity:

* $β$ with a penalty applied at each inference time-step $t$ whenever token $y_t$ appears repeated in the hypothesis prefix $y_0, ..., y_{t-1}$. Probability $p_t$ is reduced by factor $0 ≤ β ≤ 1$. Thus, reducing the likelihood of such hypotheses.

* *topk* Sampling predictions from the $k$ most likely tokens of the output distribution. This is an effective decoding strategy typically used for increasing diversity when building back-translation datasets.

We also assess the effectiveness of two large language models (LLM) with translation capabilities to overcome the repetition issue:

* *GPT3.5* consists of the *GPT3.5-turbo* version of the OpenAI LLM. Built upon the Generative Pre-trained Transformer architecture (Radford & Sutskever, 2018) which employs only a transformer decoder. Following an auto-regressive approach, the model ensures that the generated text maintains coherence and relevance to the context provided by the input text. Translations are conducted using the OpenAI API, while emphasizing the importance of minimizing word repetitions through the provided prompt<fnref n="10" />.

* *NLLB* is a family of machine translation models based on the Transformer encoder-decoder architecture, enabling translation between any of the 202 language varieties (NLLB Team et al., 2022). We use the *nllb-200-distilled-600M*<fnref n="11" /> version and perform translations with the efficient CTranslate2¹² inference toolkit.

To evaluate the presented methods, we report BLEU and COMET results computed by *sacrebleu*<fnref n="13" /> (Post, 2018) and *comet-score*<fnref n="14" /> (Rei et al., 2020) respectively over both test sets. Concerning *Repetition-test*, we also report the number of word repetitions that hinder fluency, *Degrading*, and those deemed acceptable, *Acceptable*, measured in translation hypotheses.

<footnote n="10">10. Prompt = *Translate the following text from English to French, ensuring that the translated output maintains coherence and fluency while minimizing the repetition of words or phrases. Pay attention to using synonyms, varied sentence structures, and appropriate linguistic devices to enhance the overall quality of the translation. Feel free to creatively adapt the language to achieve a natural and engaging tone in the target language. I want you to only reply the traduction, do not write explanations*</footnote>

<footnote n="11">11. https://huggingface.co/facebook/nllb-200-distilled-600M</footnote>

<footnote n="12">12. https://github.com/OpenNMT/CTranslate2</footnote>

<footnote n="13">13. https://github.com/mjpost/sacrebleu</footnote>

<footnote n="14">14. https://github.com/Unbabel/COMET</footnote>
Models fine-tuned from the *baseline* network exhibit nearly identical quality scores across the *News-test* sets. This suggests that training with the method presented to adjust the ground-truth distribution does not compromise translation quality. On the contrary, unlike Configuration $q^{\epsilon^{LS}}$, Configurations $q^{\epsilon^{LS\alpha M}}$ demonstrate a significant decrease in the number of repetitions that degrade fluency over the *Repetition-test*, while retaining most of the acceptable repetitions in the translated output. Note also the increase in quality over the *Repetition-test* set as measured by COMET score (~40 > 37.62). Adjusting $\alpha$ does not seem to have a significant impact on reducing repetitions that degrade fluency. Decreasing its value gradually (lightly) reduces the occurrence of such repetitions. As expected, the number of acceptable repetitions remains unchanged since the training input signal with acceptable repetitions remains constant across all $\alpha$ values.

Regarding inference-based configurations, *beam* with $\beta = 0$ effectively eliminates all repetitions but at the expense of a notable decrease in translation quality, in the case of *topk* = 10 the number of repetitions is lightly reduced as well as global accuracy.

Results from both LLMs demonstrate a reduced number of repetitions, suggesting an elevated level of diversity and fluency of such models. However, the translation quality scores of LLMs do not align with those achieved by the models presented in this study in either of the test sets, especially translations obtained by GPT-3.5. These findings are consistent with those presented by (Bawden & Yvon, 2023) where the authors note the challenge of controlling translations performed by BLOOM<sup>15</sup>, a multilingual LLM.

Table 4 illustrates reference translations (src and tgt) together with translations by models $q^{\epsilon^{LS}}$ and $q^{\epsilon^{LS\alpha M}}$. The first (top) examples exhibit the ability of model $q^{\epsilon^{LS\alpha M}}$ to avoid *degrading* repetitions. The last examples contain *acceptable* repetitions hypothesized by both models.

# 7 Conclusions and Further Work

We have introduced a method to reduce the occurrence of repetitions in translation hypotheses, which significantly affects the readability of the generated texts. Additionally, we have proposed a straightforward approach to identify repetitions in machine translations that detract from fluency. The method is solely implemented during fine-tuning at the conclusion of the training phase, without any modifications to the inference process. Experiments indicate the ability of our proposed methods in reducing the repetition problem. Additional experiments are necessary to confirm the applicability of the proposed methods across various language pairs and dataset conditions. We aim to further study the impact of the ratio between the number of reference sentences and synthetic translations that include repetitions during the training process. Additionally, we plan to analyze the influence of the distance (measured in number of words) between repetitions and explore the possibility of replacing the binary penalty in matrix *M* with a softer approach.

# Remerciements

The work presented in this paper was supported by the EU Horizon 2020 Programme for TRACE project (Grant Agreement No. 101022004).

<footnote n="15">15. https://huggingface.co/bigscience/bloom</footnote>
<table><caption>TABLE 4 – Models configured with ($q^{LSαM}$) and without ($q^{LS}$) penalization exhibit varying performance when encountering repetitions (outlined using blue). Reference source (src) and target (tgt) translations are also indicated. The initial (top) examples include repetitions that *degrade* fluency, whereas the final (bottom) examples feature *acceptable* repetitions.</caption><tr><td>src</td><td>(h) liabilities, including unliquidated obligations;</td></tr><tr><td>tgt</td><td>h) les dettes, y compris les engagements non réglés;</td></tr><tr><td>q<sup>LS</sup></td><td>(h) les engagements, y compris les engagements non réglés;</td></tr><tr><td>q<sup>LSαM</sup></td><td>(h) les passifs, y compris les engagements non réglés;</td></tr><tr><td>src</td><td>We talked about the tourism, hospitality and hotel sectors.</td></tr><tr><td>tgt</td><td>On a parlé des secteurs du tourisme, de l'hébergement et de l'hôtellerie.</td></tr><tr><td>q<sup>LS</sup></td><td>Nous avons parlé des secteurs du tourisme, de l'hôtellerie et de l'hôtellerie.</td></tr><tr><td>q<sup>LSαM</sup></td><td>Nous avons parlé des secteurs du tourisme, de l'accueil et de l'hôtellerie.</td></tr><tr><td>src</td><td>The home was modest and frugal.</td></tr><tr><td>tgt</td><td>C'était une maison modeste.</td></tr><tr><td>q<sup>LS</sup></td><td>La maison était modeste et modeste.</td></tr><tr><td>q<sup>LSαM</sup></td><td>La maison était modeste et économe.</td></tr><tr><td>src</td><td>Courts will tackle the question anyway, often obliquely or indirectly.</td></tr><tr><td>tgt</td><td>Les tribunaux vont se poser cette question de toute façon, souvent à mots couverts ou indirectement.</td></tr><tr><td>q<sup>LS</sup></td><td>Les tribunaux aborderont la question de toute façon, souvent indirectement ou indirectement.</td></tr><tr><td>q<sup>LSαM</sup></td><td>Les tribunaux aborderont la question de toute façon, souvent de façon oblique ou indirecte.</td></tr><tr><td>src</td><td>Technology travels fast and is swiftly adopted.</td></tr><tr><td>tgt</td><td>Les technologies se distribuent rapidement et sont rapidement adoptées.</td></tr><tr><td>q<sup>LS</sup></td><td>La technologie voyage rapidement et est rapidement adoptée.</td></tr><tr><td>q<sup>LSαM</sup></td><td>La technologie voyage vite et est rapidement adoptée.</td></tr><tr><td>src</td><td>Parliament must be able to exercise its power of scrutiny.</td></tr><tr><td>tgt</td><td>Le Parlement européen doit exercer son pouvoir de contrôle.</td></tr><tr><td>q<sup>LS</sup></td><td>Le Parlement doit pouvoir exercer son pouvoir de contrôle.</td></tr><tr><td>q<sup>LSαM</sup></td><td>Le Parlement doit être en mesure d'exercer son pouvoir de contrôle.</td></tr><tr><td>src</td><td>Interferometer apparatus and interferometric method</td></tr><tr><td>tgt</td><td>Appareil interféromètre et procédé interférométrique</td></tr><tr><td>q<sup>LS</sup></td><td>Appareil interférométrique et procédé interférométrique</td></tr><tr><td>q<sup>LSαM</sup></td><td>Appareil interférométrique et procédé interférométrique</td></tr><tr><td>src</td><td>Cardiac murmur, heart rate increased</td></tr><tr><td>tgt</td><td>Souffle cardiaque, augmentation de la fréquence cardiaque</td></tr><tr><td>q<sup>LS</sup></td><td>Souffle cardiaque, fréquence cardiaque augmentée</td></tr><tr><td>q<sup>LSαM</sup></td><td>Souffle cardiaque, fréquence cardiaque augmentée</td></tr></table>
BAWDEN R. & YVON F. (2023). Investigating the translation performance of a large multilingual language model : the case of BLOOM. In M. NURMINEN, J. BRENNER, M. KOPONEN, S. LATOMAA, M. MIKHAILEV, F. SCHIERL, T. RANASINGHE, E. VANMASSENHOVE, S. A. VIDAL, N. ARANBERRI, M. NUNZIATINI, C. P. ESCARTÍN, M. FORCADA, M. POPOVIC, C. SCARTON & H. MONIZ, Éds., *Proceedings of the 24th Annual Conference of the European Association for Machine Translation*, p. 157–170, Tampere, Finland : European Association for Machine Translation.

EDUNOV S., OTT M., AULI M. & GRANGIER D. (2018). Understanding back-translation at scale. In E. RILOFF, D. CHIANG, J. HOCKENMAIER & J. TSUJII, Éds., *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, p. 489–500, Brussels, Belgium : Association for Computational Linguistics. DOI: 10.18653/v1/D18-1045.

HOLTZMAN A., BUYS J., DU L., FORBES M. & CHOI Y. (2020). The curious case of neural text degeneration. In *8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 : OpenReview.net*.

KINGMA D. P. & BA J. (2014). Adam: A method for stochastic optimization. *arXiv preprint arXiv:1412.6980*.

LACHAUX M.-A., JOULIN A. & LAMPLE G. (2020). Target conditioning for one-to-many generation. In T. COHN, Y. HE & Y. LIU, Éds., *Findings of the Association for Computational Linguistics: EMNLP 2020*, p. 2853–2862, Online : Association for Computational Linguistics. DOI: 10.18653/v1/2020.findings-emnlp.256.

LI M., ROLLER S., KULIKOV I., WELLECK S., BOUREAU Y.-L., CHO K. & WESTON J. (2020). Don't say that! making inconsistent dialogue unlikely with unlikelihood training. In D. JURAFSKY, J. CHAI, N. SCHLUTER & J. TETREAULT, Éds., *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, p. 4715–4728, Online : Association for Computational Linguistics. DOI: 10.18653/v1/2020.acl-main.428.

LIN H., YANG B., YAO L., LIU D., ZHANG H., XIE J., ZHANG M. & SU J. (2022). Bridging the gap between training and inference : Multi-candidate optimization for diverse neural machine translation. In M. CARPUAT, M.-C. DE MARNEFFE & I. V. MEZA RUIZ, Éds., *Findings of the Association for Computational Linguistics : NAACL 2022*, p. 2622–2632, Seattle, United States : Association for Computational Linguistics. DOI: 10.18653/v1/2022.findings-naacl.200.

MÜLLER R., KORNBLITH S. & HINTON G. (2019). *When Does Label Smoothing Help?*, In *Proceedings of the 33rd International Conference on Neural Information Processing Systems*. Curran Associates Inc. : Red Hook, NY, USA.

NLLB TEAM, COSTA-JUSSÀ M. R., CROSS J., ÇELEBI O., ELBAYAD M., HEAFIELD K., HOFFERNAN K., KALBASSI E., LAM J., LICHT D., MAILLARD J., SUN A., WANG S., WENZEK G., YOUNGBLOOD A., AKULA B., BARRAULT L., MEJIA-GONZALEZ G., HANSANTI P., HOFFMAN J., JARRETT S., SADAGOPAN K. R., ROWE D., SPRUIT S., TRAN C., ANDREWS P., AYAN N. F., BHOSALE S., EDUNOV S., FAN A., GAO C., GOSWAMI V., GUZMÁN F., KOEHN P., MOURACHKO A., ROPERS C., SALEEM S., SCHWENK H. & WANG J. (2022). No language left behind : Scaling human-centered machine translation.

OCH F. J. & NEY H. (2003). A systematic comparison of various statistical alignment models. *Computational Linguistics*, 29(1), 19–51.

POST M. (2018). A call for clarity in reporting BLEU scores. In *Proceedings of the Third Conference on Machine Translation : Research Papers*, p. 186–191, Belgium, Brussels : Association for Computational Linguistics.
RADFORD, ALEC N. K. S. T. & SUTSKEVER I. (2018). Improving language understanding with unsupervised learning. Technical Report.

REI R., STEWART C., FARINHA A. C. & LAVIE A. (2020). COMET: A neural framework for MT evaluation. In B. WEBBER, T. COHN, Y. HE & Y. LIU, Éds., *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, p. 2685-2702, Online : Association for Computational Linguistics. DOI: 10.18653/v1/2020.emnlp-main.213.

SCHIOPPA A., VILAR D., SOKOLOV A. & FILIPPOVA K. (2021). Controlling machine translation for multiple attributes with additive interventions. In M.-F. MOENS, X. HUANG, L. SPECIA & S. W.-T. YIH, Éds., *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, p. 6676-6696, Online and Punta Cana, Dominican Republic : Association for Computational Linguistics. DOI: 10.18653/v1/2021.emnlp-main.535.

SENNRICH R., HADDOW B. & BIRCH A. (2016). Neural machine translation of rare words with subword units. In K. ERK & N. A. SMITH, Éds., *Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, p. 1715-1725, Berlin, Germany : Association for Computational Linguistics. DOI: 10.18653/v1/P16-1162.

SHU R., NAKAYAMA H. & CHO K. (2019). Generating diverse translations with sentence codes. In A. KORHONEN, D. TRAUM & L. MÀRQUEZ, Éds., *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, p. 1823-1827, Florence, Italy : Association for Computational Linguistics. DOI: 10.18653/v1/P19-1177.

SU Y., LAN T., WANG Y., YOGATAMA D., KONG L. & COLLIER N. (2022). A contrastive framework for neural text generation. In A. H. OH, A. AGARWAL, D. BELGRAVE & K. CHO, Éds., *Advances in Neural Information Processing Systems*.

SZEGEDY C., VANHOUCKE V., IOFFE S., SHLENS J. & WOJNA Z. (2015). Rethinking the inception architecture for computer vision. *2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, p. 2818-2826.

VAHTOLA T., CREUTZ M. & TIEDEMANN J. (2023). Guiding zero-shot paraphrase generation with fine-grained control tokens. In A. PALMER & J. CAMACHO-COLLADOS, Éds., *Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)*, p. 323-337, Toronto, Canada : Association for Computational Linguistics. DOI: 10.18653/v1/2023.starsem-1.29.

VASWANI A., SHAZEER N., PARMAR N., USZKOREIT J., JONES L., GOMEZ A. N., KAISER L. U. & POLOSUKHIN I. (2017). Attention is all you need. In I. GUYON, U. V. LUXBURG, S. BENGIO, H. WALLACH, R. FERGUS, S. VISHWANATHAN & R. GARNETT, Éds., *Advances in Neural Information Processing Systems*, volume 30 : Curran Associates, Inc.

WELLECK S., KULIKOV I., ROLLER S., DINAN E., CHO K. & WESTON J. (2019). Neural text generation with unlikelihood training. ArXiv, abs/1908.04319.

## A Corpora Statistics

Table 5 presents various statistics of the corpora used in this work, including the total number of sentences, vocabularies, words, and average sentence length. Statistics are computed after performing a light tokenization aiming to split-off punctuation.
<table><caption>TABLE 5 – Corpora statistics. M and K stand for millions and thousands respectively.</caption><thead><tr><th>Side</th><th>Sentences</th><th>Vocabulary</th><th>Words</th><th>Length</th></tr></thead><tbody><tr><td colspan="5"><em>Training set</em></td></tr><tr><td>English</td><td rowspan="2">7.6M</td><td>755K</td><td>174M</td><td>22.9</td></tr><tr><td>French</td><td>839K</td><td>208M</td><td>27.3</td></tr><tr><td colspan="5"><em>News-test</em></td></tr><tr><td>English</td><td rowspan="2">16,071</td><td>27K</td><td>401K</td><td>24.1</td></tr><tr><td>French</td><td>32K</td><td>468K</td><td>29.1</td></tr><tr><td colspan="5"><em>Repetition-test</em></td></tr><tr><td>English</td><td rowspan="2">199</td><td>1,323</td><td>2,521</td><td>12.6</td></tr><tr><td>French</td><td>1,352</td><td>3,215</td><td>16.1</td></tr></tbody></table>

<table><caption>TABLE 6 – Network hyperparameters.</caption><tr><td>size of word embedding</td><td>512</td></tr><tr><td>size of hidden layers</td><td>512</td></tr><tr><td>size of inner feed forward layer</td><td>2,048</td></tr><tr><td>number of heads</td><td>8</td></tr><tr><td>number of layers</td><td>6</td></tr><tr><td>batch size</td><td>4,000 (tokens)</td></tr><tr><td>batch accumulation</td><td>25 (batches)</td></tr></table>

## B NMT Network

Table 6 indicate the hyper-parameters employed to build our translation network.

For optimization work we use the lazy Adam algorithm (Kingma & Ba, 2014). We set warmup steps to 4,000 and update learning rate for every 8 iterations. All models are trained using a single NVIDIA V100 GPU.

We limit the source and target sentence lengths to 150 tokens based on BPE (Sennrich et al., 2016) preprocessing in both source and target sides. We use a joint vocabulary of 32K tokens for both source and target sides. In inference we use a beam size of 5.

Our *baseline* English-to-French model is trained during more than 3 million iterations using all the parallel data available in the Opus website (see Appendix A).