<figure><img src="image_1.png" /></figure>

HAL
open science

# Bilingual Synchronization: Restoring Translational Relationships with Editing Operations

Jitao Xu, Josep Crego, François Yvon

► To cite this version:

Jitao Xu, Josep Crego, François Yvon. Bilingual Synchronization: Restoring Translational Relationships with Editing Operations. The 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022), Dec 2022, Abou Dabi, United Arab Emirates. hal-03827010

HAL Id: hal-03827010

https://hal.science/hal-03827010

Submitted on 24 Oct 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Bilingual Synchronization: Restoring Translational Relationships with Editing Operations

Jitao Xu†

Josep Crego‡

François Yvon†

†Université Paris-Saclay, CNRS, LISN, 91400, Orsay, France
‡SYSTRAN, 5 rue Feydeau, 75002, Paris, France
{jitao.xu,francois.yvon}@limsi.fr, josep.crego@systrangroup.com

## Abstract

Machine Translation (MT) is usually viewed as a one-shot process that generates the target language equivalent of some source text from scratch. We consider here a more general setting which assumes an initial target sequence, that must be transformed into a valid translation of the source, thereby restoring parallelism between source and target. For this *bilingual synchronization task*, we consider several architectures (both autoregressive and non-autoregressive) and training regimes, and experiment with multiple practical settings such as simulated interactive MT, translating with Translation Memory (TM) and TM cleaning. Our results suggest that *one single generic edit-based* system, once fine-tuned, can compare with, or even outperform, dedicated systems specifically trained for these tasks.

## 1 Introduction

Neural Machine Translation (NMT) systems have made tangible progress in recent years (Bahdanau et al., 2015; Vaswani et al., 2017), as they started to produce usable translations in production environments. NMT is generally viewed as a one-shot activity process in autoregressive approaches, which generates the target translation based on the sole source side input. Recently, Non-autoregressive Machine Translation (NAT) models have proposed to perform iterative refinement decoding (Lee et al., 2018; Ghazvininejad et al., 2019; Gu et al., 2019), where translations are generated through an iterative revision process, starting with a possibly empty initial hypothesis.

This paper focuses on the revision part of the machine translation (MT) process and consider *bilingual synchronization* (Bi-sync), which we define as follows: given a pair of a source (f) and a target (ē) sentences, *which may or may not be mutual translations*, the task is to compute a revised version of ē, such that e is an actual translation of

f. This is necessary when the source side of an existing translation is edited, requiring to update the target and keep both sides synchronized. Bi-sync subsumes standard MT, where the synchronization starts with an empty target (ē =[]). Other interesting cases occur when parts of the initial target can be reused, so that the synchronization only requires a few changes.

Bi-sync encompasses several tasks: synchronization is needed in interactive MT (IMT, Knowles and Koehn, 2016) and bilingual editing (Bronner et al., 2012), with ē the translation of a previous version of f; in MT with lexical constraints (Hokamp and Liu, 2017), where ē contains target-side constraints (Susanto et al., 2020; Xu and Carpuat, 2021); in Translation Memory (TM) based approaches (Bulte and Tezcan, 2019), where ē is a TM match for a similar example; in automatic post-editing (APE) (do Carmo et al., 2021), where ē is an MT output.

We consider here several implementations of sequence-to-sequence models dedicated to these situations, contrasting an autoregressive model with a non-autoregressive approach. The former is similar to Bulte and Tezcan (2019), where the source sentence and the initial translation are concatenated as one input sequence; the latter uses the Levenshtein Transformer (LeVT) of Gu et al. (2019). We also study various ways to generate appropriate training samples (f, ē, e). Our experiments consider several tasks, including TM cleaning, which attempts to fix and synchronize noisy segments in a parallel corpus. This setting is more difficult than Bi-sync, as many initial translations are already correct and need to be left unchanged. Our results suggest that one single AR system, once fine-tuned, can favorably compare with dedicated systems for each of these tasks. To recap, our main contributions are (a) the generalization of several tasks subsumed by a generic synchronization objective, allowing us to develop a unified perspective about otherwise unrelated subdomains of MT; (b) the design of a
<figure><img src="image_3.png" /><figcaption>Figure 1: Methods for generating synthetic initial translations ē for each edit type. Rectangle purple boxes refer to separate models used to generate the desired operations. Differences in artificial initial translations (in blue boxes) are marked in bold. Initial translations ē<sub>ins</sub> for insertion are generated by randomly removing segments in the reference sentence e. For ē<sub>sub</sub>, e is first back-translated into an intermediate sentence f* using top-5 sampling, then translated back to ē<sub>sub</sub> with LCD. The first method to generate ē<sub>del</sub> randomly inserts [gap] tokens into e and decodes with a GAP insertion model (Xiao et al., 2022). ē<sub>del1</sub> is obtained by replacing [gap] with the predicted segments. The second method automatically edits e with a model trained on WikiAtomicEdits data.</figcaption></figure>

training procedure for a generic edit-based model;
(c) an empirical validation on five settings and do-
mains.

2 Methods

## 2.1 Generating Editing Data

We consider a general scenario where, given a pair
of sentences f and ē, assumed to be related, but
not necessarily parallel, we aim to generate a target
sentence e that is parallel to f. We would also like
ē and e to be close, as ē is often a valid transla-
tion of a sentence ſ that is close to f. Training
such models requires triplets (f, ē, e). While large
amounts of parallel bilingual data are available for
many language pairs, they are hardly ever associ-
ated to related translations ē (except for APE). We
therefore study ways to simulate synthetic ē from
e, while preserving large portions of e in ē. String
edits can be decomposed into a sequence of three
basic edits (insertions, substitutions and deletions),
we design our artificial samples so that edits from ē
to e only involve one type of operation (Figure 1).

**Insertions** We mainly follow Xiao et al. (2022) to generate initial translations $\tilde{e}_{ins}$ for insertion by randomly deleting segments from e. For each e, we first randomly sample an integer $k \in [1, 5]$, then randomly remove $k$ non-overlapping segments from e. The length of each removed segment is also randomly sampled with a maximum of 5 tokens. We also impose that the overall ratio of removed segments does not exceed 0.5 of e. Different from Xiao et al. (2022), $\tilde{e}_{ins}$ does not include any placeholders to locate the positions of removed segments. This makes $\tilde{e}_{ins}$ a more realistic starting point as the insertion positions are rarely known in practical settings. Our preliminary experiments also show that identifying insertion positions makes the infilling task easier than when they are unknown.

**Substitutions** To simulate substitutions, we apply round-trip translation with lexically constrained decoding (LCD, Post and Vilar, 2018) to generate initial translations for substitution $\tilde{e}_{sub}$. Round-trip translation is already used for the APE task in Junczys-Dowmunt and Grundkiewicz (2016). This requires two standard NMT models separately trained on parallel data, one for each direction. For
each training example (f, e), we first (a) translate e into an intermediate source sentence f* using top-5 sampling (Edunov et al., 2018);<fnref n="1" /> (b) generate an abbreviated version $\tilde{e}_{ins}$ using the method described above for insertions. We then translate f* using LCD, with $\tilde{e}_{ins}'$ as constraints, to obtain $\tilde{e}_{sub}$. In this way, we ensure that at least half of e remains unchanged in $\tilde{e}_{sub}$, while the other parts have been substituted. To increase diversity, $\tilde{e}_{ins}'$ (used to create $\tilde{e}_{sub}$) is sampled with a different random seed than $\tilde{e}_{ins}$ (used for the insertion task).

**Deletions** Simulating deletions requires the initial translation $\tilde{e}_{del}$ to be an extension of e. We propose two strategies to generate $\tilde{e}_{del}$. The first uses a GAP insertion model as in Xiao et al. (2022), in which word segments are randomly replaced with a placeholder [gap] to generate $\tilde{e}_{gap}$. The task is then to predict the missing segments based on the concatenation of f and $\tilde{e}_{gap}$ as input. This differs from our own insertion task, as (a) insertion positions are identified as a [gap] symbol in $\tilde{e}_{gap}$ and (b) generation only computes the sequence of missing segments e$_{seg}$, rather than a complete sentence.

We use GAP to generate extra segments for a pair of parallel sentences as follows. We randomly insert $k \in [1, 5]$ [gap] tokens into e, concatenate it with f and use GAP to predict extra segments, yielding the synthetic target sentence $\tilde{e}_{del_1}$. This method always extends parallel sentences with additional segments on the target side. However, these segments are arbitrary and may not contain any valid semantic information, nor be syntactically correct.

We thus consider a second strategy, based on actual edit operations collected in the WikiAtomicEdits dataset² (Faruqui et al., 2018), which contains edits of an original segment x and the resulting segment x', with exactly one insertion or deletion operation for each example, collected from Wikipedia edit history. This notably ensures that both versions of each utterance are syntactically correct. We treat the deletion data of WikiAtomicEdits as “reversed” insertions, and use both of them to train a seq-to-seq wiki model ($x_{short} \to x_{long}$), generating longer sentences from shorter ones. The wiki model is then used to expand e into an $\tilde{e}_{del_2}$. Compared to $\tilde{e}_{del_1}$, $\tilde{e}_{del_2}$ is syntactically more correct. However, it is also by design very close (one edit away) to e.

As both simulation methods have merits and flaws, we randomly select examples from $\tilde{e}_{del_1}$ and $\tilde{e}_{del_2}$ to build the final synthetic initial translation samples for the deletion operation $\tilde{e}_{del}$.

**Copy and Translate Operations** To handle parallel sentences that do not require any changes, we add a fourth copy operation, where the initial translation $\tilde{e}_{cp}$ is equal to the target sentence ($\tilde{e}_{cp}=e$). Hence, the data used to learn edit operations is built with triplets (f, $\tilde{e}$, e) where $\tilde{e}$ is uniformly randomly selected from $\tilde{e}_{ins}$, $\tilde{e}_{sub}$, $\tilde{e}_{del}$ and $\tilde{e}_{cp}$. Finally, to maintain the capacity to perform standard MT from scratch, we also consider samples where $\tilde{e}$ is empty. The implementation of standard MT varies slightly upon approaches, as we explain below.

## 2.2 Model Architectures

We implement Bi-sync with Transformer-based (Vaswani et al., 2017) autoregressive and non-autoregressive models. The former (Edit-MT) is a regular Transformer with a combined input made of the concatenation of f and $\tilde{e}$; the latter (Edit-LevT) is the LevT of Gu et al. (2019).

**Edit-MT** In this model, $\tilde{e}$ is simply concatenated to f, with a special token to separate the two sentences. This technique has been used e.g. in Dabre et al. (2017) for multi-source MT or in Bulte and Tezcan (2019) for translating with a similar example. The input side of the editing training data is thus f [sep] $\tilde{e}$, as shown in Figure 1 (top).

On the target side, we add a categorical prefix to indicate the type of edit(s) associated with a given training sample, as is commonly done for multi-domain or multilingual MT. For each basic edit (insertion, substitution and deletion), we use a binary tag to indicate if the operation is required. For instance, an $\tilde{e}_{ins}$ needing insertions would have tags [ins] [!sub] [!del] prepended to e. Copy corresponds to all three tags set to negative as [!ins] [!sub] [!del]. The tagging scheme provides us with various ways to perform edit-based MT: (a) we can perform inference without knowing the required edit type of $\tilde{e}$ by generating tags then translations; (b) when the edits are known, we can generate translations with desired edits by using the corresponding tags as a forced prefix; (c) inference can also only output the edit tags and predict the relation between f and $\tilde{e}$. The ability to perform standard MT is preserved by training with a balanced mixture of editing data and parallel data.

<footnote n="1">¹Early experiments showed that using sampling instead of beam search increases the diversity of the generated $\tilde{e}_{sub}$.
²https://github.com/google-research-datasets/wiki-atomic-edits</footnote>
The latter corresponds to an empty ē. For these examples, the target side does not contain any tags.

**Edit-LevT** LevT is trained with a randomly noised version of the reference as initial target, and decodes with empty sentences. In Bi-sync, we instead initialize the target side with the given ē for training and inference. To perform standard MT with the same model, we train with a tunable mixture of these two strategies, where *p* controls the proportion of each type of samples. Taking *p* = 0 is equivalent to train a LevT model with only parallel data. We use *p* = 0.5 in our experiments, making it equivalent to mixing editing and parallel data for the Edit-MT model. The value of *p* can be carefully designed with a schedule or curriculum to optimize the behavior of Edit-LevT, which we leave for future work. For Edit-LevT, we do not use any tags, as Edit-LevT already includes an internal mechanism to predict the edit operation(s).

# 3 Bilingual Re-synchronization

## 3.1 Datasets and Experimental Settings

We first evaluate Edit-MT and Edit-LevT on a basic resynchronization task where ē is assumed to be the translation of a former version of *f*, and only a limited amount of edits is sufficient to restore parallelism. We conduct experiments on WMT14 English-French data<fnref n="3" /> in both directions (En-Fr & Fr-En) and evaluate on two test sets. The first is an artificial derivation of the standard newstest2014 set, and the second is the small parallel sentence compression dataset⁴ of Ivę and Yvón (2016).

For the original newstest2014, we generate an initial version ē for each test sentence and each edit operation according to the methods of Section 2.1. For deletion, we test the performance of both generation methods, resulting in four versions (Ins, Sub, Del₁, Del₂) of newstest2014 with 3,003 sentences each. The sentence compression dataset contains a subset of documents also selected from newstest2014, where sentences are compressed by human annotators while remaining parallel in the two languages. We only retain utterances for which the compressed and original versions actually differ on both sides, resulting in 526 test sentences.

Both experiments use the same training data, where we discard examples with invalid language tag as computed by *fasttext* language identifica-

tion model<fnref n="5" /> (Bojanowski et al., 2017), yielding a training corpus of 33.9M examples. We tokenize all data using Moses⁶ and build a shared source-target vocabulary with 32K Byte Pair Encoding (BPE) units (Sennrich et al., 2016) learned with subword-nmt.⁷ Since we use both parallel and artificial editing data to train edit-based models, the total training data contains about 68M utterances.

We conduct experiments using *fairseq*⁸ (Ott et al., 2019). Edit-MT relies on the Transformer-base model of Vaswani et al. (2017). Model and training configurations are in Appendix A. Performance is computed with SacreBLEU (Post, 2018).

## 3.2 Results

We first separately evaluate the learnability of each edit operation on our synthetic newstest2014 sets. We also derive two tasks from the compression dataset: parallel sentence compression (comp) and extension (ext). For compression, the task consists of producing a compressed target sentence $e_{comp}$ given the compressed source $f_{comp}$ and the original target $e$. For extension, the model should produce $e$ with $f$ and the compressed target $e_{comp}$. These two tasks are respectively similar to the deletion and insertion tasks. There are slight differences, though, as (a) ē for these settings is always syntactically correct, (b) segments that are removed or inserted are selected for their lower informativeness. Therefore, these tasks are more about restoring an adequate, rather than a fluid, translation.

As mentioned in Section 2.2, the generation of translations in Edit-MT models is conditioned on predicted or oracle editing tags that are prefixed to the output: these two situations are contrasted using forced-prefix decoding with the correct tags. For the compression and extension tasks, we use the deletion and insertion tags, respectively.

Tables 2 and 3 report results for both directions, to be compared with a “do-nothing” baseline which simply copies ē as the output. Edit-MT is able to edit the given ē for all types of required edits much better than Edit-LevT. It obtains large gains over the *copy* baseline for insertion,⁹ substitution and deletion for both directions. When tested on

<footnote n="3">³https://www.statmt.org/wmt14
⁴https://github.com/fyvo/ParallelCompression</footnote>

<footnote n="5">⁵https://fasttext.cc/
⁶https://github.com/moses-smt/mosesdecoder
⁷https://github.com/rsennrich/subword-nmt
⁸https://github.com/pytorch/fairseq
⁹BLEU gains for the insertion operation are artificially high. This is because the baseline is hindered by a high brevity penalty.</footnote>
<table><caption>Table 1: BLEU scores for all edit types (Ins, Sub, Del₁, Del₂) broken down by the edit distance Δ between Œ and e for both En-Fr and Fr-En. Each column represents a range of distances. *N* denotes the number of sentences in each group. *All* is computed by concatenating all test sentences.</caption><thead><tr><th></th><th>0</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8-10</th><th>&gt;10</th><th>All</th></tr></thead><tbody><tr><th>En-Fr</th><td>458</td><td>2076</td><td>1133</td><td>1152</td><td>1117</td><td>1033</td><td>880</td><td>776</td><td>1852</td><td>1535</td><td>12012</td></tr><tr><th><i>N</i></th><td>100.0</td><td>88.9</td><td>84.7</td><td>81.5</td><td>77.8</td><td>75.2</td><td>70.3</td><td>68.0</td><td>65.3</td><td>52.9</td><td>75.9</td></tr><tr><th>copy</th><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><th>Edit-MT</th><td><b>96.4</b></td><td><b>92.6</b></td><td>89.3</td><td>87.3</td><td>85.8</td><td>83.3</td><td>82.3</td><td>80.4</td><td>79.4</td><td>71.7</td><td>84.1</td></tr><tr><th>+ tag</th><td>95.5</td><td><b>92.6</b></td><td><b>90.4</b></td><td><b>88.3</b></td><td><b>86.9</b></td><td><b>84.4</b></td><td><b>83.0</b></td><td><b>81.3</b></td><td><b>80.1</b></td><td><b>72.1</b></td><td><b>84.7</b></td></tr><tr><th>Edit-LevT</th><td>95.9</td><td>88.3</td><td>84.3</td><td>80.6</td><td>77.2</td><td>74.2</td><td>69.7</td><td>67.9</td><td>65.3</td><td>57.8</td><td>74.8</td></tr><tr><th>Fr-En</th><td>341</td><td>2050</td><td>1439</td><td>1178</td><td>1125</td><td>1054</td><td>945</td><td>832</td><td>1643</td><td>1405</td><td>12012</td></tr><tr><th><i>N</i></th><td>100.0</td><td>90.1</td><td>85.8</td><td>80.6</td><td>76.4</td><td>71.8</td><td>68.7</td><td>67.0</td><td>63.8</td><td>51.7</td><td>75.6</td></tr><tr><th>copy</th><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><th>Edit-MT</th><td>97.6</td><td>92.8</td><td>91.1</td><td>87.8</td><td>83.8</td><td>82.5</td><td>80.1</td><td>80.2</td><td>77.3</td><td>69.2</td><td>83.6</td></tr><tr><th>+ tag</th><td><b>98.3</b></td><td><b>93.2</b></td><td><b>91.6</b></td><td><b>88.6</b></td><td><b>84.8</b></td><td><b>83.3</b></td><td><b>80.8</b></td><td><b>80.5</b></td><td><b>77.9</b></td><td><b>69.9</b></td><td><b>84.2</b></td></tr><tr><th>Edit-LevT</th><td>97.9</td><td>89.0</td><td>84.4</td><td>79.6</td><td>75.0</td><td>71.0</td><td>67.3</td><td>66.2</td><td>64.8</td><td>54.6</td><td>73.6</td></tr></tbody></table>

<table><caption>Table 2: BLEU scores for Edit-MT and Edit-LevT on resynchronization tasks for En-Fr. Deletions are evaluated separately for two generation methods (Del₁ and Del₂). + tag refers to decoding with the oracle tag as a forced-prefix. Best performance is in bold.</caption><thead><tr><th>Model</th><th>Ins</th><th>Sub</th><th>Del<sub>1</sub></th><th>Del<sub>2</sub></th><th>Comp</th><th>Ext</th></tr></thead><tbody><tr><th>copy</th><td>54.0</td><td>71.5</td><td>71.0</td><td>78.7</td><td>68.4</td><td>66.7</td></tr><tr><th>Edit-MT</th><td>75.9</td><td>77.0</td><td>86.9</td><td><b>94.7</b></td><td>73.1</td><td>67.9</td></tr><tr><th>+ tag</th><td><b>76.9</b></td><td><b>78.5</b></td><td><b>88.6</b></td><td><b>94.7</b></td><td><b>74.0</b></td><td><b>72.7</b></td></tr><tr><th>Edit-LevT</th><td>65.3</td><td>73.9</td><td>72.5</td><td>78.7</td><td>67.8</td><td>67.7</td></tr></tbody></table>

<table><caption>Table 3: BLEU scores for Edit-MT and Edit-LevT models on resynchronization tasks for Fr-En.</caption><thead><tr><th>Model</th><th>Ins</th><th>Sub</th><th>Del<sub>1</sub></th><th>Del<sub>2</sub></th><th>Comp</th><th>Ext</th></tr></thead><tbody><tr><th>copy</th><td>51.8</td><td>70.9</td><td>71.0</td><td>78.7</td><td>63.4</td><td>61.4</td></tr><tr><th>Edit-MT</th><td>73.6</td><td>74.6</td><td>87.5</td><td>95.8</td><td>65.8</td><td>69.3</td></tr><tr><th>+ tag</th><td><b>74.6</b></td><td><b>76.2</b></td><td><b>89.1</b></td><td><b>96.2</b></td><td><b>67.0</b></td><td><b>71.6</b></td></tr><tr><th>Edit-LevT</th><td>66.5</td><td>72.4</td><td>72.3</td><td>78.4</td><td>62.5</td><td>64.3</td></tr></tbody></table>

the compression and extension tasks, which have different edit distributions to the artificial editing data, Edit-MT still improves Œ by 1.2-4.7 BLEU for En-Fr and 2.4-7.9 BLEU for Fr-En. By prefixing Edit-MT with the oracle editing type tags, we can further boost the performance on almost every task for both directions. Edit-LevT can also improve Œ in most test situations, even though the gains are lower than Edit-MT models. However, due to the non-autoregressive nature of Edit-LevT, it obtains a decoding speedup of 2.3-3× with respect to Edit-MT when tested with the same inference batch size on the same hardware, as recommended by Helcl et al. (2022).

To better understand Edit-MT and Edit-LevT on the resynchronization task, we further analyze their performance with respect to the edit distance Δ between Œ and e.

For the results in Table 1, we merge test sentences of all edit types (Ins, Sub, Del₁, Del₂) into one test set, then break them down by the value of Δ. For both directions, prefixing the oracle editing tags for Edit-MT yields a stable improvement for almost all Δ. Edit-LevT performs similar to Edit-MT when no editions are needed, but only starts to improve the copy baseline when more edits are required (Δ ≥ 8).

# 4 Translating with Translation Memories

As explained above, Bi-sync encompasses example-based MT, whereby an existing similar translation retrieved from a TM is turned into an adequate translation of the source. Edit-MT actually uses the same architecture as the retrieval-based models of Bulte and Tezcan (2019); Xu et al. (2020). In this section, we study the performance of our synchronization models in this practical scenario.

## 4.1 Datasets

We use the same multi-domain corpus as Xu et al. (2020), which contains 11 different domains for the En-Fr direction, collected from OPUS<fnref n="10" /> (Tiedemann, 2012). We search for similar translations using Fuzzy Match.<fnref n="11" /> The similarity between two English source sentences is computed as:

$$ \text{sim}(\mathbf{f}, \tilde{\mathbf{f}}) = 1 - \frac{\text{ED}(\mathbf{f}, \tilde{\mathbf{f}})}{\max(|\mathbf{f}|, |\tilde{\mathbf{f}}|)}, \quad (1) $$

where ED(f, ſ) is the edit distance between f and ſ, and |f| is the length of f. The intuition is that the closer f and ſ is, the more suitable Œ will be. To

<footnote n="10">¹⁰https://opus.nlpl.eu/</footnote>

<footnote n="11">¹¹https://github.com/SYSTRAN/fuzzy-match</footnote>
<table><caption>Table 4: BLEU scores for the multi-domain test sets. All is computed by concatenating test sets from all domains, with 11k sentences in total. Copy refers to copying the similar translation in the output. +R implies using the related segments instead of a full initial sentence for inference. Best performance in each block are in bold.</caption><thead><tr><th>Model</th><th>ECB</th><th>EMEA</th><th>Epps</th><th>GNOME</th><th>JRC</th><th>KDE</th><th>News</th><th>PHP</th><th>TED</th><th>Ubuntu</th><th>Wiki</th><th>All</th></tr></thead><tbody><tr><td>copy</td><td>59.8</td><td>64.5</td><td>34.4</td><td>70.3</td><td>67.6</td><td>55.3</td><td>12.0</td><td>38.6</td><td>30.8</td><td>51.6</td><td>47.4</td><td>52.6</td></tr><tr><td>FM</td><td><strong>72.1</strong></td><td><strong>72.3</strong></td><td><strong>58.3</strong></td><td><strong>80.6</strong></td><td><strong>83.2</strong></td><td><strong>66.9</strong></td><td>28.0</td><td><strong>47.2</strong></td><td><strong>62.9</strong></td><td><strong>69.3</strong></td><td>68.8</td><td><strong>67.3</strong></td></tr><tr><td>FM#</td><td>69.3</td><td>68.1</td><td>58.2</td><td>74.2</td><td>80.1</td><td>65.2</td><td><strong>28.6</strong></td><td>44.3</td><td>62.6</td><td>68.1</td><td><strong>69.0</strong></td><td>65.0</td></tr><tr><td>Edit-MT</td><td>59.3</td><td>62.5</td><td>34.7</td><td>69.8</td><td>68.0</td><td>50.6</td><td>12.1</td><td>38.0</td><td>31.2</td><td>52.3</td><td>45.6</td><td>51.8</td></tr><tr><td>+ tag</td><td>60.3</td><td>63.0</td><td>35.7</td><td>70.3</td><td>68.4</td><td>51.9</td><td>12.9</td><td>38.8</td><td>32.6</td><td>52.1</td><td>45.6</td><td>52.6</td></tr><tr><td>+ R + tag</td><td>56.0</td><td>53.9</td><td>45.9</td><td>64.9</td><td>68.5</td><td>50.0</td><td>17.7</td><td>39.7</td><td>44.9</td><td>59.9</td><td>52.8</td><td>53.3</td></tr><tr><td>+ FT + tag</td><td><strong>70.6</strong></td><td><strong>71.5</strong></td><td><strong>57.8</strong></td><td><strong>78.2</strong></td><td><strong>82.0</strong></td><td><strong>66.2</strong></td><td><strong>28.0</strong></td><td><strong>45.1</strong></td><td><strong>61.1</strong></td><td><strong>67.7</strong></td><td><strong>66.8</strong></td><td><strong>66.0</strong></td></tr><tr><td>+ FT + R + tag</td><td>66.4</td><td>63.6</td><td>57.3</td><td>71.3</td><td>77.6</td><td>60.5</td><td><strong>28.0</strong></td><td>42.1</td><td>60.9</td><td>66.7</td><td>65.0</td><td>62.3</td></tr><tr><td>Edit-LevT</td><td>59.5</td><td>62.6</td><td>34.8</td><td><strong>69.0</strong></td><td><strong>66.9</strong></td><td>54.2</td><td>12.2</td><td>37.8</td><td>31.0</td><td>49.0</td><td>44.8</td><td>52.0</td></tr><tr><td>+ R</td><td>49.6</td><td>49.1</td><td><strong>43.6</strong></td><td>57.8</td><td>61.0</td><td>42.9</td><td><strong>16.5</strong></td><td>34.3</td><td><strong>41.0</strong></td><td>49.2</td><td><strong>50.6</strong></td><td>47.6</td></tr><tr><td>+ FT</td><td><strong>60.5</strong></td><td><strong>63.2</strong></td><td>35.2</td><td>68.3</td><td>66.5</td><td><strong>54.5</strong></td><td>12.1</td><td><strong>39.0</strong></td><td>32.9</td><td><strong>50.7</strong></td><td>45.9</td><td><strong>52.4</strong></td></tr></tbody></table>

study the ability of our models to actually make use of TMs instead of memorizing training examples, we also test on two unseen domains: OpenOffice from OPUS and the PANACEA environment corpus<fnref n="12" /> (ENV). Detailed description and statistics about these corpora are in Appendix B. Xu et al. (2020) proposed an alternative where, for each similar translation, they masked out segments that were not aligned with the source. This means that the initial similar translation only contains segments that are directly related to the source input. We also reproduce this related setting, which is very similar to the insertion task.

As we are mostly interested in the edit behavior, we split the data by keeping 1,000 sentences with a sufficiently similar translation (sim > 0.6) in the TM as the test set for each domain. The remaining data is used for training. We use all found similar translations for training and only the best similar translation for testing. This results in 4.4M parallel sentences (para), in which 2.6M examples are also associated to a similar translation (similar) or just the related segments (related). Data preprocessing is similar as in Section 3.1.

## 4.2 Experimental Settings

Our baselines reproduce two settings for TM-based MT: the FM setting of Bulte and Tezcan (2019) and the FM# setting of Xu et al. (2020). The former is trained using *para + similar* data, and the latter uses *para + related*. These two baselines are trained with the same configuration as in Section 3.1. We also report scores obtained by simply copying the retrieved similar translations, as in Section 3.2.

Edit-MT and Edit-LevT from Section 3 differ from FM and FM# both in task, and also in training domains. Hence, we consider fine-tuning (FT) our models. For Edit-MT, we use *para+similar+related* data and fine-tune for only 1 epoch with a learning rate of $8e^{-5}$. As we do not have information about the edit operations required to change a similar translation into the reference, we set all editing tags as on for *similar* data, and prefix the output with `[ins] [sub] [del]`. For the *related* data, we conjecture that mostly insertions are needed as the irrelevant segments have already been removed. We thus only activate the insertion tag. For Edit-LevT, we only use *similar+related* and fine-tune for 1 epoch with a learning rate of $9e^{-5}$. Our fine-tuned models can perform both translation with a similar sentence and with related segments.

## 4.3 Results and Analysis

We evaluate with BLEU, and also show TER scores in Appendix C. We reproduce in Table 4 the overall good performance of FM and FM#. Both significantly improve the initial similar translations. The generic Edit-MT performs much worse, and does not even match the copy results. When prefixed with the editing tag<fnref n="13" /> (+tag), we observe small improvements (+0.8 BLEU on average), that are further increased in the *related* scenario (+R). FT yields a much larger boost in performance (+13.4 BLEU). This highlights the effect of the task and domain mismatches on our initial results with Edit-MT. The *related* setting also benefits from FT, albeit by a smaller margin (+9 BLEU).

<footnote n="12">¹²http://catalog.elda.org/en-us/repository/browse/ELRA-W0057/</footnote>

<footnote n="13">¹³We conjecture with a substitution tag for zero-shot inference as we have no information about the required edit type. Fine-tuned model uses the same tag as FT data.</footnote>
<table><caption>Table 5: BLEU scores for the multi-domain test sets broken down by the edit distance Δ between ē and e. Each column represents a range of distances. N denotes the number of sentences in each group.</caption><thead><tr><th>Model Δ(ē, e)</th><th>0</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8-10</th><th>>10</th><th>All</th></tr></thead><tbody><tr><td>N</td><td>540</td><td>2096</td><td>1107</td><td>882</td><td>827</td><td>782</td><td>689</td><td>607</td><td>1193</td><td>2277</td><td>11000</td></tr><tr><td>copy</td><td>100.0</td><td>82.3</td><td>74.2</td><td>67.2</td><td>62.1</td><td>51.7</td><td>50.5</td><td>48.2</td><td>40.7</td><td>33.8</td><td>52.6</td></tr><tr><td>FM</td><td>91.6</td><td><b>93.3</b></td><td><b>86.5</b></td><td><b>82.3</b></td><td><b>79.2</b></td><td><b>70.5</b></td><td><b>69.0</b></td><td><b>68.0</b></td><td><b>60.9</b></td><td><b>49.5</b></td><td><b>67.3</b></td></tr><tr><td>Edit-MT+ tag + FT + tag</td><td>95.3<br>91.6</td><td>80.8<br>91.1</td><td>72.9<br>85.8</td><td>68.0<br>80.9</td><td>62.7<br>77.7</td><td>52.4<br>68.8</td><td>50.7<br>68.4</td><td>49.6<br>66.6</td><td>41.5<br>59.0</td><td>34.3<br>48.0</td><td>52.6<br>66.0</td></tr><tr><td>Edit-LevT + FT</td><td>94.5<br><b>96.6</b></td><td>79.5<br>78.9</td><td>72.1<br>72.0</td><td>66.3<br>65.8</td><td>61.0<br>61.1</td><td>49.9<br>50.4</td><td>49.6<br>50.4</td><td>47.5<br>48.0</td><td>40.5<br>40.8</td><td>33.7<br>34.3</td><td>52.0<br>52.4</td></tr></tbody></table>

<table><caption>Table 6: BLEU scores on unseen domains.</caption><thead><tr><th>Model</th><th>Office</th><th>ENV</th></tr></thead><tbody><tr><td>copy</td><td>54.7</td><td>59.6</td></tr><tr><td>FM</td><td>66.8</td><td>75.4</td></tr><tr><td>FM#</td><td>64.0</td><td>70.6</td></tr><tr><td>Edit-MT+ tag + FT + tag</td><td>56.2<br><b>68.6</b></td><td>60.3<br><b>78.6</b></td></tr><tr><td>Edit-LevT + FT</td><td>53.0<br>51.9</td><td>60.0<br>59.6</td></tr></tbody></table>

Our best results overall, using FT, are superior to FM# and close to that of FM. This has practical implications, since FM# and FM are specifically trained to transform a retrieved translation, whereas the generic Edit-MT is initially trained with artificial edits then only slightly fine-tuned on the in-domain data. To appreciate this difference, we evaluate our models on two unseen domains (Office and ENV), neither of which are used to train FM and FM#, nor to fine-tune Edit-MT. Results in Table 6 unambiguously show that in this setting, the fine-tuned Edit-MT outperforms FM, suggesting that our edit-based model has not only adapted to the domain, but also to the task, as it can effectively perform zero-shot TM-based translations. Results obtained with the Edit-LevT model on these test sets lag far behind: even with FT on in-domain data, Edit-LevT still struggles to improve over the copy baseline.

We also perform the analysis with a breakdown by the edit distance Δ as in Section 3.2 on the multi-domain test sets. For the results in Table 5, all 11k test sentences are merged into one test set, then broken down by values of Δ. The generic Edit-MT starts to improve the similar translation for Δ≥3. However, it is difficult for Edit-MT to detect very small changes (Δ<3) without FT. Once fine-tuned, Edit-MT performs similar to FM for small changes, which further confirms that Edit-MT adapts to the TM-based translation task. Edit-LevT models, however, only slightly improve the copy baseline for

similar translations requiring large changes (Δ≥8). However, it is better than other models at detecting ēs that do not need edits (Δ=0). We also provide results broken down the merged test set by different edit operations in Appendix C.

# 5 Parallel Corpus Cleaning

Our model restores synchronization between a pair of sentences. This is also useful for parallel TM cleaning tasks. Given a source sentence f and a possibly incorrect translation ē, we want to detect non-parallelism and *to perform appropriate fixes*. We study how Edit-MT fares with this new problem on two publicly available datasets: first on the SemEval 2012&3 Task 8: Cross-lingual Textual Entailment (CLTE, Negri et al., 2011, 2012, 2013),<fnref n="14" /> then with the OpenSubtitles corpus (Lison and Tiedemann, 2016).

## 5.1 Cross-lingual Textual Entailment

The CLTE task aims to identify multi-directional entailment relationships between two sentences $x_1$ and $x_2$ written in different languages. We evaluate on the Fr-En direction, where $x_1$ is in French and $x_2$ is in English. The tagging mechanism of Edit-MT (see Section 2.2) can readily be used for this classification task. Data descriptions and slight adjustments of tagging scheme are in Appendix D.

We treat $x_1$ as f and $x_2$ as ē to match the input format of Edit-MT, and perform zero-shot inference reusing the same Edit-MT model as in Section 3. We concatenate $x_1$ and $x_2$ as input, and truncate the target sequence by only taking the first three edit tags as the predicted label for the corresponding input pair, treating Edit-MT as a mere classification (CLF) model. We also slightly fine-tune Edit-MT with the 500 examples of CLTE training data for 5 epochs with a learning rate of 8e-5.

<footnote n="14">¹⁴https://ict.fbk.eu/clte-benchmark/</footnote>
Results are in Table 7, together with the best scores reported in Negri et al. (2013) for years 2012 and 2013 and the scores reported in Carpuat et al. (2017), which are the best reported performance we could find. Note that these scores may be quite weak, as pre-trained language models did not exist at that time. We have tried to apply a pre-trained XLM model (Conneau and Lample, 2019) to report better baselines for CLTE. However, fine-tuning XLM with only 500 sentences for 5 epochs did not outperform the reported baselines, as the fine-tuned XLM model needs to train new parameters for the linear output layer. Our Edit-MT, on the contrary, does not require any additional parameters during fine-tuning.

<table><caption>Table 7: Accuracy scores on the SemEval CLTE tasks. FT denotes Edit-MT fine-tuned for classification.</caption><thead><tr><th>Methods</th><th>2012</th><th>2013</th></tr></thead><tbody><tr><td>Best SemEval13</td><td>0.570</td><td>0.458</td></tr><tr><td>Carpuat et al. (2017)</td><td>0.604</td><td>0.436</td></tr><tr><td>Edit-MT En-Fr</td><td>0.350</td><td>0.284</td></tr><tr><td>+ FT</td><td><strong>0.716</strong></td><td>0.466</td></tr><tr><td>Edit-MT Fr-En</td><td>0.376</td><td>0.288</td></tr><tr><td>+ FT</td><td>0.710</td><td><strong>0.530</strong></td></tr></tbody></table>

As can be seen in Table 7, out-of-the-box Edit-MT fails to clearly detect the entailment relationships. This is not surprising, as there is a significant difference between our editing data and the CLTE test sets. For instance, the insertion initial translation $\tilde{e}_{ins}$ is always grammatically incorrect, while all sentences in CLTE are syntactically correct. However, after slight fine-tuning on the CLTE data, Edit-MT for both directions can quickly adapt to the task, achieving state-of-the-art performance. This again hints that Edit-MT actually learns to identify various cases of non-parallelism.

## 5.2 Fixing OpenSubtitles Corpus

We further evaluate the ability of Edit-MT to detect parallel sentences and fix noisy data. We experiment with the OpenSubtitles<fnref n="15" /> data (Lison and Tiedemann, 2016) for the En-Fr direction. In this corpus, the French side is translated from English, but with noisy segments. A standard approach is to filter out noisy sentences from the training data when building systems. We aim to study whether Edit-MT can automatically identify and edit, rather than discard, noisy sentence pairs, so that training can use the full set of parallel data. We measure

performance on the 10,159 segments of the En-Fr Microsoft Spoken Language Translation (MSLT) task (Federmann and Lewis, 2016), which simulates a real world MT scenario.

The OpenSubtitles data is processed similarly as in Section 3.1. We first use the fine-tuned CLF model in the CLTE task to predict the relation for all sentence pairs in OpenSubtitles data. About 60% of the data is classified as parallel, indicating that no edit operation is predicted for these segments. Models trained on the 60% clean data are denoted as filtered. For the other 40% presumably noisy data, we reuse the Edit-MT En-Fr model of Section 3 to fix the translations, using the predicted edit tag as a prefix on the target side (see Sections 3 and 4). Models trained on the edited data are noted as fixed. We train NMT models with either all data (full) or just the 40% noisy data as baselines. For comparison, we also train a model using the same data size (15.8M) as the noisy subset, randomly selected from the filtered subset.

As shown in Table 8, aggressively filtering the noisy data improves over using the full training corpus (+2 BLEU for filtered) more than revising it (+1 BLEU for filtered + fixed). The second set of results yields similar conclusions with smaller datasets: here, the effect of automatically fixing a set of initially noisy data improves the BLEU score by 7.2 points and closes half of the gap with a clean corpus of the same size. Note that these results are obtained without adaptation, simply reusing the pre-trained Edit-MT model of Section 3. This suggests that in situations where the training data is small and noisy, editing-based strategies may provide an effective alternative to filtering.

<table><caption>Table 8: BLEU scores on MSLT tasks of models trained with different subsets of OpenSubtitles.</caption><thead><tr><th>Cleaning Method</th><th>BLEU</th><th>Corpus size</th></tr></thead><tbody><tr><td>full</td><td>44.7</td><td>41.6M</td></tr><tr><td>filtered</td><td><strong>46.7</strong></td><td>25.8M</td></tr><tr><td>filtered + fixed</td><td>45.7</td><td>41.6M</td></tr><tr><td>noisy</td><td>32.2</td><td>15.8M</td></tr><tr><td>fixed</td><td>39.4</td><td>15.8M</td></tr><tr><td>filtered (15.8M)</td><td><strong>46.7</strong></td><td>15.8M</td></tr></tbody></table>

## 6 Related Work

The prediction of translations based on a source sentence and an initial translation is first explored in the context of IMT, using a left-to-right sentence completing framework proposed by Langlais

<footnote n="15">¹⁵https://opus.nlpl.eu/OpenSubtitles-v2018.php</footnote>
et al. (2000). The proposals of Green et al. (2014); Knowles and Koehn (2016); Santy et al. (2019) explore ways to generate translations based on given prefix hints. A more general setting, enabling arbitrary insertions thanks to LCD, is studied for online IMT systems in Huang et al. (2021). Note that LCD was initially developed for other purposes, namely enforcing lexical or terminological constraints (Hokamp and Liu, 2017; Post and Vilar, 2018; Hu et al., 2019). As this approach induces large decoding overheads, recent works in this thread explore NAT techniques: Susanto et al. (2020) propose to inject lexical constraints into an edit-based LevT (Gu et al., 2019), an approach improved by Xu and Carpuat (2021) with an additional repositioning operator.

Recent attempts to revise initial translations are explored by Marie and Max (2015), who propose a touch-based scenario where users select usable translations segments, while the more questionable ones are automatically retranslated iteratively. This idea is revisited by Grangier and Auli (2018), where undesired words in the initial translation are crossed-out. The authors use a dual source encoder to represent the initial translation along with the source sentence, which is also explored by Wang et al. (2020) in a touch-editing case. The text infilling task is also considered by Xiao et al. (2022), based on a single source encoder; see also Yang et al. (2021) and Lee et al. (2021) for related proposals. These studies consider a slightly different task than ours, as they only predict the missing part of the initial translation. Nevertheless, they can all be adapted to our generic Bi-sync scenario.

Similar approaches have also been studied in APE. Multi-source architectures have been explored in e.g. (Junczys-Dowmunt and Grundkiewicz, 2018; Tebbifakhr et al., 2018; Shin and Lee, 2018; Pal et al., 2018), whereas Hokamp (2017); Lopes et al. (2019) jointly encode source and translation as one input. (Wisniewski et al., 2015; Libovický et al., 2016; Bérard et al., 2017) focus on learning edit operations. Junczys-Dowmunt and Grundkiewicz (2016) also propose to generate APE training data with round-trip translation.

Bi-sync also encompasses TM-based methods. Gu et al. (2018) use a second encoder to represent TM matches, an idea extended with a more compact representation of TM matches in Xia et al. (2019). As explained above, Bulte and Tezcan (2019) use a single encoder, concatenating TM seg-

ments with the source. Xu et al. (2020) further add a second embedding feature indicating related segments in TM matches, and Pham et al. (2020) propose to simultaneously consider the source and target sides of retrieved TMs. Retrieval-based MT is also explored in He et al. (2021); Khandelwal et al. (2021); Cai et al. (2021), trying to make the performance gain less dependent on the quality of the retrieved TM matches, or to enforce a tighter coupling between TM matches and translations.

## 7 Conclusion

This work introduced Bi-sync, the task of generating translations of a source sentence by editing a related target sentence. We have proposed various ways to create artificial initial translations for different editing types, that are needed for training. We have explored both autoregressive and non-autoregressive architectures, observing experimentally that our autoregressive Edit-MT model trained with artificial triplets performs bilingual resynchronization tasks in several real world scenarios. Edit-MT can also be quickly adapted to retrieval-based MT tasks, where we compared favorably to dedicated models. Finally, Edit-MT can also fix TMs by detecting parallel sentences and correct imperfect translations without adaptation. Another application that we wish to explore is APE.

Our NAT approach Edit-LevT is lagging behind Edit-MT. In the future, we would like to explore more NAT systems, which are computationally faster, and improve their performance. We intend to consider training curriculums and to modify the LevT model to better fit the Bi-sync task. We would also like to study ways to reduce the load of fully re-decoding the input sequence, especially when small changes, that need to be reproduced in the target, are iteratively applied to the source sentence.

## Limitations

The generation of editing data for each type of edit operations requires lots of efforts and resources. This requires two separately trained NMT models to generate the data for substitution via round-trip translation with LCD. The computational cost of LCD is very high compared to regular beam search, therefore consuming many computational resources. The generation of editing data for deletion also requires one separately trained model for each method with a complete decoding of the entire training corpora. Even though our data generation
procedure is effective, the generation process may
not be environmentally friendly. Due to computa-
tional limits, we were not able to conduct experi-
ments on other languages pairs and tasks such as
APE, in which large public datasets are available
for other language pairs other than En-Fr.

As we decomposed the edits from one initial
translation to the reference by basic edits (insertion,
substitution and deletion), the generic Edit-MT and
Edit-LevT models can mostly perform one type
of edits at a time. It might be worth studying to
combine several edit types into one single gener-
ated example, in order to approach more realistic
scenarios for the generic models.

We mainly measured our results with BLEU and
some additional scores in TER. However, other
metrics like COMET (Rei et al., 2020) can also be
interesting: as pointed out by Helcl et al. (2022),
BLEU may be less appropriate to measure valid
translations than COMET for NAT models.

Acknowledgement

We would like to thank the anonymous review-
ers for their valuable suggestions and comments.
This work was granted access to the HPC re-
sources of IDRIS under the allocation 2022-
[AD011011580R2] made by GENCI. The first au-
thor is partly funded by SYSTRAN and by a grant
(Transwrite) from Région Ile-de-France.

References

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-
gio. 2015. Neural machine translation by jointly
learning to align and translate. In 3rd Inter-
national Conference on Learning Representations,
ICLR 2015, San Diego, CA, USA, May 7-9, 2015,
Conference Track Proceedings.

Alexandre Bérard, Laurent Besacier, and Olivier Pietquin. 2017. LIG-CRIsTAL submission for the WMT 2017 automatic post-editing task. In *Proceedings of the Second Conference on Machine Translation*, pages 623–629, Copenhagen, Denmark. Association for Computational Linguistics.

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. *Transactions of the Association for Computational Linguistics*, 5:135–146.

Amit Bronner, Matteo Negri, Yashar Mehdad, Angela
Fahrni, and Christof Monz. 2012. Cosyne: Synchro-
nizing multilingual wiki content. In Proceedings
of the Eighth Annual International Symposium on
Wikis and Open Collaboration, WikiSym ’12, New

York, NY, USA. Association for Computing Machin-
ery.

Bram Bulte and Arda Tezcan. 2019. Neural fuzzy re-pair: Integrating fuzzy matches into neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1800–1809, Florence, Italy. Association for Computational Linguistics.

Deng Cai, Yan Wang, Huayang Li, Wai Lam, and Lemao Liu. 2021. Neural machine translation with monolingual translation memory. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7307–7318, Online. Association for Computational Linguistics.

Marine Carpuat, Yogarshi Vyas, and Xing Niu. 2017. Detecting cross-lingual semantic divergence for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 69–79, Vancouver. Association for Computational Linguistics.

Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems, volume 32, pages 7059–7069. Curran Associates, Inc.

Raj Dabre, Fabien Cromieres, and Sadao Kurohashi. 2017. Enabling multi-source neural machine translation by concatenating source sentences in multiple languages. CoRR, abs/1702.06135.

Félix do Carmo, Dimitar Shterionov, Joss Moorkens, Joachim Wagner, Murhaf Hossari, Eric Paquin, Dag Schmidtke, Declan Groves, and Andy Way. 2021. A review of the state-of-the-art in automatic post-editing. Machine Translation, 35(2):101–143.

Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500, Brussels, Belgium. Association for Computational Linguistics.

Manaal Faruqui, Ellie Pavlick, Ian Tenney, and Dipan-
jan Das. 2018. WikiAtomicEdits: A multilingual
corpus of Wikipedia edits for modeling language
and discourse. In Proceedings of the 2018 Confer-
ence on Empirical Methods in Natural Language
Processing, pages 305–315, Brussels, Belgium. As-
sociation for Computational Linguistics.

Christian Federmann and William D. Lewis. 2016. Mi-
crosoft speech language translation (MSLT) corpus:
The IWSLT 2016 release for English, French and
German. In Proceedings of the 13th International
Conference on Spoken Language Translation, Seat-
tle, Washington D.C. International Workshop on
Spoken Language Translation.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 6112–6121, Hong Kong, China. Association for Computational Linguistics.

David Grangier and Michael Auli. 2018. QuickEdit: Editing text & translations by crossing words out. In *Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)*, pages 272–282, New Orleans, Louisiana. Association for Computational Linguistics.

Spence Green, Sida I. Wang, Jason Chuang, Jeffrey Heer, Sebastian Schuster, and Christopher D. Manning. 2014. Human effort and machine learnability in computer aided translation. In *Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 1225–1236, Doha, Qatar. Association for Computational Linguistics.

Jiatao Gu, Changhan Wang, and Junbo Zhao. 2019. Levenshtein transformer. In *Advances in Neural Information Processing Systems*, volume 32, pages 11181–11191. Curran Associates, Inc.

Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O.K. Li. 2018. Search engine guided neural machine translation. *Proceedings of the AAAI Conference on Artificial Intelligence*, 32(1).

Qiuxiang He, Guoping Huang, Qu Cui, Li Li, and Lemao Liu. 2021. Fast and accurate neural machine translation with translation memory. In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, pages 3170–3180, Online. Association for Computational Linguistics.

Jindřich Helcl, Barry Haddow, and Alexandra Birch. 2022. Non-autoregressive machine translation: It's not as fast as it seems. In *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 1780–1790, Seattle, United States. Association for Computational Linguistics.

Chris Hokamp. 2017. Ensembling factored neural machine translation models for automatic post-editing and quality estimation. In *Proceedings of the Second Conference on Machine Translation*, pages 647–654, Copenhagen, Denmark. Association for Computational Linguistics.

Chris Hokamp and Qun Liu. 2017. Lexically constrained decoding for sequence generation using grid

beam search. In *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 1535–1546, Vancouver, Canada. Association for Computational Linguistics.

J. Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, and Benjamin Van Durme. 2019. Improved lexically constrained decoding for translation and monolingual rewriting. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 839–850, Minneapolis, Minnesota. Association for Computational Linguistics.

Guoping Huang, Lemao Liu, Xing Wang, Longyue Wang, Huayang Li, Zhaopeng Tu, Chengyan Huang, and Shuming Shi. 2021. Transmart: A practical interactive machine translation system. CoRR, abs/2105.13072.

Hakan Inan, Khashayar Khosravi, and Richard Socher. 2017. Tying word vectors and word classifiers: A loss framework for language modeling. In *5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings*. OpenReview.net.

Julia Ike and François Yvon. 2016. Parallel sentence compression. In *Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers*, pages 1503–1513, Osaka, Japan. The COLING 2016 Organizing Committee.

Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2016. Log-linear combinations of monolingual and bilingual neural machine translation models for automatic post-editing. In *Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers*, pages 751–758, Berlin, Germany. Association for Computational Linguistics.

Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2018. MS-UEdin submission to the WMT2018 APE shared task: Dual-source transformer for automatic post-editing. In *Proceedings of the Third Conference on Machine Translation: Shared Task Papers*, pages 822–826, Belgium, Brussels. Association for Computational Linguistics.

Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2021. Nearest neighbor machine translation. In *International Conference on Learning Representations*.

Rebecca Knowles and Philipp Koehn. 2016. Neural interactive translation prediction. In *Conferences of the Association for Machine Translation in the Americas: MT Researchers' Track*, pages 107–120, Austin, TX, USA. The Association for Machine Translation in the Americas.
Philippe Langlais, George Foster, and Guy Lapalme. 2000. TransType: a computer-aided translation typing system. In *ANLP-NAACL 2000 Workshop: Embedded Machine Translation Systems*.

Dongjun Lee, Junhyeong Ahn, Heesoo Park, and Jaemin Jo. 2021. IntelliCAT: Intelligent machine translation post-editing with quality estimation and translation suggestion. In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations*, pages 11–19, Online. Association for Computational Linguistics.

Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pages 1173–1182, Brussels, Belgium. Association for Computational Linguistics.

Jindřich Libovický, Jindřich Helcl, Marek Tlustý, Ondřej Bojar, and Pavel Pecina. 2016. CUNI system for WMT16 automatic post-editing and multimodal translation tasks. In *Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers*, pages 646–654, Berlin, Germany. Association for Computational Linguistics.

Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In *Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)*, pages 923–929, Portož, Slovenia. European Language Resources Association (ELRA).

António V. Lopes, M. Amin Farajian, Gonçalo M. Correia, Jonay Trénous, and André F. T. Martins. 2019. Unbabel’s submission to the WMT2019 APE shared task: BERT-based encoder-decoder for automatic post-editing. In *Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)*, pages 118–123, Florence, Italy. Association for Computational Linguistics.

Benjamin Marie and Aurélien Max. 2015. Touch-based pre-post-editing of machine translation output. In *Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing*, pages 1040–1045, Lisbon, Portugal. Association for Computational Linguistics.

Matteo Negri, Luisa Bentivogli, Yashar Mehdad, Danilo Giampiccolo, and Alessandro Marchetti. 2011. Divide and conquer: Crowdsourcing the creation of cross-lingual textual entailment corpora. In *Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing*, pages 670–679, Edinburgh, Scotland, UK. Association for Computational Linguistics.

Matteo Negri, Alessandro Marchetti, Yashar Mehdad, Luisa Bentivogli, and Danilo Giampiccolo. 2012.

Semeval-2012 task 8: Cross-lingual textual entailment for content synchronization. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)*, pages 399–407, Montréal, Canada. Association for Computational Linguistics.

Matteo Negri, Alessandro Marchetti, Yashar Mehdad, Luisa Bentivogli, and Danilo Giampiccolo. 2013. Semeval-2013 task 8: Cross-lingual textual entailment for content synchronization. In *Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)*, pages 25–33, Atlanta, Georgia, USA. Association for Computational Linguistics.

Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)*, pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.

Santanu Pal, Nico Herbig, Antonio Krüger, and Josef van Genabith. 2018. A transformer-based multi-source automatic post-editing system. In *Proceedings of the Third Conference on Machine Translation: Shared Task Papers*, pages 827–835, Belgium, Brussels. Association for Computational Linguistics.

Minh Quang Pham, Jitao Xu, Josep Crego, François Yvon, and Jean Senellart. 2020. Priming neural machine translation. In *Proceedings of the Fifth Conference on Machine Translation*, pages 462–473, Online. Association for Computational Linguistics.

Matt Post. 2018. A call for clarity in reporting BLEU scores. In *Proceedings of the Third Conference on Machine Translation: Research Papers*, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.

Matt Post and David Vilar. 2018. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. In *Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)*, pages 1314–1324, New Orleans, Louisiana. Association for Computational Linguistics.

Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In *Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers*, pages 157–163, Valencia, Spain. Association for Computational Linguistics.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon
Lavie. 2020. COMET: A neural framework for MT
evaluation. In Proceedings of the 2020 Conference
on Empirical Methods in Natural Language Process-
ing (EMNLP), pages 2685–2702, Online. Associa-
tion for Computational Linguistics.

Sebastin Santy, Sandipan Dandapat, Monojit Choud-
hury, and Kalika Bali. 2019. INMT: Interactive neu-
ral machine translation prediction. In Proceedings
of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th Interna-
tional Joint Conference on Natural Language Pro-
cessing (EMNLP-IJCNLP): System Demonstrations,
pages 103–108, Hong Kong, China. Association for
Computational Linguistics.

Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016. Neural machine translation of rare words
with subword units. In Proceedings of the 54th An-
nual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 1715–
1725, Berlin, Germany. Association for Computa-
tional Linguistics.

Jaehun Shin and Jong-Hyeok Lee. 2018. Multi-
encoder transformer network for automatic post-
editing. In Proceedings of the Third Conference
on Machine Translation: Shared Task Papers, pages
840–845, Belgium, Brussels. Association for Com-
putational Linguistics.

Raymond Hendy Susanto, Shamil Chollampatt, and Liling Tan. 2020. Lexically constrained neural machine translation with Levenshtein transformer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3536–3543, Online. Association for Computational Linguistics.

Amirhossein Tebbifakhr, Ruchit Agrawal, Matteo Ne-
gri, and Marco Turchi. 2018. Multi-source trans-
former with combined losses for automatic post edit-
ing. In Proceedings of the Third Conference on Ma-
chine Translation: Shared Task Papers, pages 846–
852, Belgium, Brussels. Association for Computa-
tional Linguistics.

Jörg Tiedemann. 2012. Parallel data, tools and inter-
faces in OPUS. In Proceedings of the Eighth In-
ternational Conference on Language Resources and
Evaluation (LREC’12), pages 2214–2218, Istanbul,
Turkey. European Language Resources Association
(ELRA).

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc.

Qian Wang, Jiajun Zhang, Lemao Liu, Guoping Huang,
and Chengqing Zong. 2020. Touch editing: A flexi-
ble one-time interaction approach for translation. In
Proceedings of the 1st Conference of the Asia-Pacific
Chapter of the Association for Computational Lin-
guistics and the 10th International Joint Confer-
ence on Natural Language Processing, pages 1–11,
Suzhou, China. Association for Computational Lin-
guistics.

Guillaume Wisniewski, Nicolas Pécheux, and François
Yvon. 2015. Why predicting post-edition is so hard?
failure analysis of LIMSI submission to the APE
shared task. In Proceedings of the Tenth Work-
shop on Statistical Machine Translation, pages 222–
227, Lisbon, Portugal. Association for Computa-
tional Linguistics.

Mengzhou Xia, Guoping Huang, Lemao Liu, and
Shuming Shi. 2019. Graph based translation mem-
ory for neural machine translation. Proceedings
of the AAAI Conference on Artificial Intelligence,
33(01):7297-7304.

Yanling Xiao, Lemao Liu, Guoping Huang, Qu Cui,
Shujian Huang, Shuming Shi, and Jiajun Chen. 2022.
BiTIIMT: A bilingual text-infilling method for inter-
active machine translation. In Proceedings of the
60th Annual Meeting of the Association for Compu-
tational Linguistics (Volume 1: Long Papers), pages
1958–1969, Dublin, Ireland. Association for Compu-
tational Linguistics.

Jitao Xu, Josep Crego, and Jean Senellart. 2020. Boost-
ing neural machine translation with similar trans-
lations. In Proceedings of the 58th Annual Meet-
ing of the Association for Computational Linguistics,
pages 1580–1590, Online. Association for Computa-
tional Linguistics.

Weijia Xu and Marine Carpuat. 2021. EDITOR:
An Edit-Based Transformer with Repositioning for
Neural Machine Translation with Soft Lexical Con-
straints. Transactions of the Association for Compu-
tational Linguistics, 9:311–328.

Zhen Yang, Yingxue Zhang, Ernan Li, Fandong Meng,
and Jie Zhou. 2021. Wets: A benchmark for transla-
tion suggestion. CoRR, abs/2110.05151.
## A Edit-based Model Configurations

We conduct experiments using fairseq<fnref n="16" /> (Ott et al., 2019). Edit-MT relies on the Transformer-base model of Vaswani et al. (2017). We use a hidden size of 512 and a feedforward size of 2,048. We optimize with Adam with a maximum learning rate of 0.0007, an inverse square root decay schedule, and 4,000 warmup steps. We also tie all input and output embedding matrices (Press and Wolf, 2017; Inan et al., 2017). Edit-MT is trained with mixed precision and a batch size of 8,192 tokens on 4 V100 GPUs for 300k iterations. We save checkpoints for every 3,000 iterations and average the last 10 saved checkpoints for inference. For Edit-LevT, we follow Gu et al. (2019), using a maximum learning rate of 0.0005 with 10,000 warmup steps and a larger batch size of 16,384. For inference, we set a maximum decoding round of 10.

## B Details for MT with TMs

### B.1 Datasets and Processing Details

<table><caption>Table 9: Data used for experiments of Section 4.2. *FM ratio* is the ratio of sentences with at least one matched similar translations, *FM train* is the actual number of examples augmented with a similar translation used for training, after setting aside 1,000 test sentences for each domain. Each training sentence is matched with up to 3 similar translations.</caption><thead><tr><th>Domain</th><th>Train</th><th>FM ratio</th><th>FM train</th></tr></thead><tbody><tr><td>ECB</td><td>195,956</td><td>51.73%</td><td>234,943</td></tr><tr><td>EMEA</td><td>373,235</td><td>65.68%</td><td>624,109</td></tr><tr><td>Epps</td><td>2,009,489</td><td>10.12%</td><td>465,228</td></tr><tr><td>GNOME</td><td>55,391</td><td>39.31%</td><td>42,697</td></tr><tr><td>JRC</td><td>503,437</td><td>50.87%</td><td>587,859</td></tr><tr><td>KDE</td><td>180,254</td><td>36.00%</td><td>136,456</td></tr><tr><td>News</td><td>151,423</td><td>2.12%</td><td>4,048</td></tr><tr><td>PHP</td><td>16,020</td><td>34.93%</td><td>10,350</td></tr><tr><td>TED</td><td>159,248</td><td>11.90%</td><td>39,895</td></tr><tr><td>Ubuntu</td><td>9,314</td><td>20.32%</td><td>1,738</td></tr><tr><td>Wiki</td><td>803,704</td><td>19.87%</td><td>409,755</td></tr><tr><td>Total</td><td>4,457,471</td><td>24.27%</td><td>2,557,078</td></tr><tr><td>Office</td><td>49,845</td><td>43.76%</td><td>-</td></tr><tr><td>ENV</td><td>13,632</td><td>6.81%</td><td>-</td></tr></tbody></table>

Our experiments use the same multi-domain corpus as Xu et al. (2020). This corpus contains 11 different domains for the En-Fr direction, collected from OPUS<fnref n="17" /> (Tiedemann, 2012): documents from

the European Central Bank (ECB); from the European Medicines Agency (EMEA); Proceedings of the European Parliament (Epps); legislative texts of the European Union (JRC); News Commentaries (News); TED talk subtitles (TED); parallel sentences extracted from Wikipedia (Wiki); localization files (GNOME, KDE and Ubuntu) and manuals (PHP). All these data were deduplicated prior to training. To evaluate the ability of our models to actually make use of TMs instead of memorizing training examples, we also test on two unseen domains: OpenOffice from OPUS and the PANACEA environment corpus<fnref n="18" /> (ENV). We follow Xu et al. (2020) and search for the top 3 similar translations based on Fuzzy Match with a similarity score greater than 0.6 as computed by Equation (1) on the source side and without an exact match. Note that the ratio of sentences with at least one similar translation greatly varies across domains, as shown in Table 9. When reproducing the *related* setting, Xu et al. (2020) used a placeholder token to mark the positions where segments are deleted. As discussed in Section 2.1, our models do not include such information. Therefore, we do not use placeholders for the *related* data.

## C Additional Results

Table 10 reports results on multi-domain test set for the task of translating with TMs measured on TER using SacreBLEU (Post, 2018). TER results show that even the generic Edit-MT model actually identifies useful edits, as we see improvements with respect to the copy baseline.

Table 11 shows results on BLEU broken down the aggregate test set by edit operations. We observe that the generic Edit-MT struggles to perform well when substitutions are needed. FT vastly improves the ability to substitute and delete from ē, separately or even in combination. Fine-tuned Edit-MT even outperforms FM when only deletions are required.

## D Details for Parallel Corpus Cleaning

In the CLTE task, the goal is to identify multi-directional entailment relationships between two sentences $x_1$ and $x_2$, written in different languages. Each ($x_1, x_2$) pair in the dataset is annotated with one of the following relations: *Bidirectional* ($x_1 \Leftrightarrow x_2$): the two fragments entail each

<footnote n="16">¹⁶https://github.com/pytorch/fairseq</footnote>

<footnote n="17">¹⁷https://opus.nlpl.eu/</footnote>

<footnote n="18">¹⁸http://catalog.elda.org/en-us/repository/browse/ELRA-W0057/</footnote>
<table><caption>Table 10: TER scores on multi-domain test sets. All is computed by concatenating test sets from all domains, with 11k sentences in total. Copy refers to copying the similar translation in the output. +R implies using the related segments instead of a full initial sentence for inference. Best performance in each block are in bold.</caption><thead><tr><th>TER ↓</th><th>ECB</th><th>EMEA</th><th>Epps</th><th>GNOME</th><th>JRC</th><th>KDE</th><th>News</th><th>PHP</th><th>TED</th><th>Ubuntu</th><th>Wiki</th><th>All</th></tr></thead><tbody><tr><td>copy</td><td>0.435</td><td>0.377</td><td>0.659</td><td>0.263</td><td>0.294</td><td>0.457</td><td>0.999</td><td>0.703</td><td>0.653</td><td>0.372</td><td>0.502</td><td>0.488</td></tr><tr><td>FM</td><td><strong>0.286</strong></td><td><strong>0.301</strong></td><td>0.374</td><td><strong>0.160</strong></td><td><strong>0.138</strong></td><td>0.312</td><td>0.642</td><td><strong>0.572</strong></td><td>0.328</td><td><strong>0.227</strong></td><td><strong>0.306</strong></td><td><strong>0.314</strong></td></tr><tr><td>FM#</td><td>0.299</td><td>0.332</td><td><strong>0.365</strong></td><td>0.202</td><td>0.152</td><td><strong>0.309</strong></td><td><strong>0.634</strong></td><td>0.605</td><td><strong>0.327</strong></td><td>0.234</td><td>0.309</td><td>0.327</td></tr><tr><td>Edit-MT</td><td>0.418</td><td>0.390</td><td>0.641</td><td>0.260</td><td>0.285</td><td>0.455</td><td>0.904</td><td>0.646</td><td>0.643</td><td>0.359</td><td>0.520</td><td>0.472</td></tr><tr><td>+ tag</td><td>0.412</td><td>0.382</td><td>0.638</td><td>0.256</td><td>0.283</td><td>0.452</td><td>0.899</td><td>0.643</td><td>0.635</td><td>0.360</td><td>0.520</td><td>0.468</td></tr><tr><td>+ R + tag</td><td>0.397</td><td>0.422</td><td>0.508</td><td>0.268</td><td>0.237</td><td>0.428</td><td>0.825</td><td>0.636</td><td>0.520</td><td>0.282</td><td>0.457</td><td>0.430</td></tr><tr><td>+ FT + tag</td><td><strong>0.300</strong></td><td><strong>0.308</strong></td><td>0.376</td><td><strong>0.182</strong></td><td><strong>0.146</strong></td><td><strong>0.314</strong></td><td>0.643</td><td><strong>0.595</strong></td><td>0.351</td><td><strong>0.237</strong></td><td><strong>0.329</strong></td><td><strong>0.326</strong></td></tr><tr><td>+ FT + R + tag</td><td>0.322</td><td>0.357</td><td><strong>0.375</strong></td><td>0.219</td><td>0.168</td><td>0.342</td><td><strong>0.640</strong></td><td>0.623</td><td><strong>0.350</strong></td><td>0.238</td><td>0.338</td><td>0.346</td></tr><tr><td>Edit-LevT</td><td>0.437</td><td><strong>0.390</strong></td><td>0.655</td><td><strong>0.267</strong></td><td>0.296</td><td><strong>0.463</strong></td><td>0.996</td><td>0.720</td><td>0.656</td><td>0.399</td><td>0.528</td><td>0.496</td></tr><tr><td>+ R</td><td><strong>0.434</strong></td><td>0.446</td><td><strong>0.503</strong></td><td>0.314</td><td><strong>0.294</strong></td><td>0.471</td><td><strong>0.780</strong></td><td><strong>0.649</strong></td><td><strong>0.515</strong></td><td><strong>0.370</strong></td><td><strong>0.478</strong></td><td><strong>0.458</strong></td></tr><tr><td>+ FT</td><td>0.443</td><td><strong>0.390</strong></td><td>0.663</td><td>0.276</td><td>0.306</td><td>0.467</td><td>1.008</td><td>0.722</td><td>0.677</td><td>0.389</td><td>0.531</td><td>0.502</td></tr></tbody></table>

<table><caption>Table 11: BLEU scores for the multi-domain test sets broken down by the edit operations between ē and e. Each column represents a combination of edits. N denotes the number of sentences in each group.</caption><thead><tr><th>BLEU↑</th><th>=</th><th>Ins</th><th>Sub</th><th>Del</th><th>Ins+Sub</th><th>Ins+Del</th><th>Sub+Del</th><th>Ins+Sub+Del</th></tr></thead><tbody><tr><td>N</td><td>540</td><td>316</td><td>3260</td><td>316</td><td>2865</td><td>58</td><td>2634</td><td>1011</td></tr><tr><td>copy</td><td>100.0</td><td>72.0</td><td>67.9</td><td>75.4</td><td>32.5</td><td>69.8</td><td>34.0</td><td>47.3</td></tr><tr><td>FM</td><td>91.6</td><td><strong>80.6</strong></td><td><strong>86.6</strong></td><td>82.9</td><td><strong>50.0</strong></td><td>67.4</td><td><strong>58.4</strong></td><td><strong>63.0</strong></td></tr><tr><td>Edit-MT+ tag</td><td>95.3</td><td>75.7</td><td>67.0</td><td>77.2</td><td>34.2</td><td>68.5</td><td>37.4</td><td>48.0</td></tr><tr><td>+ FT + tag</td><td>91.6</td><td>79.7</td><td>84.6</td><td><strong>85.8</strong></td><td>48.3</td><td>69.9</td><td>57.6</td><td>60.8</td></tr><tr><td>Edit-LevT</td><td>94.5</td><td>75.3</td><td>65.9</td><td>73.3</td><td>33.1</td><td>69.4</td><td>33.7</td><td>46.9</td></tr><tr><td>+ FT</td><td><strong>96.6</strong></td><td>75.5</td><td>65.6</td><td>74.1</td><td>34.3</td><td><strong>70.1</strong></td><td>33.9</td><td>47.0</td></tr></tbody></table>

other (semantic equivalence); Forward ($x_1 \Rightarrow x_2$ & $x_1 \Leftarrow x_2$): unidirectional entailment from $x_1$ to $x_2$; Backward ($x_1 \Leftarrow x_2$ & $x_1 \Rightarrow x_2$): unidirectional entailment from $x_2$ to $x_1$; No Entailment ($x_1 \Leftarrow x_2$): no entailment between $x_1$ and $x_2$. The dataset contains a training set of 500 pairs, and two test sets of the same size (test-2012 and test-2013).

<table><caption>Table 12: Label conversion scheme between CLTE task and Edit-MT editing tags.</caption><thead><tr><th>CLTE Fr-En</th><th>Edit-MT En-Fr</th><th>Edit-MT Fr-En</th></tr></thead><tbody><tr><td>Bidirectional</td><td>Copy</td><td>Copy</td></tr><tr><td>Forward</td><td>Deletion</td><td>Insertion</td></tr><tr><td>Backward</td><td>Insertion</td><td>Deletion</td></tr><tr><td>No Entailment</td><td>Substitution</td><td>Substitution</td></tr></tbody></table>

The tagging mechanism of Edit-MT described in Section 2.2 can readily be used for this classification task with slight adjustments represented in Table 12.

We have not performed hyperparameter searching for FT, even though carefully fine-tuned models may achieve even better performance.