SlowGuess/ACL_2020_OCR huggingface dataset:
Jitao Xu†‡, Josep Crego†, Jean Senellart†
$^{\dagger}$ SYSTRAN, 5 rue Feydeau, 75002 Paris, France  
 $^{\ddagger}$ firstname.lastname@systrangroup.com  
 $^{\ddagger}$ Université Paris-Saclay, CNRS, LIMSI, 91400, Orsay, France  
jitao.xu@limsi.fr
This paper explores data augmentation methods for training Neural Machine Translation to make use of similar translations, in a comparable way a human translator employs fuzzy matches. In particular, we show how we can simply feed the neural model with information on both source and target sides of the fuzzy matches, we also extend the similarity to include semantically related translations retrieved using distributed sentence representations. We show that translations based on fuzzy matching provide the model with "copy" information while translations based on embedding similarities tend to extend the translation "context". Results indicate that the effect from both similar sentences are adding up to further boost accuracy, are combining naturally with model fine-tuning and are providing dynamic adaptation for unseen translation pairs. Tests on multiple data sets and domains show consistent accuracy improvements. To foster research around these techniques, we also release an Open-Source toolkit with efficient and flexible fuzzy-match implementation.
For decades, the localization industry has been proposing Fuzzy Matching technology in CAT tools allowing the human translator to visualize one or several fuzzy matches from translation memory when translating a sentence leading to higher productivity and consistency (Yamada, 2011). Hence, even though the concept of fuzzy match scores is not standardized and differs between CAT tools (Bloodgood and Strauss, 2014), translators generally accept discounted translation rate for sentences with "high" fuzzy matches<sup>1</sup>. With improving machine translation technology
and training of models on translation memories, machine translated output has been progressively introduced as a substitute for fuzzy matches when no sufficiently "good" fuzzy match is found and proved to also increase translator productivity given appropriate post-editing environment (Plitt and Masselot, 2010).
These two technologies are entirely different in their finality - indeed, for a given source sentence, fuzzy matching is just a database retrieval and scoring technique always returning a pair of source and target segments, while machine translation is actually building an original translation. However, with Statistical Machine Translation, the two technologies are sharing the same simple idea about managing and retrieving optimal combination of longest translated n-grams and this property led to the development of several techniques like use of fuzzy matches in SMT decoding (Koehn and Senellart, 2010; Wang et al., 2013), adaptive machine translation (Zaretskaya et al., 2015) or "fuzzy match repairing" (Ortega et al., 2016).
With Neural Machine Translation (NMT), the integration of Fuzzy Matching is less obvious since NMT does not keep nor build a database of aligned sequences and does not explicitly use n-gram language models for decoding. The only obvious and important use of translation memory is to use them to train an NMT model from scratch or to adapt a generic translation model to a specific domain (fine-tuning) (Chu and Wang, 2018). While some works propose architecture changes (Zhang et al., 2018) or decoding constraints (Gu et al., 2018); a recent work (Bulté and Tezcan, 2019; Bulté et al., 2018) has proposed a simple and elegant framework where, like for human translation, translation of fuzzy matches are presented simultaneously with source sentence and the network learns to use this additional information. Even though this method has showed huge gains in quality, it also opens
1https://signsandsymptomsofftranslation. com/2015/03/06/fuzzy-matches/.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1580-1590 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
many questions.
In this work, we are pushing the concept further a) by proposing and evaluating new integration methods, b) by extending the notion of similarity and showing that fuzzy matches can be extended to embedding-based similarities, c) by analyzing how online fuzzy matching compares and combines with offline fine-tuning. Finally, our results also show that introducing similar sentence translation is helping NMT by providing sequences to copy (copy effect), but also providing additional context for the translation (context effect).
2 Translation Memories and NMT
A translation memory (TM) is a database that stores translated segments composed of a source and its corresponding translations. It is mostly used to match up previous translations to new content that is similar to content translated in the past.
Assuming that we translated the following English sentence into French: [How long does the flight last?]  $\rightarrow$  [Combien de temps dure le vol?] Both the English sentence and the corresponding French translation are saved to the TM. This way, if the same sentence appears in a future document (an exact match) the TM will suggest to reuse the translation that has just been saved. In addition to exact matches, TMs are also useful with fuzzy matches. These are useful when a new sentence is similar to a previously translated sentence, but not identical. For example, when translating the input sentence: [How long does a cold last?] the TM may also suggest to reuse the previous translation since only two replacements (a cold by the flight) are needed to achieve a correct translation. TMs are used to reduce translation effort and to increase consistency over time.
2.1 Retrieving Similar Translations
More formally, we consider a TM as a set of  $K$  sentence pairs  $\{(s_k,t_k):k = 1,\dots ,K\}$  where  $s_k$  and  $t_k$  are mutual translations. A TM must be conveniently stored so as to allow fast access to the pair  $(s_k,t_k)$  that shows the highest similarity between  $s_k$  and any given new sentence. Many methods to compute sentence similarity have been explored, mainly falling into two broad categories: lexical matches (i.e. fuzzy match) and distributional semantics. The former relies on the number of overlaps between the sentences taken into account. The latter counts on the generalisation
power of neural networks when building vector representations. Next, we describe the similarity measures employed in this work.
Fuzzy Matching Fuzzy matching is a lexicalised matching method aimed to identify non-exact matches of a given sentence. We define the fuzzy matching score  $FM(s_i, s_j)$  between two sentences  $s_i$  and  $s_j$  as:
$$
F M (s _ {i}, s _ {j}) = 1 - \frac {E D (s _ {i} , s _ {j})}{m a x (| s _ {i} | , | s _ {j} |)}
$$
where  $ED(s_i, s_j)$  is the Edit Distance between  $s_i$  and  $s_j$ , and  $|s|$  is the length of  $s$ . Many variants have been proposed to compute the edit distance, generally performed on normalized sentences (ignoring for instance case, number, punctuation, space or inline tags differences that are typically handled at a later stage). Also, IDF and stemming techniques are used to give more weight on significant words or less weight on morphological variants (Vanallemeersch and Vandeghiste, 2015; Bloodgood and Strauss, 2014).
Since we did not find an efficient TM fuzzy match library, we implemented an efficient and parameterizable algorithm in  $\mathrm{C}++$  based on suffix-array (Manber and Myers, 1993) that we open-sourced. Fuzzy matching offers a great performance under large overlapping conditions. However, in some cases, sentences with large overlaps may receive low  $FM$  scores. Consider for instance the input: [How long does the flight arriving in Paris from Barcelona last?] and the TM entry of our previous example: [How long does the flight last?]  $\rightarrow$  [Combien de temps dure le vol?] Even though the TM entry may be of great help when translating the input sentence, it receives a low score  $(1 - \frac{5}{12} = 0.583)$  because of the multiple insertion/deletion operations needed. We thus introduce a second lexicalised similarity measure that focuses on finding the longest of  $n$ -gram overlap between sentences.
$^{2}$ https://github.com/systran/FuzzyMatch
$N$ -gram Matching<sup>3</sup> We define the  $N$ -gram matching score  $NM(s_i, s_j)$  between  $s_i$  and  $s_j$ :
$$
N M (s _ {i}, s _ {j}) = \left| m a x \Big (\{\mathcal {S} (s _ {i}) \cap \mathcal {S} (s _ {j}) \} \Big) \right|
$$
where  $S(s)$  denotes the set of  $n$ -grams in sentence  $s$ ,  $\max(q)$  returns the longest  $n$ -gram in the set  $q$  and  $|r|$  is the length of the  $n$ -gram  $r$ . For  $N$ -gram matching retrieval we also use our in-house open-sourced toolkit.
Distributed Representations The current research on sentence similarity measures has made tremendous advances thanks to distributed word representations computed by neural nets. In this work, we use sent2vec (Pagliardini et al., 2018) to generate sentence embeddings. The network implements a simple but efficient unsupervised objective to train distributed representations of sentences. The authors claim that the algorithm performs state-of-the-art sentence representations on multiple benchmark tasks in particular for unsupervised similarity evaluation.
We define the similarity score  $EM(s_i, s_j)$  between sentences  $s_i$  and  $s_j$  via cosine similarity of their distributed representations  $h_i$  and  $h_j$ :
$$
E M (s _ {i}, s _ {j}) = \frac {h _ {i} \cdot h _ {j}}{| | h _ {i} | | \times | | h _ {j} | |}
$$
where  $||h||$  denotes the magnitude of vector  $h$ .
To implement fast retrieval between the input vector representation and the corresponding vector of sentences in the TM we use the faiss<sup>5</sup> toolkit (Johnson et al., 2019).
2.2 Related Words in TM Matches
Given an input sentence  $s$ , retrieving TM matches consists of identifying the TM entry  $(s_k, t_k)$  for which  $s_k$  shows the highest matching score. However, with the exception of perfect matches, not all words in  $s_k$  or  $s$  are present in the match. Considering the example in Section 2, the words the flight and a cold are not related to each other, from that follows that the TM target words le vol are irrelevant for the task at hand. In this section we
discuss an algorithm capable of identifying the set of target words  $\mathcal{T} \in t_k$  that are related to words of the input sentence  $s$ . Thus, we define the set  $\mathcal{T}$  as:
$$
\mathcal {T} = \left\{ \begin{array}{c} t \in t _ {k}: \\ \exists s \in \mathcal {S} | (s, t) \in \mathcal {A} \\ \land \forall s \notin \mathcal {S} | (s, t) \notin \mathcal {A} \end{array} \right\}
$$
where  $\mathcal{A}$  is the set of word alignments between words in  $s_k$  and  $t_k$  and  $\mathcal{S}$  is the LCS (Longest Common Subsequence) set of words in  $s_k$  and  $s$ . The LCS is computed as a by-product of the edit distance (Paterson and Dančík, 1994).
$S$  is found as a sub-product of computing fuzzy or  $n$ -gram matches. Word alignments are performed by fast_align (Dyer et al., 2013). Figure 1 illustrates the alignments and LCS words between input sentences and their corresponding fuzzy (top) and  $N$ -gram (bottom) matches.
Fuzzy Match
$N$  -gram Match
Figure 1: English-French TM entries with corresponding word alignments (right) and LCS of words with the input sentence (left). Matches are found following Fuzzy (top) and  $N$ -gram (bottom) techniques.
The TM source sentence  $s_k$  of the fuzzy matching example has a LCS set of 5 words  $S =$
$^{6}$ https://github.com/clab/fast_align
<sup>3</sup>Note that this practice is also called "subsequence" or "chunk" matching in CAT tools and is usually combined with source-target alignment in order to help human translators easily find translation fragments.
<sup>4</sup>https://github.com/epfml/sent2vec
https://github.com/facebookresearch/ faiss
{How,long,does,last,?}. The set of related target words  $\mathcal{T}$  is also composed of 5 words  $\{Combien,de,temps,dure,?\}$  , all aligned to at least one word in  $s$  and to no other word. The Ngram match example has a LCS set of 4 words  $S =$  {How,long,does,a}, while related target words consists of  $\mathcal{T} = \{\text{Combien,de, temps,un}\}$  .The target word dure is not part of  $\mathcal{T}$  as it is aligned to work and work  $\notin S$  . Notice that sets  $s$  and  $\mathcal{T}$  consist of collections of indices (word positions in their corresponding sentences) while word strings are used in the previous examples to facilitate reading.
2.3 Integrating TM into NMT
We retrieve fuzzy,  $n$ -gram and sentence embedding matches as detailed in the previous section. We explore various ways to integrate matches in the NMT workflow. We follow the work by (Bulté and Tezcan, 2019) where the input sentence is augmented with the translation retrieved from the TM showing the highest matching score (FM, NM or EM). One special integration of fuzzy matching, denoted  $\mathrm{FM}_T$ , is rescoring fuzzy matches based on the target edit distance. This special integration, that is only performed on training data, is discussed in the Target Fuzzy matches section.
Figure 2 illustrates the main integration techniques considered in this work and detailed below. The input English sentence [How long does the flight last?] is differently augmented. For each alternative we show: the TM (English) sentence producing the match; the augmented input sentence with the corresponding TM (French) translation. Note that LCS words are displayed in boldface.
$\mathbf{FM}^{\#}$  We implement the same format as detailed in (Bulté and Tezcan, 2019). The input English sentence is concatenated with the French translation with the (highest-scored) fuzzy match as computed by  $FM(s_{i}, s_{j})$ . The token  $||$  is used to mark the boundary between both sentences.
$\mathbf{FM}^*$  We modify the previous format by masking the French words that are not related to the input sentence. Thus, sequences of unrelated tokens are replaced by the || token. The mechanism to identify relevant words is detailed in Section 2.2.
$\mathbf{FM}^{+}$  As a variant of  $\mathbf{FM}^{*}$ , we now mark target words which are not related to the input sentence in an attempt to help the network identify those target
words that need to be copied in the hypothesis. However, we use an additional input stream (also called factors) to let the network access to the entire target sentence. Tokens used by this additional stream are:  $\mathbf{S}$  for source words;  $\mathbf{R}$  for unrelated target words and  $\mathbf{T}$  for related target words.
$\mathbf{NM}^{+}$  In addition to fuzzy matches, we also consider arbitrary large  $n$  -gram matching. Thus, we use the same format as for  $\mathbb{F}\mathbb{M}^{+}$  but considering the highest scored  $n$  -gram match as computed by  $NM(s_{i},s_{j})$
EM+ Finally, we also retrieve the most similar TM sentences as computed by  $EM(s_i, s_j)$ . In this case, marking the words that are not related to the input sentence is not necessary since similar sentences retrieved following EM score do not necessarily present any lexical overlap. Note from the example in Table 2 that similar sentences retrieved with distributed representations may contain many word reorderings or synonyms (i.e.: duration - last or flu - cold) that makes it difficult to align both sentences. Hence, the same format employed for FM can be used here. However, since we plan to combine different kinds of matches in a single model we adopt the format employed by  $\mathrm{NM}^+$  and  $\mathrm{FM}^+$  with a new factor label E.
FM#	How long does the flight last?
How long does a cold last? || Combien de temps dure le vol?
FM*	How long does the flight last?
How long does a cold last? || Combien de temps dure ||?
FM+	How long does the flight last?
How long does a cold last? || Combien de temps dure le vol?
S S S S S S S R T T T T R R T
NM+	How long does a vaccine work?
How long does a cold last? || Combien de temps dure un vaccin?
S S S S S S S R T T T T R R R
EM+	What is the duration of flu symptoms?
How long does a cold last? || Quelle est la durée de la gripe?
S S S S S S E E E E E E E E
Figure 2: Input sentence augmented with different TM matches: FM# (Bulté and Tezcan, 2019), FM*, FM+ and EM+.
The original paper uses ‘@@’ as break token. We made sure that || was not part of the vocabulary.
3 Experimental Framework
3.1 Corpora and Evaluation
We used the following corpora in this work $^{8}$  (Tiedemann, 2012): Proceedings of the European Parliament (EPPS); News Commentaries (NEWS); TED talk subtitles (TED); Parallel sentences extracted from Wikipedia (Wiki); Documentation from the European Central Bank (ECB); Documents from the European Medicines Agency (EMEA); Legislative texts of the European Union (JRC); Localisation files (GNOME, KDE4 and Ubuntu) and Manual texts (PHP). Detailed statistics about these are provided in Appendix A. We randomly split the corpora by keeping 500 sentences for validation, 1,000 sentences for testing and the rest for training. All data is preprocessed using the OpenNMT tokenizer $^{9}$  (conservative mode). We train a 32K joint byte-pair encoding (BPE) (Sennrich et al., 2016b) and use a joint vocabulary for both source and target.
Our NMT model follows the state-of-the-art Transformer base architecture (Vaswani et al., 2017) implemented in the OpenNMT-tf<sup>10</sup> toolkit (Klein et al., 2017). Further configuration details are given in Appendix B.
3.2 TM Retrieval
We perform fuzzy matching, ignoring exact matches, and keep the single best match if  $FM(s_i,s_j) \geq 0.6$  with no approximation. Similarly, the largest  $N$ -gram match is used for each test sentence with a threshold  $NM(s_i,s_j) \geq 5$ . A similarity threshold  $EM(s_i,s_j) \geq 0.8$  is also employed when retrieving similar sentences using distributed representations. The EM model is trained on the source training data with default fasttext params on 200 dimension, and 20 epochs.
Algorithm	Indexing (s)	Retrieval (word/s)
FM	546	607
NM	546	40,888
EM	181+342	4,142
Table 1: Indexing and retrieval time for the different matching algorithm run on single thread Intel Core i7, 2.8GHz. EM index time is the sum of embedding building for the 2M sentences and faiss index building.
The faiss search toolkit is used through python API with exact FlatIP index. Building and retrieval times for each algorithm on a 2M sentences translation memory (Europarl corpus) are provided in Table 1. Note that all retrieval algorithms are significantly faster than NMT Transformer decoding, thus, implying a very limited decoding overhead.
We compare our baseline model, without augmenting input sentences, to different augmentation formats and retrieval methods. Our base model is built using the concatenation of all the original corpora. All other models extend the original corpora with sentences retrieved following various retrieval methods. It is worth to notice that extended bittexts share the target side with the original data.
Individual comparison of Matching algorithms and Augmentation methods In this experiment, all corpora are used to build the models while matches of a given domain are retrieved from the training data of this domain. Models are built using the original source and target training data (base), and after augmenting the source sentence as detailed in Section 2.3: FM#, FM#, FM*, FM+, NM+ and EM+. Test sentences are augmented following the same technique as for training sentences $^{11}$ . Table 2 summarises the results that are divided in three blocks, showing results for the three types of matching studied in this work (FM, NM and EM).
Best scores are obtained by models using augmented inputs except for corpora not suited for translation memory usage: News, TED for which we observe no gains correlated to low matching rates. For the other corpora, large gains are achieved when evaluating test sentences with matches (up to +19 BLEU on GNOME corpus), while a very limited decrease in performance is observed for sentences that do not contain matches. This slight decrease is likely to come from the fact that we kept the corpus size and number of iterations identical while giving harder training tasks. Results are totally on par with the findings of (Bulté and Tezcan, 2019).
All types of matching indicate their suitability showing accuracy gains. In particular for fuzzy matching, which seems to be the best for our task. Among the different techniques used to insert fuzzy matching,  $\mathbf{FM}^{+}$  obtains the best results, validating
11Except for  $\mathsf{FM}_T^\#$  for which we use FM# test set
Freely available from http://opus.nlpl.eu
$^{9}$ https://github.com/OpenNMT/Tokenizer
10https://github.com/OpenNMT/OpenNMT-tf
Model	News	TED	ECB	EMEA	JRC	GNOME	KDE4	PHP	Ubuntu	Avg
%FM	3.1%	10.3%	49.8%	69.8%	50.1%	59.7%	47.3%	41.0%	23.3%	-
base FM#	37.16	43.23	49.19	50.14	59.19	51.14	50.16	30.24	45.52	47.94
57.69 - 41.95	54.88 - 44.10	66.34 - 52.84	55.80 - 47.92	53.05 - 48.77	42.19 - 25.25	56.05 - 42.27
36.68	42.93	55.15	61.16	66.35	61.82	54.37	33.10	48.26	54.32
69.79 - 41.54	70.87 - 43.53	80.46 - 53.55	73.61 - 45.83	65.57 - 47.85	47.04 - 26.08	66.72 - 42.08
FM#FTM	36.79	43.14	55.41	60.32	66.41	62.01	53.65	33.22	49.75	54.40
70.46 - 41.41	68.63 - 44.90	80.57 - 53.57	74.05 - 45.58	64.77 - 47.20	46.31 - 26.30	69.16 - 43.32
36.44	43.27	54.52	59.49	65.24	59.54	53.30	32.77	48.74	53.37
FM*FTM	37.12	42.62	68.43 - 41.68	67.64 - 44.85	77.59 - 54.10	70.16 - 45.19	62.63 - 48.00	44.50 - 26.31	68.34 - 42.20
56.18	61.97	66.91	62.68	54.59	33.81	48.62	54.97
72.26 - 41.25	71.52 - 44.72	81.58 - 53.62	74.99 - 45.83	65.95 - 48.01	47.74 - 26.27	67.49 - 42.37
%NM	45.5%	36.9%	69.9%	60.4%	69.6%	31.1%	22.9%	33.7%	14.1%	-
base
NM+	37.16	43.23	49.19	50.14	59.19	51.14	50.16	30.24	45.52	47.94
36.74	43.07	49.97 - 46.44	50.94 - 47.43	60.32 - 55.70	53.86 - 46.59	54.16 - 45.89	34.64 - 26.88	58.29 - 40.68
55.40	59.17	65.60	58.46	51.54	31.87	46.16	52.60
58.65 - 44.06	62.69 - 46.60	69.24 - 54.32	70.05 - 42.21	59.87 - 42.11	39.35 - 26.10	63.22 - 39.59
base
EM+	37.16	43.23	49.19	50.14	59.19	51.14	50.16	30.24	45.52
36.50	42.89	52.09 - 40.74	52.07 - 40.08	62.60 - 48.16	54.20 - 45.88	51.62 - 48.60	42.22 - 21.42	52.20 - 41.82
54.02	56.41	66.04	58.07	53.70	32.37	49.88
58.52 - 40.86	59.47 - 40.16	71.45 - 48.33	66.09 - 44.06	59.43 - 47.43	46.91 - 20.96	62.04 - 43.20
For instance on KDE4: the base model obtains a BLEU score of 50.16 while  $\mathsf{FM}^+$  obtains the highest score 54.59. Most of the gains are obtained over the test sentences having a fuzzy match (65.95 vs. 53.05) while for sentences without fuzzy match the best score is obtained with the base system (48.77 compared to 48.01).
Table 2: The first row in each block indicates the percentage of test sentences for which a match was found. Cells below contain the BLEU score over the entire test set (top number) and over the subset of test sentences augmented with matches (bottom left) and without matches (bottom right). Best scores of each column are outlined with bold fonts. Last column is the average of all corpus but News and TED.
Model	ECB	EMEA	JRC	GNOME	KDE4	PHP	Ubuntu	Avg
FM+	56.18	61.97	66.91	62.68	54.59	33.81	48.62	54.97
Θ(FM+,NM+)	56.83	60.60	67.52	61.97	54.67	32.38	47.13	54.44
Θ(FM+,EM+)	56.71	61.61	67.64	62.71	54.82	33.60	49.98	55.30
Θ(FM+,NM+,EM+)	56.20	61.30	67.43	62.14	55.05	32.33	48.96	54.77
⊕(FM+,EM+)	57.08	62.27	68.06	63.30	55.48	33.39	49.50	55.58
FT (base)	52.65	54.06	61.58	56.16	54.20	33.54	50.14	51.76
FT (Θ(FM+,EM+))	57.07	63.11	69.44	65.97	59.30	36.26	52.77	57.70
FT (⊕(FM+,EM+))	57.44	63.41	69.82	65.72	58.71	35.49	52.40	57.57
Table 3: BLEU scores of models combining several types of matches (2nd block) and over Fine-Tuned models (3rd block). We include again results of the FM+ model (1st block) to facilitate reading.
our hypothesis that marking related words is beneficial for the model. Masking sequences of unrelated words,  $\mathsf{FM}^*$  under-performs showing that the neural network is more challenged when
dealing with incomplete sentences than with sentences containing unrelated content.