SlowGuess/ACL_2020_OCR huggingface dataset:
Target fuzzy matches To evaluate if the fuzzy match quality is really the primary criterion for the observed improvements, we consider  $\mathsf{FM}_T^\#$  where the fuzzy matches are rescored (on the training set only) with the edit distance between the reference translation and the target side of the fuzzy match. By doing so, we reduce the fuzzy match average  $FM$  source score by about  $2\%$ , but increase target edit distance from  $61\%$  to  $69\%$ .
The effect can be seen in Table 2 in the line  $\mathsf{FM}_T^\#$  vs.  $\mathsf{FM}^\#$ . In average, this technique is performing better with large individual gains of  $+1.5$  BLEU on the Ubuntu corpus. This shows that in this configuration where we do not differentiate related and unrelated words, the model mainly learns to copy fuzzy target words.
Unseen matches Note that in the previous experiments, matches were built over domain corpora that are already used to train the model. This is a common use case: the same translation memory used to train the system will be used in run time, but now we evaluate the ability of our model in a different context where a test set is to be translated for which we have a new TM that has never been seen when learning the original model. This use case corresponds to typical translation task where new entries will be added continuously to the TM and shall be used instantly for translation of following sentences. Hence, we only use EPPS, News, TED and Wiki data to build two models: the first employs only the original source and target sentences (base) the second learns to use fuzzy matches  $(\mathsf{FM}^{+})$ . Table 4 shows results for this use case.
Model	ECB	EMEA	JRC	GNOME	KDE4	PHP	Ubuntu	Avg
%FM	49.8	69.8	50.1	59.7	47.3	41.0	23.3	-
base	36.48	26.31	45.03	27.90	23.62	19.50	25.85	29.24
FM+	43.28	36.09	53.52	38.40	30.91	23.10	30.53	36.55
Table 4: BLEU scores when models are only trained over EPPS, News, TED and Wiki datasets.
As it can be seen, the model using fuzzy matches shows clear accuracy gains. This confirms that gains obtained by  $\mathsf{FM}^{+}$  are not limited to remember an example previously "seen" during training. The model using fuzzy matches acquired the ability to actually copy or recycle words from the provided fuzzy matches and therefore is suitable for adaptive translation workflows. Note that all scores are lower than those showed in Table 2 as a result of discarding all in-domain data when training the
models showing also that online use of translation memory is not a substitute for in-domain model fine-tuning as we will further investigate in Fine Tuning.
Combining matching algorithms Next, we evaluate the ability of our NMT models to combine different matching algorithms. First, we use  $\Theta(M_1, M_2, \ldots)$  to denote the augmentation of an input sentence that considers first the match specified by  $M_1$ , if no match applies for the input sentence then it considers using the match specified by  $M_2$ , and so on. Note that at most one match is used. Sentences for which no match is found are kept without augmentation. Similar to Table 2, models are learned using all the available training data. Table 3 ( $2^{\text{nd}}$  block) illustrates the results of this experiment. The first 3 lines show BLEU scores of models combining  $\mathsf{FM}^{+}$ ,  $\mathsf{NM}^{+}$  and  $\mathsf{EM}^{+}$ . The last row illustrates the results of a model that learns to use two different matching algorithms. We use the best combination of matches obtained so far ( $\mathsf{FM}^{+}$  and  $\mathsf{EM}^{+}$ ) and augment input sentences with both matches. Figure 3 illustrates an example of an input sentence augmented with both a fuzzy match and an embedding match ( $\mathsf{FM}^{+}$  and  $\mathsf{EM}^{+}$ ). Notice that the model is able to distinguish between both types of augmented sequences by looking at the token used in the additional stream (factor). As it can be seen in Table 3 ( $2^{\text{nd}}$  block), the best combination of matches is achieved by  $\oplus(\mathsf{FM}^{+}, \mathsf{EM}^{+})$  further boosting the performance of previous configurations. It is only surpassed by  $\Theta(\mathsf{FM}^{+}, \mathsf{EM}^{+})$  in two test sets by a slight margin.
Fine Tuning Results so far evaluate the ability of NMT models to integrate similar sentences. However, we have run our comparisons over a "generic" model built from a heterogeneous training data set while it is well known that these models do not achieve best performance on homogeneous test sets. Thus, we now assess the capability of our augmentation methods to enhance fine-tuned (Luong and Manning, 2015) models, a well-known technique that is commonly used in domain adaptation scenarios obtaining state-of-the-art results. Table 3 illustrates the results of the model configurations previously described after fine-tuning the models towards each test set domain. Thus, building 7 fine-tuned models for each configuration. Note that similar sentences (matches) are retrieved from the same in-domain data sets used for fine tuning. As
How long does a cold last? || Combien de temps dure le vol? || Combien de temps dure un vaccin?
S	S	S	S	S	SR	T	T	TR	R	TE	E	E	E	E	E
Figure 3: Input sentence augmented with a fuzzy match  $\mathbf{FM}^{+}$  and an embedding match  $\mathbf{EM}^{+}$ .
Token	base FM# FM+	base NM+	base EM+	base FT (Θ(FM+,EM+))
T	66.3%	79.9%	80.3%	68.9%	83.3%	-	-
R	31.3%	54.6%	49.3%	27.0%	34.4%	-	-
E	-	-	-	-	-	45.7%	58.6%
33.0%	37.7%
Table 5: Percentage of Tokens T, R and E effectively appearing in the translation.
shown in Table 3 ( $3^{\text{rd}}$  block), models with FM/EM also increase performance of fine-tuned models gaining in average  $+6$  BLEU on fine-tuned model baselines, and  $+2.5$  compared to FM/EM on generic translation. This add-up effect is interesting since both approaches make use of the same data.
Copy Vs. Context We observe that models allowing for augmented input sentences effectively learn to output the target words used as augmented translations. Table 5 illustrates the rates of usage. We compute for each word added in the input sentence as  $\mathbf{T}$  (part of a lexicalised match),  $\mathbf{R}$  (not in the match) and  $\mathbf{E}$  (from an embedding match), how often they appear in the translated sentence. Results show that  $\mathbf{T}$  words increase their usage rate by more than  $10\%$  compared to the corresponding base models. Considering  $\mathbf{R}$  words, models incorporating fuzzy matches increase their usage rate compared to base models, albeit with lower rates than for  $\mathbf{T}$  words. Furthermore, the number of  $\mathbf{R}$  words output by  $\mathrm{FM}^+$  is clearly lower than those output by  $\mathrm{FM}^{\#}$ , demonstrating the effect of marking unrelated matching words. Thus, we can confirm the copy behaviour of the networks with lexicalised matches. Words marked as  $\mathbf{E}$  (embedding matches) increase their usage rates when compared to base models but are far from the rates of  $\mathbf{T}$  words. We hypothesize that these sentences are not copied by the translation model, rather they are used to further contextualise translations.
Our work stems on the technique proposed by (Bulté and Tezcan, 2019) to train an NMT model to leverage fuzzy matches inserted in the source sentence. We extend the concept by experimenting with more general notions of similar sentences and
techniques to inject fuzzy matches.
The use of similar sentences to improve translation models has been explored at scale in (Schwenk et al., 2019), where the authors use multilingual sentence embeddings to retrieve pairs of similar sentences and train models uniquely with such sentences. In (Niehues et al., 2016), input sentences are augmented with pre-translations performed by a phrase-based MT system. In our approach, similar sentence translations are provided dynamically to guide translation of a given sentence.
Similar to our work, (Farajian et al., 2017; Li et al., 2018) retrieve similar sentences from the training data to dynamically adapt individual input sentences. To compute similarity, the first work uses  $n$ -gram matches, the second includes dense vector representations. In (Xu et al., 2019) the same approach is followed but authors consider for adaptation a bunch of semantically related input sentences to reduce adaptation time.
Our approach combines source and target words within a same sentence - the same type of approach has also been proposed by (Dinu et al., 2019) for introduction of terminology translation.
Last, we can also compare the extra-tokens appended in augmented sentences as "side constraints" activating different translation paths on the same spirit than the work done by (Sennrich et al., 2016a; Kobus et al., 2017) for controlling translation.
6 Conclusions and Further Work
This paper explores augmentation methods for boosting Neural Machine Translation performance by using similar translations.
Based on "neural fuzzy repair" technique, we introduce tighter integration of fuzzy matches informing neural network of source and target and propose extension to similar translations retrieved
$\oplus (\mathsf{FM}^{+},\mathsf{EM}^{+})$
from their distributed representations. We show that the different types of similar translations and model fine-tuning provide complementary information to the neural model outperforming consistently and significantly previous work. We perform data augmentation at inference time with negligible speed overhead and release an Open-Source toolkit with an efficient and flexible fuzzy-match implementation.
In our future work, we plan to optimise the thresholds used with the retrieval algorithms in order to more intelligently select those translations providing richest information to the NMT model and generalize the use of edit distance on the target side.
We would also like to explore better techniques to inject information of small-size  $n$ -grams with possible convergence with terminology injection techniques, unifying framework where target clues are mixed with source sentence during translation. As regards distributed representations, we plan to study alternative networks to more accurately model the identification and incorporation of additional context.
We would like to thank Professor François Yvon for his insightful comments as well as the anonymous reviewers for the useful suggestions.
Michael Bloodgood and Benjamin Strauss. 2014. Translation memory retrieval methods. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 202-210.
Bram Bulté and Arda Tezcan. 2019. Neural fuzzy repair: Integrating fuzzy matches into neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1800-1809, Florence, Italy. Association for Computational Linguistics.
Bram Bulté, Tom Vanallemeersch, and Vincent Vandeghiste. 2018. M3tra: integrating tm and mt for professional translators. pages 69-78. EAMT.
Chenhui Chu and Rui Wang. 2018. A survey of domain adaptation for neural machine translation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1304-1319, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Georgiana Dinu, Prashant Mathur, Marcello Federico, and Yaser Al-Onaizan. 2019. Training neural machine translation to apply terminology constraints. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3063-3068, Florence, Italy. Association for Computational Linguistics.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-648, Atlanta, Georgia. Association for Computational Linguistics.
M. Amin Farajian, Marco Turchi, Matteo Negri, and Marcello Federico. 2017. Multi-domain neural machine translation through unsupervised adaptation. In Proceedings of the Second Conference on Machine Translation, pages 127-137, Copenhagen, Denmark. Association for Computational Linguistics.
Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. 2018. Search engine guided neural machine translation. In Thirty-Second AAAI Conference on Artificial Intelligence.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages 67-72, Vancouver, Canada. Association for Computational Linguistics.
Catherine Kobus, Josep Crego, and Jean Senellart. 2017. Domain control for neural machine translation. In Proceedings of the International Conference on Recent Advances in Natural Language Processing, RANLP 2017, pages 372-378, Varna, Bulgaria. INCOMA Ltd.
Philipp Koehn and Jean Senellart. 2010. Convergence of Translation Memory and Statistical Machine Translation. In Proceedings of AMTA Workshop on MT Research and the Translation Industry, pages 21-31, Denver.
Xiaoqing Li, Jiajun Zhang, and Chengqing Zong. 2018. One sentence one model for neural machine translation. In Proceedings of the 11th Language Resources and Evaluation Conference, Miyazaki, Japan. European Language Resource Association.
Minh-Thang Luong and Christopher D. Manning. 2015. Stanford neural machine translation systems for spoken language domain. In International Workshop on Spoken Language Translation, Da Nang, Vietnam.
Udi Manber and Gene Myers. 1993. Suffix arrays: a new method for on-line string searches. siam Journal on Computing, 22(5):935-948.
Jan Niehues, Eunah Cho, Thanh-Le Ha, and Alex Waibel. 2016. Pre-translation for neural machine translation. CoRR, abs/1610.05243.
John E Ortega, Felipe Sánchez-Martinez, and Mikel L Forcada. 2016. Fuzzy-match repair using black-box machine translation systems: what can be expected. In Proceedings of AMTA, volume 1, pages 27-39.
Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. 2018. Unsupervised learning of sentence embeddings using compositional n-gram features. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 528-540, New Orleans, Louisiana. Association for Computational Linguistics.
Mike Paterson and Vlado Dančík. 1994. Longest common subsequences. In International Symposium on Mathematical Foundations of Computer Science, pages 127-142. Springer.
Mirko Plitt and François Masselot. 2010. A productivity test of statistical machine translation post-editing in a typical localisation context. The Prague bulletin of mathematical linguistics, 93:7-16.
Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, and Armand Joulin. 2019. Ctermatrix: Mining billions of high-quality parallel sentences on the web. arXiv preprint arXiv:1911.04944.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Controlling politeness in neural machine translation via side constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 35-40, San Diego, California. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 2214-2218, Istanbul, Turkey. European Language Resources Association (ELRA).
Tom Vanallemeersch and Vincent Vandeghiste. 2015. Assessing linguistically aware fuzzy matching in translation memories. In Proceedings of the 18th
Annual Conference of the European Association for Machine Translation, pages 153-160, Antalya, Turkey.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Kun Wang, Chengqing Zong, and Keh-Yih Su. 2013. Integrating translation memory into phrase-based machine translation during decoding. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11-21, Sofia, Bulgaria. Association for Computational Linguistics.
Jitao Xu, Josep Crego, and Jean Senellart. 2019. Lexical micro-adaptation for neural machine translation. In International Workshop on Spoken Language Translation, Honk Kong, China.
Masaru Yamada. 2011. The effect of translation memory databases on productivity. *Translation research projects*, 3:63-73.
Anna Zaretskaya, Gloria Corpas Pastor, and Miriam Seghiri. 2015. Integration of machine translation in cat tools: State of the art, evaluation and user attitudes. Skase Journal of Translation and Interpretation, 8(1):76-89.
Jingyi Zhang, Masao Utiyama, Eiichro Sumita, Graham Neubig, and Satoshi Nakamura. 2018. Guiding neural machine translation with retrieved translation pieces. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1325-1335, New Orleans, Louisiana. Association for Computational Linguistics.
A Corpora Statistics
Corpus	#Sents (K)	Lmean	Vocab (K)
English	French	English	French
EPPS	1,992.8	27.7	32.0	129.5	149.2
News	315.3	25.3	31.7	90.5	96.7
TED	156.1	20.1	22.1	58.7	71.4
Wiki	749.0	25.9	23.5	527.5	506.6
ECB	174.1	28.6	33.8	45.3	53.5
EMEA	336.8	16.8	20.3	62.8	68.9
JRC	475.2	30.1	34.5	81.0	83.5
GNOME	51.9	9.6	11.6	19.0	21.6
KDE4	163.9	9.1	12.4	48.7	64.7
PHP	15.1	16.7	18.0	13.3	15.5
Ubuntu	7.1	6.7	8.3	7.5	7.9
B NMT Network Configuration
We use the next set of hyper-parameters: size of word embedding: 512; size of hidden layers: 512; size of inner feed forward layer: 2,048; number of heads: 8; number of layers: 6; batch size: 4,096 tokens. Note that when using factors  $(\mathsf{FM}^{+},\mathsf{NM}^{+}$  and  $\mathsf{EM}^{+})$  the final word embedding is built after concatenation of the word embedding (508 cells) and the additional factor embedding (4 cells).
We use the lazy Adam optimiser. We set warmup steps to 4,000 and update learning rate for every 8 iterations. Models are optimised during  $300K$  iterations. Fine-tuning is performed continuing Adam with the same learning rate decay schedule until convergence on the validation set. All models are trained using a single NVIDIA P100 GPU.
We limit the target sentence length to 100 tokens. The source sentence is limited to 100, 200 and 300 tokens depending on the number of sentences used to augment the input sentence. We use a joint vocabulary of 32K for both source and target sides. In inference we use a beam size of 5. For evaluation, we report BLEU scores computed by multi-bleu.perl.
C Example of Embedding Matching
The table below gives examples of retrieved EM with matching distance  $\geq 0.8$  and with Fuzzy Match distance lower than threshold 0.6.
Table 6: Corpora statistics. Note that K stands for thousands and  $L_{mean}$  is the average length in words.
Distance	Source Sentence	Matched Sentence
0.86	(i) supply of gas to power producers (CCGTs [10]);	(a) Gas supply to power producers (CCGTs)
0.87	The Commission shall provide the chairman and the secretariat for these working parties.	The Commission shall provide secretariat services for the Forum, the Bureau and the working parties.
0.93	Admission to a course of training as a pharmacist shall be contingent upon possession of a diploma or certificate giving access, in a Member State, to the studies in question, at universities or higher institutes of a level recognised as equivalent.	Admission to basic dental training presupposes possession of a diploma or certificate giving access, for the studies in question, to universities or higher institutes of a level recognised as equivalent, in a Member State.