Switching-Aligned-Words Data Augmentation for Neural Machine Translation

1 INTRODUCTION . Deep neural networks show great performances when trained on massive amounts of data . Data augmentation is a simple but effective technique to generate additional training samples when deep learning models are thirsty for data . In the area of Computer Vision , it is a standard practice to use image data augmentation methods because trivial transformations for images like random rotation , resizing , mirroring and cropping ( Krizhevsky et al. , 2012 ; Cubuk et al. , 2018 ) doesn ’ t change its semantics . This presence of of semantically invariant transformation makes it easy to use image data augumentation in Computer Vision research . Unlike image domain , data augmentation on text for Natural Language Processing ( NLP ) tasks is usually non-trivial as there is often a prerequisite to do some transformations without changing the meaning of the sentence . In this paper we will focus on data augmentation techniques in neural machine translation ( NMT ) which is special and more difficult than other NLP tasks since we should maintain semantic consistency within language pairs which is from quite possibly different domains . Data augmentation techniques in NMT can be divided into two categories dependent on whether additional monolingual corpus is uesd . If in-domain monolingual training data for NMT is available , one successful data augmentation method is back-translation ( Sennrich et al. , 2016 ) , whereby an NMT model is trained in the reverse translation direction ( target-to-source ) and then used to translate target-side monolingual data back to source language . The resulting synthetic parallel corpus can added to existing training data to learn a source-to-target model . Other more refined ideas of backtranslation include dual learning ( He et al. , 2016 ) or Iterative Back-translation ( Hoang et al. , 2018 ) . Sometimes when in-domain monolingual data is limited , existing methods including randomly swapping two words , dropping word , replacing word with another one ( Lample et al. , 2018 ) and so on are applied to perform transfromations to original training data without changing its semantics to the greatest extent . However , due to text characteristics , these random transformations often result in significant change in semantics . Gao et al . ( 2019 ) propose to replace the embedding of word by a weighted combination of mutiple semantically similar words . Also , Xiao et al . ( 2019 ) use a lattice structure to integrate multiple segmentations of a single sentence to perfrom an immediate data augmentation . In this work , we propose Switching-Aligned-Words ( SAW ) data augmentation , a simple yet effective data augmentation approach for NMT training . It belongs to the second class of data augmentation methods where in-domain monolingual data is limited . Different from the previous methods that conduct semantically invariant transformations within each language , we propose to use another language ( target language ) to help make semantically invariant transformations for current language ( source language ) by switching aligned words randomly . We use an unsupervised word aligner fast-align1 ( Dyer et al. , 2013 ) to pair source and target words that have similar meaning . To verify the effectiveness of our method , we conduct experiments on WMT14 English-to-German and IWSLT14 German-to-Englisth datasets . The experimental results show that our method can obtain remarkable BLEU score improvement over the strong baselines . 2 RELATED WORK . We describes the related work about data augmentation for NMT with or without using additional monolingual data in this section . 2.1 WITH MONOLINGUAL DATA . The most successful data augmentation techiques to leverage monolingual data for NMT training is back-translation . It requires training a target-to-source system in order to generate additional synthetic parallel data from the monolingual target data . This data complements human bitext to train the desired source-to-target system . There has been a growing body of literature that analyzes and extends back-translation . Edunov et al . ( 2018 ) demontrate that it is more effective to generate source sentences via sampling rather than beam search . Hoang et al . ( 2018 ) present iterative backtranslation , a method for generating increasingly better synthetic parallel data from monolingual data to train NMT model . Fadaee & Monz ( 2018 ) show that words with high predicted loss during training benefit most . Wang et al . ( 2019 ) propose to quantify the confidence of NMT model predictions based on model uncertainty to better cope with noise in synthetic bilingual corpora produced by back-translation . Dual learning ( He et al. , 2016 ) extends the back-translation approach to train NMT systems in both translation directions . When jointly training the source-to-target and targetto-source NMT models , the two models can provide back translated data for each other direction and perform multi-rounds back-translation . Different from back-translation , Currey et al . ( 2017 ) show that low resource language pairs can also be improved with synthetic data where the source is simply a copy of the monolingual target data . Wu et al . ( 2019 ) propose to use noised training to better leverage both back-translation and self-training data . 2.2 WITHOUT MONOLINGUAL DATA . Lample et al . ( 2018 ) randomly swap the words within a fixed small window size or drop some words in a sentence for learning an autoencoder to help train the unsupervised NMT model . Fadaee et al . ( 2017 ) propose to replace a common word by low-frequency word in the target sentence , and change its corresponding word in the source word to improve translation quality of rare words . In Xie et al . ( 2017 ) , they replace the word with a placeholder token or a word sampled from the frequency distribution of vocabulary , showing that data noising is an effective regularizer for NMT . Kobayashi ( 2018 ) propose an approach to ues the prior knowledge from a bi-directional language model to replace a word token in the sentence . Gao et al . ( 2019 ) try to replace the ids of word by a soft ids and they train Transformer language models in original training data to get soft words . Wang et al . ( 2018 ) introduce a data augmentation method for NMT called SwitchOut to randomly replace words in both source and target sentences with other words . 3 OUR APPROACH . We first describe the background and our proposed switching-aligned-words data augumentation approach . The framework can be seen as an adversarial training process like Generative Adversarial Networks ( GAN ) ( Goodfellow et al. , 2014 ; Salimans et al. , 2016 ) , see Figure 1 for an overview . For 1https : //github.com/clab/fast align I want to thank my friends Ich möchte meinen Freunden danken Figure 2 : The illustration for alignment model . English sentence is ” I want to thank my friends. ” , and corresponding German sentence is ” Ich möchte meinen Freunden danken ” . image generation , in which a discriminator and a generator compete with each other : the generator aims to generate images similar to the natural ones , and the discriminator aims to detect the generated ones from the natural ones . For data augmentaion methods in NMT , the noise generator can be any model that produces noise over parallel sentences , in our method it is an alignment model which is shown in Figure 2 . Finally , the NMT model is trained as a discriminator to distinguish generated sentences from the original ones and the process of detection noise offers NMT model an ability to learn bilingual alignment information . 3.1 BACKGROUND . Given a source and target sentence pair ( x , y ) , where x = ( x1 , x2 , · · · , x|x| ) is a source-language sentence and y = ( y1 , y2 , · · · , y|y| ) is a target-language sentence . A neural machine translation system models the conditional probability : P ( y|x ) = |y|∏ j=1 P ( yj |y < j , x ) ( 1 ) based on an encoder-decoder framework with an attention mechanism ( Sutskever et al. , 2014 ; Bahdanau et al. , 2014 ) . Encoder and decoder can be specialized using different neural architectures including GRU ( Bahdanau et al. , 2014 ) , LSTM ( Wu et al. , 2016 ) , CNN ( Gehring et al. , 2017 ) and Transformer ( Vaswani et al. , 2017 ) , among which the self-attention based Transformer is the stateof-the-art architecture for NMT . The decoder predicts a corrresponding translation y = ( y1 , · · · , y|y| ) step by step based on the last decoding state and source context . The translation probability can be formulated as follows : P ( yj |y < j , x ) = q ( yj−1 , sj , cj ) ( 2 ) where sj and cj denote the decoding state and the source context at the j-th time step respectively . Here , q ( · ) is the softmax layer . Sepcifically , sj = g ( yj−1 , sj−1 , cj ) ( 3 ) where g ( · ) is the corresponding neural architecture unit . The context vector cj is calculated as a weighted sum of the source annotations hi on the basis of attention mechanism : cj = |x|∑ i=1 αjihi ( 4 ) The alignment model αji measures the similarity between sj and hi . The whole model is jointly trained to seek the optimal parameters that can be used to correctly encode the source sentences and decode them to corresponding target sentences . 3.2 ALIGNMENT . NMT models learn the alignment between source words xi and target word yj mainly deponds on these two aspects : attention and word embeddings . Since attention weight αji measures the similarity between sj and hi , it has been widely used to evaluate the word alignment between yj and xi , so that the word alignment is explicitly modeled . NMT models also try to learn word alignment information by updating word embeddings when training . In monolingual vector space , similar words tend to have commonalities in the same dimensions of their word vectors ( Mikolov et al. , 2013 ) . These commonalities include : ( 1 ) a similar degree ( value ) of the same dimension and ( 2 ) a similar positive or negative correlation of the same dimension . In bilingual vector space , Liu et al . ( 2019 ) assume that the source and target words that have similar meanings should also have similar embedding vectors . Hence , they propose to perform a sharing techique between source and target word embedding space resulting significantly imporvement in alignment quality and translation performance . Motivated by their findings , we propose to generate new training samples by replacing one word in the original sentences with its alinged word in corresponding target sentences . According to the characteristic of bilingual embeddings , aligned words tend to have similar meanings even in different language , so our replacing method will preserve the original meaning of the sentence to a great extend . Also , when training the model we put a aligned target word in the similar context of source sentence , it is helpful for source and target words with similar meanings to learn similar embedding representation . 3.3 SWITCHING ALIGNED WORDS BY REPLACEMENT . Inspired by the above intuition , we propose to augment NMT training data by replacing a randomly chosen word in a sentence by its aligned target word . Suppose we have an extra alignment model A ( ·|· ) such as intrinsic attention mechanism ( Bahdanau et al. , 2014 ) or unspervised word aligner ( Dyer et al. , 2013 ) . Given a sentence pair ( x , y ) , each source word xi is aligned with a target word ŷi that has the highest alignment probability among the candidates , and is computed as follows : ŷi = argmax y∈a ( x ) logA ( y|xi ) ( 5 ) where a ( · ) denotes the set of aligned candidates . So the conditional probability can be written as : P ( y|x ) = |y|∏ j=1 P ( yj |y < j , C ( x ) ) = |y|∏ j=1 P ( yi|y < j , x1 , . . . , ŷk , . . . , x|x| ) ( 6 ) where k-th source word is replaced by corresponding target word . In experiments , we randomly choose a word in the training data with probability γ1 and replace it by its aligned target word .

Summary: This paper describes a method for data augmentation and/or regularization for machine translation that works by running a word aligner on the parallel data, and then with some probability \gamma, replacing a source token with its corresponding target token or vice versa. A proposed variant also mixes the embeddings of the two words. Small improvements are shown over simpler noising strategies such as replacing words with placeholder tokens or with random words from the vocabulary.