Switching-Aligned-Words Data Augmentation for Neural Machine Translation
Authors: Fengshun Xiao
Zuchao Li
hai zhao
Venue:  (2021)
Keywords: Machine Translation
Data augmentation

Abstract: In neural machine translation (NMT), data augmentation methods such as back-translation make it possible to use extra monolingual data to help improve translation performance, while it needs extra training data and the in-domain monolingual data is not always available. In this paper, we present a novel data augmentation method for neural machine translation by using only the original training data without extra data. More accurately, we randomly replace words or mixup with their aligned alternatives in another language when training neural machine translation models. Since aligned word pairs appear in the same position of each other during training, it is helpful to form bilingual embeddings which are proved useful to provide a performance boost \citep{liu2019shared}. Experiments on both small and large scale datasets show that our method significantly outperforms the baseline models.

Under review as a conference paper at ICLR 2021
SWITCHING-ALIGNED--WW DATA AUGMENTATION FOR NEURAL MACHINE TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
In neural machine translation (NMT), data augmentation methods such as back- translation make possible to use extra monolingual data to help improve trans- lation performance, while it needs extra training data and the in-domain monolin- gual data is not always available. this paper, we present a novel data augmen- tation method for neural machine translation by using only the original training data without extra data. More accurately, we randomly replace words or mixup with their aligned alternatives in another language when training neural machine translation models. Since aligned word pairs appear in the same position of each other during training, it is helpful to form bilingual embeddings which are proved useful to provide a performance boost (Liu et al.) 2019). Experiments on both small and large scale datasets show that our method significantly outperforms the baseline models.
INTRODUCTION
Deep neural networks show great performances when trained on massive amounts of data. Data augmentation is simple but effective technique to generate additional training samples when deep learning models are thirsty for data. In the area of Computer Vision, it standard practice to use image data augmentation methods because trivial transformations for images like random rotation, resizing, mirroring and cropping (Krizhevsky et al. 2012; Cubuk et al. 2018) doesn't change its semantics. This presence of of semantically invariant transformation makes it easy to use image data augumentation in Computer Vision research.
Unlike image domain, data augmentation on text for Natural Language Processing (NLP) tasks is usually non-trivial as there is often a prerequisite to do some transformations without changing the meaning of the sentence. In this paper will focus data augmentation techniques in neural machine translation (NMT) which is special and more difficult than other NLP tasks since we should maintain semantic consistency within language pairs which is from quite possibly different domains.
Data augmentation techniques in NMT can be divided into two categories dependent on whether additional monolingual corpus is uesd. If ifin-domain monolingual training data for NMT is available, one successful data augmentation method is back tack-translation (Sennrich et al., 2016). whereby an NMT model is trained the reverse translation direction (targettto-souree) and then used translate target-side monolingual data back to source language. The resulting synthetic parallel corpus can added to existing training data learn a source-to-tares to-rgget model. Other more refined ideas of back- translation include dual learning (He et al. 2016) or Iterative Back-translation (Hoang et al. 2018).
Sometimes when in-domain monolingual data limited, existing methods including randomly swapping two words, dropping word, replacing word with another one (Lample et al.) 2018) and so on are applied to perform transfromations to original training data without changing its semantics to the greatest extent. However, due to text characteristics, these random transformations often sult in significant change in semantics. Gao et al. (2019) propose to replace the embedding of word by weighted combination of mutiple semantically similar words. Also, Xiao et al.) (2019) lattice structure to integrate multiple segmentations of a single sentence to perfrom an immediate data augmentation.
In this work, we propose Switching-Aligne Words (SAW) data augmentation, a simple yet effective data augmentation approach for NMT training. It belongs the second class of data augmentation

---

Under review as a conference paper at ICLR 2021
methods where in-domain monolingual data is limited. Different from the previous methods that conduct semantically invariant transformations within each language, we propose to use another language (target language) help make semantically invariant transformations for current language (source language) by switching aligned words randomly. We use an unsupervised word aligner (Dyer al. 2013) pair source and target words that have similar meaning.
fast-aligy D
To verify the effectiveness of our method, we conduct experiments on WMT14 English-to-Germe German and IWSLT14 German-to-Enngisst datasets. The experimental results show that our method can obtain remarkable BLEU score improvement over the strong baselines.
2 RELATED WORK
We describes the related work about data augmentation for NMT with without using additional monolingual data in this section.
2.1 WITH MONOLINGUAL. DATA
The most successful data augmentation techiques to leverage monolingual data for NMT training is back-translation. It requires training target-to-source system in order to generate additional synthetic parallel data from the monolingual target data. This data complements human bitext to train the desired source-to-target system. There has been growing body of literature that analyzes and extends vack-translatioo Eduunv et al. (2018) demontrate that it is more effective to generate source sentences via sampling rather than beam search. [Hang et al. (2018) present iterative back- translation, a method for generating increasingly better synthetic parallel data from monolingual data to train NMT model. [aadaee & Monz (2018) show that words with high predicted loss during training benefit most. Wang al. (2019) propose to quantify the confidence of NMT model predic- tions based on model uncertainty to better cope with noise in synthetic bilingual corpora produced by back- takataaallatio. Dual learning (He et al. 2016) extends the back-translation approach to train NMT systems in both translation directions. When jointly training the nource-to-targer and target- to-source NMT models, the two models can provide back translated data for each other direction and perform multi-rounds back-translation.
Different from back-translation, Currey et al. (017) show that low resource language pairs can also be improved with synthetic data where the source is simply a copy of the monolingual target data. Wu et al. (2019) propose use noised training to better leverage both back- tock-translation and self-training data.
2.2 WITHOUT MONOLINGUAL DATA
Lample et al. 2018) randomly swap the words within a fixed small window size or drop some words a sentence for learning an autoencoder to help train the unsupervised NMT model. Fadace et al. (2017) propose to replace common word by wow-frequency word in the target sentence, and change its corresponding word in the source word to improve translation quality of rare words. In Xie et al. (2017), they replace the word with placeholder token or word sampled from the frequency distribution of vocabulary, showing that data noising is an effective regularizer for NMT. Kobayashi (2018) propose approach to the prior knowledge from bi-directional language model to replace a word token in the sentence. [ao et al. (2019) try to replace the ids of word by soft ids and they train Transformer language models in original training data get soft words. Wang et af.] (2018) introduce data augmentation method for NMT called SwitchOut to randomly replace words in both source and target sentences with other words.
OUR APPROACH
We first describe the background and our proposed
data augumentation
approach. The framework can be seen an adversarial training process like Generative Adversarial Networks (GAN) (Goodfellow et al. 2014 Salimans et al.) 2016), see Figure for an overview. For
hhtps///gttuub.comcchhffsst align
2

---

Under review as a conference paper at ICLR 2021
Noise
to
Generator
lo
SRC
thank
danken
want
to
thank
my
friends
Diseriminator (NMT model)
friends
friends
Ich
Ich
mbcite
müiclt
mximen
meinen
Freunden
Freunden
TGT
Ich
möchre
meinen
Freunden
danken
danken
Figure 1:
An overview of Switching- Aligneed Figure 2: The illustration for alignment model. English sentence "I want to thank my friends.". and corresponding German sentence is "Ich
Words data augumentation approach. The noise generator can be any model that produces noise over parallel sentences, and the NMT model is möchte meinen Freunden danken". trained discriminator.
image generation, in which discriminator and generator compete with each other: the generator aims to generate images similar to the natural ones, and the discriminator aims to detect the generated ones from the natural ones. For data augmentaion methods in NMT, the noise generator can be any model that produces noise over parallel sentences, our method it is an alignment model which is shown in Figure Finally, the NMT model is trained discriminator distinguish generated sentences from the original ones and the process of detection noise offers NMT model an ability to learn bilingual alignment information.
3.1 BACKGROUND
Given source and target sentence pair (æz, y). where a (æ1, ... sentence and (y1. U2. 3 v1) is target-language sentence. A neural machine translation system models the conditional probability:
;T/z) (7p is source-language
vv
P(ylæ) = II P(usluej.
(1)
based encoder-decoder framework with attention mechanism (Sutskever al., 2014 Bah- danau et al. 2014). Encoder and decoder can be specialized using different neural are architectures including GRU (Bahdanau et al. (2014), LSTM (Wu al.) (016)), CNN (Gehring al. 2017)) and Transformer (Vaswani et al. (017). among which the self attentinon based Transformer is the state- of-the-art architecture for NMT.
The decoder predicts a corrrepponing translation (V1,-.. : mn step by step based on the last decoding state and source context. The translation probability can formulated as follows:
P(y y<j: .) 1,,)-1,
where 8j and ej denote the decoding state and the source context at the j-th time step respectively. Here, g(-) is the softmax layer. Sepcifically,
$j = g(y)-1,
where g()) is the corresponding neural architecture unit. The context vector ey is caleulated weighted sum the source annotations h; on the basis attention mechanism:
Cj
aych;
The alignment model azt measures the similarity between s, and h2- The whole model is jointly trained to seek the optimal parameters that can used to correctly encode the source sentences and decode them to corresponding target sentences.
3

---

Under review as a conference paper at ICLR 2021
3.2 ALIGNMENT
NMT models learn the alignment between source words I; and target word Mj mainly deponds on these two aspects: attention and word embeddings. Since attention weight aja measures the similarity between 8, and h4, it has been widely used to evaluate the word alignment between yj and 24, so that the word alignment explicitly modeled.
NMT models also try to learn word alignment information by updating word embeddings when training. In monolingual vector space, similar words tend to have commonalities in the same di- mensions of their word vectors (Mikolov et al. (2013). These commonalities include: (1) similar degree (value) of the same dimension and (2) a similar positive or negative correlation of the same dimension. In bilingual vector space, Liu et al. (2019) assume that the source and target words that have similar meanings should also have similar embedding vectors. Hence, they propose to perform sharing techique between source and target word embedding space resulting significantly imporvement alignment quality and translation performance.
Motivated by their findings, we propose generate in the original sentences with its alinged word in corresponding target sentences. According to the characteristic of bilingual embeddings, aligned words tend to have similar meanings even in different language, so our replacing method will preserve the original meaning of the sentence to great extend. Also, when training the model we put aligned target word in the similar context of source sentence, it is helpful for source and target words with similar meanings to learn similar embedding representation.
training samples by replacing word
3.3 SWITCHING ALIGNED WORDS BY REPLACEMENT
Inspired by the above intuition, we propose to augment NMT training data by replacing a randomly chosen word in a sentence by its aligned target word. Suppose we have an extra alignment model A(-I-) such as intrinsic attention mechanism (Bahdanau et al. 2014) or unspervised word aligner Dyyer et al. 2013). Given sentence pair (æ. y), each source word v; aligned with a target word that has the highest alignment probability among the candidates, and is computed as follows:
J arg max logA(y)2;)
yEa(z)
where a(-) denotes the set of aligned candidates. So the conditional probability can be written
b
Plyl=)
Plyl C(=))
(6)
II (ullyej, T1, .k.
mmm))
where k-th source word is replaced by corresponding target word. In experiments, we randomly choose a word in the training data with probability " and replace it by its aligned target word.
3.4 SWITCHING ALIGNED WORDS BY MIXUP
Mixup is simple yet effective image augmentation techique introduced by Zhang al. (201)) The idea is to combine two random images in mini-batch in some proportion to generate synthetic examples for training. Bringing this idea with corresponding aligned target word with probability ?1, instead we mix up these two word embeddings to form a combined embedding which contain both source and target information:
work, do not directly replace source word
E(z;) = (1 (1172) E 72 E(C()))
E(1))) 7/2 B()))
where E is the embedding lookup table, 7/ is the mixup ratio which is hyper-parameter.
The intuition behind mixup is that random linear interpolations between the embeddings of source word and corresponding target word let neural models regularize the representation of word embed- dings. Mixing the aligned word pairs do not interrupt the representaion of word embeddings far from its original ones.
4

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENT
In this paper, data augmentation will only process source data of the training data.
4.1 DATASETS
Two translation tasks, IWSLT14 German-to-English (De-En) and WMT14 English-to-German (En- De), used for our evaluation.
IWSLT14 Enrrlsh IWSLT14 De-En dataset contains 153K training sentence pairs. We randomly select 7K data from the training set validation set and use the combination of dev2010, dev2012, 1sr2010, tst2011 I and tst2012 as test set with 7K sentences which are preprocessed firstly. BPE algorithm used to process words into subwords, and number of subword tokens in the shared vocabulary is 10k.
WMT14 English-German We use the WMT14 En-De dataset with 4.5M sentence pairs for training. We randomly select 40K data from the training set as validation set and use newstes(2014 as test set. Dataset is segmented by BPE and the number of subword tokens in the shared vocabulary is 32K. The sentences longer than 250 subword tokens are removed from the training dataset.
4.2 BASELINES
We compare our approach with following baselines:
Base:
The original training strategy without any data augmentation;
Swap: Randomly swap words nearby positions with a window size k (Lample et al. 2018);
Dropout: Randomly drop word tokens (Lample et al. 2018);
Blank: Ramdomly replace word tokens with a placeholder token Xie et al. 2017
Smooth: Randomly replace word tokens with sample from the unigram frequency distri-
bution over the vocabulary (Xie et al. 2017);
All above introduced methods except Swap incorporate a dlyerr-paameed the probability of each word token to be replaced in training phase. We set y with different values in 0,0.05,0.1.0.15..02 and report the best result for each method. As for Swap, use 3 window size following (Lample et al. 2018):
4.3
MODEL
We use the transformer.batt setting following Vaswani et al. (017)) for WMT14 En-De datasets, with 6-layer encoder and 6-layer decoder. The dimensions of word embeddings, hidden states and the position-wise feed-forward networks are 512, 512, 2048 respectively. The dropout 0.1 and the transformer small setting which has
attention head is 8. For IWSLT14 De-En datasets, a 6-layer encoder and 6-layer decoder, but the dimensions of word embeddings, hidden states and the position-wise feed-forward networks attention head is 4. Word embeddings between the source, target and output softmax embeddings are tied as is a normal setting. We set 7/1 and 7/ with different values in {0, 0.05, 0.1, 0.15, 0.2), and report the best result for each method. For all experiments, hyperparameters are optimized on development set and then tested using only a single hyperparameter. We use beam size and length penalty 0.6 for inference, and use multi-bler to evaluate the quality of translation.
512, 512, 1024 respectively. The dropout 0.3 and
4.4 TRAINING
All our models are trained on one TITAN RIX GPU. The implementation of model is based on fairseq toolki We choose Adam optimizer with 3 0.9, B2 0.98, = 1 and the learning

---

Under review as a conference paper at ICLR 2021
BLEU
Model
DE-EN
EN-DE
Transformer (small)
34.49
Transformer (base)
27.35
+Swap
34.40
27.12
+Dropout
34.83
27.43
+Blank
34.93
27.52
+Smooth
34.98
27.50
+Replacement 35.18
27.74
+Mixup
34.96
27.68
Table I: BLEU scores on IWSLT14 De-En and WMT14 En-De. The baselines for De-En task and En-De task are the Transformer-sman and the Transformer model respectively.
rate setting strategy, which are all the same as Vaswani et al. (0177) Ir d 0.5 mmen(step 0.5 step- warmupstep) -1.5) where d is the dimension of embeddings, step is the step number of training and wormupstep is the step number of warmup. When the number of step smaller than the step of warmup, the learning rate increases linearly and the decreases. Significantly, our replacing or mixing decision is made at runtime allowing different transformations for the same sentence pair.
4.5 RESULTS
The evalution results on IWSLT14 De-En and WMT14 En-De datasets are shown in Table [ As see, the Replacement method can achieve 0.69 and 0.39 BLEU scores improvement over the Transformer small and the Transformer base baselines and the Mixup method improve the two baselines by 0.47 and 0.33 BLEU scores respectively.
Compared with other augmentation methods, we can see that (1) the Replacement method achieves the best results all the datasets and (2) the Mixup method can achieve comparable better results. Specially, we find that our method works better on relatively small scale datasets. As small scale datasets lack bilingual information compared large scale datasets and are easy fall into the overfitting problems, these results clearly demonstrate the effectiveness of our approach.
STUDY
IMPACT OF 71 AND 7/
Base Replacement Mixup
Base
Mixup
BLEU
0.65
0.15
Figure 3: BLEU scores IWSLT De-En dataset with different replacing probability Y1- In Mixup experiment 7/ is 0.1.
Figure 4: BLEU scores on IWSLT De-En dataset with different mixup probability 72 when 71 0.1.

---

Under review as a conference paper at ICLR 2021
0.75
0.7
0.85
0.6
Wasker Water
glanhen helieve
Närf five
schán heautiful
Tenster window
baseline I +replacement D
Figure 5: Cosine similarity between some bilingual embedding pairs in different method (the results have be normalized to and 1).
We different replacing probability value 71 and mixup probability value 7/2 see the effect of our approach.
Figure the BLEU scores on IWSLT14 De-En dataset of each method with different replacing probability, from which we can see that our method can obtain a consistent BLEU improvement within large probability range and achieve the best performance when ?1 0.1 in each method. However, the performance begins to drop when meanings of original sentence begin to destroyed greatly. Also we find that Mixup more stable than Replacement.
0.1, we think the reason is that the semantic
As we can see from Figure the Mixup method can obtain consistent BLEU improvement above baseline within large probability range and the best BLEU socre is achieved in mixup probability 722 = 0.1 when 711 0.1.
5.2 ANALYSIS OF BILINGUAL EMBEDDINGS
Since suppose that aligned word pairs appear in the same position of each other during training will be helpful to form bilingual embeddings which are proved useful to provide a preformance boost (Liu al. 2019), we study whether our approach is truly useful for bilingual embeddings. We randomly sample some words and their corresponding aligned words to analyze the relation within them. Specifically, we compare the cosine similarity between the embeddings of aligned words to figure out the changes of bilingual embeddings. Formally, we have aligned word pairs (æ,, y;) and their embeddings Exx;) (e(æ;)1, e(x;)2, e(x;)a), )a). E(y,) (e(y))) (en,),ee())) e(y,)2, (yy)),.... e(y))))) where d is the embedding dimension. The cosine similarity can be defined as:
Σd e(x;)a c(U,)s
cos ((
Eimee(x)) v Ei-re(y))
where 8
the angle between embedding pairs. We finally normalize the results to and 1, and the larger the value, the more similar the two embeedings are.
From Figure 5 we can see that (1) The embedding vectors between aligned word pairs have a very strong positive correlation since the normalized cosine similarity values are all above 0.5. (2) The

---

Under review as a conference paper at ICLR 2021
Replacement method significantly imporves the positive correlation between aligned word pairs which proves our hypothesis that switching aligned words is helpful to from bilingual embeddings. (3) The Mixup method does not seem to improve the quality of bilingual embeddings. We suppose that the improvement of translation quality mainly come from the introduction of noise to word embeddings.
CONCLUSION
In this work, we have presented which randomly replace words or mixup with their aligned alternatives in another language when training. simple yet effective and can extremely useful when extra in-domain monolingual data is limited. Results on both small and large scale datasets have verified the effectiveness of our method.
alighinng-AAinne--- Words (SAW) data augmentaion for NMT,
In the future, besides focusing bilingual machine translation tasks, are interested in extending method to a multilingual scenario which needs more complex replacement and training strategies. In addition, we plan to study our approach in other cross-lingual NLP tasks.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2014.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment:
Learning augmentation policies from data. In arXiv preprint 1rxiv::8055095555 2018.
Anna Currey, Antonio Valerio Miceli-Barona and Kenneth Heafield. Copied monolingual data
improves low-resource neural machine translation. In WMT, 2017.
Chris Dyer, Victor Chahuneau, and Noah A Smith. simple, fast, and effective reparameterization of ibm model 2. In NAACL, 2013.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-translation scale. In EMNLP, 2018.
Marzieh Fadaee and Christof Monz. Back-translation sampling by targeting difficult words in neural machine translation. In EMNLP, 2018.
Marzich Fadace, Arianna Bisazza, and Christof Monz. Data augmentation for low-resource neural machine translation. In ACL, 2017.
Fei Gao, Jinhua Zhu, Lijun Wu, Yingce Xia, Tao Qin, Xueqi Cheng, Wengang Zhou, and Tie- Yan Liu. Soft contextual data augmentation for neural machine translation. In ACL, 2019.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence sequence learning. In ICML, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurlPS, 2014.
Di Hc, Yingee Xia, Tao Qin, Liwei Wang. Nenghai Yu, Tie- Yan Liu, and Wei- Ying Ma. Dual learning for machine translation. In NeurlPS, 2016.
Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. Iterative back- translation for neural machine translation. In WMT, 2018.
Sosuke Kobayashi. Contextual augmentation: Data augmentation by words with paradigmatic rela- tions. In NAACL. 2018.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurIPS, 2012.
8

---

Under review as a conference paper at ICLR 2021
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In ICLR, 2018.
Xuebo Liu, Derek F Wong, Yang Liu, Lidia S Chao, Tong Xiao, and Jingbo Zhu. Shared-private bilingual word embeddings for neural machine translation. In ACL, 2019.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. In ICLR, 2013.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurIPS, 2016.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In ACI. 2016.
Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks. In NeurlPS, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Shuo Wang, Yang Liu, Chao Wang, Huanbo Luan, and Maosong Sun. Improving hack-translation with uncertainty- confidence estimation. In EMNLP, 2019.
Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. Switchout: an efficient data augmenta- tion algorithm for neural machine translation. In Proceedings of the 2018 Conference on Empiri- cal Methods Natural Language Processing, pp. 856-861, 2018
Lijun Wu, Yiren Wang, Yingce Xia, QIN Tao, Jianhuang Lai, and Tie-Yan Liu. Exploiting mono- lingual data at scale for neural machine translation. In EMNLP, 2019.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.
Fengshun Xiao, Jiangtong Li, Hai Zhao, Rui Wang. and Kehai Chen. Lattice-based transformer encoder for neural machine translation. ACL, 2019.
Ziang Xie, Sida I Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, and Andrew Ng. Data noising smoothing in neural network language models. In ICLR, 2017.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2017.


Reviews:
[{"review_id": "rdApDRkuuiY", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a data augmentation technique where source sentences are perturbed by replacing (or mixing) source words with their aligned counterparts from the target language (while the target sentences remain as is). Alignments can be either obtained from an unsupervised aligner like fast-align or from the attention distribution of an NMT model. Perturbations are aimed to be semantically invariant to preserve the meaning of the source sentence. In addition to simply replacing the source word with the aligned word, authors also try out inputting a weighted combination of both the source word and the target word and refer to this method as “mixing”. Empirical observations suggest that simply replacing the source word with the aligned target yields better results.\n\nI believe the paper in the current form has a lot of scope for improvement. The experimental section needs to be strengthened and more thought needs to go into improving the proposed method.\n\nI have the following suggestions for improving the paper:\nMore experiments: The experiments are based on a reasonably large translation dataset, while the paper \"claims\" to improve the performance for low-resource NMT in section 4.5\n\n> Specially, we find that our method works better on a low resource settings\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "owRICe6FYYc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a method for data augmentation and/or regularization for machine translation that works by running a word aligner on the parallel data, and then with some probability \\gamma, replacing a source token with its corresponding target token or vice versa. A proposed variant also mixes the embeddings of the two words. Small improvements are shown over simpler noising strategies such as replacing words with placeholder tokens or with random words from the vocabulary.\n\nThis is a neat idea, I like the motivation that the corresponding word in the other language is likely to maintain semantic coherence (though at the cost of linguistic fluency). However, the results are simply not strong enough to warrant a strong recommendation. Table 1 shows that the proposed method performs about the same as the stronger baselines (+0.2 or +0.24 BLEU). Furthermore, with both the method and the baselines having a substantial random component, I strongly urge the authors to carry out several random replications of each experiment, so we can get error bars around these results, and perhaps carry out a replication-aware significance test, such as [1]. As it is, no attempt to do significance testing is made.\n\nI also think this work is missing some details: I assume that the swapping decision is made at runtime (and not at data construction time), allowing different swaps for the same sentence pair depending on the epoch, but it would be nice to be clear about this. I also assume that the authors have implemented shared source, target and output softmax embeddings, so as to strengthen the argument that equivalent words in different languages should have similar embeddings, but again, this should be made clear.\n\nI also think the authors missed some relevant baselines -- these are perhaps equivalent to some of the other baselines mentioned, but they should be discussed [2] [3] [4].\n\nI did not find the analogy to GANs or Figure 1 to be particularly useful for understanding this method.\n\nFinally, there are some spelling errors that indicate that a spell-checker wasn’t run:\n\nGarman → German\n\nnegrator → generator\n\n[1] Jonathan Clark, Chris Dyer, Alon Lavie, and Noah Smith, \"Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability\", Proceedings of the Association for Computational Lingustics, 2011. https://github.com/jhclark/multeval\n\n[2] SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation\nXinyi Wang, Hieu Pham, Zihang Dai, Graham Neubig, EMNLP 2018, https://arxiv.org/abs/1808.07512\n\n[3] Provilkov I, Emelianenko D, Voita E. BPE-dropout: Simple and effective subword regularization. ACL 2020. https://arxiv.org/abs/1910.13267\n\n[4] Robust Neural Machine Translation with Doubly Adversarial Inputs\nYong Cheng, Lu Jiang, Wolfgang Macherey. ACL 2019. https://arxiv.org/abs/1906.02443", "extra_scores": {}, "extra_text": {}}, {"review_id": "j5JWWZU-rHB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper shows that aligning parallel text with fastalign and then randomly replacing source words with their aligned target words, or interpolating their embeddings, improves machine translation.\n\nThis method is different from other data-augmentation methods that try to alter the source sentence without changing its meaning; here, the source sentence is altered into a mixture of the source and target. That’s interesting, but not very strongly motivated.\n\nThe paper doesn’t make clear whether the noise probability / coefficient is optimized on a development set or the test set. Based on Figures 3 and 4, it looks as though these hyperparameters may have been optimized on the test set, which is concerning. For both the baseline systems and your system, hyperparameters should be optimized on a development set and then tested using only a single hyperparameter setting on the test set. If this is what you did, please explicitly state this to reassure the reader.\n\nNot much attempt is made to explain why this method helps; the only analysis is a measurement of cosine similarity between five German-English word pairs. Do you tie word embeddings between the source and target languages (Press and Wolf, 2017)?\n\n- If so, one would expect that the transformer would already be able to place words with similar meanings close together, so the fact that your method improves this is interesting; do you know whether it helps more, e.g., for rare words, proper names, technical terms? Why is fastalign able to align some words better than the transformer? Would an even simpler method help, e.g., if (and only if) word f and word e both occur <= k times in the training data and they occur in exactly the same sentence pairs, then allow f to be switched to e?\n\n- If not, I'd suggest doing so and rerunning the experiments to see if you still get an improvement.\n\nOverall, this seems like a good trick for training NMT systems, but I would hope to see more insight either into why the proposed method works, or how NMT works or doesn’t work.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EwH-tvor1Ut", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes a GAN process for training neural machine translation models. The noise generator in this approach uses a switching-aligned-words technique where they randomly switch a word in the source sentence with its translation in the target sentence. They use fast-align to get alignments between source and target sentences. The experiments show that the noisy sentence pair generator performs best with the proposed switch and align approach in comparison with other (more random) methods. \n\n\n##########################################################################\n\nPros:\n\nUsing GANs for training an NMT model is an interesting idea. I like the idea of using word translations as replacements for data augmentation. Since the embeddings of these words are close, these can be good candidates for the noise generator to fool the discriminator.  \n\nThe paper covers the literature quite well. \n\n##########################################################################\n\nCons:\n\nThe intuition of the paper is not clearly defined. \n\nThe experiments do not cover other augmentation methods. In this paper, the authors compare their approach to the following baselines: SWAP, DROPOUT, BLANK, SMOOTH. However, these are baseline approaches for the noise generator component. It would be valuable to know how this approach performs in comparison with other augmentation approaches such as [1] and [2].\n\nAll baseline methods (SWAP, DROPOUT, BLANK, SMOOTH) have an element of randomness in them and are not strong baselines. This provides little insight for understanding the impact of the bilingual switching. \n\nThe lack of comparison with the literature makes it difficult to evaluate the reported results. \n\nIt would be insightful to cover at least another language pair where the relation between the source and the target language is different. For instance, two languages that are not similar at all (structurally, morphologically, or semantically). It's interesting to see how this model performs when the assumption that word embeddings of the same word in two different languages are close to each other. \n\n#########################################################################\n\nSome typos:\n\n(1) Typo on page 1: additional monolingual corpus is uesd -> additional monolingual corpus is used\n\n(2) Typo in Figure 1: SRT -> SRC\n\n(3) Typo on page 4:  words with similar meandings  ->  words with similar meanings  \n\n(4) Typo on page 7: we have aligned words pairs -> we have aligned word pairs\n\n(5) Typo on page 7: switching aligned words is helpeful -> switching aligned words is helpful \n\n#########################################################################\n\nReferences:\n\n[1] Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In ACL, 2016.\n\n[2] Marzieh Fadaee, Arianna Bisazza, and Christof Monz. Data augmentation for low-resource neural machine translation. In ACL, 2017.\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed to reject.
Decision: Reject