Understanding and Improving Lexical Choice in Non-Autoregressive Translation

1 INTRODUCTION . When translating a word , translation models need to spend a substantial amount of its capacity in disambiguating its sense in the source language and choose a lexeme in the target language which adequately express its meaning ( Choi et al. , 2017 ; Tamchyna , 2017 ) . However , neural machine translation ( NMT ) has a severe problem on lexical choice , since it usually has mistranslation errors on low-frequency words ( Koehn & Knowles , 2017 ; Nguyen & Chiang , 2018 ; Gu et al. , 2020 ) . In recent years , there has been a growing interest in non-autoregressive translation ( NAT , Gu et al. , 2018 ) , which improves decoding efficiency by predicting all tokens independently and simultaneously . Well-performed NAT models are generally trained on synthetic data distilled by autoregressive translation ( AT ) teachers instead of the raw training data ( Figure 1 ( a ) ) ( Stern et al. , 2019 ; Lee et al. , 2018 ; Ghazvininejad et al. , 2019 ; Gu et al. , 2019 ; Hao et al. , 2021 ) . Recent studies have revealed that knowledge distillation ( KD ) reduces the modes ( i.e . multiple lexical choices for a source word ) in the raw data by re-weighting the training examples ( Furlanello et al. , 2018 ; Tang et al. , 2020 ) , which lowers the intrinsic uncertainty ( Ott et al. , 2018 ) and learning difficulty for NAT ( Zhou et al. , 2020 ; Ren et al. , 2020 ) . However , the side effect of KD has not been fully studied . In this work , ∗Work was done when Liang Ding and Xuebo Liu were interning at Tencent AI Lab . we investigate this problem from the perspective of lexical choice , which is at the core of machine translation . We argue that the lexical choice errors of AT teacher can be propagated to the NAT model via the distilled training data . To verify this hypothesis , we qualitatively compare raw and distilled training corpora . Table 1 lists all samples whose source sentences contain the place name “ 纽马基特 ” . In the raw corpus ( “ RAW-TGT ” ) , this low-frequency word totally occurs three times and corresponds to correct translation “ Newmarket ” . However , in the KD corpus ( “ KD-TGT ” ) , the word is incorrectly translated into a person name “ Newmargot ” ( Margot Robbie is an Australian actress ) or organization name “ Newmarquette ” ( Marquette is an university in Wisconsin ) or even invalid one “ Newmarquite ” . Motivated by this finding , we explore NAT from the lexical choice perspective . We first validate our hypothesis by analyzing the lexical choice behaviors of NAT models ( §3 ) . Concretely , we propose a new metric AoLC ( accuracy of lexical choice ) to evaluate the lexical translation accuracy of a given NAT model . Experimental results across different language pairs show that NAT models trained on distilled data have higher accuracy of global lexical translation ( AoLC↑ ) , which results in better sequence generation . However , fine-grained analyses revealed that although KD improves the accuracy on high-frequency tokens , it meanwhile harms performance on low-frequency ones ( Low freq . AoLC↓ ) . And with the improvement of teacher models , this issue becomes more severe . We conclude that the lexical choice of the low-frequency tokens is a typical kind of lost information when using knowledge distillation from AT model . In order to rejuvenate this lost information in raw data , we propose to expose the raw data to the training of NAT models , which augments NAT models the ability to learn the lost knowledge by themselves . Specifically , we propose two bi-lingual lexical-level data-dependent priors ( Word Alignment Distribution and Self-Distilled Distribution ) extracted from raw data , which is integrated into NAT training via Kullback-Leibler divergence . Both approaches expose the lexical knowledge in the raw data to NAT , which makes it learn to restore the useful information of low-frequency words to accomplish the translation . We validated our approach on several datasets that widely used in previous studies ( i.e . WMT14 En-De , WMT16 Ro-En , WMT17 Zh-En , and WAT17 Ja-En ) and model architectures ( i.e . MaskPredict ( Ghazvininejad et al. , 2019 ) and Levenshtein Transformer ( Gu et al. , 2019 ) ) . Experimental results show that the proposed method consistently improve translation performance over the standard NAT models across languages and advanced NAT architectures . The improvements come from the better lexical translation accuracy ( low-frequency tokens in particular ) of NAT models ( AoLC↑ ) , which leads to less mis-translations and low-frequency words prediction errors . The main contributions of this work are : • Our study reveals the side effect of NAT models ’ knowledge distillation on low-frequency lexicons , which makes the standard NAT training on the distilled data sub-optimal . • We demonstrate the necessity of letting NAT models learn to distill lexical choices from the raw data by themselves . • We propose an simple yet effective approach to accomplish this goal1 , which are robustly applicable to several model architectures and language pairs . 2 PRELIMINARIES . 2.1 NON-AUTOREGRESSIVE TRANSLATION . The idea of NAT has been pioneered by Gu et al . ( 2018 ) , which enables the inference process goes in parallel . Different from AT models that generate each target word conditioned on previously generated ones , NAT models break the autoregressive factorization and produce target words in parallel . Given a source sentence x , the probability of generating its target sentence y with length T is calculated as : p ( y|x ) = pL ( T |x ; θ ) T∏ t=1 p ( yt|x ; θ ) ( 1 ) 1Code is available at : https : //github.com/alphadl/LCNAT where pL ( · ) is a separate conditional distribution to predict the length of target sequence . During training , the negative loglikelihood loss function of NAT is accordingly LNAT ( θ ) = − log p ( y|x ) . To bridge the performance gap between NAT and AT models , a variety approaches have been proposed , such as multi-turn refinement mechanism ( Lee et al. , 2018 ; Ghazvininejad et al. , 2019 ; Gu et al. , 2019 ; Kasai et al. , 2020 ) , rescoring with AT models ( Wei et al. , 2019 ; Ma et al. , 2019 ; Sun et al. , 2019 ) , adding auxiliary signals to improve model capacity ( Wang et al. , 2019 ; Ran et al. , 2019 ; Guo et al. , 2019 ; Ding et al. , 2020 ) , and advanced training objective ( Wei et al. , 2019 ; Shao et al. , 2019 ; Ma et al. , 2020 ) . Our work is complementary to theirs : while they focus on improving NAT models trained on the distilled data , we refine the NAT models by exploiting the knowledge in the raw data . Sentence-Level Knowledge Distillation NAT models suffer from the multimodality problem , in which the conditional independence assumption prevents a model from properly capturing the highly multimodal distribution of target translations . For example , one English source sentence “ Thank you. ” can be accurately translated into German as any one of “ Danke. ” , “ Danke schön. ” or “ Vielen Dank. ” , all of which occur in the training data . To alleviate this problem , Gu et al . ( 2018 ) applied sequence-level KD ( Kim & Rush , 2016 ) to construct a synthetic corpus , whose target sentences are generated by an AT model trained on the raw data , as shown in Figure 1 ( a ) . The NAT model is only trained on distilled data with lower modes , which makes it easily acquire more deterministic knowledge ( e.g . one lexical choice for each source word ) . While separating KD and model training makes the pipeline simple and efficient , it has one potential threat : the re-weighted samples distilled with AT model may have lost some important information . Lee et al . ( 2020 ) show that distillation benefits the sequence generation but harms the density estimation . In this study , we exploit to bridge this gap by exposing the raw data to the training of NAT models , as shown in Figure 1 ( b ) . 2.2 EXPERIMENTAL SETUP . Datasets Experiments were conducted on four widely-used translation datasets : WMT14 EnglishGerman ( En-De , Vaswani et al . 2017 ) , WMT16 Romanian-English ( Ro-En , Gu et al . 2018 ) , WMT17 Chinese-English ( Zh-En , Hassan et al . 2018 ) , and WAT17 Japanese-English ( Ja-En , Morishita et al . 2017 ) , which consist of 4.5M , 0.6M , 20M , and 2M sentence pairs , respectively . We use the same validation and test datasets with previous works for fair comparison . To avoid unknown words , we preprocessed data via BPE ( Sennrich et al. , 2016 ) with 32K merge operations . The GIZA++ ( Och & Ney , 2003 ) was employed to build word alignments for the training datasets . We evaluated the translation quality with BLEU ( Papineni et al. , 2002 ) . NAT Models We validated our research hypotheses on two SOTA NAT models : • MaskPredict ( MaskT , Ghazvininejad et al . 2019 ) that uses the conditional mask LM ( Devlin et al. , 2019 ) to iteratively generate the target sequence from the masked input . We followed its optimal settings to keep the iteration number be 10 and length beam be 5 , respectively . • Levenshtein Transformer ( LevT , Gu et al . 2019 ) that introduces three steps : deletion , placeholder prediction and token prediction . The decoding iterations in LevT adaptively depends on certain conditions . For regularization , we tune the dropout rate from [ 0.1 , 0.2 , 0.3 ] based on validation performance in each direction , and apply weight decay with 0.01 and label smoothing with = 0.1 . We train batches of approximately 128K tokens using Adam ( Kingma & Ba , 2015 ) . The learning rate warms up to 5× 10−4 in the first 10K steps , and then decays with the inverse square-root schedule . We followed the common practices ( Ghazvininejad et al. , 2019 ; Kasai et al. , 2020 ) to evaluate the translation performance on an ensemble of top 5 checkpoints to avoid stochasticity . AT Teachers We closely followed previous works on NAT to apply sequence-level knowledge distillation ( Kim & Rush , 2016 ) to reduce the modes of the training data . More precisely , to assess the effectiveness of our method under different of AT teachers , we trained three kinds of Transformer ( Vaswani et al. , 2017 ) models , including Transformer-BASE , Transformer-BIG and Transformer-STRONG . The main results employ LARGE for all directions except Ro-En , which is distilled by BASE . The architectures of Transformer-BIG and Transformer-STRONG are unchanged , but STRONG utilizes a large batch ( 458K tokens ) training strategy . 3 UNDERSTANDING LEXICAL CHOICE IN NAT MODELS . 3.1 EVALUATING LEXICAL CHOICE OF NAT MODELS . Recently , Zhou et al . ( 2020 ) argue that knowledge distillation is necessary for the uncertain nature of the machine translation task . Accordingly , they propose a metric to estimate the complexity of the data ( CoD ) , which is driven from an external word alignment model . They reveal that the distilled data is indeed less complex , which facilitates easier training for the NAT model . Inspired by this , we propose a metric to measure the lexical level accuracy of model predictions . Accuracy of Lexical Choice ( AoLC ) evaluates the accuracy of target lexicon chosen by a trained NAT model M for each source word . Specifically , the model M takes a source word f as the input , and produce a hypothesis candidate list with their corresponding word confidence : PMf = { PM ( e1|f ) , . . . , PM ( e|Vtrg||f ) } ( 2 ) where Vtrg is the target side vocabularies over whole corpus . The AoLC score is calculated by averaging the probability of the gold target word ef of each source word f : AoLC = ∑ f∈Vtestsrc PM ( ef |f ) |Vtestsrc | ( 3 ) where Vtestsrc is the set of source side tokens in test set . Each gold word ef is chosen with the help of the word alignment model PAf . The chosen procedure is as follows : Step 1 ) collecting the references of the source sentences that contains source word f , and generating the target side word bag Bf with these references . Step 2 ) Descending PAf in terms of alignment probabilities and looking up the word that first appears in Bf as the gold word until the Bf is traversed . Step 3 ) If the gold word is still not found , let the word with the highest alignment probability in PAf as the gold word . Generally , higher accuracy of lexical translation represents more confident of the predictions . We discuss the reliability of word alignment-based AoLC in Appendix A.1 .

Summary: This paper analyzes the side effect of knowledge distillation in NAT where the lexical choice errors on low-frequency words are propagated to the student model from the teacher. Tackling on this, the paper then proposes to expose raw data to restore such information. In my view, the submission is well motivated and the designed experiments and results are meaningful and convincing which deserves an accept. However, as the paper focuses on analyzing a specific point (lexical choice) in a very constrained setting (NAT), the overall contribution might be incremental compared to other works in general at such a venue like ICLR.