Dual Skew Divergence Loss for Neural Machine Translation

1 INTRODUCTION . Neural machine translation ( NMT ) ( Kalchbrenner & Blunsom , 2013 ; Sutskever et al. , 2014 ) has shown remarkable performance in terms of sequence-to-sequence learning framework for diverse language pairs . Unlike the conventional statistical machine translation ( SMT ) ( Koehn et al. , 2003 ) that explicitly models linguistic features of the training corpus , NMT aims at building an end-to-end model that directly transforms a source language sequence to the target one . With the development of attention mechanism ( Bahdanau et al. , 2015 ; Luong et al. , 2015 ) , NMT has achieved results on par with or even better than SMT . During NMT training , maximum likelihood ( ML ) is the most commonly used strategy that maximizes the likelihood of the target sentence conditioned on the source throughout training corpus . In practice , ML-based loss is often represented with a word-level cross entropy form , which has proven to be effective for NMT modeling . However , Ranzato et al . ( 2016 ) pointed out that ML training suffers from two drawbacks . First , the model is only exposed to training distribution and ignores its own prediction errors during training . Second , a word-level loss is used to optimize the model parameters at the training time while at inference the model prediction is evaluated by a sequence-level metric such as BLEU ( Papineni et al. , 2002 ) . To handle such problems , several recent work focused on the research of more effective and direct training strategies . Bengio et al . ( 2015 ) advocated a curriculum learning approach that gradually forces the model to deal with its own mistakes as it does during inference . Wiseman & Rush ( 2016 ) proposed a sequence-level loss function in terms of errors made during beam search . Shen et al . ( 2016 ) applied minimum risk training ( MRT ) from SMT to optimize NMT modeling directly in terms of BLEU score . Some other work resorted to reinforcement learning based approaches ( Ranzato et al. , 2016 ; Bahdanau et al. , 2017 ) . In this work , we introduce a novel loss called dual skew divergence ( DSD ) to compensate for the original ML-based loss . It can be proved that maximizing likelihood is equal to minimizing the Kullback-Leibler ( KL ) divergence ( Kullback & Leibler , 1951 ) KL ( P ||Q ) between the real data distribution P and the model prediction Q . According to Huszár ( 2015 ) , minimizing KL ( P ||Q ) tends to find aQ that covers all the true data distribution and ignores the rest of incorrect candidates , which will lead to models that overgeneralize and generate implausible samples during inference . Meanwhile , minimizing the opposite form KL ( Q||P ) has a distinct property that tends to model a Q that avoids assigning probability mass to the unlikely terms . To benefit from both of these two divergence and balance the tradeoff between the original ML principle and the concern about the undesirable model prediction , we interpolate KL ( P ||Q ) and KL ( Q||P ) to form a new DSD loss . We carry out all the experiments on both English-German and English-French translation tasks from WMT 2014 and compare our models to some other work with similar model size and dataset . The early experiments indicate that switching to DSD loss after the convergence of ML training , a simulated annealing mechanism is actually introduced and it indeed helps the model jump out a local optimum within a short time . Evaluations on the test sets show that our DSD-extended models are better than the ML-only ones and outperform a series of strong baselines significantly . 2 NEURAL MACHINE TRANSLATION . In this paper , we closely follow the neural machine translation model proposed by Bahdanau et al . ( 2015 ) .1 Different from the conventional statistical models , it is mainly based on a neural encoderdecoder network with attention mechanism ( Bahdanau et al. , 2015 ; Luong et al. , 2015 ) . In this section , we will give a brief introduction about this basic architecture . The encoder is a bidirectional recurrent neural network ( RNN ) such as Gated Recurrent Unit ( GRU ) ( Cho et al. , 2014 ) or Long Short-Term Memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) . The forward RNN reads an input sequence x = ( x1 , ... , xm ) from left to right and calculates a forward sequence of hidden states ( −→ h 1 , ... , −→ hm ) as the representation of the source sentence . Similarly , the backward RNN reads the input sequence in a reverse direction and learns a backward sequence ( ←− h 1 , ... , ←− hm ) . The hidden states of the two RNNs −→ h i and ←− h i are concatenated to obtain the source annotation vector hi = [ −→ h i , ←− h i ] T as the initial state of the decoder . The decoder is a forward RNN that predicts a corresponding translation y = ( y1 , ... , yn ) step by step . The translation probability can be formulated as follows : p ( yj |y < j , x ) = q ( yj−1 , sj , cj ) , where sj and cj denote the decoding state and the source context at the j-th time step respectively . Here , q ( · ) is the softmax layer and y < j = ( y1 , ... , yj−1 ) . Specifically , sj = g ( yj−1 , sj−1 , cj ) , where g ( · ) is the corresponding RNN unit . The context vector cj is calculated as a weighted sum of the source annotations hi on the basis of the attention mechanism : cj = m∑ i=1 αjihi . The alignment model αji defines the probability that how well yj is aligned to xi , which is a single layer feed-forward neural network . The whole model is jointly trained to seek the optimal parameters that can be used to correctly encode the source sentences and decode them to the corresponding target sentences . 3 DUAL SKEW DIVERGENCE LOSS FOR NMT . 3.1 CROSS ENTROPY . In information theory , cross entropy is an important concept measuring the difference between two probability distributions . In natural language processing , cross entropy is usually used as the evaluation metric of language modeling . During NMT training , word-level cross entropy broadly serves as the loss function to learn the model parameters . The NMT model aims at training a single , large neural network that directly transforms a given source sequence x = ( x1 , x2 , ... , xm ) to the corresponding target sequence y = ( y1 , y2 , ... , yn ) . 1During this study , we are aware that there come advanced models such as Transformer ( Vaswani et al. , 2017 ) and ConvS2S ( Gehring et al. , 2017 ) . To focus on the loss improvement , we still keep a simple architecture as baseline in this paper . Given a set of training examples D = { 〈x ( s ) , y ( s ) 〉 } Ss=1 , the standard ML training objective is to maximize the log-likelihood of the training data with respect to the parameters θ : θ̂ML = argmax θ { L ( θ ) } , L ( θ ) = S∑ s=1 logP ( y ( s ) |x ; θ ) = S∑ s=1 n∑ i=1 logP ( y ( s ) i |x ( s ) , y ( s ) < i ; θ ) . To fit the gradient descent method , the objective is actually transformed to minimize the negative log-likelihood . In practice , it is realized by minimizing the word-level cross entropy that can be calculated at each time step of decoding . Given an observed target sequence of length n , the cross entropy loss in the vector form can be represented as JXENT = − n∑ i=1 yi log ( ŷi ) , ( 1 ) where yi is a one-hot vector , referring to the correct target label at time step i and ŷi is the model approximation given by the softmax layer . 3.2 KULLBACK-LEIBLER DIVERGENCE . Kullback-Leibler ( KL ) divergence ( Kullback & Leibler , 1951 ) is also a measurement that calculates the distance between two random probability distributions denoted by : DKL ( P ||Q ) = Ex∼P [ logP ( x ) − logQ ( x ) ] , ( 2 ) where P is the true data distribution of target words and Q is the distribution of model prediction . Mathematically , there is some connection between KL divergence DKL ( P ||Q ) and cross entropy H ( P , Q ) , which can be described as follows : H ( P , Q ) = H ( P ) +DKL ( P ||Q ) , where H ( P ) refers to the entropy of data itself . Since the gradient of H ( P ) with respect to model parameters about Q is always equal to zero , training with H ( P , Q ) is identical to that with DKL ( P ||Q ) . In other words , minimizing cross entropy loss actually means minimizing KL divergence . 3.3 ASYMMETRY OF KL DIVERGENCE . Exchanging the position of P and Q , we can easily get DKL ( Q||P ) that is in an inverse direction with DKL ( P ||Q ) . Following Eq . ( 2 ) , DKL ( Q||P ) can be also rewritten as DKL ( Q||P ) = Ex∼Q [ − logP ( x ) + logQ ( x ) ] . Obviously , DKL ( P ||Q ) is not identical to its inverse form DKL ( Q||P ) . Huszár ( 2015 ) showed that minimizing DKL ( P ||Q ) tends to find a Q that covers all modes ( peak value ) of P at the cost of placing probability mass where P has none . However , minimizing DKL ( Q||P ) leads to a modeseeking behavior : the optimal Q will concentrate around the largest mode of P while completely ignore smaller modes . In the case of NMT , multiple target candidate words could have the same high probabilities at each step of decoding . Consequently , we may have more than one best choice at the same time . Thereby , the different effects of training with a word-level DKL ( P ||Q ) or DKL ( Q||P ) lie in two aspects : First , the model trained by minimizing DKL ( P ||Q ) will tend to memorize all possible outputs and some words with low probabilities in P will also be assigned high probabilities ; Second , the model trained by minimizing DKL ( Q||P ) will instead try to memorize only a subset of all possible outputs , while the cost is , some words with high probabilities in P will be ignored . In general , the two different objectives have their own advantages and disadvantages , so it is hard to say which is actually better . Despite of this , it still provides us with a new direction of choosing the proper loss function . 3.4 SKEW DIVERGENCE . The analysis above shows that DKL ( Q||P ) may be also a good alternative loss function for NMT training . However , DKL ( Q||P ) is only well-defined when P is never equal to zero , which is not guaranteed in the NMT training . To overcome the inconvenience , we use some kind of approximations instead . One such approximation function is the α-skew divergence ( Lee , 1999 ) : sα ( Q , P ) = DKL ( Q||αQ+ ( 1− α ) P ) , where α controls the degree to which the function approximates DKL ( Q||P ) and 0 ≤ α ≤ 1 . By assigning a small value to α , we can simulate the behavior of minimizing DKL ( Q||P ) with α-skew divergence approximation . In our experiments , α is set to 0.01 constantly .

Summary: This paper describes an alternative training objective to cross-entropy loss for sequence-to-sequence models. The key observation is that cross-entropy is minimizing KL(P|Q) for a data distribution P and a model distribution Q; they add another loss that minimizes the inverse KL(Q|P) to create their dual-skew divergence. The idea is tested in the context of neural MT, using a model similar to that proposed by Bahdanau et al. (2015) with results on English-to-French and English-to-German WNT 2014. In the context of beam search, improvements are small (<=0.5 BLEU) but statistically significant.