Minimum Divergence vs. Maximum Margin: an Empirical Comparison on Seq2Seq Models

1 INTRODUCTION . Sequence to sequence ( seq2seq ) models ( Kalchbrenner & Blunsom , 2013 ; Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ; Luong et al. , 2015 ) are a powerful end-to-end solution to sequence prediction since they allow a mapping between two sequences of different lengths . However , Maximum Likelihood Estimation ( MLE ) which is used as a standard training criterion in seq2seq has a series of drawbacks : • Training and evaluation mismatch : during training , we maximize the log likelihood , while during inference , the model is evaluated by a different metric such as BLEU or ROUGE ; • MLE fails to assign proper scores to different model outputs , which means that all incorrect outputs are treated equally . Similar to SEARN ( Daumé et al. , 2009 ) and DAGGER ( Ross et al. , 2011 ) in traditional models , many reinforcement learning ( RL ) techniques have been extended to seq2eq models recently to solve the mismatch between training and inference . The key idea is to replace MLE with a criterion that gives task-specific scores to different model outputs . Recent approaches to taking the evaluation metric into training criteria generally fall into one of the following two categories : 1. minimum divergence : Minimum divergence methods minimize some distance functions between the model output distribution and the probability distribution given by the ground truth , and by optimizing the divergence between the two distributions , the model output distribution will ideally be equal to the ground truth distribution . 2. maximum margin : During inference , in structured prediction , the goal is to simply ensure good structures have the highest predicted score . That is , margin based learning differs from minimum divergence learning in the aspect that it only requires that the predicted score of the correct output should be higher than that of the others by a margin . ∗Corresponding author . The main contribution of this paper can be summarized as : based on the analysis of existing work , we introduce a novel training criterion for seq2seq learning and show its effectiveness . Based on our experimental results on the tasks of machine translation and sentence summarization , we conclude that some models that take the evaluation metric into consideration can improve over a strong MLE baseline by a large margin , and our new training criterion can usually get better results than existing work . 2 RELATED WORK . Seq2seq models were proposed and improved by Kalchbrenner & Blunsom ( 2013 ) and Sutskever et al . ( 2014 ) . With the attention mechanism ( Bahdanau et al. , 2015 ; Luong et al. , 2015 ) , seq2seq models achieve better results than traditional models in many fields including machine translation ( Bahdanau et al. , 2015 ; Luong et al. , 2015 ) and text summarization ( Chopra et al. , 2016 ) . Before seq2seq models , previous work on optimizing task specific evaluation metric scores generally includes SEARN ( Daumé et al. , 2009 ) , DAGGER ( Ross et al. , 2011 ) , Minimum Error Rate Training ( MERT ) ( Och , 2003 ) , and softmax-margin criterion ( Gimpel & Smith , 2010 ) . Seq2seq model uses MLE as the training objective , which also has the problem of training and evaluation mismatch . Ranzato et al . ( 2016 ) incorporated the evaluation metric into training of seq2seq models and proposed the mixed incremental cross entropy REINFORCE ( MIXER ) training strategy , which is similar to the idea of Minimum Risk Training ( MRT ) ( Smith & Eisner , 2006 ; Li & Eisner , 2009 ; Ayana et al. , 2016 ; Shen et al. , 2016 ) . MIXER used decoder hidden states to predict the bias term to reduce the variance , while MRT renormalized the predicted probabilities . Inspired by MIXER , Bahdanau et al . ( 2017 ) extended the actor critic algorithm ( Konda & Tsitsiklis , 2000 ; Sutton et al. , 2000 ) to the seq2seq framework . Sokolov et al . ( 2016 ) and Kreutzer et al . ( 2017 ) proposed the pairwise preference learning framework , which enables the model to learn from the difference between a pair of samples generated by the model . Another popular model in the form of divergence minimization is the smoothed version of MERT algorithm proposed by Och ( 2003 ) . The smoothed MERT algorithm also minimizes the expected loss over samples from model outputs and is similar to MRT essentially . Norouzi et al . ( 2016 ) proposed Reward Augmented Maximum Likelihood ( RAML ) which also optimizes the divergence by data augmentation , and RAML was further combined with maximum likelihood via α-divergence recently ( Koyamada et al. , 2017 ) . In contrast to the above divergence minimization based approaches , Wiseman & Rush ( 2016 ) used a margin based training objective and their search optimization process requires performing parameter updates or inference each time a new word is generated . One main drawback of maximum margin is that given a sample subset of an arbitrary size , only two samples of the subset can contribute to the gradient . The softmax-margin criterion proposed by Gimpel & Smith ( 2010 ) can alleviate this problem , in which all samples are used in loss and gradient computation . However , Edunov et al . ( 2018 ) shows that the softmax-margin criterion still can not reach the performance of risk minimization . As the latest work , Edunov et al . ( 2018 ) is very similar to ours . However , our interests are in finding better training criteria on a consistent setting , while they mainly focused on absolute performance improvement , using techniques like combination with token level loss , online vs. offline candidate generation , etc . 3 SEQUENCE TO SEQUENCE MODEL . This paper considers the seq2seq model from Bahdanau et al . ( 2015 ) and Luong et al . ( 2015 ) , which consists of an RNN encoder and an RNN decoder . Due to the gradient vanishing problem , the vanilla RNN cell is replaced with a Gated Recurrent Unit ( GRU ) ( Cho et al. , 2014 ) or Long ShortTerm Memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) . A bidirectional RNN is used as encoder , with one RNN reading the source sequence from beginning to end , and the other RNN reading in reversed order . The bidirectional RNN encoder encodes the input sequence and gets a sequence of hidden states by concatenating the hidden states of each word in the two directions , which will be used as attention information in the decoder RNN . The decoder RNN is initialized by passing the two final hidden states of the encoder to a feed forward network . The mixed information used as input in the first step m0 is initialized by a zero vector . In decoding step i , using the input feeding approach described in Luong et al . ( 2015 ) , the decoder RNN reads the embedding of the wordwi and the mixed informationmi−1 , to generate the hidden state of current step hi : hi = f1 ( concat ( wi , mi−1 ) , hi−1 ) , where f1 is the RNN cell . The decoder then generates the attention context ci from encoder using hi . Since there are many different attention mechanisms , we omit the details here . After getting ci , the model mixes ci and hi together by a feedforward network f2 to generate mi , which is used as the mixed information in the next step : mi = f2 ( ci , hi ) Finally , mi is passed to a feedforward network to generate oi , and the softmax layer outputs the probability for each word at step i . Suppose the target word in step i is v′ , the probability of the target word in step i is given by p ( yi = v ′ |x , y1 : i−1 ) = exp ( oiv′ ) ∑ v exp ( oiv ) . For the sequence y = 〈y1 , y2 , ... , yl〉 , the probability p ( y |x ) is p ( y |x ) = p ( y1 |x ) p ( y2 |x , y1 ) ... p ( yl |x , y1 : l−1 ) . Suppose y∗ is the correct output , MLE actually maximizes the log likelihood of p ( y∗ |x ) : LMLE = − log p ( y∗ |x ) = − l∑ i=1 log p ( y∗i |x , y∗ < i ) For a detailed explanation , please refer to Luong et al . ( 2015 ) . Now we describe the algorithm to construct a subset S from all possible output sequences which will be used in Section 4 & 5 . In seq2seq model , the n-best list can only be approximated by performing beam search , as used in Wiseman & Rush ( 2016 ) , or Monte Carlo ( MC ) sampling , as used in Ayana et al . ( 2016 ) and Shen et al . ( 2016 ) . While using beam search may produce more accurate results , the time required for generating a candidate set of the same size as MC sampling using beam search is too long . In this paper , we construct the approximated n-best list S using MC sampling proposed in Shen et al . ( 2016 ) , and for completeness , we describe the sampling approach in Appendix 8.1 in details . Please note that the sampling algorithm does not guarantee the size of the sample subset , since it is inefficient to force the model to generate an exact number of distinct samples using GPU . 4 MINIMUM DIVERGENCE . In this section , we consider training criteria in the form of the following equation : L = D ( p1 || p2 ) , where p1 and p2 are two probability distributions given by the model or the ground truth , and D is a distance function , for example , Kullback Leibler ( KL ) divergence or cross entropy . The baseline training criterion MLE can also be seen as minimum divergence training , where the cross entropy between the model output distribution and the one-hot encoded ground truth is minimized . Now we will discuss training criteria in which the ground truth distribution contains information of evaluation metric scores . 4.1 REWARD AUGMENTED MAXIMUM LIKELIHOOD . Suppose that { x , y∗ } is a given input-output pair , the model learns parameters θ and gives prediction p ( y |x ; θ ) on the output space Y , the exponentiated payoff distribution ( Norouzi et al. , 2016 ) q ( y |y∗ ) is defined as follows : q ( y |y∗ ) = 1 Z ( y∗ ) exp [ r ( y , y∗ ) /τ ] , ( 1 ) where Z ( y∗ ) = ∑ y∈Y exp [ r ( y , y ∗ ) /τ ] is the normalization factor . τ is a hyperparameter that controls the smoothness of the distribution , and r ( y , y∗ ) is the score of prediction y given by the evaluation metric , e.g . BLEU ( Papineni et al. , 2002 ) , or ROUGE ( Lin , 2004 ) The training objective of Reward augmented Maximum Likelihood ( RAML ) ( Norouzi et al. , 2016 ) is defined as LRAML = −Eq ( y |y∗ ) [ log p ( y |x ; θ ) ] , ( 2 ) in which , the gradient is : Eq ( y |y∗ ) [ ∇θ log p ( y |x ; θ ) ] = − ∑ y∈Y q ( y |y∗ ) ∇θ log p ( y |x ; θ ) ( 3 ) Equation ( 3 ) requires computing q ( y |y∗ ) for all y and performing gradient descent on the whole Y . However , due to the huge search space of Y and the high computational cost of r ( y , y∗ ) , Equation ( 3 ) can not be directly computed in an efficient way . Norouzi et al . ( 2016 ) proposed using stratified sampling . One can sample from negative edit distance or Hamming distance and then do importance reweighting to reweight the samples with task specific scores . After sampling from q ( y |y∗ ) , the model is trained to maximize log p ( y |x ; θ ) . Another choice is to construct a sample subset S from the model outputs using MC sampling , and use Equation ( 1 ) to compute the probability q′ ( y |y∗ ) in the sample subset . The training criterion is then changed to L′RAML = − ∑ y∈S q′ ( y |y∗ ) log p ( y |x ; θ ) , where q′ ( y |y∗ ) = exp [ r ( y , y ∗ ) /τ ] ∑ y′∈S exp [ r ( y ′ , y∗ ) /τ ] .

Summary: In this paper the authors distinguish between two families of training objectives for seq2seq models, namely, divergence minimization objectives and max-margin objectives. They primarily focus on the divergence minimization family, and show that the MRT and RAML objectives can be related to minimizing the KL divergence between the model's distribution over outputs and the "exponentiated payoff distribution," with the two objectives differing in terms of the direction of the KL. In addition, the authors propose an objective using the Hellinger distance rather than the KL divergence, and they conduct experiments on machine translation and summarization comparing all the considered objectives.