Title: MINIMUM DIVERGENCE VS. MAXIMUM MARGIN: AN EMPIRICAL COMPARISON ON SEQ2SEQ MODELS

Abstract: Sequence to sequence (seq2seq) models have become a popular framework for neural sequence prediction. While traditional seq2seq models are trained by Maximum Likelihood Estimation (MLE), much recent work has made various attempts to optimize evaluation scores directly to solve the mismatch between training and evaluation, since model predictions are usually evaluated by a task specific evaluation metric like BLEU or ROUGE scores instead of perplexity. This paper puts this existing work into two categories, a) minimum divergence, and b) maximum margin. We introduce a new training criterion based on the analysis of existing work, and empirically compare models in the two categories. Our experimental results show that our new training criterion can usually work better than existing methods, on both the tasks of machine translation and sentence summarization.

1. INTRODUCTION:

Sequence to sequence (seq2seq) models (Kalchbrenner & Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015) are a powerful end-to-end solution to sequence prediction since they allow a mapping between two sequences of different lengths. However, Maximum Likelihood Estimation (MLE) which is used as a standard training criterion in seq2seq has a series of drawbacks:

• Training and evaluation mismatch: during training, we maximize the log likelihood, while during inference, the model is evaluated by a different metric such as BLEU or ROUGE;

• MLE fails to assign proper scores to different model outputs, which means that all incorrect outputs are treated equally.

Similar to SEARN (Daumé et al., 2009) and DAGGER (Ross et al., 2011) in traditional models, many reinforcement learning (RL) techniques have been extended to seq2eq models recently to solve the mismatch between training and inference. The key idea is to replace MLE with a criterion that gives task-specific scores to different model outputs.

Recent approaches to taking the evaluation metric into training criteria generally fall into one of the following two categories:

1. minimum divergence: Minimum divergence methods minimize some distance functions between the model output distribution and the probability distribution given by the ground truth, and by optimizing the divergence between the two distributions, the model output distribution will ideally be equal to the ground truth distribution.

2. maximum margin: During inference, in structured prediction, the goal is to simply ensure good structures have the highest predicted score. That is, margin based learning differs from minimum divergence learning in the aspect that it only requires that the predicted score of the correct output should be higher than that of the others by a margin.

The main contribution of this paper can be summarized as: based on the analysis of existing work, we introduce a novel training criterion for seq2seq learning and show its effectiveness.

Based on our experimental results on the tasks of machine translation and sentence summarization, we conclude that some models that take the evaluation metric into consideration can improve over a strong MLE baseline by a large margin, and our new training criterion can usually get better results than existing work.

2. RELATED WORK:

Seq2seq models were proposed and improved by Kalchbrenner & Blunsom (2013) and Sutskever et al. (2014) . With the attention mechanism (Bahdanau et al., 2015; Luong et al., 2015) , seq2seq models achieve better results than traditional models in many fields including machine translation (Bahdanau et al., 2015; Luong et al., 2015) and text summarization (Chopra et al., 2016) .

Before seq2seq models, previous work on optimizing task specific evaluation metric scores generally includes SEARN (Daumé et al., 2009) , DAGGER (Ross et al., 2011) , Minimum Error Rate Training (MERT) (Och, 2003) , and softmax-margin criterion (Gimpel & Smith, 2010) . , 2006; Li & Eisner, 2009; Ayana et al., 2016; Shen et al., 2016) . MIXER used decoder hidden states to predict the bias term to reduce the variance, while MRT renormalized the predicted probabilities.

Inspired by MIXER, Bahdanau et al. (2017) extended the actor critic algorithm (Konda & Tsitsiklis, 2000; Sutton et al., 2000) to the seq2seq framework. Sokolov et al. (2016) and Kreutzer et al. (2017) proposed the pairwise preference learning framework, which enables the model to learn from the difference between a pair of samples generated by the model. Another popular model in the form of divergence minimization is the smoothed version of MERT algorithm proposed by Och (2003) .

The smoothed MERT algorithm also minimizes the expected loss over samples from model outputs and is similar to MRT essentially. Norouzi et al. (2016) proposed Reward Augmented Maximum Likelihood (RAML) which also optimizes the divergence by data augmentation, and RAML was further combined with maximum likelihood via α-divergence recently (Koyamada et al., 2017) .

In contrast to the above divergence minimization based approaches, Wiseman & Rush (2016) used a margin based training objective and their search optimization process requires performing parameter updates or inference each time a new word is generated. One main drawback of maximum margin is that given a sample subset of an arbitrary size, only two samples of the subset can contribute to the gradient. The softmax-margin criterion proposed by Gimpel & Smith (2010) can alleviate this problem, in which all samples are used in loss and gradient computation. However, Edunov et al. (2018) shows that the softmax-margin criterion still cannot reach the performance of risk minimization. As the latest work, Edunov et al. (2018) is very similar to ours. However, our interests are in finding better training criteria on a consistent setting, while they mainly focused on absolute performance improvement, using techniques like combination with token level loss, online vs. offline candidate generation, etc.

3. SEQUENCE TO SEQUENCE MODEL:

This paper considers the seq2seq model from Bahdanau et al. (2015) and Luong et al. (2015) , which consists of an RNN encoder and an RNN decoder. Due to the gradient vanishing problem, the vanilla RNN cell is replaced with a Gated Recurrent Unit (GRU) (Cho et al., 2014) or Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) . A bidirectional RNN is used as encoder, with one RNN reading the source sequence from beginning to end, and the other RNN reading in reversed order. The bidirectional RNN encoder encodes the input sequence and gets a sequence of hidden states by concatenating the hidden states of each word in the two directions, which will be used as attention information in the decoder RNN.

The decoder RNN is initialized by passing the two final hidden states of the encoder to a feed forward network. The mixed information used as input in the first step m 0 is initialized by a zero vector. In decoding step i, using the input feeding approach described in Luong et al. (2015) , the decoder RNN reads the embedding of the word w i and the mixed information m i-1 , to generate the hidden state of current step h i :

h i = f 1 (concat(w i , m i-1 ), h i-1 )

, where f 1 is the RNN cell. The decoder then generates the attention context c i from encoder using h i . Since there are many different attention mechanisms, we omit the details here. After getting c i , the model mixes c i and h i together by a feedforward network f 2 to generate m i , which is used as the mixed information in the next step:

m i = f 2 (c i , h i )

Finally, m i is passed to a feedforward network to generate o i , and the softmax layer outputs the probability for each word at step i. Suppose the target word in step i is v , the probability of the target word in step i is given by

p(y i = v | x, y 1:i-1 ) = exp(o iv ) v exp(o iv )

.

For the sequence y = y 1 , y 2 , ..., y l , the probability p(y

| x) is p(y | x) = p(y 1 | x) p(y 2 | x, y 1 )...p(y l | x, y 1:l-1 ).

Suppose y * is the correct output, MLE actually maximizes the log likelihood of p(y * | x):

L MLE = -log p(y * | x) = - l i=1 log p(y * i | x, y * <i )

For a detailed explanation, please refer to Luong et al. (2015) . Now we describe the algorithm to construct a subset S from all possible output sequences which will be used in Section 4 & 5. In seq2seq model, the n-best list can only be approximated by performing beam search, as used in Wiseman & Rush (2016) , or Monte Carlo (MC) sampling, as used in Ayana et al. (2016) and Shen et al. (2016) . While using beam search may produce more accurate results, the time required for generating a candidate set of the same size as MC sampling using beam search is too long. In this paper, we construct the approximated n-best list S using MC sampling proposed in Shen et al. (2016) , and for completeness, we describe the sampling approach in Appendix 8.1 in details. Please note that the sampling algorithm does not guarantee the size of the sample subset, since it is inefficient to force the model to generate an exact number of distinct samples using GPU.

4. MINIMUM DIVERGENCE:

In this section, we consider training criteria in the form of the following equation:

L = D( p 1 || p 2 ),

where p 1 and p 2 are two probability distributions given by the model or the ground truth, and D is a distance function, for example, Kullback Leibler (KL) divergence or cross entropy. The baseline training criterion MLE can also be seen as minimum divergence training, where the cross entropy between the model output distribution and the one-hot encoded ground truth is minimized. Now we will discuss training criteria in which the ground truth distribution contains information of evaluation metric scores.

4.1. REWARD AUGMENTED MAXIMUM LIKELIHOOD:

Suppose that {x, y * } is a given input-output pair, the model learns parameters θ and gives prediction p(y | x; θ) on the output space Y, the exponentiated payoff distribution (Norouzi et al., 2016) q(y | y * ) is defined as follows:

EQUATION

where Z(y * ) = y∈Y exp[ r(y, y * )/τ ] is the normalization factor. τ is a hyperparameter that controls the smoothness of the distribution, and r(y, y * ) is the score of prediction y given by the evaluation metric, e.g. BLEU (Papineni et al., 2002) , or ROUGE (Lin, 2004) The training objective of Reward augmented Maximum Likelihood (RAML) (Norouzi et al., 2016) is defined as

L RAML = -E q(y | y * ) [ log p(y | x; θ) ],

(2) in which, the gradient is:

EQUATION

Equation ( 3) requires computing q(y | y * ) for all y and performing gradient descent on the whole Y.

However, due to the huge search space of Y and the high computational cost of r(y, y * ), Equation (3) cannot be directly computed in an efficient way. Norouzi et al. (2016) proposed using stratified sampling. One can sample from negative edit distance or Hamming distance and then do importance reweighting to reweight the samples with task specific scores. After sampling from q(y | y * ), the model is trained to maximize log p(y | x; θ).

Another choice is to construct a sample subset S from the model outputs using MC sampling, and use Equation (1) to compute the probability q (y | y * ) in the sample subset. The training criterion is then changed to

L RAML = - y∈S q (y | y * ) log p(y | x; θ),

where

q (y | y * ) = exp[ r(y, y * )/τ ] y ∈S exp[ r(y , y * )/τ ]

.

4.2. MINIMUM RISK TRAINING:

We first introduce the REINFORCE algorithm (Williams, 1992), which is a famous policy gradient method in RL, and then compare it with the Minimum Risk Training (MRT) (Smith & Eisner, 2006; Li & Eisner, 2009; Ayana et al., 2016; Shen et al., 2016) . In REINFORCE, given a state x, an action y under state x, and model parameters θ, the policy gradient g is

EQUATION

where r(y) is the reward of action y, and B(x) is the baseline or control variate that is used to reduce the variance (without introducing bias).

In MRT, the training objective is the expectation of evaluation metric scores taken with respect to the model output distribution:

EQUATION

Also due to the huge search space, Equation (5) can only be estimated by sampling from the model outputs. Suppose S is a sample subset from Y, p(y | x; θ) is replaced by p (y | x; θ; α):

EQUATION

)

where α is a hyperparameter that controls the smoothness of p (y | x; θ; α). We will later compare the effect of α here with τ in RAML.

The gradient of L MRT can be derived as:

EQUATION

If we regard the whole seq2seq framework as a one-step Markov Decision Process (MDP), in which the input sequence x is the state, the output sequence y is the action, and the evaluation metric score r is the reward, both REINFORCE and MRT actually optimize the same training criterion. The difference between them is that REINFORCE requires only a single sample and is unbiased, while MRT has multiple samples and the probability renormalization in Equation ( 6) introduces bias. The expected loss objective in Kreutzer et al. (2017) is also similar (and closer) to REINFORCE which uses one sample to estimate the gradient.

4.3. COMPARISON OF RAML & MRT:

The cross entropy is defined as:

EQUATION

Starting from RAML, in terms of cross entropy, Equation ( 2) can be rewritten as:

L RAML = D XENT (q(y | y * ) || p(y | x; θ))

The training criterion of RAML is essentially minimizing the the cross entropy loss from q(y | y * ) to p(y | x; θ).

Meanwhile, the training criterion of MRT in Equation ( 5) can be rewritten as:

L MRT = -τ E p(y | x;θ) [ r(y, y * )/τ ] = -τ E p(y | x;θ) [ log q(y | y * ) ] -τ E p(y | x;θ) [ log Z(y * ) ] = -τ E p(y | x;θ) [ log q(y | y * ) ] -τ log Z(y * ),

in which, the second term is a constant as it has no gradient with respect to θ. Thus we have:

EQUATION

Equation ( 9) actually shows that MRT minimizes the cross entropy loss from p(y | x; θ) to q(y | y * ).

By comparing the above two methods, we find that both RAML and MRT are minimizing the cross entropy loss between the model output distribution and the exponentiated payoff distribution, but with different directions of D XENT . Now we will discuss the effect of hyperparameter τ in RAML and α in MRT.

Cross entropy loss has a property that it is minimized if and only if the two probability distributions are equal. Ideally, when the cross entropy loss in RAML is minimized, we have:

p(y | x; θ) = q(y | y * ) log p(y | x; θ) = r(y, y * )/τ + Const r(y, y * ) = τ log p(y | x; θ) + Const.

And in MRT1 ,

L MRT = -E p (y | x;θ) [ log exp r(y, y * ) y exp r(y , y * ) ] + Const.

When the two probability distributions are equal, we have:

EQUATION

)

During training of both RAML and MRT, the model is trying to predict the evaluation score r by multiplying log p with a constant (either τ in RAML or α in MRT).

4.4. HELLINGER DISTANCE MINIMIZATION:

Now we introduce another popular distance metric, the Hellinger distance (Hellinger, 1909) , which also belongs to the family of Csiszár f-divergence (Csiszár, 1963) and has been used in graphical models (Beykikhoshk et al., 2015) and clustering algorithms (Abdel-Azim, 2016; Ji et al., 2018) .

Given two probability distributions p 1 and p 2 , let f and g denote the probability density functions of p 1 and p 2 . The square of the Hellinger distance H(p 1 , p 2 ) is defined as:

H 2 (p 1 , p 2 ) = 1 2 ( f (x) -g(x)) 2 dx.

The training criterion using squared Hellinger distance is:

L Hellinger = y∈S ( p (y | x; θ) -q(y | y * ) ) 2 ,

with p (y | x; θ) defined in Equation ( 6) and q(y | y * ) defined in Equation (1).

5. MAXIMUM MARGIN:

Let ∆ denote the loss given by the evaluation metric:

∆(y, y * ) = r(y * , y * ) -r(y, y * ).

∆ is 0 when y = y * and positive otherwise. Let F be the predicted score of the model:

F = τ log p(y | x; θ).

Different from minimizing the expected risk, another choice of training criterion is to optimize the evaluation score of the sequence with the highest predicted score:

EQUATION

where ȳ = arg max y F (y, θ).

For each input sequence x, we first do inference to find the output ȳ that has the highest predicted score, then we train the model to maximize the score of ȳ given by evaluation metric.

However, directly optimizing the training criterion that includes an arg max is not computationally tractable. In structured prediction, this problem has been widely discussed in the training process of Structural Support Vector Machine (SSVM) (Tsochantaridis et al., 2004; Joachims, 2006; Joachims et al., 2009) . Tsochantaridis et al. (2004) proposed using margin and slack rescaling instead. Margin Rescaling is based on the following inequality:

∆(ȳ, y * ) ≤ ∆(ȳ, y * ) + F (ȳ, θ) -F (y * , θ) ≤ max y (∆(y, y * ) + F (y, θ)) -F (y * , θ) (12)

Similarly, Slack Rescaling is:

∆(ȳ, y * ) ≤ ∆(ȳ, y * ) (1 + F (ȳ, θ) -F (y * , θ)) ≤ max y ∆(y, y * ) (1 + F (y, θ) -F (y * , θ)) (13)

As Equation ( 13) is a tighter bound on ∆(ȳ, y * ), slack rescaling generally gives better results than margin rescaling. Both Equation ( 12) and Equation ( 13) can be optimized by subgradient descent. Let y denote the sample that gives the max value. The subgradient for margin rescaling is:

∇ θ τ log p( y) -∇ θ τ log(p(y * )),

and similarly, the subgradient for slack rescaling is:

∆( y, y * )(∇ θ τ log p( y) -∇ θ τ log p(y * )).

Since doing loss augmented inference on such a huge search space is again not realistic, approximate inference is still needed. Equation ( 12) also has the name of structured hinge loss when used in SSVM. One may have found that Equation ( 13) is the same as the training criterion in Wiseman & Rush (2016) . Besides, to make a fair comparison with other training criteria and make it applicable on large datasets, we replace the beam search with sample subset.

6. EXPERIMENTS:

We evaluate the training criteria on two tasks in total. The first task is machine translation on a small dataset to explore the hyperparameter impact. The second task is sentence summarization using large datasets.

For all the experiments, we use LSTM as RNN cell. The attention mechanism used in our experiments is global attention with input feeding (Luong et al., 2015) .

6.1 MACHINE TRANSLATION Due to space limit, experimental results on hyperparameter effects and training details are provided in Appendix 8.2.1.

Data We use the IWSLT 2014 German-English translation dataset, with the same splits as Ranzato et al. ( 2016) and Wiseman & Rush (2016) , which contains about 153K training sentence pairs, 7K validation sentence pairs and 7K test sentence pairs. Sentences in the dataset are first tokenized and then converted into lowercase. During training, the maximum length for both inputs and outputs is restricted to 50. Words that appear less than three times are replaced with a UNK token.

Models:

The encoder is a single-layer bidirectional LSTM with 256 hidden units for either direction, and the decoder LSTM also has 256 hidden units. The size of word embedding for both encoder and decoder is 256. We use a dropout rate (Srivastava et al., 2014) of 0.2 to avoid overfitting. The gradient is clipped when its norm exceeds 1.0 to prevent gradient explosion and stabilize learning (Pascanu et al., 2013) . The batch size is set to 32 and the training set is shuffled at each new epoch. All models are trained with the Adam optimizer (Kingma & Ba, 2015) . The sentence level BLEU score used as reward or loss during training is smoothed by adding 1 to both numerator and denominator, as suggested by Chen & Cherry (2014) .

Results:

For the two max margin methods, τ is set to 1.0×10 -3 . We use the sample subset version of RAML, and τ for RAML is also set to 1.0 × 10 -3 . For Hellinger loss, α = 5.0 × 10 -4 and τ = 0.5. 

6.2. SENTENCE SUMMARIZATION:

The task of sentence summarization is: given a long sentence or a passage, the model will generate a short sentence that summarizes the long text. For evaluation, following Rush et al. (2015) , we use the full length F1 variant of ROUGE score (Lin, 2004) as metric. Training details are described in Appendix 8.2.2.

During training, we use the first 2K sequences of the dev corpus as validation set, and the size of the test set is also 2K. The maximum length for both input and output sequence is restricted to 50.

The encoder is a single layer bidirectional LSTM with 1000 hidden units for either direction. The single layer decoder LSTM also has 1000 dimensions. The word embedding size for both encoder and decoder is 500. Dropout is not used in the sentence summarization task, and the gradient is clipped when its norm exceeds 1.0. The batch size is set to 128. The training set is shuffled at each new epoch, and we prefetch 20 batches and sort them by the length of target sequence to form new batches to speed up training. We use ROUGE-2 as the reward or loss function.

Table 2 shows the results on the validation set and the test set. We report ROUGE-1, ROUGE-2, ROUGE-L and ROUGE-S4 score. MRT outperforms the MLE baseline by a large margin, but is still slightly lower than the newly introduced Hellinger loss. 

7. DISCUSSION:

For training criteria in the form of divergence minimization, according to our experimental results, MRT outperforms RAML. As mentioned before, MRT and RAML minimize the cross entropy in two different directions. By the definition of cross entropy loss in Equation ( 8), it is only minimized when p 1 = p 2 . Optimizing the cross entropy in either of the two directions both guarantee consistency, i.e., the training procedure can ultimately recover the true probability distribution.

For a model that minimizes D XENT (p 1 ||p 2 ), the model will sample from p 1 and maximize log p 2 , which means that if x has a high probability in p 1 , it will also have a high probability in p 2 , but if has a low probability in p 1 , p 2 (x) may still be high.

Thus in RAML which samples from exponentiated payoff distribution q and maximizes log p, sequences that have a high evaluation score will also have a high predicted probability. However, samples that have a low q, e.g., a low BLEU or ROUGE, may still have a high p.

In contrast, MRT will first sample from model outputs and ensure that a high BLEU or ROUGE score will be assigned to sequences that have a high probability in model output distribution p, at the cost that sequences that have low probabilities in p may still have high evaluation scores. This can explain the difference between RAML and MRT.

The squared Hellinger distance can be regarded as the squared error between p 1 2 and q 1 2 , which is essentially a regression loss, while the cross entropy loss usually works better in classification problems. By optimizing the cross entropy loss, the model is trained to find the candidate that has the highest probability and pays less attention to samples that have low probabilities, while by optimizing the squared Hellinger distance, the model learns to predict the evaluation metric score for every sample in the candidate set and pays equal attention to all of them. Since seq2seq models generate a predicted sequence word by word, the target sequence which has the highest evaluation metric score (sample from q and maximize log p) or the highest predicted probability (sample from p and maximize log q) may not be reached by approximate inference algorithms like greedy search or beam search. In this case, a model trained by optimizing the cross entropy loss may not work as expected, however, a model trained by optimizing the squared Hellinger distance can still make predictions normally since it pays equal attention to samples with low probabilities during training.

For maximum margin methods, slack rescaling is a tighter bound and the experimental result is slightly higher than margin rescaling on our small scale experiments. However, slack rescaling still cannot achieve the same level of performance of MRT and Hellinger distance. The reason may be that the upper bound is still not tight enough. In the training process of maximum margin methods, only the target sequence and a negative sample selected from the candidate set are used to update model parameters, which may also explain their performance.

8. APPENDIX:



8.1. THE SAMPLING ALGORITHM:

For completeness, we describe the MC sampling algorithm proposed by Shen et al. (2016) and Ayana et al. (2016) here.

Algorithm 1 The sampling approach to constructing the approximated n-best list Input:

source sequence x, model parameters θ, target sequence y * sample size m, maximum length l Output:

A set of samples S and corresponding probability P 1: S = {y * }; P = {p(y * )}; 2: for i in 1, ..., m -1 do 3: sample a sequence word by word given x and θ, stop sampling when length is l or meeting end-of-sequence symbol, and finally store the result and probability in y and q respectively 4:

if y in S then The MLE baseline is trained with a learning rate of 3.0 × 10 -4 . The model is trained for 20 epochs in total, and is evaluated every 1000 steps on the validation set.

τ for RAML For RAML, since we use BLEU instead of Hamming distance as evaluation metric, we first show the BLEU scores on the validation set for hyperparameter τ in Table 3 . RAML-I refers to importance reweighting and RAML-S refers to sample subset RAML, as mentioned in Section 4.1. Experimental results show that the BLEU score is not very sensitive to hyperparameter τ although τ varies from 1.0 × 10 -7 to 1.0 × 10 -1 , and setting τ larger will decrease the BLEU score compared with the MLE baseline. We also find that RAML-S tends to give slightly better results than RAML-I. The learning rate for RAML is 1.0 × 10 -6 . α for MRT For MRT, since Shen et al. (2016) did several experiments to study the effect of hyperparameters, the original hyperparameters are strictly followed in our experiments. Hyperparameter α is set to 5.0 × 10 -3 and the sample size is 100. The best BLEU score of MRT on the validation set is 30.55. The learning rate is set to 1.0 × 10 -5 .

α and τ for Hellinger loss As shown in Table 4 , α = 5.0 × 10 -4 , τ = 0.5 produces the best result.

The highest BLEU score on the validation set is 31.13, which is 0.58 higher than MRT. The learning rate used in Hellinger loss is also 1.0 × 10 -5 . Sample size for Hellinger loss For squared Hellinger loss, the effect of sample size is shown in Table 5 . A larger sample size generally gives a higher BLEU score, however, due to the limit of GPU memory, we cannot set sample size to a larger value. τ for max margin For the two max margin training criteria, the learning rate is 1.0 × 10 -7 . Table 6 shows the effect of τ : according to the results on the validation set, τ is not very sensitive to the BLEU scores. As shown in Table 6 , BLEU scores given by slack rescaling are usually 0.1 higher than for margin rescaling, and we use τ = 1.0 × 10 -3 for both of the two models. 

Details:

We use ROUGE-2 as reward or loss function. For the MLE baseline, we train the model for 7 epochs in total. For the first 5 epochs, we use Adam with a learning rate of 3.0 × 10 -4 , we then use SGD with a learning rate of 0.1, and decay the learning rate by a factor of 0.5 at the beginning of epoch 7.

For the other training criteria, we use the hyperparameters with the best performance described in Section 6.1.

8.2.3. ADDITIONAL EXPERIMENTS:

Details We implement the squared Hellinger distance criterion on convolutional seq2seq model (Gehring et al., 2017) and strictly follow the settings of IWSLT experiments in Edunov et al. (2018) . 

:

Here we refer to the MRT model inShen et al. (2016), which contains sampling and renormalization as described in Section 4.2.

Figures and Tables:

10 -3 29.33 29.71 29.22 5.0 × 10 -4 29.91 31.13 31.01 5.0 × 10 -5 30.75 30.83 31.00

The final results on the test set are described in Table1. Max margin methods here improve the MLE baseline less significantly thanWiseman & Rush (2016) did, since the search optimization algorithm is not used in our experiments. The training criterion of squared Hellinger distance gives the highest BLEU score on the test set. BLEU scores on IWSLT German-English translation evaluated on the test set.

Table content: <table><tr><td colspan="2">criterion greedy beam</td></tr><tr><td>MLE</td><td>26.03 27.17</td></tr><tr><td>margin</td><td>26.22 27.32</td></tr><tr><td>slack</td><td>26.32 27.37</td></tr><tr><td>RAML</td><td>26.34 27.36</td></tr><tr><td>MRT</td><td>27.35 28.10</td></tr><tr><td colspan="2">Hellinger 27.75 28.38</td></tr></table>

Results on Gigaword sentence summarization task

Table content: <table><tr><td>criterion</td><td colspan="2">ROUGE-1</td><td colspan="2">ROUGE-2</td><td colspan="2">ROUGE-L</td><td colspan="2">ROUGE-S4</td></tr><tr><td/><td>valid</td><td>test</td><td>valid</td><td>test</td><td>valid</td><td>test</td><td>valid</td><td>test</td></tr><tr><td>MLE</td><td colspan="8">48.41 34.20 24.14 15.22 45.08 31.84 22.62 14.00</td></tr><tr><td>margin</td><td colspan="8">48.43 34.20 24.16 15.24 45.09 31.85 22.65 14.00</td></tr><tr><td>slack</td><td colspan="8">48.40 34.20 24.12 15.25 45.08 31.83 22.63 14.00</td></tr><tr><td>RAML</td><td colspan="8">48.43 34.20 24.16 15.24 45.10 31.84 22.66 14.00</td></tr><tr><td>MRT</td><td colspan="8">50.41 35.63 26.42 16.96 47.05 33.24 24.47 15.31</td></tr><tr><td>Hellinger</td><td colspan="8">51.02 35.84 26.88 17.03 47.71 33.47 25.12 15.54</td></tr><tr><td>MLE + BS</td><td colspan="8">49.69 36.21 25.52 16.37 46.55 32.75 24.00 15.01</td></tr><tr><td>MRT + BS</td><td colspan="8">50.72 36.47 26.74 17.62 47.41 34.05 25.05 15.98</td></tr><tr><td colspan="9">Hellinger + BS 51.07 36.59 26.95 17.61 47.62 34.03 25.34 16.04</td></tr></table>

Tokenized BLEU scores on IWSLT 2014 German-English translation evaluated on validation set using RAML criterion.

Table content: <table><tr><td>τ</td><td colspan="2">RAML-I RAML-S</td></tr><tr><td>1.0 × 10 -7</td><td>29.25</td><td>29.28</td></tr><tr><td>1.0 × 10 -5</td><td>29.27</td><td>29.30</td></tr><tr><td>1.0 × 10 -3</td><td>29.26</td><td>29.32</td></tr><tr><td>1.0 × 10 -1</td><td>29.28</td><td>29.32</td></tr></table>

Tokenized BLEU scores on IWSLT 2014 German-English translation evaluated on the validation set using squared Hellinger distance with different α and τ as training criterion.

Table content: <table/>

Tokenized BLEU scores on IWSLT 2014 German-English translation evaluated on the validation set using squared Hellinger Distance with different sample sizes.

Table content: <table><tr><td>sample size</td><td>20</td><td>50</td><td>100</td></tr><tr><td>BLEU</td><td>29.51</td><td>30.30</td><td>31.13</td></tr></table>

Tokenized BLEU scores on IWSLT 2014 German-English translation evaluated on the validation set using max margin criteria.

Table content: <table><tr><td>τ</td><td colspan="2">Slack Margin</td></tr><tr><td colspan="2">1.0 × 10 -1 29.16</td><td>29.23</td></tr><tr><td colspan="2">1.0 × 10 -3 29.33</td><td>29.25</td></tr><tr><td colspan="2">1.0 × 10 -5 29.35</td><td>29.16</td></tr><tr><td>8.2.2 SENTENCE SUMMARIZATION</td><td/></tr></table>

BLEU scores using convolutional seq2seq models.

Table content: <table><tr><td colspan="3">criterion MLE Hellinger</td></tr><tr><td>BLEU</td><td>32.14</td><td>32.30</td></tr></table>

Peer review (iclr2019):
 The authors have updated the paper and clarified some things, and now my impression of the paper has improved. It still feels a little incremental to me, but the potential application areas of these sorts of models are quite large and therefore incremental improvements are not insignificant. This paper suggests some natural follow-up work in exploring Hellinger distance and other variations for these models. ----- original review follows: ------ This paper discusses loss functions for sequence generation tasks that take into account cost functions that reflect the task-specific evaluation metric. They compare RAML and risk (MRT) formally and empirically, and also test a loss based on Hellinger distance. They compare these to some standard max-margin losses. MRT and the Hellinger distance loss perform best in NMT and summarization experiments. Pros: There are some interesting aspects of this paper: - It is interesting to note that RAML and MRT are (similar to) different directions of KL divergences between the same two distributions. (Caveat: the entropy regularizer, which I discuss in "Cons" below.) - The new Hellinger-distance-based loss seems promising. - The empirical comparison among losses for standard NMT/summarization tasks is a potentially valuable contribution. Cons: A. The focus/story of the paper need some work. It is unclear what the key contributions are. I think the Hellinger distance loss is potentially the most important contribution, but the authors don*t spend much time on that.. it seems that they think the comparison of divergence and max-margin losses is more central. However, I think the authors* conclusion (that the divergence losses are better than the max-margin losses) is not the main story, because RAML is not much better than the max-margin losses. Also, I have some concerns about some of the details of the max-margin losses (listed and discussed below), so I*m not sure how reliable the empirical comparison is. B. As for the connections and comparison between RAML and MRT: It does not seem that MRT corresponds to a divergence of the form given at the start of Sec. 4. There is also an entropy regularizer in Eq. (9). Sec. 4.3 states: "By comparing the above two methods, we find that both RAML and MRT are minimizing the KL divergence between the model output distribution and the exponentiated payoff distribution, but with different directions of D_KL." However, this statement ignores the entropy regularizer in Eq. (9). Maybe I*m being dense, but I didn*t understand where Equation (10) comes from. I understand the equations above it for RAML, but I don*t understand the MRT case in Eq. (10). Can you provide more details? I also don*t understand the following sentence: "It turns out that the hyperparameter .tau in RAML and .alpha in MRT have the same effect." What does this mean mathematically? Also, does this equivalence also require ignoring the entropy regularizer? As formulated, L_{RAML} necessarily contains a .tau, but L_{MRT} does not necessarily contain an alpha. It is only when moving to the sample approximation does the alpha become introduced. (MRT does not require this sample approximation; Some older work on MRT developed dynamic programming algorithms to exactly compute the gradients for structured output spaces like sequences, so samples were not used in those cases.) So I think the paper needs to clarify what exactly is meant by the connection between tau and alpha, under what conditions there is a connection between the two, and what exactly is the nature of this connection. If more space is needed for this, many of the details in Sec 3 can be cut or moved to appendices because those are standard and not needed for what follows. In the experimental results (Sec. 7), MRT outperforms RAML consistently. The authors discuss the impact of the directionality of the KL divergence, but what about the entropy regularizer? It would be interesting to compare MRT both with and without the entropy regularizer. Without the regularizer, MRT would actually correspond to the KL that the authors are describing in the discussion. As it currently stands, two things are changing between MRT and RAML and we don*t know which is responsible for the sizable performance gains. C. There are several technical issues with the writing (and potentially with the claims/conclusions), most of which are potentially flexible with some corrections and more exposition by the authors: Is L_{RAML} to be maximized or minimized? Looks like maximized, but clearly both L_{MLE} and L_{MRT} are supposed to be minimized, so the use of L for all of these seems confusing. If different conventions are to be used for each one, it should be explicitly mentioned in each case whether the term is to be maximized or minimized. At the end of Sec. 4.1, q* is not defined. I can guess what it is, but it is not entirely clear from the context and should be defined. In Equation 6, please use different notation for the y in the denominator (e.g., y*) to avoid collision with the y in the numerator and that on the left-hand side. The discussion of max-margin losses in Sec. 5 has some things that should be fixed. 1. In Sec. 5, it is unclear why .Delta is defined to be the difference of two r() functions. Why not just make it -r(y, y^*)? Are there some implied conditions on .Delta that are not stated explicitly? If .Delta is assumed to be nonnegative, that should be stated. 2. In Eq. (11), F appears to be a function of y and theta, but in the definition of F, it has no functional arguments. But then further down, F appears to be a function of y only. Please make these consistent. 3. Eq. (11) is not only hard for structured settings; it is also hard in the simplest settings (binary classification with 0-1 loss). This is the motivation for surrogate loss functions in empirical risk minimization for classification settings. The discussion in the paper makes it sound as if these challenges only arise in the structured prediction setting. 4. I*m confused by what the paper is attempting to communicate with Equations 12 and 13. In Eq. 12, y on the left-hand side is not bound to anything, so it is unclear what is being stated exactly. Is it for all y? For any y? In Eq. 13, the .Delta on the right-hand side is outside the max over y -- is that really what was intended? I thought the max (the slack-rescaled loss-augmented inference step) should take into account .Delta. Otherwise, it is just doing an argmax over the score function. 5. If the authors are directly optimizing the right-hand side of the inequality in Equation 12 (as would be suggested for the formula for the gradient), then there is no global minimum of the loss. It would go to negative infinity. Typically people use a "max(0, )" outside the loss so that the global minimum is 0. Typos and minor issues follow: Sec. 1: "the SEARN" --> "SEARN" "the DAGGER" --> "DAGGER" Sec. 2: "genrally" --> "generally" "took evaluation metric into training" --> "incorporated the evaluation metric into training" "consistant" --> "consistent" Sec. 3: Use .exp instead of exp. "a detail explanation" --> "a detailed explanation" Sec. 4.1: "predition" --> "prediction" Sec. 4.2: "only single sample" --> "only a single sample" Sec. 6.1: "less significant than" --> "less significantly than"