Title: MINIMUM DIVERGENCE VS. MAXIMUM MARGIN: AN EMPIRICAL COMPARISON ON SEQ2SEQ MODELS

Abstract: Sequence to sequence (seq2seq) models have become a popular framework for neural sequence prediction. While traditional seq2seq models are trained by Maximum Likelihood Estimation (MLE), much recent work has made various attempts to optimize evaluation scores directly to solve the mismatch between training and evaluation, since model predictions are usually evaluated by a task specific evaluation metric like BLEU or ROUGE scores instead of perplexity. This paper puts this existing work into two categories, a) minimum divergence, and b) maximum margin. We introduce a new training criterion based on the analysis of existing work, and empirically compare models in the two categories. Our experimental results show that our new training criterion can usually work better than existing methods, on both the tasks of machine translation and sentence summarization.

1. INTRODUCTION:

Sequence to sequence (seq2seq) models (Kalchbrenner & Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015) are a powerful end-to-end solution to sequence prediction since they allow a mapping between two sequences of different lengths. However, Maximum Likelihood Estimation (MLE) which is used as a standard training criterion in seq2seq has a series of drawbacks:

• Training and evaluation mismatch: during training, we maximize the log likelihood, while during inference, the model is evaluated by a different metric such as BLEU or ROUGE;

• MLE fails to assign proper scores to different model outputs, which means that all incorrect outputs are treated equally.

Similar to SEARN (Daumé et al., 2009) and DAGGER (Ross et al., 2011) in traditional models, many reinforcement learning (RL) techniques have been extended to seq2eq models recently to solve the mismatch between training and inference. The key idea is to replace MLE with a criterion that gives task-specific scores to different model outputs.

Recent approaches to taking the evaluation metric into training criteria generally fall into one of the following two categories:

1. minimum divergence: Minimum divergence methods minimize some distance functions between the model output distribution and the probability distribution given by the ground truth, and by optimizing the divergence between the two distributions, the model output distribution will ideally be equal to the ground truth distribution.

2. maximum margin: During inference, in structured prediction, the goal is to simply ensure good structures have the highest predicted score. That is, margin based learning differs from minimum divergence learning in the aspect that it only requires that the predicted score of the correct output should be higher than that of the others by a margin.

The main contribution of this paper can be summarized as: based on the analysis of existing work, we introduce a novel training criterion for seq2seq learning and show its effectiveness.

Based on our experimental results on the tasks of machine translation and sentence summarization, we conclude that some models that take the evaluation metric into consideration can improve over a strong MLE baseline by a large margin, and our new training criterion can usually get better results than existing work.

2. RELATED WORK:

Seq2seq models were proposed and improved by Kalchbrenner & Blunsom (2013) and Sutskever et al. (2014) . With the attention mechanism (Bahdanau et al., 2015; Luong et al., 2015) , seq2seq models achieve better results than traditional models in many fields including machine translation (Bahdanau et al., 2015; Luong et al., 2015) and text summarization (Chopra et al., 2016) .

Before seq2seq models, previous work on optimizing task specific evaluation metric scores generally includes SEARN (Daumé et al., 2009) , DAGGER (Ross et al., 2011) , Minimum Error Rate Training (MERT) (Och, 2003) , and softmax-margin criterion (Gimpel & Smith, 2010) . , 2006; Li & Eisner, 2009; Ayana et al., 2016; Shen et al., 2016) . MIXER used decoder hidden states to predict the bias term to reduce the variance, while MRT renormalized the predicted probabilities.

Inspired by MIXER, Bahdanau et al. (2017) extended the actor critic algorithm (Konda & Tsitsiklis, 2000; Sutton et al., 2000) to the seq2seq framework. Sokolov et al. (2016) and Kreutzer et al. (2017) proposed the pairwise preference learning framework, which enables the model to learn from the difference between a pair of samples generated by the model. Another popular model in the form of divergence minimization is the smoothed version of MERT algorithm proposed by Och (2003) .

The smoothed MERT algorithm also minimizes the expected loss over samples from model outputs and is similar to MRT essentially. Norouzi et al. (2016) proposed Reward Augmented Maximum Likelihood (RAML) which also optimizes the divergence by data augmentation, and RAML was further combined with maximum likelihood via α-divergence recently (Koyamada et al., 2017) .

In contrast to the above divergence minimization based approaches, Wiseman & Rush (2016) used a margin based training objective and their search optimization process requires performing parameter updates or inference each time a new word is generated. One main drawback of maximum margin is that given a sample subset of an arbitrary size, only two samples of the subset can contribute to the gradient. The softmax-margin criterion proposed by Gimpel & Smith (2010) can alleviate this problem, in which all samples are used in loss and gradient computation. However, Edunov et al. (2018) shows that the softmax-margin criterion still cannot reach the performance of risk minimization. As the latest work, Edunov et al. (2018) is very similar to ours. However, our interests are in finding better training criteria on a consistent setting, while they mainly focused on absolute performance improvement, using techniques like combination with token level loss, online vs. offline candidate generation, etc.

3. SEQUENCE TO SEQUENCE MODEL:

This paper considers the seq2seq model from Bahdanau et al. (2015) and Luong et al. (2015) , which consists of an RNN encoder and an RNN decoder. Due to the gradient vanishing problem, the vanilla RNN cell is replaced with a Gated Recurrent Unit (GRU) (Cho et al., 2014) or Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) . A bidirectional RNN is used as encoder, with one RNN reading the source sequence from beginning to end, and the other RNN reading in reversed order. The bidirectional RNN encoder encodes the input sequence and gets a sequence of hidden states by concatenating the hidden states of each word in the two directions, which will be used as attention information in the decoder RNN.

The decoder RNN is initialized by passing the two final hidden states of the encoder to a feed forward network. The mixed information used as input in the first step m 0 is initialized by a zero vector. In decoding step i, using the input feeding approach described in Luong et al. (2015) , the decoder RNN reads the embedding of the word w i and the mixed information m i-1 , to generate the hidden state of current step h i :

h i = f 1 (concat(w i , m i-1 ), h i-1 )

, where f 1 is the RNN cell. The decoder then generates the attention context c i from encoder using h i . Since there are many different attention mechanisms, we omit the details here. After getting c i , the model mixes c i and h i together by a feedforward network f 2 to generate m i , which is used as the mixed information in the next step:

m i = f 2 (c i , h i )

Finally, m i is passed to a feedforward network to generate o i , and the softmax layer outputs the probability for each word at step i. Suppose the target word in step i is v , the probability of the target word in step i is given by

p(y i = v | x, y 1:i-1 ) = exp(o iv ) v exp(o iv )

.

For the sequence y = y 1 , y 2 , ..., y l , the probability p(y

| x) is p(y | x) = p(y 1 | x) p(y 2 | x, y 1 )...p(y l | x, y 1:l-1 ).

Suppose y * is the correct output, MLE actually maximizes the log likelihood of p(y * | x):

L MLE = -log p(y * | x) = - l i=1 log p(y * i | x, y * <i )

For a detailed explanation, please refer to Luong et al. (2015) . Now we describe the algorithm to construct a subset S from all possible output sequences which will be used in Section 4 & 5. In seq2seq model, the n-best list can only be approximated by performing beam search, as used in Wiseman & Rush (2016) , or Monte Carlo (MC) sampling, as used in Ayana et al. (2016) and Shen et al. (2016) . While using beam search may produce more accurate results, the time required for generating a candidate set of the same size as MC sampling using beam search is too long. In this paper, we construct the approximated n-best list S using MC sampling proposed in Shen et al. (2016) , and for completeness, we describe the sampling approach in Appendix 8.1 in details. Please note that the sampling algorithm does not guarantee the size of the sample subset, since it is inefficient to force the model to generate an exact number of distinct samples using GPU.

4. MINIMUM DIVERGENCE:

In this section, we consider training criteria in the form of the following equation:

L = D( p 1 || p 2 ),

where p 1 and p 2 are two probability distributions given by the model or the ground truth, and D is a distance function, for example, Kullback Leibler (KL) divergence or cross entropy. The baseline training criterion MLE can also be seen as minimum divergence training, where the cross entropy between the model output distribution and the one-hot encoded ground truth is minimized. Now we will discuss training criteria in which the ground truth distribution contains information of evaluation metric scores.

4.1. REWARD AUGMENTED MAXIMUM LIKELIHOOD:

Suppose that {x, y * } is a given input-output pair, the model learns parameters θ and gives prediction p(y | x; θ) on the output space Y, the exponentiated payoff distribution (Norouzi et al., 2016) q(y | y * ) is defined as follows:

EQUATION

where Z(y * ) = y∈Y exp[ r(y, y * )/τ ] is the normalization factor. τ is a hyperparameter that controls the smoothness of the distribution, and r(y, y * ) is the score of prediction y given by the evaluation metric, e.g. BLEU (Papineni et al., 2002) , or ROUGE (Lin, 2004) The training objective of Reward augmented Maximum Likelihood (RAML) (Norouzi et al., 2016) is defined as

L RAML = -E q(y | y * ) [ log p(y | x; θ) ],

(2) in which, the gradient is:

EQUATION

Equation ( 3) requires computing q(y | y * ) for all y and performing gradient descent on the whole Y.

However, due to the huge search space of Y and the high computational cost of r(y, y * ), Equation (3) cannot be directly computed in an efficient way. Norouzi et al. (2016) proposed using stratified sampling. One can sample from negative edit distance or Hamming distance and then do importance reweighting to reweight the samples with task specific scores. After sampling from q(y | y * ), the model is trained to maximize log p(y | x; θ).

Another choice is to construct a sample subset S from the model outputs using MC sampling, and use Equation (1) to compute the probability q (y | y * ) in the sample subset. The training criterion is then changed to

L RAML = - y∈S q (y | y * ) log p(y | x; θ),

where

q (y | y * ) = exp[ r(y, y * )/τ ] y ∈S exp[ r(y , y * )/τ ]

.

4.2. MINIMUM RISK TRAINING:

We first introduce the REINFORCE algorithm (Williams, 1992), which is a famous policy gradient method in RL, and then compare it with the Minimum Risk Training (MRT) (Smith & Eisner, 2006; Li & Eisner, 2009; Ayana et al., 2016; Shen et al., 2016) . In REINFORCE, given a state x, an action y under state x, and model parameters θ, the policy gradient g is

EQUATION

where r(y) is the reward of action y, and B(x) is the baseline or control variate that is used to reduce the variance (without introducing bias).

In MRT, the training objective is the expectation of evaluation metric scores taken with respect to the model output distribution:

EQUATION

Also due to the huge search space, Equation (5) can only be estimated by sampling from the model outputs. Suppose S is a sample subset from Y, p(y | x; θ) is replaced by p (y | x; θ; α):

EQUATION

)

where α is a hyperparameter that controls the smoothness of p (y | x; θ; α). We will later compare the effect of α here with τ in RAML.

The gradient of L MRT can be derived as:

EQUATION

If we regard the whole seq2seq framework as a one-step Markov Decision Process (MDP), in which the input sequence x is the state, the output sequence y is the action, and the evaluation metric score r is the reward, both REINFORCE and MRT actually optimize the same training criterion. The difference between them is that REINFORCE requires only a single sample and is unbiased, while MRT has multiple samples and the probability renormalization in Equation ( 6) introduces bias. The expected loss objective in Kreutzer et al. (2017) is also similar (and closer) to REINFORCE which uses one sample to estimate the gradient.

4.3. COMPARISON OF RAML & MRT:

The cross entropy is defined as:

EQUATION

Starting from RAML, in terms of cross entropy, Equation ( 2) can be rewritten as:

L RAML = D XENT (q(y | y * ) || p(y | x; θ))

The training criterion of RAML is essentially minimizing the the cross entropy loss from q(y | y * ) to p(y | x; θ).

Meanwhile, the training criterion of MRT in Equation ( 5) can be rewritten as:

L MRT = -τ E p(y | x;θ) [ r(y, y * )/τ ] = -τ E p(y | x;θ) [ log q(y | y * ) ] -τ E p(y | x;θ) [ log Z(y * ) ] = -τ E p(y | x;θ) [ log q(y | y * ) ] -τ log Z(y * ),

in which, the second term is a constant as it has no gradient with respect to θ. Thus we have:

EQUATION

Equation ( 9) actually shows that MRT minimizes the cross entropy loss from p(y | x; θ) to q(y | y * ).

By comparing the above two methods, we find that both RAML and MRT are minimizing the cross entropy loss between the model output distribution and the exponentiated payoff distribution, but with different directions of D XENT . Now we will discuss the effect of hyperparameter τ in RAML and α in MRT.

Cross entropy loss has a property that it is minimized if and only if the two probability distributions are equal. Ideally, when the cross entropy loss in RAML is minimized, we have:

p(y | x; θ) = q(y | y * ) log p(y | x; θ) = r(y, y * )/τ + Const r(y, y * ) = τ log p(y | x; θ) + Const.

And in MRT1 ,

L MRT = -E p (y | x;θ) [ log exp r(y, y * ) y exp r(y , y * ) ] + Const.

When the two probability distributions are equal, we have:

EQUATION

)

During training of both RAML and MRT, the model is trying to predict the evaluation score r by multiplying log p with a constant (either τ in RAML or α in MRT).

4.4. HELLINGER DISTANCE MINIMIZATION:

Now we introduce another popular distance metric, the Hellinger distance (Hellinger, 1909) , which also belongs to the family of Csiszár f-divergence (Csiszár, 1963) and has been used in graphical models (Beykikhoshk et al., 2015) and clustering algorithms (Abdel-Azim, 2016; Ji et al., 2018) .

Given two probability distributions p 1 and p 2 , let f and g denote the probability density functions of p 1 and p 2 . The square of the Hellinger distance H(p 1 , p 2 ) is defined as:

H 2 (p 1 , p 2 ) = 1 2 ( f (x) -g(x)) 2 dx.

The training criterion using squared Hellinger distance is:

L Hellinger = y∈S ( p (y | x; θ) -q(y | y * ) ) 2 ,

with p (y | x; θ) defined in Equation ( 6) and q(y | y * ) defined in Equation (1).

5. MAXIMUM MARGIN:

Let ∆ denote the loss given by the evaluation metric:

∆(y, y * ) = r(y * , y * ) -r(y, y * ).

∆ is 0 when y = y * and positive otherwise. Let F be the predicted score of the model:

F = τ log p(y | x; θ).

Different from minimizing the expected risk, another choice of training criterion is to optimize the evaluation score of the sequence with the highest predicted score:

EQUATION

where ȳ = arg max y F (y, θ).

For each input sequence x, we first do inference to find the output ȳ that has the highest predicted score, then we train the model to maximize the score of ȳ given by evaluation metric.

However, directly optimizing the training criterion that includes an arg max is not computationally tractable. In structured prediction, this problem has been widely discussed in the training process of Structural Support Vector Machine (SSVM) (Tsochantaridis et al., 2004; Joachims, 2006; Joachims et al., 2009) . Tsochantaridis et al. (2004) proposed using margin and slack rescaling instead. Margin Rescaling is based on the following inequality:

∆(ȳ, y * ) ≤ ∆(ȳ, y * ) + F (ȳ, θ) -F (y * , θ) ≤ max y (∆(y, y * ) + F (y, θ)) -F (y * , θ) (12)

Similarly, Slack Rescaling is:

∆(ȳ, y * ) ≤ ∆(ȳ, y * ) (1 + F (ȳ, θ) -F (y * , θ)) ≤ max y ∆(y, y * ) (1 + F (y, θ) -F (y * , θ)) (13)

As Equation ( 13) is a tighter bound on ∆(ȳ, y * ), slack rescaling generally gives better results than margin rescaling. Both Equation ( 12) and Equation ( 13) can be optimized by subgradient descent. Let y denote the sample that gives the max value. The subgradient for margin rescaling is:

∇ θ τ log p( y) -∇ θ τ log(p(y * )),

and similarly, the subgradient for slack rescaling is:

∆( y, y * )(∇ θ τ log p( y) -∇ θ τ log p(y * )).

Since doing loss augmented inference on such a huge search space is again not realistic, approximate inference is still needed. Equation ( 12) also has the name of structured hinge loss when used in SSVM. One may have found that Equation ( 13) is the same as the training criterion in Wiseman & Rush (2016) . Besides, to make a fair comparison with other training criteria and make it applicable on large datasets, we replace the beam search with sample subset.

6. EXPERIMENTS:

We evaluate the training criteria on two tasks in total. The first task is machine translation on a small dataset to explore the hyperparameter impact. The second task is sentence summarization using large datasets.

For all the experiments, we use LSTM as RNN cell. The attention mechanism used in our experiments is global attention with input feeding (Luong et al., 2015) .

6.1 MACHINE TRANSLATION Due to space limit, experimental results on hyperparameter effects and training details are provided in Appendix 8.2.1.

Data We use the IWSLT 2014 German-English translation dataset, with the same splits as Ranzato et al. ( 2016) and Wiseman & Rush (2016) , which contains about 153K training sentence pairs, 7K validation sentence pairs and 7K test sentence pairs. Sentences in the dataset are first tokenized and then converted into lowercase. During training, the maximum length for both inputs and outputs is restricted to 50. Words that appear less than three times are replaced with a UNK token.

Models:

The encoder is a single-layer bidirectional LSTM with 256 hidden units for either direction, and the decoder LSTM also has 256 hidden units. The size of word embedding for both encoder and decoder is 256. We use a dropout rate (Srivastava et al., 2014) of 0.2 to avoid overfitting. The gradient is clipped when its norm exceeds 1.0 to prevent gradient explosion and stabilize learning (Pascanu et al., 2013) . The batch size is set to 32 and the training set is shuffled at each new epoch. All models are trained with the Adam optimizer (Kingma & Ba, 2015) . The sentence level BLEU score used as reward or loss during training is smoothed by adding 1 to both numerator and denominator, as suggested by Chen & Cherry (2014) .

Results:

For the two max margin methods, τ is set to 1.0×10 -3 . We use the sample subset version of RAML, and τ for RAML is also set to 1.0 × 10 -3 . For Hellinger loss, α = 5.0 × 10 -4 and τ = 0.5. 

6.2. SENTENCE SUMMARIZATION:

The task of sentence summarization is: given a long sentence or a passage, the model will generate a short sentence that summarizes the long text. For evaluation, following Rush et al. (2015) , we use the full length F1 variant of ROUGE score (Lin, 2004) as metric. Training details are described in Appendix 8.2.2.

During training, we use the first 2K sequences of the dev corpus as validation set, and the size of the test set is also 2K. The maximum length for both input and output sequence is restricted to 50.

The encoder is a single layer bidirectional LSTM with 1000 hidden units for either direction. The single layer decoder LSTM also has 1000 dimensions. The word embedding size for both encoder and decoder is 500. Dropout is not used in the sentence summarization task, and the gradient is clipped when its norm exceeds 1.0. The batch size is set to 128. The training set is shuffled at each new epoch, and we prefetch 20 batches and sort them by the length of target sequence to form new batches to speed up training. We use ROUGE-2 as the reward or loss function.

Table 2 shows the results on the validation set and the test set. We report ROUGE-1, ROUGE-2, ROUGE-L and ROUGE-S4 score. MRT outperforms the MLE baseline by a large margin, but is still slightly lower than the newly introduced Hellinger loss. 

7. DISCUSSION:

For training criteria in the form of divergence minimization, according to our experimental results, MRT outperforms RAML. As mentioned before, MRT and RAML minimize the cross entropy in two different directions. By the definition of cross entropy loss in Equation ( 8), it is only minimized when p 1 = p 2 . Optimizing the cross entropy in either of the two directions both guarantee consistency, i.e., the training procedure can ultimately recover the true probability distribution.

For a model that minimizes D XENT (p 1 ||p 2 ), the model will sample from p 1 and maximize log p 2 , which means that if x has a high probability in p 1 , it will also have a high probability in p 2 , but if has a low probability in p 1 , p 2 (x) may still be high.

Thus in RAML which samples from exponentiated payoff distribution q and maximizes log p, sequences that have a high evaluation score will also have a high predicted probability. However, samples that have a low q, e.g., a low BLEU or ROUGE, may still have a high p.

In contrast, MRT will first sample from model outputs and ensure that a high BLEU or ROUGE score will be assigned to sequences that have a high probability in model output distribution p, at the cost that sequences that have low probabilities in p may still have high evaluation scores. This can explain the difference between RAML and MRT.

The squared Hellinger distance can be regarded as the squared error between p 1 2 and q 1 2 , which is essentially a regression loss, while the cross entropy loss usually works better in classification problems. By optimizing the cross entropy loss, the model is trained to find the candidate that has the highest probability and pays less attention to samples that have low probabilities, while by optimizing the squared Hellinger distance, the model learns to predict the evaluation metric score for every sample in the candidate set and pays equal attention to all of them. Since seq2seq models generate a predicted sequence word by word, the target sequence which has the highest evaluation metric score (sample from q and maximize log p) or the highest predicted probability (sample from p and maximize log q) may not be reached by approximate inference algorithms like greedy search or beam search. In this case, a model trained by optimizing the cross entropy loss may not work as expected, however, a model trained by optimizing the squared Hellinger distance can still make predictions normally since it pays equal attention to samples with low probabilities during training.

For maximum margin methods, slack rescaling is a tighter bound and the experimental result is slightly higher than margin rescaling on our small scale experiments. However, slack rescaling still cannot achieve the same level of performance of MRT and Hellinger distance. The reason may be that the upper bound is still not tight enough. In the training process of maximum margin methods, only the target sequence and a negative sample selected from the candidate set are used to update model parameters, which may also explain their performance.

8. APPENDIX:



8.1. THE SAMPLING ALGORITHM:

For completeness, we describe the MC sampling algorithm proposed by Shen et al. (2016) and Ayana et al. (2016) here.

Algorithm 1 The sampling approach to constructing the approximated n-best list Input:

source sequence x, model parameters θ, target sequence y * sample size m, maximum length l Output:

A set of samples S and corresponding probability P 1: S = {y * }; P = {p(y * )}; 2: for i in 1, ..., m -1 do 3: sample a sequence word by word given x and θ, stop sampling when length is l or meeting end-of-sequence symbol, and finally store the result and probability in y and q respectively 4:

if y in S then The MLE baseline is trained with a learning rate of 3.0 × 10 -4 . The model is trained for 20 epochs in total, and is evaluated every 1000 steps on the validation set.

τ for RAML For RAML, since we use BLEU instead of Hamming distance as evaluation metric, we first show the BLEU scores on the validation set for hyperparameter τ in Table 3 . RAML-I refers to importance reweighting and RAML-S refers to sample subset RAML, as mentioned in Section 4.1. Experimental results show that the BLEU score is not very sensitive to hyperparameter τ although τ varies from 1.0 × 10 -7 to 1.0 × 10 -1 , and setting τ larger will decrease the BLEU score compared with the MLE baseline. We also find that RAML-S tends to give slightly better results than RAML-I. The learning rate for RAML is 1.0 × 10 -6 . α for MRT For MRT, since Shen et al. (2016) did several experiments to study the effect of hyperparameters, the original hyperparameters are strictly followed in our experiments. Hyperparameter α is set to 5.0 × 10 -3 and the sample size is 100. The best BLEU score of MRT on the validation set is 30.55. The learning rate is set to 1.0 × 10 -5 .

α and τ for Hellinger loss As shown in Table 4 , α = 5.0 × 10 -4 , τ = 0.5 produces the best result.

The highest BLEU score on the validation set is 31.13, which is 0.58 higher than MRT. The learning rate used in Hellinger loss is also 1.0 × 10 -5 . Sample size for Hellinger loss For squared Hellinger loss, the effect of sample size is shown in Table 5 . A larger sample size generally gives a higher BLEU score, however, due to the limit of GPU memory, we cannot set sample size to a larger value. τ for max margin For the two max margin training criteria, the learning rate is 1.0 × 10 -7 . Table 6 shows the effect of τ : according to the results on the validation set, τ is not very sensitive to the BLEU scores. As shown in Table 6 , BLEU scores given by slack rescaling are usually 0.1 higher than for margin rescaling, and we use τ = 1.0 × 10 -3 for both of the two models. 

Details:

We use ROUGE-2 as reward or loss function. For the MLE baseline, we train the model for 7 epochs in total. For the first 5 epochs, we use Adam with a learning rate of 3.0 × 10 -4 , we then use SGD with a learning rate of 0.1, and decay the learning rate by a factor of 0.5 at the beginning of epoch 7.

For the other training criteria, we use the hyperparameters with the best performance described in Section 6.1.

8.2.3. ADDITIONAL EXPERIMENTS:

Details We implement the squared Hellinger distance criterion on convolutional seq2seq model (Gehring et al., 2017) and strictly follow the settings of IWSLT experiments in Edunov et al. (2018) . 

:

Here we refer to the MRT model inShen et al. (2016), which contains sampling and renormalization as described in Section 4.2.

Figures and Tables:

10 -3 29.33 29.71 29.22 5.0 × 10 -4 29.91 31.13 31.01 5.0 × 10 -5 30.75 30.83 31.00

The final results on the test set are described in Table1. Max margin methods here improve the MLE baseline less significantly thanWiseman & Rush (2016) did, since the search optimization algorithm is not used in our experiments. The training criterion of squared Hellinger distance gives the highest BLEU score on the test set. BLEU scores on IWSLT German-English translation evaluated on the test set.

Table content: <table><tr><td colspan="2">criterion greedy beam</td></tr><tr><td>MLE</td><td>26.03 27.17</td></tr><tr><td>margin</td><td>26.22 27.32</td></tr><tr><td>slack</td><td>26.32 27.37</td></tr><tr><td>RAML</td><td>26.34 27.36</td></tr><tr><td>MRT</td><td>27.35 28.10</td></tr><tr><td colspan="2">Hellinger 27.75 28.38</td></tr></table>

Results on Gigaword sentence summarization task

Table content: <table><tr><td>criterion</td><td colspan="2">ROUGE-1</td><td colspan="2">ROUGE-2</td><td colspan="2">ROUGE-L</td><td colspan="2">ROUGE-S4</td></tr><tr><td/><td>valid</td><td>test</td><td>valid</td><td>test</td><td>valid</td><td>test</td><td>valid</td><td>test</td></tr><tr><td>MLE</td><td colspan="8">48.41 34.20 24.14 15.22 45.08 31.84 22.62 14.00</td></tr><tr><td>margin</td><td colspan="8">48.43 34.20 24.16 15.24 45.09 31.85 22.65 14.00</td></tr><tr><td>slack</td><td colspan="8">48.40 34.20 24.12 15.25 45.08 31.83 22.63 14.00</td></tr><tr><td>RAML</td><td colspan="8">48.43 34.20 24.16 15.24 45.10 31.84 22.66 14.00</td></tr><tr><td>MRT</td><td colspan="8">50.41 35.63 26.42 16.96 47.05 33.24 24.47 15.31</td></tr><tr><td>Hellinger</td><td colspan="8">51.02 35.84 26.88 17.03 47.71 33.47 25.12 15.54</td></tr><tr><td>MLE + BS</td><td colspan="8">49.69 36.21 25.52 16.37 46.55 32.75 24.00 15.01</td></tr><tr><td>MRT + BS</td><td colspan="8">50.72 36.47 26.74 17.62 47.41 34.05 25.05 15.98</td></tr><tr><td colspan="9">Hellinger + BS 51.07 36.59 26.95 17.61 47.62 34.03 25.34 16.04</td></tr></table>

Tokenized BLEU scores on IWSLT 2014 German-English translation evaluated on validation set using RAML criterion.

Table content: <table><tr><td>τ</td><td colspan="2">RAML-I RAML-S</td></tr><tr><td>1.0 × 10 -7</td><td>29.25</td><td>29.28</td></tr><tr><td>1.0 × 10 -5</td><td>29.27</td><td>29.30</td></tr><tr><td>1.0 × 10 -3</td><td>29.26</td><td>29.32</td></tr><tr><td>1.0 × 10 -1</td><td>29.28</td><td>29.32</td></tr></table>

Tokenized BLEU scores on IWSLT 2014 German-English translation evaluated on the validation set using squared Hellinger distance with different α and τ as training criterion.

Table content: <table/>

Tokenized BLEU scores on IWSLT 2014 German-English translation evaluated on the validation set using squared Hellinger Distance with different sample sizes.

Table content: <table><tr><td>sample size</td><td>20</td><td>50</td><td>100</td></tr><tr><td>BLEU</td><td>29.51</td><td>30.30</td><td>31.13</td></tr></table>

Tokenized BLEU scores on IWSLT 2014 German-English translation evaluated on the validation set using max margin criteria.

Table content: <table><tr><td>τ</td><td colspan="2">Slack Margin</td></tr><tr><td colspan="2">1.0 × 10 -1 29.16</td><td>29.23</td></tr><tr><td colspan="2">1.0 × 10 -3 29.33</td><td>29.25</td></tr><tr><td colspan="2">1.0 × 10 -5 29.35</td><td>29.16</td></tr><tr><td>8.2.2 SENTENCE SUMMARIZATION</td><td/></tr></table>

BLEU scores using convolutional seq2seq models.

Table content: <table><tr><td colspan="3">criterion MLE Hellinger</td></tr><tr><td>BLEU</td><td>32.14</td><td>32.30</td></tr></table>

Peer review (iclr2019):
 In this paper the authors distinguish between two families of training objectives for seq2seq models, namely, divergence minimization objectives and max-margin objectives. They primarily focus on the divergence minimization family, and show that the MRT and RAML objectives can be related to minimizing the KL divergence between the model*s distribution over outputs and the "exponentiated payoff distribution," with the two objectives differing in terms of the direction of the KL. In addition, the authors propose an objective using the Hellinger distance rather than the KL divergence, and they conduct experiments on machine translation and summarization comparing all the considered objectives. The paper is written extremely clearly, and is a pleasure to read. While the discussion of the relationship between RAML and MRT (and MRT and REINFORCE) is interesting and illuminating, many of these insights appear to have been discussed in earlier papers, and the RAML paper itself notes that it differs from REINFORCE style training in terms of the KL direction. On the other hand, the idea of minimizing Hellinger distance is I believe novel (though related to the alpha-divergence work cited by the authors in the related work section), and it*s nice that training with this loss improves over the other losses. Since the authors* results, however, appear to be somewhat below the state of the art, I think the main question left open by the experimental section is whether training with the Hellinger loss would further improve state of the art models. Even if it would not, it would still be interesting to understand why, and so I think the paper could be strengthened either by outperforming state of the art results or, perhaps through an ablation analysis, showing what aspects of current state of the art models make minimizing the Hellinger loss unnecessary. In summary, Pros: - well written and interesting - a new loss with potential for improvement over other losses - fairly thorough experiments Cons: - much of the analysis is not new - unclear if the proposed loss will improve the state of the art, and if not why Update after author response: thanks for your response. I think the latest revision of the paper is improved, and even though state of the art BLEU scores on IWSLT appear to be in the mid 33s, I think the improvement over the Convolutional Seq2seq model is encouraging, and so I*m increasing my score to 7. I hope you*ll include these newer results in the paper.