Year: 2019

Abstract:
Sequence to sequence (seq2seq) models have become a popular framework for neural sequence prediction. While traditional seq2seq models are trained by Maximum Likelihood Estimation (MLE), much recent work has made various attempts to optimize evaluation scores directly to solve the mismatch between training and evaluation, since model predictions are usually evaluated by a task specific evaluation metric like BLEU or ROUGE scores instead of perplexity. This paper puts this existing work into two categories, a) minimum divergence, and b) maximum margin. We introduce a new training criterion based on the analysis of existing work, and empirically compare models in the two categories. Our experimental results show that our new training criterion can usually work better than existing methods, on both the tasks of machine translation and sentence summarization.

heading: 1 INTRODUCTION
text: Sequence to sequence (seq2seq) models (Kalchbrenner & Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015) are a powerful end-to-end solution to sequence prediction since they allow a mapping between two sequences of different lengths. However, Maximum Likelihood Estimation (MLE) which is used as a standard training criterion in seq2seq has a series of drawbacks:
• Training and evaluation mismatch: during training, we maximize the log likelihood, while during inference, the model is evaluated by a different metric such as BLEU or ROUGE;
• MLE fails to assign proper scores to different model outputs, which means that all incorrect outputs are treated equally.
Similar to SEARN (Daumé et al., 2009) and DAGGER (Ross et al., 2011) in traditional models, many reinforcement learning (RL) techniques have been extended to seq2eq models recently to solve the mismatch between training and inference. The key idea is to replace MLE with a criterion that gives task-specific scores to different model outputs.
Recent approaches to taking the evaluation metric into training criteria generally fall into one of the following two categories:
1. minimum divergence: Minimum divergence methods minimize some distance functions between the model output distribution and the probability distribution given by the ground truth, and by optimizing the divergence between the two distributions, the model output distribution will ideally be equal to the ground truth distribution.
2. maximum margin: During inference, in structured prediction, the goal is to simply ensure good structures have the highest predicted score. That is, margin based learning differs from minimum divergence learning in the aspect that it only requires that the predicted score of the correct output should be higher than that of the others by a margin.
∗Corresponding author.
The main contribution of this paper can be summarized as: based on the analysis of existing work, we introduce a novel training criterion for seq2seq learning and show its effectiveness.
Based on our experimental results on the tasks of machine translation and sentence summarization, we conclude that some models that take the evaluation metric into consideration can improve over a strong MLE baseline by a large margin, and our new training criterion can usually get better results than existing work.

heading: 2 RELATED WORK
text: Seq2seq models were proposed and improved by Kalchbrenner & Blunsom (2013) and Sutskever et al. (2014). With the attention mechanism (Bahdanau et al., 2015; Luong et al., 2015), seq2seq models achieve better results than traditional models in many fields including machine translation (Bahdanau et al., 2015; Luong et al., 2015) and text summarization (Chopra et al., 2016).
Before seq2seq models, previous work on optimizing task specific evaluation metric scores generally includes SEARN (Daumé et al., 2009), DAGGER (Ross et al., 2011), Minimum Error Rate Training (MERT) (Och, 2003), and softmax-margin criterion (Gimpel & Smith, 2010).
Seq2seq model uses MLE as the training objective, which also has the problem of training and evaluation mismatch. Ranzato et al. (2016) incorporated the evaluation metric into training of seq2seq models and proposed the mixed incremental cross entropy REINFORCE (MIXER) training strategy, which is similar to the idea of Minimum Risk Training (MRT) (Smith & Eisner, 2006; Li & Eisner, 2009; Ayana et al., 2016; Shen et al., 2016). MIXER used decoder hidden states to predict the bias term to reduce the variance, while MRT renormalized the predicted probabilities.
Inspired by MIXER, Bahdanau et al. (2017) extended the actor critic algorithm (Konda & Tsitsiklis, 2000; Sutton et al., 2000) to the seq2seq framework. Sokolov et al. (2016) and Kreutzer et al. (2017) proposed the pairwise preference learning framework, which enables the model to learn from the difference between a pair of samples generated by the model. Another popular model in the form of divergence minimization is the smoothed version of MERT algorithm proposed by Och (2003). The smoothed MERT algorithm also minimizes the expected loss over samples from model outputs and is similar to MRT essentially. Norouzi et al. (2016) proposed Reward Augmented Maximum Likelihood (RAML) which also optimizes the divergence by data augmentation, and RAML was further combined with maximum likelihood via α-divergence recently (Koyamada et al., 2017).
In contrast to the above divergence minimization based approaches, Wiseman & Rush (2016) used a margin based training objective and their search optimization process requires performing parameter updates or inference each time a new word is generated. One main drawback of maximum margin is that given a sample subset of an arbitrary size, only two samples of the subset can contribute to the gradient. The softmax-margin criterion proposed by Gimpel & Smith (2010) can alleviate this problem, in which all samples are used in loss and gradient computation. However, Edunov et al. (2018) shows that the softmax-margin criterion still cannot reach the performance of risk minimization. As the latest work, Edunov et al. (2018) is very similar to ours. However, our interests are in finding better training criteria on a consistent setting, while they mainly focused on absolute performance improvement, using techniques like combination with token level loss, online vs. offline candidate generation, etc.

heading: 3 SEQUENCE TO SEQUENCE MODEL
text: This paper considers the seq2seq model from Bahdanau et al. (2015) and Luong et al. (2015), which consists of an RNN encoder and an RNN decoder. Due to the gradient vanishing problem, the vanilla RNN cell is replaced with a Gated Recurrent Unit (GRU) (Cho et al., 2014) or Long ShortTerm Memory (LSTM) (Hochreiter & Schmidhuber, 1997). A bidirectional RNN is used as encoder, with one RNN reading the source sequence from beginning to end, and the other RNN reading in reversed order. The bidirectional RNN encoder encodes the input sequence and gets a sequence of hidden states by concatenating the hidden states of each word in the two directions, which will be used as attention information in the decoder RNN.
The decoder RNN is initialized by passing the two final hidden states of the encoder to a feed forward network. The mixed information used as input in the first step m0 is initialized by a zero vector. In decoding step i, using the input feeding approach described in Luong et al. (2015), the decoder RNN reads the embedding of the wordwi and the mixed informationmi−1, to generate the hidden state of current step hi:
hi = f1(concat(wi , mi−1), hi−1),
where f1 is the RNN cell. The decoder then generates the attention context ci from encoder using hi. Since there are many different attention mechanisms, we omit the details here. After getting ci, the model mixes ci and hi together by a feedforward network f2 to generate mi, which is used as the mixed information in the next step:
mi = f2(ci,hi)
Finally, mi is passed to a feedforward network to generate oi, and the softmax layer outputs the probability for each word at step i. Suppose the target word in step i is v′, the probability of the target word in step i is given by
p(yi = v ′ |x,y1:i−1) = exp(oiv′)∑ v exp(oiv) .
For the sequence y = 〈y1,y2, ...,yl〉, the probability p(y |x) is p(y |x) = p(y1 |x) p(y2 |x,y1)...p(yl |x,y1:l−1).
Suppose y∗ is the correct output, MLE actually maximizes the log likelihood of p(y∗ |x):
LMLE = − log p(y∗ |x) = − l∑ i=1 log p(y∗i |x,y∗<i)
For a detailed explanation, please refer to Luong et al. (2015). Now we describe the algorithm to construct a subset S from all possible output sequences which will be used in Section 4 & 5. In seq2seq model, the n-best list can only be approximated by performing beam search, as used in Wiseman & Rush (2016), or Monte Carlo (MC) sampling, as used in Ayana et al. (2016) and Shen et al. (2016). While using beam search may produce more accurate results, the time required for generating a candidate set of the same size as MC sampling using beam search is too long. In this paper, we construct the approximated n-best list S using MC sampling proposed in Shen et al. (2016) , and for completeness, we describe the sampling approach in Appendix 8.1 in details. Please note that the sampling algorithm does not guarantee the size of the sample subset, since it is inefficient to force the model to generate an exact number of distinct samples using GPU.

heading: 4 MINIMUM DIVERGENCE
text: In this section, we consider training criteria in the form of the following equation:
L = D( p1 || p2 ), where p1 and p2 are two probability distributions given by the model or the ground truth, and D is a distance function, for example, Kullback Leibler (KL) divergence or cross entropy. The baseline training criterion MLE can also be seen as minimum divergence training, where the cross entropy between the model output distribution and the one-hot encoded ground truth is minimized. Now we will discuss training criteria in which the ground truth distribution contains information of evaluation metric scores.

heading: 4.1 REWARD AUGMENTED MAXIMUM LIKELIHOOD
text: Suppose that {x,y∗} is a given input-output pair, the model learns parameters θ and gives prediction p(y |x;θ) on the output space Y, the exponentiated payoff distribution (Norouzi et al., 2016) q(y |y∗) is defined as follows:
q(y |y∗) = 1 Z(y∗) exp[ r(y,y∗)/τ ], (1)
where Z(y∗) = ∑ y∈Y exp[ r(y,y
∗)/τ ] is the normalization factor. τ is a hyperparameter that controls the smoothness of the distribution, and r(y,y∗) is the score of prediction y given by the evaluation metric, e.g. BLEU (Papineni et al., 2002), or ROUGE (Lin, 2004)
The training objective of Reward augmented Maximum Likelihood (RAML) (Norouzi et al., 2016) is defined as LRAML = −Eq(y |y∗)[ log p(y |x;θ) ], (2) in which, the gradient is:
Eq(y |y∗)[∇θ log p(y |x;θ) ] = − ∑ y∈Y q(y |y∗) ∇θ log p(y |x;θ) (3)
Equation (3) requires computing q(y |y∗) for all y and performing gradient descent on the whole Y. However, due to the huge search space of Y and the high computational cost of r(y,y∗), Equation (3) cannot be directly computed in an efficient way. Norouzi et al. (2016) proposed using stratified sampling. One can sample from negative edit distance or Hamming distance and then do importance reweighting to reweight the samples with task specific scores. After sampling from q(y |y∗), the model is trained to maximize log p(y |x;θ). Another choice is to construct a sample subset S from the model outputs using MC sampling, and use Equation (1) to compute the probability q′(y |y∗) in the sample subset. The training criterion is then changed to
L′RAML = − ∑ y∈S q′(y |y∗) log p(y |x;θ),
where
q′(y |y∗) = exp[ r(y,y ∗)/τ ]∑
y′∈S exp[ r(y ′,y∗)/τ ]
.

heading: 4.2 MINIMUM RISK TRAINING
text: We first introduce the REINFORCE algorithm (Williams, 1992), which is a famous policy gradient method in RL, and then compare it with the Minimum Risk Training (MRT) (Smith & Eisner, 2006; Li & Eisner, 2009; Ayana et al., 2016; Shen et al., 2016). In REINFORCE, given a state x, an action y under state x, and model parameters θ, the policy gradient g is
g = (r(y)− B(x))∇θ log p(y |x;θ), (4)
where r(y) is the reward of action y, and B(x) is the baseline or control variate that is used to reduce the variance (without introducing bias).
In MRT, the training objective is the expectation of evaluation metric scores taken with respect to the model output distribution:
LMRT = −Ep(y |x;θ)[ r(y,y∗) ]. (5)
Also due to the huge search space, Equation (5) can only be estimated by sampling from the model outputs. Suppose S is a sample subset from Y, p(y |x;θ) is replaced by p′(y |x;θ;α):
p′(y |x;θ;α) = p(y |x;θ) α∑
y′∈S p(y ′ |x;θ)α
, (6)
where α is a hyperparameter that controls the smoothness of p′(y |x;θ;α). We will later compare the effect of α here with τ in RAML.
The gradient of LMRT can be derived as:
−∇θ Ep′(y |x;θ,α)[ r(y,y∗) ] = − αEp′(y |x;θ,α)[∇θ log p(y |x;θ)(r(y,y∗)−B) ],
with B = Ep′(y′ |x;θ,α)[ r(y′,y∗) ]. (7)
If we regard the whole seq2seq framework as a one-step Markov Decision Process (MDP), in which the input sequence x is the state, the output sequence y is the action, and the evaluation metric score
r is the reward, both REINFORCE and MRT actually optimize the same training criterion. The difference between them is that REINFORCE requires only a single sample and is unbiased, while MRT has multiple samples and the probability renormalization in Equation (6) introduces bias. The expected loss objective in Kreutzer et al. (2017) is also similar (and closer) to REINFORCE which uses one sample to estimate the gradient.

heading: 4.3 COMPARISON OF RAML & MRT
text: The cross entropy is defined as:
DXENT(p1(x)||p2(x)) = − Ep1(x)[log p2(x)]. (8)
Starting from RAML, in terms of cross entropy, Equation (2) can be rewritten as:
LRAML = DXENT(q(y |y∗) || p(y |x;θ))
The training criterion of RAML is essentially minimizing the the cross entropy loss from q(y |y∗) to p(y |x;θ). Meanwhile, the training criterion of MRT in Equation (5) can be rewritten as:
LMRT = − τ Ep(y |x;θ)[ r(y,y∗)/τ ] = − τ Ep(y |x;θ)[ log q(y |y∗) ]− τ Ep(y |x;θ)[ logZ(y∗) ] = − τ Ep(y |x;θ)[ log q(y |y∗) ]− τ logZ(y∗),
in which, the second term is a constant as it has no gradient with respect to θ. Thus we have:
LMRT = τ DXENT(p(y |x;θ)||q(y |y∗)) + Const. (9)
Equation (9) actually shows that MRT minimizes the cross entropy loss from p(y |x;θ) to q(y |y∗). By comparing the above two methods, we find that both RAML and MRT are minimizing the cross entropy loss between the model output distribution and the exponentiated payoff distribution, but with different directions of DXENT. Now we will discuss the effect of hyperparameter τ in RAML and α in MRT.
Cross entropy loss has a property that it is minimized if and only if the two probability distributions are equal. Ideally, when the cross entropy loss in RAML is minimized, we have:
p(y |x;θ) = q(y |y∗) log p(y |x;θ) = r(y,y∗)/τ + Const
r(y,y∗) = τ log p(y |x;θ) + Const.
And in MRT 1,
LMRT = − Ep′(y |x;θ)[ log exp r(y,y∗)∑ y′ exp r(y ′,y∗) ] + Const.
When the two probability distributions are equal, we have:
r(y,y∗) = α log p(y |x;θ) + Const. (10)
During training of both RAML and MRT, the model is trying to predict the evaluation score r by multiplying log p with a constant (either τ in RAML or α in MRT).

heading: 4.4 HELLINGER DISTANCE MINIMIZATION
text: Now we introduce another popular distance metric, the Hellinger distance (Hellinger, 1909), which also belongs to the family of Csiszár f-divergence (Csiszár, 1963) and has been used in graphical models (Beykikhoshk et al., 2015) and clustering algorithms (Abdel-Azim, 2016; Ji et al., 2018).
1Here we refer to the MRT model in Shen et al. (2016), which contains sampling and renormalization as described in Section 4.2.
Given two probability distributions p1 and p2, let f and g denote the probability density functions of p1 and p2. The square of the Hellinger distance H(p1, p2) is defined as:
H2(p1, p2) = 1
2
∫ ( √ f(x)− √ g(x))2 dx.
The training criterion using squared Hellinger distance is: LHellinger = ∑ y∈S ( √ p′(y |x;θ)− √ q(y |y∗) )2,
with p′(y |x;θ) defined in Equation (6) and q(y |y∗) defined in Equation (1).

heading: 5 MAXIMUM MARGIN
text: Let ∆ denote the loss given by the evaluation metric:
∆(y,y∗) = r(y∗,y∗)− r(y,y∗).
∆ is 0 when y = y∗ and positive otherwise. Let F be the predicted score of the model:
F = τ log p(y |x;θ).
Different from minimizing the expected risk, another choice of training criterion is to optimize the evaluation score of the sequence with the highest predicted score:
arg min θ
∆(ȳ,y∗), (11)
where ȳ = arg max
y F (y,θ).
For each input sequence x, we first do inference to find the output ȳ that has the highest predicted score, then we train the model to maximize the score of ȳ given by evaluation metric.
However, directly optimizing the training criterion that includes an arg max is not computationally tractable. In structured prediction, this problem has been widely discussed in the training process of Structural Support Vector Machine (SSVM) (Tsochantaridis et al., 2004; Joachims, 2006; Joachims et al., 2009). Tsochantaridis et al. (2004) proposed using margin and slack rescaling instead. Margin Rescaling is based on the following inequality:
∆(ȳ,y∗) ≤ ∆(ȳ,y∗) + F (ȳ,θ)− F (y∗,θ) ≤ max y (∆(y,y∗) + F (y,θ))− F (y∗,θ) (12)
Similarly, Slack Rescaling is:
∆(ȳ,y∗) ≤ ∆(ȳ,y∗) (1+F (ȳ,θ)−F (y∗,θ)) ≤ max y ∆(y,y∗) (1+F (y,θ)−F (y∗,θ)) (13)
As Equation (13) is a tighter bound on ∆(ȳ,y∗), slack rescaling generally gives better results than margin rescaling. Both Equation (12) and Equation (13) can be optimized by subgradient descent. Let ŷ denote the sample that gives the max value. The subgradient for margin rescaling is:
∇θτ log p(ŷ)−∇θτ log(p(y∗)),
and similarly, the subgradient for slack rescaling is:
∆(ŷ,y∗)(∇θτ log p(ŷ)−∇θτ log p(y∗)).
Since doing loss augmented inference on such a huge search space is again not realistic, approximate inference is still needed. Equation (12) also has the name of structured hinge loss when used in SSVM. One may have found that Equation (13) is the same as the training criterion in Wiseman & Rush (2016). Besides, to make a fair comparison with other training criteria and make it applicable on large datasets, we replace the beam search with sample subset.

heading: 6 EXPERIMENTS
text: We evaluate the training criteria on two tasks in total. The first task is machine translation on a small dataset to explore the hyperparameter impact. The second task is sentence summarization using large datasets.
For all the experiments, we use LSTM as RNN cell. The attention mechanism used in our experiments is global attention with input feeding (Luong et al., 2015).

heading: 6.1 MACHINE TRANSLATION
text: Due to space limit, experimental results on hyperparameter effects and training details are provided in Appendix 8.2.1.
Data We use the IWSLT 2014 German-English translation dataset, with the same splits as Ranzato et al. (2016) and Wiseman & Rush (2016), which contains about 153K training sentence pairs, 7K validation sentence pairs and 7K test sentence pairs. Sentences in the dataset are first tokenized and then converted into lowercase. During training, the maximum length for both inputs and outputs is restricted to 50. Words that appear less than three times are replaced with a UNK token.
Models The encoder is a single-layer bidirectional LSTM with 256 hidden units for either direction, and the decoder LSTM also has 256 hidden units. The size of word embedding for both encoder and decoder is 256. We use a dropout rate (Srivastava et al., 2014) of 0.2 to avoid overfitting. The gradient is clipped when its norm exceeds 1.0 to prevent gradient explosion and stabilize learning (Pascanu et al., 2013). The batch size is set to 32 and the training set is shuffled at each new epoch. All models are trained with the Adam optimizer (Kingma & Ba, 2015). The sentence level BLEU score used as reward or loss during training is smoothed by adding 1 to both numerator and denominator, as suggested by Chen & Cherry (2014).
Results For the two max margin methods, τ is set to 1.0×10−3. We use the sample subset version of RAML, and τ for RAML is also set to 1.0×10−3. For Hellinger loss, α = 5.0×10−4 and τ = 0.5. The final results on the test set are described in Table 1. Max margin methods here improve the MLE baseline less significantly than Wiseman & Rush (2016) did, since the search optimization algorithm is not used in our experiments. The training criterion of squared Hellinger distance gives the highest BLEU score on the test set.

heading: 6.2 SENTENCE SUMMARIZATION
text: The task of sentence summarization is: given a long sentence or a passage, the model will generate a short sentence that summarizes the long text. For evaluation, following Rush et al. (2015), we use the full length F1 variant of ROUGE score (Lin, 2004) as metric. Training details are described in Appendix 8.2.2.
Data We use the Gigaword corpus with the same preprocessing steps as in Rush et al. (2015). In the Gigaword dataset, the first sentence of each article is regarded as the source sequence, and the title of the article as the target sequence. The training set consists of 3.7M sequence pairs, and the vocabulary size for both article and title is 50K, with the out-of-vocabulary words replaced by a unified UNK token. The whole dataset is then converted into lowercase without tokenization.
During training, we use the first 2K sequences of the dev corpus as validation set, and the size of the test set is also 2K. The maximum length for both input and output sequence is restricted to 50.
Models The encoder is a single layer bidirectional LSTM with 1000 hidden units for either direction. The single layer decoder LSTM also has 1000 dimensions. The word embedding size for both encoder and decoder is 500. Dropout is not used in the sentence summarization task, and the gradient is clipped when its norm exceeds 1.0. The batch size is set to 128. The training set is shuffled at each new epoch, and we prefetch 20 batches and sort them by the length of target sequence to form new batches to speed up training. We use ROUGE-2 as the reward or loss function.
Results Table 2 shows the results on the validation set and the test set. We report ROUGE-1, ROUGE-2, ROUGE-L and ROUGE-S4 score. MRT outperforms the MLE baseline by a large margin, but is still slightly lower than the newly introduced Hellinger loss.

heading: 7 DISCUSSION
text: For training criteria in the form of divergence minimization, according to our experimental results, MRT outperforms RAML. As mentioned before, MRT and RAML minimize the cross entropy in two different directions. By the definition of cross entropy loss in Equation (8), it is only minimized when p1 = p2. Optimizing the cross entropy in either of the two directions both guarantee consistency, i.e., the training procedure can ultimately recover the true probability distribution.
For a model that minimizes DXENT(p1||p2), the model will sample from p1 and maximize log p2, which means that if x has a high probability in p1, it will also have a high probability in p2, but if has a low probability in p1, p2(x) may still be high.
Thus in RAML which samples from exponentiated payoff distribution q and maximizes log p, sequences that have a high evaluation score will also have a high predicted probability. However, samples that have a low q, e.g., a low BLEU or ROUGE, may still have a high p.
In contrast, MRT will first sample from model outputs and ensure that a high BLEU or ROUGE score will be assigned to sequences that have a high probability in model output distribution p, at the cost that sequences that have low probabilities in p may still have high evaluation scores. This can explain the difference between RAML and MRT.
The squared Hellinger distance can be regarded as the squared error between p 1 2 and q 1 2 , which is essentially a regression loss, while the cross entropy loss usually works better in classification problems. By optimizing the cross entropy loss, the model is trained to find the candidate that has the highest probability and pays less attention to samples that have low probabilities, while by optimizing the squared Hellinger distance, the model learns to predict the evaluation metric score for every sample in the candidate set and pays equal attention to all of them. Since seq2seq models
generate a predicted sequence word by word, the target sequence which has the highest evaluation metric score (sample from q and maximize log p) or the highest predicted probability (sample from p and maximize log q) may not be reached by approximate inference algorithms like greedy search or beam search. In this case, a model trained by optimizing the cross entropy loss may not work as expected, however, a model trained by optimizing the squared Hellinger distance can still make predictions normally since it pays equal attention to samples with low probabilities during training.
For maximum margin methods, slack rescaling is a tighter bound and the experimental result is slightly higher than margin rescaling on our small scale experiments. However, slack rescaling still cannot achieve the same level of performance of MRT and Hellinger distance. The reason may be that the upper bound is still not tight enough. In the training process of maximum margin methods, only the target sequence and a negative sample selected from the candidate set are used to update model parameters, which may also explain their performance.

heading: ACKNOWLEDGMENTS
text: This paper was partially supported by National Key Research and Development Program of China (No. 2017YFB0304100), Key Projects of National Natural Science Foundation of China (U1836222 and 61733011), Key Project of National Society Science Foundation of China (No. 15-ZDA041), The Art and Science Interdisciplinary Funds of Shanghai Jiao Tong University (No. 14JCRZ04).

heading: 8 APPENDIX

heading: 8.1 THE SAMPLING ALGORITHM
text: For completeness, we describe the MC sampling algorithm proposed by Shen et al. (2016) and Ayana et al. (2016) here.
Algorithm 1 The sampling approach to constructing the approximated n-best list Input:
source sequence x, model parameters θ, target sequence y∗ sample size m, maximum length l
Output: A set of samples S and corresponding probability P
1: S = {y∗}; P = {p(y∗)}; 2: for i in 1, ...,m− 1 do 3: sample a sequence word by word given x and θ, stop sampling when length is l or meeting
〈end-of-sequence〉 symbol, and finally store the result and probability in y and q respectively
4: if y in S then 5: continue 6: end if 7: S = S ∪ {y} 8: P = P ∪ {q} 9: end for
10: return S, P

heading: 8.2 EXPERIMENTS

heading: 8.2.1 MACHINE TRANSLATION
text: The MLE baseline is trained with a learning rate of 3.0× 10−4. The model is trained for 20 epochs in total, and is evaluated every 1000 steps on the validation set.
τ for RAML For RAML, since we use BLEU instead of Hamming distance as evaluation metric, we first show the BLEU scores on the validation set for hyperparameter τ in Table 3. RAML-I refers to importance reweighting and RAML-S refers to sample subset RAML, as mentioned in Section 4.1. Experimental results show that the BLEU score is not very sensitive to hyperparameter τ although τ varies from 1.0× 10−7 to 1.0× 10−1, and setting τ larger will decrease the BLEU score compared with the MLE baseline. We also find that RAML-S tends to give slightly better results than RAML-I. The learning rate for RAML is 1.0× 10−6.
α for MRT For MRT, since Shen et al. (2016) did several experiments to study the effect of hyperparameters, the original hyperparameters are strictly followed in our experiments. Hyperparameter α is set to 5.0 × 10−3 and the sample size is 100. The best BLEU score of MRT on the validation set is 30.55. The learning rate is set to 1.0× 10−5. α and τ for Hellinger loss As shown in Table 4, α = 5.0× 10−4, τ = 0.5 produces the best result. The highest BLEU score on the validation set is 31.13, which is 0.58 higher than MRT. The learning rate used in Hellinger loss is also 1.0× 10−5.
Sample size for Hellinger loss For squared Hellinger loss, the effect of sample size is shown in Table 5. A larger sample size generally gives a higher BLEU score, however, due to the limit of GPU memory, we cannot set sample size to a larger value.
τ for max margin For the two max margin training criteria, the learning rate is 1.0 × 10−7. Table 6 shows the effect of τ : according to the results on the validation set, τ is not very sensitive to the BLEU scores. As shown in Table 6, BLEU scores given by slack rescaling are usually 0.1 higher than for margin rescaling, and we use τ = 1.0× 10−3 for both of the two models.

heading: 8.2.2 SENTENCE SUMMARIZATION
text: Details We use ROUGE-2 as reward or loss function. For the MLE baseline, we train the model for 7 epochs in total. For the first 5 epochs, we use Adam with a learning rate of 3.0 × 10−4, we then use SGD with a learning rate of 0.1, and decay the learning rate by a factor of 0.5 at the beginning of epoch 7.
For the other training criteria, we use the hyperparameters with the best performance described in Section 6.1.

heading: 8.2.3 ADDITIONAL EXPERIMENTS
text: Details We implement the squared Hellinger distance criterion on convolutional seq2seq model (Gehring et al., 2017) and strictly follow the settings of IWSLT experiments in Edunov et al. (2018).

Review summaries:
In this paper the authors distinguish between two families of training objectives for seq2seq models, namely, divergence minimization objectives and max-margin objectives. They primarily focus on the divergence minimization family, and show that the MRT and RAML objectives can be related to minimizing the KL divergence between the model's distribution over outputs and the "exponentiated payoff distribution," with the two objectives differing in terms of the direction of the KL. In addition, the authors propose an objective using the Hellinger distance rather than the KL divergence, and they conduct experiments on machine translation and summarization comparing all the considered objectives.

The authors have updated the paper and clarified some things, and now my impression of the paper has improved. It still feels a little incremental to me, but the potential application areas of these sorts of models are quite large and therefore incremental improvements are not insignificant. This paper suggests some natural follow-up work in exploring Hellinger distance and other variations for these models.