Venue: NAACL 2019

# Differentiable Sampling with Flexible Reference Word Order

for Neural Machine Translation

Weijia Xu

University of Maryland

weijia@cs.umd.edu &Xing Niu

University of Maryland

xinguiu@cs.umd.edu &Marine Carpuat

University of Maryland

marine@cs.umd.edu

###### Abstract

Despite some empirical success at correcting exposure bias in machine translation, scheduled sampling algorithms suffer from a major drawback: they incorrectly assume that words in the reference translations and in sampled sequences are aligned at each time step. Our new differentiable sampling algorithm addresses this issue by optimizing the probability that the reference can be aligned with the sampled output, based on a soft alignment predicted by the model itself. As a result, the output distribution at each time step is evaluated with respect to the whole predicted sequence. Experiments on IWSLT translation tasks show that our approach improves BLEU compared to maximum likelihood and scheduled sampling baselines. In addition, our approach is simpler to train with no need for sampling schedule and yields models that achieve larger improvements with smaller beam sizes.1

Footnote 1: The code is available at [https://github.com/Izecson/saml-nmt](https://github.com/Izecson/saml-nmt)

## 1 Introduction

Neural machine translation (NMT) models are typically trained to maximize the likelihood of reference translations Sutskever et al. (2014); Bahdanau et al. (2015). While simple and effective, this objective suffers from the exposure bias problem Ranzato et al. (2015): the model is only exposed to reference target sequences during training, but has to rely on its own predictions at inference. As a result, errors can accumulate along the generated sequence at inference time.

This is a well-known issue in sequential decision making Langford and Zadrozny (2005); Cohen and Carvalho (2005); Kaariainen and Langford (2006), i.a.) and it has been addressed in past work by incorporating the previous decoding choices into the training scheme, using imitation learning Daume et al. (2009); Ross et al. (2011); Bengio et al. (2015); Leblond et al. (2018) and reinforcement learning Ranzato et al. (2015); Bahdanau et al. (2016) techniques. In this paper, we focus on a simple and computationally inexpensive family of approaches, known as Data as Demonstrator Venkatraman et al. (2015) and scheduled sampling Bengio et al. (2015); Goyal et al. (2017). The algorithms use a stochastic mixture of the reference words and model predictions with an annealing schedule controlling the mixture probability. Despite their empirical success in various sequence prediction tasks, they are based on an assumption that does not hold for machine translation: they assume that words in the reference translations and in sampled sequences are aligned at each time step, which results in weak and sometimes misleading training signals.

In this paper, we introduce a differentiable sampling algorithm that exposes machine translation models to their own predictions during training, and allows for differences in word order when comparing model outputs with reference translations. We compute the probability that the reference can be aligned with the sampled output using a soft alignment predicted based on the model states, so that the model will not be punished too severely for producing hypotheses that deviate from the reference, as long as the hypotheses can still be aligned with the reference.

Experiments on three IWSLT tasks German-English, English-German and Vietnamese-English) show that our approach significantly improves BLEU compared to both maximum likelihood and scheduled sampling baselines. We also provide evidence that our approach addresses exposure bias by decoding with varying beam sizes, and show that our approach is simpler to train than scheduled sampling as it requires no annealing schedule.





## 2 Approach

Our approach is designed to optimize the standard sequence-to-sequence model for translating a source sentence \({\mathbf{x}}\) into a target sentence \({\mathbf{y}}\)Bahdanau et al. (2015). This model computes the probability of \({\mathbf{y}}\) given \({\mathbf{x}}\) as:

\[P({\mathbf{y}}\,|\,{\mathbf{x}})=\prod_{t=1}^{T}p(y_{t}\,|\,{\mathbf{y}}_{<t},{\mathbf{x}};\theta) \tag{1}\]

where \(\theta\) represents the model parameters. Given \({\mathbf{x}}\), the model first produces a sequence of hidden representations \({\mathbf{h}}_{1...T}\): \({\mathbf{h}}_{t}=f({\mathbf{y}}_{<t},{\mathbf{x}})\), where \(T\) is the length of \({\mathbf{y}}\), and \(f\) is usually an encoder-decoder network. At each time step \(t\), the hidden representation \({\mathbf{h}}_{t}\) is fed to a linear projection layer \({\mathbf{s}}_{t}={\mathbf{W}}{\mathbf{h}}_{t}+{\mathbf{b}}\) to obtain a vector of scores \({\mathbf{s}}_{t}\) over all possible words in the vocabulary \({\cal V}\). Scores are then turned into a conditional probability distribution: \(p(\cdot\,|\,{\mathbf{y}}_{<t},{\mathbf{x}};\theta)=\mathrm{softmax}({\mathbf{s}}_{t})\).

The traditional maximum likelihood (ML) objective maximizes the log-likelihood of the training data \({\cal D}\equiv\{({\mathbf{x}}^{(n)},{\mathbf{y}}^{(n)})\}_{n=1}^{N}\) consisting of \(N\) pairs of source and target sentences:

\[{\cal J}_{ML}(\theta)=\sum_{n=1}^{N}\sum_{t=1}^{T}\log p(y_{t}^{(n)}\,|\,{\bm {y}}_{<t}^{(n)},{\mathbf{x}}^{(n)};\theta) \tag{2}\]

At test time, prefixes \(~{}{\mathbf{y}}_{<t}\) are subsequences generated by the model and therefore contain errors. By contrast, in ML training, prefixes \({\mathbf{y}}_{<t}\) are subsequences of reference translations. As a result, the model is never exposed to its own errors during training and errors accumulate at test time. This mismatch is known as the exposure bias problem Ranzato et al. (2015).

### Limitations in Scheduled Sampling

Bengio et al. (2015) introduced the scheduled sampling algorithm to address exposure bias. Scheduled sampling gradually replaces the reference words with sampled model predictions in the prefix used at training time. An annealing schedule controls the probability of using reference words vs. model predictions. The training objective remains the same as the ML objective, except for the nature of the prefix \(\hat{{\mathbf{g}}}_{<t}\), which contains a mixture of reference and predicted words:

\[{\cal J}_{SS}(\theta)=\sum_{n=1}^{N}\sum_{t=1}^{T}\log p(y_{t}^{(n)}\,|\,\hat {{\mathbf{y}}}_{<t}^{(n)},{\mathbf{x}}^{(n)};\theta) \tag{3}\]

Despite the empirical success of scheduled sampling, one limitation is that the discontinuity of the argmax operation makes it impossible to penalize errors made in previous steps, which can lead to slow and unstable training. We address this issue using a continuous relaxation to the greedy search and sampling process, similarly to Goyal et al. (2017), which we describe in Section 2.2.

Another limitation of scheduled sampling is that it incorrectly assumes that the reference and predicted sequence are aligned by time indices which introduces additional noise to the training signal.2 We address this problem with a novel differentiable sampling algorithm with an alignment based objective called soft aligned maximum likelihood (SAML). It is used in combination with maximum likelihood to define our training objective \({\cal J}={\cal J}_{ML}+{\cal J}_{SAML}\), where \({\cal J}_{ML}\) is computed based on reference translations, and \({\cal J}_{SAML}\) is computed based on sampled translations of the same input sentences. We define \({\cal J}_{SAML}\) in Section 2.3.

Footnote 2: [https://nlpers.blogspot.com/2016/03/a-dagger-name-scheduled.html](https://nlpers.blogspot.com/2016/03/a-dagger-name-scheduled.html)

### Differentiable Sampling

To backpropagate errors made in the previous decoding steps, we use a continuous relaxation of the discrete sampling operation similar to Goyal et al. (2017), except that we use the Straight-Through (ST) Gumbel-Softmax estimator Jang et al. (2017); Bengio et al. (2013) instead of Gumbel-Softmax Jang et al. (2017); Maddison et al. (2014) to better simulate the scenario at inference time.3

Footnote 3: The Straight-Through estimator consistently outperforms the Gumbel-Softmax in preliminary experiments.

The Gumbel-Softmax is derived from the Gumbel-Max trick Maddison et al. (2014), an algorithm for sampling one-hot vector \(z\in\mathbb{R}^{k}\) from a categorical distribution \((p_{1},...,p_{k})\):

\[z=\text{one-hot}(\operatorname*{arg\,max}_{i}(\log p_{i}+\beta g_{i})) \tag{4}\]

where \(g_{i}\) is the Gumbel noise drawn i.i.d from Gumbel\((0,1)\)4, and \(\beta\) is a hyperparameter controlling the scale of the noise. Here, the trick is used to approximate the discontinuous argmax function with the differentiable softmax:

Footnote 4: \(g_{i}=-\log(-\log(u_{i}))\) and \(u_{i}\sim\mathrm{Uniform}(0,1)\).

\[\tilde{z}=\mathrm{softmax}((\log p_{i}+\beta g_{i})/\tau) \tag{5}\]

where \(\tau\) is the temperature parameter. As \(\tau\) diminishes to zero, \(\tilde{z}\) becomes the same as one-hot sample \(z\).

The Straight-Through Gumbel-Softmax maintains the differentiability of the Gumbel-Softmax estimator while allowing for discrete sampling by taking different paths in the forward and backward pass. It uses argmax to get the one-hot sample \(z\) in the forward pass, but uses its continuous approximation \(\tilde{z}\) in the backward pass. While ST estimators are biased, they have been shown to work well in latent tree learning [15] and semi-supervised machine translation [20].

### Soft Aligned Maximum Likelihood

The soft aligned maximum likelihood (SAML) is defined as the probability that the reference can be aligned with the sampled output using a soft alignment predicted by the model:

\[P_{SAML}(\boldsymbol{y}\,|\,\boldsymbol{x})=\prod_{t=1}^{T}\sum_{j=1}^{T^{ \prime}}a_{tj}\cdot p(y_{t}\,|\,\boldsymbol{\bar{y}}_{<j},\boldsymbol{x};\theta) \tag{6}\]

where \(T\) is the length of the reference sequence, \(T^{\prime}\) is the length of the sampled sequence, \(a_{tj}\) is the predicted soft alignment between the reference word \(y_{t}\) and sampled prefix \(\boldsymbol{\tilde{y}}_{<j}\).

Training with the SAML objective consists in maximizing:

\[\mathcal{J}_{SAML}(\theta)=\sum_{n=1}^{N}\log P_{SAML}(\boldsymbol{y}^{(n)} \,|\,\boldsymbol{x}^{(n)}) \tag{7}\]

The conditional probability of the next word \(p(y_{t}\,|\,\boldsymbol{\tilde{y}}_{<j},\boldsymbol{x};\theta)\) is computed as follows:

\[p(\cdot\,|\,\boldsymbol{\tilde{y}}_{<j},\boldsymbol{x};\theta)=\mathrm{softmax }(\boldsymbol{W}\boldsymbol{\tilde{h}}_{j}+\boldsymbol{b}) \tag{8}\]

where \(\boldsymbol{W}\) and \(\boldsymbol{b}\) are model parameters. \(\boldsymbol{\tilde{h}}_{j}\) is the hidden representation at step \(j\) conditioned on the source sequence \(\boldsymbol{x}\) and the preceding words \(\boldsymbol{\tilde{y}}_{<j}\) sampled from the model distribution using differentiable sampling:

\[\boldsymbol{\tilde{h}}_{j}=f(\boldsymbol{\tilde{y}}_{<j},\boldsymbol{x}) \tag{9}\]

We compute the soft alignment \(a_{tj}\) between \(y_{t}\) and \(\boldsymbol{\tilde{y}}_{<j}\) based on the model's hidden states:

\[a_{tj}=\frac{\exp(score(\boldsymbol{\tilde{h}}_{j},\boldsymbol{e}_{y_{t}}))}{ \sum_{i=1}^{T^{\prime}}\exp(score(\boldsymbol{\tilde{h}}_{i},\boldsymbol{e}_{ y_{t}}))} \tag{10}\]

where \(\boldsymbol{e}_{y_{t}}\) is the embedding of the reference word \(y_{t}\). The \(score\) function captures the similarity between the hidden state \(\boldsymbol{\tilde{h}}_{j}\) and the embedding \(\boldsymbol{e}_{y_{t}}\). We use the dot product here as it does not introduce additional parameters:

\[score(\boldsymbol{h},\boldsymbol{e})=\boldsymbol{h}^{\top}\boldsymbol{e} \tag{11}\]

Figure 1 illustrates how the resulting objective differs from scheduled sampling: (1) it is computed over sampled sequences as opposed to sequences that contain a mixture of sampled and reference words, and (2) each reference word is softaligned to the sampled sequence.

## 3 Experiments

DataWe evaluate our approach on IWSLT 2014 German-English (de-en) as prior work [16], as well as two additional tasks: IWSLT 2014 English-German (en-de) and IWSLT

\begin{table}
\begin{tabular}{l c c c c c} \hline \multirow{2}{*}{**Task**} & \multicolumn{3}{c}{sentences (K)} & vocab (K) \\ \cline{2-6}  & train & dev & test & src & tgt \\ \hline
**de-en** & 153.3 & 7.0 & 6.8 & 113.5 & 53.3 \\
**vi-en** & 121.3 & 1.5 & 1.3 & 23.9 & 50.0 \\ \hline \end{tabular}
\end{table}
Table 1: We evaluate on two translation tasks.

Figure 1: Difference between objectives used in scheduled sampling (left) and our approach (right), when computing the contribution to the objective of the reference word “dinner”. The schedule sampling hypothesis uses a mixture of the reference (black) and sampled (blue underlined) words, while the entire hypothesis sequence is sampled in our approach.

 

2015 Vietnamese-English (vi-en). For de-en and en-de, we follow the preprocessing steps in Ranzato et al. (2015). For vi-en, we use the data preprocessed by Luong and Manning (2015), with test2012 for validation and test2013 for testing. Table 1 summarizes the data statistics.

SetupOur translation models are attentional RNNs Bahdanau et al. (2015) built on Sockeye Hieber et al. (2017). We use bi-directional LSTM encoder and single-layer LSTM decoder with 256 hidden units, embeddings of size 256, and multilayer perceptron attention with a layer size of 256. We apply layer normalization Ba et al. (2016) and label smoothing (0.1). We add dropout to embeddings (0.1) and decoder hidden states (0.2). For ST Gumbel-Softmax, we use temperature \(\gamma=1\) and noise scale \(\beta=0.5\). The decoding beam size is 5 unless stated otherwise. We train the models using the Adam optimizer Kingma and Ba (2015) with a batch size of 1024 words. We checkpoint models every 1000 updates. The initial learning rate is 0.0002, and it is reduced by 30% after 4 checkpoints without validation perplexity improvement. Training stops after 12 checkpoints without improvement. For training efficiency, we first pre-train a baseline model for each task using only \({\cal J}_{ML}\) and fine-tune it using different approaches. In the fine-tuning phase, we inherit all settings except that we initialize the learning rate to 0.00002 and set the minimum number of checkpoints before early stopping to 24. We fine-tune each randomly seeded model independently.

BaselinesWe compare our model against three baselines: (1) a standard baseline trained with the ML objective, and models fine-tuned with (2) scheduled sampling **SS**Bengio et al. (2015) and (3) differentiable scheduled sampling **DSS**Goyal et al. (2017). In SS and DSS, the probability of using reference words \(\epsilon_{s}\) is annealed using inverse sigmoid decay Bengio et al. (2015): \(\epsilon_{s}=k/(k+\exp(i/k))\) at the \(i\)-th checkpoint with \(k=10\).

ResultsTable 2 shows that the SAML improves over the ML baseline by +0.5 BLEU on de-en, +0.7 BLEU on en-de, and +1.0 BLEU on vi-en task. In addition, SAML consistently improves over both the scheduled sampling and differentiable scheduled sampling on all tasks. All improvements are significant with \(p<0.002\). Interestingly, differentiable scheduled sampling performs no better than scheduled sampling in our experiments, unlike in Goyal et al. (2017).

Unlike scheduled sampling, our approach does not require an annealing schedule, and it is therefore simpler to train. We verify that the annealing schedule is needed in scheduled sampling by training a contrastive model with the same objective as scheduled sampling, but without annealing schedule (Table 2). We set the sampling rate to 0.5. The contrastive model hurts BLEU scores by at least 4.0 points compared to both the ML baseline and models fine-tuned with scheduled sampling, confirming that scheduled sampling needs the annealing schedule to work well.

We further examine the performance gain of different approaches over the baseline with varying beam sizes (Figure 2). Our approach yields larger BLEU improvements when decoding with greedy search and smaller beams, while there is no clear pattern for scheduled sampling models. These results support the hypothesis that our approach mitigates exposure bias, as it yields bigger improvements in settings where systems have fewer opportunities to recover from early errors.

## 4 Related Work

Daume et al. (2009) first addressed exposure bias in an imitation learning framework by training a classifier on examples generated using a mixture of the ground truth and the model's current predictions. DAgger Ross et al. (2011) is a similar algorithm which differs in how the training examples are generated and aggregated. Both al

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Method** & **Anneal** & **de-en** & **en-de** & **vi-en** \\ \hline Baseline & No & 27.41\({}_{\pm 0.28}\) & 22.64\({}_{\pm 0.13}\) & 23.59\({}_{\pm 0.13}\) \\ \hline +SS & Yes & 27.47\({}_{\pm 0.28}\) & 22.56\({}_{\pm 0.17}\) & 23.97\({}_{\pm 0.39}\) \\ +DSS & Yes & 27.30\({}_{\pm 0.24}\) & 22.47\({}_{\pm 0.20}\) & 23.68\({}_{\pm 0.35}\) \\ +SS & No & 22.91\({}_{\pm 0.21}\) & 17.78\({}_{\pm 0.20}\) & 19.57\({}_{\pm 0.19}\) \\ \hline +SAML & No & **27.94\({}_{\pm 0.12}\)** & **23.30\({}_{\pm 0.19}\)** & **24.60\({}_{\pm 0.35}\)** \\ \hline \hline \end{tabular}
\end{table}
Table 2: BLEU scores of our approach (SAML) and three baselines including the maximum likelihood (ML) baseline, scheduled sampling (SS), and differentiable scheduled sampling (DSS). The _Anneal_ column indicates whether the sampling rate is annealed. For each task, we report the mean and standard deviation over 5 runs with different random seeds. SAML achieves the best BLEU scores and is simpler to train than SS and DSS, as it requires no annealing schedule.

 

gorithms require an expert policy, which produces the best next token given any model predicted prefix, and assume that policy can be efficiently computed from the reference. However, for structured prediction tasks such as machine translation with large vocabulary and complex loss functions, it is intractable to find the best next token given any prefix. For time series modeling, the Data as Demonstrator algorithm (Venkatraman et al., 2015) derives the expert policy directly from the reference sequences which are aligned with the sampled sequences at each time step. Scheduled sampling algorithms (Bengio et al., 2015; Goyal et al., 2017) use the same strategy to train neural sequence-to-sequence models for a broader range of language generation tasks, even though the time alignment between reference and sampled sequences does not hold. Leblond et al. (2018) proposed to complete a predicted prefix with all possible reference suffix and picking the reference suffix that yields the highest BLEU-1 score. However, they found that this approach performs well only when the prefix is close to the reference.

Reinforcement learning (RL) algorithms Bahdanau et al. (2016); Sutton and Barto (2018); Van Hasselt et al. (2016) address exposure bias by directly optimizing a sentence-level reward for the model generated sequences. Evaluation metrics such as BLEU can be used as rewards, but they are discontinuous and hard to optimize. Techniques such as policy gradient Williams (1992) and actor-critic Sutton and Barto (2018); Degris et al. (2012) are thus required to find an unbiased estimation of the gradient to optimize the model. Due to the high variance of the gradient estimation, training with RL can be slow and unstable Henderson et al. (2018); Wu et al. (2018). Recent alternatives use data augmentation to incorporate the sentence-level reward into the training objective more efficiently Norouzi et al. (2016).

Finally, our SAML loss shares the idea of flexible reference word order with the bag-of-word loss introduced by Ma et al. (2018) to improve source coverage. However, their loss is computed with teacher forcing and therefore does not address exposure bias.

## 5 Conclusion

We introduced a differentiable sampling algorithm which exposes a sequence-to-sequence model to its own predictions during training and compares them to reference sequences flexibly to backpropagate reliable error signals. By soft aligning reference and sampled sequences, our approach consistently improves BLEU over maximum likelihood and scheduled sampling baselines on three IWSLT tasks, with larger improvements for greedy search and smaller beam sizes. Our approach is also simple to train, as it does not require any sampling schedule.

#### Acknowledgments

We thank the anonymous reviewers, Amr Sharaf, Naomi Feldman, Hal Daume III and the CLIP lab at UMD for helpful comments. This research is supported in part by an Amazon Web Services Machine Learning Research Award and by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via contract #FA8650-17-C-9117. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

Figure 2: Improvements from our method (SAML), scheduled sampling (SS), and differentiable scheduled sampling (DSS) over the maximum likelihood (ML) baseline when decoding with varying beam sizes (average of 5 runs). The SAML model consistently yields the largest improvements with smaller beams.

 



## References

* Ba et al. (2016) Lei Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. _CoRR_, abs/1607.06450.
* Bahdanau et al. (2016) Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. 2016. An actor-critic algorithm for sequence prediction. _CoRR_, abs/1607.07086.
* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _Proceedings of the 3th International Conference on Learning Representations_.
* Bengio et al. (2015) Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In _Advances in Neural Information Processing Systems 28_, pages 1171-1179.
* Bengio et al. (2013) Yoshua Bengio, Nicholas Leonard, and Aaron C. Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. _CoRR_, abs/1308.3432.
* Choi et al. (2018) Jihun Choi, Kang Min Yoo, and Sang-goo Lee. 2018. Learning to compose task-specific tree structures. In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence_, pages 5094-5101. AAAI Press.
* Cohen and Carvalho (2005) William W. Cohen and Victor R. Carvalho. 2005. Stacked Sequential Learning. In _Proceedings of the IJCAI_.
* Daume et al. (2009) Hal Daume, John Langford, and Daniel Marcu. 2009. Search-based structured prediction. _Machine Learning_, 75(3):297-325.
* Degris et al. (2012) Thomas Degris, Patrick M Pilarski, and Richard S Sutton. 2012. Model-free reinforcement learning with continuous action in practice. In _American Control Conference (ACC), 2012_, pages 2177-2182. IEEE.
* Goyal et al. (2017) Kartik Goyal, Chris Dyer, and Taylor Berg-Kirkpatrick. 2017. Differentiable scheduled sampling for credit assignment. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 366-371. Association for Computational Linguistics.
* Henderson et al. (2018) Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2018. Deep reinforcement learning that matters. In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence_. AAAI Press.
* Hieber et al. (2017) Felix Hieber, Tobias Domhan, Michael Denkowski, David Vilar, Artem Sokolov, Ann Clifton, and Matt Post. 2017. Sockeye: A toolkit for neural machine translation. _CoRR_, abs/1712.05690.
* Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with gumbel-softmax. In _Proceedings of the 5th International Conference on Learning Representations_.
* Kaariainen and Langford (2006) Matti Kaariainen and John Langford. 2006. Lower Bounds for Reductions. In _Talk at the Atomic Learning Workshop_.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _Proceedings of the 3th International Conference on Learning Representations_.
* Langford and Zadrozny (2005) John Langford and Bianca Zadrozny. 2005. Relating reinforcement learning performance to classification performance. In _Proceedings of the 22nd international conference on Machine learning_, pages 473-480. ACM.
* Leblond et al. (2018) Remi Leblond, Jean-Baptiste Alayrac, Anton Osokin, and Simon Lacoste-Julien. 2018. SEARNN: Training RNNs with global-local losses. In _International Conference on Learning Representations_.
* Luong and Manning (2015) Minh-Thang Luong and Christopher D Manning. 2015. Stanford neural machine translation systems for spoken language domains. In _Proceedings of the International Workshop on Spoken Language Translation_, pages 76-79.
* Ma et al. (2018) Shuming Ma, Xu Sun, Yizhong Wang, and Junyang Lin. 2018. Bag-of-words as target for neural machine translation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 332-338. Association for Computational Linguistics.
* Maddison et al. (2014) Chris J. Maddison, Daniel Tarlow, and Tom Minka. 2014. A* sampling. In _Advances in Neural Information Processing Systems 27_, pages 3086-3094. Curran Associates, Inc.
* Niu et al. (2019) Xing Niu, Weijia Xu, and Marine Carpuat. 2019. Bi-directional differentiable input reconstruction for low-resource neural machine translation. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics.
* Norouzi et al. (2016) Mohammad Norouzi, Samy Bengio, Zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, and Dale Schuurmans. 2016. Reward augmented maximum likelihood for neural structured prediction. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, _Advances in Neural Information Processing Systems 29_, pages 1723-1731. Curran Associates, Inc.
* Ranzato et al. (2015) Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. _CoRR_, abs/1511.06732

Stephane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In _Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics_, volume 15 of _Proceedings of Machine Learning Research_, pages 627-635. PMLR.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Advances in Neural Information Processing Systems 27_, pages 3104-3112.
* Sutton and Barto (2018) Richard S Sutton and Andrew G Barto. 2018. _Reinforcement learning: An introduction_. MIT press.
* Van Hasselt et al. (2016) Hado Van Hasselt, Arthur Guez, and David Silver. 2016. Deep reinforcement learning with double q-learning. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_, volume 2, page 5. AAAI Press.
* Venkatraman et al. (2015) Arun Venkataraman, Martial Hebert, and J. Andrew Bagnell. 2015. Improving multi-step prediction of learned time series models. In _Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence_, pages 3024-3030. AAAI Press.
* Williams (1992) Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. _Machine Learning_, 8(3):229-256.
* Wu et al. (2018) Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2018. A study of reinforcement learning for neural machine translation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3612-3621. Association for Computational Linguistics.