Venue: EMNLP 2017

# Towards Decoding as Continuous Optimisation in

Neural Machine Translation

Cong Duy Vu Hoang\({}^{\dagger}\) &Gholamreza Haffari\({}^{\ddagger}\) &Trevor Cohn\({}^{\dagger}\)

+ University of Melbourne, VIC, Australia vhoang2@student.unimelb.edu.au, gholamreza.haffari@monash.edu, t.cohn@unimelb.edu.au

###### Abstract

We propose a novel decoding approach for neural machine translation (NMT) based on continuous optimisation. We reformulate decoding, a discrete optimization problem, into a continuous problem, such that optimization can make use of efficient gradient-based techniques. Our powerful decoding framework allows for more accurate decoding for standard neural machine translation models, as well as enabling decoding in intractable models such as intersection of several different NMT models. Our empirical results show that our decoding framework is effective, and can leads to substantial improvements in translations, especially in situations where greedy search and beam search are not feasible. Finally, we show how the technique is highly competitive with, and complementary to, reranking.

## 1 Introduction

Sequence to sequence learning with neural networks Graves (2013); Sutskever et al. (2014); Lipton et al. (2015) is typically associated with two phases: training and decoding (_a.k.a._ inference). Model parameters are learned by optimising the training objective, in order that the model can produce good translations when decoding unseen sentences. The majority of research has focused on the training paradigm or network architecture, however effective means of decoding have been under-investigated. Conventional heuristic-based approaches for approximate inference include greedy, beam, and stochastic search. Greedy and beam search have been empirically proved to be adequate for many sequence to sequence tasks, and are the standard methods for NMT decoding.

However, these inference approaches have several drawbacks. Firstly, although NMT models use a left-to-right generation which would appear to facilitate efficient search, the models themselves use a recurrent architecture, and accordingly are non-Markov. This prevents exact dynamic programming solutions, and moreover, limits the potential to incorporate additional global features or constraints. Global factors can be highly useful in producing better and more diverse translations. Secondly, the sequential decoding of symbols in the target sequence, the inter-dependencies among the target symbols are not fully exploited. For example, when decoding the words of the target sentence in a left-to-right manner, the right context is not exploited leading potentially to inferior performance (see Watanabe and Sumita (2002) who apply this idea in traditional statistical MT). A natural way to capture this is to intersect left-to-right and right-to-left models, however the resulting model has no natural generation order, and thus standard decoding methods are unsuitable.

We introduce a novel decoding framework (SS 3) that relaxes this _discrete_ optimisation problem into a _continuous_ optimisation problem. This is akin to linear programming relaxation approach for approximate inference in graphical models with discrete random variables, where the exact inference is NP-hard Sontag (2010); Belanger and McCallum (2016). The resulting continuous optimisation problem is challenging due to the non-linearity and non-convexity of the relaxed decoding objective. We make use of stochastic gradient descent (SGD) and exponentiated gradient (EG) algorithms for decoding based on our relaxation approach.1 Our decoding framework is powerful and flexible, as it enables us to decode with global constraints involving intersection of multiple NMT

models (SS4). We present experimental results on Chinese-English and German-English translation tasks, confirming the effectiveness of our relaxed optimisation method for decoding (SS5).

## 2 Neural Machine Translation

We briefly review the attentional neural translation model proposed by Bahdanau et al. (2015) as a sequence-to-sequence neural model onto which we apply our decoding framework.

In neural machine translation (NMT), the probability of the target sentence given a source sentence is written as:

\[{\rm P}_{\mathbf{\Theta}}\left(\mathbf{y}|\mathbf{x}\right) = \sum_{i=1}^{|\mathbf{y}|}\log{\rm P}_{\mathbf{\Theta}}\left(y_{i}|\mathbf{y}_{< i},\mathbf{x}\right) \tag{1}\] \[y_{i}|\mathbf{y}_{<i},\mathbf{x} \sim {\rm softmax}\left(\mathbf{f}(\mathbf{\Theta},\mathbf{y}_{<i},\mathbf{x})\right)\]

where f is a non-linear function of the previously generated sequence of words \(\mathbf{y}_{<i}\), the source sentence \(\mathbf{x}\), and the model parameters \(\mathbf{\Theta}\). In this paper, we realise \(\mathbf{f}\) as follows:

\[\mathbf{f}(\mathbf{\Theta},\mathbf{y}_{<i},\mathbf{x}) =\mathbf{W}_{o}\cdot{\rm MLP}\left(\mathbf{c}_{i},\mathbf{E}_{T}^{y_{i-1}}, \mathbf{g}_{i}\right)+\mathbf{b}_{o}\] \[\mathbf{g}_{i} ={\rm RNN}_{dec}^{\phi}\left(\mathbf{c}_{i},\mathbf{E}_{T}^{y_{i-1}}, \mathbf{g}_{i-1}\right)\]

where \({\rm MLP}\) is a single hidden layer neural network with \({\rm tanh}\) activation function, and \(\mathbf{E}_{T}^{y_{i-1}}\) is the embedding of the target word \(y_{i-1}\) in the embedding matrix of the target language vocabulary \(V_{T}\) and \(n_{e}\) is the embedding dimension. The state \(\mathbf{g}_{i}\) of the decoder RNN is a function of \(y_{i-1}\), its previous state \(\mathbf{g}_{i-1}\), and the _context_\(\mathbf{c}_{i}=\sum_{j=1}^{|\mathbf{x}|}\alpha_{ij}\mathbf{h}_{j}\) summarises parts of the source sentence which are _attended_ to, where

\[\mathbf{\alpha}_{i} ={\rm softmax}(\mathbf{e}_{i})\quad;\quad e_{ij}={\rm MLP}\left(\mathbf{ g}_{i-1},\mathbf{h}_{j}\right)\] \[\mathbf{h}_{j} ={\rm biRN}_{enc}^{\theta}\left(\mathbf{E}_{S}^{x_{j}},\,\overleftarrow {\mathbf{h}}_{j-1},\,\overleftarrow{\mathbf{h}}_{j+1}\right)\]

In above, \(\overrightarrow{\mathbf{h}}_{i}\) and \(\overleftarrow{\mathbf{h}}_{i}\) are the states of the left-to-right and right-to-left RNNs encoding the source sentence, and \(\mathbf{E}_{S}^{x_{j}}\) is the embedding of the source word \(x_{j}\) in the embedding matrix \(\mathbf{E}_{S}\in\mathbb{R}^{n_{e}^{t}\times|V_{S}|}\) of the source language vocabulary \(V_{S}\) and \(n_{e}^{r}\) is the embedding dimension.

Given a bilingual corpus \(\mathcal{D}\), the model parameters are learned by maximizing the conditional log-likelihood,

\[\mathbf{\Theta}^{*}:={\rm argmax}_{\mathbf{\Theta}}\sum_{(\mathbf{x},\mathbf{y})\in\mathcal{ D}}\log{\rm P}_{\mathbf{\Theta}}\left(\mathbf{y}\mid\mathbf{x}\right). \tag{2}\]

The model parameters \(\mathbf{\Theta}\) include the weight matrix \(\mathbf{W}_{o}\in\mathbb{R}^{|V_{T}|\times n_{h}}\) and the bias \(\mathbf{b}_{o}\in\mathbb{R}^{|V_{T}|}\) - with \(n_{H}\) denoting the hidden dimension size - as well as the RNN encoder \({\rm biRNN}_{enc}^{\theta}\) / decoder \({\rm RNN}_{dec}^{\phi}\) parameters, word embedding matrices, and the parameters of the attention mechanism. The model is trained end-to-end by optimising the training objective using stochastic gradient descent (SGD) or its variants. In this paper, we focus on the decoding problem, which we turn to in the next section.

## 3 Decoding as Continuous Optimisation

In decoding, we are interested in finding the highest probability translation for a given source sentence:

\[{\rm minimise}_{\mathbf{y}}\quad-P_{\mathbf{\Theta}}\left(\mathbf{y}\mid\mathbf{ x}\right)\quad\text{ s.t.}\quad\mathbf{y}\in\mathcal{Y}_{\mathbf{x}} \tag{3}\]

where \(\mathcal{Y}_{\mathbf{x}}\) is the space of possible translations for the source sentence \(\mathbf{x}\). In general, searching \(\mathcal{Y}_{\mathbf{x}}\) to find the highest probability translation is intractable due to the recurrent nature of eqn (1) which prevents dynamic programming for efficient search. This is problematic, as the space of translations is exponentially large with respect to the output length \(|\mathbf{y}|\).

We now formulate this discrete optimisation problem as a continuous one, and then use standard algorithms for continuous optimisation for decoding. Let us assume that the maximum length of a possible translation for a source sentence is known and denote it as \(\ell\). The best translation for a given source sentence solves the following optimisation problem:

\[\mathbf{y}^{*}= \operatorname*{arg\,min}_{y_{1},\ldots,y_{\ell}}\sum_{i=1}^{\ell} -\log{\rm P}_{\mathbf{\Theta}}\left(y_{i}\mid\mathbf{y}_{<i},\mathbf{x}\right)\] (4) s.t. \[\forall i\in\{1\dots\ell\}:y_{i}\in V_{T}.\]

where we allow the translation to be padded with sentinel symbols to the right, which are ignored in computing the model probability. Equivalently, we can rewrite the above discrete optimisation problem as follows:

\[\operatorname*{arg\,min}_{\tilde{\mathbf{y}}_{1},\ldots,\tilde{\mathbf{ y}}_{\ell}}-\sum_{i=1}^{\ell}\tilde{\mathbf{y}}_{i}\cdot\log{\rm softmax}\left(\mathbf{f} \left(\mathbf{\Theta},\tilde{\mathbf{y}}_{<i},\mathbf{x}\right)\right)\] s.t. \[\forall i\in\{1\dots\ell\}:\tilde{\mathbf{y}}_{i}\in\mathbb{I}^{|V_{ T}|} \tag{5}\]

where \(\tilde{\mathbf{y}}_{i}\) are vectors using the one-hot representation of the target words 

We now convert the optimisation problem (5) to a continuous one by dropping the integrality constraints \(\tilde{\mathbf{y}}_{i}\in\mathbb{I}^{|V|}\) and require the variables to take values from the probability simplex:

\[\operatorname*{arg\,min}_{\hat{\mathbf{y}}_{1},\ldots,\hat{\mathbf{y}}_{\ell}}-\sum_{ i=1}^{\ell}\hat{\mathbf{y}}_{i}\cdot\log\mathrm{softmax}\left(\mathbf{f}\left(\mathbf{ \Theta},\hat{\mathbf{y}}_{<i},\mathbf{x}\right)\right)\] s.t. \[\forall i\in\{1\dots\ell\}:\hat{\mathbf{y}}_{i}\in\Delta_{|V_{T}|}\]

where \(\Delta_{|V_{T}|}\) is the \(|V_{T}|\)-dimensional probability simplex, i.e., \(\{\hat{\mathbf{y}}_{i}\in[0,1]^{|V_{T}|}:\lVert\hat{\mathbf{y}}_{i}\rVert_{1}=1\}\). Intuitively, this amounts to replacing \(\mathbf{E}_{T}^{y_{i}}\) with the _expected_ embedding of target language words \(\mathbb{E}_{\hat{\mathbf{y}}_{i}(w)}[\mathbf{E}_{T}^{w}]\) under the distribution \(\hat{\mathbf{y}}_{i}\).

After solving the above constrained continuous optimisation problem, there is no guarantee that the resulting solution \(\{\hat{\mathbf{y}}_{i}^{*}\}_{i=1}^{\ell}\) will comprise one-hot vectors, i.,e., target language words. Instead it can find _fractional_ solutions, that require 'rounding' in order to to resolve them to lexical items. To solve this problem, we take the \(\operatorname*{arg\,max}\),2 i.e., take the highest scoring word for each position \(\hat{\mathbf{y}}_{i}^{*}\). We leave exploration of more elaborate projection techniques to the future work.

Footnote 2: Ties are broken arbitrarily.

In the context of graphical models, the above relaxation technique gives rise to linear programming for approximate inference Sontag (2010); Belanger and McCallum (2016). However, our decoding problem is much harder due to the non-linearity and non-convexity of the objective function operating on high dimensional space for deep models. We now turn our attention to optimisation algorithms to effectively solve the decoding optimisation problem.

### Exponentiated Gradient (EG)

Exponentiated gradient Kivinen and Warmuth (1997) is an elegant algorithm for solving optimisation problems involving simplex constraints. Recall our constrained optimisation problem:

\[\operatorname*{arg\,min}_{\hat{\mathbf{y}}_{1},\ldots,\hat{\mathbf{y}}_{ \ell}}Q(\hat{\mathbf{y}}_{1},\ldots,\hat{\mathbf{y}}_{\ell})\] s.t. \[\forall i\in\{1\dots\ell\}:\hat{\mathbf{y}}_{i}\in\Delta_{|V_{T}|}\]

where \(Q(\hat{\mathbf{y}}_{1},\ldots,\hat{\mathbf{y}}_{\ell})\) is defined as

\[-\sum_{i=1}^{\ell}\hat{\mathbf{y}}_{i}\cdot\log\mathrm{softmax}\left( \mathbf{f}\left(\mathbf{\Theta},\hat{\mathbf{y}}_{<i},\mathbf{x}\right)\right). \tag{6}\]

EG is an iterative algorithm, which updates each distribution \(\hat{\mathbf{y}}_{i}^{t}\) in the current time-step \(t\) based on the distributions of the previous time-step as follows:

\[\forall w\in V_{T}:\quad\hat{\mathbf{y}}_{i}^{t}(w)=\frac{1}{Z_{i}^{ t}}\hat{\mathbf{y}}_{i}^{t-1}(w)\exp\left(-\eta\nabla_{i,w}^{t-1}\right)\]

where \(\eta\) is the step size, \(\nabla_{i,w}^{t-1}=\frac{\partial Q(\hat{\mathbf{y}}_{1}^{t-1}\ldots,\hat{\mathbf{y}}_ {\ell}^{t-1})}{\partial\hat{\mathbf{y}}_{i}(w)}\) and \(Z_{i}^{t}\) is the normalisation constant

\[Z_{i}^{t}=\sum_{w\in V_{T}}\hat{\mathbf{y}}_{i}^{t-1}(w)\exp\left(- \eta\nabla_{i,w}^{t-1}\right).\]

The partial derivatives \(\nabla_{i,w}\) are calculated using the back propagation algorithm treating \(\{\hat{\mathbf{y}}_{i}\}_{i=1}^{\ell}\) as _parameters_ and the original parameters of the model \(\mathbf{\Theta}\) as constants. Adapting EG to our decoding problem leads to Algorithm 1.

```
1:For all \(i\) initialise \(\hat{\mathbf{y}}_{i}^{0}\in\Delta_{|V_{T}|}\)
2:for\(t=1,\dots,\text{MaxIter}\)do\(\triangleright~{}Q(.)\) is defined as eqn (6)
3: For all \(i,w:\text{calculate }\nabla_{i,w}^{t-1}=\frac{\partial Q(\hat{\mathbf{y}}_{1}^{t-1}, \ldots,\hat{\mathbf{y}}_{\ell}^{t-1})}{\partial\hat{\mathbf{y}}_{i}(w)}\)\(\triangleright~{}\)using back-propagation
4: For all \(i,w:\text{update }\hat{\mathbf{y}}_{i}^{t}(w)\propto\hat{\mathbf{y}}_{i}^{t-1}(w)\cdot\exp \left(-\eta\nabla_{i,w}^{t-1}\right)\)\(\triangleright~{}\eta\)is the step size
5:return\(\operatorname*{arg\,min}_{t}Q(\hat{\mathbf{y}}_{1}^{t},\ldots,\hat{\mathbf{y}}_{\ell}^{t})\)
```

**Algorithm 1** The EG Algorithm for Decoding by Optimisation

It can be shown that the EG algorithm is a gradient descent algorithm for minimising the following objective function subject to the simplex constraints:

\[Q(\hat{\mathbf{y}}_{1},\ldots,\hat{\mathbf{y}}_{\ell})-\gamma\sum_{i=1}^{ \ell}\sum_{w\in V_{T}}\hat{\mathbf{y}}_{i}(w)\log\frac{1}{\hat{\mathbf{y}}_{i}(w)}\] \[=Q(\hat{\mathbf{y}}_{1},\ldots,\hat{\mathbf{y}}_{\ell})-\gamma\sum_{i=1}^ {\ell}\text{Entropy}(\hat{\mathbf{y}}_{i}) \tag{7}\]

In other words, the algorithm looks for the maximum entropy solution which also maximizes the 

log likelihood under the model. There are intriguing parallels with the maximum entropy formulation of log-linear models (Berger et al., 1996). In our setting, the entropy term acts as a prior which discourages overly-confident estimates in the absence of sufficient evidence.

### Stochastic Gradient Descent (SGD)

To be able to apply SGD to our optimisation problem, we need to make sure that the simplex constraints are enforced. One way to achieve this is by reparameterising using the \({\rm softmax}\) transformation, i.e. \(\hat{\mathbf{y}}_{i}={\rm softmax}\left(\hat{\mathbf{r}}_{i}\right)\). The resulting _unconstrained_ optimisation problem, now over \(\hat{\mathbf{r}}_{i}\), becomes

where \(\mathbf{E}^{\mathbf{w}_{i}}_{T}\) is replaced with the expected embedding of the target words under the distribution resulted from the \(\mathbb{E}_{{\rm softmax}\left(\hat{\mathbf{r}}_{i}\right)}\left[\mathbf{E}^{\mathbf{v}}_{ T}\right]\) in the model.

To apply SGD updates, we need the gradient of the objective function with respect to the new variables \(\hat{\mathbf{r}}_{i}\) which can be derived with the back-propagation algorithm based on the chain rule:

The resulting SGD algorithm is summarized in Algorithm2.

## 4 Decoding in Extended NMT

Our decoding framework allows us to effectively and flexibly add additional global factors over the output symbols during inference. This enables decoding for richer global models, for which there is no effective means of greedy decoding or beam search. We outline several such models, and their corresponding relaxed objective functions for optimisation-based decoding.

Bidirectional Ensemble.Standard NMT generates the translation in a left-to-right manner, conditioning each target word on its left context. However, the joint probability of the translation can be decomposed in a myriad of different orders; one compelling alternative would be to condition each target word on its right context, i.e., generating the target sentence from right-to-left. We would not expect a right-to-left model to outperform a left-to-right, however, as the left-to-right ordering reflects the natural temporal order of spoken language. However, the right-to-left model is likely to provide a complementary signal in translation, as it will be bringing different biases and making largely independent prediction errors to those of the left-to-right model. For this reason, we propose to use both models, and seek to find translations that have high probability according both models (this mirrors work on bidirectional decoding in classical statistical machine translation by Watanabe and Sumita (2002b).) Decoding under the ensemble of these models leads to an intractable search problem, not well suited to traditional greedy or beam search algorithms, which require a fixed generation order of the target words. This ensemble decoding problem can be formulated simply in our linear relaxation approach, using the following objective function:

\[\mathcal{C}_{+{\rm bidir}}:= -\alpha\log\mathrm{P}_{\mathbf{\Theta}_{\leftarrow}}\left(\mathbf{y}\mid \mathbf{x}\right)\] \[-(1-\alpha)\log\mathrm{P}_{\mathbf{\Theta}_{\leftarrow}}\left(\mathbf{ y}\mid\mathbf{x}\right); \tag{8}\]

where \(\alpha\) is an interpolation hyper-parameter, which we set to 0.5; \(\mathbf{\Theta}_{\rightarrow}\) and \(\mathbf{\Theta}_{\leftarrow}\) are the pre-trained left-to-right and right-to-left models, respectively. This bidirectional agreement may also lead to improvement in translation diversity, as shown in Li and Jurafsky (2016) in a re-ranking evaluation.

Bilingual Ensemble.Another source of complementary information is in terms of the translation direction, that is forward translation from the source to the target language, and reverse translation in the target to source direction. Decoding must find a translation which scores well under both the forward and reverse translation models. This is inspired by the direct and reverse feature functions commonly used in classical discriminative SMT Och and Ney (2002) which have been shown to offer some complementary benefits (although see Lopez and Resnik (2006)). More specifically, we decode for the best translation in the intersection of the source-to-target and target-to-source models by minimizing the following objective function:

\[\mathcal{C}_{+{\rm biling}}:= -\alpha\log\mathrm{P}_{\mathbf{\Theta}_{s\gets t}}\left(\mathbf{y} \mid\mathbf{x}\right)\] \[-(1-\alpha)\log\mathrm{P}_{\mathbf{\Theta}_{s\gets t}}\left(\mathbf{ x}\mid\mathbf{y}\right); \tag{9}\]

where \(\alpha\) is an interpolation hyper-parameter to be fine-tuned; and \(\mathbf{\Theta}_{s\rightarrow{t}}\) and \(\mathbf{\Theta}_{s\leftarrow{t}}\) are the pre-trained 

source-to-target and target-to-source models, respectively. Decoding for the best translation under the above objective function leads to an intractable search problem, as the reverse model is global over the target language, meaning there is no obvious means of search with a greedy algorithm or similar.

Discussion.There are two important considerations on how best to initialise the relaxed optimisation in the above settings, and how best to choose the step size. As the relaxed optimisation problem is, in general, non-convex, finding a plausible initialisation is likely to be important for avoiding local optima. Furthermore, a proper step size is a key in the success of the EG-based and SGD-based optimisation algorithms, and there is no obvious method how to best choose its value. We may also adaptively change the step size using (scheduled) annealing or via the line search. We return to this considerations in the experimental evaluation.

## 5 Experiments

### Setup

Datasets.We conducted our experiments on datasets with different scales, translating between Chinese\(\rightarrow\)English using the BTEC corpus, and German\(\rightarrow\)English using the IWSLT 2015 TED TalksCettolo et al. (2014) and WMT 20163 corpora. The statistics of the datasets can be found in Table 1.

Footnote 3: [http://www.statmt.org/wmt16/translation-task.html](http://www.statmt.org/wmt16/translation-task.html)

NMT Models.We implemented our continuous-optimisation based decoding method on top of the Mantidae toolkit4 Cohn et al. (2016), and using the _dynet_ deep learning library5Neubig et al. (2017). All neural network models were configured with 512 input embedding and hidden layer dimensions, and 256 alignment dimension, with 1 and 2 hidden layers in the source and target, respectively. We used a LSTM recurrent structure Hochreiter and Schmidhuber (1997) for both source and target RNN sequences. For the vocabulary, we use word frequency cut-off of 5, and words rarer than this were mapped to a sentinel. For the large-scale WMT dataset, we applied byte-pair encoding (BPE) method Sennrich et al. (2016) to better handle unknown words.6 For training our neural models, we use early stopping based on development perplexity, which usually occurs after 5-8 epochs.

Evaluation Metrics.We evaluated in terms of search error, measured using the model score of the inferred solution (either continuous or discrete), as well as measuring the end translation quality with case-insensitive BLEU Papineni et al. (2002). The continuous cost measures \(-\frac{1}{|\boldsymbol{y}|}\log P_{\boldsymbol{\Theta}}\left(\hat{\boldsymbol {y}}\mid\boldsymbol{x}\right)\) under the model \(\boldsymbol{\Theta}\); the discrete model score has the same formulation, albeit using the discrete rounded solution \(\boldsymbol{y}\) (see SS3). Note the cost can be used as a tool for selecting the best inference solution, as well as assessing convergence, as we illustrate below.

\begin{table}
\begin{tabular}{|l||c|c|c|} \hline  & **\# tokens** & **\# types** & **\# sents** \\ \hline \hline \multicolumn{4}{|c|}{BTEC 2h\(\leftarrow\)en} \\ \hline train & 422k / 454k & 3k / 3k & 44,016 \\ dev & 10k / 10k & 1k / 1k & 1,006 \\ test & 5k / 5k & 1k / 1k & 506 \\ \hline \multicolumn{4}{|c|}{TED Takls de–en} \\ \hline train & 4m / 4m & 26k / 19k & 194,181 \\ dev-test2010 & 33k / 35k & 4k / 3k & 1,565 \\ test2014 & 26k / 27k & 4k / 3k & 1,305 \\ \hline \multicolumn{4}{|c|}{WMT 2016 de–en} \\ \hline train & 107m / 108m & 90k / 78k & 4m \\ dev-test2013k14 & 154k / 152k & 20k / 13k & 6003 \\ test2015 & 54k / 54k & 10k / 8k & 2169 \\ \hline \end{tabular}
\end{table}
Table 1: Statistics of the training and evaluation sets; token and types are presented for both source/target languages.

 



### Results and Analysis

Initialisation and Step Size.As our relaxed optimisation problems are non-convex, local optima are likely to be a problem. We test this empirically, focusing on the effect that initialisation and step size, \(\eta\), have on the inference quality.

For plausible initialisation states, we evaluate different strategies: _uniform_ in which the relaxed variables \(\hat{\mathbf{y}}\) are initialised to \(\frac{1}{|V_{T}|}\); and _greedy_ or _beam_ whereby \(\hat{\mathbf{y}}\) are initialised based on an already good solution produced by a baseline decoder with greedy (gdec) or beam (bdec). Instead of using the Viterbi outputs as a one-hot representation, we initialise to the probability prediction vectors,7 which serves to limit attraction of the initialisation condition, which is likely to be a local (but not global) optima.

Footnote 7: Here, EG uses \(\mathrm{softmax}\) normalization whereas SGD uses the pre–\(\mathrm{softmax}\) vector.

Figure 1 illustrates the effect of initialisation on the EG algorithm, in terms of search error (left and middle) and translation quality (right), as we vary the number of iterations of inference. There is clear evidence of non-convexity: all initialisation methods can be seen to converge using all three measures, however they arrive at highly different solutions. Uniform initialisation is clearly not a viable approach, while greedy and beam initialisation both yield much better results. The best initialisation, beam, outperforms both greedy and beam decoding in terms of BLEU.

Note that the EG algorithm has fairly slow convergence, requiring at least 100 iterations, irrespective of the initialisation. To overcome this, we use momentum Qian (1999) to accelerate the convergence by modifying the term \(\nabla^{t}_{i,w}\) in Algorithm 1 with a weighted moving average of past gradients:

\[\nabla^{t-1}_{i,w}=\gamma\nabla^{t-2}_{i,w}+\eta\frac{\partial Q(\hat{\mathbf{y}} ^{t-1}_{1},\dots,\hat{\mathbf{y}}^{t-1}_{\ell})}{\partial\hat{\mathbf{y}}_{i}(w)}\]

where we set the momentum term \(\gamma=0.9\). The EG with momentum **(EG-MOM)** converges after fewer iterations (about 35), and results in marginally better BLEU scores. The momentum technique is usually used for SGD involving additive updates; it is interesting to see it also works in EG with multiplicative updates.

The step size, \(\eta\), is another important hyperparameter for gradient based search. We tune the step size using line search over \([10,400]\) over the development set. Figure 1 illustrates the effect of changing step size from 50 to 400 (compare **EG** and **EG-400** with **uniform**), which results in a marked difference of about 10 BLEU points, underlining the importance of tuning this value. We found that EG with momentum had less of a reliance on step size, with optimal values in \([10,50]\); we use this setting hereafter.

Continuous vs Discrete Costs.Another important question is whether the assumption behind continuous relaxation is valid, i.e., if we optimise a continuous cost to solve a discrete problem, do we improve the discrete output? Although the continuous cost diminishes with inference iterations (Figure 1 left), and appears to converge, it is not clear whether this corresponds to a better discrete output (note that the discrete cost and BLEU

Figure 1: Analysis on effects of initialisation states (uniform vs. greedy vs. beam), step size annealing, momentum mechanism from BTEC zh\(\rightarrow\)en translation. **EG-400**: EG algorithm with step size \(\eta=400\) (otherwise \(\eta=50\)); **EG-MOM**: EG algorithm with momentum.



scores do show improvements Figure 1 centre and right.) Figure 2 illustrates the relation between the two cost measures, showing that in most cases the discrete and continuous costs are identical. Linear relaxation fails only for a handful of cases, where the nearest discrete solution is significantly worse than it would appear using the continuous cost.

EG vs SGD.Both the EG and SGD algorithms are iterative methods for solving the relaxed optimisation problem with simplex constraints. We measure empirically their difference in terms of quality of inference and speed of convergence, as illustrated in Figure 3. Observe that SGD requires 150 iterations for convergence, whereas EG requires many fewer (50). This concurs with previous work on learning structured prediction models with EG Globerson et al. (2007). Further, the EG algorithm consistently produces better results in terms of both model cost and BLEU.

EG vs Reranking.Reranking is an alternative method for integrating global factors into the existing NMT systems. We compare our EG decoding algorithm against the reranking approach with bidirectional factor where the N-best outputs of a left-to-right decoder is re-scored with the forced decoder operating in a right-to-left fashion. The results are shown in Table 2. Our EG algorithm initialised with the reranked output achieves the best BLEU score. We also compare reranking with EG algorithm initialised with the beam decoder, where for direct comparison we filter out sentences with length greater than that of the beam output in the k-best lists. These results show that the EG algorithm is capable of effectively exploiting the search space.

Beyond achieving similar or better translations to re-ranking, note that EG is simpler in implementation, as it does not require \(k\)best lists, weight tuning and so forth. Instead this is replaced with iterative gradient descent. The run-time of the two methods are comparable, when reranking uses modest \(k\), however EG can be considerably faster when \(k\) is large, as is typically done to extract the full benefit from re-ranking. This performance difference is a consequence of GPU acceleration of the dense vector operations in EG inference.

Computational Efficiency.We also quantify the computational efficiency of the proposed decoding approach. Benchmarking on a GPU Titan X for decoding BTEC zh\({}^{-}\)en, the average time per sentence is 0.02 secs for greedy, 0.07s for beam=5, 0.11s for beam=10, and 3.1s for relaxed EG decoding, which uses an average of 35 EG iterations. The majority of time in the EG algorithm is in the forward and backward passes, taking 30% and 67% of the time, respectively. Our imple

Figure 3: Analysis on convergence and performance comparing SOFTMAX and EG algorithms from BTEC zh\({}^{-}\)en translation. Both algorithms use momentum and step size 50.

\begin{table}
\begin{tabular}{l c c} \hline \hline  & BLEU & AvgLen \\ \hline bdec\_left-to-right & \(26.69\) & \(20.73\) \\ filtered rerank & \(26.84\) & \(20.66\) \\ EGdec w/ beam init & \(27.34\) & \(20.73\) \\ \hline full rerank & \(27.34\) & \(21.76\) \\ EGdec w/ rerank init & \(27.78\) & \(21.70\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: The BLEU evaluation results with EG algorithm against 100-best reranking on WMT evaluation dataset.

Figure 2: Comparing discrete vs continuous costs from BTEC zh\({}^{-}\)en translation, using the EG algorithm with momentum, \(\eta=50\). Each point corresponds to a sentence.

 

mentation was not optimised thoroughly, and it is likely that it could be made significantly faster, which we defer to future research.

Main Results.Table 3 shows our experimental results across all datasets, evaluating the EG algorithm and its variants.8 For the EG algorithm with greedy initialisation (top), we see small but consistent improvements in terms of BLEU. Beam initialisation led to overall higher BLEU scores, and again demonstrating a similar pattern of improvements, albeit of a lower magnitude, over the initialisation values.

Footnote 8: Due to the space constraints, we report results for the EG algorithm only. See also translation examples in Figure 4.

Next we evaluate the capability of our inference method with extended NMT models, where approximate algorithms such as greedy or beam search are infeasible. With the **bidirectional** ensemble, we obtained the statistically significant BLEU score improvements compared to the unidirectional models, for either greedy or beam initialisation. This is interesting in the sense that the unidirectional right-to-left model always performs worse than the left-to-right model. However, our method with bidirectional ensemble is capable of combining their strengths in a unified setting. For the **bilingual** ensemble, we see similar effects, with better BLEU score improvements in most cases, albeit of a lower magnitude, over the bidirectional one. This is likely to be due to a disparity with the training condition for the models, which were learned independently of one another.

Overall, decoding in extended NMT models leads to performance improvements compared to baseline methods. This is one of the main findings in this work, and augurs well for its extension to other global model variants.

## 6 Related Work

Decoding (inference) for neural models is an important task; however, there is limited research in this space perhaps due to the challenging nature of this task, with only a few works exploring some extensions to improve upon them. The most widely-used inference methods include sampling Cho (2016), greedy and beam search Sutskever et al. (2014); Bahdanau et al. (2015), _inter alia_), and reranking Birch (2016); Li and Jurafsky (2016).

Cho (2016) proposed to perturb the neural model by injecting noise(s) in the hidden transition function of the conditional recurrent neural language model during greedy or beam search, and execute multiple parallel decoding runs. This strategy can improves over greedy and beam search; however, it is not clear how, when and where noise should be injected to be beneficial. Recently, Wiseman and Rush (2016) proposed beam search optimisation while _training_ neural models, where the model parameters are updated in case the gold standard falls outside of the beam. This exposes the model to its past incorrect predicted labels, hence making the training more robust. This is orthogonal to our approach where we focus on the decoding problem with a pre-trained model.

Reranking has also been proposed as a means of global model combination: Birch (2016) and Li and Jurafsky (2016) re-rank the left-to-right decoded translations based on the scores of a right-to-left model, learning to more diverse translations. Related, Li et al. (2016) learn to adjust the beam diversity with reinforcement learning.

Perhaps most relevant is Snelleman (2016), performed concurrently to this work, who also proposed an inference method for NMT using linear relaxation. Snelleman's method was similar to our SGD approach, however he did not manage to outperform beam search baselines with an encoder-decoder. In contrast we go much further, proposing the EG algorithm, which we show works much more effectively than SGD, and demonstrate how this can be applied to inference in an attentional

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & BTEC & TEDTalks & WMT \\  & zh \(\rightarrow\) en & de \(\rightarrow\) en & de \(\rightarrow\) en \\ \hline gdec\({}_{\text{left-to-right}}\) & \(35.98\) & \(23.16\) & \(24.41\) \\ gdec\({}_{\text{right-to-left}}\) & \(35.86\) & \(21.95\) & \(23.59\) \\ EGGecgreedy init & \(36.34\) & \(23.28\) & \(24.63\) \\ +bidir. & **36.67** & **23.91** & **25.37\({}^{\dagger}\)** \\ +bilingual & **36.88\({}^{\ddagger}\)** & **24.01\({}^{\dagger}\)** & **25.21** \\ \hline bdec\({}_{\text{left-to-right}}\) & \(38.02\) & \(23.95\) & \(26.69\) \\ bdec\({}_{\text{right-to-left}}\) & \(37.38\) & \(23.13\) & \(26.11\) \\ EGGecbeam init & \(38.38\) & \(24.02\) & \(26.66\) \\ +bidir. & **39.13\({}^{\dagger}\)** & **24.72\({}^{\dagger}\)** & **27.34\({}^{\dagger}\)** \\ +bilingual & \(38.25\) & **24.60** & \(26.82\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: The BLEU evaluation results across evaluation datasets for EG algorithm variants against the baselines; **bold**: statistically significantly better than the best greedy or beam baseline, \({}^{\dagger}\): best performance on dataset.

 

encoder-decoder. Moreover, we demonstrate the utility of related optimisation for inference over global ensembles of models, resulting in consistent improvements in search error and end translation quality.

Recently, relaxation techniques have been applied to deep models for training and inference in text classification (Belanger and McCallum, 2016; Belanger et al., 2017), and fully differentiable training of sequence-to-sequence models with scheduled-sampling (Goyal et al., 2017). Our work has applied the relaxation technique specifically for _decoding_ in NMT models.

## 7 Conclusions

This work presents the first attempt in formulating decoding in NMT as a continuous optimisation problem. The core idea is to drop the integrality (i.e. one-hot vector) constraint from the prediction variables and allow them to have soft assignments within the probability simplex while minimising the loss function produced by the neural model. We have provided two optimisation algorithms - exponentiated gradient (EG) and stochastic gradient descent (SGD) - for optimising the resulting contained optimisation problem, where our findings show the effectiveness of EG compared to SGD. Thanks to our framework, we have been able to decode when intersecting left-to-right and right-to-left as well as source-to-target and target-to-source NMT models. Our results show that our decoding framework is effective and leads to substantial improvements in translations generated from the intersected models, where the typical greedy or beam search algorithms are not applicable.

This work raises several compelling possibilities which we intend to address in future work, such as improving decoding speed, integrating additional constraints such as word coverage and fertility into _decoding_,9 and applying our method to other intractable structured prediction problems.

Figure 4: Translation examples generated by the models. \({}^{*}\): reranking with bidirectional (+bidir.) and bilingual (+biling.) produced the same translation string.

 



## Acknowledgments

We thank the reviewers for valuable feedbacks and discussions. Cong Duy Vu Hoang is supported by Australian Government Research Training Program Scholarships at the University of Melbourne, Australia. This work was supported by the Australian Research Council through their Discovery and Future Fellowship programmes.

## References

* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In _Proc. of 3rd International Conference on Learning Representations (ICLR2015)_.
* Volume 48_, ICML'17.
* Volume 48_, ICML'16, pages 983-992. JMLR.org.
* Berger et al. (1996) Adam L. Berger, Vincent J. Della Pietra, and Stephen A. Della Pietra. 1996. A maximum entropy approach to natural language processing. _Comput. Linguist._, 22(1):39-71.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow; Alexandra Birch. 2016. Edinburgh Neural Machine Translation Systems for WMT 16. In _Proceedings of the First Conference on Machine Translation, Volume 2: Shared Task Papers. Berlin, Germany_.
* Cettolo et al. (2014) M. Cettolo, J. Niehues, S. Stuker, L. Bentivogli, and M. Federico. 2014. Report on the 11th IWSLT Evaluation Campaign. In _Proc. of The International Workshop on Spoken Language Translation (IWSLT)_.
* Cho (2016) K. Cho. 2016. Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model. _ArXiv e-prints_.
* Cohn et al. (2016) Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vymolova, Kaisheng Yao, Chris Dyer, and Gholamreza Haffari. 2016. Incorporating Structural Alignment Biases into an Attentional Neural TRindal Model. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 876-885, San Diego, California. Association for Computational Linguistics.
* Globerson et al. (2007) Amir Globerson, Terry Y. Koo, Xavier Carreras, and Michael Collins. 2007. Exponentiated Gradient Algorithms for Log-linear Structured Prediction. In _Proceedings of the 24th International Conference on Machine Learning_, ICML '07, pages 305-312, New York, NY, USA. ACM.
* Goyal et al. (2017) Kartik Goyal, Chris Dyer, and Taylor Berg-Kirkpatrick. 2017. Differentiable scheduled sampling for credit assignment. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, ACL'17, Vancouver, Canada.
* Graves (2013) A. Graves. 2013. Generating Sequences With Recurrent Neural Networks. _ArXiv e-prints_.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long Short-Term Memory. _Neural Comput._, 9(8):1735-1780.
* Kivinen and Warmuth (1997) Jyrki Kivinen and Manfred K. Warmuth. 1997. Exponentiated Gradient Versus Gradient Descent for Linear Predictors. _Inf. Comput._, 132(1):1-63.
* Li and Jurafsky (2016) J. Li and D. Jurafsky. 2016. Mutual Information and Diverse Decoding Improve Neural Machine Translation. _ArXiv e-prints_.
* Li et al. (2016) J. Li, W. Monroe, and D. Jurafsky. 2016. A Simple, Fast Diverse Decoding Algorithm for Neural Generation. _ArXiv e-prints_.
* Lipton et al. (2015) Z. C. Lipton, J. Berkowitz, and C. Elkan. 2015. A Critical Review of Recurrent Neural Networks for Sequence Learning. _ArXiv e-prints_.
* Lopez and Resnik (2006) Adam Lopez and Philip Resnik. 2006. Word-based alignment, phrase-based translation: Whats the link. In _Proceedings of 7th Biennial Conference of the Association for Machine Translation in the Americas (AMTA)_.
* Mi et al. (2016) Haitao Mi, Baskaran Sankaran, Zhiguo Wang, and Abe Ittycheriah. 2016. Coverage Embedding Models for Neural Machine Translation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 955-960, Austin, Texas. Association for Computational Linguistics.
* Neubig et al. (2017) G. Neubig, C. Dyer, Y. Goldberg, A. Matthews, W. Ammar, A. Anastasopoulos, M. Ballesteros, D. Chiang, D. Clothiaux, T. Cohn, K. Duh, M. Faruqui, C. Gan, D. Garrette, Y. Ji, L. Kong, A. Kulcoro, G. Kumar, C. Malaviya, P. Michel, Y. Oda, M. Richardson, N. Saphra, S. Swayamdipta, and P. Yin. 2017. DyNet: The Dynamic Neural Network Toolkit. _ArXiv e-prints_.
* Och and Ney (2002) Franz Josef Och and Hermann Ney. 2002. Discriminative training and maximum entropy models for statistical machine translation. In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_, pages 295-302.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In _Proceedings

of the 40th Annual Meeting on Association for Computational Linguistics_, ACL '02, pages 311-318, Stroudsburg, PA, USA. Association for Computational Linguistics.

* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
* Snelleman (2016) Emanuel Snelleman. 2016. Decoding neural machine translation using gradient descent. Master's thesis, Chalmers University of Technology, Gothenburg, Sweden.
* Sontag (2010) David Sontag. 2010. _Approximate Inference in Graphical Models using LP Relaxations_. Ph.D. thesis, Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In _Proceedings of the 27th International Conference on Neural Information Processing Systems_, NIPS'14, pages 3104-3112, Cambridge, MA, USA. MIT Press.
* Volume 1_, COLING '02, pages 1-7, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Watanabe and Sumita (2002b) Taro Watanabe and Eiichiro Sumita. 2002b. Bidirectional decoding for statistical machine translation. In _Proceedings of the 19th international conference on Computational linguistics-Volume 1_, pages 1-7.
* Wiseman and Rush (2016) Sam Wiseman and Alexander M. Rush. 2016. Sequence-to-Sequence Learning as Beam-Search Optimization. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1296-1306, Austin, Texas. Association for Computational Linguistics.