Venue: EMNLP 2017

# Neural Lattice-to-Sequence Models for Uncertain Inputs

Matthias Sperber\({}^{1}\), Graham Neubig\({}^{2}\), Jan Niehues\({}^{1}\), Alex Waibel\({}^{1}\)

\({}^{1}\)Karlsruhe Institute of Technology, Germany

\({}^{2}\)Carnegie Mellon University, USA matthias.sperber@kit.edu

###### Abstract

The input to a neural sequence-to-sequence model is often determined by an up-stream system, e.g. a word segmenter, part of speech tagger, or speech recognizer. These up-stream models are potentially error-prone. Representing inputs through word lattices allows making this uncertainty explicit by capturing alternative sequences and their posterior probabilities in a compact form. In this work, we extend the TreeLSTM [30] into a LatticeLSTM that is able to consume word lattices, and can be used as encoder in an attentional encoder-decoder model. We integrate lattice posterior scores into this architecture by extending the TreeLSTM's child-sum and forget gates and introducing a bias term into the attention mechanism. We experiment with speech translation lattices and report consistent improvements over baselines that translate either the 1-best hypothesis or the lattice without posterior scores.

## 1 Introduction

In many natural language processing tasks, we will require a down-stream system to consume the input of an up-stream system, such as word segmenters, part of speech taggers, or automatic speech recognizers. Among these, one of the most prototypical and widely used examples is speech translation, where a down-stream translation system must consume the output of an up-stream automatic speech recognition (ASR) system.

Previous research on traditional phrase-based or tree-based statistical machine translation have used word lattices (e.g. Figure 1) as an effective tool to pass on uncertainties from a previous step [48, 56]. Several works have shown quality improvements by translating lattices, compared to translating only the single best upstream output. Examples include translating lattice representations of ASR output [7, 9, 10], multiple word segmentations, and morphological alternatives [47].

Recently, neural sequence-to-sequence (seq2seq) models [16, 15, 14] have often been preferred over the traditional methods for their strong empirical results and appealing end-to-end modeling. These models force us to rethink approaches to handling lattices, because their recurrent design no longer allows for efficient lattice decoding using dynamic programming as was used in the earlier works.

As a remedy, su2017deep proposed replacing the sequential encoder by a lattice encoder to obtain a lattice-to-sequence (lat2seq) model. This is achieved by extending the encoder's Gated Recurrent Units (GRUs) [12] to be conditioned on multiple predecessor paths. The authors demonstrate improvements in Chinese-to-English translation by translating lattices that combine the output of multiple word segmenters, rather than a single segmented sequence.

Figure 1: A lattice with 3 possible paths and posterior scores. Translating the whole lattice potentially allows for recovering from errors in its 1-best hypothesis.



However, this model does not address one aspect of lattices that we argue is critical to obtaining good translation results: their ability to encode the certainty or uncertainty of the paths through the use of posterior scores. Specifically, we postulate that these scores are essential for tasks that require handling lattices with a considerable amount of erroneous content, such as those produced by ASR systems. In this paper, we propose a lattice-to-sequence model that accounts for this uncertainty. Specifically, our contributions are as follows:

* We employ the popular child-sum TreeLSTM (Tai et al., 2015) to derive a lattice encoder that replaces the sequential encoder in an attentional encoder-decoder model. We show empirically that this approach yields only minor improvements compared to a baseline fine-tuned on sequential ASR outputs. This finding stands in contrast to the positive results by Su et al. (2017), and by Ladhak et al. (2016) on a lattice classification task, and suggests higher learning complexity of our speech translation task.
* We hypothesize that lattice scores are crucial in aiding training and inference, and propose several techniques for integrating lattice scores into the model: (1) We compute _weighted child-sums_,1 where hidden units in the lattice encoder are conditioned on their predecessor hidden units such that predecessors with low probability are less influential on the current hidden state. (2) We bias the TreeLSTM's _forget gates_ for each incoming connection toward being more forgetful for predecessors with low probability, such that their cell states become relatively less influential. (3) We _bias the attention mechanism_ to put more focus on source embeddings belonging to nodes with high lattice scores. We demonstrate empirically that the third proposed technique is particularly effective and produces strong gains over the baseline. According to our knowledge, this is the first attempt of integrating lattice scores already at the _training stage_ of a machine translation model. Footnote 1: This is reminiscent of the weighted pooling strategy by Ladhak et al. (2016) for spoken utterance classification.
* We exploit the fact that our lattice encoder is a strict generalization of a sequential encoder by _pre-training_ on sequential data, obtaining faster and better training convergence on large corpora of parallel sequential data.

We conduct experiments on the Fisher and Callhome Spanish-English Speech Translation Corpus (Post et al., 2013) and report improvements of 1.4 BLEU points on Fisher and 0.8 BLEU points on Callhome, compared to a strong baseline optimized for translating 1-best ASR outputs. We find that the proposed integration of lattice scores is crucial for achieving these improvements.

## 2 Background

Our work extends the seminal work on attentional encoder-decoder models (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015) which we survey in this section.

Given an input sequence \({\mathbf{x}}=(x_{1},x_{2},\ldots,x_{N})\), the goal is to generate an appropriate output sequence \({\mathbf{y}}=(y_{1},y_{2},\ldots,y_{M})\). The conditional probability \(p({\mathbf{y}}~{}|~{}{\mathbf{x}})\) is estimated using parameters trained on a parallel corpus, e.g. of sentences in the source and target language in a translation task. This probability is factorized as the product of conditional probabilities of each token to be generated: \(p({\mathbf{y}}~{}|~{}{\mathbf{x}})=\prod_{t=1}^{M}p(y_{t}~{}|~{}{\mathbf{y}}_{<t},{\mathbf{x}})\). The training objective is to estimate parameters \(\theta\) that maximize the log-likelihood of the sentence pairs in a given parallel training set \(D\): \(J(\theta)=\sum_{({\mathbf{x}},{\mathbf{y}})\in D}\log p({\mathbf{y}}~{}|~{}{\mathbf{x}};\theta)\).

### Encoder

In our baseline model, the encoder is a bi-directional recurrent neural network (RNN), following (Bahdanau et al., 2015). Here, the source sentence is processed in both the forward and backward directions with two separate RNNs. For every input \(x_{i}\), two hidden states are generated as

\[\overrightarrow{{\bf h}}_{i} ={\rm LSTM}\left(E_{\textit{fwd}}(x_{i}),\overrightarrow{{\bf h }}_{i-1}\right) \tag{1}\] \[\overleftarrow{{\bf h}}_{i} ={\rm LSTM}\left(E_{\textit{fwd}}(x_{i}),\overleftarrow{{\bf h }}_{i+1}\right), \tag{2}\]

where \(E_{\textit{fwd}}\) and \(E_{\textit{fwd}}\) are source embedding lookup tables. We opt for long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) recurrent units because of their high performance and in order to later take advantage of the Tree-LSTM extension (Tai et al., 2015). We stack multiple LSTM layers and concatenate the final layer into the final source hidden state \({\bf h}_{i}=\overrightarrow{{\bf h}}_{i}~{}|~{}\overleftarrow{{\bf h}}_{i}\), where layer indices are omitted for simplicity.

### Attention

We use an attention mechanism Luong et al. (2015) to summarize the encoder outputs into a 

fixed-size representation. At each decoding time step \(j\), a context vector \({\bf c}_{j}\) is computed as a weighted average of the source hidden states: \({\bf c}_{j}=\sum_{i=1}^{N}\alpha_{ij}{\bf h}_{i}\). The normalized attentional weights \(\alpha_{ij}\) measure the relative importance of the source words for the current decoding step and are computed as a softmax with normalization factor \(Z\) summing over \(i\):

\[\alpha_{ij}=\frac{1}{Z}\exp\big{(}\operatorname*{s}\left({\bf s}_{j-1},{\bf h }_{i}\right)\big{)} \tag{3}\]

\(\operatorname*{s}(\cdot)\) is a feed-forward neural network with a single layer that estimates the importance of source hidden state \({\bf h}_{i}\) for producing the next target symbol \(y_{j}\), conditioned on the previous decoder state \({\bf s}_{j-1}\).

### Decoder

The decoder creates output symbols one by one, conditioned on the encoder states via the attention mechanism. It contains another LSTM, initialized using the final encoder hidden state: \({\bf s}_{0}={\bf h}_{N}\). The decoding at step \(j\) assumes a special start-of-sequence symbol \(y_{0}\) and is computed as \({\bf s}_{j}=\mathrm{LSTM}\left(E_{\textit{rg}}(y_{j-1}),{\bf s}_{j-1}\right)\), and then \(\tilde{\mathbf{s}}_{t}=\tanh(W_{\textit{ks}}[{\bf s}_{j};{\bf c}_{j}]+{\bf b} _{\textit{hs}})\) The conditional probability that the \(j\)-th target word is generated is: \(p(y_{j}\mid{\boldsymbol{v}}_{<j},{\boldsymbol{x}})=\mathrm{softmax}(W_{ \textit{so}}\tilde{\mathbf{s}}_{t}+{\bf b}_{\textit{so}})\). Here, \(E_{\textit{rg}}\) is the target embedding lookup table, \(W_{\textit{ks}}\) and \(W_{\textit{so}}\) are weight matrices, and \({\bf b}_{\textit{ts}}\) and \({\bf b}_{\textit{so}}\) are bias vectors.

During decoding beam search is used to find an output sequence with high conditional probability.

## 3 Attentional Lattice-to-Sequence Model

The seq2seq model described above assumes sequential inputs and is therefore limited to taking a single output of an up-stream model as input. Instead, we wish to consume lattices to carry over uncertainties from an up-stream model.

### Lattices

Lattices (e.g. Figure 1) represent multiple ambiguous or competing sequences in a compact form. They are a more efficient alternative to enumerating all hypotheses as an \(n\)-best list, as they allow for avoiding redundant computation over subsequences shared between multiple hypotheses. Lattices can either be produced directly, e.g. by an ASR dumping its pruned search space [45], or can be obtained by merging several \(n\)-best sequences [67, 89].

A word lattice \(\mathcal{G}=\langle V,E\rangle\) is a directed, connected, and acyclic graph with nodes \(V\) and edges \(E\). \(V{\subset}\mathbb{N}\) is a node set, and \((k,i){\in}E\) denotes an edge connecting node \(k\) to node \(i\). \(C(i)\) denotes the set of predecessor nodes for node \(i\). We assume that all nodes follow a topological ordering, such that \(k{<}i~{}\forall k{\in}C(i)\). Each node \(i\) is assigned a word label \(w(i)\). 2 Every lattice contains exactly one start-of-sequence node with only outgoing edges, and exactly one end-of-sequence node with only incoming edges.

Footnote 2: It is perhaps more common to think of each edge representing a word, but we will motivate why we instead assign word labels to nodes in §3.3.

### Lattices and the TreeLSTM

One thing to notice here is that lattice nodes can have multiple predecessor states. In contrast, hidden states in LSTMs and other sequential RNNs are conditioned on only one predecessor state (\(\tilde{h}_{j}\) in left column of Table 1), rendering standard RNNs unsuitable for the modeling of lattices. Luckily Tai et al. luckily Tai et al. luckily Tai et al. (2015)'s TreeLSTM, which was designed to compose encodings in trees, is also straightforward to apply to lattices; the TreeLSTM composes multiple child states into a parent state, which can also be applied to lattices to compose multiple predecessor states into a successor state. Table 1, middle column, shows the TreeLSTM in its child-sum variant that supports an arbitrary number of predecessors. Conditioning on multiple predecessor hidden states is achieved by simply taking their sum as \(\tilde{\textbf{h}}_{i}\). Cell states from multiple predecessor are each passed through their own forget gates \({\bf f}_{jk}\) and then summed.

Encoding a lattice results in one hidden state for each lattice node. Our lat2seq framework uses this network as encoder, computing the attention over all lattice nodes.3 In other words we replace (1) by the following:

Footnote 3: This is similar in spirit to Eriguchi et al. eriguchi2016 who used the TreeLSTM in an attentional tree-to-sequence model.

\[\overrightarrow{{\bf h}}_{i}=\mathrm{LatticeLSTM}\left({\bf x}_{i},\{ \overrightarrow{{\bf h}}_{k}\mid k{\in}C(i)\}\right) \tag{4}\]

Similarly, we encode the lattice in backward direction and replace (2) accordingly. Figure 2 illustrates the result. The computational complexity of the encoder is \(\mathcal{O}(|V|+|E|)\), i.e. linear in the number of nodes plus number of edges in the graph. The complexity of the attention mechanism is \(\mathcal{O}(|V|M)\), where \(M\) is the output sequence 

length. \(|V|\) depends on both the expected input sentence length and the lattice density.

### Node-labeled Lattices

At this point we take a step back to motivate our choice of assigning word labels to lattice nodes, which is in contrast to the prior work by Ladhak et al. (2016) and Su et al. (2017) who assign word labels to edges. Recurrent states in edge-labeled lattice encoders are conditioned not only on multiple predecessor states, but must also aggregate words from multiple incoming edges. This implies that hidden units may represent more than one word in the lattice. Moreover, in the edge-labeled case hidden units that are in the same position in forward and backward encoders represent different words, but are nevertheless concatenated and attended to jointly. For these reasons we find our approach of encoding word-labeled lattices more intuitively appealing when used as input to an attentional decoder, although empirical justification is beyond the scope of this paper. We also note that it is easy to convert an edge-labeled lattice into a node-labeled lattice using the line-graph algorithm Hemminger and Beineke (1978), which we utilize in this work.

## 4 Integration of Lattice Scores

This section describes the key technical contribution of our work: integration of lattice scores encoding input uncertainty into the lat2seq framework. These lattice scores assign different probabilities to competing paths, and are often provided by up-stream statistical models. For example, an ASR may attach posterior probabilities that capture acoustic evidence and linguistic plausibility of words in the lattice. In this section, we describe our method, first explaining how we normalize scores to a format that is easily usable in our method, then presenting our methods for incorporating these scores into our encoder calculations.

### Lattice Score Normalization

Lattice scores that are obtained from upstream systems (such as ASR) are typically given in forward-normalized fashion, interpreted as the probability

Figure 2: Network structure of a bidirectional lattice encoder with one layer.

 

of a node given its predecessor. Here, outgoing edges sum up to one, as illustrated in Figure 1. However, in some of our methods it will be necessary that scores be normalized in the backward direction, so that the weights from incoming connections sum up to one, or globally normalized, so that the probability of the node is the marginal probability of all the paths containing that node.

Let \(w_{f,i}\), \(w_{m,i}\), \(w_{b,i}\) denote forward-normalized, marginal, and backward-normalized scores for node \(i\) respectively, illustrated in Figure 3. Given \(w_{f,i}\), we can compute marginal probabilities recursively as \(w_{m,i}=\sum_{k\in C(i)}w_{m,k}\cdot w_{f,i}\) by using the forward algorithm Rabiner [20]. Then, we can normalize backward using \(w_{b,i}=\frac{w_{m,i}}{\sum_{k\in C^{\prime}(i)}w_{m,k}}\), where \(C^{\prime}(i)\) denotes the successors of node \(i\). All 3 forms are employed in the sections below.

Furthermore, when integrating these scores into the lat2seq framework, it is desirable to maintain flexibility over how strongly they should impact the model. For this purpose, we introduce a peakiness coefficient \(S\). Given a lattice score \(w_{b,i}\) in backward direction, we compute \(w_{b,i}^{S}/Z_{i}\). \(Z_{i}{=}\sum_{k\in C(i)}w_{b,k}\) is a re-normalization term to ensure that incoming connections still sum up to one. In the forward direction, we compute \(w_{f,i}^{S}/Z_{i}\) and normalize analogously over outgoing connections. Setting \(S{=}0\) amounts to ignoring the scores by flattening their distribution, while letting \(S{\rightarrow}\infty\) puts emphasis solely on the strongest nodes. \(S\) can be optimized jointly with the other model parameters via back-propagation during model training.

### Integration Approaches

We suggest three methods to integrate these scores into our lat2seq model, with equations shown in the right column of Table 1. These methods can optionally be combined, and we conduct an ablation study to assess the effectivity of each method in isolation (SS5.3).

The first method consists of computing a weighted child-sum (WCS), using lattice scores as weights when composing the hidden state \(\tilde{\mathbf{h}}_{i}\). This is based on the intuition that predecessor hidden states with high lattice weights should have a higher influence on their successor than states with low weights. The precise formulas for WCS are shown in (5).

The second method biases the forget gate \(\mathbf{f}_{ik}\) for each predecessor cell state such that predecessors with high lattice score are more likely to pass through the forget gate (BFG). The intuition for this is similar to WCS; the composed cell state is more highly influenced by cell states from pre-decessors with high lattice score. BFG is implemented by introducing a bias term inside the sigmoid as in (6).

In the cases of both WCS and BFG, all hidden units have their own independent peakiness. Thus \(\mathbf{S}_{h}\) and \(\mathbf{S}_{f}\) are vectors, applied elementwise after broadcasting the lattice score. The re-normalization terms \(\mathbf{Z}_{h,k}\) and \(\mathbf{Z}_{f,k}\) are also vectors and are applied element-wise. We use backward-normalized scores \(w_{b,i}\) for the forward-directed encoder, and forward-normalized scores \(w_{f,i}\) for the backward-directed encoder.

In the third and final method, we bias the attentional weights (BATT) to put more focus on lattice nodes with high lattice scores. This can potentially mitigate the problem of having multiple contradicting lattice nodes that may confuse the attentional decoder. BATT is computed by introducing a bias term to the attention as in (7). Attentional weights are scalars, so here the peakiness \(S_{a}\) is also a scalar. We drop the normalization term, relying instead on the softmax normalization. Both BFG and BATT use the logarithm of lattice scores so that values will still be in the probability domain after the softmax or sigmoid is computed.

### Pre-training

Finally, note that our lattice encoder is a strict generalization of a sequential encoder. To reduce the computational burden, we exploit this fact and perform a two-step training process where the model is first _pre-trained_ on sequential data, then _fine-tuned_ on lattice data.4 The pre-training, like standard training for neural machine translation (NMT), allows for efficient training using mini-batches, and also allows for training on standard text corpora for which we might not have lat

Figure 3: Lattice with forward-normalized, marginal, and backward-normalized scores.

 

tices available. The fine-tuning is then performed on parallel data with lattices on the source side. This is much slower5 than the pre-training because the network structure changes from sentence to sentence, preventing us from using efficient mini-batched calculations. However, fine-tuning for only a small number of iterations is generally sufficient, as the model is already relatively accurate in the first place. In practice we found it important to use minibatches when fine-tuning, accumulating gradients over several examples before performing parameter updates. This provided negligible speedups but greatly improved optimization stability.

Footnote 4: Our implementation processed sequential inputs about 75 times faster than lattice inputs during training, and overall fine-tuning convergence was 15 times faster. Decoding was only 1.2 times slower when using lattice inputs. Note that recently proposed approaches for autobatching Neubig et al. (2017) may considerably speed up lattice training.

At test time, the model is able to translate both sequential and lattice inputs and can therefore be used even in cases where no lattices are available, at potentially diminished accuracy.

## 5 Experiments

### Setting

We conduct experiments on the Fisher and Callhome Spanish-English Speech Translation Corpus Post et al. (2013), a corpus of Spanish telephone conversations that includes automatic transcripts and lattices. The Fisher portion consists of telephone conversations between strangers, while the Callhome portion contains telephone conversations between relatives or friends. The training data size is 138,819 sentences (Fisher/Train), and 15,000 sentences (Callhome/Train). Held-out testing data is shown in Table 2. ASR word error rates (WER) are relatively high, due to the spontaneous speaking style and challenging acoustics. Lattices contain on average 3.4 (Fisher/Train) or 4.5 (Callhome/Train) times more words than the corresponding reference transcripts.

For preprocessing, we tokenized and lower-cased source and target sides. We removed punctuation from the reference transcripts on the source side for consistency with the automatic transcripts and lattices. All models are pre-trained and fine-tuned on Fisher/Train unless otherwise noted. Our source-side vocabulary contains all words from the automatic transcripts for Fisher/Train, replacing singletons by an unknown word token, totaling 14,648 words. Similarly, on the target side we used all words from the reference translations of Fisher/Train, replacing singletons by the unknown word, yielding 10,800 words in total.

Our implementation is based on lamtram Neubig (2015) and the DyNet Neubig et al. (2017) toolkit. We use the implemented attentional model with default parameters: a layer size of 256 per encoder direction and 512 for the decoder. Word embedding size was also set to 512. We used two encoder layers and two decoder layers for better baseline performance. For the sequential baselines, the LSTM variant in the left column of Table 1 was employed. We initialized the forget gate biases to 1 as recommended by Jozefowicz et al. (2015).

We used Adam Kingma and Ba (2014) for training, with an empirically determined initial learning rate of 0.001 for pre-training and 0.0001 for fine-tuning. We halve the learning rate when the dev perplexity (on Fisher/Dev) gets worse. Pre-training and fine-tuning on 1-best sequences is performed until convergence, and training on lattices is performed for 2 epochs to keep experimental effort manageable. On Fisher/Train, this took 3-4 days on a fast CPU.6 Minibatch size was 1000 target words for pre-training, and 20 sentences for lattice training. Unless otherwise noted, we employed all three proposed lattice score integration approaches, and optimized peakiness coefficients jointly during training. We repeat training 3 times with different random seeds for parameter initialization and data shuffling, and report averaged results. We set th

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & 1-best & oracle & \\  & WER & WER & \# sent. \\ \hline Fisher/Dev & 41.3 & 19.3 & 3,979 \\ Fisher/Dev2 & 40.0 & 19.4 & 3,961 \\ Fisher/Test & 36.5 & 16.1 & 3,641 \\ Callhome/Devtest & 64.7 & 36.4 & 3,966 \\ Callhome/Evltest & 65.3 & 37.9 & 1,829 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Development data statistics. Average sentence length is between 11.8 and 13.1.

 



### Main Results

We compare 4 systems: Performing pre-training on the sequential reference transcripts only (R), fine-tuning on 1-best transcripts (R+1), fine-tuning on lattices without scores (R+L), and fine-tuning on lattices including lattice scores (R+L+S). At test time, we try references, lattice oracles,7 1-best transcripts, and lattices as inputs to all 4 systems. The former 2 experiments give upper bounds on achievable translation accuracy, while the latter 2 correspond to a realistic setting. Table 3 shows the results on Fisher/Dev2 and Fisher/Test.

Footnote 7: The path through the lattice with the best WER.

Before even considering lattices, we can see that 1-best fine-tuning boosted BLEU scores quite impressively (1-best/R vs. 1-best/R+1), with gains of 1.3 and 0.7 BLEU points. This stands in contrast to Post et al. (2013) who find the 1-best transcripts not to be helpful for training a hierarchical machine translation system. Possible explanations are learning from repeating error patterns, and improved robustness to erroneous inputs. On top of these gains, our proposed set-up (lattice/R+L+S) improve BLEU scores by another 1.4 Removing the lattice scores (lattice/R+L) diminishes the results and performs worse than the 1-best baseline (1-best/R+1), indicating that the proposed lattice score integration is crucial for good performance. This demonstrates a clear advantage of our proposed method over that of Su et al. (2017).

As can be seen in the table, models fine-tuned on lattices show reasonable performance for both lattice and sequential inputs (1-best/R+L, lattice/R+L, 1-best/R+L+S, lattice/R+L+S). This is not surprising, given that the lattice training data includes lattices of varying density, including lattices with very few paths or even only one path. On the other hand, without fine-tuning on lattices, using lattices as input performs poorly (lattice/R and lattice/R+1). A closer look revealed that translations were often too long, potentially because implicitly learned mechanisms for length control were not ready to handle lattice inputs.

Table 3 reports perplexities for Fisher/Dev2. Unlike the corresponding BLEU scores, the lattice encoder appears stronger than the 1-best baseline in terms of perplexity even without lattice scores (lattice/R+L vs. 1-best/R+1). To understand this better, we computed the average entropy of the decoder softmax, a measure of how much confusion there is in the decoder predictions, independent of whether it selects the correct answer or not. Over the first 100 sentences, this value was 2.24 for 1-best/R+1, 2.39 for lattice/R+L, and 2.15 for lattice/R+L+S. This indicates that the decoder is more confused for lattices without scores, while integrating lattice scores removes this problem. These numbers also suggest that it may be possible to obtain further gains using methods that stabilize the decoder.

### Ablation Experiments

Next, we conduct an ablation study to assess the impact of the three proposed extensions for integrating lattice scores (SS4.2). We train models with different peakiness coefficients \(S\), either ignoring lattices scores by fixing \(S{=}0\), using lattice scores as-is by fixing \(S{=}1\), or optimizing \(S\) during training. Table 4 shows the results. Overall, joint training of \(S\) gives similar results as fixing \(S{=}1\), but both clearly outperform fixing \(S{=}0\). Removing confidences (setting \(S{=}0\)) in one place at a time reveals that the attention mechanism is clearly the most important point of integration, while gains from the integration into child-sum and forget gate are smaller and not always consistent.

We also analyzed what peakiness values were actually learned. We found that all 3 models that we trained for the averaging purposes converged to \(S_{a}{=}0.62\). \(\mathbf{S}_{h}\) and \(\mathbf{S}_{f}\) had per-vector means between 0.92 and 1.0, at standard deviations between 0.02 and 0.04. We conclude that while the peakiness coefficients were not particularly helpful in our experiments, stable convergence behavior makes them safe to use, and they might be helpful on other data sets that may contain lattice scores of higher or lower reliability.

### Callhome Experiments

In this experiment, we test a situation in which we have a reasonable amount of sequential data available for pre-training, but only a limited amount of lattice training data for the fine-tuning step. This may be a more realistic situation, because speech translation corpora are scarce. To investigate in this scenario, we again pre-train our models on Fisher/Train, but then fine-tune them on the 9 times smaller Callhome/Train portion of the corpus. We fine-tune for 10 epochs, all other settings are as before. We use Callhome/Evltest for testing. Table 5 shows the results. The trends are consistent to SS5.2: The proposed model (lattice/R+L+S) outperforms the 1-best baseline (1-best/R+1) by

0.8 BLEU points, which in turn beats the pre-trained system (1-best/R) by 1.5 BLEU points. Including the lattice scores is clearly beneficial, although lattices without scores also improve over 1-best inputs in this experiment.

### Impact of Lattice Quality

Next, we analyze the impact of using lattices and lattice scores as the ASR WER changes. We concatenate all test data from Table 2 and divide the result into bins according to the 1-best WER. We sample 1000 sentences from each bin, and compare BLEU scores between several models.

The results are shown in Figure 4. For very good WERs, lattices do not improve over 1-best inputs, which is unsurprising. In all other cases, lattices are helpful. Lattice scores are most beneficial for moderate WERs, and not beneficial for very high WERs. We speculate that for high WERs, the lattice scores tend to be less reliable than for lower WERs.

## 6 Conclusion

We investigated translating uncertain inputs from an error-prone up-stream component using a neu

\begin{table}
\begin{tabular}{c c c c c} \hline BATT & WCS & BFG & Fisher & Fisher \\ \(S_{a}\) & \(\mathbf{S}_{h}\) & \(\mathbf{S}_{f}\) & /Dev2 & /Test \\ \hline
0 & **0** & **0** & 36.9 & 36.1 \\
1 & **1** & **1** & **38.2** & **37.4** \\

* & **38.5*
* & **38.0*
* \\ \hline
0 & **1** & **1** & 37.2 & 36.2 \\
1 & **0** & **1** & **37.9** & **37.5** \\
1 & **1** & **0** & **38.2** & **37.8** \\ \hline
0 & * & * & 37.0 & 36.3 \\
* & **0*
* & **38.3*
* & **37.9*

* & **0*
* & **38.1*
* & **37.5*
* \\ \hline
1-best/R+1 & & 37.2 & 36.6 \\ \end{tabular}
\end{table}
Table 4: BLEU scores (4 references) for differently configured peakiness coefficients \(S_{a},\mathbf{S}_{h},\mathbf{S}_{f}\). 0/1 means fixing to that value, * indicates optimization during training. Statistically significant improvement over 1-best/R+1 is in bold.

Figure 4: BLEU score over varying 1-best WERs.

 

ral lattice-to-sequence model. Our proposed model takes word lattices as input and is able to take advantage of lattice scores. In our experiments in a speech translation task we find consistent improvements over translating 1-best transcriptions and that consideration of lattice scores, especially in the attention mechanism, is crucial for obtaining these improvements.

Promising avenues for future work are investigating consensus networks [20] for potential gains in terms of speed or quality as compared to lattice inputs, explicitly dealing with rare or unknown words in the lattice, and facilitating GPU training via autobatching [34].

## Acknowledgments

We thank Paul Michel and Austin Matthews for their helpful comments on earlier drafts of this paper.

## References

* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In _International Conference on Representation Learning (ICLR)_, San Diego, USA.
* Casacuberta et al. (2004) F. Casacuberta, H. Ney, F. J. Och, E. Vidal, J. M. Vilar, S. Barrachina, I. Garcia-Varea, D. Llorens, C. Martinez, S. Molau, F. Nevado, M. Pastor, D. Picco, A. Sanchis, and C. Tillmann. 2004. Some approaches to statistical and finite-state speech-to-speech translation. _Computer Speech and Language_, 18(1):25-47.
* Cho et al. (2014) Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation. _arXiv preprint arXiv:1406.1078_.
* Dyer et al. (2008) Christopher Dyer, Smaranda Muresan, and Philip Resnik. 2008. Generalizing Word Lattice Translation. Technical Report LAMP-TR-149, University of Maryland, Institute For Advanced Computer Studies.
* Eriguchi et al. (2016) Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2016. Tree-to-Sequence Attentional Neural Machine Translation. In _Association for Computational Linguistic (ACL)_, pages 823-833, Berlin, Germany.
* Hemminger and Beineke (1978) R.L. Hemminger and L.W. Beineke. 1978. Line graphs and line digraphs. In _Selected Topics in Graph Theory_, pages 271-305. Academic Press Inc.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long Short-Term Memory. _Neural computation_, 9(8):1735-1780.
* Jozefowicz et al. (2015) Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. 2015. An Empirical Exploration of Recurrent Network Architectures. In _International Conference on Machine Learning (ICML)_, Lille, France.
* Kalchbrenner and Blunsom (2013) Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent Continuous Translation Models. In _Empirical Methods in Natural Language Processing (EMNLP)_, pages 1700-1709, Seattle, Washington, USA.
* Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Ladhak et al. (2016) Faisal Ladhak, Ankur Gandhe, Markus Dreyer, Lambert Mathias, Ariya Rastrow, and Bjorn Hoffmeister. 2016. LatticeRnn: Recurrent Neural Networks over Lattices. In _Annual Conference of the International Speech Communication Association (InterSpeech)_, pages 695-699, San Francisco, USA.
* Luong et al. (2015) Minh-Thang Luong, Heieu Pham, and Christopher D Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. In _Empirical Methods in Natural Language Processing (EMNLP)_, pages 1412-1421, Lisbon, Portugal.
* Mangu et al. (2000) Lidia Mangu, Eric Brill, and Andreas Stolcke. 2000. Finding consensus in speech recognition: word error minimization and other applications of confusion networks. _Computer Speech & Language_, 14(4):373-400.
* Matusov et al. (2008) Evgeny Matusov, Bjorn Hoffmeister, and Hermann Ney. 2008. ASR word lattice translation with exhaustive reordering is possible. In _Annual Conference of the International Speech Communication Association (InterSpeech)_, pages 2342-2345, Brisbane, Australia.
* Neubig (2015) Graham Neubig. 2015. lamtram: A Toolkit for Language and Translation Modeling using Neural Networks. [http://www.github.com/neubig/lamtram](http://www.github.com/neubig/lamtram).
* Neubig et al. (2017a) Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaura Kumar, Chaitanya Malaviya, Paul Michel, Yusuke Oda, Matthew Richardson, Naomi Saphra, Swabha Swayamdipta, and Pengcheng Yin. 2017a. DyNet: The Dynamic Neural Network Toolkit. _arXiv preprint arXiv:1701.03980_.
* Neubig et al. (2017b) Graham Neubig, Yoav Goldberg, and Chris Dyer. 2017b. On-the-fly Operation Batching in Dynamic Computation Graphs. _arXiv preprint arXiv:1705.07860_ 

* Ney (1999) Hermann Ney. 1999. Speech Translation: Coupling of Recognition and Translation. In _International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_, pages 517-520, Phoenix, USA.
* Post et al. (2013) Matt Post, Gaurav Kumar, Adam Lopez, Damianos Karakos, Chris Callison-Burch, and Sanjeev Khudanpur. 2013. Improved Speech-to-Text Translation with the Fisher and Calhloume Spanish-English Speech Translation Corpus. In _International Workshop on Spoken Language Translation (IWSLT)_, Heidelberg, Germany.
* Rabiner (1989) Lawrence R. Rabiner. 1989. A tutorial on hidden markov models and selected applications in speech recognition. _Proceedings of the IEEE_.
* Saleem et al. (2004) Shirin Saleem, Szu-Chen (Stan) Jou, Stephan Vogel, and Tanja Schultz. 2004. Using Word Lattice Information for a Tighter Coupling in Speech Translation Systems. In _International Conference of Spoken Language Processing (ICSLP)_, pages 41-44, Jeju Island, Korea.
* Su et al. (2017)Jinsong Su, Zhixing Tan, Deyi Xiong, Ringrong Ji, Xiaodong Shi, and Yang Liu. 2017. Lattice-Based Recurrent Neural Network Encoders for Neural Machine Translation. In _Conference on Artificial Intelligence (AAAI)_, pages 3302-3308.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In _Advances in Neural Information Processing Systems (NIPS)_, pages 3104-3112, Montreal, Canada.
* Tai et al. (2015) Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks. In _Association for Computational Linguistic (ACL)_, pages 1556-1566, Beijing, China.
* Zhang et al. (2005) Ruiqiang Zhang, Genichiro Kikui, Hirofumi Yamamoto, and Wai-Kit Lo. 2005. A Decoding Algorithm for Word Lattice Translation in Speech Translation. In _International Workshop on Spoken Language Translation (IWSLT)_, pages 23-29, Pittsburgh, USA.