Introduction
Language models (LMs) are crucial components in many applications, such as speech recognition and machine translation. The aim of language models is to compute the probability of any given sentence INLINEFORM0 , which can be calculated as DISPLAYFORM0
The task of LMs is to calculate the probability of word INLINEFORM0 given its previous history INLINEFORM1 . INLINEFORM2 -gram LMs BIBREF0 and neural network based language mdoels (NNLMs) BIBREF1 , BIBREF2 are two widely used language models. In INLINEFORM3 -gram LMs, the most recent INLINEFORM4 words are used as an approximation of the complete history, thus DISPLAYFORM0
This INLINEFORM0 -gram assumption can also be used to construct a INLINEFORM1 -gram feedforward NNLMs BIBREF1 . In contrast, recurrent neural network LMs (RNNLMs) model the complete history via a recurrent connection.
Most of previous work on language models has focused on utilising history information, the future word context information has not been extensively investigated. There have been several attempts to incorporate future context information into recurrent neural network language models. Individual forward and backward RNNLMs can be built, and these two LMs combined with a log-linear interpolation BIBREF3 . In BIBREF4 , succeeding words were incorporated into RNNLM within a Maximum Entropy framework. BIBREF5 investigated the use of bidirectional RNNLMs (bi-RNNLMs) for speech recognition. For a broadcast news task, sigmoid based RNNLMs gave small gains, while no performance improvement was obtained when using long short-term memory (LSTM) based RNNLMs. More recently, bi-RNNLMs can produce consistent, and significant, performance improvements over unidirectional RNNLMs (uni-RNNLMs) on a range of speech recognition tasks BIBREF6 .
Though they can yield performance gain, bi-RNNLMs pose several challenges for both model training and inference as they require the complete previous and future word context information to be taken into account. It is difficult to parallelise training efficiently. Lattice rescoring is also complicated for these LMs as future context needs to be incorporated. This means that the form of approximation used for uni-RNNLMs BIBREF7 is not suitable to apply. Hence, N-best rescoring is normally used BIBREF4 , BIBREF5 , BIBREF6 . However, the ability to manipulate lattices is very important in many speech applications. Lattices can be used for a wide range of downstream applications, such as confidence score estimation BIBREF8 , keyword search BIBREF9 and confusion network decoding BIBREF10 . In order to address these issues, a novel model structure, succeeding word RNNLMs (su-RNNLMs), is proposed in this paper. Instead of using a recurrent unit to capture the complete future word context as in bi-RNNLMs, a feedforward unit is used to model a small, fixed-length number of succeeding words. This allows existing efficient training BIBREF11 and lattice rescoring BIBREF7 algorithms developed for uni-RNNLMs to be extended to the proposed su-RNNLMs. Using these extended algorithms, compact lattices can be generated with su-RNNLMs supporting lattice based downstream processing.
The rest of this paper is organized as follows. Section SECREF2 gives a brief review of RNNLMs, including both unidirectional and bidirectional RNNLMs. The proposed model with succeeding words (su-RNNLMs) is introduced in Section SECREF3 , followed by a description of the lattice rescoring algorithm in Section SECREF4 . Section SECREF5 discusses the interpolation of language models. The experimental results are presented in Section SECREF6 and conclusions are drawn in Section SECREF7 .
Unidirectional RNNLMs
In contrast to feedforward NNLMs, where only modeling the previous INLINEFORM0 words, recurrent NNLMs BIBREF12 represent the full non-truncated history INLINEFORM1 for word INLINEFORM2 using the 1-of-K encoding of the previous word INLINEFORM3 and a continuous vector INLINEFORM4 as a compact representation of the remaining context INLINEFORM5 . Figure FIGREF5 shows an example of this unidirectional RNNLM (uni-RNNLM). The most recent word INLINEFORM6 is used as input and projected into a low-dimensional, continuous, space via a linear projection layer. A recurrent hidden layer is used after this projection layer. The form of the recurrent layer can be based on a standard sigmoid based recurrent unit, with sigmoid activations BIBREF2 , or more complicated forms such as gated recurrent unit (GRU) BIBREF13 and long short-term memory (LSTM) units BIBREF14 . A continuous vector INLINEFORM7 representing the complete history information INLINEFORM8 can be obtained using INLINEFORM9 and previous word INLINEFORM10 . This vector is used as input of recurrent layer for the estimation of next word. An output layer with softmax function is used to calculate the probability INLINEFORM11 . An additional node is often added at the output layer to model the probability mass of out-of-shortlist (OOS) words to speed up softmax computation by limiting vocabulary size BIBREF15 . Similarly, an out-of-vocabulary (OOV) node can be added in the input layer to model OOV words. The probability of word sequence INLINEFORM12 is calculated as, DISPLAYFORM0
Perplexity (PPL) is a metric used widely to evaluate the quality of language models. According to the definition in BIBREF16 , the perplexity can be computed based on sentence probability with, DISPLAYFORM0
Where INLINEFORM0 is the total number of words and INLINEFORM1 is the number of sentence in the evaluation corpus. INLINEFORM2 is the number of word in INLINEFORM3 th sentence. From the above equation, the PPL is calculated based on the average log probability of each word, which for unidirectional LMs, yields the average sentence log probability.
Uni-RNNLMs can be trained efficiently on Graphics Processing Units (GPUs) by using spliced sentence bunch (i.e. minibatch) mode BIBREF11 . Multiple sentences can be concatenated together to form a longer sequence and sets of these long sequences can then be aligned in parallel from left to right. This data structure is more efficient for minibatch based training as they have comparable sequence length BIBREF11 . When using these forms of language models for tasks like speech recognition, N-best rescoring is the most straightforward way to apply uni-RNNLMs. Lattice rescoring is also possible by introducing approximations BIBREF7 to control merging and expansion of different paths in lattice. This will be described in more detail in Section SECREF4 .
Bidirectional RNNLMs
Figure FIGREF8 illustrates an example of bidirectional RNNLMs (bi-RNNLMs). Unlike uni-RNNLMs, both the history word context INLINEFORM0 and future word context INLINEFORM1 are used to estimate the probability of current word INLINEFORM2 . Two recurrent units are used to capture the previous and future information respectively. In the same fashion as uni-RNNLMs, INLINEFORM3 is a compact continuous vector of the history information INLINEFORM4 . While INLINEFORM5 is another continuous vector to encode the future information INLINEFORM6 . This future context vector is computed from the next word INLINEFORM7 and the previous future context vector INLINEFORM8 containing information of INLINEFORM9 . The concatenation of INLINEFORM10 and INLINEFORM11 is then fed into the output layer, with softmax function, to calculate the output probability. In order to reduce the number of parameter, the projection layer for the previous and future words are often shared.
The probability of word sequence INLINEFORM0 can be computed using bi-RNNLMs as, DISPLAYFORM0
INLINEFORM0 is the unnormalized sentence probability computed from the individual word probabilities of the bi-RNNLM. INLINEFORM1 is a sentence-level normalization term to ensure the sentence probability is appropriately normalized. This is defined as, DISPLAYFORM0
where INLINEFORM0 is the set of all possible sentences. Unfortunately, this normalization term is impractical to calculate for most tasks.
In a similar form to Equation EQREF6 , the PPL of bi-RNNLMs can be calculated based on sentence probability as, DISPLAYFORM0
However, INLINEFORM0 is often infeasible to obtain. As a result, it is not possible to compute a valid perplexity from bi-RNNLMs. Nevertheless, the average log probability of each word can be used to get a “pseudo” perplexity (PPL). DISPLAYFORM0
This is the second term of the valid PPL of bi-RNNLMs shown in Equation EQREF11 . It is a “pseudo” PPL because the normalized sentence probability INLINEFORM0 is impossible to obtain and the unnormalized sentence probability INLINEFORM1 is used instead. Hence, the “pseudo” PPL of bi-RNNLMs is not comparable with the valid PPL of uni-RNNLMs. However, the value of “pseudo” PPL provides information on the average word probability from bi-RNNLMs since it is obtained using the word probability.
In order to achieve good performance for speech recognition, BIBREF6 proposed an additional smoothing of the bi-RNNLM probability at test time. The probability of bi-RNNLMs is smoothed as, DISPLAYFORM0
where INLINEFORM0 is the activation before softmax function for node INLINEFORM1 in the output layer. INLINEFORM2 is an empirical smoothing factor, which is chosen as 0.7 in this paper.
The use of both preceding and following context information in bi-RNNLMs presents challenges to both model training and inference. First, N-best rescoring is normally used for speech recognition BIBREF6 . Lattice rescoring is impractical for bi-RNNLMs as the computation of word probabilities requires information from the complete sentence.
Another drawback of bi-RNNLMs is the difficulty in training. The complete previous and future context information is required to predict the probability of each word. It is expensive to directly training bi-RNNLMs sentence by sentence, and difficult to parallelise the training for efficiency. In BIBREF5 , all sentences in the training corpus were concatenated together to form a single sequence to facilitate minibatch based training. This sequence was then “chopped” into sub-sequences with the average sentence length. Bi-RNNLMs were then trained on GPU by processing multiple sequences at the same time. This allows bi-RNNLMs to be efficiently trained. However, issues can arise from the random cutting of sentences, history and future context vectors may be reset in the middle of a sentence. In BIBREF6 , the bi-RNNLMs were trained in a more consistent fashion. Multiple sentences were aligned from left to right to form minibatches during bi-RNNLM training. In order to handle issues caused by variable sentence length, NULL tokens were appended to the ends of sentences to ensure that the aligned sentences had the same length. These NULL tokens were not used for parameter update. In this paper, this approach is adopted to train bi-RNNLMs as it gave better performance.
RNNLMs with succeeding words
As discussed above, bi-RNNLMs are slow to train and difficult to use in lattice rescoring. In order to address these issues, a novel structure, the su-RNNLM, is proposed in this paper to incorporate future context information. The model structure is illustrated in Figure FIGREF14 . In the same fashion as bi-RNNLMs, the previous history INLINEFORM0 is modeled with recurrent units (e.g. LSTM, GRU). However, instead of modeling the complete future context information, INLINEFORM1 , using recurrent units, feedforward units are used to capture a finite number of succeeding words, INLINEFORM2 . The softmax function is again applied at the output layer to obtain the probability of the current word INLINEFORM3 . The word embedding in the projection layer are shared for all input words. When the succeeding words are beyond the sentence boundary, a vector of 0 is used as the word embedding vector. This is similar to the zero padding of the feedforward forward NNLMs at the beginning of each sentence BIBREF12 .
As the number of succeeding words is finite and fixed for each word, its succeeding words can be organized as a INLINEFORM0 -gram future context and used for minibatch mode training as in feedforward NNLMs BIBREF12 . Su-RNNLMs can then be trained efficiently in a similar fashion to uni-RNNLMs in a spliced sentence bunch mode BIBREF11 .
Compared with equations EQREF4 and EQREF9 , the probability of word sequence INLINEFORM0 can be computed as DISPLAYFORM0
Again, the sentence level normalization term INLINEFORM0 is difficult to compute and only “pseudo” PPL can be obtained. The probabilities of su-RNNLMs are also very sharp, which can be seen from the “pseudo” PPLs in Table TABREF27 in Section SECREF6 . Hence, the bi-RNNLM probability smoothing given in Equation EQREF13 is also required for su-RNNLMs to achieve good performance at evaluation time.
Lattice rescoring
Lattice rescoring with feedforward NNLMs is straightforward BIBREF12 whereas approximations are required for uni-RNNLMs lattice rescoring BIBREF7 , BIBREF17 . As mentioned in Section SECREF7 , N-best rescoring has previously been used for bi-RNNLMs. It is not practical for bi-RNNLMs to be used for lattice rescoring and generation as both the complete previous and future context information are required. However, lattices are very useful in many applications, such as confidence score estimation BIBREF8 , keyword search BIBREF9 and confusion network decoding BIBREF10 . In contrast, su-RNNLMs require a fixed number of succeeding words, instead of the complete future context information. From Figure FIGREF14 , su-RNNLMs can be viewed as a combination of uni-RNNLMs for history information and feedforward NNLMs for future context information. Hence, lattice rescoring is feasible for su-RNNLMs by extending the lattice rescoring algorithm of uni-RNNLMs by considering additional fixed length future contexts.
Lattice rescoring of uni-RNNLMs
In this paper, the INLINEFORM0 -gram approximation BIBREF7 based approach is used for uni-RNNLMs lattice rescoring. When considering merging of two paths, if their previous INLINEFORM1 words are identical, the two paths are viewed as “equivalent” and can be merged. This is illustrated in Figure FIGREF19 for the start node of word INLINEFORM2 . The history information from the best path is kept for the following RNNLM probability computation and the histories of all other paths are discarded. For example, the path INLINEFORM3 is kept and the other path INLINEFORM4 is discarded given arc INLINEFORM5 .
There are two types of approximation involved for uni-RNNLM lattice rescoring, which are the merge and cache approximations. The merge approximation controls the merging of two paths. In BIBREF7 , the first path reaching the node was kept and all other paths with the same INLINEFORM0 -gram history were discarded irrespective of the associated scores. This introduces inaccuracies in the RNNLM probability calculation. The merge approximation can be improved by keeping the path with the highest accumulated score. This is the approach adopted in this work. For fast probability lookup in lattice rescoring, INLINEFORM1 -gram probabilities can be cached using INLINEFORM2 words as a key. A similar approach can be used with RNNLM probabilities. In BIBREF7 , RNNLM probabilities were cached based on the previous INLINEFORM3 words, which is referred as cache approximation. Thus a word probability obtained from the cache may be derived from another history sharing the same INLINEFORM4 previous words. This introduces another inaccuracy. In order to avoid this inaccuracy yet maintain the efficiency, the cache approximation used in BIBREF7 is improved by adopting the complete history as key for caching RNNLM probabilities. Both modifications yielt small but consistent improvements over BIBREF7 on a range of tasks.
Lattice rescoring of su-RNNLMs
For lattice rescoring with su-RNNLMs, the INLINEFORM0 -gram approximation can be adopted and extended to support the future word context. In order to handle succeeding words correctly, paths will be merged only if the following succeeding words are identical. In this way, the path expansion is carried out in both directions. Any two paths with the same succeeding words and INLINEFORM1 previous words are merged.
Figure FIGREF18 shows part of an example lattice generated by a 2-gram LM. In order to apply uni-RNNLM lattice rescoring using a 3-gram approximation, the grey shaded node in Figure FIGREF18 needs to be duplicated as word INLINEFORM0 has two distinct 3-gram histories, which are INLINEFORM1 and INLINEFORM2 respectively. Figure FIGREF19 shows the lattice after rescoring using a uni-RNNLM with 3-gram approximation. In order to apply su-RNNLMs for lattice rescoring, the succeeding words also need to be taken into account. Figure FIGREF20 is the expanded lattice using a su-RNNLM with 1 succeeding word. The grey shaded nodes in Figure FIGREF19 need to be expanded further as they have distinct succeeding words. The blue shaded nodes in Figure FIGREF20 are the expanded node in the resulting lattice.
Using the INLINEFORM0 -gram history approximation and given INLINEFORM1 succeeding words, the lattice expansion process is effectively a INLINEFORM2 -gram lattice expansion for uni-RNNLMs. For larger value of INLINEFORM3 and INLINEFORM4 , the resulting lattices can be very large. This can be addressed by pruning the lattice and doing initial lattice expansion with a uni-RNNLM.
Language Model Interpolation
For unidirectional language models, such as INLINEFORM0 -gram model and uni-RNNLMs, the word probabilities are normally combined using linear interpolation, DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are the probabilities from INLINEFORM2 -gram and uni-RNN LMs respectively, INLINEFORM3 is the interpolation weight of uni-RNNLMs.
However, it is not valid to directly combine uni-LMs (e.g unidirectional INLINEFORM0 -gram LMs or RNNLMs) and bi-LMs (or su-LMs) using linear interpolation due to the sentence level normalisation term required for bi-LMs (or su-LMs) in Equation EQREF9 . As described in BIBREF6 , uni-LMs can be log-linearly interpolated with bi-LMs for speech recognition using, DISPLAYFORM0
where INLINEFORM0 is the appropriate normalisation term. The normalisation term can be discarded for speech recognition as it does not affect the hypothesis ranking. INLINEFORM1 and INLINEFORM2 are the probabilities from uni-LMs and bi-RNNLMs respectively. INLINEFORM3 is the log-linear interpolation weight of bi-RNNLMs. The issue of normalisation term in su-RNLMs is similar to that of bi-RNNLMs, as shown in Equation EQREF15 . Hence, log-linear interpolation can also be applied for the combination of su-RNNLMs and uni-LMs and is the approach used in this paper.
By default, linear interpolation is used to combine uni-RNNLMs and INLINEFORM0 -gram LMs. A two-stage interpolation is used when including bi-RNNLMs and su-RNNLMs. The uni-RNNLMs and INLINEFORM1 -gram LMs are first interpolated using linear interpolation. These linearly interpolated probabilities are then log-linearly interpolated with those of bi-RNNLMs (or su-RNNLMs).
Experiments
Experiments were conducted using the AMI IHM meeting corpus BIBREF18 to evaluated the speech recognition performance of various language models. The Kaldi training data configuration was used. A total of 78 hours of speech was used in acoustic model training. This consists of about 1M words of acoustic transcription. Eight meetings were excluded from the training set and used as the development and test sets.
The Kaldi acoustic model training recipe BIBREF19 featuring sequence training BIBREF20 was applied for deep neural network (DNN) training. CMLLR transformed MFCC features BIBREF21 were used as the input and 4000 clustered context dependent states were used as targets. The DNN was trained with 6 hidden layers, and each layer has 2048 hidden nodes.
The first part of the Fisher corpus, 13M words, was used for additional language modeling training data. A 49k word decoding vocabulary was used for all experiments. All LMs were trained on the combined (AMI+Fisher), 14M word in total. A 4-gram KN smoothed back-off LM without pruning was trained and used for lattices generation. GRU based recurrent units were used for all unidirectional and bidirectional RNNLMs . 512 hidden nodes were used in the hidden layer. An extended version of CUED-RNNLM BIBREF22 was developed for the training of uni-RNNLMs, bi-RNNLMs and su-RNNLMs. The related code and recipe will be available online . The linear interpolation weight INLINEFORM0 between 4-gram LMs and uni-RNNLMs was set to be 0.75 as it gave the best performance on the development data. The log-linear interpolation weight INLINEFORM1 for bi-RNNLMs (or su-RNNLMs) was 0.3. The probabilities of bi-RNNLMs and su-RNNLMs were smoothed with a smoothing factor 0.7 as suggested in BIBREF6 . The 3-gram approximation was applied for the history merging of uni-RNNLMs and su-RNNLMs during lattice rescoring and generation BIBREF7 .
Table TABREF26 shows the word error rates of the baseline system with 4-gram and uni-RNN LMs. Lattice rescoring and 100-best rescoring are applied to lattices generated by the 4-gram LM. As expected, uni-RNNLMs yield a significant performance improvement over 4-gram LMs. Lattice rescoring gives a comparable performance with 100-best rescoring. Confusion network (CN) decoding can be applied to lattices generated by uni-RNNLM lattice rescoring and additional performance improvements can be achieved. However, it is difficult to apply confusion network decoding to the 100-best .
Table TABREF27 gives the training speed measured with word per second (w/s) and (“pseudo”) PPLs of various RNNLMs with difference amounts of future word context. When the number of succeeding words is 0, this is the baseline uni-RNNLMs. When the number of succeeding words is set to INLINEFORM0 , a bi-RNNLM with complete future context information is used. It can be seen that su-RNNLMs give a comparable training speed as uni-RNNLMs. The additional computational load of the su-RNNLMs mainly come from the feedforward unit for succeeding words as shown in Figure FIGREF14 . The computation in this part is much less than that of other parts such as output layer and GRU layers. However, the training of su-RNNLMs is much faster than bi-RNNLMs as it is difficult to parallelise the training of bi-RNNLMs efficiently BIBREF6 . It is worth mentioning again that the PPLs of uni-RNNLMs can not be compared directly with the “pseudo” PPLs of bi-RNNLMs and su-RNNLMs. But both PPLs and “pseudo” PPLs reflect the average log probability of each word. From Table TABREF27 , with increasing number of succeeding words, the “pseudo” PPLs of the su-RNNLMs keeps decreasing, yielding comparable value as bi-RNNLMs.
Table TABREF28 gives the WER results of 100-best rescoring with various language models. For bi-RNNLMs (or su-RNNLMs), it is not possible to use linear interpolation. Thus a two stage approach is adopted as described in Section SECREF5 . This results in slight differences, second decimal place, between the uni-RNNLM case and the 0 future context su-RNNLM. The increasing number of the succeeding words consistently reduces the WER. With 1 succeeding word, the WERs were reduced by 0.2% absolutely. Su-RNNLMs with more than 2 succeeding words gave about 0.5% absolute WER reduction. Bi-RNNLMs (shown in the bottom line of Table TABREF28 ) outperform su-RNNLMs by 0.1% to 0.2%, as it is able to incorporate the complete future context information with recurrent connection.
Table TABREF29 shows the WERs of lattice rescoring using su-RNNLMs. The lattice rescoring algorithm described in Section SECREF4 was applied. Su-RNNLMs with 1 and 3 succeeding words were used for lattice rescoring. From Table TABREF29 , su-RNNLMs with 1 succeeding words give 0.2% WER reduction and using 3 succeeding words gives about 0.5% WER reduction. These results are consistent with the 100-best rescoring result in Table TABREF28 . Confusion network decoding can be applied on the rescored lattices and additional 0.3-0.4% WER performance improvements are obtained on dev and eval test sets.
Conclusions
In this paper, the use of future context information on neural network language models has been explored. A novel model structure is proposed to address the issues associated with bi-RNNLMs, such as slow train speed and difficulties in lattice rescoring. Instead of using a recurrent unit to capture the complete future information, a feedforward unit was used to model a finite number of succeeding words. The existing training and lattice rescoring algorithms for uni-RNNLMs are extended for the proposed su-RNNLMs. Experimental results show that su-RNNLMs achieved a slightly worse performances than bi-RNNLMs, but with much faster training speed. Furthermore, additional performance improvements can be obtained from lattice rescoring and subsequent confusion network decoding. Future work will examine improved pruning scheme to address the lattice expansion issues associated with larger future context.

Question: Which dataset do they use?
Answer:  AMI IHM meeting corpus