Abstract: Recently, bidirectional recurrent network language models (bi-RNNLMs) have been shown to outperform standard, unidirectional, recurrent neural network language models (uni-RNNLMs) on a range of speech recognition tasks. This indicates that future word context information beyond the word history can be useful. However, bi-RNNLMs pose a number of challenges as they make use of the complete previous and future word context information. This impacts both training efficiency and their use within a lattice rescoring framework. In this paper these issues are addressed by proposing a novel neural network structure, succeeding word RNNLMs (su-RNNLMs). Instead of using a recurrent unit to capture the complete future word contexts, a feedforward unit is used to model a finite number of succeeding, future, words. This model can be trained much more efficiently than bi-RNNLMs and can also be used for lattice rescoring. Experimental results on a meeting transcription task (AMI) show the proposed model consistently outperformed uni-RNNLMs and yield only a slight degradation compared to bi-RNNLMs in N-best rescoring. Additionally, performance improvements can be obtained using lattice rescoring and subsequent confusion network decoding.

Introduction: Language models (LMs) are crucial components in many applications, such as speech recognition and machine translation. The aim of language models is to compute the probability of any given sentence INLINEFORM0 , which can be calculated as DISPLAYFORM0  The task of LMs is to calculate the probability of word INLINEFORM0 given its previous history INLINEFORM1 . INLINEFORM2 -gram LMs BIBREF0 and neural network based language mdoels (NNLMs) BIBREF1 , BIBREF2 are two widely used language models. In INLINEFORM3 -gram LMs, the most recent INLINEFORM4 words are used as an approximation of the complete history, thus DISPLAYFORM0  This INLINEFORM0 -gram assumption can also be used to construct a INLINEFORM1 -gram feedforward NNLMs BIBREF1 . In contrast, recurrent neural network LMs (RNNLMs) model the complete history via a recurrent connection. Most of previous work on language models has focused on utilising history information, the future word context information has not been extensively investigated. There have been several attempts to incorporate future context information into recurrent neural network language models. Individual forward and backward RNNLMs can be built, and these two LMs combined with a log-linear interpolation BIBREF3 . In BIBREF4 , succeeding words were incorporated into RNNLM within a Maximum Entropy framework. BIBREF5 investigated the use of bidirectional RNNLMs (bi-RNNLMs) for speech recognition. For a broadcast news task, sigmoid based RNNLMs gave small gains, while no performance improvement was obtained when using long short-term memory (LSTM) based RNNLMs. More recently, bi-RNNLMs can produce consistent, and significant, performance improvements over unidirectional RNNLMs (uni-RNNLMs) on a range of speech recognition tasks BIBREF6 . Though they can yield performance gain, bi-RNNLMs pose several challenges for both model training and inference as they require the complete previous and future word context information to be taken into account. It is difficult to parallelise training efficiently. Lattice rescoring is also complicated for these LMs as future context needs to be incorporated. This means that the form of approximation used for uni-RNNLMs BIBREF7 is not suitable to apply. Hence, N-best rescoring is normally used BIBREF4 , BIBREF5 , BIBREF6 . However, the ability to manipulate lattices is very important in many speech applications. Lattices can be used for a wide range of downstream applications, such as confidence score estimation BIBREF8 , keyword search BIBREF9 and confusion network decoding BIBREF10 . In order to address these issues, a novel model structure, succeeding word RNNLMs (su-RNNLMs), is proposed in this paper. Instead of using a recurrent unit to capture the complete future word context as in bi-RNNLMs, a feedforward unit is used to model a small, fixed-length number of succeeding words. This allows existing efficient training BIBREF11 and lattice rescoring BIBREF7 algorithms developed for uni-RNNLMs to be extended to the proposed su-RNNLMs. Using these extended algorithms, compact lattices can be generated with su-RNNLMs supporting lattice based downstream processing. The rest of this paper is organized as follows. Section SECREF2 gives a brief review of RNNLMs, including both unidirectional and bidirectional RNNLMs. The proposed model with succeeding words (su-RNNLMs) is introduced in Section SECREF3 , followed by a description of the lattice rescoring algorithm in Section SECREF4 . Section SECREF5 discusses the interpolation of language models. The experimental results are presented in Section SECREF6 and conclusions are drawn in Section SECREF7 .

Question: How do Zipf and Herdan-Heap's laws differ?
Answer: Zipf's law describes change of word frequency rate, while Heaps-Herdan describes different word number in large texts (assumed that Hepas-Herdan is consequence of Zipf's)