Exploiting Syntactic Structure for Language Modeling

The paper presents a language model that develops syntactic structure and uses it to extract nmaningflfl information from the word history, thus enabling the use of long distance dependencies. The model assigns probability to every joint sequence of words-binary-parse-structure with headword annotation and operates in a left-to-right manner therefore usable for automatic speech recognition. The model, its probabilistic parameterization, and a set of experinmnts meant to evaluate its predictive power are presented; an improvement over standard tr igram modeling is achieved. 1 I n t r o d u c t i o n The main goal of the present work is to develop a language model that uses syntactic structure to model tong-distance dependencies. During the summer96 DoD Workshop a similar a t t empt was made by the dependency modeling group. The model we present is closely related to tile one investigated in (Chelba et al., 1997), however different in a few important aspects: • our model operates in a left-to-right manner, allowing the decoding of word lattices, as opposed to the one referred to previously, where only whole sentences could be processed, thus reducing its applicability to n-best list re-scoring; the syntactic structure is developed as a model component; • our model is a factored version of the one in (Chelba et al., 1997), thus enabling tile calculation of the joint probability of words and parse structure; this was not possible in the previous case due to the huge computational complexity of the model. Our model develops syntactic structure incrementally while traversing the sentence fl'om left to right. This is the main difference between our approach and other approaches to statistical natural language parsing. Our parsing strategy is similar to tile incremental syntax ones proposed relatively recently in the linguistic community (Philips, 1996). The probabilistic model, its parameterizat ion and a few experiments that are meant to evaluate its potential for speech recognition m'e presented. Ih¢_DT clmlract NN • i,Jc,! " ,M) with IN a_DT loss NN o L [ N 7 CD ~ . \ i . > alter Figure 1: Partial parse 2 T h e B a s i c I d e a a n d T e r m i n o l o g y Consider predicting the word a f t e r in the sentence: the contract ended with a loss of 7 cents after trading as low as 89 cents, A 3-gram approach would predict after from (7, c e n t s ) whereas it is intuitively clear that the strongest predictor would be ended which is outside tile reach of even 7-grams. Our assumption is that what enables humans to make a good prediction of a f t e r is the syntactic structure in the past. The linguistically correct partial parse of the word history when predicting a f t e r is shown in Figure 1. The word ended is called the headword of tile consti tucnt (ended (wi th ( . . . ) ) ) and ended is all exposed headword when predicting a f t e r topmost headword in the largest constituent that contains it,. The syntactic structure in the past filters out irrelevant words and points to the important ones, thus enabling the use of long distance information when predicting the next word. Our model will a t tempt to build the syntactic structure incrementally while traversing the sentence left-to-right. The model will assign a probability P(W, T) to every sentence W with every possible POStag assignment, binary branching parse, nonterminal label and headword annotation for every constituent of T. Let W be a sentence of length n words to which we have prepended 
 and appended 
 so that We =
 and w~+l =</ s> . Let Wk be the word k-prefix w0 . wk of the sentence and WkTk