Hierarchical Probabilistic Neural Network Language Model

Frederic Morin
Dept. IRO, Université de Montréal
P.O. Box 6128, Succ. Centre-Ville,
Montreal, H3C 3J7, Qc, Canada
morinf@iro.umontreal.ca

Abstract
In recent years, variants of a neural network architecture for statistical language modeling havebeen proposed and successfully applied, e.g. in
the language modeling component of speech recognizers. The main advantage of these architectures is that they learn an embedding for words(or other symbols) in a continuous space thathelps to smooth the language model and provide good generalization even when the number of training examples is insufficient. However, these models are extremely slow in comparison to the more commonly used n-gram models, both for training and recognition. As an alternative to an importance sampling method proposed to speed-up training, we introduce a hierarchical decomposition of the conditional probabilities that yields a speed-up of about 200 bothduring training and recognition. The hierarchicaldecomposition is a binary hierarchical clustering constrained by the prior knowledge extracted
from the WordNet semantic hierarchy.

1 INTRODUCTION
The curse of dimensionality hits hard statistical language
models because the number of possible combinations of nwords from a dictionnary (e.g. of 50,000 words) is immensely larger than all the text potentially available, at least
for n > 2. The problem comes down to transfering probability mass from the tiny fraction of observed cases to all
the other combinations. From the point of view of machinelearning, it is interesting to consider the different principles
at work in obtaining such generalization. The most fundamental principle, used explicitly in non-parametric models, is that of similarity: if two objects are similar theyshould have a similar probability. Unfortunately, using aknowledge-free notion of similarity does not work well in

Yoshua Bengio
Dept. IRO, Université de Montréal
P.O. Box 6128, Succ. Centre-Ville,
Montreal, H3C 3J7, Qc, Canada
Yoshua.Bengio@umontreal.ca

high-dimensional spaces such as sequences of words. In
the case of statistical language models, the most successful generalization principle (and corresponding notion ofsimilarity) is also a very simple one, and it is used in interpolated and back-off n-gram models (Jelinek and Mercer, 1980; Katz, 1987): sequences that share shorter subsequences are similar and should share probability mass.
However, these methods are based on exact matchingof subsequences, whereas it is obvious that two word sequences may not match and yet be very close to each othersemantically. Taking this into account, another principlethat has been shown to be very successful (in combination with the first one) is based on a notion of similarity
between individual words: two word sequences are said
to be “similar” if their corresponding words are “similar”.
Similarity between words is usually defined using wordclasses (Brown et al., 1992; Goodman, 2001b). Theseword classes correspond to a partition of the set of words
in such a way that words in the same class share statistical properties in the context of their use, and this partition
can be obtained with various clustering algorithms. This isa discrete all-or-nothing notion of similarity. Another way
to define similarity between words is based on assigningeach word to a continuous-valued set of features, and comparing words based on this feature vector. This idea has
already been exploited in information retrieval (Schutze,
1993; Deerwester et al., 1990) using a singular value decomposition of a matrix of occurrences, indexed by words
in one dimension and by documents in the other.
This idea has also been exploited in (Bengio, Ducharme
and Vincent, 2001; Bengio et al., 2003): a neural networkarchitecture is defined in which the first layer maps word
symbols to their continuous representation as feature vectors, and the rest of the neural network is conventional andused to construct conditional probabilities of the next word
given the previous ones. This model is described in detail in Section 2. The idea is to exploit the smoothness ofthe neural network to make sure that sequences of words
that are similar according to this learned metric will be assigned a similar probability. Note that both the feature vec-

tors and the part of the model that computes probabilities
from them are estimated jointly, by regularized maximum
likelihood. This type of model is also related to the popularmaximum entropy models (Berger, Della Pietra and Della
Pietra, 1996) since the latter correspond to a neural networkwith no hidden units (the unnormalized log-probabilities
are linear functions of the input indicators of presence ofwords).
This neural network approach has been shown to generalize well in comparison to interpolated n-gram models andclass-based n-grams (Brown et al., 1992; Pereira, Tishby
and Lee, 1993; Ney and Kneser, 1993; Niesler, Whittakerand Woodland, 1998; Baker and McCallum, 1998), bothin terms of perplexity and in terms of classification error
when used in a speech recognition system (Schwenk andGauvain, 2002; Schwenk, 2004; Xu, Emami and Jelinek,
2003). In (Schwenk and Gauvain, 2002; Schwenk, 2004),
it is shown how the model can be used to directly improve
speech recognition performance. In (Xu, Emami and Jelinek, 2003), the approach is generalized to form the various conditional probability functions required in a stochastic parsing model called the Structured Language Model,
and experiments also show that speech recognition performance can be improved over state-of-the-art alternatives.
However, a major weakness of this approach is the very
long training time as well as the large amount of computations required to compute probabilities, e.g. at the timeof doing speech recognition (or any other application of
the model). Note that such models could be used in otherapplications of statistical language modeling, such as automatic translation and information retrieval, but improving
speed is important to make such applications possible.
The objective of this paper is thus to propose amuch faster variant of the neural probabilistic language
model. It is based on an idea that could in principledeliver close to exponential speed-up with respect to thenumber of words in the vocabulary. Indeed the computations required during training and during probability prediction are a small constant plus a factor linearly proportional to the number of words |V | in the vocabulary V .
The approach proposed here can yield a speed-up of orderO( log|V|V| | ) for the second term. It follows up on a proposal
made in (Goodman, 2001b) to rewrite a probability function based on a partition of the set of words. The basic
idea is to form a hierarchical description of a word as a sequence of O(log |V |) decisions, and to learn to take theseprobabilistic decisions instead of directly predicting eachword’s probability. Another important idea of this paper
is to reuse the same model (i.e. the same parameters) for
all those decisions (otherwise a very large number of models would be required and the whole model would not fit
in computer memory), using a special symbolic input thatcharacterizes the nodes in the tree of the hierarchical decomposition. Finally, we use prior knowledge in the Word-

Net lexical reference system to help define the hierarchy ofword classes.

2 PROBABILISTIC NEURALLANGUAGE MODEL
The objective is to estimate the joint probability of sequences of words and we do it through the estimation of theconditional probability of the next word (the target word)
given a few previous words (the context):
Y
P (w1 , . . . , wl ) =P (wt |wt−1 , . . . , wt−n+1 ),twhere wt is the word at position t in a text and wt ∈ V ,the vocabulary. The conditional probability is estimatedby
P a normalized function f (wt , wt−1 , . . . , wt−n+1 ), withv f (v, wt−1 , . . . , wt−n+1 ) = 1.
In (Bengio, Ducharme and Vincent, 2001; Bengio et al.,2003) this conditional probability function is represented
by a neural network with a particular structure. Its mostimportant characteristic is that each input of this function(a word symbol) is first embedded into a Euclidean space(by learning to associate a real-valued “feature vector” toeach word). The set of feature vectors for all the words
in V is part of the set of parameters of the model, estimated by maximizing the empirical log-likelihood (minus
weight decay regularization). The idea of associating eachsymbol with a distributed continuous representation is not
new and was advocated since the early days of neural networks (Hinton, 1986; Elman, 1990). The idea of using neural networks for language modeling is not new either (Miikkulainen and Dyer, 1991; Xu and Rudnicky, 2000), andis similar to proposals of character-based text compression
using neural networks to predict the probability of the next
character (Schmidhuber, 1996).
There are two variants of the model in (Bengio,
Ducharme and Vincent, 2001; Bengio et al., 2003): one
with |V | outputs with softmax normalization (and the target
word wt is not mapped to a feature vector, only the context
words), and one with a single output which represents theunnormalized probability for wt given the context words.
Both variants gave similar performance in the experiments
reported in (Bengio, Ducharme and Vincent, 2001; Bengio
et al., 2003). We will start from the second variant here,
which can be formalized as follows, using the Boltzmann
distribution form, following (Hinton, 2000):e−g(wt ,wt−1 ,...,wt−n+1 )f (wt , wt−1 , . . . , wt−n+1 ) = P −g(v,w ,...,wt−1
t−n+1 )
ve
where g(v, wt−1 , . . . , wt−n+1 ) is a learned function thatcan be interpreted as an energy, which is low when the tuple(v, wt−1 , . . . , wt−n+1 ) is “plausible”.

Let F be an embedding matrix (a parameter) with row
Fi the embedding (feature vector) for word i. The above
energy function is represented by a first transformation of
the input label through the feature vectors Fi , followed byan ordinary feedforward neural network (with a single output and a bias dependent on v):
g(v, wt−1 , . . . , wt−n+1 ) = a′ . tanh(c + W x + U Fv′ ) + bv
(1)
where x′ denotes the transpose of x, tanh is applied element by element, a, c and b are parameters vectors, W and
U are weight matrices (also parameters), and x denotes theconcatenation of input feature vectors for context words:
x = (Fwt−1 , . . . , Fwt−n+1 )′ .

(2)

Let h be the number of hidden units (the number of rowsof W ) and d the dimension of the embedding (numberof columns of F ). Computing f (wt , wt−1 , . . . , wt−n+1 )can be done in two steps: first compute c + W x (requires
hd(n − 1) multiply-add operations), and second, for eachv ∈ V , compute U Fv′ (hd multiply-add operations) andthe value of g(v, ...) (h multiply-add operations). Hence
total computation time for computing f is on the order of(n − 1)hd + |V |h(d + 1). In the experiments reportedin (Bengio et al., 2003), n is around 5, |V | is around 20000,
h is around 100, and d is around 30. This gives around12000 operations for the first part (independent of |V |) andaround 60 million operations for the second part (that islinear in |V |).
Our goal in this paper is to drastically reduce the second part, ideally by replacing the O(|V |) computations byO(log |V |) computations.

3 HIERARCHICAL DECOMPOSITIONCAN PROVIDE EXPONENTIAL
SPEED-UP
In (Goodman, 2001b) it is shown how to speed-up a maximum entropy class-based statistical language model byusing the following idea. Instead of computing directly
P (Y |X) (which involves normalization across all the values that Y can take), one defines a clustering partition for
the Y (into the word classes C, such that there is a deterministic function c(.) mapping Y to C), so as to write
P (Y = y|X = x) =P (Y = y|C = c(y), X)P (C = c(y)|X = x).
This is always
P true for any functionPc(.) because
P (Y |X) =i P (Y |C =i P (Y, C = i|X) =i, X)P (C = i|X) = P (Y |C = c(Y ), X)P (C =c(Y )|X) because only one value of C is compatible withthe value of Y , the value C = c(Y ).
Although any c(.) would yield correct probabilities, generalization could be better for choices of word classes that“make sense”, i.e. those for which it easier to learn the
P (C = c(y)|X = x).
If Y can take 10000 values and we have 100 classes with
100 words y in each class, then instead of doing normalization over 10000 choices we only need to do two normalizations, each over 100 choices. If computation of conditional probabilities is proportional to the number of choices
then the above would reduce computation by a factor 50.
This is approximately what is gained according to the measurements reported in (Goodman, 2001b). The same paper suggests that one could introduce more levels to thedecomposition and here we push this idea to the limit. Indeed, whereas a one-level decomposition
should provide apspeed-up on the order of √|V | = |V |, a hierarchical de|V |composition represented by a balanced binary tree should|
provide an exponential speed-up, on the order of log|V |V|
2(at least for the part of the computation that is linear in thenumber of choices).

Each word v must be represented by a bit vector(b1 (v), . . . bm (v)) (where m depends on v). This can beachieved by building a binary hierarchical clustering ofwords, and a method for doing so is presented in the next
section. For example, b1 (v) = 1 indicates that v belongs
to the top-level group 1 and b2 (v) = 0 indicates that it belongs to the sub-group 0 of that top-level group.
The next-word conditional probability can thus be represented and computed as follows:
P (v|wt−1 , . . . , wt−n+1 ) =mY
P (bj (v)|b1 (v), . . . , bj−1 (v), wt−1 , . . . , wt−n+1 )j=1This can be interpreted as a series of binary stochastic
decisions associated with nodes of a binary tree. Each node
is indexed by a bit vector corresponding to the path fromthe root to the node (append 1 or 0 according to whether theleft or right branch of a decision node is followed). Eachleaf corresponds to a word. If the tree is balanced then the
maximum length of the bit vector is ⌈log2 |V |⌉. Note that
we could further reduce computation by looking for an encoding that takes the frequency of words into account, toreduce the average bit length to the unconditional entropy
of words. For example with the corpus used in our experiments, |V | = 10000 so log2 |V | ≈ 13.3 while the unigram
entropy is about 9.16, i.e. a possible additional speed-up
of 31% when taking word frequencies into account to better balance the binary tree. The gain would be greater forlarger vocabularies, but not a very significant improvement
over the major one obtained by using a simple balanced
hierarchy.

sense to force the word embedding to be shared across all
nodes. This is important also because the matrix of word
features F is the largest component of the parameter set.

The “target class” (0 or 1) for each node is obtained directly from the target word in each context, using the bit
encoding of that word. Note also that there will be a target(and gradient propagation) only for the nodes on the pathfrom the root to the leaf associated with the target word.
This is the major source of savings during training.

Since each node in the hierarchy presumably has a semantic meaning (being associated with a group of hopefully similar-meaning words) it makes sense to also associate each node with a feature vector. Without lossof generality, we can consider the model to predict
P (b|node, wt−1 , . . . , wt−n+1 ) where node corresponds toa sequence of bits specifying a node in the hierarchy and b
is the next bit (0 or 1), corresponding to one of the two children of node. This can be represented by a model similar tothe one described in Section 2 and (Bengio, Ducharme andVincent, 2001; Bengio et al., 2003) but with two kinds ofsymbols in input: the context words and the current node.
We allow the embedding parameters for word cluster nodes
to be different from those for words. Otherwise the architecture is the same, with the difference that there are only
two choices to predict, instead of |V | choices.

During recognition and testing, there are two main cases
to consider: one needs the probability of only one word,e.g. the observed word, (or very few) , or one needs theprobabilities of all the words. In the first case (which occurs during testing on a corpus) we still obtain the exponential speed-up. In the second case, we are back to O(|V |)computations (with a constant factor overhead). For the
purpose of estimating generalization performance (out-ofsample log-likelihood) only the probability of the observed
next word is needed. And in practical applications such as
speech recognition, we are only interested in discriminating between a few alternatives, e.g. those that are consistent
with the acoustics, and represented in a treillis of possible
word sequences.
This speed-up should be contrasted with the one
provided by the importance sampling method proposedin (Bengio and Senécal, 2003). The latter method is basedon the observation that the log-likelihood gradient is theaverage over the model’s distribution for P (v|context) of
the energy gradient associated with all the possible nextwords v. The idea is to approximate this average by abiased (but asymptotically unbiased) importance sampling
scheme. This approach can lead to significant speed-up
during training, but because the architecture is unchanged,
probability computation during recognition and test still requires O(|V |) computations for each prediction. Instead,the architecture proposed here gives significant speed-up
both during training and test / recognition.

4 SHARING PARAMETERS ACROSS
THE HIERARCHY
If a separate predictor is used for each of the nodes in thehierarchy, about 2|V | predictors will be needed. This represents a huge capacity since each predictor maps from thecontext words to a single probability. This might create
problems in terms of computer memory (not all the models
would fit at the same time in memory) as well as overfitting.
Therefore we have chosen to build a model in which parameters are shared across the hierarchy. There are clearly
many ways to achieve such sharing, and alternatives to thearchitecture presented here should motivate further study.
Based on our discussion in the introduction, it makes

More precisely, the specific predictor used in our experiments is the following:

P (b = 1|node, wt−1 , . . . , wt−n+1 ) =sigmoid(αnode + β ′ . tanh(c + W x + U Nnode))
where x is the concatenation of context word features
as in eq. 2, sigmoid(y) = 1/(1 + exp(−y)), αi is a bias
parameter playing the same role as bv in eq. 1, β is a weightvector playing the same role as a in eq. 1, c, W , U and F
play the same role as in eq. 1, and N gives feature vector
embeddings for nodes in a way similar that F gave feature
vector embeddings for next-words in eq. 1.

5 USING WORDNET TO BUILD THE
HIERARCHICAL DECOMPOSITION
A very important component of the whole model is the
choice of the words binary encoding, i.e. of the hierarchical word clustering. In this paper we combine empirical statistics with prior knowledge from the WordNet resource (Fellbaum, 1998). Another option would have been
to use a purely data-driven hierarchical clustering of words,
and there are many other ways in which the WordNet resource could have been used to influence the resulting clustering.
The IS-A taxonomy in WordNet organizes semantic
concepts associated with senses in a graph that is almost atree. For our purposes we need a tree, so we have manually
selected a parent for each of the few nodes that have morethan one parent. The leaves of the WordNet taxonomy aresenses and each word can be associated with more than one

sense. Words sharing the same sense are considered to besynonymous (at least in one of their uses). For our purpose we have to choose one of the senses for each word
(to make the whole hierarchy one over words) and we selected the most frequent sense. A straightforward extensionof the proposed model would keep the semantic ambiguityof each word: each word would be associated with several leaves (senses) of the WordNet hierarchy. This would
require summing over all those leaves (and corresponding
paths to the root) when computing next-word probabilities.
Note that the WordNet tree is not binary: each node
may have many more than two children (this is particularly a problem for verbs and adjectives, for which WordNet is shallow and incomplete). To transform this hierarchy into a binary tree we perform a data-driven binary hierarchical clustering of the children associated with eachnode, as illustrated in Figure 1. The K-means algorithm isused at each step to split each cluster. To compare nodes,
we associate each node with the subset of words that itcovers. Each word is associated with a TF/IDF (Saltonand Buckley, 1988) vector of document/word occurrence
counts, where each “document” is a paragraph in the training corpus. Each node is associated with the dimensionwise median of the TF/IDF scores. Each TF/IDF scoreis the occurrence frequency of the word in the document
times the logarithm of the ratio of the total number of documents by the number of documents containing the word.

111
000
000
111
000
111
000
111
000
111

111
000
000
111
000
111
000
111
000
111

6 COMPARATIVE RESULTS
Experiments were performed to evaluate the speed-up andany change in generalization error. The experiments also
compared an alternative speed-up technique (Bengio andSenécal, 2003) that is based on importance sampling (butonly provides a speed-up during training). The experiments
were performed on the Brown corpus, with a reduced vocabulary size of 10,000 words (the most frequent ones).
The corpus has 1,105,515 occurrences of words, split into
3 sets: 900,000 for training, 100,000 for validation (model
selection), and 105,515 for testing. The validation set wasused to select among a small number of choices for the sizeof the embeddings and the number of hidden units.
The results in terms of raw computations (time to process one example), either during training or during test
are shown respectively in Tables 1 and 2. The computations were performed on Athlon processors with a 1.2 GHz
clock. The speed-up during training is by a factor greaterthan 250 and during test by a factor close to 200. These areimpressive but less than the |V |/ log2 |V | ≈ 750 that could
be expected if there was no overhead and no constant termin the computational cost.
It is also important to verify that learning still works

Figure 1: WordNet’s IS-A hierarchy is not a binary tree:
most nodes have many children. Binary hierarchical clustering of these children is performed.

and that the model generalizes well. As usual in statistical language modeling this is measured by the model’s
perplexity on the test data, which is the exponential of
the average negative log-likehood on that data set. Training is performed over about 20 to 30 epochs according tovalidation set perplexity (early stopping). Table 3 shows
the comparative generalization performance of the different architectures, along with that of an interpolated trigram
and a class-based n-gram (same procedures as in (Bengio
et al., 2003), which follow respectively (Jelinek and Mercer, 1980) and (Brown et al., 1992; Ney and Kneser, 1993;
Niesler, Whittaker and Woodland, 1998)). The validation
set was used to choose the order of the n-gram and thenumber of word classes for the class-based models. We
used the implementation of these algorithms in the SRI
Language Modeling toolkit, described by (Stolcke, 2002)
and in www.speech.sri.com/projects/srilm/.
Note that better performance should be obtainable withsome of the tricks in (Goodman, 2001a). Combining theneural network with a trigram should also decrease its per-

architecture
original neural net
importance sampling
hierarchical modelTime perepoch (s)416 300
6 062
1 609

Time perex. (ms)
462.6
6.73
1.79

speed-up
1
68.7
258

Table 1: Training time per epoch (going once through all
the training examples) and per example. The original neural net is as described in sec. 2. The importance sampling algorithm (Bengio and Senécal, 2003) trains the samemodel faster. The hierarchical model is the one proposed
here, and it yields a speed-up not only during training but
for probability predictions as well (see the next table).

architecture
original neural net
importance sampling
hierarchical modelTime perexample (ms)
270.7
221.3
1.4

speed-up
1
1.22
193

Table 2: Test time per example for the different algorithms.
See Table 1’s caption. It is at test time that the hierarchicalmodel’s advantage becomes clear in comparison to the importance sampling technique, since the latter only brings a
speed-up during training.

plexity, as already shown in (Bengio et al., 2003).
As shown in Table 3, the hierarchical model does not
generalize as well as the original neural network, but thedifference is not very large and still represents an improvement over the benchmark n-gram models. Given the verylarge speed-up, it is certainly worth investigating variationsof the hierarchical model proposed here (in particular how
to define the hierarchy) for which generalization could bebetter. Note also that the speed-up would be greater forlarger vocabularies (e.g. 50,000 is not uncommon in speech
recognition systems).

trigramclass-based
original neural net
importance sampling
hierarchical model

Validation
perplexity
299.4
276.4
213.2
209.4
241.6

Test
perplexity
268.7
249.1
195.3
192.6
220.7

Table 3: Test perplexity for the different architectures andfor an interpolated trigram. The hierarchical model performed a bit worse than the original neural network, but
is still better than the baseline interpolated trigram and theclass-based model.

From a linguistic point of view, one of the weaknessesof the above model is that it considers word clusters as deterministic functions of the word, but uses the nodes in
WordNet’s taxonomy to help define those clusters. However, WordNet provides word sense ambiguity information
which could be used for linguistically more accurate modeling. The hierarchy would be a sense hierarchy instead ofa word hiearchy, and each word would be associated witha number of senses (those allowed for that word in WordNet). In computing probabilities, this would involve summing over several paths from the root, corresponding to thedifferent possible senses of the word. As a side effect, this
could provide a word sense disambiguation model, and it
could be trained both on sense-tagged supervised data andon unlabeled ordinary text. Since the average number ofsenses per word is small (less than a handful), the loss in
speed would correspondingly be small.
Acknowledgments
The authors would like to thank the following funding organizations for support: NSERC, MITACS, IRIS, and theCanada Research Chairs.

References
7 CONCLUSION AND FUTURE WORK
Baker, D. and McCallum, A. (1998). Distributional clustering of words for text classification. In SIGIR’98.
This paper proposes a novel architecture for speeding-up
neural networks with a huge number of output classes andshows its usefulness in the context of statistical language
modeling (which is a component of speech recognition andautomatic translation systems). This work pushes to thelimit a suggestion of (Goodman, 2001b) but also introduces the idea of sharing the same model for all nodes ofthe decomposition, which is more practical when the number of nodes is very large (tens of thousands here). The
implementation and the experiments show that a very significant speed-up of around 200-fold can be achieved, with
only a little degradation in generalization performance.

Bengio, Y., Ducharme, R., and Vincent, P. (2001). A neural probabilistic language model. In Leen, T., Dietterich, T., and Tresp, V., editors, Advances in Neural
Information Processing Systems 13 (NIPS’00), pages
933–938. MIT Press.
Bengio, Y., Ducharme, R., Vincent, P., and Jauvin, C.
(2003). A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155.
Bengio, Y. and Senécal, J.-S. (2003). Quick training ofprobabilistic neural nets by importance sampling. In

Proceedings of AISTATS 2003.
Berger, A. L., Della Pietra, V. J., and Della Pietra, S. A.
(1996). A maximum entropy approach to natural language processing. Computational Linguistics, 22:39–
71.

Niesler, T. R., Whittaker, E. W. D., and Woodland, P. C.
(1998). Comparison of part-of-speech and automatically derived category-based language models forspeech recognition. In International Conference on
Acoustics, Speech and Signal Processing (ICASSP),
pages 177–180.

Brown, P. F., Pietra, V. J. D., DeSouza, P. V., Lai, J. C.,
and Mercer, R. L. (1992). Class-based n-gram models of natural language. Computational Linguistics,
18:467–479.

Pereira, F., Tishby, N., and Lee, L. (1993). Distributional
clustering of english words. In 30th Annual Meeting of the Association for Computational Linguistics,
pages 183–190, Columbus, Ohio.

Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer,
T. K., and Harshman, R. (1990). Indexing by latentsemantic analysis. Journal of the American Society
for Information Science, 41(6):391–407.

Salton, G. and Buckley, C. (1988). Term weighting approaches in automatic text retrieval. Information Processing and Management, 24(5):513–523.

Elman, J. L. (1990). Finding structure in time. Cognitive
Science, 14:179–211.
Fellbaum, C. (1998). WordNet: An Electronic Lexical
Database. MIT Press.
Goodman, J. (2001a). A bit of progress in language modeling. Technical Report MSR-TR-2001-72, Microsoft
Research, Redmond, Washington.
Goodman, J. (2001b). Classes for fast maximum entropy
training. In International Conference on Acoustics,
Speech and Signal Processing (ICASSP), Utah.

Schmidhuber, J. (1996). Sequential neural text compression. IEEE Transactions on Neural Networks,
7(1):142–146.
Schutze, H. (1993). Word space. In Giles, C., Hanson, S.,
and Cowan, J., editors, Advances in Neural Information Processing Systems 5 (NIPS’92), pages 895–902,
San Mateo CA. Morgan Kaufmann.
Schwenk, H. (2004). Efficient training of large neural networks for language modeling. In International Joint
Conference on Neural Networks (IJCNN), volume 4,
pages 3050–3064.

Hinton, G. E. (1986). Learning distributed representationsof concepts. In Proceedings of the Eighth Annual
Conference of the Cognitive Science Society, pages 1–
12, Amherst 1986. Lawrence Erlbaum, Hillsdale.

Schwenk, H. and Gauvain, J.-L. (2002). Connectionist
language modeling for large vocabulary continuous
speech recognition. In International Conference on
Acoustics, Speech and Signal Processing (ICASSP),
pages 765–768, Orlando, Florida.

Hinton, G. E. (2000). Training products of experts byminimizing contrastive divergence. Technical Report
GCNU TR 2000-004, Gatsby Unit, University College London.

Stolcke, A. (2002). SRILM - an extensible language modeling toolkit. In Proceedings of the International Conference on Statistical Language Processing, Denver,
Colorado.

Jelinek, F. and Mercer, R. L. (1980). Interpolated estimation of Markov source parameters from sparse data.
In Gelsema, E. S. and Kanal, L. N., editors, Pattern
Recognition in Practice. North-Holland, Amsterdam.

Xu, P., Emami, A., and Jelinek, F. (2003). Training connectionist models for the structured language
model. In Proceedings of the 2003 Conference on
Empirical Methods in Natural Language Processing(EMNLP’2003), volume 10, pages 160–167.

Katz, S. M. (1987). Estimation of probabilities from sparse
data for the language model component of a speech
recognizer. IEEE Transactions on Acoustics, Speech,
and Signal Processing, ASSP-35(3):400–401.
Miikkulainen, R. and Dyer, M. G. (1991). Natural language processing with modular PDP networks anddistributed lexicon. Cognitive Science, 15:343–399.
Ney, H. and Kneser, R. (1993). Improved clustering techniques for class-based statistical language modelling.
In European Conference on Speech Communication
and Technology (Eurospeech), pages 973–976, Berlin.

Xu, W. and Rudnicky, A. (2000). Can artificial neuralnetworks learn language models. In International
Conference on Statistical Language Processing, pages
M1–13, Beijing, China.