The experiment results are showed in Table TABREF9 which suggest that, on a small corpus likes the Brown Corpus, RNNLM and LSTM-RNN did not show a remarkable advantage over FNNLM, instead a bit higher perplexity was achieved by LSTM-RNNLM. Maybe more data is needed to train RNNLM and LSTM-RNNLM because longer dependencies are taken into account by RNNLM and LSTM-RNNLM when predicting next word. LSTM-RNNLM with bias terms or direct connections was also evaluated here. When the direct connections between input layer and output layer of LSTM-RNN are enabled, a slightly higher perplexity but shorter training time were obtained. An explanation given for this phenomenon by  is that direct connections provide a bit more capacity and faster learning of the "linear" part of mapping from inputs to outputs but impose a negative effect on generalization. For bias terms, no significant improvement on performance was gained by adding bias terms which was also observed on RNNLM by  . In the rest of this paper, all studies will be performed on LSTM-RNNLM with neither direct connections nor bias terms, and the result of this model in Table TABREF9 will be used as the baseline for the rest studies.

Future Work	Various architectures of neural network language models are described and a number of improvement techniques are evaluated in this paper, but there are still something more should be included, like gate recurrent unit (GRU) RNNLM, dropout strategy for addressing overfitting, character level neural network language model and ect. In addition, the experiments in this paper are all performed on Brown Corpus which is a small corpus, and different results may be obtained when the size of corpus becomes larger. Therefore, all the experiments in this paper should be repeated on a much larger corpus. Several limits of NNLM has been explored, and, in order to achieve language understanding, these limits must be overcome. I have not come up with a complete solution yet but some ideas which will be explored further next. First, the architecture showed in Figure FIGREF19 can be used as a general improvement scheme for ANN, and I will try to figure out the structure of changeless neural network for encoder. What's more, word sequences are commonly taken as signals for LM, and it is easy to take linguistical properties of words or sentences as the features of signals. However, it maybe not a proper way to deal with natural languages.

Conclusion	In this paper, different architectures of neural network language models were described, and the results of comparative experiment suggest RNNLM and LSTM-RNNLM do not show any advantages over FNNLM on small corpus. The improvements over these models, including importance sampling, word classes, caching and BiRNN, were also introduced and evaluated separately, and some interesting findings were proposed which can help us have a better understanding of NNLM. Another significant contribution in this paper is the exploration on the limits of NNLM from the aspects of model architecture and knowledge representation. Although state of the art performance has been achieved using NNLM in various NLP tasks, the power of NNLM has been exaggerated all the time. The main idea of NNLM is to approximate the probabilistic distribution of word sequences in a natural language using ANN. NNLM can be successfully applied in some NLP tasks where the goal is to map input sequences into output sequences, like speech recognition, machine translation, tagging and ect. However, language understanding is another story. For language understanding, word sequences must be linked with any concrete or abstract objects in real world which cannot be achieved just with this probabilistic distribution.

Limits of Neural Network Language Modeling	NNLM is state of the art, and has been introduced as a promising approach to various NLP tasks. Numerous researchers from different areas of NLP attempt to improve NNLM, expecting to get better performance in their areas, like lower perplexity on test data, less word error rate (WER) in speech recognition, higher Bilingual Evaluation Understudy (BLEU) score in machine translation and etc. However, few of them spares attention on the limits of NNLM. Without a thorough understanding of NNLM's limits, the applicable scope of NNLM and directions for improving NNLM in different NLP tasks cannot be defined clearly. In this section, the limits of NNLM will be studied from two aspects: model architecture and knowledge representation.

A Study on Neural Network Language Modeling	An exhaustive study on neural network language modeling (NNLM) is performed in this paper. Different architectures of basic neural network language models are described and examined. A number of different improvements over basic neural network language models, including importance sampling, word classes, caching and bidirectional recurrent neural network (BiRNN), are studied separately, and the advantages and disadvantages of every technique are evaluated. Then, the limits of neural network language modeling are explored from the aspects of model architecture and knowledge representation. Part of the statistical information from a word sequence will loss when it is processed word by word in a certain order, and the mechanism of training neural network by updating weight matrixes and vectors imposes severe restrictions on any significant enhancement of NNLM. For knowledge representation, the knowledge represented by neural network language models is the approximate probabilistic distribution of word sequences from a certain training data set rather than the knowledge of a language itself or the information conveyed by word sequences in a natural language. Finally, some directions for improving neural network language modeling further is discussed.

Bidirectional Recurrent Neural Network	In  , significant improvement on neural machine translation (NMT) for an English to French translation task was achieved by reversing the order of input word sequence, and the possible explanation given for this phenomenon was that smaller "minimal time lag" was obtained in this way. In my opinion, another possible explanation is that a word in word sequence may more statistically depend on the following context than previous one. After all, a number of words are determined by its following words instead of previous ones in some natural languages. Take the articles in English as examples, indefinite article "an" is used when the first syllable of next word is a vowel while "a" is preposed before words starting with consonant. What's more, if a noun is qualified by an attributive clause, definite article "the" should be used before the noun. These examples illustrate that words in a word sequence depends on their following words sometimes.

Knowledge Representation	The word "learn" appears frequently with NNLM, but what neural network language models learn from training data set is rarely analyzed carefully. The common statement about the knowledge learned by neural network language models is the probabilistic distribution of word sequences in a natural language. Strictly speaking, it is the probabilistic distribution of word sequences from a certain training data set in a natural language, rather than the general one. Hence, the neural network language model trained on data set from a certain field will perform well on data set from the same field, and neural network language model trained on a general data set may show worse performance when tested on data set from a special field. In order to verify this, one million words reviews on electronics and books were extracted from Amazon reviews  ,  respectively as data sets from different fields, and 800000 words for training, 100000 words for validation, and the rest for test. In this experiment, two models were trained on training data from electronics reviews and books reviews respectively, and the other one was trained on both. Then, all three models were tested on the two test data sets.

Model Architecture	In most language models including neural network language models, words are predicated one by one according to their previous context or following one which is believed to simulate the way human deal with natural languages, and, according to common sense, human actually speak or write word by word in a certain order. However, the intrinsic mechanism in human mind of processing natural languages cannot like this way. As mentioned above, it is not always true that words in a word sequence only depend on their previous or following context. In fact, before human speaking or writing, they know what they want to express and map their ideas into word sequence, and the word sequence is already cached in memory when human speaking or writing. In most case, the cached word sequence may be not a complete sentence but at least most part of it. On the other hand, for reading or listening, it is better to know both side context of a word when predicting the meaning of the word or define the grammar properties of the word. Therefore, it is not a good strategy to deal with word sequences in a natural language word by word in a certain order which has also been questioned by the success application of BiRNN in some NLP tasks.

Feed-forward Neural Network Language Model, FNNLM	As mentioned above, the objective of FNNLM is to evaluate the conditional probability INLINEFORM0 , but feed-forward neural network (FNN) lacks of an effective way to represent history context. Hence, the idea of n-gram based LM is adopted in FNNLM that words in a word sequence more statistically depend on the words closer to them, and only the INLINEFORM1 direct predecessor words are considered when evaluating the conditional probability, this is: INLINEFORM2  The architecture of the original FNNLM proposed by  is showed in Figure FIGREF2 , and INLINEFORM0 , INLINEFORM1 are the start and end marks of a word sequence respectively. In this model, a vocabulary is pre-built from a training data set, and every word in this vocabulary is assigned with a unique index. To evaluate the conditional probability of word INLINEFORM2 , its INLINEFORM3 direct previous words INLINEFORM4 are projected linearly into feature vectors using a shared matrix INLINEFORM5 according to their index in the vocabulary, where INLINEFORM6 is the size of the vocabulary and INLINEFORM7 is the feature vectors' dimension.

All nodes of neural network in a neural network language model have parameters needed to be tunning during training, so the training of the model will become very difficult or even impossible if the model's size is too large. However, an efficient way to enhance the performance of a neural network language model is to increase the size of model. One possible way to address this problem is to implement special functions, like encoding, using changeless neural network with special struture. Not only the size of the changeless neural network can be very large, but also the structure can be very complexity. The performance of NNLM, both perplexity and training time, is expected to be improved dramatically in this way.

Nevertheless, BiRNN cannot be evaluated in LM directly as unidirectional RNN, because statistical language modeling is based on the chain rule which assumes that word in a word sequence only statistically depends on one side context. BiRNN can be applied in NLP tasks, like speech recognition and machine translation, because the input word sequences in these tasks are treated as a whole and usually encoded as a single vector. The architecture for encoding input word sequences using BiRNN is showed in Figure FIGREF18 . The facts that better performance can be achieved using BiRNN in speech recognition or machine translation indicate that a word in a word sequence is statistically determined by the words of its both side, and it is not a suitable way to deal with word sequence in a natural language word by word in an order.

Therefore, NNLM can be a good choice for NLP tasks in some special fields where language understanding is not necessary. Language understanding cannot be achieved just with the probabilistic distribution of word sequences in a natural language, and new kind of knowledge representation should be raised for language understanding. Since the training of neural network language model is really expensive, it is important for a well-trained neural network language model to keep learning during test or be improved on other training data set separately. However, the neural network language models built so far do not show this capacity. Lower perplexity can be obtained when the parameters of a trained neural network language model are tuned dynamically during test, as showed in Table TABREF21 , but this does not mean neural network language model can learn dynamically during test. ANN is just a numerical approximation method in nature, and it approximate the target function, the probabilistic distribution of word sequences for LM, by tuning parameters when trained on data set. The learned knowledge is saved as weight matrixes and vectors. When a trained neural network language model is expected to adaptive to new data set, it should be retrained on both previous training data set and new one.

In addition, it cannot be applied into RNNLM or LSTM-RNNLM directly and other simpler and more efficient speed-up techniques have been proposed now.

Inspired by the first caching technique, if the previous context can be taken into account through the internal states of RNN, the perplexity is expected to decrease. In this paper, all language models are trained sentence by sentence, and the initial states of RNN are initialized using a constant vector. This caching technique can be implemented by simply initializing the initial states using the last states of direct previous sentence in the same article. However, the experiment result (Table TABREF15 ) shows this caching technique did not work as excepted and the perplexity even increased slightly. Maybe, the Brown Corpus is too small and more data is needed to evaluated this caching technique, as more context is taken into account with this caching technique.

Introduction	Generally, a well-designed language model makes a critical difference in various natural language processing (NLP) tasks, like speech recognition  ,  , machine translation  ,  , semantic extraction  ,  and etc. Language modeling (LM), therefore, has been the research focus in NLP field all the time, and a large number of sound research results have been published in the past decades. N-gram based LM  , a non-parametric approach, is used to be state of the art, but now a parametric method - neural network language modeling (NNLM) is considered to show better performance and more potential over other LM techniques, and becomes the most commonly used LM technique in multiple NLP tasks. Although some previous attempts  ,  ,  had been made to introduce artificial neural network (ANN) into LM, NNLM began to attract researches' attentions only after  and did not show prominent advantages over other techniques of LM until recurrent neural network (RNN) was investigated for NNLM  ,  . After more than a decade's research, numerous improvements, marginal or critical, over basic NNLM have been proposed.

When INLINEFORM2 are given, INLINEFORM3 indicates that word INLINEFORM4 belongs to the sub-group 0 of current node and INLINEFORM5 indicates it belongs to the other one. The conditional probability of every word is represented as: INLINEFORM6  Theoretically, an exponential speed-up, on the order of INLINEFORM0 , can be achieved with this hierarchical architecture. In  , impressive speed-up during both training and test, which were less than the theoretical one, were obtained but an obvious increase in PPL was also observed. One possible explanation for this phenomenon is that the introduction of hierarchical architecture or word classes impose negative influence on the word classification by neural network language models. As is well known, a distribution representation for words, which can be used to represent the similarities between words, is formed by neural network language models during training. When words are clustered into classes, the similarities between words from different classes cannot be recognized directly. For a hierarchical clustering of words, words are clustered more finely which might lead to worse performance, i.e., higher perplexity, and deeper the hierarchical architecture is, worse the performance would be.

Word Classes	Before the idea of word classes was introduced to NNLM, it had been used in LM extensively for improving perplexities or increasing speed  ,  . With word classes, every word in vocabulary is assigned to a unique class, and the conditional probability of a word given its history can be decomposed into the probability of the word's class given its history and the probability of the word given its class and history, this is: INLINEFORM0  where INLINEFORM0 is the class of word INLINEFORM1 . The architecture of class based LSTM-RNNLM is illustrated in Figure FIGREF12 , and INLINEFORM2 , INLINEFORM3 are the lower and upper index of words in a class respectively.   extended word classes to a hierarchical binary clustering of words and built a hierarchical neural network language model. In hierarchical neural network language model, instead of assigning every word in vocabulary with a unique class, a hierarchical binary tree of words is built according to the word similarity information extracted from WordNet  , and every word in vocabulary is assigned with a bit vector INLINEFORM0 , INLINEFORM1 .

On the other hand, word classes consist of words with low frequency are much bigger which causes more training time. However, as the experiment results show, both perplexity and training time were improved when words were classified according to their sqrt frequency, because word classes were more uniform when built in this way. All other models in this paper were speeded up using word classes, and words were clustered according to their sqrt frequencies.

Recurrent Neural Network Language Model, RNNLM	The idea of applying RNN in LM was proposed much earlier  ,  , but the first serious attempt to build a RNNLM was made by  ,  . RNNs are fundamentally different from feed-forward architectures in the sense that they operate on not only an input space but also an internal state space, and the state space enables the representation of sequentially extended dependencies. Therefore, arbitrary length of word sequence can be dealt with using RNNLM, and all previous context can be taken into account when predicting next word. As showed in Figure FIGREF5 , the representation of words in RNNLM is the same as that of FNNLM, but the input of RNN at every step is the feature vector of a direct previous word instead of the concatenation of the INLINEFORM0 previous words' feature vectors and all other previous words are taken into account by the internal state of previous step. At step INLINEFORM1 , RNN can be described as: DISPLAYFORM0  where, weight matrix INLINEFORM0 , and the input layer's size of RNN INLINEFORM1 . The outputs of RNN are also unnormalized probabilities and should be regularized using a softmax layer.

In fact, every row of projection matrix INLINEFORM8 is a feature vector of a word in the vocabulary. The input INLINEFORM9 of FNN is formed by concatenating the feature vectors of words INLINEFORM10 , where INLINEFORM11 is the size of FNN's input layer. FNN can be generally represented as: INLINEFORM12  Where, INLINEFORM0 , INLINEFORM1 are weight matrixes, INLINEFORM2 is the size of hidden layer, INLINEFORM3 is the size of output layer, weight matrix INLINEFORM4 is for the direct connections between input layer and output layer, INLINEFORM5 and INLINEFORM6 are vectors for bias terms in hidden layer and output layer respectively, INLINEFORM7 is output vector, and INLINEFORM8 is activation function. The INLINEFORM0 -th element of output vector INLINEFORM1 is the unnormalized conditional probability of the word with index INLINEFORM2 in the vocabulary.

Another limit of NNLM caused by model architecture is original from the monotonous architecture of ANN. In ANN, models are trained by updating weight matrixes and vectors which distribute among all nodes. Training will become much more difficult or even unfeasible when increasing the size of model or the variety of connections among nodes, but it is a much efficient way to enhance the performance of ANN. As is well known, ANN is designed by imitating biological neural system, but biological neural system does not share the same limit with ANN. In fact, the strong power of biological neural system is original from the enormous number of neurons and various connections among neurons, including gathering, scattering, lateral and recurrent connections  . In biological neural system, the features of signals are detected by different receptors, and encoded by low-level central neural system (CNS) which is changeless. The encoded signals are integrated by high-level CNS. Inspired by this, an improvement scheme for the architecture of ANN is proposed, as illustrated in Figure FIGREF19 . The features of signal are extracted according to the knowledge in certain field, and every feature is encoded using changeless neural network with careful designed structure.

To verify this hypothesis further, an experiment is performed here in which the word order of every input sentence is reversed, and the probability of word sequence INLINEFORM0 is evaluated as following: INLINEFORM1  However, the experiment result (Table TABREF17 ) shows that almost the same perplexity was achieved by reversing the order of words. This indicates that the same amount statistical information, but not exactly the same statistical information, for a word in a word sequence can be obtained from its following context as from its previous context, at least for English. As a word in word sequence statistically depends on its both previous and following context, it is better to predict a word using context from its both side. Bidirectional recurrent neural network (BiRNN)  was designed to process data in both directions with two separate hidden layers, so better performance can be expected by using BiRNN. BiRNN was introduced to speech recognition by  , and then was evaluated in other NLP tasks, like NMT  ,  . In these studies, BiRNN showed more excellent performance than unidirectional RNN.

Basic Neural Network Language Models	The goal of statistical language models is to estimate the probability of a word sequence INLINEFORM0 in a natural language, and the probability can be represented by the production of the conditional probability of every word given all the previous ones: INLINEFORM1  where, INLINEFORM0 . This chain rule is established on the assumption that words in a word sequence only statistically depend on their previous context and forms the foundation of all statistical language modeling. NNLM is a kind of statistical language modeling, so it is also termed as neural probabilistic language modeling or neural statistical language modeling. According to the architecture of used ANN, neural network language models can be classified as: FNNLM, RNNLM and LSTM-RNNLM.

However, the existing experimental results of these techniques are not comparable because they were obtained under different experimental setups and, sometimes, these techniques were evaluated combined with other different techniques. Another significant problem is that most researchers focus on achieving a state of the art language model, but the limits of NNLM are rarely studied. In a few works  on exploring the limits of NNLM, only some practical issues, like computational complexity, corpus, vocabulary size, and etc., were dealt with, and no attention was spared on the effectiveness of modeling a natural language using NNLM. Since this study focuses on NNLM itself and does not aim at raising a state of the art language model, the techniques of combining neural network language models with other kind of language models, like N-gram based language models, maximum entropy (ME) language models and etc., will not be included. The rest of this paper is organized as follows: In next section, the basic neural network language models - feed-forward neural network language model (FNNLM), recurrent neural network language model (RNNLM) and long-short term memory (LSTM) RNNLM, will be introduced, including the training and evaluation of these models.

The performance of neural network language models is usually measured using perplexity (PPL) which can be defined as: INLINEFORM0  Perplexity can be defined as the exponential of the average number of bits required to encode the test data using a language model and lower perplexity indicates that the language model is closer to the true model which generates the test data.

Comparison of Neural Network Language Models	Comparisons among neural network language models with different architectures have already been made on both small and large corpus  ,  . The results show that, generally, RNNLMs outperform FNNLMs and the best performance is achieved using LSTM-NNLMs. However, the neural network language models used in these comparisons are optimized using various techniques, and even combined with other kind of language models, let alone the different experimental setups and implementation details, which make the comparison results fail to illustrate the fundamental discrepancy in the performance of neural network language models with different architecture and cannot be taken as baseline for the studies in this paper. Comparative experiments on neural network language models with different architecture were repeated here. The models in these experiments were all implemented plainly, and only a class-based speed-up technique was used which will be introduced later. Experiments were performed on the Brown Corpus, and the experimental setup for Brown corpus is the same as that in  , the first 800000 words (ca01 INLINEFORM0 cj54) were used for training, the following 200000 words (cj55 INLINEFORM1 cm06) for validation and the rest (cn01 INLINEFORM2 cr09) for test.

Caching	Like word classes, caching is also a common used optimization technique in LM. The cache language models are based on the assumption that the word in recent history are more likely to appear again. In cache language model, the conditional probability of a word is calculated by interpolating the output of standard language model and the probability evaluated by caching, like: INLINEFORM0  where, INLINEFORM0 is the output of standard language model, INLINEFORM1 is the probability evaluated using caching, and INLINEFORM2 is a constant, INLINEFORM3 .   combined FNNLM with cache model to enhance the performance of FNNLM in speech recognition, and the cache model was formed based on the previous context as following: INLINEFORM0  where, INLINEFORM0 means Kronecker delta, INLINEFORM1 is the cache length, i.e., the number of previous words taken as cache, INLINEFORM2 is a coefficient depends on INLINEFORM3 which is the distance between previous word and target word. A cache model with forgetting can be obtained by lowering INLINEFORM4 linearly or exponentially respecting to INLINEFORM5 . A class cache model was also proposed by  for the case in which words are clustered into word classes.

The lowest perplexity on each test data set was gained by the model trained on corresponding training data set, instead of the model trained on both training data set (Table TABREF23 ). The results show that the knowledge represented by a neural network language model is the probabilistic distribution of word sequences from training data set which varies from field to field. Except for the probabilistic distribution of word sequences, the feature vectors of words in vocabulary are also formed by neural network during training. Because of the classification function of neural network, the similarities between words can be observed using these feature vectors. However, the similarities between words are evaluated in a multiple dimensional space by feature vectors and it is hard to know which features of words are taken into account when these vectors are formed, which means words cannot be grouped according to any single feature by the feature vectors. In summary, the knowledge represented by neural network language model is the probabilistic distribution of word sequences from certain training data set and feature vectors for words in vocabulary formed in multiple dimensional space. Neither the knowledge of language itself, like grammar, nor the knowledge conveyed by a language can be gained from neural network language models.

In class cache model, the probability of target word given the last recent word classes is determined. However, both word based cache model and class one can be defined as a kind of unigram language model built from previous context, and this caching technique is an approach to combine neural network language model with a unigram model. Another type of caching has been proposed as a speed-up technique for RNNLMs  ,  ,  ,  . The main idea of this approach is to store the outputs and states of language models for future prediction given the same contextual history. In  , four caches were proposed, and they were all achieved by hash lookup tables to store key and value pairs: probability INLINEFORM0 and word sequence INLINEFORM1 ; history INLINEFORM2 and its corresponding hidden state vector; history INLINEFORM3 and the denominator of the softmax function for classes; history INLINEFORM4 , class index INLINEFORM5 and the denominator of the softmax function for words. In  , around 50-fold speed-up was reported with this caching technique in speech recognition but, unfortunately, it only works for prediction and cannot be applied during training.

Importance Sampling	Inspired by the contrastive divergence model  ,  proposed a sampling-based method to speed up the training of neural network language models. In order to apply this method, the outputs of neural network should be normalized in following way instead of using a softmax function: INLINEFORM0  then, neural network language models can be treated as a special case of energy-based probability models. The main idea of sampling based method is to approximate the average of log-likelihood gradient with respect to the parameters INLINEFORM0 by samples rather than computing the gradient explicitly. The log-likelihood gradient for the parameters set INLINEFORM1 can be generally represented as the sum of two parts: positive reinforcement for target word INLINEFORM2 and negative reinforcement for all word INLINEFORM3 , weighted by INLINEFORM4 : INLINEFORM5  Three sampling approximation algorithms were presented by  : Monte-Carlo Algorithm, Independent Metropolis-Hastings Algorithm and Importance Sampling Algorithm. However, only importance sampling worked well with neural network language model. In fact, 19-fold speed-up was achieved during training while no degradation of the perplexities was observed on both training and test data  .

This is another limit of NNLM because of knowledge representation, i.e., neural network language models cannot learn dynamically from new data set.

Natural languages are not natural but man-made, and linguistical knowledge are also created by human long after natural language appeared. Liguistical knowledge only covers the "right" word sequences in a natural language, but it is common to deal with "wrong" ones in real world. In nature, every natural language is a mechanism of linking voices or signs with objects, both concrete and abstract. Therefore, the proper way to deal with natural languages is to find the relations between special voices or signs and objects, and the features of voices or signs can be defined easier than a natural language itself. Every voice or sign can be encoded as a unique code, vector or matrix, according to its features, and the similarities among voices or signs are indeed can be recognized from their codes. It is really difficult to model the relation between voices or signs and objects at once, and this work should be split into several steps. The first step is to covert voice or sign into characters, i.e., speech recognition or image recognition, but it is achieved using the architecture described in Figure FIGREF19 .

In order to guarantee all the conditional probabilities of words positive and summing to one, a softmax layer is always adopted following the output layer of FNN: INLINEFORM3  where INLINEFORM0 is the INLINEFORM1 -th element of output vector INLINEFORM2 , and INLINEFORM3 is the INLINEFORM4 -th word in the vocabulary. Training of neural network language models is usually achieved by maximizing the penalized log-likelihood of the training data: INLINEFORM0  where, INLINEFORM0 is the set of model's parameters to be trained, INLINEFORM1 is a regularization term. The recommended learning algorithm for neural network language models is stochastic gradient descent (SGD) method using backpropagation (BP) algorithm. A common choice for the loss function is the cross entroy loss which equals to negative log-likelihood here. The parameters are usually updated as: INLINEFORM0  where, INLINEFORM0 is learning rate and INLINEFORM1 is regularization parameter.

Importance sampling is a Monte-Carlo scheme using an existing proposal distribution, and its estimator can be represented as: INLINEFORM0  where, INLINEFORM0 is an existing proposal distribution, INLINEFORM1 is the number of samples from INLINEFORM2 , INLINEFORM3 is the set of samples from INLINEFORM4 . Appling importance sampling to the average log-likelihood gradient of negative samples and the denominator of INLINEFORM5 , then the overall estimator for example INLINEFORM6 using INLINEFORM7 samples from distribution INLINEFORM8 is: INLINEFORM9  In order to avoid divergence, the sample size INLINEFORM0 should be increased as training processes which is measured by the effective sample size of importance sampling: INLINEFORM1  At every iteration, sampling is done block by block with a constant size until the effective sample size INLINEFORM0 becomes greater than a minimum value, and a full back-propagation will be performed when the sampling size INLINEFORM1 is greater than a certain threshold. The introduction of importance sampling is just posted here for completeness and no further studies will be performed on it. Because a quick statistical language model which is well trained, like n-gram based language model, is needed to implement importance sampling.

Because of the involvement of previous internal state at every step, back-propagation through time (BPTT) algorithm  is preferred for better performance when training RNNLMs. If data set is treated as a single long word sequence, truncated BPTT should be used and back-propagating error gradient through 5 steps is enough, at least for small corpus  . In this paper, neural network language models will all be trained on data set sentence by sentence, and the error gradient will be back-propagated trough every whole sentence without any truncation.

To explore this point further, hierarchical LSTM-NNLMs with different number of hierarchical layers were built. In these hierarchical LSTM-NNLMs, words were clustered randomly and uniformly instead of according to any word similarity information. The results of experiment on these models are showed in Table TABREF13 which strongly support the above hypothesis. When words are clustered into hierarchical word classes, the speed of both training and test increase, but the effect of speed-up decreases and the performance declines dramatically as the number of hierarchical layers increases. Lower perplexity can be expected if some similarity information of words is used when clustering words into classes. However, because of the ambiguity of words, the degradation of performance is unavoidable by assigning every word with a unique class or path. On the other hand, the similarities among words recognized by neural network is hard to defined, but it is sure that they are not confined to linguistical ones. There is a simpler way to speed up neural network language models using word classes which was proposed by  .

Then, the encoded features are integrated using a trainable neural network which may share the same architecture as existing ones. Because the model for encoding does not need to be trained, the size of this model can be much huge and the structure can be very complexity. If all the parameters of encoding model are designed using binary, it is possible to implement this model using hardware and higher efficiency can be expected.

In the third section, the details of some important NNLM techniques, including importance sampling, word classes, caching and bidirectional recurrent neural network (BiRNN), will be described, and experiments will be performed on them to examine their advantages and disadvantages separately. The limits of NNLM, mainly about the aspects of model architecture and knowledge representation, will be explored in the fourth section. A further work section will also be given to represent some further researches on NNLM. In last section, a conclusion about the findings in this paper will be made.

INLINEFORM16 , INLINEFORM17 , INLINEFORM18 , INLINEFORM19 , and INLINEFORM20 are vectors for bias terms. INLINEFORM21 is the activation function in hidden layer and INLINEFORM22 is the activation function for gates.

Words in vocabulary are arranged in descent order according to their frequencies in training data set, and are assigned to classes one by one using following rule: INLINEFORM0  where, INLINEFORM0 is the target number of word classes, INLINEFORM1 is the frequency of the INLINEFORM2 -th word in vocabulary, the sum of all words' frequencies INLINEFORM3 . If the above rule is satisfied, the INLINEFORM4 -th word in vocabulary will be assigned to INLINEFORM5 -th class. In this way, the word classes are not uniform, and the first classes hold less words with high frequency and the last ones contain more low-frequency words. This strategy was further optimized by  using following criterion: INLINEFORM6  where, the sum of all words' sqrt frequencies INLINEFORM0 . The experiment results (Table TABREF13 ) indicate that higher perplexity and a little more training time were obtained when the words in vocabulary were classified according to their frequencies than classified randomly and uniformly. When words are clustered into word classed using their frequency, words with high frequency, which contribute more to final perplexity, are clustered into very small word classes, and this leads to higher perplexity.

Long Short Term Memory RNNLM, LSTM-RNNLM	Although RNNLM can take all predecessor words into account when predicting next word in a word sequence, but it is quite difficult to be trained over long term dependencies because of the vanishing or exploring problem  . LSTM-RNN was designed aiming at solving this problem, and better performance can be expected by replacing RNN with LSTM-RNN. LSTM-RNNLM was first proposed by  , and the whole architecture is almost the same as RNNLM except the part of neural network. LSTM-RNN was proposed by  and was refined and popularized in following works  ,  . The general architecture of LSTM-RNN is: DISPLAYFORM0  Where, INLINEFORM0 , INLINEFORM1 , INLINEFORM2 are input gate, forget gate and output gate, respectively. INLINEFORM3 is the internal memory of unit. INLINEFORM4 , INLINEFORM5 , INLINEFORM6 , INLINEFORM7 , INLINEFORM8 , INLINEFORM9 , INLINEFORM10 , INLINEFORM11 , INLINEFORM12 , INLINEFORM13 , INLINEFORM14 , INLINEFORM15 are all weight matrixes.

Question: What directions are suggested to improve language models?
Answer: Improved architecture for ANN, use of linguistical properties of words or sentences as features.