RNNLM - Recurrent Neural Network Language Modeling Toolkit

We present freely available open-source toolkit for training recurrent neural network based language models. I t can be easily used to improve existing speech recognition and ma chine translation systems. Also, it can be used as a baseline for fu ture research of advanced language modeling techniques. In the p a er, we discuss optimal parameter selection and different modes of functionality. The toolkit, example scripts and basic setups are freely available at http://rnnlm.sourceforge.net/. I. I NTRODUCTION,  Statistical language modeling attracts a lot of attention, as models of natural languages are important part of many practical systems today. Moreover, it can be estimated that with further research progress, language models will becom e closer to human understanding  , and completely new applications will become practically realizable. Immedia tely, any significant progress in language modeling can be utilize d in the esisting speech recognition and statistical machine translation systems. However, the whole research field struggled for decades to overcome very simple, but also effective models based on ngram frequencies  . Many techniques were developed to beat n-grams, but the improvements came at the cost of computational complexity. Moreover, the improvements wer e often reported on very basic systems, and after application to state-of-the-art setups and comparison to n-gram models trained on large amounts of data, improvements provided by many techniques vanished. This has lead to scepticism among speech recognition researchers. In our previous work, we have compared many major advanced language modeling techniques, and found that neur al network based language models (NNLM) perform the best on several standard setups . Models of this type were introduced by Bengio in , about ten years ago. Their main weaknesses were huge computational complexity, and nontrivial implementation. Successful training of neural net works require well chosen hyper-parameters, such as learning rat e and size of hidden layer. To help overcome these basic obstacles, we have decided to release our toolkit for training recurrent neural network b ased language models (RNNLM). We have shown that the recurrent architecture outperforms the feedforward one on several se tup in . Moreover, the implemenation is simple and easy to understand. The most importantly, recurrent neural networ ks are very interesting from the research point of view, as they allow effective processing of sequences and patterns with arbitraty length these models can learn to store informati on in the hidden layer. Recurrent neural networks can have memory , and are thus important step forward to overcome the most painful and often criticized drawback of n-gram models dependence on previous two or three words only. In this paper we present an open source and freely available toolkit for training statistical language models base d or recurrent neural networks. It includes techniques for redu cing computational complexity (classes in the output layer and direct connections between input and output layer). Our too lkit has been designed to provide comparable results to the popul ar toolkit for training n-gram models, SRILM . The main goals for the RNNLM toolkit are these: • promotion of research of advanced language modeling techniques • easy usage • simple portable code without any dependencies • computational efficiency In the paper, we describe how to easily make RNNLM part of almost any speech recognition or machine translation syste m that produces lattices. II. RECURRENTNEURAL NETWORK The recurrent neural network architecture used in the toolk it is shown at Figure 1 (usually called Elman network, or simple RNN). The input layer uses the 1-of-N representation of the previous wordw(t) concatenated with previous state of the hidden layers(t − 1). The neurons in the hidden layer s(t) use sigmoid activation function. The output layer (t) has the same dimensionality as w(t), and after the network is trained, it represents probability distribution of the next word giv en the previous word and state of the hidden layer in the previous time step . The class layer c(t) can be optionally used to reduce computational complexity of the model, at a small cost of accuracy [7]. Training is performed by the standard stochastic gradient descent algorithm, and the matrix W that