Venue: NAACL 2016

# Incorporating Side Information into Recurrent Neural Network Language Models

 Cong Duy Vu Hoang

University of Melbourne

Melbourne, VIC, Australia

vhoang2@student.unimelb.edu.au &Gholamreza Haffari

Monash University

Clayton, VIC, Australia

gholamreza.haffari@monash.edu &Trevor Cohn

University of Melbourne

Melbourne, VIC, Australia

t.cohn@unimelb.edu.au

###### Abstract

Recurrent neural network language models (RNNLM) have recently demonstrated vast potential in modelling long-term dependencies for NLP problems, ranging from speech recognition to machine translation. In this work, we propose methods for conditioning RNNLMs on external side information, e.g., metadata such as keywords, description, document title or topic headline. Our experiments show consistent improvements of RNNLMs using side information over the baselines for two different datasets and genres in two languages. Interestingly, we found that side information in a foreign language can be highly beneficial in modelling texts in another language, serving as a form of cross-lingual language modelling.

## 1 Introduction

Neural network approaches to language modelling (LM) have made remarkable performance gains over traditional count-based \(n\)gram LMs Bengio et al. (2003); Mnih and Hinton (2007); Mikolov et al. (2011). They offer several desirable characteristics, including the capacity to generalise over large vocabularies through the use of vector space representation, and - for recurrent models Mikolov et al. (2011) - the ability to encode long distance dependencies that are impossible to include with a limited context windows used in conventional \(n\)gram LMs. These early papers have spawned a cottage industry in neural LM based applications, where text generation is a key component, including conditional language models for image captioning Kiros et al. (2014); Viynals et al. (2015) and neural machine translation Kalchbrenner and Blunsom (2013); Sutskever et al. (2014); Bahdanau et al. (2015).

Inspired by these works for conditioning LMs on complex side information, such as images and foreign text, in this paper we investigate the possibility of improving LMs in a more traditional setting, that is when applied directly to text documents. Typically corpora include rich side information, such as document titles, authorship, time stamp, keywords and so on, although this information is usually discarded when applying statistical models. However, this information can be highly informative, for instance, keywords, titles or descriptions, often include central topics which will be helpful in modelling or understanding the document text. We propose mechanisms for encoding this side information into a vector space representation, and means of incorporating it into the generating process in a RNNLM framework. Evaluating on two corpora and two different languages, we show consistently significant perplexity reductions over the state-of-the-art RNNLM models.

The contributions of this paper are as follows:

1. We propose a framework for encoding structured and unstructured side information, and its incorporation into a RNNLM.
2. We introduce a new corpus, the RIE corpus, based on the Europarl web archive, with rich annotations of several types of meta-data.
3. We provide empirical analysis showing consistent improvements from using side information across two datasets in two languages



## 2 Problem Formulation & Model

We first review RNNLM architecture Mikolov et al. (2011) before describing our extension in SS2.2.

### RNNLM Architecture

The standard RNNLM consists of 3 main layers: an input layer where each input word has its embedding via one-hot vector coding; a hidden layer consisting of recurrent units where a state is conditioned recursively on past states; and an output layer where a target word will be predicted. RNNLM has an advantage over conventional n-gram language model in modelling long distance dependencies effectively.

In general, an RNN operates from left-to-right over the input word sequence; i.e.,

\[{\mathbf{h}_{t}} =\operatorname{RU}\left({\mathbf{x}_{t}},{\mathbf{h}_{t-1}}\right)\] \[=\mathrm{f}\left({\mathbf{W}^{(hh)}{\mathbf{h}_{t-1}}+{\mathbf{W}^{(ih)}{ \mathbf{x}_{t}}}+{\mathbf{b}^{(h)}}}\right)\] \[{\mathbf{x}_{t+1}} \sim\operatorname{softmax}\left({\mathbf{W}^{(ho)}{\mathbf{h}_{t}}+{ \mathbf{b}^{(o)}}}\right);\]

where \(f(.)\) is a non-linear function, e.g., \(\tanh\), applied element-wise to its vector input; \({\mathbf{h}_{t}}\) is the current RNN hidden state at time-step \(t\); and matrices \({\mathbf{W}}\) and vectors \({\mathbf{b}}\) are model parameters. The model is trained using gradient-based methods to optimise a (regularised) training objective, e.g. the likelihood function. In principle, a recurrent unit (RU) can be employed using different variants of recurrent structures such as: Long Short Term Memory (LSTM) Hochreiter and Schmidhuber (1997), Gated Recurrent Unit (GRU) Cho et al. (2014), or recently deeper structures, e.g. Depth Gated Long Short Term Memory (DGLSTM) - a stack of LSTMs with extra connections between memory cells in deep layers Yao et al. (2015). It can be regarded as being a generalisation of LSTM recurrence to both time and depth. Such deep recurrent structure may capture long distance patterns at their most general. Empirically, we found that RNNLM with DGLSTM structure appears to be best performer across our datasets, and therefore is used predominantly in our experiments.

### Incorporating Side Information

Nowadays, many corpora are archived with side information or contextual meta-data. In this work, we argue that such information can be useful for language modelling (and presumably other NLP tasks). By providing this auxiliary information directly to the RNNLM, we stand to boost language modelling performance.

The first question in using side information is how to encode these unstructured inputs, \({\mathbf{y}}\), into a vector representation, denoted \({\mathbf{e}}\). We discuss several methods for encoding the auxiliary vector:

**BOW**: additive bag of words, \({\mathbf{e}}=\sum_{t}{\mathbf{y}_{t}}\), and
**average**: the average embedding vector, \({\mathbf{e}}=\frac{1}{T}\sum_{t}{\mathbf{y}_{t}}\), both inspired by Hermann and Blunsom (2014a);
**bigram**: convolution with sum-pooling, \({\mathbf{e}}=\sum_{t}\tanh\left({\mathbf{y}_{t-1}+{\mathbf{y}_{t}}}\right)\)Hermann and Blunsom (2014b); and
**RNN**: a recurrent neural network over the word sequence Sutskever et al. (2014), using the final hidden state(s) as \({\mathbf{e}}\).

From the above methods, we found that BOW worked consistently well, outperforming the other approaches, and moreover lead to a simpler model with faster training. For this reason we report only results for the BOW encoding. Note that when using multiple auxiliary inputs, we use a weighted combination, \({\mathbf{e}}=\sum_{i}{\mathbf{W}^{(ai)}{\mathbf{e}}^{(i)}}\).

The next step is the integration of \({\mathbf{e}}\) into the RNNLM. We consider two integration methods: as input to the hidden state (denoted **input**), and connected to the output \(\operatorname{softmax}\) layer (**output**), as shown in Figure 1 a and b, respectively. In both cases, we compare experimentally the following integration strategies:

**add**: adding the vectors together, e.g., using \({\mathbf{x}_{t}}+{\mathbf{e}}\) as the input to the RNN, such that

Figure 1: Integration methods for auxiliary information, \({\mathbf{e}}\): a) as input to the RNN, or b) as part of the output \(\operatorname{softmax}\) layer.



\({\mathbf{h}_{t}=\mathrm{RU}\left({\mathbf{x}_{t}+\mathbf{e},\mathbf{h}_{t-1}}\right)}\);
**stack** concatenating the vectors, e.g., using \(\left[{\mathbf{x}^{\top}_{t}\,\mathbf{e}^{\top}}\right]^{\top}\) for generating the RNN hidden state, such that \({\mathbf{h}_{t}=\mathrm{RU}\left(\left[\begin{array}{c}{\mathbf{x}_{t}}\\ {\mathbf{e}}\end{array}\right],{\mathbf{h}_{t-1}}\right)}\); and
**mlp**: feeding both vectors into an extra perceptron with single hidden layer, using a \(\tanh\) non-linearity and projecting the output to the required dimensionality; i.e.,

\[{\mathbf{h}^{\prime}_{t}=\tanh\left({\mathbf{W}^{(hh^{\prime})}\mathbf{h}_{ t}+\mathbf{W}^{(he)}\mathbf{e}+\mathbf{b}^{(h^{\prime})}}\right)}\] \[{\mathbf{x}_{t+1}\sim\operatorname*{softmax}\left({\mathbf{W}^{(ho)}\mathbf{ h}^{\prime}_{t}+\mathbf{b}^{(o)}}\right).}\]

Note that _add_ requires the vectors to be the same dimensionality, while the other two methods do not. The _stack_ method can be quite costly, given that it increases the size of several matrices, either in the recurrent unit (for _input_) or the _output_ mapping for word generation. This is a problem in the latter case: given the large size of the vocabulary, the matrix \({\mathbf{W}^{(ho)}}\) is already very large and making it larger (doubling the size, to become \({\mathbf{W}^{(h^{\prime}o)}}\)) has a sizeable effect on training time (and presumably also propensity to over-fit). The _output+stack_ method does however have a compelling interpretation as a jointly trained product model between a RNNLM and a unigram model conditioned on the side information, where both models are formulated as \(\operatorname*{softmax}\) classifiers. Considered as a product model Hinton (2002); Pascanu et al. (2013), the two components can concentrate on different aspects of the problem where the other model is not confident, and allowed each model the ability to 'veto' certain outputs, by assigning them a low probability.

## 3 Experiments

Datasets.We conducted our experiments on two datasets with different genres in two languages. As the first dataset, we use the IWSLT2014 MT track on TED Talks1 due to its self-contained rich auxiliary information, including: title, description, keywords, and author related information. We chose the English-French pair for our experiments2 . The statistics of the training set is shown in Table 1. We used dev2010 (7 talks/817 sentences) for early stopping of training neural network models. For evaluation, we used different testing sets over years, including test2010 (10/1587), tst2011 (7/768), tst2012 (10/1083).

Footnote 1: [https://wit3.fbk.eu/](https://wit3.fbk.eu/) (IWSLT’14 MT Track)

Footnote 2: Our method can be also applied to other language pairs.

As the second dataset, we crawled the entire European Parliament3 website, focusing on plenary sessions. Such sessions contain useful structural information, namely multilingual texts divided into speaker sessions and topics. We believe that those texts are interesting and challenging for language modelling tasks. Our dataset contains \(724\) plenary sessions over 12.5 years until June 2011 with multilingual texts in 22 languages4. We refer to this dataset by RIE5 (**R**ich **I**nformation **E**uroparl). We randomly select 200/5/30 plenary sessions as the training/development/test sets, respectively. We believe that the new data including side information pose another challenge for language modelling. Furthermore, the sizes of our working datasets are an order of magnitude larger than the standard Penn Treebank set which is often used for evaluating neural language models.

Footnote 3: [http://www.euorpal.europa.eu/](http://www.euorpal.europa.eu/)

Set-up and Baselines.We have used cnn6 to implement our models. We use the same configurations for all neural models: 512 input embedding and hidden layer dimensions, 2 hidden layers, and vocabulary sizes as given in Table 1. We used the same vocabulary for the auxiliary and modelled text. We trained a conventional \(5-\)gram language model using modified Kneser-Ney smoothing, with the KenLM toolkit Heafield (2011). We used the

\begin{table}
\begin{tabular}{|c||c|c|c|c|} \hline  & tokens (M) & types (K) & docs & sents (K) \\ \hline \hline TED-en & \(4.0\) & \(18.3\) & \(1414\) & \(179\) \\ TED-fr & \(4.3\) & \(22.6\) & \(1414\) & \(179\) \\ RIE-en & \(13.7\) & \(15.0\) & \(200\) & \(460\) \\ RIE-fr & \(14.9\) & \(19.4\) & \(200\) & \(460\) \\ \hline \end{tabular}
\end{table}
Table 1: Statistics of the training sets, showing in each cell the number of word tokens, types, documents (talks or plenaries), and sentences. Note that “types” here refers to word frequency thresholded at 5 and 15 for TED Talks and RIE datasets, respectively.

 

Wilcoxon signed-rank test (Wilcoxon, 1945) to measure the statistical significance (\(p<0.05\)) on differences between sentence-level perplexity scores of improved models compared to the best baseline. Throughout our experiments, punctuation, stop words and sentence markers (\(\langle{\rm s}\rangle\), \(\langle{\rm f}\rangle\), \(\langle{\rm unk}\rangle\)) are filtered out in all auxiliary inputs. We observed that this filtering was required for BOW to work reasonably well. For each model, the best perplexity score on development set is used for early stopping of training models, which was obtained after 2-5 and 2-3 epochs on TED Talks and RIE datasets, respectively.

Results & Analysis.The perplexity results on TED Talks dataset are presented in Table 2 and 3. RNNLM variants consistently achieve substantially better perplexities compared to the conventional \(5-\)gram language model baseline.7 Of the basic RNNLM models (middle), the DGLSTM works consistently better than both the standard RNN and the LSTM. This may be due to better interactions of memory cells in hidden layers. Since the DGLSTM outperformed others8, we used it for all subsequent experiments. For TED Talks dataset, there are three kinds of side information, including keywords, title, description. We attempted to inject those into different RNNLM layers, resulting in model variants as shown in Table 2. First, we chose "keywords" (+k) information as an anchor to figure out which incorporation method works well. Comparing _input_+add+k, _input_+mlp+k and _input_+stack+k, the largest decrease is obtained by _output_+mlp+k consistently across all test sets (and development sets, not shown here). We further evaluated the addition of other side information (e.g., "description" (+d), "title" (+t)), finding that +d has similar effect as +k whereas +t has a mixed effect, being detrimental for one test set (test2011). We suspect that it is due to often-times short sentences of titles in that test, after our filtering step, leading to a shortage of useful information fed into neural network learning. Interestingly, the best performance is obtained when incorporating both +k and +d, showing that there is complementary information in the two auxiliary inputs. Further, we also achieved the similar results in the counterpart of English part (in French) using _output_+mlp with both +t and +d as shown in Table 3. In French data, no "keywords" information is available. For this reason, we run additional experiments by injecting English keywords as side information into neural models of French. Interestingly, we found that "keywords" side information in English effectively improves the modelling of French texts as shown in Table 3, serving as a new form of cross-lingual language modelling.

Footnote 7: For fair comparison, when computing the perplexity with the 5-gram LM, we exclude all test words marked as \(\langle{\rm unk}\rangle\) (i.e., with low counts or OOVs) from consideration.

Footnote 8: This concurs with the finding in (Yao et al., 2015), who showed that DGLSTM produced the state-of-the-art results over Penn Treebank dataset.

We further achieved similar results by incorporating the topic headline in the RIE dataset. The consistently-improved results (in Table 4) demonstrate the robustness of th

\begin{table}
\begin{tabular}{l||c|c|c} \hline
**Method** & **test2010** & **test2011** & **test2012** \\ \hline \hline
5-gram LM & \(79.9\) & \(77.4\) & \(89.9\) \\ \hline RNNLM & \(65.8\) & \(63.9\) & \(73.0\) \\ LSTM & \(54.1\) & \(52.2\) & \(58.4\) \\ DGLSTM & \(53.1\) & \(52.1\) & \(58.8\) \\ \hline _input_+add+k & \(52.9\) & \(52.1\) & \(57.5\) \\ _input_+mlp+k & \(53.3\) & \(51.5\) & \(57.3\) \\ _input_+stack+k & \(53.7\) & \(51.9\) & \(58.1\) \\ _output_+mlp+k & **51.7** & **50.6** & **55.8** \\ _output_+mlp+t & **52.3** & \(53.5\) & \(58.3\) \\ _output_+mlp+d & **52.0** & **49.8** & **56.3** \\ _output_+mlp+k++t & **51.4** & **51.1** & **56.8** \\ _output_+mlp+k+d & **51.2** & **49.7** & **55.1** \\ _output_+mlp+t+d & **52.6** & \(51.5\) & **57.2** \\ _output_+mlp+k++td & **51.1** & **50.6** & **56.3** \\ \hline \end{tabular}
\end{table}
Table 2: **Perplexity** scores based on the **English** part of TED talks dataset in IWSLT14 MT. +k, +t, +d: with keywords, title, and description as auxiliary side information respectively. **bold**: Statistically significant better than the best baseline.

\begin{table}
\begin{tabular}{|l||c|c|c|} \hline
**Method** & **test2010** & **test2011** & **test2012** \\ \hline \hline
5-gram LM & \(65.1\) & \(60.3\) & \(64.8\) \\ \hline LSTM & \(45.0\) & \(42.5\) & \(44.0\) \\ DGLSTM & \(44.0\) & \(41.9\) & \(43.0\) \\ \hline _output_+mlp+t & **42.1** & **40.6** & \(42.5\) \\ _output_+mlp+d & **40.9** & **38.9** & \(40.3\) \\ _output_+mlp+t+d & **41.7** & **39.8** & \(42.8\) \\ \hline _output_+mlp+k & **40.8** & **38.3** & **39.7** \\ _output_+mlp+d+k & **40.2** & **38.3** & **39.4** \\ \hline \end{tabular}
\end{table}
Table 3: **Perplexity** scores based on the **French** part of TED talks dataset in IWSLT14 MT. Note that +k means with keywords in **English**.

 



## 4 Conclusion

We have proposed an effective approach to boost the performance of RNNLM using auxiliary side information (e.g. keywords, title, description, topic headline) of a textual utterance. We provided an empirical analysis of various ways of injecting such information into a distributed representation, which is then incorporated into either the input, hidden, or output layer of RNNLM architecture. Our experimental results reveal consistent improvements are achieved over strong baselines for different datasets and genres in two languages. Our future work will investigate the model performance on a closely-related task, i.e., neural machine translation Sutskever et al. (2014); Bahdanau et al. (2015). Furthermore, we will explore learning methods to combine utterances with and without the auxiliary side information.

### Acknowledgements

The authors would like to thank the reviewers for valuable comments and feedbacks. Cong Duy Vu Hoang was supported by research scholarships from the University of Melbourne, Australia. Dr Trevor Cohn was supported by the ARC (Future Fellowship).

## References

* Bahdanau et al. (2015) D. Bahdanau, K. Cho, and Y. Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In _Proceedings of International Conference on Learning Representations (ICLR 2015)_, September.
* Bengio et al. (2003) Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A Neural Probabilistic Language Model. _The Journal of Machine Learning Research_, 3:1137-1155.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. In _Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation_, pages 103-111, Doha, Qatar, October. Association for Computational Linguistics.
* Heafield (2011) Kenneth Heafield. 2011. KenLM: Faster and Smaller Language Model Queries. In _Proceedings of the EMNLP 2011 Sixth Workshop on Statistical Machine Translation_, pages 187-197, Edinburgh, Scotland, United Kingdom, July.
* Hermann and Blunsom (2014a) K. M. Hermann and P. Blunsom. 2014a. Multilingual Distributed Representations without Word Alignment. In _Proceedings of International Conference on Learning Representations (ICLR 2014)_, December.
* Hermann and Blunsom (2014b) Karl Moritz Hermann and Phil Blunsom. 2014b. Multilingual Models for Compositional Distributed Semantics. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 58-68, Baltimore, Maryland, June. Association for Computational Linguistics.
* Hinton (2002) Geoffrey E Hinton. 2002. Training Products of Experts by Minimizing Contrastive Divergence. _Neural computation_, 14(8):1771-1800.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long Short-Term Memory. _Neural Comput._, 9(8):1735-1780, November.
* Kalchbrenner and Blunsom (2013) Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent Continuous Translation Models. In _Proceedings of Empirical Methods in Natural Language Processing (EMNLP 2013)_.
* Kiros et al. (2014) Ryan Kiros, Ruslan Salakhutdinov, and Rich Zemel. 2014. Multimodal Neural Language Models. In _Proceedings of the 31st International Conference on Machine Learning (ICML-14)_, pages 595-603.
* Recurrent Neural Network Language Modeling Toolkit. In _2011 IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU)_. IEEE Automatic Speech Recognition and Understanding Workshop, December.
* Mnih and Hinton (2007) Andriy Mnih and Geoffrey Hinton. 2007. Three New Graphical Models for Statistical Language Modelling. In _Proceedings of the 24th International Conference on Machine Learning_, pages 641-648.
* Pascanu et al. (2013) R. Pascanu, C. Gulcehre, K. Cho, and Y. Bengio. 2013. How to Construct Deep Recurrent Neural Networks. _ArXiv e-prints_, December.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to Sequence Learning with Neural Networks. In _Advances in Neural Information Processing Systems (NIPS 2014)_, pages 3104-3112.
* Vinyals et al. (2015) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and Tell: A Neural Image Caption Generator. In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, June.
* Wang et al. (2015)

\begin{table}
\begin{tabular}{|l||c|c|} \hline
**Method** & **test (en)** & **test (fr)** \\ \hline \hline
5-gram LM & \(55.7\) & \(38.5\) \\ \hline LSTM & \(40.3\) & \(28.5\) \\ DGLSTM & \(36.4\) & \(25.4\) \\ \hline _output+mlp+h_ & **33.3** & **24.0** \\ \hline \end{tabular}
\end{table}
Table 4: **Perplexity scores based on the sampled RIE dataset. +h: topic headline.**

Frank Wilcoxon. 1945. Individual Comparisons by Ranking Methods. _Biometrics Bulletin_, 1 (6):80-83, Dec.
* Yao et al. (2015) K. Yao, T. Cohn, K. Vylomova, K. Duh, and C. Dyer. 2015. Depth-Gated LSTM. _ArXiv e-prints_, August.