Venue: NAACL 2016

# Inter-document Contextual Language Model

 Quan Hung Tran

Ingrid Zukerman and Gholamreza Haffari

Faculty of Information Technology

Monash University, Australia

hung.tran,ingrid.zukerman,gholamreza.haffari@monash.edu

###### Abstract

In this paper, we examine the impact of employing contextual, structural information from a tree-structured document set to derive a language model. Our results show that this information significantly improves the accuracy of the resultant model.

## 1 Introduction

Conventional Language Models (LMs) are based on n-grams, and thus rely upon a limited number of preceding words to assign a probability to the next word in a document. Recently, Mikolov _et al._ [2010] proposed a Recurrent Neural Network (RNN) LM which uses a vector representation of all the preceding words in a sentence as the context for language modeling. This model, which theoretically can utilize an infinite context window within a sentence, yields an LM with lower perplexity than that of n-gram-based LMs. However, the model does not leverage the wider contextual information provided by words in other sentences in a document or in related documents.

Several researchers have explored extending the contextual information of an RNN-based LM. Mikolov and Zweig [2012] proposed a context-dependent RNN LM that employs Latent Dirichlet Allocation for modeling a long span of context. Wang and Cho [2015] offered a bag-of-words representation of preceding sentences as the context for the RNN LM. Ji _et al._ [2015] used a Document-Context LM (DCLM) to leverage both intra- and inter-sentence context.

These works focused on contextual information at the document level for LM, but did not consider information at the inter-document level. Many document sets on the Internet are structured, which means there are connections between different documents. This phenomenon is prominent in social media, where all the posts are directly linked to several other posts. We posit that these related documents could hold important information about a particular post, including the topic and language use, and propose an RNN-based LM architecture that utilizes both intra- and inter-document contextual information. Our approach, which was tested on the social media dataset reddit, yielded promising results, which significantly improve on the state of the art.

## 2 Dataset

We used pre-collected reddit data,1 which as of December, 2015, consists of approximately 1.7 billion comments in JSON format. A comment thread starts with a "topic", which might be a link or an image. The users then begin to comment on the topic, or reply to previous comments. Over time, this process creates a tree-structured document repository (Figure 1), where a level indicator is assigned to each comment, e.g., a response to the root topic is assigned level 1, and the reply to a level \(n\) comment is assigned level \(n+1\). We parsed the raw data in JSON format into a tree structure, removing threads that have less than three comments, contain deleted comments, or do not have comments above

level 2. We randomly selected 2100 threads that fit these criteria. The data were then split into training/testing/validation sets. Table 1 displays some statistics of our dataset.

## 3 Baseline Neural Language Models

Our inter-document contextual language model scaffolds on the RNN LM Mikolov et al. (2010) and DCLMs Ji et al. (2015), as described below.

Rnn-Lstm.Given a sentence \(\{x_{t}\}_{t\in[1,...,N]}\), where \(x_{t}\) is the vector representation of the \(t\)-th word in the sentence, and \(N\) is the length of the sentence, Mikolov 's (2010) RNN LM can be defined as:

\[\mathbf{h}_{t}=\mathbf{f}(\mathbf{h}_{t-1},\mathbf{x}_{t}) \tag{1}\] \[\mathbf{y}_{t}\sim\text{softmax}\left(\mathbf{W}_{o}\mathbf{h}_{t-1}+\mathbf{b} _{o}\right) \tag{2}\]

where \(\mathbf{h}_{t}\) is the hidden unit at word \(t\), and \(\mathbf{y}_{t}\) is the prediction of the \(t\)-th word given the previous hidden unit \(\mathbf{h}_{t-1}\). The function \(\mathbf{f}\) in Equation 1 can be any non-linear function. Following the approach in Sundermeyer et al. (2012) and Ji et al. (2015), we make use of Long Short-Term Memory (LSTM) units Hochreiter and Schmidhuber (1997) rather than the simple hidden units used in the original RNN LM. In our work, the word representation \(\mathbf{x}_{t}\) is obtained from the one-hot representation using an affine transformation, as follows:

\[\mathbf{x}_{t}=\mathbf{W}_{p}\mathbf{o}_{t}+\mathbf{b}_{p} \tag{3}\]

where \(\mathbf{o}_{t}\) is the one-hot representation, \(\mathbf{W}_{p}\) is the projection matrix, and \(\mathbf{b}_{p}\) is a bias term.

Document Context LMs (DCLMs).We re-implemented two of Ji 's (2015) DCLMs as our baselines,2 viz Context-to-context (Figure 1(a)) and Context-to-output (Figure 1(b)). These models extend the RNN-LSTM model by leveraging information from preceding sentences.

Footnote 2: Ji _et al.’s three options performed similarly._

The context-to-context model (ccDCM) concatenates the final hidden unit of the previous sentence with the word vectors of the current sentence. Thus, Equation 1 becomes:

\[\mathbf{h}^{i}_{t}=\mathbf{f}(\mathbf{h}^{i}_{t-1},\mathbf{x}^{\prime}_{i,t}) \tag{4}\] \[\mathbf{x}^{\prime}_{i,t}=\text{concat}\left(\mathbf{x}_{i,t},\mathbf{h}^{i-1 }_{N_{i-1}}\right) \tag{5}\]

where \(N_{i-1}\) is the length of the previous sentence in the document, \(\mathbf{x}_{i,t}\) is the vector representation of the \(t\)-th word in the \(i\)-th sentence, \(\mathbf{x}^{\prime}_{i,t}\) is the concatenation of the vector representation \(\mathbf{x}_{i,t}\) and the previous sentence's final hidden unit \(\mathbf{h}^{i-1}_{N_{i-1}}\).

The context-to-output model (coDCLM) applies the additional information directly to the word-decoding phase. Thus, Equation 2 becomes:

\[\mathbf{y}_{i,t}\sim\text{softmax}\left(\mathbf{W}_{o}\mathbf{h}^{i}_{t-1}+\mathbf{W}^{\prime }_{o}\mathbf{h}^{i-1}_{N_{i-1}}+\mathbf{b}_{o}\right) \tag{6}\]

## 4 Inter-document Context Language Model

We now extend the DCLM by leveraging the information at the inter-document level, taking advantage of the structure of the repository -- a tree in reddit. Specifically, by harnessing the information in documents related to a target document, i.e., its siblings and parent, the LM is expected to contain additional relevant information, and hence lower perplexity. Formally, let's call the sentence-level context vector \(h_{s}\), the parent document context

\begin{table}
\begin{tabular}{l r r r r} \hline  & \# of & \# of & \# of & \# of \\  & threads & posts & sentences & tokens \\ \hline training & 1500 & 14592 & 40709 & 648624 \\ \hline testing & 500 & 5007 & 13612 & 217164 \\ \hline validation & 100 & 968 & 2762 & 44575 \\ \hline \end{tabular}
\end{table}
Table 1: Dataset statistics

Figure 1: reddit example



vector \(h_{p}\), the sibling context vector \(h_{l}\), and the overall context vector \(h_{c}\). Our framework is defined as:

\[\mathbf{h}_{c} = \mathbf{g}_{h}(\mathbf{h}_{s},\mathbf{h}_{l},\mathbf{h}_{p}) \tag{7}\] \[\mathbf{x}^{\prime}_{i,t} = \mathbf{g}_{i}(\mathbf{x}_{i,t},\mathbf{h}_{c})\] (8) \[\mathbf{h}_{t} = \mathbf{f}(\mathbf{h}_{t-1},\mathbf{x}^{\prime}_{t})\] (9) \[\mathbf{y}_{t} \sim \text{softmax}\left(\mathbf{g}_{o}(\mathbf{h}_{t-1},\mathbf{x}^{\prime}_{t}, \mathbf{h}_{c})\right) \tag{10}\]

We use the last hidden vector of the RNNs as the representation of the parent post, the older-sibling, and the previous sentence. The definition of the context function (\(\mathbf{g}_{h}\)), the input function (\(\mathbf{g}_{i}\)), and the word-decoding function (\(\mathbf{g}_{o}\)) yields different configurations.

We also explored two strategies of training the models: _Disconnected (disC)_ and _Fully Connected (fulC)_. In the disC-trained models, the error signal within a time step (i.e. a post or sentence) only affects the parameters in that time step. This is in contrast to the fulC-trained models, where the error signal is propagated to the previous time steps, hence influencing parameters in those time steps too.

### Analysis of our modelling approach

In this section, we empirically analyze different training and modelling decisions within our framework, namely DC vs FC training, as well as contextual information from parent vs sibling.

The Setup.For our analysis, we employed a subset of the data described in Table 1 which contains 450 threads split into training/testing/validation sets with 300/100/50 threads respectively. The hidden-vector and word-vector dimensions were set to 50 and 70, respectively. The models were implemented in Theano Bastien et al. (2012); Bergstra et al. (2010), and trained with RMSProp Tieleman and Hinton (2012).

disC vs fuIC.We first compared the disC and fuIC strategies, at the sentence level only, in order to select the best strategy in a known setting. To this effect, we re-implemented Ji _et al._'s (2015) DCLMs with the disC strategy, noting that Ji _et al._'s original sentence-based models are fulC-trained. The results of this experiment appear in Table 2 which further compares these models with the following baselines: (1) vanilla RNN-LSTM, and (2) a 6-gram LM with Kneser-Ney smoothing3Kneser and Ney (1995). The disC-trained models showed no improvement over the RNN-LSTM, and lagged behind their fulC-trained counterparts. The lower performance of the disC-trained models may be due to not fully leveraging the contextual information; disC-training lose information, as the error signal from the current time step is not used to calibrate the parameters of previous time steps. Therefore, we make use of fuIC strategy to train our models in the rest of this paper.

Footnote 3: footnotemark:

Parent vs Sibling Context.The inter-document information in reddit's case may come from a parent post, sibling posts or both. We tested our models with different combinations of inter-document con

Figure 2: Contextual language models; see Sections 3 and 4 for detailed descriptions.

 

text information to reflect these options. At present, we consider only the closest older-sibling of a post, as it is deemed the most related; different combinations of sibling posts are left for future work. We tested the following three context-to-context configurations: parent only (P-ccDCLM), sibling only (S-ccDLM), and parent and sibling (PS-ccDCLM), which define the context function as Equation 11, 12 and 13 respectively. The three configurations use the same word-decoding function (Equation 15) and the same input function (Equation 14).

\[{\mathbf{h}_{c}} = \mbox{concat}\left({\mathbf{h}_{s},{\mathbf{h}_{p}}}\right) \tag{11}\] \[{\mathbf{h}_{c}} = \mbox{concat}\left({\mathbf{h}_{s},{\mathbf{h}_{l}}}\right)\] (12) \[{\mathbf{h}_{c}} = \mbox{concat}\left({\mathbf{h}_{s},{\mathbf{h}_{l}},{\mathbf{h}_{p}}}\right)\] (13) \[{\mathbf{x}^{\prime}_{i,t}} = \mbox{concat}\left({\mathbf{x}_{i,t},{\mathbf{h}_{c}}}\right)\] (14) \[{\mathbf{y}_{i,t}} \sim \mbox{softmax}\left({\mathbf{W}_{o}{\mathbf{h}_{t-1}}+{\mathbf{b}_{o}}}\right) \tag{15}\]

The results of this experiment appear in the first three rows of Table 3, which shows that the best-performing model is PS-ccDCLM.

As discussed by Ji et al. (2015), the coDCLM makes the hidden units of the previous sentence have no effect on the hidden units of the current sentence. While this configuration might have some advantages Ji et al. (2015), applying it directly to a larger context may lead to complications. Suppose we use the last hidden unit of the previous document as the context for the next document. With the context-to-output approach, the last hidden unit summarizes only the information in the last sentence of the previous document, and doesn't reflect the entire document. We address this problem by not using the context-to-output approach in isolation. Instead, we use the context-to-output approach in tandem with the context-to-context approach of ccDCLM. This approach was tested in an additional parent-sibling configuration (PS-ccOCLM), as an alternative to the best performing context-to-context configuration. The PS-ccODCLM is similar to the PS-ccDCLM except for the decoding equation, which is changed into Equation 16.

\[{\mathbf{y}_{i,t}}\sim\mbox{softmax}\left({\mathbf{W}_{o}{\mathbf{h}_{t-1}^{i}}+{\mathbf{ W}^{\prime}_{o}{\mathbf{h}_{c}}}+{\mathbf{b}_{o}}}\right) \tag{16}\]

Based on the results of these trials, we chose the best-performing PS-ccDCLM (Figure 1(c)) as our final system.

### Results

The model perplexity obtained by the baselines and our best-performing model for the test set (Table 1) is shown in Table 4 -- our system (PS-ccDCLM) statistically significantly outperforms the best baseline (ccDCLM), with \(\alpha=0.01\), using the Friedman test. The inter-sentence contextual information under the context-to-context regime (ccDCLM) decreases model perplexity by 9% compared to the original RNN-LSTM, while the inter-document contextual information (PS-ccDCLM) reduces perplexity by a further 5% compared to ccDCLM.

## 5 Discussion and Future Work

Our results show that including inter-document contextual information yields additional improvements to those obtained from inter-sentence information. However, as expected, the former are smaller than the latter, as sentences in the same post are more related than sentences in different posts. At present, we rely on the final hidden-vector of the sentences and the posts for contextual information. In the future, we propose to explore other options, such as additional models to combine the contextual information from all siblings in the tree structure, and extending our model to structures beyond trees.

\begin{table}
\begin{tabular}{l c} \hline \hline
**Systems** & **Perplexity** \\ \hline
6-gram & 209 \\ \hline RNN-LSTM & 184 \\ \hline ccDCLM & 168 \\ \hline coDCLM & 176 \\ \hline PS-ccDCLM & **159** \\ \hline \end{tabular}
\end{table}
Table 4: Results on the entire dataset.

\begin{table}
\begin{tabular}{l c} \hline \hline
**Systems** & **Perplexity** \\ \hline P-ccDCLM & 172 \\ \hline S-ccDCLM & 174 \\ \hline PS-ccDCLM & **168** \\ \hline PS-ccoDCLM & 175 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparing models incorporating parent (P) and/or sibling (S) contextual information.

 



## References

* Bastien et al. (2012) Frederic Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua Bengio. 2012. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop.
* Bergstra et al. (2010) James Bergstra, Olivier Breuleux, Frederic Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. 2010. Theano: a CPU and GPU math expression compiler. In _Proceedings of the Python for Scientific Computing Conference (SciPy)_, June. Oral Presentation.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Ji et al. (2015) Yangfeng Ji, Trevor Cohn, Lingpeng Kong, Chris Dyer, and Jacob Eisenstein. 2015. Document context language models. _arXiv preprint arXiv:1511.03962_.
* Kneser and Ney (1995) Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for M-gram language modeling. In _ICASSP-95_, volume 1, pages 181-184. IEEE.
* Mikolov and Zweig (2012) Tomas Mikolov and Geoffrey Zweig. 2012. Context dependent recurrent neural network language model. In _SLT_, pages 234-239.
* Mikolov et al. (2010) Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In _INTERSPEECH 2010_, pages 1045-1048.
* Stolcke et al. (2011) Andreas Stolcke, Jing Zheng, Wen Wang, and Victor Abrasch. 2011. SRILM at sixteen: Update and outlook. In _Proceedings of IEEE Automatic Speech Recognition and Understanding Workshop_, page 5.
* Sundermeyer et al. (2012) Martin Sundermeyer, Ralf Schluter, and Hermann Ney. 2012. LSTM neural networks for language modeling. In _INTERSPEECH_, pages 194-197.
* Tijmen and Hinton (2012) Tijmen. Tieleman and Geoffrey Hinton. 2012. Neural Networks for Machine Learning. [http://www.youtube.com/watch?v=O3sxAc4hxZU](http://www.youtube.com/watch?v=O3sxAc4hxZU). [Online].
* Wang and Cho (2015) Tian Wang and Kyunghyun Cho. 2015. Larger-context language modelling. _arXiv preprint arXiv:1511.03729_.