Venue: NAACL 2016

# Top-down Tree Long Short-Term Memory Networks

Xingxing Zhang, Liang Lu and  Mirella Lapata

School of Informatics, University of Edinburgh

10 Crichton Street, Edinburgh EH8 9AB, UK

{x.zhang,liang.lu}@ed.ac.uk,mlap@inf.edu.ac.uk

###### Abstract

Long Short-Term Memory (LSTM) networks, a type of recurrent neural network with a more complex computational unit, have been successfully applied to a variety of sequence modeling tasks. In this paper we develop Tree Long Short-Term Memory (TreeLSTM), a neural network model based on LSTM, which is designed to predict a tree rather than a linear sequence. TreeLSTM defines the probability of a sentence by estimating the generation probability of its dependency tree. At each time step, a node is generated based on the representation of the generated subtree. We further enhance the modeling power of TreeLSTM by explicitly representing the correlations between left and right dependents. Application of our model to the MSR sentence completion challenge achieves results beyond the current state of the art. We also report results on dependency parsing reranking achieving competitive performance.

## 1 Introduction

Neural language models have been gaining increasing attention as a competitive alternative to n-grams. The main idea is to represent each word using a real-valued feature vector capturing the contexts in which it occurs. The conditional probability of the next word is then modeled as a smooth function of the feature vectors of the preceding words and the next word. In essence, similar representations are learned for words found in similar contexts resulting in similar predictions for the next word. Previous approaches have mainly employed feed-forward [2, 34] and recurrent neural networks [15, 6] in order to map the feature vectors of the context words to the distribution for the next word. Recently, RNNs with Long Short-Term Memory (LSTM) units [7, 1] have emerged as a popular architecture due to their strong ability to capture long-term dependencies. LSTMs have been successfully applied to a variety of tasks ranging from machine translation [20], to speech recognition [1], and image description generation [21].

Despite superior performance in many applications, neural language models essentially predict sequences of words. Many NLP tasks, however, exploit syntactic information operating over tree structures (e.g., dependency or constituent trees). In this paper we develop a novel neural network model which combines the advantages of the LSTM architecture and syntactic structure. Our model estimates the probability of a sentence by estimating the generation probability of its dependency tree. Instead of explicitly encoding tree structure as a set of features, we use four LSTM networks to model four types of dependency edges which altogether specify how the tree is built. At each time step, one LSTM is activated which predicts the next word conditioned on the sub-tree generated so far. To learn the representations of the conditioned sub-tree, we force the four LSTMs to share their hidden layers. Our model is also capable of generating trees just by sampling from a trained model and can be seamlessly integrated with text generation applications

Our approach is related to but ultimately different from recursive neural networks [14] a class of models which operate on structured inputs. Given a (binary) parse tree, they recursively generate parent representations in a bottom-up fashion, by combining tokens to produce representations for phrases, and eventually the whole sentence. The learned representations can be then used in classification tasks such as sentiment analysis [23] and paraphrase detection [26]. Tai et al. TaiEtAl:2015 learn distributed representations over syntactic trees by generalizing the LSTM architecture to tree-structured network topologies. The key feature of our model is not so much that it can learn semantic representations of phrases or sentences, but its ability to predict tree structure and estimate its probability.

Syntactic language models have a long history in NLP dating back to Chelba and Jelinek Chelba:2000 (see also Roark Ranzani:2001 and Charniak Charniak:2001). These models differ in how grammar structures in a parsing tree are used when predicting the next word. Other work develops dependency-based language models for specific applications such as machine translation [7, 8, 9], speech recognition [10] or sentence completion [11]. All instances of these models apply Markov assumptions on the dependency tree, and adopt standard n-gram smoothing methods for reliable parameter estimation. Emami et al.Emami:2003 and SennrichSennrich:2015 estimate the parameters of a structured language model using feed-forward neural networks [12]. Mirowski and Vlachos mirowski:2015 re-implement the model of Gubbins and Vlachos (Vlachos:2013 with RNNs. They view sentences as sequences of words over a tree. While they ignore the tree structures themselves, we model them explicitly.

Our model shares with other structured-based language models the ability to take dependency information into account. It differs in the following respects: (a) it does not artificially restrict the depth of the dependencies it considers and can thus be viewed as an infinite order dependency language model; (b) it not only estimates the probability of a string but is also capable of generating dependency trees; (c) finally, contrary to previous dependency-based language models which encode syntactic information as features, our model takes tree structure into account more directly via representing different types of dependency edges explicitly using LSTMs. Therefore, there is no need to manually determine which dependency tree features should be used or how large the feature embeddings should be.

We evaluate our model on the MSR sentence completion challenge, a benchmark language modeling dataset. Our results outperform the best published results on this dataset. Since our model is a general tree estimator, we also use it to rerank the top \(K\) dependency trees from the (second order) MSTParsser and obtain performance on par with recently proposed dependency parsers.

## 2 Tree Long Short-Term Memory Networks

We seek to estimate the probability of a sentence by estimating the generation probability of its dependency tree. Syntactic information in our model is represented in the form of dependency paths. In the following, we first describe our definition of dependency path and based on it explain how the probability of a sentence is estimated.

### Dependency Path

Generally speaking, a dependency path is the path between root and \(w\) consisting of the nodes on the path and the edges connecting them. To represent dependency paths, we introduce four types of edges which essentially define the "shape" of a dependency tree. Let \(w_{0}\) denote a node in a tree and \(w_{1},w_{2},\ldots,w_{n}\) its left dependents. As shown in Figure 1, Left edge is the edge between \(w_{0}\) and its first left dependent denoted as \((w_{0},w_{1})\). Let \(w_{k}\) (with \(1<k\leq n\)) denote a non-first left dependent of \(w_{0}\). The edge from \(w_{k-1}\) to \(w_{k}\) is a Nx-Left edge (Nx stands for Next), where \(w_{k-1}\) is the right adjacent sibling of \(w_{k}\). Note that the Nx-Left edge \((w_{k-1},w_{k})\) replaces edge \((w_{0},w_{k})\) (illustrated with a dashed line in Figure 1) in the original dependency tree. The modification allows information to flow from \(w_{0}\) to \(w_{k}\) through \(w_{1},\ldots,w_{k-1}\) rather than directly from \(w_{0}\) to \(w_{k}\). Right and Nx-Right edges are defined analogously for right dependents.

Given these four types of edges, dependency paths (denoted as \(\mathcal{D}(w)\)) can be defined as follows 

bearing in mind that the first right dependent of root is its only dependent and that \(w^{p}\) denotes the parent of \(w\). We use \((\dots)\) to denote a sequence, where \(()\) is an empty sequence and \(\|\) is an operator for concatenating two sequences.

1. if \(w\) is root, then \(\mathcal{D}(w)=()\)
2. if \(w\) is a **left dependent** of \(w^{p}\)  1. if \(w\) is the first left dependent, then \(\mathcal{D}(w)=\mathcal{D}(w^{p})\|(\langle w^{p},\textsc{Left}\rangle)\)  2. if \(w\) is not the first left dependent and \(w^{s}\) is its right adjacent sibling, then \[\mathcal{D}(w)=\mathcal{D}(w^{s})\|(\langle w^{s},\textsc{Nx-Left}\rangle)\]
3. if \(w\) is a **right dependent** of \(w^{p}\)  1. if \(w\) is the first right dependent, then \(\mathcal{D}(w)=\mathcal{D}(w^{p})\|(\langle w^{p},\textsc{Right}\rangle)\)  2. if \(w\) is not the first right dependent and \(w^{s}\) is its left adjacent sibling, then \[\mathcal{D}(w)=\mathcal{D}(w^{s})\|(\langle w^{s},\textsc{Nx- Right}\rangle)\]

A dependency tree can be represented by the set of its dependency paths which in turn can be used to reconstruct the original tree.1

Footnote 1: Throughout this paper we assume all dependency trees are projective.

Dependency paths for the first two levels of the tree in Figure 2 are as follows (ignoring for the moment the subscripts which we explain in the next section). \(\mathcal{D}(\text{sold})=(\langle\textsc{root},\textsc{Right}\rangle)\) (see definitions (1) and (3a)), \(\mathcal{D}(\text{year})=\mathcal{D}(\text{sold})\|(\langle\text{sold}, \textsc{Left}\rangle)\) (see (2a)), \(\mathcal{D}(\text{manufacturer})=\mathcal{D}(\text{year})\|(\langle\text{ year},\textsc{Nx-Left}\rangle)\) (see (2b)), \(\mathcal{D}(\text{cars})=\mathcal{D}(\text{sold})\|(\langle\text{sold}, \textsc{Right}\rangle)\) (see (3a)), \(\mathcal{D}(\text{in})=\mathcal{D}(\text{cars})\|(\langle\text{cars},\textsc{ Nx-Right}\rangle)\) (according to (3b)).

### Tree Probability

The core problem in syntax-based language modeling is to estimate the probability of sentence \(S\) given its corresponding tree \(T\), \(P(S|T)\). We view the probability computation of a dependency tree as a generation process. Specifically, we assume dependency trees are constructed top-down, in a breadth-first manner. Generation starts at the root node. For each node at each level, first its left dependents are generated from closest to farthest and then the right dependents (again from closest to farthest). The same process is applied to the next node at the same level or a node at the next level. Figure 2 shows the breadth-first traversal of a dependency tree.

Under the assumption that each word \(w\) in a dependency tree is _only_ conditioned on its _dependency path_, the probability of a sentence \(S\) given its dependency tree \(T\) is:

\[P(S|T)=\prod_{w\in\textsc{BFS}(T)\setminus\textsc{Root}}P(w|\mathcal{D}(w)) \tag{1}\]

where \(\mathcal{D}(w)\) is the dependency path of \(w\). Note that each word \(w\) is visited according to its breadth-first search order (BFS(T)) and the probability of root is ignored since every tree has one. The role of root in a dependency tree is the same as the begin of sentence token (BOS) in a sentence. When computing \(P(S|T)\) (or \(P(S)\)), the probability of root (or BOS) is ignored (we assume it always exists), but is used to predict other words. We explain in the next section how TreeLSTM estimates \(P(w|\mathcal{D}(w))\).

### Tree LSTMs

A dependency path \(\mathcal{D}(w)\) is subtree which we denote as a sequence of \(\langle\textit{word}\), _edge-type_\(\rangle\) tuples. Our

Figure 1: Left and Nx-Left edges. Dotted line between \(w_{1}\) and \(w_{k-1}\) (also between \(w_{k}\) and \(w_{n}\)) indicate that there may be \(\geq 0\) nodes inbetween.

Figure 2: Dependency tree of the sentence The luxury auto manufacturer last year sold 1,214 cars in the U.S. Subscripts indicate breadth-first traversal. root has only one dependent (i.e., sold) which we view as its first right dependent.

 

innovation is to learn the representation of \({\cal D}(w)\) using four LSTMs. The four LSTMs (Gen-L, Gen-R, Gen-Nx-L and Gen-Nx-R) are used to represent the four types of edges (Left, Right, Nx-Left and Nx-Right) introduced earlier. Gen, Nx, L and R are shorthands for Generate, Next, Left and Right. At each time step, an LSTM is chosen according to an edge-type; then the LSTM takes a word as input and predicts/generates its dependent or sibling. This process can be also viewed as adding an edge and a node to a tree. Specifically, LSTMs Gen-L and Gen-R are used to generate the first left and right dependent of a node (\(w_{1}\) and \(w_{4}\) in Figure 3). So, these two LSTMs are responsible for going deeper in a tree. While Gen-Nx-L and Gen-Nx-R generate the remaining left/right dependents and therefore go wider in a tree. As shown in Figure 3, \(w_{2}\) and \(w_{3}\) are generated by Gen-Nx-L, whereas \(w_{5}\) and \(w_{6}\) are generated by Gen-Nx-R. Note that the model can handle any number of left or right dependents by applying Gen-Nx-L or Gen-Nx-R multiple times.

We assume time steps correspond to the steps taken by the breadth-first traversal of the dependency tree and the sentence has length \(n\). At time step \(t\) (\(1\leq t\leq n\)), let \(\langle w_{t^{\prime}},z_{t}\rangle\) denote the last tuple in \({\cal D}(w_{t})\). Subscripts \(t\) and \(t^{\prime}\) denote the breadth-first search order of \(w_{t}\) and \(w_{t^{\prime}}\), respectively. \(z_{t}\in\left\{\textsc{Left,Right,Nx-Left,Nx-Right}\right\}\) is the edge type (see the definitions in Section 2.1). Let \(\mathbf{W}_{e}\in\mathbb{R}^{s\times|V|}\) denote the word embedding matrix and \(\mathbf{W}_{ho}\in\mathbb{R}^{|V|\times d}\) the output matrix of our model, where \(|V|\) is the vocabulary size, \(s\) the word embedding size and \(d\) the hidden unit size. We use tied \(\mathbf{W}_{e}\) and tied \(\mathbf{W}_{ho}\) for the four LSTMs to reduce the number of parameters in our model. The four LSTMs also share their hidden states. Let \(\mathbf{H}\in\mathbb{R}^{d\times(n+1)}\) denote the _shared_ hidden states of all time steps and \(e(w_{t})\) the one-hot vector of \(w_{t}\). Then, \(\mathbf{H}[:,t]\) represents \({\cal D}(w_{t})\) at time step \(t\), and the computation2 is:

Footnote 2: We ignore all bias terms for notational simplicity.

\[\mathbf{x}_{t} =\mathbf{W}_{e}\cdot e(w_{t^{\prime}}) \tag{2a}\] \[\mathbf{h}_{t} =\mathrm{LSTM}^{z}\left(\mathbf{x}_{t},\mathbf{H}[:,t^{\prime}]\right)\] (2b) \[\mathbf{H}[:,t] =\mathbf{h}_{t}\] (2c) \[\mathbf{y}_{t} =\mathbf{W}_{ho}\cdot\mathbf{h}_{t} \tag{2d}\]

where the initial hidden state \(\mathbf{H}[:,0]\) is initialized to a vector of small values such as 0.01. According to Equation (2b), the model selects an LSTM based on edge type \(z_{t}\). We describe the details of \(\mathrm{LSTM}^{z}\) in the next paragraph. The probability of \(w_{t}\) given its dependency path \({\cal D}(w_{t})\) is estimated by a _softmax_ function:

\[P(w_{t}|{\cal D}(w_{t}))=\frac{\exp(\mathbf{y}_{t,w_{t}})}{\sum_{k^{\prime}=1} ^{|V|}\exp(\mathbf{y}_{t,k^{\prime}})} \tag{3}\]

We must point out that although we use four jointly trained LSTMs to encode the hidden states, the training and inference complexity of our model is no different from a regular LSTM, since at each time step only one LSTM is working.

We implement \(\mathrm{LSTM}^{z}\) in Equation (2b) using a deep LSTM (to simplify notation, from now on we write \(z\) instead of \(z_{t}\)). The inputs at time step \(t\) are \(\mathbf{x}_{t}\) and \(\mathbf{h}_{t^{\prime}}\) (the hidden state of an earlier time step \(t^{\prime}\)) and the output is \(\mathbf{h}_{t}\) (the hidden state of current time step). Let \(L\) denote the layer number of \(\mathrm{LSTM}^{z}\) and \(\hat{\mathbf{h}}_{t}^{l}\) the internal hidden state of the \(l\)-th layer of the \(\mathrm{LSTM}^{z}\) at time step \(t\), where \(\mathbf{x}_{t}\) is \(\hat{\mathbf{h}}_{t}^{0}\) and \(\mathbf{h}_{t^{\prime}}\) is \(\hat{\mathbf{h}}_{t^{\prime}}^{L}\). The LSTM architecture introduces multiplicative gates and memory cells \(\hat{\mathbf{c}}_{t}^{l}\) (at \(l\)-th layer) in order to address the _vanishing gradient_ problem which makes it difficult for the standard RNN model to learn long-distance correlations in a sequence. Here, \(\hat{\mathbf{c}}_{t}^{l}\) is a linear combination of the current input signal \(\mathbf{u}_{t}\) and an earlier memory cell \(\hat{\mathbf{c}}_{t^{\prime}}^{l}\). How much input information \(\mathbf{u}_{t}\) will flow into \(\hat{\mathbf{c}}_{t}^{l}\) is controlled

Figure 3: Generation process of left (\(w_{1},w_{2},w_{3}\)) and right (\(w_{4},w_{5},w_{6}\)) dependents of tree node \(w_{o}\) (top) using four LSTMs (Gen-L, Gen-R, Gen-Nx-L and Gen-Nx-R). The model can handle an arbitrary number of dependents due to Gen-Nx-L and Gen-Nx-R.

 

by input gate \({\bf i}_{t}\) and how much of the earlier memory cell \(\hat{\mathbf{c}}^{l}_{t^{\prime}}\) will be forgotten is controlled by forget gate \({\bf f}_{t}\). This process is computed as follows:

\[{\bf u}_{t}=\tanh({\bf W}^{z,l}_{ux}\cdot\hat{\mathbf{h}}^{I-1}_{t}+{ \bf W}^{z,l}_{uh}\cdot\hat{\mathbf{h}}^{I}_{t^{\prime}}) \tag{4a}\] \[{\bf i}_{t}=\sigma({\bf W}^{zz}_{ix}\cdot\hat{\mathbf{h}}^{I-1}_{ t}+{\bf W}^{zd}_{ih}\cdot\hat{\mathbf{h}}^{I}_{t^{\prime}})\] (4b) \[{\bf f}_{t}=\sigma({\bf W}^{zd}_{fx}\cdot\hat{\mathbf{h}}^{I-1}_{ t}+{\bf W}^{zd}_{fh}\cdot\hat{\mathbf{h}}^{I}_{t^{\prime}})\] (4c) \[\hat{\mathbf{c}}^{l}_{t}={\bf f}_{t}\odot\hat{\mathbf{c}}^{l}_{t^ {\prime}}+{\bf i}_{i}\odot{\bf u}_{t} \tag{4d}\]

where \({\bf W}^{zc,l}_{ux}\in\mathbb{R}^{d\times d}\) (\({\bf W}^{zc,l}_{ux}\in\mathbb{R}^{d\times s}\) when \(l=1\)) and \({\bf W}^{zd}_{uh}\in\mathbb{R}^{xd\times d}\) are weight matrices for \({\bf u}_{t}\), \({\bf W}^{zc,l}_{ix}\) and \({\bf W}^{zd}_{ih}\) are weight matrices for \({\bf i}_{t}\) and \({\bf W}^{zc,l}_{fx}\), and \({\bf W}^{zd}_{fh}\) are weight matrices for \({\bf f}_{t}\). \(\sigma\) is a sigmoid function and \(\odot\) the element-wise product.

Output gate \({\bf o}_{t}\) controls how much information of the cell \(\hat{\mathbf{c}}^{l}_{t}\) can be seen by other modules:

\[{\bf o}_{t} =\sigma({\bf W}^{zc,l}_{ox}\cdot\hat{\mathbf{h}}^{I-1}_{t}+{\bf W }^{zc,l}_{oh}\cdot\hat{\mathbf{h}}^{I}_{t^{\prime}}) \tag{5a}\] \[\hat{\mathbf{h}}^{I}_{t} ={\bf o}_{t}\odot\tanh(\hat{\mathbf{c}}^{l}_{t}) \tag{5b}\]

Application of the above process to all layers \(L\), will yield \(\hat{\mathbf{h}}^{L}_{t}\), which is \({\bf h}_{t}\). Note that in implementation, all \(\hat{\mathbf{c}}^{l}_{t}\) and \(\hat{\mathbf{h}}^{I}_{t}\) (\(1\leq l\leq L\)) at time step \(t\) are stored, although we only care about \(\hat{\mathbf{h}}^{L}_{t}\) (\({\bf h}_{t}\)).

### Left Dependent Tree LSTMs

TreeLSTM computes \(P(w|\mathcal{D}(w))\) based on the dependency path \(\mathcal{D}(w)\), which ignores the interaction between left and right dependents on the same level. In many cases, TreeLSTM will use a verb to predict its object directly without knowing its subject. For example, in Figure 2, TreeLSTM uses \(\langle\text{root}\), Right\(\rangle\) and \(\langle\text{sold}\), Right\(\rangle\) to predict _ears_. This information is unfortunately not specific to cars (many things can be sold, e.g., chocolates, candy). Considering manufacturer, the left dependent of sold would help predict cars more accurately.

In order to jointly take left and right dependents into account, we employ yet another LSTM, which goes from the furthest left dependent to the closest left dependent (Id is a shorthand for left dependent). As shown in Figure 4, Id LSTM learns the representation of all left dependents of a node \(w_{0}\); this representation is then used to predict the first right dependent of the same node. Non-first right dependents can also leverage the representation of left dependents, since this information is injected into the hidden state of the first right dependent and can percolate all the way. Note that in order to retain the generation capability of our model (Section 3.4), we only allow right dependents to leverage left dependents (they are generated before right dependents).

The computation of the LDTreeLSTM is almost the same as in TreeLSTM except when \(z_{t}=\textsc{Gen-R}\). In this case, let \({\bf v}_{t}\) be the corresponding left dependent sequence with length \(K\) (\({\bf v}_{t}=(w_{3},w_{2},w_{1})\) in Figure 4). Then, the hidden state (\({\bf q}_{k}\)) of \({\bf v}_{t}\) at each time step \(k\) is:

\[{\bf m}_{k} ={\bf W}_{e}\cdot e({\bf v}_{t,k}) \tag{6a}\] \[{\bf q}_{k} =\mathrm{LSTM}^{\mathrm{LD}}({\bf m}_{k},{\bf q}_{k-1}) \tag{6b}\]

where \({\bf q}_{K}\) is the representation for all left dependents. Then, the computation of the current hidden state becomes (see Equation (2) for the original computation):

\[{\bf r}_{t} =\begin{bmatrix}{\bf W}_{e}\cdot e(w_{t^{\prime}})\\ {\bf q}_{K}\end{bmatrix} \tag{7a}\] \[{\bf h}_{t} =\mathrm{LSTM}^{\textsc{Gen-R}}({\bf r}_{t},{\bf H}[:,t^{\prime}]) \tag{7b}\]

where \({\bf q}_{K}\) serves as additional input for \(\mathrm{LSTM}^{\textsc{Gen-R}}\). All other computational details are the same as in TreeLSTM (see Section 2.3).

### Model Training

On small scale datasets we employ Negative Log-likelihood (NLL) as our training objective for both TreeLSTM and LDTreeLSTM:

\[\mathcal{L}^{\mathrm{NLL}}(\emptyset)=-\frac{1}{|\mathcal{S}|}\sum_{S\in \mathcal{S}}\log P(S|T) \tag{8}\]

where \(S\) is a sentence in the training set \(\mathcal{S}\), \(T\) is the dependency tree of \(S\) and \(P(S|T)\) is defined as in Equation (1).

Figure 4: Generation of left and right dependents of node \(w_{0}\) according to LDTreeLSTM.

 

On large scale datasets (e.g., with vocabulary size of 65K), computing the output layer activations and the _softmax_ function with NLL would become prohibitively expensive. Instead, we employ Noise Contrastive Estimation (NCE; Gutmann and Hyvarinen (2012), Mnih and Teh (2012)) which treats the normalization term \(\hat{Z}\) in \(\hat{P}(w|\mathcal{D}(w_{t}))=\frac{\exp(\mathbf{W}_{h_{u}}[w_{:}]\cdot\mathbf{ h}_{t})}{2}\) as constant. The intuition behind NCE is to discriminate between samples from a data distribution \(\hat{P}(w|\mathcal{D}(w_{t}))\) and a known noise distribution \(P_{n}(w)\) via binary logistic regression. Assuming that noise words are \(k\) times more frequent than real words in the training set Mnih and Teh (2012), then the probability of a word \(w\) being from our model \(P_{d}(w,\mathcal{D}(w_{t}))\) is \(\frac{\hat{P}(w|\mathcal{D}(w_{t}))}{\hat{P}(w|\mathcal{D}(w_{t}))+kP_{n}( w)}\). We apply NCE to large vocabulary models with the following training objective:

\[L^{\text{NCE}}(\theta)= -\frac{1}{|\mathcal{S}|}\sum_{T\in S}\sum_{t=1}^{|T|}\bigg{(}\log P _{d}(w_{t},\mathcal{D}(w_{t}))\] \[+\sum_{j=1}^{k}\log[1-P_{d}(\tilde{w}_{t,j},\mathcal{D}(w_{t}))] \bigg{)}\]

where \(\tilde{w}_{t,j}\) is a word sampled from the noise distribution \(P_{n}(w)\). We use smoothed unigram frequencies (exponentiating by 0.75) as the noise distribution \(P_{n}(w)\)Mikolov et al. (2013b). We initialize \(\ln\hat{Z}=9\) as suggested in Chen et al. (2015), but instead of keeping it fixed we also learn \(\hat{Z}\) during training Vaswani et al. (2013). We set \(k=20\).

## 3 Experiments

We assess the performance of our model on two tasks: the Microsoft Research (MSR) sentence completion challenge Zweig and Burges (2012), and dependency parsing reranking. We also demonstrate the tree generation capability of our models. In the following, we first present details on model training and then present our results. We implemented our models using the Torch library Collobert et al. (2011) and our code is available at [https://github.com/XingxingZhang/td-treelstm](https://github.com/XingxingZhang/td-treelstm).

### Training Details

We trained our model with back propagation through time Rumelhart et al. (1988) on an Nvidia GPU Card with a mini-batch size of 64. The objective (NLL or NCE) was minimized by stochastic gradient descent. Model parameters were uniformly initialized in \([-0.1,0.1]\). We used the NCE objective on the MSR sentence completion task (due to the large size of this dataset) and the NLL objective on dependency parsing reranking. We used an initial learning rate of 1.0 for all experiments and when there was no significant improvement in log-likelihood on the validation set, the learning rate was divided by 2 per epoch until convergence Mikolov et al. (2010). To alleviate the exploding gradients problem, we rescaled the gradient \(g\) when the gradient norm \(||g||>5\) and set \(g=\frac{5g}{||g||}\) Pascanu et al. (2013); Sutskever et al. (2014). Dropout Srivastava et al. (2014) was applied to the 2-layer TreeLSTM and LdTreeLSTM models. The word embedding size was set to \(s=d/2\) where \(d\) is the hidden unit size.

### Microsoft Sentence Completion Challenge

The task in the MSR Sentence Completion Challenge Zweig and Burges (2012) is to select the correct missing word for 1,040 SAT-style test sentences when presented with five candidate completions. The training set contains 522 novels from the Project Gutenberg which we preprocessed as follows. After removing headers and footers from the files, we tokenized and parsed the dataset into dependency trees with the Stanford Core NLP toolkit Manning et al. (2014). The resulting training set contained 49M words. We converted all words to lower case and replaced those occurring five times or less with UNK. The resulting vocabulary size was 65,346 words. We randomly sampled 4,000 sentences from the training set as our validation set.

The literature describes two main approaches to the sentence completion task based on word vectors and language models. In vector-based approaches, all words in the sentence and the five candidate words are represented by a vector; the candidate which has the highest average similarity with the sentence words is selected as the answer. For language model-based methods, the LM computes the probability of a test sentence with each of the five candidate words, and picks the candidate completion which gives the highest probability. Our model belongs to this class of models 

Table 1 presents a summary of our results together with previoulsy published results. The best performing word vector model is ivLBL Mnih and Kavukcuoglu (2013) with an accuracy of 55.5, while the best performing single language model is LBL Mnih and Teh (2012) with an accuracy of 54.7. Both approaches are based on the log-bilinear language model Mnih and Hinton (2007). A combination of several recurrent neural networks and the skip-gram model holds the state of the art with an accuracy of 58.9 Mikolov et al. (2013). To fairly compare with existing models, we restrict the layer size of our models to 1. We observe that LtTreeLSTM consistently outperforms TreeLSTM, which indicates the importance of modeling the interaction between left and right dependents. In fact, LtTreeLSTM (\(d=400\)) achieves a new state-of-the-art on this task, despite being a single model. We also implement LSTM and bidirectional LSTM language models.3 An LSTM with \(d=400\) outperforms its smaller counterpart (\(d=300\)), however performance decreases with \(d=450\). The bidirectional LSTM is worse than the LSTM see Mnih and Teh (2012) for a similar observation). The best performing LSTM is worse than a LpTreeLSTM (\(d=300\)). The input and output embeddings (\({\bf W}_{e}\) and \({\bf W}_{ho}\)) dominate the number of parameters in all neural models except for RNNME, depRNN+3gram and ldepRNN+4gram, which include a ME model that contains 1 billion sparse n-gram features Mikolov (2012); Mirowski and Vlachos (2015). The number of parameters in TreeLSTM and LtTreeLSTM is not much larger compared to LSTM due to the tied \({\bf W}_{e}\) and \({\bf W}_{ho}\) matrices.

Footnote 3: LSTMs and BiLSTMs were also trained with NCE (\(s=d/2\); hyperparameters were tuned on the development set).

### Dependency Parsing

In this section we demonstrate that our model can be also used for parse reranking. This is not possible for sequence-based language models since they cannot estimate the probability of a tree. We use our models to rerank the top \(K\) dependency trees produced by the second order MSTParser McDon

\begin{table}
\begin{tabular}{l|c c|c} \hline
**Model** & \(d\) & \(|\theta|\) & **Accuracy** \\ \hline \hline Word Vector based Models & & & \\ \hline LSA & — & — & 49.0 \\ Skip-gram & 640 & 102M & 48.0 \\ ivLBL & 600 & 96.0M & 55.5 \\ \hline \hline Language Models & & & \\ \hline KN5 & — & — & 40.0 \\ UDPNgram & — & — & 48.3 \\ LDepNgram & — & — & 50.0 \\ RNN & 300 & 48.1M & 45.0 \\ RNNME & 300 & 1120M & 49.3 \\ depRNN+3gram & 100 & 1014M & 53.5 \\ ldepRNN+4gram & 200 & 1029M & 50.7 \\ LBL & 300 & 48.0M & 54.7 \\ \hline LSTM & 300 & 29.9M & 55.00 \\ LSTM & 400 & 40.2M & 57.02 \\ LSTM & 450 & 45.3M & 55.96 \\ Bidirectional LSTM & 200 & 33.2M & 48.46 \\ Bidirectional LSTM & 300 & 50.1M & 49.90 \\ Bidirectional LSTM & 400 & 67.3M & 48.65 \\ \hline \hline Model Combinations & & & \\ \hline RNNMEs & — & — & 55.4 \\ Skip-gram + RNNMEs & — & — & 58.9 \\ \hline \hline Our Models & & & \\ \hline TreeLSTM & 300 & 31.6M & 55.29 \\ LtdTreeLSTM & 300 & 32.5M & 57.79 \\ TreeLSTM & 400 & 43.1M & 56.73 \\ LlTreeLSTM & 400 & 44.7M & **60.67** \\ \hline \end{tabular}
\end{table}
Table 1: Model accuracy on the MSR sentence completion task. The results of KN5, RNNME and RNNMEs are reported in Mikolov (2012), LSA and RNN in Zweig et al. (2012), UDepNgram and LDepNgram in Gubbins and Vlachos (2013), dePRNN+3gram and depRNN+4gram in Mirowski and Vlachos (2015), LBL in Mnih and Teh (2012), Skip-gram and Skip-gram+RNNMEs in Mikolov et al. (2013), and ivLBL in Mnih and Kavukcuoglu (2013); \(d\) is the hidden size and \(|\theta|\) the number of parameters in a model.

\begin{table}
\begin{tabular}{p{113.8pt}|p{28.5pt} p{28.5pt}|p{28.5pt}} \hline
* & \multicolumn{2}{c|}{Development UAS LAS} & \multicolumn{2}{c}{Test UAS LAS} \\ \hline \hline MSTParser-2nd & 92.20 & 88.78 & 91.63 & 88.44 \\ TreeLSTM & 92.51 & 89.07 & 91.79 & 88.53 \\ TreeLSTM
* & 92.64 & 89.09 & 91.97 & 88.69 \\ LdTreeLSTM & **92.66*
* & **89.14*
* & **91.99*
* & **88.69*
* \\ \hline \hline NN parser
* & 92.00 & 89.70 & 91.80 & 89.60 \\ S-LSTM
* & **93.20*
* & **90.90*
* & **93.10*
* & **90.90*
* \\ \hline \end{tabular}
\end{table}
Table 2: Performance of TreeLSTM and LüTreeLSTM on reranking the top dependency trees produced by the 2nd order MSTParser McDonald and Pereira (2006). Results for the NN and S-LSTM parsers are reported in Chen and Manning (2014) and Dyer et al. (2015), respectively.
* indicates that the model is initialized with pre-trained word vectors.
 

ald and Pereira, 2006).4 We follow closely the experimental setup of Chen and Manning (2014) and Dyer et al. (2015). Specifically, we trained TreeLSTM and LdTreeLSTM on Penn Treebank sections 2-21. We used section 22 for development and section 23 for testing. We adopted the Stanford basic dependency representations De Marneffe et al. (2006); part-of-speech tags were predicted with the Stanford Tagger Toutanova et al. (2003). We trained TreeLSTM and LdTreeLSTM as language models (singletons were replaced with UNK) and did not use any POS tags, dependency labels or composition features, whereas these features are used in Chen and Manning (2014) and Dyer et al. (2015). We tuned \(d\), the number of layers, and \(K\) on the development set.

Footnote 4: [http://www.seas.upenn.edu/](http://www.seas.upenn.edu/) strccltrn/MSTParser

Table 2 reports unlabeled attachment scores (UAS) and labeled attachment scores (LAS) for the MSTParser, TreeLSTM (\(d=\) 300, 1 layer, \(K=\) 2), and LdTreeLSTM (\(d=\) 200, 2 layers, \(K=\) 4). We also include the performance of two neural network-based dependency parsers; Chen and Manning (2014) use a neural network classifier to predict the correct transition (NN parser); Dyer et al. (2015) also implement a transition-based dependency parser using LSTMs to represent the contents of the stack and buffer in a continuous space. As can be seen, both TreeLSTM and LdTreeLSTM outperform the baseline MSTParser, with LdTreeLSTM performing best. We also initialized the word embedding matrix \({\bf W}_{e}\) with pre-trained GLOVE vectors Pennington et al. (2014). We obtained a slight improvement over TreeLSTM (TreeLSTM* in Table 2; \(d=\) 200, 2 layer, \(K=\) 4) but no improvement over LdTreeLSTM. Finally, notice that LdTreeLSTM is slightly better than the NN parser in terms of UAS but worse than the S-LSTM parser. In the future, we would like to extend our model so that it takes labeled dependency information into account.

### Tree Generation

This section demonstrates how to use a trained LdTreeLSTM to generate tree samples. The generation starts at the root node. At each time step \(t\), for each node \(w_{t}\), we add a new edge and node to the tree. Unfortunately during generation, we do not know which type of edge to add. We therefore use four binary classifiers (Add-Left, Add-Right, Add-Nx-Left and Add-Nx-Right) to predict whether we should add a Left, Right, Nx-Left or Nx-Right edge.5 Then when a classifier predicts true, we use the corresponding LSTM to generate a new node by sampling from the predicted word distribution in Equation (3). The four classifiers take the previous hidden state \({\bf H}[:,t^{\prime}]\) and the output embedding of the current node \({\bf W}_{ho}\cdot e(w_{t})\) as features.6 Specifically, we use a trained LdTreeLSTM to go through the training corpus and generate hidden states and embeddings as input features; the corresponding class labels (true and false) are "read off" the training dependency trees. We use two-layer rectifier networks Glorot et al. (2011) as the four classifiers with a hidden size of 300. We use the same LdTreeLSTM model as in Section 3.3 to generate dependency trees. The classifiers were trained using AdaGrad Duchi et al. (2011) with a learning rate of 0.01. The accuracies of Add-Left, Add-Right, Add-Nx-Left and Add-Nx-Right are 94.3%, 92.6%, 93.4% and 96.0%, respectively. Fig

Figure 5: Generated dependency trees with LdTreeLSTM trained on the PTB.

 

ure 5 shows examples of generated trees.

## 4 Conclusions

In this paper we developed TreeLSTM (and LdTreeLSTM), a neural network model architecture, which is designed to predict tree structures rather than linear sequences. Experimental results on the MSR sentence completion task show that LdTreeLSTM is superior to sequential LSTMs. Dependency parsing reranking experiments highlight our model's potential for dependency parsing. Finally, the ability of our model to generate dependency trees holds promise for text generation applications such as sentence compression and simplification Filippova et al. (2015). Although our experiments have focused exclusively on dependency trees, there is nothing inherent in our formulation that disallows its application to other types of tree structure such as constituent trees or even taxonomies.

We would like to thank Adam Lopez, Frank Keller, Iain Murray, Li Dong, Brian Roark, and the NAACL reviewers for their valuable feedback. Xingxing Zhang gratefully acknowledges the financial support of the China Scholarship Council (CSC). Liang Lu is funded by the UK EPSRC Programme Grant EP/I031022/1, Natural Speech Technology (NST).

## References

* Bengio et al. (2003) Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. _The Journal of Machine Learning Research_, 3:1137-1155.
* Charniak (2001) Eugene Charniak. 2001. Immediate-head parsing for language models. In _Proceedings of the 39th Annual Meeting on Association for Computational Linguistics_, pages 124-131. Association for Computational Linguistics.
* Chelba and Jelinek (2000) Ciprian Chelba and Frederick Jelinek. 2000. Structured language modeling. _Computer Speech and Language_, 14(4):283-332.
* Chelba et al. (1997) Ciprian Chelba, David Engle, Frederick Jelinek, Victor Jimenez, Sanjeev Khudanpur, Lidia Mangu, Harry Printz, Eric Ristad, Ronald Rosenfeld, Andreas Stolcke, et al. 1997. Structure and performance of a dependency language model. In _EUROSPEECH_. Citeseer.
* Chen and Manning (2014) Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 740-750, Doha, Qatar, October. Association for Computational Linguistics.
* Chen et al. (2015) X Chen, X Liu, MJF Gales, and PC Woodland. 2015. Recurrent neural network language model training with noise contrastive estimation for speech recognition. In _In 40th IEEE International Conference on Acoustics, Speech and Signal Processing_, pages 5401-5405, Brisbane, Australia.
* Collobert et al. (2011) Ronan Collobert, Koray Kavukcuoglu, and Clement Farabet. 2011. Torch7: A matlab-like environment for machine learning. In _BigLearn, NIPS Workshop_, number EPFL-CONF-192376.
* De Marneffe et al. (2006) Marie-Catherine De Marneffe, Bill MacCartney, Christopher D Manning, et al. 2006. Generating typed dependency parses from phrase structure parses. In _Proceedings of LREC_, volume 6, pages 449-454.
* Duchi et al. (2011) John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. _The Journal of Machine Learning Research_, 12:2121-2159.
* Dyer et al. (2015) Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 334-343, Beijing, China, July. Association for Computational Linguistics.
* Eisner (1996) Jason M Eisner. 1996. Three new probabilistic models for dependency parsing: An exploration. In _Proceedings of the 16th conference on Computational linguistics-Volume 1_, pages 340-345. Association for Computational Linguistics.
* Emami et al. (2003) Ahmad Emami, Peng Xu, and Frederick Jelinek. 2003. Using a connectionist model in a syntactical based language model. In _Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing_, pages 372-375, Hong Kong, China.
* Filippova et al. (2015) Katja Filippova, Enrique Alfonseca, Carlos A Collmenares, Lukasz Kaiser, and Oriol Vinyals. 2015. Sentence compression by deletion with lstms. In _EMNLP_, pages 360-368.
* Glorot et al. (2011) Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep sparse rectifier neural networks. In _International Conference on Artificial Intelligence and Statistics_, pages 315-323 

Alan Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In _Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on_, pages 6645-6649. IEEE.
* Gubbins and Vlachos [2013] Joseph Gubbins and Andreas Vlachos. 2013. Dependency language models for sentence completion. In _EMNLP_, pages 1405-1410, Seattle, Washington, USA, October. Association for Computational Linguistics.
* Gutmann and Hyvarinen [2012] Michael U Gutmann and Aapo Hyvarinen. 2012. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. _The Journal of Machine Learning Research_, 13(1):307-361.
* Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Hochreiter [1998] Sepp Hochreiter. 1998. Vanishing gradient problem during learning recurrent neural nets and problem solutions. _International Journal of Uncertainty, Fuzziness and Knowledge-based Systems_, 6(2):107-116.
* Manning et al. [2014] Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In _Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 55-60.
* McDonald and Pereira [2006] Ryan T McDonald and Fernando CN Pereira. 2006. Online learning of approximate dependency parsing algorithms. In _EACL_.
* Mikolov et al. [2010] Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In _INTERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan, September 26-30, 2010_, pages 1045-1048.
* Mikolov et al. [2013a] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In _Proceedings of the 2013 International Conference on Learning Representations_, Scottsdale, Arizona, USA.
* Mikolov et al. [2013b] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In _Advances in Neural Information Processing Systems 26_, pages 3111-3119.
* Mikolov [2012] Tomas Mikolov. 2012. _Statistical Language Models based on Neural Networks_. Ph.D. thesis, Brno University of Technology.
* Piotr Mirowski and Vlachos [2015] Piotr Mirowski and Andreas Vlachos. 2015. Dependency recurrent neural language models for sentence completion. In _ACL_, pages 511-517, Beijing, China, July. Association for Computational Linguistics.
* Mnih and Hinton [2007] Andriy Mnih and Geoffrey Hinton. 2007. Three new graphical models for statistical language modelling. In _Proceedings of the 24th International Conference on Machine Learning_, pages 641-648.
* Mnih and Kavukcuoglu [2013] Andriy Mnih and Koray Kavukcuoglu. 2013. Learning word embeddings efficiently with noise-contrastive estimation. In _Advances in Neural Information Processing Systems 26_, pages 2265-2273.
* Mnih and Teh [2012] Andriy Mnih and Yee Whye Teh. 2012. A fast and simple algorithm for training neural probabilistic language models. In _Proceedings of the 29th International Conference on Machine Learning_, pages 1751-1758, Edinburgh, Scotland.
* Pascanu et al. [2013] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In _Proceedings of the 31st International Conference on Machine Learning_, pages 1310-1318, Atlanta, Georgia, USA.
* Pennington et al. [2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. _EMNLP_, 12:1532-1543.
* Pollack [1990] Jordan B. Pollack. 1990. Recursive distributed representations. _Artificial Intelligence_, 1-2(46):77-105.
* Roark [2001] Brian Roark. 2001. Probabilistic top-down parsing and language modeling. _Computational linguistics_, 27(2):249-276.
* Rumelhart et al. [1988] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1988. Learning representations by backpropagating errors. _Cognitive modeling_, 5:3.
* Sennrich [2015] Rico Sennrich. 2015. Modelling and optimizing on syntactic n-grams for statistical machine translation. _Transactions of the Association for Computational Linguistics_, 3:169-182.
* Shen et al. [2008] Libin Shen, Jinxi Xu, and Ralph Weischedel. 2008. A new string-to-dependency machine translation algorithm with a target dependency language model. In _Proceedings of ACL-08: HLT_, pages 577-585, Columbus, Ohio, USA.
* Socher et al. [2011a] Richard Socher, Eric H. Huang, Jeffrey Pennington, Christopher D. Manning, and Andrew Ng. 2011a. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In _Advances in Neural Information Processing Systems_, pages 801-809.
* Socher et al. [2011b] Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011b. Semi-supervised recursive autoencoders for predicting sentiment distributions. In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_, pages 151-161, Edinburgh, Scotland, UK 

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. _The Journal of Machine Learning Research_, 15(1):1929-1958.
* [Sutskever et al.2014] Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In _Advances in Neural Information Processing Systems_, pages 3104-3112.
* [Tai et al.2015] Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1556-1566, Beijing, China, July. Association for Computational Linguistics.
* [Toutanova et al.2003] Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In _Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1_, pages 173-180. Association for Computational Linguistics.
* [Vaswani et al.2013] Ashish Vaswani, Yinggong Zhao, Victoria Fossum, and David Chiang. 2013. Decoding with large-scale neural language models improves translation. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1387-1392, Seattle, Washington, USA.
* [Vinyals et al.2015] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In _The IEEE Conference on Computer Vision and Pattern Recognition_, Boston, Massachusetts, USA.
* [Zhang2009] Ying Zhang. 2009. _Structured language models for statistical machine translation_. Ph.D. thesis, Johns Hopkins University.
* [Zweig and Burges2012] Geoffrey Zweig and Chris J.C. Burges. 2012. A challenge set for advancing language modeling. In _Proceedings of the NAACL-HLT 2012 Workshop: Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT_, pages 29-36, Montreal, Canada.
* [Zweig et al.2012] Geoffrey Zweig, John C Platt, Christopher Meek, Christopher JC Burges, Aniru Yessenalina, and Qiang Liu. 2012. Computational approaches to sentence completion. In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1_, pages 601-610. Association for Computational Linguistics