Venue: NAACL 2019

# Unsupervised Latent Tree Induction with

Deep Inside-Outside Recursive Autoencoders

Andrew Drozdov1 , Pat Verga1 , Mohit Yadav1 ,

**Mohit Iyyer, and Andrew McCallum**

College of Information and Computer Sciences

University of Massachusetts Amherst

{adrozdov, pat, ymohit, miyyer, mccallum}@cs.umass.edu

Equal contribution, randomly ordered.

Footnote 1: footnotemark:

###### Abstract

We introduce deep inside-outside recursive autoencoders (DIORA), a fully-unsupervised method for discovering syntax that simultaneously learns representations for constituents within the induced tree. Our approach predicts each word in an input sentence conditioned on the rest of the sentence and uses inside-outside dynamic programming to consider all possible binary trees over the sentence. At test time the CKY algorithm extracts the highest scoring parse. DIORA achieves a new state-of-the-art F1 in unsupervised binary constituency parsing (unlabeled) in two benchmark datasets, WSJ and MultiNLI.

## 1 Introduction

Syntactic parse trees are useful for downstream tasks such as relation extraction Gamallo et al. (2012), semantic role labeling Sutton and McCallum (2005); He et al. (2018), machine translation Aharoni and Goldberg (2017); Eriguchi et al. (2017); Zaremoodi and Haffari (2018), and text classification Li and Roth (2006); Tai et al. (2015). Traditionally, supervised parsers trained on datasets such as the Penn Treebank Marcus et al. (1993) are used to obtain syntactic trees. However, the treebanks used to train these supervised parsers are typically small and restricted to the newswire domain. Unfortunately, models trained on newswire treebanks tend to perform considerably worse when applied to new types of data, and creating new domain specific treebanks with syntactic annotations is expensive and time-consuming.

Motivated by the desire to address the limitations of supervised parsing and by the success of large-scale unsupervised modeling such as ELMo and BERT Peters et al. (2018); Devlin et al. (2019), we propose a new deep learning method of unsupervised parser training that can extract both shallow parses (i.e., noun phrases or entities) and full syntactic trees from any domain or language automatically _without requiring any labeled training data_. In addition to producing parses, our model simultaneously builds representations for internal constituents that reflect syntactic and semantic regularities which can be leveraged by downstream tasks.

Our model builds on existing work developing latent tree chart parser Socher et al. (2011); Le and Zuidema (2015); Yogatama et al. (2017); Maillard et al. (2017); Choi et al. (2018). These methods produce representations for all internal nodes in the tree (cells in the chart), each generated as a soft weighting over all possible sub-trees (SS2). Unfortunately, they still require sentence-level annotations during training, as they are all trained to optimize a downstream task, typically natural language inference.

To address these limitations, we present deep inside-outside recursive autoencoders (DIORA) which enable unsupervised discovery and representation of constituents without requiring any supervised training data. DIORA incorporates the inside-outside algorithm Baker (1979); Lari and Young (1990) into a latent tree chart parser. The bottom-up inside step calculates a representation for all possible constituents within a binary tree over the input sentence. This step is equivalent to the forward-pass of previous latent tree chart parsers Maillard et al. (2017). These inside representations only encode the current subtree, ignor

Figure 1: An unlabeled binary constituency parse from DIORA matching the ground truth.



ing all outside context. Thus, we perform an additional top-down outside calculation for each node in the tree, providing external context into the subtree representations in each chart cell. The model is then trained with the objective that the outside representations of the leaf cells should reconstruct the corresponding leaf input word, analogous to masked language model Devlin et al. (2019) pretraining, except by using dynamic programming we predict every word from a completely unmasked context. The single most likely tree can be recovered using the CKY algorithm and compatibility scores between constituents. Previous work either predict trees that are not well aligned with known treebanks Yogatama et al. (2017); Choi et al. (2018), or has no mechanism for explicitly modeling phrases, requiring a complex procedure to extract syntactic structures Shen et al. (2018).

To probe different properties of our model, we run experiments on unsupervised parsing, segment recall, and phrase representations. DIORA achieves multiple new state-of-the-art results for unsupervised constituency parsing (absolute improvements of **13.7%**, **11.5%**, and **7.8%** on WSJ, WSJ-40, and MultiNLI), has a greater recall on more constituent types than a strong baseline, and produces meaningful phrase representations.

## 2 DIORA: Deep Inside-Outside Recursive Autoencoders

Our goal is to design a model and unsupervised training procedure that learns structure from raw text. The design of DIORA is based on our hypothesis is that the most effective compression of a sentence will be derived from following the true syntactic structure of the underlying input. Our approach builds on previous latent tree chart parsers which are augmented with the inside-outside algorithm Baker (1979); Lari and Young (1990) and trained to reproduce each input word from its outside context. Based on our hypothesis, loosely inspired by the linguistic "substitution principle" Frege (1960), the model will best reconstruct the input by discovering and exploiting syntactic regularities of the text.

The inside pass of our method recursively compresses the input sequence, at each step inputting the vector representations of the two children into a composition function (SS2.1.1) that outputs an inside vector representation of the parent. This process continues up to the root of the tree, eventually yielding a single vector representing the entire sentence (Figure 2a). This is loosely analogous to the compression step of an autoencoder and equivalent to existing latent tree chart parsers Morawal et al. (2017). Following this, we initiate the outside pass of our algorithm with a generic (root) representation that is learned as a separate parameter. As the outside step of the inside-outside algorithm (Figure 2b), we unfold until finally producing representations of the leaf nodes. These leaves are then optimized to reconstruct the input sentence as done in an autoencoder-based deep neural network.

### Filling the Chart with Inside-Outside

Each inside representation is the root of a particularly sub-tree, and that representation is generated by considering only the descendant constituents within that sub-tree, ignoring any outside context. After the inside representations are calculated, we

Figure 2: The illustrated inside and outside pass of DIORA operating over an input of length three, ‘the cat drinks’. a) The inside pass: The inside vector \(\bar{a}(k)\) for the phrase ‘the cat drinks’ is a weighted average of the compositions for the two possible segmentations - ((the cat), drinks) and (the, (cat drinks)). The scalar weights come from a learned compatibility function. b) The outside pass: The outside vector \(\tilde{b}(k)\) for the phrase ’the cat’ is a function of the outside vector of its parent ‘the cat drinks’ and the inside vector of its sibling ‘drinks’.

 

perform a top-down outside pass to compute outside representations. The outside representations are encoded by looking at only the context of a given sub-tree. Once the chart is filled, each constituent \(k\) (cell in the chart) is associated with an inside vector \(\bar{a}(k)\), an outside vector \(\bar{b}(k)\), inside compatibility score \(\bar{e}(k)\) and outside compatibility score \(\bar{f}(k)\).

The input to our model is a sentence \(\mathbf{x}\) made up of \(T\) tokens, \(x_{0},x_{1},...,x_{T-1}\). Each token \(x_{i}\) has a corresponding pre-trained embedded vector \(v_{i}\).

#### 2.1.1 Inside Pass

For each pair of neighboring constituents \(i\) and \(j\)1, we compute a _compatibility_ score and a _composition_ vector. The score and vector that represent a particular span \(k\) are computed using a soft weighting over all possible pairs of constituents, that together fully cover the span (we refer to this set of constituent pairs as \(\{k\}\)).

Footnote 1: The symbols \(i\), \(j\), and \(k\) are identifiers of spans from the input \(\mathbf{x}\). The symbol \(i^{*}\) identifies a token from the set of negative examples \(\{\mathbf{x}^{*}\}\).

Vectors for spans of length 1 are initialized as a non-linear transformation 2 of the embedded input \(v_{i}\), and the scores associated with these spans are set to \(0\):

Footnote 2: This function shares its bias term \(b\) with \(\mathrm{Compose}_{a}\), although \(U_{\psi}\) is not tied to any other weights.

Higher levels of the chart are computed as a weighted summation of constituent pairs:

\[\bar{a}(k)=\sum_{i,j\in\{k\}}e(i,j)\;a(i,j)\] \[\bar{e}(k)=\sum_{i,j\in\{k\}}e(i,j)\;\hat{e}(i,j)\]

The compatibility function \(\hat{e}\) is meant to produce a score for how likely a pair of neighboring cells are to be merged. We implement this as a bilinear function of the vectors from neighboring spans, using a learned parameter matrix \(S\). We additionally add the individual scores from each two merging cells. Intuitively, these individual scores correspond to how likely each of the cells would exist in the final binary tree independently. The formula for the compatibility function (and its normalized form \(e\)) is defined as follows:

\[e(i,j)=\frac{\exp(\hat{e}(i,j))}{\sum\limits_{i,j\in\{k\}}\exp( \hat{e}(\hat{i},\hat{j}))}\] \[\hat{e}(i,j)=\phi(\bar{a}(i),\bar{a}(j);S_{\alpha})+\bar{e}(i)+ \bar{e}(j)\]

Where the bilinear projection \(\phi\) is defined as:

\[\phi(u,v;W)=u^{\top}Wv\]

For the composition function \(a\) we used either a \(\mathrm{TreeLSTM}\)(Tai et al., 2015) or a 2-layer \(\mathrm{MLP}\) (see Appendix A.2 for more precise definitions on both methods). In order for the remainder of equations to remain agnostic to the choice of composition function, we refer to the function as \(\mathrm{Compose}\), which produces a hidden state vector \(h\) and, in the case of \(\mathrm{TreeLSTM}\), a cell state vector \(c\), resulting in:

\[a(i,j)=\mathrm{Compose}_{\alpha}(\bar{a}(i),\bar{a}(j))\]

#### 2.1.2 Outside Pass

The outside computation is similar to the inside pass (depicted in Figure 1(b)).

The root node of the outside chart is learned as a bias. Descondant cells are predicted using a disambiguation over the possible outside contexts. Each component of the context consists of a sibling cell from the inside chart and a parent cell from the outside chart.

The function \(f\) is analogous to the function \(e\). It is normalized over constituent pairs \(i,j\) for the span \(k\), and is used to disambiguate among the many outside contexts. The function \(b\) generates a phrase representation for the missing sibling cell. Equations for the outside computation follow:

\[\bar{b}(k)=\sum_{i,j\in\{k\}}f(i,j)\;b(i,j)\] \[\bar{f}(k)=\sum_{i,j\in\{k\}}f(i,j)\;\hat{f}(i,j)\] \[b(i,j)=\mathrm{Compose}_{\beta}(\bar{a}(i),\bar{b}(j))\] \[\hat{f}(i,j)=\phi(\bar{a}(i),\bar{b}(j);S_{\beta})+\bar{e}(i)+ \bar{f}(j)\]

In the majority of our experiments, the \(\mathrm{Compose}\) used in \(b\) shares parameters with \(a\) used in the inside pass, as do the compatibility functions \(\hat{e}\) and \(\hat{f}\) (see SS3.4 for results on the effects of parameter sharing).

 



### Training Objective

To train our model we use an autoencoder-like language modeling objective. In a standard autoencoder, the entire input \({\mathbf{x}}\) is compressed into a single lower dimensional representation. This representation, \({\mathbf{z}}\), is then decompressed and trained to reconstruct \({\mathbf{x}}\). In our model, we never condition the reconstruction of \({\mathbf{x}}\) on a single \({\mathbf{z}}\) because the root's outside representation is initialized with a bias rather than the root's own inside vector. Instead, we reconstruct \({\mathbf{x}}\) conditioned on the many sub-tree roots, each of which is only a compression of a _subset_ of the input.

To approximate this reconstruction we use a max-margin loss considering a set \(\{{\mathbf{x^{*}}}\}\) of \(N\) negative examples that are sampled according to their frequency from the vocabulary (further details in Appendix A.1). The terminal outside vector \(\bar{b}(i)\) is trained to predict its original input \(v_{i}\).

The per-instance loss function is described in Equation 1:

\[L_{\mathbf{x}}=\sum_{i=0}^{T-1}\sum_{i^{*}=0}^{N-1}\max(0,1 -\bar{b}(i)\cdot\bar{a}(i)\] \[+\bar{b}(i)\cdot\bar{a}(i^{*})) \tag{1}\]

The max-margin loss does not provide a gradient if the predicted vector is closer to its ground truth than the negative example by a margin greater than \(1\). For that reason, we also experimented with an objective based on cross-entropy, described in Equation 2:

\[Z^{*} =\sum_{i^{*}=0}^{N-1}\exp(\bar{b}(i)\cdot\bar{a}(i^{*}))\] \[L_{\mathbf{x}} =-\sum_{i=0}^{T-1}\log\frac{\exp(\bar{b}(i)\cdot\bar{a}(i))}{\exp (\bar{b}(i)\cdot\bar{a}(i))+Z^{*}} \tag{2}\]

```
1:procedureCKY(chart) Initialize terminal values.
2:forach\(k\in\mathrm{chart}~{}|~{}\textsc{size}(k)=1\)do
3:\(x_{k}\gets 0\) Calculate a maximum score for each span, and record a backpointer.
4:foreach\(k\in\mathrm{chart}\)do
5:\(x_{k}\leftarrow\max\limits_{i,j\in\{k\}}[x_{i}+x_{j}+e(i,j)]\)
6:\(\pi_{k}^{i},\pi_{k}^{j}\leftarrow\underset{i,j\in\{k\}}{\arg\max}[x_{i}+x_{ j}+e(i,j)]\) Backtrack to get the maximal tree.
7:procedureBacktrack(\(k\))
8:if\(\textsc{size}(k)=1\)then
9:return k
10:\(i\leftarrow\textsc{Backtrack}(\pi_{k}^{i})\)
11:\(j\leftarrow\textsc{Backtrack}(\pi_{k}^{j})\)
12:return (i, j)
13:return Backtrack(\(k\leftarrow\text{root}\))
```

**Algorithm 1** Parsing with DIORA

### DIORA CKV Parsing

To obtain a parse with DIORA, we populate an inside and outside chart using the input sentence. We can extract the maximum scoring parse based on our single grammar rule using the CKY procedure (Kasami, 1966; Younger, 1967). The steps for this procedure are described in Algorithm 1 and its runtime complexity in Appendix A.4.

## 3 Experiments

To evaluate the effectiveness of DIORA, we run experiments on unsupervised parsing, unsupervised segment recall, and phrase similarity. The model has been implemented in PyTorch Team (2018) and the code is published online.3 For training details, see Appendix A.1.

Footnote 3: [https://github.com/iesl/diora](https://github.com/iesl/diora)

### Unsupervised Parsing

We first evaluate how well our model predicts a full unlabeled constituency parse. We look at two data sets used in prior work Htut et al. (2018), The Wall Street Journal (WSJ) section of Penn Treebank Marcus et al. (1993), and the automatic parses MultiNLI Williams et al. (2018). WSJ has gold human-annotated parses and MultiNLI contains automatic parses derived from a supervised parser Manning et al. (2014).

In addition to PRPN Shen et al. (2018),4 we compare our model to deterministically constructed left branching, right branching, balanced, and random trees. We also compare to ON-LSTM Shen et al. (2019), an extension of the PRPN model, RL-SPINX Yogatama et al. (2017), an unsupervised shift-reduce parser, and ST-Gumbel Choi et al. (2018), an unsupervised chart parser. The latter two of these models are trained to predict the downstream task of natural language inference (NLI).





#### 3.1.1 Binarized WSJ and MultiNLI results

For the full WSJ test set and MultiNLI datasets we follow the experimental setup of previous work Williams et al. (2018). We binarize target trees using Stanford CoreNLP Manning et al. (2014) and do not remove punctuation (experiments in SS3.1.2 do remove punctuation).

Latent tree models have been shown to perform particularly poorly on attachments at the beginning and end of the sequence Williams et al. (2018). To address this, we incorporate a post-processing heuristic (denoted as \(+{\rm PP}\) in result tables)5. This heuristic simply attaches trailing punctuation to the root of the tree, regardless of its predicted attachment.

Footnote 5: We did not have access to predictions or an implementation of the concurrent ON-LSTM model and therefore could not apply the \(+{\rm PP}\) heuristic.

In Table 1, we see that DIORA\({}^{+{\rm PP}}\) achieves the highest average and maximum F1 from five random restarts. This model achieves a mean F1 7 points higher than ON-LSTM and an increase of over 6.5 max F1 points. We also see that DIORA exhibits much less variance between random seeds than ON-LSTM. Additionally, we find that PRPN-UP and DIORA benefit much more from the \(+{\rm PP}\) heuristic than PRPN-LM. This is consistent with qualitative analysis showing that DIORA and PRPN-UP incorrectly attach trailing punctuation much more often than PRPN-LM.

On the MultiNLI dataset, PRPN-LM is the top performing model without using the \(+{\rm PP}\) heuristic while DIORA matches PRPN-UP (Table 2. Using the heuristic, DIORA greatly surpasses both variants of PRPN. However, it is worth noting that this is not a gold standard evaluation and instead evaluates a model's ability to replicate the output of a trained parser Manning et al. (2014). A second caveat is that SNLI Bowman et al. (2015) and MultiNLI contain several non-newswire domains. Syntactic parsers often suffer significant performance drops when predicting outside of the newswire domain that the models were trained on.

#### 3.1.2 WSJ-10 and WSJ-40 results

We also compare our models to two subsets of the WSJ dataset that were used in previous unsupervised parsing evaluations. WSJ-10 and WSJ-40 contain sentences up to length 10 and 40 respectively after punctuation removal. We do not binarize either of these two splits in order to compare to previous work (see Appendix A.3 for more details on WSJ split differences). Not binarizing the target trees sets an upper-bound on the performance of our models, denoted as UB in Table 3.

We compare against previous notable models for this task: CCM Klein and Manning (2002) uses the EM algorithm to learn probable nested

\begin{table}
\begin{tabular}{l|c c c} \hline \hline
* \\ ST-Gumbel
- GRU\(\dagger\) & 22.8 \(\pm_{1.6}\) & 25.0 &
- \\ \hline PRPN-UP & 38.3 \(\pm_{0.5}\) & 39.8 & 5.9 \\ PRPN-LM & 35.0 \(\pm_{5.4}\) & 42.8 & 6.2 \\ ON-LSTM & 47.7 \(\pm_{1.5}\) & 49.4 & 5.6 \\ DIORA & 48.9 \(\pm_{0.5}\) & 49.6 & 8.0 \\ \hline PRPN-UP\({}^{+{\rm PP}}\) &
- & 45.2 & 6.7 \\ PRPN-LM\({}^{+{\rm PP}}\) &
- & 42.4 & 6.3 \\ DIORA\({}^{+{\rm PP}}\) & **55.7**\(\pm_{0.4}\) & **56.2** & 8.5 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Full WSJ (test set) unsupervised unlabeled binary constituency parsing including punctuation. \(\dagger\) indicates trained to optimize NLI task. Mean and max are calculated over five random restarts. PRPN F1 was calculated using the parse trees and results provided by Htut et al. (2018). The depth (\(\delta\)) is the average tree height. \(+{\rm PP}\) refers to post-processing heuristic that attaches trailing punctuation to the root of the tree. The top F1 value in each column is bolded.

\begin{table}
\begin{tabular}{l|c c c} \hline \hline
* & 4.9 \\ PRPN-LM & 50.4 &
- & 5.1 \\ DIORA & 51.2 & 53.3 & 6.4 \\ \hline PRPN-UP\({}^{+{\rm PP}}\) &
- & 54.8 & 5.2 \\ PRPN-LM\({}^{+{\rm PP}}\) &
- & 50.4 & 5.1 \\ DIORA\({}^{+{\rm PP}}\) & 59.0 & **59.1** & 6.7 \\ \hline \hline \end{tabular}
\end{table}
Table 2: NLI unsupervised unlabeled binary constituency parsing comparing to CoreNLP predicted parses. PRPN F1 was calculated using the parse trees and results provided by Hutt et al. (2018). F1 median and max are calculated over five random seeds and the top F1 value in each column is bolded. Note that we use median rather than mean in order to compare with previous work.


bracketings over a sentence using gold or induced part-of-speech tags, and PRLG (Ponvert et al., 2011) performs constituent parsing through consecutive rounds of sentence chunking.

In Table 3, we see that DIORA outperforms the previous state of the art for WSJ-40, PRLG, in max F1. The WSJ-10 split has been difficult for latent tree parsers such as DIORA, PRPN, and ONLSTM, none of which (including our model) are able to improve upon previous non-neural methods. However, when we compare trends between WSJ-10 and WSJ-40, we see that DIORA does a better job at extending to longer sequences.

### Unsupervised Phrase Segmentation

In many scenarios, one is only concerned with extracting particular constituent phrases rather than a full parse. Common use cases would be identifying entities, noun phrases, or verb phrases for downstream analysis. To get an idea of how well our model can perform on phrase segmentation, we consider the maximum recall of spans in our predicted parse tree. We leave methods for cutting the tree to future work and instead consider the maximum recall of our model which serves as an upper bound on its performance. Recall here is the percentage of labeled constituents that appear in our predicted tree relative to the total number of constituents in the gold tree. These scores are separated by type and presented in Table 4.

In Table 4 we see the breakdown of constituent recall across the 10 most common types. DIORA achieves the highest recall across the most types and is the only model to perform effectively on verb-phrases. Interestingly, DIORA performs worse than PRPN-LM at prepositional phrases.

### Phrase Similarity

One of the goals of DIORA is to learn meaningful representations for spans of text. Most language modeling methods focus only on explicitly modeling token representations and rely on ad-hoc postprocessing to generate representations for longer spans, typically relying on simple arithmetic functions of the individual tokens.

To evaluate our model's learned phrase representations, we look at the similarity between spans of the same type within labeled phrase datasets. We look at two datasets. CoNLL 2000 (Tjong Kim Sang and Buchholz, 2000) is a shallow parsing dataset containing spans of noun phrases, verb phrases, etc. CoNLL 2012 (Pradhan et al., 2012) is a named entity dataset containing 19 different entity types.

For each of the labeled spans with length greater than one, we first generate its phrase representation. We then calculate its cosine similarity to all other labeled spans. We then calculate if the label for that query span matches the labels for each of the \(K\) most similar other spans in the dataset. In Table 5 we report precision@\(K\) for both datasets and various values of \(K\).

The first baseline we compare against produces phrase representations from averaging context-insensitive (CI) ELMo vectors of individual tokens with the span. The second uses sentence-insensitive (SI) ELMo vectors, running the full ELMo over only the relevant tokens and ignoring the rest of the sentence. We also look at ELMo's output when given the entire sentence. When analyzing our baselines that run the full ELMo, we follow the procedure described in Peters et al. (2018) and represent phrases as a function of its first and last hidden state. We extract these states from the final ELMo layer (3rd BiLSTM) as these consistently gave the best performance among other options. For DIORA, we use the concatenation of the inside and outside representations (\([\bar{a};\bar{b}]\)).

\begin{table}
\begin{tabular}{l|c c|c c} \hline \hline  & \multicolumn{2}{c}{**WSJ-10**} & \multicolumn{2}{c}{**WSJ-40**} \\
* & 63.2 &
- \\ CCM\({}_{gold}\dagger\) &
- & 71.9 &
- & 33.7 \\ PRLG † &
- & **72.1** &
- & 54.6 \\ \hline PRPN\({}_{NLI}\) & 66.3 \({}_{\pm 0.8}\) & 68.5 &
- \\ PRPN�
 

For CoNLL 2000, we find that our model outperforms all baselines for all values of \(K\). This demonstrates DIORA's ability to capture and represent syntactic information within phrases. For CoNLL 2012, we find that DIORA outperforms both ELMoCI and ELMoSI while ELMo performs best overall. ELMoCI is surprisingly effective on this dataset even though it performed more poorly on CoNLL 2000. These results indicate that DIORA is capturing syntax quite well, but still has room to improve on more fine-grained semantic representations.

### Impact of Modeling Choices

To test the impact of our modeling choices, we compared the performance of two different losses and four different composition functions on the full WSJ validation set. The losses were covered in Equations 1 (Margin) and 2 (Softmax). The two primary methods of composition we considered were TreeLSTM [35] and MLP (a 2-hidden layer neural network). In addition, we experimented with a simple _kernel_ of the MLP input \([x;y;x\odot y;x-y]\) and with a setting where both the inside and outside parameters are _shared_.

The results are shown in Table 6. We see that MLP composition consistently performs better than with TreeLSTM, that MLP benefits from the Softmax loss, and that the best performance comes from sharing parameters. All other experimental results use this highly performant setting unless otherwise specified.

Figure 4: At times, DIORA exhibits contrary behavior to the ground truth inevitably leading to some error. DIORA’s output is shown above the ground truth.6

Footnote 6: Ground truth parses are binarized unless otherwise specified. All examples of DIORA parses are already binary. Some punctuation has been removed for easier readability.

Figure 5: DIORA often groups verbs and particles (top), sometimes exactly as the ground truth (middle). Occasionally, errors are particle-like (bottom). DIORA’s output is shown above the ground truth.7

Figure 3: DIORA can match the ground truth exactly.

 



### Qualitative Results

Looking at our model's output, we see that some trees are an exact replication of the binarized ground truth (Fig. 3), or very close (Fig. 4). For future work we intend to explore common patterns in DIORA's learned structure, although some patterns are already recognizable, such as the affinity to group particles and verbs (Fig. 5).

## 4 Related Work

**Latent Tree Learning** A brief survey of neural latent tree learning models was covered in Williams et al. (2018a). The first positive result for neural latent tree parsing was shown in Hut et al. (2018), which used a language modeling objective. The model in Liu et al. (2018) uses an inside chart and an outside procedure to calculate marginal probabilities in order to align spans between sentences in entailment.

**Neural Inside-Outside Parsers** The Inside-Outside Recursive Neural Network (IORNN) Le and Zuidema (2014) is closest to ours. It is a graph-based dependency parser that uses beam search and can reliably find accurate parses when retaining a \(k\)-best list. In contrast, our model produces the most likely parse given the learned compatibility of the constituents. The Neural CRF Parser Durrett and Klein (2015), similar to DIORA, performs exact inference on the structure of a sentence, although requires a set of grammar rules and labeled parse trees during training. DIORA, like Liu et al. (2018), has a single grammar rule that applies to any pair of constituents and does not use structural supervision.

**Learning from Raw Text** Unsupervised learning of syntactic structure has been an active research area Brill et al. (1990), including for unsupervised segmentation Ando and Lee (2000); Goldwater et al. (2009); Ponvert et al. (2011) and unsupervised dependency parsing Spitkovsky et al. (2013). Some models exploit the availability of parallel corpora in multiple languages Das Petrov (2011); Cohen et al. (2011). Others have shown that dependency parsing can be used for unsupervised constituency parsing Spitkovsky et al. (2013); Klein and Manning (2004), or that it's effective to prune a random subset of possible trees Bod (2006). These approaches aren't necessarily orthogonal to DIORA. For instance, our model may benefit when combined with an unsupervised dependency parser.

## 5 Conclusion

In this work we presented DIORA, an unsupervised method for inducing syntactic trees and representations of constituent spans. We showed

\begin{table}
\begin{tabular}{c c|c c} \hline \hline  & & \multicolumn{2}{c}{**F1\({}_{\mu}\)**} \\
**Composition** & **Loss** & \(\varnothing\) & \(+\mathrm{PP}\) \\ \hline TreeLSTM & Margin & 49.9 & 53.1 \\ TreeLSTM & Softmax & 52.0 & 52.9 \\ MLP & Margin & 49.7 & 54.4 \\ MLP & Softmax & 52.6 & 55.5 \\ MLP\({}_{\text{Kernel}}\) & Softmax & 51.8 & 54.8 \\ MLP\({}_{\text{Shared}}\) & Softmax & 50.8 & 56.7 \\ \hline \hline \end{tabular}
\end{table}
Table 6: F1 for different model variants on the binary WSJ validation set with included punctuation. The binary trees are as-is (\(\varnothing\)) or modified according to the post-processing heuristic (\(+PP\)). The mean F1 is shown across three random seeds.

\begin{table}
\begin{tabular}{l|c|c c c|c c c} \hline \hline  & & \multicolumn{4}{c|}{**CoNLL 2000**} & \multicolumn{4}{c}{**CoNLL 2012**} \\
**Model** & Dim & P@1 & P@10 & P@100 & P@1 & P@10 & P@100 \\ \hline Random & 800 & 0.684 & 0.683 & 0.680 & 0.137 & 0.133 & 0.135 \\ ELMo\({}_{CI}\) & 1024 & 0.962 & 0.955 & 0.957 & 0.708 & 0.643 & 0.544 \\ ELMo\({}_{SI}\) & 4096 & 0.970 & 0.964 & 0.955 & 0.660 & 0.624 & 0.533 \\ ELMo & 4096 & 0.987 & 0.983 & 0.974 & **0.896** & **0.847** & **0.716** \\ \hline DIORA\({}_{Int/Out}\) & 800 & **0.990** & **0.985** & **0.979** & 0.860 & 0.796 & 0.646 \\ \hline \hline \end{tabular}
\end{table}
Table 5: P@1, P@10, and P@100 for labeled chunks from CoNLL-2000 and CoNLL 2012 datasets. For all metrics, higher is better. The top value in each column is bolded. Diora uses the concatenation of the inside and outside vector at each cell which performed better than either in isolation.



inside-outside representations constructed with a latent tree chart parser and trained with an autoencoder language modeling objective learns syntactic structure of language effectively. In experiments on unsupervised parsing, chunking, and phrase representations we show our model is comparable to or outperforms previous methods, achieving the state-of-the-art performance on unsupervised unlabeled constituency parsing for the full WSJ (with punctuation), WSJ-40, and NLI datasets. We also show our model obtains higher segment recall than a comparable model and outperforms strong baselines on phrase representations on a chunking dataset.

While the current model seems to focus primarily on syntax, future work can improve the model's ability to capture fine-grained semantics. Potential avenues include training larger models over much larger corpora, extra unsupervised or weakly-supervised phrase classification objectives, and other modeling enhancements. We are also eager to apply DIORA to other domains and languages which do not have rich linguistically annotated training sets.

## Acknowledgements

We are grateful to Carolyn Anderson, Adina Williams, Phu Mon Htut, and our colleagues at UMass for help and advice, and to the UMASS NLP reading group and the anonymous reviewers for feedback on drafts of this work. This work was supported in part by the Center for Intelligent Information Retrieval, in part by the National Science Foundation (NSF) grant numbers DMRR-1534431, IIS-1514053 and CNS-0958392. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsor.

## References

* Aharoni and Goldberg [2017] Roee Aharoni and Yoav Goldberg. 2017. Towards string-to-tree neural machine translation. In _Association for Computational Linguistics (ACL)_.
* Ando and Lee [2000] Rie Kubota Ando and Lillian Lee. 2000. Mostly-unsupervised statistical segmentation of japanese: Applications to kanji. In _North American Association for Computational Linguistics (NAACL)_.
* Baker [1979] James K Baker. 1979. Trainable grammars for speech recognition. _The Journal of the Acoustical Society of America_, 65(S1):S132-S132.
* Bod [2006] Rens Bod. 2006. An all-subtrees approach to unsupervised parsing. In _Association for Computational Linguistics (ACL)_.
* Bowman et al. [2015] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Brill et al. [1990] Eric Brill, David Magerman, Mitchell Marcus, and Beatrice Santorini. 1990. Reducing linguistic structure from the statistics of large corpora. In _Information Technology, 1990:Next Decade in Information Technology', Proceedings of the 5th Jerusalem Conference on (Cat. No. 90TH0326-9)_, pages 380-389. IEEE.
* Choi et al. [2018] Jihun Choi, Kang Min Yoo, and Sang-goo Lee. 2018. Learning to compose task-specific tree structures. In _Advancement of Artificial Intelligence Conference on Artificial Intelligence (AAAI)_.
* Cohen et al. [2011] Shay B. Cohen, Dipanjan Das, and Noah A. Smith. 2011. Unsupervised structure prediction with nonparallel multilingual guidance. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Das and Petrov [2011] Dipanjan Das and Slav Petrov. 2011. Unsupervised part-of-speech tagging with bilingual graph-based projections. In _Association for Computational Linguistics (ACL)_.
* Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. _North American Association for Computational Linguistics (NAACL)_.
* Durrett and Klein [2015] Greg Durrett and Dan Klein. 2015. Neural crf parsing. In _Association for Computational Linguistics (ACL)_.
* Eriguchi et al. [2017] Akiko Eriguchi, Yoshimasa Tsuruoka, and Kyunghyun Cho. 2017. Learning to parse and translate improves neural machine translation. In _Association for Computational Linguistics (ACL)_.
* Frege [1960] Friedrich Ludwig Gottlob Frege. 1960. On sense and reference. In _Zeitschrift fur Philosophile und philosophische Kritik 100 (1892) 25-50; translated in Translations from the Philosophical Writings of Gotllob Frege (ed. by P. Geach and M. Black)_. Oxford.
* Gamallo et al. [2012] Pablo Gamallo, Marcos Garcia, and Santiago Fernandez-Lanza. 2012. Dependency-based open information extraction. In _Joint workshop on unsupervised and semi-supervised learning in NLP_. Association for Computational Linguistics (ACL).
* Goldwater et al. [2009] Sharon Goldwater, Thomas L Griffiths, and Mark Johnson. 2009. A bayesian framework for word segmentation: Exploring the effects of context. _Cognition_, 112:21-54 

Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018. Jointly predicting predicates and arguments in neural semantic role labeling. In _Association for Computational Linguistics (ACL)_.
* Htut et al. [2018] Phu Mon Hutt, Kyunghyun Cho, and Samuel R Bowman. 2018. Grammar induction with neural language models: An unusual replication. In _Empirical Methods in Natural Language Processing (EMNLP): Short Paper_.
* Kasami [1966] Tadao Kasami. 1966. An efficient recognition and syntax-analysis algorithm for context-free languages. _Coordinated Science Laboratory Report no. R-257_.
* Kingma and Ba [2014] Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _CoRR_, abs/1412.6980.
* Klein and Manning [2002] Dan Klein and Christopher D. Manning. 2002. A generative constituent-context model for improved grammar induction. In _Association for Computational Linguistics (ACL)_.
* Klein and Manning [2004] Dan Klein and Christopher D. Manning. 2004. Corpus-based induction of syntactic structure: Models of dependency and constituency. In _Association for Computational Linguistics (ACL)_.
* Lai and Young [1990] Karim Lai and Steve J Young. 1990. The estimation of stochastic context-free grammars using the inside-outside algorithm. _Computer speech & language_, 4(1):35-56.
* Le and Zuidema [2014] Phong Le and Willem Zuidema. 2014. The inside-outside recursive neural network model for dependency parsing. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Le and Zuidema [2015] Phong Le and Willem Zuidema. 2015. The forest convolutional network: Compositional distributional semantics with a neural chart and without binarization. In _Empirical Methods in Natural Language Processing (EMNLP)_, pages 1155-1164.
* Li and Roth [2006] Xin Li and Dan Roth. 2006. Learning question classifiers: the role of semantic information. _Natural Language Engineering_, 12(3):229-249.
* Liu et al. [2018] Yang Liu, Matt Gardner, and Mirella Lapata. 2018. Structured alignment networks for matching sentences. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Maillard et al. [2017] Jean Maillard, Stephen Clark, and Dani Yogatama. 2017. Jointly learning sentence embeddings and syntax with unsupervised tree-lstms. _arXiv preprint arXiv:1705.09189_.
* Manning et al. [2014] Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In _Association for Computational Linguistics (ACL): System Demonstrations_.
* Marcus et al. [1993] Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. _Computational linguistics_, 19(2):313-330.
* Peters et al. [2018a] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In _North American Association for Computational Linguistics (NAACL)_.
* Peters et al. [2018b] Matthew E Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018b. Dissecting contextual word embeddings: Architecture and representation. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Ponvert et al. [2011] Elias Ponvert, Jason Baldridge, and Katrin Erk. 2011. Simple unsupervised grammar induction from raw text with cascaded finite state models. In _Association for Computational Linguistics (ACL)_.
* Pradhan et al. [2012] Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In _Joint Conference on EMNLP and CoNLL-Shared Task_.
* Shen et al. [2018]Yikang Shen, Zhouhan Lin, Chin-Wei Huang, and Aaron Courville. 2018. Neural language modeling by jointly learning syntax and lexicon. In _International Conference on Learning Representations (ICLR)_.
* Shen et al. [2019] Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. In _International Conference on Learning Representations (ICLR)_.
* Socher et al. [2011a] Richard Socher, Eric H Huang, Jeffrey Pennin, Christopher D Manning, and Andrew Y Ng. 2011a. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In _Advances in Neural Information Processing Systems (NeurIPS)_.
* Socher et al. [2011b] Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, and Christopher D Manning. 2011b. Semi-supervised recursive autoencoders for predicting sentiment distributions. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Spitkovsky et al. [2013] Valentin I Spitkovsky, Hiyan Alshawi, and Daniel Jurafsky. 2013. Breaking out of local optima with count transforms and model recombination: A study in grammar induction. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Sutton and McCallum [2005] Charles Sutton and Andrew McCallum. 2005. Joint parsing and semantic role labeling. In _Conference on Computational Natural Language Learning (CoNLL)_.
* Tai et al. [2015] Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations 

from tree-structured long short-term memory networks. In _Association for Computational Linguistics (ACL)_.
* Core Team [2018] Pytorch Core Team. 2018. Pytorch: Tensors and dynamic neural networks in python with strong gpu acceleration. [http://pytorch.org/](http://pytorch.org/). Accessed: 2018-09-26.
* Sang and Buchholz [2000] Erik F Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the conll-2000 shared task: Chunking. In _Proceedings of CoNLL-2000 and LLL-2000_, pages 127-132.
* Williams et al. [2018a] Adina Williams, Andrew Drozdov, and Samuel R Bowman. 2018a. Do latent tree learning models identify meaningful structure in sentences? _Transactions of the Association of Computational Linguistics (TACI)_, 6:253-267.
* Williams et al. [2018b] Adina Williams, Nikita Nangia, and Samuel Bowman. 2018b. A broad-coverage challenge corpus for sentence understanding through inference. In _North American Association for Computational Linguistics (NAACL)_.
* Yogatama et al. [2017] Dani Yogatama, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Wang Ling. 2017. Learning to compose words into sentences with reinforcement learning. In _International Conference on Learning Representations (ICLR)_.
* Younger [1967] Daniel H Younger. 1967. Recognition and parsing of context-free languages in time n3. _Information and control_, 10(2):189-208.
* Zaremoodi and Haffari [2018] Poorya Zaremoodi and Gholamreza Haffari. 2018. Incorporating syntactic uncertainty in neural machine translation with a forest-to-sequence model. In _International Conference on Computational Linguistics_.

## Appendix A Appendices

### Training Details

_Training Data_. Sentences of length \(\leq 20\) from the SNLI and MultiNLI training sets.

_Optimization_. We train our model using stochastic gradient descent with the Adam optimization algorithm Kingma and Ba [2014]. Cells were normalized to have magnitude of 1, following Socher et al. [2011a]. For instance, \(\bar{a}(k)\coloneqq\tilde{a}(k)/\left\lVert\bar{a}(k)\right\rVert^{2}\). Gradients are clipped to a maximum L2-norm of 5.

_Hyperparameters_. Chosen using grid search over cell-dimension \(\{400\mathrm{D},800\mathrm{D}\}\) and learning rate \(\{2,4,8,10,20\}\cdot 10^{-4}\).

_Early Stopping_. Using unlabeled parsing F1 against the binarized WSJ validation set.

_Vocabulary_. The model is trained in an open-vocabulary setting using pre-trained context-insensitive character embeddings. The embedder is taken from ELMo Peters et al. [2018a].

_Batching_. Batches were constructed such that they contained sentences of uniform length. Using batch size 128 for 400D and 64 for 800D.

_Sampling_. \(N\) negatives are sampled for each batch. All experiments use \(N=100\).

_Training Steps_. 1M parameter updates, taking 3 days using 4x Nvidia 1080ti.

### Composition and Input Transform

**TreeLSTM**. The \(\operatorname{TreeLSTM}\) Tai et al. [2015] function produces a hidden state vector \(h\) and cell state vector \(c\) given two input vectors \(h_{i}\) and \(h_{j}\).

\[\begin{bmatrix}x\\ f_{i}\\ f_{j}\\ o\\ u\end{bmatrix} =\begin{bmatrix}\sigma\\ \sigma\\ \sigma\\ \tanh\end{bmatrix}\left(U\begin{bmatrix}h_{i}\\ h_{j}\end{bmatrix}+b+\begin{bmatrix}0\\ \omega\\ \omega\\ 0\\ 0\end{bmatrix}\right)\] \[c =c_{i}\odot f_{i}+c_{j}\odot f_{j}+x\odot u\] \[h =o+\tanh(c)\]

The constant \(\omega\) is set to \(1\) for the inside, \(0\) for the outside. \(U\) and \(b\) are learned.

**MLP**. MLP (Multi-Layer Perceptron) is a deep non-linear composition with the following form:

\[h=W_{1}\;(W_{0}\;\langle h_{i},h_{j}\rangle+b)+b_{1}\]

The operator \(\langle h_{i},h_{j}\rangle\) is a concatenation \([h_{i};h_{j}]\). For the \(\text{MLP}^{Kernel}\;\langle h_{i},h_{j}\rangle\) is more involved to support further interaction between the two input vectors \([h_{i};h_{j};h_{i}\odot h_{j};h_{i}-h_{j}]\). The variables \(W_{0},W_{1},b,b_{1}\) are learned and \(c\) is unused.

### Reproducing Parsing Results

In Table 7, we've organized a reference for creating various splits of the WSJ for the purpose of evaluating unsupervised parsing. Some splits use only the test set (section 23), others use all of the training, validation, and test data. Optionally, punctuation is stripped and sentences greater than a specified length are ignored. Predictions can be compared to the full parse trees in the annotated data, or to a binarized version. The PARSEVAL specification calculated bracketing F1 considering all spans, although some previous work diverts from PARSEVAL and ignores spans that are trivially correct (ones over the entire sentence) 



### Runtime Complexity

The runtime complexities for DIORA's methods are shown in Table 8. The parallel column represents the complexity when the values for all constituent pairs are computed simultaneously, assuming that these computations are independent and do not depend on values that have yet to be computed. Linear complexity is theoretically feasible depending on batch size, input length, and number of computational cores. In practice, one might experience super-linear performance.

Although both the inside pass and outside pass have an upper bound of \(n^{3}\) operations, the outside pass will have more operations than the inside pass for sentences of length \(>1\).

As a point of reference, our implementation computes the loss over the entire WSJ corpus in 5 minutes 30 seconds at a rate of 3,500 words per second using a single GPU.

### Parse Trees

Examples of parse trees derived from the compatibility scores are shown in Figures 6 and 7. Some punctuation has been removed for readability.

Figure 6: Examples where DIORA achieves 100% recall compared with the raw (n-ary) ground truth, but less than 100% accuracy on the binarized ground truth. DIORA’s output is shown above the ground truth.

\begin{table}
\begin{tabular}{l|c c c} \hline \hline  & WSJ & WSJ-10 & WSJ-40 \\ \hline Split & Test & All & Test \\ w/ Punctuation & Yes & No & No \\ Max Length & \(\infty\) & 10 & 40 \\ Binarized & Yes & No & No \\ Trivial Spans & Yes & No & No \\ \hline \hline \end{tabular}
\end{table}
Table 7: Settings for unlabeled binary bracketing evaluation for different splits of the WSJ corpus.



* [108] M. C. Bader, A. D. Krizhevsky, and S. I. Jordan. The impact of the deep learning on the performance of a neural network. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 235-246, 2017.
* [109] J. L. Friedman, R. G. Olsson, and E. Rabinovich. Learning to improve the quality of the model. _Journal of Machine Learning Research_**26**:105-118, 2002.
* [110] P. F. Donoho, T. Darrell, and Y. Bengio. Generative adversarial nets for image recognition. In _Advances in Neural Information Processing Systems (NIPS'06)_, volume 1, pages 1-10, 2006.
* [111] H. Chen, X. Li, and Z. Sun. Deep residual learning for image classification. In _IEEE International Conference on Acoustics and Speech (ICASSP)_, pages 1-10, 2006.

[MISSING_PAGE_POST]