Venue: ACL 2020

# Representations of Syntax [MASK] Useful:

Effects of Constituency and Dependency Structure in Recursive LSTMs

 Michael A. Lepori\({}^{1}\) Tal Linzen\({}^{1,2}\) R. Thomas McCoy\({}^{2}\)

\({}^{1}\)Department of Computer Science  \({}^{2}\)Department of Cognitive Science

Johns Hopkins University

{mleporil, tal.linzen, tom.mcocy}@jhu.edu

###### Abstract

Sequence-based neural networks show significant sensitivity to syntactic structure, but they still perform less well on syntactic tasks than tree-based networks. Such tree-based networks can be provided with a constituency parse, a dependency parse, or both. We evaluate which of these two representational schemes more effectively introduces biases for syntactic structure that increase performance on the subject-verb agreement prediction task. We find that a constituency-based network generalizes more robustly than a dependency-based one, and that combining the two types of structure does not yield further improvement. Finally, we show that the syntactic robustness of sequential models can be substantially improved by fine-tuning on a small amount of constructed data, suggesting that data augmentation is a viable alternative to explicit constituency structure for imparting the syntactic biases that sequential models are lacking.

## 1 Introduction

Natural language syntax is structured hierarchically, rather than sequentially Chomsky (1957); Everaert et al. (2015). One phenomenon that illustrates this fact is **English subject-verb agreement**, the requirement that verbs and their subjects must match in number. The hierarchical structure of a sentence determines which noun phrase each verb must agree with; sequential heuristics such as agreeing with the most recent noun may succeed on simple sentences such as (1a) but fail in more complex cases such as (1b):

* [noitemsep,topsep=0pt]
* The **boys kick** the ball.
* The **boys** by the red truck **kick** the ball.

We investigate whether a neural network must process input according to the structure of a syntactic parse in order for it to learn the appropriate rules governing these dependencies, or whether there is sufficient signal in natural language corpora for low-bias networks (such as sequential LSTMs) to learn these structures. We compare sequential LSTMs, which process sentences from left to right, with tree-based LSTMs that process sentences in accordance with an externally-provided, ground-truth syntactic structure.

We consider two types of syntactic structure: **constitucency structure**Chomsky (1993); Pollard and Sag (1994) and **dependency structure**Tesminere (1959); Hudson (1984). We investigate models provided with either structure, both structures, or neither structure (see Table 1), and assess how robustly these models learn subject-verb agreement when trained on natural language.1

Footnote 1: Code, data, and models are at [https://github.com/mleporil/Representations_Of_Syntax](https://github.com/mleporil/Representations_Of_Syntax)

Even with the syntactic biases present in tree-based LSTMs, it is possible that natural language might not impart a strong enough signal to teach a network how to robustly track subject-verb dependencies. How might the performance of these tree-based LSTMs change if they were fine-tuned on a small dataset designed to impart a stronger syntactic signal? Furthermore, would we still need these tree structures, or could a sequential LSTM now learn to track syntactic dependencies?

We find that building in either type of syntactic structure improves performance over the BiLSTM

\begin{table}
\begin{tabular}{l l l} \hline \hline  & No Constituency & Constituency \\ \hline No & BiLSTM & Constituency \\ Heads & & LSTM \\ Heads & Dependency & Head-Lexicalized \\  & LSTM & LSTM \\ \hline \hline \end{tabular}
\end{table}
Table 1: Linguistic properties of our four models.



baseline, thus showing that these structures are learned imperfectly (at best) by low-bias models from natural language data. Of the two types of structure, constituency structure turns out to be more useful. The dependency-only model performs well on natural language test sets, but fails to generalize to an artificially-constructed challenge set. After fine-tuning on a small dataset that is designed to impart a strong syntactic signal, the BiLSTM generalizes more robustly, but still falls short of the tree-based LSTMs.

We conclude that for a network to robustly show sensitivity to syntactic structure, stronger biases for syntactic structure need to be introduced than are present in a low-bias learner such as a BiLSTM, and that, at least for the subject-verb agreement task, constituency structure is more important than dependency structure. Both tree-based model structure and data augmentation appear to be viable approaches for imparting these biases.

## 2 Related Work

Prior work has shown that neural networks without explicit mechanisms for representing syntactic structure can show considerable sensitivity to syntactic dependencies Goldberg (2019); Gulordava et al. (2018); Linzen et al. (2016), and that certain aspects of the structure of the sentence can be reconstructed from their internal representations Lin et al. (2019); Giulianelli et al. (2018); Hewitt and Manning (2019). Marvin and Linzen (2018) showed that sequential models still have substantial room for improvement in capturing syntax, and other work has shown that models with a greater degree of syntactic structure outperform sequential models on syntax-sensitive tasks Yogatama et al. (2018); Kuncoro et al. (2018, 2017), including some of the tree-based models used here Bowman et al. (2015); Li et al. (2015). One contribution of the present work is to tease apart the two major types of syntactic structure to see which one imparts more effective syntactic biases.

## 3 Models

### BiLSTM

As our baseline model, we used a simple extension to the LSTM architecture Hochreiter and Schmidhuber (1997), the **bidirectional LSTM**BiLSTM; Schuster and Paliwal (1997). This model runs one LSTM from left to right over a sequence, and another from right to left, without appealing to tree structure. Bidirectional LSTMs outperform unidirectional LSTMs on a variety of tasks Huang et al. (2015); Chiu and Nichols (2016), including syntax-sensitive tasks Kiperwasser and Goldberg (2016). Ravfogel et al. (2019) also employs BiLSTMs for a similar agreement task.

### Tree LSTMs

To study the effects of explicitly building tree structure into the model architecture, we used the **Constitency LSTM** and the **Dependency LSTM**Tai et al. (2015), which are types of recursive neural networks Goller and Kuchler (1996). The Constituency LSTM operates in accordance with a binary constituency parse, composing together vectors representing a left child and a right child into a vector representing their parent. Models similar to the Constituency LSTM have been proposed by Le and Zuidema (2015) and Zhu et al. (2015).

In a Dependency LSTM, the representations of a head's children are summed, and then composed with the representation of the head itself to yield a representation of the phrase that has that head. See Appendix A for more details on both models.

### Head-Lexicalized Tree LSTMs

To create a model where composition is simultaneously guided by both a dependency parse and a constituency parse, we modified the constituency model described in Section 3.2, turning it into a **head-lexicalized tree LSTM**. In a standard Constituency LSTM, the input for all non-leaf nodes is a vector of all 0's. To add head lexicalization, we instead feed in the word embedding of the correct headword of that constituent as the input, where the choice of headword is determined using the Stanford Dependency Parser Manning et al. (2014). See Appendix B for more details, as well as an example of a head-lexicalized constituency tree. This model is similar to the head-lexicalized tree LSTM of Teng and Zhang (2017). However, their model learns how to select the heads of constituents in an unsupervised manner; these heads may not correspond to the syntactic notion of heads. Because we seek to understand the effect of using the heads derived from the dependency parse, we provide our models with explicit head information.

## 4 Task

We adapted a syntax-sensitive task that previous work has used to assess the syntactic capabilities 

of LSTMs--the number prediction task (Linzen et al., 2016). The most standard version of this task is based on a left-to-right language modeling objective; however, tree-based models are not compatible with left-to-right language modeling. Therefore, we made two modifications to this objective, both of which have precedents in the literature: First, we gave the model an entire present-tense sentence with main verb masked out, following Goldberg (2019). Second, the model's target output was the number of the masked verb: singular or plural; we follow Linzen et al. (2016) and Ravfogel et al. (2019) in framing number prediction as a classification task. To solve the task, the model must identify the subject whose head is the main verb (in the dependency formalism), and use that information to determine the syntactic number of the verb; e.g., for (2), the answer is singular.

The girl *MASK* the ball.

Linzen et al. (2016) pointed out that there are several incorrect heuristics which models might adopt for this task because these heuristics still produce decent classification accuracy. One salient example is picking the syntactic number of the most recent noun to the left of the verb. We hypothesize that tree-based models will be less susceptible to these non-robust heuristics than sequential models.

## 5 Experiment 1: Natural Language

Data:We train our models on a subset of the dataset from Linzen et al. (2016) that is chosen to have a uniform label distribution (50% singular and 50% plural). We made this choice because our task format differs from that used in some past work (see Section4), so performance on the task as we have framed it cannot be directly compared to prior work. In the absence of baselines from the literature, we use chance performance of 50% as a baseline; to ensure that this baseline is reasonable, we balance the label distribution during training to discourage models from becoming biased toward one label.

We use two types of test sets: those that contain adversarial attractors, and those that do not. An **adversarial attractor** is a noun that is between the subject and the main verb of a sentence and that has the opposite syntactic number from the subject noun. Adversarial attractors have been found to produce agreement errors in humans (Bock and Miller, 1991) and neural models (Goldberg, 2019; Gulordava et al., 2018; Linzen et al., 2016). We use code from Goldberg (2019)2 to extract adversarial datasets containing varying numbers of attractors, from 0 to 4 attractors. Sentence (3) provides an example of a sentence with 4 attractors. Footnote 2: [https://github.com/yoavg/bert-syntax](https://github.com/yoavg/bert-syntax)

Footnote 2: [https://github.com/yoavg/bert-syntax](https://github.com/yoavg/bert-syntax)

Algorithmic **problems** such as _[type]__[checking]_ and _[type]__[inference]_are more difficult for equircursive types as well.

See AppendixD for details on our corpus and on preprocessing, and AppendixC.1 for training.

Natural language evaluation:All of the tree-based models outperformed the BiLSTM in the presence of attractors (Figure1a). Compared to prior work with the number prediction task, our BiLSTM performed very poorly on the 4 Attractors dataset. However, our results cannot be directly compared to previous work because of the modifications we have made to the task, data, and training procedure in order to accommodate tree

Figure 1: Results on binary classification of masked verbs as singular or plural. All results are averages across 3 runs. Chance performance is 50%.

 

-based models. In light of these modifications, there are several reasons why the BiLSTM's low accuracy is unsurprising. First, we used a balanced label distribution during training. In the standard dataset from Linzen et al. (2016), the class labels are not balanced, so models evaluated on that dataset might outperform our BiLSTM by exploiting the biased label distribution--a heuristic that our balanced training set discourages. Another potential cause for the BiLSTM's poor performance is that, in order to balance the label frequencies, we used a smaller training set than was used in past work (81,000 sentences instead of 121,000 sentences). Finally, it is possible that allowing models to see the entire sentence may allow them to acquire non-robust heuristics related to the words following the main verb. For example, a model might learn spurious correlation between the syntactic number of subjects and their direct objects. See Appendix E, Table 2 for results on all test sets.

**Constructed sentence evaluation:** With naturally occurring sentences, it is possible that models perform well not because they have mastered syntax, but rather because of statistical regularities in the data. For example, given _The players *MASK* the ball_, the model may be able to exploit the fact that animate nouns tend to be subjects while inanimate nouns do not. As pointed out by Gulordava et al. (2018), this would allow the model to correctly predict syntactic number, but for the wrong reasons. To test whether our models were leveraging this statistical heuristic, we constructed a 400-sentence test set where this heuristic cannot succeed. We did so using a probabilistic context-free grammar (PCFG) under which all words of a given part of speech are equally likely in all positions; each sentence from this grammar is of the form Subject-Verb-Object, and all noun phrases can optionally be modified by adjectives and/or prepositional phrases (see Appendix F), as in (4):

1. The **fern** near the sad teachers **hates** the singer.

The Dependency LSTM is especially likely to fall prey to word cooccurrence heuristics, as it lacks the ability for a parent to account for the sequential position of its children. This can be an issue when determining whether a verb is supposed to be singular or plural, because the model has no robust way to distinguish a verb's subject from its direct object. The dependency model did indeed perform at chance (See the bar graph in Figure 0(a)).3 This suggests that the dependency model's high accuracy is partially due to lexical heuristics rather than syntactic processing. In contrast, the other models performed well, suggesting that they are less susceptible to relying on word cooccurrence.

Footnote 3: Most sentences in the test set have only two nouns. 50% of the time, they will agree in number, and the syntactic number is unambiguous. Random guessing on the other 50% of cases would yield about 75% accuracy.

## 6 Experiment 2: Fine-tuning

In Experiment 1, tree-based models dramatically outperformed the BiLSTM in the presence of attractors. This difference may have arisen because most natural language sentences are simple, and thus they do not generate enough signal to illustrate the importance of tree structure to a low-bias learner, such as a BiLSTM. Recent work has shown the effectiveness of syntactically-motivated fine-tuning at increasing the robustness of neural models Min et al. (2020). Would our models generalize more robustly if we added a few training examples that do not lend themselves to non-syntactic heuristics?

To provide the model with a stronger signal about the importance of syntactic structure, we fine-tuned our models on a dataset designed to impart this signal. We used a variant of the PCFG (see Appendix F) from Section 5 to generate a 500-sentence **augmentation set**. This augmentation set cannot be solved using word cooccurrence statistics, and contains some sentences with attractors. The models were then fine-tuned on the augmentation set for just _one epoch_ over the 500 examples. See Appendix C.2 for training details.

**Results:** The head-lexicalized model and the BiLSTM benefited most from fine-tuning, with the head-lexicalized model now matching the performance of the Constituency LSTM, and the BiLSTM showing dramatic improvement on sentences with multiple attractors (Figure 0(b); see Appendix E, Table 3 for detailed results). While the BiLSTM's accuracy increased on sentences with attractors, it decreased on the No Attractors test set. We suspect that this is because augmentation discouraged the model from using heuristics: while this makes performance more robust overall, it may hurt accuracy on simple examples where the heuristics give the correct answer Min et al. (2020). As expected from its architectural limitations, the Dependency LSTM did not noticeably benefit from fine-tuning 

because it cannot extract the relevant information from the augmentation set. There was no clear effect of augmentation on the Constituency LSTM.4

Footnote 4: Note that the constructed test set used here is controlled to have no overlap with the augmentation set. Thus, it is not exactly the same as the set used in Section 5, but both corpora are generated from the same CFG.

## 7 Discussion

Overall, we found that neural models trained on natural language achieve much more robust performance on syntactic tasks when syntax is explicitly built into the model. This suggests that the information we provided to our tree-based models is unlikely to be learned from natural language by models with only general inductive biases.

In Experiment 1, the network provided with a dependency parse did the best on most of the natural language test sets. This is unsurprising, as the task is largely about a particular dependency (i.e., the dependency between a verb and its subject). At the same time, as demonstrated by the constructed sentence test, the syntactic capabilities of the Dependency LSTM are inherently limited. Thus, it must default to non-robust heuristics in cases where the unlabeled dependency information is ambiguous. In future work, these syntactic limitations may be overcome by giving the model typed dependencies (which would distinguish between a subject-verb dependency and a verb-object dependency).

One might expect the head-lexicalized model to perform the best, since it can leverage both syntactic formalisms. However, it performs no better than the constituency model when trained on natural language, suggesting that there is little benefit to incorporating dependency structure into a Constituency LSTM. In some cases, the head-lexicalized model without fine-tuning even performs worse than the Constituency LSTM. When fine-tuned on more challenging constructed examples, the head-lexicalized model performed similarly to the Constituency LSTM, suggesting that there is not enough signal in the natural language training set to teach this model what to do with the heads it has been given.

Our results point to two possible approaches for improving how models handle syntax. The first approach is to use models that have explicit mechanisms for representing syntactic structure. In particular, our results suggest that the most important aspect of syntactic structure to include is constituency structure, as constituency models appear to implicitly learn dependency structure as well. Though the models we used require parse trees to be provided, it is possible that models can learn to induce tree structure in an unsupervised or weakly-supervised manner Bowman et al. (2016); Choi et al. (2018); Shen et al. (2019). Another effective approach for improving the syntactic robustness of neural models is data augmentation, as demonstrated in Experiment 2. With this approach, it is possible to bring the syntactic performance of less-structured models closer to that of models with explicit tree structure, even with an augmentation set generated simply and easily using a PCFG.

Future work should further explore both of these approaches. Our conclusions about the importance of explicit mechanisms for representing syntactic structure can be strengthened by developing different formulations of the tree LSTMs. It seems particularly promising to explore alternative formulations of the Dependency LSTM (as mentioned above) and the effect of learning embeddings of non-terminal symbols for the Constituency LSTM. Finally, future work should investigate whether data augmentation can fully bridge the gap between low-bias learners and structured tree LSTMs, and whether our conclusions apply to other syntactic phenomena besides agreement.

This research was supported by a Google Faculty Award to Tal Linzen, NSF Graduate Research Fellowship No. 1746891, and NSF Grant No. BCS-1920924. Our experiments were conducted using the Maryland Advanced Research Computing Center (MARCC).

## References

* Bock and Miller (1991) Kathryn Bock and Carol Ann Miller. 1991. Broken agreement. _Cognitive Psychology_, 23:45-93.
* Bowman et al. (2016) Samuel R. Bowman, Jon Gauthier, Abhinav Rasutogi, Raghav Gupta, Christopher D. Manning, and Christopher Potts. 2016. A fast unified model for parsing and sentence understanding. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1466-1477, Berlin, Germany. Association for Computational Linguistics.
* Bowman et al. (2015) Samuel R. Bowman, Christopher D. Manning, and Christopher Potts. 2015. Tree-structured composition in neural networks without tree-structured ar 

chitectures. In _Proceedings of the 2015 International Conference on Cognitive Computation: Integrating Neural and Symbolic Approaches_, volume 1583 of _COCO'15_, pages 37-42, Aachen, Germany, Germany.
* Chiu and Nichols (2016) Jason P. C. Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. _Transactions of the Association for Computational Linguistics_, 4:357-370.
* Choi et al. (2018) Jihun Choi, Kang Min Yoo, and Sang-goo Lee. 2018. Learning to compose task-specific tree structures. In _32nd AAAI Conference on Artificial Intelligence_.
* Chomsky (1957) Noam Chomsky. 1957. _Syntactic structures_. Mouton and Co., The Hague.
* Chomsky (1993) Noam Chomsky. 1993. A minimalist program for linguistic theory. _The View from Building 20: Essays in Linguistics in Honor of Sylvain Bromberger_, pages 1-53.
* Everaert et al. (2015) Martin B. H. Everaert, Marius A. C. Huybregts, Noam Chomsky, Robert C. Berwick, and Johan J. Bolhuis. 2015. Structures, not strings: Linguistics as part of the cognitive sciences. _Trends in Cognitive Sciences_, 19(12):729-743.
* Giulianelli et al. (2018) Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 240-248, Brussels, Belgium. Association for Computational Linguistics.
* Goldberg (2019) Yoav Goldberg. 2019. Assessing BERT's syntactic abilities. _arXiv preprint arXiv:1901.05287_.
* Goller and Kuchler (1996) Cristoph Goller and Andreas Kuchler. 1996. Learning task-dependent distributed representations by backpropagation through structure. In _1996 IEEE International Conference on Neural Networks_, volume 1, pages 347-352 vol.1. IEEE.
* Gulordava et al. (2018) Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1195-1205, New Orleans, Louisiana. Association for Computational Linguistics.
* Hewitt and Manning (2019) John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4129-4138, Minneapolis, Minnesota. Association for Computational Linguistics.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Computation_, 9(8):1735-1780.
* Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. _arXiv preprint arxiv:1508.01991_.
* Hudson (1984) Richard A. Hudson. 1984. _Word grammar_. Blackwell Oxford.
* Kingma and Ba (2015) Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _International Conference for Learning Representations_.
* Kiperwasser and Goldberg (2016) Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. _Transactions of the Association for Computational Linguistics_, 4:313-327.
* Konucoro et al. (2017) Adhiguna Konucoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. 2017. What do recurrent neural network grammars learn about syntax? In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, pages 1249-1258, Valencia, Spain. Association for Computational Linguistics.
* Kulcoro et al. (2018) Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. 2018. LSTMs can learn syntax-sensitive dependencies well, but modeling structure makes them better. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1426-1436, Melbourne, Australia. Association for Computational Linguistics.
* Le and Zuidema (2015) Phong Le and Willem Zuidema. 2015. Compositional distributional semantics with long short term memory. In _Proceedings of the 4th Joint Conference on Lexical and Computational Semantics_, pages 10-19, Denver, Colorado. Association for Computational Linguistics.
* Li et al. (2015) Jiwei Li, Thang Luong, Dan Jurafsky, and Eduard Hovy. 2015. When are tree structures necessary for deep learning of representations? In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 2304-2314, Lisbon, Portugal. Association for Computational Linguistics.
* Lin et al. (2019) Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019. Open sesame: Getting inside BERT's linguistic knowledge. In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 241-253, Florence, Italy. Association for Computational Linguistics.
* Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. _Transactions of the Association for Computational Linguistics_, 4:521-535 

Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In _Association for Computational Linguistics (ACL) System Demonstrations_, pages 55-60.
* Marvin and Linzen (2018) Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1192-1202.
* Min et al. (2020) Junghyun Min, R. Thomas McCoy, Dipanjan Das, Emily Pitler, and Tal Linzen. 2020. Syntactic data augmentation increases robustness to inference heuristics. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, Seattle, Washington. Association for Computational Linguistics.
* Pennington et al. (2014) Jeffrey Penington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_, pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
* Pollard and Sag (1994) Carl Pollard and Ivan A. Sag. 1994. _Head-driven phrase structure grammar_. University of Chicago Press.
* Ravfogel et al. (2019) Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. 2019. Studying the inductive biases of RNNs with synthetic variations of natural languages. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 3532-3542.
* Schuster and Paliwal (1997) Mike Schuster and Kuldip K. Paliwal. 1997. Bidirectional recurrent neural networks. _IEEE Transactions on Signal Processing_, 45:2673-2681.
* Shen et al. (2019) Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron C. Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. In _International Conference on Learning Representations_.
* Tai et al. (2015) Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1556-1566, Beijing, China. Association for Computational Linguistics.
* Teng and Zhang (2017) Zhiyang Teng and Yue Zhang. 2017. Head-lexicalized bidirectional tree LSTMs. _Transactions of the Association for Computational Linguistics_, 5:163-177.
* Tesniere (1959) Lucien Tesniere. 1959. _Elements de syntaxe structurale, Klincksieck_. Pang.
* Yogatama et al. (2018) Dani Yogatama, Yishu Miao, Gabor Melis, Wang Ling, Adhiguna Kuncoro, Chris Dyer, and Phil Blunsom. 2018. Memory architectures in recurrent neural network language models. _International Conference on Learning Representations_.
* Zhu et al. (2015) Xiaodan Zhu, Parinaz Sobhani, and Hongyu Guo. 2015. Long short-term memory over recursive structures. In _Proceedings of the 32nd International Conference on Machine Learning_, volume 37 of _ICML'15_, pages 1604-1612.

 

Appendix: Tree LSTM Details

The constituency-based model that we use is the \(N\)-ary Tree-LSTM from Tai et al. (2015), with \(N\) fixed at 2 such that the tree is strictly binary; the equations for this model are shown below. Each \(W\) is an input weight matrix, each \(U\) is a hidden state update weight matrix, each \(b\) is a bias term, each \(x\) is an input word embedding, and each \(h\) is a hidden state. These equations are adaptations of the typical LSTM equations that allow the LSTM to be structured according to a constituency parse. The \(x_{j}\) is the input embedding for a particular node in the constituency tree. In a Constituency LSTM, all leaf nodes receive the embedding for the word at that leaf, while all other nodes receive a vector of 0's. Every non-leaf node is thus a composition of the hidden states of its two children. In these equations, \(k\) = 1 or 2, which allows "the left hidden state in a binary tree to have either an excitatory or inhibitory effect on the forget gate of the right child" (Tai et al., 2015). Importantly, this model distinguishes between a node's left and right children.

\[i_{j} =\sigma(W^{(i)}x_{j}+\sum_{l=1}^{2}U_{l}^{(i)}h_{jl}+b^{(i)}) \tag{1}\] \[f_{jk} =\sigma(W^{(f)}x_{j}+\sum_{l=1}^{2}U_{kl}^{(f)}h_{jl}+b^{(f)})\] (2) \[o_{j} =\sigma(W^{(o)}x_{j}+\sum_{l=1}^{2}U_{l}^{(o)}h_{jl}+b^{(o)})\] (3) \[u_{j} =tanh(W^{(u)}x_{j}+\sum_{l=1}^{2}U_{l}^{(u)}h_{jl}+b^{(u)})\] (4) \[c_{j} =i_{j}\odot u_{j}+\sum_{l=1}^{2}f_{jl}\odot c_{jl}\] (5) \[h_{j} =o_{j}\odot tanh(c_{j}) \tag{6}\]

The following equations, also from Tai et al. (2015), define a child-sum Tree LSTM, which we structure according to a dependency parse. Here, the input \(x_{j}\) is the embedding of the headword of that node in the DAG that defines a dependency parse. Note that in this model, the hidden representations of the children of a node are summed. Thus, this model cannot distinguish the linear order of its children.

\[\tilde{h}_{j} =\sum_{k\in C(j)}h_{k} \tag{7}\] \[i_{j} =\sigma(W^{(i)}x_{j}+U^{(i)}\tilde{h}_{j}+b^{(i)})\] (8) \[f_{jk} =\sigma(W^{(f)}x_{j}+U^{(f)}h_{k}+b^{(f)})\] (9) \[o_{j} =\sigma(W^{(o)}x_{j}+U^{(o)}\tilde{h}_{j}+b^{(o)})\] (10) \[u_{j} =tanh(W^{(u)}x_{j}+U^{(u)}\tilde{h}_{j}+b^{(u)})\] (11) \[c_{j} =i_{j}\odot u_{j}+\sum_{k\in C(j)}f_{jk}\odot c_{k}\] (12) \[h_{j} =o_{j}\odot tanh(c_{j}) \tag{13}\]

## Appendix B Appendix: Details of the Head-Lexicalized Tree LSTM Variant

Our head-lexicalized tree LSTM architecture is structured exactly the same as the Constituency LSTM. Thus, Equations 1 through 6 characterize the parameters and operations performed by the head-lexicalized tree LSTM. The difference between the two architectures lies in the input, \(x_{j}\). In the Constituency LSTM, a node \(j\) was provided an input vector \(x_{j}\) only if \(j\) was a leaf node. In the head-lexicalized tree LSTM model, we use a dependency parse to generate a tag for each node in the constituency tree, which identifies which word in the corresponding constituent is the most dominant word in the dependency tree. The word embedding corresponding to the most dominant word in constituent \(j\) is then provided as input \(x_{j}\). Thus, every node in the tree receives an input vector, and the root node is guaranteed to have the headword of the whole sentence provided as input.

More formally, a dependency parse forms a tree, \(T_{D}\). For each word, \(w\), in a given sentence, denote its score, \(s(w)\), as the depth of \(w\) in \(T_{D}\). A constituency parse forms a tree \(T_{C}\). For every node \(j\) in \(T_{C}\), let \(l_{j}\) denote the set of words corresponding to children of \(j\) that are leaves of \(T_{C}\). The input vector \(x_{j}\) is then just the embedding of \(w=\operatorname*{arg\,min}_{w\in l_{j}}s(w)\). Ties should not exist within a constituent, but if they do (due to parsing errors), then they are broken arbitrarily.

See Figure 2 for an example of a head-lexicalized constituency tree.

## Appendix C Appendix: Training Details

### Experiment 1

We use an embedding size and hidden cell size of 100 for every model. Our word embeddings 

are 100-dimensional pretrained GloVe embeddings from the Wikipedia 2014 + Gigaword 5 distribution (glove.6b.zip) Pennington et al. (2014), and we do not tune them during training. We also employ the Adam optimizer Kingma and Ba (2015) with the PyTorch default learning rate of 0.001. Because this is a binary classification problem, we use binary cross entropy as our loss function. These hyperparameter choices are based on Linzen et al. (2016), but we increase the hidden size from 50 to 100, in order to create slightly more capacity. Though this may seem small, the models achieved high overall accuracy, suggesting that model size was not a bottleneck.

We cap training at 50 epochs, but also employ early stopping. The early stopping procedure is as follows: Train for 10,000 sentences, then evaluate on the validation data. Stop when the average decrease in validation loss over the previous five evaluations is less than 0.0005. For all models, this occurs after about 1 or 1.5 epochs. During training, the parameters that resulted in the best validation loss are saved, and these weights are used during testing. We repeat this procedure for three random initializations of each model. The reported results are averages over these three models.

In order to turn a tree LSTM into a binary classifier, we feed the hidden state of the root into a linear layer that condenses the output into a single value, and squash the result to the range [0, 1] using a sigmoid activation function. If the result of that process is greater than 0.5, then we predict label 1, else we predict label 0. For the bidirectional LSTM, we take the representation of the masked verb from both the left to right and right to left passes and feed both of these into a linear classifier. Then we repeat the process described above, using a sigmoid activation function to constrain the prediction to the range [0, 1], and classifying based on this value.

### Experiment 2

We take the same models from Experiment 1 and fine tune them on the augmentation set. We train for one epoch with the same parameters used in Experiment 1, and then use the resulting weights to evaluate the models.

## Appendix D Appendix: Data

The original dataset contains approximately 1.3 million sentences. We use the Stanford constituency parser and Stanford dependency parser Manning et al. (2014) to generate the two types of parse trees for each of these sentences, and then convert these objects into suitable representations for our models. In this process, a small percentage of examples were discarded due to the parser failing to parse them. We deviate from past work by ensuring that both classes (SINGULAR and PLURAL) are of equal size. This results in more data from the majority class (singular verb class) being thrown away. After these exclusions, we have approximately 903,000 sentences remaining. We provide our models 9% of this (81,300 sentences) to train on, 0.1% (904 sentences) to validate, and then generate our test sets from the remainder of the data. All sentences were stripped of quotation marks, apostrophes, parentheses and hyphens in order to minimize parsing failures.

The sizes of our test sets are as follows: No Attractors (50,000 sentences), Any Attractors (52,815 sentences), One Attractor (41,902 sentences), Two Attractors (8,473 sentences), Three Attractors (1,884 sentences), and Four Attractors (556 sentences). Note also that the Any Attractors dataset is the union of the One, Two, Three, and Four Attractors datasets.

## Appendix E Appendix: Full Results

Table 2 contains the full results after training all models on natural language. Table 3 contains the full results after augmentation.

## Appendix F Appendix: Probabilistic Context Free Grammars

Figure 3 contains the probabilistic context free grammar used to generate the constructed corpora.

Figure 2: Head-lexicalized constituency tree for the sentence _The bakers near the table bake the cake_.

 

* [150] M. C. Bader, J. R. Friedman, and A. S. G. Hinton, _A Practical Guide to Machine Learning: An Introduction_. Springer Science & Business Media, 2007.
* [151] D. P. Kingma and J. Ba, "Adam: a method for stochastic optimization," in _Proceedings of the Thirtieth International Conference on Neural Information Processing Systems_, pp. 349-368, 2014.
* [152] K. L. Lehner, T. W. Teh, and H. E. Stanley, "Learning with deep learning," _arXiv preprint arXiv:1412.6058_, 2014.
* [153] K. L. Lehner, T. W. Teh, and H. E. Stanley, "Deep learning with deep neural networks," _Neural Networks_**2**(1), 1-12, 2015.
* [154] K. L. Lehner, T. W. Teh, and H. E. Stanley, "Deep learning with deep neural networks," _Advances in Neural Information Processing Systems_, pp. 1-10, 2015.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{c c c c c} \hline \hline Attractors & BiLSTM & Dependency & Constituency & Head \\ \hline No & 96.4\% & 95.5\% & **97.3\%** & 97.2\% \\ Any & 70.8\% & **91.4\%** & 90.2\% & 87.0\% \\
1 & 74.6\% & **91.9\%** & 91.3\% & 88.7\% \\
2 & 59.7\% & **89.7\%** & 87.1\% & 82.0\% \\
3 & 48.4\% & **87.6\%** & 83.7\% & 77.0\% \\
4 & 41.0\% & **87.0\%** & 80.8\% & 73.1\% \\ Constructed & 96.0\% & 73.8\% & **97.6\%** & 97.3\% \\ \hline \hline  & & & & \\ \end{tabular}
\end{table}
Table 2: Natural language results for all datasets. Best performances are bolded. All numbers are averaged over three models.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline Attractors & BiLSTM & Dependency & Constituency & Head \\ \hline No & 88.8\% & 94.9\% & **98.1\%** & 97.2\% \\ Any & 77.1\% & 90.1\% & **91.9\%** & 91.5\% \\
1 & 76.6\% & 90.8\% & **93.2\%** & 92.8\% \\
2 & 78.5\% & 87.9\% & **88.1\%** & 87.6\% \\
3 & 80.1\% & **85.2\%** & 83.5\% & 83.3\% \\
4 & 81.1\% & **85.9\%** & 78.4\% & 79.4\% \\ Constructed & 95.3\% & 75.8\% & 99.7\% & **99.8\%** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Results for all datasets after augmentation. All numbers are averaged over three models.