SlowGuess/ACL_2020_OCR huggingface dataset:
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arxiv:1508.01991.
Richard A. Hudson. 1984. Word grammar. Blackwell Oxford.
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference for Learning Representations.
Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics, 4:313-327.
Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. 2017. What do recurrent neural network grammars learn about syntax? In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1249-1258, Valencia, Spain. Association for Computational Linguistics.
Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. 2018. LSTMs can learn syntax-sensitive dependencies well, but modeling structure makes them better. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1426-1436, Melbourne, Australia. Association for Computational Linguistics.
Phong Le and Willem Zuidema. 2015. Compositional distributional semantics with long short term memory. In Proceedings of the 4th Joint Conference on Lexical and Computational Semantics, pages 10-19, Denver, Colorado. Association for Computational Linguistics.
Jiwei Li, Thang Luong, Dan Jurafsky, and Eduard Hovy. 2015. When are tree structures necessary for deep learning of representations? In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2304-2314, Lisbon, Portugal. Association for Computational Linguistics.
Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019. Open sesame: Getting inside BERT's linguistic knowledge. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 241-253, Florence, Italy. Association for Computational Linguistics.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521-535.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55-60.
Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192-1202.
Junghyun Min, R. Thomas McCoy, Dipanjan Das, Emily Pitler, and Tal Linzen. 2020. Syntactic data augmentation increases robustness to inference heuristics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Seattle, Washington. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Carl Pollard and Ivan A. Sag. 1994. Head-driven phrase structure grammar. University of Chicago Press.
Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. 2019. Studying the inductive biases of RNNs with synthetic variations of natural languages. In Proceedings of the 2019 Conference of the North American Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 3532-3542.
Mike Schuster and Kuldip K. Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45:2673-2681.
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron C. Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. In International Conference on Learning Representations.
Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1556–1566, Beijing, China. Association for Computational Linguistics.
Zhiyang Teng and Yue Zhang. 2017. Head-lexicalized bidirectional tree LSTMs. Transactions of the Association for Computational Linguistics, 5:163-177.
Lucien Tesniere. 1959. *Éléments de syntaxe structurale*, Klincksieck. Pans.
Dani Yogatama, Yishu Miao, Gabor Melis, Wang Ling, Adhiguna Kuncoro, Chris Dyer, and Phil Blunsom. 2018. Memory architectures in recurrent neural network language models. International Conference on Learning Representations.
Xiaodan Zhu, Parinaz Sobhani, and Hongyu Guo. 2015. Long short-term memory over recursive structures. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of ICML'15, pages 1604-1612.
A Appendix: Tree LSTM Details
The constituency-based model that we use is the  $N$ -ary Tree-LSTM from Tai et al. (2015), with  $N$  fixed at 2 such that the tree is strictly binary; the equations for this model are shown below. Each  $W$  is an input weight matrix, each  $U$  is a hidden state update weight matrix, each  $b$  is a bias term, each  $x$  is an input word embedding, and each  $h$  is a hidden state. These equations are adaptations of the typical LSTM equations that allow the LSTM to be structured according to a constituency parse. The  $x_{j}$  is the input embedding for a particular node in the constituency tree. In a Constituency LSTM, all leaf nodes receive the embedding for the word at that leaf, while all other nodes receive a vector of 0's. Every non-leaf node is thus a composition of the hidden states of its two children. In these equations,  $k = 1$  or 2, which allows "the left hidden state in a binary tree to have either an excitatory or inhibitory effect on the forget gate of the right child" (Tai et al., 2015). Importantly, this model distinguishes between a node's left and right children.
$$
i _ {j} = \sigma \left(W ^ {(i)} x _ {j} + \sum_ {l = 1} ^ {2} U _ {l} ^ {(i)} h _ {j l} + b ^ {(i)}\right) \tag {1}
$$
$$
f _ {j k} = \sigma (W ^ {(f)} x _ {j} + \sum_ {l = 1} ^ {2} U _ {k l} ^ {(f)} h _ {j l} + b ^ {(f)}) \quad (2)
$$
$$
o _ {j} = \sigma \left(W ^ {(o)} x _ {j} + \sum_ {l = 1} ^ {2} U _ {l} ^ {(o)} h _ {j l} + b ^ {(o)}\right) \tag {3}
$$
$$
u _ {j} = \tanh  \left(W ^ {(u)} x _ {j} + \sum_ {l = 1} ^ {2} U _ {l} ^ {(u)} h _ {j l} + b ^ {(u)}\right) \tag {4}
$$
$$
c _ {j} = i _ {j} \odot u _ {j} + \sum_ {l = 1} ^ {2} f _ {j l} \odot c _ {j l} \tag {5}
$$
$$
h _ {j} = o _ {j} \odot \tanh  \left(c _ {j}\right) \tag {6}
$$
The following equations, also from Tai et al. (2015), define a child-sum Tree LSTM, which we structure according to a dependency parse. Here, the input  $x_{j}$  is the embedding of the headword of that node in the DAG that defines a dependency parse. Note that in this model, the hidden representations of the children of a node are summed. Thus, this model cannot distinguish the linear order of its children.
$$
\tilde {h} _ {j} = \sum_ {k \in C (j)} h _ {k} \tag {7}
$$
$$
i _ {j} = \sigma \left(W ^ {(i)} x _ {j} + U ^ {(i)} \tilde {h} _ {j} + b ^ {(i)}\right) \tag {8}
$$
$$
f _ {j k} = \sigma \left(W ^ {(f)} x _ {j} + U ^ {(f)} h _ {k} + b ^ {(f)}\right) \tag {9}
$$
$$
o _ {j} = \sigma \left(W ^ {(o)} x _ {j} + U ^ {(o)} \tilde {h} _ {j} + b ^ {(o)}\right) \tag {10}
$$
$$
u _ {j} = \tanh  \left(W ^ {(u)} x _ {j} + U ^ {(u)} \tilde {h} _ {j} + b ^ {(u)}\right) \tag {11}
$$
$$
c _ {j} = i _ {j} \odot u _ {j} + \sum_ {k \in C (j)} f _ {j k} \odot c _ {k} \tag {12}
$$
$$
h _ {j} = o _ {j} \odot \tanh  \left(c _ {j}\right) \tag {13}
$$
B Appendix: Details of the
Head-Lexicalized Tree LSTM Variant
Our head-lexicalized tree LSTM architecture is structured exactly the same as the Constituency LSTM. Thus, Equations 1 through 6 characterize the parameters and operations performed by the head-lexicalized tree LSTM. The difference between the two architectures lies in the input,  $x_{j}$ . In the Constituency LSTM, a node  $j$  was provided an input vector  $x_{j}$  only if  $j$  was a leaf node. In the head-lexicalized tree LSTM model, we use a dependency parse to generate a tag for each node in the constituency tree, which identifies which word in the corresponding constituent is the most dominant word in the dependency tree. The word embedding corresponding to the most dominant word in constituent  $j$  is then provided as input  $x_{j}$ . Thus, every node in the tree receives an input vector, and the root node is guaranteed to have the headword of the whole sentence provided as input.
More formally, a dependency parse forms a tree,  $T_{D}$ . For each word,  $w$ , in a given sentence, denote its score,  $s(w)$ , as the depth of  $w$  in  $T_{D}$ . A constituency parse forms a tree  $T_{C}$ . For every node  $j$  in  $T_{C}$ , let  $l_{j}$  denote the set of words corresponding to children of  $j$  that are leaves of  $T_{C}$ . The input vector  $x_{j}$  is then just the embedding of  $w = \arg \min_{w\in l_j}s(w)$ . Ties should not exist within a constituent, but if they do (due to parsing errors), then they are broken arbitrarily.
See Figure 2 for an example of a head-lexicalized constituency tree.
C.1 Experiment 1
We use an embedding size and hidden cell size of 100 for every model. Our word embeddings
Figure 2: Head-lexicalized constituency tree for the sentence The bakers near the table bake the cake.
are 100-dimensional pretrained GloVe embeddings from the Wikipedia 2014 + Gigaword 5 distribution (glove.6b.zip) (Pennington et al., 2014), and we do not tune them during training. We also employ the Adam optimizer (Kingma and Ba, 2015) with the PyTorch default learning rate of 0.001. Because this is a binary classification problem, we use binary cross entropy as our loss function. These hyperparameter choices are based on Linzen et al. (2016), but we increase the hidden size from 50 to 100, in order to create slightly more capacity. Though this may seem small, the models achieved high overall accuracy, suggesting that model size was not a bottleneck.
We cap training at 50 epochs, but also employ early stopping. The early stopping procedure is as follows: Train for 10,000 sentences, then evaluate on the validation data. Stop when the average decrease in validation loss over the previous five evaluations is less than 0.0005. For all models, this occurs after about 1 or 1.5 epochs. During training, the parameters that resulted in the best validation loss are saved, and these weights are used during testing. We repeat this procedure for three random initializations of each model. The reported results are averages over these three models.
In order to turn a tree LSTM into a binary classifier, we feed the hidden state of the root into a linear layer that condenses the output into a single value, and squash the result to the range [0, 1] using a sigmoid activation function. If the result of that process is greater than 0.5, then we predict label 1, else we predict label 0. For the bidirectional LSTM, we take the representation of the masked verb from both the left to right and right to left passes and feed both of these into a linear classifier. Then we repeat the process described above, using a sigmoid activation function to constrain the prediction to the range [0, 1], and classifying based on this value.
C.2 Experiment 2
We take the same models from Experiment 1 and fine tune them on the augmentation set. We train for one epoch with the same parameters used in Experiment 1, and then use the resulting weights to evaluate the models.
D Appendix: Data
The original dataset contains approximately 1.3 million sentences. We use the Stanford constituency parser and Stanford dependency parser (Manning et al., 2014) to generate the two types of parse trees for each of these sentences, and then convert these objects into suitable representations for our models. In this process, a small percentage of examples were discarded due to the parser failing to parse them. We deviate from past work by ensuring that both classes (SINGULAR and PLURAL) are of equal size. This results in more data from the majority class (singular verb class) being thrown away. After these exclusions, we have approximately 903,000 sentences remaining. We provide our models  $9\%$  of this (81,300 sentences) to train on,  $0.1\%$  (904 sentences) to validate, and then generate our test sets from the remainder of the data. All sentences were stripped of quotation marks, apostrophes, parentheses and hyphens in order to minimize parsing failures.
The sizes of our test sets are as follows: No Attractors (50,000 sentences), Any Attractors (52,815 sentences), One Attractor (41,902 sentences), Two Attractors (8,473 sentences), Three Attractors (1,884 sentences), and Four Attractors (556 sentences). Note also that the Any Attractors dataset is the union of the One, Two, Three, and Four Attractors datasets.
E Appendix: Full Results
Table 2 contains the full results after training all models on natural language. Table 3 contains the full results after augmentation.
F Appendix: Probabilistic Context Free Grammars
Figure 3 contains the probabilistic context free grammar used to generate the constructed corpora.
Figure 3: Probabilistic Context-free grammar used for creating constructed datasets. For the constructed language test set, the probabilities for the three potential expansions of  $\mathrm{NP}_s$  and  $\mathrm{NP}_p$  are .1, .1, .8, respectively. For the augmentation set, these probabilities are .69, .04, .27. For all other nonterminals, all possible expansions have uniform probability in both test and augmentation sets. PPs are present in approximately one third of sentences in both the test and augmentation sets.
Attractors	BiLSTM	Dependency	Constituency	Head
No	96.4%	95.5%	97.3%	97.2%
Any	70.8%	91.4%	90.2%	87.0%
1	74.6%	91.9%	91.3%	88.7%
2	59.7%	89.7%	87.1%	82.0%
3	48.4%	87.6%	83.7%	77.0%
4	41.0%	87.0%	80.8%	73.1%
Constructed	96.0%	73.8%	97.6%	97.3%
Table 2: Natural language results for all datasets. Best performances are bolded. All numbers are averaged over three models.
Attractors	BiLSTM	Dependency	Constituency	Head
No	88.8%	94.9%	98.1%	97.2%
Any	77.1%	90.1%	91.9%	91.5%
1	76.6%	90.8%	93.2%	92.8%
2	78.5%	87.9%	88.1%	87.6%
3	80.1%	85.2%	83.5%	83.3%
4	81.1%	85.9%	78.4%	79.4%
Constructed	95.3%	75.8%	99.7%	99.8%
Table 3: Results for all datasets after augmentation. All numbers are averaged over three models.