SlowGuess/ACL_2020_OCR huggingface dataset:
LSTM-LM	Additional margin loss (δ = 10)	Additional loss (α = 1000, β = 1)	Distilled
M&L18	Ours	Sentence-level	Token-level	Binary-pred.	Unlike.	K19
AGREEMENT:
Simple	94.0	98.1 (±1.3)	100.0 (±0.0)	100.0 (±0.0)	99.1 (±1.2)	99.7 (±0.6)	100.0 (±0.0)
In a sent. complement	99.0	96.1 (±2.0)	95.8 (±0.7)	99.3 (±0.4)	96.9 (±2.4)	92.7 (±3.1)	98.0 (±2.0)
Short VP coordination	90.0	93.6 (±3.0)	100.0 (±0.0)	99.4 (±1.1)	93.8 (±3.3)	95.6 (±3.0)	99.0 (±2.0)
Long VP coordination	61.0	82.2 (±3.4)	94.5 (±1.0)	99.0 (±0.8)	83.9 (±3.2)	90.0 (±2.4)	80.0 (±2.0)
Across a PP	57.0	92.6 (±1.4)	98.8 (±0.4)	98.6 (±0.3)	92.7 (±1.3)	95.2 (±1.2)	91.0 (±3.0)
Across a SRC	56.0	91.5 (±3.4)	99.6 (±0.4)	99.8 (±0.2)	91.9 (±2.5)	97.1 (±0.7)	90.0 (±2.0)
Across an ORC	50.0	84.5 (±3.1)	93.5 (±4.0)	93.7 (±2.0)	86.3 (±3.2)	88.7 (±4.1)	84.0 (±3.0)
Across an ORC (no that)	52.0	75.7 (±3.3)	86.7 (±4.2)	89.4 (±2.7)	78.6 (±4.0)	86.4 (±3.5)	77.0 (±2.0)
In an ORC	84.0	84.3 (±5.5)	99.8 (±0.2)	99.9 (±0.1)	89.3 (±6.2)	92.4 (±3.5)	92.0 (±4.0)
In an ORC (no that)	71.0	81.8 (±2.3)	97.0 (±1.0)	98.6 (±0.9)	83.0 (±5.1)	88.9 (±2.4)	92.0 (±2.0)
REFLEXIVE:
Simple	83.0	94.1 (±1.9)	99.4 (±1.1)	99.9 (±0.2)	91.8 (±2.9)	98.0 (±1.1)	91.0 (±4.0)
In a sent. complement	86.0	80.8 (±1.7)	99.2 (±0.6)	97.9 (±0.8)	79.0 (±3.1)	92.6 (±2.9)	82.0 (±3.0)
Across an ORC	55.0	74.9 (±5.0)	72.8 (±2.4)	73.9 (±1.3)	72.3 (±3.0)	78.9 (±8.6)	67.0 (±3.0)
NPI:
Simple	40.0	99.2 (±0.7)	98.7 (±1.6)	97.7 (±2.0)	98.0 (±3.1)	98.2 (±1.2)	94.0 (±4.0)
Across an ORC	41.0	63.5 (±15.0)	56.8 (±6.0)	64.1 (±13.8)	64.5 (±14.0)	48.5 (±6.4)	91.0 (±7.0)
Perplexity	78.6	49.5 (±0.2)	56.4 (±0.5)	50.4 (±0.6)	49.6 (±0.3)	50.3 (±0.2)	56.7 (±0.2)
Table 1: Comparison of syntactic dependency evaluation accuracies across different types of dependencies and perplexities. Numbers in parentheses are standard deviations. M&L18 is the result of two-layer LSTM-LM in Marvin and Linzen (2018). K19 is the result of distilled two-layer LSTM-LM from RNNGs (Kuncoro et al., 2019). VP: verb phrase; PP: prepositional phrase; SRC: subject relative clause; and ORC: object-relative clause. Margin values are set to 10, which works better according to Figure 1. Perplexity values are calculated on the test set of the Wikipedia dataset. The values of M&L18 and K19 are copied from Kuncoro et al. (2019).
Figure 1: Margin value vs. macro average accuracy over the same type of constructions, or perplexity, with standard deviation for the sentence and token-level margin losses.  $\delta = 0$  is the baseline LSTM-LM without additional loss.
notice that our baseline LSTM-LM (Section 2.2) performs much better than Marvin and Linzen (2018)'s LM. A similar observation is recently made by Kuncoro et al. (2019). This suggests that the original work underestimates the true syntactic ability induced by LSTM-LMs. The table also shows the results by their distilled LSTM-LM from RNNGs (Section 1).
Higher margin value is effective For the two types of margin loss, which margin value should we use? Figure 1 reports average accuracies within the same types of constructions. For both token and sentence-levels, the task performance increases along  $\delta$ , but a too large value (15) causes a nega
tive effect, in particular on reflexive anaphora. Increases (degradations) of perplexity are observed in both methods but this effect is much smaller for the token-level loss. In the following experiments, we fix the margin value to 10 for both, which achieves the best syntactic performance.
Which additional loss works better? We see a clear tendency that our token-level margin loss achieves overall better performance. Unlikelihood loss does not work unless we choose a huge weight parameter  $(\alpha = 1000)$ , but it does not outperform ours, with a similar value of perplexity. The improvements by binary-classification loss are smaller, indicating that the signals are weaker than other methods with explicit negative exam
<sup>8</sup>We omit the comparison but the scores are overall similar.
Figure 2: Accuracies on "Across an ORC" (with and without complementizer "that") by models trained on augmented data with additional sentences containing an object RC. Margin is set to 10. X-axis denotes the total number of object RCs in the training data. 0.37M roughly equals the number of subject RCs in the original data. "animate only" is a subset of examples (see body). Error bars are standard deviations across 5 different runs.
plies. Sentence-level margin loss is conceptually advantageous in that it can deal with any type of sentence-level grammaticality including NPIs. We see that it is overall competitive with token-level margin loss but suffers from a larger increase of perplexity (4.9 points), which is observed even with smaller margin values (Figure 1). Understanding the cause of this degradation as well as alleviating it is an important future direction.
5 Limitations of LSTM-LMs
In Table 1, the accuracies on dependencies across an object RC are relatively low. The central question in this experiment is whether this low performance is due to the limitation of current architectures, or other factors such as frequency. We base our discussion on the contrast between object (7) and subject (8) RCs:
(7) The authors (that) the chef likes laugh.
(8) The authors that like the chef laugh.
Importantly, the accuracies for a subject RC are more stable, reaching  $99.8\%$  with the token-level margin loss, although the content words used in the examples are common.[9]
It is known that object RCs are less frequent than subject RCs (Hale, 2001; Levy, 2008), and it could be the case that the use of negative examples still does not fully alleviate this factor. Here, to understand the true limitation of the current LSTM architecture, we try to eliminate such other factors as much as possible under a controlled experiment.
Setup We first inspect the frequencies of object and subject RCs in the training data, by parsing them with the state-of-the-art Berkeley neural parser (Kitaev and Klein, 2018). In total, while subject RCs occur 373,186 times, object RCs only occur 106,558 times. We create three additional training datasets by adding sentences involving object RCs to the original Wikipedia corpus (Section 2.2). To this end, we randomly pick up 30 million sentences from Wikipedia (not overlapped to any sentences in the original corpus), parse by the same parser, and filter sentences containing an object RC, amounting to 680,000 sentences. We create augmented training sets by adding a subset, or all of these sentences to the original training sentences. Among the test cases about object RCs we only report accuracies on subject-verb agreement, on which the portion for subject RCs also exists. This allows us to compare the difficulties of two types of RCs for the present models. We also evaluate on "animate only" subset, which has a correspondence to the test cases for subject RCs with only differences in word order and inflection (like (7) and (8); see footnote 9). Of particular interest to us is the accuracy on these animate cases. We expect that the main reason for lower performance for object RCs is due to frequency, and with our augmentation the accuracy will reach the same level as that for subject RCs.
Results However, for both all and animate cases, accuracies are below those for subject RCs (Figure 2). Although we see improvements from the original score (93.7), the highest average accuracy by the token-level margin loss on the "animate" subset is 97.1 ("with that"), not beyond  $99\%$ . This result indicates some architectural limitations of LSTM-LMs in handling object RCs robustly at a near perfect level. Answering why the accuracy
9 Precisely, they are not the same. Examples of object RCs are divided into two categories by the animacy of the main subject (animate or not), while subject RCs only contain animate cases. If we select only animate examples from object RCs the vocabularies for both RCs are the same, remaining only differences in word order and inflection, as in (7, 8).
3381
Figure 3: An ablation study to see the performance of models trained with reduced explicit negative examples (token-level and construction-level). One color represents the same models across plots, except the last bar (construction-level), which is different for each plot.
does not reach (almost)  $100\%$ , perhaps with other empirical properties or inductive biases (Khandelwal et al., 2018; Ravfogel et al., 2019) is future work.
6 Do models generalize explicit supervision, or just memorize it?
One distinguishing property of our margin loss, in particular token-level loss, is that it is highly lexical, making a contrast explicitly between correct and incorrect words. This direct signal may make models acquire very specialized knowledge about each target word, not very generalizable one across similar words and occurring contexts. In this section, to get insights into the transferability of syntactic knowledge induced by our margin losses, we provide an ablation study by removing certain negative examples during training.
Setup We perform two kinds of ablation. For token-level ablation (-TOKEN), we avoid creating negative examples for all verbs that appear as a target verb $^{10}$  in the test set. Another is construction-level (-PATTERN), by removing all negative examples occurring in a particular syntactic pattern. We ablate a single construction at a time for -PATTERN, from four non-local subject-verb dependencies (across a prepositional phrase (PP), sub
Models	Second verb (V1 and V2)
All verbs	like	other verbs
LSTM-LM	82.2 (±3.4)	13.0 (±12.2)	89.9 (±3.6)
Margin (token)	99.0 (±0.8)	94.0 (±6.5)	99.6 (±0.5)
-TOKEN	90.8 (±3.3)	51.0 (±29.9)	95.2 (±2.6)
-PATTERN	90.1 (±4.6)	50.0 (±30.6)	94.6 (±2.2)
Table 2: Accuracies on long VP coordinations by the models with/without ablations. "All verbs" scores are overall accuracies. "like" scores are accuracies on examples on which the second verb (target verb) is like.
Models	First verb (V1 and V2)
likes	other verbs
LSTM-LM	61.5 (±20.0)	93.5 (±3.4)
Margin (token)	97.0 (±4.5)	99.9 (±0.1)
-TOKEN	63.5 (±18.5)	99.2 (±1.1)
-PATTERN	67.0 (±21.2)	98.0 (±1.4)
Table 3: Further analysis of accuracies on the "other verbs" cases of Table 2. Among these cases, the second column ("likes") shows accuracies on examples where the first verb (not target) is likes.
ject RC, object RC, and long verb phrase (VP)).<sup>11</sup> We hypothesize that models are less affected by token-level ablation, as knowledge transfer across words appearing in similar contexts is promoted by language modeling objective. We expect that construction-level supervision would be necessary to induce robust syntactic knowledge, as perhaps different phrases, e.g., a PP and a VP, are processed differently.
Results Figure 3 is the main results. Across models, we restrict the evaluation on four nonlocal dependency constructions, which we select as ablation candidates as well. For a model with -PATTERN, we evaluate only on examples of construction ablated in training (see caption). To our surprise, both -TOKEN and -PATTERN have similar effects, except "Across an ORC", on which the degradation by -PATTERN is larger. This may be related to the inherent difficulty of object RCs for LSTM-LMs that we verified in Section 5. For such particularly challenging constructions, models may need explicit supervision signals. We observe lesser score degradation by abating prepositional phrases and subject RCs. This suggests that, for example, the syntactic knowledge strengthened for prepositional phrases with negative examples could be exploited to learn the syntactic patterns about
<sup>11</sup>We identify all these cases from the parsed training data, which we prepared for the analysis in Section 5.
swim, smile, laugh, enjoy, hate, bring, interest, like, write, admire, love, know, and is.
subject RCs, even when direct learning signals on subject RCs are missing.
We see approximately 10.0 points score degradation on long VP coordination by both ablations. Does this mean that long VPs are particularly hard in terms of transferability? We find that the main reasons for this drop, relative to other cases, are rather technical, essentially due to the target verbs used in the test cases. See Table 2, 3, which show that failed cases for the ablated models are often characterized by the existence of either like or likes. Excluding these cases ("other verbs" in Table 3), the accuracies reach 99.2 and 98.0 by -TOKEN and -PATTERN, respectively. These verbs do not appear as a target verb in the test cases of other tested constructions. This result suggests that the transferability of syntactic knowledge to a particular word may depend on some characteristics of that word. We conjecture that the reason for weak transferability to likes and like is that they are polysemous; e.g., in the corpus, like is much more often used as a preposition and being used as a present tense verb is rare. This type of issue due to frequency may be one reason for lessening the transferability. In other words, like can be seen as a challenging verb to learn its usage only from the corpus, and our margin loss helps for such cases.
Our results with explicit negative examples are overall positive. We have demonstrated that models exposed to these examples at training time in an appropriate way will be capable of handling the targeted constructions at near perfect level except a few cases. We found that our new token-level margin loss is superior to the other approaches and the remaining challenging cases are dependencies across an object relative clause.
Object relative clauses are known to be harder for a human as well, and our results may indicate some similarities in the sentence processing behaviors by a human and RNN, though other studies also find some dissimilarities between them (Linzen and Leonard, 2018; Wilcox et al., 2019a). The difficulty of object relative clauses for RNN-LMs has also been observed in the prior work (Marvin and Linzen, 2018; van Schijndel et al., 2019). A new insight provided by our study is that this difficulty holds even after alleviating the frequency effects by augmenting the target structures along with direct supervision signals. This
indicates that RNNs might inherently suffer from some memory limitation like a human subject, for which the difficulty of particular constructions, including center-embedded object relative clauses, are known to be incurred due to memory limitation (Gibson, 1998; Demberg and Keller, 2008) rather than purely frequencies of the phenomena. In terms of language acquisition, the supervision provided in our approach can be seen as direct negative evidence (Marcus, 1993). Since human learners are known to acquire syntax without such direct feedback we do not claim that our proposed learning method itself is cognitively plausible.
One limitation of our approach is that the scope of negative examples has to be predetermined and fixed. Alleviating this restriction is an important future direction. Though it is challenging, we believe that our final analysis for transferability, which indicates that the negative examples do not have to be complete and can be noisy, suggests a possibility of a mechanism to induce negative examples themselves during training, perhaps relying on other linguistic cues or external knowledge.
We would like to thank Naho Orita and the members of Computational Psycholinguistics Tokyo for their valuable suggestions and comments. This paper is based on results obtained from projects commissioned by the New Energy and Industrial Technology Development Organization (NEDO).
Vera Demberg and Frank Keller. 2008. Data from eyetracking corpora as evidence for theories of syntactic processing complexity. Cognition, 109:193-210.
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 199-209, San Diego, California. Association for Computational Linguistics.
Émile Enguehard, Yoav Goldberg, and Tal Linzen. 2017. Exploring the syntactic abilities of RNNs with multi-task learning. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 3-14, Vancouver, Canada. Association for Computational Linguistics.
Edward Gibson. 1998. Linguistic complexity: Locality of syntactic dependencies. Cognition, 68(1):1-76.
3383
Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1195-1205. Association for Computational Linguistics.
John Hale. 2001. A probabilistic earley parser as a psycholinguistic model. In Second Meeting of the North American Chapter of the Association for Computational Linguistics.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration.
Jiaji Huang, Yi Li, Wei Ping, and Liang Huang. 2018. Large margin neural language model. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1183-1191, Brussels, Belgium. Association for Computational Linguistics.
Hakan Inan, Khashayar Khosravi, and Richard Socher. 2016. Tying word vectors and word classifiers: A loss framework for language modeling. In International Conference on Learning Representations.
Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018. Sharp nearby, fuzzy far away: How neural language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 284-294, Melbourne, Australia. Association for Computational Linguistics.
Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2676–2686, Melbourne, Australia. Association for Computational Linguistics.
Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. 2018. Lstms can learn syntax-sensitive dependencies well, but modeling structure makes them better. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1426-1436, Melbourne, Australia. Association for Computational Linguistics.
Adhiguna Kuncoro, Chris Dyer, Laura Rimell, Stephen Clark, and Phil Blunsom. 2019. Scalable syntax-aware language models using knowledge distillation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3472-3484, Florence, Italy. Association for Computational Linguistics.
Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126-1177.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521-535.
Tal Linzen and Brian Leonard. 2018. Distinct patterns of syntactic agreement errors in recurrent networks and humans. In Proceedings of the 40th Annual Conference of the Cognitive Science Society, pages 692-697, Austin, TX. Cognitive Science Society.
Gary F. Marcus. 1993. Negative evidence in language acquisition. Cognition, 46(1):53 - 85.
Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192-1202, Brussels, Belgium. Association for Computational Linguistics.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and optimizing LSTM language models. In International Conference on Learning Representations.
Tomas Mikolov, Stefan Kombrink, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. 2011. Extensions of recurrent neural network language model. In ICASSP, pages 5528-5531. IEEE.
Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. 2019. Studying the inductive biases of RNNs with synthetic variations of natural languages. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3532-3542, Minneapolis, Minnesota. Association for Computational Linguistics.
Marten van Schijndel, Aaron Mueller, and Tal Linzen. 2019. Quantity doesn't buy quality syntax with neural language models. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, Hong Kong, China. Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 conference of the North American chapter of the association for computational linguistics on human language technology volume 1, pages 173-180. Association for Computational Linguistics.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In International Conference on Learning Representations.
Ethan Wilcox, Roger P. Levy, and Richard Futrell. 2019a. What syntactic structures block dependencies in rnn language models? In Proceedings of the 41st Annual Meeting of the Cognitive Science Society. Cognitive Science Society.
Ethan Wilcox, Peng Qian, Richard Futrell, Miguel Ballesteros, and Roger Levy. 2019b. Structural supervision improves learning of non-local grammatical dependencies. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3302-3312, Minneapolis, Minnesota. Association for Computational Linguistics.