SlowGuess/ACL_2020_OCR huggingface dataset:
To address all biases together using only a single techniques we modified margin approaches to minimize all margin violations in the beam instead of just the single one. When gold hypothesis falls out of the beam BSO-Early and BSO-LaSO use only the most violated hypothesis to update the parameters. However, there is no good reason not to update against all violations present in the beam. LeCun et al. (2006) argue that the good property of CRF models is that they simultaneously decrease weight of all bad hypotheses simultaneously. Our BSO- $*$ -All approach can be seen as an approximation of this idea using a beam. This small modification does not slow down training in any significant manner (we already have a forward pass for all the additional hypotheses because they are in the beam) and it gives significant improvements in parsing accuracy.
We have conducted experiments on English CCGbank (Hockenmaier and Steedman, 2007). For evaluation we use F1 score over labelled-directed and unlabelled-undirected dependencies. The parser is implemented in Scala and uses DyNet (Neubig et al., 2017) for the neural computation. The code is available on github.<sup>5</sup>
There are two dependency types often used in CCG parsing research: first one from Clark et al., 2002) which is much closer to the typical CCG notion of dependencies and the second one from Clark and Curran, 2007) which is more formalism-neutral but less expressive. The only implementation of the second method is the one in C&C parser and is not able to handle all the categories that come from CCGbank. This is the rea
son why most previous work on incremental CCG parsing has used the dependencies of Clark et al. (2002). In order to be able to compare to them we use the same dependencies.
5.1 Models tested
We have tested the following methods:
Disc Incremental discriminative model (the baseline).
Disc-REINFORCE Discriminative model trained using REINFORCE to maximise the expected reward (F1 score of CCG dependencies).
Gen Generative model that additionally has word generation transitions.
Gen-WordSync Same generative model but decoded with word-synchronous beam with main beam size 100, word-beam size 10 and no fast-tracking.
Gen-Rescaled Generative model that uses additional three weights to put the probabilities of all actions on the same scale.
BSO-Early-Single and BSO-LaSO-Single
Un-normalised model trained with Early and LaSO updates but only with single violation per update as proposed in Wiseman and Rush (2016).
BSO-Early-All and BSO-LaSO-All Same as above but with minimizing all violations present in the beam. We refer to them together as BSO-\*-All.
CRF-Early Globally normalized model with Early update as proposed in (Zhou et al., 2015; Andor et al., 2016).
CRF-LaSO Same as above but modified to use LaSO instead of Early update.
All beam approximation methods used beam of size 32. The number of samples in REINFORCE is 32 and it includes a gold hypothesis for stability as suggested by Fried and Klein (2018).
CRF-Early achieved accuracy of  $36.9\%$ , BSO-Early-Single of  $51.7\%$  and Gen-WordSync of  $58.1\%$  which are all way below the baseline. CRF-Early and BSO-Early-Single update methods probably gave bad results because the training is too unstable with Early heuristic that often does not get to learn from the whole transition sequence. We are not sure why Gen-WordSync gave
5https://github.com/stanojevic/ Rotating-CCG/tree/incremental_max_margin
Figure 1: Reranking 100 samples of dev set sentences generated by discriminative non-incremental model.
bad results. It could be that word-synchronous decoding while addressing the imbalanced probability search bias introduces some other search bias that is even more harmful. Another reason could be that, unlike RNNG, we have introduced an additional bottleneck of supertagging transitions that would require additional modifications. We will not consider these methods in the rest of the paper.
5.2 Results: Incremental Beam Search
Figure 2 shows the results for all the other methods with different beam sizes. REINFORCE training does improve the robustness of the discriminative model. It improved greedy decoding by  $10\%$  more than any other method, but due to label-bias it cannot exploit the benefits of a larger beam.
The generative model addresses the label bias which is evident from relatively good results with a bigger beam. When on top of the generative model we add Rescaling parameters the model gets even more benefit as the beam gets larger.
The BSO-LaSO-Single model that addresses all three biases at the same time gets very good results and is outperformed by Gen-Rescaled model only in the context of a very large beam. Gen-Rescaled and BSO-LaSO-Single get close to  $80\%$  but do not go above it. Our BSO-\*All modification to beam search optimisations gives significantly better results already with a very small beam. With beam of size 8 BSO-LaSO-All crosses the border of  $80\%$  and it improves all the way to  $82.7\%$ . This is only  $4.8\%$  lower than the upper bound set by the non-incremental model. BSO-LaSO-All is a small modification over BSO-LaSO-Single but is responsible for more than  $5\%$  of improvement
over it. The importance of updating for all violations is particularly striking with the case of BSO-Early where the accuracy increases by  $29\%$ .
CRF-Early already has the property of updating against all bad hypotheses in the beam but it differs from our best method in the type of loss (logistic vs max-margin) and the update heuristic (Early vs LaSO). We have also tried modifying the CRF method to use LaSO (CRF-LaSO) which made the model significantly better than the original CRF-Early but still much lower than BSO- $*$ -All.
5.3 Results:Reranking
Is the gap between non-incremental models and incremental models due to the imperfect search or to the imperfect prediction models? To test that we have conducted an experiment where the models need only to rerank a list of 100 derivations sampled from non-incremental model for each sentence in the development set. This puts beam search out of the equation and tests only how good are the models as discriminators between good and bad trees. The samples have trees of mixed quality: the worst score a parser could get by reranking the trees is 67.8 F1 while the best is 95.8 F1.
The results in Figure 1 show that the gap between incremental and non-incremental models is around one point of F1-score. This is much smaller than the results with beam search would lead us to expect. Also here the generative model outperforms BSO-LaSO-All. This means that the primary reason for success of BSO-LaSO-All over Gen in beam search is probably due to its incremental scoring (a property that was also noticed by Goyal et al. (2019) for similar models) and/or lack of imbalanced probability bias.
We have also conducted reranking using Minimum-Bayes Risk (MBR) method (Kumar and Byrne, 2004) which finds the hypothesis that would minimise the expected loss under some metric. In the parsing context that means finding the tree with the best expected F1-score (Goodman, 1996; Titov and Henderson, 2006; Stanojevic and Sima'an, 2015). MBR is defined only for probabilistic models, but as Titov and Henderson (2006) show it could also be adapted and applied to non-probabilistic models, such as our BSO-LaSO-All model.
Figure 1 shows that while MBR does not make any significant difference for the non-incremental model, it makes a huge difference for the incre
Figure 2: Influence of beam size on the dev results.
mental models. With MBR they all manage to outperform the non-incremental model. However, we should not credit this right away to the quality of the incremental models. As Fried et al. (2017) point out, improvements in reranking with a different model could be a result of model ensembling.
5.4 Results: Test set performance
Table 1 compares our strongest method on the test set against all the previously published incremental CCG models. The results show that it outperforms all the previous incremental models when using beams of the same size. The improvement is even bigger with the larger beam. Even thought our primary goal is not to compete with non-incremental parsers, our incremental model outperforms some widely used non-incremental CCG parsers such as EasyCCG (Lewis and Steedman, 2014). The result is particularly good for unlabelled dependencies.
We also report the results of applying MBR reranking using incremental model over the samples generated by the non-incremental model. This model outperforms other incremental and non-incremental models on all metrics.
6 Other relevant work
The incremental CCG parser of Ambati (2016) uses the linear model trained with a structured perceptron objective and the early update heuristic. Given the simplicity of that model, it performs surprisingly well. The reason is the fact that the structured perceptron addresses all the biases identified in our paper. Our work has been an attempt to
Tag	UF	LF
Incremental	Hassan et al. (2008)	beam= 1	—	59.0
Ambati (2016)	beam= 1	74.6	67.5
this work	beam= 1	78.8	69.9
Goyal et al. (2019)	beam= 5	85.5	—
this work	beam= 5	90.1	92.2
Ambati (2016)	beam=16	90.8	88.3
this work	beam=16	91.4	91.5
this work	beam=64	92.0	92.3
Non-Incremental	Lewis and Steedman (2014)	93.0	88.6
Ambati et al. (2015)	91.2	89.0
Hockenmaier (2003)	92.2	92.0
Zhang and Clark (2011)	93.1	—
Xu et al. (2016)	93.9	—
Clark and Curran (2007)	94.3	93.0
Stanojevic and Steedman (2019)	95.4	95.8
this work	MBR reranking	95.6	95.9
Table 1: Results on the test set. The results of Non-Incremental parsers are shown only as a reference.
bring these benefits to more modern neural models.
Another interesting approach to tackle label-bias while keeping the probabilistic interpretation is the error-states model of Vaswani and Sagae (2016). This model in its original formulation would not be computationally efficient in our setting because there are too many instances of error-states to be trained on in CCG parsing caused by large number of transitions. Possibly some modification based on sampling could remedy this.
There has also been some recent work on reducing the imbalanced probability bias. Mabona et al. (2019) propose an algorithmic solution for organising beam search into buckets that have the same number of expensive transitions. Crabbé et al. (2019) propose a sampling based approach with
the same motivation of controlling which hypotheses are being compared.
Of relevance for the CCG incrementality are Sturt and Lombardo (2005) and Demberg et al. (2013) who claimed that human sentence processing is more incremental than CCG allows under SCH for sentences like:
The pilot embarrassed Mary and put herself in a very awkward situation.
Here a male gender-biased interpretation of the antecedent "the pilot" conflicts with a feminine bound reflexive "herself". The eye-movements show processing difficulty as soon as "put herself" is read, rather than being delayed until the completion of the VP by the PP. This allows subject binding to be established by VP coordination.
Stanojevic et al. (2020) argue that Sturt and Lombardo's result is explained by the fact that the category for "put" is predictive of a future PP, allowing establishment of binding in advance of parsing without strict incrementality or compromising SCH.
The methods discussed here have been applied to the task of incremental CCG parsing, but they are not limited to CCG or even to parsing as a task. In principle, they could be applied to any task involving sequential structure prediction. We see this as the most interesting use case not only for the BSO*-All training method but also for having an incremental CCG parser. Such parsers can potentially make much more informed decisions about the next word, compared to the models based on mere sequence of words prefix, by including semantic and referential meaning (Altmann and Steedman, 1988), as well as syntax.
This work was supported by ERC H2020 Advanced Fellowship GA 742137 SEMANTAX grant.
Steven P. Abney and Mark Johnson. 1991. Memory requirements and local ambiguities of parsing strategies. Journal of Psycholinguistic Research, 20:233-249.
Anthony Ades and Mark Steedman. 1982. On the order of words. Linguistics and Philosophy, 4:517-558.
Gerry Altmann and Mark Steedman. 1988. Interaction with context during human sentence processing. Cognition, 30(3):191-238.
Bharat Ram Ambati. 2016. Transition-based combinatory categorial grammar parsing for English and Hindi. Ph.D. thesis, University of Edinburgh.
Bharat Ram Ambati, Tejaswini Deoskar, Mark Johnson, and Mark Steedman. 2015. An Incremental Algorithm for Transition-based CCG Parsing. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 53-63. Association for Computational Linguistics.
Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. arXiv preprint arXiv:1603.06042.
L. Bottou. 1991. Une Approche Theorique de l'Apprentissage Connexionniste: Applications a la Reconnaissance de la Parole. Ph.D. thesis.
Stephen Clark and James R Curran. 2007. Wide-coverage efficient statistical parsing with CCG and log-linear models. Computational Linguistics, 33(4):493-552.
Stephen Clark, Julia Hockenmaier, and Mark Steedman. 2002. Building deep dependency structures with a wide-coverage CCG parser. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 327-334. Association for Computational Linguistics.
Michael Collins and Brian Roark. 2004. Incremental parsing with the perceptron algorithm. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, page 111. Association for Computational Linguistics.
Benoit Crabbé, Murielle Fabre, and Christophe Pallier. 2019. Variable beam search for generative neural parsing and its relevance for the analysis of neuroimaging signal. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1150-1160, Hong Kong, China. Association for Computational Linguistics.
James Cross and Liang Huang. 2016. Span-based constituency parsing with a structure-label system and provably optimal dynamic oracles. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1-11.
Fernando Cuetos and Don C. Mitchell. 1988. Cross-linguistic differences in parsing: Restrictions on the use of the late closure strategy in spanish. Cognition, 30(1):73 - 105.
Hal Daumé III and Daniel Marcu. 2005. Learning as search optimization: Approximate large margin methods for structured prediction. In Proceedings of the 22nd international conference on Machine learning, pages 169-176. ACM.
Vera Demberg, Frank Keller, and Alexander Koller. 2013. Incremental, predictive parsing with psycholinguistically motivated tree-adjoining grammar. Computational Linguistics, 39:1025-1066.
Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 334-343, Beijing, China. Association for Computational Linguistics.
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A Smith. 2016. Recurrent Neural Network Grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 199-209.
Lyn Frazier. 1979. On comprehending sentences: Syntactic parsing strategies. Ph.D. thesis, University of Massachusetts, Amherst.
Daniel Fried and Dan Klein. 2018. Policy gradient as a proxy for dynamic oracles in constituency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 469-476.
Daniel Fried, Mitchell Stern, and Dan Klein. 2017. Improving Neural Parsing by Disentangling Model Combination and Reranking Effects. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 161-166.
Yoav Goldberg and Joakim Nivre. 2012. A dynamic oracle for arc-eager dependency parsing. Proceedings of COLING 2012, pages 959-976.
Joshua Goodman. 1996. Parsing Algorithms and Metrics. In Proceedings of the 34th Annual Meeting on Association for Computational Linguistics, ACL '96, pages 177-183, Stroudsburg, PA, USA. Association for Computational Linguistics.
Kartik Goyal, Chris Dyer, and Taylor Berg-Kirkpatrick. 2019. An empirical investigation of global and local normalization for recurrent neural sequence models using a continuous relaxation to beam search. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 1724-1733.
Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. 2005. Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition. In Proceedings of the 15th International Conference on Artificial Neural Networks: Formal Models and Their Applications - Volume Part II, ICANN'05, pages 799-804, Berlin, Heidelberg. Springer-Verlag.
John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies, pages 1-8. Association for Computational Linguistics.
John Hale, Chris Dyer, Adhiguna Kuncoro, and Jonathan Brennan. 2018. Finding syntax in human encephalography with beam search. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2727-2736. Association for Computational Linguistics.
John T. Hale. 2014. Automaton Theories of Human Sentence Comprehension. CSLI, Stanford.
Hany Hassan, Khalil Sima'an, and Andy Way. 2008. A syntactic language model based on incremental CCG parsing. In SLT, pages 205-208. IEEE.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. *Neural Computation*, 9(8).
Julia Hockenmaier. 2003. Data and models for statistical parsing with Combinatory Categorical Grammar. Ph.D. thesis, University of Edinburgh. College of Science and Engineering. School of Informatics.
Julia Hockenmaier and Mark Steedman. 2007. CCGbank: a corpus of CCG derivations and dependency structures extracted from the Penn Treebank. Computational Linguistics, 33(3):355-396.
Shankar Kumar and William Byrne. 2004. Minimum Bayes-Risk Decoding for Statistical Machine Translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In ICML '01: Proceedings of the Eighteenth International Conference on Machine Learning, pages 282-289, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Yann LeCun, Sumit Chopra, Raia Hadsell, Marc Aurelio Ranzato, and Fu Jie Huang. 2006. A tutorial on energy-based learning. In *Predicting structured data*. MIT Press.
Mike Lewis and Mark Steedman. 2014. A* CCG parsing with a supertag-factored model. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 990-1000.
Amandla Mabona, Laura Rimell, Stephen Clark, and Andreas Vlachos. 2019. Neural generative rhetorical structure parsing. arXiv preprint arXiv:1909.11049.
William Marslen-Wilson. 1973. Linguistic structure and speech shadowing at very short latencies. Nature, 244:522-523.
Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Paul Michel, Yusuke Oda, Matthew Richardson, Naomi Saphra, Swabha Swayamdipta, and Pengcheng Yin. 2017. Dynet: The dynamic neural network toolkit. arXiv preprint arXiv:1701.03980.
Fernando C. Pereira. 1985. A new characterization of attachment preferences. In David R. Dowty, Lauri Kartunnen, and Arnold M. Zwicky, editors, Natural language parsing: Psychological, Computational, and Theoretical perspectives, pages 307-319.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL.
Philip Resnik. 1992. Left-corner parsing and psychological plausibility. In Proceedings of the 14th International Conference on Computational Linguistics, COLING 92, pages 191-197.
Brian Roark. 2001. Probabilistic top-down parsing and language modeling. Computational linguistics, 27(2):249-276.
Brian Roark and Mark Johnson. 1999. Efficient probabilistic top-down and left-corner parsing. In Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, pages 421-428. Association for Computational Linguistics.
Edward P Stabler. 1991. Avoid the pedestrian's paradox. In *Principle-based parsing*, pages 199-237. Springer.
Miloš Stanojevic and Edward Stabler. 2018. A Sound and Complete Left-Corner Parsing for Minimalist Grammars. In Proceedings of the Eight Workshop on Cognitive Aspects of Computational Language Learning and Processing, pages 65-74. Association for Computational Linguistics.
Miloš Stanojevic, John Hale, and Mark Steedman. 2020. Predictive Processing of Coordination in CCG. In Proceedings of the 33rd Annual CUNY Conference on Human Sentence Processing, Amherst, Massachusetts. University of Massachusetts.
Miloš Stanojevic and Khalil Sima'an. 2015. Reordering Grammar Induction. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 44-54, Lisbon, Portugal. Association for Computational Linguistics.
Miloš Stanojevic and Mark Steedman. 2019. CCG Parsing Algorithm with Incremental Tree Rotation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, Volume 1 (Long Papers). Association for Computational Linguistics.
Mark Steedman. 1989. Grammar, interpretation, and processing from the lexicon. In William Marslen-Wilson, editor, Lexical Representation and Process, pages 463-504. MIT Press, Cambridge, MA.
Mark Steedman. 2000. The Syntactic Process. MIT Press, Cambridge, MA.
Mitchell Stern, Daniel Fried, and Dan Klein. 2017. Effective inference for generative neural parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1695-1700.
Andreas Stolcke. 1995. An efficient probabilistic context-free parsing algorithm that computes prefix probabilities. Computational linguistics, 21(2):165-201.
Patrick Sturt and Vincenzo Lombardo. 2005. Processing coordinated structures: Incrementality and connectedness. Cognitive Science, 29:291-305.
Ben Taskar, Carlos Guestrin, and Daphne Koller. 2004. Max-Margin Markov Networks. In Advances in neural information processing systems, pages 25-32.
Ivan Titov and James Henderson. 2006. Loss Minimization in Parse Reranking. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 560-567, Sydney, Australia. Association for Computational Linguistics.
I. Tsochantaridis, T. Hofmann, T. Joachims, and Y. Al-tun. 2004. Support Vector Machine Learning for Interdependent and Structured Output Spaces. In International Conference on Machine Learning (ICML), pages 104-112.
Ashish Vaswani and Kenji Sagae. 2016. Efficient structured inference for transition-based parsing with neural networks and error states. Transactions of the Association for Computational Linguistics, 4:183-196.