SlowGuess/ACL_2020_OCR huggingface dataset:
It is crucial to understand whether and to what extent parsers are indeed robust to learner errors. We re-analyse the results from two aspects. First, we modify the original SMATCH evaluation metric and enable it to be sensitive to distances from errors. Then we make a distinction among typical error types proposed in CoNLL-2014 Shared Task (Ng et al., 2014). Results show that standard parsers can not handle learner errors well enough and their behaviors vary among different
6788
Data	LEX	CXG	FAC
Node	Edge	All	Node	Edge	All	Node	Edge	All
DeepBank	94.05	92.96	93.50	95.83	92.87	94.34	96.85	95.19	96.01
L1	88.41	86.44	87.41	90.32	86.04	88.14	92.28	89.12	90.91
L2	84.38	82.23	83.29	86.47	81.70	84.04	88.68	84.45	86.91
Δ	4.03	4.21	4.12	3.85	4.34	4.10	3.60	4.67	4.00
Table 3: SMATCH scores of semantic parsing on different test data. Henceforth,  $LEX$ ,  $CXG$  and  $FAC$  refer to lexicalized and constructional composition-based parsers and the factorization-based parser, respectively.  $\Delta$  refers to the gap between  $L1$  and  $L2$ .
Model	Data	Node	Edge	All
LEX	✓	86.31	81.95	84.23
×	68.94	79.81	75.74
CXG	✓	89.04	82.14	85.75
×	71.46	79.77	76.66
FAC	✓	90.96	84.48	87.86
×	73.55	80.27	77.75
Table 4:  $\checkmark$  refers to error-ignored  $(\sigma_{k} = 0$  when the  $k$ th triple in  $G_{g}$  is involved with errors,  $\sigma_{k} = 1$  otherwise) SMATCH scores while  $\times$  refers to error-oriented  $(\sigma_{k} = 1$  when the  $k$ th triple in  $G_{g}$  is involved with errors,  $\sigma_{k} = 0$  otherwise) SMATCH scores.
error types.
It should be noticed that only several points in a sentence are occupied by errors while most of the structure is still well-formed. The scores of  $L2$  in Table 3 may be not able to exactly reflect the robustness of models. Therefore, we modify the original SMATCH evaluation metric by paying additional attention to erroneous points. The original metric can be formulated into an Integer Linear Programming (ILP) problem. Suppose there are gold and predicted graphs  $G_{g}$  ( $m$  variables) and  $G_{p}$  ( $n$  variables). Semantic relations in graphs are represented as triples which can illustrate both the concepts (represented as (variable, concept, relation)) and edges (represented as (variable1, variable2, relation)). We define  $v_{ij} = 1$  iff the  $i$ th variable in  $G_{g}$  is mapped to the  $j$ th variable in  $G_{p}$  in the current alignment,  $v_{ij} = 0$  otherwise. We have  $t_{kl} = 1$  iff the  $k$ th triple ( $x, y, \text{relation1}$ ) in  $G_{g}$  and the  $l$ th triple ( $w, z, \text{relation2}$ ) in  $G_{p}$  are matched, which means  $v_{xw} = 1$ ,  $v_{yz} = 1$  and  $\text{relation1=relation2}$ . In the original metric,  $t_{kl}$  takes the value of 1 or 0 and all triple pairs are treated equally. In order to focus on the error
neous points, we put various weights on different triple pairs depending on their distance from errors. Then the optimization problem can be stated as:
$$
\begin{array}{l} \max \sum_ {k l} \sigma_ {k} t _ {k l} \\ \begin{array}{l l} s. t. & \sum_ {j} v _ {i j} \leq 1, \hskip 2 8. 4 5 2 7 5 6 p t i = 1, 2, 3 \ldots , m \\ & \sum_ {i} v _ {i j} \leq 1, \hskip 2 8. 4 5 2 7 5 6 p t j = 1, 2, 3 \ldots , n \end{array} \\ t _ {r _ {x y} r _ {w z}} \leq v _ {x w}, \\ t _ {r _ {x y} r _ {w z}} \leq v _ {y z}, \quad r _ {x y} r _ {w z} \in \mathcal {R} \\ \end{array}
$$
Here,  $r_{xy}$  means the triple describing the relationship between  $x$  and  $y$ , and  $\mathcal{R}$  means the set of all triple pairs.  $\sigma_k$  refers to the weight of the  $k$ th triple in  $G_g$ . If we want to explore the performance on erroneous points, triples related to these points will be assigned a larger weight. If we want to find out the performance on good part, we can just set the weight of triples involved with errors to zero.
Table 4 compares the error-oriented and error-ignored results. We can see that although the average gap in Table 3 is about 4 points, the actual performance pertaining to the ill-formed part is much lower. Especially, the F-score of nodes drops heavily. The gray line in Figure 6 illustrates the tendency of scores changing with the distance from abnormal points. It clearly shows that farther nodes suffer less.
Moreover, we explore the relationship between learner errors (LEs) and parsing errors (PEs). We find that  $21.40\%$  PEs are caused by LEs and  $66.80\%$  LEs cause at least one PE. It indicates that parsing models are really struggling with learner errors.
Furthermore, we look into the produced graphs with regard to different error types. We refer to the list of error types introduced in the CoNLL-2014 Shared Task (Ng et al., 2014). Detailed results are illustrated in Figure 5. This dia
Figure 5: Overall SMATCH scores with regard to different grammatical error types. Detailed descriptions of errors are provided in Ng et al. (2014).
Model	F0.5
Chollampatt and Ng (2018)	45.36
Zhao et al. (2019)	61.15
Table 5: Performances of the two GEC models on CoNLL-2014 test set.
gram reflects a clear comparison among different error types. The four best-performing types are ArtOrDet, Nn, Pform and Wform, referring to errors of article or determinier, noun number, pronoun form and general word form, respectively. We can see that most of them are related to morphological variations and can be disambiguated at the word level. In contrast, WOadv and WOinc, meaning incorrect adjective/adverb order and other word order errors, are much more complex. They are involved with reorganizations of the sentence structure and hence more difficult to handle. Factorization-based model is more robust to these hard cases than composition-based models since it is grounded on graph structures and can reduce the influence from broken sequential syntax.
6 Parsing to Intended Meanings
Previous evaluation indicates the difficulty to adopt a standard semantic parsing model to handle competence errors. Motivated by this fact, we are concerned with whether it is feasible to automatically normalize the texts first. Specifically, our strategy is correcting the grammatical errors contained in the input sentences, and then parsing the revised texts into semantic struc
tures with standard models. The first step can resort to Grammatical Error Correction (GEC), the task of correcting different kinds of errors in text. It has attracted a lot of attention and considerable effort has been made to promote the performance on specific benchmark data. We utilize two off-the-shelf GEC models. One is a multilayer convolutional encoder-decoder neural network proposed in Chollampatt and Ng (2018). We choose the basic model introduced in the paper. The other model copies the unchanged words from the source sentence to the target sentence using a pretrained copy-augmented architecture with a denoising auto-encoder (Zhao et al., 2019). It achieves the state-of-the-art performance without extra pseudo data. Performances of the two GEC models on CoNLL-2014 test set are shown in Table 5.
We train the factorization-based model on DeepBank and examine the performance on L2 and L1 sentences as well as the revised sentences by two GEC models. The produced graphs are compared with  $I$ -silver which represents the intended meaning. We notice that during the computation of SMATCH, some disagreements of nodes result from the discrepancy of morphological variation or different collocations between the input and the standard sentence. Hence the node score may be underestimated. Therefore, we relax the standards of matching nodes. We establish a paraphrase table based on the statistical machine translation between a parallel learner corpus<sup>3</sup>. As long as the labels of two aligned nodes have the same stem or they form a paraphrase pair in our table, then the two nodes can be considered "matching". We call the new evaluation metric as "node-relaxed SMATCH".
Table 6 summarizes the results. The gap between the first and the last rows demonstrates that it may be difficult to automatically infer the intended meaning based on the literal representation. GEC does help us to understand the learner English, but it seems to be a small step on the progress bar. Although the second GEC model (Zhao et al., 2019) outperforms the first model (Chollampatt and Ng, 2018) a lot on benchmark data (Table 5), its superiority on semantic parsing is not so obvious. There is still a long way to go before automatically capturing the intended mean
<sup>3</sup>https://sites.google.com/site/naistlang8corpora/
Test Data	Standard	Error-oriented	Node-relaxed
Node	Edge	All	Node	Edge	All	Node	Edge	All
L2 sentence	83.91	84.86	84.39	45.91	78.93	66.73	57.18	78.45	70.59
Chollampatt and Ng (2018)	84.98	85.13	85.06	53.06	80.06	70.08	62.05	79.26	72.90
Zhao et al. (2019)	86.10	85.85	85.98	58.73	80.56	72.49	65.39	80.09	74.66
L1 sentence	92.28	89.60	90.92	86.08	85.72	85.85	89.64	85.48	87.02
Table 6: Results of SMATCH scores compared to  $I$ -silver. Chollampatt and Ng (2018) and Zhao et al. (2019) mean the revised sentences with GEC models introduced in the two studies. "Error-oriented" means only focusing on the parts aligned to grammatical errors in  $I$ -silver. "Node-relaxed" is an error-oriented metric that relax the standards of matching nodes.
ing like humans.
Figure 6: Overall SMATCH scores with regard to the distance from errors.  $L$ -silver and  $I$ -silver mean the silver standards of literal and intended meanings, respectively. L2-predicted refers to the predicted semantic graphs produced by neural parsers on L2 sentences.
In order to figure out to what extent grammatical errors influence the good part and hence the whole sentence structure, we draw curves concerning distance from errors, which is displayed in Figure 6. The red line compares the two kinds of silver representations, which indicates the deviation from the intended meaning due to ungrammatical expressions. It appears as a smooth curve which goes steadily up. The overall trend indicates that the damage to farther parts from errors is less extensive. We assume that the propagation process is limited by the syntactic architecture. However, the situation of automatically predicted graphs by neural models is slightly different. It is depicted by the blue line in Figure 6 and the gradient is much smaller. We suggest it results from the great power of neural models to encode contextual information. In the L2 circumstance, while such characteristic enables the encoder to capture long-distance dependencies, it also expands the scope of errors' influence.
In this paper, we formulate the ESL semantic parsing task based on the divergence on literal and intended meanings. We establish parallel meaning representations by combining the complementary strengths of knowledge-intensive ERG-licensed analysis and dependency tree annotations through a new reranking model. For literal meaning, we probe the semantic parsing of multiple state-of-the-art neural parsers and give detailed analysis of effects from grammatical errors. For intended meaning, we investigate how grammatical errors affect the understanding of sentences as well as how grammatical error correction (GEC) can contribute to the parsing. Results reveal three facts: 1) semantic parsing is sensitive to non-canonical expressions, and the parsing performance varies with regard to the distribution as well as types of grammatical errors; 2) Factorization-based parser is the most promising parser to process learner English; and 3) GEC has a positive, but limited influence on the parsing of intended meaning.
This paper shows a pilot study on the semantic parsing for learner language. Future research may involve tailoring existing parsers to learner data, combining literal and intended meanings in a unified framework, evaluating GEC models in terms of speakers' intention and parsing for other languages.
This work is supported in part by the National Hi-Tech R&D Program of China (No. 2018YFB1005100). We thank the anonymous reviewers and the area chair for their useful feedback and suggestions, Yufei Chen and Yajie Ye for providing the parsers and Ben Roberts for proofreading. Weiwei Sun is the corresponding author.
6791
Omri Abend and Ari Rappoport. 2013. Universal conceptual cognitive annotation (ucca). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 228-238.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, pages 178-186.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010. A theory of learning from different domains. Machine learning, 79(1-2):151-175.
Emily M Bender, Dan Flickinger, Stephan Oepen, Woodley Packard, and Ann Copestake. 2015. Layers of interpretation: On grammar and compositionality. In Proceedings of the 11th international conference on Computational Semantics, pages 239-249.
Yevgeni Berzak, Jessica Kenney, Carolyn Spadine, Jing Xian Wang, Lucia Lam, Keiko Sophie Mori, Sebastian Garza, and Boris Katz. 2016a. Universal dependencies for learner english. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 737-746, Berlin, Germany. Association for Computational Linguistics.
Yevgeni Berzak, Jessica Kenney, Carolyn Spadine, Jing Xian Wang, Lucia Lam, Keiko Sophie Mori, Sebastian Garza, and Boris Katz. 2016b. Universal dependencies for learner english. arXiv preprint arXiv:1605.04278.
John Blitzer and Fernando Pereira. 2007. Domain adaptation of natural language processing systems. University of Pennsylvania, pages 1-106.
Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 748-752.
Yufei Chen, Sheng Huang, Fang Wang, Junjie Cao, Weiwei Sun, and Xiaojun Wan. 2018a. Neural maximum subgraph parsing for cross-domain semantic dependency analysis. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 562-572.
Yufei Chen, Weiwei Sun, and Xiaojun Wan. 2018b. Accurate shrg-based semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 408-418. Association for Computational Linguistics.
Yufei Chen, Yajie Ye, and Weiwei Sun. 2019. Peking at mrp 2019: Factorization-and composition-based parsing for elementary dependency structures. In Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning, pages 166-176.
Shamil Chollampatt and Hwee Tou Ng. 2018. A multilayer convolutional encoder-decoder neural network for grammatical error correction. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence.
Noam Chomsky. 2014. Aspects of the Theory of Syntax, volume 11. MIT press.
David Crystal. 2012. English as a global language. Cambridge university press.
Ana Diaz-Negrillo, Detmar Meurers, Salvador Valera, and Holger Wunsch. 2010. Towards interlanguage pos annotation for effective learner corpora in sola and flt. In Language Forum, volume 36, pages 139-154.
Markus Dickinson and Marwa Ragheb. 2009. Dependency annotation for learner corpora. In Proceedings of the Eighth Workshop on Treebanks and Linguistic Theories (TLT-8), pages 59-70.
Timothy Dozat and Christopher D. Manning. 2018. Simpler but more accurate semantic dependency parsing. page fromto484490.
Rebecca Dridan and Stephan Oepen. 2011. Parser evaluation using elementary dependency matching. In Proceedings of the 12th International Conference on Parsing Technologies, pages 225-230.
Hady Elsahar and Matthias Galle. 2019. To annotate or not? predicting performance drop under domain shift. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2163-2173.
Dan Flickinger. 1999. The english resource grammar.
Dan Flickinger. 2000. On building a more efficient grammar by exploiting types. *Natural Language Engineering*, 6(1):15-28.
Dan Flickinger. 2011. Accuracy vs. robustness in grammar engineering. Language from a cognitive perspective: Grammar, usage, and processing, 201:31-50.
Dan Flickinger, Emily M Bender, and Stephan Oopen. 2014. Towards an encyclopedia of compositional semantics: Documenting the interface of the english resource grammar. In LREC, pages 875-881.
6792
Dan Flickinger, Emily M. Bender, and Woodley Packard. 2016. English resource semantics. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorial Abstracts, pages 1-5, San Diego, California. Association for Computational Linguistics.
Dan Flickinger, Yi Zhang, and Valia Kordoni. 2012. Deepbank: a dynamically annotated treebank of the wall street journal. In Proceedings of the 11th International Workshop on Treebanks and Linguistic Theories, pages 85-96.
Susan M Gass. 2013. Second language acquisition: An introductory course. Routledge.
Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2017. Deep semantic role labeling: What works and what's next. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 473-483.
Karl Moritz Hermann and Phil Blunsom. 2013. The role of syntax in vector space models of compositional semantics. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 894-904.
Hagen Hirschmann, Seanna Doolittle, and Anke Lüdeling. 2007. Syntactic annotation of non-canonical linguistic structures.
Alexander Koller, Stephan Oepen, and Weiwei Sun. 2019. Graph-based meaning representations: Design and processing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 6-11, Florence, Italy. Association for Computational Linguistics.
Irene Langkilde and Kevin Knight. 1998. Generation that exploits corpus-based statistical knowledge. In Proceedings of the 17th international conference on Computational linguistics-Volume 1, pages 704-710. Association for Computational Linguistics.
Zi Lin, Yuguang Duan, Yuanyuan Zhao, Weiwei Sun, and Xiaojun Wan. 2018. Semantic role labeling for learner chinese: the importance of syntactic parsing and 12-11 parallel data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3793-3802.
Matthias Lindemann, Jonas Groschwitz, and Alexander Koller. 2019. Compositional semantic parsing across graphbanks.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank.
Ryo Nagata and Keisuke Sakaguchi. 2016. Phrase structure annotation and parsing for learner english. In Proceedings of the 54th Annual Meeting of the
Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1837-1847.
Ryo Nagata, Edward Whittaker, and Vera Sheinman. 2011. Creating a manually error-tagged and shallow-parsed learner corpus. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 1210-1219. Association for Computational Linguistics.
Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The conll-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 1-14.
Joakim Nivre, Marie-Catherine De Marneffé, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1659-1666.
Stephan Oepen and Jan Tore Lønning. 2006. Discriminant-based mrs banking. In LREC, pages 1250-1255.
Carl Pollard and Ivan A. Sag. 1994. Head-Driven Phrase Structure Grammar. The University of Chicago Press, Chicago.
Feng Qian, Lei Sha, Baobao Chang, LuChen Liu, and Ming Zhang. 2017. Syntax aware lstm model for semantic role labeling. In Proceedings of the 2nd Workshop on Structured Prediction for Natural Language Processing, pages 27-32.
Marwa Ragheb and Markus Dickinson. 2012. Defining syntax for learner language annotation. In Proceedings of COLING 2012: Posters, pages 965-974, Mumbai, India.
Stefano Rastelli. 2013. Learner corpora without error tagging. Linguistik online, 38(2).
Alexandr Rosen, Jirka Hana, Barbora Štindlova, and Anna Feldman. 2014. Evaluating and automating the annotation of a learner corpus. Language Resources and Evaluation, 48(1):65-92.
Alla Rozovskaya and Dan Roth. 2010. Annotating esl errors: Challenges and rewards. In Proceedings of the NAACL HLT 2010 fifth workshop on innovative use of NLP for building educational applications, pages 28-36. Association for Computational Linguistics.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80.
6793
Antonella Sorace. 2006. Gradedness and optionality in mature and developing grammars. Gradience in grammar: Generative perspectives, pages 106-123.
Antonella Sorace. 2011. Pinning down the concept of aAIjinterfaceaAI in bilingualism. Linguistic approaches to bilingualism, 1(1):1-33.
Lydia White. 2011. Second language acquisition at the interfaces. *Lingua*, 121(4):577-590.
Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019. Amr parsing as sequence-to-graph transduction.
Wei Zhao, Liang Wang, Kewei Shen, Ruoyu Jia, and Jingming Liu. 2019. Improving grammatical error correction via pre-training a copy-augmented architecture with unlabeled data. arXiv preprint arXiv:1903.00138.
6794