Question: What kind of annotation scheme do they describe?
Evidence: The majority of the English text available worldwide is generated by non-native speakers BIBREF0 . Such texts introduce a variety of challenges, most notably grammatical errors, and are of paramount importance for the scientific study of language acquisition as well as for NLP. Despite the ubiquity of non-native English, there is currently no publicly available syntactic treebank for English as a Second Language (ESL).
To address this shortcoming, we present the Treebank of Learner English (TLE), a first of its kind resource for non-native English, containing 5,124 sentences manually annotated with POS tags and dependency trees. The TLE sentences are drawn from the FCE dataset BIBREF1 , and authored by English learners from 10 different native language backgrounds. The treebank uses the Universal Dependencies (UD) formalism BIBREF2 , BIBREF3 , which provides a unified annotation framework across different languages and is geared towards multilingual NLP BIBREF4 . This characteristic allows our treebank to support computational analysis of ESL using not only English based but also multilingual approaches which seek to relate ESL phenomena to native language syntax.
While the annotation inventory and guidelines are defined by the English UD formalism, we build on previous work in learner language analysis BIBREF5 , BIBREF6 to formulate an additional set of annotation conventions aiming at a uniform treatment of ungrammatical learner language. Our annotation scheme uses a two-layer analysis, whereby a distinct syntactic annotation is provided for the original and the corrected version of each sentence. This approach is enabled by a pre-existing error annotation of the FCE BIBREF7 which is used to generate an error corrected variant of the dataset. Our inter-annotator agreement results provide evidence for the ability of the annotation scheme to support consistent annotation of ungrammatical structures.
Finally, a corpus that is annotated with both grammatical errors and syntactic dependencies paves the way for empirical investigation of the relation between grammaticality and syntax. Understanding this relation is vital for improving tagging and parsing performance on learner language BIBREF8 , syntax based grammatical error correction BIBREF9 , BIBREF10 , and many other fundamental challenges in NLP. In this work, we take the first step in this direction by benchmarking tagging and parsing accuracy on our dataset under different training regimes, and obtaining several estimates for the impact of grammatical errors on these tasks.
To summarize, this paper presents three contributions. First, we introduce the first large scale syntactic treebank for ESL, manually annotated with POS tags and universal dependencies. Second, we describe a linguistically motivated annotation scheme for ungrammatical learner English and provide empirical support for its consistency via inter-annotator agreement analysis. Third, we benchmark a state of the art parser on our dataset and estimate the influence of grammatical errors on the accuracy of automatic POS tagging and dependency parsing.
The remainder of this paper is structured as follows. We start by presenting an overview of the treebank in section SECREF2 . In sections SECREF3 and SECREF4 we provide background information on the annotation project, and review the main annotation stages leading to the current form of the dataset. The ESL annotation guidelines are summarized in section SECREF5 . Inter-annotator agreement analysis is presented in section SECREF6 , followed by parsing experiments in section SECREF7 . Finally, we review related work in section SECREF8 and present the conclusion in section SECREF9 .
Second, we describe a linguistically motivated annotation scheme for ungrammatical learner English and provide empirical support for its consistency via inter-annotator agreement analysis.
Answer: A linguistically motivated annotation scheme for ungrammatical learner English.