SlowGuess/NAACL_2021_OCR huggingface dataset:
Beyond the grammatical errors and their types, the annotator had access to more information about the errors, such as the context surrounding the erroneous utterance and the Extensible Markup Language (XML) data extracted from the FCE dataset. These two features proved useful to elucidate semantic errors. A semantic error initially looks like an annotation error, since the utterance's grammatical structure is not problematic. However, when the annotator checked the context around the error, they would often find its cause to be context-related. In the sentence "I have never been to in my life," the word "never" does not seem incorrect, although it is tagged as such. By looking at the context surrounding this error, "It was the worst show and theatre I have never been to in my life," it is possible to see that indeed the word "never" should be replaced with the word "ever".
3.3 Ambiguous cases
During the annotation procedure, ambiguous cases were discussed and reviewed among the annotator and the research group in weekly meetings. The annotator highlighted entries that she found hard to label and those were discussed within the group. Such cases ranged from entries that were deemed erroneous by the FCE annotators due to language variety (e.g., British or American idioms), entries that did not have an equivalent structure in the learners' L1 (e.g., hyphenated words, which do not exist in Chinese), and semantic errors (e.g., errors in
Incorrect utterance	Correct utterance	Ambiguity	Number of cases
British vs American English varieties	We all would like to go there.	We would all like to go there.	The incorrect version of the sentence is more commonly used in the American variety of English. It is not incorrect in that variety.	18
Chinese does not have an equivalent structure	I'm standing on your left hand side.	I'm standing on your left-hand side.	Hyphens do not have a parallel structure in Chinese.	17
Semantic errors tagged as structural errors	You could find a restaurant.	You can find a restaurant.	Although the verb “could” is in the past tense, some learners may choose to use it to indicate respect.	10
Table 5: Ambiguous errors from the FCE dataset
which the grammatical structure is not incorrect, but the utterance does not fit the overall essay context). Table 5 presents examples of errors that were discussed during the annotation process. These errors are considered ambiguous with regards to whether they should be labelled as transfer related.
4 Annotation scheme
The annotation scheme was designed to highlight the relationship between the learner error and the learner's L1. Other than the boolean label representing whether an error is related to negative language transfer, each entry carries information about the possible reason behind that learner mistake. Even when the relationship between the error and the learner's L1 is not apparent, the annotation scheme will provide a possible cause for the error. This cause is not related to language transfer.
The error cause feature was heavily influenced by language teacher guides, books that aim to make teachers aware of the learner errors they can encounter in the classroom, e.g., "Learner English: A Teacher's Guide to Interference and other Problems" by Swan and Smith (2001). Guides like these have been written based on years of in-classroom experience and contain information about error causes along with potential learner feedback. These guides were used as a baseline for negative language transfer detection during the annotation process. Other important sources of guidance for the error cause feature annotation were Chinese and English grammar books and guides<sup>3</sup> (Faigley, 2015;
Li and Thompson, 1989). These sources allowed the direct contrast of erroneous utterances with language rules and this contrast enabled the derivation of possible causes for learner mistakes.
5 Dataset application
Second language acquisition researchers and language teachers are well acquainted with learner errors that are related to the learners' L1s. These communities have produced comprehensive guides to learner errors and their causes. Learner language guides allow learners and teachers to identify the reasons behind certain error types and, with that, better understand and prevent those mistakes. In some of these guides the reader will find information that connects learners' L1s with common error types committed by native speakers of that language. Our new dataset, enables the use of other indicators, such as linguistic features and proficiency levels, to identify errors related to negative language transfer.
To understand the effect of linguistic features in negative language transfer prediction, we built classification models to predict when a learner error is related to negative language transfer. We wanted to explore the relationship between negative language transfer and the linguistic features of errors, such as part-of-speech (POS) tags and dependency labels, since these features are made available by this new dataset.
5.1 Negative language transfer classification
In this experiment, we used the new negative language transfer dataset to compare the predictive
3https://www.grammarly.com/blog/ category/handbook/
Incorrect utterance	Error length	Error type	POS tags
This are only my immature views.	1	AGA (anaphoric pronoun agreement error)	DT VBP RB
power of classification models for negative transfer $^{4}$ . These models are trained on error features from the new negative language transfer dataset. The models output whether the errors are related to negative transfer. This is a binary classification problem in which most of the features available are categorical. For this reason, we converted the categorical features, such as error types, into one-hot-encoding columns and binary vectors. The one-hot-encoding conversion creates one new column on the dataset for each unique categorical value. The binary vector conversion creates one new column on the dataset containing a binary number in which the position of the digit one corresponds to the category of the entry.
The conversion of categorical features into one-hot-encoding columns increased the number of dimensions in our data. Hence, we decided to experiment with a random forest classifier, a classification model that is known to perform well with high-dimensional data (Xu et al., 2012). For our baseline model we decided to use a logistic regression model trained only on the error type data features. This choice was based on the parallel that can be drawn between the dataset's error type information and the teacher guide descriptions of connections between L1s and specific error patterns. A strong baseline for the experiment relies solely on error types to predict negative language transfer. Both classifiers were trained using the models available in the Python library scikit-learn<sup>5</sup>.
Since the new dataset contains actual learner writing, it is possible to extract a wide range of linguistic features from the sentences in the dataset. We used the Python library spaCy<sup>6</sup> to extract dependency labels, Universal Dependencies POS tags (Nivre et al., 2016), and Penn Treebank POS tags (Marcus et al., 1993) from the erroneous utterances and their surrounding tokens. These features were then converted to one-hot-encoding columns and binary vectors, as described above.
Given the wide range of features in the new
Table 6: Best performing feature set for the random forest classifier
Acc	P	R
Logistic regression baseline	0.72	0.79	0.73
Random forest model	0.78	0.82	0.79
Table 7: Accuracy, precision, and recall results on the held-out test set for the baseline logistic regression model trained with the error type information and the random forest classifier trained with the best performing feature set
dataset, we performed an initial step of feature selection to determine the most relevant features to predict the negative language transfer label. The feature selection process consisted in performing 10-fold cross validation with  $90\%$  of the dataset as training data. The remaining  $10\%$  was held out for testing. We performed the cross validation on all feature set combinations training a random forest classifier on nine folds and testing it on the remaining one. The mean score for each feature set combination was used to select the best performing set. The best performing model in the feature selection process was trained with three features: the error length (the number of words in the error), the error type (described in Nicholls (2003)), and the Penn Treebank POS tags of the erroneous utterance plus the POS tags of the error's two subsequent words. Table 6 presents an example of the features selected. The columns "Error type" and "POS tags trigram" were converted into one-hot-encoding columns during the feature selection, training, and testing processes but are presented here as categorical data for intelligibility.
After feature selection, a random forest classifier was trained on the three best performing features using  $90\%$  of the dataset, i.e., 2952 error instances. This model accurately classified  $78.04\%$  of the test samples as negative language transfer or not. The baseline model, a logistic regression model trained on the error type features, achieved  $72.56\%$  accuracy on the test set. Table 7 presents the accuracy, precision, and recall scores yielded by both baseline and random forest models on the test set, which contained 328 error instances.
Analyzing the models' outputs, it becomes clear
4The experimental code is available in our research group's repository (see footnote 2).
that more information about learners' incorrect utterances captures more of the language transfer phenomenon. One of the most common baseline model misclassifications was the prediction of "replace punctuation" errors as negative language transfer when they were not negative transfer related. The model misclassified  $30\%$  of the "replace punctuation" errors. Although Chinese learners are known to replace periods with commas incorrectly (Liu, 2011), the error category by itself is not enough to make an accurate classification. In contrast, the random forest classifier mislabelled  $16\%$  of the "replace punctuation" errors. The random forest approach misclassified entries as negative transfer and not negative transfer related, demonstrating that this approach does not simply associate one error type with one output label. Another error category in which the random forest classifier outperformed the baseline model was when classifying the "wrong verb tense" errors. The error type features do not provide information about the verb tense that was used incorrectly, but the POS tags extracted from the incorrect utterance do. This extra information helps the random forest classifier make more accurate predictions about negative language transfer related errors.
These results suggest that negative language transfer classification can benefit from features other than the error type. Furthermore, it shows that linguistic features are important in the identification of negative language transfer errors.
Our dataset is the first we are aware of that annotates a large amount of learner English data with negative language transfer features and error causes. It has the potential to improve the performance of computational linguistics tasks, such as native language transfer identification and grammatical error correction. More importantly, its content can benefit English teachers and learners by making more personalized error feedback available. Another potential application of the dataset is in the automatic detection of negative language transfer. This application could help provide real-time L1-informed feedback to English learners.
Our research group is currently working on annotating errors from other L1 learner groups. We are also expanding our annotation process to learner data from sources other than the FCE dataset, such as the Lang-8 English corpus (Mizumoto et al.,
2011). With that, we hope to broaden the scope of English learners supported by L1 informed error feedback.
The new datasets presented in this paper are built on top of the FCE dataset described in Yannakoudakis et al. (2011). The FCE dataset contains anonymised essays from the First Certificate in English test-takers between the years of 2000 and 2001. The essays' meta-data contains information about the age range and native language of the learners. Although the original dataset description does not address how the learners' consent was obtained, Cambridge Assessment should be governed by the same consent procedures as other UK researchers. The candidate privacy policy<sup>7</sup> from the Cambridge Assessment website states that the learners' data could be used in "developing and delivering publications and other resources that support learning".
The annotation procedure described in this paper was performed by undergraduate and graduate students as part of individual project courses and research assistantships, respectively. All three authors<sup>8</sup> are fluent in at least one variety of English. Two of them also have deep familiarity with other English varieties. This knowledge was used to ensure that the negative language transfer annotations do not reinforce existing power structures around language varieties and standard forms of English. That said, cases of linguistic imperialism are likely to remain in the dataset.
Another facet that may limit these datasets' applicability is the fact that the FCE annotated essays were collected 20 years ago. As both the Chinese and English languages have evolved and, possibly intersected over time, the occurrence of some negative language transfer errors may have decreased. For example, the prevalence of determiner omis
<sup>7</sup>https://www.cambridgeenglish.org/ch/fr/footer/data-protection/candidates/
$^{8}$ Demmans Epp has specialized in computer-assisted language learning and she has taught English as a second or foreign language in a variety of educational contexts. She has training in several areas of linguistics that include language acquisition and sociolinguistics. She is a first language speaker of Canadian English and has experience working in American and British English contexts.
Wanderley speaks Portuguese as her first language and is proficient in English. She has experience working in British and Canadian English contexts.
Zhao grew up speaking Mandarin Chinese and is familiar with various dialects of Chinese. She received her education in American and Canadian English and is familiar with British English to a certain extent.
sion errors in Chinese L1 English communication has been targeted from an instructional perspective in recent years which may have lowered the occurrence rate of this negative language transfer error. On the English language front, the usage of the pronoun “they” has changed and may have rendered some of the entries in our dataset obsolete.
The new dataset's main purpose is to aid English language learning by providing personalized error causes according to the learner's L1. It aims to help English as a Second Language learners acquire a better understanding of the English language by contrasting it to the learner's L1. Although this type of information tends to be helpful to language learners, there might be learners who do not benefit from it. The data available in the dataset was reviewed by our research group to ensure clarity and correctness. We do not foresee additional risks stemming from the usage of the new dataset.
We acknowledge the financial support of the Natural Sciences and Engineering Research Council of Canada (NSERC), [RGPIN-2018-03834], and the Social Sciences and Humanities Research Council (SSHRC). We would also like to acknowledge the anonymous reviewers for their insightful and valuable feedback.
Yevgeni Berzak, Jessica Kenney, Carolyn Spadine, Jing Xian Wang, Lucia Lam, Keiko Sophie Mori, Sebastian Garza, and Boris Katz. 2016. Universal Dependencies for learner English. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 737-746, Berlin, Germany. Association for Computational Linguistics.
Adriane Boyd, Jirka Hana, Lionel Nicolas, Detmar Meurers, Katrin Wisniewski, Andrea Abel, Karin Schöne, Barbora Štindlova, and Chiara Vettori. 2014. The MERLIN corpus: Learner language and the CEFR. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 1281-1288, Reykjavik, Iceland. European Language Resources Association (ELRA).
Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019. The BEA-2019 shared task on grammatical error correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52-75, Florence, Italy. Association for Computational Linguistics.
Christopher Jack Bryant. 2019. Automatic annotation of error types for grammatical error correction (Doctoral thesis). Ph.D. thesis, University of Cambridge.
Tonje M Caine. 2008. Do you speak global?: The spread of English and the implications for English language teaching. Canadian Journal for New Scholars in Education/Revue canadienne des jeuneschescherchees etchercheurs en education, 1(1).
Shamil Chollampatt, Duc Tam Hoang, and Hwee Tou Ng. 2016. Adapting grammatical error correction based on the native language of writers with neural network joint models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1901-1911, Austin, Texas. Association for Computational Linguistics.
Carrie Demmans Epp and Gordon McCalla. 2011. *Protutor: Historic open learner models for pronunciation tutoring*. In *Artificial Intelligence in Education*, pages 441-443, Berlin, Heidelberg. Springer Berlin Heidelberg.
L. Faigley. 2015. *Little Penguin Handbook*. Pearson Australia.
Brendan Flanagan, Chengjiu Yin, Takahiko Suzuki, and Sachio Hirokawa. 2015. Prediction of learner native language by writing error pattern. In Learning and Collaboration Technologies, pages 87-96, Cham. Springer International Publishing.
Nan Rae Han, Martin Chodorow, and Claudia Leacock. 2006. Detecting errors in English article usage by non-native speakers. *Natural Language Engineering*, 12.
Zhao Hong Han. 2001. Fine-tuning corrective feedback. Foreign Language Annals, 34.
Institute of International Education. 2020. International student totals by place of origin, 2000/01-2019/20.
Khaled Karim and Hossein Nassaji. 2020. The revision and transfer effects of direct and indirect comprehensive corrective feedback on ESL students' writing. Language Teaching Research, 24(4):519-539.
Irit Kupferberg. 1999. The cognitive turn of contrastive analysis: Empirical evidence. Language Awareness, 8.
Robert Lado. 1957. Linguistics Across Cultures: Applied Linguistics for Language Teachers. University of Michigan Press.
C.N. Li and S.A. Thompson. 1989. Mandarin Chinese: A Functional Reference Grammar. Number v. 3 in Linguistics: Asian studies. University of California Press.
Amna Liaqat, Cosmin Munteanu, and Carrie Demmans Epp. 2020. Collaborating with Mature English Language Learners to Combine Peer and Automated Feedback: a User-Centered Approach to Designing Writing Support. International Journal of Artificial Intelligence in Education, pages 1-42.
Xing Liu. 2011. The not-so-humble "Chinese comma": Improving English CFL students' understanding of multi-clause sentences. In Proceedings of the 9th New York International Conference on Teaching Chinese.
Photis Lysandrou and Yvonne Lysandrou. 2003. Global English and preregression: understanding English language spread in the contemporary era. Economy and Society, 32(2):207-233.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330.
Tomoya Mizumoto, Mamoru Komachi, Masaaki Nagata, and Yuji Matsumoto. 2011. Mining revision log of language learning SNS for automated Japanese error correction of second language learners. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 147-155, Chiang Mai, Thailand. Asian Federation of Natural Language Processing.
Maria Nadejde and Joel Tetreault. 2019. Personalizing grammatical error correction: Adaptation to proficiency level and L1. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 27-33, Hong Kong, China. Association for Computational Linguistics.
Ryo Nagata. 2019. Toward a task of feedback comment generation for writing learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3206-3215, Hong Kong, China. Association for Computational Linguistics.
Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 1-14, Baltimore, Maryland. Association for Computational Linguistics.
Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013. The CoNLL-2013 shared task on grammatical error correction. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task, pages 1–12, Sofia, Bulgaria. Association for Computational Linguistics.
Diane Nicholls. 2003. The Cambridge Learner Corpus: Error coding and analysis for lexicography and ELT. In Proceedings of the Corpus Linguistics 2003 conference, volume 16, pages 572-581.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1659-1666, Portoroz, Slovenia. European Language Resources Association (ELRA).
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR - grammatical error correction: Tag, not rewrite. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 163-170, Seattle, WA, USA  $\hat{a}^{\dagger}$  Online. Association for Computational Linguistics.
Zahra Rahimi, Diane Litman, Richard Correnti, Elaine Wang, and Lindsay Clare Matsumura. 2017. Assessing students' use of evidence and organization in response-to-text writing: Using natural language processing for rubric-based automated scoring. International Journal of Artificial Intelligence in Education, 27(4):694-728.
Alla Rozovskaya and Dan Roth. 2011. Algorithm selection and model adaptation for ESL correction tasks. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 924-933, Portland, Oregon, USA. Association for Computational Linguistics.
Younghee Sheen. 2007. The Effect of Focused Written Corrective Feedback and Language Aptitude on ESL Learners' Acquisition of Articles. TESOL Quarterly, 41(2):255-283.
Natsuko Shintani and Rod Ellis. 2013. The comparative effect of direct written corrective feedback and metalinguistic explanation on learners' explicit and implicit knowledge of the English indefinite article. Journal of Second Language Writing, 22(3):286-306.
Karen L. Smith. 1982. Avoidance, overuse, and misuse: Three trial and error learning strategies of second language learners. Hispania, 65.
Michael Swan and Bernard Smith. 2001. *Learner English: A Teacher's Guide to Interference and Other Problems*, 2 edition. Cambridge Handbooks for Language Teachers. Cambridge University Press.
Joel Tetreault, Daniel Blanchard, and Aoife Cahill. 2013. A report on the first native language identification shared task. In Proceedings of the Eighth