SlowGuess/NAACL_2022_OCR huggingface dataset:
Xun Yuan\*2, Derek Pham\*1, Sam Davidson3 and Zhou Yu
$^{1}$ Columbia University,  $^{2}$ Zhejiang University,  $^{3}$ University of California, Davis
{dp3081,zy2461}@columbia.edu
yuanxun@zju.edu.cn, ssdavidson@ucdavis.edu
Currently available grammatical error correction (GEC) datasets are compiled using essays or other long-form text written by language learners, limiting the applicability of these datasets to other domains such as informal writing and conversational dialog. In this paper, we present a novel GEC dataset consisting of parallel original and corrected utterances drawn from open-domain chatbot conversations; this dataset is, to our knowledge, the first GEC dataset targeted to a human-machine conversational setting. We also present a detailed annotation scheme which ranks errors by perceived impact on comprehension, making our dataset more representative of real-world language learning applications. To demonstrate the utility of the dataset, we use our annotated data to fine-tune a state-of-the-art GEC model. Experimental results show the effectiveness of our data in improving GEC model performance in a conversational scenario.
In recent years, both researchers and businesses have attempted to build effective educational chatbots to help language learners improve their conversational skills in a second language (primarily English) (Huang et al., 2021). However, many such systems, such as GenieTutor Plus (Huang et al., 2017), use rule-based dialog engines, and thus do not take advantage of recent developments in dialog generation using Transformer models, which have vastly improved the quality of modern chatbots (Liang et al., 2020). Extant dialog systems for conversational language learning can be broadly classified into two types. In the first type, the chatbot serves as a teacher and repeatedly asks the user questions to test acquisition of specific words, syntax, and other pedagogical targets.
In the second type, the chatbot serves as a conversational partner, encouraging users to chat with it and, in some cases, providing corrective feedback to learners (Fryer et al., 2020). It is this latter type we hope to improve using our proposed dataset.
Grammatical error correction (GEC) models are needed to generate appropriate corrective feedback for this second type of educational chatbot. However, nearly all current GEC datasets focus on written essays, a domain which differs markedly from conversational speech in both syntax and style. As a result, datasets drawn from written sources, such as student essays, produce poor results when applied to dialog (Davidson et al., 2019). There currently exists one dataset of error-annotated conversational utterances by English second language learners on which researchers can train and evaluate conversational GEC models, the Teacher-Student Chatroom Corpus (Caines et al., 2020); this data is generated in the context of human-human interaction, specifically interactions between a teacher and a second language learner student. However, no similar conversational dataset focuses on the human-machine conversational setting. In this work we seek to address this gap in the available data by developing a high-quality, error-annotated dataset of learner dialog collected from an online educational chatbot. To appropriately annotate our data for language learning applications, we introduce a 3-level grammatical error classification structure in order to categorize errors based on severity. Our motivation for this error classification structure is to give users the opportunity to first focus on improving their most serious grammatical errors. To demonstrate the utility of the proposed dataset, we fine-tune and evaluate a state-of-the-art GEC model using our newly developed dataset.
$^{1}$ Data is available at https://github.com/yuanxun-yx/eracond
Human Language Technologies, pages 76 - 84
As with many NLP tasks, the current state-of-the-art in grammatical error correction involves using large Transformer-based language models such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and XLNet (Yang et al., 2019). To evaluate the utility of our dataset, we use Omelianchuk et al. (2020)'s GECToR model, which reframes GEC as a sequence labelling task rather than a monolingual machine translation task. GECToR achieves SoTA results on the test corpus used for the BEA 2019 Shared Task on Grammatical Error Correction (Bryant et al., 2019). Other promising supervised GEC models include those of Stahlberg and Kumar (2021) and Rothe et al. (2021), who achieve SoTA results on the JFLEG (Napoles et al., 2017) and CoNLL-2014 (Ng et al., 2014) GEC datasets, respectively. Both models combine innovative synthetic data generation methods with large pretrained transformer language models.
Recent work related to the development of datasets for grammatical error correction include Naples et al. (2019) who presents a dataset of native and non-native English writing. Trinh and Rozovskaya (2021) proposes a new parallel dataset of Russian student writing. These datasets add to the growing number of GEC datasets available to the research community. However, as previously mentioned, no GEC dataset that contains chatbot-based human-machine conversational data, in English or any other language, is currently available. We seek to begin closing this gap with the present research.
3.1 Data Collection Process
We collected 186 dialogs containing 1735 user utterance turns of open-domain dialog data by deploying BlenderBot (Roller et al., 2020) on Amazon Mechanical Turk (AMT) via LEGOEval. (Li et al., 2021). We decided to deploy BlenderBot, because it is open-sourced and because it has relatively good coherence, and is known for its engagement and human-like conversational qualities.
The AMT crowd-workers who conversed with our bot are L2 English speakers of at least intermediate proficiency. The workers were asked to converse with our chatbot for at least 10 turns
(a turn is defined as a bot/user utterance pair) either about movies or the COVID-19 pandemic; we chose these because of their universal experience and subjectivity of the two topics, resulting in a rich and diverse set of utterances in the dataset. Workers interacted with the bot using a typed interface, similar to a messaging app. We plan to expand this to an ASR-driven system as well as to additional conversational topics in future work.
After collecting open-domain dialog data, we manually revised each user utterance to correct any non-standard or ungrammatical English usage. A subset of the dialogs are corrected by two annotators to provide multiple corrected targets for system evaluation—the remaining dialogs are corrected by a single annotator. Both annotators are graduate student native speakers of English.
3.2.1 Annotation scheme
We followed an annotation method similar to that proposed in Náplava et al. (2022), in which we asked annotators to revise any sentences containing ungrammatical elements. Our goal was to apply the minimum number of edits needed to make the utterance conform to standard written English while remaining as faithful to the source as possible. With this goal in mind, we designed our annotation scheme to conform to the rules of standard written English with two exceptions: internet shorthand and slang, and short responses which are incomplete sentences; both forms, while not acceptable in formal written English, are frequently considered acceptable in the context of informal dialog. We also made fluency edits (Napoles et al., 2017) of semantic and sentence construction errors, particularly those related to lexical choice, omission, and word order. For example, the source line "The movie tell about a poor girl that meet a prince and in love for him", suffers from non-native-like word choice. We corrected this utterance to "the movie tells about a poor girl that meets a prince and falls in love with him". We made these corrections with the intention of creating ground truth utterances which are as semantically and syntactically similar to the source as possible.
3.2.2 Inter-annotator agreement
We took several steps to ensure that annotators were meeting the goal of revising ungrammatical
and disfluent sentences while retaining the semantic content of the source dialog. During our annotation training phase, we had both annotators correct an identical set of 26 dialogs. The annotators then reviewed each other's annotations on this subset and noted specific area of disagreement, which were then discussed with the lead researchers who provided specific instructions on how to resolve these discrepancies. Annotators then repeated the annotations of the same 26 dialogs to ensure that the provided instructions were being followed. During this process, we also asked annotators to note any changes which they believed changed the underlying semantics of a given dialog, so that such changes could be eliminated in subsequent passes. At each stage, we calculated inter-annotator agreement (as described below), and only continued with the annotation of the remaining 160 dialogs once our second-pass agreement levels were on-par with previously reported GEC corpora such as Trinh and Rozovskaya (2021).
For many ungrammatical sentences, there are multiple acceptable ways to correct the error. As a result GEC annotations can be quite variable, and traditional methods of calculating inter-annotator agreement are not informative (Rozovskaya and Roth, 2019). We therefore utilized two metrics for calculating inter-annotator agreement. The first, originally proposed by Rozovskaya and Roth (2010), asks each annotator to review and correct the corrections of the other annotators, and then calculates the percent of sentences which are unchanged on this second pass; these figures are shown as "Judged correct" in Table 1. The second method, used in work such as Trinh and Rozovskaya (2021) calculates the  $\mathrm{F}_{0.5}$  by setting one annotator as reference and the other as hypothesis; these figures are also provided in Table 1.
Ref	Judged Correct	TP	FP	FN	Prec	Rec	F0.5
0	95.9%	59	109	57	0.351	0.509	0.374
1	96.2%	59	57	109	0.509	0.351	0.467
3.3 Error Types
One of our key goals in developing an error correction model using the proposed dataset is to en
Table 1: Annotator agreement by  $\mathrm{F}_{0.5}$  score. Only dialogs with two annotators are compared. The first column indicates which annotator is selected as reference. The "Judged Correct" column indicates second-pass agreement between annotators.
Level	Impact on Meaning	Error Types
1	Trivial	Punctuation (excl. apostrophe) & Casing
2	Moderate	Acronyms, Abbreviations, Non-English Internet Slang, & Apos-trophe
3	Significant	SV Agreement, Verb Form, Word Confusion, etc.
Table 2: Categorization of grammatical errors.
able users to focus on specific language skills on which they wish to improve. Since we are dealing with online chat conversations, our data is more casual than the more formal written data seen in previous GEC datasets. Moreover, because our data consists of human-machine conversations involving English language learners of intermediate level, users are assumed to know basic English grammar. Therefore, we wanted to give users the flexibility of choosing to limit feedback, such as only receiving feedback on major lexical and syntactic errors. Specifically, we want to avoid overwhelming users with an excessive number of proposed corrections, and to enable users to improve their conversational skills by first focusing on their most serious errors. Importantly, suggesting an excessive number of corrections could overwhelm a less proficient user or possibly irritate a more proficient participant, resulting in reduced user enjoyment and engagement (Koltovskaia, 2020). To that end, we organized our annotated corrections into a 3-level structure based on a perceived ranking of how errors impact the ability of interlocutors to understand what the user is saying, as shown in Table 2. As such, we focus primarily on lexical, syntactic and usage errors (Ferris, 2011; Touchie, 1986), while leaving mechanical errors to the lowest-priority category. Error priority tags are attached to each edit proposed by the annotators automatically in a post-processing step using a modified version of the ERRANT toolkit (Bryant et al., 2017).
For Level 1, our logic is that conversational partners are generally still able to understand a message when it is missing sentence-final punctuation or when a word is not properly capitalized. Because they are of at least intermediate English proficiency, participants can be assumed to know the underlying rules related to punctuation and capitalization; their errors result rather from inattentiveness (Sermsook et al., 2017) and the informal nature of the conversational genre (Cohen
Example	Message	Error
1	yes, johnny depp, and brad pitt	Punctuation & Casing
2	Ok, what are you talking about? Kkkkkk	Non-English Internet Slang
3	I also like SF movies. It makes me think differently.	Acronym
4	What's your fav movie right now?	Abbreviation
5	IT SEEMS DRAMATIC. ILL WATCH.	Apostrophe
6	She is not on the line now. Maybe its nighttime there.	Apostrophe
7	I'd say you could help Zhou Yu. He's either unable to create a non-broken hit or he's cheating, exploring low-wage workers. What do you think?	Word Confusion
8	It just don't work	SV Agreement
9	I have a friend from the US. We have a conversation and I don't know the word bangus in English. So it was hard for me to communicate with her.	Verb Form
and Robbins, 1976). Consider Ex.1 in Table 3: the syntactic structure of the sentence makes it clear that the user is listing names of actors despite the lack of capitalization and punctuation.
For Level 2, our logic is that interlocutors are likely able to understand a message despite usage of acronyms, abbreviations, non-English internet slang, or a missing apostrophe. An example of such non-English internet slang is shown in Ex. 2 in Table 3. The use of such forms in text-based online conversation is to be expected, since these types of abbreviations are common in all student writing (Purcell et al., 2013; Thangaraj and Maniam, 2015). However, such cases could potentially lead to misunderstanding, especially when conversing with someone of a different generation or linguistic background. Therefore, we categorize these non-standard forms as moderate "errors" (though they are not errors in the traditional sense). We do not consider these non-standard forms as significant because our assumption is that the writer intentionally chose to use these forms for brevity and in the spirit of informality common in online chat (Forsythand and Martell, 2007).
Finally, we include errors which are likely to result misunderstanding or misinterpretation of a message in Level 3. As we can see in Ex. 7 in Table 3, the user incorrectly uses the term nonbroken instead of unbroken, and exploring instead of exploiting. These lexical errors, particularly the latter, are likely to result in misinterpretation of the speaker's intended meaning. Similarly, the user makes a subject-verb agreement error in Ex. 8 and a verb tense error in Ex. 9. In the former, the user mistakenly uses a plural verb for a singular subject, while in the latter, the user uses a present tense verb when a past tense verb is needed. Because these errors relate to some of the most fundamental rules in English grammar, such errors must be addressed promptly. Thus, we treat these errors as "significant" in our annotation scheme.
Table 3: Examples user utterances with error type from ErAConD dataset.
Dialogs	186
User turns	1735
User sentences (source)	2454
Word tokens (source)	24616
Word types	2860
Error annotations	2346.5
Level 3 error annotations	684.5
# of turns per dialog	9.33
# of sentences per turn (source)	1.41
# of tokens per turn (source)	14.19
# of error annotations per turn	1.35
# of Level 3 error annotations per turn	0.39
# of Level 3 error annotations per 100 tokens	2.78
Table 4: Overview of ErAConD dataset.
Table 4 reports statistics related to the composition of the ErAConD dataset. All statistics are based on user turns; we omit turns generated by our dialog system, as these are not relevant to training a GEC system to provide feedback to users. Additionally, we exclude utterances which include only stop phrases (i.e. "stop", "goodbye", etc.) since these are intended to terminate the conversation. Our 3-level structure is reflected in our modified ERRANT (Bryant et al., 2019) toolkit and M2 format. Error type tags are generated from annotated parallel data automatically with our modified version of ERRANT $^2$ , and related figures are averaged across multiple annotators. Inspired by Rozovskaya and Roth (2021), our version of ERRANT also enables users to provide grammatically equivalent edits (i.e. changing "I'm" to "I am"), so that ERRANT can recognize them as identical edits.
As shown in Table 4, Level 3 edits account for  $29.17\%$  of all errors, which supports the necessity of our proposed categorization feature. The error distribution in our dataset is comparable to that of essay-based GEC datasets, according to statis
$^{2}$ Code is available at https://github.com/yuanxun-yx/errant
tics provided in Bryant et al. (2019), with the exception of spelling and morphological (inflection) errors, which are substantially higher. While the higher rate of spelling errors is unsurprising in a conversation dataset, the difference in morphological errors warrants further investigation.
5 Grammar Error Correction Model
To demonstrate the utility of our proposed dataset in improving GEC for the open-domain dialog setting, we use the ErAConD dataset to fine-tune a state-of-the-art GEC model, GECToR (Omelianchuk et al., 2020), which we then test on held-out dialog data. Our results show that ErA-ConD is useful for adapting GEC models to open-domain dialog.
5.1 Training process
To train a model catered to conversations, we fine-tune the GECToR model<sup>3</sup> proposed by Omelianchuk et al. (2020) on our collected data. The GECToR model is a grammatical error correction model that generates a set of encoded edit operations to correct the input text rather than directly outputting corrected text. In other words, rather than outputting superficially corrected text, the model outputs the operations necessary to convert the uncorrected text to its corrected version. This set of encoded edit operations can then be applied to the original uncorrected text in a post-processing step to generate the final corrected output (Omelianchuk et al., 2020). The GECToR model training pipeline starts with a large pretrained language model (i.e. XLNet or RoBERTa) that is then fine-tuned on both synthetic and collected data. To test our proposed dataset, we further fine-tune this GECToR model on our data.
We only choose to fine-tune the GECToR model using Level 3 edits in our dataset and ignore the Level 1 and 2 edits so that our model can perform better in real-world pedagogical settings. As previously mentioned, overwhelming students with trivial errors, such as punctuation and capitalization, can decrease user enjoyment and engagement (Koltovskaia, 2020). In future work, we plan to train the GECToR model on targeted conversational data across all stages of the pipeline, and determine which errors to present to the user in a post-processing step. We also plan to integrate
conversational context.
5.2 Result and Analysis
Setting	TP	FP	FN	Prec	Rec	F0.5
XLNet	72.4	444.6	147.2	0.140	0.330	0.158
FT XLNet	27.1	13.2	191.1	0.683	0.124	0.352
Table 5: Performance of GECToR with each setting. Scores are averaged among 5 runs. Table 7 provides detailed score of every run. XLNet is the baseline GECToR model based on XLNet, and FT XLNet is the finestured GECToR using level 3 edits.
Table 5 indicates the efficacy of our data in terms of improving the performance of the GEC-ToR model. The fine-tuned model outperforms the original in terms of  $\mathrm{F}_{0.5}$ , a metric commonly used in GEC (Omelianchuk et al., 2020). The significant increase in  $\mathrm{F}_{0.5}$  score results from a massive reduction of false positives. In other words, after we fine-tune GECToR on our dataset, the model produces far fewer edits, which helps improve the precision greatly. This is of particular importance in a GEC model, as model precision is considered more important than recall in GEC tasks since false positives could lead to serious confusion in language learners.
Due to the limited size of the dataset, and the uneven distribution of errors in user utterances, we use 5-fold cross-validation to ensure the reliability of our results. We report the average of five cross-validation runs. One note, we modified ERRANT to allow equivalent edits, our reported results on all models might be slightly higher than original ERRANT-based results.
We provide the first high-quality, fine-grained error-correction conversation dataset between English second language learner and an educational chatbot. To demonstrate the utility of our dataset, we train and evaluate a SoTA GEC model on the dataset, resulting in a significant improvement in overall model performance for conversational setting. This project lays the groundwork for future work on conversational grammatical error correction (such as adding other dialog domains and incorporating information about the native languages of users) and customized educational dialog system for second language learners.
3https://github.com/grammarly/gector# pretrained-models
Collecting these dialogs for our dataset is difficult in that it requires substantial commitment from participants. In order to provide as large of a dataset as possible, we utilized the services of Amazon Mechanical Turk as previously mentioned. Given ethical concerns in recent years regarding data acquisition through crowdworkers, we verified that the crowdworkers assigned to our tasks were compensated fairly and treated humanely.
The annotators also examined the dataset to ensure that it does not contain personally identifiable information (which was anonymized) or potentially offensive content (which was removed).
Christopher Bryant, Mariano Felice, Øistein E Andersen, and Ted Briscoe. 2019. The BEA-2019 shared task on grammatical error correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52-75.
Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 793-805.
Andrew Caines, Helen Yannakoudakis, Helena Edmondson, Helen Allen, Pascual Pérez-Paredes, Bill Byrne, and Paula Buttery. 2020. The Teacher-Student Chatroom Corpus. In Proceedings of the 9th Workshop on NLP for Computer Assisted Language Learning, pages 10-20, Gothenburg, Sweden. LiU Electronic Press.
Andrew D Cohen and Margaret Robbins. 1976. Toward assessing interlanguage performance: The relationship between selected errors, learners' characteristics, and learners' explanations. Language learning, 26(1):45-66.
Sam Davidson, Dian Yu, and Zhou Yu. 2019. Dependency Parsing for Spoken Dialog Systems. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1513-1519.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language