SlowGuess/NAACL_2022_OCR huggingface dataset:
Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Dana Ferris. 2011. Treatment of error in second language student writing. University of Michigan Press.
Eric N Forsythand and Craig H Martell. 2007. Lexical and discourse analysis of online chat dialog. In International Conference on Semantic Computing (ICSC 2007), pages 19-26. IEEE.
Luke Fryer, David Coniam, Rollo Carpenter, and Diana Lăpusaneanu. 2020. Bots for language learning now: Current and future directions.
Jin-Xia Huang, Kyung-Soon Lee, Oh-Woog Kwon, and Young-Kil Kim. 2017. A chatbot for a dialogue-based second language learning system. CALL in a climate of change: adapting to turbulent global conditions–short papers from EUROCALL, pages 151–156.
Weijiao Huang, Khe Foon Hew, and Luke K Fryer. 2021. Chatbots for language learning—Are they really useful? A systematic review of chatbots-supported language learning. Journal of Computer Assisted Learning.
Svetlana Koltovskaia. 2020. Student engagement with automated written corrective feedback (AWCF) provided by Grammarly: A multiple case study. Assessing Writing, 44:100450.
Yu Li, Josh Arnold, Feifan Yan, Weiyan Shi, and Zhou Yu. 2021. LEGOEval: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcing. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations.
Kaihui Liang, Austin Chau, Yu Li, Xueyuan Lu, Dian Yu, Mingyang Zhou, Ishan Jain, Sam Davidson, Josh Arnold, Minh Nguyen, et al. 2020. Gunrock 2.0: A user adaptive social conversational system. arXiv preprint arXiv:2011.08906.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Jakub Náplava, Milan Straka, Jana Straková, and Alexandr Rosen. 2022. Czech Grammar Error Correction with a Large and Diverse Corpus. arXiv preprint arXiv:2201.05590.
Courtney Naples, Maria Nădejde, and Joel Tetreault. 2019. Enabling robust grammatical error correction in new domains: Data sets, metrics, and analyses. Transactions of the Association for Computational Linguistics, 7:551-566.
Courtney Naples, Keisuke Sakaguchi, and Joel Tetreault. 2017. JFLEG: A fluency corpus and benchmark for grammatical error correction. arXiv preprint arXiv:1702.04066.
Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 1-14.
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR-Grammatical Error Correction: Tag, Not Rewrite. arXiv preprint arXiv:2005.12592.
Kristen Purcell, Judy Buchanan, and Linda Friedrich. 2013. The impact of digital tools on student writing and how writing is taught in schools. Washington, DC: Pew Research Center.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, and Jason Weston. 2020. Recipes for building an open-domain chatbot.
Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2021. A Simple Recipe for Multilingual Grammatical Error Correction. arXiv preprint arXiv:2106.03830.
Alla Rozovskaya and Dan Roth. 2010. Annotating ESL errors: Challenges and rewards. In Proceedings of the NAACL HLT 2010 fifth workshop on innovative use of NLP for building educational applications, pages 28-36.
Alla Rozovskaya and Dan Roth. 2019. Grammar Error Correction in Morphologically Rich Languages: The Case of Russian. Transactions of the Association for Computational Linguistics, 7:1-17.
Alla Rozovskaya and Dan Roth. 2021. How Good (really) are Grammatical Error Correction Systems? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2686-2698.
Kanyakorn Sermsook, Jiraporn Liamnimit, and Rattaneekorn Pochakorn. 2017. An Analysis of Errors in Written English Sentences: A Case Study of Thai EFL Students. English Language Teaching, 10(3):101-110.
Felix Stahlberg and Shankar Kumar. 2021. Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models. In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pages 37-47.
Shalini Raj Thangaraj and Mahendran Maniam. 2015. The Influence of Netspeak on students' writing. Journal of Education and Learning, 9(1):45-52.
Hanna Y Touchie. 1986. Second language learning errors: Their types, causes, and treatment. JALT journal, 8(1):75-80.
Viet Anh Trinh and Alla Rozovskaya. 2021. New Dataset and Strong Baselines for the Grammatical Error Correction of Russian. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4103-4111.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32.
A.1 Annotation Exceptions
Even though they violate the rules of standard English, we left the following types of errors unchanged in our annotated dataset:
1. Utterances that are not complete sentences. For example, response utterances such as Yes, Very good, and Me too are considered correct in our annotation due to their prevalence in informal dialog, although they are not correct in formal writing.
2. Use of common English internet slang and shorthand expressions. Slang and shorthand expressions such as lol ("laugh out loud") and  $u$  (short for "you") are not only distinctive to online chat conversations, but also reflective of their casual nature. Additionally, they may be language, culture, and even sub-culture specific. While these terms may not be suitable to a more formal register, they are generally acceptable in the context of informal dialog (Forsythand and Martell, 2007); thus, we do not classify such usage as errors.
A.2 Dataset Statistics
As described in Section 4, Table 6 shows the type distribution of edit type in ErAConD. Type labels were generated using our version of ERRANT. Levels of edits were first generated by ERRANT, and then manually checked to label Type 2 edits that are hard to be recognized by code (non-English Internet slangs, acronyms and abbreviations). To take all annotators into consideration, the number of edits was averaged among multiple annotators.
The statistics give us several important insights. First, the number of "significant" errors is slightly
Level	Type	Number	%
1	PUNCT	824.5	63.28
ORTH	478.5	36.72
Total	1303.0	55.45
2	SPELL	0.5	0.14
PUNCT	229.5	63.31
PREP	1.0	0.28
OTHER	124.5	34.34
NOUN:POSS	3.5	0.97
NOUN	2.0	0.55
DET	0.5	0.14
ADJ	1.0	0.28
Total	362.5	15.43
3	WO	9.5	1.39
VERB:TENSE	37.5	5.48
VERB:SVA	19.0	2.78
VERB:INFL	1.0	0.15
VERB:FORM	37.5	5.48
VERB	40.0	5.84
SPELL	115.5	16.87
SPACE	11.0	1.61
PRON	34.0	4.97
PREP	69.0	10.08
PART	4.0	0.58
OTHER	110.0	16.07
NOUN:POSS	3.5	0.51
NOUN:NUM	35.5	5.19
NOUN:INFL	2.5	0.37
NOUN	35.5	5.19
MORPH	28.0	4.09
DET	57.0	8.33
CONTR	4.0	0.58
CONJ	3.5	0.51
ADV	15.0	2.19
ADJ:FORM	2.5	0.37
ADJ	9.5	1.39
Total	684.5	29.13
Table 6: Error type distribution.
higher than in written GEC datasets, such as NUCLE. This result shows that grammatical errors are relatively rare in both the conversational and written domain. Additionally, the average length of each sentence is significantly shorter than written GEC datasets. Finally, the error rate data supports our tiered categorization of errors, as the frequency of errors would be much higher than nonconversational datasets if all less significant errors, such as capitalization and punctuation, were included.
Table 7 is the full version of Table 5. Some details of experiment are mentioned at Section 5.2.  $20\%$  of the dialogs were chosen randomly for the test set and the rest were used for training. Then 5-fold cross-validation was applied and the whole process was run 5 times in total, so as to observe the reliability of our results. We used the recommended parameters of XLNet to train and test
GECToR. From the table we can see that the variance of performance among these runs is small. The distribution of Level 3 edits in test and train sets for each run is also represented in Table 8.
Run No.	Setting	TP	FP	FN	Prec	Rec	F0.5
1	XLNet	54	395	157	0.120	0.256	0.135
FT XLNet	21.4	6.8	184.6	0.759	0.104	0.336
2	XLNet	71	506	134	0.123	0.346	0.141
FT XLNet	24.4	11.0	179.6	0.690	0.120	0.353
3	XLNet	77	437	168	0.150	0.314	0.167
FT XLNet	25.4	14.6	219.6	0.637	0.104	0.313
4	XLNet	74	404	146	0.155	0.336	0.173
FT XLNet	22.6	10.4	196.4	0.686	0.103	0.321
5	XLNet	86	481	131	0.152	0.396	0.173
FT XLNet	41.6	23.2	175.4	0.642	0.192	0.437
Avg.	XLNet	72.4	444.6	147.2	0.140	0.330	0.158
FT XLNet	27.1	13.2	191.1	0.683	0.124	0.352
Table 7: Performance of GECToR with each setting in 5 runs.
Type	1	2	3	4	5
Test	Train	Test	Train	Test	Train	Test	Train	Test	Train
WO	1.03	1.48	1.23	1.42	1.14	1.45	1.74	1.32	0.87	1.49
VERB:TENSE	8.28	4.73	5.33	5.51	5.68	5.43	2.17	6.15	6.11	5.35
VERB:SVA	4.14	2.41	2.46	2.84	2.27	2.90	0.43	3.25	3.06	2.72
VERB:INFL	0.34	0.09	0.00	0.18	0.38	0.09	0.00	0.18	0.00	0.18
VERB:FORM	4.83	5.65	5.33	5.51	4.55	5.70	5.22	5.53	6.55	5.26
VERB	6.21	5.75	4.92	6.04	6.06	5.79	6.09	5.79	4.37	6.14
SPELL	15.17	17.33	17.21	16.80	17.42	16.74	19.13	16.42	18.78	16.49
SPACE	1.38	1.67	1.64	1.60	3.03	1.27	1.30	1.67	0.87	1.75
PRON	8.97	3.89	4.51	5.07	4.92	4.98	3.04	5.36	3.93	5.18
PREP	9.31	10.29	11.07	9.87	10.23	10.05	8.70	10.36	13.97	9.30
PART	1.38	0.37	0.82	0.53	0.38	0.63	0.00	0.70	1.31	0.44
OTHER	16.90	15.85	16.39	16.00	15.53	16.20	17.83	15.72	10.48	17.19
NOUN:POSS	0.00	0.65	0.41	0.53	0.38	0.54	0.87	0.44	0.44	0.53
NOUN:NUM	5.52	5.10	8.61	4.44	5.30	5.16	7.39	4.74	6.55	4.91
NOUN:INFL	0.34	0.37	0.41	0.36	1.52	0.09	0.87	0.26	0.87	0.26
NOUN	1.38	6.21	5.74	5.07	3.41	5.61	4.78	5.27	2.62	5.70
MORPH	2.41	4.54	2.05	4.53	3.79	4.16	5.65	3.78	3.49	4.21
DET	7.59	8.53	7.38	8.53	9.47	8.05	7.83	8.43	11.79	7.63
CONTR	0.00	0.74	0.41	0.62	0.38	0.63	1.30	0.44	0.87	0.53
CONJ	0.34	0.56	0.41	0.53	0.00	0.63	0.87	0.44	0.00	0.61
ADV	2.07	2.22	1.64	2.31	2.65	2.08	2.61	2.11	1.31	2.37
ADJ:FORM	0.34	0.37	0.82	0.27	0.38	0.36	0.43	0.35	0.87	0.26
ADJ	2.07	1.20	1.23	1.42	1.14	1.45	1.74	1.32	0.87	1.49
Table 8: Level 3 error type distribution (%) in train and test sets of 5 runs.