# Automatic Correction of Indonesian Grammatical Errors Based on Transformer
## 1 Introduction  
Grammar is one of the most important features in human language which provides understanding and readability of text for readers or listeners [1]. Grammar and related errors in text and speech lead to confusion and misunderstanding for readers. Therefore, numerous research studies in natural language processing have tried to address the challenge of automatic grammar error correction (GEC). Recently, English and Chinese have received more research attention thanks to large-scale pretraining models and massive training data for GEC, which reported close to human-like performance [2; 3]. However, languages such as Arabic, Russian and Indonesian have received little research attention due to a lack of training data, being classified as low-resource languages. The Indonesian language is listed as the 10 th-most-spoken language in the world ([[https://www.visualcapitalist.com/100-most-spoken-languages/](https://www.visualcapitalist.com/100-most-spoken-languages/)]([https://www.visualcapitalist.com/100-most-spoken-languages/](https://www.visualcapitalist.com/100-most-spoken-languages/)), accessed on 1 May 2022) and is used by more than 270 million people ([[https://en.wikipedia.org/wiki/List](https://en.wikipedia.org/wiki/List)]([https://en.wikipedia.org/wiki/List](https://en.wikipedia.org/wiki/List)), accessed on 1 May 2022) in countries and dependencies by population. However, the lack of research in Indonesian natural language processing (NLP), especially in the GEC task, is associated with the lack of training data, since no parallel GEC training data are available. In the last decade, the growth of neural networks in NLP has increased significantly and has even penetrated various tasks including text classification [4; 5], information extraction [6; 7], summarization [8], question answering [9] and GEC [10]. The automatic correction of grammar and spelling has increased due to the recent increase in second language learners and also the increasing advancement of deep learning technology, which can be felt directly in real life [11; 12]. Recently, automatic GEC systems based on sequence-to-sequence (seq2 seq) machine translation techniques have been preferred, such as recurrentneural networks (RNN) [13] convolutional neural networks (CNNs) [14] and multi-head attention networks (Transformer) [15; 16], which has been found to outperform the other models in resource NLP tasks, including the GEC task. GEC based on NMT techniques uses parallel data to learn how to map the source (correct sentence) to the target (incorrect sentence), as shown in Figure 1.  
Indonesian GEC is still growing, and few studies have been performed, such as that by [PERSON] et al. [17], who introduced an analysis-based descriptive research method to analyze grammatical errors. [PERSON] et al. [4] proposed an Indonesian GEC framework for classification tasks that corrected 10 part of speech (POS) errors in text. [PERSON] et al. [18] proposed the Markov probability model to correct spelling and grammatical errors. [PERSON] et al. [19] specified 10 types of grammatical errors that often occur in writing using a morphological and syntactic approach. [PERSON] et al. [20] introduced a grammar-checking method based on the rule for checking sentence patterns and punctuation. Most of the previous methods are applied using rule-based and statistical methods, and to the best of our knowledge, end-to-end neural-based approaches have not been investigated in Indonesian GEC. In this paper, we propose automatic GEC based on Transformer [15] to correct Indonesian grammar errors, which can also be used to correct the text of other low-resource languages. The proposed model was equipped with a copy mechanism to address the unknown and special words that appeared in the source sentences, which is one of GEC's challenges. Aside from that, a semi-supervised method has been proposed to construct parallel training data from a monolingual corpus that is out of domain.