# Automatic Correction of Indonesian Grammatical Errors Based on Transformer
## 2 Related Works  
The attention in research and industry toward automatically detecting and correcting grammatical errors in text has increased in the last decade. However, different approaches have been proposed, such as rule-based systems that use hundreds of grammar rules to detect and correct grammatical errors [21]. Furthermore, n-gram language models have been applied to measure the probability of each sequence of words (n-gram) from a large corpus of text [22]. GECs are based on a statistical classifier model that uses classification algorithms such as decision trees, naive Bayes classifiers and logistic regression to detect grammatical errors [4]. Statistical machine translation (SMT) is phrase-based statistical machine translation for the incorrect input sentence and maximizes the conditional probability of the suggested output correct sentence over possible corrections [10]. More recently, neural machine translation (NMT) deep neural networks have been used in GEC as classes of models that learn how to map an ill-formatted input (an incorrect sentence) to a well-formatted output (a correct sentence) via a set of hidden layers [23]. [PERSON] et al. [24] proposed GEC based on a five-gram language model to overcome the limitations of the training data and also claimed that it outperformed the language model in statistical machine translation. The model was trained on a large corpus containing a collection of more than nine billion web pages. [PERSON] and [PERSON] [25] used the trigram and bigram language models to detect grammatical errors in a large corpus. [PERSON] et al. [26] used a combination of the n-gram language model and a string-matching algorithm to resolve the problem of Chinese spelling with Cantonese usage. Another study conducted by [PERSON] et al. [27] used the n-gram model to detect and correct Chinese grammatical errors that achieved high accuracy in correction and was not satisfactory in detection. The weakness of these language models is that they are not suitable in the representation of sequence probability and handling sentence construction or rare words. GEC classifier systems are used to classify input sentences into category sentences or words. In this case, most of the previous work used classified specific error types such as verbs, prepositions and articles based on the large size of the possible correction. [PERSON] et al. [13] developed a GEC system to correct preposition errors, missing commas, articles, nouns and verb forms. [PERSON] et al. [4] introduced an Indonesian GEC framework that used a combination of pretraining language with a BRNN to correct 10 types of errors in the text. Another good example is presented by [PERSON] et al. [28], who employed a bidirectional LSTM tagger for the proper word choice for English GEC. This appends the part of speech (POS) tagger and filters out the suggested word by verb form, article, preposition, comma and noun number from the source tokens. The disadvantage of this approach is that the error type classification target is extremely specific, and it is difficult to extend the error types. However, SMT achieved remarkable improvement in GEC. For example, [PERSON] et al. [29] used an SMT-based method with task-specific features to correct grammatical errors. [PERSON] and [PERSON] [3] proposed a hybrid system of SMT and NMT methods for automatic correction of grammatical errors. However, SMT systems could be used to correct all types of errors, whereas they still rely heavily on large amounts of training data and are customized to specific error types. Recently, NMT approaches have been used in GEC. For example, [PERSON] [30] utilized the NMT approach to overcome the challenge of rare words, and it outperformed SMT systems, which were evaluated in both the First Certificate in English (FCE) and CoNLL-2014 test sets. Another work presented by [PERSON] [31] proposed an NMT model that extends the basic encoder-decoder architecture to handle the problem of the context vector in the source sentence. [PERSON] et al.