Leveraging LLMs for Bangla Grammar Error Correction: Error Categorization, Synthetic Data, and Model Evaluation
Authors: Pramit Bhattacharyya, Arnab Bhattacharya
Accepted at: acl25

Leveraging LLMs for Bangla Grammar Error Correction: Error Categorization, Synthetic Data, and Model Evaluation

Pramit Bhattacharyya    Arnab Bhattacharya
Dept. of Computer Science and Engineering,
Indian Institute of Technology Kanpur,
India
pramitb@cse.iitk.ac.in    arnabb@cse.iitk.ac.in

Abstract
Large Language Models (LLMs) perform exceedingly well in Natural Language Understanding (NLU) tasks for many languages including English. However, despite being the fifth most-spoken language globally, Grammatical Error Correction (GEC) in Bangla remains underdeveloped. In this work, we investigate how LLMs can be leveraged for improving Bangla GEC. For that, we first do an extensive categorization of 12 error classes in Bangla, and take a survey of native Bangla speakers to collect real-world errors. We next devise a rule-based noise injection method to create grammatically incorrect sentences corresponding to correct ones. The Vaiyākarana dataset, thus created, consists of 5,67,422 sentences of which 2,27,119 are erroneous. This dataset is then used to instruction-tune LLMs for the task of GEC in Bangla. Evaluations show that instruction-tuning with Vaiyākarana improves GEC performance of LLMs by 3–7 percentage points as compared to the zero-shot setting, and makes them achieve human-like performance in grammatical error identification. Humans, though, remain superior in error correction. The data and code are available from https://github.com/Bangla-iitk/Vaiyakarana.

1 Introduction

Grammatical Error Correction (GEC) aims to detect and correct grammatical errors in a text automatically. For example, given the following sentence in English, “A ten year oldest boy go to school.” a GEC system detects errors in the use of the superlative degree and verb number and corrects it to “A ten-year-old boy goes to school.”. The CoNLL 2013 and 2014 shared tasks (Ng et al., 2013, 2014) significantly advanced GEC research, but focus primarily on English.

Bangla (Bengali, বাংলা/বাঙ্গলা Bāṅgalā1) is the fifth most spoken language in the world. However, to our knowledge, only a handful of GEC works exist for Bangla. Alam et al. (2007) proposed a rule-based statistical grammar checker, but its coverage of grammatical rules is limited, leading to failure in detecting and correcting complex errors. Data-driven GEC methods require pairs of correct and corresponding incorrect sentences for training. Islam et al. (2018) attempted to generate erroneous Bangla sentences by randomly inserting, deleting, and swapping words in a corpus of 250K sentences. While such methods have been effective for English, they are less suitable for Bangla and other morphologically rich Indian languages with free word order.

Word-level operations such as swapping, deletion, and insertion often fail to produce grammatically incorrect Bangla sentences. Being morphologically rich, Bangla exhibits free word order. Thus, different permutations of subject-verb-object (SVO) are grammatically valid. For instance, consider the Bangla sentence অমর গীতা-কে ভালোবাসে (amara gītāē bhālōvāsē.. Amar loves Geeta.) shown in Table A1 of Appx. A. The first five variants contain the same words in different orders, all of which are grammatically correct. The next three variations introduce word substitution, deletion, and insertion; even these forms remain grammatically correct. Thus, a more nuanced error generation approach, beyond simple word-level operations, is needed for Bangla GEC.

Large Language Models (LLMs), particularly those trained with instruction fine-tuning, have demonstrated strong capabilities in generating synthetic data for various NLP tasks, particularly in English (Li et al., 2024; Jin et al., 2024).

In this paper, we explore the ability of LLMs to perform GEC for Bangla. For that, we first do an extensive categorization of possible grammati-

1We use ISO15919 transliteration scheme for Bangla: https://en.wikipedia.org/wiki/ISO_15919
cal errors found in Bangla into 12 error categories. We then collected real-world errors by doing an essay writing survey of native speakers. We use the distribution of real errors found from this survey (of 2,576 sentences) to generate synthetic data that contains grammatical errors. The generation is done using a rule-based noise injection scheme on grammatically correct sentences to guarantee that the generated sentences are actually grammatically incorrect.

We curate a dataset, Vaiyākarana2, consisting of 2,27,119 erroneous and 5,67,422 total Bangla sentences to instruction-tune LLMs. We use the Vaiyākarana dataset to evaluate the ability of LLMs for GEC. We compare the performance of LLMs thus instruction-tuned with Vaiyākarana against both zero-shot performance as well as human evaluators. We also assess their performance for another task, namely, paraphrasing.

Experiments show that using Vaiyākarana improves LLM performance by approximately 3-7 percentage points compared to the zero-shot setting for both GEC and paraphrasing tasks. However, humans still significantly outperform LLMs in correcting grammatically erroneous sentences.

Our key contributions in this paper are:
1. We do an extensive categorization of possible grammatical errors in Bangla into 12 distinct categories (Sec. 3), making this the first such extensive attempt. Standardization of grammatical error categories is essential for addressing challenges in low-resource language (Nigatu et al., 2024). This categorization of errors is applicable for many major Indian languages (Appx. C).
2. We collect and analyze 2,576 human-annotated sentences to identify common grammatical errors. We also present statistics on error distributions based on real-world usage (Sec. 4).
3. We propose a structured error generation approach (Sec. 6) for systematically injecting grammatical errors into Bangla sentences. This approach allows for scalable generation of error-annotated corpora. Our approach is extendable to other Indian languages (Appx. C).
4. We curate Vaiyākarana, a dataset containing 2,27,119 erroneous Bangla sentences, which can be directly used instruction-tuning LLMs for Bangla GEC.
5. We evaluate the effectiveness of instruction-tuned LLMs (decoder-based) for GEC in Bangla against both zero-shot setting and human evaluators (Sec. 7.5). Experiments show that instruction-tuning using Vaiyākarana improves LLM performance by 3-7 percentage points compared to the zero-shot setting for both GEC and paraphrasing tasks. However, humans still significantly outperform LLMs in correcting grammatically erroneous sentences.

2 Related Work

Grammatical Error Correction (GEC) for Indian languages, including Bangla, is still in its early stages. While existing methods have explored rule-based, statistical, and neural approaches, they lack comprehensive error categorization and robust datasets. Early works on GEC focused on rule-based and statistical models. Sonawane et al. (2020) categorized inflectional errors for Hindi GEC, and Rachel et al. (2023) proposed Vyakaranly, a toolkit for Hindi grammar correction. Alam et al. (2007) introduced a rule-based statistical approach, but it failed to generalize beyond simple sentences. Islam et al. (2018) attempted to generate erroneous Bangla sentences via random word swaps, insertions, and deletions. However, this method of generating wrong sentences does not always give the desired result as shown in Table A1 of Appx. A. Rahman et al. (2023) developed a CNN-based spelling correction model, while Oshin et al. (2023) curated a 10K-sentence dataset for Bangla text error classification, with only 3,140 erroneous sentences. Hossain et al. (2023) proposed Panini, a Vaswani-style monolingual transformer for Bangla GEC, and synthetically generated a 7.7M+ sentence corpus over 10 error categories. However, their error classification lacked key categories such as tense errors, Gurucandālī Doṣa, and semantic errors, which are significant as reported in Table 2. Hence, we did not use this dataset for instruction-tuning LLMs. Maity et al. (2024) generated a dataset of only 3,412 sentences curated by amalgamating 1,678 sentences (only 50 erroneous samples are publicly available) from essays written by school students and 1,724 sentences by crawling social media websites. This work does not consider number errors, gender errors and semantic errors in Bangla, which may not be significant but do occur occasionally (Sec. 4). Additionally, they do not classify POS and homonym errors, which

2The word means “grammarian” in Bangla.
<table>
  <tr>
    <th>Error Class</th>
    <th>Sub-class</th>
    <th>Example of Error Sentence (in <span style="color:purple;">violet</span>) followed by Correct Sentence (in <span style="color:blue;">blue</span>)</th>
  </tr>
  <tr>
    <td rowspan="4">Spelling</td>
    <td>Non-Dictionary</td>
    <td>আমি কারখানায় কাজ করি। (āmi kārkhānāya kāj kari., I <span style="color:blue;">non-word</span> in factory.)<br>আমি কারখানায় কাজ করি। (āmi kārkhānāya kāj kari., I work in factory.)</td>
  </tr>
  <tr>
    <td>Dictionary</td>
    <td>আমি কাল বাড়ি যাবো (āmi kāl bāṛi yāba., I will go water tomorrow.)<br>আমি কাল বাড়ি যাবো (āmi kāl bāṛi yāba., I will go home tomorrow.)<br>আমি কাল শহরে যাবো (āmi kāl śhara yāba., I will go source tomorrow.)<br>আমি কাল বাড়ি যাবো (āmi kāl bāṛi yāba., I will go home tomorrow.)</td>
  </tr>
  <tr>
    <td>Tense</td>
    <td>আমি গতকাল পড়াশোনা করেছি। (āmi gatakāla parāsōna karāba., I will study yesterday.)<br>আমি গতকাল পড়াশোনা করেছি। (āmi gatakāla parāsōna karēchilāma., I studied yesterday.)<br>বন্ধ হলে বাসায় তখন ফুল ফুটেছে। (yakhana sīta āsāb takhana phula phutēchula., When winter comes, flowers bloomed.)<br>বন্ধ হলে বাসায় তখন ফুল ফুটেছে। (yakhana sīta āsāb takhana phula phutabē., When winter comes, flowers will bloom.)</td>
  </tr>
  <tr>
    <td>Person</td>
    <td>আমি কারখানায় কাজ করি। (āmi kārkhānāya kāj kari., I works in factory.)<br>আমি কারখানায় কাজ করি। (āmi kārkhānāya kāj kari., I work in factory.)</td>
  </tr>
  <tr>
    <td rowspan="2">Word</td>
    <td>Number</td>
    <td>আমি একজন চরিত্র ছিলি (amara ekhāne carijana thāki., I four stay here.)<br>আমি একজন চরিত্র ছিলি (amara ekhāne carijana thāki., We tour stay here.)</td>
  </tr>
  <tr>
    <td>Gender</td>
    <td>উন্ম একজন বাবার অভিনেত্রী (uttama ekājana asādhārāna abhinēṭī., Uttam is an outstanding actress.)<br>উন্ম একজন বাবার অভিনেত্রী (uttama ekājana asādhārāna abhinēṭī., Uttam is an outstanding actor.)</td>
  </tr>
  <tr>
    <td rowspan="2">Case</td>
    <td></td>
    <td>আমি রান্নাঘরে ভাত খাই (āmi rānnāgharē bhāṭa khāi., I eat rice to kitchen.)<br>আমি রান্নাঘরে ভাত খাই (āmi rānnāgharē bhāṭa khāi., I eat rice in kitchen.)</td>
  </tr>
  <tr>
    <td>Parts-of-Speech</td>
    <td>হিমালয়ের সুন্দর অভিমানী (himālayēra sundara abhisamarāṇī., The beautiful of Himalaya is unforgettable.)<br>হিমালয়ের সুন্দর অভিমানী (himālayēra saundarya abhisamarāṇī., The beauty of Himalaya is unforgettable.)</td>
  </tr>
  <tr>
    <td rowspan="3">Missing</td>
    <td></td>
    <td>আমি কাল বাড়ি (āmi kāl bāṛi .•I home tomorrow.)<br>আমি কাল বাড়ি যাবো (āmi kāl bāṛi yāba., I will go home tomorrow.)</td>
  </tr>
  <tr>
    <td></td>
    <td>উন্ম একজন বাবার অভিনেত্রী (uttama ekājana asādhārāna abhinēṭī., Uttam is an outstanding actress.)<br>উন্ম একজন বাবার অভিনেত্রী (uttama ekājana asādhārāna abhinēṭī., Uttam is an outstanding actor.)</td>
  </tr>
  <tr>
    <td>Gurucandāli dōṣa</td>
    <td>নন্দবাবু হঠাৎ কাজ করেছেন (nandabābu hāṭhā kāj karēchēna., Nanda has noticed this.)<br>নন্দবাবু হঠাৎ কাজ করেছেন (nandabābu hāṭhā kāj karēchēna., Nanda has noticed this.)<br>নন্দবাবু হঠাৎ কাজ করেছেন (nandabābu hāṭhā kāj karēchēna., Nanda has noticed this.)<br>নন্দবাবু হঠাৎ কাজ করেছেন (nandabābu hāṭhā kāj karēchēna., Nanda has noticed this.)</td>
  </tr>
  <tr>
    <td>Punctuation</td>
    <td></td>
    <td>আমি গতকাল পড়াশোনা করেছি। (āmi gatakāla parāsōna karēchilāma?, I studied yesterday?)<br>আমি গতকাল পড়াশোনা করেছি। (āmi gatakāla parāsōna karēchilāma., I studied yesterday.)</td>
  </tr>
  <tr>
    <td rowspan="4">Semantic</td>
    <td></td>
    <td>মানস আকাশে খেতে ভালবাসে (mānasa ākāśe khēṭe bhālōbāse., Manas loves to eat the sky.)<br>মানস আকাশে খেতে ভালবাসে (mānasa ākāśe khēṭe bhālōbāse., Manas loves to see the sky.)<br>মানস আকাশে খেতে ভালবাসে (mānasa ākāśe khēṭe bhālōbāse., Manas loves to eat the sky.)<br>মানস মাছ খেতে ভালবাসে (mānasa mācha khēṭe bhālōbāse., Manas loves to eat fish.)</td>
  </tr>
</table>

Table 1: Grammatical Error Types in Bangla

are significant in Bangla. We have discussed about GEC for English and other languages in Appx B. Back-translation has been used for data augmentation in GEC (Sennrich et al., 2016; Rei and Yannakoudakis, 2017; Zhou et al., 2020). However, round-trip translation for Bangla using English as a bridge fails to consistently produce grammatically incorrect sentences. Moreover, errors generated via back-translation are challenging to categorize and localize, making them unsuitable for Bangla GEC (Sec. 5).

Since Bangla lacks large-scale GEC datasets, we adopt a structured noise injection approach based on real-world error patterns, ensuring the controlled generation of incorrect sentences across specific error types. This approach improves sentence quality while maintaining category-wise error distributions observed in real time. Unlike previous methods, our dataset explicitly covers common Bangla errors and is adaptable to other Indian languages (Appx. C).

3 Grammar Error Categories

Standardization of error categories is necessary for GEC to alleviate problems associated with low-resource languages (Nigatu et al., 2024). In this section, we categorize grammatical errors in Bangla formally. We follow a standard Bangla grammar book (Chakroborty, 2018) as reference for grammar error types. The book explicitly does not have any error categories; however, following the grammatical rules described in the book, we have formalised these error categories. To our humble understanding, these categories are exhaustive and cover all possible error types in Bangla. They can be also used for other major Indian languages (Appx. C).

First, we classified the grammatical errors in Bangla into 5 broader categories. These broad categories are then further sub-divided into 12 finer distinctions. Table 1 lists example sentences for all the categories3 of the error classes. Appx. D

3The text in violet shows the erroneous portion of a sentence corresponding to the correct text in blue.
explains the category of errors in detail with examples. A sentence may contain multiple errors of one class or different classes.

4 Manual Generation

To understand the nature of real-life grammatical errors made by native speakers, we organized a survey in which participants were asked to write an essay on a specific topic within a specific time. In this way, we collected handwritten sentences from native Bangla speakers and analyzed the occurrence of various grammatical errors in those sentences. Each participant was allowed 30 minutes to write an essay comprising at least 15 sentences and 150 words, choosing from a set of topics provided. The survey was conducted in a proctored environment to simulate an exam-like situation, allowing us to collect real-time data (including errors) on Bangla writing. We collected 123 essays, resulting in 2,576 sentences and 28,713 words, produced by 51 participants (30 participants wrote 2 essays each, while 21 participants wrote 3 essays). The longest sentence contained 69 words, while the shortest had just 1 word. Detailed information about the topics and participants can be found in Appx. E. A team of 3 Bangla language experts then evaluated the written sentences and the errors were categorised based on majority voting. Of the 2,576 sentences written, 1,045 (41%) were grammatically incorrect.4 Of the erroneous sentences, 804 (77%) contained single errors, while 241 (23%) had multiple errors. Among the sentences with multiple errors, 185 contained 2 errors (18%), and 42 sentences contained 3 errors (4%). The remaining 9 sentences contained up to 6 errors. A total of 678 (4%) words were erroneous in these 1,045 sentences.

Table 2 presents the number of errors for each category outlined in Table 1, along with their respective percentages for the total number of erroneous words. Spelling mistakes emerged as the most prevalent type of error, representing over 62% of occurrences. Further analysis indicates that more than 45% of spelling errors stem from the confusion between the characters ‘র’(n)/‘ণ’(n); ‘র’(r)/‘ঘ’(r)/‘ঘ’(rh); and ‘স’(s)/‘শ’(ś)/‘ষ’(s). The most significant dictionary-based spelling error involves the mixing of “কি” (ki, whether) and “কি” (kī, what). Al-

<table>
  <tr>
    <th>Error Class</th>
    <th>#Occurences</th>
    <th>Percentage</th>
  </tr>
  <tr>
    <td>Non-Dictionary</td>
    <td>677</td>
    <td>49.85</td>
  </tr>
  <tr>
    <td>Dictionary</td>
    <td>174</td>
    <td>12.81%</td>
  </tr>
  <tr>
    <td>Spelling Errors</td>
    <td>851</td>
    <td>62.66%</td>
  </tr>
  <tr>
    <td>Tense Errors</td>
    <td>30</td>
    <td>2.21%</td>
  </tr>
  <tr>
    <td>Person Errors</td>
    <td>26</td>
    <td>1.91%</td>
  </tr>
  <tr>
    <td>Number Errors</td>
    <td>4</td>
    <td>0.29%</td>
  </tr>
  <tr>
    <td>Gender Errors</td>
    <td>1</td>
    <td>0.07%</td>
  </tr>
  <tr>
    <td>Case Errors</td>
    <td>162</td>
    <td>11.93%</td>
  </tr>
  <tr>
    <td>POS Errors</td>
    <td>29</td>
    <td>2.14%</td>
  </tr>
  <tr>
    <td>Missing Words</td>
    <td>64</td>
    <td>4.71%</td>
  </tr>
  <tr>
    <td>Word Errors</td>
    <td>316</td>
    <td>23.26%</td>
  </tr>
  <tr>
    <td>Punctuation Errors</td>
    <td>156</td>
    <td>11.49%</td>
  </tr>
  <tr>
    <td>Semantic Errors</td>
    <td>2</td>
    <td>0.15%</td>
  </tr>
  <tr>
    <td>Gurucaṇḍālī Dōṣa</td>
    <td>33</td>
    <td>2.43%</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>1,358</th>
    <th>100.00%</th>
  </tr>
</table>

Table 2: Grammatical errors in manual survey

though tense, person, number, gender, and semantic errors are not as frequent in Bangla, they do occur once in a while. Case and punctuation errors are also rather common. Additionally, we observed that sentences containing multiple errors generally include several spelling mistakes. Notably, the combination of (spelling, punctuation) and (spelling, case, and punctuation) errors occurs the most. In one case, as many as six different types of errors (spelling errors of both kinds, case, missing word, Gurucaṇḍālī Dōṣa, and punctuation) appeared in a single sentence. We next use the relative frequency of these different kinds of errors for our synthetic data generation.

5 Error Injection Methods

In this section, we focus on generating synthetic data mimicking the real-world error distribution. Bryant et al. (2023) proposed various synthetic data generation methods, including back-translation (Sennrich et al., 2016), round-trip translation (Zhou et al., 2020) and error injection methods (Bryant et al., 2023) using grammatical methods. We next show the advantages and disadvantages of each type of methodology for synthetic data generation for Bangla GEC.

5.1 Translation Methods

We start by evaluating back translation and round-trip translation methods to see if they can generate erroneous sentences. Back translation involves translating another language (here, English) text to Bangla. This method introduces subtle variations that mimic realistic errors in text, creating a diverse dataset for model training. However, this method does not always guarantee the genera-

4This is likely due to the exam-like time situation where participants did not get a time to revise and correct.
tion of erroneous sentences, as shown in Table A3 of Appx. F. Zhou et al. (2020) used round-trip translation, a variant of back-translation, to synthesize noisy sentences using a bridge language, e.g., English-Chinese-English, where Chinese is the bridge language. We tried to generate the wrong sentences by following the same methodology using English as a bridge language. Example sentences are shown in Table A4 of Appx. F.

5.2 LLM-generated Sentences

The same phenomenon is observed while trying to generate erroneous sentences using GPT-4o. We used the prompt ব্যাকরণগত ভুল বাকা লেখ (byākaranagata bhula bākya lēkha, Write a grammatically wrong sentence). While it generates sentences such as আমি কাল রাতি সিনেমা দেখছি-লাম (āmi kāla rāti sinēmā dēkhachilāma,, I saw a cinema last <non-word>.) which are grammatically wrong, it also generates sentences like তারা ফুটবল খেলেছে গতকাল (tārā phutabala khēlēchē gatakāla,, They played football yesterday.) which are grammatically correct. Other methods, including adding paraphrased sentences also suffers from similar issues.

From the above discussion, it is clear that none of these methods guarantees a generation of grammatically wrong sentences. Further, it does not categorize and localize errors in the generated sentences. Hence, we adopted the method of error injection following grammatical rules as the preferred method of generating erroneous sentences.

5.3 Rule-based Error Injection

Through our rule-based error injection methodology, not only can we guarantee the presence of grammatical errors in a sentence but also categorize and localize the type of error in the sentence. We introduced noise into grammatically correct sentences following the error distribution shown in Table 2, and curated Vaiyākarana with 2,27,119 erroneous sentences. The total number of sentences is 5,67,422.

For errors related to homonyms, parts-of-speech (POS), tense, person, and case, we primarily replaced the correct word with a corresponding incorrect word (all such pairs were sourced from (Chakroborty, 2018)). In this substitution process, we ensured that for generating tense at least two verbs are present in the sentence. Similarly, for Gurucandāli Dōṣa errors, at least two verb/pro-noun are present. We intentionally do not replace one of these words to ensure the generation of an erroneous sentences. Since we inject noise according to the rules outlined in the book (Chakroborty, 2018), the resulting sentences are guaranteed to be always incorrect. The detailed steps taken to introduce noise for various types of errors are described in Appx. G.

6 Instruction-tuning Dataset: Vaiyākarana

To leverage LLMs for Bangla GEC, we curate a corpus of grammatically incorrect sentences by following the rule injection methodology as described in the previous section.

We use the Vācaspati (Bhattacharyya et al., 2023) corpus as the base set of grammatically correct sentences. We chose this corpus since it consists of only literature data and, hence, sentences sampled from this corpus are grammatically correct, as reported by the authors. Further, the corpus captures stylistic, linguistic, spatial and temporal variations of Bangla. The temporal diversity, in particular, is especially useful for Gurucandāli Dōṣa errors since that particular writing style started becoming rare from 1960s. (Newspapers, blogs, and social media data are not suitable for this.)

We collected 5,67,172 sentences from Vācaspati, which serve as our gold-standard sentences. Additionally, we incorporated 250 sentences from a well-known grammar book (Chakroborty, 2018) to enhance the grammatical and linguistic richness of the dataset. These sentences are beneficial for generating errors related to number, gender, and semantics according to the rules specified in the book, which are less frequently found in literary data. Consequently, we curated a dataset containing a total of 5,67,422 sentences. We followed the data cleaning and pre-processing steps outlined in Appx. H to ensure that this dataset is suitable for error generation.

We inject errors into randomly selected sentences following grammatical rules, with the type of error and the words to inject errors chosen randomly. We choose roughly 40% sentences for error injection. This process ensures a diverse range of errors. Since most real-world erroneous sentences typically involve single-word mistakes, we generate sentences that reflect this pattern. To cover various error categories, we carefully monitor the number of erroneous words to
<table>
  <tr>
    <th>Error Class</th>
    <th>#Occurrences</th>
    <th>Total %</th>
    <th>Error %</th>
  </tr>
  <tr>
    <td>Non-Dictionary Dictionary</td>
    <td>113,244<br>26,046</td>
    <td>19.97<br>4.59</td>
    <td>49.91<br>11.48</td>
  </tr>
  <tr>
    <td>Spelling Errors</td>
    <td>139,290</td>
    <td>24.56</td>
    <td>61.40</td>
  </tr>
  <tr>
    <td>Tense Errors</td>
    <td>4,983</td>
    <td>0.88</td>
    <td>2.20</td>
  </tr>
  <tr>
    <td>Person Errors</td>
    <td>4,530</td>
    <td>0.80</td>
    <td>2.00</td>
  </tr>
  <tr>
    <td>Number Errors</td>
    <td>200</td>
    <td>0.035</td>
    <td>0.090</td>
  </tr>
  <tr>
    <td>Gender Errors</td>
    <td>100</td>
    <td>0.018</td>
    <td>0.044</td>
  </tr>
  <tr>
    <td>Case Errors</td>
    <td>26,046</td>
    <td>4.59</td>
    <td>11.48</td>
  </tr>
  <tr>
    <td>POS Errors</td>
    <td>4622</td>
    <td>0.81</td>
    <td>2.04</td>
  </tr>
  <tr>
    <td>Missing Words</td>
    <td>10,690</td>
    <td>1.88</td>
    <td>4.71</td>
  </tr>
  <tr>
    <td>Word Errors</td>
    <td>51,171</td>
    <td>9.02</td>
    <td>22.55</td>
  </tr>
  <tr>
    <td>Punctuation Errors</td>
    <td>26,046</td>
    <td>4.59</td>
    <td>11.48</td>
  </tr>
  <tr>
    <td>Semantic Errors</td>
    <td>100</td>
    <td>0.018</td>
    <td>0.044</td>
  </tr>
  <tr>
    <td>Gurucandālī Dōṣa</td>
    <td>6,402</td>
    <td>1.13</td>
    <td>2.82</td>
  </tr>
  <tr>
    <td>Multiple Errors</td>
    <td>3,860</td>
    <td>0.68</td>
    <td>1.70</td>
  </tr>
  <tr>
    <td><i>InCorrect</i></td>
    <td>2,26,869</td>
    <td>40.00</td>
    <td>100.00</td>
  </tr>
  <tr>
    <td><i>Correct</i></td>
    <td>340,303</td>
    <td>60.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td><b>Total</b></td>
    <td>5,67,422</td>
    <td>100.00</td>
    <td>100.00</td>
  </tr>
</table>

Table 3: Grammatical Error in Vaiyākarana

ensure it does not exceed 30% of the total words in any given sentence. Since most real-world spelling errors arise from confusion between characters such as ‘ন’(n)/‘র’(r); ‘র’(r)/‘ঘ’(gh); ‘স’(s)/‘শ’(ś)/‘ষ’(ṣ); ‘কী’(ki)/‘কি’(ki) and other similar sounding words, we place additional emphasis on sentences containing these characters or words when introducing spelling errors. For all other types of mistakes, errors are injected randomly with equal probabilities. The detailed steps taken to introduce noise for various types of errors are described in Appx. G.

Table 3 shows the number of sentences generated and their corresponding category of errors. A total of 2,23,259 sentences contains a single error, whereas 3,860 sentences contain multiple errors. Table A6 in Appx. J shows the distribution of multiple errors in Vaiyākarana. The maximum number of errors in a sentence in Vaiyākarana is 10. We validated that the distribution of error categories are comparable between real-world data and generated data (Appx I).

Sec. 4 showed that ~77% of sentences contain single errors, while the remaining contain multiple errors. Amongst them, the most common is multiple spelling errors. If there are multiple errors of the same category in a sentence, we have categorized it in that error category only, and not included them into the multiple error category. Table A6 shows the distribution of sentences with multiple error types in Vaiyākaraṇa. We have considered only those category of multiple errors that were present in the human survey in Sec. 4, and not all possible combinations.

Although we focused on generating a GEC dataset Vaiyākarana for Bangla, the aforementioned procedure of injecting noise to generate grammatically wrong sentences can also be applied to other major Indian languages with little or no modification (Appx. C).

We next show some anecdotal examples in Table 4 to highlight that our error injection method generates grammatically incorrect sentences that resemble real-life ones. It covers homonym errors as well as the popular confusion of শ (ś) / স (s).

Additionally, we validated these sentences by a group of 12 individuals. Each participant was provided with a unique set of 50 sentences and was asked to mark whether they believed the errors were naturally occurring, as well as the minimum education level required to detect the errors. Of the 12 participants, 10 felt that the generated sentences were natural and that a minimum education level of 10th grade would be sufficient to identify the errors. The remaining 2 participants noted that the generated sentences sometimes seemed artificial. Participants also rated the level of naturalness on a Likert scale from 1 (least natural) to 5 (most natural), and the average score for the generated sentences was 3.62.

<table>
  <tr>
    <th>Correct</th>
    <th>Generated</th>
    <th>Error Type</th>
  </tr>
  <tr>
    <td>কেউ উক্তিকৃত মারে না, ভিন্নভাবে করতে নামে না। (kēu umkijhumki mārē nā, distārvā umkijhumki karatē nāmē nā., No one pecks, no one comes to disturb.)</td>
    <td>কেউ উক্তিকৃত মারে না, ভিন্নভাবে করতে নামে না। (kēu umkijhumki mārē nā, distārvā karitē nāmē nā.)</td>
    <td>Gurucandālī Dōṣa</td>
  </tr>
  <tr>
    <td>পন্টুর যে ফাসির হকুম হয়েছে এটা সে নানাকে হকুম হয়েছে এটা বলেন। (paltura yē phānsira hukuma hayēchē ētā sē nabinakē balēni.) He did not tell Navin that Paltu has been sentenced to death.)</td>
    <td>পন্টুর যে ফাসির হকুম হয়েছে এটা সে নানাকে হকুম হয়েছে এটা বলেন। (paltura yē phānsira hukuma hayēchē ētā sē nabinakē balēni.)</td>
    <td>Non-Dictionary</td>
  </tr>
  <tr>
    <td>অদ্ধকারে গাছপালা অফকারে গাছপালা Dictionary দেখে করে সে কি দেখে করে সে কি দেখে? (andhakārē gāchāpālā bhēda karē sē ki chūta!, What a run through the trees in the dark!)</td>
    <td>অদ্ধকারে গাছপালা অফকারে গাছপালা Dictionary দেখে করে সে কি দেখে করে সে কি দেখে? (andhakārē gāchāpālā bhēda karē sē ki chūta!, What a run through the trees in the dark!)</td>
    <td></td>
  </tr>
</table>

Table 4: Example of sentences generated by noise-injection method present in Vaiyākarana.
7 Evaluation

In this section, we evaluate the effectiveness of our rule-based noise-injection method and the instruction-tuned dataset Vaiyākarana by evaluating large language models (LLMs) on grammatical error detection and correction, as well as on a separate task of paraphrasing.

7.1 Efficacy of the Rule-based Method

We prompted GPT-4o with the instruction “ব্যাকরণ-নত ফুল বাকা লেখ” (byākaraṇanagata bhula bākya lēkha, Write a grammatically incorrect sentence) and analyzed the generated outputs. From a set of 50 generated sentences, each containing less than five words, we observed that it generated 9 correct sentences and 41 incorrect sentences. Among the 41 incorrect sentences, 32 sentences were of Tense and Person errors, whereas the rest are Spelling errors. Notably, although spelling errors are generally the most frequent error type in Bangla, GPT-4o generated a disproportionate number of tense and person errors, ignoring all the other categories of grammatical errors.

We repeated the same experiment with a minimum word length constraint of at least 8. We observed that GPT-4o generated 11 correct sentences and 39 incorrect sentences. All 39 incorrect sentences contained either Tense or Person errors only. Additionally, GPT-4o misclassified 3 incorrect sentences as grammatically correct.

We further experimented with back-translation and round-trip translation for generating erroneous sentences. While 23 out of 50 sentences in the back translation were grammatically correct, the corresponding number for round-trip translation was even higher—40. The common error types in these cases were Spelling (non-dictionary words), Tense, and Person while other error categories were practically non-existent.

These experiments indicate that LLMs like GPT-4o struggle to generate grammatically incorrect sentences reliably. Even when incorrect sentences are generated, the types of errors are limited in diversity, and are predominantly tense and person errors. In contrast, our proposed rule-based noise injection approach involves injecting errors based on a set of predefined rules, ensuring a more diverse range of error types. Also, it guarantees generation of incorrect sentences. Further, even when LLM-generated sentences are erroneous, categorizing and localizing the errors remains a significant challenge. This validates the necessity and effectiveness of our controlled rule-based error generation framework for Bangla.

7.2 Grammatical Error Detection and Classification

We first test the abilities of humans and LLMs on error classification. We have segregated this into three types. The first is binary classification, where the task is to indicate if a given sentence is grammatically correct or wrong. The next two tasks are on classifying an erroneous sentence into the type of error it has. Assuming the correct sentence to be another class, the task is to either classify broadly into 5+1 broad classes or in a fine manner into 12+1 classes.

For human evaluation, we developed an interface where participants could mark the error class for a given sentence, including the option for the correct classification. We enlisted the help of 12 Bangla speakers, each of whom was assigned a set of 50 non-overlapping sentences. These sentences were randomly selected from a pool of 2,500 sentences from the Vaiyākarana dataset, comprising 650 correct sentences and 1,850 incorrect ones. A very high score on the part of humans would indicate that the dataset is very easy to classify, and may not be realistic. Otherwise, the wrong sentences generated by our noise injection method would be realistic and non-trivial.

Table 5 shows the results of encoder-decoder based transformer models against humans for the three kinds of classification tasks on these 600 sentences. The mean macro-F1 scores achieved by the 12 evaluators for the three classification tasks were between 82% and 89%. The highest macro-F1 scores recorded by an individual evaluator were 91.10%, 87.50%, and 83.33% for the binary, broad and fine classification tasks respectively. The high macro-F1 scores indicate that humans can identify and categorize the errors fairly well, which indicates that the synthetic data is following the naturally occurring error trends. However, the scores are not very high, thereby implying the test sentences are realistic and not trivial to correct.

These results do not include decoder-based models since they are known to be not good for classification tasks (Nielsen et al., 2025). Nevertheless, we tried using them for the simplest task—that of binary classification. Results shown in Table A8 in Appx. K confirm that their performance is indeed not up to the mark. Hence, we have not
<table>
  <tr>
    <th>Model</th>
    <th>Parameters</th>
    <th>Binary</th>
    <th>Broad</th>
    <th>Fine</th>
  </tr>
  <tr>
    <td>Google-ByT5</td>
    <td>300M</td>
    <td>81.25±0.65</td>
    <td>79.30±0.87</td>
    <td>76.65±0.80</td>
  </tr>
  <tr>
    <td>BanglaT5</td>
    <td>277M</td>
    <td>88.90±0.10</td>
    <td>84.50±0.08</td>
    <td>82.48±0.14</td>
  </tr>
  <tr>
    <td>Panini</td>
    <td>70.46M</td>
    <td>89.25±0.25</td>
    <td>84.75±0.15</td>
    <td>82.88±0.08</td>
  </tr>
  <tr>
    <td>Human</td>
    <td>–</td>
    <td>88.30±3.46</td>
    <td>84.20±3.75</td>
    <td>82.30±3.90</td>
  </tr>
</table>
Table 5: Macro-F1 for error classification on 600 sentences.

continued with our evaluation for multi-class scenarios using decoder-based models.

For each of the other transformer models, we employed 5-fold cross validation. When the transformers are tested on a zero-shot setting, they failed to perform any classification, and simply classified every sentence into 1 class. Fine-tuning these same models with our Vaiyākaraṇa dataset makes them achieve results that are at par with humans. Panini achieves the highest mean score and the best macro-F1 score for both multi-class and binary classification tasks. On average, both Panini and Bangla-T5 outperform human evaluators, although the maximum scores achieved by individual humans are higher.

We next evaluated the performance of various neural models including multilingual LLMs such as GPT-4o (OpenAI et al., 2023), GPT-2-XL (Black et al., 2022), Bloom-1.1B (Workshop et al., 2023), BanglaT5 (Hossain et al., 2023), and Google-ByT5. Each model was run five times for at least 20 iterations, utilizing different seed values, and we report the means and standard deviations in Table 6. Details regarding the hyperparameters for all models can be found in Appx. M.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Zero-Shot</th>
    <th colspan="3">With Fine-tuning</th>
  </tr>
  <tr>
    <th>GLEU</th>
    <th>F<sub>0.5</sub></th>
    <th>BERE-score</th>
    <th>BLEU</th>
    <th>F<sub>0.5</sub></th>
    <th>BERE-score</th>
    <th>BLEU</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>70.40</td>
    <td>81.85</td>
    <td>46.40</td>
    <td>73.66</td>
    <td>83.53</td>
    <td>86.86</td>
    <td>51.45</td>
  </tr>
  <tr>
    <td>GPT-2-XL</td>
    <td>62.85</td>
    <td>58.03</td>
    <td>77.90</td>
    <td>69.05</td>
    <td>57.27</td>
    <td>83.77</td>
    <td>59.10</td>
  </tr>
  <tr>
    <td>BLOOM-1.1B</td>
    <td>61.45</td>
    <td>54.43</td>
    <td>75.86</td>
    <td>66.50</td>
    <td>57.77</td>
    <td>79.24</td>
    <td>46.60</td>
  </tr>
  <tr>
    <td>BanglaT5</td>
    <td>62.55</td>
    <td>54.00</td>
    <td>76.82</td>
    <td>41.67</td>
    <td>68.88</td>
    <td>87.01</td>
    <td>81.50</td>
    <td>48.90</td>
  </tr>
  <tr>
    <td>Google-ByT5</td>
    <td>64.90</td>
    <td>56.17</td>
    <td>74.40</td>
    <td>64.10</td>
    <td>50.30</td>
    <td>80.30</td>
    <td>50.45</td>
  </tr>
  <tr>
    <td>Panini</td>
    <td>64.90</td>
    <td>56.17</td>
    <td>76.90</td>
    <td>40.80</td>
    <td>71.10</td>
    <td>60.50</td>
    <td>81.70</td>
    <td>50.35</td>
  </tr>
  <tr>
    <td>LLaMA3-BB</td>
    <td>66.52</td>
    <td>59.13</td>
    <td>79.84</td>
    <td>44.63</td>
    <td>71.49</td>
    <td>61.91</td>
    <td>85.22</td>
    <td>50.77</td>
  </tr>
</table>
Table 6: Performance of models with and without fine-tuning with Vaiyākaraṇa on grammatically correct sentence generation on 52,100 generated sentences.

the gold standard sentence. However, some correct answers deviated more, and the maximum deviation was 4 words (the average sentence length is around 10 words).

We created a test corpus of 52,100 sentences (details of the sentences are in Appx. L) which were not present in Vaiyākaraṇa and tested the ability of the models to generate correct grammatical sentences in two conditions, one without instruction tuning (zero-shot), and the other after instruction tuning the models with Vaiyākaraṇa. The hyperparameters used for instruction-tuning the models are described in Appx. N. Table 6 shows the performance of models in generating grammatically correct sentences with and without Vaiyākaraṇa. There is an average increase of more than 5% on \( F_{0.5} \) score as well as GLEU. In all the architectures, the average \( F_{0.5} \) and GLEU increased from the zero-shot paradigm. It indicates the effectiveness of Vaiyākaraṇa in building better GEC models for Bangla. In addition to the standard GEC metric GLEU, we used the other metrics since they are proposed in the CoNLL 2013 task (Ng et al., 2013) and Panini (Hossain et al., 2023).

To evaluate the quality of Vaiyākaraṇa, we experimented with generating grammatically correct sentences with generative models. In this experiment, we tested these models’ performance on 2,576 manually written sentences obtained from essay writing surveys that were not included in Vaiyākaraṇa. Like our previous experiments, we assessed the generative models in zero-shot and after instruction-tuning with Vaiyākaraṇa. Table 7 demonstrates that the performance of all models and architectures improved by 3-5 percentage points after instruction-tuning with Vaiyākaraṇa. Thus, the same trends hold across both the generated and the manual sentences.

Fig. 1 shows the GLEU score for each of the 12 error categories for GPT-4o on the 1513 manually written sentences. Fig. 3 of Appx. L, on the other hand, shows the GLEU score for each of the
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Zero-Shot</th>
    <th colspan="3">With Fine-tuning</th>
  </tr>
  <tr>
    <th>BLEU</th>
    <th>F1</th>
    <th>BERT-score</th>
    <th>BLEU</th>
    <th>F1</th>
    <th>BERT-score</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>72.40</td>
    <td>82.55</td>
    <td>48.50</td>
    <td>73.66</td>
    <td>83.15</td>
    <td>55.15</td>
  </tr>
  <tr>
    <td>GPT-2-XL</td>
    <td>64.85</td>
    <td>58.53</td>
    <td>70.16</td>
    <td>70.85</td>
    <td>61.30</td>
    <td>44.65</td>
  </tr>
  <tr>
    <td>BLOOM-1B</td>
    <td>63.75</td>
    <td>53.85</td>
    <td>77.56</td>
    <td>68.60</td>
    <td>58.40</td>
    <td>81.35</td>
    <td>48.80</td>
  </tr>
  <tr>
    <td>BanglaT5</td>
    <td>65.75</td>
    <td>59.20</td>
    <td>79.54</td>
    <td>47.50</td>
    <td>68.50</td>
    <td>59.80</td>
    <td>83.10</td>
    <td>51.40</td>
  </tr>
  <tr>
    <td>Google-ByT5</td>
    <td>62.60</td>
    <td>56.20</td>
    <td>76.15</td>
    <td>66.50</td>
    <td>58.10</td>
    <td>81.60</td>
    <td>52.20</td>
  </tr>
  <tr>
    <td>Panini</td>
    <td>66.80</td>
    <td>57.40</td>
    <td>79.50</td>
    <td>47.40</td>
    <td>71.20</td>
    <td>72.30</td>
    <td>85.00</td>
    <td>52.25</td>
  </tr>
  <tr>
    <td>LLaMA3-SB</td>
    <td>68.52</td>
    <td>59.39</td>
    <td>76.13</td>
    <td>46.48</td>
    <td>73.24</td>
    <td>62.94</td>
    <td>87.26</td>
    <td>53.31</td>
  </tr>
</table>

Table 7: Performance of models with and without fine-tuning with Vaiyākaraṇa on grammatically correct sentence generation on 2,576 manual sentences.

![Bar chart showing GLEU score of GPT-4o on handwritten sentences, comparing Scenario 1 (Zero-shot) and Scenario 2 (Instruction-tuning)](page_184_370_495_312.png)

Figure 1: Performance of GPT-4o on different error categories in Bangla on 2,576 manual sentences.

![Bar chart showing GPT-4o performance by error category on different input sentences for that category](page_774_370_495_312.png)

Figure 2: Performance of GPT-4o on different error categories on different input sentences for that category.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Zero-Shot</th>
    <th colspan="2">With Fine-tuning</th>
  </tr>
  <tr>
    <th>BLEU</th>
    <th>BERT-score</th>
    <th>BLEU</th>
    <th>BERT-score</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>49.00</td>
    <td>62.70</td>
    <td>57.60</td>
    <td>66.80</td>
  </tr>
  <tr>
    <td>GPT-2</td>
    <td>46.38</td>
    <td>57.58</td>
    <td>50.74</td>
    <td>62.41</td>
  </tr>
  <tr>
    <td>BLOOM</td>
    <td>45.50</td>
    <td>55.86</td>
    <td>49.00</td>
    <td>59.50</td>
  </tr>
  <tr>
    <td>BanglaT5</td>
    <td>48.66</td>
    <td>57.07</td>
    <td>55.38</td>
    <td>62.53</td>
  </tr>
  <tr>
    <td>Google-ByT5</td>
    <td>45.88</td>
    <td>56.52</td>
    <td>56.35</td>
    <td>61.42</td>
  </tr>
  <tr>
    <td>Panini</td>
    <td>48.46</td>
    <td>57.50</td>
    <td>55.56</td>
    <td>62.75</td>
  </tr>
  <tr>
    <td>LLaMA3-SB</td>
    <td>47.68</td>
    <td>60.11</td>
    <td>54.12</td>
    <td>64.59</td>
  </tr>
</table>

Table 8: Performance of models with and without fine-tuning with Vaiyākaraṇa on paraphrasing

12 error categories for the best performing model, GPT-4o, on the 20,100 sentences used for GEC evaluation of transformer models.

Appx O shows qualitative examples of performance of human and GPT-4o for GEC in Bangla.

7.4 Performance of LLMs on Error Categories

We evaluated the performance of GPT-4o (which is the best-performing model) for each error class after instruction tuning with different sizes for each category on the Vaiyākaraṇa dataset. The detailed performance of GPT-4o for each error class is shown in Fig. 2. It indicates that gpt4-o struggles to correct Spelling errors (specially Homonym errors), Case errors, POS errors, and Gurucandālī Dōsa. It fairly struggles in correcting Missing Word and Punctuation and Semantic errors, while it is quite good in correcting Number, Gender, Tense and Person errors.

7.5 Paraphrasing

To demonstrate the quality of Vaiyākaraṇa as a corpus, we evaluated the performance of neural models on a completely different task, that of paraphrasing, before and after instruction-tuning with Vaiyākaraṇa. We used the paraphrasing dataset developed by (Akil et al., 2022) containing 5,763 sentences. Table 8 shows the performance of models with and without instruction-tuning using Vaiyākaraṇa. The models’ performances were enhanced by 3-7 percentage points after instruction-tuning with Vaiyākaraṇa.

8 Conclusions and Future Work

In this paper, we proposed a rule-based noise injection methodology for generating grammatically wrong sentences in Bangla. We generated erroneous sentences across 12 categories, which is the most extensive categorization of grammatical errors for Bangla. We curated a dataset Vaiyākaraṇa consisting of 2,27,119 wrong and 5,67,422 total sentences. We also collected a set of 2,576 sentences (of which 1,045 were grammatically wrong) from manually written essays. The results show that neural models perform similar to human evaluators in detecting error categories and words whereas humans outperform these models in correcting grammatically wrong sentences. After instruction-tuning with Vaiyākaraṇa, the performance of LLMs improve by 3-7 percentage points on both these tasks. We have released Vaiyākaraṇa, the Alpaca format of Vaiyākaraṇa, manual hand-written data and code for the rule-based noise injection methodology of the paper under a non-commercial license at https://github.com/Bangla-itk/Vaiyakarana.

In future, we would like to use this dataset to develop better GEC models with explainability. Further, this methodology can be applied to generate benchmarks for most other major Indian languages, since their grammatical structures are similar to Bangla.
9 Limitations

Curating a large quality benchmark for GEC requires a good quality lemmatizer and POS tagger. Bangla suffers from a lack of quality lemmatizers and POS taggers. Hence, we had to manually add words from a grammar book (Chakroborty, 2018).

Also, hand-written Bangla data is not readily available. We conducted a survey for several weeks and could still collect only 2,576 hand-written sentences.

Finally, while the 12 human evaluators are all native speakers of Bangla, evaluating against Bangla grammarians could have given us more insights into the process. We are planning to do that in the future.

Additionally, we have evaluated only GPT-4o and not other commercially available models like Claude and Gemini due to resource constraints. In contrast, we have experimented with different architectures and loss functions to show the generalizability of the results.

10 Ethics Statement

The Vaiyākaraṇa benchmark is curated by merging sentences from Vacaspati corpus (Bhattacharyya et al., 2023) and (Chakroborty, 2018). The authors of Vacaspati provided us with the corpus, and (Chakroborty, 2018) is publicly available. Hence, there is no copyright infringement in curating Vaiyākaraṇa. We have made efforts to ensure that Vaiyākaraṇa is also devoid of any objectionable statements. We have also conducted a manual essay writing survey for gathering real word errors. The participants have kindly allowed us to use their essays for research purpose.

Acknowledgements

We thank Arghya Mukherjee, a PhD student in the department of Mathematics and Statistics, IIT Kanpur, for his unconditional support that played a pivotal role in the completion of the project. We also acknowledge the SERB Core Research Grant (file no. CRG/2023/006064) and the Research-I Foundation (RIF), Dept. of Computer Science and Engineering, IIT Kanpur grants, which were essential for completion of the project.

References

Ajwad Akil, Najrin Sultana, Abhik Bhattacharjee, and Rifat Shahriyar. 2022. BanglaParaphrase: A high-quality Bangla paraphrase dataset. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 261–272, Online only. Association for Computational Linguistics.

Md. Jahangir Alam, Naushad UzZaman, and Mumit Khan. 2007. N-gram based statistical grammar checker for bangla and english.

Pramit Bhattacharyya, Joydeep Mondal, Subhadip Maji, and Arnab Bhattacharya. 2023. VACASPATI: A diverse corpus of Bangla literature. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1118–1130, Nusa Dua, Bali. Association for Computational Linguistics.

Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. Gpt-neox-20b: An open-source autoregressive language model. Preprint, arXiv:2204.06745.

Adriane Boyd. 2018. Using Wikipedia edits in low resource grammatical error correction. In Proceedings of the 2018 EMNLP Workshop W-NUT: The 4th Workshop on Noisy User-generated Text, pages 79–84, Brussels, Belgium. Association for Computational Linguistics.

Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019. The BEA-2019 shared task on grammatical error correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52–75, Florence, Italy. Association for Computational Linguistics.

Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. 2023. Grammatical error correction: A survey of the state of the art. Computational Linguistics, page 159.

Bamandev Chakroborty. 2018. Uchchatara Bangla Byakaran, volume 2nd. Akshay Malancha.

Sam Davidson, Aaron Yamada, Paloma Fernandez Mira, Agustina Carando, Claudia H. Sanchez Gutierrez, and Kenji Sagae. 2020. Developing NLP tools with a new corpus of learner Spanish. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7238–7243, Marseille, France. European Language Resources Association.
Nahid Hossain, Mehedi Hasan Bijoy, Salekul Islam, and Swakkhar Shatabda. 2023. Panini: a transformer-based grammatical error correction method for bangla. Neural Comput. Appl., 36(7):34633477.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.

Budi Irmawati, Hiroyuki Shindo, and Yuji Matsumoto. 2017. Generating artificial error data for indonesian preposition error corrections. International Journal of Technology, 8(3):549–558.

Sadidul Islam, Mst. Farhana Sarkar, Towhid Hussain, Md. Mehedi Hasan, Dewan Md Farid, and Swakkhar Shatabda. 2018. Bangla sentence correction using deep neural network based sequence to sequence learning. In 2018 21st International Conference of Computer and Information Technology (ICCIT), pages 1–6.

Yiqiao Jin, Mohit Chandra, Gaurav Verma, Yibo Hu, Mummun De Choudhury, and Srijan Kumar. 2024. Better to ask in english: Cross-lingual evaluation of large language models for healthcare queries. In Proceedings of the ACM Web Conference 2024, WWW ’24, page 26272638, New York, NY, USA. Association for Computing Machinery.

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.

Katerina Korre and John Pavlopoulos. 2022. Enriching grammatical error correction resources for Modern Greek. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4984–4991, Marseille, France. European Language Resources Association.

Myunghoon Lee, Hyeonho Shin, Dabin Lee, and Sung-Pil Choi. 2021. Korean grammatical error correction based on transformer with copying mechanisms and grammatical noise implantation methods. Sensors, 21(8).

Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ali Payani, Ninghao Liu, and Mengnan Du. 2024. Quantifying multilingual performance of large language models across languages. Preprint, arXiv:2404.11553.

Jared Lichtarge, Chris Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar, and Simon Tong. 2019. Corpora generation for grammatical error correction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3291–3301, Minneapolis, Minnesota. Association for Computational Linguistics.

Subhankar Maity, Aniket Deroy, and Sudeshna Sarkar. 2024. How ready are generative pre-trained large language models for explaining bengali grammatical errors? Preprint, arXiv:2406.00039.

Jakub Näplava and Milan Straka. 2019. CUNI system for the building educational applications 2019 shared task: Grammatical error correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 183–190, Florence, Italy. Association for Computational Linguistics.

Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. JFLEG: A fluency corpus and benchmark for grammatical error correction. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 229–234, Valencia, Spain. Association for Computational Linguistics.

Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–14, Baltimore, Maryland. Association for Computational Linguistics.

Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013. The CoNLL-2013 shared task on grammatical error correction. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task, pages 1–12, Sofia, Bulgaria. Association for Computational Linguistics.

Dan Saattrup Nielsen, Kenneth Enevoldsen, and Peter Schneider-Kamp. 2025. Encoder vs decoder: Comparative analysis of encoder and decoder language models on multilingual nlu tasks. Preprint, arXiv:2406.13469.

Hellina Hailu Nigatu, Atanfu Lambebo Tonja, Benjamin Rosman, Thamar Solorio, and Monojit Choudhury. 2024. The zeno’s paradox of ‘low-resource’ languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17753–17774, Miami, Florida, USA. Association for Computational Linguistics.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, and Paul Baltescu et al. 2023. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Nabilah Oshin, Syed Hoque, Md Fahim, Amin Ahsan Ali, M Ashraful Amin, and Akmmahhubur Rahman. 2023. BaTEClacor: A novel dataset for Bangla text error classification and correction. In Proceedings of
the First Workshop on Bangla Language Processing (BLP-2023), pages 124–135, Singapore. Association for Computational Linguistics.

S. Rachel, S. Vasudha, T. Shriya, K. Rhtuja, and Lakshmi Gadhikar. 2023. Vyakaranly: Hindi grammar & spelling errors detection and correction system. In 2023 5th Biennial International Conference on Nascent Technologies in Engineering (ICNTE), pages 1–6.

Chowdhury Rahman, MD.Hasibur Rahman, Samiha Zakir, Mohammad Rafsan, and Mohammed Eunus Ali. 2023. BSspell: A CNN-blended BERT based Bangla spell checker. In Proceedings of the First Workshop on Bangla Language Processing (BLP-2023), pages 7–17, Singapore. Association for Computational Linguistics.

Gaoqi Rao, Qi Gong, Baolin Zhang, and Endong Xun. 2018. Overview of NLPTEA-2018 share task Chinese grammatical error diagnosis. In Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications, pages 42–51, Melbourne, Australia. Association for Computational Linguistics.

Marek Rei and Helen Yannakoudakis. 2017. Auxiliary objectives for neural error detection models. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 33–43, Copenhagen, Denmark. Association for Computational Linguistics.

Alla Rozovskaya and Dan Roth. 2019. Grammar error correction in morphologically rich languages: The case of Russian. Transactions of the Association for Computational Linguistics, 7:1–17.

Sagor Sarker. 2021. BNLP: Natural language processing toolkit for Bengali language. arXiv preprint.

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational Linguistics.

Aiman Solyman, Zhenyu Wang, Qian Tao, Arafat Abdulgader Mohammed Elhag, Rui Zhang, and Zeinab Mahmoud. 2022. Automatic arabic grammatical error correction based on expectation-maximization routing and target-bidirectional agreement. Knowledge-Based Systems, 241:108180.

Ankur Sonawane, Sujeet Kumar Vishwakarma, Bhavana Srivastava, and Anil Kumar Singh. 2020. Generating inflectional errors for grammatical error correction in Hindi. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 165–171, Suzhou, China. Association for Computational Linguistics.

Oleksiy Syvokon, Olena Nahorna, Pavlo Kuchmiichuk, and Nastasia Osidach. 2023. UA-GEC: Grammatical error correction and fluency corpus for the Ukrainian language. In Proceedings of the Second Ukrainian Natural Language Processing Workshop (UNLP), pages 96–102, Dubrovnik, Croatia. Association for Computational Linguistics.

BigScience Workshop, .; Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili, Daniel Hesslow, Roman Castagné, Alexandra Sasha Lucioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, and Pawan Sasanka Ammanamanchi et al. 2023. Bloom: A 176b-parameter open-access multilingual language model. Preprint, arXiv:2211.05100.

Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. 2023. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. Preprint, arXiv:2312.12148.

Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 180–189, Portland, Oregon, USA. Association for Computational Linguistics.

Wangchunshu Zhou, Tao Ge, Chang Mu, Ke Xu, Furu Wei, and Ming Zhou. 2020. Improving grammatical error correction with machine translation pairs. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 318–328, Online. Association for Computational Linguistics.
Appendix

A  Word Order

Table A1 shows that all possible word order of sentence অমর গীতাকে ভালোবাসে। (amara gitākē bhālōvāsē.) is correct.

<table>
  <tr>
    <th>Original sentence</th>
    <td>অমর গীতাকে ভালোবাসে।<br>amara gitākē bhālōvāsē.</td>
  </tr>
  <tr>
    <th>Word order 1</th>
    <td>গীতাকে অমর ভালোবাসে।<br>gitākē amara bhālōvāsē.</td>
  </tr>
  <tr>
    <th>Word order 2</th>
    <td>গীতাকে অমর ভালোবাসে।<br>gitākē amara bhālōvāsē.</td>
  </tr>
  <tr>
    <th>Word order 3</th>
    <td>অমর ভালোবাসে গীতাকে।<br>amara bhālōvāsē gitākē.</td>
  </tr>
  <tr>
    <th>Word order 4</th>
    <td>ভালোবাসে অমর গীতাকে।<br>bhālōvāsē amara gitākē.</td>
  </tr>
  <tr>
    <th>Word order 5</th>
    <td>গীতাকে অমর ভালোবাসে।<br>gitākē amara bhālōvāsē.</td>
  </tr>
  <tr>
    <th>Word substitution</th>
    <td>শ্যামল গীতাকে ভালোবাসে।<br>śyāmala gitākē bhālōvāsē.</td>
  </tr>
  <tr>
    <th>Word deletion</th>
    <td>অমর ভালোবাসে।<br>amara bhālōvāsē.</td>
  </tr>
  <tr>
    <th>Word insertion</th>
    <td>অমর গীতাকে খুব ভালোবাসে।<br>amara gitākē khuva bhālōvāsē.</td>
  </tr>
</table>

Table A1: Word order shuffling, substitution, deletion and insertion may not necessarily generate wrong sentences in Bangla.

B  Related Work

In this section, we discuss GEC in English and other non-Indian languages.

English: CoNLL-shared task 2013(Ng et al., 2013) and CoNLL-shared task 2014(Ng et al., 2014) played a pivotal role in advancing GEC works in English. Other than providing 55,000+ grammatically incorrect sentences in English, they also categorized grammatical errors in English into 5 broad classes and 27 finer classes. Napoles et al. (2017) presented a parallel corpus of 1,511 sentences for English representing a wide range of language proficiency. It incorporates holistic edits that make the original text sound more native. Additionally, Yannakoudakis et al. (2011) curated a collection of 1,238 scripts from distinct learners. The BEA-2019 shared task (Bryant et al., 2019) contributed a benchmark dataset of 43,169 sentences curated from the Write&Improve-LOCNESS corpus. This represents a broader range of native English learners.

Other Languages: Unlike English, low-resource Asian languages suffer from the unavailability of large corpora for neural models. Attempts have been made to enrich resources for GEC in many languages: Spanish (Davidson et al., 2020), German (Boyd, 2018), Russian (Rozovskaya and Roth, 2019), Czech (Náplava and Straka, 2019), Greek (Korre and Pavlopoulos, 2022), and Chinese (Rao et al., 2018). Syvokon et al. (2023) presented a corpus annotated for GEC and fluency edits for Ukrainian. Lee et al. (2021) gave four different noising methods, such as grapheme-to-phoneme noising rules and, heuristic-based noising rules, and others, to generate incorrect sentences for Korean. Lichtarge et al. (2019) proposed a rule-based system for deliberately injecting noises for low-resource languages like Indonesian (Irmawati et al., 2017). Solymán et al. (2022) proposed semi-supervised noising methods to generate 13,333,929 synthetic parallel examples from a monolingual corpus for Arabic.

C  Human Evaluation for Hindi

We conducted a survey for Hindi grammatical correction to evaluate whether the error injection methodology can be extended to other Indian languages. Five native speakers participated in this survey. Each participant was provided with 40 randomly selected sentences from a pool of 250 sentences (both correct and incorrect). The other setting for this experiment is similar to the Bangla evaluation described in Sec. 7.2.

The average macro-F1 score for all three classification tasks is 83%, 74% and 63%, with the highest being 87%, 87%, and 73.33%. All the participants opined that the generated sentences were confusing and that the sentences identified as incorrect were indeed wrong. This supports our assertion that our method for generating incorrect sentences for Bangla can also be effectively used for other Indian languages, such as Hindi.

D  Grammar Error Categories

This section expands upon Sec. 3 by illustrating the various error categories with examples. Grammatical errors in Bangla can be classified into five broader categories, which are further detailed into twelve specific distinctions. A sentence may contain multiple errors from the same category or from different categories. Table 1 provides example sentences for each of the error classes.
D.1 Spelling Errors

Spelling errors are amongst the most frequent types of errors. In Bangla and major Indian languages, there are almost similar sounding consonants and, thus, mistakes between ন / ণ (n / ṇ), শ / ষ / স (ś / ś / s), র / ঱ / রং (r / ṛ / ṛh), etc. are prominent among even the native speakers. Spelling errors are further classified into 2 types.

1. Non-Dictionary Words: Spelling errors of this type result in words that are not in a dictionary. We have considered Vācaspati (Bhattacharyya et al., 2023) as the vocabulary of Bangla words since it covers literary works of almost 8 centuries and works from both India and Bangladesh. In the example shown in Table 1, কাজ (kāja) gets changed to কাব (kāva) which is not a word.

2. Dictionary Words: A spelling error of this type produces another word which is in the dictionary. However, in the context of the sentence, it is an error. For example, in Table 1, changing রং (ṛ) of বারি (vāri) to র (r) produces a perfect word বারি (vāri). The sentence, however, ceases to have any valid meaning. Mostly these errors are of homonym types, i.e., similar sounding words. Simple non-homonym typos may, however, also result in a dictionary word শার্ট (sārṭ) that does not make sense in the sentence, as shown in the second example.

D.2 Word Errors

A prominent class of grammatical errors in almost any language including Bangla is word errors. We have categorized word errors further into different sub-classes as explained next.

1. Tense Error: In Bangla, like most other languages, there are specific verb forms for the three tenses. Failing to use the correct form leads to errors, as illustrated in the example in Table 1. Tense errors are particularly prevalent when multiple verbs are used within a single sentence, resulting in mismatches among the verb tenses. In the second example in the table, while the first verb আসবে (āsavē) is in future tense, the second verb ফুটেছিল (phutēchila) is in past tense.

2. Person Error: Similar to tenses, there are different verb forms and pronouns for different persons in Bangla. It is, thus, an error to use the wrong person of a verb. The sentence in Table 1 shows an example where instead of the first person form করি (kari), the third person form করে (karē) is used with the pronoun আমি (āmi, I). These errors are common in Indian languages.

3. Number Error: In Bangla, the verb forms for both singular and plural numbers are the same. However, there are distinct forms for pronouns. The example in Table 1 shows such a wrong usage where the singular form আমি (āmi) is used instead of the plural form আমরা (āmarā). Number errors are more common in other Indian languages compared to Bangla.

4. Gender Error: In Bangla, the verb forms and pronouns for different genders are the same. However, there are distinct forms for adjectives as well as nouns. Moreover, the gender and number of an the adjective should match that of the noun it qualifies. Hence, in the example in Table 1, since the proper noun উত্তম (uttama) is masculine, the correct adjective used should be the masculine form অভিনেতা (abhinētā) and not the feminine form অভিনেত্রী (abhinētrī). While strictly speaking, masculine forms of adjectives should not be used for feminine nouns, it is a common practice to accept them. In such sentences, the masculine form takes the role of a gender-neutral form. Hence, the sentence সুচিত্রা একজন অসাধারণ অভিনেতা। (sucitṛā ēkajana asādhāraṇa abhinētā.. Suchitra is an outstanding actor.) where সুচিত্রা (sucitṛā) is a feminine proper noun, but the adjective অভিনেতা (abhinētā) is masculine is not considered as incorrect. Many Indian languages, such as Hindi, have different forms of verbs for different genders and, thus, this kind of error is more common in those languages as compared to Bangla.

5. Case Error: Bangla and other Indian languages use a lot of inflected words. For different cases, different word forms are used that modify the original word. Case endings loosely correspond to prepositions in English. In the example in Table 1, the wrong case accusative is used instead of the correct case locative.

6. Parts-of-Speech Error: Sometimes, a word is used in a wrong parts-of-speech (POS). Since Indian languages, including Bangla, use a lot of nouns and their corresponding adjectives, these errors are common. Instead of a noun form, the adjective form is sometimes erroneously used, as shown in the example in Table 1.

7. Missing Word Error: These sentences are incomplete because of a missing word. Missing
a verb in Bangla will always generate this kind of error, as shown in the example in Table 1, while missing a random word may or may not be grammatically wrong. Missing a noun corresponding to its adjective will also generate an erroneous sentence. The second example in Table 1 shows such a sentence.

D.3 Mixing of Language Variants: Gurucandālī Dōṣā

Bangla has a unique temporal language feature. All written works in Bangla till the 19th century were exclusively in সাধু ভাষা (sādhū bhāśā, “refined language”). Authors started switching to চলিত ভাষা (calita bhāśā, “colloquial language”) during the 20th century and, currently, almost all the works are in this variant of the language. The two differ mostly in verb forms and pronouns and use exclusive sets of these. It is similar to the old English usage of “thou shalt” versus the modern “you shall”, etc., but is more elaborate. A sentence should be written in either of the variants. Thus, it is an error to mix, for example, pronouns of one variant with verbs of another. The example in Table 1 shows two cases. The sentence নন্দবাবু ইহা লক্ষ্য করেছেন। (nandavāvu ihā lakṣya karēchēna.) mixes the sādhū bhāśā pronoun form ইহা (ihā) with the calita bhāśā verb form করিয়াছেন (kariyāchēna). Either the verb form or the pronoun can be corrected, as shown in the examples. This mixing error is known as “গুরুচন্দালী দোষ” (Gurucandālī dōṣā) in Bangla.

D.4 Punctuation Errors

Punctuation errors occur due to the usage of wrong punctuation marks, absence of punctuation marks where needed, or spurious usage of punctuation marks. Thus, while a simple imperative sentence ends with a ! (full-stop mark), putting ? (interrogative mark) results in an error, as shown in Table 1.

D.5 Semantic Errors

Semantic error is a special class of error where the sentence’s semantic meaning becomes inconsistent or fictitious in the real world. For example, consider the sentence মানস আকাশ খেতে ভালোবাসে (mānas akāśa khētē bhālōbāsē.) which literally means “Manas loves to eat the sky.” Although this sentence is grammatically correct as far as the usage of words, spellings, etc. are concerned, it is still considered a wrong sentence due to its semantics. Note that this is for ordinary usage in a language, and such sentences may be correct in science fiction or other fantasy novels. Table 1 shows two correct sentences corresponding to the above wrong one. While in the first example, the verb is modified, in the second, the noun is modified to produce a semantically meaningful sentence.

D.6 Multiple Errors

These sentences suffer from multiple errors of the same category or a combination of different categories of errors. For example, the sentence আমরা বন্ধুরা গতকাল কাশীর যাবো। (āmarā bāndhurā gatakālā kāśmīra yābō.. We <non-word> will go to Kashi tomorrow.) consists of spelling errors (non-dictionary), person errors and tense errors. The correct sentence can be আমরা বন্ধুরা আগামীকাল কাশী যাবো। (āmarā bandhurā āgāmikālā kāśī yābā!. We friends will go to Kashi tomorrow.)

E Manual Generation

Table A2 provides the details of the essays given for the manual annotation survey. All 9 essays in this survey are commonly asked in 10th standard board exams. Each participant was asked to write an essay on a randomly picked topic. 36 participants undertook the study. Each annotator was paid on an hourly basis according to the standard rates prescribed by the university.

F Methodology

Table A3 shows example sentences generated by back-translating English sentence to Bangla, whereas and Table A4 shows example sentences generated by round-trip translation with English as bridge language.

G Generation of Different Types of Errors

• Spelling Errors: Spelling errors are those for which the original intention was to write the correct word, but some characters are wrongly written. Typically, the misspelled word should be within one or at most two edit distance from the original word. They can be, thus generated by substituting, inserting, or deleting one or two characters of a randomly chosen word in a sentence. These generated spelling errors may be of non-dictionary or dictionary types. We further collected 300 homonym word pairs from (Chakraborty, 2018). These homonyms are very
common in Bangla. We replaced the original word in sentences with their corresponding homonyms to generate dictionary-based spelling errors.

• Word Errors: We have followed different procedures to generate different types of word errors in Bangla.

1. Tense Error: We collected 24 most commonly used verbs and their forms across three tenses and three persons, resulting in 470 verb forms from (Chakroborty, 2018). This verb forms are replaced against the original word to generate erroneous sentences. These errors are difficult to generate since if the sentence contains only one verb and it is in its present form, then replacing it with past or future tense will not generate an error. For example in the sentence আমি বাড়ি যাব (āmi vārī yāva.) (I will go home) if we replace "যাব" (yāva) with "গিয়েছিলাম" (giyēchilāma) the resulting sentence আমি বাড়ি গিয়েছিলাম (āmi vārī giyēchilāma.) (I went to home) is not grammatically incorrect. So, to generate Tense errors, we need at least two verbs in the sentence, and we need to change only one, which we did randomly to generate Vaiyākarana. For example, রাম যখন পড়তে বসবে, তখন লিখবে (rāma yakhana paratē vasavē, takhana likhavē. (Ram will write when he sits down to read.) if we change one among the verbs "বসবে" (vasavē) or "লিখবে" (likhavē) the resulting sentences রাম যখন পড়তে বসবে, তখন লিখেছিল (rāma yakhana paratē vasavē, takhana likhēchila. (Ram wrote when he sits down to read.) and রাম যখন পড়তে বসেছিল, তখন লিখেছিল (rāma yakhana paratē vasēchila, takhana likhēchila.) (When Ram sat down to read, he wrote.) These issues compelled us to adopt the noise injection methodology to generate erroneous sentences, which would not have been possible if we had adopted other methodologies discussed in Sec: 2. There is one more type of tense error, which occurs with respect to time, like for sentence গতকাল আমি বাড়ি গিয়েছি-

ছিলাম (gatakālā āmi bārī giyēchilāma.) (I went to home yesterday) if changed to গতকাল আমি বাড়ি যাব (gatakālā āmi bārī yāba. (I will go to home yesterday) or আগমিকাল আমি বাড়ি গিয়েছিলাম (āgāmikālā āmi bārī giyēchilāma. (I went to home tomorrow) will generate grammatically incorrect sentences. We have crafted 15 such sentences manually and added them to Vaiyākarana.

2. Person Error: To generate these types of errors, we replaced the original verb form with its corresponding verb form from the other two types of persons.

3. Number Error: To generate this kind of error, we collected 23 pronouns with both of their singular-plural forms from (Chakroborty, 2018). We injected this error by deliberately replacing the original singular (respectively, plural) pronoun with its corresponding plural (respectively, singular) form. For pronoun detection, we used the POS tagger by Sarker (2021) since pronouns are typically a frozen list and taggers do well at detecting them.

4. Gender Error: We handcrafted 100 sentences for this kind of error. In each sentence, we chose a random word and changed its case. We employed three native speakers to validate the error category, and based on majority voting, we added the sentences in Vaiyākarana

5. Case Error: We collected a list of 20 cases and inflections and randomly interchanged them in the sentences to generate the wrong sentences.

6. POS Error: We collected 350 noun-adjective word pairs from (Chakroborty, 2018). We replaced a noun (respectively, adjective) with its corresponding adjective (respectively, noun) to generate errors.

7. Missing Word Error: We ran the POS tagger (Sarker, 2021) and deleted verbs from the sentence to generate erroneous sentences. We applied the same technique to delete the noun corresponding to its adjective to generate errors. For other cases, we randomly deleted some words from the sentences. We asked three native speakers to validate whether the generated sentence was an error, and based on majority voting, we marked the sentences. If it is an error, we add the sentence to Vaiyākarana. Else, we discard it.
• Semantic Error: We handcrafted 100 sentences for this kind of error. We employed three native speakers to validate the error category of the sentences, and based on majority voting, we added the sentences in Vaiyākarana.
• Gurucandālī Dōṣā: We collected 140+ verbs and pronouns with their corresponding sādhū and calita forms from (Chakroborty, 2018). We then replaced the original word with its counterpart to generate this kind of error. To generate these sentences, we make sure that at least one verb or pronoun retains its original form so that the resulting sentence is an error that mixes the two variants. For example আমি খেতে খেতে হাঁটছি (āmi khētē khētē hāṃṭachi.) we randomly changed "খেতে" (khētē) to "খাইতে" (khāitē) generating a wrong sentence আমি খাইতে খেতে হাঁটছি (āmi khāitē khētē hāṃṭachi.). Changing all three "খেতে" (khētē), "খেতে" (khētē) and "হাঁটছি" (hāṃṭachi) will lead to a grammatically correct sentence আমি খাইতে খাইতে হাঁটিতেছিলাম। (āmi khāitē khāitē hāṃṭiṭeṭhilām.).

Following these steps, we generated 2,26,869 grammatically incorrect sentences, as outlined in Table 3. Following the procedure outlined above, we can generate any number of grammatically incorrect sentences for Bangla.

H Data Cleaning

• Cleaning of Unicode characters: Unicode characters “0020” (space), “00a0” (no-break space), “200c” (zero width non-joiner), “1680” (ogham space mark), “180e” (mongolian vowel separator), “202f” (narrow no-break space), “205f” (medium mathematical space), “3000” (ideographic space), “2000” (en quad), “200a” (hair space) are separated from the texts.
• Cleaning of different punctuation marks: In Bangla, usage of punctuation marks has also evolved alongside words. In particular, we have treated the following as punctuation marks: “...”, “।..”, “।।”, “।-”, “–”.

I Statistical Comparison of Real and Synthetic Data

In this section, we assess whether the distribution of error categories in the manual analysis is similar to that in Vaiyākarana. We conducted this validation in three different ways. First, we performed a binary task to compare the distribution of correct and incorrect sentences in both the manual annotation and Vaiyākarana. The second task involved a multi-class validation across five broad error categories (excluding sub-classes) and their respective distributions. Lastly, we examined the distribution of finer classes (totalling 12) between the manual analysis and Vaiyākarana. For these validation tasks, we employed the Jensen-Shannon divergence and will discuss the significance of our findings.

Table A5 presents the JSD values for all three tasks, which are all less than 0.05. This finding suggests that the distribution of erroneous sentences and error categories in the manual data and Vaiyākarana are comparable. Therefore, we can use Vaiyākarana as a benchmark dataset for Grammatical Error Correction (GEC) in Bangla.

J Multiple Error Categories

We have observed from the manual survey that there are single errors in a sentence 77% of the time. The remaining 23% of the sentences contain multiple errors. Amongst them, the most common is multiple spelling errors in a sentence. We have categorised multiple errors of the same type in that category of error only, not inducted them in the multiple error category. Vaiyākarana has 3,860 such sentences. Very few sentences have more than one type of error. Here, we show the distribution of sentences having different types of errors in a sentence. We have only considered that category of errors prevalent in the human survey in Sec. 4. Table A6 show the distribution of multiple errors in Vaiyākarana.

K Results of Transformer-based Models

Table A7 shows the performance of all neural models and Random Forest classifier on the 5,67,422 sentences in Vaiyākarana.

We also evaluated decoder-based models to detect whether a sentence is grammatically correct. Table A8 presents the performance of decoder-based models for prompt with prompt ‘বাক্যটি সঠিক অথবা ভুল কিনা তা নির্ধারণ কর’ (bākyati satihika athabā bhula kina tā nirdhāraṇa kara., is this sentence grammatically correct?). Since the performance of binary classification is itself poor we have not continued with the multi-class classifications.
Figure 3: Figure showing performance of GPT-4o on different error categories in Bangla on 20,100 generated erroneous sentences.

L Grammatical Error Correction

We created a test set of 52,100 sentences to evaluate the efficacy of Vaiyākarana. The distribution of error categories on the test set is shown in Table A9.

Fig. 3 shows the GLEU score for each of the 12 error categories for GPT-4o on the 52,100 sentences used for GEC evaluation of transformer models.

M Model Hyper Parameters

We fine-tuned the transformer-based models with Adam optimizer (Kingma and Ba, 2015) and learning rate of 2e-5 for 20 epochs. Each transformer-based model’s batch size is 16, with a maximum length of 512.

N Hyperparameters for Instruction-tuning

All models were instruction-tuned using the Low-Rank Adaptation (LoRA) method (Hu et al., 2021), a parameter-efficient fine-tuning approach for pre-trained models (Xu et al., 2023), via the Hugging Face peft module. The LoRA hyperparameters were set as follows:
• Rank (r): 16
• LoRA alpha (\( \alpha \)): 32
• LoRA dropout: 0.05
• Bias: none

All other hyperparameters were set at their default values. For all the models, the following hyperparameter values have been used for generation:
• temperature=0.7
• top_k=50
• num_beams=10
• max_length=1800

The default values have been used for all other hyperparameters.

O Anecdotal Examples

Table A10 shows a few anecdotal examples of Human and GPT-4o (best-performing model) generated outputs on test sentences from Vaiyākarana dataset. In the first example, even though the generated output is grammatically correct, the edit distance between the generated sentence and ground truth is significant, resulting in a decreased GLEU score. The generated output and ground truth are the same in the second case. Hence, the GLEU score is 1.0.
<table>
  <tr>
    <th>Essay topic</th>
    <th># Essays</th>
  </tr>
  <tr>
    <td>বিজ্ঞান আধীনের না অভিশাপ</td>
    <td>15</td>
  </tr>
  <tr>
    <td>vijñāna āśīrvāda nā abhiśāpa<br>Is science boon or bane?</td>
    <td></td>
  </tr>
  <tr>
    <td>একটি বৃষ্টির দিন</td>
    <td>12</td>
  </tr>
  <tr>
    <td>ēkati vṛṣṭira dina<br>A rainy day</td>
    <td></td>
  </tr>
  <tr>
    <td>একটি নদীর আত্মকথা</td>
    <td>6</td>
  </tr>
  <tr>
    <td>ēkati nadira ātmakathā<br>Autobiography of a river</td>
    <td></td>
  </tr>
  <tr>
    <td>একটি স্মরণীয় দিন</td>
    <td>15</td>
  </tr>
  <tr>
    <td>ēkati smaraniyā dina<br>A memorable day</td>
    <td></td>
  </tr>
  <tr>
    <td>খেলা ও খেলা নয়</td>
    <td>12</td>
  </tr>
  <tr>
    <td>khēlā śūdhu khēlā naẏā<br>Sports is not just sports</td>
    <td></td>
  </tr>
  <tr>
    <td>হাত আলাদিনের আশ্চর্য প্রদীপ কুড়িয়ে পেলে কী করবে</td>
    <td>8</td>
  </tr>
  <tr>
    <td>hathāt alādinēra āścarya pradipa kuriyē pēlē kī karavē<br>What will you do if you suddenly get Aladdin’s lamp?</td>
    <td></td>
  </tr>
  <tr>
    <td>সামাজিক মাধ্যম আধীনের না অভিশাপ</td>
    <td>20</td>
  </tr>
  <tr>
    <td>sāmājika mādhyaṃ āśīrvāda nā abhiśāpa<br>Is social media boon or bane?</td>
    <td></td>
  </tr>
  <tr>
    <td>ঈন গ্রহের প্রাণী ও বোমার কথাপাকথন</td>
    <td>10</td>
  </tr>
  <tr>
    <td>bhinā grahēra prāṇi ō tōmāra kathōpakathana<br>Dialogue between an extraterrestrial being and you</td>
    <td></td>
  </tr>
  <tr>
    <td>পনেরো বছর বড় আগের তুমি আর আজকের তুমির মধ্যে কথাপাকথন</td>
    <td>15</td>
  </tr>
  <tr>
    <td>panērō vachara āgēra tumi āra ajakēra tumira madhyē kathōpakathana<br>Dialogue between 15-years older you and present you</td>
    <td></td>
  </tr>
  <tr>
    <td>একটি বটাগাছের আত্মকথা</td>
    <td>10</td>
  </tr>
  <tr>
    <td>ēkati vātagāchēra ātmakathā<br>Autobiography of a Banyan Tree</td>
    <td></td>
  </tr>
  <tr>
    <td>Total</td>
    <td>123</td>
  </tr>
</table>

Table A2: Grammar essays for manual survey

<table>
  <tr>
    <th>English</th>
    <th>Bangla</th>
    <th>Grammar</th>
  </tr>
  <tr>
    <td>He helped me succeed.</td>
    <td>তিনি আমাকে সফল করতে সাহায্য করেছেন। (tini amākē saphala karatē sāhāyya karēchēna.)</td>
    <td>Semantic</td>
  </tr>
  <tr>
    <td>I go to school.</td>
    <td>আমি স্কুলে যাই। (āmi skulē yāi.)</td>
    <td>Correct</td>
  </tr>
</table>

Table A3: Example of sentences generated by back translation from English to Bangla.

<table>
  <tr>
    <th>Bangla</th>
    <th>English</th>
    <th>Bangla</th>
    <th>Grammar</th>
  </tr>
  <tr>
    <td>নে আমার ভন্দু। (sē amārā bandhū.)</td>
    <td>He is my friend</td>
    <td>নে আমার ভন্দু। (sē amārā bandhūta.)</td>
    <td>Case</td>
  </tr>
  <tr>
    <td>নন্দালু এই লক্ষ্যে নন্দালু ব্যাপারটা লক্ষ্যে। (nandālubu ētā lakṣya nandā lābhu vyāpārata lakṣya karalēsa.)</td>
    <td>Nandalu noticed this.</td>
    <td>নন্দালু ব্যাপারটা লক্ষ্যে। (nandā lābhu vyāpārata lakṣya karalēsa.)</td>
    <td>Correct</td>
  </tr>
</table>

Table A4: Example of sentences generated by round-trip translation using English as bridge language.

<table>
  <tr>
    <th>Type</th>
    <th>JSD-Score</th>
  </tr>
  <tr>
    <td>Binary</td>
    <td>0.048</td>
  </tr>
  <tr>
    <td>Broad</td>
    <td>0.037</td>
  </tr>
  <tr>
    <td>Finer</td>
    <td>0.046</td>
  </tr>
</table>

Table A5: Jensen-Shannon Divergence score of distribution of manual analysis and Vaiyākaraṇa.

<table>
  <tr>
    <th>Error Class</th>
    <th>#Occurences</th>
  </tr>
  <tr>
    <td>Non_Dictionary-Dictionary</td>
    <td>1,860</td>
  </tr>
  <tr>
    <td>Case-Non_Dictionary-Dictionary</td>
    <td>500</td>
  </tr>
  <tr>
    <td>Gurucandali Dōsa-Non_Dictionary-Dictionary</td>
    <td>250</td>
  </tr>
  <tr>
    <td>Non_Dictionary-Dictionary-Punctuation</td>
    <td>250</td>
  </tr>
  <tr>
    <td>Punctuation-Non_Dictionary-Dictionary</td>
    <td>250</td>
  </tr>
  <tr>
    <td>Case-Non_Dictionary-Gurucandali Dōsa</td>
    <td>250</td>
  </tr>
  <tr>
    <td>Non_Dictionary-Dictionary-Gurucandali Dōsa-Punctuation</td>
    <td>250</td>
  </tr>
  <tr>
    <td>Non_Dictionary-Dictionary-Gurucandali Dōsa-Punctuation-Case-Missing Word</td>
    <td>250</td>
  </tr>
  <tr>
    <td>Total</td>
    <td>3,860</td>
  </tr>
</table>

Table A6: Distribution of Multiple Errors in Vaiyākaraṇa

<table>
  <tr>
    <th>Model</th>
    <th>Parameters</th>
    <th>Binary</th>
    <th>Broad</th>
    <th>Finer</th>
  </tr>
  <tr>
    <td>Google-ByT5</td>
    <td>300M</td>
    <td>81.25 ± 0.65</td>
    <td>79.30±0.87</td>
    <td>76.65±0.80</td>
  </tr>
  <tr>
    <td>BanglaT5</td>
    <td>270M</td>
    <td>88.90±0.10</td>
    <td>84.50±0.68</td>
    <td>82.48±0.14</td>
  </tr>
  <tr>
    <td>Panini</td>
    <td>70.46M</td>
    <td>89.25±0.25</td>
    <td>84.75±0.15</td>
    <td>82.88±0.08</td>
  </tr>
</table>

Table A7: Macro-F1 of transformer models on Vaiyākaraṇa.

<table>
  <tr>
    <th>Model</th>
    <th>Without Instruction Tuning</th>
    <th>With Instruction Tuning</th>
  </tr>
  <tr>
    <td>GPT-4.0</td>
    <td>40.30</td>
    <td>56.30</td>
  </tr>
  <tr>
    <td>GPT-2-XL</td>
    <td>38.10</td>
    <td>49.65</td>
  </tr>
  <tr>
    <td>OPT</td>
    <td>36.50</td>
    <td>48.30</td>
  </tr>
  <tr>
    <td>GPT-Neo</td>
    <td>36.38</td>
    <td>50.90</td>
  </tr>
  <tr>
    <td>BLOOM</td>
    <td>35.00</td>
    <td>48.00</td>
  </tr>
</table>

Table A8: Macro-F1 of different LLMs for prompts with and without instruction tuning for 600 sentences.

<table>
  <tr>
    <th>Error Class</th>
    <th>#Occurences</th>
  </tr>
  <tr>
    <td>Non-Dictionary Dictionary</td>
    <td>8,000</td>
  </tr>
  <tr>
    <td>Spelling Errors</td>
    <td>2,000</td>
  </tr>
  <tr>
    <td>Tense Errors</td>
    <td>1,000</td>
  </tr>
  <tr>
    <td>Person Errors</td>
    <td>1,000</td>
  </tr>
  <tr>
    <td>Number Errors</td>
    <td>40</td>
  </tr>
  <tr>
    <td>Gender Errors</td>
    <td>30</td>
  </tr>
  <tr>
    <td>Case Errors</td>
    <td>2,000</td>
  </tr>
  <tr>
    <td>POS Errors</td>
    <td>750</td>
  </tr>
  <tr>
    <td>Missing Words</td>
    <td>750</td>
  </tr>
  <tr>
    <td>Word Errors</td>
    <td>5,570</td>
  </tr>
  <tr>
    <td>Punctuation Errors</td>
    <td>2,000</td>
  </tr>
  <tr>
    <td>Semantic Errors</td>
    <td>30</td>
  </tr>
  <tr>
    <td>Gurucandali Dōsa</td>
    <td>2,000</td>
  </tr>
  <tr>
    <td>Multiple Errors</td>
    <td>500</td>
  </tr>
  <tr>
    <td>Different Categories</td>
    <td></td>
  </tr>
  <tr>
    <td><i>InCorrect</i></td>
    <td>20,100</td>
  </tr>
  <tr>
    <td><i>Correct</i></td>
    <td>32,000</td>
  </tr>
  <tr>
    <td><b>Total</b></td>
    <td>52,100</td>
  </tr>
</table>

Table A9: Distribution of different error categories in the test set of 52,100 sentences.
<table>
  <tr>
    <th>Test Sentence</th>
    <th>Correct</th>
    <th>Human</th>
    <th>GPT-4o (Zero-Shot)</th>
    <th>GPT-4o (Instruction Tuned)</th>
  </tr>
  <tr>
    <td>কানে গাইতেন বাজতেই লাগল।<br>(kānē gāitēṇa bājatēi lāgala.., incorrect sentence)</td>
    <td>কানে গানটা বাজতেই লাগল।<br>(kānē gānatā bājatēi lāgala.., The song kept ringing in my ears.)</td>
    <td>কানে গানটা বাজতেই লাগল।<br>(kānē gānatā bājatēi lāgala.., The song kept ringing in my ears.)</td>
    <td>কানে গাইতেন বাজতেই লাগল।<br>(kānē gāitēṇa bājatēi lāgala.., incorrect sentence)</td>
    <td>কানে ইয়ারফোন পরতেই গান বাজতে লাগল।<br>(kānē iyāraphōna parafēi gāna bājatē lāgala.., As soon as I put on the earphones, the music started playing)</td>
  </tr>
  <tr>
    <td>শিস দিতে গিয়ে দেখলেন, শব্দ হয়েছিল না তো!<br>(sīsa ditē giyē dēkhalēna, śabda hayēchila nā tō!, incorrect sentence)</td>
    <td>শিস দিতে গিয়ে দেখলেন, শব্দ হল না তো!<br>(sīsa ditē giyē dēkhalēna, śabda hala nā tō!, When he whistled, he saw that there was no sound!)</td>
    <td>শিস দিতে গিয়ে দেখলেন, শব্দ হল না তো!<br>(sīsa ditē giyē dēkhalēna, śabda hala nā tō!, When he whistled, he saw that there was no sound!)</td>
    <td>শিস দিতে গিয়ে দেখলেন, শব্দ হয়নি তো!<br>(sīsa ditē giyē dēkhalēna, śabda hayani tō!, wrong use of verb form)</td>
    <td>শিস দিতে গিয়ে দেখলেন, শব্দ হল না তো!<br>(sīsa ditē giyē dēkhalēna, śabda hala nā tō!, When he whistled, he saw that there was no sound!)</td>
  </tr>
</table>

Table A10: Anecdotal examples comparing human versus GPT-4o at both zero-shot and instruction-tuned

Reviewer paper summaries: The authors showcase a representative dataset for the GEC task in Bangla. Since real hand- and keyboard-written data are hardly available, the study suggests injecting errors into correct sentences from the corpus of Bangla fiction written in the 14th-21th cc. The generated data mimics the distribution of grammatical, semantic, spelling, punctuation, and modern vs. historical variants of the language in the pilot dataset of real-world errors. The synthetic dataset is used to instruction-tune a number of LLM models in GED, GEC, and paraphrasing tasks. This method improves the results by 3-7%.

The study tackled GEC for Bangla. Researchers introduced 12 classes of Bangla errors and surveyed native speakers to gather real-world examples. They introduced a rule-based method to inject grammatical errors into correct sentences, creating a dataset with over 500k sentences. Instruction-tuning LLMs on this dataset were used to assess GEC performance and an increase of 3–7 percentage points was observed.
Summary of strengths: * This is a well-organized, well-structured and very detailed paper about creating synthetic datasets for low-resource languages ​​for which there is not enough hand-labeled data. The choice of error labels is presented clearly and provided with examples. While in the GED task, instruct-tuning models allows reaching human-like performance, it is shown that there is still room for improvement in the GEC solutions. 
* The detailed report on the native speakers' performance in the GEC task including individual biases is interesting and can serve as a scheme for analysis in similar works in the field.

- Multiple experiments and valuable resources
- Human validation of the error injection process
Summary of weaknesses: * It is not evident whether the test part will be included in the Vaiyākaraṇa dataset as a separate part (cf. "the Vaiyākaraṇa dataset... is then used to instruction-tune LLMs for the task of GEC in Bangla" (lines 19-20), "We created a test corpus of 52,100 sentences ... which were not present in Vaiyākaraṇa" (lines 559-561)). Or do the authors plan to publish it as a separate dataset?
* Description of the particular methods for LLM instruction-tuning is missing 
* Section 7.3 (evaluation on the paraphrasing task with and without tuning with Vaiyākaraṇa) despite the fact that it is not without interest, is too short and does not add much to the main argument of the work. At the same time, a lot of information important for research is presented in the appendices.

- “We evaluate the effectiveness of instruction-tuned LLMs for GEC in Bangla against both zero-shot setting and human evaluators” – It seems normal that finetuning has better results than zero-shot, but this is not by a large margin. This does not prove the effectiveness of the dataset. Maybe try to compare finetuning on this dataset with finetuning on other GEC Bengali datasets (or other synthetic data generation methods) and observe the model performance. 
- Unfortunately, the method does not bring something new/innovative, a twist for GEC

Inquiry:
- There is already an arxiv paper (https://arxiv.org/pdf/2406.14284) from June 2024 introducing the dataset - what is the status of that study? what does this bring new, and how are they different?
Comments, suggestions and typos: * A coreference resolution problem in the following fragment: 
Lines 368-373: "The temporal diversity, in particular, is especially useful for Gurucan.d.a ̄l ̄i Do ̄s.a errors since that particular writing style started becoming rare from 1960s. (Newspaper, blog, and social media data are not 373 suitable **for this**)
* Vacaspati is written differently in Sections 6 and 10.
* Section 8 is more about "Conclusions and Future Work" rather than "Discussions"
* Table 4 would benefit from English translations 
* I would suggest including visualizations or detaled tables wrt the human performance in the GED and GEC tasks in appendices.

It would have been best to compute an agreement between the human annotators, to see the complexity of the task
What is Gurucandali Dosa as error class? It would be better to have an English name
- Clarify “Evaluations show that instruction-tuning with Vaiyākaraṇa improves GEC performance of LLMs” - this was done only for paraphrasing
- Paper structure needs improving - 5.1 and 5.2 were not actually considered in the final experiments; the insights are interesting, but summarizing Appendix F in the main text instead of failed experiments would have been more insightful
- Marking the best models in all tables would have been useful. GPT-4o expectedly has high performance due to extensive language exposure, but it would be interesting to emphasize more the situations when finetuning on your dataset surpasses GPT-4o in the zero-shot setup.
- I appreciate the effort put into introducing details in Appendices, but the paper needs to be self-contained. 14 appendices & 8 pages is simply too much
- An analysis on the size of the synthetic dataset to be used for fine-tuning would have been opportunistic. The current gain with fine-tuning is not large; what could have been done more?
Overall assessment scores: 4

2.5
Metareview score: 3
Metareview: This paper focuses on using Large Language Models (LLMs) for Bangla grammar error correction. It classifies Bangla grammar errors, creates the Vaiyākaraṇa dataset for fine-tuning LLMs, and evaluates the performance of models in Grammar Error Correction (GEC) and paraphrasing tasks.

All reviewers gave constructive and valuable suggestions. The authors can address most concerns raised by four reviewers. I suggest to accept this paper to the Findings of the ACL according to the new dataset for low-resource language and the detailed and interesting analysis.
Summary of reasons to publish: 1) This paper pioneered the classification of Bangla grammar errors, providing a foundation for related research.

2) It constructed a dataset that can be used to fine-tune LLMs for Bangla GEC.

3) The detailed report on the native speakers' performance including individual biases is interesting.