Prompting open-source and commercial language models for grammatical error correction of English learner text (2024)

Abstract: Thanks to recent advances in generative AI, we are able to prompt large language models (LLMs) to produce texts which are fluent and grammatical. In addition, it has been shown that we can elicit attempts at grammatical error correction (GEC) from LLMs when prompted with ungrammatical input sentences. We evaluate how well LLMs can perform at GEC by measuring their performance on established benchmark datasets. We go beyond previous studies, which only examined GPT∗ models on a selection of English GEC datasets, by evaluating seven open-source and three commercial LLMs on four established GEC benchmarks. We investigate model performance and report results against individual error types. Our results indicate that LLMs do not always outperform supervised English GEC models except in specific contexts – namely commercial LLMs on benchmarks annotated with fluency corrections as opposed to minimal edits. We find that several open-source models outperform commercial ones on minimal edit benchmarks, and that in some settings zero-shot prompting is just as competitive as few-shot prompting.

heading: 1 Introduction
text: Grammatical error correction (GEC) of second language learner English text is an important task in Educational AI. Its main applications include: i) enabling learners to receive instant feedback on their written work, ii) providing features for automarking, and iii) profiling learners’ grammatical knowledge in such a way as to facilitate personalised learning (Andersen et al., 2013; Yannakoudakis et al., 2018; Zaidi et al., 2019).
There is a long history of GEC research in the field of computational linguistics, developing from
rule-based to statistical approaches to neural network models, as has happened with other tasks in natural language processing (Bryant et al., 2023). Given the recent emergence of large language models (LLMs), such as OpenAI’s GPT∗ and Meta’s Llama LLMs, it is natural to ask how well they can perform at GEC and how they compare to existing state-of-the-art supervised approaches (Caines et al., 2023).
To answer this question, we aim to elicit minimal edit style corrections from LLMs through zeroshot and few-shot prompting. Minimal edit correction of text aims to resolve any grammatical errors in a text while staying as close as possible to the phrasing and lexical choices of the original. This is sometimes held up as a distinct alternative to fluency correction – where texts are rewritten for naturalness – though in reality the two annotation methods are not completely separable from each other (Bryant et al., 2023). The tendency so far has been to annotate GEC datasets with minimal edits only, and so that is the way past systems have been trained. This is an important point to note, as LLMs by default will output a transformative fluency correction of ungrammatical text (Coyne et al., 2023; Fang et al., 2023; Loem et al., 2023). In this paper, we attempt to prompt LLMs to perform minimal edit correction rather than fluency correction, so that the outputs are comparable to previous systems. Minimal edit corrections are additionally valuable within an educational setting, for example displaying grammatical errors to a learner (Yannakoudakis et al., 2018).
We evaluate three commercial and seven open source LLMs on four publicly available GEC benchmarks: CoNLL 2014 (Ng et al., 2014), the FCE Corpus (Yannakoudakis et al., 2011), JFLEG
11952
(Napoles et al., 2017), and Write&Improve + LOCNESS (W&I) from the BEA-2019 shared task (Bryant et al., 2019). This builds on previous work, which involved GPT∗ models only and at most three GEC benchmarks (Coyne et al., 2023; Fang et al., 2023; Loem et al., 2023). We find that some of our chosen open-source models outperform GPT3.5 Turbo on English GEC benchmarks which have been annotated in a minimal edit fashion. In contrast, GPT-3.5 performs best on a test set annotated with fluency corrections.
We evaluate several zero-shot and few-shot prompts, but find that different models require different styles of prompting. Some models appear to respond better to few-shot prompting than others, and certain prompt templates work best with a specific LLM and dataset rather than universally across the board. We provide strong empirical evidence that LLMs do not always outperform existing state-of-the-art, supervised GEC models – though the search space over LLMs, prompt templates, and few-shot learning is so great that our results can only be considered a building block in the full picture of LLM evaluation on English GEC. Our investigations can serve as a comparison point for future work on GEC with LLMs. We make our code, prompt templates, few-shot examples, and model predictions publicly available.1

heading: 2 Related work
text: Grammatical error correction (GEC). GEC is the task of returning an edited version of an input text such that any errors are corrected. It is a longstanding task in research on NLP for educational applications.
Thanks to large-scale annotation projects and the public release of labelled data, GEC systems can be built aiming at general correction of all error types. Machine translation GEC systems pioneered this general purpose approach, at first with statistical and later neural models (Brockett et al., 2006; Junczys-Dowmunt et al., 2018; Yuan and Bryant, 2021). More recently, edit-based approaches have been proposed in which corrections are applied on a sequence labelling (Omelianchuk et al., 2020) or sequence-to-sequence basis (Stahlberg and Kumar, 2020). Bryant et al. (2023) offers a comprehensive survey of the history and current state of GEC.
1https://github.com/chrisdavis90/ gec-prompting-public
GEC with large language models (LLMs). The term ‘large language model’ is currently used to refer to a variety of neural networks developed by a number of organisations and businesses. These models reached the mainstream media through GPT-3 and ChatGPT, and as a result there is now a widespread awareness of ‘generative AI’ – in particular relating to text generation – amongst the general public. OpenAI’s GPT∗ models feature in this paper, alongside others for comparison. We have selected ten open-source and proprietary LLMs for reasons described in Section 4.
Recent studies indicate that LLMs from OpenAI can be prompted to generate corrected versions of ungrammatical inputs. Wu et al. (2023) compares ChatGPT to Grammarly and GECToR (Omelianchuk et al., 2020) on a sample of 100 sentences from the CoNLL-14 test set. Coyne et al. (2023) compares GPT-3.5 and GPT-42 to two GEC systems on English benchmarks (Yasunaga et al., 2021; Liu et al., 2021), whilst Fang et al. (2023) compare ChatGPT with multiple baselines including TagGEC (Stahlberg and Kumar, 2021) and T5 (Rothe et al., 2021). Finally Loem et al. (2023) compare ChatGPT and GPT3.5 (text-davinci-003) with models trained on synthetic data (Grundkiewicz and JunczysDowmunt, 2019; Grundkiewicz et al., 2019).
Coyne et al. (2023); Fang et al. (2023); Loem et al. (2023) all perform evaluation of English GEC on some combination of the JFLEG, CoNLL-14 and W&I+LOCNESS test sets. They find that the GPT∗ models set new state-of-the-art (SOTA) performance on the JFLEG dataset in which the annotators were permitted to carry out naturalistic fluency rewrites (Napoles et al., 2017). However, they perform worse than SOTA on CoNLL-14 and W&I+LOCNESS, which are much larger, more popular datasets that were annotated on the basis of minimal edit corrections. Coyne et al. (2023) and Fang et al. (2023) found through further investigation that the GPT∗ models have a tendency to over-correct and make extraneous fluency edits. This explains why it is that they can score so highly on JFLEG but not on minimal edit data. Loem et al. (2023) meanwhile investigated the possibility of prompting for minimal edits rather than fluency rewrites, and obtained promising improvements which motivate further work.
2Specifically the text-davinci-003 GPT-3.5 model, and a GPT-4 model gpt-4-0314.
Both Wu et al. (2023) and Coyne et al. (2023) additionally carried out human evaluation to rate the output from each system and found a preference amongst human raters for the GPT∗ outputs because they were considered to be more fluent. They also found instances of under-correction in the reference sentences derived from human annotators: in other words the LLMs were able to catch and correct errors which had not been corrected by the original annotators. These human evaluations are tentative only, since they involve only small samples of 100 sentences at a time from each test set.
While more fluent corrections may be preferred by human evaluators, they may not aid language learners if they diverge too greatly from the original text. Existing annotation guidelines for error correction state that edits should be as minimal as possible so that the learner can be helped to express what they are trying to say, rather than told how to express it differently (which may otherwise discourage them); i.e. how to amend an error rather than avoid it (Nicholls, 2003). Consequently, although both minimal and fluent corrections may be valuable to different user groups, we focus on minimal corrections for educational applications in this paper.

heading: 3 Datasets
text: We compare model performance on four publicly available and well-known English language GEC datasets: CoNLL-14 Test (Ng et al., 2014), JFLEG Dev and Test (Napoles et al., 2017), FCE Dev and Test (Yannakoudakis et al., 2011), and W&I+LOCNESS Dev3 (Bryant et al., 2019). We additionally sample 2,000 sentences uniformly from FCE train and W&I+LOCNESS train to con-
3The test set for W&I+LOCNESS is not publicly available.
struct a development set in order to filter the set of prompt templates. Table 1 presents the number of sentences and tokens per dataset.
The CoNLL-14 test set contains 50 essays written by undergraduate students at the National University of Singapore on one of two topics. It has featured in multiple GEC studies, and new SOTA performance was reported by Zhou et al. (2023) in a recent paper describing decoding interventions.
JFLEG dev and test contain approximately 1.5k sentences randomly sampled from essays by learners of English of unknown proficiency levels, and corrected by crowdworkers. Annotators were permitted to make fluency corrections to the sentences: not just minimal edits for grammaticality. Stahlberg and Kumar (2021) achieved the current SOTA performance on JFLEG for a single system with their guided approach to synthetic generation of training data based on error type distributions found in annotated corpora.
FCE dev and test feature essays written by intermediate learners of English (CEFR levels B1 and B2). It is a subset of the Cambridge Learner Corpus and has also been used in multiple GEC studies. Current SOTA was established by Yuan and Bryant (2021) with a multi-encoder model which encodes a given sentence and the preceding one separately, integrating them in the decoder.
W&I+LOCNESS is a hybrid dataset made up of native speaker essays written by undergraduate students (LOCNESS; Granger (1998)) and essays submitted to the Write&Improve learning platform by learners of English at varying levels of proficiency (W&I). It was prepared for the BEA 2019 Shared Task on GEC (Bryant et al., 2019), and SOTA was achieved by Qorib et al. (2022) with system combination across multiple GEC models.
Each dataset was processed with ERRANT (Felice et al., 2016; Bryant et al., 2017), an automatic error annotation tool, in order to be standardised into a common format. Consequently the datasets are in tokenised M2 format, and we first need to detokenise them as LLMs expect untokenised inputs. To carry out this task, we use the Moses detokeniser4 and a rule-based heuristic to combine negative contractions which are not fully handled by the detokeniser5.
4https://github.com/luismsgomes/ mosestokenizer
5The detokeniser transforms token sequences such as “couldn ’t” to “couldn’t” in a satisfactory manner but sequences such as “could n’t” are missed.

heading: 4 Models
text: We evaluate three commercial and seven opensource LLMs. We include more open-source than commercial models as we assume that the latter will have a performance advantage and wish to investigate whether open-source models can perform in comparable ways. If so, this would be positive news from an open-science perspective. For the commercial LLMs, we include OpenAI’s GPT3.5-turbo and GPT-4 models (OpenAI, 2023), and Cohere’s Command model.6 Many more are available but due to budget constraints we work only with these three and we only evaluate GPT-4 in the zero-shot setting. We choose not to work with ChatGPT as it has been engineered to function as a chatbot.
For the open-source models, we select instruction-tuned models because the majority of our prompt templates contain instructions, and we evaluate the largest model from each model type that fits on a server with two A100 80GB NVIDIA GPUs. Our upper bound on model size relates to the computing resources available to us at the time of writing.
The open-source models are: OPT-IML-Max30B (Iyer et al., 2022), Llama-2-70B-chat (Touvron et al., 2023), Stable Beluga 2 (Mahan et al.), Falcon40B-Instruct (Almazrouei et al., 2023), Flan-T5XXL (Chung et al., 2022), BLOOMZ-7B1 (Muennighoff et al., 2022), InstructPalmyra-20B (Writer
6Both GPT-3.5-turbo and GPT-4 are the 0613 versions. Cohere’s Command is “v1”.
Inc., 2023). This is a representative sample of the models available, with a range of sizes and architectures. Approximate model sizes are given in Table 7 in the Appendix. We use HuggingFace (Wolf et al., 2020) to run the models and load them with float16 precision.7

heading: 4.1 Prompting LLMs for grammatical error correction
text: Prior work has shown that prompt format and wording can have a significant impact on task performance (Jiang et al., 2020; Shin et al., 2020; Schick and Schütze, 2021). We therefore evaluate and compare models across a selection of prompt templates (hereinafter prompts). In order to constrain the scope of experiments, we carry out two filtering and evaluation steps to construct and evaluate a set of zero- and few-shot prompts as follows.
We collect eleven zero-shot prompts based on a survey of NLP colleagues and related work.8 We first evaluate the zero-shot prompts with each model on a development set of 2,000 sentences sampled uniformly from the FCE and W&I+LOCNESS training sets. From these results we exclude four prompts with the lowest maximum scores, leaving seven prompts to evaluate in the zero-shot setting on the three development datasets: FCE, JFLEG, and W&I+LOCNESS.9
7We use bfloat16 for Falcon-40B-instruct. 8We considered a wide set of prompts used in related work but ultimately decided against their inclusion due to the estimated difficulty in replication and time/budget constraints.
9Details are provided in Appendix D.
We then select the 3 best-performing zero-shot prompts and create few-shot versions using 1, 2, 3, and 4 examples – 12 few-shot prompts in total. While related work samples few-shot examples, we make the decision to use a fixed set and order of examples to control the experimental parameters. A dynamic set of few-shot examples would require multiple samples per sentence in order to obtain a clear view of few-shot performance for each model. In addition, we evaluate models using the best performing few-shot prompt from Coyne et al. (2023).
Table 2 lists the prompts we evaluate in zeroand few-shot settings (see Appendix B for the complete set). Briefly, prompts MIN and DN contain general instructions to make minimal corrections, prompt ELT uses an “English language teacher” expert, TOOL uses a “grammatical error correction tool” expert, CYN is the prompt from Coyne et al. (2023), and CON frames the GEC instruction as a continuation. The set of few-shot examples are listed in Appendix Table 9.
Generation hyper-parameters We use the following settings for all models – we set temperature to 0.1, top-K to 50, and top-P to 1.0. Preliminary work has shown that lower temperature values result in better GEC performance (Coyne et al., 2023), and importantly, we want the model to make minimal edits and stay as close as possible to the original sentence. For some models the lowest temperature is 0.1, and so we set the parameter to this value to be constant across all models.
Evaluation As per the recommendations in Bryant et al. (2023), we evaluate the FCE and W&I corpus in terms of F0.5 using ERRANT (Bryant et al., 2017), the CoNLL-2014 test set in terms of F0.5 using the M2 scorer (Dahlmeier and Ng, 2012), and the JFLEG corpus using GLEU (Napoles et al., 2015).
Along with the open search space in prompt design, a practical question arises as to how much time and effort to dedicate to implement a modelor prompt-specific post-processing step to extract the generated hypothesis sentence from the model output. Due to the number and variety of models and prompts, it’s possible that each model–prompt combination will generate a different output format, and clearly, the quality of the post-processing step will impact evaluation measures. For all models, we replace all new line tokens with blank spaces, replace sequences of multiple spaces with a sin-
gle space, and remove all trailing quotation marks. We also remove strings from the start and end of sentences based on keyword matching – for example, we remove “Output sentence: ”, “Corrected sentence: ”, and “Input sentence: ” from the start of sentences. The output from Llama-2-chat was particularly noisy and required more rules – we detail our processing steps in Appendix C.

heading: 5 Results
text: Table 3 presents the top-1 results for each model on the development sets for the FCE, JFLEG and W&I+LOCNESS. From the models we test, the results show GPT-4 scores highest on every development dataset, though Stable Beluga 2 and GPT-3.5 Turbo obtain comparable performance to GPT-4 on JFLEG. Amongst the open-source models, Falcon40B-Instruct and Stable Beluga 2 have relatively high performance across the board, whilst Flan-T5 scores highly on FCE dev specifically.
Contrary to expectations set by previous work, adding few-shot examples to the three zero-shot prompts does not always lead to an improvement in performance. Indeed for FCE dev, zero-shot prompts perform best for most models. The picture is mixed for JFLEG dev, whilst the majority of models benefit from few-shot learning for W&I dev. It remains a matter for future work to investigate whether more dynamic approaches to data sampling (as opposed to a fixed selection of examples) will aid with few-shot GEC prompting.
Table 5 shows the performance of each model (except GPT-4) on the three test sets in our study: FCE, JFLEG and CoNLL-14. We compare with previous work on GEC with LLMs, and SOTA results from GEC-specific systems in the literature. For FCE and JFLEG, we use the prompt template that resulted in the best performance on the corresponding development set. For example, for GPT-3.5 Turbo on the FCE test set, we use the ELT zero-shot prompt because it resulted in the best performance on FCE dev. For CoNLL-14, we do the same based on model performance for W&I+LOCNESS dev.
Our LLM results are well short of SOTA performance, established by task-specific supervised models, for FCE test and CoNLL-14 test – the corpora annotated in minimal edit fashion – whereas the performance of GPT 3.5 Turbo is much closer to the SOTA on JFLEG test. These findings reinforce initial experiments by Coyne et al. (2023),
Fang et al. (2023) and Loem et al. (2023). It is apparent that supervised GEC systems, trained on each corpus, are best for minimal edit style corrections, whereas LLMs generate SOTA fluency corrections more similar to the style found in JFLEG.
We find that the four smallest models are biased towards precision over recall, while the larger models are more balanced (Table 4). The GPT∗ models have the best recall, which is a finding that deserves further investigation in future work.

heading: 5.1 Error type analysis
text: We use ERRANT to obtain the grammatical error types found in the W&I+LOCNESS development set – the largest development set we evaluate. ERRANT can identify 55 error classes. Table 6 presents F0.5 scores for the 18 most frequent error types for the three best performing models:
Falcon-40B-Instruct, Stable Beluga 2, and GPT-3.5 Turbo. Performance for each error type is comparable across the models, though GPT-3.5 Turbo is notably better at replacement punctuation errors.
Generally, the LLMs excel at spelling, missing determiners, replacement subject–verb agreement, replacement noun number, and orthography errors, while struggling on the open-class replacement of nouns and verbs, and the catch-all “other” error type. It seems that the LLMs perform better on morphological or character-based corrections which are not too distant from the original form, whereas lexical or phrasal replacement within the minimal edit paradigm are much more challenging.

heading: 6 Discussion
text: We set out to investigate the performance levels of LLMs on the task of English GEC. Previous work has shown that GPT∗ models could perform GEC with mixed success: outdoing existing SOTA models on the JFLEG dataset, which contains fluency corrections, whilst performing poorly on benchmarks annotated with minimal edit corrections – namely CoNLL-14, the FCE and W&I+LOCNESS (Fang et al., 2023; Loem et al., 2023; Coyne et al., 2023). We aimed to elicit minimal edit corrections through exploration of different prompting strategies, and evaluated models other than GPT∗ – including more open-source than commercial LLMs.
Our findings echo those in previous papers: our chosen LLMs perform well on JFLEG test – above
all Falcon-40B-Instruct, Stable Beluga 2 and GPT3.5 – though not outdoing SOTA, possibly because our prompts were designed to discourage fluency style corrections. Based on experiments with JFLEG dev, GPT-4 might perform best on JFLEG test, but full investigation of this question requires additional funding as GPT-4 is currently an order of magnitude more expensive than GPT-3.5 Turbo.
In contrast, the LLMs perform poorly on the FCE and CoNLL-14 test sets, lagging far behind SOTA in both cases. For these datasets, open-source models outperform or compete with the commercial models: the best performing model is FLAN-T5 on the FCE, and Stable Beluga 2 matches GPT 3.5 Turbo in the case of CoNLL-14. Again, performance on the FCE and W&I+LOCNESS dev sets suggests that GPT-4 could outperform the other LLMs on the test sets.
We narrowed down our initial 11 zero-shot prompts to the 7 which performed best on a sample of sentences from the FCE and W&I+LOCNESS training sets. We created few-shot prompts from the 3 best performing zero-shot prompts and varied the number of examples from 1 to 4. The re-
sults for zero-shot versus few-shot learning do not clearly show a best method for prompting. The open-source models which perform best on the test sets are FLAN-T5, Falcon-40B-Instruct, and Stable Beluga 2: of these, FLAN-T5 and Falcon-40BInstruct work best with few-shot learning, whereas Stable Beluga 2 is best with a zero-shot prompt. For GPT-3.5 Turbo, zero-shot is best for FCE test, few-shot is best for JFLEG and CoNLL-14 test.
In terms of prompt wording, the TOOL and CYN prompts are best for the three best open-source models: FLAN-T5, Falcon-40B-Instruct, and Stable Beluga 2. For GPT-3.5 Turbo, the ELT prompt is best for the FCE test set and the TOOL one is best for JFLEG and CoNLL-14 test. Note that two of the three best performing prompts are those in which a role is clearly specified to the LLM – either as an English language teacher or a grammatical error correction tool (Table 2).
The fact that the other best performing prompt, the one from Coyne et al. (2023), replicates the strong results from that paper is evidence for convergence around optimal prompt crafting. Further exploration of the huge prompt search space is possible, but we show that the CYN prompt holds up
well against a set of alternatives, and can therefore be considered a strong baseline for future GEC experiments.
Another provision we make for replication in future studies is to supply the list of examples we used in few-shot learning (Table 9). This allows others to use them for their own novel prompts, while holding constant the nature of the examples. Furthermore we believe that alternative methods for sourcing few-shot examples could be explored in future work, as discussed below.
Finally, we note that the comparison between commercial and open-source LLMs is not entirely even, as the former sit behind APIs and a black box processing pipeline. We recognise that GPT3.5 Turbo shows great promise for English GEC, at least for fluency corrections, but we also find that several open-source models perform relatively well – in fact better than GPT-3.5 on benchmarks annotated with minimal edits. This is a boon for open science, because models which researchers can obtain and work with directly lead to greater transparency in GEC and beyond.

heading: 7 Conclusion
text: We have shown that LLMs do not always outperform existing SOTA models for English GEC: for minimal edit style datasets such as the FCE, CoNLL-14 and W&I+LOCNESS, their performance is far below that of supervised GEC systems. We attempted to elicit minimal edit corrections
from LLMs through prompt crafting, but it may be that LLMs are still biased towards fluency rewrites as has been shown in previous work (Coyne et al., 2023; Fang et al., 2023; Loem et al., 2023). This is consistent with our finding, echoing that of others, that LLMs perform best on JFLEG, which was annotated with a fluency correction style.
We arrive at the following conclusions: (i) Supervised models are still best for English GEC with minimal edit corrections; (ii) Further explorations of prompt crafting, few-shot learning, and dynamic sampling are justified, as is work with open-source models as opposed to commercial ones; (iii) Methods for improving LLM performance on specific error types could be explored.
Other potential areas for future work include document-level GEC and human evaluation of proposed corrections. We worked with sentence-level GEC, but this deviates from the greater amount of essay context given to annotators. Document-level GEC has been proposed and recommended in previous work (Yuan and Bryant, 2021; Coyne et al., 2023). Exploratory work by Fang et al. (2023) showed that ChatGPT could not perform documentlevel GEC well, and speculated that it may not be able to handle long inputs requiring “high levels of coherence and consistency between sentences”. We notice that LLMs are better at GEC on beginner and intermediate texts, rather than advanced or native-speaker ones (Appendix Table 13): further investigation is needed on this matter.
Initial human evaluation studies suggest a preference for the corrections generated by LLMs over the reference corrections contained in GEC corpora (Coyne et al., 2023; Fang et al., 2023). It may be that human raters prefer to read the more fluent LLM-derived corrections but minimal edit corrections are actually more helpful for language learning since they are more faithful to the original intended meaning of the writer. Investigating learning benefits from receiving minimal edit grammatical feedback as opposed to fluency rewrites is a matter for future work which will involve longitudinal data collection, a focus on different feedback styles, and tracking how learners respond.

heading: Acknowledgements
text: This paper reports on research supported by Cambridge University Press & Assessment. Some of this work was performed using resources provided by the Cambridge Service for Data Driven Discov-
ery (CSD3) operated by the University of Cambridge Research Computing Service, provided by Dell EMC and Intel using Tier-2 funding from the Engineering and Physical Sciences Research Council (capital grant EP/T022159/1), and DiRAC funding from the Science and Technology Facilities Council. We acknowledge the Cohere For AI Research Grant that supported our evaluation of the Cohere Command model. We thank Yulong Lin for his exploratory work on GEC with LLMs.

heading: A Model sizes

heading: B Prompt Templates
text: Table 8 includes the complete list of zero-shot prompt templates used to perform GEC with LLMs. While the majority of the models use these templates, four models recommend a predefined prompt format – we describe model-specific prompts below. Table 9 contains the list of examples used in the few-shot prompts.
B.1 OpenAI GPT-∗ We use the OpenAI ChatCompletion endpoint that formats prompts with separate System and User messages. We adapt our prompts and put the instruction in the System message, and the learner sentence with any “Input” tags in the User message. For few-shot prompts, we format each example using separate User and Assistant messages, to mimic a chat-history as context – see Table 10 for an example.
B.2 Llama-2-chat Llama-2-chat is trained with the following structure for the first turn in chat applications:10


 
 {system_prompt} 

{input} 

We insert the entire GEC instruction into the system_prompt, and the learner sentence into the input. Where a prompt template uses “Input:”/“Output:” tags, we append the output tags after the final [\INST].
For the few-shot prompts, we follow the conversational setup and include examples as:
10https://huggingface.co/blog/llama2# how-to-prompt-llama-2
{input 1} 
 {hypothesis 1} 


 {input 2} 

B.3 Stable Beluga 2 Stable Beluga 2 recommends structuring prompts with System, User, and Response tags: ### System: This is a system prompt, please behave and help the user.
### User: {input}
### Assistant: {The output of Stable Beluga 2}
B.4 InstructPalmyra-20B InstructPalmyra recommends the following prompt format, including a preamble followed by Instruction, Input, and Response tags: Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. \n\n### Instruction:\n {instruction} \n\n### Input:\n {input}\n\n### Response:

heading: C Post-processing model output
text: For each model, we process the output with the following rules:
1. Remove “Output sentence: ”, “Corrected sentence: ”, and “Input sentence: ” from the start of sentences.
2. Strip double-quotes.
3. If there is an odd number of quotations, we remove trailing quotations.
4. For Llama 2, we search for and remove strings from a keyword list (included in Table 11).
5. For Llama 2, we split model generations based on the keyword list in Table 11.
6. For Falcon-40B-Instruct, we split model generations based on “Input sentence:” – this mainly impacts the few-shot setting, where the model tends to continue the few-shot pattern and generate a novel learner sentence after the correction.

heading: D Filtering zero-shot prompts with a sampled development set
text: We evaluated a long-list of eleven zero-shot prompts with each model on a development set of 2,000 sentences sampled uniformly from the
FCE and W&I+LOCNESS training sets. We report F0.5 scores as calculated using the automatic scorer in ERRANT. Table 12 presents the score for the top-1 performing prompt for each model and prompt-type.
We find Stable Beluga 2 and GPT-3.5 Turbo perform the best and obtain comparable performance using different prompts: the former using the “GEC tool” expert and the latter using the “English language teacher”. Indeed, we observe that the two expert prompts and the prompt from Coyne et al. (2023) result in the best performance across the models.
Figure 1a and 1b illustrate F0.5 scores for models using the zero-shot prompts, evaluated on the sampled development set. In the former, we can see that Dolly-v2-12B stands out with particularly low performance across all prompts. While in the latter, we can see that prompts 2, 4, and 9 have the lowest maximum scores. Additionally, prompts 4 and 5 are paired: they both use the “English language teacher” expert template, but prompt 5 contains more detailed instructions. It is clear from the plot that the more detailed instructions tend to result in
higher performance. From these results, we exclude zero-shot prompts 1, 2, 4, and 9 from the final evaluation due to their relatively low performance with every model. We additionally exclude Dolly-v2-12B due to its low performance across every prompt.

heading: E Results on the development sets
text: Table 4 shows precision, recall and F0.5 on the FCE development set. We find that the four smallest models have a bias towards precision over recall, while the larger models are more balanced.
Figure 2 presents the scores for each model on the FCE, JFLEG and W&I+LOCNESS development sets with our seven zero–shot prompts. We observe that InstructPalmyra and Stable Beluga 2 have much smaller variance in both zero- and fewshot settings. On the other hand we observe high variability with different prompts for OPT-IML and Falcon-40B-Instruct. For most models, we observe more consistent performance in the few-shot settings.
Figure 3 presents the scores for each model on the FCE development set with the prompts CYN, ELT, TOOL in zero- and few-shot settings. BLOOMZ, OPT-IML, and Falcon-40B-Instruct stand out as particularly sensitive to the choice of prompt – in particular, OPT-IML scores ∼0 F0.5 using the MIN, ELT, and DN prompts on each development set.11
11OPT-IML generates empty hypotheses for the majority of sentences with prompts MIN, ELT, and DN.
E.1 Proficiency level analysis We report performance on the W&I+LOCNESS development set grouped by CEFR level in Table 13. The majority of models perform relatively well on A-level learner text (beginners), followed by intermediate B-level text, English text written by native speakers, and finally advanced learner C-level text.
Interestingly, BLOOMZ, FLAN-T5, and OPTIML perform best on native speaker text. A closer inspection of the precision and recall results show all of these models have a bias towards high precision and low recall.

heading: Future Work
text: Other potential areas for future work include document-level GEC and human evaluation of proposed corrections. We worked with sentence-level GEC, but this deviates from the greater amount of essay context given to annotators. Document-level GEC has been proposed and recommended in previous work (Yuan and Bryant, 2021; Coyne et al. , 2023). Exploratory work by Fang et al. (2023) showed that ChatGPT could not perform documentlevel GEC well, and speculated that it may not be able to handle long inputs requiring “high levels of coherence and consistency between sentences”. We notice that LLMs are better at GEC on beginner and intermediate texts, rather than advanced or native-speaker ones (Appendix Table 13): further investigation is needed on this matter. Initial human evaluation studies suggest a preference for the corrections generated by LLMs over the reference corrections contained in GEC corpora (Coyne et al. , 2023; Fang et al. , 2023). It may be that human raters prefer to read the more fluent LLM-derived corrections but minimal edit corrections are actually more helpful for language learning since they are more faithful to the original intended meaning of the writer. Investigating learning benefits from receiving minimal edit grammatical feedback as opposed to fluency rewrites is a matter for future work which will involve longitudinal data collection, a focus on different feedback styles, and tracking how learners respond