Venue: NAACL 2024

# Ungrammatical-syntax-based In-context Example Selection

for Grammatical Error Correction

Chenming Tang  Fanyi Qu   Yunfang Wu1

National Key Laboratory for Multimedia Information Processing, Peking University

MOE Key Laboratory of Computational Linguistics, Peking University

School of Computer Science, Peking University

{tangchenning@stu, fanyiqu@, wuyf@}pku.edu.cn

Footnote 1: Corresponding author.

###### Abstract

In the era of large language models (LLMs), in-context learning (ICL) stands out as an effective prompting strategy that explores LLMs' potency across various tasks. However, applying LLMs to grammatical error correction (GEC) is still a challenging task. In this paper, we propose a novel ungrammatical-syntax-based in-context example selection strategy for GEC. Specifically, we measure similarity of sentences based on their syntactic structures with diverse algorithms, and identify optimal ICL examples sharing the most similar ill-formed syntax to the test input. Additionally, we carry out a two-stage process to further improve the quality of selection results. On benchmark English GEC datasets, empirical results show that our proposed ungrammatical-syntax-based strategies outperform commonly-used word-matching or semantics-based methods with multiple LLMs. This indicates that for a syntax-oriented task like GEC, paying more attention to syntactic information can effectively boost LLMs' performance. Our code is available at [https://github.com/JamyDon/SynICL46EC](https://github.com/JamyDon/SynICL46EC).

## 1 Introduction

Recently, large language models (LLMs) have shown awesome power in many areas and ended the contest on many tasks Chowdhery et al. (2023); Bubeck et al. (2023); Touvron et al. (2023). Unfortunately for LLMs, grammatical error correction (GEC), which aims at automatically correcting grammatical errors in erroneous text Bryant et al. (2022), is still a challenging task where they cannot beat conventional models yet. Fang et al. (2023b) and Loem et al. (2023) explore the performance of LLMs on GEC, demonstrating mainstream LLMs lag over 10 points behind the state-of-the-art result. Therefore, it is significant to explore new strategies to further improve the power of LLMs on GEC.

As a helpful prompting strategy, in-context learning (ICL) has achieved impressive results on many tasks Dong et al. (2022); Min et al. (2022). In ICL, several in-context examples are presented to LLMs as demonstrations before the input test sample in order to make LLMs aware of the requirement and output format of the specific task, thereby enhancing LLMs' performance during the subsequent generation process. Since the quality of context examples plays a crucial role under the few-shot setting, some strategies of example selection have been proposed Agrawal et al. (2023); Li et al. (2023a); Ye et al. (2023); Gupta et al. (2023).

To the best of our knowledge, no existing work on ICL example selection has taken syntactic information into consideration. However, GEC aims to correct grammatical errors and is a typical syntax-oriented task. In GEC, common errors can be classified into four types: _misuse_, _missing_, _redundancy_ and _word order_Bryant et al. (2017); Zhang et al. (2022a), and the last three of which are closely related to syntactic structure. That is, the _missing_, _redundancy_ or _disorder_ of text constituents may lead to ill-formed syntax Zhang et al. (2022b), suggesting the important role syntax plays in GEC. Hence, selecting in-context examples based on syntactic structure is likely to benefit LLMs more than conventional word-matching-based or semantics-based approaches.

Comparing with semantic similarity, syntactic similarity of text is less-studied. Previous works have leveraged the similarity of dependency trees to help multi-document summarization Ozates et al. (2016) and semantic textual similarity Le et al. (2018). To compute syntactic similarity, several effective algorithms computing similarity between syntactic trees have been proposed. Tree Kernel is a typical one, which counts the shared sub-structures of two trees to measure their similarity Collins and Duffy (2002); Vishwanathan et al. (2004); Moschitti (2006). Polynomial Distance is another handy one,

which converts syntactic trees into polynomials and then computes the distances Liu et al. (2022).

In this paper, we propose a novel ICL example selection strategy for GEC, by computing similarities of syntactic trees on ungrammatical sentences. Specially, we apply the syntactic similarity algorithms Tree Kernel and Polynomial Distance) to dependency trees generated by a GEC-oriented parser (GOPar) proposed by Zhang et al. (2022b), which is more reliable and provides error information when parsing ungrammatical sentences. Moreover, we carry out a two-stage process. In the first stage, namely _selection_, a fast and general method like BM25 is applied to filter out most of the irrelevant instances from the training data and obtain a much smaller candidate set. In the second stage, namely _ranking_, the more powerful syntax-based method is implemented to find out the best \(k\) instances as the final in-context examples.

To give a quick view of the superiority of our method, Table 1 shows an example illustrating the difference between BM25 selection and our ungrammatical-syntax-based method with Polynoimal Distance selection. BM25 only selects examples with similar words while Polynomial Distance is able to select those with similar grammatical errors, which will benefit the GEC task more.

We conduct experiments on two mainstream English GEC datasets, BEA-2019 Bryant et al. (2019) and CoNLL-2014 Ng et al. (2014). According to experimental results, Polynomial Distance and its weighted version achieve competitive results even under the single-stage setting, improving the performance by around 3 points and 2 points on BEA-19 and CoNLL-14 respectively. With the help of our two-stage selection, Tree Kernel gets its power unlocked and Polynomial Distance also benefits, leading to a further 1-point and 0.4-point improvement on BEA-19 and CoNLL-14 respectively. Overall, our ungrammatical-syntax-based in-context example selection methods secure the best results under all settings, outperforming conventional baselines by a margin of nearly 3 F\({}_{0.5}\) points on average.

Our contributions can be summarized as follows:

* We propose a novel ICL example selection method based on ungrammatical syntactic similarity to improve LLMs' performance on GEC. To the best of our knowledge, this is the first time that knowledge of syntactic structure is introduced to ICL example selection for GEC.
* We explore a two-stage selection strategy on GEC, where superficial word-similarity-based or semantics-based methods are used in the first stage and deep syntax-similarity-based ones are used in the second stage. It further improves LLMs' performance and achieves competitive results.
* We want to re-draw the natural language processing (NLP) community's attention to the significance of syntactic information. In this work, we show that syntax-related knowledge helps LLMs correct grammatical errors better. We believe our methods can be smoothly transferred to many other syntax-related tasks, like machine translation (MT) and information extraction (IE).

## 2 Related Work

### Grammatical Error Correction

In the past few years, the GEC task has been dominated by sequence-to-sequence machine translation models Junczys-Dowmunt et al. (2018); Rothe et al. (2021) and sequence-to-edit tagging models Omelianchuk et al. (2020); Tarnavskyi et al. (2022), both based on Transformer Vaswani et al. (2017).

Nowadays, with the finalization of mainstream models, further explorations on GEC mainly focus on two aspects. For one thing, injecting all kinds of additional knowledge into GEC models has proved helpful. The additional knowledge can be part-of-speech (POS) Wu and Wu (2022), syntax tree Zhang et al. (2022b), speech representation

\begin{table}
\begin{tabular}{l l p{160.7pt}} \hline \hline  & **Source (Erroneous Sentence)** & **Target (Corrected Sentence)** \\ \hline Input & No smoking in _the_ public places. & No smoking in public places. \\ \hline BM25 & 1 am writing to complain about the suggested _bar_ on smoking in public areas. & I am writing to complain about the suggested _ban_ on smoking in public areas. \\ Poly. & No future for _the_ public transport? & No future for public transport? \\ \hline \end{tabular}
\end{table}
Table 1: An example comparing the selection results of BM25 and Polynomial Distance (“Poly.” in the table).

 

(Fang et al., 2023a), abstract meaning representation (AMR) (Cao and Zhao, 2023), error type (Yang et al., 2023), etc. For another, multi-stage strategies help refine models' predictions. The multi-stage workflow can be permutation & decoding (Yakovlev et al., 2023), detection & correction (Li et al., 2023b), re-ranking (Zhang et al., 2023), etc.

With the rising of powerful large language models (LLMs), some works have begun exploring their performance on GEC (Loem et al., 2023; Fang et al., 2023b), showing that LLMs cannot beat conventional models on GEC yet.

### Syntactic Similarity

In computational linguistics (CL), previous works compared syntax trees of different languages to measure their similarities (Oya, 2020; Liu et al., 2022). In NLP, most works on text similarity focus on the semantic perspective (Gomaa et al., 2013, Reimers and Gurevych, 2019; Chandrasekaran and Mago, 2021), syntactic similarity of text is lessstudied. Ozates et al. (2016) used similarity of dependency trees to help multi-document summarization. Le et al. (2018) proposed ACV-tree (Attention Constituency Vector-tree), which combines word weight, word representation and constituency tree, to help the task of semantic textual similarity.

Syntactic similarity is usually represented by similarity between syntax trees. Tree similarity can be measured by various algorithms including Edit Distance (de Castro Reis et al., 2004), Polynomial Distance (Liu et al., 2022), Subset Tree Kernel (SSTK) (Collins and Duffy, 2002), SubTree Kernel (STK) (Vishwanathan et al., 2004), and Partial Tree Kernel (PTK) (Moschitti, 2006).

### Large Language Models and In-context Learning

In recent years, LLMs have shown their awesome power in many areas (Brown et al., 2020; Chowdhivery et al., 2023). Due to the limitation of computing resources, the focus of research on LLMs turns to the inference stage, trying to exploit the potency of LLMs with inference-only strategies.

ICL is a successful inference strategy that can make LLMs perform as well as fine-tuned models on many tasks (Brown et al., 2020; Von Oswald et al., 2023), where several in-context examples are given to LLMs as demonstrations before the actual test input. Instead of randomly sampling

Figure 1: Our two-stage selection and ICL workflow. For each input test sample, Stage **I** computes word similarities with BM25 or BERT representation between the input and all training data and select the top-\(1000\) as candidates. Then, Stage **II** computes ungrammatical syntactic similarities with tree kernel or polynomial distance between the input and candidates to select the most similar \(k\) example(s). After that, we concatenate the input after the \(k\) examples to construct the prompt for LLM inference. In the end, the LLM outputs the final result.

 

examples from the training set, recent works have boosted the performance of ICL by selecting in-context examples using various strategies. Agrawal et al. (2023) proposed R-BM25, a word overlap and coverage based selection strategy for machine translation. Li et al. (2023a) proposed training a Unified Demonstration Retriever (UDR) for ICL on a wide range of tasks. Ye et al. (2023) treated interest examples as a whole and selected examples on a subset level with the help of Determinantal Point Processes (DPPs). Gupta et al. (2023) also selected examples as an entire set, with contextual-embedding-level coverage as the goal.

Besides normal ICL, Chain-of-thought (CoT) Wei et al. (2022); Kojima et al. (2022) is another effective inference strategy in current favor, where LLMs are prompted to think step by step and answer with intermediate rationales.

## 3 Preliminaries

### Syntax Parser for Ungrammatical Sentences

Unlike most NLP tasks, which take correct sentences as input, the GEC task considers erroneous text as input. This gives rise to an issue that mainstream parsers may fail to obtain the expected dependency tree for the erroneous text.

To solve this problem, Zhang et al. (2022b) built a tailored GEC-Oriented dependency Parser (GOPar) based on parallel GEC training data, which is much more reliable when handling ungrammatical sentences than conventional parsers. Concretely, GOPar sets "S" (Substituted), "R" (Redundant) or "M" (Missing) labels to deal with different kinds of grammatical errors in the sentence, which inject additional information of errors into the dependency tree. Figure 2 shows the original illustration of GOPar from Zhang et al. (2022b).

Most previous works computing syntactic similarity were based on grammatical sentences with standard parsing trees Ozaates et al. (2016); Oya (2020). However, in GEC, we only have the ungrammatical source sentences, on which conventional parsers may perform poorly. So we apply the algorithms of tree similarity on the parsing results of GOPar, to compute syntactic similarities between test sample and training instances. We follow the official guidance of SynGEC 1 to run GOPar. We use biaffine-dep-electrora-en-gopar provided by SynGCE as the model for parsing.

Footnote 1: [https://github.com/HillZhang1999/SynGEC](https://github.com/HillZhang1999/SynGEC)

### Syntactic Similarity with Tree Kernel

We follow the unified Tree Kernel method proposed by Moschitti (2006), which can compute kernels of subset trees defined by Collins and Duffy (2002), subtrees defined by Vishwanathan et al. (2004) and partial trees defined in their own work.

For simplicity, we imitate the algorithm described in Le et al. (2018) and design the following algorithm (shown in Algorithm 1) to implement a simple version of Tree Kernel.

```
procedureCompSim(\(N_{1},N_{2}\))  \(K\gets 0\) foreach node \(n_{i}\) in \(N_{1}\)doforeach node \(n_{j}\) in \(N_{2}\)doif\(n_{i}.label=n_{j}.label\)then if\(n_{i}\) and \(n_{j}\) are both leaves then \(K\gets K+1\) elseif\(n_{i}\) and \(n_{j}\) are both non-leaves then \(K\gets K+\textsc{CompSim}(n_{i},n_{j})\) endif endif endfor endfor \(K\leftarrow K/(N_{1}.size\times N_{2}.size)\) return \(K\) endprocedure
```

**Algorithm 1** Similarity with Tree Kernel

For two trees \(T_{1}\) and \(T_{2}\), we conduct CompSim between their root nodes \(N_{1}\) and \(N_{2}\) to get a similarity score.

### Syntactic Similarity with Polynomial Distance

Liu et al. (2022) converted trees into polynomials and took the distances between polynomials as

Figure 2: Original illustration of GOPar from Zhang et al. (2022b). \(\emptyset\) denotes the missing word.

 

tree distances to measure syntactic similarities of dependency trees.

Given the number of dependency labels \(d\), the dependency trees will be represented into polynomials recursively on two variable set: \(X=\{x_{1},x_{2}...x_{d}\}\) and \(Y=\{y_{1},y_{2},...y_{d}\}\). In the dependency tree, for each leaf \(n^{l}\) with label \(l\), the corresponding polynomial is \(P(n^{l},X,Y)=x_{l}\). Then, for each non-leaf \(m_{l}\) with label \(l\), the corresponding polynomial is \(P(m^{l},X,Y)=y_{l}+\prod_{i=1}^{k}P(n_{i},X,Y)\), where \(n_{1},...,n_{k}\) are all child nodes of \(m_{l}\). In this way, the polynomial of the root node is regarded as the polynomial representation of a tree.

To compute similarity more conveniently, for each term \(cx_{1}^{e_{x_{1}}}x_{2}^{e_{x_{2}}}...c_{d}^{e_{xd}}y_{1}^{e_{yu}}y_{2}^{e_{yz}} ...y_{d}^{e_{yd}}\) in the dependency polynomial, we write it as a term vector with \(2d+1\) entries:

\[t=[e_{x_{1}},e_{x_{2}},...,e_{x_{d}},e_{y_{1}},e_{y_{2}},...,e_{y_{d}},c],\]

where each entry represents the exponent of the corresponding variable. In this way, a polynomial \(P\) can be written as a set of term vectors \(\mathcal{V}_{P}\). Then, we compute the distance between two polynomials as:

\[d(P,Q)=\frac{\sum\limits_{s\in\mathcal{V}_{P}}\min\limits_{t\in V_{Q}}\parallel s -t\parallel_{1}+\sum\limits_{t\in V_{Q}}\min\limits_{s\in\mathcal{V}_{P}} \parallel s-t\parallel_{1}}{|\mathcal{V}_{P}\mid+|\mathcal{V}_{Q}|}, \tag{1}\]

where \(\parallel s-t\parallel_{1}\) denotes the Manhattan distance (Craw, 2017) between term vector \(s\) and \(t\).

## 4 Methodology

### In-context Learning Workflow for GEC

Based on LLMs, our ungrammatical-syntax-based example selection and few-shot ICL workflow is illustrated in Figure 1. Specially, when faced with a test input, we search through the training data to find the best example(s) for in-context learning. Then, both the source (erroneous) and the target (corrected) sentences of the example(s) are inserted into the prompt as demonstrations, with the test sample concatenated at the end. In this way, LLMs can learn the GEC task from the demonstrations and perform better correction on the test input. In this framework, a set of high-quality in-context examples are crucial to lead LLMs to a better performance. Prompts used in this work are shown in Table 2.

### Ungrammatical-syntax-based Selection

We parse all source sentences in both training and test data with GOPar introduced in Section 3.1. Then, for each test input, we search the training data to find most syntactically similar examples with the help of Tree Kernel or Polynomial Distance. These examples will serve as in-context demonstrations in the prompts shown in Table 2.

Weighting Ungrammatical Nodes with Polynomial DistanceWe hypothesize that LLMs benefit more from similar grammatical errors, and error nodes with similar neighboring syntactic structure lead to similar error patterns. Therefore, assigning higher weights to ungrammatical nodes can select examples with error patterns closer to the test sample. Hence, besides the original Polynomial Distance algorithm, we also explore a weighted version. When computing the Manhattan distance between two term vectors, we assign a higher weight to entries corresponding to labels with error information ("S", "R" and "M"). In our experiment, as a preliminary attempt, we set the weight to 2.

\begin{table}
\begin{tabular}{p{85.4pt} p{196.7pt}} \hline  & **LLMaA-2** \\ \hline There is an erroneous sentence between “cerronous sentence” and “c/erronous sentence”. The grammatical errors in the erronous sentence will be corrected. The corrected version will be between ‘correct sentence’ and “c/orreonest sentence”; \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textascirowal sentence’. \textasci 



### Two-stage Selection

In previous works, a two-stage select-then-rank strategy performs well in in-context learning Wu et al. (2023); Agrawal et al. (2023). To be specific, a fast and general method is used to filter out most of the not-so-relevant instances from training data and get a much smaller candidate set with high quality, which is called _selection_. After that, a specific and powerful method is used to rank the instances in the candidate set and obtain the top-\(k\) best training instances, which is called _ranking_. Motivated by this, we also design a two-stage sample selection mechanism for GEC.

Stage 1: BM25/BERT SelectionFirst, we explore _selection_ with BM25 or BERT representation to obtain candidate examples, and the size of candidate set is 1000 in our experiment.

**BM25**Robertson et al. (1994) is a widely-used retrieval algorithm based on term frequency, inverse document frequency and length normalization. Many recent works regard BM25 as a strong baseline for in-context example selection Agrawal et al. (2023); Li et al. (2023). In our work, we take the input test sample as the query and source sentences of all training data as the document.

**BERT Representation**Li et al. (2023) make use of SentenceBERT Reimers and Gurevych (2019) to get sentence representations and then compared similarities of sentences. For simplicity, we adopt the more frequently-used BERT Devlin et al. (2019) instead. In our work, we take the BERT representation of the [CLS] token as the representation of the sentence. Then we compute the cosine similarities between the representations of the input test sentence and all source sentences in the training data.

For comparison, we also experiment on single-stage BM25 and BERT representation selection, which serve as baselines in Section 5.

**Stage 2: Ungrammatical-syntax-based Ranking** Further, we employ _ranking_ via syntactic similarity computing with Tree Kernel or Polynomial Distance, to obtain the best \(k\) matching examples from the candidate set.

## 5 Experimental Results

### Datasets and Evaluation Metrics

We carry out experiments on English GEC datasets. Since no model training is involved, most large-scale GEC data is unnecessary, while the data quality matters for example selection. Thus in this work, we only use the relatively small but high-quality Write&Improve+LOCNESS W&I+LOCNESS Bryant et al. (2019) as the training data.

For evaluation, we report P (Precision), R (Recall) and F\({}_{0.5}\) results on BEA-19 test set Bryant et al. (2019) evaluated by ERRANT Bryant et al. (2017) and on CoNLL-14 test set Ng et al. (2014) evaluated by M2Scorer Dahlmeier and Ng (2012). We primarily compare the F\({}_{0.5}\) among different methods, which shows the comprehensive performance of models on GEC.

Statistics of datasets mentioned above are shown in Table 3.

### Large Language Models

We use two mainstream LLM series: LLaMA-2 Touvron et al. (2023) and GPT-3.5 OpenAI (2023) for experiment.

For LLAMA-2, we use llama-2-7b-chat and llama-2-13b-chat with 7B and 13B parameters respectively. For GPT-3.5, we use the official gpt-3.5-turbo API for inference.

For the sake of reproductivity, we turn off the sampling and set the temperature to zero for all these models we use.

### Results

Experimental results are shown in Table 4. With different LLMs and on both datasets, our ungrammatical-syntax-based selection strategy obviously outperforms conventional methods. On BEA-2019 data, the method with first BM25 selection and then Tree Kernel ranking improves the performance by 3.7, 4.6 and 2.4 F\({}_{0.5}\) points, using llama-2-7b-chat, llama-2-13b-chat and gpt-3.5-turbo respectively.

Performance of Tree KernelWhen applied as a single-stage method, the Tree Kernel similarity performs poorly and even achieves a lower F\({}_{0.5}\) score than conventional baselines. However, with the help of a preliminary _selection_ stage, it improves by a margin of about 2 to 3 percentage points, and

\begin{table}
\begin{tabular}{l c c c} \hline \hline
* & Testing \\ CoNLL-14-Test & 1,312 & 72 & Testing \\ \hline \hline \end{tabular}
\end{table}
Table 3: Statistics of GEC datasets used in this work. **#Sentences** refers to the number of sentences. **%Error** refers to the percentage of erroneous sentences.


even achieves the highest F\({}_{0.5}\) score on BEA-2019 data with LLaMA-2.

Performance of Polynomial DistanceDifferent from Tree Kernel, Polynomial Distance performs fairly well even without a preliminary _selection_. Among those single-stage approaches, both polynomial-based methods outperform traditional baselines by an average margin of 2 to 3 percentage points in all cases, which indicates the superiority of syntactic similarity on GEC. The weighted version, with a higher weight on labels with error tags, brings a slight improvement in most cases, which shows the effectiveness of error information in GOPar-based dependency trees.

Performance of Two-stage SelectionAs for Tree Kernel, the two-stage selection strategy consistently boosts performance, whether using BM25 or BERT representation as the preliminary selection approach. But for Polynomial Distance, the two-stage selection brings less improvement and even fails to improve performance in some cases. We leave it for future research.

Comparison with State-of-the-artWe compare our method with previous supervised approaches, as shown in Table 5. Even with the help of ungrammatical-syntax-based selection, the GEC performance of LLMs is still far from state-of-the-art. We look forward to more advanced foundation models in the future.

## 6 Model Analysis

### Experiments with Different Numbers of In-context Examples

To explore the consistency and robustness of our methods, we conduct 1-shot, 2-shot, 4-shot and 8-shot experiments on llama-2-7b-chat. The results on BEA-2019 test set and CoNLL-2014 test set are shown in Table 6 and 7 respectively.

When there is only one example, the model performs relatively poor. When the number of examples comes to two, the performance improves significantly. Then, further increasing the number of examples brings a slight but consistent performance gain.

When the number of examples is small, the superiority of syntax-based methods compared with

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c c c c} \hline \hline  & & \multicolumn{6}{c}{**BEA-2019**} & \multicolumn{6}{c}{**CoNLL-2014**} \\ \cline{3-14}
**I** & **II** & **LLaMA-2-7B** & **LLaMA-2-13B** & **GPT-3.5-turbo** & **LLaMA-2-7B** & **LLaMA-2-7B** & **LLaMA-2-13B** & **GPT-3.5-turbo** \\ \cline{2-14}  & & **P** & **R** & **F\({}_{0.5}\)** & **P** & **R** & **F\({}_{0.5}\)** & **P** & **R** & **F\({}_{0.5}\)** & **P** & **R** & **F\({}_{0.5}\)** & **P** & **R** & **F\({}_{0.5}\)** \\ \hline \multirow{6}{*}{-} & Rand. & 50.1 & 57.7 & 51.5 & 49.0 & 61.2 & 51.0 & 47.0 & 70.4 & 50.3 & 59.4 & 48.8 & 56.9 & 58.6 & 51.3 & 57.0 & 56.5 & 59.9 & 57.1 \\  & BM25 & 50.9 & 58.2 & 52.2 & 51.6 & 61.1 & 53.3 & 46.8 & 69.6 & 50.1 & 59.7 & 47.7 & 56.8 & 59.3 & 50.1 & 57.2 & 56.6 & 60.8 & 57.4 \\  & BERT & 50.7 & 56.8 & 51.8 & 51.0 & 61.2 & 52.8 & 47.6 & 70.0 & 50.9 & 58.6 & 45.4 & 55.4 & 60.1 & 52.0 & 58.3 & 56.0 & 60.8 & 56.9 \\ \cline{2-14}  & T. K. & 50.0 & 57.0 & 51.2 & 52.5 & 59.0 & 53.6 & 47.2 & 69.8 & 50.5 & 57.9 & 47.5 & 58.5 & 61.8 & 48.0 & 58.5 & 57.3 & 60.3 & 57.9 \\  & Poly. & 53.1 & 57.9 & 54.0 & 52.9 & 60.2 & 54.3 & 49.5 & 70.0 & 52.6 & 59.5 & 49.5 & 57.2 & 61.7 & 51.8 & 59.4 & 58.2 & 59.9 & 58.6 \\  & W. Poly. & 53.2 & 58.2 & **54.2** & 53.4 & 60.5 & **54.7** & 50.3 & 69.6 & **53.2** & 60.1 & 49.2 & **57.5** & 61.6 & 52.3 & **59.5** & 58.4 & 60.5 & **58.8** \\ \hline \multirow{6}{*}{BM25} & T. K. & 55.1 & 55.9 & **55.2** & 54.9 & 58.7 & **55.6** & 49.7 & 69.3 & **52.7** & 62.2 & 45.7 & 58.0 & 61.9 & 47.3 & 58.3 & 58.3 & 59.7 & **58.6** \\  & Poly. & 51.2 & 57.1 & 52.3 & 50.9 & 59.8 & 52.5 & 48.8 & 69.5 & 51.9 & 62.1 & 47.7 & **58.6** & 60.9 & 49.8 & 58.3 & 57.2 & 59.7 & 57.7 \\ \cline{1-1}  & W. Poly. & 54.4 & 57.4 & 55.0 & 54.0 & 59.7 & 55.0 & 49.3 & 69.8 & 52.4 & 61.4 & 47.7 & 58.1 & 60.8 & 50.4 & **58.4** & 57.6 & 60.4 & 58.1 \\ \hline \multirow{6}{*}{BERT} & T. K. & 53.6 & 56.0 & 54.1 & 53.7 & 59.3 & 54.7 & 50.0 & 69.7 & **53.0** & 60.7 & 46.3 & 57.1 & 60.8 & 49.9 & **58.3** & 57.6 & 59.2 & 57.9 \\ \cline{1-1}  & Poly. & 53.3 & 57.2 & 54.0 & 53.8 & 60.4 & 55.0 & 49.0 & 69.5 & 52.1 & 60.5 & 47.6 & 57.4 & 59.8 & 50.8 & 57.8 & 57.6 & 60.7 & **58.2** \\ \cline{1-1}  & W. Poly. & 53.8 & 57.4 & **54.5** & 54.2 & 60.7 & **55.4** & 49.9 & 69.7 & 52.9 & 61.0 & 48.3 & **57.9** & 59.8 & 51.5 & 57.9 & 57.3 & 60.5 & 57.9 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Experimental results under the in-context few-shot setting with 4 examples. **I** and **II** denote the first (_selection_) and second (_ranking_) stage of the two-stage selection respectively. ”-” means the Stage **I** is absent and these are single-stage models. “Rand.”, ”T. K.”, “Poly.” and “W. Poly.” refer to “Random”, ”Tree Kernel” “Polynomial Distance” and “Weighted Polynomial Distance”, respectively. The dashed line separates results of conventional baselines and our proposed methods: the former on the upper side and the latter on the lower side. The best F\({}_{0.5}\) scores of each group are displayed in **bold**, and the best F\({}_{0.5}\) scores of all settings are displayed in **underlined bold**.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**System** & **CoNLL-14** & **BEA-19** \\ \hline GECTRo [2] & 65.3 & 72.4 \\ SyngEGC [23] & 66.7 & 72.0 \\ T5 xrl [14] & **68.9** & **75.9** \\ \hline ChaMGPr zero-shot CofT [22] & 51.7 & 36.1 \\ \hline LLaMA-2-7B with BM25 + Tree Kernel (ours) & 55.2 & 58.0 \\ LLaMA-2-13B with Weighted Polynomial (ours) & 54.7 & 59.5 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results of state-of-the-art GEC systems and our proposed methods on two datasets. The evaluation metric is F\({}_{0.5}\).

 

those conventional is evident. When the number of examples increases, conventional baselines improve a lot while syntax-based methods gain relatively less, which shows a marginal benefit. But syntax-based methods always secure the highest score, indicating the consistency of their advantages, especially under settings of fewer shots.

### Ungrammatical Parser or Standard Parser?

To explore the affect of different parsers on model performance, we also experiment with Stanford Parser [10], which is a widely-used conventional parser. We use stanford-corenlp-4.5.5 as the model for parsing and run Stanford Parser with stanfordcorenlp2, which is a Python wrapper for Stanford CoreNLP. For a clear demonstration, an example is illustrated in Figure 3 to show the

\begin{table}
\begin{tabular}{c c c c c c c c c c c c c c} \hline \hline  & \multirow{2}{*}{**II**} & \multicolumn{6}{c}{**1-shot**} & \multicolumn{6}{c}{**2-shot**} & \multicolumn{6}{c}{**4-shot**} & \multicolumn{6}{c}{**8-shot**} \\ \cline{3-17}  & & **P** & **R** & \(\mathbf{F_{0.5}}\) & **P** & **R** & \(\mathbf{F_{0.5}}\) & **P** & **R** & \(\mathbf{F_{0.5}}\) & **P** & **R** & \(\mathbf{F_{0.5}}\) \\ \hline \multirow{9}{*}{BART} & Rand. & 47.3 & 29.8 & 42.3 & 49.6 & 50.9 & 49.8 & 50.1 & 57.7 & 51.5 & 52.2 & 58.8 & 53.4 \\  & BM25 & 48.4 & 35.8 & **45.2** & 50.4 & 53.1 & 50.9 & 50.9 & 58.2 & 52.2 & 52.5 & 59.0 & 53.7 \\  & BERT & 47.3 & 33.8 & 43.8 & 50.0 & 51.4 & 50.2 & 50.7 & 56.8 & 51.8 & 53.6 & 59.3 & 54.6 \\ \cline{1-1}  & T. K. & 47.1 & 27.4 & 41.2 & 49.0 & 53.2 & 49.8 & 50.0 & 57.0 & 51.2 & 53.6 & 55.9 & 54.0 \\ \cline{1-1}  & Poly. & 50.1 & 31.5 & 44.8 & 53.9 & 51.9 & **53.5** & 53.1 & 57.9 & 54.0 & 54.3 & 58.3 & **55.1** \\ \cline{1-1}  & W. Poly. & 50.4 & 31.5 & 45.0 & 52.7 & 51.5 & 52.4 & 53.2 & 58.2 & **54.2** & 53.3 & 58.0 & 54.2 \\ \hline \multirow{9}{*}{BM25} & T. K. & 51.7 & 37.5 & **48.1** & 53.3 & 53.8 & **53.4** & 55.1 & 55.9 & **55.2** & 57.2 & 55.6 & **56.9** \\  & Poly. & 51.3 & 36.6 & 47.5 & 52.9 & 54.5 & 53.2 & 51.2 & 57.1 & 52.3 & 55.5 & 56.9 & 55.8 \\ \cline{1-1}  & W. Poly. & 51.1 & 36.6 & 47.4 & 52.8 & 54.7 & 53.2 & 54.4 & 57.4 & 55.0 & 56.3 & 57.0 & 56.4 \\ \hline \multirow{9}{*}{BERT} & T. K. & 50.7 & 35.6 & 46.8 & 53.3 & 52.4 & **53.1** & 53.6 & 56.0 & 54.1 & 57.1 & 57.0 & **57.1** \\ \cline{1-1}  & Poly. & 50.9 & 35.5 & **46.9** & 52.1 & 53.4 & 52.4 & 53.3 & 57.2 & 54.0 & 55.5 & 58.2 & 56.1 \\ \cline{1-1}  & W. Poly. & 50.6 & 35.7 & 46.7 & 52.1 & 53.8 & 52.4 & 53.8 & 57.4 & **54.5** & 56.5 & 57.8 & 56.7 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Results of llama–2–7b-chat with different numbers of shots on BEA-19 test set.

Figure 3: An example of parsing tree by GOPar and Stanford Parser.

 

different parsing results of GOPar and Stanford Parser.

The experimental results comparing GOPar and Stanford Parser on BEA-2019 test set are shown in Table 9. Here, we adopt llama-2-7b-chat as the LLM and Tree Kernel as the ranking method.

Without using the two-stage selection, Stanford Parser performs slightly worse than GOPar. With the two-stage selection, GOPar gains more improvement than Stanford Parser and outperforms it by a margin of more than 2 points. This indicates GOPar is more suitable for GEC, and its superiority lies in two aspects. First, it performs more robust on ungrammatical sentences (e.g., it correctly recognizes the prepositional object "pool" in the sentence shown in Figure 3 while Stanford Parser fails to). Second, it provides extra information about the grammatical errors (e.g., the _Missing_ error in Figure 3).

### Effect of Two-stage Selection

In order to find out how the two-stage strategy benefits the Tree Kernel method, we conduct a case study and compare three selection settings: BM25 only ("BM25"), Tree Kernel only ("T.K.") and Tree Kernel after the BM25 _selection_ ("BM25+T.K.").

In the example shown in Table 8, the input sentence is ungrammatical in word order. "BM25" selects a sentence with a punctuation missing error that is similar to the input sample in words ("also", "they" and "prepare"). "T. K." selects a sentence with an improper expression "get around constantly" which is similar to "prepare mentally" in syntactic structure but has little to do with the grammatical errors. "BM25 + T. K." selects a sentence that is similar to the input sample both in word occurrences ("also" and "have") and in error form (improper word order).

Since similar words are more likely to form similar errors, with the help of a preliminary selection, Tree Kernel can select from a more relative candidate set, leading to a better example selection involving both word and syntactic similarity in erroneous constituents. Moreover, it also shows the disadvantage of conventional selection method BM25 on GEC, which cannot effectively select examples similar in syntax.

## 7 Conclusion

In this work, we make use of two conventional tree-based syntactic similarity algorithms and the selecthen-rank two-stage framework to select in-context examples for the GEC task. Empirical results show that our syntax-based in-context example selection method is effective for GEC. We call on the NLP community to pay more attention to the help of syntactic information for many other syntax-related tasks besides GEC.

#### Acknowledgements

This work is supported by the National Natural Science Foundation of China (62076008) and the Key Project of Natural Science Foundation of China (61936012).

#### Limitations

First, we only experiment on English datasets. The performance of our method on other languages requires further exploration. Second, besides dependency tree, constituent tree is also worth trying. However, unfortunately, we do not have access to GEC-oriented constituent trees Zhang and Li (2022) at the time of writing this paper. Third, many previous outstanding methods of both in-context example selection and tree similarity computation have not been explored in our work. Fourth, due to limited time, we do not explore the effect of the size of candidate set after the _selection_ stage and the choice of weight of ungrammatical nodes in the Polynomial Distance method. There may

\begin{table}
\begin{tabular}{c|l} \hline  & **Source (Erroneous Sentence)** & **Target (Corrected Sentence)** \\ \hline Input & So, they _have to also_ prepare manually. & So, they _also have to_ prepare mentally. \\ \hline BM25 & Also you can see how they prepare your food in front of you. & Also, you can see how they prepare your food in front of you. \\ T. K. & Nowadays people get _around constantly_. & Nowadays, people _are constantly on the move._ \\ BM25 + T. K. & _that have limitation also_ there. & _There are also limitations_ there. \\ \hline \end{tabular}
\end{table}
Table 8: A one-shot example showing the tree kernel method benefiting from the two-stage selection.

\begin{table}
\begin{tabular}{c c c c c c c} \hline \multirow{2}{*}{**I**} & \multirow{2}{*}{**II**} & \multicolumn{4}{c}{**GOPar**} & \multicolumn{4}{c}{**Stanford Parser**} \\ \cline{3-6}  & & **P** & **R** & \(\mathbf{F}_{0.5}\) & **P** & **R** & \(\mathbf{F}_{0.5}\) \\ \hline - & & 50.0 & 57.0 & 51.2 & 49.6 & 56.4 & 50.8 \\ BM25 & T. K. & 55.1 & 55.9 & **55.2** & 51.8 & 56.2 & **52.7** \\ BERT & 53.6 & 56.0 & 54.1 & 50.6 & 57.2 & 51.8 \\ \hline \end{tabular}
\end{table}
Table 9: Results on BEA-2019 test set with 4 examples, using GOPar and Stanford Parser respectively.

 

exist a better size than the values we use in our experiments. Fifth, except for the Stanford Parser (which splits sentences itself), our experiments do not split instances with multiple sentences into single-sentence instances. Some instances in GEC datasets contains more than one sentence. Directly parsing these instances without splitting them into single sentences may hurt the parsing performance and lead to unreliable results. Last, we do not treat in-context examples as a whole, which might lead to a lower level of diversity of examples and sub-optimal performance, as addressed in Ye et al. (2023) and Gupta et al. (2023).

## Ethics Statement

Use of Scientific Artifacts.We make use of GOPar provided by Zhang et al. (2022b), which is publicly available based on the MIT license 3.

Footnote 3: [https://github.com/HillZhang1999/SynGEC](https://github.com/HillZhang1999/SynGEC)

About Computational Budget.Computation time is shown in Table 10.

About Reproducibility.All the experiments are completely reproducible since we disable sampling and set the temperature to zero for all LLMs we use, as discussed in Section 5.2.

## References

* Agrawal et al. (2023) Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2023. InContext examples selection for machine translation. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 8857-8873, Toronto, Canada. Association for Computational Linguistics.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877-1901.
* Bryant et al. (2019) Christopher Bryant, Mariano Felice, Oistein E. Andersen, and Ted Briscoe. 2019. The BEA-2019 shared task on grammatical error correction. In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_, pages 52-75, Florence, Italy. Association for Computational Linguistics.
* Bryant et al. (2017) Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 793-805, Vancouver, Canada. Association for Computational Linguistics.
* Bryant et al. (2022) Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. 2022. Grammatical error correction: A survey of the state of the art. _Computational Linguistics_, pages 1-59.
* Bubeck et al. (2023) Sebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. _arXiv preprint arXiv:2303.12712_.
* Cao and Zhao (2023) Hejing Cao and Dongyan Zhao. 2023. Leveraging denoised Abstract Meaning Representation for grammatical error correction. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 7180-7188, Toronto, Canada. Association for Computational Linguistics.
* Chandrasekaran and Mago (2021) Dhivya Chandrasekaran and Vijay Mago. 2021. Evolution of semantic similarity--a survey. _ACM Computing Surveys (CSUR)_, 54(2):1-37.
* Chowdhery et al. (2023) Akanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling pathways. _Journal of Machine Learning Research_, 24(240):1-113.
* Collins and Duffy (2002) Michael Collins and Nigel Duffy. 2002. New ranking algorithms for parsing and tagging: Kernels over discrete structures, and the voted perceptron. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 263-270, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Craw (2017) Susan Craw. 2017. Manhattan distance. _Encyclopedia of Machine Learning and Data Mining_, pages 790-791.
* Dahlmeier and Ng (2012) Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 568-572, Montreal, Canada. Association for Computational Linguistics.

\begin{table}
\begin{tabular}{c c} \hline \hline
**Method** & **Time** \\ \hline BM25 & 440 \\ BERT & 4500 \\ Tree Kernel & 3600 \\ Polynomial Distance & 3200 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Computation time of different methods on BEA-19 test set, all in seconds. BERT runs on an NVIDIA GeForce RTX 2080 Ti and the other three run on an Intel® Xeon® Gold 5218 CPU.

 

Davi de Castro Reis, Paulo Braz Golgher, Altigraan Soares da Silva, and Alberto H. F. Laender. 2004. Automatic web news extraction using tree edit distance. In The Web Conference.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics.
* Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. _arXiv preprint arXiv:2301.00234_.
* Dozat and Manning (2017) Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In International Conference on Learning Representations.
* Fang et al. (2023a) Tao Fang, Jinpeng Hu, Derek F Wong, Xiang Wan, Lidia S. Chao, and Tsung-Hui Chang. 2023a. Improving grammatical error correction with multimodal feature integration. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9328-9344, Toronto, Canada. Association for Computational Linguistics.
* Fang et al. (2023b) Tao Fang, Shu Yang, Kaixin Lan, Derek F Wong, Jinpeng Hu, Ladia S Chao, and Yue Zhang. 2023b. Is chatgtpt a highly fluent grammatical error correction system. _A comprehensive evaluation. ArXiv, abs/2304.01746_.
* Gomaa et al. (2013) Wael H Gomaa, Ay Ay Ay Fahmy, et al. 2013. A survey of text similarity approaches. _international journal of Computer Applications_, 68(13):13-18.
* Gupta et al. (2023) Shivanshu Gupta, Matt Gardner, and Sameer Singh. 2023. Coverage-based example selection for incons context learning. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 13924-13950, Singapore. Association for Computational Linguistics.
* Junczys-Dowmunt et al. (2018) Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. 2018. Approaching neural grammatical error correction as a low-resource machine translation task. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 595-606, New Orleans, Louisiana. Association for Computational Linguistics.
* Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Michel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. _Advances in neural information processing systems_, 35:22199-22213.
* Le et al. (2018) Yuquan Le, Zhi-Jie Wang, Zhe Quan, Jiawei He, and Bin Yao. 2018. Acv-tree: A new method for sentence similarity modeling. In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18_, pages 4137-4143. International Joint Conferences on Artificial Intelligence Organization.
* Li et al. (2023a) Xiaoran Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023a. Unified demonstration retriever for incontext learning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 4644-4668, Toronto, Canada. Association for Computational Linguistics.
* Li et al. (2023b) Yinghao Li, Xuebo Liu, Shuo Wang, Peiyuan Gong, Derek F. Wong, Yang Gao, Heyan Huang, and Min Zhang. 2023b. TemplateGEC: Improving grammatical error correction with detection template. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 6878-6892, Toronto, Canada. Association for Computational Linguistics.
* Liu et al. (2022) Pengyu Liu, Tinghao Feng, and Rui Liu. 2022. Quantifying syntax similarity with a polynomial representation of dependency trees. _arXiv preprint arXiv:2211.07005_.
* Loem et al. (2023) Mengsay Loem, Masahiro Kaneko, Sho Takase, and Naoxaki Okazaki. 2023. Exploring effectiveness of GPT-3 in grammatical error correction: A study on performance and controllability in prompt-based methods. In _Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)_, pages 205-219, Toronto, Canada. Association for Computational Linguistics.
* Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? _arXiv preprint arXiv:2202.12837_.
* Moschitti (2006) Alessandro Moschitti. 2006. Efficient convolution kernels for dependency and constituent syntactic trees. In _Machine Learning: ECML 2006_, pages 318-329, Berlin, Heidelberg. Springer Berlin Heidelberg.
* Ng et al. (2014) Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In _Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task_, pages 1-14, Baltimore, Maryland. Association for Computational Linguistics.
* grammatical error correction: Tag, not rewrite. In _Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational
 

Applications_, pages 163-170, Seattle, WA, USA. Online. Association for Computational Linguistics.
* OpeAI (2023) OpenAI. 2023. GPT-3.5 API. [https://platform.openai.com/docs/models/gpt-3-5](https://platform.openai.com/docs/models/gpt-3-5).
* Oya (2020) Masanori Oya. 2020. Syntactic similarity of the sentences in a multi-lingual parallel corpus based on the Euclidean distance of their dependency trees. In _Proceedings of the 34th Pacific Asia Conference on Language, Information and Computation_, pages 225-233, Hanoi, Vietnam. Association for Computational Linguistics.
* Ozates et al. (2016) Saziye Betul Ozates, Arzucan Ozguir, and Dragomir Radev. 2016. Sentence similarity based on dependency tree kernels for multi-document summarization. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)_, pages 2833-2838, Portoroz, Slovenia. European Language Resources Association (ELRA).
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
* Robertson et al. (1994) Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1994. Okapi at trec-3. In _Text Retrieval Conference_.
* Rothe et al. (2021) Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2021. A simple recipe for multilingual grammatical error correction. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 702-707, Online. Association for Computational Linguistics.
* Taravskyi et al. (2022) Maksym Taravskyi, Artem Chernodub, and Kostiantyn Omelianchuk. 2022. Ensembling and knowledge distilling of large sequence taggers for grammatical error correction. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3842-3852, Dublin, Ireland. Association for Computational Linguistics.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. 2023. LIama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. _Advances in neural information processing systems_, 30.
* Vishwanathan et al. (2004) SVN Vishwanathan, Alexander Johannes Smola, et al. 2004. Fast kernels for string and tree matching. _Kernel methods in computational biology_, 15(113-130):1.
* Oswald et al. (2023) Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Joao Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. 2023. Transformers learn in-context by gradient descent. In _International Conference on Machine Learning_, pages 35151-35174. PMLR.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denmy Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In _Advances in Neural Information Processing Systems_, volume 35, pages 24824-24837. Curran Associates, Inc.
* Wu and Wu (2022) Xiuyu Wu and Yunfang Wu. 2022. From spelling to grammar: A new framework for Chinese grammatical error correction. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 889-902, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Wu et al. (2023) Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. 2023. Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1423-1436, Toronto, Canada. Association for Computational Linguistics.
* Yakovlev et al. (2023) Konstantin Yakovlev, Alexander Podolskiy, Andrey Bout, Sergey Nikolenko, and Irina Piontkovskaya. 2023. GEC-DePenD: Non-autoregressive grammatical error correction with decoupled permutation and decoding. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1546-1558, Toronto, Canada. Association for Computational Linguistics.
* Yang et al. (2023) Lingyu Yang, Hongjia Li, Lei Li, Chengyi Xu, Shutao Xia, and Chun Yuan. 2023. LET: Leveraging error type information for grammatical error correction. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 5986-5998, Toronto, Canada. Association for Computational Linguistics.
* Ye et al. (2023) Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. In _Proceedings of the 40th International Conference on Machine Learning_, volume 202 of _Proceedings of Machine Learning Research_, pages 39818-39833. PMLR.
* Zhang et al. (2023) Ying Zhang, Hidetaka Kamigaito, and Manabu Okumura. 2023. Bidirectional transformer reranker for grammatical error correction. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 3801-3825, Toronto, Canada. Association for Computational Linguistics.

 

* Zhang and Li (2022) Yue Zhang and Zhenghua Li. 2022. Csyngec: Incorporating constituent-based syntax for grammatical error correction with a tailored gec-oriented parser. _arXiv preprint arXiv:2211.08158_.
* Zhang et al. (2022a) Yue Zhang, Zhenghua Li, Zuyi Bao, Jiacheng Li, Bo Zhang, Chen Li, Fei Huang, and Min Zhang. 2022a. MuCGEC: a multi-reference multi-source evaluation dataset for Chinese grammatical error correction. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3118-3130, Seattle, United States. Association for Computational Linguistics.
* Zhang et al. (2022b) Yue Zhang, Bo Zhang, Zhenghua Li, Zuyi Bao, Chen Li, and Min Zhang. 2022b. SynGEC: Syntax-enhanced grammatical error correction with a tailored GEC-oriented parser. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 2518-2531, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.