Venue: NAACL 2024

# MEDIT: Multilingual Text Editing via Instruction Tuning

Vipul Raheja\({}^{1}\) Dimitris Alikaniotis\({}^{1}\) Vivek Kulkarni\({}^{1}\) Bashar Alhafni\({}^{2*}\) Dhruv Kumar\({}^{1}\)

\({}^{1}\)Grammarly \({}^{2}\)New York University Abu Dhabi

firstname.lastname@grammarly.com

###### Abstract

We introduce mEdIT, a multilingual extension to CoEDIT - the recent state-of-the-art text editing models for writing assistance. mEDT models are trained by fine-tuning multilingual large, pre-trained language models (LLMs) via instruction tuning. They are designed to take instructions from the user specifying the attributes of the desired text in the form of natural language instructions, such as _Grammatik korrigieren_ (German) or o ! q? & c# ! (Korean). We build mEdit by curating data from multiple publicly available human-annotated text editing datasets for three text editing tasks (GEC), Text Simplification, and Paraphrasing) across diverse languages belonging to six different language families. We detail the design and training of mDIT models and demonstrate their strong performance on many multilingual text editing benchmarks against other multilingual LLMs. We also find that mEdIT generalizes effectively to new languages over multilingual baselines. We publicly release our data, code, and trained models.1

Footnote 1: [https://github.com/vipulraheja/medit](https://github.com/vipulraheja/medit)

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 979–1001 June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

Large language models (LLMs) have made remarkable progress toward generating fluent and coherent text in a wide variety of tasks and domains to support writing assistance Brown et al. (2020); OpenAI (2023); Touvron et al. (2023); _inter alia_). In particular, LLMs have been adapted to perform many complex text editing tasks like GEC Wu et al. (2023); Coyne and Sakaguchi (2023); Fang et al. (2023), text simplification Baez and Saggion (2023); Saggion et al. (2022), paraphrasing Witteveen and Andrews (2019); Niu et al. (2021), and formality and tone rewriting Reif et al. (2022); Luo et al. (2023), among others. However, most of these works are restricted to single tasks, with few works adapting LLMs to perform high-quality text editing across multiple tasks Schick et al. (2023); Raheja et al. (2023); Laban et al. (2023). A lot of these improvements have been driven by fine-tuning large language models (LLMs) with task-specific instruction tuning, resulting in remarkable zero-shot generalization abilities Sanh et al. (2022); Ouyang et al. (2022); Chung et al. (2022); _inter alia_).

At the same time, significant research effort has been dedicated to leveraging and enhancing the multilingual capabilities of LLMs Lin et al. (2022). These abilities can be improved using methods such as continued pre-training with abundant monolingual data Yang et al. (2023); Cui et al. (2023) or language-specific instruction-tuning Zhu et al. (2023); Li et al. (2023). However, in the case of continued pre-training, the lack of high

Figure 1: **Examples illustrating multilingual and cross-lingual text editing. The editing instructions are described in bold. Note that the input and output texts are always in the same language. The monolingual vs. cross-lingual setting is determined by comparing the language of the edit instruction to the language of the input text.**

quality web-scale data often restricts the ability to improve LLMs capabilities in less-represented languages in the same way that English data can be expanded. Moreover, while numerous multilingual instruction-tuned models have been developed (Muennighoff et al., 2023; Workshop, 2023; Xue et al., 2021; Li et al., 2023; Wei et al., 2023), our analyses show that without further task-specific fine-tuning, these models are not suitable for carrying out high-quality text-editing tasks (SS 5.1). In the context of text editing tasks, multiple previous works have developed high-quality, general-purpose LLMs on non-English languages, restricting themselves, however, on either specific tasks (Rothe et al., 2021; Sun et al., 2022; Kementchedjhieva and Sogaard, 2023; Ryan et al., 2023; Krishna et al., 2022; Lai et al., 2022) or specific languages (Alhafni et al., 2023; Anschutz et al., 2023). Overall, the aforementioned factors have limited the availability of high-quality _multilingual text editing_ (MTE) models, which has limited their usability for writing assistance across multiple tasks in languages beyond English.

We address these gaps with mEdIT, a multitask, multilingual extension of CoEDiTRaheja et al. (2023). mEdIT models can perform text editing operations for three popular tasks: Grammatical Error Correction, Paraphrasing, and Text Simplification, in multilingual and cross-lingual settings (Figure 1) across a diverse set of seven languages, spanning six different language families (Table 1).

To build mEdIT, we fine-tune several multilingual LLMs of varying sizes on carefully curated, largely human-annotated, parallel corpora of over 200k instructional input-output pairs, using publicly available datasets (Table 4) for different text editing tasks. We evaluate the performance of our models extensively on text editing benchmarks in both multilingual and cross-lingual settings to demonstrate their effectiveness. Our contributions are as follows:

* This work, to the best of our knowledge, is the first to investigate multi-task, multilingual text editing via instruction tuning.
* Our models achieve strong performance on multiple text editing tasks across numerous languages and are publicly released for fostering further MTE research.
* Through a comprehensive set of controlled experiments, we provide insights on how model performance on multilingual text editing tasks is affected by various choices like model architecture, model scale, and training data mixtures.

## 2 Related Work

Multi-lingual LLMs for Text EditingThere is an extensive body of prior literature that has leveraged LLMs for various multi-lingual text editing tasks. These works have proposed models for text editing tasks like GEC Rothe et al. (2021); Sun et al. (2022), paraphrasingChowdhury et al. (2022), formality style transfer Briakou et al. (2021), and text simplificationMallinson et al. (2020); Martin et al. (2022); Ryan et al. (2023). However, all of these prior approaches have proposed task-specific multi-lingual models. In contrast, we propose a single unified text-editing model for all the considered tasks by leveraging the power of instruction-tuning and task-specific fine-tuning, which enables our multi-lingual models to generalize to multiple text-editing tasks.

Multi-lingual Instruction-TuningWhile numerous multi-lingual instruction fine-tuned models like Muennighoff et al. (2023); Wei et al. (2023) and Li et al. (2023) have been developed, they are not focused or tailored for text editing tasks, which we address by task-specific fine-tuning. Specific

\begin{table}
\begin{tabular}{l l l} \hline \hline
**Language** & **ISO-639-1** & **Family** \\ \hline Arabic & ar & Semitic \\ Chinese & zh & Sino-Tibetan \\ English & en & Germanic \\ German & de & \\ Japanese & ja & Japonic \\ Korean & ko & Koreanic \\ Spanish & es & Romance \\ \hline \hline \end{tabular}
\end{table}
Table 1: **Set of Languages. The seven languages, along with the ISO-639-1 code and their language family, on which we train and evaluate our models.**

Figure 2: **Data distribution for each of the three tasks and seven languages on which we train. The amount of data is shown in a log scale to aid visualization.**



to text editing, many prior works have explored instruction tuning capable of performing multiple text editing tasks with a single model, such as GEC, simplification, sentence fusion, style transfer, and paraphrasing, to name a few (Mallinson et al., 2022; Du et al., 2022; Kim et al., 2022; Schick et al., 2023; Raheja et al., 2023). However, while they are able to support multi-task text editing, they are generally mono-lingual and typically restricted to specific languages (predominantly English). Thus, our work addresses this gap by proposing multilingual, instruction-tuned models for multiple text editing and revision tasks.

## 3 mEdIT

### Tasks and Languages

We chose a broad set of languages to ensure coverage and chose text editing tasks that had multilingual, publicly available human-annotated datasets to ensure high data quality. Another criteria was to choose languages at the intersection of the publicly available corpora we could find across a large set of languages for all the tasks we considered. We refer to this as the mEdIT dataset.

Table 1 describes the languages covered in our work, whereas Figure 2 depicts the amounts of training datasets that were available for all tasks and languages we considered. Appendix A details all the training and testing datasets.

### Models

We fine-tune different versions of pre-trained multilingual LLMs (both encoder-decoder/sequence-to-sequence (Seq2Seq) and decoder-only/causal language models (CLM)) on the mEdIT dataset using cross-entropy loss. The details of the mEdIT models are described in SS 4.2, whereas the training details are summarized in SS 4.3.

## 4 Experiments

### No-Edits Baseline

We first evaluate a no-edits baseline, where the output is simply a copy of the source input without the instruction. This strategy performs reasonably well on tasks where the target output largely overlaps with the input (e.g., GEC).

### Multilingual LLMs

mT5(Xue et al., 2021) is a multilingual variant of T5 (Raffel et al., 2020), trained on the mC4 dataset, a multilingual variant of the C4 dataset extended to 101 languages. We experiment with three variants of mT5 - Large (770M), XL (3B), and XXL (13B) parameters.

mT0(Muennighoff et al., 2023) is a family of multilingual Seq2Seq models capable of zero-shot following human instructions in dozens of languages. We use the mt0-large (1.2B), mt0-XL (3.7B), and mt0-xxl (13B) models. These models are constructed by fine-tuning mT5 models on the xP3 cross-lingual task mixture dataset, which consists of multilingual datasets with English prompts. As a result, mT0 models are better suited for following English prompts. We also use the mt0-XXL-mt variant, which is fine-tuned on the xP3mt dataset and is better suited for prompting in non-English.

Bloomz(Muennighoff et al., 2023) is a family of multilingual Causal Language Models (CLMs) constructed by fine-tuning BLOOM (Workshop, 2023) on the xP3 dataset. We use BLOOMLZ-3b and BLOOMZ-7b1 models for our experiments.

PolyLM(Wei et al., 2023) is a set of multilingual LLMs trained on 640B tokens. We experiment with the PolyLM-MultiAlpaca-13B model, which is PolyLM-13B model fine-tuned on the MultiAlpaca dataset, consisting of 132k samples of multilingual instructions.

Bactrian-X(Li et al., 2023) is a collection of lightweight adapters for LLaMA (7B and 13B) (Touvron et al., 2023) and BLOOM (7B) (Workshop, 2023) on the Bactrian-X dataset, which is a multilingual parallel dataset comprising 3.4 million instruction-response pairs across 52 languages. For simplicity, we only compare against its higher-performant LLaMA-adapted versions.

#### 4.2.1 Large-Pretrained Decoder-only Models

We also conduct zero-shot evaluations against state-of-the-art decoder-only LLMs that have shown impressive multilingual capabilities on a variety of NLP tasks leveraging the power of in-context learning (Lei et al., 2023; OpenAI, 2023).

Gpt3.5(also referred to as ChatGPT),2 is an improved version of GPT3 (Brown et al., 2020) optimized for chat. We use the gpt-3.5-turbo0613 model from the OpenAI API.3 

Gpt4(OpenAI, 2023) is the latest iteration of the GPT models and is also optimized for chat. We use the gpt-4-0613 model from the OpenAI API.

While we recognize that these models may not be explicitly optimized or trained for multi-lingual settings, considering that they have been trained on massive amounts of web-scale data, these models have been shown to have multi-lingual capabilities Lai et al. (2023), hence, we consider them as one of our baseline groups.

### Training Setup

We perform instruction tuning for all our models by crafting custom prompts for each of the 21 task-language combinations (seven languages, three tasks). Similar to CoEdIT, for each task-language combination, depending on the number of ways the instructions can be translated without altering the meaning, we write between 14 and 27 instructions by automatically translating each one from English and verifying the accuracy of the translations by asking native language speakers to evaluate and correct them. The total number of task-language instructions is 365, which can be found in Appendix D. We explore three different multilingual and cross-lingual instructional settings, depending on the language of the prompt, where the editing instruction could be in (a) **English**, (b) the same language as the text being edited (**Native**), and (c) a random language which may or may not be the same as the language of the text being edited (**Random**). With this definition, English and Random are cross-lingual text editing tasks, and Native is a multilingual text editing task. In all settings, the input-output pairs are in the same language, but only the language of the instruction changes.

We train all models on 8xA100 80G GPU instances for five epochs. For the PolyLM and Bactrian-X models (>7B parameters), we also use LoRA Hu et al. (2022) to lower the number of trainable parameters and increase the batch size.

### Evaluation

For GEC evaluation, we follow prior work on each language we report on and use the appropriate GEC metric accordingly. Mainly, we use the MaxMatch (M\({}^{2}\))Scorer Dahlmeier and Ng (2012), ERRANT Bryant et al. (2017), and GLEU Napoles et al. (2015, 2016) as our evaluation metrics. The M\({}^{2}\) Scorer and ERRANT compare the edits made by a GEC system against annotated reference edits and calculate the precision (P), recall (R), and F\({}_{0.5}\) (i.e., weighing precision twice as much as recall). GLEU computes the precision of the n-grams that overlap with the references but not the original texts and penalizes n-grams that overlap with the original texts but not the references.

For simplification, we follow Ryan et al. (2023) and use SARI Xu et al. (2016) and BLEU Papineni et al. (2002) for evaluation. SARI is the average of the F1 score for adding, keeping, and deleting n-grams \((n\in 1,2,3,4)\) and has been shown to correlate with human judgments of simplicity Xu et al. (2016). BLEU, on the other hand, is a common metric in machine translation and is used as a check for grammatical and meaning preservation. We compute all metrics using the EASSE evaluation suite Alva-Manchego et al. (2019).

We evaluate paraphrasing on two criteria and metrics: diversity and semantic similarity. For diversity, we use Self-BLEU Zhu et al. (2018) to measure the diversity of the paraphrases relative to the given source and reference texts. We use Semantic Similarity to measure meaning preservation. Specifically, we use _m_USE Yang et al. (2020) for this, as it is the best-performing multilingual sentence similarity model that supports all the languages in our work. We also considered other notable works that have made significant progress on multilingual sentence similarity, such as Multilingual-SBERT Reimers and Gurevych (2020) and LaBSE Feng et al. (2022). However, we found them unsuitable for our purposes as they were either limited by the languages they support or suffered from lower performance for multilingual sentence similarity.

## 5 Quantitative Results: MTE Quality

We split the models into three main groups. The first group (a) consists of the "no edits" baseline, the second group (b) is the untrained baseline, where models are evaluated in a zero-shot setting without any task-specific fine-tuning, while the third group (c) is our set of multi-lingual models trained on task-specific datasets. For all our experiments, we aggregate the models' performance by text editing tasks. Specifically, we aggregate the metrics for each task using the harmonic means of its constituents. Specifically, we use (1-Self-BLEU)4 and Semantic Accuracy for Paraphrasing, SARI and BLEU for Simplification, and \(F_{0.5}\) and GLEU for GEC. We scale all metrics to lie between 

0 and 100. We show full results on all models in Appendix C for the best-performing setup.

### Baselines

In Figure 3, we report the results of our trained models against various baselines by aggregating the performance on all tasks (as detailed in SS 5).

No Edits (Copy) BaselineWe observe that not making any edits leads to a performance that is on par with the untrained versions of all models, which highlights the limitations of the n-gram overlap-based metrics.

Untrained BaselineSimilar to Raheja et al. (2023), a core contribution of this work is to push the performance of small- (\(\sim\)1B parameters) to medium-sized (1-15B parameters) LLMs for common text editing tasks across multiple languages. This drives the need for fine-tuning task-specific and language-specific datasets. For this work, we compare our fine-tuned models against their non-fine-tuned counterparts. We find a substantial gap between the untrained models and their trained counterparts, highlighting the impact of task- and language-specific fine-tuning for the tasks.

English-Only BaselineIn this experiment, we analyze the ability of multilingual LLMs to adapt to different text editing tasks across different languages by fine-tuning them in the most prominent high-resource language (English). We fine-tune all the multi-lingual models on just the English subsets of the training data, as it is the largest in terms of quality and quantity. This experiment tests the need for language-specific data. Similar to the previous result, we observe that the gap between the untrained versions of English-only models is relatively small vs. the ones trained on the full dataset; it increases significantly with language-specific fine-tuning.

State-of-the-art LLMsAdditionally, we also evaluate against the most powerful commercially available LLMs on their ability to perform MTE. Specifically, we evaluate GPT3.5 and GPT-4 in a zero-shot setting. Although these models have been shown to exhibit strong zero-shot performance on a variety of NLP tasks, we find that the overall performance of both models is close to most untrained baselines. This can be attributed to the rather limited multilingual capabilities of GPT3.5, which often lead to outputs being generated in other languages (English in particular). To some extent, the verbosity of responses is highly detrimental to the performance, especially for GPT4 as it gets penalized by the automatic metrics (especially for GEC).

The rest of this section analyzes different aspects of the quantitative performance of our models.

### Model Performance by Language

In this section, we analyze the performance of different MTE models by language (Figure 5). It is interesting to note that Paraphrasing exhibits a rather steady performance across languages. This can partially be attributed to the weakness of the evaluation metrics as they rely mostly on n-gram overlap, on the multilingual pre-training of the LLMs where they are exposed to medium-large corpora of nearly all the languages, but also that with the increase in the model size and fine-tuning, they

Figure 4: **Aggregated performance on different tasks broken down by instruction language**. Apart from some minor fluctuation, there is no significant impact of instruction language on our results.

Figure 3: **Overall performance comparison of all baselines against trained models.** We calculate the aggregated performance across all tasks using the harmonic mean of task-specific scores. Baselines are “No Edits (Copy)”, “English-only” (“-En,”), and our trained models are marked as “CLM” and “Seq2Seq,” respectively. The aggregated performance is calculated as described in § 5.

 

tend to make fewer changes, thus, leading to higher scores. For simplification, the variance in the performance across languages can be attributed to the amount and quality of training data available for each task. For instance, for German, only 1.1k training data points were available, which leads to models not only showing a great improvement in performance with fine tuning, but also a great variance. Similarly, the training data is very noisy for Japanese, which leads to a similar effect. For GEC, we observe that the performance varies a lot by language, indicating the challenging nature of the task. This can be partially attributed to the frequency and the type of errors in each dataset, a phenomenon we see in Arabic datasets. For instance, Arabic datasets contain subtle and frequent errors made by native/L1 speakers, whereas the data for all other languages consists of sparse and infrequency errors made by L2 learners, which could explain the abnormally low quality (on Arabic GEC) of our best-performing model. Moreover, the quality of the training GEC data available also leads to varying performance across languages.

### Language of Instruction

Here, we analyze the effect of the language of the instruction used to instruct the model. As mentioned in SS 4.3, we have three configurations for this set of experiments:

English-language InstructionsWe train the first set of multilingual mEdIT models with just English instructions. These are MTE models capable of performing cross-lingual text editing, trained on data where the instruction is always in English.

Native-language InstructionsWe train the next set of multi-lingual mEdIT models with instructions in their native language. These models are capable of performing MTE, where the language of edit instructions is the same as the language of texts being edited.

Randomized-language InstructionsFinally, we also explore the cross-lingual text editing (Figure 1) abilities of multi-lingual LLMs. To do so, we modify our dataset by appending an edit instruction from a randomly chosen language (different from the language of the edited source-target text pair) and train our models on this cross-lingual-prompted dataset.

Figure 4 shows the effect of instruction language on performance on all three tasks. We note that there is no significant difference in performance between the different settings. This is likely because in each setting, owing to its multilingual instructional pre-training, the model is able to adapt well to the language of the instruction in the fine-tuning phase, hence focusing mostly on the specific tasks.

### Effect of Model Architecture

We present the task-specific results across each model type in Figure 6, observing that CLMs generally are either on par or outperform the rest of the models with GPT3.5, yielding the lowest results.

BLEU relies on n-gram overlap artificially boosting the scores, which highlights the disadvantage of the Copy baseline. Also, GPT3.5 and GPT4 consistently perform poorly in comparison to the rest of the models, which is especially surprising given GPT4's multilingual capability. In addition, this could be an artifact of the metrics since RLHF can produce excellent results that have little overlap with the references, and future human studies may shed more light on the issue.

CLMs and Seq2Seq models perform similarly on GEC and Paraphrasing, while Seq2Seq performs better on simplification. We posit that this discrepancy happens due to shorter generations from the Seq2Seq models since we observe that the seq2seq models generate significantly shorter sequences than the expected distribution (\(D=0.06,p<0.001\)),5 increasing the BLEU scores, which are

Figure 5: **Aggregated model performance by language** (for GEC, Paraphrasing, and Simplification). For each task, we aggregate the relevant metrics as described in § 5 and split them by model training.

 

sensitive to the prediction length, whereas CLMs tend to generate longer sequences (\(D=0.27,p<0.001\)). Looking at the SARI scores, we observe that the two model types do not differ significantly (\(p~{}>~{}.05\)), indicating similar performance overall.

Footnote 6: [https://github.com/google-learning/tree-](https://github.com/google-learning/tree-) 



### Generalization to new languages

We also explore the capabilities of our MEDIT models on new languages. For every task, we chose two new languages not present in the training set: one related to the language families covered in our training dataset (Table 1) and one belonging to a language family not present in the training dataset. We only considered the languages that the underlying LLM had as part of its pre-training corpora so as to ensure the models had some understanding of the languages in question. Table 2 provides a summary of the languages considered and the datasets they were sourced from, as well as the results of the language generalization experiments. We follow the same metrics for all the tasks as in SS 4.4.

MEDIT models are competitive on many unseen languages as compared to the monolingual state-of-the-art, especially on i t-Simplification, and hiGEC and Paraphrasing.7

Footnote 7: Since no multilingual models have attempted to perform the considered tasks in the respective languages, we are unable to report the metrics on others. We also do not provide any comparisons if the language-specific metrics either do not exist or are reported using different metrics.

## 6 Human Evaluations

We conduct human evaluations of our model outputs by proficient linguists (and native language speakers of the respective languages) on 50 test inputs (per task- language) to ensure they meet the instructional task-specific constraints across the various languages since text editing is often subjective, and automatic metrics are often limited in their effectiveness and accuracy. We evaluate our best-performing MEDIT model8 based on its strong performance (Table 5). Specifically, we conduct a qualitative evaluation where each annotator is shown an instructional input and output from the model and asked to rate the quality of the output on three criteria: **fluency** (Is the output grammatically correct and sound like it was written by a native speaker of the language?), **adequacy** (does the output preserve the meaning of the input?), and **accuracy** (did the model make the desired edits according to the given edit instructions?) of the edited texts, on a Likert scale ranging from _Very Bad_ to _Very Good_. We collect two annotations for each data point and adjudicate the conflicting judgments with the annotators. The annotation guidelines are provided in Appendix E. Table 3 shows the results of the evaluation. The expert annotators generally rate the model outputs as _Good_ or _Very Good_ across nearly all languages. For Arabic, the preferences are more balanced across the scale and sometimes even leaning towards the _Bad_ or _Very Bad_ across all criteria, which confirms our findings on the automatic metrics as well, indicating that while our model performs very well on languages such as English, German, Chinese, and Spanish, it still has a long way to go in terms of performance for languages such as Arabic in terms of quality, and on Chinese, Japanese, Korean, and Spanish on accuracy.

Figure 8: **Aggregated model performance by varying amounts of data samples: (0% to 100%) by task (in the order: GEC, Paraphrasing, Simplification). We aggregate the scores as described in § 5.**

Conclusions

We present mEdIT - an open-sourced dataset and set of multilingual instruction-tuned LLMs capable of following natural language instructions in seven languages to perform various textual editing tasks. It is the first publicly available set of models that heavily outperforms numerous multilingual LLMs on multiple tasks in different languages. Positive feedback from human evaluations shows that mEdIT can assist writers with various aspects of the text revision process at scale by following natural language instructions in multiple languages. Experiments on various multilingual NLP tasks demonstrate that mEdIT models outperform both their corresponding non-fine-tuned and fine-tuned models on other multilingual instruction datasets for text editing, in addition to achieving strong performance on languages unseen during the task-specific fine-tuning. By making our data/models publicly available, we hope to help make advances in multilingual intelligent writing assistants.

### Limitations

In this work, we have developed and evaluated instruction-tuned LLMs capable of editing text in multiple languages. However, this work has several limitations that can be improved in future research.

Despite our attempts to cover a diverse set of seven languages, there are still a number of languages that have not been included in our research, largely because of the lack of high-quality, human-annotated text editing data. Our system can be extended to include more languages in the future to better understand the generalization of these models to new languages, and create more accessible and ubiquitous writing assistants.

Secondly, for training and evaluation, we primarily use datasets that are publicly available in specific languages, and sometimes we generate instructions in English and translate them into multiple languages using Google Translate (for example, for simplification tasks). Despite the fact that our approach allows us to support multiple languages with reasonable development costs, data generated and translated might contain unexpected noise. Moreover, they might not best represent expert-annotated edits in different languages. In order to further improve multilingual LLMs for text editing, future research can use human-generated data for training and evaluation.

Thirdly, our system leverages numerous LLMs with billions of parameters. Considering the computing resources required for running and developing these models, replicating the results may prove difficult (which we try to address by sharing our models publicly).

Lastly, our evaluations focus only on the performance of the models on benchmark datasets for text editing, which, in turn, focus primarily on measuring superficial characteristics based on n-gram overlaps. These evaluations are limited as they do not test for the more nuanced aspects of text editing, such as fluency, coherence, and meaning preservation. The lack of human evaluations makes assessing these nuanced characteristics difficult. Future work in this direction could look at robust and scalable evaluation metrics for multilingual text editing.

## Ethics Statement

While our models offer several advantages to make intelligent writing assistance more accessible, we do recognize their potential limitations. Since our work mainly focuses on text editing, we are able to avoid many issues involving generating harmful text. Although there is still a possibility of small meaning changes for stylistic tasks due to the lack of user-specific context Kulkarni and Raheja (2023), we try to reduce the chance of hallucinations by constraining the generation to strictly editing tasks in order to reduce the chance of adding any new information or perpetuating biases.

Moreover, due to the multilingual settings, there is a risk of our models generating responses that are discriminatory, biased, or contain false information. Hence, our models, when fine-tuned on the text editing datasets, may inadvertently learn or propagate these problematic patterns. To address these concerns and minimize potential harm, we are dedicated to mitigating the risks associated with the use of our models in future research. We strongly advocate for the responsible use of our models to prevent any unintended negative consequences.

## Acknowledgments

We sincerely thank Christoph Stuber, David Rojas, Ryan Koo, Samuel Lou, Sofia Mac Gregor, Yuko Okubo, and all the annotators for their immense help, invaluable linguistic expertise, and insightful feedback on the model quality and evaluations. We also thank Max Gubin, Dongyeop Kang, and the anonymous reviewers for their helpful suggestions.

 



## References

* Alhafni et al. (2023) Bashar Alhafni, Go Inoue, Christian Khairallah, and Nizar Habash. 2023. Advancements in Arabic grammatical error detection and correction: An empirical investigation. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 6430-6448, Singapore. Association for Computational Linguistics.
* Ailan et al. (2019) Marwah Ailan, Arafat Awajana, Ahmad Al-Hasan, and Raeda Akuzhia. 2019. Towards building arabic paraphrasing benchmark. In _Proceedings of the Second International Conference on Data Science, E-Learning and Information Systems_, DATA '19, New York, NY, USA. Association for Computing Machinery.
* Alva-Manchego et al. (2020) Fernando Alves-Manchego, Louis Martin, Antoine Bordes, Carolina Scarton, Benoit Sagot, and Lucia Specia. 2020. ASSET: A dataset for tuning and evaluation of sentence simplification models with multiple rewriting transformations. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4668-4679, Online. Association for Computational Linguistics.
* Alva-Manchego et al. (2019) Fernando Alva-Manchego, Louis Martin, Carolina Scarton, and Lucia Specia. 2019. EASSE: Easier automatic sentence simplification evaluation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations_, pages 49-54, Hong Kong, China. Association for Computational Linguistics.
* Anschutz et al. (2023) Miriam Anschutz, Joshua Oehms, Thomas Wimmer, Bartlomiej Jezierski, and Georg Groh. 2023. Language models for German text simplification: Overcoming parallel data scarcity through style-specific pre-training. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 1147-1158, Toronto, Canada. Association for Computational Linguistics.
* Baez and Saggion (2023) Anthony Baez and Horacio Saggion. 2023. LSLlama: Fine-tuned LLaMA for lexical simplification. In _Proceedings of the Second Workshop on Text Simplification, Accessibility and Readability_, pages 102-108, Varna, Bulgaria. INCOMA Ltd.., Shoumen, Bulgaria.
* Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
* Boyd (2018) Adriane Boyd. 2018. Using Wikipedia edits in low resource grammatical error correction. In _Proceedings of the 2018 EMNLP Workshop W-NUT: The 4th Workshop on Noisy User-generated Text_, pages 79-84, Brussels, Belgium. Association for Computational Linguistics.
* Briakou et al. (2021)Eleftheria Briakou, Di Lu, Ke Zhang, and Joel Tetreault. 2021. Ola, bonjour, salve! XFORMAL: A benchmark for multilingual formality style transfer. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3199-3216, Online. Association for Computational Linguistics.
* Brown et al. (2020a) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020a. Language models are few-shot learners. In _Advances in Neural Information Processing Systems_, volume 33, pages 1877-1901. Curran Associates, Inc.
* Brown et al. (2020b)Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020b. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877-1901.
* Brunato et al. (2016) Dominique Brunato, Andrea Cimino, Felice Dell'Orletta, and Giulia Venturi. 2016. PaCCSS-IT: A parallel corpus of complex-simple sentences for automatic text simplification. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 351-361, Austin, Texas. Association for Computational Linguistics.
* Bryant et al. (2019) Christopher Bryant, Mariano Felice, Qistein E. Andersen, and Ted Briscoe. 2019. The BEA-2019 shared task on grammatical error correction. In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_, pages 52-75, Florence, Italy. Association for Computational Linguistics.
* Bryant et al. (2017)Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 793-805, Vancouver, Canada. Association for Computational Linguistics.
* Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_, pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
* Chowdhury et al. (2022)Jishn Ray Chowdhury, Yong Zhuang, and Shuyi Wang. 2022. Novelty controlled paraphrase generation with retrieval augmented conditional prompt tuning. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 36, pages 10535-10544.
* Chung et al. (2018) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xue Zhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.



2022. Scaling instruction-finetuned language models. _arXiv preprint arXiv:2210.11416_.
* Cotet et al. (2020) Teodor-Mihai Cotet, Stefan Ruseti, and Mihai Dascalu. 2020. Neural grammatical error correction for romanian. In _2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI)_, pages 625-631.
* Coyne and Sakaguchi (2023) Steven Coyne and Keisuke Sakaguchi. 2023. An analysis of gpt-3's performance in grammatical error correction. _arXiv preprint arXiv:2303.14342_.
* Cui et al. (2023) Yiming Cui, Ziqing Yang, and Xin Yao. 2023. Efficient and effective text encoding for chinese llama and alpaca. _arXiv preprint arXiv:2304.08177_.
* Dahlmeier and Ng (2012) Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 568-572, Montreal, Canada. Association for Computational Linguistics.
* Davidson et al. (2020) Sam Davidson, Aaron Yamada, Paloma Fernandez Mira, Agustina Carando, Claudia H. Sanchez Gutierrez, and Kenji Sagae. 2020. Developing NLP tools with a new corpus of learner Spanish. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_, pages 7238-7243, Marseille, France. European Language Resources Association.
* Du et al. (2022) Yupei Du, Qi Zheng, Yuanbin Wu, Man Lan, Yan Yang, and Meirong Ma. 2022. Understanding gender bias in knowledge base embeddings. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1381-1395, Dublin, Ireland. Association for Computational Linguistics.
* Fang et al. (2023a) Tao Fang, Jinpeng Hu, Derek F. Wong, Xiang Wan, Lidia S. Chao, and Tsung-Hui Chang. 2023a. Improving grammatical error correction with multimodal feature integration. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 9328-9344, Toronto, Canada. Association for Computational Linguistics.
* Fang et al. (2023b) Tao Fang, Shu Yang, Kaixin Lan, Derek F. Wong, Jinpeng Hu, Lidia S. Chao, and Yue Zhang. 2023b. Is chatgtr a highly fluent grammatical error correction system? a comprehensive evaluation.
* Feng et al. (2022) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 878-891, Dublin, Ireland. Association for Computational Linguistics.
* Flachs et al. (2021) Simon Flachs, Felix Stahlberg, and Shankar Kumar. 2021. Data strategies for low-resource grammatical error correction. In _Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications_, pages 117-122, Online. Association for Computational Linguistics.
* Habash and Palfreyman (2022)Nizar Habash and David Palfreyman. 2022. ZAEBUC: An annotated Arabic-English bilingual writer corpus. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_, pages 79-88, Marseille, France. European Language Resources Association.
* Hu et al. (2022) Edward J Hu, velong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In _International Conference on Learning Representations_.
* Jiang et al. (2020) Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. 2020. Neural CRF model for sentence alignment in text simplification. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7943-7960, Online. Association for Computational Linguistics.
* Katsuta and Yamamoto (2018) Akihiro Katsuta and Kazuhide Yamamoto. 2018. Crowdsourced corpus of sentence simplification with core vocabulary. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_, Miyazaki, Japan. European Language Resources Association (ELRA).
* Kementchedjhieva and Sogaard (2023) Yova Kementchedjhieva and Anders Sogaard. 2023. Grammatical error correction through round-trip machine translation. In _Findings of the Association for Computational Linguistics: EACL 2023_, pages 2208-2215, Dubrovnik, Croatia. Association for Computational Linguistics.
* Kim et al. (2022) Zae Myung Kim, Wanyu Du, Vippal Raheja, Dhruv Kumar, and Dongyeop Kang. 2022. Improving iterative text revision by learning where to edit from other revision tasks. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 9986-9999, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Koyama et al. (2020) Aomi Koyama, Tomoshige Kiyuna, Kenji Kobayashi, Moi Arai, and Mamoru Komachi. 2020. Construction of an evaluation corpus for grammatical error correction for learners of Japanese as a second language. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_, pages 204-211, Marseille, France. European Language Resources Association.
* Krishna et al. (2022) Kalpesh Krishna, Deepak Nathani, Xavier Garcia, Bidisha Samanta, and Partha Talukdar. 2022. Fewshot controllable style transfer for low-resource multilingual settings. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 7439-7468, Dublin, Ireland. Association for Computational Linguistics.
* Kulkarni and Raheja (2023) Vivelk Kulkarni and Vipul Raheja. 2023. Writing assistants should model social factors of language.
* Kumar et al. (2022)Aman Kumar, Himani Shrotriya, Prachi Sahu, Amogh Mishra, Raj Dabre, Ratish Puduppully, Anoop Kamchukuttan, Mitesh M. Khapra, and Pratyush Kumar. 2022. IndicNLG benchmark: Multilingual datasets for diverse NLG tasks in Indic languages. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 5363-5394, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics 

Philippe Laban, Jesse Vig, Marti A Hearst, Caiming Xiong, and Chien-Sheng Wu. 2023. Beyond the chat: Executable and verifiable text-editing with llms. _arXiv preprint arXiv:2309.15337_.
* Lai et al. (2022) Huiyuan Lai, Antonio Toral, and Malvina Nissim. 2022. Multilingual pre-training with language and task adaptation for multilingual text style transfer. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 262-271, Dublin, Ireland. Association for Computational Linguistics.
* Lai et al. (2023) Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dermoncourt, Trung Bui, and Thien Huu Nguyen. 2023. Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning. _arXiv preprint arXiv:2304.05613_.
* Li et al. (2023) Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023. Bactrian-x : A multilingual replicable instruction-following model with low-rank adaptation.
* Lin et al. (2022) Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Sheleifer, Punit Singh Kotara, Vishrav Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2022. Few-shot learning with multilingual generative language models. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 9019-9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Luo et al. (2023) Guoqing Luo, Yu Han, Lili Mou, and Mauajama Firdaus. 2023. Prompt-based editing for text style transfer. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 5740-5750.
* Mallinson et al. (2022) Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliakasei Severyn. 2022. EdiT5: Semi-autoregressive text editing with t5 warm-start. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 2126-2138, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Mallinson et al. (2020) Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2020. Zero-shot crosslingual sentence simplification. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5109-5126, Online. Association for Computational Linguistics.
* Martin et al. (2022) Louis Martin, Angela Fan, Eric de la Clergerie, Antoine Bordes, and Benoit Sagot. 2022. MUSS: Multilingual unsupervised sentence simplification by mining paraphrases. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_, pages 1651-1664, Mariseille, France. European Language Resources Association.
* Maruyama and Yamamoto (2018) Takumi Maruyama and Kazuhide Yamamoto. 2018. Simplified corpus with core vocabulary. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_, Miyazaki, Japan. European Language Resources Association (ELRA).
* Mizumoto et al. (2011) Tomoya Mizumoto, Mamoru Komachi, Masaaki Nagata, and Yuji Matsumoto. 2011. Mining revision log of language learning SNS for automated Japanese error correction of second language learners. In _Proceedings of 5th International Joint Conference on Natural Language Processing_, pages 147-155, Chiang Mai, Thailand. Asian Federation of Natural Language Processing.
* Mohit et al. (2014) Behrang Mohit, Alla Rozovskaya, Nizar Habash, Wajdi Zaghouani, and Ossama Obeid. 2014. The first QALB shared task on automatic text correction for Arabic. In _Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP)_, pages 39-47, Doha, Qatar. Association for Computational Linguistics.
* Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalil Almubark, Samuel Albanke, Zaid Mayfaei, Albert Webson, Edward Raff, and Colin Rapffel. 2023. Crosslingual generalization through multitask finetuning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15991-16111, Toronto, Canada. Association for Computational Linguistics.
* Napoles et al. (2015) Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical error correction metrics. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 588-593, Beijing, China. Association for Computational Linguistics.
* Naples et al. (2016) Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2016. Gleu without tuning.
* Napoles et al. (2017) Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. JFLEG: A fluency corpus and benchmark for grammatical error correction. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, pages 229-234, Valencia, Spain. Association for Computational Linguistics.
* Niu et al. (2021) Tong Niu, Semih Yavuz, Yingbo Zhou, Nitish Shirish Keskar, Huan Wang, and Caiming Xiong. 2021. Unsupervised paraphrasing with pretrained language models. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 5136-5150, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* OpenAI (2023) OpenAI. 2023. GPT-4 Technical Report.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhnii Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in Neural Information Processing Systems_, 35:27730-27744 

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318. Cited by: SS5.
* C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2020)Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research21 (140), pp. 1-67. External Links: Document, ISSN 1939-100X Cited by: SS5.
* V. Raheja, D. Kumar, R. Koo, and D. Kang (2023)Coedit: text editing by task-specific instruction tuning. arXiv preprint arXiv:2305.09857. Cited by: SS5.
* E. Reif, D. Ippolito, A. Yuan, A. Coenen, C. Callison-Burch, and J. Wei (2022)A recipe for arbitrary text style transfer with large language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Dublin, Ireland, pp. 837-848. External Links: Link, Document Cited by: SS5.
* N. Reimers and I. Gurevych (2020)Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 4512-4525. External Links: Link, Document Cited by: SS5.
* S. Rothe, J. Mallinson, E. Malmi, S. Krause, and A. Severyn (2021)A simple recipe for multilingual grammatical error correction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Online, pp. 702-707. External Links: Link, Document Cited by: SS5.
* A. Rozovskaya, H. Bouamor, N. Habash, W. Zaghouani, O. Obied, and B. Mohit (2015)The second QALB shared task on automatic text correction for Arabic. In Proceedings of the Second Workshop on Arabic Natural Language Processing, Beijing, China, pp. 26-35. External Links: Link, Document Cited by: SS5.
* M. Ryan, T. Naous, and W. Xu (2023)Revisiting non-English text simplification: a unified multilingual benchmark. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 4898-4927. External Links: Link, Document Cited by: SS5.
* H. Saggion, S. Stajner, D. Ferres, K. Cheng Sheang, M. Shardlow, Kai North, and Marcos Zampieri (2022)Findings of the TSAR-2022 shared task on multilingual lexical simplification. In Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022), Abu Dhabi, United Arab Emirates (Virtual). Association for Computational Linguistics. Cited by: SS5.
* V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafaei, A. Chaffin, A. Stiegler, T. Le Scao, A. Raja, et al. (2022)Multitask prompted training enables zero-shot task generalization. In ICLR 2022-Tenth International Conference on Learning Representations, Cited by: SS5.
* T. Schick, J. A. Yu, Z. Jiang, F. Petroni, P. Lewis, G. Izacard, Q. You, C. Nalampantis, E. Grave, and S. Riedel (2023)PEER: a collaborative language model. In The Eleventh International Conference on Learning Representations, Cited by: SS5.
* short papers, Vol. , ITonto, Italy, pp. 1-8. External Links: Link, Document Cited by: SS5.
* L. Seiffe, F. Kallel, S. Moller, B. Naderi, and R. Roller (2022)Subjective text complexity assessment for german. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, Marseille, France, pp. 707-714. External Links: Link, Document Cited by: SS5.
* Ankur Sonawane, S. Kumar Vishwakarma, B. Srivastava, and A. Singh (2020)Generating inflectional errors for grammatical error correction in Hindi. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: Student Research Workshop, Suzhou, China, pp. 165-171. External Links: Link, Document Cited by: SS5.
* X. Sun, T. Ge, S. Ma, J. Li, F. Wei, and H. Wang (2022)A unified strategy for multilingual grammatical error correction with pretrained cross-lingual language model. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pp. 4367-4374. External Links: Link, Document Cited by: SS5.
* Toshikazu Tajiri, M. Komachi, and Y. Matsumoto (2012)Tense and aspect error correction for esl learners using global context. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Jeju Island, Korea, pp. 198-202. External Links: Link, Document Cited by: SS5.
* Y. Tanaka (2001)Compilation of a multilingual parallel corpus. Proceedings of PACLING 2001, pp. 265-268. External Links: Link, Document Cited by: SS5.
* S. Tonelli, A. Palmero Aprosio, and F. Saltori (2016)Simpitikit: a simplification corpus for italian. Proceedings of the Third Italian Conference on Computational Linguistics (CLiC-it) 2016, pp. 291-296. External Links: Link, Document Cited by: SS5.
* Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. (2023)LIMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

 

Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, Tianxiang Hu, Shangjie Li, Binyuan Hui, Bowen Yu, Dayiheng Liu, Baosong Yang, Fei Huang, and Jun Xie. 2023. Polylm: An open source polyglot large language model.
* Witteveen and Andrews (2019) Sam Witteveen and Martin Andrews. 2019. Paraphrasing with large language models. In _Proceedings of the 3rd Workshop on Neural Generation and Translation_, pages 215-220, Hong Kong. Association for Computational Linguistics.
* Workshop (2023) BigScience Workshop. 2023. Bloom: A 176b-parameter open-access multilingual language model.
* Wu et al. (2023) Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. 2023. Chatgpt or grammar? evaluating chatgpt on grammatical error correction benchmark.
* Xu et al. (2015) Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. _Transactions of the Association for Computational Linguistics_, 3:283-297.
* Xu et al. (2016a) Wei Xu, Courtney Napoles, Ellie Pavlick, Quanz Chen, and Chris Callison-Burch. 2016a. Optimizing statistical machine translation for text simplification. _Transactions of the Association for Computational Linguistics_, 4:401-415.
* Xu et al. (2016b) Wei Xu, Courtney Napoles, Ellie Pavlick, Quanz Chen, and Chris Callison-Burch. 2016b. Optimizing statistical machine translation for text simplification. _Transactions of the Association for Computational Linguistics_, 4:401-415.
* Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Ramil Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 483-498, Online. Association for Computational Linguistics.
* Yang et al. (2023a) Shiping Yang, Renliang Sun, and Xiaojun Wan. 2023a. A new dataset and empirical study for sentence simplification in Chinese. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8306-8321, Toronto, Canada. Association for Computational Linguistics.
* Yang et al. (2023b) Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. 2023b. Bigtranslate: Augmenting large language models with multilingual translation capability over 100 languages. _arXiv preprint arXiv:2305.18098_.
* Yang et al. (2020) Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jay Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2020. Multilingual universal sentence encoder for semantic retrieval. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 87-94, Online. Association for Computational Linguistics.
* Yang et al. (2019) Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3687-3692, Hong Kong, China. Association for Computational Linguistics.
* Yoon et al. (2023) Soyoung Yoon, Sungjoon Park, Gyufwan Kim, Junhee Cho, Kihyo Park, Gyu Tae Kim, Minjoon Seo, and Alice Oh. 2023. Towards standardizing Korean grammatical error correction: Datasets and annotation. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 6713-6742, Toronto, Canada. Association for Computational Linguistics.
* Zhang and Lapata (2017) Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 584-594, Copenhagen, Denmark. Association for Computational Linguistics.
* Zhang et al. (2023) Ying Zhang, Hidetaka Kamigaito, and Manabu Okumura. 2023. Bidirectional transformer reranker for grammatical error correction. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 3801-3825, Toronto, Canada. Association for Computational Linguistics.
* Zhang et al. (2019) Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1298-1308, Minneapolis, Minnesota. Association for Computational Linguistics.
* Zhao et al. (2018) Yuanyuan Zhao, Nan Jiang, Weiwei Sun, and Xiaojun Wan. 2018. Overview of the nlpcc 2018 shared task: Grammatical error correction. In _Natural Language Processing and Chinese Computing_.
* Zhu et al. (2023) Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. Extrapolating large language models to non-engish by aligning languages.
* Zhu et al. (2018) Yaoming Zhu, Sidii Du, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_, SIGIR '18, page 1097-1100, New York, NY, USA. Association for Computing Machinery 

Training and Evaluation Datasets

### Grammatical Error Correction

ArabicWe report on three publicly available Arabic GEC datasets. The first two come from the QALB-2014 (Mohit et al., 2014) and QALB-2015 (Rozovskaya et al., 2015) shared tasks. The third is the newly created ZAEBUC dataset (Habash and Palfreyman, 2022; Alhafni et al., 2023). QALB-2014 consists of native/L1 user comments from the Aljazeera news website, whereas QALB-2015 consists of essays written by Arabic L2 learners with various levels of proficiency. It is worth noting that the QALB-2015 dataset has two test sets consisting of L1 and L2 data. In this work, we report results on the L1 test set. The ZAEBUC dataset comprises essays written by native Arabic speakers, which were manually corrected. We use the MaxMatch (M\({}^{2}\))Scorer (Dahlmeier and Ng, 2012) for the evaluation.

EnglishWhen it comes to English, we use the Write & Improve + LOCNESS (W&I) corpus released in the Building Educational Applications (BEA) shared task on GEC (Byvant et al., 2019). We also use the NAIST Lang-8 corpus (Tajiri et al., 2012), which is one of the largest and most widely used datasets for English GEC. To test our systems, we use the JFLEG (Napoles et al., 2017) dataset. We use GLEU (Napoles et al., 2015, 2016) for the evaluation.

GermanFor German, we use the Falko-MERLIN corpus (Boyd, 2018), which consists of sentences written by L2 learners that were manually corrected. We use the MaxMatch (M\({}^{2}\))Scorer Dahlmeier and Ng, 2012) for the evaluation.

SpanishFor Spanish, we use the publicly available COWS-L2H (Davidson et al., 2020) dataset. COWS-L2H consists of essays written by Spanish L2 learners at the university level in the United States. We use ERRANT (Byrant et al., 2017) for the evaluation.

ChineseFor Chinese, we use the data that is part of the NLPCC18 shared task (Zhao et al., 2018). The training data used in the shared task was collected from the NAIST Lang-8 corpus (Tajiri et al., 2012), whereas the test data consists of manually corrected sentences written by Chinese L2 learners. We use GLEU (Napoles et al., 2015, 2016) for the evaluation.

JapaneseFor Japanese, we use the NAIST Lang-8 corpus (Mizumoto et al., 2011) to train our systems. For evaluation, we use the Japanese L2 TECJ-JL dataset (Koyama et al., 2020). We use GLEU (Napoles et al., 2015, 2016) for the evaluation.

KoreanWe use the recently created Kor-Union dataset (Yoon et al., 2023). Kor-Union was created by collecting and combining GEC data from various sources. This includes essays written by Korean native/L1 speakers and L2 learners. We use the MaxMatch (M\({}^{2}\))Scorer (Dahlmeier and Ng, 2012) for the evaluation.

### Paraphrasing

ArabicFor training, we use the Arabic SemEval Paraphrasing (ASEP) corpus, which sourced three existing Arabic semantic similarity datasets released during SemEval 2017 Task 1 (Cer et al., 2017), consisting of roughly 1100 sentence pairs. For our purposes, similar to them, we only keep the sentence pairs with a semantic similarity score \(\geq\) 3.25, which leads to 603 pairs. We also inverted the pairs for training, leading to a total of 1.2k training pairs. For evaluation, we use the evaluation dataset that was used for SemEval 2017 Track 1, but with the same similarity threshold as the training data, consisting of 67 sentence pairs. This evaluation set consists of sentences from the SNLI Corpus (Bowman et al., 2015) that were human-translated into Arabic, provided by CMU-Qatar by native Arabic speakers with strong English skills.

We also source from the Arabic Question Similarity (Shared Task 8) organized at the Workshop on NLP Solutions for Under-Resourced Languages (NSURL 2019) (Seelawi et al., 2019). The dataset was developed by mawdoo, and consists of 12k pairs for training and 3715 for testing. For both training and evaluation, we filter the semantically similar pairs (similarity score of 1), which leaves us with 10.7k training and 1.7k test pairs.

We also use the Arabic Paraphrasing Benchmark (APB) dataset (Alian et al., 2019), which consists of 1010 Arabic sentence pairs that are collected from different Arabic books. Paraphrasing was performed manually using six transformation procedures (i.e., addition, deletion, expansion, permutation, reduction, and replacement). Similar to other evaluation sets, we only keep the sentence pairs with a semantic similarity score \(\geq\) 3.25, which leads to 286 pairs 

EnglishParaphrase Adversaries from Word Scrambling (PAWS) is a dataset that contains pairs of sentences with a high lexical overlap Zhang et al. (2019). We use the PAWS dataset for training and evaluation.

German, Spanish, Japanese, Korean, ChineseWe use the Cross-lingual Paraphrase Adversaries from Word Scrambling Yang et al. (2019) dataset (PAWS-X), which was created by translating a subset of the PAWS validation and test sets to six other languages by professional translators.

### Simplification

We draw on a variety of existing text simplification datasets in various languages. Table 4 shows the different simplification datasets we draw on in our work and also outlines the training, development, and test settings.

A major issue with text simplification is the absence of publicly available, human-annotated, sentence-level parallel corpora for some of the languages we considered, such as Arabic, Spanish, and Korean. Therefore, we addressed this by translating the Text Simplification datasets for English to these three languages, in which the parallel data is absent. One potential limitation of this approach could be the poor quality of the translation models, which could negatively impact the overall data quality. Therefore, we use the latest Google Translate API9 to construct the translated data, and further verify the quality of the translated text with human annotators (native speakers) for a subset of the data. We chose the Google API since it performed best amongst the other open-source machine translation

\begin{table}
\begin{tabular}{c c l l l} \hline \hline
**Task** & **Language** & **Dataset** & **Split** & **Size** \\ \hline \multirow{8}{*}{G grammatical} & & BEA Bryan et al. (2016) & Train & 1.3M \\  & \(a\) & FELBO Angelos et al. (2017) & Dev, Test & 754, 747 \\ \cline{2-5}  & & QALB-2014 Okutdinov et al. (2014) & & 19k, IR, 968 \\  & at & QALB-2015 Okutdinov et al. (2015) & Train, Dev, Test & 310, 354, 920 \\  & & ZABUC Mirabul and Palfreyniemi (2022) & & 180, 33, 31 \\ \cline{2-5} Enzer & \(d\) & Fabric-MEDLIN Boyd (2018) & Train, Dev, Test & 19k, 2.5k, 2.3k \\ \cline{2-5} Correction & \(e\) & COS-LSTM Davidson et al. (2020) & Train, Dev, Test & 39k, 85, 86 \\ \cline{2-5}  & & Lang’s Affennero et al. (2011) & Train & 1.58M \\  & ja & TECL Kioyama et al. (2020) & Test & 1.9k \\ \cline{2-5}  & \(b\) & Kor-Union Yoon et al. (2023) & Train, Dev, Test & 108.9k, 2.3k, 2.3k \\ \cline{2-5}  & _zh_ & NLPC-2018 Zhao et al. (2018) & Train, Dev, Test & 540k, 53, 8x, 2k \\ \hline \multirow{8}{*}{Praplressing} & _en_ & PAWS Zhang et al. (2019) & Train, Dev, Test & 49k, 8x, 8x \\ \cline{2-5}  & & SunfeiBoard’Dick et al. (2017) & Train & 1.2k, 67 \\ \cline{2-5}  & at & NNKRL Zhou et al. (2019) & Train, Tech & 243, 37k \\ \cline{2-5}  & APB Alain et al. (2019) & & 286 \\ \cline{2-5}  & _de_ & & & \\ \cline{2-5}  & _re_ & PawS-X Ying et al. (2019) & Train, Dev, Test & 49k, 2x, 2x \\ \cline{2-5}  & _ja_ & & & \\ \cline{2-5}  & _zh_ & & & \\ \hline \multirow{8}{*}{Simplification} & _en_ & Wikdar.large Zhang and Lapstan (2017) & Train, Dev, Test & 296k, 2.359 \\  & _wikkarin Jiang et al. (2020) & & 57k, 84, 8x \\  & _nwebsite_ & NWEBLA Xu et al. (2015) & Train & 94k \\  & _assert_ (Axo-Mnchregelvi et al., 2020) & Dev, Test & 2000, 359 \\ \cline{2-5}  & _ar_ & NEWSIKA-Auto-AR & Train & 94k \\  & _assert_ Auto-AR & Dev, Test & 100, 359 \\ \cline{2-5}  & dp & GOLLE Mullison et al. (2020) & Train, Dev, Test & 95k, 122, 118 \\  & TerCompiexplexplvBeiffe 

models and APIs we tested1011.

Footnote 10: [https://libretranslate.com/](https://libretranslate.com/)

Footnote 11: [https://www.deepl.com/translator](https://www.deepl.com/translator)

EnglishFor English, we used Wikilarge Zhang and Lapata (2017), WikiAuto Jiang et al. (2020), and Newsela Xu et al. (2015) datasets for training. WikiAuto is a neural CRF-aligned corpus of original and simple Wikipedia documents that are automatically aligned to generate sentence pairs, whereas the Newsela Xu et al. (2015) dataset contains automatically aligned sentence pairs from documents that are generated by professional writers at Newsela for various grade levels. For testing, we use ASSET Alva-Manchego et al. (2020), which contains ten high-quality human written simplifications for each of the 2,390 sentences from the TurkCorpus Xu et al. (2016b).

GermanWe use the GEOLino Mallinson et al. (2020) and TextComplexityDE Seiffe et al. (2022) datasets for both training and testing. GEOInoTest contains text about nature, physics, and people from GeoLino, a children's magazine that was manually simplified by a German linguist to a five to seven-year-old reading level. TextComplexityDE contains 250 complex sentences from German Wikipedia that native speakers manually simplified.

JapaneseWe use the EasyJapanese Maruyama and Yamamoto (2018) and EasyJapaneseExtended Maruyama and Yamamoto (2018) datasets for training and testing. EasyJapanese contains 50k sentence pairs that were manually created by five students by simplifying text from the Tanaka corpus Tanaka (2001). The EasyJapaneseExtended dataset contains an additional 34.4k sentences from the Tanaka corpus with simplifications crowdsourced.

Arabic, Spanish, KoreanFor Arabic, Spanish and Korean, as there were no publicly available sentence-level parallel datasets available, we translated the English simplification datasets. Specifically, we translated the English Newsela dataset for training and ASSERT for testing using the Google Translate API, giving us 94k and 359 examples for training and testing, respectively.

ChineseWe found no publicly available dataset for training Chinese Simplification. Therefore, we again translated the English Newsela training dataset into Chinese. However, for the testing set, we use the CSS Yang et al. (2023) dataset. CSS consists of two human-written simplifications for each of the 383 original sentences from the PFR corpus. 12

Footnote 12: [https://www.heywhale.com/mm/dataset](https://www.heywhale.com/mm/dataset)

## Appendix B Data Preparation

For Seq2Seq models, we prepend the task-specific instructions to the input to the encoder for each language, performing a full parameter update on the entire sequence. We construct the CLM datasets by wrapping each example in model-specific instructions13 computing the loss _only_ on the target text; hence, in the Native and Random settings, the model does not optimize for the specific "translated" instructions.

Footnote 13: PolyLM and Batcrian-X follow different prompt templates. Details in Appendix D.

We randomly sample 10k examples from the original datasets for each language-task combination and keep the original validation and test tests (see Table 4). We chose this quantity as a balance between computational cost and qualitative performance based on the insights from Raheja et al. (2023). Moreover, in our experiments, we did not find a significant impact of increasing the data quantity per task-language combination beyond 10k. In the Spanish GEC task, we only have 398 data points, so this portion of our data is considerably smaller than the rest of the languages. Furthermore, in the GEC and Simplification training data for all languages, we reserve 20% of the set as input-output pairs without any edits to avoid over-corrections by the model.

## Appendix C Results

We present the full set of results for our best-performing random-language setting. In addition to the marginally higher performance that all trained models demonstrate in this setting, we choose this as our representative setting due to the fact that it is the most capable setting for our models, allowing them to perform cross-lingual editing.

We present the results for all trained models on all datasets, as well as the No Edits (Copy) baseline and State-of-the-art LLMs. In the interest of space and interpretability, we skip the detailed results for the untrained baseline as we already show them to massively underperform relative to the trained models. We also compare our results to the previously reported SOTA results across tasks and languages: 

Grammatical Error CorrectionFor GEC, we compare against the following SOTA results: Arabic (Alhafini et al., 2023), English (Zhang et al., 2023), German (Fang et al., 2023a), Korean (Yoon et al., 2023), Spanish (Flachs et al., 2021), and Japanese (Koyama et al., 2020).

SimplificationFor simplification, we take the results from the fine-tuned mT5 models from Ryan et al. (2023). They compare multiple settings in their work, such as using data from a single training dataset, from a single language, and from multiple languages. We pick the score from any setting that gives the highest score. Thus, the SARI score for German might be picked from one setting while the BLEU score for German from some other setting, and so on. This ensures that we take their strongest models for each use case. We still see that our models perform better than them in most languages and datasets.

ParaphrasingTo paraphrase, most of the related works that utilize the PAWS-X dataset Yang et al. (2019) have used it for paraphrase detection, and not for paraphrase generation. Hence, we are unable to report any comparable SOTA results for this task.

## Appendix D Task Verbalizers

We present the full list of our manually curated task-specific verbalizers used for training and evaluations in Table 6, Table 7, and Table 8.

 



\begin{table} (b) Paraphrasing 
 (c) Text Simplification (c) Part 1: **Full set of results on the best-performing setting of 10k random-language-prompted data.** For GEC, we report GLEU or \(F_{0.5}\) depending on the metric as described in Appendix A. For Paraphrasing, the first quantity is \(1-\text{Self-BLEU}\) and the second one is the accuracy of semantic similarity as calculated by \(m\)USE (explained in SS 4.4). Finally, for Simplification, the first quantity is SARI, and the second one is BLEU. In terms of models, bx-llama-7b denotes the MBZUAI/bactrian-x-llama-7b-merged checkpoint, and bx-llama-7b denotes the MBZUAI/bactrian-x-llama-13b-merged one. \\ \hline \hline  & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} \\ \cline{2-9} Model & Version & Size & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} \\ \cline{6-8} \cline{10-11} \multirow{-2}{*}{Model} & Version & Size & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} \\ \cline{6-8} \cline{10-11} \multirow{-2}{*}{Code} & & & & & & & & & & & \\ \hline Copy & – & – & 00.01/1000 & 0.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 \\ \hline GMP3 & ggt-1+1-symptote11 & - & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 0.01/1004.0 \\ \hline GMP4 & ggt-1+1-symptote11 & - & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 \\ \hline Reflexes & ggt-1+1-symptote11 & - & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 \\ \hline Reflexes & ggt-1+1-symptote11 & - & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 \\ \hline Reflexes & ggt-1+1-symptote11 & - & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 0.01/1004.0 \\ \hline Reflexes & ggt-1+1-symptote11 & - & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 0.01/1004.0 \\ \hline Reflexes & ggt-1+1-symptote11 & - & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 \\ \hline Reflexes & ggt-1+1-symptote11 & - & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 \\ \hline Reflexes & ggt-1+1-symptote11 & - & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 \\ \hline Reflexes & ggt-1+1-symptote11 & - & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 \\ \hline Reflexes & ggt-1+1-symptote11 & - & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 \\ \hline Reflexes & ggt-1+1-symptote11 & - & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 \\ \hline Reflexes & ggt-1+1-symptote11 & - & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 \\ \hline Reflexes & ggt-1+1-symptote11 & - & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 \\ \hline Reflexes & ggt-1+1-symptote11 & - & 00.01/1004.0 & 00.01/1004.0 & 00.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 \\ \hline Reflexes & ggt-1+1-symptote11 & - & 00.01/1004.0 & 00.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 \\ \hline Reflexes & ggt-1+1-symptote11 & - & 00.01/1004.0 & 00.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 & 0.01/1004.0 \\ \hline Reflexes & ggt-1+1-symptote11 & - & 00.01/1004.0 & 00.





\begin{table}
\begin{tabular}{c c l l} \hline \multicolumn{2}{l}{**Language**} & \multicolumn{1}{c}{**Verbalizers**} \\ \hline \multirow{4}{*}{Arabic} & \multirow{3}{*}{Chinese} & Fix grammar in this sentence & Fix grammar in the sentence \\  & Fix grammar errors & Fix grammatical errors & Fix grammatical errors \\  & Fix all grammatical errors & Fix grammatical errors in this sentence \\  & Fix grammatical mistakes in this sentence & Fix grammars in this sentence \\ English & Fix affordworks in the sentence & Make the sentence grammatical & Make the sentence different from this text \\  & Fix errors on its text & Update to remove grammatic errors & Remove and grammatical errors from this text \\  & Improve the grammar of this text & Improve the grammaticality & Improve the grammaticality of this text \\  & Improve the grammaticality of this sentence & Grammar improvements & Remove grammar mistakes \\  & Remove grammatical mistakes & Fix the grammar mistakes & Fix the grammatical mistakes \\ \hline \multirow{5}{*}{German} & Gammaanki Romigerian & Gammaanki in theorem Star korpiengrigent \\  & Gammaankiewicz bleuben & Alle Gramankieller betheben & Atomandexikier in descent Star korpiengrigent \\  & Graphus des Stores Serieurgenuvers & Unambigerenation in Star betheben & MapNet so we Sor Sorgangangankules hertteh \\  & Muchan Sie den Genstät Belfoel & Fribler in diesem Text leebhen & Update man faitrenken von Gammaankikloren \\  & Enformed Sea die Gemanankiewicz absens Stern Tert & Vetesen Sie de Gemanankiess Stees & Verbeenschen Sie der Gemmatikhe Messersung \\  & Viewsement SNe de Gemanankiewicz dieses Teates & Verbeenschen Sie de Gemanankiewicz \\  & Gammaankiewicz extension & Bellecten Sie die Gemanankiewicz \\ \hline \multirow{6}{*}{Japanese} & \multirow{7}{*}{Japanese} & \(\times\)\



\begin{table}
\begin{tabular}{c l p{142.3pt} p{142.3pt}} \hline
**Language** & **Verbalizers** \\ \hline \multirow{5}{*}{A Arabic} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\  & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\  & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\ Chinese & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\  & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\  & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\  & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\ English & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\  & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\  & \multicolumn{1}{p{142.





\begin{table}
\begin{tabular}{l l l l} \hline \multicolumn{2}{c}{**Language**} & \multicolumn{1}{c}{**Verbalizers**} \\ \hline Arabic & & & \\  & & & \\ \cline{3-4} \multirow{-2}{*}{Chinese} & \(\#\times\) & \(\#\times\) & \(\#\times\) \\  & \(\#\times\) & \(\#\times\) & \(\#\times\) \\  & \(\#\times\) & \(\#\times\) & \(\#\times\) \\  & \(\#\times\) & \(\#\times\) & \(\#\times\) \\  & \(\#\times\) & \(\#\times\) & \(\#\times\) \\ \hline English & Paephrase the sentence & Paephrase this sentence & Paephrase this text \\  & Paephrase & Write a panelizes for the sentence & Write a paragraphed version of the sentence \\  & Rewrite the sentence with different working & Use different working & Reviewing this sentence \\  & Reward this sentence & Referring this sentence & Rescribe this text \\  & Reward this text & Referring this text \\ \hline German & Unstenchenbe Set den Stat & Unschebenben Sie deuen Star & Unscheben Sie deren Text \\  & Unischerbenbene & Substitution see une Umscherbung fur den Staz & Schurchie She eine parfuglicence Vossie des Stattes \\  & Schrehen Sie das Strze mit citem anners Werdelt um & Aborde Fermentimen verwenden & Shorneh Siech diesen Satz mit \\  & Formulner bis Ges descent Satz mit & Desen Test unterschten & Desen Test untermefnemente \\ \hline Japanese & \(\#\times\) & \(\#\times\) & \(\#\times\) \\  & \(\#\times\) & \(\#\times\) & \(\#\times\) \\  & \(\#\times\) & \(\#\times\) & \(\#\times\) \\  & \(\#\times\) & \(\#\times\) & \(\#\times\) \\ Korean & \(\#\times\) & \(\#\times\) & \(\#\times\) \\  & \(\#\times\) & \(\#\times\) & \(\#\times\) \\  & \(\#\times\) & \(\#\times\) & \(\#\times\) \\ Spanish & Parafilree la motion & Performance est erosion & Postfracse ceite texto \\  & Pavelfasser & Escribe une paaffirations do he carición & Electric plus aversion parallazona del lo torsion \\  & Receible la position com une reflection differente & User una endecide difference & Rescribe este curción \\  & Reformella eta creation & Rescribe este texto & Referntals

Human Evaluation Annotation Guidelines

The human experts were asked to rate the **fluency**, **adequacy**, and **accuracy** separately, following the guidelines in Figure 9, Figure 10 and Figure 11, respectively.

Given the instruction to edit the text and the text to be edited, rate the output on the following dimensions:

**Adequacy**: does the output preserve the meaning of the original sentence? Please use the following scale for your evaluations along every dimension:

**Very Good**: The output fully preserves the meaning of the text.

**Good**: The output is semantically similar to the input with minor errors.

**Average**: The output is semantically similar to the input but has significant errors.

**Bad**: The output is barely similar to the input.

**Very Bad**: The output has opposite meaning to the input.

Given the instruction to edit the text and the text to be edited, rate the output on the following dimensions:

**Accuracy**: how well do the edits made in the output follow the given instructions? Please use the following scale for your evaluations along every dimension:

**Very Good**: The output follows the instructions exactly.

**Good**: The output generally follows the instructions with minor errors.

**Average**: The instructions are partially followed but has errors.

**Bad**: The output did not follow the instructions but did not significantly make the text unusable.

**Very Bad**: The instructions were completely ignored, and wrong edits were made to make the text unusable.

Figure 11: **Annotation guidelines for human evaluations for Accuracy.**

Figure 9: **Annotation guidelines for human evaluations for Fluency.**