Reducing Sequence Length by Predicting Edit Spans with Large Language Models
Authors: Masahiro Kaneko
Naoaki Okazaki
Venue: EMNLP 2023 Main (2023)
Keywords: Large language model
efficiency
instruction tuning
Local sequence transduction task

Abstract: Large Language Models (LLMs) have demonstrated remarkable performance in various tasks and gained significant attention.
LLMs are also used for local sequence transduction tasks, including grammatical error correction (GEC) and formality style transfer, where most tokens in a source text are kept unchanged.
However, the models that generate all target tokens in such tasks have a tendency to simply copy the input text as is, without making needed changes, because the difference between input and output texts is minimal in the training data.
This is also inefficient because the computational cost grows quadratically with the target sequence length with Transformer.
This paper proposes predicting edit spans for the source text for local sequence transduction tasks.
Representing an edit span with a position of the source text and corrected tokens, we can reduce the length of the target sequence and the computational cost for inference.
We apply instruction tuning for LLMs on the supervision data of edit spans.
Experiments show that the proposed method achieves comparable performance to the baseline in four tasks, paraphrasing, formality style transfer, GEC, and text simplification, despite reducing the length of the target text by as small as 21%.
Furthermore, we report that the task-specific fine-tuning with the proposed method achieved state-of-the-art performance in the four tasks.

Reducing Sequence Length by Predicting Edit Spans with Large Language Models
Masahiro Kaneko Naoaki Okazaki²
MBZUAI
Tokyo Institute of Technology okazakiic. titech.ac...
Masahiro www....annnnkbbaaaaaa..... ac. ae
Abstract
are becoming larger and larger. However, the com- putational cost of inference is a severe bottleneck of many practical applications, especially when the number of parameters in an LLM is massive (Ben- der et al., 2021; Kraus et al., 2023).
Large Language Models (LLMs) have demon- strated remarkable performance various tasks and gained significant attention. L.Mss are also used for local sequence transduction tasks, including grammatical errоr correction (GEC) and formality style transfer, where most tokens in a source text are kept unchanged. However, the models that generate all target tokens in such tasks have a tendency to sim- ply copy the input text as is, without mak- ing needed changes, because the difference between input and output texts is minimal in the training data. This is also inefficient be- cause the computational cost grows quadrat- ically with the target sequence length with Transformer. This paper proposes predicting edit spans for the source text for local sequence transduction tasks. Representing an edit span with position of the source text and corrected tokens, we can reduce the length of the target sequence and the computational cost infer- ence. We apply instruction tuning for LLMs on the supervision data of edit spans. Experi- ments show that the proposed method achieves comparable performance to the baseline in four tasks, paraphrasing. formality style transfer, GEC, and text simplification, despite reducing the length of the target text by as small as %. Furthermore, we report that the task-specific fine-tune fine-tuning with the proposed method achieved state-of-the-art performance in the four tasks.
Meanwhile, LLMs are also used for local se- quence transduction tasks, such as paraphrasing, formality style transfer, Grammatical Error Cor- rection (GEC), and simplification (Kaneko et al., 2022; Reif et al., 2022; Wu et al., 2023a; Wang et al., 2022; Kaneko and Okazaki, 2023), where only small portion of the source text is edited. Most tokens in source text are kept unchanged in these tasks. For example, the source text, "Many years ago, the situation is different," " and the target text, "Many years ago, the situation was different," of the GEC task mostly share the common tokens except for the underlined tokens (is and was).
Existing methods of downstream tasks do not make use of the characteristics of local sequence transduction (Reif et al.. 2022; Wu et al., 2023a; Wang et al., 2022), simply generating all target tokens. In this paper, hypothesize that this treat- ment is disadvantagcout in achieving high perfor- mance in terms of task accuracy and computational time. More specifically, is inefficient generate unchanged tokens (e.g. Many, years, ago, the, sit- uation, different) in the previous example because the model must copy many source tokens only to increase the length of the target sequence.
1 Introduction
This study proposes to predict a set of edit spans, which represent the changed parts of the target text relative to the source tokens. Omitting unedited tokens that occupy most of the target text, we can reduce the length of the target text and the infer- time for local sequence transduction tasks. Figure shows the process of creating set of edit spans from source and target texts in GEC. First, we align tokens the source and target texts extract the edit locations and tokens and convert them into set of edit spans. In the example shown
Large Language Models (LLMMs), including Chat- GPT and Bard?, have exhibited exceptional per- formance across a range of natural language pro- cessing (NLP) tasks and amassed significant user base (Brown et al., 2020; Chowdhery et al., 2022; OpenAL, 2023). As performance gains are brought from the increases in model size (Kaplan et al., 2020; Wei et al., 2022; Zhao et al., 2023), LLMS
'https://chate oww.oi..eccooooooc com/ com/

---

Through the thousands of years, most Chinese scholars have been affected by Confucianism.
Output
Rule-based Conversion
1, 'the"), (8, "have been"), (11, 12,
Pasition & Edit
Through thousands scholars are affected by the Confucienism Token Set
LLM
Instruction
Source
Rewrite the inpu: text into grammetically correct text
Input
Through thousands of years, most Chinese scho ars are affected by the Confucianism,
10
1314
Figure I: Inference of instruction tuned L.MSs using edit spans. LLMs take instruction text and source text as input and output only the positions and tokens for rewriting. Rule-based conversion applies the outputted positions and tokens of the rewriting to the source text and produces the plaintext output.
in Figure 1, the edit spans (1, 1, "the"), (8, 9, "have been"), (12, 13, """) are created from the source text "Through thousands of years, most Chinese scholars are greatly affected by the Confucianism." and the target text "Through the thousands of years, most Chinese scholars have been greatly affected by Confucianism... LLMs are fine-tuned using pairs of source text and edit spans with the instruc- tions.
accurate compared alignment methods that only use surface information. Furthermore, linguistic alignment merges token alignments using recursive rules to create alignments for multiple tokens, such as "have been" in Figure 1.
To indicate the edit position identified by the alignment, 0 is assigned before the first token of the source text, and an index is sequentially assigned the space after each token. When the length of the source text is N, N assigned after the last token. The edit span is represented by the tuple of the start position of the source text, the end position of the source text, and the token result after being edited.
We conducted experiments four local quence transduction tasks: paraphrasing. formality style transfer, GEC, and simplification. The pro- posed method achieved comparable performance to the baseline that directly outputs the target text. In these tasks, the proposed method could reduce the sequence length on the target side by 32% on aver- age and by small 21 % in GEC. Furthermore, the proposed method with task task-specific fine-tuning achieved state-of-the-art (SoTA) performance in the four tasks.
There are three types of edit operations: insert, replace, and delete; we explain them using the ex- ample in Figure 1. The tuple (1, 1, "the") represents the operation to insert "the". : In an insertion oper- ation, both the start and end positions are set to the same position where token is inserted in the source. The tuple stands for inserting "the" be- tween the tokens located the 1st position. The tuple (8, 9, "have been") presents the operation replace "are" with "have been".. : By specifying the 8th and 9th positions of the source text, this tuple targets the "are" and rewrites them as "have been". The tuple (12, 13, """) represents the operation to delete "the". It points "the" by specifying the 12th and 13th positions in the source text. Because the target token after this edit operation is empty, this tuple corresponds to removing "the".
2 Edit Spans
2.1 Edit Span Extraction
To extract the editing locations and results of the source and target texts, we calculate the alignment between the tokens in each text. We use linguis- tical alignment, which incorporates linguistic in- formation, to perform the alignment (Felice et al., 2016). Linguistical alignment is a method based the Damerau-Levenshinn tokens by considering not only the distance be- tween tokens but also the match of their lemma, part-of-speech, and character features, weighted accordingly. Taking into account the linguistic in- formation of tokens, linguistical alignment is more
algorithm that aligns
2.2 Instruction Tuning with Edit Spans
Instruction tuning fine-tunes LLMs by using nat- ural language instructions describing a task (Wei

---

al., 2021). Compared to the conventional fine- tuning that specializes the model for specific task, instruction tuning aims for generalization to var- ious tasks by training LLMs to respond well to many kinds of instructions. Therefore, instruction tuning is used for training many LLMs in an open- ended setting (Ouyang et al., 2022; Chung et al., 2022; Wang et al., 2022; Wu et al., 2023b). We use the description of local sequence transduction tasks as instructions to perform instruction tuning of LLMs. We provide the LLMs with instructions and source text, and train the LLMs to generate edit spuns. When there are multiple edits, they are concatenated with commas like "1 I the, 8 9 have been, 12 13". When no editing is required in the source text, "None" is given as the gold text.
ation data. The dataset is comprised of essays com- posed by college students from the National Uni- versity of Singapore, covering broad spectrum of subjects, including environmental pollution, health- care, and more. We used the M score (Dahlmeier and Ng, 2012) as the evaluation metric. For GEC, we provide the instruction text "Rewrite the inpul text into grammatically correct text.".
Paraphrasing Quora published a dataset that in- cludes more than 400K lines of potential question duplicate pairs Of these pairs, 150K question pairs were labeled as paraphrases. Only those la- beled paraphrase question pairs used training, development, and test sets. We used BLEU-4 (Pap- ineni et al., 2002), ROUGE-I, and ROUGE-2 (Lin, 2004) to evaluate LLMs, following previous re- search (Kumar et al., 2020; Meng et al., 2021; Li et al., 2022). For paraphrasing, we provide the instruction text "Rewrite the input text into para- phrased text."
Recent LLMs are expected to have the ability to handle unknown tasks and various tasks, achieve generality. is important that learning through edit spans does not degrade the performance of tasks other than local sequence transduction tasks. Therefore, we add edit span data to the existing training data for instruction tuning, which includes various tasks, and fine-tune LLMs.
Style transfer We used FST benchmark Gram- marly Yahoo Answers Corpus (GYAFC) (Rao and Tetreault, 2018) for formality style transfer. GYAFC is a plain corpus that contains pairs of in- formal and formal sentences conveying the same meaning. It covers domains such as Entertainment & Music (E&M) and Family & Relationship (F&R). We utilized the corpus BLEU in NLTK (Bird and Loper, 2004) as described Chawla and Yang (2020). For formality style transfer, we provide the instruction text "Rewrite the input text into formal text."
2.3 Conversion from Edit Spans to Output Text
To convert the edit spans output by LLMs into plaintext, we use rule-based approach. If LLMS generate "None", we use the source text as the final output text. Otherwise, we split the edit spans by commas and extract the edits. From each edit, we extract the starting position, ending position, and edited result. If LLMs generate edits in an incorrect format that do not include start or end positions edits where the start or end positions exceed the source text range, we ignore them. To ensure that the token indices do not shift, we apply the edits to the source text in descending order of starting positions. This conversion is implemented by simple rules with minimal computational cost.
Simplification
We used WikiSmall (Zhu et al., 2010; Zhang and Lapata, 2017) as the training data and ASSET (Alva-Manchego et al., 2020) and TurkCorpus (Xu et al., 2016) as the evaluation data. We used SART (Xu et al., 2016) to evaluate LLMs, which compares the generated text with the target text and calculates the average F1 score for addi- tion, keep, and deletion operations. For text simpli- fication, we provide the instruction text "Rewrite the input text into simpler text." "
3 Experiment Setting
3.1 Local Sequence Transduction Taskes
3.2 Open-ended Tasks
We conducted experiments on local sequence trans- duction tasks such as GEC,, paraphrasing, formality style transfer, and simplification.
The rules of edit spans differ from the rules in the raw text, which could potentially have a negative impact on the performance of tasks other than local
GEC We used NUCLE as the training data, CoNLL2013 (Ng et al., 2013) as the development data, and C (Ng et al., 2014)aa as the evalu-
'tt tpt: //ww. wwwegle..oo//ee quora- perrr--uussiioopppeee pairs "ht thub. com/ homm/iiinxingghag/dreess

---

sequence transduction. By combining open-ended instruction tuning data and edit spans instruction tuning data, we can train LLMs and investigate their impact on other tasks as well.
NVIDIA A100 GPUs. We used the code for lin- guistical alignment provided by Felice et (2016).
Baselines We compare the results of the pro- posed method with the results of LLMs fine-tuned for instruction tuning using the target text the ground truth instead of using edit spans. This com- parison examines whether edit spans can reduce computational costs during inference without com- promising performance.
We utilize the databricks-dolly: -oarricck--dlll-1155 15lly-15kkkk dataset by randomly dividing it into 13K for training, 1K for development, and IK for evaluation. databricks- d5lly-15k is publicly available dataset consist- ing of instructional records created by numerous Databricks employees. covers various behavioral categories described InstructGPT (Ouyang et al., 2022), such as brainstorming, classification, closed QA, generation, information extraction, open QA, and summarization. We sampled 3K instances for each of the tasks: GEC, paraphrasing, style trans- fer, and simplification, resulting in total of 12K instruction instances. We fine-tuned LLMs using combined dataset of all these instructions, totaling 25K instances.
4
Experiment
4.1
Performance on Local Sequence Transduction Tasks
To demonstrate the contribution of edit spans performance improvement, we first compare the baseline performance with fine-tuned data using plain text. Table I shows the results of performance comparison between the baseline and the proposed method in the GEC, paraphrasing, style transfer, and simplification tasks. Out of 32 cases, perfor- mance improvement was observed in 19 cases, and edit spans contributed to the performance enhance- ment. Furthermore, it can be observed that the LLaMA trained with edit spans achieves the high- est performance in most cases.
We used BERTScore (Zhang et al., 2019) as our evaluation metric. BERTScore is an evalua- tion method that measures the similarity between generated text and target text using contextual em- beddings from pre-trained models. We utilized RoBERTa (Liu et al., 2019) (roberta-large") as the BERTScore models.
3.3 Instruction Tuning Settings
4.2 Reducing Text Length
We used the following four LLMs for our ex- periments: MPT (mpt-7b)8 (Team, 2023), OPT (opt-6.7b) (Zhang et al., 2022), LLaMA (Ilama- 7 (Touvron et al., 2023), and BLOOM (bloom- 7b1)"! (Scao et al., 2022).
We examine how much the fine-tuning of LLMs with edit span data reduced the length of the output text. Figure 2 shows the ratio of output text length to target text length when fine-tuned with plain data and edit span data, respectively, on the devel- opment data for each task. The proposed method successfully compresses the output text across all tasks, independent of the model used; it achieves text compression in the range of 21 % in the most compressed cases and 41% even in the least com- pressed cases. In GEC, there cases where gram- matically correct text is provided as source text. In such cases, the model does not need to make any revisions and can simply output "None", resulting in significant compression in GEC.
We used the code for instruction tuning from Stanford Alpaca (Taori et al., 2023) code for in- struction tuning. We set the number of epochs to 3 and used a batch size of 32. The learning rate was set to 2e-5, with a warmup rate of 0.03, and we employed a cosine learning rate schedule. These hyperparameters were determined following Stan- ford Alpaca. We report the average results of three models trained with different seeds for instruction tuning. We used four nodes, each containing eight
'tt "https://hugggiggfaae https///huggnngffeee.oottttsssss co/dataett// data babbrrcccckttarrrisssdllllll//-/ Barbarikks-dattabrrddd--llll----
4.3 Performance on Open-ended Task
In open-ended tasks, the target texts written in plain text, while edit spans introduce significant dif- ferences in text formatting. This misalignment in text representation may potentially impact the per- formance open-ended tasks. Therefore, we aim
"I "httts://huggiingface hhapps/s//iiiiiiiieeeeeeeeeeeeer
Chttps:://githin
13 htttps: http.tbbcmmm/mrriispraeeerrrattt

---

GEC
Paraphrasing
Style transfer
Simplifictiion
MPT
68.0
37.9/66.5/47
78.9/81.2
46.3/41.1
OPT
65.7
35.2/63.2/45.
75.0/77.2
43.7/40.5
Plain
LLaMA
68.2
39.3/69.00 .994..4722
79.5/81.0
48.0/41.9
BLOOM
66.4
37.0666.446.1
78.2/79.9
45.0V41.0
MPT
68.5
28.2/66.7/47.1
78.2/81.3 1.3
46.6/41.3
OPT
66.2
34.1/61. 2/43,9
75.6/77.9
43.9/40.3
Edit spans
LLaMA
69.1
39.0/69.2/47.6
79.3/81.2
48.3/4.00
B1.OMM
65.8
37.2/66. 1/46.3
78.0/80.3
44.8/40.7
Table The performance of four LLMs fine-tuned with edit spans and plain data instructions on four local sequence transduction tasks. The bold values indicate the highest performance for each task. The underlined values indicate when edit spans exceed the baseline.
100%
100%
80%
60%
60%
20%
20%
GEC
Paraphrasing Style transfer Simplifiction
GEC
Paraphrasing
Style transfer
Simpiification
I Parallel
Position-ban based editing
Parallel
Position based editing
(a) MPT
(b) OPT
100%
100%
60%
60%
20%
20%
0%
GEC
Parephrosing Style tronsfer
Simpiification
GFC
Paraphrasing
Slyle transfer Simpification
Parallel
Position-baseee editing
Paraliel
Position- based editing
(c) LLaMA
(d) BLOOM
Figure 2: The ratio of output text length to target text length when MPT, OPT, LLaMA, and BLOOM are fine-tuned with plain data and edit span data, respectively.
to demonstrate that edit spans do not significantly degrade the performance of open-ended tasks.
transduction task data.
Table 2 shows the scores for each LLM when using RoBERTa BERTScore models on the 1K subset of the fatabricks-dolly-11 dataset, which was divided for evaluation. This indicates that the proposed method achieves efficient computational cost during inference without significantly sacrific- ing open-ended task performance.
Table demonstrates the performance difference on the 1K split of the databricks-doll dolky--1k dataset, evaluating LLMs trained on both open-ended task and local sequence transduction task data versus LLMss trained solely on local sequence transduction task data. The performance decreases when not ing open-ended task data for training, both in terms of plain text and edit spans. This is likely because open-ended task data consists of plain text, while edit spans include totally different text formats, leading to a larger disparity in task requirements.
To maintain performance in open-ended tasks, the proposed method combines data from both lo- cal sequence transduction tasks and open-ended tasks. To demonstrate the effectiveness of com- bining open-ended task data, we also investigate the open-ended task performance of instruction- tuned LLMss when solely trained on local sequence

---

BERTScore
tween the edit spans output by the LLMs and the edit spans extracted by the linguistic alignment in the development data for each task. The proposed method achieves more than 90% agreement in 13 out of 16 settings. This indicates that LLMss are able to learn the extraction rules for linguistical alignment through instruction tuning.
MPT
81.5
OPT
79.3
Plain
LLaMA
81.8
BLOOM
79.9
MPT
81.0
OPT
78.6
Edit span
LLMMA
81.3
BLOOM 79.5
4.5 Task-specific Fine-tuning
Table 2: Scores using BERTScore on the databricks- dolly-15k dataset, which was divided for evaluation.
In the previous experiments, LLMs were trained by combining data from the four local sequence transduction tasks and the open-ended task. To explore the maximum potential performance of the proposed method, we fine-tune LLMs with task- specific focus using edit span data. We fine-tune LLMs for each task using all available training data. In this case, we specialize LLMs for specific tasks without the need for instruction texts. Therefore, trained the LLMs by providing only the source texts as input.
BERTScore diff.
MPT
-5.2
OPT
-5.7
Plain
LLaMA
-4.4
BLOOM
-6.2
MPT
-8.1
OPT
-8.6
Edit span
LLaMA
-6.9
BLOOM
-7.6
Table 3: The performance difference between instruc- tion tuned LLMs using local sequence transduction task and open-ended task datasets, and instruction tuned LLMs using only local sequence transduction task datasets.
We trained LLaMA, which showed the high- est performance in the local sequence transduction tasks. We set the number epochs to and used a batch size of 32. The learning rate was set to le-5, with a warmup rate of 0.03, and we employed a cosine learning rate schedule. Following the explo- ration method described in Section 3.3, we deter- mined the hyperparameters for experiments.
4.4
The Accuracy of Edits Generated by the LLMs
Table 5 shows the results of performance com- parison with existing studies on GEC, paraphras- ing, style transfer, and simplification tasks. The proposed method outperforms existing studies by 1.8 points in GEC, 0.9, 1.2, and 2.3 points in para- phrasing. 1.9 and 1.3 points in style transfer, and 1.2 and 0.7 points in simplification tasks, respec- tively. Thus, the proposed method achieves the SoTA performance in all tasks. From these results, can be concluded that edit spans are an effective method, even in specifc fine-tuning scenarios.
Even if the edit span text is different, there are cases where the text is transformed by the rule, and the text matches. For example, in GEC, the model is given the input "This technology could also be seen as invasion of human privacy." and the model outputs "7 9 invading". In this case, even with the alternate edit span text "7 8 invading. 8 9", the conversion based on the rules would re- sult in the same output text. However, this would increase the sentence length by the index, creating room for improvement in terms of computational cost. Therefore, we investigate how well edit span of the model matches the results using linguistic alignment.
4.6 Example of LLMs Output Using Edit Spans
Table shows the output in CoNLL2013 for LLaMA using edit span and LLaMA outputting plain text. The normal model outputting plain text outputs 23 tokens, while the model using edit span outputs only 3 tokens. The output of the model using the edit span is much shorter sequence than the original model that outputs plain text. Fur- thermore, LLaMA, which outputs in plain text, is unable to correct a grammatical error. In a local se- quence transduction task, most tokens in the source
First, we convert the edit spans generated by the model to plain text using rules. From the converted plain text and the source text, we create edit spans using linguistic alignment and calculate the per- centage of agreement with the edit spans output by the model. Only when the start position end position and the edit token all match exactly is it considered correct answer.
Table shows the percentage of agreement be-

---

GEC
Paraphrasing
Style transfer
Simplification
MPT
96.6
95.0
89.2
94.7
OPT
93.3
91.9
88.8
92.7
LLaMA
99.0
96.2
92.6
95.4
BLOOM
94.2
92.5
89.4
93.5
Table 4: Agreement between edit spans generated by LLMs and edit spans extracted by linguistic alignment.
Paraphrasing
GEC
(Kumar al., 2020)
38.0/68, 1/45.7
(Kaneko al., 2020)
65.2
(Meng al., 2021)
26.8/65.0/3$
(Omelianchuk al., 2020)
66.5
(Li al.. 2022)
39.3/70. 8/48.3
(Qorih al., 2022)
69.5
Edit span
41.2/72.0/50.6
Edit span
71.3
(b) BLEU-4, ROUGE-1, and ROUGE-2 scores the Quora dataset.
(a) M scores the CoNLL2204 dataset.
Style transfer
Simplification
(Chawla and Yang, 2020)
76.2/79.9
(Martin al., 2020)
40.1/41.4
(Lai al,, 2021) (Liu al., 2022)
76.5/79.3
(Martin al., 2022)
44.2/42.6
77.8/
(Feng et al., 2023a)
47.9/41.8
Edit span
80.7/82.7
Edit span
49.1/43.5
(c) NLTK BLEU scores the F&M and F&R datasets.
(d) SARI scores ASSET and TurkCorpus datasets.
Table 5: Performance comparison with previous studies on GEC, paraphrasing, style transfer, and simplification tasks.
text and target text are common, and the model tends to learn just to copy the input tokens (Rastogi et al., 2016). Contrarily, our model that uses edit spans outputs only the edited parts. Thus simply copying the input is not issue for model.
terns is the utilization of learnable patterns. Mod- els that incorporate learnable patterns aim to ac- quire the access pattern through data-driven meth- ods. One crucial aspect of learning patterns is to establish a concept of token relevance and subse- quently assign tokens to buckets clusters (Vyas al., 2020; Wang al., 2021; Kitaev et al., 2020; Tay al., 2020; Roy et al., 2021).
5 Related Work
5.1 Efficient LLMs
Another approach to utilize a trainable side memory module capable of accessing multiple tokens simultancously (Sukhbaatar et al., 2019; Ainslie et al., 2020; Beltagy et al., 2020). A preva- lent example is the global neural memory, which access the entire sequence. The global tokens function type of model memory, learning gather information from the input sequence tokens. Another method to enhance efficiency is by utilizing low-rank approximations of the self- attention matrix to improve computational perfor- mance (Wang et al., 2020), and to view the atten- tion mechanism through kernelization (Choroman- ski al., 2020; Peng et al., 2021). Sparse models selectively activate fraction of the parameters, sulting in an improved parameter FLOPs ratio in general (Fedus et al., 2022).
Most of the methods for achieving efficient LLMs involve improving the memory complexity of self- attention mechanisms or enhancing the overall effi- ciency of the Transformer urchitecture (Tay et al., 2022; Loem et al., 2022). In the initial stages, the modifications made to self-attention focused reducing the computational complexity by intro- ducing sparsity in the attention matrix. This was accomplished by restricting the attention's 's scope to predetermined patterns, such as local windows and fixed stride block patterns (Liu et al., 2018; Qiu et al., 2020; Beltagy et al., 2020). A natural extension to the blockwise method is to connect these blocks via recurrence. Dai et al. (2019) in- troduced a mechanism of segment-level recurrence that establishes connections among multiple seg- ments and blocks.
As a way to reduce the length of the text, Cheng et al. (2023) proposed including multiple examples
An expansion upon fixed, predetermined pat-

---

Source text
Since we do not to bring cash to pay for the transportation enormous time has been saved everybody
Target lext
Since we do not need to bring cash to pay for transportation fee, enormous time has been saved everybody.
Target edit span
need
Plain
Since do to bring cash to pay for the transportation fee enormous time has been saved everybody.
System edit span
need
Table 6: Outputs in plain text and edit span formats respectively by LLaMA in the CoNLL
in one prompt and inferring in parallel.
tures from LLMs. Therefore, it is not easy to apply them to LLMs, unlike our method, which can train models by simply changing the output text.
These techniques, unlike our research, do not al- ter the writing style of the target text, and edit spans can be used in conjunction with these methods.
5.3 LLMs for Local Sequence Transduction Tasks
5.2 Edit-based Model
Since the question of necessarily using the seq2seq model for local sequence transduction tasks was raised (Rastogi et al., 2016; Schnober et al., 2016), various edit-based models have been proposed. Guu et al. (2018) proposed language model that initially selects prototype sentence from the train- ing dataset and subsequently modifies it to create a new sentence. Ribeiro al. (2018) introduced a method for representing general string transduc- tion problems as sequence labeling. Koide et al. (2018) proposed the model implemented to ana- lyze the evolution of biological sequences driven by substitution, insertion, and deletion edit opera- tions, achieving improved accuracy on protein sec- ondary structure prediction. Awasthi al. (2019) presented a parallel iterative edit model reducing decoding time for local sequence transduction tasks. Gu et al. (2019) developed the Levenshtein Trans- former, a non-autoregressivee model using edit oper- ations. (Mallinson et al., 2020) introduced FELIX, an adaptable text-editing approach for the genera- tion that aims to leverage the advantages of decod- ing with bi-directional contexts and self-supervised pretraining to the fullest extent. (Xu and Carpuat, 2021) presented an Edit-Based Transformer with Repositioning, which enhances sequence genera- tion flcxibility by scamlessly incorporating user- specified preferences in output lexical choice. Reid and Neubig (2022) proposed the modeling of edit- ing processes, encompassing the iterative gener- ation of sequences as whole. They establish conceptual framework to explain the probability of multi-step edits and outline neural models capa- ble of learning a generative model of sequences by leveraging these multi-step edits.
In GEC, the model based on GPT-3 achieves state- of-the-art in unsupervised settings (Loem et al., 2023). Fang et al. (2023) showed that ChatGPT corrects input text very fluently. Yamashita et al. (2020); Rothe al. (2021); Sun et al. (2022) pro- posed a method for multilingual GEC using multi- lingual LLMs. Feng et al. (2023b) investigated the performance of few-shot and zero-shot of GPT3 and ChatGPT in the simplification. Anschütz et al. (2023) used LLMs for German simplification and found them to be effective in languages with little parallel data. (Witteveen and Andrews, 2019) ver- ified the performance of GPT-2 (Radford al., 2019) in paraphrasing. Wahle et al. (2022) in- vestigated the utilization of T5 and GPT3 in gen- erating machine-generatee paraphrases for scien- tific articles sourced from arXiv, student theses, and Wikipedia. Reif et al. (2022) introduced a method based on GPT-3 that solely relies nat- ural language instruction and does not necessitate model fine-tuning or exemplars in the desired style. Malmi et al. (2020) proposed a method of using LLMs for style transfer where no parallel data is available. On the other hand, these studies did not target the efficiency of LLMs based on the edit.
Conclusion
In this study, we proposed to predict a set of edit spans, which represent the changed parts of the target text relative to the source tokens. We showed our method omits unedited tokens that occupy most of the target text and reduces the length of the tar- get text and the inference time for local sequence transduction tasks. Moreover, we reported that in- struction tuning with the proposed method achieves
However, these methods have different architec-

---

state-of-the-art performance in the four tasks.
be too big?. In Proceedings of the 2021 ACM confer- ence on fairness, accountability, and transparency. pages 610-623.
Limitations
In our preliminary experiments, even high- performance LLMs such as GPT-3 (Brown et al., 2020) and ChatGPT (OpenAI, 2023) could not gen- erate edit spans with zero-shot and few-shot. In particular, indexes could not be generated correctly. Therefore, it is future work apply the proposed method zero-shot and few-shot. Moreover, the use of edit span not necessarily effective for tasks, such as machine translation and dialogue, other than the local sequence transduction task, where many tokens in the source and target texts are not common.
Steven Bird and Edward Loper. 2004. NLTK: The natu- ral language toolkit. In Proceedings of the ACL In- leractive Poster and Demonstration Sessions, pages 214-217, Barcelona, Spain. Association for Compu- tational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter. Chris Hesse. Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020, Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
References
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Va- clav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. ETC: Encoding long and structured inputs in tranffrmeers Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 268-284, Online. Association for Computational Linguisties.
Kunal Chawla and Diyi Yang. 2020. Semi-supervised formality style transfer using language model dis- criminator and mutual information maximization. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 2340-2354, Online. Association for Computational Linguisties.
Fernando Alya-Manchego, Louis Martin, Antoine Bor- des, Carolina Scarton, Benoît Sagot, and Lucia Spe- cia. 2020. ASSET: A dataset for tuning and evalua- tion of sentence simplification models with multiple rewriting transformations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4668 4679, Online. Association for Computational Linguistics.
Zhoujun Cheng, Jungo Kasai, and Tao Yu. 2023. Batch prompting: Efficient inference with large language model apis. arXiv preprint arXhv:2301.08221
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song. Andreea Gane, Tamas Sarlos. Peter Hawkins, Jared Davis, David Belanger, Lucy Colwell, et al. 2020. Masked language modeling for proteins via linearly scalable long-context transform- ers. arXiv preprini arXiv:2006. 03555..
Miriam Anschütz, Joshua Oehms, Thomas Wimmer, Bartlomiej Jezierski, and Georg Groh. 2023. Lan- guage models for german text simplification: Over- coming parallel data scarcity through stylc-specific pre-training, arXiv preprint 2rxiv:2305.120000 12908.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint 2rXiv.2004.02311.
Abhijeet Awasthi, Sunita Sarawagi, Rasna Goyal, Sabyasachi Ghosh, and Vihari Piratla. 2019, Par- allel iterative edit models for local sequence trans- duction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-UJCNLP), pages 4260-4270, Hong Kong, China. Association for Com- putational Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghami, Siddhartha Brahma, et al. finetune language models. arXiv preprint arXiv:2210.114411
2022. Scaling
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv prepriat arX4v:2004.05
Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Pro- ceedings of the 2012 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies. pages Montréal, Canada. Association for Compu- tational Linguistics.
Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models

---

Zihang Dai. Zhilin Yang, Yiming Yang, Jaime Car- bonell, Quoe Le, and Ruslan Salakhutdinov. 2019. Transformer-XI: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988 Florence, Italy. Asso- ciation for Computational I Linguisties.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray. Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.0853
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. arXiv preprint vrxiv:2000.0455!
Tao Fang. Shu Yang. Kaixin Lan, Derek F Wong, Jin- peng Hu, Lidia S Chao, and Yue Zhang, 2023. Is chatgpt a highly fluent grammatical error correction system? a comprehensive evaluation. arXiv preprint arXiv:230..0144.
Satoshi Koide, Keisuke Kawano, and Takuro Kutsuna. 2018. Neural edit operations for biological se- quences. Advances in Neural Information Processing Systems, 31. 31.
William Fedus, Barret Zoph. and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parame- ter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23()):2232- 5270.
Mathias Kraus, Julia Anna Bingler, Markus Leip- pold, Tobias Schimanski, Chiara Colesanti Senni, Dominik Stammbach, Sacid Ashraf Vaghefi, and Nicolas Webersinke. 2023. Enhancing large lan- guage models with climate resources. arXiv preprint arXiv:2304.00116.
Mariano Felice, Christopher Bryant, and Ted Briscoe. 2016. Automatic extraction of learner emrors in ESL sentences using linguistically enhanced alignments. In Proceedings of COLING 2016, the 26th Inter- national Conference on Computational Linguistics: Technical Papers, pages 825-835, Osaka, Japan. The COLING 2016 Organizing Committee.
Ashutosh Kumar, Kabir Ahuja, Raghuram Vadapalli, and Partha Talukdar. 2020. Syntax-Guider Con- trolled Generation of Paraphrases. Transactions of the Association for Computational Linguistics, 8:330- 345.
Huiyuan Lai, Antonio Toral, and Malvina Nissim. 2021. Thank you BART! rewarding pre-trained models im- proves formality style transfer. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 2: Short Popers), pages 484-494, Online. Asso- ciation for Computational Linguistics.
Yutao Feng, Jipeng Qiang, Yun Li, Yunhao Yuan, and Yi Zhu, 2023a. Sentence simplification via large language models. arXiv preprint arXiv:2302. 11977.
Yutao Feng. Jipeng Qiang, Yun Li, Yunhao Yuan, and Yi Zhu, 2023b. Sentence simplification via large language models. ArXiv, abs/2302.11957.
Jiatao Gu, Changhan Wang, and Junbo Zhao. 2019. Lev- enshtein transformer. Advances in Neural Informa- tion Processing Systems, 32.
Zhigen Li, Yanmeng Wang, Rizhao Fan, Ye Wang, Jian- feng Li, and Shaojun Wang. 2022. Learning to adapt to low- low-resource paraphrase generation. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1014-1022, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Kelvin Guu, Tatsunori B. Hashimoto, Yonatan Oren, and Percy Liang. 2018 Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437-450.
Chin-Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- fion Brunches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2020. Encoder-decoder models can benefit from pre-trained masked language models in grammatical error correction. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4254, On- line. Association for Computational Linguistics.
Ao Liu, An Wang, and Nacaki Okazaki. 2022. Semi- supervised formality style transfer with consistency training. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4689-4701, Dublin, Ireland. Association for Computational Linguistics.
Masahiro Kaneko and Naoaki Okazaki. 2023. Con- trolled generation with prompt insertion for natural language explanations in grammatical error correc- tion. arXiv preprint arXiv:2309.1 11439.
Peter J Liu, Mohammad Salch, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10199.
Masahiro Kaneko, Sho Takase, Ayana Niwa, and Naoaki Okazaki. 2022. Interpretabiliity for language learners using example- example-based grammatical error correction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume Long Pupers), pages 7176-7187, Dublin, Ireland. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019.

---

Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint (eXii:1907.1169)
Computational Natural Language Learning: Shared Task, pages 1-12, Sofia, Bulgaria. Association for Computational Linguistics.
Mengsay Loem, Masahiro Kaneko, Sho Takase and Naoaki Okazaki. 2023. Exploring effectiveness of gpt-3 in grammatical error correction: A study on per- formance and controllability in prompt-basee meth- ods. arXiv preprinl arXiv:2305. 18156.
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR - grammatical error correction: Tag, not rewrite. In Proceedings of the Fifteenth Workshop on Innovative Use of NL. for Building Educational Applications, pages 163-170, Seattle, WA, USA + Online. Association for Computational Linguistics.
Mengsay Loem, Sho Takase, Masahiro Kaneko, and Naoaki Okazaki. 2022. Are neighbors enough?
multi-head neural n-gram can be alternative to self- attention. arXiv preprint arXiv:2207.1338
OpenAL. 2023. Introducing ChatGPT.
Jonathan Mallinson, Aliaksei Severyn, Eric Malmi, and Guillermo Garrido. 2020. FELIX: Flexible text edit- ing through tagging and insertion. In Findings of the Association for Computational Linguistics: EMNLP 2020. pages 1244-1255 Online. Association for Computational Linguistics.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 31 311-318.
Eric Malmi, Aliaksei Severyn, and Sascha Rothe. 2020. Unsupervised text style transfer with padded masked language models. In Proceedings of the 2020 Con- ference on Empiricul Methods in Natural Language Processing (EMNLPP), pages 8671-8680, Online. As- sociation for Computational Linguistics.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Lingpeng Kong. 2021. Random feature attention. arXiv preprint wrXv:2213..02143
Louis Martin, Éric de la Clergerie, Benoît Sagot, and Antoine Bordes. 2020. Controllable sentence sim- plification. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4689- 4698, Marseille, France. European Language Re- sources Association.
Jiezhong Qiu, Hao Ma, Omer Levy, Wen-tau Yih, Sinong Wang, and Jie Tang. 2020. Blockwise self- attention for long document understanding. In Find- ings of the Association for Computational Linguistics: EMNLP 2020, pages 2555-2565, Online. Association for Computational Linguistics.
Louis Martin, Angela Fan, Éric de la Clergerie, Antoine Bordes, and Benoît Sagot. 2022. MUSS: Multilin- gual unsupervised sentence simplification by mining paraphrases. In Proceedings of the Thirteemth Lan- guage Resources and Evaluation Conference, pages 1651-1664, Marseille, France. European Language Resources Association.
Muhammad Qorib, Seung-Hoon Na, and Hwee Tou Ng. 2022. Frustratingly easy system combination for grammatical error correction. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1964-19744 Seattle, United States. Association for Computational Linguistics.
Yuxian Meng. Xiang Ao, Qing He, Xiaofei Sun, Qinghong Han, Fei Wu, Chun Fan, and Jiwei Li. 2021. ConRPG: Paraphrase generation using con- texts as regularizer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2551-2562, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Hwee Tou Ng. Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christo- pher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natu- ral Language Learning: Shared Task, pages 1-14, Baltimore, Maryland. Association for Computational Linguistics.
Sudha Rao and Joel Tetreault. 2018 Dear sir or madam, may introduce the GYAFC dataset: Corpus, bench- marks and metrics for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long Pupers), pages 129-140, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Hwee Tou Ng. Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013. The CoNLL- 2013 shared task on grammatical error correction. In Proceedings of the Seventeenth Conference on
Pushpendre Rastogi, Ryan Cotterell, and Jason Eisner. 2016. Weighting finite-state transductions with neu- ral context. In Proceedings of the 2016 Conference

---

of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 623-633, San Diego, California. Association for Computational Linguisties.
multilingual grammatical error correction with pre- trained cross- -ingual language model. arXiv preprint arXiv: 1070.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An ensrruction-oolloiing 1lama model. https:/ // gi thub. www..c.ootttaaaeeeeeeeeeeeeeeaee tatsu- watlllaa/staaaa stanford_affaaaaa alpcc.
Machel Reid and Graham Neubig. 2022. Learning to model editing processes. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2022, pages 3822-38322, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yi Tay, Dara Bahri, Liu Yang. Donald Metzler, and Da-Cheng Juan. 2020. Sparse sinkhorn attention. In International Conference on Machine Learning. pages 9438-9447. PMLR.
Emily Reif. Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Kallison-Burchhe and Jason Wei. 2022. A recipe for arbitrary text style transfer with large language models. In Proceedings of the 60th Annual Meet- ing of the Association for Compulational Linguistics (Volume Short Papers), pages 837-848, Dublin, Ireland. Association for Computational Linguisties.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Met- zler. 2022. Efficient transformers: A survey. ACM Computing Surveys, 55(6):1-28.
MosaicML NLP Team. 2023, Introducing mpt-7b: A
Joana Ribeiro, Shashi Narayan, Shay B. Cohen, and Xavier Carreras. 2018. Local string transduction as sequence labeling. In Proceedings of the 27th Inter- national Conference on Computational Linguisties, pages 1360-1371, Santa Fe, New Mexico, USA. As- sociation for Computational Linguistics.
new standard for open- source, ly usable llms.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez. Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv preprint urXiv:2302.1397.
Sascha Rothe, Jonathan Mallinson, Fric Malmi, Sebas- tian Krause, and Aliaksei Severyn. 2021. A simple recipe for multilingual grammatical error correction. In Proceedings of the 59th Annual Meeting of the As- sociation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Pupers), pages 702-707, Online. Association for Computational Linguisties.
Apoorv Vyas, Angelos Katharopoulos, and François Fleuret. 2020. Fast transformers with clustered at- tention. Advances in Newral Information Processing Systems, 33:21665-21674.
Jan Philip Wahle, Terry Ruas, Frederic Kirstein, and Bela Gipp. 2022. How large language models are transforming machine-paraphrass plagiarism. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 952-963, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2021. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53- 68.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ili,, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon. Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint 2rXiv:2211.051000
Shuohang Wang, Luowei Zhou, Zhe Gan. Yen- Cen-Chun Chen, Yuwei Fang, Siqi Sun, Yu Cheng, and Jingjing Liu. 2021. former:-ormmr: Clustering- based sparse transformer for question answering. In Findings of the A Association for Computational Linguistics: ACI- IJCNLP 2021, pages 3958-39686 Online. Association for Computational Linguistics.
Carsten Schnober. Steffen Eger, Erik-Lån Do Dinh, and Tryna Gurevych. 2016. Still not there? comparing traditional srquence-to-sqqenne models encoder- decoder neural networks on monotone string trans- lation tasks. In Proceedings of COLING 2016. the 26th International Conference on Computational Lin- guistics: Technical Papers, pages 1703-1714, Osaka, Japan. The COLING 2016 Organizing Committee.
Sinong Wang. Belinda Z Li. Madian Khabsa, Han Fang. and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.00 247660668..
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith, Daniel Khashabi, and Han- nanch Hajishirzi. 2022. Self-instruct: Aligning lan- guage model with self generated instructions. arXiv preprint arXiv:22 10560.
Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lam- ple, Herve Jegou, and Armand Joulin. 2019. Aug- menting self-attention with persistent memory. arXiv preprint xrviv: 1907.01470.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoe V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.01652.
Xin Sun, Tao Ge, Shuming Ma, Jingjing Li, Furu Wei, and Houfeng Wang. 2022. A unitied strategy for

---

Jason Wei. Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprini urXiv:2206.07658
Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplificatin. In Proceedings of the 23rd International Conference on Computational Lin- guistics (Coling 2010), pages 1353-1361, Beijing, China. Coling 2010 Organizing Committee.
Sam Witteveen and Martin Andrews. 2019. Paraphras- ing with large language models. In Proceedings of the 3rd Workshop on Neural Generation and Truns- lation, pages 215 210- Hong Kong. Association for Computational Linguistics.
Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenx- lang Jiao, and Michael Lyu. 2023a. Chatgpt or grammarly? evaluating chatgpt on grammat- ical error correction benchmark. arXiv preprint arXiv:2303. 13648.
Minghao Wu, Abdul Waheed, Chiyu Zhang, Muham- mad Abdul-Mageed, and Alham Fikri Aji. 2023b. Lamini-Im: A diverse herd of distilled mod- els from large-scale instructions. arXiv preprint arXiv:2304. 14402.
Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Gallison-Burces 2016. Optimizing sta- tistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401 415.
Weijia Xu and Marine Carpuat. 2021. Editor: An edit- based transformer with repositioning for neural ma- chine translation with soft lexical constraints. Trans- actions of the Association for Computational Linguis- tics, 9:311-328.
Ikumi Yamashita, Satoru Katsumata, Masahiro Kaneko, Aizhan Imankulova, and Mamoru Komachi. 2020. Cross-lingual transfer learning for grammatical er- ror correction. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 4704 4715, Barcelona, Spain (Online). Inter- national Committee Computational Linguistics.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprini arXiv:2205.0000
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:190400965..
Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 584- 594, Copenhagen, Denmark, Association for Compu- tational Linguistics.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang. Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303. 18223.


Reviews:
[{"review_id": "YIcIZPRQOv", "reviewer": "Reviewer_xjk9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The paper distinguishes text generation tasks into local sequence transduction tasks and open ended tasks, but does not explain the differences between these two series of tasks or cite other work to distinguish them. This may confuse readers who are not familiar with this field.\n\nLocal sequence transduction and open-ended tasks: Local sequence transduction tasks have massive overlapping tokens between source and target texts. On the other hand, open-ended tasks don't have such a restriction. We will add this explanation referring to some papers (https://aclanthology.org/D19-1435, https://aclanthology.org/2020.findings-emnlp.136/, https://arxiv.org/abs/2203.02155) in final-version.\n\n> The motivation mentioned in the abstract is that \"the models (referring to LLMs according to the context) have a tendency to simply copy the input text as is...\" This statement was not mentioned or verified in the subsequent sections. However, to my understanding, LLMs usually perform better than other models on so-called local sequence translation tasks and do not show this tendency.\n\nThank you for pointing that out. We compare the original LLM and the LLM using the edit spans with the rate of overlapped tokens between source and target texts in order to show LLMs have a tendency to copy tokens in local sequence transduction tasks.\n\n> I think reducing the sequence length is a by-product of changing the form of the target, with the biggest benefit being reduced computational cost in generation. So I think the matter of \"reducing the sequence length\" may not be that important. The comparison of computing costs should be given in the article, such as how much the memory and time are reduced during inference.\n\nThere is a correlation between decreased output length and efficiency. However, we plan to compare the time and memory of the original LLM and LLM using edit spans to show obviously our method is efficient.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It is a natural idea to use LLMs to generate edit spans for tasks where the source and target sentences are close， which is a simple but effective method.\n2. In order to validate this method, the authors conduct solid experiments on multiple tasks.\n3. The experimental results demonstrate the superiority of the method.", "reasons_to_reject": "1. The paper distinguishes text generation tasks into local sequence transduction tasks and open ended tasks, but does not explain the differences between these two series of tasks or cite other work to distinguish them. This may confuse readers who are not familiar with this field.\n\n2. The motivation mentioned in the abstract is that \"the models (referring to LLMs according to the context) have a tendency to simply copy the input text as is...\" This statement was not mentioned or verified in the subsequent sections. However, to my understanding, LLMs usually perform better than other models on so-called local sequence translation tasks and do not show this tendency.\n\n3. I think reducing the sequence length is a by-product of changing the form of the target, with the biggest benefit being reduced computational cost in generation. So I think the matter of \"reducing the sequence length\" may not be that important. The comparison of computing costs should be given in the article, such as how much the memory and time are reduced during inference.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper propose a novel method for applying the large language models to downstream tasks.\nThe method uses instruction tuning to make LLMs generate edit spans of the source sentence.\nExperiments demonstrate that this method performs well on tasks where the source and target sentences are close, outperforms previous methods and achieving significant efficiency improvements.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qlptO7g6BU", "reviewer": "Reviewer_R7G8", "summary": "", "questions": "A. This tuple consists only of position and inserts words; is there any reason why you did not include labels for insert, replace, and delete?\nB. Section 4.2 states that the proposed method can output \"none\", what is the percentage of \"none\" in the actual test set?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> I could not find any merit in shortening the target sequence length.\n\nWe wrote, ``We can reduce the length of the target text and the inference time for local sequence transduction tasks`` to explain the motivation to reduce the output length in L74-L75 in the introduction section.\n\n> (Section 4.1) No performance improvement was observed in 13 of the 32 cases, but the reasons for this were not discussed.\n\nThe performance, especially in style transfer, has decreased. Fig. 2 shows that the style transfer dataset has the longest output sequence when using edit spans. This means that a lot of tokens are rewritten. It is difficult for LLMs that use edit spans to fully benefit from too many edit spans.\n\n> A. This tuple consists only of position and inserts words; is there any reason why you did not include labels for insert, replace, and delete?\n\nThe aim of this study is to decrease the output length and make inference time more efficient. The edit positions can represent which edit operation (insert, replace, or delete) has been performed. Therefore, we did not use explicit edit operation tokens to prevent an increase in the output length.\n\n> B. Section 4.2 states that the proposed method can output \"none\", what is the percentage of \"none\" in the actual test set?\n\nOnly the GEC dataset has ``none``, due to including some texts that do not need to be corrected. For example, 10% of the CoNLL2014 dataset has ``none''.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors propose a simpler method than conventional methods that output the entire target sequence.\nSuch an output-oriented method could be a new baseline for the NLP community, where LLM is becoming mainstream.\nIn my opinion, such output-oriented methods have various possibilities and should be actively adopted.", "reasons_to_reject": "- I could not find any merit in shortening the target sequence length.\n- (Section 4.1) No performance improvement was observed in 13 of the 32 cases, but the reasons for this were not discussed.", "questions_to_authors": "A. This tuple consists only of position and inserts words; is there any reason why you did not include labels for insert, replace, and delete?\nB. Section 4.2 states that the proposed method can output \"none\", what is the percentage of \"none\" in the actual test set?", "paper_topic_and_main_contributions": "The authors noted that most source sentences are not transformed in local sequence transformation tasks.\nThey proposed a method that outputs the modified sentences as tuples instead of outputting them.\nThe proposed method employs instructional tuning instead of conventional fine-tuning, and the performance was improved in several local sequence transformation tasks.\nBy fine-tuning each task, SOTA was achieved for GEC, paraphrasing, formal style transformation, and simplification.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Mo6r4uu95P", "reviewer": "Reviewer_JGA8", "summary": "", "questions": "* L186: \"To ensure that the token indices do not shift, we apply the edits to the source text in descending order of starting positions\". This sentence is confusing - it would be good to clarify this further. \n* L254: It would be good to present some examples from the open-ended tasks.\n* L300: The phrase: \"fine-tuned data\" is confusing. Did you mean: \"comparing the performance of LLM fine-tuned on plain text vs edit spans\"?\n* Table 1: The columns should labeled with the metrics.\n* L337: \"This indicates that the proposed method achieved efficient computational cost during inference without significantly sacrificing open-ended task performance\" This sentence is confusing. It would be good to clarify that a) for open-ended tasks, the target is the plain text representation i.e. not the edit spans, b) the savings in computational cost during inference is only applicable to the localized sequence transduction tasks such as grammatical error correction.\n* L349: \"Table 3 demonstrates the performance difference on … evaluating LLMs trained on both open-ended task and local sequence transduction task data versus LLMs trained solely on local sequence transduction task data\". Given the signs of metrics in Table 3, it looks like the above sentence should be \"performance difference between LLMs trained solely on local sequence transduction task data (A) and LLMs trained on both open-ended task and local sequence transduction task data (B)\". i.e. A-B. The same error is also present in the Table 3 caption. \n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The idea of using edit representations in sequence-to-sequence models is not novel. See for example: Stahlberg and Kumar, Seq2Edits: Sequence Transduction Using Span-level Edit Operations, EMNLP 2020 https://aclanthology.org/2020.emnlp-main.418/\n\nThere are two differences between our study and the previous study. First, we demonstrate that the models can generate correct output using only the edit span prompt without changing the architecture. On the other hand, the previous study requires architectural changes, and it is not readily applicable to LLMs. Second, the previous method generates the edit spans for both rewrite and non-rewrite parts. In contrast, our method generates edit spans only for the rewrite part. Consequently, our method has a shorter output sequence length than the previous method. We will reference this study in the camera-ready version and clarify the differences between our study and the previous one.\n\n> The paper ignores some prior work in editing models e.g. Mallinson et al. EdiT5: Semi-Autoregressive Text-Editing with T5 Warm-Start https://arxiv.org/abs/2205.12209\n\nThank you for sharing the paper. We did not refer to some papers due to the page limit. Therefore, we will add related studies in the final version.\n\n> L300: The phrase: \"fine-tuned data\" is confusing. Did you mean: \"comparing the performance of LLM fine-tuned on plain text vs edit spans\"?\n\nYes, you are right. We rewrite to ``comparing the performance of LLM fine-tuned on plain text vs. edit spans``.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Given the popularity of LLMs in the NLP community, the proposed approach is likely to encourage researchers think about alternative output representations which are concise and better suited to the task at hand.\n* The paper demonstrates that LLMs can be fine-tuned using a mixture of plain text and edit representations.\n", "reasons_to_reject": "* The idea of using edit representations is not entirely novel. \n* Some parts of the paper are confusing. See 'questions to the authors'.\n", "questions_to_authors": "* L186: \"To ensure that the token indices do not shift, we apply the edits to the source text in descending order of starting positions\". This sentence is confusing - it would be good to clarify this further. \n* L254: It would be good to present some examples from the open-ended tasks.\n* L300: The phrase: \"fine-tuned data\" is confusing. Did you mean: \"comparing the performance of LLM fine-tuned on plain text vs edit spans\"?\n* Table 1: The columns should labeled with the metrics.\n* L337: \"This indicates that the proposed method achieved efficient computational cost during inference without significantly sacrificing open-ended task performance\" This sentence is confusing. It would be good to clarify that a) for open-ended tasks, the target is the plain text representation i.e. not the edit spans, b) the savings in computational cost during inference is only applicable to the localized sequence transduction tasks such as grammatical error correction.\n* L349: \"Table 3 demonstrates the performance difference on … evaluating LLMs trained on both open-ended task and local sequence transduction task data versus LLMs trained solely on local sequence transduction task data\". Given the signs of metrics in Table 3, it looks like the above sentence should be \"performance difference between LLMs trained solely on local sequence transduction task data (A) and LLMs trained on both open-ended task and local sequence transduction task data (B)\". i.e. A-B. The same error is also present in the Table 3 caption. \n", "paper_topic_and_main_contributions": "Summary:\nThis paper proposes the use of edit representations when instruction-tuning/fine-tuning LLMs for localized sequence transduction tasks (e.g. grammatical error correction) where the input and output have substantial overlap. The paper shows that LLMs fine-tuned on edit-span representations for localized transduction tasks and plain text representations for open-ended tasks yield improvements on localized transduction tasks without a large loss in performance on the open ended tasks which use plain text representations. Furthermore, LLMs fine-tuned on task specific data with edit representation, yield large gains in performance relative to existing baselines.\n\nStrengths:\n* Proposes a simple approach to fine-tune an LLM on a combination of plain text and edit representations which improves performance on localized sequence transduction tasks without causing a large degradation in open ended tasks with plain text representations\n* Demonstrates that using edit representations can substantially reduce the output length of the LLM.\n* Shows that LLMs when fine-tuned on task specific data for localized transduction tasks with edit representations can yield good performance relative to existing baselines in the literature.\n\nWeaknesses:\n* The idea of using edit representations in sequence-to-sequence models is not novel. See for example: Stahlberg and Kumar,  Seq2Edits: Sequence Transduction Using Span-level Edit Operations, EMNLP 2020\nhttps://aclanthology.org/2020.emnlp-main.418/\n* The paper ignores some prior work in editing models e.g. Mallinson et al. EdiT5: Semi-Autoregressive Text-Editing with T5 Warm-Start https://arxiv.org/abs/2205.12209\n* There are some confusing statements in the paper - see 'questions to the author' below.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "* Stahlberg and Kumar,  Seq2Edits: Sequence Transduction Using Span-level Edit Operations, EMNLP 2020\nhttps://aclanthology.org/2020.emnlp-main.418/\n* Mallinson et al. EdiT5: Semi-Autoregressive Text-Editing with T5 Warm-Start https://arxiv.org/abs/2205.12209"}}]

Author rebuttal:

Meta-review: The paper proposes a method based on instruction tuning of LLMs for localized sequence transduction tasks where source and target sequences have substantial overlap. Experiments demonstrate that model performance was improved in several tasks, such as grammatical error correction, paraphrasing, formal style transformation, and simplification.

Reviewers agree that the method is simple and effective, and that it could possibly be adopted as a new baseline for the NLP community. It was also suggested that such an approach could encourage researchers to think about alternative concise output representations which are better suited to the task at hand.

Some aspects to improve include a comparison of computing costs, such as how much the memory and time are reduced during inference (which the authors' rebuttal agree with and commit to including). In addition, a couple missing references were pointed out.
Decision: Accept-Main