G-SPEED: General SParse Efficient Editing MoDel
Authors: Haoke Zhang
Yue Wang
Juntao Li
Xiabing Zhou
Min Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: Text Editing

Abstract: Large Language Models (LLMs) have demonstrated incredible capabilities in understanding, generating, and manipulating languages.
Through human-model interactions, LLMs can automatically understand human-issued instructions and output the expected contents, which can significantly increase working efficiency.
In various types of real-world demands, editing-oriented tasks account for a considerable proportion, which involves an interactive process that entails the continuous refinement of existing texts to meet specific criteria.
Due to the need for multi-round human-model interaction and the generation of complicated editing tasks,  there is an emergent need for efficient general editing models.
In this paper, we propose \underline{\textbf{G}}eneral \underline{\textbf{SP}}arse \underline{\textbf{E}}fficient \underline{\textbf{E}}diting Mo\underline{\textbf{D}}el~(\textbf{G-SPEED}), which can fulfill diverse editing requirements through a single model while maintaining low computational costs.
Specifically, we first propose a novel unsupervised text editing data clustering algorithm to deal with the data scarcity problem.
Subsequently, we introduce a sparse editing model architecture to mitigate the inherently limited learning capabilities of small language models.
The experimental outcomes indicate that G-SPEED, with its 508M parameters, can surpass LLMs equipped with 175B parameters.
Our code and model checkpoints are available at \url{https://github.com/Banner-Z/G-SPEED}.

G-SPEED: General SParse Efficient Editing MoDel
Haoke Zhang; Yue Wang; Juntao Li, Xiabing Zhou, Min Zhang
Soochow University
(hkzhangnlp, ywangnlp]@stu. suda edu. cn (1jt, zhouxiabing, (achhnng)ssuaa edu. cn
Abstract
Therefore, due the powerful ability of large mod- els as assistants, users can conveniently accomplish a wide range of open-domain tasks by collaborat- ing with LLMs. One particularly important applied scenario of the LLM assistants is editing.
Large Language Models (LLMs) have demon- strated incredible capabilities understand- ing, generating, and manipulating languages. Through human-model interactions, LLMs can automatically understand numan-issuee instruc- tions and output the expected contents, which can significantly increase working efficiency. In various types of real-world demands, editing- oriented tasks account for considerable pro- portion, which involves an interactive process that entails the continuous refinement of isting texts to meet specific criteria. Due to the need multi-round human-modee inter- action and the generation of complicated edit- ing tasks, there an emergent need for effi- cient general editing models. In this paper, we propose General SParse Efficient Editing MoDel (G-SPEED), which can fulfill diverse editing requirements through single model while maintaining low computational costs. Specifically, we first propose a novel unsuper- vised text editing data clustering algorithm to deal with the data scarcity problem. Subse- quently, we introduce a sparse editing model architecture to mitigate the inherently limited learning capabilities of small language mod- els. The experimental outcomes indicate that G- SPEED, with its 508M parameters, can surpass LLMs equipped with 175B parameters. Our code and model checkpoints are available https:///ithhu wow/Banner-r///e//eooooo Z/GSSSEEE
Editing is a crucial process in writing, requiring a diverse range of skills to refine texts from multiple perspectives. Consequently, there is long-tern goal of using machines to assist in automating the editing process. However, the inherent difficulty of editing, coupled with the limited availability of annotated editing data, hinders the development of general editing models capable of meeting a wide range editing requirements. Therefore, before the era of LLMs, previous works mainly focused on developing specialized editing models tailored to specific tasks, such grammatical error correc- tion (GEC) (Bryant al., 2019), style transfer (Rao and Tetreault, 2018), and sentence fusion (Geva et al., 2019). Due to their specific capabilities, these specialized editing models have limited practical applications. Besides, although recent work has explored the potential of LLMs as general editing models (Faltings et al., 2021: Schick et al., 2023; Madaan et al., 2023), the iterative nature of editing often requires multiple interactions with models to edit a single piece of text. Therefore, relying solely on LLMs meet diverse editing needs becomes impractical and cost-prohibitive.
In this work, to develop a lightweight gen- eral edit model that caters to various editing re- quirements, we propose General SParse Efficient Editing MoDel (G-SPEED). Specifically, we intro- duce novel unsupervised editing data clustering method to eliminate noise and annotate the intent, considering the limited quality of current edit intent annotations and the presence of noisy data in the vision history of Wikipedia passages. Besides, we propose a novel sparse editing model structure enhance the learning capubilities of small models. Experimental results demonstrate that G-SPEED
1 Introduction
Recently, the Natural Language Processing (NLP) community has witnessed the rapid development of Large Language Models (LLMs). With the help of instruction tuning, LLMs can effectively utilize the knowledge acquired during pre-trale traininii and have the ability to follow user instructions (Wei et al., 2021; Sanh et al., 2021; Mishra et al., 2022; Wang et al., 2022; Ouyang et al., 2022; OpenAl, 2023).
Equal Contribution
Corresponding Author

---

achieves state-of-the-an performance on the EditE- val benchmark and exhibits strong generalization ability to unseen tasks with limited data resources. In conclusion, our contributions are as follows:
2019) is a general approach that predicts edit oper- ations by sequence labeling and then generates in- serted words with a fixed vocabulary. Omelianchuk et al. (2020) proposes more complete tags (such as pluralization and capitalization) to make generation casier and improve the GEC task. Mallinson et al. (2020) and Mallinson et al. (2022) perform arbi- trary content insertion via non von-auutorerressive and semi-autoregressivee methods, respectively. How- ever, the current editing model is stuck on solving single task. Editing models do not perform well in multi-task problems (Du et al., 2022b).
In this work, we first propose G-SPEED, a lightweight framework with a novel sparse editing model structure, which can satisfy var- ious editing needs:
To deal with the limited editing data source, we propose a novel unsupervised data cluster- ing method, which will be released to promote the development of editing models;
Sparse Language Model Sparse language mod- els achieve promising results with larger model sizes while maintaining almost the same computa- tional costs (Jaszezur al., 2021; Du al., 2022a; Fedus et al., 2022b.a). There are various routing algorithms to determine where to send examples, such as hash routing (Roller et al., 2021), base layer (Lewis et al., 2021), and Multi-gate MoE (Ma et al., 2018). Zuo et al. (2022), Gupta et al. (2022) and Lee- Lee-Thorp and Ainslic (2022) design sparse feed-forward layers on BERT (Devlin al., 2019) and make boost on GLUE (Wang et al., 2018). In this paper, we explore the use of sparse layers on Editing models for General Text Editing tasks with a BERT backbone.
Experimental results show that G-SPPED can achieve slatc-of-the-an the- results EditEval, surpassing the best LLMs by 2.8 points erage, and have strong generalization ability to unseen tasks in data-limited scenarios.
2 Related work
General Text Editing Existing work on general text editing broadly be divided into two cate- gories: (1) instruction-based text editing (Faltings al., 2021; Schick et al., 2023; Madaan et al., 2023):(2) multi-intent editing (Du et al., 2022b; Kim et al., 2022). Instruction- polishes up existing texts following user instruc- tions, which is based on LLMs (Chung et al., 2022; Brown et al., 2020; Ouyang et al., 2022) and needs long time to generate and large-scale datasets for training. Multi-intent editing uses intents to perform diverse editing actions. Specifically, Kim al. (2022) improves text editing with intent span detection. However, edit intents are defined by hu- mans, which have high annotation costs and may not cover all editing intents (Yang et al., 2017; An- thonio et al., 2020; Du et al., 2022b). In this work, we propose G-SPEED, which can handle general text editing without the dependence on LLMs and human-annotate editing intents.
text editing
3 Task Formulation
During text editing, the documents D undergo modifications based on user-selected intents like fluency, clarity, simplification, and neutralization. Each modification corresponds to a pair of docu- ments (D:-1, Dr) and editing intent Le where t represents the number of modifications. Text edit- ing models are tailored modify documents based on intent:
D f(D-(DD,
(1)
where f denotes text editing models.
Unsupervised Editing Data Clustering
Editing Model Editing models are efficient alter- natives to Seq2Seq models (Sutskever et al., 2014) 4) when the source and target texts in the task have a large amount of overlap, e.g., grammatical er- ror correction (GEC) (Bryant et al., 2019), style transfer (Rao and Tetreault, 2018), and sentence fusion (Geva et al., 2019). Editing models learn to predict edit operations and directly leave the correct text unchanged, while Seq2Seq models generate target text from scratch. LaserTagger (Malmi et al.,
To facilitate text editing tasks, a dataset comprising editing intent and text pairs, both before and after editing, is required for pre-training purposes. IT- ERATER (Du et al., 2022b) collects text revisions from Wikipedia, ArXiv, and Wikinews. It catego- rizes text intents into five categories: fluency, clar- ity, coherence, style, and meaning-changed The intent then predicted using a RoBERTa-baas model (Liu et al., 2019) that takes both the original

---

Cluster
Source
Target
User Comment
Fluency
At the end of the 1986 season, he unnounced that would retine after completing the 1987 NFL season.
At the end of the 1986 season, he un- nounced that he would retire after complet- ing the 1987 NFL season.
Minor grammatical fix
Readability
The journey to Loadon takes approx. 53 minutes,
The journey to London takes about one
'about one hour" more readable than 'ap prox. min."
Simplification
Europeun sales were slowed when 150,000 faulty CD copies of the album were recalled by record company EMI. Dises sent to Germany, France, Spain, Portugal, the Netherlands and Belgium had heen affected by a mastering error, CDs started with second live recording of a different band Pearl Jam according
150,000 Furopeun CD capies of the alhum were recalled by EMI after mastering ror was discovered.
unnecessary detail
site.
Neutralization
CORBA aims bring to the table many benefits single technology hrings in package.
CORBA supports feeatures which claims that other single technology brings in one package.
more neutrality. Let's unwrap this rhetorical language
Table 1: Representative examples in our pre-training dataset.
and revised texts as input. However, automatic annotation performs poorly on small categories due to the unbalanced distribution of categories, and certain categories, such as simplification, are not taken into account. Formulating perfect intent categories and annotating large-scale data for pre- training pose significant challenges.
embeddings represented 3 22 .... k-means clustering aims to partition these embeddings into sets (Si, .k.Skk so as to minimize the intra- cluster distance:
min Σ
dist(r, /4);
(2)
11
Similar to existing work (Zhang et al., 2019;
where dist represents the Euclidean distance, and p4 denotes the mean of embeddings within Si. In our study, we utilize Sentence- BERT (Reimers and Gurevych, 2019) generate comment embeddings, which are then decomposed using Singular Value Decomposi- tion (SVD). Sentence-BERT provides a more effective representation of text semantics compared to algorithms based on word frequency.
Faltings et al., 2021), we first collect data from the revision histories in March 1st 2023 dump of English Wikipedia.' Each revision consists of an original text, revised text, and user comment that is associated with the editing intent. We ex- tract revisions from the XML format dump and exclude revisions that do not have user comments. To ensure data clustering quality, we additionally perform data cleaning based on various factors, such as the BLUE value of the source and target, as well as the length of comments and text. We extract the source and target sentences using Punkt Sen- tence Tokenizer2 and difflib?, and then we eliminate Wikipedia markup using w
To discern the purpose of each cluster, also organize the designed prompts into clusters (such as fixing grammar errors and improving text cohe- sion). Based on the prompts they contained, we then selected four clusters, namely fluency, read- ability, simplification, and neutralization, as illus- trated in step I of Figure 1. Table 1 presents the data instances. Clusters that focus information updates, citation modifications, punctuation mod- ifications, and other similar tasks are disregarded. For more details, please refer Appendix A.
To classify user comments into distinct intent categories, we utilize k-means clustering initial- ized with k-means++ algorithm (Arthur and Vas- silvitskii, 2007).5 With collection of n comment
htttps///umms.. wikimedia. org.///ikk.g///kkk
tht wttp///wwwwwwt....mmm.k/...... org/api/n//i api/nllk. tokenize. punkt.
ht ttml##odule--ntk. tow.eeennnnnttttttteeeeee punkt
G-SPEED
org/3 htmd
5.1 Editing Model
"https://githur http:///ittuuu..m//ttttttt htw./attard//ikkeetrreee
We decompose editing into two steps: (1) tagging. label each word with an editing operation by se- quence labeling; (2) generation, insert new words into the original sentence. As shown in Figure I, we share the encoder parameters of tagging and
'We also employ the DBSCAN algorithm (Ester et al., 1996). which segments categories by detecting changes in data point density and eliminates the need to specify the number of clusters. However, due to the high and uneven density of data points, DBSCAN struggles to segment meaningful clusters offectively.

---

Step I: Data Clustering
Step II: Model Training
Qutput:
You
improved (PAD)
Ganeration Classitee
C2
Sparse Bart Layer
Nx
Sparse Bert Layer
Add Normelize
Fiurnc
Rawdib.
Simpli
Neutral
Embedding
C3
Cluster =
[D] made (I) MASK MASK
sharing
MoE Gale
Add Normalize
noging Classifee
Salf-Attention
comment: minor tweak to improve clarity sentence. cluster:
Sparse Bert Layer
KEEP;
R- REPLACE
DELEEE
(O) (DELETE)
You made it better than before
Embedding
inpul:
target: You improved it
You
made
belter
than
before
Figure
The illustration of unsupervised editing data elustering and G-SPEED training. Initially, utilize unsupervised data clustering by leveraging user comments. Subsequently, sparse BERT layers are utilized for training G-SPEED with multi-task data, wherein the feed-forward layer associated with the respective cluster is activated. Text editing is divided into two distinct steps: initially predicting the operation tags and subsequently inserting the words. The encoders for the two steps are shared. "Readab." means readability, "Simpli." denotes simplification, and "Neutral." " means neutralization.
generation so that the two modules are trained to- gether:
total of 14 tags which encompass case conver- sion, singular and plural conversion, verb tense conversion, as well as split operations like hyphen splitting or merging. For automatic operation an- notation, we utilize dynamic programming and set the editing distance as the cost function. Detailed information about the categories and a comparison of various designs are provided in Appendix B.
L - Liagging + XL generation
(3)
where X is hyper- parameter.
Tagging Editing operations are marked at the word level. A linear classification layer is used to predict tags following transformer encoder. Training is performed using cross- loss:
The tagging phase primarily handles transformations that do not involve the generation of new words. During the generation phase, our task is to predict new words at the positions of REPLACE, APPEND, and TRANSFORM-VERRY We employ an efficient non-autoregressive mask language model (MLM) similar to Mallinson et al. (2020). As shown in Figure 1, we insert n [MASK) tokens at each required position and subsequently perform mask prediction on the new text. Any [MASK] tokens beyond the length of the desired tokens will be predicted as [PAD]. The MLM is trained with cross-entropy loss:
Generation
|od Σ
Ltagging
log (hi)),
(4)
where y is golden tags, is the index of each word, ht is the output of encoder, xf is the input text, |x*] represents the total number of words in the text, fet is classification layer to predict tags.
The most basic type of editing operation rep- resented by the Levenshtein transition types, which encompass KEEP, DELETE, REPLACE, and AP. PEND. Omelianchuk al. (2020) expands the sum of types 4971, 29 of which represent fundamen- tal text transformations, and the rest are used to insert new words. To maximize the number of tags for reducing generation difficulties while ensuring sufficient training data for each tag, we develop
Iv*1
Lgeneration = > log ppee)))) (5)
where y is the golden results of [MASK], is the index of [MASK), ly^ denotes the total number of IIn our experiments, n = 4.

---

[MASK], hef is the hidden states, and Spred is linear layers to predict tokens in vocabulary.
6 Experiments
6.1 Datasets
To retain the maximum amount of information from the original sentence, we retain the deleted words and the verbs with the wrong tense. As demonstrated in Figure 1, we enclose the deleted words within [DELETE] and [/DELETE] tags, and the wrong verbs within (TRANSFORM_ VERBB] and [/TRANFFORM_VERT _VRB| tags.
We conduct evaluation using EditE- val (Dwivedi-Yu Yu al., 2022), which is an instruction-baset text improvement benchmark, The study includes six datasets and six tasks; their details are provided below.
Fluency The primary objective of the fluency task is rectify grammatical and spelling errors. The evaluation of this task utilizes ITERATER (Du et al., 2022b) and JFLEG (Napoles et al., 2017). ITERATER an edit-intention annotated corpus of iteratively revised text. We utilize the subset that focuses on fluency. In comparison to ITERATER, JFLEG not only corrects grammatical errors but also makes the original text more native sounding.
5.2 Sparse Bert Layer
Then, we introduce efficient and compact sparse Bert layer for multi-intenn editing model. Fig- ure illustrates the use of four fully connected fera-forward networks (FFN), referred to as "ex- perts". The experts correspond to four editing in- tents present in our training data. The experts are activated when training data that corresponds to their respective intents. The data point shown in the figure is clustered as "C2" (readability) and, there- fore, utilizes the red module. Each layer comprises 8 experts, with the tagging and generation modules being divided due their distinct effects. By em- ploying this approuch, we can efficiently address multi-intent editing tasks with a single encoder:
Clarity and Coherence Both tasks use the corre- sponding subsets in the test dataset of ITERATER. The clarity task aims to enhance the formality, con- ciseness, readability, and comprehensibility of the text. The coherence task aims to enhance the co- hesiveness, logical connectivity, and overall coher- ence of the text.
Paraphrasing The purpose of the Paraphrasing task is to rephrase sentences without specific im- provement goal. EditEval opts utilize text pairs from SemEval-2018 (Cer et al., 2017) that have high similarity scores.
h' fo r², 2)), 10,n), E [0,
where x' represents the input for either tagging or generation, r² represents the index of the intent, is the total number of intents, 2 = 0 indicates the tagging mode and 2 = 1 indicates the generation mode, fo represents the sparse Bert model, and h' is the hidden states in Equation 4 and
Simplification TurkCorpus (Xu et al., 2016) and ASSET (Alva-Manchego et al., 2020) are uesed for evaluation of this task. ASSET and TURK contain identical test examples, but their references are simplified in different ways.
For the balance between each task and the two modules, we sequentially train the tagging module and the generation module of each task units of batch, so that the number of training steps for each task and each module is the same.
Neutralization
The process of neutralization in- volves eliminating subjective bias in text. For example, in the text of "Jimi hendrix (musician), great musician and vocalist", the word "great" lacks neutrality. The WNC dataset (Pryzant al., 2020) serves as a representative dataset for this task.
5.3 Additional Fine-tune
6.2 Metries
In practice, text text editing tasks may not align with our expert configuration. We employ additional fine- tuning for specific tasks. To enhance the model's generalization, we copy the experts in the pre- trained model and train targeted tasks accordingly. Specifically, we freeze all parameters except the experts, enabling the model to adapt flexibly to any editing task with minimal training cost.
SARI (Xu et al., 2016) measures the goodness of words that are added, deleted, and kept. It cal- culates the average n-gram F1 score for addition, deletion, and retention operations. GLUE (Napoles et al., 2015) an additional n-gram- based metric better suited for evaluating single sentences than BLEU. is applied to JFLEG and ITERATER.

---

Cluster (size)
Fine-tune
Size
Test
Abbrer.
base-v2 to generate comment embeddings. We use scikit-learn (Pedregosa et al., 2011) to employ
WAI+LOCNESS
SK
JALEG
JFL.
Fluency (140K)
ITERATER-V2
5K
ITERATER
ITR-F
SVD and set the output dimension to 100.
ITERATER-VE
ITERATER
Reudability (180K)
ITERATER V2
10K
ITERATER
ITR-O
For pre-training. we choose pert-base-cased (De- vlin et al., 2019)² as our backbone model with the Huggingface Transformers (Wolf et al., 2020) toolkit. We initialize all experts based on the cor- responding counterparts in Bert. We use Adam Optimizer (Kingma and Ba, 2014) and clip the norms of gradients 1. À is set to 1.
ParaSCI
10K
SemEival
Turk
2K
Turk
TRK
Simplification (91X)
Asset
Neutralization (95K)
WNC
10K
WNC
WNC
Table 2: Statistics of the pre-training dataset, fine-tuning dataset, and test set. "Abbrev." denotes the abbreviation of the test dataset. Readability corresponds to the three tasks of clarity, coherence, and paraphrasing.
During additional fine-tuning, the expert respon- sible for the readability task duplicated into three instances. Each instance is utilized to train the tasks of clarity, coherence, and paraphrasing, as shown in Table 2. All experiments are carried out on 4 Nvidia GeForce RTX 3090 GPUs.
Exact Match (EM) calculates the percentage of predictions that exactly match the reference. This metric the official measure used by WNC.
6.3 Baselines
6.5 Main Results
Initially, we assess our models using non-pre- trained baseline, which employs identical fine- tuning data and edit model structure as G-SPEED. Then, we assess various existing LLMs using the EditEval setup, employing prompts to enable di- verse editing operations. Our primary compari- son is PEER (Schick et al., 2023), a collaborative language model trained on Wikipedia edit history that is initialized from the LM Adapted variant of T5 (Schick al., 2023). Scores for 3b and 1lb parameters PEER reported on EditEval. TO, To++ (Sanh al.) and Tk-Instruct (Wang al.. 2022) are other models that are initialized from the LM Adapted variant of T5 and then fine-tuned using prompts or instructions. We compare against GPT-3 (Brown et al., 2020), OPT (Zhang et al., 2022), InstructGPT (Ouyang et al., 2022), and ChatGPT decoder-only LLMs.
The main results are presented in Table The met- rics are compared among the following: Copy the text of source (Copy), supervised state-of-the-art models (SotA), LLMs, and our model (G-SPEED). The average SARI score is calculated as the mean of each task, while the score for specific task is computed the mean of the corresponding test datasets. Out of the models with fewer than 11 billion parameters, only PEER achieves a score more than 5 points higher than the average Copy score. Despite their large size, models such as OPT and GPT-3, which have 175B parameters, are un- able to ensure satisfactory performance in general text editing tasks. Among the LLMs, InstructGPT performs the best, slightly greater than ChatGPT. Additionally, PEER demonstrates favorable results with a relatively small number of parameters. G-SPEED surpasses the performance of LLMs with only 508M parameters. Compared with the best LLMs. InstructGPT and ChatGPT, G-SPEED achieves satisfactory results on coherence and nou- tralization tasks. G-SPEED exhibits noticeable gap compared to ChatGPT in fluency, paraphras- ing, and simplification tasks. However, these cas can be significantly improved through addi- tional fine-tuning. We compare our model with two variants: one without the pre-training step (de- noted as "w/o PRE.") and another without the fine- tuning step (denoted as "w/o ADDI.") as shown in Table 3. We find that both of these steps are crucial for our model. The model without either
6.4
Implementation Details
We present the information of the datasets in Table 2. For additional fine-tuning, we select data from various sources: W&I+LOCNESS (Bryant et al., 2019), ParaSCI (Dong et al., 2021), Turk (Xu et al., 2016), Wikiauto (Jiang et al., 2020), WNC (Pryzant et al., 2020), and the subsets of ITERATER- V2 (Kim al., 2022) with intent confidence scores higher than 0.8. Each expert has 10k data points for fine-tuning.
For data clustering, we employ K-Means++ with cuML (Raschka et al., 2020) toolkit, setting the number of clusters to 10. We choose all-mpnet-
hnttps://huggingfacc toosseeeneeetaanne transrorerrsffss/s/// all 1a--mpeet-basevv2
hhttps://chat.oppnaa
"https://huggingaace co co/bert-based base-cased

---

Fluency
Clarity ITR-L
Coherence ITR-O
Para. STS
Simptification
Neutral. WNC
Method
Size
Avg.
JFL
ITR-F
TRK
AST
Copy
26.7/ 40.5
32.3 86.0
62.9 46.2
3
21.1
26.3
20.7
319/0.0
27.8
SotA
6/62.4
37.2/ /-
38.3/-
34.4
37.2
-/45.8
Tk-instruct
31,8
32./61.6
38.4/ 58.4
33.8/70.4
30.2
32.8
29.9
31.3/0.4
32.9
TO
3B
38.0/38.8
24.6/34.9
32.6/30.2
22.2/21.6
34.3
34.4
32.3
22.3/0.0
29.7
T0++
1IB
34.7/43.2
35.3/75.8
37.6/56.5
32.7/59.9
28.4
32.9
28.2
29,
32.3
PEER-3B
3B
54.5
514/84.3
32.1/47.11
32.1/59.8
28.6
32.5
30,5
53.3/21.
38.5
PEER-118
11B
55.8/
54.3
58.1/85.
32.5/5133
32.7/62.7
28.2
32.1
29.5
54.5/
38.8
OPT
175B
47.3/47.5
34.7/70.6
31,5/31.5
27.6/ 36.1
29.1
32.6
31.8
31.2/0.4
32.1
GPT-3
175B
50.3/518
32.1/56.7
33.5
36.1
27.2
33.0
30.5
/..7
32,0
InstructGPT
175B
59.8/59.3
88.8/82.7
35.1/48.4
35.9 60.2
42,5
38.8
38.0
3.,4
40.4
ChatGT
65.6 61.3
45.6/70.4
3 30.5
33.7 33.5
36.9
39.7
44.6
/0.1
39.0
G-SPEED (Ours)
508M
54.2/519
42.2/78.8
53.6
41.0/ 68.4
36.0
38.5
40.6
60.2/31.2
43,2
w/o PRE.
508M
48.4/47.1
36.6/77.2
34.1/
40.2 / 20.1
33.6
39.8
40.3
57.7 / 28.8
41.4
w/o ADDI,
508M
46.6
73.6
36.9 /3.9
36.8 65.5
29.4
34.4
37.1
60.1
30.9
40.6
Table 3: The performance of G-SPPED and all baseline models on EditEvvl.. The hest score among LLMs and G-SPEED is Bold, and the second is Underlined. "Para." " denotes paraphrasing, "Neutral." means neutralization, and "Avg." stands for the average SARI score for each task. The first numbers for each task are SARI scores: additional metrics are GLEU for fluency, clarity, and coherence, and EM for neutralization. "w/o PRE." means the model fine-tuned without pre-training, and "w/o ADDI." denotes the pre-trained model without additional fine-tuning.
step performs poorly. In particular, the pre-training step greatly improves fluency, clarity, and neutral- ization tasks, while the fine-tuning step enhances coherence, paraphrasing, and simplification tasks. This can be attributed to the varying proportions of these tasks in the history of Wikipedia editing. Fur- thermore, our model, even without additional fine- tuning. performs slightly better than InstructGPT and ChatGPT, thereby confirming the effectiveness of our training and clustering method.
Router Structure We compare one static routing algorithm and two dynamic routing algorithms. (1) Select the expert corresponding to the task (referred to as "Task ID"). (2) Select the expert using a linear classification layer (referred to as "Linear"), which predicts the probabilities by softmax for each ex- pert and multiplies the highest probability with the output the corresponding expert. (3) Select the expert using a linear classification layer specific to each task (referred to as "Task ID + Linear"), which differs only in the number of classifiers with "Linear". 10 As shown in Table 4, the routing al- gorithms combined with task information perform better. We believe this is because the editing inten- tion cannot be directly inferred from the input text. The external knowledge about editing intentions benefits model training. Furthermore, we employ a "Token Level" routing algorithm in our model, which selects the experts for each token based on the hidden states of the token. However, this ap- proach yields inferior performance. Selecting an expert based the semantics of a whole sentence is more reasonable and efficient.
6.6 Further Analysis
Expert Structure In Table 4, we present the re- sults for two types of expert structures: (1) sparse last encoder layer (referred to as "Last Layer"), which often impacts the final output, and (2) sparse feed-forward layer (referred to "Feed Forward") in each encoder layer. Additionally, we include the results of dense model (referred to "Dense"), where the entire encoder is shared, and differen- tiation between different tasks occurs solely at the linear classification layer. Our findings indi- cate that sparse models generally outperform the dense model, with inference times being almost identical. Furthermore, models with sparse feed- forward layers show slightly better performance than those with sparse last encoder layers. Further- more, we examine a model that shares the experts in tagging and generation (referred to as "Sharing T&G"), which can further reduce the model's size. However, this approach yields inferior performance compared to the other models.
Few Shot Learning Table 5 presents the results of few-shot learning using the G-SPEED back- bone for general text editing tasks. We main- tain the same settings as in the additional fine- tuning, setting the data volume for each task at 500, 1 5k, and 10k, respectively. We compare the
10ehe router weights are initialized using a normal distribu- tion with a mean of ( and a standard deviation of 0.001. The temperature of the softmax in the router is 0.7.

---

Fluency
Clarity ITR-L
Coherence ITR-0
Para.
Simplification
Neutral. WNC
Method
Avg.
JFL
ITR-F
STS
TRK
AST
Dense
42.8 46.0
42.0/ 79.6
35.5 / 55.3
36.8 66.8
24.6
32.6
35.7
$7.5 27.8
38.5
Last Layer
Task ID
44.8/47.2
42.6/ 80.4
36.1 53.4
36.9 65.66
28.3
32.7
37.1
59.1 28.5
39.8
Linear
43.1 44.2
42.5
36.0
37.7 /
34.0
58.1
39.4
Task ID + Lincar
45.0 47.3
41,8/79.7 79.7
35.9/54.2
39.1 67.9
28.6
33.0
37.1
59.0/
28.4
40.2
Sharing T&G
40.0% 44.3
40.8/ 80.7
34.7 54.8
36.9 / 69.2
24.5
33.4
37.4
56.9
26.3
38.1
Feed Forward
Task ID
45.6/ 46.6
43.6 79.3
36.9 /3.99
36.8 65.5
29.4
34.4
37.1
60.1 30.9
40.6
Sharing T&G
43.1 46.00
42.7 80.4
35.5 54.5
36.0/ 68.1
26.7
33.7
37.4
57.1 27.0
39.0
Linear
44.6/47.0
42.9/79.9
36.5/ 54.6
68.4
32.4
58.8/28.2
40.0
Token Level
44.2
71.4/7911
36.1 54.77
35.6/ 64.0
29.6
31.9
37.6
59.3 29.2
39.8
Task Lincar
45.3/ 46.8
41.2/78.3
37.0/ 54.7
38.0 68.1
28.6
35.0
37.7
59.3
29.8
40.4
Token Level
45.0/44.4
40.2/77.0
36.8 /5.44
36.5 65.0
28.2
34.3
37.6
59.0
30.4
39.8
Table 4: Comparison of various sparse layers on EditEval. The best score is Bold, and the second is Underlined. "Linear" means a linear classifier for routing; "Task ID + Linear" denotes a classifier specific to each task; "Sharing T&G" means using the same experts for tagging and generation model; "Token Level" indicates that experts are activated separately for each token.
Train Data Size
Method
0.1
0.01
Method
0.5K
1K
5K
10K
BERT2BERT(Rothe al., 2020)
3.4
0.0
TS-hasc(Raffel al., 2020)
33.8
10.8
G-SPEED
41.4
41.8
42.4
42.9
LASERTAGGER (Malmi et al., 2019)
25.7
12.3
w/o PRE.
38.9
39.5
40.8
41.4
FELIX (Mallinson al., 2020)
36.9
17.0
EDIT5 (Mallinson et al., 2022)
43.8
28.6
Table 5: Comparison of our pre-trained and non-pon pre- trained models in few-shot learning. The head of the column is the amount of training data for each task. We report the average SARI score for each task.
G-SPEED (Ours)
47.8
33.2
Table Comparison of Seq2Seq models, single editing models, and G-SPEED on sentence fusion task (Exact Match) under various data conditions.
model that skips the pre-training step (referred to as "w/o PRE") with our pre-trained model. We find that the pre-trained model outperforms the non- pre-trained model. Furthermore, the fine-tuning results with 500 samples are nearly equivalent to those of the un-pre-trained model trained on 10k samples, demonstrating its generalization abil- fine-
ing data. Furthermore, fine-tuning our pre-trained model yields significantly better performance than other editing models, demonstrating the generaliza- tion ability of our method on downstream tasks.
7 Conclusion
ity downstream tasks. Additionally, tune our pre-trained model on the sentence fusion task and compare with editing models. Follow- ing the work of Mallinson et al. (2022), we use the "balanced Wikipedia" subset of the DiscoFuse dataset (Geva et al., 2019) and compare the results with those of editing models using 4,500 (0.1 %) and 450 (0.01%) () training data points. We report the Exact Match score of G-SPEED, while the other results are reported by Mallinson al. (2022,
In this work, propose the General SParse Ef- ficient Editing MoDel (G-SPEED), model de- signed fulfiil diverse editing needs while ensur- ing computational efficiency. Specifically, we first propose a novel unsupervised clustering strategy to obtain a large amount of multi-intent editing data, which is collected from editing histories from Wikipedia and can be used to conduct pre-tnnin
Subsequently,
propose novel sparse editing model architecture improve the learning abili- ties of small models. The experimental results on EditEval show that, with the use of Bert-base as the backbone model, G-SPEED can outperform LLMs while maintaining an efficient inference speed. Ad-
2020). In Table 6, our initial observation is that the Seq2Seq model (BERT2BERT (Rothe et al., 2020),
T5 base (Raffel et al., 2020)) performs poorly, in- dicating that editing models are effective on train-

---

ditionally, we discuss different sparse structures and show the strong generalization capability of our method across downstream tasks.
task on grammatical error correction. In Proceed- ings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52-75.
Limitations
Daniel Cer, Mona Diab, Eneko E Agirre, lñigo Lopez- Gazpio, and Lucia Specia. 2017. Semeval-201 2217 task 1: Semantic textual similarity multilingual and cross- lingual focused evaluation. In The 11th International Workshop on Semantic Evaluation (SemEval-2017)) pages 1-14.
A limitation of unsupervised clustering is that cannot deal with revisions that contain more than one editing intent. Although text editing in the history of Wikipedia is iterative, the degree of cach modification by the user is still uncontrollable.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Fric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. 2022. Scaling instruction-finee finetuned language models. arXiv preprint arXiv: 221.1112....111...
Ethies Statement
We collect all data from publicly available sources and test our model on public datasets. Since our work mainly focuses on non-meaning-changing text edits, we are able to avoid many issues involv- ing generating harmful text. Therefore, our work has no possibility of generating harmful informa- tion in practical applications.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume I (Long and Short Papers), pages 4171- 4186.
A Acknowledgements
Qingxiu Dong, Xiaojun Wan, and Yue Cao. 2021. Parasci: large scientific paraphrase dataset for longer paraphrase generation. In Proceedings of the 16th Conference the European Chapter of the Asso- ciation for Computational Linguisticx: Main Volume, pages 424-434.
This work is supported by the National Sci- ence Foundation of China (No. 62176174, No. 62206194) and the Natural Science Foundation of Jiangsu Province (No. BK20220488).
Nan Du Yanping Huang, Andrew M Dai. Simon Tong. Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, al. 2022a. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference Machine Learning, pages 5547-5569. PMLR.
References
Fernando Alva-Manchego, Louis Martin, Antoine Bor- des, Carolina Scarton, Benoft Sagot, and Lucia Spe- cia. 2020. Asset: A dataset for tuning and evalua- tion of sentence simplification models with multiple rewriting transformations. Annual Meeting of the Association for Computational Linguistics, pages 4668-4679.
Wanyu Du, Vipul Raheja, Dhruv Kumar. Zae Myung Kim, Melissa Lopez, and Dongyeop Kang. 20226. Understanding iterative revision from human-writter text. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Vol- ume I: Long Papers), pages 3573-3590.
Proceedings of the 58th
Talita Anthonio, Irshad Bhat, and Michael Roth. 2020. wikihowtoimprove: A resource and analyses on edits in instructional texts. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5721-5729.
Jane Dwivedi- Yu, Timo Schick, Zhengbao Jiang, Maria Lomeli, Patrick Lewis, Gautier Izacard, Edouard Grave, Sebastian Riedel, and Fabio Petroni. 2022. Editeval: An instruction baseed benchmark for text improvements. arXiv preprint arXiv:2209 13331.
David Arthur and Sergei Vassilvitskii. 2007. K-
means++ the advantages of careful seeding, In Pro- ceedings of the eighteenth annual ACM-SIAM sym- posium Discrete algorithms,, pages 1027-1035.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, al. 1996. A density- -based algorithm for dis- covering clusters in large spatial databases with noise. In kdd, volume 96, pages 226-232
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et 2020. Language models are few-shot learners. Advances in neural information processing systems, 1911
Felix Faltings, Michel Galley, Gerold Hintz, Chris Brockett, Chris Quirk, Jianfeng Gao, and William B Dolan. 2021. Text editing by command. ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Techmologies, pages 5259-5274.
Pro-
Christopher Bryant, Mariano Felice, Øistein Ander- sen, and Ted Briscoe. 2019. The bea-2019 shared

---

William Fedus. Jeff Dean, and Barret Zoph. 2022a. Ae-- view of sparse expert models in deep learning. arXiv preprint 2XXi::209.01666
Jiaqi Ma, Zhe Zhao, Xinyang Yi. Jilin Chen, Lichan Hong, and Ed H Chi. 2018. Modeling task relation- ships in multi-task learning with multi-gate mixture- of-experts. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1930-1939.
William Fedus, Barret Zoph. and Noam Shazeer. 2022b. Switch transformers: Scaling to trillion parame- ter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):5232- 5270.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. et al. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint ar arXiv:2303.1765.
Mor Geva, Eric Malmi, Idan Szpektor, and Jomathan Berant. 2019. Discofuse: A large-scale dataset for secourse-bbsed sentence fusion. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 1443-3455.
Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2022. EdiT5: Semi- semi-autoreerressive text editing with warm-start. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2126 2138, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Shashank Gupta, Subhabrata Mukherjee. Krishan Sub- udhi, Eduardo Gonzalez. Damien Jose, Ahmed Awadallah. and Jianfeng Gao. 2022. Sparsely acti- vated mixture-ee of experts are robust muulti-task learn- ers. arXiv preprint Nww:2044076899
Jonathan Mallinson, Aliaksei Severyn, Eric Malmi, and Guillermo Garrido. 2020. Felix: Flexible text editing through tagging and insertion. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1244-1255.
Sebastian Jaszezur, Aakanksha Chowdhery, Afroz Mo- hiuddin, Lukasz Kaiser, Wojeiech Gajewski, Henryk Michalewski, and Jonni Kanerva. 2021. Sparse is enough in scaling transformers. Advances in Neural Information Processing Systems, 34:9895-9907.
Eric Malmi, Sehastian Krause, Sascha Rothe, Daniil Mirylenka, and Aliaksei Severyn. 2019. Encode, tag. realize: fiigh-precisoon text editing. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIPP-JCNIPP), pages 5054-5065.
Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. 2020. Neural erf model for sentence aligmment in text simplification. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7943-7960.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generaliza- tion via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3470-3487, Dublin, Ireland. Association for Computational Linguistics.
Zae Myung Kim, Wanyu Du Vipul Raheja, Dhruv Ku- mar, and Dongyeop Kang. 2022. Improving iterative text revision by learning where to edit from other revi- sion tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9986 (8669999,,.44 Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical error correction metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Confer- ence on Natural Language Processing (Volume 2: Short Papers). pages 588-593.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint
James Lee- Lee-Thorp and Joshua Ainslic. 2022. Sparse mixers: Combining moe and mixing to build a more efficient bert, arXiv preprint arXiv:2205. 12399.
Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. Jfleg: A fluency corpus and bench- mark for grammatical error correction. In Proceed- ings of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 229- -944,
Mike Lewis, Shruti Bhosale, Tim Dettmers. Naman Goyal, and Luke Zettlemoyer. 2021. Base layers: Simplifying training of large, sparse models. In In- ternational Conference Machine Learning, pages 6265-6274. PMLRR
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. Gector-grammaticll error correction: Tag. not rewrite. In Proceedings of the Fifteenth Workshop Innovative Use of NLP for Building Educational Applications, pages 163-170.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint Xixv:19077 11692.
OpenAI. 2023. Introducing chatgpt.

---

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances in Nearal Information Processing Systems, 35:27730-27744,
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, al. 2021. Multitask prompted training en- ables zero-shot task generalization. arXiv preprint 1rXiv:2110.082077
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay. 2011. Scikit-leam: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830
Timo Schick, Jane A. Yu, Zhengbao Jiang. Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You. Christoforos Nalmpantis, Edouard Grave, and Sebas- tian Riedel. 2023. PEER: A collaborative language model. In The Eleventh International Conference on Learning Representations.
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. 2014. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27.
Reid Pryzant, Richard Diehl Martinez, Nathan Dass, Sadao Kurohashi, Dan Jurafsky, and Diyi Yang. 2020. Automatically neutralizing subjective bias in text. In Proceedings of the aaai conference on artificial intel-
ligence, volume 34, pages 480-489.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018, Glue A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLPP. Analyz- ing and Interpreting Neural Networks for NIP, pages 353-355
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. Journal of Machine Learning Research, 21:1- 67.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasckaran, An- jana Arunkumar, David Stap, et al. 2022. Super- naturalinstrrutiios:: Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 5085-5109.
Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may i introduce the gyafc dataset: Corpus, bench- marks and metrics for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long Papers). pages 129-140.
Sebastian Raschka, Joshua Patterson, and Corey Nolet. 2020. Machine learning in python: Main develop- ments and technology trends in data science, machine learning, and artificial intelligence. arXiv preprint arXiv:2002.0444 048003.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. 2021. Finetuned lan- guage models are shot learners. arXiv preprint arXiv:2109.01652.
Nils Reimers and Tryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- (JCNLPP pages 3982-3992.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz. et al. 2020. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 con- ference on empirical methods in natural language processing: system demonstrations, pages 38-45.
Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. 2021. Hash layers for large sparse models. Advances in Neural Information Processing Systems, 34:11555-17566.
Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing sta- tistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401-415.
Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for se- quence generation tasks. Transactions of the Associ- ation for Computational Linguistics, 8:264-280.
Diyi Yang, Aaron Halfaker, Robert Kraut, and Eduard Hovy. 2017. Identifying semantic edit intentions from revisions in wikipedia. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2000-2010.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, et al. Multitask prompted training enables zero-shot task generalization. International Conference on Learning Representations.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel
Astetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.

---

Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0106.
minimizing the presence of inaccuracies. Addition- ally, another filtering criterion involves considering differences in sentence length to identify and ex- clude incorrect simplification data.
Xuchao Zhang, Dheeraj Rajagopal, Michael Gamon, Sujay Kumar Jauhar, and ChangTien Lu. 2019. Mod- eling the relationship between user comments and edits in document revision. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing SOMNLP-JCCNLE pages
Table 2 presents the central words and a selec- tion of sampled comments within each cluster. A concise description is provided for each cluster. The clusters employed for training purposes are clusters 1, 2, 3, and 5. The central words corre- spond to the most frequently occurring words in each cluster, excluding stop words, and they exhibit a high degree of concentration within their respec- tive clusters. Detailed information regarding the dataset instances is presented in Table 1. Figure 2 illustrates the distribution of semantic clustering sult maps in a two-dimensional space. To enhance the visualization of clustering effects, we employed a random subsample from the dataset and applied the Uniform Manifold Approximation and Projec- tion (UMAP) technique to condense the semantic data into a wwo-dimensional vector representation.
Simiao Zuo, Qingru Zhang, Chen I Liang, Pengcheng He, Tuo Zhao, and Weizhu Chen. 2022. Moebert: from bert to mixture-of of experrs via importance- guided adaptation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies, pages 1610-1623.
A Details and Studies on Datasets
In addition to the filtering rules discussed in Sec- tion 4, we employ additional filtering criteria for revisions. Firstly, we exclude revisions whose com- ments contain any of the following terms: "tem- plate", "image", "infobox", "pic". "link", "photo". "comment", "http:", "https:". "jjg". "-png". or "re- ply". The presence of these terms suggests the involvement of unwanted revision types, such as image alterations and citation modifications, which can complicate the clustering process (e.g., due to lengthy URL links in comments). Furthermore, we remove any links present in the source and target texts. Additionally, we replace certain im- portant shorteuts with their full meanings to en- hance data clustering. For instance, we substitute "[[P.NPOOVPOV]I" with "neutral point of view", "[[WP-TYPO]" with "typo", "[[WP:RS]]" with "reliable sources", and "[[WP:SYN]]" with "syn- thesis". This modification contributes to improved clustering outcomes. Furthermore, we exclude re- visions that solely pertain to number or time modi- fications, such as updates on the real-time number of COVID-19 cases in specific location. More- over, we filter out revisions in which both the com- ments and the source texts exhibit significant sim- ilarities. This type of overlap revision typically corresponds to low-quality comments. Since our extraction of source and target relies solely on doc- ument comparison, there are instances where may encounter incorrect incomplete source and target pairs. Consequently, it becomes necessary to conduct a basic screening of the clustered data. One approach involves utilizing BLUE to identify and filter out sentence pairs that do not match, thereby
We have undertaken meticulous examination to determine the presence of any data contamination between our training and test datasets. Specifically, in the evaluation benchmark, JFLEG, Turk, ASSET, and ITERATER are manually annotated; STSB is collected from forums and news. Only the WNC dataset is collected from the revision history of Wikipedia passages. However, WNC is collected from the revisions made between 2004 and 2019, while our dataset is collected from the revision histories in the March I st, 2023 dump of English Wikipedia. For further confirmation, we first con- catenate the source and target of each data sample and use traversal search to filter the training sam- ples of our pre- pre-training datasets that have a BLUE value of 0.9 with any data sample of WNC. How- ever, no such data sample satisfies this requirement, which further confirms there is no contamination.
B Details and Experiments on Tags
Similar to the approach taken in GEC- TOR (Omelianchuk et al., 2020), our work considers line-grained tag design. We present three different tag designs in Table 8, where the tag names and meanings correspond to those used in GECTOR. The decision to reduce the number of tags from 34 to 14 is based on the frequency of tag occurrence in the pre-training dataset and its compatibility with the pre-training process. Table displays the performance results of the three

---

Cluster Center Words
Comment Samples
Description
0 born, father, wife. died, birth, death
*+A little history the Spanish Monastery": "Source for Jane Merriett Netterville party and year of birth";
Supplementation and modification of infor- mation about character relationships, char- acter information, history, etc.
rewrote, ediy. tweaks, flw, rewording, read- ability, awkward, improve, clarity, phrasing
"reworded for clarity". ": "paragraph split, wording"; : "Fix-
Modification of xenience structure and wording, improvement of readability and clarity, and copy-edit
ing awkward sentence"; : "Small sentence change to make sentence seem more natural." "; "fix awkward phrasing." "; "Made more clear and precise.";
2 neutral, view, opinion. evi- dence, true, point, claim
"and skepticism, the other main scientific bias": "it was coninoversial"; ; 'I. More accurate info. 2, No proof or source that anyone lost their savings.", " "removed bias and point of view pushing":
Neutralization. verification of evidence, and removeal of hias and confusion
3 cleanup. typos, fixing, er-
"Typo fixing, replaced: anatomy anatomy 2192 anatomy"; "Clean typo fixed: the USA 2192 the US": "minor grammatical change"
Correction of grammar, typo, misspelling and other emors
rors, density, mistakes, mis- spelling, grammatical
4 updating, updated, add, inlo
"Add date with source." "; "Addition of storyline info..; "adding page and info about copyright";
Information updates and additions
5 deletion, vandalism, dupli- cate, delete, redundant, super-
"Removing Hurrian deities, who have separate cate- down helow. Inara is not Hurrian, neither is Hamma- hanna. Also, Hebat was listed multiple times.":
Text simplitication, information removal
fluous
6 units, numbers, %,
"In units where c=1, mass and energy expressed in the same units c.g. in Kg.": "Mexican Navy orders 2 units.": 'It's only the sccond aircraft operated."
Messy statements
7
citations, cite, references, ref, reliahle
"new inline reference"; "help pleuse similar problem, bui the cite butlon has disappeared"; Addedd needed citations."
Correction and verification of reffrences
8 capiitaived, draft, plural, hy-
"Hy "Hypataation and consistency"; "what is with the caprtal letters?":
Modification of hyphenation, capitaliza. tion, etc.
phen, capitalization
9 italicize, commis itaiee, semi- colon, dashes, colon, marks, parenthesis
"putting in () marks tn make the puge betler"; "changed hold font into quotes": "changed hyphen to colon":
Modification of punctuation and font
Table 7: The center words and comments of each cluster with a description.

---

Tags
4(KDRA) KEER DELETE, REPLACE, APPEND
(ours)
APPEND, DELETE, KEEP, MERGE
MERGE_HYPHEN HYPHEN, REPLACE, TRANSFORM AGREEMENT PLURAL, TRANS-
DORM_AGREEMENT SINGULAR, SINGUUL
TRANSFORM, CASE_CAPITAL TRANSFORM 1AAE.AAAL
TRANSFORM, CASE LOWER, TRANSFORM CASE_ TRANSFORM CASE, UPPER_II. FORM_SPLIT
TRANS-
HYPHEN, SEAESSPPIT_HYPHEEN
TRANSFORM VERB
APPEND, DELETE, KEEE MERGE HYPHEN, MERGE,
MERGE_SPACE, SPACE, REPLACE, TRANSFORM AGREEMENT PLURAL,
TRANSFORM, AGREEMENT SINGULAR, TRANSFORM CASE, CAPITAL, TRANSFORM CASE CAPITAL
TRANSFORM CASE, LOWER,
TRANSFORM CASE, UPPER,
TRANSFORM CASE, UIPPER -I,
TRANS-
FORM HYPHEN.
VERB,
FORM VERB VBD VB,
TRANSFORM, VERB VBD. VEZ, VBG TRANSFORM, VERB. VENVED,
TRANSFORM, VERB VBG, VE.
TRANS-
FORM VERB VBN, VB,
TRANSFORM, VERB VEN VBG.
TRANS
FORM VERB VBN, VBZ,
TRANSFORM VERB, VBZ VB,
TRANSFORM VERB VBZ, VBD. TRANSPORM VERBVB VBD,
TRANS-
FORM VERB_VBB
TRANSFORM VERB VBN,
TRANS-
FORM_ VERB. VBG. TRANSFORM VERB VB, VBN, TRANSFORM VERB. VB2
Table 8: Details of the tag sets we compared.
Fluency
Clarity ITR-L
Coherence ITR-O
Para. STS
Simplification TRK
Neutral. WNC
Tag Design
Avg.
JEL
ITR-F
AST
Last Layer
14ugs (Ours)
47.3
41.8/ 79.7
54.2
39.1 67.9
28.6
33.0
37.1
59.0
40.2
KDRA
44.6 46.8
12.9/79.9
54.8/
37.5/67.3 67.3
29.2
32.8
36.9
58.9
40.0
34tags
42.6/78.8 45.3/46.9
36.6 /3.4
37.1 66.2
29.7
33.4
37.1
28.7
40.2
Feed Forward
14ugs (Ours)
46.6
43.6/ 79.3
36.9 53.9
65.5
29.4
34.4
37.1
60.1 60.1//300
30.9
40.6
KDRA
45.9/47.3
41.9778 78.5
37.0 54.2
36.1 65.4
28.6
34.4
37.1
58.9 29.6
40.0
34tags
45.3 46.6
41.1 78.2
36.4/
64.2
30.3
34,5
37.1
59.4 30.9
40.1
Table 9: Comparison of the impact of different tag designs on the models.
Laser JaserTagger
FELIX G-SPEED
33.0
33.5
43.2
Comparison of three editing models that were trained with the same dataset. The results in the table are average SARI scores across multiple tasks.
Table 10:
UMAP projection of the training dataset
tag designs on two sparse models. Notably, the design comprising 14 tags achieves the highest performance among all designs.
C Efficiency Comparison
As demonstrated in Table 10, G-SPEED exhibits superior multi-tasking capabilities when contrasted with previous editing models due to its sparse architecture. Furthermore, as shown Table 1, compared to fine-tuned Large Language Mod- els (LLMs). G-SPEED boasts conspicuous advan- tages in model size, computational speed, and per- formance. Specifically, use the WNC test set for inference on single NVIDIA GeForce RTX 3090. We conducted fine-tuning on both TO and LLaMa using the identical dataset. We also adjusted the data input format to "instruction: input," as was performed during instruction fine-tuning. Notably,
2-
0-
Figure 2: UMAP projection of the training dataset. The four colors represent the four clusters used in pre- training.

---

Model
Model Size 4
SARI f
Tokens/s f
Speed Up1 f
LLaMa-7B
1.00x
54.1
21.05
1.00x
TO-3B
0.43x
30.5 (-23.6)
39,46
1.87x
G-SPEED (508M)
0.07x
60.2 (+6.1)
754.33
35.84x
Table II: Comparison between fine-tuned LLMs and G-SPEED.
Model
SARI
GLEU
ChatGPT
36.4
30.9
G-SPEED
36.6
48.3
Table 12: Comparison multi-intent iterative text mod- ification capabilities.
G-SPEED achieves a remarkable speed boost up to 35 times faster than LLaMa-7B. Morcover, the training costs of G-SPEED are substantially more economical than those of LLMs.
D Iteration Study
There currently no large for multi-intent iterative editing tasks. In our study, we employed the
doc dataset!! which comprises a total of 51 manually annotated instances, to assess and contrast the iterative editing proficiency of both ChatGPT and G-SPEED, Table 12 shows that the two models are similar in the SARI score, but G- SPEED retains the original text better.
We also explored multiple iterations within sin- gle task. As indicated in Table 13, we conducted a comparison our pre-trained model at various editing depths. Our findings reveal that the flu- ency task consistently benefits from iterative edit- ing, whereas most tasks are more inclined toward single-pass modifications. Table 14 presents spe- cific instances from the JFLEG dataset, illustrating the incremental revision process employed by the editing model address grammatical errors.

---

Fluency
Clarity ITR-L
Coherence ITR-O
Para. STS
Simplification TRK
Neutral.
Method
JFL
ITR-F
AST
WNC
Last Layer
depth
47.0
41.8/ 79.7
35.9/54.
39.1 201/677 67.9
28.6
33.0
37.1
59.0 28.4
depth =
48.7 / 49.4
79.11
/3.1
65.7
28.6
28.4
36.0
55.5 12.6
depth 3
49.5 49.7
42.6/ 72.7
/
37.5/ 65.6
29.1
24.6
33.2
54.7 12.8
Feed Forward
depth 45.6/ 46.6
43.6 79.3
36.9/53.9
36.8 65.5
29.4
34.4
37.1
60.1 / 30.9
depth 49.6 48.7
43.7 / 78.3
/6.3/
35.7 63.2
29.6
30.0
35.8
54.8 / 8.8
depth 3
50.6/ 48.9
41.4/ 76.4
36.6/
35.0/60.5
30.2
26.3
33.4
53.3/9.4
Table 13: Comparison of the results after multiple iterations. Bold indicates the best score each model.
Depth Outputs
Bigger farming are use more chemical product and substance to feed fish.
Bigger farming uses more chemical product and substance feed fish.
2
Bigger farming uses more chemical products and substances to feed fish.
3
Bigger farming uses more chemical products and substances to feed fish.
they did not get the ideas or any concepts about what they learn.
1
They did get the ideas or any concepts about what they learn.
2
They did not get the ideas any concepts about what they learned.
3
They did not get the ideas or any concepts about what they learned. larned many kind of subject, also could make difrent types friends
1
studied many kind of subject, also could make different types friends.
2
studied many kind of subjects, also could make different types of friends. studied many kind of subjects, also could make different types of friends.
3
Table 14: The iteration cases on JFLEG, The depth of 0 refers to the source text.


Reviews:
[{"review_id": "Qkr4XEZfch", "reviewer": "Reviewer_qsvy", "summary": "", "questions": "1. The EditEval includes 7 kinds of editing intents. I would like to know the reasons for not choosing the Update editing task.\n\n2. Are there any quality studies about the pre-training dataset?\n\n3. Please discuss the pros and cons of LLM-based generative editing approaches (e.g., ChatGPT) and G-SPEED. It is observed from Table 2 that G-SPEED does not win across the board on EditEval. LLM-based generative editing approaches still outperform 3 of 6 tasks. Also, though G-SPEED is light-weight and powerful, it has limitations in generalizing to unseen editing tasks. To the best of my knowledge, G-SPEED needs to be finetuned with labeled data before being used for new editing tasks, such as the clarity task in EditEval, but ChatGPT like approaches do not have such limitations. Hence, to give comprehensive guidance for the community and practitioners, it is recommended to holistically discuss the pros and cons of G-SPEED and ChatGPT like approaches.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your reviewing efforts and valuable comments! The following responses are intended to respond to your questions and alleviate your concerns.\n\n\n\n> Reject Reason: Limited Technical Novelty.\n\nA: In this work, to the best of our knowledge, we are the first to propose a lightweight general text editing model (G-SPEED) with only 508M parameters, which can surpass LLMs equipped with 175B parameters. Due to the data scarcity problem of editing tasks and small language models' inherently limited learning capabilities, simply combining existing methods cannot obtain a general editing model based on small language models. Therefore, to meet this challenging goal, apart from testing different variants of existing methods empirically, we propose a novel unsupervised text editing data clustering method and a sparse editing model architecture. Both of these strategies are under-explored in the field of text editing, and the experimental results fully confirm the effectiveness of our proposed methods. Hence, we believe this work makes significant and nontrivial technical contributions to promote the development of text editing.\n\n> Question 1: The EditEval includes 7 kinds of editing intents. I would like to know the reasons for not choosing the Update editing task.\n\nA: Sorry for the missing information. The Update editing task requires references from external sources that are relevant to the particular task.  This task is inconvenient to evaluate because text updates can be multi-directional, especially when we have multiple references without a clear update direction (such as correcting hallucinations) and only one reference answer. One parallel work [1] doesn't choose the Update editing task for the same reason. We will supplement this information in the next version.\n\n\n\n> Question 2: Are there any quality studies about the pre-training dataset?\n\nA: In Table 7, we have shown the center words and comments of each cluster of the pre-training dataset. In the next version, we will further use UMAP to visualize the data clustering results and combine the high-frequency word statistics to analyze the differences and similarities between multiple clusters. We will also conduct more case studies of the pre-training dataset in the next version.\n\n\n\n> Question 3: Please discuss the pros and cons of LLM-based generative editing approaches (e.g., ChatGPT) and G-SPEED. \n\nA: Thanks for your suggestions. In lines 441-464, we have compared the performance of G-SPEED and the LLM-based approaches on different tasks. Following your guidelines, in the next version, we will supplement more discussion of the pros and cons of performance, cost, and generalization. (1). In terms of performance, due to the different requirements of various text editing tasks and the different characteristics of models, G-SPEED and the LLM-based approaches excel in different types of tasks. Specifically, G-SPEED tends to make more precise modifications with caution, while the LLM-based approaches are generated from scratch in an autoregressive manner and hence more likely to encounter over-correction problems; (2) In terms of time and space cost, G-SPEED has an absolute advantage over LLMs. G-SPEED not only calculates fewer parameters but also generates responses in a non-autoregressive way, so it is very suitable as a flexible and lightweight application (3). In terms of generalizing to unseen editing tasks, because the pre-training stage is irrelevant to downstream tasks,  G-SPEED can generalize to unseen editing tasks without the need for additional fine-tuning on downstream tasks. Specifically, in Table 3, we can observe that without additional fine-tuning on the train data of downstream tasks, G-SPEED can still outperform InstructGPT and ChatGPT, which shows the strong generalization ability of G-SPEED to unseen editing tasks.\n\n\n\nReference:\n\n[1] Zhang Y, Cui L, Cai D, et al. Multi-Task Instruction Tuning of LLaMa for Specific Scenarios: A Preliminary Study on Writing Assistance[J]. arXiv preprint arXiv:2305.13225, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy to follow.\n\n2. The proposed clustering method and its resulting editing data is useful for the community to advance the field of text editing. Also, the impressive performance of the light-weight G-SPEED model could benefit more applications and practitioners.", "reasons_to_reject": "1. Limited technical novelty. The tagging technique, non-autoregressive editing tokens generation, and sparse FFN layer have been explored extensively in the literature. The ablation study and analysis cannot comprehensively tell the reasons why G-SPEED outperforms prior work by a large margin. It is suggested that the authors elaborate the differences with prior work in the Editing Model subsection in Related Work.", "questions_to_authors": "1. The EditEval includes 7 kinds of editing intents. I would like to know the reasons for not choosing the Update editing task.\n\n2. Are there any quality studies about the pre-training dataset?\n\n3. Please discuss the pros and cons of LLM-based generative editing approaches (e.g., ChatGPT) and G-SPEED. It is observed from Table 2 that G-SPEED does not win across the board on EditEval. LLM-based generative editing approaches still outperform 3 of 6 tasks. Also, though G-SPEED is light-weight and powerful, it has limitations in generalizing to unseen editing tasks. To the best of my knowledge, G-SPEED needs to be finetuned with labeled data before being used for new editing tasks, such as the clarity task in EditEval, but ChatGPT like approaches do not have such limitations. Hence, to give comprehensive guidance for the community and practitioners, it is recommended to holistically discuss the pros and cons of G-SPEED and ChatGPT like approaches.", "paper_topic_and_main_contributions": "This paper targets the task of text editing, where an input document is edited according to a given intent. To tackle the task, this paper proposes G-SPEED, a lightweight general editing model. G-SPEED is made up of a tagging model and a generation model. The tagging model learns to mark the editing operation for each word in the document, while the generation model learns to generate new words or transformed words based on the tagging results in a non-autoregressive manner. To further boost the model performance, this paper employs a mixture-of-expert strategy for both the tagging and generation models. Specifically, each expert is instantiated as an independent FFN in a Transformer block. To pre-train the G-SPEED for general editing, this paper proposes unsupervised editing data clustering, a method to curate datasets of <Text before edited, Text after edited, Edit intent>. To evaluate the performance of G-SPEED, the paper conducts experiments on the EditEval benchmark. Experimental results show that G-SPEED achieves state-of-the-art performance on some editing tasks.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. As editing data clustering is one of the major contributions of the paper, it is necessary to discuss relevant editing dataset construction methods in Related Work."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Missing full stop in Line071.\n\n2. The sparse BERT layer is not well-motivated in current writing. From my point of view, it is a solution for multi-task training. The authors could try to re-frame the introduction and motivation of using sparse layer."}}, {"review_id": "JFX4EaS4k2", "reviewer": "Reviewer_XAag", "summary": "", "questions": "A.\tCould you check if there is any contamination between data collected by your clustering algorithm and the test sets of the evaluation tasks?\n\nB.\tFor an input with a single editing intent (say fluency), could different experts (corresponding to different intents) be activated at different sparse BERT layers? If yes, what does this mean? If we assume single intent input, were any model variations tried to enforce that the same expert is activated at all layers?\n\nC.\tIn Table 3, the performance of G-SPEED on clarity seems to improve without pre-training or without fine-tuning. Discussion on this could be added.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate your thoughtful reviews! We hope the following response can address your questions and concerns well.\n\n> Reject Reason 1: Unsure of contamination between data collected in an unsupervised manner and the test set of the evaluation tasks.\n>\n> Question A: Could you check if there is any contamination between the data collected by your clustering algorithm and the test sets of the evaluation tasks?\n\nA: This is a really thoughtful comment. After carefully checking data sources, we think there is no contamination between our pre-training data and the test sets of the evaluation tasks.  Specifically, in the evaluation benchmark, JFLEG, Turk, ASSET, and ITERATER are manually annotated; STSB is collected from forums and news. Only the WNC dataset is collected from the revision history of Wikipedia passages. However, WNC is collected from the revisions made between 2004 and 2019, while our dataset is collected from the revision histories in the March 1st, 2023 dump of English Wikipedia. For further confirmation, we first concatenate the source and target of each data sample and use traversal search to filter the training samples of our pre-training datasets that have a BLUE value of 0.9 with any data sample of WNC. However, no such data sample satisfies this requirement, which further confirms there is no contamination. We will supplement these details in the next version.\n\n> Reject Reason 2: Comparisons could be performed with other editing models like Levenshtein Transformer, etc.\n\nA: Thanks for your suggestions. Because Levenshtein Transformer mainly focuses on machine translation tasks, we select two popular editing models (e.g., LaserTagger [1] and FELIX [2]) as baselines. Below are the SARI scores of different models:\n\n|                 | JFL      | ITR-F    | ITR-L    | ITR-O    | STS      | TRK      | AST      | WNC      |\n| --------------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| LaserTagger [1] | 40.0     | 34.1     | 33.0     | 33.1     | 25.6     | 33.1     | 35.7     | 35.1     |\n| FELIX [2]       | 36.1     | 34.2     | 31.5     | 34.8     | 29.1     | 34.5     | 36.3     | 35.2     |\n| G-SPEED         | **54.2** | **42.2** | **34.0** | **41.0** | **36.0** | **38.5** | **40.6** | **60.2** |\n\nWe can observe that G-SPEED outperforms LaserTagger and FELIX. We will report these results in the next version.\n\n\n\n>  Question B: For an input with a single editing intent (say fluency), could different experts (corresponding to different intents) be activated at different sparse BERT layers? If yes, what does this mean? If we assume single intent input, were any model variations tried to enforce that the same expert is activated at all layers?\n\nA: Different experts will not be activated at different sparse BERT layers for an input with a single editing intent. The routing method we use in G-SPEED enforces the activation of the same expert at all layers. In the next version, we will revise the Section 5 for more clarity.\n\n> Question C: In Table 3, the performance of G-SPEED on clarity seems to improve without pre-training or without fine-tuning. Discussion on this could be added. \n\nA: Thanks for your suggestions. The result without pre-training is only 0.1 higher than the G-SPEED result in SARI, and GLEU is reduced by 0.3, so the major problem is that the effect of fine-tuning is not good for the pre-training model. We think the quality of the fine-tuning dataset is responsible for this problem. We use the same data to fine-tune LLaMa-7B and the SARI score of LLaMa-7B also drops from 34.1(before training ) to 33.3(after training), which confirms our hypothesis. We will supplement these details in the next version.\n\n\n\nReferences:\n\n[1] Malmi E, Krause S, Rothe S, et al. Encode, Tag, Realize: High-Precision Text Editing[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019: 5054-5065.\n\n[2] Mallinson J, Severyn A, Malmi E, et al. FELIX: Flexible Text Editing Through Tagging and Insertion[C]//Findings of the Association for Computational Linguistics: EMNLP 2020. 2020: 1244-1255.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe unsupervised data clustering algorithm is shown to be effective in training editing models.\n2.\tThe sparse BERT-based editing model when either pre-trained or fine-tuned is shown to have competitive performance with larger LMs used in a prompting-based manner.", "reasons_to_reject": "1.\tUnsure of contamination between data collected in an unsupervised manner and the test set of the evaluation tasks. \n2.\tComparisons could be performed with other editing models like Levenshtein Transformer, etc.", "questions_to_authors": "A.\tCould you check if there is any contamination between data collected by your clustering algorithm and the test sets of the evaluation tasks?\n\nB.\tFor an input with a single editing intent (say fluency), could different experts (corresponding to different intents) be activated at different sparse BERT layers? If yes, what does this mean? If we assume single intent input, were any model variations tried to enforce that the same expert is activated at all layers?\n\nC.\tIn Table 3, the performance of G-SPEED on clarity seems to improve without pre-training or without fine-tuning. Discussion on this could be added.", "paper_topic_and_main_contributions": "This paper presents an unsupervised data clustering algorithm for data collection to train editing models, and a sparse BERT-based editing model.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "o39nnoaI1f", "reviewer": "Reviewer_VHDk", "summary": "", "questions": "1. Why does Table 6 evaluate the model's performance only on the sentence fusion task?\n2. In Table 3, the performance of w/o ADDI. outperforms G-SPEED on ITR-F and ITR-L. Please explain the reason for this discrepancy.\n\nQuestion for Rebuttal\n\nI've carefully reviewed the authors' thoughtful responses and the additional experimental results have certainly addressed part of my concerns. However, I still have some concerns regarding the contributions of this paper:\n\n(1) The authors claim that prior work is specifically designed for a particular type of editing task. However, PEER is proposed to address a variety of editing tasks and has leveraged data from Wikipedia to effectively mitigate the issue of data scarcity in text editing. Furthermore, the results in Table 3 demonstrate that PEER still performs competitively in a zero-shot setting across multiple editing tasks.\n\n(2) Additionally, as mentioned in Lines 061-067, some existing work based on LLMs has already achieved general text editing capabilities. The authors emphasize the importance of efficiency in text editing, particularly in interactive scenarios. However, the main experiments in the paper focus only on single-turn editing and do not provide any inference time comparisons between different models.  Moreover, these LLM-based methods can edit texts based on user feedback to cover a broader range of editing needs, making them more practical. Consequently, I think that the experimental results do not adequately support the authors' motivation.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the informative reviews and inspiring comments on our paper! We hope the following responses can address your questions well.\n\n> Reject Reason 1: Insufficient experimental comparisons. a) Most of the main baselines are evaluated in zero-shot setting, while G-SPEED is further fine-tuned with supervised task data, leading to an unfair comparison. It is suggested to include more supervised baseline models, such as PEER with further fine-tuning. b) The evaluation metrics may not be fully appropriate. Consider adding human evaluation or automated evaluation with ChatGPT/GPT4.\nA: Thanks for your suggestions.\n\na) In Table 3, we have reported the performance of G-SPEED without fine-tuning(\"w/o ADDI\"). We can observe that G-SPEED can still outperform all the baselines without further fine-tuning. Since the datasets and models of PEER are not open source so far, we choose to use downstream supervised data to fine-tune LLaMa-7B with LoRA to further confirm the effectiveness of G-SPEED. The results are as follows:\n\n|          | JFL      | ITR-F    | ITR-L    | ITR-O    | STS      | TRK  | AST      | WNC      |\n| -------- | -------- | -------- | -------- | -------- | -------- | ---- | -------- | -------- |\n| LLaMa-7B | 52.0     | 36.3     | 33.9     | 36.8     | **36.3** | 38.5 | 37.1     | 54.1     |\n| G-SPEED  | **54.2** | **42.2** | **34.0** | **41.0** | 36.0     | 38.5 | **40.6** | **60.2** |\n\nThe results show that G-SPEED performs better than LLaMa-7B when both conduct additional fine-tuning. \n\nb) We conduct human evaluation between G-SPEED and ChatGPT. The following are the results of human evaluation:\n\n| ChatGPT win | Tie   | G-SPEED win |\n| ----------- | ----- | ----------- |\n| 28.3%       | 40.3% | 31.4%       |\n\nThe results of the human evaluation show that G-SPEED outperforms ChatGPT. The next version will report these results and more human evaluation details.\n\nWe will report these results in the next version.\n\n> Reject Reason 2: The contribution needs further elaboration.\n\nA: Due to the data scarcity problem of editing tasks, there is a line of existing works exploring utilizing the revision history of Wikipedia passages. All these methods are either introduced for specific editing tasks or designed for LLMs. In our preliminary study, both of these methods fail to achieve the goal of achieving an efficient general editing model. Therefore, we think the efficient general editing model has a much higher requirement for training data quality than the specific editing model and general editing LLMs. Due to the challenge of obtaining an efficient general editing model, we propose a novel unsupervised editing data clustering method, which can achieve higher data quality than existing methods.\n\n\n\n> Reject Reason 3: Text editing often involves multi-turn interactions, but the proposed model is only tested for single-turn editing requests. Evaluating text editing performance under multi-turn settings, possibly using human evaluation, would make the experimental results more convincing.\n\nA: We tested the interactive modification in IteraTeR_human_doc and compared ChatGPT. The results are as follows:\n\n|         | SARI     | GLEU     |\n| ------- | -------- | -------- |\n| ChatGPT | 36.4     | 30.9     |\n| G-SPEED | **36.6** | **48.3** |\n\nThe results show that the two models have similar modification capabilities, but G-SPEED retains the source text more.\n\n> Question 1: Why does Table 6 evaluate the model's performance only on the sentence fusion task?\n\nA: We supplement the results on WikiSplit and GYAFC datasets. We randomly sample 500 data points for training on LLaMa-7B and G-SPEED respectively. GYAFC-E&M and GYAFC-F&R are the subsets of the GYAFC corpus in the Entertainment & Music and Family & Relationships domains. The evaluation metrics of WikiSplit and GYAFC are SARI and BLEU respectively. The following results still support our original conclusion:\n\n|          | WikiSplit | GYAFC-E&M | GYAFC-F&R |\n| -------- | --------- | --------- | --------- |\n| LLaMa-7B | 50.1      | 44.2      | 46.0      |\n| G-SPEED  | **54.4**  | **46.2**  | **46.6**  |\n\n> Question 2: In Table 3, the performance of w/o ADDI. outperforms G-SPEED on ITR-F and ITR-L. Please explain the reason for this discrepancy.\n\nA: In the Fluency task, the model has a significant improvement on the JFL test set after additional fine-tuning, while the performance on the ITR-F test set is worse. Because the data volume of ITR-F is too small (88), some slight changes will lead to large differences in results. In actual observation, we did not find obvious quality degradation. In ITR-L, we think the quality of the fine-tuning dataset is responsible for this problem. We use the same data to fine-tune LLaMa-7B and the SARI score of LLaMa-7B also drops from 34.1(before training ) to 33.3(after training), which confirms our hypothesis.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a model capable of handling various text editing tasks with superior performance on multiple tasks.\n2. The unsupervised edit data clustering algorithm can effectively leverage text editing data from real-world platforms.", "reasons_to_reject": "1. Insufficient experimental comparisons.\na) Most of the main baselines are evaluated in zero-shot setting, while G-SPEED is further fine-tuned with supervised task data, leading to an unfair comparison. It is suggested to include more supervised baseline models, such as PEER with further fine-tuning.\nb) The evaluation metrics may not be fully appropriate. Consider adding human evaluation or automated evaluation with ChatGPT/GPT4.\nc) In Table 6, the comparison involves G-SPEED and other SeqSeq models trained with only a small amount of supervised data. However, the selected models in Table 6 are not pre-trained with massive edit data, and it is essential to compare G-SPEED with the supervised models (T0, PEER).\n2. The contribution needs further elaboration. While the paper claims that they propose an unsupervised edit data clustering method to mitigate data scarcity problem (Lines 021-023), previous work (PEER) already utilize edit data from Wikipedia to train a general text editing model. The advantages of the proposed algorithm over prior work are not clearly articulated.\n3. Text editing often involves multi-turn interactions, but the proposed model is only tested for single-turn editing requests. Evaluating text editing performance under multi-turn settings, possibly using human evaluation, would make the experimental results more convincing.", "questions_to_authors": "1. Why does Table 6 evaluate the model's performance only on the sentence fusion task?\n2. In Table 3, the performance of w/o ADDI. outperforms G-SPEED on ITR-F and ITR-L. Please explain the reason for this discrepancy.\n\nQuestion for Rebuttal\n\nI've carefully reviewed the authors' thoughtful responses and the additional experimental results have certainly addressed part of my concerns. However, I still have some concerns regarding the contributions of this paper:\n\n(1) The authors claim that prior work is specifically designed for a particular type of editing task. However, PEER is proposed to address a variety of editing tasks and has leveraged data from Wikipedia to effectively mitigate the issue of data scarcity in text editing. Furthermore, the results in Table 3 demonstrate that PEER still performs competitively in a zero-shot setting across multiple editing tasks.\n\n(2) Additionally, as mentioned in Lines 061-067, some existing work based on LLMs has already achieved general text editing capabilities. The authors emphasize the importance of efficiency in text editing, particularly in interactive scenarios. However, the main experiments in the paper focus only on single-turn editing and do not provide any inference time comparisons between different models.  Moreover, these LLM-based methods can edit texts based on user feedback to cover a broader range of editing needs, making them more practical. Consequently, I think that the experimental results do not adequately support the authors' motivation.", "paper_topic_and_main_contributions": "The article proposes a general text editing model called G-Speed. The authors pretrain the model using text editing data collected from Wikipedia, arXiv, and Wikinews, to alleviate data sparsity issues. To enable the model to handle multiple editing tasks simultaneously, the authors employ a mixture-of-experts architecture for training. Results on six datasets demonstrate that G-SPEED achieves superior performance.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In Section \"Related work (General Text Editing),\" there should be  a space between the main text and the citations.\n2. It is suggested to provide descriptions of supervised SOTA in the appendix, such as the model size and whether additional pre-training is used."}}]

Author rebuttal:

Meta-review: The paper "G-SPEED: General Text Editing with Sparse BERT and Mixture-of-Experts" presents a lightweight text editing model called G-SPEED, which is composed of a tagging model and a generation model. The paper's main contributions include the proposal of G-SPEED, the unsupervised data clustering method, and the mixture-of-experts strategy for the tagging and generation models. The strengths of the paper include the well-written and easy-to-follow presentation of the proposed methods, the usefulness of the clustering method and the resulting editing data for advancing the field of text editing, and the impressive performance of the lightweight G-SPEED model. The paper also provides experimental results that demonstrate the effectiveness of G-SPEED, achieving state-of-the-art performance on some editing tasks. However, the reviewers also raise a number of concerns or suggestions to improve this paper. Overall, the technical novelty of the proposed techniques can be discussed further and enhanced. The paper does not provide a comprehensive analysis of the differences with prior work in the Editing Model subsection in Related Work. Secondly, the experimental setting and results analysis can be strengthened. Thirdly, the paper does not provide a comprehensive discussion of the pros and cons of LLM-based generative editing approaches, such as ChatGPT, compared to G-SPEED. Although G-SPEED is lightweight and powerful, it has limitations in generalizing to unseen editing tasks, whereas LLM-based approaches do not have such limitations. Finally, the paper should discuss the potential contamination between the data collected in an unsupervised manner and the test sets of the evaluation tasks, which raises concerns about the validity of the results.
Decision: Accept-Findings